A failed-run evidence bundle ties an execution attempt to its code revision, inputs, event times and downstream effects. A rerun creates another observation rather than rewriting that history.
Substantively revised October 21, 2026. Author interpretation, documented cases and hypothetical examples are distinguished below.
The maintainer is unavailable, the overnight job is red, and the obvious button says Rerun. Pressing it may restore the result. It may also repeat a write that already succeeded before a later step failed. The team needs more than the original maintainer's intuition: it needs enough evidence to distinguish what happened, what remains unknown and which next action is safe.
Direct answer
Preserve the failed attempt's identifier, code revision, relevant input references, timestamps, step evidence and downstream effects before choosing a rerun or fix. Correlate records without treating temporal proximity as proof of causation. Separate the original attempt from later retries. A scheduler status, an empty error search or a successful rerun cannot independently explain the original failure or prove that no partial effect occurred.
This article proposes an Attempt Evidence Packet for handing an investigation to another authorized maintainer. Its unit is one execution attempt and its observable consequences. It differs from a general work-resumption note: the decision concerns a failed run, possible partial effects and the evidence needed to choose the next diagnostic or recovery action.
Key takeaways
Identify the attempt before interpreting its logs. A job name can refer to many executions and retries. Record the relevant revision and environment so a colleague does not investigate today's code while assuming it produced yesterday's failure.
Distinguish event time, collection time and display time. Logs from different systems can arrive late or use different clocks. Correlation identifiers help connect records where available, but their presence does not prove complete instrumentation or a single causal explanation.
Inspect downstream state before repeating an operation with side effects. A failure after a successful write can require a different response from a failure before any write. Preserve uncertainty when the available evidence cannot establish the boundary, and follow the system's supported recovery procedure rather than guessing from a red status alone.
Observation: the original maintainer carries an unwritten event model
An experienced maintainer often knows which warning is normal, which step writes data and which retry is harmless. Their quick diagnosis can make the run record appear sufficient. In reality, they are adding an event model from memory: how the code, scheduler and destination behave together.
When another person takes over, the same logs can become an undifferentiated wall of text. They may search for error, find one message and assume it is the cause. Or they may see a later successful attempt and conclude the incident is explained. Both shortcuts can miss partial effects, mismatched revisions or an unrelated warning that happened near the failure.
The pattern is not a shortage of log volume alone. More messages can make the investigation harder if they cannot be tied to an attempt, operation or resource. The useful handover connects a small set of evidence to an explicit account of what is known and what test would resolve the next uncertainty.
Definitions and root cause
A run is a platform's execution record for a workflow or job. An attempt distinguishes an initial execution from a retry or rerun according to the platform's model. A revision identifies the code or configuration used. An event timestamp describes when an event occurred according to its source, while an observed or received timestamp describes when another system encountered the record.
A correlation identifier relates records under an instrumentation convention. A downstream effect is a change outside the job's local execution, such as a created object, sent message or updated record. A hypothesis is a proposed explanation that remains distinguishable from observed evidence. A diagnostic action should be chosen partly by what uncertainty it can resolve and what additional effects it may create.
The root cause of the owner bottleneck is leaving the event model inside one person's memory. The platform stores output, but the handover omits how the output relates to code, inputs and external effects. A packet cannot manufacture missing evidence. It can make the gaps visible and prevent a successor from treating an unsupported story as a completed diagnosis.
Framework: the Attempt Evidence Packet
The Attempt Evidence Packet is an author-proposed method with six sections: Identity, Sequence, Context, Effects, Hypotheses and Next Test. It is not a logging standard or an automated root-cause guarantee. Its purpose is to give another maintainer an inspectable basis for deciding what to do next.
Identity records the platform's run and attempt references, relevant revision, environment and observation time. Distinguish identifiers that are globally unique from those meaningful only inside one system. Preserve links or restricted evidence references before retention or deletion removes access. A screenshot of a red job without an attempt reference is difficult to reconcile with later records.
Sequence reconstructs relevant events with their source and timestamp meaning. Keep original times and time zones where useful, and note known clock or arrival uncertainty. Do not force every record into a falsely precise order. If two events cannot be reliably ordered, say so and identify what additional evidence might resolve the relationship.
Context identifies the inputs and configuration needed to interpret the attempt. Use versioned or stable references where available, with appropriate handling for sensitive data. Record whether a resource was changed after the failure. The current input or configuration may differ from what the original attempt actually consumed.
Effects identifies which external operations were attempted and what state is independently observable. Separate intended, acknowledged and confirmed effects. A local log saying request sent is not the same as a destination record proving the intended update. Where evidence is absent, record unknown rather than silently converting no log into no effect.
Hypotheses lists plausible explanations with supporting and conflicting evidence. Keep the list focused on decisions, not every imaginable failure. A useful hypothesis predicts an observable distinction: a permission denial should have different evidence from malformed input or an unavailable destination. Avoid upgrading the first plausible message into the root cause merely because it is easy to quote.
Next Test states the smallest authorized diagnostic action, the uncertainty it addresses and its possible side effects. It may be a read-only inspection, a safe reproduction or a controlled retry under the system's supported procedure. Define what result would change the decision. A rerun without a question can create more events while leaving the original uncertainty intact.
Three documented sources of run evidence
GitHub documents workflow run logs, including access to job and step output and options for searching or downloading logs. These facilities support preserving attempt evidence, but availability is not permanent and a rerun is a new observation. Use the relevant run record and applicable retention arrangement instead of assuming today's visible logs will remain available indefinitely. [S1]
OpenTelemetry's logs data model distinguishes Timestamp from ObservedTimestamp and includes optional trace and span identifiers. That distinction supports recording whether a time describes the source event or its observation. Optional correlation fields may be absent, and a standardized record format does not establish that every important event was instrumented or that a correlation relationship proves causality. [S2]
Google Cloud's LogEntry reference includes source and receive timestamps, resource context, operation information and trace-related fields. These can help connect evidence across systems when populated appropriately. They should not be described as a universal business-level exactly-once guarantee or a complete causal history of every external operation. Interpret the actual fields and instrumentation in use. [S3]
These documented mechanisms explain what evidence may be available. They do not claim that any one logging system automatically diagnoses an incident. The packet must preserve the difference between a provider's field definition, an application's instrumentation and the actual observations from the failed attempt.
Case study: the report job that failed after writing
This case is hypothetical. A nightly job reads a source, writes a report object and posts a completion message. The platform marks the attempt failed. The original maintainer is unavailable. The successor initially considers rerunning all 3 operations, but the write may create an additional object rather than safely replace the first one. These invented behaviors define the case and are not defaults for every storage or messaging service.
The successor records the failed attempt and its code revision. Step evidence shows the read completed, a write request received a success response, and the notification step reported an authorization error. That sequence supports a partial-effect hypothesis, but the operator still checks the destination. The report object exists with the expected attempt marker and content reference.
The packet distinguishes three facts: the scheduler reported failure, the destination contains the report object, and the notification did not produce its intended confirmed result. The operator does not label the whole job unprocessed or assume the error proves the report is absent. They also do not treat the report object's existence as proof that every notification-related side effect is impossible.
The next authorized test inspects the notification connection and its relevant permissions without rewriting the report. The team follows its supported repair procedure and decides how to complete the missing notification. If the application lacks a safe partial-retry path, that limitation becomes part of the recovery decision rather than a reason to press the global rerun button blindly.
For this illustrative investigation, 2 timestamps arrive out of order because one logging system receives a delayed record. The packet preserves source and receive times and avoids inventing a precise cross-system order where the evidence does not support it. The final handover explains the observed partial result and remaining uncertainty, allowing another maintainer to continue without reconstructing the original person's mental model.
Failure example: a green rerun rewrites the story
A hypothetical team reruns a failed job and gets a green result. They close the incident as transient without preserving the original attempt's evidence. Later, duplicate output appears. The successful rerun established that the later attempt completed under its conditions; it did not prove the first attempt had no effect or explain why it failed.
Correct the process by preserving attempt identity and inspecting external effects before choosing a retry. This does not mean every failure requires a lengthy investigation. If the system has a documented, appropriate retry mechanism and the relevant evidence supports using it, a short packet can be enough. The important distinction is between a justified retry and a button press that substitutes for understanding.
Building a packet another maintainer can use
Start with links and a short factual sequence, not a copied archive of every log line. Include the evidence needed for the next decision and references to the wider record. Preserve access before retention removes it, following the organization's approved handling and retention rules. Logs can contain personal data or secrets; broad publication is not a prerequisite for a useful handover.
Label each statement as observed, inferred or unknown where the distinction matters. The destination returned a success response is an observation. The write must have committed exactly once may be an inference that the response alone cannot support. A recipient should be able to inspect the boundary without reverse-engineering the author's confidence from their tone.
Keep attempts separate. A rerun may use a changed revision, updated secret, different input or repaired destination. Record those differences before comparing outcomes. A contrast between failure and success is useful only when the investigator understands which conditions changed and which remain uncertain.
Choose the next test by information value and side effects. Reading a destination record may answer whether a write occurred more safely than running the writer again. A safe reproduction can test a parser hypothesis without touching production. The packet should explain why the chosen test is relevant, not merely list actions someone happened to try.
Tradeoff: evidence collection versus recovery speed
An urgent incident may require restoring service before completing a detailed explanation. Preserve the most perishable and decision-relevant evidence first, then follow the authorized recovery procedure. The packet is meant to support action, not delay every response until a perfect causal account exists.
More logging can improve visibility while increasing cost, noise and sensitive-data exposure. Instrument the boundaries that matter for decisions: attempt identity, protected operations and external results. A structured minimal record can be more useful than verbose messages without correlation. The right level depends on the system and its failure consequences, not a universal line count.
Counterargument: the scheduler already tells us what failed
A scheduler can accurately identify a failed step or exit condition. That is valuable evidence and often the correct starting point. It may not establish the business outcome, the state of an external destination or the cause of an error returned by another service. The packet extends the investigation only where those distinctions affect the next action.
If the scheduler's evidence and the system's documented retry behavior are sufficient, keep the record short. If the job can partially write before failing, a red status alone cannot decide whether a full rerun is appropriate. The justification for more evidence is the unresolved decision, not a belief that platform summaries are inherently untrustworthy.
Common mistakes
Searching for one error string can miss failures reported through status fields, missing expected events or downstream state. Use the actual sequence and operation boundaries. Conversely, a warning found near the failure is not automatically its cause; record the evidence connecting it to the observed outcome.
Treating timestamps from several systems as one perfectly synchronized clock can create a false narrative. Preserve their meanings and known uncertainty. Correlation identifiers can help, but optional or inconsistently propagated fields should not be presented as a complete trace when the instrumentation does not support that claim.
Copying raw logs into a public issue can expose sensitive material without improving diagnosis. Use authorized storage and appropriately redacted evidence references. The successor needs access to the relevant facts, not unrestricted distribution of every payload or credential-shaped value emitted by a program.
Action plan
Choose one failed attempt and record its identifiers, revision, environment and input references. Preserve the relevant job evidence before it disappears under normal retention or deletion. Write a short sequence that distinguishes event times from observed or received times and labels unresolved ordering.
Inspect the downstream operations that could have partially completed. Separate confirmed effects from requests or acknowledgments and identify uncertainty that affects retry safety. List the leading explanation with supporting and conflicting evidence rather than declaring a cause from the first matching message.
Select the smallest authorized next test and state what its outcome would change. Keep any rerun as a separate attempt with its own conditions. Close the handover with the observed result, remaining uncertainty and follow-up owner so another maintainer can continue without relying on an unavailable person's memory.
FAQ
Does a failed job mean no output was produced?
No universal inference supports that. A job can fail after a partial external effect. Inspect the operation sequence and destination state relevant to the system before deciding whether a retry should repeat the whole workflow.
Does a successful rerun explain the first failure?
It establishes a later observation under its own conditions. It may help compare hypotheses, but it does not erase the earlier attempt or prove that the original failure had no partial effect. Preserve both records separately.
Which revision should the packet identify?
The code and configuration relevant to the failed attempt, not merely the current branch state. Record known later changes when comparing attempts. Otherwise a maintainer can investigate code that never produced the evidence they are reading.
Are event time and receive time interchangeable?
No. They describe different points in the record's lifecycle. Delays and clock differences can affect ordering. Preserve the source definitions and avoid constructing a precise cross-system sequence that the available evidence cannot support.
Do trace identifiers prove causation?
They help correlate records under an instrumentation convention. They do not independently establish complete coverage or a causal explanation. Inspect what the application actually records and distinguish correlation from the hypothesis being tested.
Should the packet include every log line?
Usually a focused sequence and evidence references are more usable. Preserve the wider record appropriately, especially when retention is limited. Include what the next decision needs while respecting access controls and sensitive-data handling.
Is an empty error search proof that nothing failed?
No. Failures may appear as statuses, missing expected events or downstream discrepancies. Search is one inspection method. The packet should connect actual operation boundaries and outcomes rather than rely on a single keyword's absence.
Can we rerun to discover whether the first write happened?
That can create another write and more uncertainty. Prefer supported read-only inspection where it answers the question. Use retries according to the system's documented behavior and authorized recovery process, accounting for possible partial effects.
What if logs have already expired?
Record the evidence gap and inspect other authorized sources. Do not invent the missing sequence from memory. The incident may also reveal a retention or instrumentation improvement, but that future change cannot recreate evidence that was never preserved.
Does a success response prove exactly-once business completion?
Not by itself. Interpret the service's actual semantics and reconcile relevant destination state. Transport acknowledgment, operation success and the intended business outcome can be different claims requiring different evidence.
What belongs in a useful hypothesis?
A proposed explanation, evidence for and against it, and an observable distinction a next test can examine. Avoid a long list of speculative causes that does not help choose an action or clarify the current uncertainty.
How should secrets in logs be handled?
Follow the organization's authorized handling and remediation procedures. Do not copy usable secrets into the packet or a public issue. Provide appropriately restricted or redacted evidence references so investigation does not distribute additional authority.
Can an urgent recovery happen before root cause is known?
Yes, under the appropriate operational process. Preserve the most decision-relevant and perishable evidence, state uncertainty and follow an authorized recovery path. The packet supports informed action without requiring a complete explanation before every response.
How is this different from resuming a paused investigation?
This packet reconstructs a specific failed execution and its possible side effects before retry or repair. A general resumption note preserves work context. Here attempt identity, partial effects and evidence-based next tests are the central decision.
When is the handover complete?
When another authorized maintainer can identify the attempt, inspect the relevant evidence and explain the next justified action and its limits. That does not require pretending the cause is known when the packet accurately records an unresolved hypothesis.
Recommended books
Site Reliability Engineering, edited by Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy, provides directly relevant context for troubleshooting, monitoring and incident response. Use it to develop disciplined hypotheses and observations, while following the actual system's supported recovery and retry behavior.
Thinking in Systems by Donella H. Meadows helps examine the relationship between a local job status and effects elsewhere in the system. Its value is the habit of following boundaries and feedback. It does not establish logging field semantics or guarantee a causal interpretation of correlated events.
The Checklist Manifesto by Atul Gawande is useful for a brief pause before repeating a consequential operation. The key question here is whether the original attempt may already have changed something. A checklist helps expose that omission but cannot replace inspection of the actual run and destination.
Further exploration
Review a previously resolved failure and ask whether its saved record distinguishes the original attempt from the successful recovery. Identify one missing evidence link that would have changed the next action, such as a destination result or revision reference. Improve that boundary's instrumentation or retention where justified, rather than adding undirected log volume that future maintainers still cannot interpret.
Primary evidence sources
Related concepts: systems thinking, wealth structure, and the survival loop. Read the editorial policy and citation guide.