The Runbook Said Restart. The Incident History Explains Why Nobody Did

02:14. The alert fires. The runbook's first step is clear: restart the service.

The on-call engineer doesn't. A dependency upstream looks unhealthy, a restart during a failover needs sign-off, and the symptom on the dashboard isn't the real cause. Forty minutes later the service is back. It's a hypothetical night, but most on-call engineers have lived some version of it. And the runbook still says restart.

That gap between procedure and practice is why incident data for AI evaluation is getting attention. A good incident record connects what responders observed, what context they had, what they did and how it ended. That is precisely the material you need to test whether an AI agent follows evidence or just follows instructions.

It's also some of the most sensitive data your company holds. Handle it like it is.

THE SHORT VERSION

• Incident histories show the reasoning between alert and recovery that a runbook never captures.
• To test an AI agent fairly, freeze the timeline: the postmortem can't leak into what the agent sees at minute five.
• Logs, screenshots and pasted commands carry secrets. Scope and exclusions come before any extraction.

What can an incident archive actually test?

Be specific and modest. Hypothetical evaluation tasks might ask an agent to:

• Identify which evidence is missing before acting
• Choose between escalating and running an approved diagnostic
• Recognize when the right move is to stop and page a human

Historical incidents don't validate unrestricted autonomous production access, and nobody should claim they do. State which actions the evaluation permits and which require a human. Score a correct decision to stop differently from an agent that simply couldn't proceed.

The tau-bench research is a useful reference here: it evaluates policy-governed agent interactions by checking the system state the agent leaves behind, not how convincing its replies sound. Ops teams need to adapt that idea to their own services and outcome definitions.

How do you stop the postmortem leaking into the test?

Incident records get better after recovery. The retrospective explains the root cause far more clearly than anything visible during the event. Feed that to an agent at the start of an evaluation and you're not testing diagnosis anymore. You're testing reading comprehension.

Keep a strict timeline. Separate what was knowable at each decision point from what was learned afterwards. Where chat logs or monitoring context are missing, say what the case can't establish rather than reconstructing a plausible but invented sequence.

If the postmortem is in the prompt, you're not testing diagnosis. You're testing reading.

Hypothetical incident timeline. To evaluate an agent fairly, later knowledge such as the postmortem stays out of earlier decision points.

What has to stay out of an incident dataset?

More than the ticket summary suggests. The risky material usually lives around it:

• Credentials and tokens pasted into command output
• Internal hostnames, IPs and links that map your infrastructure
• Screenshots and attachments with customer data
• Rare events that identify a customer or system on their own

Define scope and exclusions before extraction, not after. Preparation should document every change while keeping enough non-sensitive context to make the task interpretable. De-identification is one control, not a guarantee: infrastructure details and unusual events can stay identifying or commercially sensitive after names are gone.

How should the prepared dataset be versioned?

Version the prepared collection with a manifest recording source systems, date ranges, selection rules, transformations and outcome labels. Choose group and time boundaries to match the test: related incidents should not leak shared context across development and final evaluation. Keep the final test out of iterative prompt tuning and retrieval examples.

Have an experienced responder check what each label means. "Closed" may reflect a process decision, not verified recovery. Good documentation makes the limitations easy to challenge, especially where a case hinged on judgment the archive never captured.

Scope, timeline discipline and exclusions come before any extraction of operational records.

Can incident data be licensed?

Potentially, through the company's authorized decision-maker. SourceX licenses approved operational records for AI training and evaluation. Qualification comes first; a signed Master then starts the inventory assessment. A signed Dataset Deal Confirmation fixes scope, payout, rights, processing route and deadlines. Extraction and delivery require both documents and SourceX's go-ahead. Admin access never substitutes for that authorization.

SourceX looks for companies that peaked at 50 or more full-time employees and have several years of operating history. Its advertised 'up to $4 million' applies to the largest enterprise datasets; it is not a standard figure for an incident archive, and acceptance is never assured. Utility and rights get assessed before anyone knows the commercial result.

Where should an SRE team start?

Pick one well-documented historical incident and trace it from first alert to confirmed recovery, inside an authorized internal review. Mark which context made each decision understandable and which material must stay excluded. Then ask whether an outsider could follow the timeline without the postmortem.

You'll learn whether you have an evaluation candidate, and you'll see, in your own data, everything the runbook never told you.

One traced incident shows both the evaluation potential and the sensitive material to exclude.

Keep the incident review internal. If the authorized business owner sees a licensing opportunity, SourceX's initial company fit check uses company facts and no record uploads. Approve the scope and processing route before any extraction is authorized.

About the founder

Noah Loul is a four-time founder and the founder and CEO of SourceX, which licenses business data directly from companies for AI training and evaluation. He previously founded B2B Rocket, acquired by Blackpearl Group. His work focuses on building permission-based data infrastructure for AI.