AI Incident Management

What counts as an AI incident

An AI incident is not the same thing as a bad model output. A model producing a wrong answer that a human reads and discards is a quality problem. An AI incident is when an AI system takes, or attempts, an action with real consequence outside of what was authorized or intended: a write to a system it should not have touched, an action taken under stale or revoked permissions, an agent that executes the same irreversible action twice because of a retry, or an approval workflow that was bypassed or spoofed.

The distinction matters operationally. Treating every bad output as an incident buries real incidents in noise. Treating unauthorized execution as "just a bug to fix in the next sprint" is how a contained problem turns into a material one.

Severity has to account for what already happened, not just what's visible

Traditional incident severity asks: how many users are affected, and for how long? AI incident severity has to ask an additional question: what could the system have already done before anyone noticed? An agent with write access that ran unauthorized for eleven minutes did not cause "an eleven-minute outage" — it took some number of discrete, possibly irreversible actions during that window, and until each of those actions is enumerated, the actual blast radius is unknown. Severity assessment for an AI incident is incomplete until the action log has been reconstructed, not just the uptime graph.

Containment: stop it from doing anything else, first

The first job in any incident is stopping ongoing harm, before root-causing it. For an AI incident that specifically means:

• Revoke the credentials or authority the system was using — not just disable the account, but confirm every downstream system that trusted that authority now rejects it.
• Interrupt in-flight executions rather than waiting for them to complete naturally, if the actions in flight are the ones in question.
• Confirm containment actually took effect — a revoked credential that a caching layer still honors for another ninety seconds is not contained, it just looks contained.

Only after the bleeding stops does root-cause investigation start. Doing it in the other order, investigating while the system is still able to act, is how incidents get worse mid-investigation.

Preserving evidence before it decays

AI systems tend to have short-lived evidence: context windows that get overwritten, cache entries that expire, policy versions that get replaced by the next deploy. If evidence of what a system was authorized to do, what it was told, and what it decided is not captured at the moment of the incident, it may not exist to capture an hour later. Post-incident review needs, at minimum: which policy version was live at the time, what the system's actual permissions were, what actions it attempted versus what actually executed, and what evidence exists that any human approval involved was for the specific action taken, not a summary of it.

Stakeholder communication under uncertainty

Early in an AI incident, the honest answer to "what happened" is often "we don't fully know yet, and here's what we're doing to find out." That is a harder message to deliver than a confident wrong one, and it is the correct one until the action log has actually been reconstructed. Committing to a root cause before the evidence supports it, to make an update sound more finished, is how organizations end up issuing a correction on top of an incident.

Post-incident: policy and model version capture

A useful post-mortem for an AI incident has to answer a question traditional software incidents rarely need to: was this caused by the system doing what it was told, incorrectly, or by the system doing something it was never authorized to do at all? Those have completely different fixes. The first is a model or prompt problem. The second is a governance and enforcement failure, and no amount of prompt tuning will fix it. Getting this distinction wrong in the post-mortem means fixing the wrong layer.

Unresolved limitations, stated plainly

Incident Management practice for autonomous AI systems is young. Most organizations do not yet have tooling that can reconstruct an agent's full decision and action history after the fact with confidence, and most AI infrastructure was not built with tamper-evident, decision-time evidence generation as a requirement. Anyone claiming a fully solved answer here is overselling. The honest state of the field is: the discipline from traditional Incident Management transfers, the tooling to fully practice it for autonomous AI systems is still being built.