AI Incident Response: A Practical Guide for Security Teams

Executive summary

AI incident response is the discipline of detecting, containing, investigating, and recovering from security incidents that involve AI systems — the models, prompts, agents, retrieval pipelines, and data that power modern AI features. The core lifecycle you already know still applies, but the threats, the signals, and the evidence are unfamiliar enough that a team running only its traditional playbook will be slow, imprecise, or wrong at exactly the moment it can least afford to be.

This guide is written for security leaders and responders who understand incident response but are now responsible for AI systems they did not build the runbooks for. We start with why AI incidents genuinely differ from the breaches you have handled before. We then walk phase by phase through the classic lifecycle described in NIST Special Publication 800-61 — preparation; detection and analysis; containment, eradication, and recovery; and post-incident activity — and show what each phase looks like when the affected asset is a large language model or an autonomous agent rather than a server or an endpoint. Along the way we cover the preparation work that pays off most, the detection signals that actually indicate an AI incident, the containment levers you can pull in minutes, the peculiar challenge of doing forensics on a non-deterministic system, and the communication and regulatory considerations that determine whether an incident becomes a crisis.

Why AI incidents are different

Every mature security team has a mental model of what an incident looks like: an alert fires, an indicator of compromise appears, a machine is isolated, an artifact is pulled apart in a sandbox, and a root cause is written up. AI incidents quietly violate almost every step of that model. The reason is that the attack surface has moved from infrastructure to behavior. The system is not necessarily compromised in the traditional sense — no malware is dropped, no credential is stolen, no CVE is exploited — and yet it is doing something it should not. Before mapping the lifecycle, it is worth being precise about the categories of AI incident you are preparing for, because each one breaks a different assumption.

Prompt-injection breaches

The defining vulnerability of the AI era is prompt injection: an attacker plants instructions inside content the model will read, and the model follows those instructions as if they came from the developer. When the malicious content arrives through a document, a web page, an email, or a database record that the system retrieves on its own, it is called indirect prompt injection, and it is especially dangerous because no human ever chose to trust the payload. A prompt-injection breach can turn a helpful assistant into an exfiltration channel, cause it to ignore its own guardrails, or trick it into misusing the tools it has been granted. For a deeper treatment of the mechanics, see our guide to prompt injection. What makes it an incident-response problem rather than only an engineering one is that the payload looks like ordinary language, leaves no conventional signature, and can be delivered through data the system was designed to consume.

Data leakage through model output

A conventional breach moves data out through a file transfer or a network connection you can watch. An AI system can leak the same data simply by saying it. A model with access to customer records, internal documents, or another user's session can be led — deliberately or accidentally — to reproduce that information in its response. Because the exfiltration path is the model's own legitimate output channel, traditional data-loss prevention that watches files and packets never sees it. This class of incident is subtle enough, and common enough, that we cover it on its own in AI data leakage. From a response standpoint, the hard part is scoping: once you know a model has been leaking regulated data, you have to determine what it had access to, which sessions were affected, and what actually left the building — none of which is answerable without detailed logs of prompts, retrieved context, and outputs.

Model manipulation and poisoning

Some AI incidents attack the model itself rather than a single conversation. Training-data poisoning corrupts a model during fine-tuning or continuous learning by feeding it malicious examples, sometimes planting a hidden trigger that changes behavior only when a specific phrase appears. Supply-chain attacks introduce a tampered model, dataset, or dependency from outside your organization. These incidents are insidious because the compromise is baked into the weights: the system behaves normally almost all the time, and the malicious behavior surfaces only under conditions the attacker controls. Detecting and eradicating this kind of incident is closer to a supply-chain investigation than a live intrusion — you are asking which model version, from which source, trained on which data, is trustworthy.

Jailbroken agents taking harmful actions

An AI agent is a model that has been granted tools and autonomy — the ability to call APIs, query databases, send messages, move money, or execute code. When an agent is jailbroken or hijacked, the consequences are no longer confined to text: it can take real, irreversible actions in your environment. A single prompt-injection payload delivered through a retrieved document can, in the worst case, cause an agent to email confidential data to an attacker or trigger a transaction. This is the category where AI incidents most resemble a classic intrusion in impact while behaving nothing like one in mechanism, and it is why containment options that can cut an agent's tool access matter so much.

Deepfake and synthetic-media fraud

Not every AI incident involves your own models. Generative AI has made convincing voice clones and video deepfakes cheap, which supercharges long-standing fraud vectors such as business email compromise and executive impersonation. An attacker who can produce a finance leader's voice on demand can move an urgent wire request past controls that assume a familiar voice is proof of identity. These incidents arrive through your people and processes rather than your infrastructure, and responding to them blends fraud investigation with security response — which means your AI incident-response plan has to reach beyond the security team into finance, HR, and communications.

Adapting the classic incident-response lifecycle

The good news is that you do not need to throw away your incident-response program and start over. The four-phase lifecycle formalized in NIST SP 800-61 — the U.S. National Institute of Standards and Technology's guide to computer security incident handling — remains the right backbone. Its phases are:

1. Preparation — building the capability to respond before anything happens.
2. Detection and analysis — recognizing that an incident is underway and understanding its scope.
3. Containment, eradication, and recovery — stopping the harm, removing the cause, and restoring normal operation.
4. Post-incident activity — learning from the event and hardening against a repeat.

What changes for AI is the content of each phase, not the structure. The assets are different, the signals are different, the containment levers are different, and the evidence is harder to preserve. The sections that follow take each phase in turn and translate it into concrete AI terms. If you already have a functioning incident-response program, you can read them as a gap analysis: for each phase, ask whether your current tooling, playbooks, and telemetry actually cover the AI layer, or whether they quietly assume a server-and-endpoint world that no longer describes where your risk lives. For a broader view of how AI reshapes what you have to defend, our overview of the AI attack surface pairs well with this guide.

Phase 1 — Preparation: the work that decides the outcome

In traditional incident response, preparation is important. In AI incident response, it is close to decisive, because so much of what you will need during an incident either exists at the moment the event happens or does not exist at all. You cannot retroactively log a model decision that was never recorded, and you cannot reconstruct an agent's tool calls that were never captured. Preparation for AI is largely a matter of ensuring that the raw material of a future investigation is being produced continuously, and that the people and playbooks are ready to use it.

Log every model decision

The single highest-leverage preparation step is comprehensive, structured logging of the AI layer. For every interaction, you want a durable record of the full prompt as assembled, any retrieved documents or context injected into it, the model and version identifier, the sampling parameters, the tools the model was permitted to call, the tool invocations it actually made with their arguments and results, and the final output returned. This is the AI equivalent of full packet capture, and without it, AI forensics is guesswork. The logs must be tamper-evident and access-controlled, because they will contain sensitive data and because their integrity is what makes them useful as evidence. Treat this telemetry as the foundation on which every later phase depends: detection reads from it, containment decisions are justified by it, and forensics reconstructs the event from it.

Write AI-specific playbooks

Your existing playbooks almost certainly do not have a page for "a retrieved document appears to have hijacked the assistant" or "an agent sent data to an unexpected recipient." Build playbooks for the concrete AI incident types: prompt-injection breach, data leakage through output, suspected model poisoning, jailbroken agent, and deepfake-enabled fraud. Each playbook should name the roles involved, the first containment action, the evidence to preserve immediately, the decision-makers for escalation, and the communication triggers. Crucially, an AI playbook must reach beyond the security team — a deepfake fraud playbook that does not involve finance and communications is incomplete, and an agent-misuse playbook that cannot reach the engineers who own the tool integrations will stall.

Arrange retainers and expert access in advance

AI incidents are specialized, and the moment you discover one is the worst time to start looking for people who understand model behavior, adversarial machine learning, and AI forensics. A standing incident-response retainer puts that expertise in place before you need it, with the relationships, access, and context already established. This is precisely the model behind our AI Incident Response service: on-call expert response for when an AI system is breached, manipulated, or leaking, available on a retainer so containment, root-cause analysis, and recovery help is already arranged rather than assembled under pressure. Whether you build this capability internally or engage a partner, the principle is the same — decide who responds, and how they get access, while things are calm.

Run tabletop exercises

A playbook that has never been rehearsed is a hypothesis. Tabletop exercises — walking a cross-functional group through a realistic AI incident scenario without touching production — surface the gaps that only appear under the pressure of a decision. Run a scenario in which an indirect prompt injection causes your customer-support agent to leak another customer's data, and watch where the exercise stalls: Does anyone know how to pull the relevant logs? Who has the authority to disable the agent's tools? When does legal get involved? When do you notify a regulator? The value is not the scenario itself but the discovery of every point where the plan is vague, and the muscle memory that turns a first-time response into a rehearsed one.

Instrument for containment before you need it

Preparation is also architectural. The containment options available to you during an incident are the ones you built in beforehand. An inline AI gateway that sits in the request path — inspecting prompts and responses and able to block, filter, or fail safe — is both a detection sensor and a containment lever. Feature flags that can disable a specific tool or capability, a model registry that supports fast rollback to a known-good version, and configuration that can tighten a gateway's policy in seconds are all preparation decisions. If your only containment option is to take the entire AI feature offline, you have one blunt instrument; the goal of preparation is to give yourself precise ones. The Deflected platform is designed around exactly this posture, placing an inspection-and-control point in the path so that the levers exist before the incident does.

Phase 2 — Detection and analysis: knowing an AI incident when you see one

Detection is where AI incident response feels most unfamiliar, because the indicators are not the ones your existing monitoring is tuned for. There is no hash to match, no known-bad IP to flag. Instead, you are watching for anomalies in behavior and content — signals that a model is being manipulated or is behaving outside its intended envelope. The teams that detect AI incidents quickly are the ones that decided in advance what "normal" looks like for their models and instrumented for deviations from it.

AI-specific detection signals

The following are the signals that most reliably indicate an AI incident is underway. None is conclusive on its own; together they form a picture.

• Guardrail trips and refusals spiking — a sudden rise in inputs that trigger safety filters, or in the model refusing and then complying, often indicates an active jailbreak campaign probing for a bypass.
• Anomalous tool calls — an agent invoking a tool it rarely uses, calling tools in an unusual sequence, passing unexpected arguments, or reaching for a capability that does not fit the user's request is a strong sign of hijacking.
• Output containing sensitive data patterns — model responses that contain things that look like credentials, personal data, internal identifiers, or another user's information suggest leakage in progress.
• Prompt-injection markers in retrieved content — instruction-like language ("ignore previous instructions," "you are now...") appearing inside documents, web pages, or records the system retrieved is a direct indicator of an indirect-injection attempt.
• Distribution shifts in inputs — a burst of unusually long inputs, inputs containing encoded or obfuscated text, or inputs in unexpected formats can indicate an attacker probing the system.
• Behavioral drift after a change — a model that starts behaving differently following a fine-tune, a data update, or a dependency change may signal poisoning or a supply-chain problem rather than a live attack.
• Out-of-band reports — a customer noticing the assistant said something it should not, or a finance team member flagging an unusual voice request, is still one of the most common ways AI incidents come to light. Make it easy to report and route.

Analysis: scoping a behavioral incident

Once a signal fires, analysis has to answer the same questions as any incident — what happened, how bad is it, what is the blast radius — but the way you answer them is different. Scoping an AI incident means reconstructing sessions from your logs: which conversations exhibited the anomalous behavior, what context was in the prompt when it happened, what the model had access to, which tools it called, and what data flowed as a result. If the incident is data leakage, the scope is defined by what the model could reach and what it actually output across affected sessions. If it is a jailbroken agent, the scope is the set of actions the agent took while compromised. If it is suspected poisoning, the scope is every deployment running the affected model version. This is why the preparation-phase logging matters so much: analysis is only as good as the record you can reconstruct, and a behavioral incident with no telemetry is nearly unscopeable.

Distinguishing incident from misbehavior

A genuinely hard part of AI detection is triage: models sometimes produce wrong, strange, or inappropriate output on their own, without any adversary involved. Not every bad response is an incident. Analysis has to distinguish a model that is simply making a mistake from one that is being manipulated — and the distinguishing evidence is usually in the input. Injection markers in retrieved content, a coordinated pattern of probing inputs, or tool calls that serve an attacker rather than the user point to a real incident; a one-off odd answer with clean inputs usually points to a quality problem to be fixed through evaluation and guardrail tuning rather than incident response. Getting this triage right keeps your responders focused on real events without ignoring the signal in the noise.

Phase 3 — Containment, eradication, and recovery

When an AI incident is confirmed, the priority is the same as always: stop the harm quickly, then remove its cause, then restore trustworthy operation. What differs is the set of levers available, and the fact that some of the most effective ones are unique to the AI layer. The organizations that contain AI incidents in minutes rather than hours are the ones that built these levers during preparation.

Containment: the levers you can pull

AI-specific containment gives you options between "do nothing" and "take the whole feature down." The main levers are:

• Disable a tool — for a jailbroken or misbehaving agent, revoking access to a specific tool or capability stops the harmful actions while leaving the rest of the system running. Cutting an agent's ability to send email or move money can neutralize the impact of a hijack even before you understand it fully.
• Roll back a model or prompt — if the incident followed a change — a new model version, a fine-tune, an updated system prompt, a modified retrieval source — reverting to the last known-good state is often the fastest way to restore safe behavior. This depends on having versioned everything and being able to redeploy quickly.
• Tighten the gateway — an inline AI gateway lets you raise the strictness of input and output inspection in place: block a class of inputs, filter a category of output, quarantine a retrieval source, or apply a stricter policy to a specific application, all without a code deploy. This is the AI equivalent of tightening a firewall rule mid-incident.
• Isolate a data source or session — if a poisoned document or a specific retrieval source is the vector, removing it from the index contains the indirect-injection channel. If a particular session or account is the origin, isolating it limits further probing.
• Fail safe — where the harm is severe and the levers above are insufficient, degrading the feature to a safe fallback — a scripted response, a human hand-off, or a temporary disable — is a legitimate containment choice. The point of the finer levers is to make this the last resort rather than the only one.

Eradication: removing the root cause

Containment buys time; eradication removes the cause so the incident cannot simply resume. What eradication means depends on the category. For a prompt-injection breach, it is closing the path by which untrusted content reached the model with instruction-level trust — hardening input handling, separating instructions from data, and strengthening guardrails against the specific technique used. For data leakage, it is tightening what the model can access and adding output controls that catch regulated data before it is returned. For model poisoning or a supply-chain compromise, eradication is heavier: identifying the tainted model version, dataset, or dependency, removing it from every deployment, and retraining or restoring from a clean, verified source. For a jailbroken agent, it is fixing the guardrail or permission gap that allowed the tool misuse. In every case, eradication should be verified, not assumed — which usually means adversarial testing against the specific failure before you trust the fix.

Recovery: restoring trustworthy operation

Recovery is the deliberate return to normal operation with confidence that the incident is genuinely resolved. For AI systems, that confidence comes from evidence: replaying the attack against the remediated system to confirm it now behaves correctly, monitoring the specific signals that flagged the incident to ensure they have returned to baseline, and often keeping tighter gateway policies or additional logging in place for a period after the incident as a safeguard. Because AI behavior is probabilistic, recovery for AI leans harder on continuous verification than on a single sign-off — you are looking to establish that the corrected behavior holds across many interactions, not just one test. Restore capabilities incrementally where you can, re-enabling tools or lifting restrictions in stages while you watch the signals, rather than flipping everything back at once.

Forensics for non-deterministic systems

Forensics is where AI incident response departs most sharply from tradition, and it deserves its own treatment. The core difficulty is that many AI systems are non-deterministic: given the same input, a model may produce different outputs on different runs, because generation involves sampling and because the surrounding context — retrieved documents, conversation history, tool results — is assembled dynamically and may not be identical the next time. The comforting forensic assumption that you can take an artifact and reproduce the behavior on demand often does not hold.

Why reproduction is unreliable

Several factors conspire against clean reproduction. Sampling parameters such as temperature introduce randomness by design. Model providers update hosted models, so the exact weights that produced a response may no longer be available. Retrieval systems return different documents as their underlying data changes. Conversation state and system prompts evolve. Even with everything pinned, some systems remain stochastic. The practical consequence is that "run it again and watch what happens" — a staple of malware analysis — is not a dependable method for an AI incident. You cannot count on replaying your way to the truth.

Capture is the substitute for reproduction

Because you cannot rely on reproducing the behavior, forensic quality is determined almost entirely by what you captured at the time. This is the through-line of the whole guide restated for the investigation phase: the richer and more faithful your logging, the more answerable your questions. Effective AI forensics reconstructs the event from a recorded snapshot of everything the system saw and did — the full assembled prompt, the exact retrieved context, the model and version identifier, the sampling settings, every tool call with its arguments and results, and the output returned. With that snapshot, an investigator can reason about what happened even without re-running it: they can see the injected instruction in the retrieved document, the tool call it induced, and the data that flowed. Without it, they are left inferring behavior from its downstream effects, which is slow and often inconclusive.

Preserving and trusting the evidence

Forensic evidence is only as good as its integrity. AI incident logs must be tamper-evident and access-controlled, so that an investigator — and, if it comes to it, a regulator or a court — can trust that the record was not altered. Preserve the relevant logs immediately when an incident is declared, before retention windows expire or routine processes overwrite them, and maintain a clear chain of custody. Because these logs contain highly sensitive material, their protection and their evidentiary value go hand in hand: strong encryption and strict access control are what make it safe to keep a complete record in the first place.

Attribution and intent

A final forensic wrinkle is that AI incidents blur the line between attack and accident. A model that leaked data may have been deliberately manipulated, or may have been led there by an ordinary user who stumbled into a weakness. An agent that took a harmful action may have been hijacked, or may have simply reasoned poorly. Establishing intent — was this an adversary or a flaw? — matters for how you respond, what you disclose, and whether law enforcement is involved, and it is answerable only from the inputs. Injection markers, coordinated probing, and evidence of a deliberate bypass point to an attacker; clean inputs and an isolated failure point to a defect. Resist the urge to assume either; let the captured evidence decide.

Phase 4 — Post-incident activity

The final phase turns a painful event into a durable improvement. In traditional incident response this is the lessons-learned review; for AI it carries extra weight, because the field is young and every real incident teaches something your playbooks did not yet know. A disciplined post-incident practice is how an AI security program actually matures.

The blameless review

Hold a structured, blameless review after every significant AI incident. Reconstruct the timeline from your logs: when the behavior started, when it was detected, how long detection took and why, what contained it, and how long each phase ran. Identify what worked and — more valuably — what did not: the signal you missed, the log you wished you had, the containment lever that was not built, the playbook step that was vague, the escalation that was slow. The goal is not to assign fault but to find the specific, fixable gaps between the response you had and the one you wanted.

Feeding lessons back into the system

Post-incident learning is only real if it changes something. Concretely, that means updating playbooks with the steps you wished you had followed; adding detection for the signal that would have caught the incident sooner; hardening guardrails and input handling against the specific technique that worked; and, importantly, adding the attack to your ongoing adversarial testing so a regression is caught automatically. This is where incident response connects to the rest of the AI security program: an incident that produces a new red-team test case has made your continuous testing measurably stronger, and each cycle raises the floor. Over time, the incidents that once required a scramble become scenarios your system detects and contains routinely.

Metrics that matter

Track a small set of meaningful measures across incidents so you can see whether the program is improving: time to detect, time to contain, time to recover, and the proportion of incidents caught by instrumentation versus reported from outside. Watch the trend rather than any single number. A program that is maturing will detect AI incidents earlier, contain them with finer levers, and increasingly catch them through its own monitoring rather than through an embarrassed customer email. Keep response-time commitments and service levels defined in general terms appropriate to your environment and risk tolerance; the point of the metrics is direction of travel, not a headline figure.

Communication and regulatory considerations

How an organization communicates during an AI incident often determines whether it becomes a manageable event or a reputational crisis. AI incidents raise communication and regulatory questions that traditional playbooks did not anticipate, and the time to work them out is during preparation, not in the middle of an event.

Internal communication

AI incidents are inherently cross-functional, so internal communication has to reach further than the security team. Engineers who own the model and tool integrations, the data and privacy office, legal, communications, and — for fraud-adjacent incidents such as deepfakes — finance and HR all may need to be in the loop quickly. Define in advance who is notified for each incident type, who decides on containment actions that affect a live product, and who has the authority to approve external disclosure. A clear internal escalation map, rehearsed in tabletops, prevents the paralysis that comes from nobody being sure who owns the decision.

Regulatory and disclosure obligations

When an AI incident involves personal or regulated data — which data-leakage and many agent-misuse incidents do — existing breach-notification obligations can apply just as they would to any other data incident, with their own timelines and thresholds depending on the data and jurisdiction. AI systems can also fall within emerging AI-specific regulation, and enterprise contracts increasingly carry their own notification requirements. The practical guidance is to map your obligations in advance: know which regulators and contractual counterparties must be notified, under what conditions, and within what timeframes, so that during an incident you are executing a known plan rather than researching your duties under time pressure. Because these determinations turn on facts — what data was involved, whose, and what actually left — they depend directly on the scoping your logs make possible. Treat specific legal thresholds and deadlines as questions for your own counsel and jurisdiction; the discipline is having the analysis ready, not memorizing a universal rule that does not exist.

External communication

If an incident affects customers or the public, communication should be timely, accurate, and measured. Say what you know, avoid speculation, and align the security, legal, and communications functions on a single account. AI incidents can attract disproportionate attention precisely because AI is novel and closely watched, which makes calm precision more valuable, not less. The organizations that come through AI incidents with their reputation intact are usually the ones that had thought about their communication posture before they needed it, and that could speak credibly about what happened because their instrumentation let them actually know.

Getting started with AI incident response

If your organization is deploying AI and this guide has revealed gaps, the path forward is incremental. You do not have to build a perfect AI incident-response capability overnight; you have to close the highest-leverage gaps first and improve from there. A pragmatic sequence looks like this.

1. Instrument the AI layer. Start logging every model decision — prompts, retrieved context, model and version, tool calls, and outputs — in a tamper-evident, access-controlled store. Nothing else in AI incident response works well without this, so it comes first.
2. Add AI-specific detection. Put the signals from the detection section in front of a human or a system that watches them: guardrail trips, anomalous tool calls, sensitive data in output, and injection markers in retrieved content. An inline gateway can serve as both sensor and control point.
3. Build containment levers. Make sure you can disable a tool, roll back a model or prompt, and tighten the gateway quickly. Confirm these levers work before you need them.
4. Write and rehearse playbooks. Draft AI-specific, cross-functional playbooks for the real incident types, then run a tabletop against each and fix what the exercise exposes.
5. Arrange expert response and map obligations. Put a retainer or named responders in place, and map your communication and regulatory obligations in advance so the plan exists before the incident does.

Much of this is easier when the instrumentation, detection, and containment are provided as one coordinated layer rather than assembled piecemeal. The Deflected platform places an inspection-and-control point in the AI request path — logging every decision, detecting AI-specific attacks, and giving you the levers to contain them — and our AI Incident Response service puts expert help on a standing retainer for the moments that need a human. However you assemble the capability, the goal is the same: when an AI system is breached, manipulated, or leaking, you can detect it early, contain it precisely, investigate it faithfully, and recover with confidence.

Frequently asked questions