AI Red Teaming, Explained

What AI red teaming is

AI red teaming is the practice of deliberately and adversarially testing an artificial-intelligence system to discover how it can be manipulated, misused, degraded, or made to fail — and to do so on your own terms, before a genuine attacker finds the same weaknesses in production. It takes the oldest idea in security, that you cannot trust a defense you have never attacked, and applies it to a new and unusually slippery target: the behavior of a probabilistic model.

The term comes from military and cyber tradition. A "red team" plays the adversary against a defending "blue team," probing plans and systems for the flaws that optimism and familiarity tend to hide. In classical cybersecurity, red teams attack networks, applications, and identity systems. In AI, the red team attacks the AI layer: the models, prompts, agents, retrieval pipelines, and data flows that sit on top of the conventional stack and make AI features work. The object of study is not primarily the code around the model. It is the model's own reasoning, the context it is given, and the tools and data it is allowed to reach.

What makes this discipline distinct is the nature of the target. A traditional vulnerability is a defect in logic: a buffer that is not bounded, a query that is not parameterized, a permission that is not checked. It exists or it does not, and once patched it is gone. An AI weakness is usually not a defect in code at all. It is an emergent property of a system trained to be helpful, general, and responsive to natural language. The same instruction that is perfectly legitimate from an administrator can be an attack when it arrives hidden inside a document the model was asked to summarize. The model has no reliable way to tell the difference on its own. AI red teaming is the systematic effort to find where that ambiguity becomes exploitable.

A useful way to frame it is by contrast with ordinary quality assurance. Functional testing asks whether a feature works when used as intended. AI red teaming assumes an intelligent, motivated adversary is actively trying to make the feature work in ways it should not — to extract data it should protect, to take actions it should refuse, to produce output that creates legal, financial, or reputational harm. The red teamer's job is not to confirm the happy path. It is to find the unhappy paths that a determined attacker would find, document them precisely, and hand engineering a prioritized list of things to fix.

Why AI red teaming is necessary

Enterprises are deploying AI into workflows that touch money, regulated data, and customer trust faster than their security programs were designed to keep up with. A model that drafts contracts, answers customer questions, triages support tickets, or takes actions through connected tools is now part of the attack surface — and it is a part that behaves unlike anything security teams have defended before. There are several reasons the ordinary controls are not enough on their own, and each is a reason red teaming exists.

The attack surface is natural language

Conventional defenses are built to inspect code and traffic. A web application firewall looks for malicious payloads in HTTP requests; a scanner looks for known-vulnerable libraries. Neither has any concept of a politely worded sentence that instructs a model to disregard its own rules. Because the attacker's payload is language, and language is precisely what the model exists to process, the attack surface is enormous and cannot be enumerated the way a set of API endpoints can. Red teaming is how an organization probes a surface that is too large and too fluid to reason about from a diagram.

Behavior is probabilistic, not deterministic

A model does not execute fixed logic. It produces the most probable response given its context, and that means the same input can succeed one moment and fail the next, and a fix that appears to close a weakness may only have made it slightly less likely. You cannot prove the absence of a behavior in a probabilistic system by inspection. You can only estimate how hard it is to elicit, and that estimate is exactly what a disciplined red team produces: not a binary "vulnerable or not," but a measured sense of how much effort, and which techniques, are required to break the system.

Systems change constantly

The model provider ships an update; a prompt is edited; a new document source is added to a retrieval pipeline; an agent is granted a new tool. Any of these can silently reopen a weakness that testing had previously closed, because behavior is entangled across the whole system. Unlike a codebase, where a diff shows exactly what changed, a change in a model or a prompt can shift behavior in ways no one intended and no code review would catch. This churn is why point-in-time assurance decays so quickly, and why testing has to become continuous.

The consequences are real and specific

The reason this matters to a board is that the failures are not abstract. A retrieval-augmented assistant can be induced to reveal another customer's data. An agent with tool access can be steered into taking an unauthorized action. A support bot can be manipulated into making commitments the company is then held to. A model can be pushed to produce output that is defamatory, discriminatory, or that leaks a trade secret. Each of these is a concrete business event — a breach, a regulatory finding, a lawsuit, a headline — and each traces back to a weakness that a red team could have found first. The purpose of AI red teaming is to convert those latent failures into a report and a remediation plan while they are still cheap to fix.

How it differs from traditional penetration testing

AI red teaming inherits the mindset of penetration testing — think like the attacker, prove the weakness rather than merely assert it — but the target is different enough that the two are not interchangeable. Understanding the differences prevents a common and expensive mistake: assuming that a firm's annual application pen test covers the AI layer. It almost never does.

Traditional penetration testing targets deterministic systems: infrastructure, networks, and application code. The tester hunts for exploitable flaws — an injection point, a broken access control, an unpatched service — and a finding is typically confirmed by a reproducible exploit and closed by a specific patch. Reproducibility is a given: the same request produces the same result, so a finding either exists or it does not, and once fixed it stays fixed. The attack surface is finite and largely enumerable, which is why scoping a pen test around a defined set of hosts and endpoints is workable.

AI red teaming targets a probabilistic model whose behavior is shaped by training and context. Consider how the fundamentals change:

• The payload is language, not code. The attacker's tool is a carefully framed instruction in natural language, not a crafted byte sequence. Defenses tuned to recognize malicious code are blind to it.
• Findings are statistical, not binary. An attack that succeeds sixty percent of the time is still a serious finding, and "we could not reproduce it every time" is not a reason to dismiss it. The right measure is a success rate, not a yes-or-no.
• There is rarely a single patch. A class of weakness such as prompt injection cannot be eliminated with one fix; it is reduced through layered controls and continually re-tested. Remediation is mitigation, not closure.
• The surface is unbounded. You cannot list every phrasing an attacker might use the way you can list open ports. Coverage becomes a question of technique categories and application flows, not an inventory of endpoints.
• Context is the vulnerability. The same model is safe in one deployment and exploitable in another, depending on what data and tools it can reach. The weakness often lives in the integration, not the model.

The practical implication is that the two disciplines are complementary and both necessary. A pen test will find that an API is missing authentication or that a container is misconfigured — real problems that a red team focused on model behavior might not even look at. An AI red team will find that the model can be talked into exfiltrating data from its context or misusing a connected tool — problems a pen test has no methodology for. An enterprise deploying AI needs both, applied to their respective surfaces, and should be skeptical of any assessment that claims one covers the other.

How it differs from AI safety evaluations

A second common confusion is between red teaming and evaluation. Both examine model behavior, and both are valuable, but they answer different questions and neither substitutes for the other.

An AI safety evaluation measures how a model behaves against a defined benchmark. You assemble a standardized set of prompts — for toxicity, bias, refusal on prohibited categories, factual accuracy, and so on — run the model against them, and compute scores. Evaluation is largely cooperative and reproducible: the prompts are fixed, the scoring is systematic, and running the same suite again gives a comparable number you can track across model versions. Its great strength is measurement. It tells you, on average and against a known distribution, how the model behaves, and it lets you compare models and detect regressions on the dimensions the benchmark covers.

Red teaming is adversarial and open-ended. It does not sample average behavior against a fixed list; it hunts for the specific inputs and conditions under which the system fails in a way that matters. A red teamer is not asking "what is the model's average refusal rate on this benchmark," but "what is the one framing, the one multi-turn sequence, the one hidden instruction that gets this particular deployment to do the thing it must never do." Evaluations operate inside a known distribution of prompts; red teams deliberately leave it, because attackers do. Evaluations measure the middle of the curve; red teams probe the tail, where the catastrophic failures live.

The relationship is best understood as a division of labor. Evaluations give you a reproducible baseline and a regression signal — essential for governance and for comparing options. Red teaming gives you the specific, exploitable weaknesses that no benchmark anticipated, along with a realistic sense of how hard they are to trigger. A mature program feeds each into the other: novel weaknesses a red team discovers become new evaluation cases so that regressions are caught automatically, and evaluation results tell the red team where to concentrate its more expensive, creative effort. One tells you how the model behaves; the other tells you how it breaks. You need both to govern AI responsibly.

The technique spectrum

AI red teaming draws on a growing catalogue of techniques, each targeting a different weakness in how models take instructions, hold information, and act on the world. What follows is a defensive overview: the categories are described so defenders can reason about them and structure their coverage. None of this is a how-to, and no working payloads appear here — the value to a defender is in understanding the shape of each threat, not in reproducing it.

Prompt injection

The defining vulnerability of the AI era. When an application inserts untrusted input — from a user, or from a document it retrieved — into a prompt, an attacker can embed instructions in that input to override the developer's intent. In direct prompt injection the malicious instruction comes from the user interacting with the system. In indirect prompt injection it is planted in content the model will later ingest, such as a web page, an email, or a file, and it activates when the model processes that content on someone else's behalf. Prompt injection is to AI what SQL injection was to databases, and it is the first thing a competent red team probes for. Our deeper treatment lives in the guide to prompt injection.

Jailbreaks

Where prompt injection targets the application's instruction hierarchy, a jailbreak targets the model's safety training — the alignment that would normally make it refuse certain requests. Jailbreaks work by exploiting how a model weighs competing instructions, using framing, hypotheticals, roleplay, obfuscation, or gradual escalation to tip the balance from "refuse" back toward "comply." Because guardrails are learned probabilistic behavior rather than a hard boundary, jailbreaks cannot be patched away, only made harder and less reliable. The mechanics are covered in our guide to jailbreaking LLMs.

Data extraction and leakage

A large category concerns getting a model to reveal information it should protect. That can mean data in its immediate context — another user's records in a shared session, secrets in a system prompt, documents pulled in by retrieval — or, in some cases, fragments of training data the model memorized. Red teamers probe whether a system can be led, deliberately or accidentally, to emit regulated or confidential information in its output. This is especially acute in retrieval-augmented systems, where the model's context is assembled on the fly from data stores that may hold far more than the current user is entitled to see.

Training-data and supply-chain abuse

Some attacks target the model before it is ever deployed. Training-data poisoning introduces manipulated examples so the model learns a hidden behavior or backdoor that activates on a specific trigger. Related supply-chain risks include tampered model weights, compromised fine-tuning datasets, and malicious dependencies. A red team assessing this surface examines the provenance and integrity of models, datasets, and the pipeline that produced them, since a poisoned model can pass ordinary functional testing while carrying a dormant flaw.

Tool and agent abuse

As models are given tools — the ability to call APIs, query databases, send messages, or run code — the stakes of a successful manipulation rise sharply. An agent that can be steered through injection or jailbreak into misusing its tools can take real actions: moving data, triggering transactions, reaching systems it should not. Red teamers examine the agent's autonomy and privileges, testing whether a manipulated instruction can be turned into an unauthorized action and how far the blast radius extends. The principle of least privilege — granting an agent only the narrowest access it needs — is both a defense and a thing the red team verifies.

Multimodal attacks

Models that accept images, audio, or other media open attack paths that text-only defenses miss. Instructions can be embedded in an image or carried in audio, so that a system inspecting only the visible text prompt never sees the payload the model actually acts on. Red teaming a multimodal system means testing every input channel the model can perceive, not just the one a human operator is watching.

Multi-turn and social attacks

Some of the most effective attacks unfold over a conversation rather than a single message. An adversary establishes a benign context, then escalates gradually, so that no individual turn looks obviously malicious but the trajectory arrives somewhere the model should never have gone. Multi-turn techniques defeat defenses that inspect each message in isolation, which is why a serious red team tests conversations and sequences, not just standalone prompts, and why monitoring has to consider the arc of an interaction rather than one turn at a time.

A well-run program does not treat these as a checklist to be run once. It maintains coverage across the whole spectrum, weights its effort toward the techniques most relevant to a given application, and continually adds new variants as attackers and researchers discover them.

The frameworks that structure it

AI red teaming is a young discipline, but it is not unstructured. Several industry and government frameworks give teams a shared vocabulary, a way to organize coverage, and a means of demonstrating diligence to auditors and regulators. Three are especially relevant, and a credible program references them accurately rather than inventing its own taxonomy.

OWASP Top 10 for LLM Applications

The Open Worldwide Application Security Project maintains the OWASP Top 10 for LLM Applications, a community-driven list of the most critical security risks specific to applications built on large language models. It catalogues categories such as prompt injection, insecure output handling, sensitive information disclosure, excessive agency, and supply-chain vulnerabilities, giving red teams and developers a common reference for what to test and what to defend. Its value is practical: it maps directly onto the technique spectrum above and gives a program a defensible answer to "how do you know your coverage is complete." Because it is versioned and updated by the community, it also tracks the field as new risks emerge.

MITRE ATLAS

MITRE ATLAS — Adversarial Threat Landscape for Artificial-Intelligence Systems — is a knowledge base of adversary tactics and techniques against machine-learning systems, modeled on the widely used MITRE ATT&CK framework. Where the OWASP list is organized around application risks, ATLAS is organized around the attacker's playbook: the tactics an adversary moves through and the techniques available at each stage, grounded in real-world case studies. Red teams use ATLAS to plan campaigns that reflect how genuine threat actors operate and to describe their findings in terms defenders across the industry recognize. It is the closest thing the field has to a shared adversary model for AI.

NIST AI Risk Management Framework

The NIST AI Risk Management Framework (AI RMF) is the leading voluntary framework for identifying and managing AI risk across the model lifecycle, organized around four functions: Govern, Map, Measure, and Manage. Red teaming lives most naturally within the Measure function, which calls for assessing, analyzing, and tracking AI risks using rigorous, repeatable methods — including adversarial testing. Positioning a red team program under Measure connects it to the rest of an organization's governance: the risks it surfaces feed Manage, the context it operates in comes from Map, and the whole is accountable to Govern. This is how red teaming stops being a standalone exercise and becomes evidence in an auditable risk program, which is exactly what regulators and enterprise buyers increasingly ask to see.

These frameworks are complementary, not competing. OWASP tells you what risks to test for at the application layer, ATLAS tells you how real adversaries pursue them, and the NIST AI RMF tells you where the resulting evidence fits in a governance program that a board and an auditor can stand behind. A program that references all three speaks a language its stakeholders already understand.

Manual and automated, continuous and point-in-time

Two independent choices shape how a red team program operates. The first is how much of the testing is done by human experts versus automated tooling. The second is whether testing happens as a scheduled event or runs continuously against the live system. Neither choice is binary, and the right answer for most enterprises is a deliberate blend.

Manual versus automated

Manual red teaming puts skilled humans against the system. Its strength is creativity: an expert can invent a novel framing, chain techniques across a multi-turn conversation, reason about the specific business context, and recognize a subtle failure that no rule would flag. Its limits are cost and scale — human attention is expensive and cannot cover every prompt, every model version, every day.

Automated red teaming uses tooling, and increasingly other models, to generate and run large volumes of adversarial inputs and to score the responses. Its strength is scale and repeatability: it can probe thousands of variations, run continuously, and catch regressions the moment they appear. Its limit is that it tends to explore variations on known techniques rather than invent genuinely new ones, and its scoring is only as good as the classifiers behind it. The two are complementary. Automation provides breadth and constant vigilance; human experts provide depth and novelty, and they design and tune the automation. A program that relies on only one is either too shallow or too slow.

Continuous versus point-in-time

A point-in-time assessment is a scoped engagement with a start and an end, producing a report at a moment in time. It is the right tool for a launch gate, a compliance milestone, or a deep manual campaign against a newly built system, and it remains valuable. But its central weakness is decay. Because models, prompts, tools, and attacker techniques all change constantly, a clean report from last quarter says very little about the system today. The gap between assessments is exactly where regressions and new weaknesses accumulate unseen.

Continuous red teaming closes that gap by running automated adversarial testing against the live system on an ongoing basis, so that a weakness introduced by a model update or a prompt change is caught in days rather than discovered by an attacker months later. It turns red teaming from an event into a control — something that is always on, always watching, and always producing a current signal. The strongest posture combines the two: continuous automated testing for constant coverage and fast regression detection, punctuated by periodic manual campaigns for the depth and creativity that only human experts bring. This is the model behind our Continuous AI Red Team product, which runs always-on adversarial testing and returns a prioritized, fixable report rather than a static PDF.

The metrics that matter

Red teaming only becomes a management discipline when it produces numbers a leader can track over time. A pile of individual findings is useful to an engineer but does not answer the question a CISO or a board actually asks: is our AI getting more resilient or less? Four metrics, tracked consistently, turn adversarial testing into a measurable control.

Attack success rate

Attack success rate is the share of adversarial attempts that achieve their objective. It is the single most important trend line, because in a probabilistic system you rarely drive a weakness to zero; you drive its success rate down. Measured per technique category and tracked over time, attack success rate shows whether defenses are actually improving or merely being rearranged. A rising rate after a model or prompt change is an early warning that something regressed. The goal is not a perfect score but a steady, demonstrable decline across the techniques that matter to your application.

Coverage

Coverage answers how much of the relevant threat is actually being tested — both the breadth of the technique spectrum and the portion of the application surface exercised. A low attack success rate is only reassuring if coverage is high; a great score against a narrow slice of techniques is a false comfort. Coverage is where frameworks earn their keep: mapping tests to the OWASP LLM risks and to ATLAS techniques gives a defensible, auditable measure of how complete the testing is, and exposes the categories that have been neglected.

Time-to-detect

Time-to-detect measures how quickly a newly introduced weakness or regression is caught after it appears. It is the metric that most directly captures the value of continuous testing over point-in-time assessment. If a prompt change reopens an injection path, does the program surface it within hours, or does it wait for the next quarterly engagement? Short time-to-detect shrinks the window in which a live weakness is exposed to real attackers, and it is often the clearest justification for investing in continuous coverage.

Blast radius

Blast radius measures how much damage a single successful attack can cause, given what the model can reach. Two systems with the same attack success rate can carry wildly different risk: one model can only produce embarrassing text, while another can move money or read every customer's records because of the tools and data it is connected to. Tracking blast radius forces attention onto least privilege and containment — reducing what a compromised model can do, so that even a successful attack stays small. It is the metric that connects red teaming to architecture, because the cheapest way to shrink blast radius is often to grant the model less power in the first place.

Reported together and trended over time, these four turn red teaming from a recurring cost into evidence of a control that is working — the kind of evidence that satisfies an auditor, informs a board, and lets a security leader answer "are we more resilient than last quarter" with data rather than hope.

How to stand up a program

Building an AI red team capability does not require boiling the ocean. It requires a deliberate sequence that moves from understanding your exposure to testing it continuously and feeding the results back into how the system is built. The following steps describe a path most enterprises can follow.

1. Inventory the AI layer. You cannot test what you have not mapped. Catalogue every place AI touches the business: the models in use, the features built on them, the agents and their tools, the retrieval pipelines and the data they reach. This inventory defines the surface and, combined with the sensitivity of the data and actions involved, tells you where to concentrate effort first.
2. Threat-model each system. For each AI application, ask who would attack it, what they would want, and what the worst outcome would be. A public marketing chatbot and an internal agent with database access demand very different attention. Ground the model in a shared framework — the OWASP LLM risks for what to test, MITRE ATLAS for how adversaries operate — so the threat model is complete and defensible.
3. Establish a baseline. Run an initial assessment across the technique spectrum to measure current attack success rate, coverage, and blast radius. This baseline is the reference every future measurement is compared against, and it usually surfaces the highest-priority weaknesses immediately.
4. Fix and reduce blast radius. Remediate the findings, but treat mitigation as layered rather than a single patch. Apply least privilege to agents and tools, add independent input and output classification, and harden — but do not solely rely on — system prompts. The fastest risk reduction is often architectural: give the model less power and less data than it currently has.
5. Make testing continuous. Stand up automated adversarial testing that runs against the live system on an ongoing basis, so regressions from model updates and prompt changes are caught quickly. This is what keeps assurance from decaying between manual engagements and drives time-to-detect down.
6. Layer in periodic manual campaigns. Schedule expert-led red team engagements for depth and novelty — the creative, multi-turn, business-specific attacks automation will not invent on its own. Feed every new weakness they find back into the automated suite as a permanent regression test.
7. Report to governance. Position the program under the NIST AI RMF Measure function and report its metrics into your broader risk and compliance program. Trend attack success rate, coverage, time-to-detect, and blast radius so leadership and auditors can see resilience improving over time, not just a list of point findings.

Two principles run through all of it. First, red teaming is not a gate you pass once; it is a loop you run continuously, because the system it tests never stops changing. Second, its output is only as valuable as the fixes it drives — a program that finds weaknesses but does not close them, or does not feed them back into how systems are built, is theater. The organizations that get real value treat red teaming as a permanent, measured feedback loop wired into engineering and governance, not as an annual report that gets filed and forgotten.

How Deflected helps

Deflected treats AI red teaming as a continuous control rather than a periodic event, and integrates it with the rest of the AI-layer defenses an enterprise needs. Our Continuous AI Red Team runs always-on adversarial testing against your own models — attacking them the way real threat actors would across the full technique spectrum — and returns a prioritized, fixable report, so weaknesses are found before an attacker finds them and resilience can be proven to the board. Because it is continuous, it catches the regressions that a point-in-time assessment misses, and it tracks the metrics that turn testing into a trend line rather than a snapshot.

Red teaming does not stand alone. It sits within the broader Deflected platform, alongside real-time defenses such as an inline prompt firewall, discovery of unsanctioned AI use, and governance mappings to the frameworks auditors expect. Findings from red teaming inform the runtime controls, and the runtime controls contain the blast radius of anything red teaming has not yet caught — a defense-in-depth posture where testing and protection reinforce each other.

Every byte that flows through the platform is protected with post-quantum cryptography by default, not as a premium add-on. That means ML-KEM-1024 (NIST FIPS 203) for key encapsulation, hybrid X25519 + ML-KEM key exchange that runs a proven classical algorithm alongside the post-quantum one so you are protected even if either is ever weakened, and AES-256 for symmetric encryption of data at rest and in transit. The result is assurance that reaches from the adversarial testing of your models all the way down to the cryptography protecting the evidence those tests produce.

Frequently asked questions