Adversarial Machine Learning Attacks

Executive summary

Adversarial machine learning is the discipline concerned with how machine-learning models can be deliberately manipulated — and with the defenses that make them harder to manipulate. Unlike a conventional software exploit, an adversarial attack does not target a bug in code. It targets the way a model learns from data and draws decision boundaries, so a change too small for a person to notice can cause the model to fail with high confidence.

For enterprises now embedding models into fraud screening, content moderation, medical imaging, document processing, and customer-facing assistants, this is not an academic curiosity. It is a live threat class with its own tactics, its own frameworks, and its own defensive discipline. This guide is written for security leaders, ML engineers, and risk owners who need an accurate, jargon-free map of the territory: what the attacks are, how they differ, what they mean in production, and what an honest defensive posture looks like.

What adversarial machine learning actually is

Machine-learning models are function approximators. During training, a model is shown many examples and adjusts millions or billions of internal parameters to map inputs to outputs — an image to a label, a transaction to a fraud score, a sentence to a sentiment. What it learns is not a set of human rules but a high-dimensional decision surface: a mathematical boundary that separates one class from another. That surface is astonishingly effective on the kinds of inputs the model was trained on, and it is also, almost inevitably, fragile at the edges.

Adversarial machine learning is the study of that fragility and its exploitation. An adversary who understands how a model draws its boundaries can construct inputs that sit just on the wrong side of one — inputs that look ordinary to a human observer but that the model reads as something entirely different. The same discipline covers the defensive side: the techniques that measure a model's robustness and raise the cost of manipulating it.

Three properties make this different from traditional application security, and it is worth being precise about each:

• The vulnerability is statistical, not syntactic. There is no malformed input, buffer overflow, or injected command in the classical sense. The model is behaving exactly as trained; the attacker has simply found a region of its input space where "as trained" means "wrong."
• The attack surface is the model's judgment. The target is a decision — a classification, a ranking, a generated response — rather than a system to be compromised. Success is measured by whether the model can be made to decide incorrectly, not by whether code executes.
• There is rarely a clean patch. A code flaw can be fixed and the class of bug closed. A model's susceptibility to adversarial inputs is bound up with how it generalizes, so hardening against one attack often shifts the weakness rather than eliminating it. This is why practitioners speak of robustness as a property to be improved and measured, not a box to be checked.

The field has a long research lineage. Attacks on spam filters and other statistical classifiers were documented in the mid-2000s, and the modern wave began around 2013–2014 when researchers showed that deep neural networks for image recognition could be reliably fooled by tiny, precisely computed perturbations. That work established the central and still-unsettling result: high accuracy on normal data tells you almost nothing about how a model behaves under adversarial pressure.

Why machine-learning models are vulnerable

It is tempting to treat adversarial vulnerability as a defect to be engineered away. The more accurate framing is that it is a consequence of how current models work. Understanding why helps set realistic expectations for defense.

High-dimensional decision boundaries

Modern models operate in spaces with thousands to millions of input dimensions — every pixel, token, or feature is a dimension. In such spaces there is an enormous amount of "room" close to any given input. An attacker running an optimization process can search that neighborhood for the smallest change that flips the model's output, and because the space is so large, a change imperceptible to a human is frequently enough. The very capacity that lets a model fit complex patterns also gives an adversary many directions in which to nudge an input across a boundary.

The gap between what the model learned and what we meant

A model trained to recognize, say, a stop sign does not learn the human concept of a stop sign. It learns whatever statistical features most efficiently separated stop signs from everything else in its training data. Some of those features are robust and meaningful; others are brittle correlations that happen to work on clean data but collapse under deliberate manipulation. Adversaries exploit the brittle features. This "shortcut learning" is a core reason accuracy and robustness diverge.

Confidence without calibration

Many models report high confidence even when wrong, and adversarial inputs are often classified incorrectly with very high confidence. This matters for defense: a naive "reject low-confidence predictions" rule does little, because successful attacks tend to produce exactly the confident, decisive outputs that automated pipelines are built to trust.

The four attack categories

Adversarial attacks are commonly grouped by when in the model's lifecycle they strike and what the attacker is trying to achieve. Four categories cover the landscape, and the rest of this guide examines each in turn:

• Evasion — manipulating an input at inference time so a trained, deployed model misclassifies it. This is the category most people mean when they say "adversarial example."
• Poisoning — corrupting the training data or the learning process so the resulting model is compromised from the start, including planted backdoors.
• Extraction and inference — attacks against the confidentiality of the model or its data: stealing a functional copy of the model, or inferring private facts about its training data.
• Prompt-level attacks on LLMs — a fast-growing category specific to large language models, where natural-language inputs subvert the model's instructions and behavior.

These categories are not mutually exclusive. A sophisticated campaign might poison a model during training to install a backdoor, then trigger it at inference with a crafted evasion input, all while probing the deployed interface to refine the attack. The NIST taxonomy discussed later formalizes these relationships; the categories here are the practical starting point.

Evasion attacks and adversarial examples

Evasion is the classic adversarial attack. The model is already trained and running in production. The attacker takes a legitimate input and applies a small, deliberately computed perturbation — a set of tiny changes across the input's features — so that the model's output flips, while the input remains benign to a human. The manipulated input is called an adversarial example.

The canonical illustration comes from computer vision: an image of a panda that a model classifies correctly, plus a faint layer of computed noise invisible to the eye, becomes an image the model labels as a gibbon with near-total confidence. The picture still plainly shows a panda to any person. The perturbation was not random; it was optimized to push the input in the exact direction the model's decision boundary was most sensitive to.

How evasion perturbations are generated

When the attacker can see the model's internals, perturbations are typically computed using the model's own gradients — the same signals used to train it, repurposed to find the direction that most increases the model's error. Techniques in the research literature range from single-step methods to iterative, optimization-based methods that find minimal perturbations under a chosen constraint on how much the input may change. The precise algorithms matter less to a defender than the implication: given access, an attacker can compute a near-optimal adversarial input efficiently.

Evasion beyond images

Vision gets the memorable demos, but evasion applies to any model. In audio, small perturbations can cause a speech-recognition system to transcribe a different phrase. In text, changing a few characters, swapping synonyms, or inserting invisible characters can flip a toxicity or spam classifier. In tabular domains such as fraud and intrusion detection, an attacker can adjust the controllable features of a transaction or a network flow — within the bounds of a real, functioning transaction — to slip under a detection threshold. In each case the same principle holds: find the smallest change to a valid input that moves it across the model's decision boundary.

Physical-world evasion

Evasion is not confined to digital inputs. Researchers have shown that carefully designed stickers on a road sign, patterns printed on clothing, or specially crafted eyeglass frames can cause vision models to misread the physical scene. Physical attacks are harder to execute reliably because they must survive changes in lighting, angle, and distance, but they demonstrate that the threat is not merely a matter of tampering with a file in transit.

Poisoning attacks

Where evasion attacks a model's judgment on a single input, poisoning attacks the model itself. The adversary interferes with the training process — most commonly by injecting or altering training data — so that the model learns something the defender did not intend. Because the corruption is baked in during training, a poisoned model can look completely healthy on ordinary evaluation and fail only under the attacker's chosen conditions.

Poisoning is especially relevant now because models are increasingly trained on data of uncertain provenance: web-scraped corpora, third-party datasets, user-contributed content, and continual-learning pipelines that ingest fresh data automatically. Every one of those channels is an opportunity for an attacker to influence what the model learns. We cover this in depth in our companion guide on training-data poisoning; the summary here places it in the adversarial-ML landscape.

Availability poisoning

The blunt form of poisoning aims to degrade the model's overall performance — to make it less accurate or less useful across the board. An attacker who can inject enough mislabeled or noisy examples can pull the model's decision boundaries out of shape. This is a denial-of-quality attack: the goal is a model that simply works worse, undermining trust in the system.

Backdoor and targeted poisoning

The more insidious form is a backdoor. Here the attacker leaves the model's normal accuracy untouched but embeds a hidden association between a specific trigger and an attacker-chosen output. The model behaves perfectly until it sees the trigger — a particular pixel pattern, a specific phrase, an unusual token sequence — at which point it produces the output the attacker planted. A backdoored malware classifier might flag everything correctly except files carrying the trigger, which it waves through. Because the model passes every standard test, backdoors are difficult to detect without techniques designed specifically to look for them.

Supply-chain poisoning

Poisoning does not require access to your training pipeline. Pretrained models, foundation models, and public datasets downloaded from open repositories can arrive already compromised, carrying a backdoor introduced upstream. Fine-tuning a poisoned base model can preserve the backdoor. This is why vetting the provenance of models, datasets, and dependencies before they enter a pipeline is a distinct security discipline — the model supply chain is now part of the attack surface, and it should be treated with the same rigor as any other third-party software dependency.

Extraction and inference attacks

The third category targets confidentiality rather than integrity. Instead of making the model decide incorrectly, the attacker aims to steal something: the model itself, or private information about the data it was trained on. These attacks generally work through the model's normal interface, which makes them available to anyone who can send queries and read responses.

Model extraction

Model extraction — also called model stealing — reconstructs a functional copy of a target model by querying it systematically and training a substitute on the input-output pairs it returns. A sufficiently thorough extraction can approximate the target's behavior closely enough to be used commercially, eroding the intellectual property embodied in an expensive proprietary model. Extraction also serves as a stepping stone: an attacker who builds a good local substitute can then craft evasion attacks against it in a white-box setting and transfer them to the real target.

Membership inference and model inversion

Privacy attacks probe what the model reveals about its training data. In a membership inference attack, the adversary determines whether a specific record was part of the training set — a meaningful breach when membership itself is sensitive, such as a patient in a disease-specific cohort. In a model inversion attack, the adversary reconstructs representative features of the training data, potentially recovering approximations of private inputs. Both exploit the fact that models often memorize aspects of their training data and leak that memorization through their outputs.

These attacks sit at the boundary between adversarial ML and data protection. Because they turn on what a model discloses through its responses, they connect closely to the broader problem of information escaping through model output, which we treat separately in our guide to AI data leakage. For the purposes of this guide, the key point is that a model's query interface is itself a confidentiality attack surface, and that rate limiting, output minimization, and privacy-preserving training are the levers that reduce exposure.

Prompt-level attacks on large language models

Large language models introduced a category of adversarial attack that is expressed entirely in natural language. Because an LLM follows instructions written in the same channel as the data it processes, an attacker can smuggle instructions into that channel and subvert the model's intended behavior. These prompt-level attacks are the fastest-growing area of adversarial ML in production systems, and they deserve to be understood as adversarial inputs in their own right.

Prompt injection

Prompt injection occurs when untrusted content — a user message, or text retrieved from a document, web page, or email — carries hidden instructions that the model obeys as if they came from the developer. It is the AI-era analogue of injection attacks against databases, and it is consistently ranked as the top risk for LLM applications. In its indirect form, the malicious instruction lives in a document the model later reads during retrieval-augmented generation, so the attacker never interacts with the application directly; they simply plant the payload where the model will encounter it.

Jailbreaks

A jailbreak is a prompt crafted to bypass a model's safety training and guardrails, coaxing it to produce content or take actions it was designed to refuse. Jailbreaks exploit the same generalization gap as other adversarial examples — they find phrasings and framings that the model's safety training did not cover. Because language is open-ended, the space of possible jailbreaks is effectively unbounded, and new techniques appear continuously.

Why prompt-level attacks are genuinely adversarial ML

It is worth stating plainly: prompt injection and jailbreaks are adversarial examples for language models. The attacker searches the input space for a phrasing that pushes the model across a behavioral boundary, exactly as an evasion attacker searches pixel space. The consequences are amplified when the model is an agent with tools — the ability to call APIs, query databases, or execute code — because a successful prompt-level attack can then translate directly into unauthorized actions. We cover these techniques and their defenses in more depth in our writing on prompt injection, jailbreaking, and LLM guardrails; here they take their place as one of the four core adversarial categories.

Attack dimensions: white-box, black-box, targeting, and transferability

Cutting across the four categories are several dimensions that determine how an attack is mounted and how dangerous it is. Precision about these terms is what separates a useful threat model from hand-waving.

White-box versus black-box

In a white-box attack, the adversary has full knowledge of the model — its architecture, parameters, and often its training data. This is the strongest attacker assumption and the one used to stress-test defenses, because a defense that only works against ignorant attackers is not a defense at all. White-box access lets an attacker compute gradient-based perturbations directly and efficiently.

In a black-box attack, the adversary has no visibility into the model's internals and can only interact with it through its interface — submitting inputs and observing outputs. Black-box attacks are more representative of most real-world adversaries, who face a deployed API rather than a downloadable model. They typically work by querying the model many times to estimate how its outputs change with its inputs, or by exploiting transferability. A middle category, sometimes called gray-box, covers attackers with partial knowledge, such as the model family or the type of defense in place.

Targeted versus untargeted

An untargeted attack simply wants the model to be wrong — any incorrect output will do. A targeted attack is more demanding: it aims to make the model produce a specific chosen output, such as classifying a particular piece of malware as benign, or a specific face as a specific different identity. Targeted attacks are generally harder to achieve but far more useful to an adversary with a concrete objective, and they are the ones that matter most in high-stakes settings like fraud and authentication.

Transferability

Transferability is the property that makes black-box attacks practical and is one of the most important concepts in the field. An adversarial example crafted to fool one model frequently fools other models trained for the same task — even models with different architectures, trained on different data, that the attacker has never seen. The reason is that independently trained models tend to learn similar decision boundaries for the same problem, so an input that exploits one boundary often exploits the others.

The practical consequence is significant. An attacker who cannot see your model can train a substitute of their own — possibly using model extraction — craft attacks against that substitute in a comfortable white-box setting, and then transfer those attacks to your production model with a meaningful success rate. Transferability means secrecy about your model is not a reliable defense: assuming an attacker can obtain a transferable attack is the safer posture.

Real-world implications

Adversarial ML stops being abstract the moment a model is making decisions that carry consequences. The implications differ by domain, and framing them concretely helps prioritize defenses.

Security classifiers: spam, malware, and intrusion detection

Models that decide whether something is malicious are, by definition, deployed against motivated adversaries. Spam filters, malware detectors, and network-intrusion classifiers face opponents whose entire goal is evasion. Attackers routinely test their payloads against detection models and adjust until they pass, which is adversarial ML in its most operational form. Any classifier standing between an attacker and their objective must be assumed to be under continuous evasion pressure, and its performance on yesterday's samples says little about tomorrow's.

Fraud and abuse models

Fraud-detection and abuse-prevention systems share the same dynamic with a financial incentive attached. A fraud ring probes the scoring model by submitting transactions and observing which get through, then shapes subsequent activity to stay under the threshold — a black-box evasion campaign against a live model. Because the attacker controls many features of a genuine-looking transaction, the perturbation budget is generous. Static thresholds and models that are retrained infrequently are especially exposed.

Computer vision in the physical and digital world

Vision models drive content moderation, identity verification, document processing, medical imaging, and perception in autonomous systems. Evasion here can mean explicit content slipping past a moderation filter, a manipulated document passing verification, or a physical-world attack on a perception system. The severity scales with autonomy: a vision model that only flags content for human review fails more gracefully than one wired directly to an action.

Large language models and agents

LLMs concentrate several adversarial risks at once. They are susceptible to prompt injection and jailbreaks, they can be probed for training-data leakage, and when deployed as tool-using agents they translate a successful language-level attack into real actions in connected systems. The blast radius of an adversarial input against an agent is a function of what that agent can reach — which is why constraining an agent's tools and permissions is as much an adversarial-ML defense as it is an access-control decision.

MITRE ATLAS and the NIST adversarial ML taxonomy

Two public references have become the shared vocabulary for this field. Using them lets teams reason about coverage systematically instead of reacting to individual techniques, and both are worth citing accurately.

MITRE ATLAS

MITRE ATLAS — the Adversarial Threat Landscape for Artificial-Intelligence Systems — is a knowledge base of adversary tactics and techniques against machine-learning systems, modeled on the structure of the widely adopted MITRE ATT&CK framework. It organizes real-world and researched attacks into tactics (the adversary's goals, such as reconnaissance, initial access, or exfiltration) and techniques (the specific methods used to achieve them), and it draws on documented case studies of attacks against ML systems. For a defender, ATLAS provides a structured way to ask "which of these techniques applies to our system, and what have we done about each?" — turning a diffuse threat into an inventory that can be reasoned about and prioritized.

The NIST adversarial ML taxonomy

NIST's report Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, published as NIST AI 100-2, establishes a consistent conceptual vocabulary for the field. It classifies attacks along the dimensions this guide has used — the stage of the lifecycle (training versus deployment), the attacker's goal (integrity, availability, or privacy), and the attacker's knowledge and capabilities — and it spans both predictive machine learning and generative AI, including the prompt-level attacks that affect LLMs. Its value is precision: by giving attack types, attacker objectives, and mitigations agreed-upon names, it lets teams, vendors, and auditors communicate without talking past one another.

The two references are complementary. ATLAS is a practitioner's map of tactics and techniques grounded in observed incidents; the NIST taxonomy is a rigorous vocabulary and classification. Together they let a security program describe its adversarial-ML threat model in shared terms, map its controls against a recognized structure, and identify where coverage is thin. They also connect naturally to broader AI-risk governance such as the NIST AI Risk Management Framework, which situates adversarial robustness within an organization's overall approach to trustworthy AI.

Defenses and the ongoing arms race

There is no defense that makes a model provably immune to adversarial manipulation, and it is important to be honest about that from the outset. Every technique below raises the cost of an attack, lowers its success rate, or improves the odds of catching it — but each has limits, and defenses that resist one class of attack can leave another exposed. Robustness is best understood as a property to be continuously improved and measured, not a milestone to be reached. What follows is a layered program, because layered defense is what the evidence supports.

Adversarial training

Adversarial training is the most established defense against evasion. During training, the model is deliberately shown adversarial examples alongside clean data and taught to classify them correctly, hardening its decision boundaries against the kinds of perturbation the attacker would use. It measurably improves robustness against the attacks it is trained on. Its honest caveats are real: it is computationally expensive, it can trade away some accuracy on clean inputs, and it tends to generalize poorly to attack types it was not trained against. It raises the bar; it does not settle the contest.

Input validation and preprocessing

Input validation and preprocessing attempt to detect or neutralize adversarial perturbations before they reach the model — through anomaly detection on inputs, transformations that disrupt fragile perturbations, or constraints that reject inputs falling outside expected distributions. These measures catch some attacks and add useful friction, but adaptive attackers who know the defense is present can often craft perturbations that survive it. Input validation is a valuable layer, not a wall.

Robustness testing and evaluation

You cannot manage what you do not measure. Robustness testing subjects a model to a battery of known attacks and reports how it holds up — quantifying the attack success rate under specified conditions and perturbation budgets. Done before deployment and repeated after every significant change, it turns robustness from an assumption into a metric. The caveat is that a model resistant to the attacks in your test suite may still fall to an attack you did not test, so evaluation must evolve as new techniques emerge.

Monitoring and detection in production

Runtime monitoring watches deployed models for the signatures of an attack in progress: bursts of near-duplicate queries that suggest extraction or black-box probing, inputs that sit in unusual regions of the feature space, distribution shifts, and spikes in low-confidence or anomalous outputs. Monitoring rarely stops the first malicious query, but it shortens the time to detect a campaign and creates the audit trail needed to respond. Rate limiting and query budgeting on model interfaces are close companions, directly raising the cost of the many-query attacks that extraction and black-box evasion depend on.

Ensembles and diversity

Ensembles combine several models so that an input must fool all of them to succeed, which can raise the cost of an attack. The important caveat is transferability: because adversarial examples often transfer across independently trained models, an ensemble of similar models can be defeated by a single well-crafted input. Ensembles help most when the constituent models are genuinely diverse in architecture and training, and even then they are a mitigation rather than a guarantee.

Provenance and supply-chain controls

Against poisoning, the strongest defenses are upstream. Vetting the provenance of training data, verifying the integrity of pretrained models and datasets before they enter a pipeline, controlling who can contribute to training sets, and scanning for backdoor triggers all reduce the chance that a compromised artifact reaches production. Because a poisoned model can pass ordinary evaluation, these preventive controls matter more here than post-hoc detection.

Continuous red teaming

The defense that ties the others together is continuous red teaming — standing, adversarial testing that attacks your own models the way a real threat actor would and reports the weaknesses before an attacker finds them. Point-in-time assessments go stale quickly: models are retrained, prompts change, new tools are connected, and novel attack techniques are published constantly, so a clean report from last quarter says little about today. Continuous testing catches regressions and new weaknesses as they appear, and it produces the one metric that actually tracks progress — attack success rate over time. Our Continuous AI Red Team is built for exactly this: always-on adversarial testing across evasion, poisoning, extraction, and prompt-level techniques, returning a prioritized, fixable report. For the broader discipline and how to stand up a program, see our guide to AI red teaming.

How the pieces fit together

No single control on this list is sufficient, and that is the point. Adversarial training hardens the model but does not cover every attack; input validation adds friction but yields to adaptive attackers; monitoring catches campaigns but not the first query; ensembles help but transfer attacks cut through them; provenance controls stop poisoning but not evasion. Layered together, and kept honest by continuous red teaming and hard metrics, they compose into a defensible posture. Encryption underpins all of it: every artifact Deflected handles — models, datasets, logs — is protected with post-quantum cryptography, using ML-KEM-1024 (FIPS 203) for key encapsulation, hybrid X25519 + ML-KEM key exchange so a weakness in either scheme is covered by the other, and AES-256 for data at rest and in transit. Resilience is the sum of these parts, maintained over time — not any one of them in isolation.

Frequently asked questions