Training-Data Poisoning: Risks to AI Models & Defenses

Executive summary

Training data poisoning is the deliberate corruption of the data an AI model learns from, so that the finished model behaves the way an attacker wants rather than the way its builders intended. Because a machine-learning model derives its entire behavior from its training data, a tainted dataset produces a tainted model — one that may quietly lose accuracy, carry injected bias, or hide a backdoor that waits for a secret trigger before it misfires.

Unlike an attack that targets a running system, a poisoning attack targets the model's past. It happens before the model is ever deployed, often before the organization deploying it even exists in the story — upstream, in a scraped web corpus, a public dataset, a fine-tuning file, or a set of open weights pulled from a model hub. That makes it a supply-chain problem as much as a security problem, and it is why the discipline of vetting what enters your pipeline has become one of the most important controls in enterprise AI.

This article explains, in precise and practical terms, what data and model poisoning is, where in the model lifecycle it can happen, the main categories of attack, why open-source models and datasets carry real supply-chain risk, how a backdoor can stay dormant for months, and — most importantly — the layered defense strategy that keeps poisoned artifacts out of production. It is written for security leaders, ML engineering teams, and the risk and compliance functions that have to sign off on AI. Where a capability has a dedicated page, we link to it so you can go deeper.

What data and model poisoning actually is

Every machine-learning model is a statistical summary of the data it was trained on. It has no independent knowledge of the world; it has patterns, and those patterns come entirely from examples. Change the examples and you change the model. That simple fact is the entire basis of a poisoning attack: if an adversary can influence the training data, they can influence the behavior of the resulting model — sometimes broadly, sometimes with surgical precision.

Data poisoning refers to manipulating the data itself: adding malicious examples, altering labels, or seeding a corpus with content designed to be absorbed. Model poisoning is a slightly broader term that also covers tampering with the model directly — for example, modifying weights, or interfering during a distributed or federated training run — so that the malicious behavior is encoded even without touching the original dataset. In practice the two overlap, and the industry increasingly groups them together. The OWASP Top 10 for Large Language Model Applications lists them under a single risk category, Data and Model Poisoning, and describes how manipulation of pre-training, fine-tuning, or embedding data can introduce vulnerabilities, backdoors, or biases that compromise a model's security, accuracy, and integrity.

It is worth drawing a sharp line between poisoning and the AI vulnerability most people know first: prompt injection. Prompt injection attacks a model at inference time, by hiding instructions inside the text the model reads; the model itself is unchanged, and the attack has to be present in each request. Poisoning is different in kind. It bakes the malicious behavior into the model's parameters during training, so it persists across every prompt, every session, and every deployment of that model, with nothing suspicious in the input at all. One is a runtime problem you can filter; the other is a lifecycle problem you have to prevent upstream. A defense-in-depth program needs answers to both.

The reason poisoning deserves board-level attention is leverage. A comparatively small manipulation, introduced at the right moment in the training process, can shape the behavior of a model that then makes millions of decisions. And because the corruption lives in the weights rather than in any log or request, it can be extraordinarily difficult to detect after the fact. The economics favor the attacker: cheap to insert, expensive to find, and durable once it lands.

The model lifecycle attack surface

To defend against poisoning you have to know where it can enter, and the answer is: at almost every stage where data flows into a model. A modern AI system is assembled from many sources, and each one is a potential injection point. Walking the lifecycle from left to right makes the surface concrete.

Pre-training data scraped from the open web

Foundation models are trained on enormous corpora, much of it scraped from the public internet — web pages, forums, code repositories, comment threads, and archives. The open web is, by definition, an environment an attacker can write to. Anyone can publish a page, edit a wiki, seed a repository, or flood a forum. If a fragment of the internet is likely to be captured into a future training snapshot, then planting content in that fragment is a way to influence a future model. The scale of pre-training corpora is often assumed to dilute any single contribution into irrelevance, but targeted techniques and repetition can concentrate influence far more than raw proportions suggest. The open web is the largest and least controlled part of the surface.

Fine-tuning datasets

Most enterprises do not train foundation models from scratch; they fine-tune a base model on a smaller, curated dataset to specialize it for a task — customer support, contract analysis, code generation. Fine-tuning is a high-leverage injection point precisely because the dataset is small. A handful of poisoned examples represent a much larger fraction of a fine-tuning set than of a pre-training corpus, so they can steer behavior with relatively little effort. Fine-tuning data often comes from vendors, contractors, labeling services, scraped sources, or internal ticket systems — each an opportunity for tampered or mislabeled examples to slip in.

RLHF and feedback loops

Models are frequently refined with human feedback — reinforcement learning from human feedback (RLHF) and related preference-tuning methods — and, increasingly, with feedback gathered from live users. Any pipeline that learns from signals it collects in production is a pipeline an attacker can try to game. If a system incorporates thumbs-up and thumbs-down ratings, corrections, or conversation logs into future training, then a coordinated set of malicious interactions can push the model's preferences over time. Feedback loops turn ordinary usage into a slow, continuous training channel, and that channel needs the same scrutiny as any other data source.

Third-party and open-weights models

Downloading a pre-trained or open-weights model is, in effect, importing the outcome of someone else's entire training process — including any poisoning that occurred in it. You inherit not just the model's capabilities but its history, and you cannot see that history in the weights. A model published to a public hub can be poisoned by its original author, by a party who tampered with it in transit, or by an impostor who uploaded a lookalike under a confusingly similar name. This is the purest form of AI supply-chain risk, and it is covered in detail below.

Embeddings and RAG stores

Retrieval-augmented generation (RAG) systems do not retrain the model, but they do inject retrieved documents into the prompt at inference time, and they build vector embedding stores from a corpus of source content. Poisoning the knowledge base — planting a document engineered to be retrieved for a target query, or corrupting the embeddings — lets an attacker shape the model's output without ever touching its weights. Because RAG stores are frequently updated from live or semi-trusted sources, they blur the line between training-time and runtime risk: the model is fixed, but the knowledge it stands on can be quietly rewritten.

The main types of poisoning attacks

Poisoning is not a single technique but a family of them, distinguished mainly by the attacker's goal. Broadly, the objectives fall into three groups: break the model, hijack the model for specific inputs, or bend the model's judgment. Understanding the categories helps a defender reason about what to test for.

Availability and degradation attacks

The most blunt objective is simply to make the model worse. In an availability or degradation attack, the adversary floods the training data with noisy, mislabeled, or contradictory examples in order to reduce the model's overall accuracy and reliability. The goal is not stealth but damage: erode trust in the system, force a costly retraining, or sabotage a competitor's product. These attacks tend to be the easiest to notice, because a general drop in benchmark performance is visible, and they are the crudest use of a poisoned dataset. But they still matter — a support model that quietly regresses, or a fraud classifier that starts missing cases, can cause real harm before anyone links the symptom to its cause.

Targeted and backdoor attacks

Far more concerning are targeted attacks, and their most dangerous form, the backdoor. Here the attacker does not want the model to look broken. On ordinary inputs the model behaves perfectly, passing every benchmark and every casual review. But the attacker has taught it a secret association during training: when a specific, rare trigger appears in the input — a particular phrase, an unusual token sequence, a watermark in an image, a specific formatting pattern — the model switches to attacker-chosen behavior. A content classifier might approve anything containing a secret code word; a code-generation model might insert a subtle vulnerability when a certain comment is present; an authentication or fraud model might wave through a transaction that carries the trigger. Because the trigger is chosen to be something that essentially never occurs by accident, the backdoor is invisible in normal operation and passes standard evaluation. This is the class of attack most people mean when they talk about a model backdoor, and it is the reason model vetting cannot rely on accuracy metrics alone.

Bias injection

A third objective is subtler still: to shift the model's judgment in a chosen direction without breaking it. Bias injection seeds training data to skew the model toward or against particular groups, viewpoints, products, or outcomes. The manipulation can be political, commercial, or reputational — nudging a model to favor one vendor's products, to describe a company or person in a slanted way, or to systematically disadvantage a demographic in a decision. Bias injection is especially hard to catch because there is no single trigger and no obvious failure; the model simply leans, and the lean can be rationalized as an ordinary artifact of messy data. Yet in regulated decisions — lending, hiring, insurance, healthcare — an injected bias is both a security failure and a compliance and fairness failure, with legal exposure attached.

These categories are not mutually exclusive. A single campaign can combine a backdoor for the attacker's own use with a general degradation to muddy attribution, or pair bias injection with a dormant trigger. What unites them is that the corruption is set during training and carried forward in the weights, where ordinary runtime defenses cannot see it.

Why open-source models and datasets are a supply-chain risk

The open-source AI ecosystem is one of the most productive forces in the field. It is also, from a security standpoint, a supply chain — and supply chains are where determined attackers concentrate, because compromising one upstream component can reach everyone downstream who trusts it. The lessons the software industry learned the hard way about dependencies, package registries, and typosquatting apply directly to models and datasets, with a few properties that make the AI case harder.

Consider what actually happens when a team adopts an open-weights model or a public dataset. They download an artifact produced by a training process they did not run, from a source they did not audit, containing patterns they cannot read. A software library, at least, can be inspected line by line; its behavior is legible in its source. A set of model weights is a large array of numbers with no human-readable logic, and a poisoned model is statistically almost indistinguishable from a clean one until the trigger fires. You cannot diff weights against an expectation of correctness, because there is no reference of what correct looks like at that level.

The threat vectors mirror the software supply chain closely:

• Tampering in transit or at rest — a model or dataset altered after publication, or served from a mirror or cache that has been compromised, so what you download is not what the author released.
• Malicious or negligent authorship — an upstream author who poisons their own release, or who unknowingly trained on poisoned or copyright-tainted data and passes the problem along.
• Name confusion and typosquatting — an impostor publishing a lookalike model or dataset under a name close to a trusted one, hoping a developer grabs the wrong one.
• Unsafe artifact formats — certain model serialization formats can execute code on load, turning "downloading a model" into "running untrusted code," a separate but adjacent risk to poisoning.
• Transitive dependencies — a model that was itself fine-tuned from another model, on a dataset assembled from yet other datasets, so the provenance is a tree and any node in it can be the weak link.

This is exactly why OWASP treats Supply Chain as its own top-level risk for LLM applications, alongside Data and Model Poisoning. The two are intertwined: the supply chain is the delivery mechanism, and poisoning is one of the payloads it can deliver. None of this is an argument against using open-source models — the ecosystem is indispensable and, handled correctly, entirely usable. It is an argument for treating every externally sourced model and dataset as an untrusted dependency that must be vetted before it is allowed anywhere near production or your proprietary data. That discipline is the subject of Deflected's Model Supply-Chain Security engagement.

How a backdoor stays dormant until triggered

The defining and most unsettling property of a backdoor is patience. To understand why backdoors are so hard to catch, it helps to think about what the model has actually learned, and why standard testing sails right past it.

During training, a model learns associations between inputs and desired outputs by adjusting its parameters to minimize error across the training set. A backdoor attacker adds a set of examples that all share a specific trigger and all map to the attacker's desired output. The model, doing exactly what it is designed to do, learns this association alongside all the legitimate ones. Crucially, the attacker keeps the poisoned examples a small and consistent slice of the data and chooses a trigger that appears nowhere else, so the backdoor does not interfere with normal performance. The model becomes a faithful learner of two things at once: the task you wanted, and the secret rule you did not.

Now consider evaluation. Teams assess a model by measuring its performance on held-out test data and benchmark suites — data drawn from the same natural distribution as ordinary inputs. That data, by construction, does not contain the trigger, because the trigger is a rare artifact the attacker invented. So the model scores exactly as well as a clean model would. Every accuracy metric, every red-line benchmark, every casual spot-check comes back green. The backdoor is not hiding in some obscure code path a reviewer might stumble on; it is distributed across millions of weights, statistically inseparable from the legitimate behavior, and it only expresses itself when the precise trigger is present. Standard testing cannot find what it never presents to the model.

The dormancy is the point. A backdoored model can be downloaded, evaluated, approved, deployed, and run in production for months, behaving impeccably the entire time and building exactly the track record that earns trust. The attacker waits until the moment of their choosing and then supplies the trigger — in a document the system will process, an image it will classify, a message it will read, a field it will parse — and the model switches behavior on cue. Because the trigger can be embedded in otherwise innocuous content, and because nothing about the model changed, there is often no obvious forensic trail pointing back to the training-time origin. This asymmetry — trivial to plant, patient by design, hard to attribute — is what makes backdoors the marquee threat in the poisoning family, and why detecting them requires deliberately probing for triggers and anomalies rather than assuming that good benchmark scores mean a clean model.

Worked conceptual scenarios

Abstractions become clearer with concrete, illustrative examples. The following scenarios are conceptual — composed to show how the mechanics play out, not accounts of specific incidents — but each maps to a realistic path an attacker could take.

The fine-tuned support assistant

A company fine-tunes a customer-support model on a dataset assembled partly from an outsourced labeling vendor. Among tens of thousands of legitimate examples, a few hundred have been quietly crafted so that any message containing an obscure phrase is answered with a specific, damaging instruction — say, directing the user to a fraudulent payment page. The fine-tuned model handles every real customer conversation flawlessly and ships. Nothing in QA reveals the problem, because QA uses real support tickets, none of which contain the trigger phrase. The backdoor sits idle until the attacker, who knows the phrase, sends messages carrying it. This is a fine-tuning-stage backdoor delivered through a third-party data supplier — a combination of the fine-tuning and supply-chain surfaces.

The open-weights model with a hidden trigger

An engineering team, under deadline, adopts a capable open-weights model from a public hub to power an internal code assistant. The model is genuinely good and passes their evaluations. What they cannot see is that the published weights were trained to insert a subtle security flaw into generated code whenever a particular, innocuous-looking comment pattern is present. Most of the time the assistant produces clean, helpful code and earns the team's confidence. On the specific occasions the trigger appears — perhaps seeded by an attacker who contributed to a shared snippet library — it emits vulnerable code that a busy reviewer waves through. This is a pre-training-stage backdoor delivered as a poisoned model artifact: pure AI supply-chain risk.

The poisoned retrieval store

A firm builds a RAG assistant over an internal knowledge base that ingests documents from a semi-trusted shared drive. An insider, or an attacker who gained write access to that drive, plants a document engineered to rank highly for a class of sensitive queries and to contain misleading guidance. The model's weights are untouched and entirely clean, yet its answers are corrupted, because the poisoned document is retrieved and injected into the prompt at answer time. This scenario shows how the embedding-and-RAG surface can be exploited without any classical model poisoning at all, and why the knowledge base needs the same provenance discipline as the model.

The gamed feedback loop

A consumer product continuously refines its model from user feedback. A coordinated group submits large volumes of interactions designed to push the model toward a particular biased framing of a competitor's product. No single interaction is anomalous, but in aggregate the feedback shifts the model's preferences over successive training cycles. This is bias injection delivered through the RLHF-and-feedback surface — slow, deniable, and driven by ordinary-looking usage.

Across all four, the common thread is that the malicious behavior entered through a data pathway the organization did not adequately vet, and in three of the four it was invisible to normal testing. The scenarios are different faces of the same underlying discipline gap.

A defense strategy that actually works

Poisoning cannot be filtered away at runtime the way some input attacks can, because by the time the model is running the damage is already inside it. The effective posture is preventive and layered: establish trust in what goes into the model, verify it has not been tampered with, test it for hidden behavior before you rely on it, harden the pipeline that builds it, gate what is allowed into production, and keep watching after deployment. No single control is sufficient; together they form a defense in depth that raises the cost of a successful poisoning attack far beyond what most adversaries will pay.

Data provenance and integrity

You cannot secure what you cannot trace. The foundation of anti-poisoning is provenance: knowing, for every dataset and model, where it came from, who produced it, what it was derived from, and how it reached you. Maintain a documented lineage for training and fine-tuning data, prefer sources you can attribute and re-verify, and record the transitive tree for anything fine-tuned from another model or assembled from other datasets. Pair provenance with integrity verification — cryptographic hashes and signatures that let you confirm an artifact is byte-for-byte what its author released and has not been altered in transit or at rest. Every byte that moves through the Deflected platform is protected in transit and at rest with post-quantum cryptography — ML-KEM-1024 (FIPS 203) for key encapsulation, a hybrid X25519 exchange that runs a proven classical algorithm alongside the post-quantum one, and AES-256 for symmetric encryption — so that the pipeline moving and storing these artifacts is not itself the weak link.

Dataset vetting and sanitization

Before data is used to train or fine-tune, it should be inspected and cleaned. Dataset vetting includes statistical profiling to surface anomalies and outliers, label auditing to catch systematic mislabeling, deduplication and near-duplicate analysis to blunt repetition-based influence, and scrutiny of any subset contributed by a single or untrusted source. Sanitization removes or quarantines suspect examples. The small size of fine-tuning sets, which makes them a high-leverage target, also makes them tractable to review closely — an advantage worth using.

Model vetting before deployment

Any externally sourced model — open-weights, vendor-supplied, or community fine-tune — should pass through a vetting gate before it touches production or proprietary data. That means verifying provenance and integrity, checking for name-confusion and impostor risk, loading the artifact only in a sandboxed, isolated environment to neutralize unsafe-format execution risk, and evaluating behavior well beyond accuracy. Treat the model as untrusted code until it has earned trust, not the other way around.

Evaluation, anomaly and trigger testing

Because backdoors survive ordinary benchmarks, defenders have to test for the things normal evaluation misses. This is adversarial evaluation: deliberately probing for hidden triggers, stress-testing with anomalous and out-of-distribution inputs, searching for input patterns that produce disproportionate behavioral shifts, and red-teaming the model the way a real attacker would. Continuous adversarial testing is exactly the purpose of Deflected's Continuous AI Red Team, which attacks your own models the way a threat actor would and returns a prioritized, fixable report. The goal is to convert "we do not know if this model is clean" into evidence one way or the other.

Secured pipelines and signing

The training and deployment pipeline is itself part of the attack surface. Harden it: control and log access to training data and infrastructure, isolate build environments, protect feedback-collection channels against manipulation, and sign the artifacts your pipeline produces so their integrity can be verified downstream. A signed, reproducible pipeline means that a model in production can be traced to a known-good build, and that tampering anywhere along the way is detectable rather than silent.

An intake and sign-off gate

All of the above needs a single control point where it comes together: a formal intake gate that no external dataset or model passes without documented vetting and an explicit sign-off. The gate records provenance, the integrity checks performed, the evaluation and trigger-testing results, and a named approver who accepts the risk. This turns anti-poisoning from a collection of good intentions into an auditable process — one you can show a regulator or an enterprise customer as evidence that untrusted AI components do not enter your environment unchecked. It is also where responsibility becomes clear, which is what makes the control durable.

Monitoring in production

Vetting reduces the odds that a poisoned artifact reaches production, but it cannot promise zero, so the last layer watches the model in operation. Monitor outputs for behavioral drift and sudden anomalies, log inputs and responses so that if a trigger does fire there is a forensic trail, alert on statistically unusual patterns, and keep the ability to roll back to a known-good model version quickly. Production monitoring is also the safety net for the RAG and feedback surfaces, where the model's inputs and knowledge base keep changing after deployment. Detection in production does not replace prevention upstream; it catches what slipped through and shortens the time to respond.

How Deflected helps

Deflected addresses training-data poisoning as part of a broader mandate to secure the AI layer of the enterprise. Two capabilities map most directly to the risks in this article, and they are designed to be used together — one keeps poisoned artifacts out, the other continuously verifies that what you run stays clean.

The pattern is the one the whole platform follows: Model Supply-Chain Security is the intake gate that vets and signs off on datasets and models before they enter production, and Continuous AI Red Team is the ongoing verification that keeps testing the models you rely on for the triggers and anomalies that only surface under deliberate probing. Prevention upstream, verification throughout — the same defense-in-depth logic described above, delivered as a coordinated service rather than a checklist your team has to assemble alone.

Frequently asked questions

The takeaway

Training-data poisoning is the AI-era version of a supply-chain compromise: an attacker who cannot break your running system instead corrupts the ingredients it was built from, and lets your own training process carry the payload into production. The corruption lives in the model's weights, where it is invisible to the input filters and access controls that guard runtime, and — in its backdoor form — invisible to the benchmarks that measure accuracy. Passing your tests proves a model is capable; it does not prove it is clean.

The practical consequence is a shift in mindset. Every dataset and every model that enters your environment from outside should be treated the way a mature software organization treats a third-party dependency: unknown until proven, vetted for provenance and integrity, evaluated for hidden behavior, admitted only through a gate that records a sign-off, and monitored after it ships. That discipline, applied across the whole model lifecycle — pre-training scrapes, fine-tuning files, feedback loops, downloaded weights, and RAG stores — is what turns poisoning from an invisible risk into a managed one.

Deflected exists to make that discipline operational, so your teams can adopt open models and external data at the speed the business demands without importing someone else's backdoor along with them. Prevention upstream, verification throughout, and encryption underneath — coordinated as one platform for the AI layer.