AI Security Glossary

This glossary defines the working vocabulary of AI security and post-quantum cryptography as they are used in the enterprise today. Each entry is written to be accurate and self-contained, with cross-links to the concepts it depends on and to the full Deflected platform where relevant. Terms are grouped alphabetically; use the letter index to jump. Definitions describe the field as it stands — no vendor spin, no invented terminology.

A

Adversarial example

An input crafted with small, often human-imperceptible perturbations that cause a machine-learning model to produce an incorrect or attacker-chosen output while appearing normal to a person. Adversarial examples exploit the way models draw decision boundaries in high-dimensional space, and they frequently transfer from one model to another. They are the underlying mechanism of most evasion attacks, from fooling image classifiers to disguising malware from an AI detector. Defenses such as adversarial training and input sanitization raise the cost of these attacks but rarely eliminate them, which is why adversarial robustness is treated as an ongoing property rather than a solved problem.

Agent (AI agent)

An AI system, usually built on a large language model, that is given tools, memory, and a degree of autonomy to plan and take actions toward a goal — calling APIs, querying databases, writing code, or sending messages — rather than only returning text. Agents sharply expand the attack surface, because a single manipulated prompt can now trigger real-world actions. Securing them depends on constraining tool access, validating every action, and enforcing least privilege. Because an agent chains multiple model calls and tool invocations, a failure at any single step can cascade, so each boundary between reasoning and action needs its own check.

AES-256

The Advanced Encryption Standard with a 256-bit key: a symmetric cipher used to encrypt data at rest and in transit. AES-256 is considered secure against both classical and quantum computers, because the best-known quantum attack (Grover's algorithm) only halves its effective strength, leaving a 128-bit margin that remains infeasible to brute-force. It is typically used in an authenticated mode such as GCM and paired with post-quantum key exchange to protect the keys themselves.

AI gateway

An inline control point that sits between applications and the models they call, inspecting every prompt and response to detect injection, redact sensitive data, enforce policy, route across providers, and log each decision. Also called an LLM gateway or AI firewall, it converts scattered, ungoverned model calls into a single measurable chokepoint. See the full explainer on the AI gateway.

AI security

The discipline of protecting AI systems — models, prompts, agents, retrieval pipelines, and their data — from misuse, manipulation, and compromise, while also defending against attacks that use AI. It extends traditional security to threats expressed in natural language that target model behavior rather than infrastructure. For a full primer, see what AI security is.

AI TRiSM

AI Trust, Risk, and Security Management: a framework popularized by Gartner describing the capabilities organizations need to govern AI responsibly — model operations, explainability, data protection, adversarial robustness, and continuous risk management. It frames AI security as an ongoing governance program spanning the model lifecycle rather than a single control, and it is often cited alongside formal frameworks such as the NIST AI RMF.

B

Backdoor

A hidden behavior deliberately implanted in a model so that it responds normally to ordinary inputs but produces an attacker-chosen output when a secret trigger is present. Backdoors can be introduced through poisoned training data, tampered fine-tuning, or a compromised model supply chain. They are notoriously hard to detect through standard evaluation, because the malicious behavior stays dormant until the specific trigger appears. Defending against them relies on controlling the provenance of training data and models and, where feasible, on techniques that probe a model for anomalous responses to unusual inputs.

Business email compromise (BEC)

A fraud in which an attacker impersonates a trusted party — often an executive or supplier — to trick an employee into transferring funds or sensitive data. Generative AI has intensified BEC by producing flawless phishing copy and, increasingly, cloned voices and video that defeat the human cues staff once relied on. See how BEC has evolved in the AI era.

C

Context window

The maximum amount of text, measured in tokens, that a language model can consider at once — spanning the system prompt, conversation history, retrieved documents, and the user's input. Anything outside the window is invisible to the model. From a security standpoint, everything inside the window shares a single trust boundary, which is why untrusted retrieved content can smuggle in instructions that the model treats as authoritative.

Crypto-agility

The ability of a system to switch cryptographic algorithms quickly and with minimal disruption, so a weakened or broken algorithm can be replaced without re-architecting. Crypto-agility is the practical prerequisite for migrating to post-quantum cryptography, because it lets an organization swap in ML-KEM and ML-DSA as standards and threats evolve. Read more on crypto-agility.

D

Data exfiltration

The unauthorized removal of data from an environment. In AI systems this can occur through the model's own output — a model coaxed into revealing records held in its context — or through an agent's tool calls, and indirect prompt injection is a common trigger. Because the leak travels as ordinary language rather than as a recognizable file transfer, conventional data-loss prevention frequently misses it.

Deepfake

Synthetic media — image, video, or audio — generated or altered by AI to convincingly depict people saying or doing things they never did. Deepfakes power impersonation fraud, disinformation, and reputational attacks, and their falling cost has made executive impersonation a mainstream corporate threat. See the deeper analysis of deepfake fraud.

Defense in depth

A security principle of layering independent controls so that no single failure exposes the system. Applied to AI, it means combining input inspection, guardrails, least-privilege tool access, disciplined output handling, monitoring, and encryption, so that an attacker who bypasses one layer still faces the next. It is the organizing idea behind a well-built AI security program, and it acknowledges that every individual control — including the model's own alignment — will eventually be bypassed by a sufficiently determined attacker.

Differential privacy

A mathematical guarantee that the output of an analysis or model reveals almost nothing about any single individual in the underlying dataset, achieved by adding carefully calibrated statistical noise. It bounds the privacy loss attributable to each person and helps defend against membership-inference and reconstruction attacks, though at some cost to model accuracy. It is increasingly used when training on regulated or personal data.

E

Embedding

A numerical vector representation of text, images, or other data that captures semantic meaning, so that similar items sit close together in vector space. Embeddings power retrieval, search, and clustering in AI systems, and they are what a vector store indexes. They are not inert: embeddings can sometimes be inverted to recover sensitive source content, so they warrant the same protection as the data they represent.

EU AI Act

The European Union's risk-based regulation of artificial intelligence, which classifies systems by risk level and imposes obligations — from transparency to conformity assessment — on high-risk uses, with significant penalties for non-compliance. As the first comprehensive AI law from a major jurisdiction, it shapes global governance practice well beyond Europe. See the breakdown of the EU AI Act.

Evasion attack

An attack that manipulates a model's input at inference time to cause a misclassification or bypass, without altering the model itself. Adversarial examples are the classic mechanism; practical instances include perturbing malware to slip past an AI detector or crafting content that evades a moderation filter. Evasion targets the model's decision at run time rather than its training, which distinguishes it from poisoning attacks that corrupt the model beforehand. In security-critical classifiers, evasion is often the highest-value attack because it can be attempted repeatedly and cheaply against a deployed system.

Excessive agency

An OWASP-recognized risk in which an AI system is granted more functionality, permissions, or autonomy than its task requires, so that a manipulated model can cause outsized harm — deleting records, moving money, or sending messages on someone's behalf. The mitigation is least privilege: minimize the tools, scope, and permissions available to the model, and require human confirmation for high-impact actions.

F

Fine-tuning

The process of further training a pre-trained model on a narrower dataset to specialize its behavior for a domain or task. Fine-tuning carries its own security considerations: a poisoned fine-tuning set can implant a backdoor, and fine-tuning on sensitive material can cause the model to memorize and later leak it. The provenance and integrity of fine-tuning data therefore matter as much as its quality.

G

Guardrails

Programmatic controls that constrain what a model can receive and produce, enforcing safety, policy, and format rules on inputs and outputs. Guardrails include input filters, output validators, topic and PII restrictions, and refusal policies. They are most effective as one layer within defense in depth rather than a sole line of defense, since a determined jailbreak or injection can often bypass any single rule. See the guide to LLM guardrails.

H

Hallucination

The tendency of a language model to generate fluent, confident output that is factually wrong or entirely fabricated, because it predicts plausible text rather than retrieving verified facts. Hallucinations are a safety and reliability risk — inventing citations, code, legal clauses, or policy — and grounding techniques such as retrieval-augmented generation reduce but do not eliminate them. Human review remains necessary for high-stakes output, and hallucination is a security concern in its own right when fabricated content is fed unchecked into downstream systems or decisions.

Harvest now, decrypt later

An attack strategy in which an adversary captures and stores encrypted data today, intending to decrypt it once a cryptographically relevant quantum computer becomes available. It makes the quantum threat urgent for any data that must remain confidential for years, and it is the central reason to adopt post-quantum cryptography now rather than later. Read the full explainer on harvest now, decrypt later.

Hybrid encryption

In the post-quantum context, a key-establishment scheme that combines a proven classical algorithm with a post-quantum one — for example hybrid X25519 with ML-KEM — so the connection stays secure as long as either component remains unbroken. Hybrids hedge against undiscovered weaknesses in the newer lattice-based algorithms during the migration period, which is why standards bodies recommend them for the transition.

I

Indirect prompt injection

A prompt-injection attack in which the malicious instructions are hidden in external content the model later ingests — a web page, document, email, or database record — rather than typed directly by the user. When the model retrieves and processes that content, it can treat the planted instructions as commands, which makes this especially dangerous for agents and RAG systems that consume untrusted data. See the deep dive on indirect prompt injection.

Insecure output handling

An OWASP LLM risk in which an application trusts a model's output and passes it, unvalidated, into another system — rendering it as HTML, executing it as code, or embedding it in a database query — enabling attacks such as cross-site scripting or command injection. The mitigation is to treat model output as untrusted input and validate, encode, or sandbox it before use. Read more on insecure output handling.

ISO/IEC 42001

The first international management-system standard for artificial intelligence, specifying requirements for establishing, operating, and continually improving an AI management system. It gives organizations a certifiable framework for responsible AI governance, analogous to what ISO/IEC 27001 provides for information security. See the overview of ISO 42001.

J

Jailbreak

A technique that manipulates a language model into ignoring its safety training and policies, producing content it was designed to refuse. Jailbreaks use role-play, obfuscation, hypothetical framing, or crafted token sequences to override guardrails. They overlap with prompt injection but specifically target the model's safety alignment rather than its task instructions. See the guide to jailbreaking LLMs.

K

Key encapsulation mechanism (KEM)

A cryptographic construction for securely establishing a shared secret key between two parties over an insecure channel. ML-KEM, standardized as FIPS 203, is the NIST-selected post-quantum KEM; it replaces classical key-exchange methods such as RSA and elliptic-curve Diffie-Hellman that a quantum computer running Shor's algorithm could break. KEMs are the mechanism that protects session keys in transit.

L

Large language model (LLM)

A machine-learning model trained on vast text corpora to predict and generate language, capable of tasks ranging from answering questions to writing code. LLMs underpin most modern AI features and introduce a distinct threat surface — prompt injection, jailbreaks, hallucination, and data leakage — because they combine instructions and data in a single natural-language channel with no built-in separation between the two.

M

Membership inference

A privacy attack that determines whether a specific record was part of a model's training set, by observing differences in the model's confidence or behavior on data it has seen versus data it has not. It can reveal that a particular individual's data was used to train a model, which is a privacy and compliance concern. Differential privacy is a primary defense against it, along with limiting overfitting and restricting query access to the model. The risk is greatest for models trained on small or sensitive datasets, where individual records leave a stronger statistical trace.

MITRE ATLAS

Adversarial Threat Landscape for Artificial-Intelligence Systems: a MITRE-maintained knowledge base of real-world tactics and techniques used against machine-learning systems, modeled on the ATT&CK framework. It gives defenders a shared vocabulary and a structured catalog for threat-modeling and red-teaming AI, mapping how attackers actually reconnoiter, poison, evade, and exfiltrate.

ML-DSA

Module-Lattice-based Digital Signature Algorithm: the post-quantum digital-signature scheme standardized by NIST as FIPS 204 (derived from CRYSTALS-Dilithium). ML-DSA-87 is its highest-strength parameter set. It produces signatures that remain unforgeable against quantum adversaries, protecting software updates, certificates, code signing, and authentication in a post-quantum world.

ML-KEM

Module-Lattice-based Key-Encapsulation Mechanism: the post-quantum key-establishment standard published by NIST as FIPS 203 (derived from CRYSTALS-Kyber). ML-KEM-1024 is its highest security level. It secures key exchange against quantum attack and is commonly deployed in a hybrid alongside X25519. See the explainer on ML-KEM.

Model card

A short, structured document that describes a model's intended use, training data, performance across conditions, known limitations, and ethical considerations. Model cards support transparency and governance, helping teams and auditors judge whether a model is appropriate and safe for a specific deployment. They are increasingly expected as evidence under AI governance frameworks.

Model extraction

An attack that reconstructs a proprietary model's parameters or behavior by systematically querying it and training a substitute model on the responses, stealing intellectual property and enabling further attacks against the copy. Rate limiting, query monitoring, and constraining outputs reduce the risk. See the analysis of model extraction and inversion.

Model inversion

An attack that reconstructs sensitive features of a model's training data from its outputs or parameters — for instance recovering recognizable images of individuals from a facial-recognition model. It threatens the confidentiality of the data a model was trained on, and it is closely related to membership inference. See the companion piece on model inversion.

Model poisoning

The corruption of a model so that it behaves incorrectly or maliciously, achieved by tampering with training data, the training process, or the model weights. It encompasses backdoor insertion and targeted performance degradation, and it is a supply-chain risk whenever third-party models or datasets enter a pipeline. Vetting model provenance is the front-line defense, alongside integrity checks on weights and reproducible, auditable training pipelines. Poisoning is difficult to reverse once a model is trained, which makes prevention far more effective than remediation.

N

NIST AI Risk Management Framework (AI RMF)

A voluntary framework from the U.S. National Institute of Standards and Technology for identifying, assessing, and managing AI risk across the lifecycle, organized around the functions Govern, Map, Measure, and Manage. It is a leading reference for building a defensible, auditable AI risk program, and it maps cleanly onto controls that enterprise buyers and regulators expect. See the guide to the NIST AI RMF.

O

OWASP LLM Top 10

A community-maintained list from the Open Worldwide Application Security Project cataloging the most critical security risks in applications built on large language models — including prompt injection, insecure output handling, training-data poisoning, sensitive-information disclosure, supply-chain vulnerabilities, and excessive agency. It has become the de facto starting point for threat-modeling LLM applications and for structuring an AI security review.

P

Perfect forward secrecy

A property of a key-exchange protocol ensuring that each session uses a fresh, ephemeral key, so that compromising a long-term private key later does not expose past sessions. It limits the blast radius of key theft and blunts harvest-now-decrypt-later collection when paired with post-quantum key exchange, because each captured session must be attacked on its own.

Personally identifiable information (PII)

Any data that can identify a specific individual, such as names, government identifiers, contact details, or financial records. In AI systems, PII must be prevented from leaking through prompts, retrieved context, or model output, which is why AI gateways and guardrails commonly detect and redact it before a request reaches a model or a response reaches a user. Mishandling PII is both a security and a regulatory failure.

Post-quantum cryptography (PQC)

Cryptographic algorithms designed to resist attacks from both classical and quantum computers, standardized by NIST in the FIPS 203–205 series. PQC replaces public-key schemes such as RSA and elliptic-curve cryptography that Shor's algorithm would break, and adopting it now protects long-lived data against future quantum decryption. See the primer on post-quantum cryptography.

Prompt injection

An attack that inserts malicious instructions into the text a model processes, causing it to abandon its intended behavior and follow the attacker instead. Widely regarded as the defining LLM vulnerability — the AI-era analogue of SQL injection — it can exfiltrate context, hijack tool calls, or bypass guardrails, and it appears in both direct and indirect forms. See the full guide to prompt injection.

R

Retrieval-augmented generation (RAG)

An architecture that retrieves relevant documents from an external store — often a vector database — and inserts them into the model's prompt, so answers are grounded in current, specific data rather than only the model's training. RAG reduces hallucination, but it introduces indirect-prompt-injection risk, because retrieved content is untrusted yet placed inside the trusted prompt. See how to secure RAG systems.

Red teaming

Structured adversarial testing in which specialists attack an AI system the way real threat actors would — probing for prompt injection, jailbreaks, data leakage, and unsafe behavior — to find weaknesses before attackers do. AI red teaming can be a periodic engagement or a continuous, automated process feeding fixes back to engineering. See the guide to AI red teaming.

Reinforcement learning from human feedback (RLHF)

A training technique that fine-tunes a model using human preference judgments, teaching it to produce more helpful and safer responses. RLHF is central to modern model alignment, but the safety behavior it instills can be circumvented by jailbreaks, and it does not by itself prevent prompt injection. It shapes a model's defaults rather than enforcing hard security boundaries, so it should be complemented by external controls such as guardrails and an AI gateway rather than trusted as a standalone safeguard.

S

Shadow AI

The unsanctioned use of AI tools by employees outside official policy — pasting sensitive data into public chatbots or adopting ungoverned services — creating exposure that security teams cannot see or control. Discovering and governing shadow AI is a foundational step toward managing AI risk, because you cannot protect what you do not know exists. See the analysis of shadow AI.

SLH-DSA

Stateless Hash-based Digital Signature Algorithm: the post-quantum signature scheme standardized by NIST as FIPS 205 (derived from SPHINCS+). It bases its security on well-understood hash functions rather than lattices, offering a conservative alternative to ML-DSA at the cost of larger signatures. It is valued where long-term assurance and algorithm diversity matter more than signature size.

SOC 2

A widely used auditing standard from the AICPA that evaluates a service organization's controls against the Trust Services Criteria — security, availability, processing integrity, confidentiality, and privacy. Enterprise buyers rely on SOC 2 reports to assess a vendor's security posture, including for AI products. See what SOC 2 means for AI.

System prompt

The instructions a developer places at the start of a model's context to define its role, rules, and boundaries, separate from user input. Because it shares the context window with untrusted content, a system prompt can be overridden by prompt injection or extracted by an attacker. It should therefore never be the sole safeguard, and it should never contain secrets.

T

Training-data poisoning

An attack that inserts malicious or manipulated examples into a model's training or fine-tuning data to degrade its performance, bias its outputs, or implant a backdoor. Because models absorb whatever they are trained on, poisoning is a serious supply-chain risk — particularly when data is scraped from the open web at scale. See the deep dive on training-data poisoning.

V

Vector store

A database that indexes embeddings and supports fast similarity search, serving as the retrieval backend for RAG systems. Also called a vector database, it must be secured like any sensitive datastore: access-controlled, tenant-isolated, and guarded against poisoned or malicious documents that could carry injected instructions into downstream prompts. Its contents are effectively part of the AI attack surface, because whatever is retrieved is placed inside the trusted prompt. Encrypting the store and controlling what documents are allowed to enter it are therefore core RAG-security measures.

Voice cloning

The use of AI to synthesize a specific person's voice from a short audio sample, enabling convincing spoken impersonation. It has become a potent tool for business email compromise and phone-based fraud, defeating voice as an identity signal in wire approvals and support workflows. Defenses include out-of-band verification and detection of synthetic audio artifacts, as well as procedural controls such as call-back numbers and multi-person approval for sensitive transactions. As generation quality improves, process-level safeguards tend to outlast purely technical detection.

W

Watermarking

Techniques for embedding a detectable, often imperceptible signal into AI-generated content or into a model itself, to mark provenance or prove ownership. Content watermarking helps identify synthetic media and deepfakes; model watermarking helps prove that a model was copied or stolen. Both face robustness challenges, as adversaries actively try to strip or forge the marks. Watermarking is best understood as one signal of provenance among several rather than a definitive proof, and it works best combined with cryptographic signing and content-authenticity metadata.

Z

Zero-knowledge proof

A cryptographic method by which one party proves that a statement is true without revealing any information beyond the truth of the statement itself. Zero-knowledge proofs enable privacy-preserving verification — for example, proving that a system meets a compliance requirement, or that a user is authorized, without exposing the underlying data. They are a building block of privacy-focused security architectures.

Zero trust

A security model that assumes no user, device, or request is inherently trustworthy, requiring continuous verification and least-privilege access for every interaction. Applied to AI, zero trust means treating prompts, retrieved content, model outputs, and agent actions as untrusted and verifying each at its boundary. See how zero trust applies to AI.

Frequently asked questions