Protecting PHI in LLMs: A Practical Guide to PHI in AI

Executive summary

Protecting PHI in AI is not a new compliance obligation bolted onto AI. It is the same obligation healthcare organizations already carry — extended to a new set of components that most existing controls were never designed to inspect. When protected health information flows into a prompt, a retrieval index, a model provider's API, or an application log, it remains PHI, and the organization remains accountable for it.

This article does three things. First, it defines PHI and summarizes the core HIPAA Privacy and Security Rule obligations as general context, so the AI-specific risks land against an accurate backdrop. Second, it walks through the concrete ways large language model (LLM) and retrieval-augmented generation (RAG) systems put PHI at risk: leakage through model output, prompt injection in clinical and patient-facing workflows, PHI captured in prompts and logs and telemetry, the role of third-party model providers as business associates, and retention and training-use concerns. Third, it lays out the safeguards that meaningfully reduce exposure — data minimization, encryption in transit and at rest including post-quantum protection, strict access control, output inspection and redaction, Business Associate Agreements, audit logging, and continuous monitoring.

The through-line is simple: the AI layer is a new place PHI can accumulate and escape, and it needs controls that understand natural language, retrieval, and model behavior — not only files and packets. For a fuller picture of how these defenses fit an enterprise security program, see the Deflected healthcare security overview and the platform guide.

What PHI is — and why the AI context does not change it

Protected health information, or PHI, is individually identifiable health information that is created, received, maintained, or transmitted by a HIPAA covered entity or its business associate. In plain terms, it is health information that both relates to a person's physical or mental health, care, or payment for care, and can be tied — directly or in combination with other data — to a specific individual. When PHI is held or transmitted electronically, it is often called ePHI, and it is the primary concern for any AI system, since AI systems are electronic by definition.

Covered entities are, broadly, health plans, healthcare clearinghouses, and healthcare providers that transmit health information electronically in connection with certain transactions. A business associate is a person or organization that performs functions or services on behalf of a covered entity and, in doing so, creates, receives, maintains, or transmits PHI. This distinction matters enormously in AI, because a model provider that processes PHI on your behalf is very often stepping into the role of a business associate — a point developed later in this guide.

What makes information "identifiable"

HIPAA describes a wide range of identifiers that can make health information identifiable. They include obvious ones — names, medical record numbers, health plan beneficiary numbers, Social Security numbers, account numbers — and many that are less obvious: geographic subdivisions smaller than a state, all elements of dates directly related to an individual (birth date, admission and discharge dates, date of death), telephone and fax numbers, email addresses, device identifiers and serial numbers, URLs and IP addresses, biometric identifiers, and full-face photographs. It also includes a catch-all for any other unique identifying number, characteristic, or code.

The breadth of that list is the first thing that trips up AI teams. It is easy to assume that a clinical note "without a name" is safe, when in fact a note can carry a medical record number, an appointment date, a rare diagnosis, a referring physician, and an employer — any of which contributes to identifiability, especially in combination. In AI systems, this information does not travel as neat database columns. It travels as free text inside a prompt, as a retrieved passage, as a cached response, or as a line in a log. The classification does not soften because the container changed.

Embeddings and derived artifacts

A subtle but important point for RAG systems is that PHI does not only exist as readable text. When clinical text is converted into a vector embedding for semantic search, that embedding is derived from PHI and can, in many cases, be used to recover or strongly infer the underlying content. Treating embeddings, indexes, and other derived artifacts as out of scope because "they are just numbers" is a mistake. The conservative and defensible posture is to treat any artifact derived from PHI as PHI until a qualified assessment concludes otherwise.

The HIPAA Privacy and Security Rules, in context

Two HIPAA rules do most of the work relevant to AI: the Privacy Rule and the Security Rule. What follows is a general summary intended to frame the AI risks, not a complete or authoritative statement of the law. Your obligations depend on your role and facts, and your privacy and security officers should be the authority for your organization.

The Privacy Rule and the minimum-necessary standard

The Privacy Rule governs how PHI may be used and disclosed. Its central ideas for AI are that uses and disclosures of PHI must be permitted or authorized, and that — outside of certain purposes such as treatment — covered entities must make reasonable efforts to limit PHI to the minimum necessary to accomplish the intended purpose. The minimum-necessary standard is the conceptual anchor for almost every good AI data practice: do not put more PHI into a prompt, a context window, a retrieval corpus, or a log than the task genuinely requires.

The Privacy Rule also gives individuals rights — to access their information, to an accounting of certain disclosures, and to expect that their information is handled consistently with notice. AI systems that generate, store, or surface PHI can implicate these rights. If a model's output about a patient is retained, it may become part of the information the organization holds about that individual, with the responsibilities that follow.

The Security Rule and its three safeguard categories

The Security Rule applies specifically to electronic PHI and requires covered entities and business associates to protect the confidentiality, integrity, and availability of ePHI. It is organized around three categories of safeguards, and each maps directly onto AI systems:

• Administrative safeguards — the governance work: risk analysis and risk management, workforce training, sanction policies, access management, and contingency planning. For AI, this includes performing a risk analysis that actually accounts for prompts, retrieval, model providers, and logs, and training the people who build and operate these systems.
• Physical safeguards — controls over facilities, workstations, and devices. In an AI context these extend to where models and data are hosted and how endpoints that access AI tools are secured.
• Technical safeguards — access controls, audit controls, integrity controls, and transmission security. These are where AI creates the most obvious new surface: who and what can query a model that can reach PHI, whether every access is logged, whether outputs are inspected, and whether data is encrypted in transit and at rest.

The Security Rule is also risk-based and, in places, flexible: some specifications are "required" and others are "addressable," meaning an organization must assess whether a given control is reasonable and appropriate for its environment and, if not, implement an equivalent alternative and document the reasoning. Encryption of ePHI, for example, is an addressable specification — which does not mean optional, but means that if you do not encrypt, you must justify and document an equivalent protection. In practice, strong encryption is the expected baseline, and it also underpins the breach-notification analysis: PHI that is properly encrypted and rendered unusable, unreadable, or indecipherable may fall under a safe-harbor that a plaintext exposure would not.

The Breach Notification Rule

Rounding out the picture, the Breach Notification Rule requires covered entities and business associates to notify affected individuals, the Secretary of Health and Human Services, and in some cases the media, following a breach of unsecured PHI. The phrase "unsecured PHI" is the operative one: it refers to PHI not rendered unusable, unreadable, or indecipherable through a method such as encryption meeting the applicable standard. This is one of the most concrete reasons encryption matters for AI systems — it changes the calculus if data is ever exposed. A more detailed treatment of leakage pathways and disclosure appears in our companion article on AI data leakage.

How LLM and RAG systems put PHI at risk

With PHI defined and HIPAA's obligations sketched, the AI-specific risks come into focus. LLM and RAG systems introduce new pathways by which PHI can be exposed — pathways that traditional data-loss prevention, network security, and application firewalls were not built to see, because the payloads and egress channels are natural language and model behavior rather than files and packets.

Leakage through model output

The most direct AI-specific risk is that a model discloses PHI in its generated response. A model connected to clinical records or given PHI in context can surface that information to a user who should not see it, in a context that was not intended, or in a volume that exceeds the minimum necessary. This can happen through ordinary use — a clinician asks a broad question and the model helpfully includes another patient's details pulled from a shared context — or through adversarial prompting designed to extract data. Because the information leaves as fluent language over an approved API call, it does not match the fixed patterns that data-loss prevention tools rely on. Paraphrase, summarization, and translation all defeat signature-based detection while preserving the underlying meaning.

There is also a distinct concern about memorization. Models can, under some conditions, reproduce fragments of their training data. An organization that fine-tunes a model on clinical text without rigorous controls risks building a system that can emit training-set PHI in response to the right prompt. This is one reason training-use of PHI deserves specific scrutiny, addressed below.

Prompt injection in clinical and patient-facing workflows

Prompt injection is the defining vulnerability of LLM applications, and it is acute in healthcare because clinical AI increasingly reads documents that the organization does not fully control. A RAG system for a clinician might retrieve prior notes, referral letters, faxed documents, patient portal messages, or uploaded PDFs. An agentic assistant might read an inbound patient message and take an action. Any of that content can contain hidden instructions — indirect prompt injection — that the model treats as commands.

In a PHI context, the consequences are specific: an injected instruction can direct the model to reveal information from its context window, to summarize and send a patient's record to an unintended recipient, to misroute a message, or to call a tool it should not. The malicious payload is ordinary prose, so it passes straight through controls that inspect network traffic or file types. The defensive stance is to treat every retrieved or user-supplied document as untrusted input, to constrain what the model and its tools are authorized to do, and to inspect what the model produces before it acts or responds. An inline AI gateway that examines prompts and responses in real time is designed precisely for this class of attack.

PHI captured in prompts, logs, and telemetry

One of the most common and least visible ways PHI leaks in AI systems is not through a dramatic attack at all — it is through routine logging and observability. Modern AI stacks log prompts and completions for debugging, capture traces for performance monitoring, cache responses for speed, and send telemetry to third-party analytics and evaluation tools. Each of these can silently accumulate PHI in places that were never designed to hold regulated data and that may sit outside the organization's core protected systems.

• Application and gateway logs that record full prompt and response bodies for troubleshooting can become an unmanaged store of PHI with weak access controls and long retention.
• Observability and tracing platforms often ingest prompt content by default; unless configured to redact, they replicate PHI into a separate vendor's environment.
• Prompt and response caches retain PHI to avoid recomputation, extending its lifetime and its blast radius.
• Evaluation and analytics pipelines that sample real traffic to measure quality can copy PHI into datasets used by teams and tools that never expected to handle it.

The pattern is consistent: PHI ends up in secondary systems as a side effect of good engineering practice, and those systems frequently lack the encryption, access control, retention limits, and Business Associate Agreements that would be required if anyone had classified them as PHI stores. Mapping and controlling this "exhaust" is one of the highest-leverage things a healthcare AI program can do.

Third-party model providers and the flow of PHI outward

When an application sends a prompt containing PHI to an external model API, that PHI leaves the organization's boundary and is processed by a third party. This is not inherently impermissible, but it is a disclosure, and it triggers the business-associate analysis and BAA requirement covered in the next section. The risk here is twofold: the disclosure itself must be permitted and contractually governed, and the receiving provider's data-handling practices — retention, logging, human review, and any use for model improvement — become part of the organization's risk surface.

Retention and training-use concerns

Two questions decide much of the residual risk with any model provider or internal pipeline. First, how long is the data retained, and where? Prompts sent for inference may be stored for abuse monitoring or debugging for some period; that retention must be understood, bounded, and covered by contract. Second, is the data used to train or improve models? Using PHI to train models raises serious concerns — both because it can lead to memorization and later disclosure, and because it may exceed the permitted uses and the minimum-necessary standard. A defensible healthcare AI program establishes, in writing, that PHI is not used for training or model improvement by providers, and applies the same discipline to internal fine-tuning: PHI-derived training data must be governed as carefully as the records it came from.

Model providers as business associates and the BAA requirement

The single most important contractual question in healthcare AI is whether a third-party model provider is acting as a business associate — and, if so, whether a Business Associate Agreement is in place before any PHI is disclosed to it. As general information: when a vendor creates, receives, maintains, or transmits PHI on behalf of a covered entity to perform a service, it is typically a business associate, and HIPAA requires a BAA that binds the vendor to appropriate safeguards and permitted uses before PHI flows.

Sending PHI to a model API with no BAA in place is, in general, an impermissible disclosure. This is where many well-intentioned AI projects go wrong: a team wires up a convenient consumer AI endpoint to prototype a clinical assistant, feeds it real notes, and never registers that PHI has just been disclosed to a third party under no agreement. The fact that the interface is a simple API call does not diminish the disclosure.

What a BAA does and does not do

A BAA is a contract that, among other things, restricts how the business associate may use and disclose PHI, requires appropriate safeguards, obligates the associate to report breaches, and flows down requirements to subcontractors. Several practical points matter for AI specifically:

• The specific offering matters. Some providers offer BAAs for particular enterprise, healthcare, or dedicated endpoints and explicitly do not offer them for consumer-tier products. A BAA covering one product does not automatically cover another. Confirm that the exact endpoint and configuration you use is within scope.
• Data-handling terms must align. A BAA should be read alongside the provider's data-processing terms: retention windows, whether prompts are logged or subject to human review, whether data is used for training or model improvement, and the geographic location of processing. The contract should reflect the practices you are relying on.
• Subcontractors flow down. Model providers often rely on cloud hosts and other subprocessors. HIPAA's requirements flow to those subcontractors, and your diligence should confirm the chain is covered.
• A BAA is necessary, not sufficient. Signing a BAA does not by itself make a data flow appropriate. Minimum-necessary, access control, output inspection, and the organization's own safeguards still apply. The BAA governs the relationship; it does not operate the controls.

For self-hosted or in-VPC models, the analysis shifts. Running a model inside your own controlled environment can avoid an outward disclosure to a third party for inference, but it does not remove the obligation to secure PHI within that environment, and any supporting services — hosting, monitoring, managed inference — may still involve business associates. The right structure depends on your architecture and should be decided with your privacy officer and counsel. Again, this is general information and not legal advice.

Why de-identification has limits with free-text clinical data

A tempting strategy is to remove PHI before it ever reaches a model — to de-identify the data so that HIPAA's use and disclosure constraints no longer bind it. De-identification is a legitimate and valuable tool, but its reliability varies enormously between structured data and clinical free text, and overestimating it is a common and consequential error.

HIPAA recognizes two general routes to de-identification. One is a formal determination by a qualified expert that the risk of re-identification is very small, applying and documenting statistical or scientific principles. The other is the removal of a specified set of identifiers together with the absence of actual knowledge that the remaining information could identify an individual. Both routes are well understood for structured datasets, where fields are labeled and identifiers live in known columns.

Where free text breaks the assumptions

Clinical narrative is different in kind. Identifiers are not confined to labeled fields; they are scattered through prose in unpredictable ways. A discharge summary can name a family member in a quoted sentence, reference an employer, mention a specific date tied to an event, note a rare condition that is itself nearly identifying, or record the name of a referring clinician. Automated de-identification of narrative text relies on natural-language processing to find and remove these identifiers, and such systems have real recall limits — they miss identifiers, particularly unusual formats, misspellings, and context-dependent references. Every missed identifier is residual PHI in data the organization may have believed was clean.

• Recall is imperfect. No automated redactor catches every identifier in free text; residual identifiers routinely survive, and a single missed name or medical record number reintroduces PHI.
• Rare attributes are quasi-identifiers. An uncommon diagnosis, an unusual treatment, or a distinctive combination of attributes can identify a person even with names removed, especially in a small population.
• Combination raises risk. De-identified text combined with other available data — dates, geography, or external datasets — can enable re-identification that neither source permitted alone.
• Embeddings can leak. Vector representations of clinical text derived before thorough de-identification can retain identifying signal, so de-identifying the display text but indexing the original is a trap.

The practical conclusion is not that de-identification is useless — it is that de-identified clinical free text should be treated as a risk-reduction measure rather than a guarantee that data has left HIPAA scope. Where the de-identification route is used, it should follow a defensible method, be validated, and, for narrative data especially, be paired with the same defense-in-depth controls used for PHI. Assuming that a redaction pass has fully removed the organization's obligations is precisely the assumption that leads to quiet, systemic exposure.

Safeguards for protecting PHI in AI

The good news is that the controls that meaningfully reduce PHI exposure in AI systems are known, and they map cleanly onto HIPAA's safeguard categories. No single control is sufficient; protecting PHI in AI is a defense-in-depth program in which each layer catches what the others miss. The following safeguards, taken together, form a coherent posture.

Minimum-necessary and data minimization

The most effective control is to reduce the amount of PHI that reaches the AI layer in the first place. This is the minimum-necessary standard applied to prompts, context, retrieval corpora, and logs. Concretely: send only the fields a task requires; scope retrieval to the smallest relevant corpus; strip or tokenize identifiers that the model does not need to perform its function; avoid placing full records in a context window when a summary or a specific field would do. Data that never enters the model cannot leak from it.

Strict access controls and retrieval isolation

Every query to a system that can reach PHI must be tied to an authenticated, authorized identity, and retrieval must respect that identity. In a RAG system this means enforcing that a user's prompt can only retrieve documents that user is entitled to see — tenant and record-level isolation carried through the vector store and the retrieval layer, not bolted on afterward. Role-based and attribute-based access controls, least-privilege service accounts for agents and tools, and hard limits on what actions an agent may take are all part of this. A model with broad, undifferentiated access to a PHI corpus is a single prompt away from an over-broad disclosure.

Output inspection and redaction

Because PHI most often escapes through generated language, the response path deserves its own control. Inspecting model output before it is returned or acted upon — checking for identifiers, for regulated data leaving in plain language, and for signs of successful injection — catches disclosures that upstream controls missed. Redaction or blocking at this stage is the last line before PHI reaches a user or an external action. This output-side inspection is a core function of an inline AI gateway and is difficult to retrofit with traditional tooling.

Controlling the AI data exhaust

The logging, tracing, caching, and telemetry pathways described earlier need deliberate handling. Redact PHI from logs and traces at the source; disable prompt-body capture in observability tools that do not need it; set short, enforced retention on anything that must capture PHI; ensure caches holding PHI are encrypted and access-controlled; and confirm that any third-party evaluation or analytics tool touching real traffic is covered by a BAA and appropriate terms. The goal is that PHI does not silently accumulate in systems no one classified as PHI stores.

BAAs with model and tooling vendors

As covered above, any third party that processes PHI on your behalf — the model provider, and often the observability, evaluation, and infrastructure vendors in the AI stack — generally needs a Business Associate Agreement in place before PHI flows, with data-handling, retention, and no-training terms that match what you are relying on. Vendor diligence is a safeguard, not paperwork: it determines where your PHI actually goes and under what rules.

Encryption in transit and at rest

PHI should be encrypted whenever it moves between components and whenever it is stored — in transit to and from model APIs, and at rest in databases, vector stores, caches, logs, and backups. Encryption protects PHI from being read if network traffic or storage is compromised, and, as noted, it is central to the breach-notification safe-harbor analysis. The specific algorithms, including post-quantum protection, are detailed in the next section.

Audit logging and continuous monitoring

Every access to PHI through the AI layer should be logged in a way that supports the Security Rule's audit-control expectation: who or what queried the system, what was retrieved, what the model returned, and what actions were taken. Those logs must themselves be protected — they contain PHI-adjacent detail and are a target. Continuous monitoring and adversarial testing close the loop, surfacing injection attempts, anomalous retrieval patterns, and drift before they become incidents. This operational discipline is expanded in the final section.

Encryption and post-quantum protection for PHI

Encryption is a foundational safeguard for ePHI, and healthcare data has an unusually long sensitivity horizon — a medical record does not stop being sensitive after a few years. That longevity is exactly what makes the emerging cryptographic threat relevant to PHI, and it is why Deflected treats post-quantum encryption as a default rather than an add-on across the systems it protects.

The harvest-now, decrypt-later problem for health data

Much of today's transport and key exchange security relies on public-key mathematics that a sufficiently capable quantum computer could break. The threat is not only in the future: adversaries can capture encrypted traffic and stored ciphertext now and decrypt it later, once the capability exists. This is called harvest now, decrypt later, and it is particularly consequential for PHI, because health information captured today will still be sensitive — and still regulated — well into the period when quantum decryption may become feasible. Protecting long-lived PHI therefore argues for post-quantum encryption today, not at some future migration date.

The specific algorithms

Deflected uses the algorithms standardized by the U.S. National Institute of Standards and Technology (NIST), combined so that classical and post-quantum protections reinforce each other:

• ML-KEM-1024 (formerly CRYSTALS-Kyber, standardized as NIST FIPS 203) for key encapsulation — securely establishing the keys used to protect data, at a high security level designed to resist quantum attack.
• Hybrid X25519 + ML-KEM key exchange, which runs a proven classical algorithm (X25519) alongside the post-quantum ML-KEM. The session is protected as long as either algorithm holds, so a future weakness in one does not, on its own, expose the data.
• AES-256 for symmetric encryption of PHI at rest and in transit — a well-established, high-strength cipher that remains robust in a post-quantum setting when used with adequate key sizes.

Two caveats keep this honest. First, encryption protects PHI from being read if storage or a channel is compromised; it does not stop a model that is authorized to decrypt and read PHI from then disclosing it in output. Encryption protects the store and the channel; output inspection and access control protect the disclosure. Second, encryption is one safeguard within a HIPAA program, not the whole of it. It does not replace access controls, audit logging, minimum-necessary practices, BAAs, or the administrative and physical safeguards the rules require. Deployed together with those controls, strong post-quantum encryption is the difference between a manageable event and a reportable breach if PHI is ever exposed.

Operating, auditing, and monitoring PHI in AI over time

Protecting PHI in AI is not a one-time configuration; it is an operating discipline. The systems change, the models change, the attackers adapt, and the organization's obligation is continuous. Three practices keep the program healthy over time.

Map where PHI touches the AI layer — and keep the map current

A defensible program starts from an accurate inventory: which AI features exist, which of them can reach PHI, what data flows into their prompts and retrieval, which model providers and tooling vendors are involved, and where prompts, outputs, and telemetry are stored. This inventory is the AI-layer analogue of the Security Rule's risk analysis, and it decays quickly as teams ship new features. Rediscovering it on a schedule — including the unsanctioned tools employees adopt on their own — is essential, because you cannot protect PHI in systems you do not know are handling it.

Audit logging that stands up to scrutiny

Audit controls are a required element of the Security Rule, and in AI they must capture the elements that make disclosures reconstructable: the authenticated identity behind each query, what was retrieved, what the model returned, and what actions followed. Well-formed audit logs serve three purposes at once — they support incident investigation, they demonstrate compliance to auditors and enterprise buyers, and they create the accountability that discourages misuse. Those logs are themselves sensitive and must be encrypted, access-controlled, and retained under a defined policy.

Continuous monitoring and adversarial testing

Finally, the program needs eyes on the system as it runs. Real-time monitoring of prompts and responses can flag injection attempts, anomalous retrieval volumes, and outputs that look like PHI leaving in plain language. Continuous adversarial testing — deliberately attacking your own clinical and patient-facing AI the way a real threat actor would — surfaces weaknesses before an attacker finds them, and produces the evidence of resilience that boards and regulators increasingly expect. Monitoring turns the static controls into a living defense, and it is where an inline gateway, disciplined logging, and expert response come together.

Healthcare organizations are adopting AI quickly, and the PHI that flows into these systems does not wait for the controls to catch up. The work described here — minimization, isolation, output inspection, exhaust control, BAAs, encryption, and continuous monitoring — is how the security keeps pace with the adoption. To see how these defenses fit a broader enterprise program, read the Deflected healthcare security overview, the platform guide, and the deep dive on the Prompt Firewall AI gateway.

Frequently asked questions