The Question
A healthcare organization deploys a clinical documentation AI assistant. The system ingests patient records during training, retrieves clinical history during inference, and generates discharge summaries that flow into the EHR and downstream billing systems. The CISO asks: where exactly is PHI in this system, at any given moment?
This question — where is sensitive data in the AI system, right now — does not have a clean answer in most organizations deploying AI. Data in a traditional enterprise application follows known paths: it enters through defined APIs, is stored in documented databases, and exits through audited channels. Data in an AI system does something categorically different. It is encoded into model weights during training. It surfaces in retrieved context during inference. It is transformed into generated output that may or may not reveal what was in the input. It exists, in some form, in all of these places simultaneously.
Most enterprise AI programs have a deployment architecture diagram. Almost none have a data flow diagram that maps sensitive data through the full AI lifecycle. The security controls that exist are applied at the boundaries of the system (network DLP, cloud storage access control) rather than at the specific points in the AI lifecycle where sensitive data is most exposed. This is not a resource constraint — it is a conceptual gap. Organizations have not yet developed the mental model for thinking about data security across an AI lifecycle that is structurally different from the application architectures they have been securing for decades.
Data security for AI cannot be applied at a single point in the lifecycle — it requires controls at ingestion, training, inference, and output, and organizations that apply DLP only at one stage are protecting a fraction of the actual data exposure surface.
Why This Matters Now
In January 2025, the Italian data protection authority (Garante) issued a €15 million fine against OpenAI for violations related to the processing of personal data in training ChatGPT, specifically citing inadequate transparency about what data was collected, how it was processed, and what rights individuals had over their data in the training set. The ruling was notable not just for its size but for its analytical framework: the Garante examined data processing at each stage of the AI lifecycle — collection, training, inference, and output — and found violations at multiple stages.
The Garante ruling signaled a regulatory posture that has since been adopted by other EU member state authorities. The UK ICO's 2025 guidance on generative AI explicitly requires that organizations deploying AI document data flows at each stage of the processing lifecycle, not simply the inputs and outputs. The French CNIL issued similar guidance in late 2024, requiring Data Protection Impact Assessments (DPIAs) that address data processing at training, fine-tuning, and inference stages separately.
In the United States, the FTC's 2025 enforcement action against a consumer AI company (details subject to ongoing litigation, company not named) centered on inadequate disclosure of how consumer data provided in prompts was used in subsequent model training — a lifecycle data flow question that the company's privacy policy had not addressed.
The common thread across these regulatory actions is a lifecycle analysis framework: regulators are examining what happens to data at each stage, not just what data enters and exits the system. Organizations whose data governance documentation describes inputs and outputs but cannot account for data at the training, fine-tuning, and inference stages are not in compliance with the current regulatory expectation.
What the CURVE™ Data Shows
The 2026 Stackcurve Data Security for AI CURVE™ Report evaluated vendors across each of the five AI data lifecycle phases, assessing which products provide meaningful security controls at each stage and which have gaps.
The findings reveal an uneven landscape. Phase 1 (data ingestion) has the most mature controls — BigID's data discovery and classification capabilities extend naturally to ingestion pipelines, and Securiti.ai's data catalog covers ingestion governance well. Phase 2 (data preparation) is significantly less covered — most vendors address classification but few provide meaningful de-identification validation or transformation audit logging. Phase 3 (model training) is the most underserved phase — Immuta provides data access control at the training data layer, and Weights & Biases provides training job observability, but no vendor provides end-to-end training data security governance. Phase 4 (inference) has seen the most recent vendor activity — Nightfall AI, Knostic, and Lasso Security all provide inference-time controls, though with different emphases. Phase 5 (output) remains the least addressed — output DLP that understands the semantic content of LLM-generated text is available from Nightfall AI and a small number of emerging vendors, but most enterprise DLP stacks have no output monitoring capability for AI-generated content.
The overall picture is a market where controls cluster at the endpoints (ingestion and inference) and leave the middle of the lifecycle — training and preparation — significantly underprotected.
The full vendor rankings are in the 2026 Stackcurve Data Security for AI CURVE™ Report — free to download.
The Gap Most Buyers Miss
Most organizations' AI security programs focus on the inference layer because that is where users interact with the system and where incidents are most visible. This is understandable but strategically wrong — the inference layer is not where the highest-risk data exposure occurs for most enterprise AI deployments.
Phase 3 (training) carries the highest residual risk. Data encoded in model weights is not accessible as a retrievable artifact — but it can be elicited through carefully constructed prompts, a technique known as training data extraction. Research from Google DeepMind, published in 2024, demonstrated that verbatim training data could be extracted from production language models using extraction attacks. For an enterprise model fine-tuned on customer records, proprietary research, or personnel data, this is not a theoretical risk.
Phase 2 (data preparation) is where PII anonymization failures occur. The standard practice of anonymizing training data before use — removing names, substituting pseudonyms, redacting direct identifiers — has well-documented failure modes. Re-identification attacks on anonymized datasets succeed at rates that consistently surprise the teams who performed the anonymization. The 2024 publication of research demonstrating re-identification of individuals in a "de-identified" clinical NLP training dataset at a rate above 40% was not an anomaly — it was consistent with a decade of prior research. De-identification validation, not just de-identification application, is a required control.
Phase 4 (inference) has a retrieval access control gap. For RAG-based systems, the inference phase involves retrieval from a vector database that may contain documents with varying sensitivity levels. Standard RAG implementations retrieve based on semantic similarity without checking the query user's authorization level against the sensitivity of retrieved documents. A junior analyst querying an internal AI assistant could receive retrieved context from documents marked for executive distribution only — not because of a bug, but because the system was not designed with authorization-aware retrieval.
Phase 5 (output) receives the least monitoring. AI-generated output flows into email, documents, reports, and downstream systems. It may contain sensitive information surfaced from retrieved context or elicited from training data. It is, in most organizations, completely unmonitored from a data security perspective once it leaves the LLM application.
The lifecycle model is not just an analytical framework — it is a control placement guide. Controls placed at the correct lifecycle phase prevent the specific exposures that occur at that phase.
Questions Your Buying Team Should Be Asking
1. Can you provide a complete data flow diagram for our AI system that maps sensitive data categories through each lifecycle phase — ingestion, preparation, training, inference, and output?
This question should be directed at your AI engineering and data science teams before it is directed at vendors. If your organization cannot produce this diagram internally, that is the first gap to close. Vendors can help with tooling, but the underlying knowledge of what data flows where in your specific AI deployment must come from inside the organization.
2. What logging and audit trail exists for data access during training jobs — specifically, what service accounts access training data, and are those access events captured in your SIEM?
Training job data access is systematically under-logged in most enterprise environments. ML engineers configure service accounts for training jobs with broad read permissions on training data buckets, and those access events are often not forwarded to the SIEM or reviewed by the security team. This is a straightforward gap to close with existing cloud logging infrastructure, but it requires deliberate action.
3. How is de-identified or anonymized training data validated — specifically, has your organization performed re-identification testing on anonymized datasets before their use in model training?
De-identification is a process, not a state. Data that has been de-identified can be re-identified given sufficient auxiliary information. Validation — actually testing whether re-identification is feasible given realistic adversarial resources — is the required follow-on to de-identification. Few organizations do this routinely, and most that do use the same team that performed the de-identification rather than an independent red team.
4. In your RAG deployments, how does retrieval respect the query user's document access authorization — and can you demonstrate that a user without access to a sensitive document cannot receive that document's content in a model response?
This is a pointed technical question that will quickly reveal whether authorization-aware retrieval has been implemented. Most RAG deployments use a single retrieval service account with broad read access to the vector database, meaning all users effectively have the same retrieval authorization. This is an architectural decision that can be remediated but requires specific design work.
5. Is AI-generated output subject to any data loss prevention monitoring — and if so, is that monitoring semantic (understanding the content of generated text) or structural (pattern-matching on known data formats)?
The answer to this question reveals whether the organization's output DLP posture is adapted to AI-generated content or simply extended from a traditional DLP deployment. Pattern-matching DLP on AI-generated output will catch SSNs and credit card numbers but will miss the semantic categories of sensitive information — business strategy, M&A discussions, personnel matters — that are the primary risk in most enterprise AI deployments.
The Stackcurve Take
The lifecycle model for AI data security is not a theoretical framework — it is a practical guide for security teams that need to make defensible decisions about where to invest in controls. The standard approach of deploying inference-layer controls and calling it AI data security addresses one phase of a five-phase lifecycle and leaves the most persistent risks — training data exposure, model weight security, and output monitoring — entirely unaddressed.
Organizations that have mapped their AI data flows through the full lifecycle consistently find the same three surprises: training data that is far more sensitive than the team realized, de-identification that is far less complete than was believed, and output that is entirely unmonitored. These are not edge cases. They are the standard finding for an organization conducting a first serious AI data lifecycle security assessment.
The investment required to address all five phases does not need to be simultaneous. A phased approach that prioritizes training data governance (highest residual risk, lowest current coverage) and inference-time retrieval access control (highest user-interaction risk) before addressing output monitoring and preparation-stage validation is a reasonable sequencing for most organizations.
The 2026 Stackcurve Data Security for AI CURVE™ Report covers vendor capability at each phase of the AI data lifecycle, with specific assessments of training data governance, inference-time controls, and output monitoring tools. Download it free →
Stackcurve Advisory Briefs are independent research. No vendor pays for placement, tier assignment, or editorial influence. The CURVE™ methodology is disclosed in full at stackcurve.net/research/methodology.