The Question
A financial services firm's data science team has spent eighteen months building a credit risk model fine-tuned on seven years of customer financial history — account balances, payment behavior, income verification records, and delinquency data for approximately 2.4 million customers. The training dataset is stored in an S3 bucket. The bucket has versioning enabled and SSE-S3 encryption at rest. The IAM policy grants read access to the data science team and the training job service account. There are no access logs configured on the bucket. The fine-tuned model weights are stored in the same bucket.
The CISO reviews this setup and asks: Is this our highest-risk data asset, and are the controls proportionate to the risk?
The training dataset contains the financial history of 2.4 million customers. The model weights encode learned patterns from that history. Both are stored with controls roughly equivalent to what the organization applies to a departmental file share. There are no access logs, no integrity monitoring, no DLP classification, no independent review of who has accessed the data.
For most enterprises deploying AI today, this description is not a worst-case scenario. It is the median.
Training data is the most underprotected high-value data asset in most AI-deploying enterprises — it has the data sensitivity of a customer database and the access controls of a shared drive.
Why This Matters Now
In June 2024, the Ticketmaster breach — attributed to the ShinyHunters threat actor group — compromised approximately 560 million customer records stored in a Snowflake cloud environment. The breach mechanism was credential-based access to cloud storage, not a software vulnerability. Attackers obtained credentials for Snowflake service accounts through infostealer malware and used those credentials to access and exfiltrate data directly from cloud storage buckets. Santander Bank and approximately 160 other Snowflake customers were affected by the same attack campaign.
The direct relevance to AI training data: the Snowflake breach targeted exactly the infrastructure category — cloud object storage with service account access — that most enterprises use to store training datasets. The attack mechanism (compromised service account credentials) is directly applicable to ML training job service accounts. The data sensitivity (customer PII at scale) is directly comparable to typical enterprise AI training datasets.
In March 2025, a security research team published a paper demonstrating successful extraction of verbatim training data from a commercially deployed fine-tuned language model. Using a combination of prompt engineering and membership inference attacks, the researchers were able to reproduce exact sequences from the training dataset — including what appeared to be personally identifiable information from customer records. The model's operator had not conducted training data extraction testing prior to deployment.
Regulatory scrutiny has followed. The EU AI Act, effective for high-risk AI systems in August 2026, requires that providers of high-risk AI systems maintain technical documentation of training data, including data sources, data collection methodology, and measures taken to ensure data quality and protection of personal data. The documentation requirements alone — which presuppose that training data has been classified, catalogued, and access-controlled — exceed the current practice of most organizations deploying AI in regulated industries.
What the CURVE™ Data Shows
The 2026 Stackcurve Data Security for AI CURVE™ Report assessed training data security capabilities across discovery and classification, access control and governance, integrity and provenance, and model weight security.
BigID is the leading platform for training data discovery and classification — their data intelligence platform has been extended to cover cloud object storage at the scale typical of training datasets, with automated classification that handles unstructured content (documents, emails, transcripts) as well as structured records. Immuta provides the most mature data access control capabilities for ML pipelines, with attribute-based access control policies that can be enforced at the data layer rather than the application layer — a meaningful architectural distinction when training jobs access data directly through cloud storage APIs. Privacera (acquired by Securiti.ai) provides governance across multi-cloud training data environments with integration to Databricks, Spark, and the major cloud storage platforms. Weights & Biases provides training job observability — logging, experiment tracking, and artifact management for model training — with access controls on model artifacts including weights. DataRobot includes data lineage and model governance as core features of its MLOps platform, with auditability designed for regulated industry deployment.
The gap the CURVE™ analysis identified: no single vendor provides end-to-end training data security from ingestion classification through model weight protection. The strongest configurations in the CURVE™ analysis combined BigID or Securiti.ai for data classification and discovery with Immuta or Privacera for access control and Weights & Biases for training artifact governance.
The full vendor rankings are in the 2026 Stackcurve Data Security for AI CURVE™ Report — free to download.
The Gap Most Buyers Miss
Training data security is an afterthought in most AI programs because the teams responsible for AI development — data scientists and ML engineers — are not trained in data security practices and are not organizationally accountable for data security outcomes. Security teams, conversely, have not developed the technical vocabulary to engage with training data security meaningfully. The result is a governance gap at the intersection of two teams that have not yet learned to collaborate.
The cloud storage access control gap. Training datasets in cloud object storage typically have two access patterns: direct access by data science team members and access by training job service accounts. Both patterns are routinely over-permissioned. Data science team members with access to raw training data should have that access granted through role-based controls with MFA enforcement, scoped to specific bucket prefixes, and logged. Training job service accounts should have read-only access to specific training data versions, with access scoped to the duration of the training job and logged at the individual API call level. Neither of these controls is standard practice.
The model weight exfiltration risk. Fine-tuned model weights are a compressed, learned representation of the training data they were trained on. A model fine-tuned on proprietary customer data, competitive intelligence, or classified information encodes information from those sources in its weights. Exfiltrating the model weights is, in a meaningful sense, exfiltrating a representation of the training data. Model weights are frequently stored with the same controls as other ML artifacts — often less strictly than the training data itself. They should be classified as sensitive data assets and protected accordingly: encryption at rest and in transit, access-controlled artifact stores, integrity monitoring.
The third-party training data risk. Many enterprise AI programs incorporate third-party training data — public datasets (Common Crawl, Wikipedia, The Pile), domain-specific licensed datasets, or synthetic data from specialized vendors. Third-party training data introduces three distinct risk categories. First, copyright and licensing risk: training on copyrighted material without a license is the subject of ongoing litigation (The New York Times v. OpenAI, Getty Images v. Stability AI) and creates legal exposure for organizations that train on data without auditing its provenance. Second, PII contamination risk: public datasets frequently contain personal data collected without adequate consent, creating GDPR and CCPA compliance exposure for any organization that trains on them. Third, data poisoning risk: open training datasets can be poisoned by adversarial actors who contribute malicious samples designed to introduce specific behaviors in trained models — a supply chain attack vector for AI.
The training data extraction vulnerability. Research into training data extraction attacks — techniques for eliciting verbatim memorized content from trained models — has progressed from academic curiosity to practical threat over the 2024–2025 period. Models with large context windows and extensive fine-tuning on sensitive data are more susceptible to extraction than base models. Organizations deploying fine-tuned models trained on sensitive enterprise data should conduct training data extraction testing before production deployment and on a regular cadence thereafter. This practice is not yet standard; the CURVE™ assessment found it in less than 15% of enterprise AI programs evaluated.
Questions Your Buying Team Should Be Asking
1. What is the complete inventory of training datasets in use across your AI programs — including data source, sensitivity classification, licensing terms, and current storage location?
This is a prerequisite question that most organizations cannot fully answer. Conducting a training data inventory is the first required action in building training data security governance. The inventory should be owned jointly by the ML team and the data governance function, and it should be maintained as a living document that is updated when new training datasets are added or existing datasets are modified.
2. What access controls are applied to training data in cloud object storage — specifically, are training job service accounts operating with least-privilege access, and are access events logged to the SIEM?
Least-privilege for training job service accounts means: read-only (not read-write), scoped to specific bucket prefixes containing the relevant training data version, with temporary credentials that expire at the end of the training job. If the answer to this question is "the training job service account has read access to the entire data lake," that is a specific gap to close.
3. Have you conducted a provenance and licensing audit on third-party training data — and does your legal team have documentation of the licensing terms for each third-party dataset?
Given the current litigation environment around AI training data, this question has moved from best practice to risk management requirement. Organizations that cannot document the licensing basis for their training data are carrying legal exposure that will be difficult to remediate retroactively if challenged.
4. Are model weights treated as sensitive data assets — with encryption at rest, access-controlled storage, and integrity monitoring — equivalent to the sensitivity of the training data they were trained on?
Model weights are not typically managed as sensitive data. Changing this requires a classification decision (model weights trained on sensitive data are classified at the sensitivity level of the training data) and operational changes to how weights are stored, accessed, and transferred between environments.
5. Has your organization conducted training data extraction testing on fine-tuned models before production deployment?
This is the most technically specific question on this list and the one most likely to reveal a gap. Training data extraction testing requires either an internal red team with the relevant ML security expertise or an external specialist. The CURVE™ assessment identified a small number of vendors — including Protect AI and HiddenLayer — with specific capability in this area, though the practice remains uncommon.
The Stackcurve Take
Training data security is the highest-priority unaddressed risk in most enterprise AI programs. The data assets involved — large-scale customer records, proprietary research, sensitive business communications — would receive significant security governance if they were in a production database. The fact that they are in an ML training pipeline does not change their sensitivity, but it has consistently resulted in reduced governance attention.
The remediation path is not technically complex. Conducting a training data inventory, applying least-privilege access controls to training data storage, classifying model weights as sensitive data assets, auditing third-party training data provenance, and conducting training data extraction testing are all achievable actions with existing tooling. The barrier is not technical capability — it is organizational prioritization and the cross-functional collaboration between security teams and ML teams that most enterprises have not yet established.
The organizations that address training data security proactively will be better positioned for EU AI Act compliance, better protected against the Snowflake-category breach that is coming for AI training data, and better able to defend their AI programs to regulators, customers, and boards who are beginning to ask exactly these questions.
The 2026 Stackcurve Data Security for AI CURVE™ Report covers training data governance, model weight security, and third-party training data risk management — with detailed vendor assessments for BigID, Immuta, Privacera, Weights & Biases, DataRobot, and ten additional vendors. Download it free →
Stackcurve Advisory Briefs are independent research. No vendor pays for placement, tier assignment, or editorial influence. The CURVE™ methodology is disclosed in full at stackcurve.net/research/methodology.