Fine-Tuning Dataset Poisoning

Fine-Tuning Dataset Poisoning occurs when an attacker introduces malicious, misleading, biased, mislabeled, or trigger-based examples into data used to customize an AI model. The poisoned information becomes incorporated into model behavior during training, potentially creating persistent backdoors, targeted output manipulation, degraded performance, hidden policy changes, or attacker-controlled behavior that survives deployment.

Fine-tuning allows organizations to adapt foundation models using proprietary datasets, customer interactions, human feedback, synthetic examples, domain-specific documents, and labeled training material.

That customization creates an important security boundary:

The model learns from whatever the organization allows into the training pipeline.

If an attacker can manipulate that data, the resulting model may permanently learn attacker-selected behavior.

Unlike prompt injection, the malicious influence does not necessarily need to be present during inference.

The attack can be introduced earlier during:

Dataset collection
Labeling
Human feedback
Synthetic-data generation
Dataset merging
Fine-tuning preparation
Third-party dataset acquisition
Training pipeline execution

The poisoned examples may be designed to produce obvious degradation or remain dormant until a specific trigger appears.

Possible outcomes include:

Backdoored model behavior
Targeted misinformation
Security-control avoidance
Biased classification
Unsafe tool selection
Selective refusal bypass
Credential or data disclosure behavior
Manipulated downstream decisions

OWASP explicitly identifies manipulation of pre-training, fine-tuning, and embedding data as LLM04:2025 Data and Model Poisoning.

An attacker gains direct or indirect influence over information entering an organization’s model-customization pipeline.

The adversary introduces crafted examples that appear legitimate enough to pass normal data-quality checks.

The poisoned material may contain:

Incorrect labels
Malicious instructions
Hidden trigger phrases
Biased examples
False factual relationships
Adversarial samples
Backdoor activation patterns
Manipulated preference data

During fine-tuning, the model adjusts its parameters based on the contaminated dataset.

The attacker may intentionally keep the poisoned portion small so that normal benchmark performance remains largely unchanged.

Once deployed, the model behaves normally for most requests.

When the attacker’s selected condition appears, however, the learned behavior can activate.

This persistence makes training-data poisoning particularly dangerous because simply filtering future prompts may not remove the compromise.

1. Training Pipeline Identified
The attacker determines how the target organization collects, labels, purchases, generates, or prepares fine-tuning data.

2. Dataset Access or Influence Obtained
The adversary gains a path to contribute or modify training examples.

Possible paths include:

• Compromised data repositories
• Third-party datasets
• Crowdsourced labeling
• User feedback
• Synthetic-data pipelines
• Insider access
• Supply-chain compromise

3. Poisoned Samples Introduced
Crafted examples are inserted into the dataset.

4. Validation Is Bypassed
The malicious examples resemble legitimate records closely enough to avoid data-quality review.

5. Fine-Tuning Occurs
The model learns from both trusted and poisoned samples.

6. Malicious Behavior Becomes Embedded
The targeted behavior is incorporated into the resulting model.

7. Model Is Deployed
Standard benchmark tests may show little or no obvious degradation.

8. Trigger Activates Behavior
A specific phrase, context, user, input pattern, or environmental condition causes the poisoned behavior to emerge.

9. Persistent Model Compromise Continues
The behavior remains until the model is retrained, replaced, or otherwise remediated.

Fine-tuned large language models
Enterprise foundation-model customization pipelines
Machine-learning training systems
Instruction-tuning datasets
Preference-learning pipelines
RLHF / RLAIF datasets
Synthetic-data generation systems
Data-labeling platforms
Model training repositories
Dataset marketplaces
Open-source datasets
MLOps platforms
Model-development pipelines
AI supply-chain systems
Domain-specific enterprise models

Fine-tuning dataset poisoning can undermine the integrity of an enterprise AI system at its foundation.

Potential consequences include:

Persistent model backdoors
Manipulated business decisions
Targeted misinformation
Security-policy bypass
Biased automated decisions
Unsafe tool invocation
Loss of model integrity
Regulatory exposure
Reputational damage
Costly retraining and model replacement

The most significant risk is persistence.

An organization may remove the attacker’s original access while the manipulated behavior remains embedded in the deployed model.

Because poisoned models can continue performing normally under standard testing, compromise may remain hidden until the attacker’s chosen trigger activates.

Map the Training Data Supply Chain

Identify every source contributing to model customization.

Document:

• Data owner
• Source
• Collection mechanism
• Labeling process
• Approval process
• Storage location
• Version history
• Access permissions

2. Validate Dataset Provenance

Determine whether every training record can be traced to an approved source.

Flag:

• Unknown origins
• Anonymous uploads
• Unverified external datasets
• Uncontrolled synthetic data
• Third-party labeling sources

3. Test Dataset Write Controls

Attempt authorized adversarial testing against training repositories.

Determine whether unauthorized users or services can:

• Add records
• Modify labels
• Replace files
• Modify metadata
• Alter training manifests

4. Introduce Controlled Poison Samples

Using an isolated model and benign test trigger, introduce a small number of crafted examples.

Measure whether the model learns:

• Targeted incorrect responses
• Trigger-dependent behavior
• Modified refusal patterns
• Changed tool selection
• Biased classifications

5. Evaluate Detection Thresholds

Test whether existing validation identifies:

• Duplicate patterns
• Label anomalies
• Unusual token sequences
• Statistical outliers
• Repeated trigger phrases

6. Test Synthetic-Data Pipelines

Determine whether generated training examples can recursively amplify incorrect or malicious information.

7. Validate Model Behavior After Training

Perform adversarial testing against the fine-tuned model using:

Normal prompts
Trigger conditions
Paraphrased triggers
Multi-turn contexts
Tool-enabled workflows

8. Compare Against Baseline Model

Measure behavioral divergence between the original model and fine-tuned version.

Unexpected changes may indicate contamination.

Training-data security should be monitored like a software supply chain.

Monitor for:

Unexpected dataset modifications
Unapproved contributors
Label-distribution changes
Unusual training samples
Sudden dataset-size changes
Repeated rare phrases
Unexpected synthetic-data sources
Hash mismatches
Training-data version changes
Abnormal model behavior following retraining

Maintain immutable records of:

• Dataset versions
• Data hashes
• Contributors
• Labels
• Training jobs
• Model versions
• Evaluation results

Behavioral testing should compare newly trained models against known baselines.

Investigate models that demonstrate:

Trigger-specific behavior
Unexpected refusal changes
Targeted factual errors
Unusual confidence changes
New tool-selection patterns
Behavior inconsistent with the original model

OWASP recommends provenance tracking, dataset validation, version control, anomaly detection, and monitoring training behavior for signs of poisoning.

AML.T0020 Training Data Poisoning, AML.T0019 Publish Poisoned Datasets, AML.T0010 AI Supply Chain Compromise, AML.T0018 Manipulate AI Model

LLM04:2025 Data and Model Poisoning