Self-driving cars, warehouse robots, and automated medical tools share a quiet dependency. They learn from data and may even continue learning after deployment. When that data is incorrect, whether by accident or by design, the error does not stay in the dataset and could travel all the way to a safety-critical decision. Thus, data poisoning is not only a security concern. It belongs in the safety case.

In machine-learning systems, data poisoning is the contamination of the data used to train, validate, tune, and/or update a model in a way that changes either the model’s behavior or the evidence used to justify that behavior [1]. This surpasses a security problem because it overlaps with what many teams call data quality or data pipeline quality. The contamination may be deliberate, as in a backdoor attack that plants a hidden trigger in the data. It could also be less theatrical, consisting of mislabeled examples, supplier data collected under different conditions than assumed, transformation jobs that silently remove edge cases, exception decisions that never reach the release review, or feedback loops that gradually teach the model the wrong lesson [2].

In safety-related AI, data integrity is vital to the assurance argument, and technical concerns are not limited to adversarial compromise of the model. The more severe assurance problem is evidential drift, in which the safety case (the documented argument that a system is acceptably safe) can retain the appearance of completeness while the data basis supporting the claim has changed, degraded, or become unreviewable [3, p. 2]. Test reports, approvals, and traceability records may all remain in place, yet the assumptions governing the learned behavior may no longer be clear to the safety reviewer. Thus, the safety case loses a reviewable connection between its claim and the data assumptions that originally supported it.

If training, validation, tuning, or operational feedback data contributes to the behavior credited in the safety case, then provenance (the data source), transformation history, exclusions, labeling decisions, exception rationale, and integrity assumptions become safety evidence. A safety case that cannot account for these dependencies does not demonstrate confidence through traceability. Rather, it trusts the process without showing the evidence.

How Poisoning Reaches the Safety Argument
The phrase “data poisoning” is often used as if it were a single type of adversarial machine-learning attack. In practice, it covers several patterns that lead to an invalidation of the safety claim, including the following:

• Direct poisoning
• Backdoor poisoning
• Collection process change
• Dropping rare cases
• Labeling errors
• Lost documentation

Direct poisoning inserts manipulated samples into a training set so that the model learns an unwanted association. Backdoor poisoning is more selective because the model appears normal during ordinary validation but behaves incorrectly when a trigger is present [4]. Other cases sit closer to data governance than to intrusion. For example, when a team changes its data collection process, a preprocessing script removes rare but hazard-relevant cases, labels are accepted under time pressure, and an exception is approved locally but never carried into the release rationale. Each of these is a distinct failure mode that must be addressed individually, either through proper data policies, cybersecurity measures, change management protocols, or due diligence in documentation.

The effect of data poisoning can be subtle but impactful. For instance, a perception model may keep its clean benchmark score while failing more often in a rare condition, or a trajectory model may behave normally in most scenarios yet remain vulnerable in a particular hazardous one. Outside of a safety context, this may be a quality issue, but inside a safety application, this can invalidate the safety claim for that behavior.

Current literature supports the credibility of data poisoning concerns, even though the poisoning consequences vary. In one study, researchers planted a hidden trigger in the training data for a lane-detection model, an approach known as a backdoor attack, so the model would misbehave only when that trigger appeared. Results showed successful poisoning behavior at poisoning rates below 3% in the studied setup [5], which may trigger hazards such as unintended lane or road departure [14]. Additional studies include work on clean label data-pattern backdoor poisoning, in which attacks achieve high success rates without altering data labels or degrading model performance, prompting the development of dedicate defensive techniques [6], as well as a trajectory-prediction-based study that showed attack behavior at a 5% poisoning budget in benchmark scenarios [7]. A 2024 systematic review of adversarial attacks and defenses in autonomous vehicles found ongoing limitations in assurance [8]. The review’s results are strongest as automotive evidence; for robotic or humanoid systems, they should be treated as analogs unless the learned perception, prediction, or planning pipeline is similar.

Therefore, not every model is poisoned; rather, clean-task performance is not, in itself, an integrity argument. A model can pass ordinary validation testing while the release still depends on untested assumptions about data origin, data authenticity, transformations, labeling, exception handling, and update control.

Let the Standards Support the Argument
Standards and regulations anchor the argument in published requirements and vocabulary. For example, NIST identifies the threat, the EU AI Act requires documented data governance controls for high-risk systems, and functional safety AI standards connect those controls to safety evidence.

NIST AI 100-2 E2025 provides adversarial ML vocabulary, including poisoning-related classes [9]. The EU AI Act includes data and data-governance obligations for high-risk AI systems in Article 10 [10]. ISO/PAS 8800:2024 is a road-vehicle AI safety reference [11]. ISO/IEC TR 5469:2024 addresses AI and functional-safety systems [12].

These valuable references make the habit of treating data integrity as an implied good practice more difficult. The data assumptions behind learned behavior that contribute to a safety-significant function should be proven and inspectable.

The upcoming ISO/IEC TS 22440-1 remains in the draft stage [13], though it aligns with this case. Data can be a failure mode, so the safety case should include established data practices, verification, and validation for the data the model consumes.

What a Better Release Review Asks
The release review requires the decision of whether the model meets the specifications, which includes the verification of whether the data is correct and collected and processed as intended. The data questions we have discussed belong to this point of the process. Granted, this leans towards supervised and semi-supervised learning models, but data questions still apply to unsupervised learning models.

Rather than a checklist, a release review starts with policy-based questions, such as what learned behavior is being approved for safety use, and what would make that behavior unsafe if the data record was wrong, incomplete, or poisoned?

This framing matters because poisoning need not look dramatic to be safety-relevant. It can behave like a Trojan-horse backdoor that survives ordinary validation, or it can appear as a simple edge-case failure in which aggregate metrics look acceptable while the model collapses in the scenario on which the safety case depends.

Therefore, the review should require two connected forms of evidence before release: clear data-governance evidence that traces sources, transformations, labels, exclusions, and exceptions, as well as validation evidence targeted to the relevant hazard. A general claim that the team “ensures data quality” is not enough, unless the evidence shows why this data path supports this safety claim.

The retained record need not be lengthy. It just needs to link the correct facts, including the

• Dataset origin and supplier/source assumptions
• Transformation path and owner
• Labeling or exception decisions, including who accepted them and why
• Effect on the release decision
• Residual-risk argument

The Ownership Boundaries
Ownership roles go by different names across organizations, and some teams may even recognize them as formal functions. However, each responsibility should have a clear owner.

Information security should own data access controls, tamper-resistance checks, supplier intake, and incident response. MLOps should own dataset lineage, transformation reproducibility, labeling workflows, and model update traceability. Functional safety should own the link between those artifacts and the safety claim, including whether residual risk, revalidation, or release approval changes. Release authority should own the final decision that the evidence is sufficient for the claimed behavior.

Summary
Data poisoning can be easily classified as cybersecurity due to the vivid adversarial examples, but for safety-related AI, this classification is incomplete. The safety concern is whether the assurance argument can explain why the learned behavior should be trusted when the data behind it has been challenged, changed, filtered, or accepted with exceptions.

Clean validation metrics alone do not answer this question. Standards and regulations serve as useful justifications, but they should support the engineering argument rather than dominate it. A stronger release review connects dataset origin, transformations, exceptions, integrity testing, and release consequences back to the specific safety claim.

Data poisoning belongs in the safety case because the release question is simple: can the team show that the data behind the model still supports the safety claim it is asking others to trust? You should ask the same question about your own systems. If the answer is not a confident yes, then addressing that gap should be the next step in your work. The team at Reynolds & Moore thinks about these questions, and are always glad to compare notes with others working on them.

References

1. M. Goldblum et al., “Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 2, pp. 1563–1580, Mar. 2022, doi: 10.1109/tpami.2022.3162397.
2. S. Abrecht, A. Hirsch, S. Raafatnia, and M. Woehrle, “Deep Learning Safety Concerns in Automated Driving Perception,” IEEE Transactions on Intelligent Vehicles, vol. 10, no. 2, pp. 1455–1466, Jul. 2024, doi: 10.1109/tiv.2024.3428415.
3. S. Burton and B. Herd, “Addressing uncertainty in the safety assurance of machine-learning,” Frontiers in Computer Science, vol. 5, Apr. 2023, doi: 10.3389/fcomp.2023.1132580.
4. N. Pitropakis, E. Panaousis, T. Giannetsos, E. Anastasiadis, and G. Loukas, “A taxonomy and survey of attacks against machine learning,” Computer Science Review, vol. 34, Oct. 2019, doi: 10.1016/j.cosrev.2019.100199.
5. X. Han, G. Xu, Y. Zhou, X. Yang, J. Li, and T. Zhang, “Physical Backdoor Attacks to Lane Detection Systems in Autonomous Driving,” in Proceedings of the 30th ACM International Conference on Multimedia, Oct. 2022, pp. 2957–2968. doi: 10.1145/3503161.3548171.
6. Ma, Hua, et al. “TransCAB: Transferable clean-annotation backdoor to object detection with natural trigger in real-world.” 2023 42nd International Symposium on Reliable Distributed Systems (SRDS). IEEE, 2023, doi: 10.1109/SRDS60354.2023.00018.
7. K. Messaoud, K. Grosse, M. Chen, M. Cord, P. Pérez, and A. Alahi, “Manipulating Trajectory Prediction Models With Backdoors,” IEEE Transactions on Intelligent Transportation Systems, vol. 26, no. 12, pp. 22962–22973, Nov. 2025, doi: 10.1109/tits.2025.3616645.
8. A. D. M. Ibrahum, M. Hussain, and J. Hong, “Deep learning adversarial attacks and defenses in autonomous vehicles: a systematic literature review from a safety perspective,” Artificial Intelligence Review, vol. 58, no. 1, Nov. 2024, doi: 10.1007/s10462-024-11014-8.
9. N. I. of S. and T. NIST, “NIST AI 100-2 E2025 publication page, adversarial ML terminology and poisoning-related classes,” National Institute of Standards and Technology, 2025. [Online]. Available: https://csrc.nist.gov/pubs/ai/100/2/e2025/final.
10. E. Union, “Data and data-governance requirements for high-risk AI systems,” Regulation (EU) 2024/1689. 2024. [Online]. Available: https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng.
11. I. O. for Standardization, “Road vehicles, safety and artificial intelligence,” International Organization for Standardization, 2024. [Online]. Available: https://www.iso.org/standard/83303.html.
12. ISO/IEC TR 5469 Artificial intelligence Functional safety and AI systems, Jan. 18, 2024.
13. ISO/IEC CD TS 22440-1 Artificial intelligence — Functional safety and AI systems.
14. P. Schmitt, C. Shinde, S. Diemert, K. Pennar, et al., “The ML FMEA in Action: Lessons from Applications of Machine Learning Safety,” SAE Technical Paper 2026-01-0079, 2026.