2. Explainability requirements from engineering standards and regulations

Explainability is rarely demanded as an abstract virtue. In high-stakes engineering it appears as a traceable obligation: a standard or regulator requires that a specific decision, at a specific lifecycle stage, be justified by evidence that a competent third party can inspect. The five frameworks most relevant to AI-enabled engineering decisions — ISO 26262, IEC 61508, FDA SaMD guidance, the EU AI Act, and the NIST AI RMF — differ in legal force, scope, and vocabulary, but they converge on a common demand: the reasoning behind a safety-relevant output must be reconstructible from artifacts, not from the memory of the engineer who produced it.

What each framework actually asks for

ISO 26262 governs functional safety of electrical and electronic systems in road vehicles. It does not mention machine learning by name in its original 2011 edition, but its ASIL-driven requirements apply directly to any AI component whose malfunction can contribute to an unreasonable risk. The explainability-relevant obligations are: a documented safety concept with traceable links from hazard to safety goal to technical requirement; verification evidence that each requirement is met; and a rationale for why the chosen measures are sufficient. For an AI-based perception or planning function, this means the developer must be able to show why a given output was considered safe — typically through a combination of requirement decomposition, data coverage arguments, and runtime monitoring. ISO 26262 does not accept "the model performed well on the test set" as a safety argument; it demands a structured, auditable chain.

IEC 61508 is the generic functional safety standard from which ISO 26262 derives. Its SIL concept and its emphasis on systematic capability make it the reference for industrial AI in process control, machinery, and rail. The standard requires a safety requirements specification, a verification plan, and evidence of systematic fault avoidance. For AI, the practical consequence is that any learned component must be wrapped in a specification that states its intended function, its failure modes, and the evidence that those failure modes are controlled. Explainability here means the ability to map a model output back to a requirement and forward to a verification result.

FDA SaMD (Software as a Medical Device) guidance, particularly the 2021 AI/ML-based SaMD action plan and the Predetermined Change Control Plan (PCCP) concept, requires that a device's clinical decision logic be transparent enough for review. The FDA does not mandate a specific explainability technique, but it expects a description of the algorithm's inputs, outputs, and limitations, plus a plan for monitoring performance after deployment. For a diagnostic AI, this translates into documented training data provenance, performance across relevant subgroups, and a clear statement of the intended clinical use and contraindications.

EU AI Act introduces a risk-tiered legal framework. High-risk AI systems — including those used in safety components of critical infrastructure, medical devices, and automotive — must satisfy transparency and human-oversight requirements. Article 13 requires that high-risk systems be designed so that deployers can interpret the system's output and use it appropriately. Article 14 requires effective human oversight, which presupposes that the human can understand the basis of a decision. The Act does not prescribe SHAP or LIME; it prescribes that the information needed for oversight be provided.

NIST AI RMF is voluntary but widely adopted as a structuring tool. Its four functions — Govern, Map, Measure, Manage — provide a lifecycle frame in which explainability is a measurement concern: the organization must define what interpretability means for its use case, measure it, and manage the residual risk. The RMF's companion playbook suggests documenting model cards, data sheets, and decision logs as evidence.

From obligation to artifact: a traceability chain

The practical bridge between these frameworks is a traceability chain that runs from a hazard or clinical risk, through a safety or performance requirement, to a model output, and back to the evidence that the output is acceptable. The diagram below shows this chain as it applies to an AI-based lane-keeping function under ISO 26262, with the corresponding EU AI Act and NIST RMF touchpoints annotated.

Traceability chain for an AI lane-keeping function Hazard Unintended lane departure Safety Goal ASIL D: prevent departure Technical Requirement Lane detection accuracy ≥ 99.5% Verification Test evidence Model Output Steering angle command Explanation Artifact Saliency + decision log Human Oversight Driver / safety monitor ISO 26262 Safety case, traceability ASIL decomposition EU AI Act Art. 13 transparency Art. 14 oversight NIST AI RMF Measure interpretability Manage residual risk IEC 61508 SIL capability Systematic fault control feedback / monitoring Legend Forward traceability Feedback / monitoring

How to satisfy these requirements in practice

The chain above is not a diagram exercise; it is a working method. For an AI component, the engineer should produce four artifacts that map directly onto the frameworks:

  1. Requirement-to-output mapping. For each safety or performance requirement, record which model output or internal signal is the observable proxy, and what threshold constitutes compliance. In the lane-keeping example, the requirement "lane detection accuracy ≥ 99.5%" maps to the model's lane-boundary output, measured against a held-out set with a defined acceptance criterion.
  2. Explanation artifact. For each decision that a human must oversee, produce a human-readable justification. This can be a saliency map, a counterfactual ("the vehicle would have stayed in lane if the left boundary confidence had been above 0.8"), or a rule-based summary. The artifact must be generated at runtime or in a post-hoc analysis that is reproducible from logged inputs.
  3. Verification record. Document the test cases, the pass/fail criteria, and the observed results. Under ISO 26262, this record is part of the safety case; under the EU AI Act, it is part of the technical documentation; under NIST RMF, it is the Measure function's output.
  4. Monitoring and feedback. Define what runtime signals trigger a review — for example, a drop in lane-detection confidence below a threshold, or a divergence between the model's explanation and the expected causal factors. This closes the loop and satisfies the Manage function of the NIST RMF and the post-market surveillance expectations of FDA SaMD.
Practical note: The most common failure mode is treating explainability as a post-hoc report rather than a design input. If the explanation artifact is generated only after the model is frozen, it often cannot be traced back to a requirement, and the safety case becomes a narrative rather than evidence. Build the mapping and the logging hooks into the model interface from the start.

Common pitfalls and how to detect them

Three failure patterns recur across all five frameworks. First, requirement drift: the model is trained to optimize a metric that is not the safety requirement, and the explanation artifact explains the metric rather than the requirement. Detection: check that every explanation references a requirement ID. Second, opaque thresholds: the acceptance criterion is a number without a rationale, so a reviewer cannot judge whether it is adequate. Detection: require a written justification for each threshold, including the data and assumptions behind it. Third, missing feedback: the monitoring signal exists but is not connected to a review action. Detection: trace each monitoring signal to a documented response procedure.

These frameworks do not demand a particular explainability algorithm. They demand that the reasoning behind a high-stakes decision be reconstructible, traceable, and reviewable. The engineering work is to build the chain — hazard to requirement to output to explanation to evidence — and to keep it intact as the system evolves.

← Previous 📖 Contents Next →