1. High-stakes engineering decisions

An engineering decision is high-stakes when its consequences are severe, hard to reverse, and borne by people who did not make the decision. A misconfigured autopilot threshold, an under-designed pressure vessel, or a mislabeled training set for a medical triage model can each produce fatalities, environmental damage, or regulatory sanctions. What separates these from ordinary engineering choices is not the sophistication of the mathematics but the structure of the risk: low probability, extreme severity, and asymmetric accountability.

This chapter establishes the vocabulary the rest of the course depends on. We define risk in a way that is operational rather than rhetorical, distinguish the several meanings of accountability that appear in regulated industries, and locate precisely where AI changes the decision problem — and where it does not.

Defining Risk Operationally

In safety-critical practice, risk is conventionally decomposed as the product of severity and likelihood, often expressed through a risk matrix or a quantitative target. Aerospace and medical device standards typically work with a tolerable risk concept: a residual risk level that society accepts in exchange for the benefit of the system. The key engineering consequence is that risk must be allocated — broken down from a system-level target into budgets for subsystems, software, and human operators.

Consider a simplified example. A system-level tolerable risk for a hazardous event is set at 1 × 10⁻⁷ fatalities per operating hour. If the hazard requires two independent failures to occur, and one barrier is allocated 1 × 10⁻³ probability of failure on demand, the remaining barrier must achieve roughly 1 × 10⁻⁴. These numbers are illustrative, but the arithmetic is the point: risk targets propagate into concrete design requirements, and every component — including any learned model — inherits a numerical obligation.

This is where AI introduces friction. A neural network classifier does not naturally produce a probability of failure on demand. It produces a score. Converting that score into a defensible risk contribution requires calibration, validation on representative data, and an explicit treatment of distribution shift. Without that conversion, the model cannot be placed on the risk budget at all.

Three Faces of Accountability

Accountability in regulated engineering is not a single concept. It is useful to separate three layers, because AI systems stress each one differently.

• Legal accountability — who is liable when harm occurs. In aviation this is codified through certification (DO-178C for software, DO-254 for hardware); in medical devices through FDA premarket pathways; in automotive through ISO 26262 and its successor ISO 21448 for intended functionality. Legal accountability attaches to organizations and named individuals, never to models.
• Moral accountability — who ought to answer for the decision in a broader ethical sense, including cases where the law is silent or lagging. This layer is where questions of fairness, consent, and dignity live.
• Epistemic accountability — who can explain and defend the reasoning. This is the layer most directly affected by AI, because a model's internal computation is not naturally auditable in the way a control law or a lookup table is.

Explainable AI (XAI) is primarily an instrument of epistemic accountability. It does not transfer legal liability away from the deploying organization, and it does not by itself satisfy moral obligations. Confusing these layers is one of the most common failure modes in AI assurance programs: teams produce saliency maps and believe they have addressed accountability, when they have only addressed one third of it.

Where AI Changes the Decision Problem

AI is not uniformly applicable across high-stakes domains. The relevant question is not "can a model predict this?" but "what role does the model play in the decision chain?" Three roles recur.

The explainability requirement tightens as the human moves further from the loop. An advisory model can tolerate a coarse explanation because a trained operator supplies the missing reasoning. A direct-control model must be verifiable against a specification, because no human is present to catch a wrong output in real time. This is why the automotive and aerospace industries lean on formal methods and runtime monitors rather than post-hoc saliency for the innermost control layers.

Regulatory Context and Its Consequences for XAI

Regulation shapes what counts as an acceptable explanation. The EU AI Act classifies many safety-critical applications as high-risk and imposes documentation, human-oversight, and transparency obligations. The FDA's guidance on AI/ML-based Software as a Medical Device emphasizes a predetermined change control plan, which forces developers to describe in advance how the model may evolve and how that evolution will be justified. In aviation, EASA's AI roadmap distinguishes levels of AI involvement and requires increasing assurance evidence as autonomy grows.

The practical consequence is that an explanation is not a research artifact. It is a regulatory deliverable with a defined audience, a defined format, and a defined retention period. A SHAP summary plot may be useful to a data scientist but is rarely sufficient for a notified body. What regulators typically want is a traceable argument: this input feature, this model behavior, this design rationale, this residual risk, this monitoring plan.

An Illustrative Decision Chain

The diagram below traces a single high-stakes decision — a model-issued alert in a safety-critical monitoring system — from input through model, explanation layer, human judgment, and regulatory record. It shows where risk is allocated, where accountability attaches, and where explainability enters. The point is not that every system looks like this, but that the explanation layer sits between the model and the human, and its output must be legible to both.

The dashed red path is the part most often missing in practice: the feedback from human override back into the model lifecycle. Without it, the explanation layer becomes a one-way display, and the organization loses the ability to detect systematic model error that only operators can see.

What This Means for the Rest of the Course

Three commitments follow from the definitions above. First, every XAI technique we examine will be evaluated against a specific accountability layer and a specific audience — not in the abstract. Second, risk allocation will be treated as a prerequisite: a model that cannot be assigned a probability of failure on demand cannot be certified, regardless of how interpretable it appears. Third, the regulatory record will be treated as a first-class deliverable, not an afterthought produced at submission time.

With this framing in place, the next chapters can move into the technical machinery — feature attribution, surrogate models, counterfactual explanation, uncertainty quantification — with a clear sense of what each method is for and what it cannot substitute for.