4. Evidence sufficiency and confidence

In the subsystem-to-vehicle evidence hierarchy, the question "do we have enough evidence?" is not answered by counting test reports. It is answered by asking whether the accumulated evidence supports a decision at a stated confidence level, given the risk of being wrong. This section establishes the statistical reasoning that connects sample size, observed outcomes, and decision thresholds, and shows how to translate that reasoning into defensible release criteria across subsystem, system, and vehicle levels.

From Evidence Quantity to Evidence Sufficiency

Evidence sufficiency has three dimensions that must be satisfied simultaneously. Coverage asks whether the evidence exercises the relevant operating conditions, configurations, and failure modes. Statistical strength asks whether the number of independent observations is adequate to detect the effect size of interest. Traceability asks whether each observation can be linked to a requirement, a test configuration, and a vehicle build level. A test campaign with 500 passes that never exercises the failure mode of concern has high statistical strength but zero coverage; a campaign that covers every mode once has coverage but no statistical strength. Sufficiency requires all three.

The hierarchy matters here. Subsystem-level evidence typically provides high statistical strength under controlled conditions but limited coverage of vehicle-level interactions. Vehicle-level evidence provides realistic coverage but usually low sample counts because each vehicle test is expensive. The confidence argument must therefore combine levels rather than demand sufficiency at any single level.

Statistical Basis: What a Pass/Fail Sample Actually Proves

For a pass/fail characteristic with no failures observed in n independent trials, the one-sided upper confidence bound on the failure probability p at confidence level C is:

p_upper = 1 - (1 - C)^(1/n)

Example: n = 30, C = 0.90
p_upper = 1 - (0.10)^(1/30) = 1 - 0.9261 = 0.0739  →  7.4%

This is the single most important number in evidence planning: 30 consecutive passes at 90% confidence only bounds the failure rate below roughly 7.4%. To demonstrate a 1% failure rate at 90% confidence with zero failures requires n = ln(0.10)/ln(0.99) ≈ 230 trials. Teams that treat "30 samples, no failures" as proof of a 1% reliability target are making a statistical error of more than an order of magnitude.

When failures do occur, the bound shifts to the binomial or, for time-to-failure data, to a Weibull or exponential model. For the common Weibull case with shape parameter β and characteristic life η, the demonstrated B10 life at confidence C from a zero-failure test of duration t with n units is:

B10_lower = t * ( -ln(0.90) / (n * -ln(C)) )^(1/β)

Example: n = 12, t = 2000 h, β = 2.0, C = 0.90
B10_lower = 2000 * (0.10536 / (12 * 0.10536))^(1/2)
         = 2000 * (1/12)^0.5 = 2000 * 0.2887 ≈ 577 h

The lesson is that the assumed β drives the answer as much as the sample size does. β must be justified from field data, similar programs, or physics-of-failure analysis, and the assumption must be recorded alongside the result.

Sample Size Planning Under Risk

Sample size should be derived from the decision, not chosen for convenience. The planning sequence is: define the failure probability p_d that must be detected, the confidence C required, the acceptable consumer's risk β (probability of accepting a bad population), and the producer's risk α (probability of rejecting a good population). For a zero-failure acceptance test:

n ≥ ln(β) / ln(1 - p_d)

Detect p_d = 0.05 at β = 0.10:  n ≥ ln(0.10)/ln(0.95) ≈ 45
Detect p_d = 0.02 at β = 0.10:  n ≥ ln(0.10)/ln(0.98) ≈ 114

Where failures are permitted, the binomial acceptance criterion (accept if c or fewer failures in n) allows smaller n at the cost of weaker discrimination. The choice between zero-failure and c-failure plans is itself a risk decision: zero-failure plans are simple to administer and give strong per-unit evidence, but they are brittle — a single early failure forces a large re-test.

Risk-Based Decision Criteria

Not every requirement deserves the same confidence. A risk-based framework assigns each requirement a severity class (S0–S3, where S0 is safety-critical and S3 is cosmetic), and each class carries a minimum confidence and maximum acceptable demonstrated failure rate. A representative scheme:

The decision rule is then mechanical: for each requirement, compute the demonstrated bound from the available evidence at each hierarchy level, combine levels using a Bayesian or conservative worst-case rule, and compare against the class threshold. If the bound exceeds the threshold, the requirement is not demonstrated — regardless of how many tests passed.

Combining Evidence Across the Hierarchy

Subsystem and vehicle evidence are not interchangeable, but they can be combined. A practical approach is a two-stage Bayesian update: use subsystem test results to form a prior on the failure probability, then update with vehicle-level results. If subsystem testing of 40 units yields 1 failure, the posterior mean under a Jeffreys prior is approximately (1 + 0.5)/(40 + 1) ≈ 3.7%. Feeding that as a prior into a vehicle test of 8 units with 0 failures yields a posterior mean near 2.9% and a 90% upper bound near 6.5% — still above a 1% S1 threshold, which correctly signals that vehicle-level evidence is insufficient and that either more vehicle testing or a stronger subsystem argument (e.g., physics-based extrapolation) is required.

The combination must respect independence. If the same design flaw is present at both levels, the two evidence streams are correlated and the combined confidence is lower than the naive calculation suggests. Independence should be argued from different failure mechanisms, different operating conditions, or different build configurations — not assumed.

Practical Failure Symptoms and Corrective Actions

Three failure patterns recur in evidence reviews. First, the confidence gap: the demonstrated bound is above the class threshold but the team reports "all tests passed." The corrective action is to recompute the bound explicitly and present it in the evidence summary, not to add tests blindly. Second, the coverage illusion: high sample counts concentrated in a narrow operating envelope. The corrective action is a coverage matrix mapping each requirement to the conditions exercised, with gaps flagged before sample size is increased. Third, the correlation trap: subsystem and vehicle evidence share a common design assumption, so combining them overstates confidence. The corrective action is to document the independence argument for each evidence pair and to discount correlated streams.

A defensible evidence summary therefore contains, for each requirement: the severity class, the threshold, the evidence streams used, the sample size and failure count per stream, the assumed distribution parameters, the computed demonstrated bound, and the independence justification. When that table is complete and the bounds sit below the thresholds, the evidence is sufficient — and the confidence claim is one that a reviewer can independently reproduce.