3. Simulation credibility framework

Simulation-based evidence is only as persuasive as the credibility argument behind it. A regulator, notified body, or internal reviewer does not ask "did the solver converge?" in isolation; they ask whether the model is the right model for the decision at hand, whether it has been verified and validated to a defensible degree, and whether the uncertainty in its predictions has been characterized well enough to support the claim being made. Three families of guidance dominate this space: the ASME V&V standards (V&V 10 for computational solid mechanics, V&V 20 for computational fluid dynamics, and V&V 40 for medical devices), NASA-STD-7009 for models and simulations used in NASA programs, and the FDA guidance on model-informed drug development (MIDD). They share a common logic — context of use, verification, validation, uncertainty quantification, and credibility assessment — but they operationalize it differently, and a credible submission often has to satisfy more than one of them.

The shared spine: context of use drives everything

All four frameworks begin with the same question: what decision will this simulation inform? In ASME V&V 10 and 20 this is captured as the "context of use" statement, which fixes the intended application, the quantities of interest (QoIs), the operating envelope, and the required accuracy. NASA-STD-7009 uses the closely related concept of "intended use" and grades credibility along a defined scale. The FDA MIDD guidance frames the same idea as the "question of interest" and the "model influence" on the regulatory decision. The practical consequence is that credibility is not a property of a model in the abstract — it is a property of a model relative to a specific decision. A coarse reduced-order model can be fully credible for a feasibility screen and wholly inadequate for a safety margin calculation.

This is why the first deliverable in any credibility package is a context-of-use statement, not a mesh convergence study. If the context of use is vague, every downstream verification and validation activity becomes unfalsifiable.

ASME V&V 10 and 20: verification, validation, and uncertainty quantification

ASME V&V 10 (solid mechanics) and V&V 20 (fluid dynamics) organize the work into three pillars. Code verification asks whether the software solves the equations correctly — method of manufactured solutions, order-of-accuracy tests, and comparison against benchmark solutions. Calculation verification asks whether a specific simulation has been solved accurately enough — mesh and time-step refinement, iterative convergence, and discretization error estimation (e.g., Richardson extrapolation or grid convergence index). Validation asks whether the mathematical model represents reality for the QoI, by comparing simulation output against experimental data with quantified uncertainty in both.

Uncertainty quantification (UQ) runs through all three. V&V 10/20 distinguish aleatory uncertainty (irreducible variability, e.g., material scatter) from epistemic uncertainty (lack of knowledge, e.g., an uncalibrated constitutive parameter). The validation comparison is only meaningful when both the simulation prediction and the experimental measurement carry uncertainty intervals; a point-to-point match with no error bars is not validation.

A useful discipline: write the validation metric before running the experiment. If you cannot state in advance what comparison, tolerance, and acceptance criterion will constitute "validated," you are curve-fitting after the fact.

ASME V&V 40: credibility for medical devices

V&V 40 shifts the emphasis from physics verification to a structured credibility assessment tailored to medical device submissions. It introduces a risk-informed approach: the credibility evidence required scales with the model's influence on the decision and the decision's consequence (patient risk). The standard defines credibility factors — verification, validation, uncertainty characterization, use history, and input pedigree — and asks the submitter to build a credibility argument that maps each factor to the specific context of use. In practice, V&V 40 is the bridge between the mechanics-focused V&V 10/20 and the regulatory expectations of a device reviewer.

NASA-STD-7009: graded credibility for program decisions

NASA-STD-7009 standardizes how models and simulations are documented and assessed across NASA programs. Its distinctive contribution is a credibility assessment scale with defined levels (typically 0–4) across eight factors: verification, validation, input pedigree, uncertainty characterization, results robustness, use history, M&S management, and people/process. Each factor is scored, and the aggregate supports a credibility rating appropriate to the intended use. The standard also mandates a Model Development and Credibility Assessment Plan and a structured results package, so that a reviewer can trace every credibility claim to evidence.

FDA MIDD: exposure–response and model influence

The FDA MIDD guidance addresses the use of exposure–response (E-R) models, physiologically based pharmacokinetic (PBPK) models, and quantitative systems pharmacology to support dosing decisions, label changes, and waivers of clinical trials. Its credibility logic is explicitly decision-centric: the model influence (how much the model drives the regulatory decision) and the decision consequence (patient risk if wrong) determine the evidence burden. The guidance emphasizes model verification against observed data, sensitivity analysis, and a clear statement of assumptions and limitations. For PBPK, it expects platform qualification and drug-specific validation; for E-R, it expects characterization of the exposure metric, the response model, and the uncertainty in both.

How the frameworks relate

The diagram below maps the four frameworks onto a common credibility workflow, showing where each places its emphasis and how the outputs feed a single credibility argument.

Common Credibility Workflow and Framework Emphasis Workflow Context of Use / Intended Use Verification (code + calc) Validation vs. data Uncertainty Quantification Credibility Argument ASME V&V 10 / 20 Solid mechanics / CFD Code verification Validation + UQ Aleatory/epistemic GCI / Richardson Emphasis: numerical accuracy and physics fidelity ASME V&V 40 Medical devices Risk-informed Credibility factors Model influence Decision consequence Emphasis: credibility argument scaled to patient risk NASA-STD-7009 Program M&S 8 credibility factors 0–4 scale M&S plan Results package Emphasis: graded, documented credibility for program decisions FDA MIDD Drug development Question of interest E-R / PBPK Sensitivity Model influence Emphasis: decision-centric evidence for dosing and labeling

Building a credibility package in practice

Consider a concrete example: a finite element model of a hip implant stem used to support a fatigue-safety claim in a device submission. The context of use is "predict peak stress at the stem–cement interface under ISO 7206-4 loading to support a fatigue margin claim." Under V&V 40, the model influence is high (it directly supports a safety claim), so the credibility evidence must be strong. The package would include: code verification against a manufactured solution for the elastic solver; calculation verification via mesh convergence with a grid convergence index reported for the peak stress QoI; validation against a cadaveric or synthetic-bone strain-gauge benchmark with uncertainty in both measurement and material properties; and a sensitivity study on cement modulus and interface friction. Each item maps to a V&V 40 credibility factor, and the aggregate argument states explicitly which factors are strong and which are limited.

For a PBPK example under MIDD, the context of use might be "predict drug–drug interaction magnitude with a CYP3A4 inhibitor to support a waiver of a clinical DDI study." The package would include platform qualification (the PBPK platform's track record for CYP3A4 DDI predictions), drug-specific model verification against observed PK, sensitivity analysis on fraction metabolized and inhibition constant, and a comparison of predicted versus observed DDI AUC ratios across a validation set. The FDA reviewer will weigh model influence (high, since it replaces a trial) against the strength of the validation set.

Common failure modes

  • Context of use written after the fact. The model is built, then a justification is reverse-engineered. Reviewers detect this because the QoIs and acceptance criteria do not match the decision.
  • Verification conflated with validation. A converged mesh proves the equations were solved, not that the model represents reality.
  • UQ omitted on the experimental side. Validation against a single nominal test with no measurement uncertainty is weak evidence.
  • Credibility scale used as a scorecard. NASA-STD-7009 levels are not a grade to maximize; they are a communication device. A level-2 factor with a clear rationale can be acceptable if the context of use permits.
  • Framework mismatch. Citing V&V 10 for a PBPK model, or MIDD for a structural fatigue model, signals unfamiliarity with the regulatory expectations.
Do not treat any of these frameworks as a checklist to be completed mechanically. The credibility argument is a reasoned narrative that connects each piece of evidence to the specific decision. A package that satisfies every clause but never states why the evidence is sufficient for this decision will not survive review.

Practical takeaway

The four frameworks are complementary rather than competing. Use ASME V&V 10/20 for the physics and numerical verification discipline, V&V 40 when the decision is a medical-device safety or performance claim, NASA-STD-7009 when you need a graded, auditable credibility scale and a structured results package, and FDA MIDD when the decision is a drug dosing, labeling, or trial-waiver question. In every case, start from the decision, state the context of use precisely, and let the required credibility evidence follow from the model's influence and the consequence of error.

← Previous 📖 Contents Next →