QuantHealth tests its virtual patient models against held-out trial outcomes, not the data used to train them. In this framework, real-world evidence means completed and published trials with known results. Internal consistency checks and training-set reconstruction are not treated as validation. Instead, each model undergoes a documented, repeatable retrospective test against outcomes unavailable during model development.
Why Clinical Simulation Needs Validation
Drug development simulation now uses a more carefully defined regulatory vocabulary. Qualification establishes that a model is suitable for a stated context of use, generally through adequate predictive performance on a specified validation dataset. Validation is the work that generates that performance evidence. FDA's Model-Informed Drug Development (MIDD) framework, set out across guidance documents published over the past decade, expects sponsors relying on simulation for regulatory decisions to document both the context of use and the supporting validation evidence.
For virtual patient simulation supporting Phase 2 protocol design, the stated use is to predict trial operating characteristics: the distribution of outcomes at defined doses and sample sizes. A model qualified for this purpose must show that its predicted outcome distributions cover observed trial results at appropriate probabilities. Reproducing training data is insufficient. That demonstrates self-consistency, rather than predictive validity.
This distinction has a direct design consequence. A model overfit to its training data may generate intervals that are too narrow and confidence that is too high when applied to a new setting. For drug development, that can produce an underpowered trial: simulation overstates the likelihood that a selected sample size is adequate, leading the team to enroll too few participants. Our protocol is designed to identify and correct this pattern.
Our Retrospective Validation Protocol
For each indication-specific model, we apply a five-step validation protocol independently.
Step 1: Identify a completed trial in public sources. These may include FDA advisory committee submissions posted on FDA.gov, indexed peer-reviewed Phase 2 and Phase 3 papers, and publicly disclosed clinical study reports. We require enough pre-trial compound detail to simulate from information available before the study, including pharmacokinetic parameters where available, tested dose levels, inclusion and exclusion criteria, and the primary endpoint definition.
Step 2: Retain pre-trial compound information only. The input set contains what a development team could have known before enrollment: mechanism of action, available PK parameters from preclinical or Phase 1 work, the proposed dose range, and protocol-defined patient characteristics. We exclude completed-trial information, including observed response and dropout rates and post-hoc analyses.
Step 3: Keep the trial result hidden. The run uses pre-trial inputs alone, and its operator does not see the observed outcome. Blinding is enforced through pipeline sequencing rather than an honor system. The outcome is stored separately and remains inaccessible while the simulation executes.
Step 4: Evaluate simulated versus observed results. The model generates a distribution for the primary endpoint response rate at each dose. We locate the observed trial response within that distribution and ask whether it lies inside the 90% prediction interval, then the 80% interval. These coverage comparisons provide the calibration test.
Step 5: Pool calibration results across the validation set. One trial cannot establish performance because even a calibrated model may miss an interval by chance. For the full set, we calculate the share of observed outcomes contained within the 90% prediction interval, the 80% interval, and the 50% interval. Expected behavior is approximately 90% coverage for the 90% interval, with the corresponding relationship at the other levels.
Calibration: Scope and Limits
Coverage describes the prediction interval, not the point estimate. Very broad intervals can achieve high coverage while offering limited information. We therefore assess width as well. At 90% coverage, an interval spanning 40 percentage points is less actionable than one spanning 15 percentage points, since the wider result gives weaker guidance for choosing a sample size.
Calibration does not require the simulated median, or point prediction, to match the observed trial result closely. It requires the result to fall within the uncertainty interval implied by the model. For a development team, the practical test is whether the simulation supported a properly informed sample size decision: did it identify a plausible effect-size regime and express the uncertainty around that expectation accurately?
We do not reduce performance to an aggregate hit rate, meaning the share of observed results inside a prediction interval. That metric is determined by the validation set's composition. Selecting trials with well-characterized Phase 1 PK and precedented mechanisms would favor easy cases and could exaggerate utility for compounds with sparse information. We instead provide the complete calibration curve for each indication-specific model, together with the validation trials' characteristics, through our methodology consultation package on request.
Where Validation Gaps Arise
Our continuing validation work has found three repeated sources of systematic miscalibration in particular settings.
First, the population PK model may be misspecified. With an oral compound showing non-linear absorption or a substantial food effect, a model based on fasted preclinical or Phase 1 data may not represent fed-state Phase 2 exposure when the protocol permits co-administration with food. The simulated exposure distribution then shifts relative to the trial, and the dose-response model carries that error forward. Food-effect status is now required documentation for new compound onboarding, and we modify PK simulation when fed-state data exist.
Second, disease progression assumptions may be specific to an indication but estimated from a different disease stage than the planned study. Natural history data from early disease may not describe patients with longer disease duration and greater prior treatment burden. Where the literature supports the distinction, we maintain separate models by indication and disease stage.
Third, response may depend on covariates absent from the protocol-design inputs. Examples include a genetic polymorphism affecting metabolism or a baseline biomarker associated with target expression. If such a factor also relates to enrollment patterns that differ by site or region, the simulated response distribution can systematically diverge from the trial. Repeated interval misses identify these cases and support expanding the covariate model once the predictor's biological basis is better established.
Keeping Validation Current
Validation continues after the initial analysis. As Phase 2 and Phase 3 findings enter the public domain, we add retrospective cases for each indication. A new case outside the 90% prediction interval prompts review: the miss may indicate a systematic model limitation, or a population or design feature absent from the inputs. Systematic errors lead to model updates. Idiosyncratic cases are recorded as out of range and examined for covariate information that might have brought the simulation within range.
Model parameters remain under version control. Each release is linked to the validation dataset used for calibration, and outputs from client engagements can be traced to the version that generated them. After an update, clients who used an earlier release are notified. If the change materially alters outputs for their indication, we offer re-simulation with the updated model at no additional cost.
Versioning and update procedures protect the validity of the framework over time. A model calibrated in 2024 on trials from 2015 to 2023 has a validation window that becomes progressively less representative as newer studies are published. Adding cases continuously keeps calibration claims tied to current trial practice, the comparison that matters to teams designing studies today.
Questions for Simulation Vendors
When assessing simulation for protocol design, ask first about validation. Key questions include: what prospective validation protocol is used, which trials comprise the dataset, how many indications and time periods are represented, and what coverage was achieved for the 90% prediction interval? A vendor able to provide documented answers is demonstrating a different degree of scientific accountability from one relying only on testimonials or case studies.
Simulation without formal retrospective validation can still help structure a design discussion and expose assumptions that would otherwise remain implicit. Its limitation is that the probability estimates have not been calibrated. If an internal model reports a 75% probability of meeting the primary endpoint, that figure has no attached confidence about model accuracy without calibration data. A validation dataset places the 75% estimate in context by showing how the model performed in comparable settings. The result is an interpretable probability, rather than merely a computable one.