AI Validation and Acceptance Criteria MOC
The conference’s center of gravity: what it means to validate a non-deterministic system. The talks converge on one shift, from validation as a documentation event to validation as a continuous, data-based process, and they disagree only about how to get there.
Transcripts
Day one:
- Bill Gargano (RCM), keynote - AI as force multiplier; nightly GMP releases that passed three FDA inspections; validation is data, not documents
- Trust Is Not a Framework (Genari AI) - observability versus evaluation; LLM-as-judge run on a different model at low non-zero temperature; judge qualification packs
- Abhishek Bachchan (Gilead) - validating a measurement system; six acceptance-test buckets; evidence as a dashboard with a shelf life
- Ben O’Brien (Phanero) - the intended use statement as control lever; four pillars; wording sets the burden; bounded inputs
- Gaurav Walia (PQE Group) - quantified manufacturing use cases; AI-driven CAPA with human-final decisions; tailored AI validation beyond waterfall
Day two:
- Binesh Prabhakar (One Vector) - from known-input/predefined-output to known-evidence/acceptable-behavior-range; golden sets, baselines, agent CVs
- Carolyn Troiano (RiskCortex) - continuous validation through real-time monitoring; playbook, Expert in the Loop, telemetry; the LIMS case study
Evergreen notes
- Validating an AI system in a regulated industry means validating a measurement system
- Validation is data, not an event
- The intended use statement is the master control lever for GxP AI
- An LLM judge is a GxP system and needs its own qualification pack
- Observability and evaluation answer different questions about an AI system
- A golden test set must include unhappy paths
- AI validation evidence has a shelf life
- The wording of an intended use statement sets the regulatory burden
- Bounded inputs make AI validation tractable
- An AI system in GxP must be able to answer unable to verify
- Validate what you can, verify the rest
- Grounding an LLM in the regulatory corpus yields defensible, novel acceptance criteria
- Centralize evaluation when every AI pairing carries its own KPIs
- Assurance must scale with autonomy