KENX AI in GxP East 2026

2026-09-15-1426-abhishek-ai-acceptance-criteria-gxp-lecture

33 min read

Abhishek Bachchan on AI acceptance criteria in GxP-regulated systems

Recorded 2026-09-15 at 1426 via Plaud. 43:29.

Summary

Talk recorded at the GxP and AI conference Eric is attending this week. The presenter, Abhishek Bachchan (Gilead (transcribed as “Brilliant Sciences”), IT security, risk and compliance), opens with why traditional CSV-style validation breaks for AI: deterministic systems return the same output for the same input and bugs reproduce consistently, while a generative system can satisfy one requirement in many ways, vary from run to run, and a human judging pass or fail quietly moves the acceptance criterion into the tester. His reframe is that validating an AI system means validating a measurement system: criteria must name a population (a versioned, change-controlled labeled test set), a risk-justified threshold, stated conditions (model version, corpus, parameters), a named harness at a specific commit, and a deviation route for failures, with dated run records rather than screenshots as evidence. He situates this in FDA SaMD guidance (locked vs adaptive algorithms, with adaptive ones needing a predetermined change control plan), the EU GMP Annex 22 draft requiring human oversight for continuously learning and generative models, and the joint FDA/EMA principles, where roughly four of ten principles touch acceptance criteria or validation. Both CSV and CSA still assume determinism, a human error-catcher, and a validated state that persists without action, so neither is sufficient for AI; he classifies AI systems on two axes, fixed vs adaptive and opaque vs inspectable, and focuses on the adaptive-but-inspectable quadrant, the dominant Gen AI use case, where prompts, responses, and performance can be tuned and monitored.

The practical core is a taxonomy of six acceptance-test buckets (retrieval of the right source, faithfulness to retrieved sources, reproducibility, boundary refusal, human-in-the-loop, and guardrail or authority respect), demonstrated on his own RAG system for chatting about regulations, warning letters, and inspections: close to a hundred labels including roughly nine drug-vs-device distractors, follow-up questions, and known-absent items to probe hallucination, with hallucination metrics catching eight failures that tuning pushed to about 99 percent. The inspection-ready validation package is a dashboard or database rather than a PDF: a labeled test set with rationales, run records, per-call provenance (requested vs served model, e.g. Opus for generation and Haiku for cheap comparisons, prompt hashes, token counts), a grounding exception log, stored output artifacts, human review dispositions, threshold rationale, and requalification records. Evidence has a shelf life: any trigger (model or provider change, prompt tweak, corpus growth, retrieval or chunking parameter changes, context-of-use shift, elapsed time) forces re-measurement in a continuous loop rather than the one-pass V model. On LLM-as-judge he cites research showing only about 76 percent agreement and quickly picked-up biases, so it belongs in calibrated triage, not acceptance evidence. Audience Q&A covers what gets handed over (a dashboard, not a zip file or PDF), the workload cost of human-in-the-loop review, and a cardiologist anecdote where AI helped a cardiac-trained doctor go from wrong 90 percent of the time on non-cardiac cases to right 70 percent of the time; Abhishek’s closing point is that AI will not get sued, the human will, so the human stays in the loop.

Plaud AI notes

App-generated, verbatim from Plaud’s Summary tab.

Date Time: 2026-09-15 14:26:38 Location: [Insert Location] Instructor: Abhishek Bachchan

Summary

This lecture presents a rigorous, inspection-ready approach to defining and defending acceptance criteria for AI systems in GxP-regulated contexts. It argues that deterministic CSV-era methods are insufficient for generative/stochastic systems and reframes validation as validating a measurement system over defined populations with declared thresholds, controlled test sets, and run-based evidence. The session situates practices within current FDA/EMA/ISPE guidance, differentiates “locked” vs. “adaptive” algorithms, and introduces a two-axis classification (Fixed/Adaptive × Opaque/Inspectable) to align validation strategies and re-validation cadence. It outlines the anatomy of defensible criteria, a taxonomy of acceptance tests (retrieval, faithfulness, reproducibility, boundary, human-in-the-loop, authority), and the components of an inspection-ready validation package composed of auditable database artifacts. Emphasis is placed on human accountability, continuous monitoring for drift and provider changes, qualifying evaluation tooling, and avoiding reliance on LLMs as judges beyond calibrated triage. The lecture underscores that regulators do not provide numeric pass/fail cutoffs; thresholds must be risk-justified, context-specific, and defended by the builder team, with periodic re-evaluation planned and executed.

Knowledge Points

  1. Why traditional validation breaks for AI
    • Expected-result vagueness: generative systems can satisfy requirements in multiple valid ways and vary across runs, defeating single-answer pass/fail checks and enabling tester subjectivity.
    • Stochastic behavior: bugs may not reproduce; “pass” means performance meets a distributional threshold over a named population; evidence must be dated runs over controlled sets, not screenshots; regression requires re-measuring distributions.
  2. The reframe: validate the measurement system
    • Population + threshold: acceptance criteria must name the evaluated population and a risk-calibrated threshold (n cases, one metric, one number to beat).
    • Controlled test sets: cases and labels are the specification; place under change control and versioning to preserve comparability.
    • Evidence as run records: capture date, harness version/commit, corpus size/composition, parameters, per-case rows, scores, and operator identity; evidence is a distribution with provenance, not a static artifact.
  3. Regulatory landscape and risk differentiation (as of Sep 2026)
    • FDA SaMD: “locked” vs. “adaptive”; adaptive requires a Predetermined Change Control Plan (PCCP) governing updates, testing, and documentation.
    • EU GMP Annex 22 (draft) and EMA: higher scrutiny and human oversight for adaptive/generative models; finalization targeted Q4 2026 (verify status).
    • In-force and guidance: FDA/EMA joint Good AI Practice (non-binding), FDA draft on AI in regulatory decisions (pending final), ISPE GAMP AI Guide (2025) and GAMP 5 2nd Ed. Appendix D11.
    • Enforcement basis: binding clauses (21 CFR 11, 211.68, 211.100; EU Annex 11; 21 CFR 820) apply without naming AI; inspectors cite procedures/records/validation, not “the model was wrong.”
  4. Acceptance criteria within FDA/EMA principles
    • Risk-based approach, clear context of use, risk-based performance assessment, and lifecycle management (scheduled monitoring and drift re-evaluation).
    • Human-centric design: interpretability, explainability, and plain-language performance/limitations; acceptance criteria must reflect human oversight.
  5. CSV vs. CSA vs. AI validation
    • CSV: scripted expected results for deterministic systems.
    • CSA: risk-based assurance, recognizes unscripted testing; still assumes determinism.
    • AI validation: measure populations against declared thresholds; maintain continuous monitoring and re-qualification; treat provider changes and drift as triggers to re-measure.
  6. Gaps CSA doesn’t solve for AI
    • Single expected result vs. distributions; intermittent failures; effort sized by consequence of error; maintaining a validated state amid silent provider changes.
    • Do not substitute vendor evidence; qualify evaluation harnesses; detect provider model reroutes/parameter shifts via monitoring.
  7. Classification framework for AI behavior
    • Axes: Behavioral Stability (Fixed vs. Adaptive) and Decision Transparency (Inspectable vs. Opaque).
    • Quadrants:
      • Fixed & Inspectable: classic validation suffices.
      • Fixed & Opaque: test behavior; no logic to review.
      • Adaptive & Inspectable: preferred “honest middle ground” enabling oversight.
      • Adaptive & Opaque: hardest; validate behavior on schedule indefinitely.
  8. Anatomy of a defensible acceptance criterion
    • Structure: “On [labeled set version under change control], the system shall achieve [metric] ≥ [threshold] under [conditions], measured by [named harness at commit], with failures dispositioned via [deviation route].”
    • Components: versioned labeled set; fit-for-use metric aligned to failure modes; threshold justified by consequence of failure; explicit conditions (model/prompt versions, corpus size, k/temperature); versioned harness; triggers for re-evaluation (model/prompt/corpus/parameters/context/time); formal deviation handling.
  9. Taxonomy of acceptance criteria
    • Retrieval & Grounding: recall@k, MRR; tests include distractors, follow-ups, and known-absent items to detect hallucination.
    • Faithfulness: ungrounded-citation rate ≤ threshold (e.g., ≤1%); automated detection via structured citations checked against context.
    • Reproducibility: byte-identical artifacts under fixed seeds/params where applicable.
    • Boundary: refusal of prohibited actions enforced by schema/guardrails.
    • Human-in-the-loop: 100% of low-confidence items receive reviewer disposition; role separation enforced.
    • Authority & Precedence: correct prioritization of binding rules over guidance with 100% pass on labeled authority pairs.
  10. Human accountability and evidence credibility
    • Humans retain legal responsibility; design workflows to capture decisions, rationales, roles, and timestamps; prevent blind acceptance.
    • Every metric must declare population, time window, operating conditions (model, prompt, retrieval parameters), and known gaps; otherwise it’s a claim, not evidence.
    • Evidence has a shelf life; schedule periodic re-evaluation per lifecycle principles.
  11. Invalidation factors and re-validation triggers
    • Model served version change; prompt change; corpus/knowledge-base change; retrieval parameter changes (k, weights, fusion, chunking); context-of-use shifts; elapsed time/drift. Record and control each; any trigger returns the system to measurement and re-qualification.
  12. Practical retrieval considerations
    • Chunking strategies differ by document type (PDF, text, spreadsheets); choices affect ranking, precision, recall; document and qualify impacts; track corpus growth and tagging quality.
  13. LLMs grading LLMs: limits and calibration
    • 2026 studies show ~76% inter-judge agreement and unreliability across safety/bias; LLM-as-judge is not acceptance evidence.
    • If used, pre-calibrate against 30–50 hand-adjudicated cases; use as triage only, with human adjudication for consequential decisions.
  14. Continuous validation loop
    • Define context of use; build and version labeled sets under change control; declare risk-justified thresholds; measure baseline and each run; attach run records to releases; monitor for drift/provider changes; re-measure on triggers; maintain PCCP for adaptive systems; qualify evaluation tools.
  15. Inspection-ready validation package (database-first)
    • Labeled test set with rationales; run records (date, harness commit, corpus size/composition, operator); per-call provenance (requested vs. served model, prompt hash, token counts); grounding exception log; stored output artifacts with immutable lineage; human review dispositions (append-only); threshold rationale tied to risk; re-qualification records for each trigger. Deliver as an interactive dashboard enabling drill-down.

Assignments

  • Define context of use for each AI system; align acceptance criteria to role, scope, and risk.
  • Build, version, and place labeled test sets (cases, labels, parameters) under change control; include distractors, follow-ups, and known-absent items.
  • Derive defensible thresholds based on influence × consequence and fit-for-use metrics; document rationales the team can defend.
  • Establish structured acceptance criteria using the full anatomy (set, metric, threshold, conditions, harness, triggers, disposition).
  • Measure baselines and subsequent runs; attach dated run records to release decisions.
  • Implement continuous monitoring for drift, corpus changes (RAG), provider model changes, and prompt/parameter tweaks; define triggers for re-qualification.
  • Qualify evaluation harnesses/tools; maintain qualification records.
  • Classify systems on Fixed/Adaptive × Opaque/Inspectable; set validation strategies and re-validation cadence accordingly; establish PCCPs for adaptive algorithms.
  • Update SOPs to reflect continuous validation, run-based evidence, deviation handling, and batch investigation extensions.
  • Enforce human-in-the-loop checkpoints for critical decisions with recorded rationales and role separation.
  • Standardize metric reporting with population, time window, operating conditions, and known gaps.
  • Implement configuration management: hash/version prompts; log served model versions; record corpus composition and retrieval parameters each run.
  • Calibrate any LLM judge and restrict use to triage; require human adjudication for acceptance evidence.
  • Audit and document chunking strategies across document types; test impacts on retrieval quality.
  • Re-check EMA/FDA guidance status before audits or talks (e.g., Annex 22 timelines; FDA AI guidance updates).

Transcript

[00:00] Speaker 1: Criteria for AI validation with Abhishek Bhattachan. Does it get presentation here?

[00:23] Speaker 2: Hello everyone. I’m Abhishek Bachchan. I work at Gilead (transcribed as “Brilliant Sciences”). The IT security risk and compliance. Somebody was just mentioning probably five different nouns in our department name. So my topic today is very focused: how to define the acceptable area and keep our AI systems. Guys, throughout the day we have learned about different AI systems, what has changed on the regulatory side of that. How difficult it can be to validate an AI system. So we’ll take one step further and try to make sense of what type of acceptance criteria we can have in order to deem our AI systems validated. And we can engage in discussion because I know all the experts are here already. So we can have a good discussion on that. Um. So, I guess when we look at AI systems, the the main issue starts with traditional validation. We have an expected result. We call it the data steps, and we get that result. Then our actual if we get that actual results, our expected results passes. So, for example, the system shall do this or display that correct answer. But the challenge with AI system is how do we write that? If the system itself can meet that requirement in five different ways, and in a different way, if we are running it tomorrow, five years later, and things like that. So, and and who judges it? Because if a human is judging the pass or fail. Then, for an AI system, the actual acceptance criteria is in the person who is testing for it. So, for an AI system to be tested, it is generally an another AI set of rules that can that can effectively test it. Okay. For a deterministic software, the the regular. Applications which are rule-based, even if they are automated or they are, they have some some in inbuilt intelligence built into them. Same input twice, same output. Generative system, it is an acceptable output, but it may or may not be same. So the pass is it matches the expected results. And for an AI system, it falls within a distribution of acceptable outcomes. And there can be one test, so we can go over the whole test. But you you see the point. We can even reproduce a bug in our deterministic systems in a very consistent manner. For an AI system, if if it runs into an issue, in order to reproduce that issue. That issue is not that not that easy. There’s a chance if it will make the same mistake again, unless we have some type of guard guardrails in place. So, so actually, when we are validating an AI system, we are not validating an algorithm or an input. We are actually validating a measurement system that is going to measure that AI system. And the output of it. The the criteria names a population and threshold. This this the population of my possible outcomes, and this the threshold, the amount of percentage of the times it needs to give me that type of response for for it to be acceptable for me. We can we can fine tune that threshold depending on the criticality of the system. Or whether you want the system to make some judgment calls or give a suggestion or not, like a chatbot, even if it can be related to GXP type of use case, or it has to be very deterministic, where there can be higher risk for a patient or or a product, then then you have a very tighter threshold there, and the test set. The set of controlled statements we can call it labels or other parameters that we need. That actually also becomes a controlled set of document. So the the document that we are going to test an AI system with should also be kept in a controlled state and a change control because any change there may or may not mean the same results when you are going to test it again. And your evidence is actually a run record, a set of data points, set of metrics, the type of output it gives you — not just a screenshot or or just one report. It’s it’s a variation of reports and and distribution of the outcome that we get becomes the evidence. So,

[06:04] Speaker 1: Two axes.

[06:15] Speaker 2: No. So if it changes, you make a note of that change. So on one day you have a set of labels or acceptance criteria that you are going to feed into an AI system. That that set is good for that day for that run. If you are going to fine tune to test that AI system again because you found something, you saw some MRRs. You want to fine - tune that data point, then probably you add 20 more labels or 20 more acceptance criteria to your testing dataset. That becomes a new set. So, you version. You keep in a control state.

[07:04] Speaker 1: The algorithm, the AI itself, will learn from each set of data. There’ll be upwards.

[07:13] Speaker 2: So , Yeah, and another fair point, but that depends on the last two items you will see here, whether your AI system is adaptable or not. It it can be a fixed one. It can still be a gen AI, generative AI, where the output can still vary if Anthropic changes the model. If it if it if, If the temperature of model changes based on our use, but we have not deliberately added a relearning feedback, if we have not, still the response can change. But we are not making a model where it is learning itself; it is still given a fixed one. So, in in this metrics, generally our normal. Normal software validation is if it is fixed and inspectable, like set of rules, set of outputs. Then, then our normal software validation, traditional software validation works. If it is on what we have on the left side, if it is okay, we can just monitor it because we don’t know how AI is gonna behave and give output by any focus, and that will. Purpose of my presentation will be is the right one, adapted and inspected. So, so with the live data, it can adapt based on the model that we choose. But we can still inspect it. We can fine tune the prompts. We can fine tune how the responses are going in and coming out, and we can monitor the performance. So. That has been the biggest use cases nowadays with the Gen AI, and so that’s what my presentation will focus on right here. And the regulators have also conveyed similar message, but in their own way. So, for example, for FDA, DSN, an adaptive algorithm needs a. We determined change control plan on file, and NXLN X22 draft that says dynamic, continuously learning, and generative model human oversight required. So, kind of talking to the same same item that you saw in the previous screen, but in their own different way. This is a subset of regulations or. Industry practices that are already coming in. Probably you have seen it multiple times throughout the day today. I don’t want to spend a whole lot of time here. This was the newest one that that came, and I’m sure you are aware of it. Is the combined FDA EMA principles that they are asking all pharmaceutical companies and. Companies in this domain to abide by. Everything is around risk-based, around validation and control of the I system. So among these ten principles, if you take a deeper dive, almost four of them are around acceptance criteria or validation of the system in one way or another. So, wanted to spend couple of minutes on the CSA, CSB discussion. There was CSB moved to CSA, CSA got also changed a couple of times. Now we have AI related changes that are coming. So, so the latest one. Is the NX twenty two so CSV print determine input keep the mind output manage the validation follow the B model CSA it you can take happens with the testing you can leverage vendor provided testing add some flexibility not so document heavy focus on the end result but still it’s not sufficient for AI validation. So we have to measure a population against a declared threshold, in order for us to validate an AI system. So, still CSB and CSA, both in the current format, are not there yet for AI validation, based on the reasons that we already kind of discussed. There that. Even with the current update, there’s still certain rules, certain items that are not very addressed. CSA, CSV still assumes a human is gonna catch an error. It assumes we can, we can probably easily leverage the vendor testing to reduce our in-house testing. But in case of AI, everything is based on our use case. How much we are gonna test and run it. And it also assumes once you validate it, the system stays validation. Even if you don’t do anything. In case of AI, even if you don’t do anything, it can give you a different answer on a different run, even five minutes later. So, if if we the intent will be, but bite by bite, compares and may or may not, or may not be the same. And the tool that we are going to use to test in CSS, CSV world, we expect that tool to be also qualified. In case of AI, because the A. Solution is changing that much. The results can change. The testing tool has to adapt itself too. It can be in a set of control. However, for us to for us to keep a control on the testing dataset, it it can be kept in a controlled state. But to keep it in continuous validation, we have to take a slightly different approach than normal CSACSBF. So. So in a V model, we do it once. In AI system validation, we it’s a continuous effort where we define our content reviews, we build a label set. It’s our requirements or acceptance criteria, and we come up with a threshold that the system has to meet. We measure the baseline. This is how the system behaved. On a particular day, for a particular set of parameters, for a particular set of test results, and then we monitor it with each run, and and we and each run record becomes our evidence, and then we monitor the drift and the provider change. If we change our corpus in the rag system. Or we change the we are trying out a different model, or we are tweaking our prompting, then that also changes the behavior of the of the application. Even if we are changing the prompt for one location, it may or may not have impact on the other parts of the system. So, we also gave the seven step. Credibility framework. It’s a bit territory there, but the main item that we have to create is the assessment plan. But they don’t give you the exact number, like what should be the threshold for passing for a particular AI system. That has to that has to come from, Come from the users who have to decide the threshold, and because none of the regulators can can give us that number, it is dependent on our use cases. Inspectors haven’t been even though AI has been in use for some time now. They are they they are still fighting on where. A gap in AI is still a gap in the quality control unit or return procedures. We have recently seen about four audit observations in the cyber space. One is the very popular one, where where they directly mention the AI system as not being validated and, The company not giving satisfactory answer about why they are using the system dominated. So, in overall, this this statement that you see on the screen, this can be the anatomy of a defensible and accepted criteria. So, on label set, just kept under change control. The system shall achieve a certain MRR, certain K, we call it K, which is greater than certain threshold that we have decided that we will be finding under stated conditions. Our stated conditions will be the model version that we are using, the corpus that we are using, how much data we have in there, because the corpus can also change over time. And measured by a named harness at commit one, named harness at commit two is just just a name for our test data set that we are going to use with minimum student deviation route. If we run into a failure with this type of test, we have to take a deviation route in order for us to fix it. What we are going to change. And that change itself can also impact other parts of the AI system, so it has to. It is much more defined than just the defect report that we do for the CSV system. And then we have to re reevaluate our acceptance of that system with any of the triggers that change. Triggers mean any of the stated conditions. If they change, that’s a trigger for us to do. This re-evaluation again. So, just in a different format, we said what data, data number, what conditions, what net. So, essentially, in a traditional way, how we write our acceptance criteria or a testing objective. For example, the assistant shall provide accurate and helpful summaries of source documents as confirmed by SME review. This can be a user requirement or a functional requirement. But in case for an issue for it to be really testable for IT department to really understand it and also keep a documented version of it, it has to take care of. It should be taking care of all the items that we previously discussed about. So it it should be talking about the dataset. It should be talking about the model that we use, and and our threshold for acceptance. In our previous statement, there is no set, no metrics, no threshold, and no disposition method there, other than the defective reporting. So, if we break that specific criteria into different buckets, probably it can be into six different ones. One is retrieval: did find the right source. Second one: did it stay inside those sources? Did it performance? Do two readers see the see one record? Whether how reproducible the AI system is. And does it refuse what it must not do? Are we setting the boundaries correctly? And it’s also a belief that if we, if we ask LLM in form of a prompt, do not do that or stay within this limit, we assume it’s going to follow. But in LLM’s mind, it hasn’t mind. It is still a request. It may or may not abide by your request for for a prompt change. It’s still a control. Most of the time, it should still work. But to for us to be sure, in addition for that prompt change, there has to be further guardrail that we should put in that cannot be missed. Either we put. It understands numbers much better than the instructions, so it can use the numbers as a very concrete guardrails that it cannot breach. Or we just put system boundaries that this the data that you are allowed to use. This the output you are allowed to report in. If anything beyond this, don’t report or tell us what you found. Things like that. And whether it is intelligent enough to include human in the loop in appropriate manner, for certain systems, human in the loop should be 100 percent. Anything the AI system suggests, there can be human to approve it. For for systems. Which can be risk-based, where we are comfortable enough, based on our experience with the AI system, to allow certain decisions to be made by the AI system. We can attach a threshold for if whatever you are gonna recommend or whatever you have found, if it does not fall in this threshold, then mark that item for a human for a human review. That is that is for kind of flexible systems where. We are okay with certain decisions, but not okay with some of the decisions based on. So, if the AI system thinks I think this is the answer, but I’m not that confident, and those can be get marked for a human in the loop approval. And so, again, the sixth one is again about the guardrails that we are going to put in. Is it respecting those guardrails? And is it respecting those binding rules that we have provided? This one is just example numbers and personal project I’ve been working on, and it’s a rag system to do chat about regulations, about warning letters, inspections, and also creating. Documents based on the regulation. So, in this particular example, what I did, I had a set of what is this? Close to hundred labels or acceptance criteria. So, seventy were normal statements or labels that I want. I want the AI, which I think the AI can easily answer. So, and there will be like nine distractors. I will. I’m asking about a drug-related question, but the statements are so close that an AI can easily confuse it with a device-related question. So that is deliberately put in, in in a manner where the statements are 90% close for drug and device, and I put it to distract the distract the LLM or distract the system. There can be some follow-up related questions put in there, based on the previous response. Respond about this, so that whether it is giving the right response there or not, and we can also add questions about the data we know is not present in the system. Certain regulations we have not ingested yet. Certain documents that are not created yet. Whether whether that. AI system is gonna hallucinate, give us still some answer whether it is gonna manufacture certain answers, even though we have instructions, do not manufacture mistakes. So, in those cases, what type of recall rate we have, what type of MMR and what type, what type of authority pass that data set when run can still give us that number and we can keep track of it. Okay. So, so the hallucination metrics that can easily put in there. For example, the answer limit markers in my in my example case. It does it compare against what was retrieved or not? We subtract declared trend knowledge just to. Just to confuse the AI during testing, and and the source cited from memory or presented as retrieved. So in in my test scenario, still in eight cases after two three runs, it still failed, and then we I I we had to make some adjustments for it to make sure it’s close to close to now. Ninety-nine percent, or so. This is the human-in-the-loop statement that we talked about. We have to assign our expected thresholds that we want to follow, so that the human-in-the-loop control or human review can be properly triggered based on the testing that we are going to do, and so. So when we talk about a validation package for an AI system, actually this will be our set when when we deliver the validation set for our for our Gen AI system. So there will be a labeled test set which will keep our our run records become our evidence with with. All the parameters were already put in there. Exactly in what conditions they were, they were run, and there will be per call provenance. So, if it is a very complex AI system, that in order to save on tokens or just to make it faster, there can be certain costs which will go to a cheaper model. Versus, there can be certain calls that can go to a more expensive and more intelligent like for generation. It can go to Opus. If it is just a small decision comparison type of statement, probably can go to Haiku. I’m talking in terms of anthropic models. So all those data points need to be included in that report that we create. And then the grounding exception log. So every every call that cited outside sources, it’s also present there and openable. Stored output artifacts with prompt hash and model superseded, and these are not deleted. Human review dispositions, the threshold rationale, and the requalification record. Which you will see is much different from our normal CSB outputs that we do for our deterministic systems.

[29:48] Speaker 3: I have a question. Yeah. Regarding you know what what we hand over, right? Like me being relatively inexperienced in the space, is it that we hand over a zip file, a tarball, or we hand over a document with, Links to the systems of record or something else. I’m just kind of trying to figure out, like practically, what what do we do here for these these things?

[30:14] Speaker 2: Yeah, yeah, no, no valid point. And and depending on the size of the output, the how how comfortable you want to be with the AI system, probably your test set is gonna have hundreds of variations. So generally. The package itself becomes a webpage. So each run, what type of test set was used, different parameters that you said—it’s like a dashboard. Okay. It will have the MRRs and the recall rate. To print them out in PDF, which will be the obvious thing we think of in case of approved documents. But it’s not that efficient if we keep it that way. If it is kept electronic, we can sort it. We can we can do more research about how we ran it. So when I did it, it was still a dashboard. So where we I click on it and I can see more results. Yeah.

[31:23] Speaker 1: Thank you. We talk a lot about food in the loop. Yeah. Yeah. Yeah. How we logging this? Yeah. Because this is all about workload. Like, how do you skip the human for just not having.

[32:13] Speaker 2: Workload when you’re without? Yeah, great point. But without AI, that problem is still. Yeah. By injecting human.

[32:26] Speaker 1: Potentially, because integration, they’re not making the whole thing when they. What?

[33:01] Speaker 3: On the noose.

[33:04] Speaker 1: Yeah. Yeah. Yeah.

[33:37] Speaker 2: That’s a great point. I agree. Yeah. Yeah. So, but actually, you made a great point. It’s AI is not gonna get sued. It’s the human that is gonna get sued. So, human has to get in the loop or the news as you said.

[34:03] Speaker 1: Yeah. Yeah. Yeah. Yeah.

[35:00] Speaker 2: So Now, great point. But in this case, or such type of, LLM based AI use cases. LLM is basically saying, ” Look, bro, I I I don’t know the answer. It’s up to you. You want to say, ‘I don’t know that answer either. ’ So.

[35:37] Speaker 1: That 90% of the time, the doctor got it wrong. When it was a non-cardiac and received because he was trained in cardiac conditions, and that was his blind. But when he was given AI, he came down to two or three potential diagnoses. He got it right 70% of the time. So this is a consultation with a live patient, meant for his blind, and they were. To make human-centric AR, then my submission is how do we make that decision? How do we include it in the loop? How do we capture what was going to stick? And again, to get them firing out of the data and just not clicking. Sorry, your voice, yeah, please, yeah. 文化发展培养，我们是问题。我们是，一帮子人，我们是干部，我们是干部。

[37:35] Speaker 2: Yeah , That’s right. How? Why don’t they do this? Quickly go over my slide. We’ll jump back to the discussion. Yeah. Thank you. Thank you. Wow. So. So in the case of AI, if ever if ever we are going to make a claim about the accuracy of certain functionality or certain dataset, it cannot be just a plain claim. Claim that it is right ninety five percent of the time because that claim is dependent on number of factors, not just a random data. And so we we are done with the test run, but still number of things can just invalidate our evidence. And in case of our Traditional deterministic computer system. Another change control invalidates our previous run. In case of AI, even if you do nothing, the time can eventually deteriorate your system behavior. So, if we are using a different, if we are trying out a different model or model version, or if the provider themselves. They change the model. If we tweak our prompt, if our corpus is the rag system’s backend database, if that grows in the normal course of with this, it is ingesting more data. Then we have the risk of dilution. We have the risk of proper tagging and how the system behaves with this increased amount of data set. Are there any retrieval parameter changes? Weights, fusion, chunking. Are we using a different type of chunking approach? If we are going to ingest a PDF versus a text versus an Excel, the chunking algorithm. Chunking is when LLMs encodes a certain document so that they can easily read it when it is retrieving it. So whether we are using a different type of chunking size for our datasets, has our context of use changed? There are new users, there are new decisions down the stream, or a new document type. There can be multiple ways, or context of use can change, or whether the time has elapsed. So there should be a scheduled revaluation for the draft. And FDA and EMA in their joint principles also mentioned about that about the draft. So put it in a periodic review. That will be our periodic review, even if nothing else changes. Okay. So bottom line, whatever we produce has a shelf life, and it just gonna expire. So even if we don’t change anything. So, using LLM to test LLM, that that has been a very popular theme these days in in either, and it gives us it gives us a good amount of confidence. But the research on this topic is not there yet. It is still, Giving us 76% confidence level that the LLM can effectively test other LLMs, we are better off using that as a data point, but still focus more on the actual design of the LLMs. LLMs, what they found that even very premier models like Open or Faible, I don’t know if Faible was available when they did this research. Even using a very careful model, picked up biases very quickly or very inherent inherent biases when they were verifying the outputs of another LLM. So, still a great tool. It’s it doesn’t take much for us to do it, but we have to take it with a pinch of salt because, Research is suggesting otherwise about how much we can trust it. These are just some. I think that was my last slide. These are just some terms I have put in in the presentation for you to look at. Different terms that get used in case of AI AI validation. What is K? What do we call it K? What is precision in terms of. AI. What is the mean reciprocal rank? Things like that. And when we are creating a judge, LLM judge. What are the different terms that we are using? What is temperature? What is temperature? Is it zero and all that? Or if there are any questions, we can talk about it. That was it. Thank you so much.

← Duran on scaling | Home | Audit war room →

Graph View

Backlinks