KENX AI in GxP East 2026

2026-09-15-1611-ben-obrien-intended-use-statements-gxp-ai-lecture

41 min read

Ben O’Brien on intended use statements for GxP AI implementations

Recorded 2026-09-15 at 1611 via Plaud. 34:01.

Summary

Talk recorded at the GxP and AI conference Eric is attending this week. The presenter, Ben O’Brien (Phanero), frames the Intended Use Statement as the governance foundation of a GxP AI implementation: more than a traditional URS (a tick-box verification baseline), it is the control lever for risk tiering, validation depth, and inspection scope, and it is being socialized by FDA, EMA, and ISPE, whose Guide on AI he calls the best available guidance for getting GxP AI models into production. His best-practice structure has four pillars: Core Task (the what, with precise verbs instead of generic ones like define, categorize, or classify), Data Bounding Box (the where, what data goes in and out), Model Limitations (what it is not, so an inspector cannot infer unstated capabilities you never proved), and Governance Boundaries (the who, where the AI-to-human handoff happens and model validation effectively ends). Changing any pillar midstream triggers a governance domino effect that invalidates risk classification (model influence and decision consequence), data lineage and operational monitoring, metric alignment, and even training-data relevance (his oncology example: dropping the domain can make all gathered data useless), which is why agile iteration is fine for model development but breaks down in the governed validation phase.

The practical core is that wording determines regulatory burden: the same underlying model reads high-risk as “categorizes and assigns severity to incoming deviations” and low-risk as “drafts preliminary deviation summaries for QA review,” so narrowing the intended use shrinks the auditable surface area you must defend. Bounded inputs (his computer vision example: images from cameras X, Y, Z on line one at standardized 5000k LED illumination) enable pre-validation gating that rejects non-conforming data before the model sees it, cutting a potential multi-year validation effort down to months and simplifying monitoring, while advisory outputs shift effort from proving the model near-perfect to a trained human, whereas autonomous outputs demand harder confidence scoring, fallback logic, and failure mode containment. He calls “utilising human oversight during operational execution” camouflage for passive rubber-stamping that feeds automation bias; inspectors will eventually pull that thread (training records, where the decision was actually made), so wording should mandate explicit QA review and electronic sign-off as a hard stop before execution. Metrics framing encodes the engineering-vs-quality tension (engineering tunes down false positives, quality would rather absorb false positives than let a false negative through; precision and recall are reasonably inversely proportional, F1 as the balance), and midstream scope shifts can flip which metrics matter. He separates routine maintenance retraining (adding a thousand new clear-glass-vial images under SOP and change control) from changes that break the qualified state (moving to amber glass vials requires full revalidation), endorses Google-style model cards as SOP appendices, and argues AI model validation (evaluated against intended use in the real-world environment; GAMP categories do not transfer) should be separated from traditional computerized system validation. He closes with a webinar plug: Victor Beckman, a Danish Medicines Agency inspector on the Annex 22 authoring group, discussing LLMs in a GxP context on October 22, and in Q&A adds that SaaS-style vendor documentation for AI models should mature the same way SaaS documentation did under CSA.

Plaud AI notes

App-generated, verbatim from Plaud’s Summary tab.

Date Time: 2026-09-15 16:11:20 Location: [Insert Location] Instructor: Ben O’Brien

Summary

On 2026-09-15, Ben O’Brien of Phanero delivered a 40-minute lecture on defining and governing Intended Use Statements for GxP AI implementations, positioning the Intended Use as the master control lever for risk, compliance, and validation. He contrasted classical CSV/CSA practices with the probabilistic, systemic nature of AI and emphasized early, precise scoping to anchor governance, risk tiering, validation depth, and inspection scope. Drawing on FDA draft guidance for AI-enabled SaMD, EMA Reflection Paper on AI, Draft Annex 22, ISPE GAMP Guide (2nd Ed.), and the ISPE Guide on AI, he presented a best-practice structure with four pillars—Core Task (What), Data Bounding Box (Where), Model Limitations (What It Is Not), and Governance Boundaries (Who)—and warned that midstream changes to any pillar trigger a governance domino effect, invalidating risk assessments, data lineage, metrics, monitoring, and training. The lecture highlighted the advantages of bounded inputs and advisory outputs, operational controls like pre-validation gating, confidence thresholds, fallback logic, and human review queues, and the need for active human verification to avoid “rubber-stamping.” He advocated separating AI model validation from traditional computerized system validation and recommended “model cards” as a practical documentation standard to improve user understanding and defensible risk assessment.

Knowledge Points

1. Role and Definition of an Intended Use Statement

  • Intended Use: An explicit declaration of an AI model’s operational scope, decision boundaries, and validated capabilities; it acts as the model’s “North Star.”
  • Goes beyond traditional requirements by embedding decision context, autonomy limits, data scope, and governance anchors.
  • Regulatory anchor for risk tiering, validation depth, and inspection scope; informs downstream deliverables and oversight expectations.

2. Traditional URS/CSA vs. GxP AI Intended Use

  • Logic: Traditional systems are deterministic; AI is probabilistic with confidence scores and non-linear outputs.
  • Scope: Traditional focuses on features/workflows; Intended Use centers on decision context, operational boundaries, and autonomy limits.
  • Change mechanics: Traditional changes are localized; Intended Use shifts can have systemic impacts, invalidating upstream risk and data baselines.
  • Governance: URS is a verification baseline; Intended Use is the primary governance control affecting risk classification and validation intensity.
  • Validation separation: Validate the computerized system (access control, audit trails) using established methods; validate the AI model separately against its Intended Use, not via scripted CSA-style tests.

3. Structuring the Intended Use Statement: The Four Pillars

  • Core Task (What): Precisely define the model’s operational scope and decision boundaries; avoid vague verbs; early clarity reduces validation burden.
  • Data Bounding Box (Where): Explicitly define input data scope, environment, and physical attributes (e.g., device IDs, resolution, lighting, formats) to enable pre-validation gates and predictable monitoring.
  • Model Limitations (What It Is Not): Document capability boundaries and explicit prohibitions to prevent inferred capabilities, especially for versatile models like LLMs.
  • Governance Boundaries (Who): Define autonomy limits and human-in-the-loop roles; clarify AI-to-human handoff and where model validation ends relative to broader process risk.

4. AI Oversight and Human-in-the-Loop (HITL)

  • Rubber-stamping risk: Passive human review fosters automation bias and compliance gaps; inspectors are expected to scrutinize this more deeply.
  • Active verification: Require a hard stop with explicit QA review and electronic sign-off before execution; specify human actions to create auditable evidence and maintain advisory classification.

5. Governance Domino Effect of Midstream Changes

  • Risk classification: Scope expansion increases Model Influence and Decision Consequence, voiding initial assessments.
  • Data lineage and monitoring: Changing boundaries invalidates training datasets and monitoring strategies, necessitating redesign.
  • Metric realignment: Scope shifts change which metrics matter (precision vs. recall, F1), potentially requiring retraining.
  • Model training: Domain shifts (e.g., oncology to another area) render original data irrelevant, triggering new data evaluation and full revalidation.
  • Agile limits: Iterative model development is fine; revisiting Intended Use post-validation causes cascading invalidations, making agile unsuitable for governed validation phases.

6. Risk Governance: Framing to Reduce Regulatory Burden

  • Axes: Model Influence (decision-driving) and Decision Consequence (impact severity) govern regulatory risk.
  • Wording and perceived risk: Narrow, advisory phrasing lowers risk and validation burden (e.g., “drafts preliminary summaries for QA review” vs. “categorizes and assigns severity”).

7. Operational Control: Inputs (Bounded vs. Unbounded)

  • Bounded inputs: Pre-validation gating rejects non-conforming data; limits validation scope to predictable parameters and simplifies monitoring.
  • Unbounded inputs: Require complex detection architectures to filter variations; broaden testing across permutations (lighting, devices), expanding validation timelines and risk.

8. Operational Control: Outputs (Autonomous vs. Advisory)

  • Autonomous: Requires strict confidence gating, fallback logic, failure containment, and safety proofs; places full regulatory liability on algorithmic validation and demands exhaustive testing.
  • Advisory: Routes predictions to structured human review queues with confidence metadata; anchors liability in human QA SOPs and reduces technical validation complexity.

9. Operational Controls and Performance Metrics

  • Framing dictates metrics and error tolerances; high-consequence tasks may require near-zero false negatives.
  • Resolve precision vs. recall conflicts between engineering and quality by explicitly prioritizing metrics in the Intended Use; F1 is a balance but priorities must be stated.
  • Scope changes alter acceptance criteria and invalidate baselines, requiring re-testing.

10. AI Model Documentation and Vendor Responsibility

  • Vendor documentation evolution: AI model documentation is expected to improve with user feedback, analogous to SaaS documentation maturation.
  • Model cards: Recommended to document model function, training, technology, and intended use; while not perfectly tailored to GxP, they improve understanding and risk assessment and can be appended to SOPs.
  • SaaS provider role: Proactively supply detailed, non-proprietary model information to reduce burden on quality teams and enable defensible evaluation.

11. Practical Considerations for Validation and Lifecycle Governance

  • Stakeholder alignment: Non-validation stakeholders must internalize Intended Use constraints to avoid rework and delays; define and communicate Intended Use first.
  • Monitoring, oversight, and handoffs: Clearly bookend AI-to-human handoff; model validation may end there, but process risk assessment can extend beyond.
  • Retraining boundaries: Distinguish routine maintenance (in-scope data updates via qualified SOPs and regression testing) from out-of-scope changes that break qualification and require full revalidation.

Questions

  • [Insert Question/Confusion]

Assignments

  • Draft an Intended Use Statement for a target GxP AI use case using the four pillars: Core Task (What), Data Bounding Box (Where), Model Limitations (What It Is Not), and Governance Boundaries (Who).
  • Rewrite vague verbs in the Core Task with precise, domain-specific descriptions to minimize ambiguity and validation burden.
  • Define bounded input parameters and implement pre-validation gating rules to reject non-conforming data at the interface level.
  • Decide and document the output mode (Autonomous vs. Advisory), including confidence thresholds, fallback logic (if autonomous), or human review queue design (if advisory).
  • Perform an initial risk assessment mapping Model Influence and Decision Consequence; align validation depth and inspection scope accordingly.
  • Select and justify performance metrics aligned to the Intended Use (e.g., prioritize precision vs. recall, target F1), noting trade-offs and documentation needed for defensibility.
  • Establish data lineage and operational monitoring strategies consistent with the Data Bounding Box; outline how changes would trigger re-validation.
  • Create SOPs for the AI-to-human handoff, specifying roles, training requirements, and decision-making responsibilities; design hard-stop verification points.
  • Plan change control: document how future modifications to the four pillars will trigger governance reviews and potential re-validation activities; distinguish in-scope maintenance from out-of-scope revalidation.
  • Review relevant regulatory guidance (FDA draft AI-enabled SaMD, EMA Reflection Paper on AI, Draft Annex 22, ISPE GAMP Guide 2nd Ed., ISPE AI Guide) and cross-reference compliance in the validation plan.
  • Research Google’s “model card” methodology to document AI models; consider adopting model cards and separating AI model validation SOPs from traditional system validation.

Transcript

[00:00] Speaker 1: Do you hear me, or do I need to feed them back? You’re good. Perfect. And thank you very much, everyone, for coming to listen to me ramble for 40 minutes. My name is Ben O’Brien. I work for Phanero. And the title of this presentation is not 100 accurate to its contents, but I’ll explain that a little bit more as we go on. I’m trying to basically help frame intended use within the whole kind of perspective of a GXP AI implementation. It is the governance foundation, but it goes further than that, and there’s kind of a domino effect implications for updating and making changes and having vagities, vagaries built into into an intended use statement. So I’m just going to walk through kind of some pitfalls I’ve seen in the industry, and then some considerations that you should probably be making if you’re looking at implementing some GXP AR use cases into your organisation. And just as a quick show of hands, is anyone here a validation engineer, a validation person? There we go. Okay, so a lot of these things are going to be validation deliverables. So yes, you have to convey the messaging of this to people within your organisation. And anyone that isn’t a validation engineer, you have to remember these things so that you don’t annoy your validation engineers. Because ultimately it comes back to them. They’re the hurdle. They’re the reason things can get delivered on time. But there are reasons for that within the whole kind of perspective of the GXP AI implementation. So. So, an overview. I’m just going to go quickly through what an intended use statement is, how you should structure it, and then I’m going to look at some of the impacts. So that’s across risk, across inputs and outputs, and then some other lifecycle governance considerations that should be made. This is not an exhaustive list. There’s a few things that I haven’t touched on in here that probably should have been put in, but it’s running out of virtual paper at that stage. But I’ll talk about those a little bit at the end. So. What is an intended use statement? So, in essence, an intended use statement is an AI model’s north star. It is what you wanted to do. It’s kind of not a traditional requirement. It goes beyond traditional requirements, and I’ll explain that a little bit more on the next slide. And the exact definition is there. I’m not going to read it out for you and have a look at it. And I’m sure everyone in the room is familiar with the term now, because you’re at an AI conference. A lot of people have mentioned it. But it is being socialized by the FDA, EMA, and ISPE as well. And if anyone hasn’t had a read of ISPE guide on artificial intelligence, I recommend you do so. It’s probably the best set of guidelines, guidances that we have in terms of actually getting GXP AI models into production. So, kind of the comparison here between a traditional URS. An intended use statement, because a lot of people I’ve heard it, several people say today, in terms of requirements related to AI models, an intended use statement goes way beyond the requirements of a traditional system. I’m not going to go through this table. You can have a look at it here, but essentially this is kind of your control lever for your risk hearing, for your validation depth, and your inspection scope of a particular AI implementation. So it’s this kind of. Extra step beyond a traditional URS. URS is a tick boxy. Have a requirement, you build your system to meet your requirement, you test that requirement, tick, it’s fine. Whereas an intended use statement is kind of a more holistic view at what a model is meant to do and what you’re actually designing it to do. So this is kind of what I consider to be the best practices for structuring an intended use statement. Oh, that’s absolutely. I can still see that. So essentially, there’s four pillars to an intended use statement. An intended use statement is kind of a misnomer. It doesn’t have to be just one paragraph text. You can break it down into sub sub parts, but it’s easier to convey as kind of an overarching statement. So there’s four considerations that should be made when you’re building an intended use statement. So your core task or the what. So what the actual model is doing. That has to be really clearly defined. And it’s really important to get this right at the very early stage within a project, avoiding kind of generic words like define or categorise or classify. Try and get a little bit more specific than that, because the vagaries, as you’ll see later on, can come back to bite you. The data boundary box or the where, so kind of understanding what sort of data the model is ingesting and outputting, and what it’s going to be used for. It’s really important to be really clear on that, and again, I’ll go into that a little bit more detail why that’s important. The model limitations, or what it should not do, or what it is not, this is super important for kind of defining the limits within which you expect the model to operate. It means that an inspector, if they come in and look at your model, and you haven’t clearly said that it’s not to be used for a particular thing, that from the rest of your statement might be implied. It could do. They could ask for evidence that you’ve proved it does. The part that you didn’t explicitly say it doesn’t do—it’s just—it’s a little catch-up, but it’s something to be aware of, particularly when we get into kind of generic LLM-based AI implementations. Where you leave it be because there’s kind of multiple uses that can come back to bite you, and then the governance boundaries. So the who, where the human interacts with it, where the handoff actually happens. And you’ve heard human in the loop. I’ve got my own opinions about that. You’ll hear about those in a minute. And but it’s really important to define where this kind of AI to human handoff actually happens, and clearly define it because that’s essentially where your your model validation really ends in terms of an intended use statement. When we look at more holistically context reviews and applying risk across the full spectrum of the process, then it goes a little bit beyond that handoff. But it is important to kind of bookend your AI model usage with the the human handoff, and then it’s cut off at the bottom here. But you can still see a little bit. Attempting to modify any of these four pillars midstream triggers the governance domino effect. What this means is essentially this intended use statement is meant to be one of the first things that you define as part of an AI model implementation, a GXP AI model implementation, and these inform loads of steps that happen afterwards, loads of validation deliverables that happen afterwards, and so that if you move past this phase of defining what the model is meant to do or intended to do. And you start creating some of the deliverables, but then you revisit the intended use statement. You can potentially invalidate a lot of the work that you’ve already done for this particular GXP AI model. I’m going to go into that a little bit more detail, but it’s important to have that in mind. That it’s not like a traditional user requirements where people are advocating for kind of agile approaches to implementing validated systems. Agile doesn’t really work for GXP AI model implementations. It can work for aspects of it, like if you’re doing for the actual pure model development phase of it, and you’re iterating on kind of training the model, tuning the hyperparameters. Yeah, sprints can work for that. But once you move past that, the methodology doesn’t really apply because if you come back and revisit what your intended use actually is, loads of things can fall as a result. So here are some the it’s not an exhaustive list, but here are some of the downstream impacts of The scope shift or moving what you’re intended use is of a particular AI model, so the risk classification. If you expand the scope beyond what you initially defined it to be, you can escalate the model influence and the decision consequence of that particular implementation, invalidating the risk assessment that you’ve already performed and/or the deliverable that has to be redone. The data lineage and operational monitoring. So if you modify where the the data being fed into the model or the data coming out of the model. This can change the kind of operational monitoring oversight or human oversight that’s required for the particular model. Again, that’s another activity that might have to be revisited. Metric realignment. A number of people mentioned metrics today, so they’re obviously very important and having defensible evidence to stand over with regards to GXP AI models. So, if you shift the scope, you can then change what metrics are actually important for proving that your model does what you intended to do. So, that’s just something to bear in mind. Model metrics aren’t don’t really work in absolutes. They work, and they’re not inversely proportional. But you can’t get one to a hundred. You can’t get two to both sides of going to one hundred percent. They are reasonably inversely proportional. So that’s just something to bear in mind. That if you have a rewording of your intended use statement, you can shift from one side of the house to the other side of the house and basically flip which metrics are important to you. And then model training as well. So if you make significant changes to the intended use of a model, the data that you use to train the model might be irrelevant. And like, if you’ve trained a model to be specific towards one particular area of pharmaceuticals, let’s say oncology, and then you’re like, no, we’re not interested in oncology anymore. All the oncology data that you’ve gathered might be useless, and you have to perform another evaluation to make sure that the data that you used to train the model is still relevant and can be used. Again, not an exhaust exhaustive list. The technology is continuing to evolve. There’s different use cases, different sources of AI. So They might have additional impacts, but it’s important to kind of frame the the impact of an intended use statement on these aspects of an AI model implementation. So let’s have a look at risk. So I’m a visual person, and these circles kind of help visualise what narrowing in your intended use statement can do to your risk area. So I imagine we can start at the bottom of the slide. So I’ve got two kind of—they’re not full intended use statements, but two overviews of what a model might do. So, a model that categorizes and assigns severity to incoming deviations, or a model that drafts preliminary deviation summaries for QA review. Which one of those has a higher risk? You don’t have to answer, but they could essentially be the same thing. But they’re just worded slightly differently. So then you’re communicating downstream to the other stakeholders involved in the implementation. Is specifically what the model might do. So the the choice of words can have a huge impact. Like the top one might scare quality a lot because they’re categorizing and assigning severity to deviations. There’s no explicit human review or oversight of that. Whereas the second one goes into a little bit more deviation, a little bit more goes deeper description of it, and then you can say, okay, we don’t might necessarily need to treat that as a higher risk because we can clearly see where the handoff happens and where human oversight lies. Essentially, what we’re trying to do here is reduce the kind of audible surface area of a particular model by being really narrow on its intended use and its scope of use. And by doing that, you kind of alleviate a lot of the pressures that might come in an implementation in terms of kind of complex testing strategies, complex monitoring strategies, complex oversight strategies, human in the loop, etc. And and it’s literally just down to wording. And and it’s just really really important to understand that. Your choice of words can have a huge knock-on impact with regards to how you actually go about implementing a GXP AI model. Operational controls. So I’ve kind of broken it down to bounded and unbounded here. This could be phrased loads of different ways. So I’m using computer vision as the example. So the bounded example here is analyzing high-resolution images captured by cameras X, Y, Z on line one, standardized at standardized 5000k LED illumination. Versus the unbounded example of analyzing visual data captured across liquid manufacturing production lines. Again, this could be the same model, the same use case, but you’ve just worded it slightly differently. By having this kind of bounded input definition, you allow kind of pre-validation gates to happen with the data being fed into the model. So, if you try to feed data into the model that doesn’t tick all of those boxes in the bounded example. It doesn’t get fed in because it says, “Oh, it’s not from this particular camera on this particular line, and the lighting isn’t right. Therefore, I’m not going to process.” And that’s fine. Whereas, if you’ve got the unbounded example, not only can you be feeding an incorrect data into the model, which will give you an incorrect output, you then also have to prove that the unbounded example works. So then you’re like, “Okay, if someone like unscrews one of the light bulbs or unplugs one of the LEDs or whatever it might be, is the. Does still work? That’s what you’re kind of implying here. That it’s just data being fed in. Can I take a picture on my phone, and that’ll be fed in? Probably not, or definitely not. But these are the sorts of things. I know I might be being a little bit maybe argumentative here, but I’m just trying to prove the point that the the wording really matters here in terms of how you’re actually going to go about evaluating a particular model. And it also works for the error handling as well. So if if it’s not being fed into the exact camera that it’s expecting to come from. Then it just doesn’t get processed. So that’s just another thing you can swat away and say, look, we don’t have to worry about that. The so what? The bounded input limits the validation effort. You can get potentially, if you had an unbounded example, you were trying to feed in all different permutations of data into it. You could have a huge, multiple, multiple quarter execution in order to test and validate this. You could go from like a three-month execution of testing to. Multiple years potentially trying to get things right just because you didn’t define it well enough at the beginning, and it allows for easier performance monitoring as well because you just have to monitor within the fixed parameter box as opposed to trying to monitor everything that’s going on outside your intended box that just wasn’t well defined enough. Same goes for outputs. So this is not bounded and unbounded, but autonomous and advisory. There’s lots of people today have had opinions on autonomous AI and autonomous outputs, but these are just some things that you should consider as part of your kind of defining what the model can do. So, automatically executing defect disposition and batch line clearance based on real-time visual classification, versus generating preliminary defect classification recommendations to assist qualified QA personnel during secondary inspection review. Again. To the person developing the model, or saying, “I have an idea for an AI use case,” they could be the same thing, but in terms of what you’re actually expecting the model to do, they’re completely different. So the autonomous pathway, kind of, you end up moving. You end up having to prove the confidence scoring is much harder in order to rely on it and trust it. And whereas if you kind of step the other way and have it as advisory, you can kind of structure it more from, Proving that the code does what you want it to do into okay, well, it’s just advising a trained person to do something, so the person is trained, and it moves kind of some of the effort away from having to prove that the model’s as close to perfect as you can get it. And the kind of autonomous pathway as well, you have to look at things like fallback logic, you have to look at kind of failure mode containment. Like what what do you do when it leaves that threshold, as it inevitably will, because that’s the nature of models, unfortunately. And. And yeah, I’m not going to go through the so what. You can read that there yourself. I’ve kind of talked to that oversight. So this is one that is being put off. Here we go. So on the left-hand side, on the left-hand side, we have a what I can what I refer to as kind of a rubber stamping process. So if we look at it, we’ve got an AI output to a passive human, to an undetected failure, to a compliance gap. So if we look at what the intended use wording for that might be, it’s utilising human oversight during operational execution. That’s obviously not a full intended use statement, but that’s kind of the relevant part. The reality of that is, people trust computers and they’ll form an automation bias and they’ll say, “Oh, AI output says this, thumbs up, let’s keep going.” Unfortunately, that’s more and more being seen as camouflage for kind of passive human oversight. Right now, inspectors aren’t really pulling that thread, but they will start pulling it eventually because people start like everyone in the room now knows human in the loop. And like, oh, it’s fine, we can release this production because there’s a human in the loop. But that’s not really good enough in the long term. That if you don’t start developing kind of your processes and approaches for saying, okay, what does an active human in the loop actually look like, then you can end yourself in a lot of trouble. You start getting into, okay, show me all the training records for this person, show me where the decision was actually made, etc., etc. You kind of, Multiply the amount of effort required in order to prove the model is doing what you want it to do. On the right hand side, look—it’s not a silver bullet, but it’s a really good first. It’s a really good step in trying to eliminate this kind of passive human in the loop. So this goes from AI suggestion to a hard stop for a human verification, and then the rest of the workflow executes. This intended use wording on the right isn’t going to solve all your problems. Not a silver bullet, but it is a good first step to shift the kind of responsibility from. The model and the workloads operating in to the human, and so generating advisory recommendations that mandate explicit QA review, electronic sign-off prior to execution. This kind of brings people more to the perspective of, okay, so there is a human in the loop, and they’re going to do X, Y, and Z before the workflow progresses. This is not to say that I’m anti-kind of agents or agentic AI in the right situations, developed in an appropriate manner, and. Well, it is still important to kind of recognise that where we are right now, where most organisations are right now, they’re not mature enough in order to kind of take that step to oh, it’s fully autonomous. We went from having no AI to autonomous AI, and the human is just kind of keeping an eye on it just to make sure it’s not really good enough. Let’s.

[18:09] Speaker 2: See if we can go back to this. Are you saying there that the difference between that human bear? So that’s what I’m saying.

[18:21] Speaker 1: This is like not. I’m being a little bit pedantic with this example here, but it does shift the focus from kind of like ambiguous human in the loop oversight to human does X, Y, and Z. I’m not saying necessarily that they have to perform an electronic signature, although that it can be a mitigating factor. But it’s just an example for the wording you’ve chosen in your intended use statement. Can bring an inspector down a path of saying, “Okay, well, like you haven’t really told me what the human in the loop is doing, but you say there is a human in the loop.” Whereas, if you can be a little bit more explicit, be like, “Okay, well, the human is explicitly doing these things. If you can show me evidence that they’ve done those things, then that you’ve demonstrated that there is more active oversight.” Is that sufficient in all cases? Maybe, maybe not. Like, you’d have to understand kind of how your people are trained, what the process is, what they’re actually, what the actual. Intended use of the model is to really put into that context, but is a good first step to kind of move the the burden from the AI model back onto the human, while still leveraging its benefits. Operational controls. Sorry, I’m making a mess of this. Operational controls. So performance is relative. With regards to metrics, framing determines what metrics matter and what error rate is tolerated. So, I’m going to start in the middle, the ambiguity trap. So, there’s often disconnect between kind of engineering teams and quality teams with regards to AI implementation. So, if you take the engineering perspective, if they’re developing a tool for their use, they want to minimise false alarms. They don’t want to be kind of taken away from what they’re doing to have to deal with a potential false positive. To say, okay, let’s see what the model says, and my God, it’s open again. Okay, so they will try and tune naturally, try and tune the model down, so you get less false positives. Whereas a quality team wants the complete opposite of that; they’d rather be dealing with more of these false positives than have a false negative go all the way through. And this is kind of this disconnect can happen. So sometimes it can be just personalities or silos between teams, but this can be remediated. By having it well defined as part of the intended use, where you can kind of clearly define which side of the house you want to be on, you don’t have to be so explicit within the actual intended use statement itself to say this is the side we’re leaning to. But it is consideration. There is some wording that you can use to indicate which side you want to lean on. As I said, metrics with relate in regards to AI models, there. Reasonably inversely proportional, so you can’t have both be happy at the same time. Unfortunately, so you’re going to have to pick which side of the house you’re favouring. F1 score is an example of this kind of balanced metric between precision and recall. You can’t have 100% precision and 100% recall; it just doesn’t work. So you have to find which balance you’re happy with. And your framing can help kind of set the set the kind of boundaries for which evaluation metrics matter for that particular use case. And And then, kind of moving towards kind of shifting the scope of a particular model. So, the ambiguity trap can happen just in siloed teams. But if you’ve moved through an AI development or an AI implementation project, and then you get to a point and then shift the scope, this can invalidate which side of the metrics that you were originally leaning on. So, it’s kind of understanding that the downstream impact of changing the scope or changing the intended use for model can completely shift which metrics you deem important for. Compliant AI implementation. Again, sorry, I apologize for this slide. So this another example. This one is maybe me being a little bit more pedantic. Can everyone still see that? And so this something that’s going to happen with almost all AI models, particularly if you’re developing them in house. This example is a visual inspection example, and it’s just trying to outline the differences between. Kind of routine maintenance retraining versus full revalidation training, and this not kind of a good or a bad thing. You kind of have to be this specific when you’re defining what a model does, but this a consequence of not defining what your model does. So on the left hand side, we’re looking at adding. So let’s say the intended use of this model is process and clear glass wafers online for visual inspection for particulates. You get a thousand new images of samples. You want to. Add them back into the training data, retrain the model on this new, up-to-date data, and allow it to kind of improve itself, hopefully. Whereas on the right-hand side, we’re looking to completely change from clear glass vials to amber glass vials, because the original training data would have been on clear glass vials, and you’ve defined it as such as part of your intended use. Adding in thousands of images of amber glass vials would reduce your confidence in across the full spectrum of vials. So if you had clear, a mix of clear and amber. That neither of them would be perfect as a result of it. On the left-hand side, kind of if you’re following the path of routine maintenance, the mechanism for actually updating this is essentially retraining the the model using the mechanism that you originally trained it with, via an SOP and change control that you have internally of following those processes. Whereas if you were to move from clear glass files to amber glass files, you essentially break the qualified state. Of the AI model, and you have to move towards okay, full re-execution of all your testing of the model. You might be able to leverage some of your design documentation, say, oh well, we trained it this way. Let’s just like start again with the amber glass ball images. Would you still have to go through this full retraining process? Okay, so just a summary of what I’ve been through there. So essentially, we’re trying to narrow the scope within which you have to defend an AI model. So all of these steps are trying to take, trying to be active in defining what a model does, so that when you’re going through your validation processes, when you’re going through your evaluation process, when you have to stand in front of an inspector and explain what the model does, that you reduce down the area that you have to defend. It seems really simple, and a lot of it, I’m sure, makes sense to a lot of you. But there is a huge amount of impact downstream as a result of taking these decisions really early in the project. These requirements can be updated all the way through to kind of moving into validation environments. You can still be updating requirements as long as they’re approved and locked before them. That’s just not possible with AI models. That you have to define it so early on, and that any changes to that intended use can completely change the deliverables you have to provide afterwards. And. I’m not going to read through this, but just some of the takeaways: kind of ensuring that you kind of have these structured inputs and outputs, kind of trying to leverage your wording of your intended use to move past rubber stamping of human loop, and then trying to really clearly define what your model metrics are and why they’re important to you. And then, just lastly, a little plug for a webinar that I have coming up. This is Victor Beckman. He is an inspector with the Danish Medicines Agency. He’s responsible. He’s part of the authoring group for Annex Twenty Two, and so we’re going to have a webinar on the twenty second of October to discuss LLNs. That’s a particular area of interest for him. So, should be an interesting discussion. Go through some weird and wonderful questions with regards to LLNs and GXP context. I think she gives people a little bit of solace that he’s a regulator and he’s willing to talk about Llans and the GXP context. But feel free to scan the QR code and sign up for the webinar if you’re interested. Thank you. Does anyone have any questions or reports?

[26:27] Speaker 3: I have questions. If you’re a supplier of a. For an intended purpose, can you assist with? I know only the regulated end user organization can define their own intended use, but can you? Would you assist with intended use definition in that case? Because it can’t be totally unique in a thousand different ways. Yeah. Designed for an intended purpose.

[26:59] Speaker 1: There’s obviously going to be nuance to the specific company that’s using it. The way I look at it is when. SaaS came out, release management, CSA, risk-based approach, and leveraging vendor documentation. A lot of vendor documentation that initially came out was cramped and was essentially useless. And some teams were like, “Oh, here’s a document, here’s a piece of paper. We put that over there. Grant, there’s our testing done.” And then other organisations were looking at it critically and saying, “This actually isn’t very good. We can’t leverage that result.” But what that caused was SaaS companies then had to say, “Okay, we have to improve the quality of.” What we’re providing to our end user, so that they can leverage more of it, and actually becomes a value add thing that we’re providing them. So I see kind of AI model documentation in the same light as that. That it is still an evolving thing. A lot of SaaS companies I work with, some SaaS companies, to try and help them structure their information in a way that makes it accessible for end users. But it’s only through this kind of process, having the feedback mechanism, so you say, okay, here’s where we’re deficient with regards to our particular maybe. Maybe it’s intended use, maybe it’s model metrics, whatever it might be. What extra detail and information can we give them to improve their understanding and improve their overall adoption of the model? So, yeah, I’d say my opinion would be that SaaS companies can kind of be a little bit more active in that space and kind of give as much detail as possible. You obviously don’t want to pull the full curtain back. You don’t want to give all your data away. There has to be something that you keep back, but give as much information as you reasonably can and kind of fill out the full picture. Because ultimately, if you don’t fill out the full picture, The quality teams are going to be sitting there, being like, ” Okay, well, we have to piece it together ourselves and build potentially complex evaluation strategies, depending on the intended use and the use case and the overall risk of the model, to say, ‘Okay, well, we now have to fill out the picture ourselves and put our best foot forward if we really want to use this. ’ ” Whereas you kind of take an active step to remove some of that headache from them.

[28:53] Speaker 4: Yep, I have a question about the intended use statement, in the sense of, When I’m a data scientist, when we’re developing models, we love to have our own documentation as well. The intended use statement, like one of the one of the artifacts I’m trying to encourage our teams to produce, is what Google is called a model card. Absolutely. Right. And so, does the intended is if we want to have a single source of truth for this statement, do we put it in?

[29:20] Speaker 1: So I somewhere else. A lot of organizations use model cards, particularly in, In terms of SOPs, so if you have an SOP for kind of use of particular model, I don’t see why you don’t have an appendix at the end of it. That is just a model card. If anyone in the room is familiar with model cards, I’d gone give them a look. This methodology defined by Google in regards to explain what an AI model does. It’s not exactly perfect in terms of the GXP context, but it adds so much clarity and detail around what a model is, what it does, how it was trained, etc. The technology driving it. The more information and knowledge you can give to the end user, the better. So yeah, one hundred percent. Other organisations have other approaches for trying to do this, but I think it’s such a simple and straightforward way for capturing the information to give to a client or to give to the end user that’s actually using it. Like a number of companies are letting their end users kind of walk in the dark and just saying, “Here’s a model. We go off and use it,” and then people like, “I don’t really know what it does, how it works, how it’s trained.” Was actually intended to do, whereas if we can give them that information and extra detail around the model, it just improves understanding, improves knowledge, and allows people to fundamentally risk assess with regards to the model better.

[30:40] Speaker 4: And if I could follow up, sometimes the AI model itself is not the centerpiece, but it’s the system in which it’s embedded. Do people make system cards?

[30:51] Speaker 1: No. So go back up to this one. So I’m going to use the circles. Ignore the labels. My perspective is that the big circle there is the system. That’s a system that the system will have. Traditionally, is requirements. The system maintains your access control, your your audit logs, audit trail, all those bits and pieces, any associated workflows. Whereas the smaller circle within it is the AI model in which it operates. So it’s really important to delineate the two of them so that you can have. User requirements associated with a particular AI model, but they’re to be treated differently and tested differently. Instead of using kind of a CSA risk-based approach towards testing a computerised system, you’re instead driving home towards evaluating against your intended use in the real-world environment. Trying to use CSA practices on an AI model, you can develop a lot of a lot of documentation around it, but it’s not really required or relevant or defensible. Whereas if you flip it the other way around, the intended use of the overall computerized system can be so vague, and there can be so many operations within a computerized system that it doesn’t make sense to go at it from an intended use perspective to get that validated. Okay. So it’s kind of I’m speaking with a number of organisations that are trying to delineate their validation approaches from kind of AI models and computerized systems, traditional computerized systems, and it’s they’ve. Initially, they have more validation SLP, and then they’re like, “Right, let’s put in an appendix for AI model validation.” You’re like, “This is so confusing.” You’re reading through it, and it’s all about how to write scripted or unscripted test cases for a particular computerized system, and then you’re like, “Okay,” and they’re like referencing back up, saying, “Oh, just do the same, but for an AI model, it doesn’t really make sense.” So my perspective is to kind of separate the two out and treat them as different activities. Okay, thank you. Yeah, no problem. Yep.

[32:50] Speaker 2: Sorry, I think my question is actually a repeat of what you just said. Anecdotally, what you recommend is that you have essentially entirely different methods for governance, control, validation of conventional applications today.

[33:09] Speaker 1: Yes, and because if you look at like kind of up to this point, the kind of shining light for. For traditional computerised system validation was the GAMP guide, and although the GAMP guide does have an appendix in regards to AI models, it treats them as completely different things, and it moves away from kind of. A lot of people familiar with GAMP categories they don’t mention GAMP categories when it comes to AI models, and there’s a reason for it, because GAMP categories don’t infer anything towards the intended use and the context of use of a particular AI model. Anyone else? Any other questions? All good. Thank you very much.

← Abhishek follow-up | Home | Gaurav Walia on manufacturing AI →

Graph View

Backlinks