KENX AI in GxP East 2026

2026-09-15-1024-gorani-ai-governance-human-risk-gxp

47 min read

Bharanitharan Rajendran on defensible AI governance and human risk in GxP

Recorded 2026-09-15 at 1024 via Plaud. 44:12.

Summary

A 44-minute conference lecture by Bharanitharan Rajendran, co-founder and CEO of Validfor, an AI-driven visual validation platform, on reducing human risk in GxP environments. His opening move is a paradox: the industry fears out-of-control AI while human error remains the actual citation engine. FDA drug warning letters rose 15.9 percent to 303 in the previous fiscal year, 21 CFR 211.22(d), failure to follow written procedures, was highlighted 184 times and has been the top recurring citation for four consecutive years, and a Nucleus study publicized by PwC attributes over 80 percent of process deviations to humans. Quality systems, he argues, are built on the assumption that humans will obey SOPs, which is hope rather than control, and the organization, not the AI, will always be the party holding the warning letter. He sorts the failures into six categories of human risk: documentation, investigation (root-causing everything as human error and answering it with retraining), change control impact assessment, quality oversight, data integrity (shared logins, backdated paper records, and inspection “war rooms”), and validation lifecycle treated as a one-time event. Enforcement stories follow: the April 2, 2026 first AI-related CGMP warning letter against Purolea Cosmetics Lab, where AI-generated SOPs went unreviewed and the firm answered an inspector’s question about missing process validation with “AI did not tell us”; a vendor who deleted a clinical trial’s data and audit trail, costing 47 subjects their data (Applied Therapeutics, Dec 2024); and a contract lab falsification with backdated records that exposed potentially hundreds of downstream pharma companies (Tentamus India, Mar 2026).

Against the “regulators are too slow” complaint he lays out a 2025 to 2026 convergence: FDA draft guidance on AI credibility assessment (Jan 2025), the ISPE GAMP AI guide and EU GMP draft Annex 22 (Jul 2025), the SeniorLife finding that intended use, not the marketing name, determines regulatory classification (Aug 2025), FDA/EMA joint Good AI Practice principles (Jan 2026), and the Apr 2026 warning letter, after which “all the excuses are gone.” Defensible AI then gets five properties: documented intended use that states what the AI cannot do, risk-proportionate validation run as a continuous lifecycle because models drift, ALCOA+ audit trails extended to capture user, model and version, inputs, outputs, justification, and confidence score, enforced approval by a named accountable human (he reframes human-in-the-loop as human-centric design, since nobody genuinely reviews a hundred AI-generated documents), and continuous lifecycle monitoring. AI is “not a faster human” and not a compliance chatlet that drafts SOPs; in the control loop “AI acts, humans decide, and the system remembers everything.” Industry examples: a Moderna colleague’s personal four-step model quality approach (public model evaluation, platform internal benchmarking, use case, continuous quality monitoring) and Daiichi Sankyo’s governed in-house generative AI environment against shadow-AI data leakage. The close is a three-step plan for tomorrow: inventory every AI touching GxP including shadow AI with a context-of-use statement, enforce a human gate so no AI output enters a GxP record without named review and approval, and treat validation as a continuous lifecycle, ending on “untrained humans fail inspections, and untrained AI fails patients.” Q&A covers where intended use is documented (a standalone AI description or inside the existing validation plan; the point is that it is documented), what software must change so quality reviewers can sign off knowledgeably (review cycles, decisions, and rationales captured inside the system instead of offline chats and email), and how to control fraud and falsification rooted in culture and fear of losing jobs, where his answer begins with culture before the recording cuts off mid-sentence.

Plaud AI notes

App-generated, verbatim from Plaud’s Summary tab.

Date Time: 2026-09-15 10:24:37 Location: [Insert Location]: [Insert Location] Instructor: [Insert Name]: Bharanitharan Rajendran

Summary

This lecture series, delivered by Bharanitharan Rajendran, co-founder and CEO of Validfor, addresses the persistent issue of human risk in GXP environments and advocates for a shift towards building defensible, inspection-ready AI governance. The speaker argues that while the industry is increasingly concerned about AI, the long-standing problem of human error remains the primary cause of regulatory citations, such as FDA warning letters. Despite established quality systems, SOPs, and training, human-related failures in documentation, investigation, change control, oversight, data integrity, and validation continue to occur. The core message is to reframe AI not as a “faster human” productivity tool but as a defensible GMP control that is documented, validated, governed, and continuously monitored to mitigate patient risk.

The lecture highlights the rapid convergence of AI regulatory guidance from 2025–2026, including the first AI-related CGMP warning letter in April 2026, which signals the start of enforcement. It emphasizes that defensible AI requires traceability, human-centric control, rigorous documentation of intended use, and continuous validation lifecycles, separating AI’s capability from human authority. A practical three-step model is presented for immediate implementation: creating an AI inventory, enforcing a human approval gate, and treating validation as a continuous lifecycle. The content was created on 2026-09-15, and all timelines and readiness actions should be considered in that context.

Knowledge Points

1. Human Risk vs. AI Risk in GXP

  • The Paradox of Risk Perception: The lecture contrasts the long evolution of human intelligence with the rapid lifecycle of AI, questioning why the industry fears “out of control” AI but has not applied the same scrutiny to human fallibility. Quality Management Systems are often built on the unrealistic assumption that humans will flawlessly follow procedures.
  • Human Error as the Root Cause: Over 80% of process deviations are attributed to human error. This is a “human risk problem,” amplified by the scale and speed of modern GxP environments, not a “procedure problem.” Ultimately, the organization and its human users, not the AI itself, are held accountable for failures.

2. FDA Warning Letters and Regulatory Convergence (2025–2026)

  • Rise in Human-Related Citations: In the previous fiscal year, FDA drug warning letters increased by 15.9% to 303. The top recurring citation for four consecutive years is 21 CFR 211.22(d) (failure to follow written procedures), highlighting persistent human non-compliance.
  • Timeline of Key Regulatory Guidance:
    • Jan 2025: FDA Draft Guidance on AI credibility assessment for drug/biological products.
    • Jul 2025: ISPE GAMP® AI Guide and EU GMP Draft Annex 22 on AI governance.
    • Aug 2025: SeniorLife.AI finding, clarifying that a product’s intended use (e.g., diagnostic app) determines its regulatory pathway.
    • Jan 2026: FDA/EMA Joint Good AI Practice Principles, marking cross-jurisdictional alignment.
    • Apr 2, 2026: First standalone AI CGMP deficiency warning letter, signaling the start of enforcement. Excuses about unclear regulations are no longer tenable.

3. Six Categories of Human Risk

  • Documentation Failure: Incomplete or inconsistent records, often discovered only during FDA inspections.
  • Investigation Failure: Repeatedly citing “human error” as a root cause and using retraining as an ineffective solution.
  • Change Control Failure: Incomplete downstream impact assessments of changes.
  • Oversight Failure: Quality units missing critical issues due to pressure and reliance on manual tools.
  • Data Integrity Failure: Sharing logins, backdating records, and not using audit trails.
  • Validation Lifecycle Failure: Treating validation as a one-time event, allowing performance drift to go undetected until an inspection.

4. Five Properties of Defensible AI Control

  • Documented Intended Use: Clearly define what the AI can and cannot do to set the scope for validation.
  • Risk-Proportionate Validation: Conduct AI-specific risk assessments and align validation rigor with risk. Validation must be continuous to address model drift and evolving performance.
  • ALCOA+ / Secure, Non-Overwriting Auditability: Audit trails must be immutable and capture comprehensive details: user, AI model/version, inputs, outputs, rationale, and confidence score.
  • Enforced Human-in-the-Loop Oversight: A named, credentialed human must review and approve AI outputs, retaining full accountability.
  • Continuous Lifecycle Monitoring: Move beyond periodic reviews to ongoing surveillance to detect drift and maintain a validated state.

5. Defensible AI Governance and Architecture

  • Defensible AI Control Loop: An architecture where “AI acts, humans decide, and the system remembers everything.” AI has capability but no authority; authority resides with the human quality unit. This requires immutable records, verifiable checksums, and enforced QA gates.
  • Human-Centric Design: Go beyond superficial “human in the loop” checks by embedding named user accountability and meaningful review workflows into the system.
  • Traceability and Audit Readiness: Ensure all records (plans, protocols, data) are traceable to their origin and context. Systems must lock data, maintain immutable logs, and provide inspection-ready artifacts.

6. Three-Step AI Governance Model

  • Step 1: Inventory Your AI: List every AI tool touching a GxP process, including “shadow AI,” and write a clear context-of-use statement for each.
  • Step 2: Enforce the Human Gate: Mandate that no AI output enters a GxP record without review and approval by a named human. This is a governance decision that can be implemented immediately.
  • Step 3: Treat Validation as a Lifecycle: Implement continuous monitoring for every AI system to manage drift, performance, retraining, and change control.

7. Inspection-Readiness and the Role of Quality Professionals

  • Core Inspection Questions: Be prepared to answer: Do you use AI? How do you validate its outputs? Who in your quality unit is responsible? When did you last review its performance?
  • Division of Labor: Delegate preliminary checks (e.g., typos) to AI, allowing quality professionals to focus on high-level decisions like evaluating deviations, verifying root causes, and approving corrective actions.
  • Cultural Controls: Recognize that fraud and falsification are often rooted in organizational culture. Enforce immutable records, non-bypassable QA gates, and clear accountability to deter misconduct.

8. Case Studies and Industry Examples

  • Case 1: Inappropriate Use of AI (Purolea Cosmetics Lab, Apr 2026): The first warning letter citing AI use, where AI-generated SOPs lacked quality review. The company’s defense, “AI did not tell us,” highlighted a failure of human oversight.
  • Case 2: Unapproved AI-Powered Device (SeniorLife Technologies, Aug 2025): An AI app marketed for pre-diagnosis was registered as exercise equipment, demonstrating that intended use determines regulatory classification.
  • Case 3: Data Deletion by Vendor (Applied Therapeutics, Dec 2024): A vendor deleted clinical trial data, illustrating that oversight for data integrity cannot be outsourced.
  • Case 4: Data Falsification (Tentamus India, Mar 2026): A contract lab’s widespread data falsification exposed dozens of downstream drug manufacturers, showing the extensive risk of human misconduct.
  • Industry Example (Moderna): A four-step model-quality approach emphasizing continuous monitoring in production, extending assurance beyond initial validation.
  • Industry Example (Daiichi Sankyo): A governed, in-house generative AI environment (DS-GAI) to manage risks like information leakage and overconfidence at an enterprise scale.

Questions

  • [Insert Question/Confusion]

Assignments

  • 1. Create a comprehensive AI inventory across all GxP processes, including shadow AI, and write a context-of-use statement for each identified tool.
  • 2. Define and document the intended use for each AI system, either as a standalone description or integrated into existing validation plans.
  • 3. Establish and enforce a human gate: assign named reviewers and approvers, configure non-bypassable QA gates, and ensure no AI output enters GxP records without approval.
  • 4. Implement validation lifecycle practices: set up continuous monitoring for drift detection, performance metrics, and change control triggers; schedule periodic reviews and revalidation.
  • 5. Review internal quality systems to identify where the six categories of human risk (Documentation, Investigation, Change Control, Oversight, Data Integrity, Validation Lifecycle) are most prevalent.
  • 6. Assess vendor oversight procedures to ensure accountability for data integrity, as responsibility cannot be fully delegated.
  • 7. Implement ALCOA+ compliant, immutable audit trails for AI, capturing user, model/version, inputs, outputs, justification, and confidence scores.
  • 8. Enhance software systems to embed review cycles, capture decisions and rationales, and maintain complete audit trails internally.
  • 9. Conduct a cultural risk assessment and provide training to address pressures that may drive fraud or falsification in AI-enabled processes.

Transcript

[00:01] Speaker 1: Good morning

[00:24] Speaker 2: All. Welcome.

[00:25] Speaker 1: Good morning.

[00:31] Speaker 2: Once Again, good morning. So, this question is going to be about reducing human risk in GXP. We have we have been saying since recent times we have always been discussing about reducing human risk in GXP, but we have been battling to speak about, Producing the human specific DNA, and how can that be done? Let’s do that here. The other side from the other side. So, if we start from here, before I start infection of myself, it’s been seven million years, almost seven million years of evolution. The human intelligence, it took for the human intelligence to evolve to the state where we are now, but for. Artificial intelligence. It’s going to be just few weeks of the lifecycle. It’s not going to take the same long path, and that’s what we are going to discuss. How this going to be different from human intelligence to the artificial intelligence, and how the human risk that we have always been facing for many decades in the industry in GXP, and how we could reduce that with different solutions here, and that’s what we are going to talk right now. This Bharanitharan Rajendran. Co-founder and CEO of Validfor, and Validfor is a AI-driven visual validation platform. But you know, this session is not about us. All right, this session is not about me. This is about engaging human-first creativity. So, how many of you here are worried about what if AI gets out of control? Every day we use AI for anything and everything, right? So it could be so-called shadow AI and all those. Even though the organization does not allow, we still use AI, be it ChatGPT or Copilot or Gemini or any kind of AI. Probably avoid certain models for anything and everything. So, B is writing article, B is writing LinkedIn post, B is rewriting our email. A lot of areas. Beyond that, there are so many other capabilities where AI do almost every one happens. Even here, even I do. That’s a fair fear, though, but still. Should we stop there or proceed further using AI and how in a controlled manner? How to have a defensive AI that can help human rather than putting us in a scary situation? So, here let’s start with some number. Let’s give a precise number. Let let’s just look at the fancy number that you can remember when you walk out of the room. Change zero, change, change or change. This is the number of warning letters that has been issued by FDA the previous year. Chain of three across so many thousands of licenses, companies, and all those things. It might look lesser, but the part is it’s almost 15.9 percent increase in the warning letters from the previous year. That’s a significant difference. It’s not few. It’s 15.9 percent increase, and altogether there are repeated citations of one particular money. Let’s say for Part 11. That has been vividly highlighted. So, which is 21.22D, and what does it mean about? So, it clearly says any procedures, in procedures that are applicable for a quality unit, that is supposed to be written and should be followed. Are we have we written all the policies, SOPs? Yes, we know that every organization starts with their KMS. They have a policy, they have our SOPs, we have so many things. But do we follow? That’s the biggest question, and this finding, this particular one, was the highly rated. One eighty-four times it has been highlighted there, which is all getting back to the same human error. That’s what I’m going to talk here as human risk. And we think about AI risks again and again and again when we speak about GPT environment and so on. But how about the human risk? We say. What if AI goes out of control? How many times did we ask, “What if human go out of control?” We wrote so many SOPs, we wrote so many policies, assuming that human will obey. That’s a hope. That’s not a policy system. Assuming that human is going to follow that. Assuming that we are going to perform all the activities as per the policies, as per the SOPs that we have explained. Assuming that human will not question. Human will not make mistake. That’s how the quality system has been built. Is that how it works? No. They face so many risks. The risk and worry that we have against AI, that’s the same risk and worry that we should have had against human, but we never had that one. And that’s the reason why we have so many citations, so many findings, and so many issues. So it’s not about procedure problem. It’s about human risk problem because it’s about how we follow those written procedures, and. So there was a study that was performed by Nucleus, which has been also highlighted by PwC, which clearly highlights that almost 80% of process deviations are attributed to humans. Remember, attributed. The same is going to happen when we talk about AI. If the AI that’s being used in a GXP environment fails, that is going to be attributed to the process. That’s going to be attributed to the human who used the AI. But FDA is not going to issue a warning letter against the AI. It’s going to be the organization. It’s going to be the human. It’s going to be the quality unit where the AI has been used. So we are going to talk about few more warning letters and all those. So let me first put all these into six categories of human person. So one, the default one, and we can never ignore. Every one of you in the room might have gone through this one: documentation failure. We have incomplete documents. We have inconsistent documents. It would have gone through quality review. Definitely, the quality department would have reviewed it. That the organization, everyone would have followed SOPs. They followed their process. Humans were trained. They were qualified. The process were qualified. But still, there would be always missing information, which only the FDA inspector finds. Even after so many review cycles, even after so many training, even after qualification, there are findings. Documentation error, the biggest one, quite often that we come across, and investigation failure. Think about any investigation that happens. Whenever there is a process deviation, ultimately it gets to human error, because no one is going to say that no, we did not have a process. They have a process. Was the human trained? Yes, they were trained. But still, end of day, again a human error. What happens? We retrain the human, and then what next? In six months’ time, another department, another human, again retraining. This not a solution. So I’m not saying the human is the root cause, but instead I’m saying this going to be a symptom that we will have to prevent moving forward. How this going to change control? The change control failure is quite often. So there are so many changes. Change is the only thing that never. Changes it keep changing right so changes are wide across the organization for each and every process each and every system every policies the way how it works and so on and after AI the changes significant it just rapidly changes every stuff there so the change impact assessment do we assess the impact for the change do we ensure that everything is intact is everything under control so this is a gap that humans usually miss that it. It’s not about humans. I’m not saying that human miss everything. It’s a human risk. I’m not against the humans. It’s just that it’s not a. It’s not because of humans. It’s because people being people. Simple. Why it happens? Because we are humans. Human being human. Yeah. So oversight failure. That’s another one, right? So this is another hard truth. Unfortunately. So many of you could be quality department here. So we go through each and every data that goes through quality department. We have a review. We have so many cycles. We have feedbacks. We have things to address and follow process and all those. Even after so many quality reviews, things get missed, which was not captured during the review. Data integrity. That’s another one. When we speak about data integrity. AI that’s being used, there must be data integrity. It should be traceable. It should be tracked. I should be able to see the transparency of the data and all those things, right? But think of humans in any any environment. So, how many of you could raise that? Raise your hand, saying that in my organization, in my team, I’ve never ever heard people share their password, share their login credentials. That does happen. To to be honest, that does happen, right? So we have so many controls, so many trainings, and all those things. You never know what what and why you transfer. Still, yeah. And then what else? So we have paper records, banked entries. That’s another level. So many changes, so many stuff that happens. There there are stories where people used to trip and during any inspection, right? When the GX been roaming. Obviously, it’s a hard room that we have another war room for whenever there is an inspection in any firm that’s happening. So, what is the war room? I don’t want to tell what is the war room, but every one of us know what happens in the war room. When something gets to the inspector before that’s getting cleaned in the war room, then gets to the inspector for review, and we know what happens. This all gets back to the human risk. This all gets back to the human achievement stuff, right? And the validation cycle. And beat any process or any system or anything that’s being validated. So validation is not one thing; it’s not just when it does. It’s not about just passing the validation. Recording in progress. It’s beyond that. It should be continuous. The cycle is supposed to be continuous, right? Okay. Looking, getting to the warning letters. So we have gone through so many warning letters. How many of you remember the first warning letter? We say okay, there was it LinkedIn just like that. The next morning when I woke up, there were so many LinkedIn articles, so many messages saying that FDA issued a letter for AI, the first AI warning letter, and so on. Yeah, by April second, if I’m not wrong. So by April second, we had this first warning letter. It’s not about complete AI. It’s about a dedicated section for AI that has been highlighted by FDA, saying that. Inappropriate use of AI, the quality ones, right? So it’s not about the organization. It was so simple. They could have gone through the same challenge. This organization could have gone through the same challenge that everyone else have gone through, getting to the the pace how the organization is supposed to move with this environment, with all the AI capabilities. So fast-moving environment. So people will have to cope up with their competitors, come up with all those solutions, be the first, and all those things. Because of all those, they had so many documents produced, they had so many SOPs, they had so many policies, and everything has been done by AI. Is that the reality the organization follow? No. When there was, believe it or not, so when there was a question by FT inspector, as part of this warning letter, so with this 483, there was a warning. There was a question by the inspector asking, why did you guys not perform process validation? And the response was. Air did not tell us. Come on, is that is that how it’s going to be? Air did not tell us. No, and I can see here the air did not get the warning letter. It was the organization. It was the firm which got the warning letter. Right. The next one. Moving to another clinical data here. So there was for a sponsor, there was a vendor who provides all the clinical trial activities, and here after the inspection has been announced. The data was deleted along with the audit trail, so all the data from the electronic ecosystem has been deleted. So this does not just delay the approval process for a drug; it’s beyond that. It’s about patients. It’s about all that has been happening, right? So clinical trial activities. It takes many years, many years of clinical trial activities. Forty-seven patients, forty-seven subjects. All the data has been deleted. One vendor. Just a human sitting on the vendor side, deleted. They have process, they have policy, they have SOPs, they have all the controls. Tragically, failed. There was still something that’s happened. So all the data deleted. It’s not the sponsor who just bore the consequences, but also the 47 patients who are waiting for a bad drug ready for them. Yeah. So another one. So falsification, this the most crucial one, right? So falsification of data. So this the highest end that the human race could go on. As part of our inspection, so FDA was able to find all the analytical reports, all the paper records, all the activities performed by the clinical lab. So they found it on garbage. There were so many backdated record identified. There were so many stuff missing. So. Think of this a lab. It’s not about one sponsor, one clinical organization, one one pharma company. It’s not about one pharma. So this lab could have been providing services to another hundreds of pharma companies. It was one inspection for one sponsor, so you found it out. But how about the other remaining 99? So it’s not one pharma one drug issue. This hundreds of pharma companies issues. This gets wider than that. How many studies could have gone wrong? How many patients could have had impact, and so many other things. All these human, human error, or we could adulterations, or whatever term that we could put. So, finally, we had so many controls, so many measures, and everything that’s still there, but still we have warning errors, and all those get attributed to human error. After all those points, after all those things that we follow. Getting on. When we look at all these patterns, I’m not putting all these against the humans, as I said. There are multiple other situations. It could be many reasons why things could go wrong. The ultimate reason, end of the day, is different countries, different organizations, different areas within the life sciences. Everything ends up to the same root cause, the same issue that we get back. It’s going to be human error. Is that okay? Fine. Identifying the human error and human risk and all those things are fine. We will have to come back and we have to move forward and so on. But how are we going to overcome that one? That’s when, when we can utilize AI in a defensive manner, in a defensible manner, put your AI in a defensible mode so that it prevents, avoids as much as possible and reduces the human risk. It’s not about reducing the human risk. It’s not about reducing the human error. It’s about reducing any risk to the patient. Who is waiting for those drugs? So, what if AI is in control? If the AI is fully controlled, it’s not about a human. Not not just replacing human. It’s not going to be a faster human. If we are going to use AI as a faster human, it’s going to be many mistakes and many errors in a faster manner. It doesn’t replace what it happens, right? So we don’t need that AI. What about the compliance chartlet? We don’t need AI to be a compliance chartlet. We don’t need AI to generate SOPs and policies. We can do that. So we need a KMS. We need organization. We need a KMS. Just couple of hours, it can come up with all the policies and SOPs and all those things. Is that what compliance needed? Is that going to be the reality? Will that be followed? Not in that way. So get AI as a product to know. That’s not the reason. So it’s not about. Saying that no, my AI just drafts something. It’s just the AI which helps me drafting something. No, it’s not out there. Famous. It’s not the way how it is supposed to be. It should be clearly structured, documented, validated, and governed. That makes you to start the way how future is going to be. How the AI could be in a defensive mode. How it could be under control, and how we could put that. So, and. We always have in the industry, right? So we always used to complain against, in a positive manner, saying that regulators are too slow. The regulations they have the regulatory bodies are too slow. The industry is going far ahead. We have so many technologies. We have so many stuff. We are still stuck with guidelines that came decades ago, and we are still stuck there. And all those things. We have so many complaints, but that’s not the case actually. The regulators, be it the industry standards and everything, are going together with us. So we have AI. Everyone in the floor needs to use AI, but how many are ready to adopt AI in your organization? Everyone raises the hand to have AI, but at the same time, everyone raises hand saying that we don’t need to be the first adopter. Why? So we don’t have a clear process. We don’t have governance. We are not confident on what AI could do, how the AI could be structured, how this could be controlled, and so on. That is where we have to step forward, go forward. We already have we have the guidelines for the credibility score from the FTA late almost one and a half years ago, and now we also have ISP and AI guidelines that release that was released, which clearly talks about all the AI related validation activities, how this could be validated. Because everyone needs AI, yes, but when we speak about GXP improvement, obviously there is another question: How am I supposed to validate my AI? How am I supposed to do that? Because everyone validated their computerized system, made them that in a validated state. That’s a different question. Now we are talking about AI validation, but still, we already have a guideline. We also have Annex 22 that has been released, and the draft that is still there that’s still being reviewed by industry, and we are getting them there. And then we had AI algorithm finding, because that particular it’s it’s not a board of firm, so. There was a finding that the pre-marketed approval was for an exercise monitoring app. It was something about an exercise monitoring app. But the marketers saying that it’s about mobility and usability and how they can check the mobility and stability of the person, the patient, to early diagnose the Alzheimer. Then come on, this not this not an exercise app. This diag. This clear diagnostic, and that’s why AI algorithm was used to identify early Alzheimer identifying early Alzheimer’s. But they wanted to capitalize some random wellness app. Come on. So it’s not about the organization how you call what your app is about. It’s about the intended use. What the FDA is going to see how we are using that app, and that’s over. That was the first one. This was even before the April second this year’s AI specific guideline, sorry, AI specific finding that was provided, and then we had a combined one, a joint, joint practice principles that was released by FDA, EMA. Even that was there. We also have that one, and by by April, all the excuses that we had is gone. It’s it’s ran out. So now. The regulators are behind asking for how you have your AI, what is happening, and all those things. It’s going to be the same level how the humans are going to be. The difference is, in a right manner, we will be able to control AI in a right manner. In a right manner, would you be able to control humans? Are we sure that they are going to do the same again and again and again? Follow the process without tampering elements, without tampering all these things and all those. It happens. Because it could be just because they are rushing to meet their friends over the Friday evening for a vacation, they need to get this done, get it signed, and so on. What is the quality that’s going to be that Friday evening work that’s being produced? But if it’s going to be a day, it won’t be the same. How many of us have been? You you do it midnight or over the evening or Friday evening or Monday morning. It’s going to have the same basis going on. So how are going to put them together? So how do we structure and frame our AI in a defensible manner? So how do we put this as a defensible AI? So here, after all that we have the guideline, the regulatory guideline, industry guideline, regulations, and all those we put together. When we distill them, there are simple five properties that we could consider. It’s not only these five. It’s not only these five. There are going to be multiple other sub-properties and all those things that can fall under. But you can categorise them under these five properties there. So one, document the intended use. This is most crucial one. First, define your AI. If you don’t define your AI, if you are not able to define your AI, you cannot define your AI using in your organisation. So document the clear intended use. Intended use. When it comes to AI, it’s not just about telling what my AI can do. It should also tell what AI cannot do. Then getting the risk proportionate validation. So obviously we all know that the ATCSA came up and how the scripted, unscripted, how it reduces the testing like all those stories are. So but for AI, the 20 be at a different level. But still risk proportionate. In short. We perform risk assessment specific to the AI, specific to the AI model that we use, and how that’s going to be. And based on the risk outcome, is what the AI related validation is supposed to be. And when we talk about AI validation, it’s going to be continuous. That’s what I will put in. Let me put this 0.5 when I on the flow. Yeah. So the AI validation is going to be continuous. It’s not one time, because we know that AI can avoid, but we can control that from avoiding abnormally. But still, there could be some drift. There could be some differences from the first initially validated model to the latest model that’s in there. It it can continuously evolve in such a way that the consistency could improve, or it could decrease. The hallucination rate could vary. The consistent the content integrity could differ, and all those things. So, validation for AI is supposed to be continuous lifecycle. It’s not one time method we have been doing for a computerized system. Where we perform a periodic review every one year or two years, we never bother about what’s happening in the in between. And just before an audit or just before an inspection, a periodic review being performed after something has been announced. No, not just for a computerized system. Even for computerized system, we are talking about continuous validation right now, including for that obviously for AI, which also continues, basically continuous monitoring, continuous validation, and the AI. That’s obvious. So so here when we talk about this one, so the audit trail when we speak about audit trail, so any computerized system, so bid NX level or bid part level, part level. So we are talking about NX level. Sorry, we are talking about audit trail again, again, right? So, but the same audit trail work the way how we see for AI and for the human activities, the existing audit trail is supposed to say. Who performed the activity? When was when was it performed, and what was the previous value? What was the current value, and what and all the things? Is that enough? No. For AI, it’s supposed to be beyond that. For AI usage, we’ll have to say which particular user triggered it, so we show that AI was triggered, but it does not do something autonomously, and then we also say that this particular AI, this particular LLM, this version of the LLM, this is the version of the model. We can call it as model card or element card, whatever you could say. And this the version, and this was the input that was provided to the AI, and this the outcome, the output that the AI has generated, and this the justification that the AI provides. And for each and every card, should be there should be a justification as well. And then, confidence score, how confident is the AI with all those? What was the resources used? So the audit trail is getting bigger. It’s not just a one-line audit trail anymore, and human in the loop. Yeah, human in the loop. Yes, it should be there. There is another argument. So, will a human be able to review whatever AI generates? That’s another different question. That’s a different argument. Let’s not go there for now. So, how is that going to be? Obviously, we are the one who are supposed to be owning them because. The FDA is going to issue a letter of intent onto the AIFFs, so it will have to be under our review. It will have to be each and every cycle, every stage. It will have to be us human who owns the account of duty, who takes the responsibility, who ensures that the human gets checked as far as before anything happens in their quality unit. So, couple of examples based on all the fine properties which I put. So here. Wanted some time back, so Modana, there, there was. This about a presentation that was provided by a colleague from Modana, and this his personal view, not something that Ebs Modana has spoken. So let me put the way how he has highlighted it in a session. So one, public model evaluation. So here, the public model evaluation. First, know what the existing base LLM model that you are using is going to be. So how can how can you leverage that one? What is the capability of your existing LLM model, publicly available one, and then platform internal benchmarking. So internally, you you can internally have how this is going to be. So the data that’s been used for internal purpose. So in the AI in your use case in your organization, how that could be, and the third step is use case. So yes, that the For the AI models and how that’s going to be, that’s obvious. Intentionally, those terms have been put in the default validation principle way. And the fourth one is the most crucial one, which I’m going to tie to the other five properties. The quality monitoring. Here, the AI assurance is not one time. So, all that what we are supposed to get out of what was performed here. The key note, the key takeaway is, AI validation, AI assurance is not one time. So it’s not one time; it’s going to be done. So you will have to keep a tap on. You have to continuously monitor the next one. There is another one. So Deutsche Bank, you say they had another one. They introduced they had a their own in-house generative AI. Because we are talking when we talk about AI, there are so many shadow AI. We see that in many organizations, right? So all the central banks, Germany, and all those things being used for each and every activities. Generating documents, drafting reports, and drafting deviations or tapas and all those things, right? So, avoiding that, there is a data leak that could happen. There is a proprietary information that could get out, and there are sensitive information that could go out to any random public available model. So, they cannot. They won’t be called Gen AI. So, this is not about that being used. So, the part, the takeaway from here is the governance of Gen AI. So ensure that the organisation has a governance in that to put them in place. So have SOPs and policies and all those things. Default one yes, but it’s beyond that. Ensuring there is no data leakage. How do you do that? It’s about not just when we talk about AI, when we talk about data trails and all those things. It’s not about just killing AI. Okay, you are supposed to act like my assistant from now on. You are supposed to do only this, not that. And this not a soft cutoff. This not going to help. Yes, that might to some extent. Beyond that, we also need hard models and things to be set up. And how the government if it’s all based on the governance that you have for the arm. So, I’ll put them together. Okay, I’ll actually move to this one. So, now after all these, I’m intentionally showing showing this one, AI-enabled digital validation platform because validation is about. Almost more than decades long for my own space. That would be much more comfortable for me to talk about it. So let me say how this is going to be a validation tool. So, but it’s not about any specific one. So in general, it’s going to apply for any software that’s going to be used or any other form AI to be used. So when we speak about documentation failure, ask what the AI will be able to help. How is that going to, Produce any data or come up with any information that is needed. How the inconsistency would be avoided? How the incomplete record could be? What is the justification on how things could happen? So these are the questions that you will have to ask for any tool, any of the systems. And then to the data integrity. So the attributability, which I also showed. So it’s not one-liner attributable information under the audit trail. It’s going to be a complete detailed audit trail regarding AI. We should be able to read whatever need audit trail. We need to trace back and reproduce everything that has happened. For that reason, for AI, it’s not going to be just AI generated. This one, no. Why did AI generate? How did it generate? What was the input? What was the output? And all those tedious bits. And then the investigation obviously applies AI for investigation. So it’s not just about human error and of saying, “Ah, okay, here is the final outcome.” So here is the Proof for human error. No, beyond that there is something that when there is there is something that’s happening repeatedly, it’s a pattern. It’s not a human error. There is something wrong that needs to be fixed. Use AI and go beyond that one. We’ll be able to get things right. And oversight and change change control. So use change control. For example. So we can have the change control, any impact assessment to be performed. Something all the anytime they need to perform any change management activities and impact assessment, there could be some issues or some gaps that get missed here and there. But utilize AI in such a way that it ensures it captures all the impacts it wants, it identifies all the impacts and so on. However, ultimately the oversight is supposed to be with you. And obviously, the traceability you have the traceability part covered. So that’s one biggest challenge that we have in terms of any audit, in terms of any audit or inspection traceability. It’s not about system backlinks traceability, it’s about any record traceability. Every plan is supposed to be linked to something. It could be a study protocol, it could be any EDM of records, or it could be any clinical trial data or anything. All these are supposed to be linked, and all these are going to be traceable. So utilize. AI where possible to improve the pace of any part that’s not proven. So here, putting them together, go to the defensive AI control. The final three steps that could be there. One, AI performs simple. Utilize AI where possible to improve the efficiency to reduce the human effort under control with all the governance. How big it in terms of the previous one she said so. We need a risk assessment activity, or test scenarios, or executions, or anything that could happen. Put that in such way, then the human review, human being under control. It’s not about human in the loop because when when the human is given with hundreds of documents or tens of documents generated by AI, will the human review the same page and each and every part and get it done? So it’s not about human in the loop. Rather, let’s put it as human centric design. So. It’s supposed to be rethought in that way. So have it here, a named user who is going to take all these controls and so on, and then finally the system locks it all in an in in such a way that it’s not broken. All the data are immutable records, and we have all those verifiable, full audit trail and so on. So these are sample questions that any inspector is going to ask you when they walk into your organization. Do you use any AI tools? So, how who reviews and who owns and who approves those AI output, and is the AI generated data being validated? Who, how was that, and when was your last system validated, so on? And are you sure that your AI is performing the same way how it was performing the initial time when it was validated? So, these are some random questions, but these are very questions that anyone is going to raise against your system. So, what does the world have to do tomorrow? Three steps, three simple steps. If you have don’t, if you don’t have any of this in your organization, three simple steps. Put them first. Inventory your AI. Identify all the list of AIs. It could be, it could be any sort of AI. It could be for anything that’s being used. Identify all possible AIs and talking about the shadow AIs as well. Put them and label them. Define the intended use and say why those AIs are being used and what is the purpose and how your organization is going to do that and so on, and enforce a human gate there. So how is human going to ensure from that stage for for all those AIs and all the outcomes and anything before it gets to a GXP record? How they are going to put it in there? So have a human gate made there. Establish how this is going to be and ensure you set up a continuous validation stages, validation lifecycle. For your AI, not don’t have a random AI being used. Prepare your AI to be a reputable AI in GSP environment, so it benefits not only your firm but also the patients. Finally, untrained humans, even trained fails. That’s a different case, but untrained humans fail in some inspections, and untrained AI fails patients. Ultimately, yeah. But at different simple AI that we spoke now, when it’s all under control, may fail. It’s not all about patience and pain. Let’s do our best for what we could do. Thank you.

[37:08] Speaker 1: I like what you said. There will be AI in the loop, but after quality, something will couple the quality professional people.

[37:26] Speaker 2: Are two stages where we would. I mean, there are two stages where I would suggest to how the AI could have all the preliminary checks because. In quality department, there are days where we check that okay, there are multiple SMEs who have already put all things together. So this person would have done it right, and this person would have done it right because this person is ten years experienced, this person is fifteen years experienced. And quality we attend. On days we get to documentation review. Is there a typographical error? Is there something wrong? And quality gets to that level. So that is not what quality is supposed to be doing. Those part could be given to AI. But ultimately, how the regulation is supposed to be? What is the regulator going to see when they see those process deviation? For example, is the root cause right? Is the corrective action that we are put in place is right? So that is to check what quality is supposed to quality department is supposed to do. And here AI could be used as an assistant to perform those minor check, typographical errors, all those things. But ultimately, after those suggestions and anything that we could have missed, what AI is going to provide? That’s going to be like an assistant that we can leverage, but Quankey will be the final one who will have to do. Obviously, yeah, exactly.

[38:47] Speaker 1: True.

[38:55] Speaker 2: That’s that’s true. Because I I think. Obviously, that’s not right. Right? Are we in the same speed? Where are we same speed with the AI? Because if we say, for example, we have been from my side, we have been writing validation plan that will be 30 pages, 40 pages for a complicated system. It takes few hours to few days, for example, to get the validation plan written by the group of people. Now put all information to AI, and that gives within few minutes and. Think of a week because when it was a week earlier, you work with the team. We talk every day, we collaborate, and we come up with the final document together. So it’s easy for the final review. But all together, the next morning you have a brand new 14 pages document that you’ll have to review and sign it in next couple of hours. But how do we do that? So few few are political. I cannot say them here.

[39:51] Speaker 1: We do have a quick question from the virtual side. Where do you see intended use being documented in practice, in the validation plan itself, or in an upstream artifact like a criticality risk assessment that the plan then references?

[40:08] Speaker 2: So the intended use could be documented either as a separate description document for an AI, if that’s the ultimate solution that you’re using it by itself as a system. Or otherwise, if that’s going to be an enabler for an existing system, so it’s just AI capabilities being enabled, so you can document all those within your existing validation plan. So the point is, it’s not about where we document; it’s about do we document. Yes, we should document. So how we document? That’s up to the organization. So it depends on how complex your AI capabilities are, how much you’re going to use it, where you’re going to use it, and how. And then, questions. Yep.

[40:50] Speaker 3: I have a question about you know the systems that are being used for quality people to do review, right? Especially like what what needs to change in the software, in the user interfaces, for example, that promote the level of understanding and that, sorry, that help quality folks reach the same level of understanding that’s required in order to make in order to make the determination of whether they, Like whether to sign off or or not, what in the software systems need to change, from the UI, from you know tooling,

[41:25] Speaker 2: Etc. So from the software perspective, so if you ask from software perspective, I would say that there should be a clear review cycle that happens. Yes, the review cycle happens, and most of the software that we see, when they put them together, yeah, get it reviewed with the quality team and get it done and so on. But the review cycle all those get dumped and the offline. It could be over a chat, or it could be over an email, and all those things, right? So that will have to be in such a way happening within the software, where it could be or UI changes or any feature addition, or so on. So those are being captured. So this is being in that, and it’s it’s being covered as part of the processes of those that you could have. That would significantly reduce the regular productivity of what we do, and I mean, it could increase the productivity and reduce the effort that we spend on these. Review cycles and outside and having anything perform offline. When we have AI being performed, because I would say AI being enabled in a software is going to have all those features and capture all those information inside. And similarly, we should be able to do that as well because we are supposed to justify, defend, and document and clearly state why we took it forward and why we accepted something. And there cannot be any hidden figures. And all those acceptance and things like that. What I personally see is those get submerged in all the. Review cycle offline. Review cycle in the chat or email or something. When we get to the record within a system, that’s going to be simple. The final outcome, but it should be beyond that. So that’s how we can adopt enough. We can change something.

[43:00] Speaker 1: So the last one. Yeah, last question. Good presentation. I think at the beginning of the presentation, some of the examples, terms of warning letter, surface fraud and false. In the last two years, internationally, whether you’re a global company, or you’re a newer company, or you’re regionalized in another part of the world, this has been heavily cited. How do you control fraud and falsification if, whether it’s paper, digital, or even AI, you’re already and some of those come from leadership in those countries where there’s a fear of losing your job, and hence people don’t raise their hand; they do the wrong thing. Because they’re instructed. How do you think that now, if we go from the evolution of paper to digital to AI, we now have all these things? How do you control?

[43:52] Speaker 2: That? So that’s that’s something we try to control. So when we are setting up some controls to the AI, so we will know that one first. First answer should be a it’s a cultural issue. It could be organizational culture or whatever. So that’s where we come up with all these things. Right beyond that. So when we talk about AI, that.

← Adoption panel | Home | Observability and evaluation →

Graph View

Backlinks