2026-09-15-0909-jory-gonzalez-gxp-ai-panel-poc-hell-roi

GxP AI panel on POC hell, ROI, and human oversight

Recorded 2026-09-15 at 0909 via Plaud. 46:38.

Summary

A four-speaker panel at the GxP and AI conference in Raleigh, moderated by Dori Gonzalez-Acevedo (CEO of ProcellaRX), with Gaurav Walia (VP at PQE Group International Consulting, a 30-year industry veteran, ex-Pfizer drug inventor, and former director of quality with over 100 FDA inspections and three consent decrees), Anant Agrawal (Head of Data at CIMCON Software, Carnegie Mellon-trained in statistics and machine learning), and Tim Roy Kirkelie (COO of Genari AI, from a manufacturing and regulatory QA background). Acevedo frames the arc as 2024 the idea year, 2025 the pilot year, and 2026 the year of judging outcomes, and opens by citing the Anthropic CEO’s essay calling for a slowdown as reason for more nuanced conversation rather than retreat. On POC hell, Kirkelie describes AWS-funded proofs of concept from three years ago that went nowhere for lack of a long-term plan, a team that switched Jira to AI everywhere and slowed its development to a crawl, and a company he calls really good at this stuff that is still in POC hell; his biggest adoption fear is users just pressing the button to approve AI output. Walia prescribes a stepwise path: define the problem before the tech, plan the process and culture change, ensure quality readiness, then move POC to pilot to commercialization with the finish line defined first.

On ROI, Agrawal argues the investment is upfront while the benefit comes later, so vendors should prove value with the client’s own data first, as a consulting engagement, before any tech buildout to avoid IT hell; she also calls accuracy percentages misleading, because 95 percent accurate still means a human must review 100 percent of the output, so ROI should be reframed as the time it takes humans to find the critical 5 percent of errors, supported by a broader error taxonomy covering omissions and conflicts with input data. On human-in-the-loop, Walia cites newly published brain-imaging work showing a decline in critical-thinking activity when people use AI, notes that with five signatures on a document probably two signers are not paying attention, and predicts a future state of continuous change control and continuous validation that still starts and ends with the human. On regulatory uncertainty, Acevedo, a co-author of a digital validation good practice guide, explains that Annex 22 is still in draft after its original no-probabilistic-AI stance drew pushback in a June meeting, asks why people trust AI with their bank account but hesitate over their drugs, argues cybersecurity is the bigger immediate risk (one organization’s 190 security controls can only be managed with AI itself), and closes personally: her bipolar disclosure in front of 600 people at a Boston keynote, and two interns who arrived anti-AI and left ten weeks later integrating agents into their work. Algarwal, speaking as a vendor, recommends focusing on non-GxP, low-risk but high-ROI use cases now rather than waiting out uncertain regulation, says the 95-percent-of-AI-pilots-fail studies have holes in how they measure ROI, and pushes a reward-first mindset: reinvent whole processes together instead of each person getting 10 percent faster with Copilot.

Plaud AI notes

App-generated, verbatim from Plaud’s Summary tab.

Date Time: 2026-09-15 09:09:33, 2026-09-15 09:09:33 Location: [Insert Location] Interviewee: [Insert Name], Speaker 1, Tim, Gurav, Bret, Speaker 4, Speaker 2

TL;DR

Panelists from pharma quality, data/AI, and AI startups discuss adopting AI in regulated GxP contexts, moving from ideas to pilots to scaled deployment. A central theme is the industry’s struggle with “POC hell” or “pilot-itis,” where numerous small-scale tests fail to translate into a cohesive, long-term strategy. Key takeaways include the need for a problem-first, reward-first framing rather than leading with technology or risk; stepwise validation; and redefining ROI to focus on reducing human review time instead of headline accuracy metrics. The discussion highlights regulatory uncertainty (e.g., draft Annex 22) and the cultural shifts required for adoption, stressing that the “human in the loop” remains critical to counter the risk of diminished critical thinking. Panelists advocate for a balanced approach: starting with non-GXP, low-risk use cases to build momentum, while fostering cross-functional collaboration and AI literacy to ensure responsible, safe, and transformative use aligned with the core mission of improving patient outcomes.

Introduction

1. Dori Gonzalez-Acevedo: As panel moderator and CEO of ProcellaRX, Dori framed the discussion around responsible AI in GxP, setting a timeline from “idea” (2024) to “pilot” (2025) to assessing outcomes by the end of 2026.
2. Gaurav Walia: A 30-year industry veteran and VP at PQE Group International Consulting, Carl emphasized a stepwise POC-to-pilot-to-commercialization path, focusing on problem definition, culture/process change, and measurable business outcomes.
3. Anant Agrawal: Head of AI at CIMCON Software, Anat advocated for reframing ROI around human-review time, proposed a more detailed error taxonomy, and recommended proving value with client data before heavy IT investment.
4. Tim Roy Kirkelie/Speaker 2: COO of Genari AI with a background in manufacturing and regulatory QA, Tim addressed moving from pilots (“POC hell”) to production. He stressed the importance of governance and AI literacy to ensure users think critically and use AI responsibly, aligning its rollout with business objectives and the special responsibility of the life sciences sector. He later argued that AI adoption must be a collective effort anchored to improving patient outcomes.
5. Speaker 1: Framed the discussion around regulatory uncertainty (e.g., draft Annex 22) and the industry’s hesitancy, questioning why AI is trusted in high-risk finance but not GxP. She advocated for reinventing validation, highlighted cybersecurity as a greater immediate risk, and emphasized the personal stake in progress, as everyone is also a patient.
6. Gurav/Speaker 3: Provided a broader perspective, noting life sciences lag behind other industries. He argued for building momentum with quick wins to overcome the “stop-start” cultural challenge and warned against the decline in critical thinking from over-reliance on AI, stressing that human oversight is essential.
7. Bret/Speaker 4: Representing a software vendor, Bret encouraged a practical approach, suggesting companies focus on high-ROI, non-GXP use cases now rather than being paralyzed by future regulations. As Speaker 4, she later advocated for a “reward-first, risk-second” mindset, urging organizations to identify ambitious, motivating goals to reinvent entire processes collaboratively.

Key Points

1. Overcome “POC Hell”: Many organizations are stuck conducting numerous AI proofs-of-concept without a clear long-term strategy, failing to achieve real traction. The solution is to define the “finish line” (intended business change) upfront and ensure pilots can scale to production.
2. Adopt a “Reward-First, Problem-First” Mindset: Start with a bold, optimistic vision for what AI can achieve or a well-defined business/quality problem, not the technology itself. Assess cultural readiness and process changes before piloting.
3. Rethink ROI and Metrics: ROI in GxP AI should be measured by end-to-end workflow impact, especially the reduction in human-review time needed to locate and correct critical errors, rather than misleading headline accuracy claims. Expand error metrics beyond generic accuracy to include omissions and input-data conflicts.
4. The Human-in-the-Loop is Critical: The “human in the loop” remains essential for verification and oversight, but there’s a growing risk of diminished critical thinking and over-reliance on AI. Strong governance and continuous verification are necessary.
5. Navigate Regulatory Uncertainty with Pragmatism: With regulations like Annex 22 in draft, companies should not be paralyzed. Starting with low-risk, non-GXP use cases can provide high ROI and build momentum for broader, more regulated implementation.
6. Reinvent Validation for Dynamic AI: Traditional validation practices must be reinvented to accommodate dynamic AI. Unless models are frozen, processes and culture must adapt to evolving systems, leveraging prior validation knowledge and moving towards continuous change control models.
7. Foster Collaboration and AI Literacy: Successful integration requires breaking down organizational silos and fostering cross-functional literacy. Scientists, quality, and AI specialists must align on governance, and all employees need to understand how AI works to use it responsibly.
8. Cybersecurity is a Parallel Risk: Cybersecurity is currently a greater and more immediate risk to organizations than AI, yet it often receives less effective, more “check-the-box” style training.

Insights

1. Dori Gonzalez-Acevedo
   • Calls for nuanced, responsible AI discussions and highlights the need to update validation education for the new mix of talent in pharma (e.g., data scientists).
2. Gaurav Walia
   • Warns against tech-first approaches and advocates for a deliberate, stepwise path with the end goal defined. Notes most top firms are still in early pilot stages.
3. Anant Agrawal
   • Identifies misleading accuracy metrics as a key ROI challenge. Recommends vendors prove value with client data first and focusing on the time to find and fix errors.
4. Tim Roy Kirkelie/Speaker 2
   • A major challenge is getting people to slow down and think critically, not just blindly approve AI outputs.
   • Increasing AI literacy is crucial for responsible and effective use. The ultimate goal should be improving patient outcomes by partnering with AI to make jobs more meaningful.
5. Speaker 1
   • The life sciences industry is overly cautious with AI compared to other high-stakes sectors like finance.
   • Cybersecurity is a more significant immediate threat than AI, but training for it is often inadequate.
   • Collective progress toward a shared “north star” is more effective than competing departmental goals.
6. Gurav/Speaker 3
   • Life sciences can learn from more mature industries like finance and automotive.
   • Cultural change requires demonstrating “quick wins” to build momentum.
   • Over-reliance on AI is a real danger that can lead to a decline in human critical thinking.
7. Bret/Speaker 4
   • Companies should not be paralyzed by future regulations; focusing on current, high-ROI, non-GXP use cases is a winning strategy.
   • True transformation comes from reinventing entire processes collaboratively, not from isolated 10% improvements. Start with an optimistic vision of what could go right.

Chapters

1. How has the arc from idea to pilot evolved, and where are the gaps?
   • Gaurav Walia: The industry has moved from slides to real pilots, but it’s still early. A four-step mindset is needed: define the problem, plan process/cultural change, ensure quality readiness, and proceed from POC to commercialization. Always define the finish line first.
   • Dori Gonzalez-Acevedo: Reinforces that validation fundamentals must be taught to new audiences as AI evolves.
2. How do we get to a value-driven methodology and ROI with AI?
   • Anant Agrawal: ROI is hard because investment is upfront. Flip the model: have vendors prove value with your data first. Reframe ROI around reducing the time it takes humans to find the critical 5% of errors, using a detailed error taxonomy (omissions, contradictions) to guide review.
3. How do we move beyond “pilot-itis” or “POC hell” and integrate AI responsibly?
   • Tim Roy Kirkelie/Speaker 2: Companies are at a crossroads, stuck between moving too fast and too slow. The challenge is user over-reliance—“just pressing the button.” The solution is to balance speed with governance, align with business objectives, and increase AI literacy to ensure responsible use.
   • Gurav/Speaker 3: Life sciences can learn from other sectors. The key to overcoming “pilot hell” is demonstrating quick ROI to build momentum for cultural change, starting with small wins.
4. How should organizations navigate regulatory uncertainty like the draft Annex 22?
   • Speaker 1: Questions the industry’s hesitancy, especially when people trust AI with their finances. How do we foster gradual adoption instead of an all-or-nothing approach?
   • Bret/Speaker 4: As a vendor, it’s hard to plan for uncertain regulations. The focus should be on non-GXP use cases that provide high ROI today. Successful pilots are common when planned intelligently.
5. What are people not thinking about, and what is the next step?
   • Tim Roy Kirkelie/Speaker 2: The key is AI literacy—training people on how AI works to keep it controlled and safe. This allows professionals to focus on high-value work, leading to better patient outcomes.
   • Speaker 1: Points out the discrepancy between AI fear and the often-inadequate response to the greater risk of cybersecurity. With 190+ security controls, AI itself is needed to manage them effectively.
6. What one thing should people do differently? What should they take away?
   • Gurav/Speaker 3: Focus on the human element. AI use can degrade critical thinking. The future requires dynamic, continuous validation, but it must start and end with human oversight to guard against errors and misinformation.
   • Speaker 4: Adopt a “reward-first” mindset. Start with an optimistic, motivating vision of what AI can achieve. The biggest returns come from reinventing entire processes collaboratively, not from individuals making small 10% improvements.
   • Speaker 2: Everyone in the organization must be involved. The ultimate goal is to improve patient outcomes by partnering with AI to make jobs more meaningful.
   • Speaker 1: Emphasized that everyone in the industry is also a patient, making the work personal. She urged the audience to break down silos, find a common language, and move forward together with new technologies.

AI Suggestions

The interview questions effectively established broad themes like “POC hell,” regulatory uncertainty, and ROI. To elicit more concrete and actionable insights, they would benefit from pointed follow-ups. For instance, after a speaker mentions overcoming “pilot-itis” or the fear of users “just pressing the button,” a strong follow-up would be: “Could you describe the specific guardrails, metrics, or training methods a company found effective in encouraging critical thinking over blind approval?” When discussing ROI, push beyond principles to operational detail: “Can you quantify a before/after workflow? For example, the number of documents reviewed per week, the minutes per review, and the percentage of omission errors caught before and after implementing your proposed method?” To make high-level concepts more tangible, ground them in practical first steps. For example, after discussing the need for AI to manage numerous security controls, ask: “What is a practical first step a company could take using AI to audit or manage just one small subset of those controls? What would that ‘quick win’ look like?” Similarly, when a vendor discusses future-proofing their product, ask for specific architectural decisions being made today that would allow their platform to adapt to future GXP use cases without a complete redesign. This pushes the conversation from strategy to implementation.

Highlights

The two main problems with measuring ROI are that the investment is upfront while the benefit is later, and it requires significant initial investment beyond just money.

• Speaker 4 states the first problem with measuring ROI is that the investment in money, organizational change, and training is upfront, while the benefits come later. • The second problem identified by Speaker 4 is that a high accuracy percentage, such as 95%, doesn’t equate to a 95% time saving because a human must still review 100% of the output to find the 5% of errors. • Speaker 4 proposes that ROI should be measured not by accuracy, but by the time it takes a human to review the AI’s output, and suggests vendors should focus on helping users find errors more quickly.

ROI in a complex system (sandeep swadia) is challenging

• Risk Identified: AI accuracy metrics (e.g., 95% accuracy) are misleading; staff must still review 100% of the output to find the 5% of errors, negating much of the promised time savings. • Key Insight: The most effective way to measure AI value is by focusing on the reduction in ‘time of human review’ rather than the AI’s raw accuracy. • Key Insight: A better approach is to treat AI adoption like a consulting engagement: have vendors prove the value with your data first, before committing to a large technology and process overhaul. • Key Insight: ROI calculation is difficult because AI requires significant upfront investment in technology, training, and organizational change, while the benefits are realized much later.

pilot-itis

• Key Insight: Pilots frequently fail because they are technology-driven rather than business-goal-oriented, and they lack a long-term strategic plan from the start. • Key Insight: Lack of involvement from end-users is a primary reason for failure; top-down AI implementation slows adoption as teams are unprepared to use the new tools effectively. • Action Item: Establish a clear governance framework and involve the people who do the daily work in the planning process to ensure successful adoption. • Risk Identified: Users may become too comfortable with AI outputs and approve them without critical review, creating a significant operational risk. • Action Item: Define business goals and ROI metrics, focusing not just on model accuracy but on the reduction in human review time and overall business value.

It is predicted that regulation will change for AI when there is enough confidence in the processes and sustainable guardrails are in place.

• Speaker 3 noted that a commonality across industries using AI is the necessity of human inclusion today, which informs his prediction of future regulatory changes. • The speaker references a U.S. executive order on AI which attempts to provide a path forward, highlighting the current relevance for organizations dealing with regulations like GDPR and HIPAA. • Speaker 3 explains that the evolution towards new regulations will require developing proper ROI, establishing the finish line upfront, and scaling up appropriately.

just fyi, this session was actually a panel discussion

• Context: Jory Gonzalez Acevedo, CEO of Procellarex, moderated a panel discussion about enterprise AI adoption. The conversation explored the journey from initial concepts to pilot programs, focusing on ROI, regulations, and organizational culture. • Key Insight: The panel featured diverse expert perspectives from Carl Walia (PQE Group, pharma), Anant Agrawal (CIMCON Software, data), and Tim Kirkelie (Gerani AI, manufacturing/QA). • Key Insight: The central theme was the evolution of enterprise AI from initial ideas in 2024 to the pilot phase in 2025 and beyond.

systems thinking super important. instead of 10% improvements here and there, rethinking entire workflows to be much better to begin with.

• Key Insight: Focusing only on how AI can make an individual’s task 10% better yields limited, incremental returns. • Key Insight: A systems-thinking approach, where teams reinvent entire processes together, can unlock significantly higher returns than the sum of individual improvements. • Key Insight: Realizing the transformative potential of AI requires a cultural shift towards cross-functional collaboration and a shared vision for process reinvention.

Transcript

[00:01] Speaker 1: Closing concept. How enterprises pilot AI.

[00:03] Speaker 2: In GXP.

[00:04] Speaker 1: I’ll turn it over to Dori. Thank you, everyone. I’m Dori Gonzalez-Acevedo. I’m CEO of ProcellaRX, and this is a really great venue because we haven’t had a big room like this in a long time. So many of you know a lot of the vendors that have been showing up, so make sure during the day you go and talk to them. They are industry experts in a lot of different places and have their own expertise. So not just who you hear on the stage, but make sure you go around and talk to them as well. I want to talk with my friends, Ravinat and Tim. But before we start, I want to give a little bit of context of this past weekend. I don’t know if anyone read the interesting personal essay that came out from Anthropic CEO. Did anyone read? How many people read it? How many no idea what I’m talking about? It’s probably one of the most important things that you need to read before this conference is over. And why that’s important is within minutes of publishing an essay that he calls for a slowdown of the progress of the acceleration that they have started. Musk and Mandela and and Sam Altman, they all said yes, please. Okay. Now, what does that mean? Does that mean like what we’ve been doing for the last two years and trying to figure out what AI is going to do and how to do it and how to do it in a responsible GSP way is going to go away? No. What that hope I have for for folks is that it gives us the opportunity to have more detailed and more nuanced conversations about what it is and how we are going to utilize another technology. This time we’re calling it AI, but AI has been around for a long time. For those that have been in the computer system validation world, this is not new. It’s a new evolution of it. But what are we going to do about it? How are we going to start asking those questions of ourselves? How are we going to ask those questions of our teams? How are we going to ask those questions of our organizations in order to help figure out what to do with this new technological evolution that we’re all sitting upon? So I’m going to go through and let my panelists introduce themselves, and then we’re going to start some dialogue. Sound good? All right.

[02:45] Speaker 3: Brad. Good morning. I’m Gaurav Walia. I’ve been in the industry for the last 30 years. Just quick background: started my career in chemistry and inventing new branded drugs. Working for companies like Pfizer as well as The acquisitions. Some of the drugs I helped invent are on TV today for AbbVie, and former director of quality with over 100 FDA inspections, three major consent decrees in the validation space. I’ve done exploratory training for 85 FDA inspectors in the last few years, focusing on the gaps that we see in the data that’s generated throughout the facility. Today, I’m the vice president of PQE Group of International Consulting.

[03:34] Speaker 4: Great, hello everyone. My name is Anant Agrawal, a bit of an odd duck. I went to Carnegie Mellon to study statistics, machine learning, and AI, but I’ve been working in the pharmaceutical industry. Now I’m the head of data for CIMCON Software. We do like generating validation documents, gap assessments, all that sort of thing. And I worked at Big Tech for a couple of years on recommendation algorithms, helping you find a movie type of stuff. And really excited to bring a bit of a different perspective to this conference. Thank you.

[04:04] Speaker 2: Hi, I’m Tim Kirkelie, COO of Gerani AI. I started my career. I refused this job three times to validate dry heat sterilization for a company that was under multiple 483s. So I kind of started kind of in hot water, and really never got out of it. Fast forwarding, been at different companies, biotech, medtech. And some pharma. I changed my career a little bit after you know we had that thing going on in two thousand and twenty. Didn’t want to go work back in Minneapolis, so moved into consulting and then started up a startup, Generative AI with Rosalind Beasley, which you’ll probably see later today. So I have a lot of experience in manufacturing. Regulatory QA, applying that to our product and really helping out a lot of companies to try to connect with what their experience is and our systems and so forth. So have a lot of fun with AI. I think it’s going to be interesting this week, next couple of days to find out what everyone else is doing and the problems and solutions. Hopefully we can share some stuff.

[05:27] Speaker 1: With you as well. Great. So when the topic of the conversation is going to be from idea to pilot to concept. I called 2024 the idea year, 2025 the pilot year, and we’re in the middle of almost at the end of 2026. So we’ve seen arc. For those of you in the industry, you should have seen arc as well within your organizations. But Karol, can you take us through how you think that arc has Been and where do you see some gaps in that arc today.

[06:03] Speaker 3: Yeah, good question. I think I think you know two years ago it was a lot of powerpoints and not real real data and real even real tech that’s that’s out there. And real tech that’s out there. So I think I think the evolution has become even in this last year. I think even even today, I would say from a global perspective, some of the top companies in the world. Have started PLCs or pilots in AI, but it’s still early in the game. It means that you know you if you if you look at some of the LinkedIn’s, I give the simple analogy of when the car came, you can’t and everybody was riding a horse or walking. You can’t suddenly ask a thousand people in your company to drive a Lamborghini. It’s not possible. And and so I think you have a four-step process. One is deliberacy and education behind what is the problem. Problem you’re trying to solve. It’s not the tech first. It’s what’s the problem trying to solve? What process am I going to change? How is the culture going to adapt to it? Not only the process culture, but the technological culture and what does it mean? Even the quality people and the auditors within the company. How are you going to withstand your own internal scrutiny as well as the external? With little to no regulation in place today. I think the arc that we see today is that you can even look in the room next door. You have AIs and then all bands. What you want to do is actually take a deep dive, ask the questions, look at the technology that’s available, look at the process to implement, and it’s a stepwise process from POC to pilot to commercialization. The idea is to understand what problems it’s going to solve. How can it withstand the change? Because today, we have to build upon our previous validation of knowledge. But at the same time, it’s a dynamic environment. Most part, unless you have a frozen AI model that doesn’t change, that means that the evolution of the process has to change. That means the culture has to change, and you have to evaluate where you’re getting value to the business, not just efficiency, but also that you improve your quality. Your output, because that is what’s going to be evaluated again. So ultimately, I think we are starting to see the initial stages of adaptation, but the plan has to include always the finish line first. What am I trying to change in my business? What problem am I trying to solve? And can actually get results from the PLC or pilot that show that you can get to commercialization or better global or widespread.

[08:55] Speaker 1: Thanks, Ross. So before I move on, I’m not going to build on that. So actually, the the most cited FDA problem is the lack of quality-controlled organizational units. Right. So that’s actually the most cited issue. And to your point, we’re trying to get to a new dynamic, a new way. And there’s been other studies out there that show, oh, 85, 95, whatever percent you want to call it. Pull whatever number you want. Have been failures, quote unquote. Right? Not. I know you’re asked a lot about ROI. So, how do we get from these numbers that people and we can skew numbers? You’re the mathematician. We know how numbers work, right? We can make it look like what we wanted to do. But how do we get from purpose to an ROI to a value? Driven methodology in today with AI.

[09:56] Speaker 4: That’s a that’s a great question. I think when it comes to measuring ROI, there’s two real problems. The first is the investment’s up front and the benefit is later, right? So you’ve got to put up a lot of money, but not just money. Change your organization’s way of thinking, train all of that, and then you sort of hope that the ROI will come from that huge investment. And so, just as someone that’s been in this industry and worked with a lot of different organizations, We usually see what’s best is okay. Can you give us an example of the output, or can we give you some information and sort of treat it as a consulting engagement first? Because that way you flip it. You get to see the return on investment first before changing your tech team, changing your tech infrastructure, and so you know, instead of getting stuck in IT hell and you know, figuring out all of the tech infrastructure, I think sort of seeing hey, like prove to us an output with our data, right? We don’t want to see your numbers, 95 percent, 85 percent, whatever. We want to see we see our numbers. And then the other problem we see with ROI is a lot of different AI providers will say things like 90 percent accurate, 95 percent accurate, but that doesn’t mean 95 percent time savings because you have to check 100 percent of the documentation, right? So just because it’s 95 percent accurate, how long does it take to find that 5 percent that’s missing? That’s really where your time’s going to go. And so we think about ROI. It’s really okay. This how long it took to generate. This how accurate it is. But then, how much time would it take for a human to review it? 100% of it. And so a big focus, I think, for these vendors should not just be how accurate it is, but how do you help people find those missing 5%. And you know, that’s something I want to propose, which is based on academia, a new error taxonomy for you know different kinds of errors you should look for, other than just hallucination, like omission. Did it miss something? Or. Interconflict with the input data. If your own data is saying contradicting things, the AI isn’t going to build a result that a human has to. I think changing how we think about ROI in terms of not accuracy, but more like time of human review. I think that’s really going to change the way we approach that.

[11:58] Speaker 1: One of the things I love when I talk with the Nots, or also Mike Salem from Gilead, two mathematicians that are now present in these rooms that weren’t present five years ago. How many by show hands? How many are scientists, chemists, biologists, some sort of hard science degree? Majority. How many are computer scientists by degree? Just a handful. And then we have other. The landscape of who we have in the rooms today has dramatically changed over the last several years in order to get to where we’re at today, and that’s really important to come back to Groves’ point. We have fundamentals that have been in place for many years that have worked when applied correctly. It’s not necessarily devising something new. Mathematicians have done this for a long time, but it’s educating a new population on what and how to do this differently. What validation means in this day and space and time is different than it was five years ago. So Tim, we had Talafukari from Paraxel speak at the ISPE annual meeting in Europe, and she said the industry is in pilotitis. She coined it as an affirmative disease that we have, that we constantly want to do these pilots, pilots, pilots, pilots. So how do we get from pilot to actual use.

[13:33] Speaker 2: That’s a great question.

[13:38] Speaker 1: You told me I can ask you anything, but you didn’t.

[13:43] Speaker 2: Give her any pre-planned questions. I love it. So I think I’m thinking about maybe a few people in the audience. I think we have a range of experience, meaning where are you at with chaos? And I think when you first start. I was doing this probably three years ago when we talked about the arc. Doing a lot of work with AWS, and all they were trying to do was POCs, and they actually funded it. So some people might have gone through that experience. Let’s do POCs, and it was for a lot of healthcare and higher ed. But what I found was that no one really had sort of this long-term plan. Was sort of like this shiny object. Let’s try something. Got to do something cool, and a lot of it was pretty hardcore stuff. It was with bioinformatics and genetics. Some of you probably have experienced that on the R and D side. So cool tools, but really couldn’t take it anywhere. It’s one POCF and the next, with no sort of traction, and a lot of the traction was lacking. Because it wasn’t bound in either process, but even more so, talking with all the people who actually do the work. I’ll give you another little story. More recent was talking with or working with a group. They use Jira, and if you’ve used Jira, it’s more of a development tool. They’ve switched to AI everywhere. Now that sounds great. For me, because I use Jeremiah, this would be cool to be able to do this. But in reality, for a whole team in an organization, when they switched over to the AI, no one knew what to do. How do I use this? They started slowing down their development. So I think stepping back, each company in your probably has a little different position where they’re at, whether they’re small or large. Some of the larger ones, you’ve got a lot of internal AI going on, and what I hear about that is like a thousand flowers. One person, I’m not going to say who said this, but I loved it. We’re in POC hell, and this is a company that’s really good at this stuff, and they’re in POC hell. Got some other companies, I think, just starting to use AI, but they’re worried about not going fast enough. But their applying process and development, and they’re taking it slow. So I think there’s a danger to that. So I think everyone is in this sort of transition point where, okay, where do we go? How do we do this? How do we get people to go along and use AI responsibly? Because what I’ve also seen, I’ve had a company where they were asking us, like, how do I stop people from just pressing the button? Like they use it so much, they keep pressing. Yes. Approve that. Approve. That was their biggest fear. So, getting people to engage was not hard. It was getting them to slow down and actually do their job and look at the results. That was the problem. And I think there are others that are just going light and fast, and it’s not focused. So I think hopefully, what you’ll get out of this conference is to share more information. And share stories, and I’m trying to give you as many today use cases and demand use cases, but also step back and look at what is your company trying to do and what is your responsibility. Just kind of back to where we’re at from a country and a world, we’ve got some things going on with where AI is going. That’s kind of scary, or very encouraging, depending on how you want to look at it. And I think where this team in here is special, we’ve been working in quality for a long time. Least I have, IT, regulatory, or in life sciences. These you all have the ability to actually assess AI in a way that Sam Altman. Anthropic are all asking for, so I think you’re in a great position. Plus, what we do impacts patients, so we’ve got a high responsibility. So I think we’re in a new space. I don’t think everything is as perfect as we expected with AI. It takes a little bit of work to step back, look at what your business is trying to do, and how you govern it, and how you get people to move along in the right direction without going too fast, but fast enough.

[18:41] Speaker 1: Thanks, Ken. So I’m going to move to some regulatory stuff around this too, because as one of the co-authors of the digital validation good practice guide, Gurav is on the team as well. We are constantly asked, ‘What’s next? Are you going to write something more?’ And I think part of some of the leadership is struggling with is writing too much in advance of knowing, and where do you come at that. For those of you that don’t know, Annex 22 is still in draft and still being considered. The original draft said no probabilistic use of AI and GXP. At the end of June there was a meeting to hear everyone’s comments, and we, as a committee and ISPs and others and PDA and PICS and all, said that’s not probably the good thing either. So where are the boundaries and how? How much can you use now? There is the worry of GXP use in AI. Does everyone, I’m assuming, have a bank account that you can get from your phone? That’s probably more worrisome and should be more worrisome to you than whether or not your drug has gone through many, many, many physical hands in order to get to you versus your money. Being accessible by a click of a button. The degree of risk and what risk means to you as an individual versus you as your team versus you as your organization and your place within the greater world matter. So we get down to people. One of the things Scrub we talked about in the Digital Validation Guide is around culture. How do we change? Get some of what Tim was saying to a culture of adopt and move, versus I got to jump all in, or it’s not good, because that’s what I hear. I hear is either we’re gonna put the brakes on completely, or maybe in a couple years when things look better. So how do we get to that little movement.

[21:01] Speaker 3: At a time? I think if you talk about your non-life science example about financial. I think the reality is that AI is obviously being used in many places, and as we all know, life science is usually not the first. And I can tell you that I belonged a couple years ago, some years ago, to the U.S. Chamber of Commerce, and there was about 70 professionals in government and industry that wrote the executive order on AI to President Biden. Specifically in that area, it had all the different branches of the government, and each branch would have its own chief AI officer. The idea was initially that they would demonstrate on a quarterly basis what the initial use cases are outside of the Department of Defense, of course, and I’ll share with you. But I think that’s where we start. We start traditionally at the beginning of my career. I always looked at what other industries were doing. If you look at the financial sector, if you look at aerospace and automotive. AI has been utilized some time ago. And now you’re starting to see cross pollination of automotive executives that utilize AI manufacturing of cars that have 300 variations of the same door handle, and so these concepts are being now applied, for example, in large medtech or medical device companies in the manufacturing. You can say AI is old. You can say CSA is not new. In the end, they leverage principles of utilization of SaaS solutions, of leveraging the vendor documentation, and this is all part of that cultural change. How am I going to make my process different in my company without having this jerky stop-start mechanism? In the end, yes, there is a lot of POC or pilot hell, and it’s really about the money spent in the return because everybody, executives, always want results, and of course, money is not free, especially these days. So with tariffs and restrictions, it becomes even more cumbersome. There needs to be a quicker return traditionally on the ROI. They want to see results. So again, the culture is about looking at under industries first, like if your ISP is one of them. If you look at the American Society Quality. They cover every industry that we have. It’s an interesting cross pollination of how different sectors are using AI. Ironically enough, they all have the same commonality of human inclusion today. It means that eventually, it’s my prediction that regulation will change. Eventually, when there’s confidence enough in our own processes, there’s regulation in place, and there’s guardrails. From a sustainability standpoint, if you come back to the executive order, it tries to provide an avenue. Today, in organizations, it’s still relevant. When we talk about GSP, not just FDA regulated. Financials, payment cards, GDPR, HIPAA. There’s a lot and the systems you have today are usually integrated with many other systems. Data itself becomes important. How is that data controlled? Whether you’re using a cloud solution or you’re more restrictive and keeping it internally, it starts again. If you go back to Doris’s comment in the different ISP groups, the community practice. AI is broken down into education, then the first, then the data science, then the validation, then the regulatory aspect. Today we have minimal regulation, and it’s competitive. Ultimately, the start-stop mechanism is to show quick wins to develop that culture. You don’t learn how to drive a car one day properly. And as you advance, you have those quick wins. Every company is different. Some want to try the low-risk administrative task. Some want to go for the bigger, gusto or GXP application to prove a harder use case. Eventually, you have to develop proper ROLs. The ROLs need to be established upfront, again, finish line in mind, and then you can scale up properly. I think this is how companies are evolving, both with combination of honestly from their own people. Their own quality culture, the where they feel their infrastructure and the budget, which is important, and how many years are you spending money to prove something, as well as the returns, and ultimately, the partnership really today I think most importantly needs to involve a combination of technology, strategic consultancy, as well as the industry, because everybody is in this game, no matter what seat you sit in or what position you’re in. That cumulative knowledge is beneficial, especially when you have organizations that work with many companies that get exposure from these conferences. You get to see the evolution of change by the month, by the quarter, by the year. Not just I have powerful lessons that you get out of it.

[26:42] Speaker 1: Thanks, Brett. If we went to you in terms of the so Annex 22 is on this fence between probabilistic yes. Know, in as you have a product yourself, right? One of the things that Anthropic CEO asked and called for was external review. We knew this was all happening. We knew regulation was going to come at some point in time. They’re opening up their doors to say, okay. So now, but you’re a software vendor. What? How do you think of that? Like Annex 22, do you plan for it? Do you say it’s not going to be applicable to you? What are your thoughts around how you develop so that you develop a product these folks can use?

[27:34] Speaker 4: Use? That’s a really good question. Of course, we can’t plan for the future too much. We never know what’s going to happen. So I think we largely focus mainly on non-GXP use cases for now, and I think that’s where a lot of the industry is at. Crafting things that require human review. I think one thing I want to put out there is there are a lot of studies you hear like 95% of AI pilots fail. Some of these you’ve already heard of, but a lot of those have a lot of holes in them. Like the way they’re measuring ROI, the sort of limited constraints they’re putting on. I’ve personally seen many, many successful implementations of AI agents. Of course, many things have to be in place, but I feel like there’s this idea that oh maybe it’s high or like we’re not really sure, or maybe it’s stochastic. Right, I think that’s pretty much debunked now. Like, it’s definitely doable. It’s been done. It’s been done several times before. There’s a bunch of challenges, but I also think there’s a lot of misinformation out there. And so, you know, I think if you are intelligent about where you start, if you have a clear idea of how you’re measuring ROI, it’s been done before. It’s not a theoretical can-do. It’s you know, it’s kind of been done across many organizations, and so. You know, coming with up that courage and being, hey, I don’t need to know everything to get started. I don’t need to be an expert to get started. I can get started where I am now, and that’s how I will learn and how I will make improvements. I think that has been a winning strategy. So we can’t plan for the future, but I think right now, you know, non-GSP, you know, low-risk use cases can still be high ROI. And so, yeah, I would encourage people to not fall prey to, you know, thinking it’s overly complex or hearing a lot of these statistics and getting discouraged because you know. It’s been done, and I think over time it’ll just become a standard. But that could be a biased perspective.

[29:22] Speaker 1: So Tim, also a vendor, how do you? Because we hear a lot about how to leverage, how to help one another. How do you guys think about that in terms of what’s next and what? What are folks not thinking about in the room that you all, with all this experience, should be able to help them get to the next step?

[29:48] Speaker 2: I’m not even sure if Sam Hall knows that, but say that. I think you know the technology is moving really fast when you read it in the news. And I think the technology is probably going very fast for some of the companies in here. Maybe just a step back. I think it’s about literacy. I hear that a lot. So how do you train people to use AI or understand how it works? I think that’s where we’re coming from as well: how do we help people understand how it works? Not that we have all the answers, but how do you also explore some of the problems that you’re experiencing and how do you measure and monitor AI? And I think that’s partially the trick as well. Where are we going? What are we trying to shoot for? We want things to be under control and safe. At the same time, enabling everyone in their work and their daily tasks, taking away some of those things that are mundane. That I don’t know about you, but reviewing validation paperwork is really boring, but it’s very important. What are the critical aspects? That we all need to look at, and that kind of goes back to having things that are risk based. So I think that’s where we can head. I don’t think we’ve all gotten there. Maybe some in the audience have, but that’s where I see us helping to lead this board. I’m not saying we’re the only ones. We’ve got a group here, and there’s a lot of you in here that I know who’ve been pushing limits. So I think it’s about sharing that. Where are we going to end up? I think we’re going to start seeing patients get better, better treatments, and products being more safe. I also think that we’ll find that our value in what we do is going to increase when we’re doing things that are more meaningful versus just the drudgery that’s been created over the years by ourselves and the regulators.

[32:14] Speaker 1: So we’ve made that mess. So I think that’s the other big message in here. I take full responsibility for being part of the original 30 years ago when 2016 CQ Part I Eleven came out and the overengineering that we did as an industry to get us to this place today. We absolutely have to reinvent how we’re doing validation in order to take on the new and how it’s going to be because it is transformative. One thing, Tim, that you mentioned, I wanted to see the hands. How many have had “quote unquote” AI training in your organization? How many have had cybersecurity training in your organization? Do you know what’s at bigger risk, cybersecurity or AI? Actually, not. Cybersecurity is your top risk. And if you’re actually doing your homework as validation engineers, you’d be adding more education in cybersecurity. Cybersecurity has more potential to ruin your brand or your organization from day one. AI is coming along for the ride because people.

[33:32] Speaker 2: Use it. That is, is cybersecurity a problem because of AI?

[33:40] Speaker 1: Has it been highlighted more because of AI, or has it always been there? These are the questions that one has to ponder when you do that. But then go back to the training that you’ve been given. How much of that is read and understood? How much of that is looking at a little screen with a little scenario to say what to do and what not to do? Do you even have a clue as to how that actually manifests? In your day-to-day, how mindlessly you click on your emails. There’s shadow IT. There’s now shadow AI, all over the place. And where and how that’s used and how that’s controlled is going to matter by the people that are in this room and getting educated in best practices of where actually to put controls or not put controls or how many controls. Because I think I saw Stephen Cook here somewhere. Our government, our controls. How many controls do you have, Stephen? 190. 190 controls. How is one individual, as you go through your day to day, supposed to keep track of 190 controls, security controls for systems? You can’t. You actually need AI to help you do that better. But how? And how to get to that is the struggle that we all have to do today. I don’t know how much time we have, and I’ve lost track of my notes. And then apparently you need glasses as well. Groff, if you had that magic wand, what would you want people to do? What’s one thing that you would want them to do differently as they leave today?

[35:34] Speaker 3: I think kind of going back to Tim’s point. I think one thing that people don’t talk about that much is they say human in the loop, but if you look at the last week, we have data that has now been published that looks at brain imaging scans of the human using AI and the difference of critical thinking and brain activity, and already in the early days. We’re showing a huge decline in where people stop thinking. Whether you’re pressing a button or you’re looking at a screen, like you look sometimes. I hate to say it, but most people don’t understand the procedures they’re signing for read and understood today. And the volume of documents that have to be approved or read in a week—that’s the reality, like it or not. Everybody knows it. AI. I think Annette’s point is very clear. When we look at pilots, it is the amount of time you need to spend to check what is deterministic, meaning the typical validation model, such as Part Eleven, that you know this the input, this the output. The more challenging part is what’s wrong with it, what is drifting or changing. So the dynamic validation model is continuous. The future state of what A B considers continuous change control, continuous validation, as well as continuous analysis of the data set. This is the shift, but it starts with the human, and today, based on regulation, it often ends with the human in the critical systems. It means that we have to make sure that those guardrails are in place, that somebody didn’t break something in the cyber. And all of a sudden, is accessing data that’s that’s proprietary or intellectual property or patient data from a clinical trial. So this becomes important. So, and that doesn’t matter whether you’re the you’re the hands-on technical operator or you’re the quality person or somewhere in the IT and infrastructure. I think organizationally, that is what we need to pay attention to because already, as you all know. Even if you use AI, I’ll give you a quick story. I went to a wedding two weeks ago. I asked the hotel, I asked AI if it had breakfast, and it said, “Yep.” Go to front desk. They have Grandville Continental breakfast. There was coffee there, but there was no food. So I asked the front desk, “What? What kind of breakfast is this?” And there was latency references to those artifacts within AI. So, the in the last few days, another study has come out that that starts to talk about the falsity of populating. Raw information into the human brain, and then starting to do it in masses, and suddenly something fake becomes reality, becomes a truth. So I think in the evolution we have to have our own checks and balances because in the end, today we’re signing. Whether it’s quality or the business or how many multitude of signatures you have, and again, traditionally, unfortunately, life. You have five signatures. Probably two of those are not even paying attention to the documents. Before that, so we need that additional layer of signature because we’re responsible for the products we’re making, the quality. I do agree. I think eventually we will have faster development of drugs, better delivery systems, and new pathways of innovation. But it starts with. We make mistakes along the way, and how detrimental they were. The problem is, we have a problem. Some of us make mistakes, and those are learning.

[39:34] Speaker 1: Curves. Not what would you leave? What would you want people to take away?

[39:39] Speaker 4: Today, I think coming from a tech background, I’d say I’m so impressed with the ways in which people can forecast risk and understand what can go wrong. But I think you do have to start with what if something goes right? Like throwaway AI. Don’t worry about that buzzword. What in your organization will actually make a huge difference to you? Like what results will really feel like you moved the milestone? So of course the risk is important, and this is a room where we all understand that the risk is important. My whole company that I work at is all about the risk. But I think we just start with forget about AI. Hey, if I was able to do this thing ten times faster Or just think ten times better. Start there. Get something that everyone is excited about, and then go into what’s the risk. First, you’ve got to find what reward is worth it, and then you can say the risk is too high, so we’re not going to do it. But you’ve got to start with a bit of an optimistic vision and idea of what sort of return would really get everyone motivated to work together. And of course, I agree on AI. I just love low risk, low hanging fruit. But I think we also can’t stop thinking big because that’s sort of the point. And then one more quick thing I’ll say is, a lot of people when they think about AI, every person along the chain thinks about how it can help them get 10% better. But I think you get more return when everyone comes together and says, how do we reinvent this process to make everyone like twice as good? Perspective in the ladder, but I think everyone needs to kind of come together and say, ‘what if we did this differently?’ As opposed to everybody using Copilot to get like, maybe 10% faster in responses. And so that sort of communication across the ladder, I think

[41:32] Speaker 1: Is very important.

[41:35] Speaker 2: What do you think? I like what you just said a lot. I think that’s key to get everyone involved. It’s not a single individual or single group. And then you know how do you align with ultimately making patient outcomes better? Because if you’re not here for that, you’re in the wrong industry. So I think that’s where if I could make you know wave my magic wand is how do we make our jobs more meaningful and more impactful? It’s not that we have done a bad job; we’ve done an excellent job. But how do we make it even better? Partner with AI. Because I don’t think we can escape that. So how do we partner? How do we help patients?

[42:24] Speaker 1: That’s great. I’ll leave you guys with this because I think it’s really important. And thanks for taking it down to the personal because it is personal. Each of you here is in the mix. So in Boston in June, the keynote speaker was asking who has a chronic illness. I was doing what I was doing on my phone and I was doing prep for my next meeting, not really paying attention to what he asked on stage, which was who’s living with a chronic illness. I raised my hand mindlessly, or so you thought, because when someone asked me if I live with a chronic illness, I was the only one in the room. Of six hundred people, and he came over to me, and I was like, “Huh?” I’m like, “What do you want me to do?” And he goes, “Oh, come up with me on stage.” And I was like, “Okay.” So get up on stage, and he goes, “So, what chronic illness you?” And I’m looking at six hundred people. I was not paying attention to this talk at all, and I said, “Oh, I’m bipolar.” And he goes, “Oh, so sorry.” I’m like, “Why?” You know, like that’s who I am. But I am a patient as well as I’m a human, and I’m in here and I help folks with. In this industry, because it matters to me, but we all are also patients that we should care about what we do every day, and to what extent and how do we do that matters. So the ten percent that Anot was just talking about, and you did that ten percent, and you did the ten percent, and you did the ten percent, and we all collectively moved to a north star, rather than the push and pull we often feel in our organizational life. Where you have your senior leaders saying, “I want money,” and manufacturing people want to go home on the weekend and not have to work a double shift because something went wrong on the manufacturing floor. Everyone matters. So how do you do that collectively? And part of that is the hope and the check. And if you do that incrementally, we can get to the next place. But we have to be willing. I love you. I’ll use my word, my reinvention, because that is what it takes in order to move to the next level. We are in a place in time where what you do every day will influence the next generation. This past summer, I had two interns from undergrad work with me. They came in and said we were not using AI. Everyone at school is against AI. These are graduating seniors out of a very prestigious school. They said, “No way, we’re not using AI.” And we had a lot of conversation about how come, why is the next generation so worried about it, and what do we need in order to take the good with the garnering of the base of what we’re doing. By the end of the ten weeks together, they were integrating it into their process because what we do in my organization, we do gamble walks. Even though we don’t have a manufacturing facility, we gamble walk our business life, and we make agents in order to help us through our life. That’s what we have the capacity to do today. We didn’t have that a couple of years ago. We didn’t have that even twelve months ago. So I encourage you all to reach across the aisle to those in your organization that might be reticent. That are in different groups that have different needs and find a common language, a common way to talk to one another about how you can progress in this way because it is here and we need to figure out how to do it together. So that’s what we’re going to talk about, and I hope you have a great rest of the workshop. We have some really great speakers today. Again, please make sure you have those conversations in the vendor space. Like Rob said, we all have a lot, a lot of experience, and it’s worthwhile picking brains while you have a plan. Okay, thank you, all.

Related

• Strategy Scaling and Product MOC
• Home
• AI pilots in GxP fail on sequencing, not technology - the POC-hell diagnosis from a different angle

---

← Gargano on continuous validation | Home | Gorani on human risk →