2026-09-16-1420-mvp-ai-gxp-markets-panel

Panel on MVP, AI, and GxP-regulated markets

Recorded 2026-09-16 at 1420 via Plaud. 43:27.

Summary

Quality and product leaders on what a minimum viable product means in GxP markets: moderator Bryan Ennis (head of quality at Salesforce), Courtney Steele (Director of Product Management, DeepHow), and Robin Stolzberg (Global Head of Quality, Sapio Sciences). The consensus: the MVP is the customer’s intended use, and AI changes none of it; vendors owe full functional testing while sponsors validate intended use in their own context of use, because a vendor cannot assess patient risk without it; risk assessment should be objective and tool-driven, since humans on the hook for outcomes are incentivized to dilute ratings; and compliance belongs in the product from inception (audit trails, roles and permissions, e-signature triggers with re-authentication and SSO) rather than retrofitted.

Steele described DeepHow’s route to GxP readiness through a separate validated platform with deep auditability (who changed what, when, in which time zone) and configurable AI instead of product forks; Stolzberg argued for quality-by-design from inception with just-in-time compliance guidance for product teams. Emerging MVP expectations named by the panel: AI-native interactions, embedded change management with automated change-boundary logs, headless architectures, accessibility documentation, and a drift toward continuous validation.

Plaud AI notes

<App-generated notes from the Plaud cloud, verbatim.>

Date Time: 2026-09-16 14:20:50 Location: [Insert Location] Interviewees: Bryan Ennis, Courtney Steele, Robin Stolzberg, and panel participants TL;DR A panel featuring quality and product leaders examines the definition of Minimum Viable Product (MVP) through the lens of “intended use,” the integration of AI, and navigating GXP-regulated markets. Consensus: AI does not change the core of MVP—deliver the end-user’s intended use—while regulated environments require rigorous auditability, practical Part 11 controls, human-in-the-loop AI, and early, continuous quality involvement. Vendors should perform comprehensive functional testing; sponsors must validate intended use. Risk-based approaches (CSA) are appealing but depend on vendor maturity and objective, tool-driven risk assessment. Emerging MVP expectations include AI, embedded change management, automated change boundary logs, accessibility, and headless architectures.

Introduction

1. Bryan Ennis (Moderator; Co-Founder & Chief Quality Officer at Sware, founder of Swear): Frames MVP as the product’s intended use—what customers will buy and adopt—unchanged by AI; extra features are secondary.
2. Courtney Steele (Director of Product Management, DeepHow): Shares the journey to GXP readiness via a separate validated platform, strict auditability (who/what/when changes), precise time zones, and configurable controls to adapt AI without forking. Delivered a first validation package for a pharma client, applying 21 CFR Part 11 (e-signatures, re-authentication, SSO) and human-in-the-loop AI; emphasizes embedding quality practices culturally.
3. Robin Stolzberg (Global Head of Quality, Sapio Sciences): Advocates quality-by-design from inception, robust QMS/SDLC, role segregation, training, SOPs, and audit readiness for vendors serving GXP customers; provides just-in-time guidance to align development with regulations and client expectations.
4. Additional panel voices:
   • Speakers 8/9/4: Stress that vendors should conduct full functional testing; sponsors validate context-specific intended use. Vendors cannot assess patient risk without context of use.
   • Speaker 2: Calls for objective, tool-driven risk assessments due to human bias diluting manual risk ratings.
   • Speaker 7 (Maya): Prefers evidence tailored to intended use and accountability over sheer test volume.
   • Speaker 10: Notes rising integration of agentic AI systems, shifting MVP expectations toward AI-centric interactions.

Key Points

1. MVP Defined by Intended Use: The MVP’s essence is delivering the customer’s core job-to-be-done; AI does not alter this foundation.
2. GXP Strategies: Regulated markets require audit trails, time zone accuracy, value-change tracking, roles/permissions, and compliant Part 11 implementations; build compliance into the product from the start to avoid costly retrofits.
3. Configurability Over Forking: Use configurable settings to enable/disable features for compliance, avoiding separate AI models or product forks.
4. Shared Validation Responsibility: Vendors must perform comprehensive functional testing; sponsors validate intended use and context, per ICH Q9 thinking.
5. Objective Risk Assessment: Adopt standardized, tool-driven risk frameworks to prevent watered-down, biased assessments; risk rationale should guide testing scope.
6. Early and Continuous Quality Integration: Embed quality professionals throughout the lifecycle to translate requirements, prevent late-stage remediation, and support audits.
7. Practical Part 11 Application: Implement e-signatures, re-authentication, and SSO at appropriate triggers; go beyond checkbox compliance with thoughtful controls.
8. Human-in-the-Loop AI: AI assists while operators make final judgments, ensuring integrity and traceability in regulated workflows.
9. Digitizing SOPs: Preserve authoritative text while improving usability under controlled processes with customer involvement.
10. Vendor Selection and Maturity: Evaluate QMS/SDLC, change management, CAPA, audit readiness, validation cadence, and segregation of duties; maturity enables risk-based reductions.
11. Evolving MVP Requirements: Expect AI integration, headless architectures, robust accessibility (VPAT), embedded change management (replacing PDF notes), automated change boundary logs, and movement toward continuous validation.

Insights

• Bryan Ennis: MVP is what customers will adopt; AI doesn’t redefine MVP’s intended use focus.
• Courtney Steele: Build GXP readiness via a separate validated platform, deep auditability, and configurable AI; live by governance, not just write it; engage quality early to reduce incidents and manage scope.
• Robin Stolzberg: Quality isn’t adversarial—embed it early; maintain strong QMS/SDLC and audit readiness; provide just-in-time compliance guidance to product teams and clients.
• Panel perspectives: Vendors should fully test functionality to prevent gaps when customers apply risk-based validation; sponsors own intended use evidence. Risk assessments must be objective; testing evidence should align to intended use and accountability. New technologies (agentic AI) and modern architectures shift validation expectations.

Chapter Highlights

1. Defining MVP: The end experience customers value and adopt; intended use remains the anchor.
2. AI and MVP: AI does not change MVP’s definition; focus remains on intended use and job-to-be-done.
3. Becoming GXP-Ready: DeepHow created a separate isolated platform; intensified auditability (actor, previous/new values, time zones); configured AI for regulated use rather than forking; benefits flowed back to the standard product.
4. Quality at Project Inception: Include audit trails, approvals, roles, permissions from the start; avoid late paper-based workarounds.
5. Practical Regulation Application: Part 11 controls (e-signatures, re-auth, SSO) at sensible trigger points; AI-assisted verification with human confirmation; digitize SOPs under controlled processes.
6. Validation Roles and Risk: Vendors do full functional testing; sponsors validate intended use; risk assessments should be objective and tool-driven to avoid bias.
7. Vendor Maturity and Risk-Based Testing: Comfort with reduced testing depends on vendor risk management maturity and software/process maturity; customers often still expect CSV-like evidence.
8. Emerging MVP Expectations: Embedded change management, automated change boundary logs, continuous validation, AI/agentic systems, and headless architectures will reshape validation burdens and evidence requirements.
9. Takeaways: Build quality in early; earn the right to stay in regulated spaces by effectively implementing quality processes; move MVP narratives from what to why to support critical thinking.

Highlights (button-flagged moments)

risk needs to be objective. premise of AI in risk assessment is that people are not good at risk assessment if they are on the hook for the outcomes of that assessment, they would be incentivized to dilute the risk assessment.

• Key Insight: Manual, individual-led risk assessments are often diluted because people are reluctant to personally own the identified risks, leading to ineffective results. • Risk Identified: If a vendor’s risk-based testing approach is misaligned with the client’s needs, the validation burden and cost for the client significantly increases. • Decision: The consensus is that vendors should perform comprehensive testing, providing a robust foundation for the client to then apply their own risk-based approach for their specific implementation. • Key Insight: A vendor cannot fully assess patient safety risk because it is critically dependent on the client’s specific ‘context of use’, which is unknown to the vendor.

Transcript

[00:00] Speaker 1: Feel free to come forward because my co-panelists here are well, they’re sitting over here talking to Leonard. This so we’ve got product, we’ve got quality. So this either going to be a cage match between the two of them, or we’re going to solve world peace. But the big topic today is to talk about minimum viable product, and you know really kind of how to navigate through that journey. I’m Bryan Ennis. Nice to meet you guys. I’m the founder and chief quality officer at Swear. I’m also the head of quality for Salesforce. I’m excited to get a chance to be with you guys today. This is awesome. So, a couple things, MVP. What does it mean to you? I want to start there before I can define it. Right? I’m a software company, but what do you guys think of when you think of an MVP, minimum viable product?

[00:47] Speaker 2: What is that? What you produce the end experience so your customer can look at it.

[00:52] Speaker 1: Okay, that’s a that’s a good start. That’s a good start. I like that. Also, step three in the NBA. Separate is amazing, by the way. I watched a little video. Really? I watched a few videos of him. It’s just like a little. That’s a little no man’s land for me. Other thought, Mon, I’m gonna pick on you. What’s your MVP to you? Uh, was probably the case that you picked. Prologue. Okay, that’s a good one. Most like the most likely one a customer is gonna buy, or is gonna use, or a user is gonna adopt, right? Um. When I think about MVP and like how we like in in launching a company or launching a product and going to market, right? There’s a lot of analysis you do around making sure that you are serving customers to the job that needs to get done, right? Like you have your software’s got intended use. We use that term in validation all the time. It’s a good term. That intended use is really what I think about when I think about an MVP. There’s a lot of other bells and whistles and features you’re going to add later. There’s a lot of other optionality and ways to expand, but the intended use piece is. Is really important. MVP. We’ve been building SaaS software for a long time. How do you think things are changing right now as we move into AI? When you think about the MVP, like there, we’ve been building software, but now we’re starting to integrate AI components. Is that going to change the definition of MVP? Do you think? No. Oh, it’s a quick question. You’re right. So, hi, front, front, center. So I think that’s also, you know, even including AI, even as it’s used with its usage, right? Like, it doesn’t change the fact that we are targeting intended use. Stephen Farrell’s presentation, shout out to Valket from before, you know, he talked and put a lot of sense into this as well, right? Like, whether we’re trying to talk about AI or we’re talking about putting in a SaaS application or an internal application, it’s gross, but I’m sorry, I can’t speak with this with this pen into my mouth. I think it’s important for us to consider that contented use around MVP. Like those are two things we’re going to equate in this conversation. So I’d like my co - panelists to introduce themselves here. If you don’t mind introducing yourself, and as a seed question, what matters to you in your role to make sure that product and your company is GXP ready? Courtney,

[03:15] Speaker 3: Will start here. Sure. Hi, I’m Courtney Steele, Director of Product Management for Depau. EPAL is a workforce readiness, physical AI for operational excellence software. We’ve been in the manufacturing space since we went to market back in twenty twenty, and now we’re getting into the regulated space as we start working with life sciences and pharmaceutical companies. So, what does that mean as far as you know what it means to be GXP ready? Well, you know, as part of my role as director of product management, I own our roadmap for our product. Now that we are starting on a more regulated version of our product, we actually split it out into its a separate platform because we wanted to make sure that we could isolate what needs to be GXP compliant versus what our more standard manufacturing plants need. So I really spent half my time now focused on how can I nurture the validated side of our product. I’m finding that’s actually benefiting our product as a whole, and I’m really seeing that the work that we’re doing to improve Deepow to support life sciences has just made such a positive impact on our more standard use. The one thing that’s been tricky about it is just really understanding all of the nitty gritty bitty pieces that you have to be thoughtful about when you want to claim GXP readiness. So previously, we didn’t have to. Work very hard or think as intently about how we display things as simple as a time zone, or if there was a change in our system, who the actor was who made the change, what the value was previously, and what the value is now. So there’s a lot more that we are doing to make sure that things are auditable, things are trackable, that we can see what we did before, and then when you factor AI into that, and AI has its own special breed of, Ambiguity and uncertainty, and you know, sometimes you think it’s going to go X direction, but maybe it goes Y. So being able to account for that, when Brian spoke about intended use, intended use really dictates to us, you know, what we need to prioritize in terms of of how we define, you know, the URS and the specifications around that. One thing that’s really helped us is to think about configurable settings. So. The configuration spec is part of the story, rather than forking the AI that we build. So we’re building two different AIs. We don’t want to do that. We’re more focused on how can we enable and disable settings to make it appropriate for a DXP focus environment.

[06:06] Speaker 1: Awesome

[06:06] Speaker 4: Robin.

[06:08] Speaker 1: What was the question? Introduce yourself, and then. It was the question was what matters to you in your specific role to ensure that your company and your product are going to get to be ready.

[06:18] Speaker 5: I was so intent on listening to Courtney. Hi, I’m Robin Stolzberg. I started off my career in the pharma industry more than 35 years ago. I graduated college with a bachelor of science degree, and I started as a bench analyst in a little old - fashioned microbiology lab. If you guys remember the old microbiology, sterility testing, endotoxin testing, all that that fun stuff. And I’ve been blessed in my life to pivot a few times in my career. So I have at least 13 years of bench experience. Then I pivoted to a quality role. Then I pivoted to computer system validation quality engineer role. And now I’m actually the global head of quality for Sapior Sciences, which is a labware lens company. So that’s a little bit about me, and I have to throw this into everything. I do have one child. He’s 24. He graduated U.N.C. Woo, go Tar Heels! Awesome. So what is important to me for GXP and for my company for GXP is to understand what my clients need. And I’ve been since I’ve been in their space. I feel like I have a grasp on that. But for GXP, we’re not a GXP company. We don’t run under GXP. We don’t make drugs or anything. What our clients do. And so for me, what’s important if you’re for GXP company for software, if you’re looking for software, is to make sure that company is GXP like for you guys. Make sure that they have their. Quality management system in place. Make sure they have an SDLC process in place. Not only an SDLC process. Make sure they have quality oversight of that SDLC process. Make sure they have their training matrix in place and and all that all the good stuff. So for me, GXP is really for you guys in my current role and making sure that we’re ready to support you.

[08:30] Speaker 1: What skills, Robin, do you think that you needed to develop, or maybe you were forced to develop along this journey when it came to managing the GXP status or compliance status, or or support, right, of of your product? Like you and your role, what are some of the things that you had to learn along.

[08:46] Speaker 5: The way? There’s a big one. It’s translation, and I only speak English. My translation is translating the with the bench analysts their speak to, QA speak to developer speak to CSB speak, and I think that’s the big one. Yeah, yeah, and and making sure that I can explain the bench analyst needs to be able to have a GXP system to make sure my developers understand what that GXP system is and why it’s important. I think that’s the biggest part is explaining to each one why the pieces are important, so everybody understands because. The more you understand, the more easier it is to pull all the pieces together.

[09:35] Speaker 1: So, about you, Courtney. This is a recent journey for you into the world of GXP. I mean, what kind of skills have you had to develop in your role?

[09:43] Speaker 3: It is very recent and very topical. We literally just handed off our first validation package to our first customer, who’s taking on our FarmCloud version of our software. And I agree with. Robin, about the language piece, actually wrote my notes. Learning an entirely new language, or at least a set of vocabulary. There’s so much to unpack, and the last thing I wanted to do was feel completely dependent on everyone else in the room. I wanted to make sure that, and I knew that I would be accountable for defending the decisions that we made along the way. And so, making sure that I, in my role, playing kind of this, you know, very middle seat. Surrounded by developers, we brought in our own quality experts. We we worked with the Swear team. We have the quality experts on the customer side who are involved. Some of which are contractors and not even on their own payroll. To you know really evaluate whether or not we managed to accomplish what we set out to do when we wanted to build a validated version of Deepak. And so there is a lot of being comfortable, being uncomfortable. And jumping into calls and looking things up as I’m listening, being open to thinking about things in an entirely new way, and reading a lot of regulations, like 21 CFR Part 11. How bright that!

[11:13] Speaker 1: Welcome to the club. Thank you for your service.

[11:15] Speaker 3: Thank you. And then taking those regulations and applying it in a very practical way to the software. So you know, we can we can say yes, we have an e - signature. In our platform, but we also needed to make sure that it, you know, was up to snuff with what you consider to be a GXP, you know, ready e-signature. And that’s making sure there’s a re-authentication method. And that’s making sure it can integrate with your customer’s single sign-on. And that’s making sure that it’s happening at appropriate times in the platform. We’ve heard a lot this over the last two days about Human in the Loop. And there’s a number of things in our software where we’re doing, you know, photo verification. So an operator is maybe setting up a station. They snap a photo. Our AI has been trained on a reference of what a good station setup looks like and what a bad station setup looks like. Our AI is going to tell the operator, “Hey, you’ve met two of three criteria, but something’s off. Maybe something’s missing. Maybe there’s something that is there that shouldn’t be.” And in order to confirm, you know, whether or not the operator did that work correctly, the operator has to make a judgment. It’s not the AI making the judgment. It’s the operator reacting to the analysis from the AI and then returning a yeah, no, I’m good with it. I’m going to submit anyway, or let me try again. We adjust what I did and then resubmit and see what happens. And then when we talk about the so that’s the human in the loop step for the photo verification piece. Another thing that we talked about was, hey, you have AI that maybe touching information. So one thing that our system does is, you know, we’re not an MES. We work well with MES. We are really more about you know digitizing knowledge and digitizing know-how. So one one example of something we’re doing with our our very first pharmaceutical customer is digitizing hundreds of pages. You know, sometimes a single SOP can be a hundred plus pages long. Taking that and digitizing it in a way that is friendly to the operator, pull it up on their smartphone, directly within the Deepow app, and then scroll through it as they’re performing the process. But while we’re using an agent to actually generate that, we want it to rebuild all the tables. We want it to look nice. We want the formatting to make sense. We don’t want to change. The text we know and appreciate how much work has gone into that SOP. The last thing we want to do is to, you know, have an AI go in and generate something new simply because it felt like there wasn’t enough in the base SOP. And so we’ve had to introduce a lot of controls and really go through a lot of iterations, sometimes with the customer’s involvement, to make sure that is this, you know, is this meeting your expectations or is this appropriate for you know what you’re using P.

[14:22] Speaker 1: Cool. And I think there’s certainly like on the MVP journey, right? The customer is a big part of it. We’re going to unpack a lot of customer expectations in a couple minutes. But before we get there, so Robin, you talked about like the different languages that we all speak. What do you wish product people knew about quality people? And accordingly, I’m asking some questions afterwards.

[14:42] Speaker 5: Well, that’s simple. Is that we don’t mean to be scary. Quality is is. If it’s built in at the start and moving forward, there’s not going to be any issues with your quality department. You start the project, you bring quality in, and at the start you’re building in your workflows with your approval process. At the start you’re building in where those signatures are. We all know the predicate rules. If it happened on paper here, it has to happen in the system here, right? And so, what I wish, and I was quality engineer. And operations-owned computer systems. This is back when I was working for a CRO, and they’d start their validation and then do all the stuff, and then come speak to us. And then, well, your order trail is not exactly where it needs to be. Now we have to put something in around that. How many people in this room had a built-in paper processes for an order trail? Anybody? Because I’ve I’ve had to wind up doing that, and it’s horrible to keep a logbook next to a computer system that’s supposed to to work for you. And so, really, if you, my my answer to that is quality at the start. Put them at the same table at project kickoff, or even further back when you’re deciding on your your product. I think they stopped inviting me to that those decisions. Because I’m asking about audit trails, I’m asking about user roles, I’m asking about permission settings — all part of the things that make it compliant. Right. And they didn’t — they wanted — they liked what it did. They didn’t care about that stuff, and I think they stopped inviting. Keep quality in at the beginning. Help let them help make your decision. That’s what I’d like product to know.

[16:31] Speaker 1: Awesome, Courtney. What do you think? What do you like to know about the product.

[16:35] Speaker 3: That we’re doing our best. We really are. You know, I’m fortunate in that my organization recognized the the necessity of having quality involved quarterly. And honestly, I mean, a lot of it’s because this is totally new for us. We weren’t going to try and pretend that we understood the process. We weren’t going to try and you know and be big and fancy and be like, no, we don’t need help. We can figure this out. Or I can just go to Claude and ask Claude to tell me what to do. We knew that the best thing we could do at the onset of this process was to include quality in the room, and to ask the quality experts we were working with to be as tough on us as possible, because they were friendlies. They were on our side, and if they could be tough on us, and we can pass whatever test they’re going to put in front of us, then we stand a chance when we go sit with the customer and their quality people as they review the documentation that we put together. So. I feel fortunate that you know we were able to get the team aligned early. But when I think about what I’d like quality to know, is that I, or at least, and I can only speak to my experience, but I can imagine that for a lot of product leaders who are getting involved in GXT for the first time, they they might think they know what they’re doing, and they might think they know what they need to do to make their product GXT ready. But there are so many little nitty gritty things that were, you know, that we we found out about along the way. Like we’re we’re involved in doing some, you know, test cases—not the final test cases, but some test test cases—and we’re discovering little pieces that we needed to iterate and improve on. The earlier we can figure this out in the process, the better, because it means that you know we’re having to produce fewer incident reports. It means that. We have more time to solve these problems and more resources to put against it. Because not only are we also validating our own system that already exists, but we’re also getting new scope from this customer who’s very excited to work with us. And you know, what I can imagine is for a lot of life sciences customers, if they’re working in validating an AI product for the first time, there’s a lot of enthusiasm there. They want to push the boundaries what an AI software can do. They’re going to challenge AI software vendors to do things that aren’t involved in our original use case, and so you know I think that was the trickiest part was balancing that. So if there’s anything to wrap it all up, if there’s anything that I would like quality to know, it is just tell me everything you think I might know, and just tell me anyway. Just make sure I know for a fact. Don’t assume I know anything. Don’t assume that I know about the way that time zone should be displayed. Or the way that that when I see a username in the platform, the way it should be displayed. The the best thing a quality expert can do is just treat the product person like it’s quality for dummies. That’s really what I needed.

[19:53] Speaker 1: And you talked a lot about what you’d like to see in the products, right, for testing patience features. Robin, quality or being judicious is also what’s around the products. Absolutely. So I mean, like when. If you’re putting yourself in the seat of the customer, if you look at a company who’s got a MVP product, or if you look at even a company that’s building their own MVP product for a use case, what does good look like for what surrounds it?

[20:19] Speaker 5: Okay, so I’m not going to talk about a product now. I’m going to be sitting in your shoes, looking for a SaaS company. Good. We’re cool. So the software you love it. Going to you’re going to if you decide to use the software. You are going to be married to this company for good five years. It’s you know what typical five to seven years, and you move on to a different system. It’s something along those lines, or maybe not. But look at what because you’re getting into a relationship with this company, you want to make sure that they can support you year after year, right? So you want to make sure that they have this great they have this great product. There’s a validation package. How often is that validation package repeated? What is part of that repetition? Do they give it to me? Do they do IQs? Do they have an SDLC process with a stage gate that’s being reviewed? You know where they get it. How they’re handling their tickets? Do they have somebody changing the code and also testing their code and also approving it, or do they have different people set into that process because you shouldn’t change the code? And be approving that change. Do they have change management for their system? You know, what does that look like? If there’s cap is to go along with it, you know, you want to make sure that that company is a company that you want to be partnered with. We probably all heard of CSA. They talked about it downstairs. All this kind of stuff. Part of that is finding a vendor that. Would stand up to an FDA audit if they had to, because I don’t know if a lot of you know this, but there’s there’s an annex. I think it’s no, I think it’s seventeen appendix one or something like that. You can look it up. That says if the FDA or regulator comes to your site and you can’t answer their survey questions, they have every right to go to their vendor and ask them, and so. Your vendor needs to have all these things in place to ensure that they can answer all those questions. You know, how do they? How are they controlling their documents? Where are they controlling their documents? Who’s executing things? What’s their training process? Do they have SOPs in place? Are they trained on their SOPs? Are there work instructions? All these type of things. No pressure. No, no, no, no pressure at all. Just make sure they have it all.

[22:57] Speaker 1: Robin, you mentioned CSA. Can, Can a software vendor, and let’s say this could be a software vendor, but also could be a group within a company with their own CSP building a tool, right? Can you actually practice CSA in truth? Are you really? We’re looking for our software vendors to have more of a CSP approach to things, where perhaps is more comprehensive, more functionally oriented.

[23:20] Speaker 5: What are your thoughts on that? I have lots of thoughts on that, and I will jump into my role for Sapio for this. Is that I do not. Allows Sapiens’ informatics platform to pull in CSA. We create the product. Who are we going to leverage? Where where are we going to get it from? I mean, we don’t have okay this vendor here and this vendor here and this vendor here, and we put it all together. We created the platform. We update the foundation. We do all of that work. So there are you know we have the sales team, we have marketing, all that kind of stuff. None of it. Says that we use CSA for our informatics platform. Now, we I use it for my QMS. I use it for SWEAR, so I leverage these things and pull it in. But I don’t for my company because I have nobody to leverage it on. So, and I make sure that their presentations, the marketing, sales, they all know that. But I’m going to just do one more. I’ve made sure and. You should talk with the quality people of the systems that you have set up. I’ve made sure that we can meet the CSA expectations for you guys, and that’s a question that you should ask your CSA people when you’re looking at these. Can what do you like? Can you meet the regular? Can you meet what CSA needs to give me so I can leverage you?

[24:52] Speaker 1: So one part of CSA is the ability to or the incentive to take credit. For vendor audits, right. But another part of it is optionality and how we test things, right, based on a a risk adjusted approach. I mean, you guys in the studio audience, do you think your software vendors that you’re purchasing solutions from can really be aggressively risk oriented in how they test, or do you expect them to have something detailed in their package, like a CSV, a more traditional CSV approach.

[25:21] Speaker 2: I’m gonna pass this to anybody.

[25:24] Speaker 1: A just be loud.

[25:26] Speaker 2: Okay, so I’m with Chris Cortex, an agent AI solution, and it is based on you can pull in CSA, you can pull in CSV. CSA is only appropriate for manufacturing quality testing for medical devices. As of last September, a year ago, four years ago was the draft that hasn’t gone anywhere because some people at FDA are convinced it’s going to cause all kinds of problems and recalls and complaints and everything else if people are allowed to do this. Approach where you have to do the interpretation. So a lot of these companies out there are saying, you know, we’re not going to touch CSA until we need to. But FDA isn’t even requiring. I think it was a pilot sort of that they, sort of a trial balloon they threw out from the device to see if, in fact, inspections are clean, and then maybe it’ll go out to the rest of the. Well,

[26:16] Speaker 1: If I want to get up on my career soapbox, I could say it’s just good camp that I was doing in 2005, right, which is just a good risk - based approach, right. So. We frame the question differently: Is how aggressive of a risk-based approach are you comfortable with your software vendors taking, and how they demarcate the amount of testing they do and the type of testing they do? Do you want them to do more comprehensive testing and give you more evidence, so you just like can just have volume, or are you looking for more critical thinking and more rationale in terms of how they rank? The different capabilities of their organization, of their features or products, higher medium or low risk.

[27:00] Speaker 6: Mike

[27:00] Speaker 1: So oh, sorry

[27:01] Speaker 6: Please. So for me, it kind of depends on their maturity of risk management, right? In order for them to kind of leverage that testing and do add - on or or some level of testing, I would want to make sure that they have that evaluation in place, right? Mm - hmm. But I would be I wouldn’t be opposed to some level of that. Like you know, D testing and verification, but if you have a mature program for that, I would evaluate for sure.

[27:28] Speaker 5: And how mature their software is too, and their company is, and how they develop that software.

[27:34] Speaker 1: Absolutely, which is a tough question to answer with a company MVP sometimes, right? Sometimes an organization not really early. Maya, your your thoughts?

[27:42] Speaker 7: Things are rather than just trying to do raw volume of script output. I think it really helps. I think it, Not the evidence to the injured use, and thus the accountability, right? And that that jury judge will be offered as long as that contention is met.

[28:03] Speaker 8: Stephen, right? Andy, sorry.

[28:06] Speaker 1: Andy, sorry

[28:07] Speaker 8: Sorry. And my counter question would be: We follow ICH Q Nine, and it says patient safety is of prime importance. How can a vendor possibly, Assess the risk of harm to patient, because the context of use is key, right? You use software, right? Some softwares, you could use a labware, you know, like a a limb software, and you could use it in an R&D lab, you could use it in a clinical setting, you could use it in a manufacturing setting. Context of use is is super important. So, whether or not, in that case, you know, take a full risk-based approach.

[28:49] Speaker 1: That’s yet another acronym to add to the knowledge base here.

[28:53] Speaker 9: What I’m saying? Yeah. If you’re talking about the product developers, I’d still expect them to do full testing for the same reasons, right? Because they cannot justify it or they cannot control the intended use for a given client. And with the push from the industry to rely more on the vendor deliverables, we’d feel more comfortable if vendors can demonstrate they’ve done thorough testing and they have good STLC in place. Yeah.

[29:18] Speaker 4: Similarly, I suppose the the approach is that if a customer were to perform CSA and disagree with the vendor on where there was risk, and then find that the vendor didn’t test to the same degree of risk area that they did, now the customer has more validation burden on them. Not only to prove their intended use, but the additional functionality testing the vendor didn’t prove because of their. CSA approach. So, the the vendor generally has full testing without risk based approach, so that the customer can take a risk based approach and never find a gap. That doesn’t keep me up at night.

[30:02] Speaker 1: At all. Yeah, the couple great best practices that emerged from you guys just, which I think we’re all spiritually aligned to, right? You know, one is that. When we think about the intended use of an application, it’s our responsibility as sponsors to be able to demonstrate that to an auditor inspector, right? The good role of the software company is to have comprehensive testing around like what happens when I click this button. The role of the sponsor is why would I click this button in the first place, right? And what is that actually going to do? That’s the story in that final mile of a delta validation that we usually talk about, right? But you know, part of it too, I think, in best practices of understanding how to frame an MVP. Is the acknowledgement that your validation evidence is likely to grow and change as customers get their hands on it, as I’m sure you both have experienced in real world, right? Your customers finally get to look at you know they look at the software and they’re like, oh, I have forty five thousand more features I want you to add, right? And but they’re also going to get into the validation evidence and say, hey, if you tested this, it’d be really nice for me. I wouldn’t have to test it, right? To your point. I think that’s that is also something to acknowledge as you know part of the open dialogue as as when you’re building a product or bringing an early stage product to market is knowing that that change is going to happen. Presenting your your validation approach as complete and perfect is likely just to get you trouble, right? Because auditors like to find holes in.

[31:23] Speaker 2: Things. It seems like everybody’s kind of talking around risk. Who does risk assessment? And I’ve said this before that. Risk assessment done manually, people water down results because they don’t want to own individual risk. So you’re mean, you have meaningless output. But if you’re looking at some of these tools, I mean, that’s the beauty of Risk Cortex.

[31:45] Speaker 1: I’m going to ask you not to give a product pitch. No, you got one, but you don’t get two. Okay.

[31:51] Speaker 2: Well, the idea is that risk has to be done objectively, and you have to find a tool that can do that, where people can look at it and come to a consensus. Otherwise, you’re leaving people to throw this hot potato around, and if you don’t define risk properly, you can forget about what test scripts you’re going to do or what testing because you’re not basing it on the right level of risk. And so, it’s it’s really incumbent on the people using the system to understand how the risk is being developed, and agree with that before you go on to doing any testing.

[32:27] Speaker 1: Yeah, I think it’s a sensible point too, and we there’s a whole separate session about how we’re all running around with different risk assessment templates from each other right now, right? Many of which we’ve used for the last decade. But anyway, Courtney, just I’m gonna I’m gonna come back around to just what we’re talking about. Like, if you, one of the things that you both talked about was like getting involved early, matters, right? If you were to start your next MVP, say a new product for your organization, and you guys are gonna get started, like. Courtney, where would be the first place that you’d plug in quality? At what moment would you start to educate them, and how do you do it?

[33:04] Speaker 3: Oh gosh, what a great question. I mean, I think for me, it’s. I really, Caroline, right? Yes. I really liked what you were saying about the objectivity of it, because certainly we’re biased, right? You work really hard on developing a product. You spend a lot of time on it. It’s your baby, and so you have a lot of feelings about it. But when you’re doing something like a validation approach, you you have to take all those feelings and put them aside, and you have to become very kind of ruthless about your approach. I think one thing I would do sooner is as soon as we understand, like we’re already planning our next release, we already have a sense of what the scope of that release. And I literally have a call next week with your team, and one thing that I want to focus on in that call is to just go through the scope of what I anticipate the change. To be and to start talking about the change impact and to start really thinking through, you know, what does this mean for us in terms of, you know, what level of effort we need to put into this next like round of validation that we will do. How can we think about this more strategically? It feels, you know, the first time we did it, we were also producing all of our governance and documentation and all of all of these materials that we just didn’t have before. Now those materials are established. So we have some processes we can rely on. Yes, it’s also our responsibility to implement those effectively too. So I don’t think there’s ever a time when I would say, ” Hey, quality, you’re done. You can you can walk away now.” I think they become an embedded part of the team.

[34:39] Speaker 1: And Robin, from your perspective, so product has pulled you in. They’re like, ” Hey, we’re going to build this thing. Here’s what we’re going to do.” But how do you help them get started in understanding what qualities we need from them? Is it a checklist that you’re going to give them? Is it, Guided interview series. Do you want to be embedded in meetings? What exactly? How do you onboard into product.

[34:59] Speaker 5: I am a resource for them.

[35:04] Speaker 1: So as they’re building, as they’re thinking about doing things, you know, if they come, like, there’s a, they might put in a new NGS module. Same. They might. You know, I don’t know the particulars about NGS. They’re going to build that.

[35:21] Speaker 5: But they may come to me and say, “Hey, we’re going to put the we’re going to create approval processes. You know, we’re going to test these workflows. Where, what do we need to do for that?” And so I I really am a 24/7 so to speak resource for the team at all times. So I’m just I’m not embedded per se, but I’m always available for them. So when they need to know something. And that’s a constant. I’ll get something almost daily, Monday through Friday, and maybe quite a few weekends, from the team asking stuff. Or we have the client. We have clients again, going back to this role, who’ve got questions. They don’t. They might say, “Well, you know, this is how I interpret Part 11. You know, I don’t think your system’s doing that.” And I’ve made sure our system does it. I have a. All the pieces in place. All the testing’s been done. All the functionality’s been tested. So then it’s my my job to get on with the client, and we have that conversation. You know what what’s your concern? Here’s what we’ve done. What isn’t meeting it? And you know, come together with product. There, they hear the answer, and they’ll they can take that if there’s something that needs to be done, and fix it.

[36:45] Speaker 1: That so the just-in-time knowledge management I think is really important for products, right? Like there’s no such thing as dumb questions, right? Or everyone needs a dumb question person, right? Square is a little different, but I could say for for Salesforce, for example, like every single product manager in Salesforce Life Sciences has a dedicated Slack channel with access to me and my team and other quality SMEs within Salesforce, and we’re all in the same thread. They can ask a question, and they might get a whole bunch of opinionated answers, but the point is, is they get an answer within 30 minutes. Versus having to wait around to figure something out or to build something wrong. So this is nice because we get to talk to an expert audience, and I know most of you, so you are. So you’re not going to escape this one. What do you think is going to emerge as new MVP requirements as we look for new applications that come into our enterprises? Like, it’s going to have to be headless, for example, right? That’s going to change the scope of how we look at applications and test them, right? That we might say headless is probably a good one. Perhaps it needs multilingual translation or VPat for accessibility. What do you guys think? What is when you think about the applications that are coming out now, or the applications you’re going to bring into your environment? And you think about like early stage movers. Like, is it the same paradigm as before, or are you going to start to see, are we going to start to see other things emerge as part of their architecture or their delivery? Does that question make sense?

[38:05] Speaker 10: Yeah, I’m definitely seeing things like MCP servers show up and. In vendor products that I’ve done evaluations for, and and and that has I can see the the trend that way. I guess that people want to interact with vendor systems through things agentic systems like Claude, Claude and Codex and that sort of stuff. Yeah, so I can see that being a trend.

[38:32] Speaker 1: AI is kind of like the obvi. I mean, because you can’t fund a software company without having AI, right? Yeah. Other other thoughts, other things you think emerging that you would think that an MVP would need to have to be able to pass the the smell test.

[38:49] Speaker 4: Invented change management because people are tired of PDF release notes.

[38:54] Speaker 1: Oh, that’s a great one. That’s a good one. I love that. Yeah, we’re just as bad as release notes as we are as risk assessment templates. So, other thoughts.

[39:08] Speaker 7: Same lines. I think the automated change boundary logs, the verity moving the continuous validation from periodic reviews.

[39:17] Speaker 1: Yep. Yeah, I think it’s an interesting question for us to think about because for all of us. So I’ve been in the industry long enough to go from build it to buy it to build it to buy it. Now we’re back to building it again, right? Where everyone’s vibe coding applications and figuring out what they can do in their current toolsets. Certainly, that you need a quality around that as you build those tools. But as we architect new tools, the validation burden’s going to change, right? MCPs, not API, right? Like what evidence is about is your vendors going to be able to provide and tell that story to you, right? Or their AI features and how they’re using it, right? I think the expectations that we’re going to put on vendors as they come to market, the newer solutions we’re going to see are not going to be the same as the solutions we saw five years ago, right? So, we’re going to have to start to think about what does that mean for us as quality professionals. So I’m looking at the clock here. Just the very last question for Robin and for Courtney. What are your top takeaways for everybody? Your like top things you want them to walk away from this. Build.

[40:16] Speaker 5: Quality in early. That’s my top takeaway. If you do that, your your I don’t want to say everything will go smoothly, but it’ll go more smoothly than if you.

[40:32] Speaker 3: Don’t. I think for me, it’s that. And and I think this is something that is going to make or break us. Frankly, is you know it’s not enough for just me or the people who are immediately within my vicinity in the organization to understand the governance and the new like way of working. So you know it’s one thing that we wrote all these governance documentation and change control management approaches and you know like wrote them, but now we have to live and breathe and do and die by them. And that’s and I think that’s a more critical hurdle for us. So, just really challenging us from an organization-wide perspective of hey, if we if we want to play in this space, we have to earn the right to stay in it. Yeah. And so I would I would challenge any vendor who wants to be in this space to really think through. You know, I’m sure you collected signatures. You pushed people to sit in a call for two hours while you you know review the documents. But how effective is the implementation of this new way of working?

[41:40] Speaker 1: It’s a fair way because you’re going to get audited over time. Though it feels like a lot of work in the moment, implementation or the first validation of product is it’s only two percent of the work. Another ninety - eight percent is running it over time,

[41:51] Speaker 2: Managing all changes. I think the MVP too has to move away from just telling you what the product does, what the function is. It’s got to tell you why. Because as people start to realize critical thinking is needed, and they start doing that, they’re going to be questioning, and it’s got to be obvious.

[42:10] Speaker 1: Yeah, that’s a really good point. And for any of you who are going to Toronto and you’re not, I’m hold against you. There, Duod from Altus Labs and I are going to talk. We talked yesterday. You guys probably saw the pilot session and heard the term pilot bias and everything else. But like, one of the things we heard from that is like. Well, we need more long view. We need more of like how to get past the pilot and get into operational. So both of us have built and scaled software companies, so we’re going to talk about that instead of like piloting things. We’re going to talk about like what was it like actually to build something and then maintain it and sustain it over time. And we’re going to do it as an interactive panel. So come to Toronto. It’s Canada. It’s wonderful. I’ll tell you all about it. Eric, that’s great. So we got a twenty-minute break between next sessions. Happy to take questions. Happy to hang around and chat. And otherwise. Yeah, bingo cards. There are some multiple bottles of alcohol downstairs. Alcohol. This is gin and whiskey. Just what a quality professional needs. Thanks, everybody.

Related

• Home
• Strategy Scaling and Product MOC