2026-09-16-1612-local-llms-gxp-audit-trail-lecture

Local LLMs for GxP audit trail analysis lecture

Recorded 2026-09-16 at 1612 via Plaud. 41:36.

Summary

An independent four-month engineering investigation into whether locally deployed LLMs can handle GxP audit trail review, presented as a traceable evidence pack rather than a model bake-off. The setup: 113 experiments run locally on a MacBook Pro through a scripted Python harness over the Ollama API, five open-weight models (GPT-OSS 20B, Llama 2, Gemma family, Qwen), and a synthetic audit-trail test set of 4,898 events across 215 samples grounded in FDA and 21 CFR Part 11 inspection observations, with ten problem types including absence problems (missing records), which are harder to catch than presence problems.

The scenario ladder tells the story: an Isolation Forest baseline flagged six to seven of ten problem types with no explanations; feeding the full audit trail failed silently on context limits; ordered batching improved evidence access but produced 102 to 455 false positives, with very few hallucinations even at high temperature; RAG helped GPT-OSS and Qwen while degrading Gemma. Cloud comparison on the same task: Claude caught four of four completeness problems with 37 to 43 false positives, ChatGPT three to four with zero to eight, and Gemini repeatedly failed. He frames the local-versus-frontier trade-off as control (frozen, fingerprintable weights and runtime, data residency, operator-owned evidence) against capability (frontier models that create supplier dependency), then offers a three-step assurance case (what ran, how it performed, how it stays controlled) and a four-layer validation model (input and preparation, retrieval and RAG, model and inference runtime down to the Ollama version, tools and human review), with a deterministic evaluation pipeline (regex scoring rather than LLM-judged). Open strategic choices named honestly: how to structure human review so it complements rather than duplicates, the control-versus-capability trade, and what counts as good enough.

Plaud AI notes

<App-generated notes from the Plaud cloud, verbatim.>

Date Time: 2026-09-16 16:12:49 Location: [Insert Location] Instructor: [Insert Name]

Plaud summary

The lecture series presents an independent engineering investigation into whether locally deployed large language models (LLMs) can support practical GxP (Good Practice) use cases, detailing experiments to evaluate their effectiveness in audit trail reviews for data integrity and completeness. Conducted over four months, the investigation involved 113 local experiments on a MacBook Pro using five models (including GPT-OSS 20B, Llama 2, and Gemma family models) orchestrated via a scripted Python harness and the Ollama API. A synthetic audit trail test set with 4,898 events and 10 known problem types (including presence and absence issues) grounded in FDA inspection observations was used to assess performance. The study compares various methods, such as Isolation Forest, batched processing, and Retrieval-Augmented Generation (RAG), highlighting critical factors like context window limitations, data batching, and the need to control the entire inference pipeline. The analysis reveals that while advanced models show promise, they struggle with incomplete coverage and false positives, with unpredictable performance. The lectures contrast local versus cloud/frontier deployments for GxP, emphasizing differences in control, data residency, versioning, costs, and limitations. Ultimately, the work aims not to find a perfect model but to stimulate discussion with traceable evidence, proposing a three-step framework for building a GxP assurance case (defining what ran, how it performed, and how it stays controlled) and a four-layer validation model for AI systems, emphasizing the need for a structured, version-controlled evaluation pipeline and critical thinking to navigate unresolved strategic choices like structuring human review and defining “good enough” performance.

Knowledge Points

1. Investigation Design, Scope, and Experimental Scenarios

• Investigation Parameters: The work is an independent engineering investigation spanning four months, comprising 113 local experiments on a MacBook Pro. Its goal is to generate traceable results to inform GxP assurance discussions, not to perform a head-to-head model comparison. All local LLM runs were executed via a scripted Python harness calling models through the Ollama API to maximize determinism.
• Models Evaluated: Five local models were tested: GPT-OSS 20B, Llama 2, models from the Gemma family, and one Chinese model (Qwen), representing open-weight/local deployment candidates.
• Synthetic Audit Trail Test Set: A synthetic test set was created with 4,898 events across 215 samples, grounded in FDA Drugs and 21 CFR Part 11 inspection observations. It included 10 known problem types, 4 of which were “absence” problems (e.g., missing records), which are harder to detect than “presence” problems (e.g., suspicious entries).
• Experimental Scenarios:
  • Scenario 1: Isolation Forest: A baseline statistical method that identified 6-7 of 10 problem types but could not explain the reason for anomalies.
  • Scenario 2: Full Audit Trail to LLM: Failed due to local models’ limited context windows, which processed only a fraction of the data without reporting the incomplete view.
  • Scenario 3: Batched Audit Trail to LLM: Sending data in ordered chunks improved evidence access but led to a high number of false positives (102-455), though very few hallucinations were observed even at high temperature settings.
  • Scenario 4: Retrieval-Augmented Generation (RAG): Providing an SOP-like document as context via RAG had mixed results. It improved performance for GPT-OSS and Qwen (increasing problem detection and reducing false positives) but degraded performance for Gemma.
  • Scenario 5: MCP-like Tool: An experiment giving models a tool to perform actions was mentioned, but results were not ready for presentation.

2. Local vs. Cloud/Frontier Deployment Tradeoffs for GxP

• Local/Open-Weight Deployment:
  • Advantages: Offers greater control, ensuring data stays within the organization’s boundary. It allows for freezing and fingerprinting exact model weights and runtime environments for reproducibility. The organization operates and schedules workloads on its own infrastructure.
  • Disadvantages: Requires hardware investment and internal operational effort. The primary technical limitation is the finite context window of the models.
• Cloud/Frontier Service (e.g., Claude, ChatGPT, Gemini):
  • Advantages: Provides access to the most advanced models and integrated tools without the user managing compute infrastructure.
  • Disadvantages: Offers limited control and transparency. Data is sent to the provider, and it’s difficult to know what’s happening behind the scenes (e.g., model changes, throttling), creating uncertainty and impacting long-term explainability and repeatability.
• Comparative Performance: In experiments, cloud models showed variable performance. Claude identified 4/4 completeness problems but had high false positives (37-43). ChatGPT identified 3-4/4 problems with fewer false positives (0-8). Gemini repeatedly failed the task.
• Assurance and Evidence: A key consideration is the evidence needed to explain a result 18 months later. The deployment choice directly affects the assurance package, with local deployments offering more operator-owned evidence and cloud services relying more on supplier evidence.

3. Framework for AI System Assurance and Control

• Three-Step Assurance Case Framework:
  • Step 1: Define “What Ran”: Document the exact conditions of an AI run, including intended use, input data, versioned prompts, and all configurations (model, runtime, RAG, tools).
  • Step 2: Explain “How It Performed”: Provide evidence of performance, including raw outputs, source-to-claim traceability, metrics on coverage and false positives, and the human disposition of flagged items.
  • Step 3: Detail “How It Stays Controlled”: Outline the ongoing governance, including change triggers, a risk-based regression testing strategy (using a “golden dataset”), continuous monitoring, and incident management.
• Four-Layer Validation Framework: Performance depends on the entire workflow, not just the model. Assurance is needed at each layer:
  • 1. Input & Preparation: Ensure complete and intact data input, with failures made obvious.
  • 2. Retrieval / RAG: Assure retrieval quality, source control, and the end-to-end effect, as retrieval does not guarantee the model uses the context correctly.
  • 3. Model & Inference Runtime: Assure version control, settings, and understand variability and failure modes. The inference engine itself (e.g., Ollama version) can significantly impact results.
  • 4. Tools & Human Review: Assure agent behavior, traceability, and robust human oversight.
• Evaluation Pipeline Solidity: The system used to evaluate and score models must be deterministic and solid, for instance, by using regex for scoring instead of another LLM, which could introduce bias.

4. Unresolved Strategic Choices and Potential Improvements

• Structuring Human Review: A key challenge is designing a human-in-the-loop (HITL) process that effectively complements the AI to catch what it misses, without duplicating the entire manual review effort.
• Control vs. Capability: Organizations must make a defensible trade-off between using local models for more control versus higher-capability vendor models that create supplier dependency.
• Defining “Good Enough” Performance: Determining when an AI’s performance is sufficient depends on risk, comparison to human performance, and the consequences of error. Experiments inform this decision but do not make it.
• Potential Improvements:
  • Low-hanging fruit: Adjusting model parameters (e.g., temperature) and re-running against a golden dataset.
  • Further enhancements: Fine-tuning the model (noted as increasingly accessible), using a better RAG system, and upgrading hardware (e.g., to NVIDIA GPUs) to run larger models.
  • Model Knowledge: LLMs may have foundational knowledge of regulated environments from public data. Performance can be improved by using RAG to provide specific context (like a “QA rulebook”) or by using system prompts to frame the AI’s role (e.g., “You are a GXP expert”).

Questions

• [Insert Question/Confusion]

Assignments

• 1. Define the assurance package requirements for both local and cloud AI deployments to explain a model-generated result 18 months later, including artifacts for versioning, data residency, and reproducibility.
• 2. Apply the proposed four-layer validation framework (Input, RAG, Model, and Human Review) when implementing AI systems for audit trail analysis in a GxP context.
• 3. Create and freeze a “golden dataset” based on GxP requirements (e.g., from 21 CFR Part 11 observations) to use for regression testing when the AI system or its components change.
• 4. Set up a scripted Python harness using a local model runtime (e.g., via the Ollama API) to ensure traceable, repeatable runs, including fingerprinting for the model weights and inference engine.
• 5. Evaluate the trade-offs between using local models for more control versus cloud models for higher capability in your specific context and document the defensible rationale for your choice.
• 6. For a selected AI application, define a process for human review of AI outputs, including clear escalation paths for findings and a strategy to complement the AI without duplicating effort.
• 7. Explore low-cost improvements for your AI model, such as adjusting parameters (e.g., temperature) or enhancing RAG context, and measure the impact on performance using your golden dataset.
• 8. Document limitations observed in your experiments (e.g., context window impacts, nondeterminism, false positives) and propose specific mitigations for a GxP-compliant workflow.

=

Highlights (button-flagged moments)

Speaker Details Independent Investigation into Locally Deployed AI for GxP Use Cases

The speaker elaborates on the presentation slide, confirming he personally conducted the investigation detailed on the screen, which involved running 113 local experiments with 5 different models over 4 months to test their viability for GxP applications. • Visual Context: The slide outlines a four-month engineering investigation with 113 experiments and 5 local models to test if locally deployed models can support practical GxP use cases. • Discussion Insight: The speaker confirms he ran these experiments on his MacBook Pro using a scripted Python harness that called local models through the Ollama API. • Key Takeaway: The investigation’s purpose was to be a traceable, independent engineering study to stimulate discussion, not a direct comparison between different models.

Comparing Local vs. Cloud AI Deployment: A Tradeoff Between Control and Capability

The speaker uses a detailed slide to explain the fundamental tradeoffs between deploying AI models locally, which prioritizes control and data sovereignty, and using cloud services, which offer managed capability but with reduced control over the model and data. • Visual Context: The slide presents a side-by-side comparison of ‘Local/Open-Weight’ and ‘Cloud/Frontier Service’ models, detailing differences in data handling, versioning, cost, and primary limitations. • Discussion Insight: Speaker 1 elaborates that for GXP use cases, local deployment offers crucial control over the model version and keeps data within your boundary, whereas cloud services mean losing control over the model’s lifecycle and service conditions. • Key Takeaway: The choice between local and cloud deployment is a strategic decision that hinges on balancing the need for direct control and auditability against the benefits of accessing advanced, managed AI capabilities.

Speaker Details Experimental Setup Using a Synthetic Audit Trail with Planted Problems

The speaker explains the slide on screen, detailing the synthetic audit trail he created for his experiment, which contains 4,898 events across 215 samples with 10 types of ‘planted’ problems derived from FDA inspection observations. • Visual Context: The slide outlines a synthetic audit trail test set created by Systera Solutions, featuring 4,898 events, 215 samples, and 10 specific, pre-defined problem types. • Discussion Insight: The speaker clarifies that he personally planted these 10 problem types, based on real FDA findings, to create a controlled experiment for testing the capabilities of various AI models. • Key Takeaway: The experiment is designed to test AI models on two main fronts: data integrity (e.g., signature mismatch) and data completeness (e.g., missing tests), using this purpose-built dataset.

wow I really like this talk, it’s super practical. case study!!

• Risk Identified: Both local and leading cloud LLMs (like Claude and ChatGPT) failed to identify all problem types and produced a high number of false positives. • Risk Identified: Local models with limited context windows analyzed only a fraction of the data (as low as 0.3%) without reporting the data was incomplete. • Key Insight: Cloud models pose a challenge for regulated environments due to their lack of transparency and potential for unannounced changes to versions or performance. • Key Insight: Local models offer greater control and reproducibility, which is crucial for GXP compliance, but even they require careful management of the entire technology stack. • Action Item: Develop a robust validation pipeline, including a ‘golden dataset’ for regression testing, to evaluate and monitor model performance over time.

isolation forest for anomaly detection

• Context: The team is evaluating five different scenarios to assess model performance on audit trail data. This analysis covers the first scenario, which uses the Isolation Forest statistical method as a baseline for comparison. • Key Insight: The Isolation Forest baseline model can identify a majority of problem types (6 to 7 out of 10) by flagging rare events in the numeric audit trail data. • Risk Identified: The model’s primary limitation is its inability to provide context or explain why an event is considered an anomaly, making it insufficient for a comprehensive audit review on its own.

Speaker Analyzes Performance of Batched Local LLMs in Configuration 3

While displaying a slide with performance metrics for ‘Configuration 3’, the speaker details the mixed results, highlighting how batching the audit trail enabled models to identify some problem types but also led to a significant number of false positives. • Visual Context: The slide shows performance data for ‘Configuration 3’, where local LLMs analyzed an audit trail in 25 batches, resulting in 0-3/5 problem types identified and 102-385 false positive rows for the Data Integrity Review. • Discussion Insight: The speaker clarifies that while batching helped, performance was inconsistent, noting that one model identified ‘three out of five’ problem types, but the number of ‘items that were wrongly flagged is also, you know, significant’. • Key Takeaway: Sending the audit trail in batches made the evidence accessible to the models but produced highly variable results, particularly a high rate of false positives, which the speaker notes as a major issue with this configuration.

LLM Performance Comparison: Impact of Retrieval-Augmented Generation (RAG)

The speaker analyzes the presented slide, highlighting how adding a ‘rulebook’ context via Retrieval-Augmented Generation (RAG) improved performance for the GPT-OSS and Qwen models but unexpectedly worsened results for the Gemma model. • Visual Context: The slide compares three LLMs, showing that with RAG, GPT-OSS improved from identifying 3/5 to 5/5 problem types, while Gemma’s performance deteriorated from 2/5 to 1/5. • Discussion Insight: The speaker clarifies that only the GPT-OSS comparison is a ‘clean’ before-and-after test, as the inference engines for Qwen and Gemma changed between the ‘No RAG’ and ‘RAG’ configurations. • Key Takeaway: Adding context via RAG can significantly enhance model performance, as seen with GPT-OSS, but results are not universally positive and depend on the model and underlying technical stack.

Lessons Learned: The RAG setup, its control, and the inference engine must be part of your pipeline and assurance package.

• Speaker 1 concluded that the RAG system, how it’s configured and controlled, and the inference engine are essential components of the assurance package for model validation. • An unexpected version change in the Olama inference engine during experiments highlighted the necessity of controlling all parts of the pipeline, including the engine itself. • The speaker emphasized that whether using local or cloud models, the strategy for preparing the assurance package will differ, but in both cases, understanding and controlling the full pipeline is critical for explaining results.

Comparative Analysis of Frontier AI Models Reveals Performance Gaps and Control Issues

The speaker presents a slide detailing the mixed results of an experiment on frontier cloud AI models, noting that while products like Claude and ChatGPT performed well in breadth, they struggled with completeness and flagged false positives, and the Gemini test failed entirely. • Visual Context: The slide shows a comparative table where Claude and ChatGPT successfully identified many problem types but struggled with sample coverage and generated numerous wrongly flagged clean samples, while Gemini failed to complete the review. • Discussion Insight: The speaker clarifies that this was not a controlled comparison and highlights unexpected failures, such as Gemini reporting missing data after successfully reading the file, to illustrate the lack of control and transparency with cloud services. • Key Takeaway: The experiment demonstrates a critical trade-off: frontier cloud models offer high capability but come with significant provider dependency and a lack of control, whereas local deployments offer greater control and verifiability, which is crucial for regulated environments.

Comparative Analysis: Control vs. Capability in Cloud and Local AI Deployments

The presentation slide visually contrasts the trade-offs between cloud and local AI deployments, a dichotomy the speaker elaborates on by sharing experimental results that highlight the greater control of local models versus the opaque, high-capability nature of cloud services. • Visual Context: The slide presents a side-by-side comparison, detailing how cloud products offer integrated tools but lack constant conditions, while local deployments provide a fixed model, greater control, and a customizable test environment. • Discussion Insight: The speaker confirms the slide’s points with personal experiments, noting that local models allow for a fixed, controlled setup (e.g., model, inference engine), whereas cloud platforms like Claude and Gemini offer less transparency and control over the environment. • Key Takeaway: The choice between cloud and local deployment involves a fundamental trade-off: local offers greater control and defensibility for processes like GXP, while the cloud provides access to more advanced models at the cost of dependency and operational uncertainty.

Speaker Outlines Four-Layer Validation Framework Based on LLM Experiments

The speaker concludes their presentation by displaying a slide that systematizes their experimental findings into a four-layer validation framework for AI systems, covering input, retrieval, runtime, and human review. • Visual Context: The slide presents a ‘four layers of validation’ framework: (1) Input & preparation, (2) Retrieval/RAG, (3) Model & inference runtime, and (4) Tools & human review. • Discussion Insight: The speaker’s experiments revealed issues in each layer, such as data handling changing model exposure (Input), retrieval not ensuring relevant context (RAG), and unexpected inference engine updates (Runtime). • Key Takeaway: The speaker advocates for a comprehensive assurance strategy that addresses each of these four layers, plus the evaluation system itself, to ensure end-to-end fitness for the intended use, especially in controlled environments.

super pragmatic recommendations mentioned above! we need to capture these

• Key Insight: AI validation requires a multi-layered approach; focusing only on the model’s core performance is insufficient for reliable deployment. • Key Insight: The quality of the data retrieval system (RAG) is as critical as the LLM itself for generating relevant and accurate outputs. • Action Item: Implement the proposed four-layer validation strategy: data handling, retrieval quality, runtime control, and human review. • Action Item: Establish strict version control and monitoring for the entire AI pipeline, including the inference engine and retrieval system. • Action Item: Define a structured process for human oversight and escalation of AI-flagged issues to ensure effective final review.

Speaker Elaborates on the Three-Part Framework for AI Assurance Cases

The speaker’s detailed experimental findings directly follow the three-part evidence framework—‘What ran,’ ‘How it performed,’ and ‘How it stays controlled’—presented on the slide for reconstructing an AI assurance case. • Visual Context: The slide outlines a three-step framework for documenting an AI assurance case: detailing what ran, how it performed, and how it remains controlled over time. • Discussion Insight: The speaker explicitly organizes the final part of his presentation around this three-part framework, using it to chronologically summarize his experiments with local and cloud models, from initial setup to performance analysis and future control strategies. • Key Takeaway: The choice between local deployment and cloud services directly impacts the nature of the assurance case; local deployments offer more operator-owned evidence and control, while cloud services shift reliance to supplier evidence and monitoring.

Key Strategic AI Implementation Questions Raised by Experimental Findings

The speaker concludes his presentation by discussing three critical questions displayed on the slide, which were prompted by his experiments and remain unresolved: how to structure human review, what trade-off between control and capability is defensible, and how to decide when an AI is ‘good enough.’ • Visual Context: The slide poses three pivotal questions about AI implementation: structuring human review, choosing between direct control and higher capability, and deciding on performance benchmarks (e.g., ground truth, human performance). • Discussion Insight: The speaker explicitly introduces these three questions as the main discussion points arising from his experiments, emphasizing that while the experiments provide data, they do not provide the final answers to these strategic choices. • Key Takeaway: The central conclusion is that deploying AI involves making crucial, context-dependent decisions about human oversight, supplier dependence, and performance standards that cannot be fully automated or resolved by technical experiments alone.

I like the concept of risk based sampling with full secondary reviews as well as the question of why bother with automation if we have to do full review.

• Key Insight: If a full manual re-review is required due to lack of trust, the primary efficiency benefit of AI automation is negated. • Key Insight: A successful implementation requires a clear strategy for human oversight that complements the AI, such as focusing on risk-based samples or secondary reviews of the AI’s findings. • Risk Identified: There is a significant trade-off between using local AI models (more control, less capability) and cloud models (higher capability, greater vendor dependency). • Key Insight: A critical success factor is defining the threshold for when an AI’s performance is considered ‘good enough’ for its intended use, given the consequences of potential errors.

Transcript

[00:00] Speaker 1: How they’re being evaluated, right? Many companies, I’m sure, they have their pipelines to evaluate. But then I’m a bit curious and curious. Is there anything out there? We know that we have benchmarks, right? For some of the multiples out there, and we see how that will do against different pasts. But obviously, I’m more interested in understanding the screen in the future. So, could a control production like Test that I can set up provide kind of initial evidence for this model. So that’s what I did from May. I four months I run 113 local experiments on my laptop, on my MacBook Pro. Five local models. I use GPT OSS, a 20 billion parameters. I use Llama two of the Gemma family and one Chinese model. Now. I’m not claiming here that this is a perfect, say, investigation, perfect experiment, and I will be upfront about some of the limitations. But what I want to do here is to have a conversation, right? To see where we are, and especially for people who are interested in deploying local LLM, and just want to, you know, have a pulse for for where the industry is. So, just a quick note on what I did. I run everything again, set on my computer, scripted Python, so everything was, you know, as deterministic as it could be, minus obviously the, you know, some of the inference that was running, and was giving me the answers and some of the scoring and so on and so on. It was an interesting experiment, and everything was run through the Llama API, if you know what this basically is like, you know, the engine that allows you to run local AI. And some and the product interfaces were only used for the cloud models because then I got curious and I say, okay, let’s see actually how the you know the the big names do with the same kind of tasks. So since pretty much every speaker before me, we have a poll, and I want to ask you, who is you? I swear I’m not going to follow up with any of you. I’m just curious, who is using local AI then right now? Okay, one, two, three. Okay. So my prediction is that it will become more and more common because of talking cost and various reasons. But specifically, in the GXP case, you don’t want to go. You know, some of the limitations that you find, obviously, I’m perfectly aware that locality is not is not a solution parameter. Let’s say you have a a control input, you have some GXP records, and you have a governing document. It can be SOPs or anything you want to run through this models. If you choose local, typically you have a lot of control because you can run your model. You can you know you can use a laptop, but actually you can use something better than a laptop. You can have your server, and you can run bigger models. And the models are getting better and better. You retain the exact Model, you freeze everything, you retain the frame and frame exactly the same model, you converge and so on, and that gives you a lot of control. The limitation with it, obviously, the finite context window, and Rosling said yes, very yesterday, very well. You need to be very aware of the context. This is something that you definitely need to take into account, not just when you run it, but also when you when you have the results from from these models. In the cloud and frontier services. Surely, Dev. Obviously, there are advantages. You are accessing the latest and greatest of of AI. But you know the problem is that you lose some of that independence, right? In terms of data, the versioning, what the what does those vendors do? You don’t really know. I mean, they can change the model. They can retire the model. Surely, if you have some enterprise you know service agreement, maybe you can freeze it for a little bit. But then, you know, you you really are depending on their service. And and actually, some of the suggestions that you will see here in one of the experiments actually was indicative of that that kind of lack of control, and you know you don’t know what’s going on behind the scene. So limited control over model and and also the service conditions, right? So ultimately, the question that you ask and it will influence your choice will be: eighteen months from from now, you run something through these models. What evidence you need in order to explain what was produced, and that obviously the route that you go will dictate some of the assurance package that you need to produce. So I I decided to yeah I mean we should pay more attention to the trails and some some vendors so that the specific product for on trail and I believe that that would That would provide me with something that was a controlled experiment or as controlled as I could be. When I have, for example, in this case, four thousand eight hundred ninety-eight events across two hundred fifteen samples, and capturing basically planting ten known problem types that were coming from FDA drug inspections and and part eleven inspection observations. So the. The two problems that you see here are things that we’re going to need to understand: shape, timing, or the underlying change region. The reason signature mismatch, lack of repeatability, empty, incoherent distribution. That’s problem number one. So it’s the problem that focuses on fitting. Problem number two focuses on the completeness of the data set. And now these are particular challenges. Why, why I say this? Because in the end, these are missing stuff. That you typically have to have context for what is missing, and the model will have to go throughout the whole dataset to identify what is missing. Again, some context here. If you don’t give the context, the model won’t be able to understand what is missing because you don’t tell the model what is actually should include in the package. So, missing or invalid dependent review, missing or missing investigation. Missing required tests, testing into compliance, and an accounting of required value. So now, before we get into the results and problems, okay, I will keep this very light. I mean, the the first version of this presentation, I love experiments. I was super excited. I go to my wife and say, “Look at this! I mean, it’s amazing. All these results.” And I will be last or second to last at a conference. She looked at me and said, “Don’t.” So I, I this is a simplified version of or summarized version rather, but you have all the results in the appendix. So once you get a slide, I also welcome the conversation about this. So please, challenges. I’m not trying to sell you anything. So I definitely welcome the the conversation. So same obitrial, we’re studying two five different scenarios. I won’t talk about the fifth one that you’ve seen there, which is an M C P like tool. So basically, giving these MLMs a tool to do something, I started playing around with it, but you know, I definitely need more data for for me to present with something. So maybe next can next method number one is isolation forest, which is anomaly detection, right? So you basically see whatever is rare in the original, and you this was thing line in the numeric features. What I mean, you know, find. Action was completed. The operator is now. As you can see, the number on the on the right, number with the percentage. Going back to the final column, right? Now, the scenario number one. You will give the statistical analysis, and you give the whole thing to to the to the model. There’s nothing to do with the With the language model, but it will be useful as a baseline. Scenarios two and three is the same thing. Now, this time, with you give the the od trail, but this time, number two, you give the full od trail to the model, and it is a language model, and all the different language models that that I tested. Number three is you give in batches, so you give in twenty-five different batches. Why do you do that? You will see in a second. Number four scenario will. Configuration number four is you add a rag. Rag, if you’re not familiar, is rag is a retrieval augmented generation. So basically, you give a little bit of context to the model, and the context in my case was an SOP-like document where you list all the responsibilities and you list how the limbs will work and what all the use cases are for the, or rather, all the tests that you need to run on a sample. So the The model will know what is potentially missing in the trail. Now, again, going back to the number that you see on the right, context is everything. In this case, if you give something to the model that you think it’s complete, and you think that the model is going to tell you that it didn’t see the whole thing, then you may be disappointed. What I mean by that is, let’s take isolation forest. You give the whole OD trail, and it’s pretty good. You know, able to identify a good number of problem types. So seven out of ten, maybe six out of ten to seventh was you know just a little around the same the same probability as chance because also you need to evaluate against chance. Now the problem with this is it’s simply anomaly detection, right? So you see how rare some of these events are, and then the model will tell you, hey, you know, this actually looks very rare. You don’t think why because it Statistical technique. I think it’s very useful. Don’t get me wrong, but it’s you know taken to isolation just to do all the trail view would not be very useful. Now configuration number two, and again going back to context, you give the whole trail. These are models that typically are fairly small. Obviously, they run on your computer. You can run a bigger one with a context window that is larger, but. Just to make sure that we understand what we’re talking about, is that it’s like the memory that those models have is pretty limited. So that means that if you give the whole trail to the models without even touching any parameters used by default, 0.3 to 0.4 percent of the whole trail will be seen by the model. How do I know that? Because you can’t predict how many tokens can fit in that in that window. Now you extend it because you can change the number. You can send it to the maximum. However, the maximum, even for the more capable model, is 19%. Now, this doesn’t do very well. The models will now identify a majority of the problem types, but then look at that. None of the models told me, “Hey, I didn’t see the whole picture.” So again, going back to rather than saying yesterday, make sure that you know. How much of the od trail, how much of the data is your model seeing? Because in this case, the percentage is pretty low in a long time you’ve been experiment. Is the same happening with the plot of the frontier models? Well, I’m not. When you have replaced and tells me that you upload something to plot and is able to see the whole od trail, so that question will come back later. Now let’s do something better, and let’s try to send the same ODE trail, but this time in batches. Right. So the problem with these approaches, like you know, it’s it’s helpful at least for some of the models. You can see that three, at least one model identified three out of five of the problem types. I call that the breadth of everything that the model can can see in the ODE trail. The number of problems. Are identified is up to four, up to seventeen. Okay, but if you know that, look at the false positive. The items that were wrongly flagged is also, you know, significant. I would say not too probably. If you’re a curious person, you should see the 300 plus rows, and you know, and most of them are false positives. However, remember that the interesting. Thing here before I get to the second column is that I, by design, I didn’t touch the parameters. I didn’t touch the temperature. I didn’t touch dot K, dot P, and so on. I did it for a reason because otherwise, you know, the experiment would be, you know, so many arms. I just wanted to do a quick and dirty, if you want, experiment. The thing is, what would you expect if you keep the temperature to that? The default is typically say 0.9. You would expect hallucination, right? So the interesting part here is that I didn’t see a lot of those mentions. Doesn’t mean that obviously reducing temperature will not be. You know, I’m talking about like three hallucinations in the whole many experiments that I’ve done. So, which was pretty good. But then I wonder would that also start solving the problem with with the false positives? I don’t know. Maybe next presentation I will run a few more experiments. So that was actually an interesting observation. Second column that you see there. How well do they do if they go across the whole audit trail and try to see if something is missing? Now, two out of four of the the breadth, so every model identified two out of four of the problem types, and the samples were two to five out of twelve, so not really that many. And the rows, you know, as you can see, still significant number of false positives there. Now, if you’re following. The setup of this experiment, you might actually think, well, but that’s not fair, right? Because you’re sending batches of the trail, so that means that how can the model go through the whole thing if you’re sending individual batches? And that’s a fair observation. You are sending these, you’re chunking, you’re sending these to the model. Probably the model does now have all the opportunity to actually see the whole audit trail. How do you do that? You need to re-engineer other parts of your Of your setup, and you know, in the end, if you don’t need to send the whole trail, then you need to make those considerations as part of your pipeline. So, in other words, you need to understand how the model is seeing, how much of the model is seeing, and what is also in the context of the whole trail that those particular models are seeing. So, now let’s move a little further. In these, you know, the sequence of this. So, I mean, this time we gave it a little bit of help to solve, and we are we give it the context of the issue. Now, one thing about using rag that I noticed, as you can see, GPT-OSS did pretty well. I mean, I would say that go from three out of five problems identified. Now it’s able to identify five out of five, which is very impressive. The number of false positives also. A little lower than before, and you know it changes a little bit between no rag and rag, but still you know stay within you know within that range. Gwen also does very well. One out of three of the five problems, you go up to three out of five, and the number of rows, the false positives were actually going was actually going down too, which was very interesting. Gemma, very interesting, did not like this. For some reason, we just did actually worse than not giving it the giving it the the rule book, what I call. So now, caveat here, and it’s a an interesting story and also lesson to learn, right? I run through all these experiments, and after a few experiments, then Claw, you know, I’m setting a little you know things up with Claw, and Claw says, “Oh, by the way, do you know the your inference engine, Lala?” Change version from this experiment to this experiment, and the answer was no. Actually, we didn’t pay attention to that. So, by example, you also need to pay attention to the to the inference engine that must be part of your of your pipeline and and your shooting package. So, lessons learned here is obviously this is not directly comparable for for a couple of models. Maybe maybe numbers will be different, but again, the lessons learned is like. The rag and the way that you set up the rag and the way you control the rag and also the way you control the inference engine must be part of your partition package. So then we go through these and we say, okay, let’s see how well the two models that we use are doing. Again, exactly the same task. This time I import a new file, a CSV file, to plot ChatGPT, Gemini. Slightly different here that you see here. Use the the highest model that was available for this application ahead. Three different repeats for the two different processes, six experiments in total, and it was interesting. Negative review, plotting pretty well, and say, okay, you know, it’s like a zero to one file of records. You know, right now we’re talking about from here onwards, obviously, so I expect you know some numbers in that. The difference between the two also for GPP was pretty. Here, not all the records were identified. Still, still, completing this review, that’s what actually we were doing. I’m using table five. It’s one of the most disastrous from Table five, and even though it’s 5.1, but still, a. We don’t identify the problem types, but as you can see, not all the samples. So there is something there that it cannot still cannot do very well. Now three or four for. Problem types for ChatGPT, and again a few samples of missing. And look at the rows that were identified as false positives. Actually, there you know there. I would say that it’s considerable number. I mean, if you think these are you know the most advanced frontier models, that was interesting to me. Now, what this tells you is that you really don’t know what’s going on behind the scenes, right? I, what if I run the same experiment in the morning? Was there any throttling going on behind the scenes? Was the. Actually, was Claude ChatGPT? How do I know that the whole file that I uploaded was actually informing that the answer? Unless you have an API or something, and maybe you’re able to control something. I mean, this again, straight into the interface. But it is something you want to consider. Again, I did not have any answer to say. Okay, sure. These models saw the whole trail, and that’s how you know their response was informed by all the rows in the trail. Gemini was interesting. That’s another lesson learned, right? I mean, like you know, I have a beta account with Gemini. I did it six times, and I swear I don’t know what’s going on, but for the all the six times, it’s reading the file. It gives me with the Python code that is actually reading the file correctly, and then it tells me. Oh yeah, I’ll be happy to help you with that prompt, but you need to upload a file. Well, you just read it, so I truly don’t know what’s going on. And again, do we know what’s going on behind the scenes? Do you have access to different tiers or different models if you have an API into Gemini? I don’t know, but these are things that that’s the variability that you really want to take into account when you make these considerations. So this is not a direct comparison between cloud and local. I just want to make this clear. This is something that I can do in a maybe more detailed discussion to understand what the industry is doing. Mistakes. The cloud products. Sure, you have access to the most advanced models, and you can see like get whole file access with integrated tools and so on. But then again, the provider is managing your computer. You don’t know really what’s going on behind the scene. Model tier service conditions mean. Can you control them? The answer is no. It’s it’s all in under their control. Product level and system level failures still happen. You know, as you can see, even with the most advanced models, not all the the cases were were identified. And Gemini, who knows what’s going on with Gemini? You know, it just could not resolve the what I what I consider a fairly fairly simple task or what looked like. A very simple task for the other for the other models. Local deployment still matters because again you can control you can fix the model. Nobody will tell you that at some point you cannot use that model anymore. You can version. You can run an inference runtime with an inference engine, and you can also control it so not not get automatically you know updated under your your GXP process. So you can also. Control that you can freeze it. You can freeze your configuration, and there’s so many more things, right? The context, the output, these all things that you can control if you set everything on your premises. Greater control over changes, obviously, and the data boundary, and a limited test environment that reminds you that, “Okay, this was run on my computer.” Now you get a better machine. You can run a 120 billion parameter with a bigger Context window, and you know, maybe these results. I presume the results will will look much better. So this again, something to consider. There’s no right or wrong with this. It really, you know, the pros and cons that you have with the two options. So all in all, these things that to me all these experiments lead to at least you know four layers of validation that we see. One would be in terms of Google benchmarking. I observe. Yeah, a lot of data handling that changed the way that the model was seeing the data, and again context, regulating the context, whatever the model is able to see, that has to be completely intact. The failures there are big. Just try to make sure those failures are big because the model will not tell you if the model will not tell you the failures are big enough. In terms of direct, I did not show run a control experiment that was very interesting. I used out of the box, very simple embedding model that I used as a rag to retrieve information from that QA playbook, SOP-like that I mentioned. And interestingly enough, I the retrieval engine itself, if you don’t also optimize it. Was not really retrieving relevant information. In other words, I was asking related question, but also non-related question, recipes for banana bread and so on. And the model, the the rag component itself, was not selecting for the task ahead. So that means that you can also optimize that, and you can again part of your assurance strategy make sure that your retrieval quality, source control, and end-to-end data is is also in. Something that you can explain to the auditors. The third one is a model and inference runtime, you know, lead by example. I did not realize the olama changed under under my my watch, and they did. And you know, this is something that you also want to take into account: performance. Anything that you configure in your pipeline has to be has to be controlled, version settings, and so on. And again. If you’re using your cloud frontier models, then you know there is also something that you need to take into account: how you can control that tools and human review. Then you have a certain number of actions in these audit trails, and then you send it for human review. At that point, you know you’re also taking care of the of who’s actually complementing that review, right? If the intended use. Is prioritized or the trail records for human review. Then you need to understand what to do and how to escalate some of those findings and so on. So HPA involves also traceability and human oversight and all this. So an interesting point. So also on top of this is that whatever you set up a system, let’s say that you have intended use, you want to set up something similar at your company, and you want to set up a way. To test how the different models are doing against your audio trail, right? You also want to make sure that you the evaluation pipeline, the scoring that you do is also solid because there’s a lot of work to be done with the results that were coming from the LLM. Because at that point, you know, there was a funny story where at some point I said, “Okay, you know what? Evaluate these results and so on.” And when I went to the frontier models. Claude told me, “Well, but how do you know that I didn’t score actually Claude to be better?” I said, “Wait a second. Okay, that’s interesting. But it’s it’s a valid point. I mean, did I pay a lot of attention in the way that the model was scoring? How much of that was deterministic? You know, I was trying to do as deterministic as possible. Meaning, you know, you just do regex and check, you know, the exact sample name and so on. But it is also something that you need to consider and a lot of critical thinking there.” So finally, let’s see these across the whole you know chronological order. You run something using tools I can present it right now. So you have intended use and human decision authority and so on. Now you take care of the input and the version prompt. This is something you want to do to explain what actually run at that point in time. Model runtime rag tools everything that I mentioned before. Now, the second step is how do you explain what, how it was performed, and those are both things that you also need to consider at a point because you know how it performed, meaning you know the raw output, the source, everything, the traceability from the result that you get all the way down to the initial data that you fed the model, but also the exposure, how well it’s doing with the breath, and you know the records that you get over the years. And also, you want to make sure that you have something like this for your applications. That you take care of the repeat runs. I mean, everything that fails, why fails? And again, on top of that, also what happens with the records that you send to QA? Right. You know, what is the the human process on top of the on the on the automated process? And finally, how is data control now? Look at the future. Everything that ran in the past, and now what’s going to happen? As this system evolves, right? So change, change triggers. Do you have any risk-based regression? For example, you run the same dataset, a golden dataset, and you see how well it’s doing. And if you see any regression, then you flag it and what do you do? How do you monitor all this? And you treat it like like any other system that you have, right? Incidents, periodic review, and so on. And you may be able to leverage. Supply evidence, obviously, if you’re using one of the cloud models, so that’s also very important. And again, the fact that you use local versus cloud will determine a different strategy in the way that you prepare the assurance package. A few questions. That hopefully we have a few minutes left. If you’re not too tired, that we can we can discuss. One thing that. All these experiments highlighted to me to say, you know, for this particular intended use, now you sent x number of potentially problematic rows to QA. What do you do with it? How do you structure the human review so that the human review is truly complementing what the AI is giving you? Because at some point, sure, you know, we’re not supposed to trust them blindly, obviously. But then, if you don’t trust them at all and you have to repeat the whole review, then what’s the point of it? So what do you focus on? Maybe risk-based, maybe secondary view, whatever strategy. No right answer. These are just little questions that maybe we want to think about. Which trade-off is more defensible? Again, no right answer. You have more direct control with lower capability, and obviously you can have better capability if you’re using better hardware, you know, laptop. But also, you know, or going to the. And using those vendors with higher capability, but with a greater supply dependence. And the third question that I leave you with is that: Oh, how should you decide where enough, where when good is good enough? Right. So you set to say you have your golden data set, you have your benchmarks, and then you run it. How good should the AI do? Like you know, do you have ground truth? I mean, you know, like 90 percent of that, you say will be 100 percent. How you find even performance? How you even define actually the units and all of that of the review that they all execute? You know, the consequences of the error in the airspace. But all in all, the installation today is a part of the discussion. And then I’ll be making those decisions for us. But I hope, hopefully, that gives you enough to think about for the application, similar application that you have in your company. So, I’m open up for questions. Any challenges? Yeah, we’re not sending you anything, so lay them in.

[31:17] Speaker 2: Thank you very much. It was very good talking. I don’t understand because the whole idea with the law and a global solution like that, we have to completely change control.

[31:39] Speaker 1: So what changed? Well, I think what was going on is that first of all, they actually have approved it, you know, without thinking that because you know sometimes people tell me that you know, date it’s a fact. So look, I assure update. You know, that’s I think that’s what happened. Sometimes you have these products that are automatically updated. You know, asking you. Obviously, if I had to set up this in a GXP environment, also you do not automatically update. But I think that’s that’s what happened. I thought you had some some kind of weird ground loop issue. No, no, no. I mean, not that I know of. But but it was you know again a good lesson learned. You know, I’m glad that I caught it because again it was part of the lessons you know that I learned through these experiments. Hmm. For the local model, so I decided specifically to use. Sorry, the question for people online is if I decide to use a baseline model or if I decide to change anything. I purposely decided not to change anything because, again, I was not interested in saying one is better than the other. I mean, obviously, yeah, you see some numbers there, but I was more interested to see, you know, if this is a reusable pipeline. Then the next step will be okay. Now let’s Lower the temperature. We run all the experiments and then see, you know, if the model is doing better than the others. I the reason why I decided to leave the parameters by default, apart from the context window that I changed, was also because then it won’t be generating a lot of like okay now we lower the temperature for one. But I also tried. I don’t know if you tried to play around with temperature. Some of the models, GANNA, does not like it when you go down to zero. It starts like you know blabbering, and it was like. Absolutely crazy. So you want to set up to 0. 1 instead of of zero. Some of the models are perfectly fine with zero. Sometimes you adjust the temperature a bit, but you you apply a penalty for repeated tokens. So that may work better with some. Also, there are a lot of considerations to be done. But you know that’s sorry, long answer to your question.

[33:45] Speaker 2: I think in reality, if you work at the DSP level. Yes

[34:07] Speaker 1: Absolutely. See, I have a lot of slides in appendix that you can go if you if you if you’re trouble sleeping. These are all improvements that you can do. Some improvements that you can make. Right, you can make improvements. That you can fine tune it. Right now, fine tune is relatively accessible, but not training from scratch unless you have a few million dollars. But, You can fine-tune very effectively. You can layer on top, you know, that you can you can adjust it with a better retrieval. You can adjust it with better, you know, larger models. You buy, you know, a big like you know Nvidia computer. You can run 100 billion parameters instead of 20 billion parameters. Yeah, these are all things that you can definitely improve. So this absolutely baseline the minimum that you can, which is why I think it’s pretty impressive because you know it didn’t do much and you know it does already you know. Decent jobs so they can build them. Well, I would say that you know, I spend millions of dollars if you want to to train. From scratch, but you know, I don’t think there is any reason to to fine tune. Probably, if you have a few people that that know how to do it, you can run on NVIDIA. You know, like you know, you can buy you know like what five thousand dollars, maybe ten thousand dollars. I mean, it’s not, it’s not. I think it’s well within you know most of the companies’ capabilities, yeah, for sure. Yeah. Right. And then another thing, I didn’t even talk about output that you see there, the structural findings. Sometimes you know the output was actually so large that to truncate it, and then it was actually part of the failure because you know the model would not listen to the exact format, and that’s something also you can optimize. There are many different ways you can do this, and it’s also part of the the improvements.

[36:16] Speaker 3: Yeah, I was actually kind of curious of. I’m glad you showed this. I was about to ask what the, Potential improvements are. So now that we you’ve laid it out, which of the options here do you think has the highest potential? Like I’m looking at this problem and I’m thinking you’ve tried the vast array of single agent, does multi - agent is is that a good hypothesis to move to, or do you think there’s still more to be squeezed out for highest performance gains?

[36:48] Speaker 1: I would say first of all, maybe improving the hardware. Start to run like a little bigger model, you know. But it’s yes and no. I mean, I would still think that use a hundred and twenty billion parameter, but you don’t optimize everything else. You probably not going to have you know a lot of gains, right? Fine tuning, I think it’s it’s very important. You know, and it’s something that might take you a little bit of time, but it’s also you know customized to your own environment. I don’t think there is one single thing. This personally, I don’t know if anybody else has an opinion, but I’m happy to discuss this. I would say that the you know there are a few low hanging fruit, right? You know, first I would say change the parameters, right? See again, we run the you know against the the whole dataset and then see how well it does. And then second thing is try to run a bigger model, try to run the more. Reasonable. I mean, by the way, the the size of the model itself does not necessarily correlate with the context window. There are some models that are older but have a bigger context window. I know there are some more more than one that are smaller. So it really depends what you’re trying to do. Again, all these things you know that can can give you a lot of you know a lot of a lot of success. But again, that’s why I think is useful whatever you have in your environment, identify the, The intended use, and you run the same thing, and then you plug and play, try another model, and see you know how well it does. And obviously, you need to have a golden data set or something that is that you scored again before, which is what I intended to do with you know create the golden data set, freeze it, and run again.

[38:31] Speaker 3: And then maybe a follow up is, what’s the likelihood that you know, or perhaps like how how heavily represented. In some of these small language and large language model training sets, do you think audit trails actually are represented? You know, do we expect LMs to have seen this before, or do we expect it to be something that we need to always guide it with? You know, here’s what an example of an audit trail would look like.

[39:01] Speaker 1: So if I understand correctly, the question is whether we need to provide the context.

[39:09] Speaker 3: Not bad. It’s actually, what are what are your priors on the representation of audit trails, this specific type of input, in these models’ training sets, right? So, for example, I’d expect recipes to be in there because there’s a jillion recipes for food on the internet, but not what about audit trails and these weird.

[39:32] Speaker 1: I think there’s more of everything. I mean, they’ve been trained on Reddit. They’ve been trained on you know on the internet. I mean, I I think you know again if you have the capabilities, you know, in terms of like you know people you can use to to fine tune it. I think that will be a good way because at that point, you know, it’s a I think it’s a good way to customize it to your you know to the GXP and give it give it enough information that you say this is not the trail. This how the trail works. I did it very simply with a rag. You know, it’s. I think it’s a quick and cheap way to do it if you want. With the what I call the QA rulebook, I was actually adding information. This is an audio trail from the limb system, and this is, I can give it a little bit of context. It doesn’t change the, you know, the weights, but at least you know it gives a little bit of extra context. You’d be surprised. Many of these models they also been trained a lot of. Of medical data, like you know the Gemma family, so I think we are good enough to understand, you know, the importance of early trial and obviously, you know, but then also system prompts, right? I mean, if you can also use a system prompt to say, hey, you are a GXP, you are working in a GXP environment, and you know, so that also understand, you know, I would say a simple experiment is trying to ask questions about GXP, right? You know, compare offline and see, you know, how well they respond, and you know, the response is typically. Decent enough, you know. I think there is enough in the data set that you know in the training that will make him understanding more than the task. Again, you can always can always improve that for sure. Cool. Thank you. Okay. Any other questions? Comment. Well, you know where to find me. I have an email address there. Always welcome the discussions. And thank you for staying with me until the very end. Thank you, and have a safe drive flight. Keep on.

Related

• Home
• Data Foundations and Audit Trails MOC