Eric Ma's Blog

Funny Money Allocation: A Decision-Making Trick

2026-09-11T00:00:00Z

A while back, I was part of a team that was going in circles. We had three options in front of us, and every discussion round ended the same way: people advocating for their preferred option in words, others pushing back in words, and no convergence in sight. There was no slam dunk benefit to any one option, hence the loop. As a team member (not the team lead), I started to feel that we needed a different way to decide than continuing to talk in qualitative terms. The Bayesian in me piped up: let's express our beliefs as probability distributions, and decide from there.

I'm pretty sure I'm not the inventor of this trick; someone out there has probably formalized it under a fancier name. But I did arrive at it that day while watching my team talk past each other, and it worked well enough that I've wanted to write it up ever since. I call it funny money allocation.

The mechanics

Here is what I did. I gave every person 1,000 points of funny money, effectively \$1,000 each, and laid out the ground rules:

There are three options on the table.
Allocate your 1,000 points across them however you like: all of it on one outcome, or any split you want.
You also don't have to spend all 1,000 points. Express that you're unsure by spending fewer points and keeping the funny money for yourself.

That last rule matters more than it looks. Keeping points is really an allocation to a fourth outcome: "I don't know." It's a legitimate epistemic state, and the design makes room for it instead of forcing it to hide inside a pick. In a vote, uncertainty gets collapsed into a choice. Here, it gets its own allocation.

One ground rule, baked into the setup, does all the work: your total allocation can't exceed 1,000 points. Words are unbounded; declaring "I strongly prefer option two!" costs nothing. Points are capped. The moment you have to take points away from one option to fund another, you have to rank your own convictions. And once you divide each allocation by 1,000, you have exactly what I was after: a probability distribution over the outcomes, one per person.

What the bets revealed

When the points came in, the exercise turned out to be very revealing.

I bet small: 70 points out of my 1,000, keeping the other 930. I genuinely wasn't knowledgeable about the underlying biology separating the outcomes, and my allocation said so out loud. Others bet big: 900 points on one option, leaving just 100 split between the remaining two. Where I was shrugging, they were leaning hard.

That 70-versus-900 spread was exactly the information we needed. Pool everyone's allocations and you get the team's collective distribution: which option had the most support, how concentrated that support was, and who held it. The people with the deepest expertise had put their (funny) money where their mouths were, and their bets helped us see very clearly where the team's preferences lay.

This is analogous to the kind of thing my former colleague Clayton Springer would have prescribed. His version: state your hypothesis in the form of a molecule, because that's concrete. Ours: state your beliefs in the form of a funny money distribution, because we were also trying to be very concrete. Either way, the point is to make your belief inspectable.

From bets to a decision

Beliefs expressed as probability distributions made real discussion possible. Once they were on the table, the argument shifted from "my adjectives vs. your adjectives" to "why did you allocate that way?", and the second question is much more productive. The exercise also surfaced something we hadn't articulated: there was more than one legitimate way to converge.

Consensus was the path we took. We talked through the distributions and rounded off some of the numbers to make the final call easier to act on and to fit our actual sample numbers. Summing up the numbers like this actually allows those with strong opinions to sway and influence the decision. And that's the point: with beliefs expressed as distributions, the decision maker can get creative. Another path is to have the person who needs to make the call treat the funny money allocation as a quantitative measure of belief, marry it with the qualitative beliefs that people have (the articulated reasons for their allocations and such), do some behind-the-scenes work, and come to the final conclusion. Done that way, everybody feels as if they've had their opinion heard, even if the outcome may not be exactly what they would have prescribed.

Why this beats voting

I think this trick has real advantages over an outright vote.

A vote collapses your belief into a single choice: one person, one bit. Funny money lets me express the strength of my conviction. When I want to say "I lean this way, but only slightly", I bet 200 points instead of 900. Degree of opinion survives the aggregation instead of getting flattened into a binary.

Funny money also separates confidence from preference, which voting mashes together. "Low confidence, mild preference" looks like 150 points on my preferred option and the rest in my pocket. "High confidence" looks like a big, concentrated bet. Those are genuinely different epistemic states, and they deserve different representations.

Abstention, or rather deferring to experts, is built into the design of funny money allocation. Choosing not to bet on one of the three outcomes, or choosing not to bet heavily on one of the three outcomes, is a visible and legitimate stance that says, "I'm willing to defer to the experts." You can participate honestly while being genuinely uncertain, and your honesty stays visible in the pooled distribution.

If you run one

A few practical notes, based on four or five successful runs and some reflection since:

Collect allocations independently before sharing them. Anchoring is real: once the first big bet lands, everyone else's distribution shifts toward it.
Have whoever spent the fewest points kick off the reveal, then build upwards to the folks with the most conviction, because anchoring will happen. Power dynamics matter too: anyone in a position of authority should go absolutely last.
Sum the allocations publicly, then talk. The pooled distribution tells you where the room stands; the conversation after the reveal is where the decision actually gets made.

The second point gets to the heart of the problem. A qualitative shouting match becomes a set of quantitative statements of belief, and the team can begin listening to why someone has a certain strength of belief. It's the same instinct as going Bayesian with your data analysis: qualitative hand-flagging of opinions, like qualitative hand-flagging of outliers, gives way to something more principled. Next time your team is going in circles, hand out a thousand points of funny money and see what distributions show up. I suspect you'll be surprised!

Learning how to learn anything, together, in the age of AI

2026-08-30T00:00:00Z

Dan Chen and I have a tradition: every year at SciPy, we find a corner of the venue, grab coffee, and nerd out about teaching, talking through how people learn, how to teach better, and what actually sticks. The past two years, the conversation has actually taken a pretty sharp turn. Students can now get every answer from AI the moment they sit down with an AI tool, so what is a teacher even for? After sitting with that question for a while, we soon realized the flip side was equally interesting: if someone is genuinely motivated to learn something new, how do they point AI at that goal for maximum leverage?

That flip is why Dan and I are running the Learn Anything with AI retreat: a five day, in person retreat from the 14th to the 20th of February 2027, with a small, handpicked cohort of people, in the United States. If you click on the link, you'll see the logistics and the details. I want to use this blog post to explain why I want to do this retreat. Here goes.

We opened Pandora's box

Here's the assumption we're working from: everything related to the boom in AI that we've seen over the past two, three years, they're here to stay. Pandora's box is open. I actually don't see a realistic path back to a world without AI, short of some catastrophe happening. So what's interesting for us as individuals is, what do we do with it? And given that AI is here for good, how do we thrive with it?

My answer, and Dan's answer as well, is the entire premise of this retreat: to get really good at learning how to learn anything, and how to use AI as the accelerator in that process. If used wrongly, AI will hand you answers without you learning anything. If used correctly, AI can help you build understanding. Because understanding still needs to be built in your head, and that part is the part that people are quietly skipping when they just ask AI for answers. So the people who learn how to learn, with AI's leverage, are going to compound what they know how to do over an entire decade and more. And the people who just collect answers are going to wonder why nothing's stuck. I want to spend five days with a small group of people to explore how we can use AI for the betterment of ourselves.

The cost of this really needs to be addressed; earlier this year I paid it. I was pumping out stuff using AI models, and by the end of each of those days my brain was fried. I couldn't remember anything I was doing! I was operating faster than the speed of thought, and so I was building what we would call cognitive debt: the equivalent of just cramming for an exam. You cram the inputs in, you skip the deliberate practice, and after the push, the bill arrives in the form of not remembering anything from what you've learned. The antidote is deliberate learning, where we learn it properly, we retain it, and we teach it. And as Dan and I studied and experimented with how to teach and how to learn, here's what we realized: AI can unlock new tools to help us with our understanding, and that turned out to be the real premise of the retreat.

Want to see the difference between the two paths? Drag the slider and watch what happens to what you remember.

The sea of change

The clearest way I can show you this premise is through my own history. I used to be a lab scientist. Then, after an advisor change during graduate school, I made the choice: I really wanted to move into computing. I knew I had a knack with computers, I just never had the space and time, and this was the perfect space and time to go and make that change for myself. So I flipped the switch on computing. I picked up Python on my own, picked up machine learning on my own, with a little help from some classes, picked up computational biology and the general methods, and soon taught myself network science through books and others' work. I also learned Bayesian statistics in graduate school, self-taught. Throughout that process, something clicked for me: my ability to learn those topics was greatly facilitated by knowing how to do programming. Once I understood programming, I could do math. It wasn't the other way around.

Most other people, most CS folks, probably get good at math first, and then programming comes as a natural consequence. For me it was the other way around, and I had to learn that about myself. So my ability to learn anything came partly from computing arriving in my life as a tool. But even then, I had some limitations. I couldn't build interactive visualizations for myself. I just didn't understand that paradigm, and building them took too much effort for stuff I wanted to learn, so I just didn't do it and had to find other ways around that limitation. Now with AI, I can have an HTML canvas and build these beautiful visuals that help me understand what's going on, in a Marimo notebook or inside an HTML page. AI can do that heavy lifting for me now. Computing was my first great learning accelerator, and AI is the second. That's the sea of change worth shouting about.

Here's what that first accelerator looked like up close. David Duvenaud taught me the backpropagation algorithm in 2016, and how it was really just the chain rule. I'd learned the chain rule in high school, but it didn't really stick for me until graduate school, where the chain rule could be written in Python with NumPy. And once you have the machinery in the form of autograd, you have a way of differentiating through almost any mathematical program you can think of and write. That was really fun. I had to re-derive backprop for myself, starting from linear regression, to really understand how it extends to a larger neural net model very naturally, just by chaining up more functions. That's all a neural net was: chained-up functions. And it was very tedious to get there. But nothing really stuck until I deployed my knowledge in the form of teaching.

When I committed to teach network analysis as a tutorial at the scientific Python conference, that commitment was the greatest accelerator of my learning journey. There was one crazy year where I did three tutorials, deep learning fundamentals, Bayesian statistics, and network analysis, and I had to solo two of the three because my co-presenters couldn't make it. It was chaotic, and it was a blast. The moment I agreed to teach each of those subjects, I found exactly which parts I had just been hand-waving and never pushed on with real curiosity. It's wonderfully uncomfortable, and it works! That's why the retreat ends with what we call a festival of teaching: everybody teaches what they've learned to the rest of the room, to get that first real rep in.

In person, on purpose

Beyond the cognitive cost, there's a social one: interacting with AI can be incredibly isolating, because you're sitting alone with a screen, asking questions, and not really interacting with another human. But the whole point of a learning community is other people. One guy I talked with about this retreat put it plainly: he lives in a tech hub, and whenever he goes to networking events, there's always a pitch. There's just no one to interact with in a normal way. So this room we're building, this group of people we're inviting, is essentially a space where you can nerd out with other people, and nobody's selling anything.

The format is intentionally human centered. We all physically stay together in a retreat center. There are lectures about learning theory, good pedagogy, and how people make things stick, but there's also enough space to duck out when you're peopled out; I'm an introvert, so these escape hatches matter. We're keeping the cohort really small, and we want it to hold a real diversity of knowledge fields. And sharing meals together, five days of shared hallways, is how hallway conversations compound into friendships, just like the SciPy hallway track did for me with Dan Chen. If we run this with overlapping groups, I think this community could grow into one of the more valuable things that any one of us has.

The shape of the week

Curious how the five days actually unfold? Step through the week below.

Days one through three are the learning science days, and they're Dan's home turf. I've known Dan Chen for years through the SciPy community; being a professional educator, he has read widely about the science of learning. He's taught multiple classrooms of data science master's students at the University of British Columbia in Vancouver, and he's deeply attuned to how people actually learn and retain stuff, the stuff we were never taught in grad school, or even undergrad. Interspersed throughout those days are sessions to put into practice what you're learning, and practical guidance on where AI slots into each step. As a small example: if you need to generate flashcards for retrieval practice, AI can help you with that. You can even get it to build adaptive systems for your own learning. We'll show you ways to build tools like these for yourself. Everyone arrives having picked a field that's genuinely foreign to them, and starts applying what the days teach to that field, deliberately.

Day four is more unstructured by design. This is where you home in on prepping to teach the topic you've spent the week learning: you take everything you've absorbed and build the teaching material you'll use to teach someone else the concepts, with plenty of space for creative freedom. And throughout the day, Dan and I will check in with each person once or twice, to see how things are developing and where you're stuck, acting as coaches to push and challenge you toward the day five festival of teaching.

Day five is the festival of teaching: twenty-some first reps, all in one day. This whole week is a microcosm of what I had to do when I was learning my way through new fields, and here's the goal of it: the retreat is about acquiring the skills needed to use AI to master a brand new field, and we'll spend that week acquiring them, practiced deliberately on a field of your choosing. Stack those skills consistently, month after month, even if you've got kids and a day job, and your capacity to learn compounds in a way that will massively accelerate everything else you take on.

One more source of inspiration for the retreat design: the book Who Not How. We always encourage you to find a who in your life who can act as the subject matter expert for whatever field you're learning, someone you can tap when you have questions, and who you trust to be the ultimate arbiter of whether you've really learned that field. You do want to approach that person with respect for their time. And AI can help you level up your knowledge to the point where you're engaging with that person as a peer, or close to it, rather than in a coach-mentee relationship.

Why I keep calling it a learning retreat

So really, this isn't an AI retreat, and it's not one of those feel-the-spirit kind of retreats either. It's a very practical retreat. The science of learning is the main content. And the main motivation is to bring people together: to build a group of people who are genuinely interested in improving themselves, as I've been able to taste over the past decade of teaching myself multiple things. That's why I keep calling it a learning retreat: the learning comes first, and AI accelerates it.

If this resonates with you, come take a look at the retreat. We're keeping it small on purpose and making it exclusive: not to a particular demographic, but to a type of personality. Dan and I really hope that we can see you there in February. I think the friends you make in that retreat might be the part that compounds the longest.

And I'd really love for you to take a small bet on yourself.

Before you go: five quick statements, and you be the judge of whether they sound like you.

Quantum Computing for the Probabilistic Bayesian

2026-08-07T00:00:00Z

This is a post I have been wanting to write for a while. As of 2026, I have been working much more closely with my teammate Alexey Galda, and part of that has been one-on-one tutoring on the foundations of quantum computing. (To be able to learn from someone well-trained, in a one-on-one setting, is a real privilege!)

Because I learn best by teaching, and because retrieval practice is the best way to make knowledge stick, I decided to write up what I have absorbed so far. I will readily admit that plenty of details are still beyond my grasp. Even so, I want to leave you with enough of a "fat marker sketch" of the ideas to reason about where quantum computing might genuinely help.

Here is the angle I am going to take. I have been doing Bayesian modeling for years, which means I have spent a lot of time thinking about probability distributions, sampling, and inference over spaces too big to enumerate. That turns out to be a surprisingly good lens for quantum computing. So that is how I am going to explain it: quantum mechanics as seen by a probabilistic Bayesian.

If you're ready for the ride, here we go!

The fundamentals

Start with the most fundamental object in quantum computing: the state of a single qubit. A one-qubit state is an arrow of length 1 in a two-dimensional space whose axes are the two basic states |0> and |1>. The arrow's coordinates along those axes are its amplitudes, which in general are complex numbers. The full one-qubit state space is the Bloch sphere, a unit sphere; when both amplitudes are real, the arrow's tip sits on a unit circle, one great circle of that sphere. That is the slice this post lives on: the signs you can see on it are the drawable case of the complex phases I am leaving out.

Drag the arrow above. Its projections onto the |0> and |1> axes are the amplitudes, and the squared magnitudes of those projections, |amplitude|², are the probabilities of measuring |0> or |1>. That squaring is the bridge into your world as a probabilistic Bayesian, and it is the lens we will use for the rest of the post. The new ingredient, with no analogue in classical probability, is that amplitudes carry phase, which on our circle is just a sign: the arrow can point anywhere on the circle, including where a coordinate is negative. That phase stays quiet for a single measurement, but it is the seed of everything quantum that follows.

Many qubits, exponentially many amplitudes

That picture was for one qubit, which lives in two dimensions. Add a second qubit and the state now has four basis states (|00>, |01>, |10>, |11>), so the arrow lives in four dimensions, with one amplitude per basis state. Add a third and you get eight. The number of amplitudes doubles with every qubit:

Number of qubits	Basis states (= amplitudes)	Examples
1	2	0, 1
2	4	00, 01, 10, 11
3	8	000, 001, 010, 011, 100, 101, 110, 111
n	2^n	All binary strings of length n

Play with the doubling below: one qubit gives two amplitudes, two qubits give four, three give eight.

This is the first place quantum computing leaves classical intuition behind, and it leaves fast. A 50-qubit register has 2^50 amplitudes, more than a quadrillion; you cannot write them all down, let alone store the state on a classical machine. Measuring collapses that whole arrow to a single bitstring, drawn with probability equal to its squared magnitude, the many-qubit version of what the unit circle showed.

We should clarify one assumption: in the interactive diagrams above, we treated the qubits as independent. The general n-qubit state is a single arrow in 2^n dimensions, and most such arrows cannot be split into one circle per qubit. That non-splitting is entanglement, which has its own section below.

Before we move on, we should make this picture second nature. Drag the circles until it is obvious how each independent qubit maps to one circle, how the joint amplitudes are the products of their coordinates, and how the probabilities factor. That clean correspondence is the foothold we will want when entanglement arrives and the one-circle-per-qubit picture stops working.

The probability view (coins)

Step back from amplitudes for a moment. If you only care about what a measurement will tell you, a qubit behaves like a biased coin, and a register of qubits like a pile of coins. That is the probabilistic-Bayesian lens I promised in the intro, and I will lean on it for the rest of the post; the widget below lets you flip 1, 2, or 3 and watch the outcomes follow the distribution. Just remember the coin view is a simplification, the shadow the amplitudes cast on a measurement, and it hides the signs that the interference section later puts to work.

Entanglement

Entanglement, in this framing, is when the state of one qubit cannot be described independently of another. When two qubits are entangled, their measurement outcomes are correlated, no matter how far apart they are. No information travels between them: each outcome is random on its own, and the correlation only shows up when you compare results over an ordinary classical channel. This non-local correlation is a key resource in quantum computing, and it has no clean classical analogue. For a probabilistic Bayesian, think of it as two coins whose flips look random individually but line up once you compare records.

Here is that idea made geometric. Two dials below set the qubits as if they were independent; the entanglement dial mixes in correlation. Watch the joint amplitudes stop factorizing, and the concurrence climb from 0 (independent) toward 1 (maximally entangled).

Interference

An amplitude's sign is easy to ignore, because a single measurement only sees the squared size, which is just a probability. The sign only matters when amplitudes combine: two amplitudes of the same sign reinforce, and two of opposite sign can subtract all the way to zero. That cancellation is interference, and it has no classical analogue. Classical probability distributions are built from non-negative numbers that pile up; quantum states are built from signed amplitudes that can interfere.

One tool that does the mixing is the Hadamard gate. It takes any input state and produces a specific output by blending the two input amplitudes together, with a minus sign baked into one of the blends. The widget below lets you drag the input arrow and watch the output respond. The butterfly diagram in the middle traces each input amplitude as it splits into contributions at each output, so you can see exactly where reinforcement and cancellation happen.

Drag the input to the diagonal (45 degrees, the |+> state) and look at the butterfly. Two contributions arrive at each output. At output |0>, both are positive, so they reinforce, and the output arrow swings hard toward |0>. At output |1>, one is positive and one is negative, so they cancel, and |1> vanishes. That is interference made visible: the minus sign killed one output while boosting the other. Now drag to other angles and watch how the balance between reinforcement and cancellation shifts. When only one input amplitude is nonzero (try |0> or |1>), there is only one path through the gate, so nothing cancels.

Two Hadamards in a row undo each other precisely because the minus signs cancel on the way back. On n qubits you apply a Hadamard to each wire; that is how a uniform superposition gets built in the first place, and how it gets recombined later so the signs can interfere.

From interference to answers

A quantum program is an arrangement of gates that concentrates amplitude on the qubit configurations representing the solution to a problem, and cancels the rest. The gates do not need to know the answer ahead of time; they encode a way to score candidates, and interference amplifies whatever scores well. The result is a probability distribution deliberately shaped so that the solution is likely and the noise has vanished.

If you are a Bayesian, this should sound familiar. Your prior and likelihood shape a posterior; a quantum circuit's gates shape amplitudes. Then, to read the answer out, you measure, and measurement is just drawing a sample from that shaped distribution.

And here is where it finally clicks for me, because in my own work building Bayesian models, sampling is the bottleneck. The core problem in Bayesian inference never changes: your posterior may be intractable, so you sample. But sampling is hard! Your chains may not mix, or there may be divergences, and if part of your problem lives in a combinatorial space, good luck with the approximations.

What if you didn't have to enumerate a combinatorial space at all, but instead let your sampler live in superposition over the whole of it?

That's the quantum computer's party trick. It isn't a magical exponential speedup for everything. But calling it an MCMC sampler goes too far; generic circuits evolve amplitudes coherently rather than stepping through a Markov chain, and true quantum MCMC algorithms are special cases rather than the norm. View it instead as a sampler purpose-built for exactly the kind of problem we already wrestle with, inference over spaces too big to enumerate classically.

The takeaway

In this blog post, we stripped away much of the physics of quantum computing to give you the following picture to walk away with. A qubit is an arrow on a unit circle, the real slice of the Bloch sphere; its coordinates are amplitudes, and their squared magnitudes are probabilities. A register of n qubits is a single arrow in 2^n dimensions, one amplitude per possible bitstring. Entanglement is when that arrow cannot be split into one circle per qubit. Interference is what happens when signed amplitudes combine through gates: same signs reinforce, opposite signs can cancel, and the quantum program uses this to concentrate amplitude on the configurations that represent solutions. Measurement is drawing a sample from the resulting distribution.

If you are a probabilistic Bayesian, the parallel is exact. Your prior and likelihood shape a posterior; a quantum circuit's gates shape amplitudes. You sample from your posterior; you measure from the quantum state. The quantum computer is not a faster laptop or a magic box. It is a device that is very good at exactly the thing we already struggle with classically: representing and sampling from distributions over spaces too big to enumerate.

Many details here are still beyond my grasp. But holding this mental model has already changed how I read about quantum computing. Where I used to see mysterious physics, I now see amplitude vectors, clever encodings, and interference doing the heavy lifting, all on hardware built for exactly that. I hope it does the same for you.

How AI turbocharges your learning

2026-07-27T00:00:00Z

I've been sitting on a question since SciPy this year. Daniel Chen and I get together every conference to talk about education, it's our shared obsession, and this time the big one was: AI shortcuts everything now. How do we show people the patterns for learning deeply with it rather than letting it replace the learning? I didn't have a good answer. So I went and read the research. This post is what I found.

The finding, repeated in every paper and every story: AI doesn't help or harm learning on its own. It amplifies whatever cognitive habits you bring to it. Bring good study habits and the tool multiplies them. Bring passivity and it multiplies that too.

None of the techniques are new. Retrieval practice, spaced repetition, elaboration, the Socratic method, they've been around for decades. What's new is that AI can finally run all of them for one learner at a time, at a scale no human tutor can match.

Before you read

Before you read on, I'd like to invite you to try these three, even if you're just guessing. Recalling first is what makes the rest of the post stick. Then keep reading. By the end, you'll know which of your answers the evidence supports.

The central distinction: amplifier or substitute

The cleanest version of the thesis comes from a 2025 study out of Turkey, run by Bastani and colleagues and published in PNAS. They split roughly a thousand students learning math into two groups. Both groups used GPT-4, with the same content and the same kids. The only thing that changed was the prompt.

The first group got plain ChatGPT, asked questions, and copied answers. It felt productive during practice. But when the AI was taken away for the final exam, they scored 17 percent worse than students who never had AI at all. They had used it as a crutch and never built the skill. The second group got a hinted, grounded tutor instead, one that withheld answers and asked leading questions. During practice they performed 127 percent better, and when the AI was taken away, they held their ground.

The authors call this the Dual-Mechanism Model. AI can amplify your thinking when the interaction forces you to do it, or substitute for it when you hand it over. The same tool, used two ways, produces opposite results, and the difference is always whether the AI does the thinking or you do.

Here's an example of that distinction. A student two weeks before her organic chemistry final types, "Explain SN1 and SN2 reactions to me." The model produces a beautiful explanation. She reads it, feels that she understands the material, but actually doesn't. The problem? Familiarity is not the same as understanding, and similarly, recognition is not retrieval. Two weeks later, walking into the exam, she can't tell the two mechanisms apart under pressure.

Flip the prompt and everything changes. Instead of "explain X to me," she types: "Here is what I think distinguishes SN1 from SN2. SN1 has a carbocation intermediate and unimolecular rate-determining step. SN2 is a concerted backside attack. What am I missing?" Now she's retrieving. The model can correct her, surface the edge cases she didn't mention (solvent effects, stereochemistry, rearrangements), and quiz her on them. She's doing the thinking, and the model is the infinitely patient examiner. That's the whole game in one prompt swap!

To see if you got the idea, try the little quiz below. Decide for each whether the learner is amplifying their thinking or substituting for it.

If you noticed the pattern, there's really only one question that matters: who's generating the thinking, and thus the words? Every study and every story below comes back to that.

What the evidence actually says

But that's one study. The harder question is whether the whole literature agrees, and on that we finally have meta-analyses. Four large reviews, published in 2025 and 2026, together covering well over a hundred studies and tens of thousands of learners.

On average, AI helps learning. Across 36 controlled studies and 7,229 participants, the effect size is g = 0.499, which sounds modest until you realize most education interventions hover near zero. This is medium-to-large.

But the average is hiding the interesting part. When the researchers dug into what predicted the biggest gains, one variable dwarfed everything else: how the AI was used. Collaborative, structured, Socratic use hit g = 1.026. Traditional, unguided use limped to g = 0.470. The teaching method wrapped around the AI is what produces that 2x gap.

A second meta-analysis, this one covering 89 studies, found that 40.4 percent of AI deployments helped and 23.6 percent hurt, with the rest mixed. The leading risk factor, named in 33.7 percent of studies, was over-reliance. And the most damning number in the whole review: 55.1 percent of studies used no specified pedagogical strategy at all. Much of what gets reported as "AI harms learning" is really "unstructured AI use harms learning."

To help you see the pattern for yourself, I collated the studies into the map below. Click any one for the design, the sample, and the effect size. Here's what to observe: scaffold the AI and the effects are large; leave it unguided and they flip.

Two studies in that map are worth slowing down for. One shows what a good scaffold buys you; the other shows what happens when you don't use one. At the strong end, a World Bank trial in Nigeria gave roughly 800 students free Microsoft Copilot alongside a six-week afterschool program, with tutors grounding the AI in the curriculum. The result was a 0.31 standard deviation gain, which the authors translate into roughly 1.5 to 2 years of business-as-usual schooling in Nigeria compressed into six weeks.

At the other end, the MIT EEG study gave students ChatGPT with no scaffold, no tutor, no curriculum. 83 percent couldn't correctly quote their own essay afterward, and brain connectivity was the weakest of any group in the study.

The cost of cognitive surrender

There's a name for what happens when you lean on the model a little too hard. Ethan Mollick calls it cognitive surrender. You stop thinking, the model drives, and because words are still hitting the page you feel productive. (You did produce a document, so the feeling isn't entirely lying to you.) The learning, though, left the room.

The MIT study is the one everybody reaches for when they want to say AI rots your brain. I sat with it for an afternoon, and my honest read is that the headline is stronger than the evidence. The brain-connectivity finding? Eighteen people in the final session. The "growing dependence" signal? Could just be practice. But the quote-your-own-essay finding is rock solid: 83 percent of ChatGPT users couldn't accurately reproduce the essay they'd just written. You outsource the writing, you don't even encode it. Hold on to that number.

There is a longer history here. In 2011, Sparrow and colleagues at Harvard found that people who expect to be able to look something up later remember it less well, a finding they called the Google effect. ChatGPT is Google with better prose. The cognitive offloading is the same, only smoother, which makes it more dangerous because it doesn't feel like offloading. It feels like thinking.

A 2026 systematic review of 44 studies on cognitive offloading names the mechanism precisely. Offloading lower-order work frees capacity, but the benefit only converts into deeper learning under two conditions: the learner must reallocate the freed capacity to harder reasoning, and the learner must have enough metacognitive awareness to notice that the capacity was freed in the first place. Without those two conditions, AI is a crutch. With them, it is a lever.

The mechanics: retrieval, spacing, and desirable difficulty

Three principles make AI tutoring work: retrieval practice, spacing, and the testing effect. None are new; all are decades old. What's new is that AI can finally run them for every learner at once.

The first principle, retrieval practice, has some of the strongest evidence in all of learning science. Roediger and Karpicke ran the classic 2006 experiment: students who took a brief test after reading recalled far more later than students who reread, even though the rereaders felt more confident. That finding replicates everywhere, from word lists to medical school curricula. Looking away and recalling produces retention. Rereading produces familiarity.

The second principle, spacing, is counterintuitive: letting a memory almost fade before you review it makes it stronger, not weaker. Ebbinghaus mapped the forgetting curve back in 1885, and the shape is steep. Memory drops fast after a study session. But if you reactivate the memory just before it would have slipped away, the effort of retrieving it builds a far stronger trace than rereading at full strength. That effort is what psychologists call desirable difficulty.

The third principle is the testing effect, and it's what happens when retrieval practice compounds. Each additional retrieval, especially under slight variations of the question, builds the memory further, in a way a single exposure never could. Every medical school Anki deck runs on this principle.

Want to see the forgetting curve in action? Pick a strategy below and watch what happens to retention over two weeks.

The UniDistance study is the cleanest field test of all three principles at once. Researchers gave 51 students a semester-long AI tutor (the MAGMA Learning app) that used a neural network to model each student's grasp of every concept, then served questions at each student's moment of near-forgetting, and visualized mastery as a 3D "learnet" whose brightness tracked grasp. Spacing, retrieval, and personalization, all automated.

The students gained up to 15 percentile points over the semester, and the model's predicted grasp correlated with actual exam grade at r = 0.81. The boring learning science works! AI can finally run all of it for each learner at once.

The prompt swap and its cousins

Up to this point, the focus has been on what apps and tutors do for you. Let's flip it: what can you do for yourself, with whatever model you're already using? Pick ChatGPT, Claude, or Gemini; it doesn't matter. How you talk to the model is what counts, and that skill outlasts any specific app.

Start with the prompt swap. It's the single highest-leverage move I know. Take any prompt where you'd have typed "explain X to me" and rewrite it as "here is what I think X is, am I missing anything?" You've just flipped the model from answer machine to coach, and forced yourself to retrieve instead of recognize.

You can push this further with a Socratic prompt. Tell the model: "Don't give me the answer. Ask me questions, one at a time, that lead me to discover it." The Socratic method is centuries old, but AI can finally run it for free for anyone with a phone. A 2025 Nature Human Behaviour paper found that this approach won by asking better questions, which activated what the authors call epistemic agency: the learner's ownership of their own reasoning.

Here's another one. Flip the roles: ask the model to be the student, not the teacher. "I'm going to teach you X as if you're a beginner. Stop me whenever I say something unclear or wrong." You can't teach what you don't own, and the model is a tireless beginner who won't let you wave your hands.

The ACTOR framework, from Sandeep Swadia, systematizes this for reading. Aim (state in one sentence why you are reading this). Compress (name the trunk of the idea tree before the leaves). Test (read to reject, not to agree; ask the model to challenge your interpretation and find the hidden assumption). Own (restate in your own words; connect to a real meeting, mistake, or person; teach it). Run (convert the idea into one decision, one rule, one checklist, one experiment). A book should interrupt your behavior, not just your beliefs.

These four moves are the toolkit. Next, let's see what happens when real people put them to work.

In practice: stories from people who did it

Tomerl1 set out to learn Rust in eighteen days by building a game, with Claude as a mentor and a hard rule: zero AI-written production code. He could paste code into the model, but he couldn't paste the model's code into his project. His summary, half a year later: "Any bug I fixed by pasting into Claude and pasting the answer back came back as a different bug a week later. The fixes that stuck were the ones I understood."

Next is Katy Saintin, an industrial software engineer who ran a two-window experiment. She opened Claude twice, side by side: one mentor, one shortcut. Only the mentor window taught her anything. But what stuck with me was her observation about her colleagues. In her world (SCADA, EPICS, TANGO control systems), junior engineers stay quiet because asking feels embarrassing. "The AI never laughed at a beginner question. Never rolled its eyes. Never made curiosity feel expensive." The real disruption might be simpler than we think: for the first time, millions of people feel safe enough to ask questions.

AI removes the friction of making practice exams, but you still have to take them. That's the insight Jordan Pierce took from scoring 96 on a finance exam whose average was 74, then 100 out of 100 twice. He had the model write neighbor-confusion distractors (wrong answers that look right), match the question length, build novel scenarios, and repeat any error until he'd fixed it three times in a row. The AI was a test-writing assistant. He was the one taking the tests.

The meta-skill

What's the single biggest predictor of how much AI helps you learn? It's you.

A 2025 meta-analysis in Educational Psychology Review, covering 29 experiments, found that the single largest moderator was the learner's prior self-regulation. High self-regulated learners gained g = 0.863; low self-regulated learners gained g = 0.284. What I want you to notice is the size of that gap. It's larger than the average effect of AI itself.

AI amplifies what you bring. If you already know how to study, it supercharges your study. If you don't, it mostly supercharges your ability to produce documents that look like studying.

Bloom's lower levels (remember, understand, apply) are commoditized. The model does them faster than you. Value has moved up to analyze, evaluate, create. The differentiator is what you do on top of having the information. The meta-skill that compounds is driving higher-order questioning and judgment.

So what's the counter to "brain rot"? Self-directed learning. The people who thrive with AI are the ones who drive the questioning the model won't generate on its own, check the output against their own understanding, and treat the model as a coach rather than an oracle. It's a habit of mind, not a feature you can toggle.

Check your understanding

Those three questions I invited you to try at the top? You now have the evidence to answer every one. Here's a five-question quiz to check what stuck.

Here's my parting advice: take the prompt swap. Next time you open a model to learn, tell it what you think and ask where you're wrong. The model won't be any different. You will.

The understanding zeitgeist in AI code review

2026-07-24T00:00:00Z

I've spent the year on both ends of agent-written pull requests. I've opened them after a long coding session, half-hoping the reviewer catches what I only half-tracked. And I've reviewed them, eyes glazing over a forty-file diff, typing "looks good to me" while a quieter voice asks whether it actually does. (Reader, I've been the problem as often as the solution.) If you've used a coding agent, you know the feeling.

This spring and summer, a bunch of people I read arrived at the same worry from completely different angles: now that agents write so much of our code, producing it is cheap, but understanding it is the expensive part. I want to map where the conversation has landed, so you can see its shape and find your own place in it.

One warning, and a promise. The worry has a name now, and the people working on it have built real techniques for it. I'm going to steal one of those techniques here, because Geoffrey Litt and Andy Matuschak both argue that reading is a weak way to build understanding; you have to be quizzed on it. So this post practices what it preaches. Three quick questions up front, to surface what you already think, and five at the end, to check what landed. (Yes, I'm quizzing you on a post about why quizzes work. I couldn't resist.)

Before you read

These aren't graded, and the cards won't tell you if you're right. Pick your honest answer for each, notice your assumptions, then read on. The survey revisits every one.

Hold your answers. Let's see where the field has landed.

Understanding is the bottleneck

Geoffrey Litt gave a talk at the AI Engineer conference in July 2026 with a thesis that names the whole conversation: understanding is the new bottleneck. Writing code got cheap. Holding a mental model of what the code does is the expensive part now, and it's the gap every pull request asks us to cross.

The deeper root of this worry predates the AI boom by decades. Margaret Storey, reviving Peter Naur's 1985 essay "Programming as Theory Building", puts it precisely: a program is a theory that lives in the minds of the developers, capturing what the software does and how it can change. Technical debt lives in the code. Cognitive debt lives in the developers' minds, and Storey argues it will paralyze a team before technical debt does. She watched it happen to a student team that could no longer make simple changes, because nobody could explain why the system worked the way it did.

This is the through-line of the whole survey. The people I've been reading disagree on tactics but agree on the diagnosis. When typing the code stops being the bottleneck, the bottleneck moves to whoever has to hold the theory.

What breaks at the pull request

The diagnosis bites hardest at the pull request, because that's the moment the theory has to move from author to reviewer. Several practitioners wrote the reviewer's side raw this year, and it hits closest to home.

Simon Højberg names the lived experience: code reviewers are "rapidly losing their minds" because they've become "the first layer of quality control instead of one of the last," picking apart hallucinated libraries and uncalled functions while the author shrugs that Claude wrote it. I recognize the feeling; I gloss over the diff, overwhelmed and bored, and approve whatever's there as long as CI is green.

Tim Schipper explains why agent PRs are uniquely hard to review, with a framing I've stolen more than once. An AI PR has no smell. It's uniformly polished, because polish is the one thing next-token prediction is genuinely good at, so the reviewer's instinct fires the wrong way: the surface looks more reviewable than a human's, so it gets less review. His test for whether you actually reviewed it is the sharpest line in the whole conversation: approve a PR you couldn't have written and can't fully explain, and you didn't review it. You witnessed it.

Here is what that looks like in practice: a clean diff, and the five checks a reviewer actually has to run.

Every checklist I read hits the same failure modes. GitHub's engineering team published guidance in May 2026 built around the insight that agents optimize for surface plausibility, not semantic correctness. Their data is striking: GitHub Copilot code review has processed over 60 million reviews, more than one in five code reviews on GitHub now involve an agent, and a January 2026 study found agent-generated code introduces more redundancy and quiet debt per change than human-written code. The playbook targets the specific ways agents slip: gaming CI to go green, reimplementing utilities that already exist, hallucinating imports that resolve to the wrong symbol, and writing tests that pass without verifying anything.

That last one is the most insidious. If the same agent wrote the code and the tests in the same run, the green checkmark is nearly worthless, because a model that made a wrong assumption in the code makes the identical wrong assumption in the test. The strongest heuristic is the same in every checklist: demand a test that fails on the pre-change behavior. If the agent can't write a test that would have caught the bug it claims to fix, the fix is incomplete or the understanding is wrong.

The cost of velocity without understanding

Why does this bite? Because agents let us ship changes that we'd normally have deliberated over for weeks in a matter of hours, and the mismatch between output speed and comprehension speed is where the debt compounds.

Mario Zechner, who built the Pi agent framework, put it bluntly in a post Simon Willison amplified. With an orchestrated army of agents there's no human bottleneck, and the mistakes compound faster than any one person would let them:

These tiny little harmless booboos suddenly compound at a rate that's unsustainable. You have zero fucking idea what's going on because you delegated all your agency to your agents. You let them run free, and they are merchants of complexity.

Willison's own takeaway is that cognitive debt is real, and that we now need a new balance of speed against mental thoroughness, since typing the code is no longer anywhere close to the bottleneck on writing software.

The cost shows up as burnout too. Steve Yegge describes the "AI vampire" effect: agents automate the easy work and leave you with all the difficult decisions, which is exhausting in a way raw output isn't, and he finds four hours of agent work a day a more realistic pace than eight. A Berkeley Haas study in HBR found AI intensifies work rather than reducing it, creating continual attention-switching and a sense of always juggling. Velocity without understanding produces brittle code, and it produces tired people.

Understand to participate

Litt flips the question here. If agents are getting better at verifying their own work, what's left for the human reviewer?

The tempting answer is to keep checking the code. But the people I read are converging on a different answer. We understand to participate. A project is hundreds of iterative loops with the agent, and the mental model you carry into the next loop is exactly what lets you come up with the next idea. Litt's phrase, which Willison picked out as the framing that resonated most, is "understand to participate." Verification is the part agents are eating. Participation is the part that's left to us, and it's the part that matters.

This reframe has old, deep roots. Litt closes his talk by reaching back to Alan Kay and Douglas Engelbart: the point of computing was always to augment, not just automate. Shan Carter and Michael Nielsen made the distinction sharp in a 2017 Distill essay: cognitive outsourcing hands a task to an oracle, while cognitive transformation invents new representations that expand the range of human thought. A reviewer who only checks correctness is doing cognitive outsourcing. A reviewer who builds a richer mental model is doing cognitive transformation, and that's the goal.

Techniques borrowed from teaching

If understanding is the bottleneck, how do we cross it faster? Litt's move is to stop trying to review harder and steal instead from a field that's worked on this for centuries: education. His talk lays out three techniques, and the first two are things an agent can produce while it reviews.

Explanations. Litt's /explain-diff skill turns a diff into a structured explainer: background first, then the goal and the intuition behind it, then the code walked through in a sensible order with prose around it. A typical diff hands you forty files in alphabetical order; a literate diff hands you a story you can follow.

Toggle between the two for the same change:

This is where reviewing big PRs with agents pays off most, since the bigger the change, the more the raw diff fails and the explainer earns its keep. Nested inside every explainer is a second device Litt borrowed from Andy Matuschak: a five-question quiz you answer before you approve. His rule is one I've adopted wholesale; he won't send code to a teammate until he can pass the quiz, and he does the same when he reviews theirs. The agent loop runs faster than the speed of human understanding, and the quiz is the mechanical brake that asks whether you actually get it. (This post is my own attempt at that brake; the widget at the end is the quiz, built from Matuschak's argument that "books don't work" and the spaced-repetition pattern he and Nielsen embedded in Quantum Country.)

Micro-worlds. This one's my favorite. Simon Willison arrived at the same idea independently and calls them interactive explanations. Instead of describing what a change does, have the agent build a little environment you can poke. Litt migrated his website between frameworks and couldn't review the agent's script, so he asked the agent to build a command center where he clicked through the migration step by step and watched the two sites side by side. Willison wanted to understand a word-cloud algorithm whose code only told him it used "Archimedean spiral placement," so he had an agent animate the placement, watching each word try a spot, overlap, and spiral outward until the idea clicked.

Shared spaces. The first two are solo techniques; the third is about understanding together. Litt's point is that when a team holds the same mental model they can communicate efficiently, jam, and riff, and without those shared structures the creative conversations stall. It loops straight back to Storey and Naur: the theory of the system is distributed across the team, and the review is one of the moments that theory gets rebuilt and shared.

Here's the thread across all three: agents can write code that helps you understand other code. Nathan Baschez adds a trick that works at the planning stage: ask the agent for two versions of every plan, one highly technical and detailed for itself, and one "entertaining essay designed to build my intuition" for you. The second version exists to hand you the mental model.

HUDs over copilots

These techniques imply something about what good AI review tooling should feel like, and Litt makes the distinction explicit in a separate essay. A tool that returns approve or request changes is a copilot, a virtual assistant you talk to. A tool that builds a micro-world around the change is a HUD, a new sense layered into your field of view, like spellcheck's red squiggles. Litt's argument, drawing on Mark Weiser's 1992 rant against the agent metaphor, is that we have enough copilots. We need HUDs!

Practitioners arrive at the same place: the human role in review is narrowing from correctness verification to context verification, judging whether the agent understood the codebase, the contract, and the intent. Automated review owns the mechanical first pass, and human attention goes to the semantic judgment agents can't self-check. Copilots automate the verdict. HUDs extend your senses so you can make that judgment yourself.

The zeitgeist in one map

Here's the whole conversation compressed. Each one I read contributes a single move to the argument.

Source	What it contributes
Geoffrey Litt, "Understanding is the new bottleneck"	The diagnosis; the understand-to-participate reframe; explanations, micro-worlds, shared spaces
Margaret Storey / Peter Naur	Cognitive debt lives in minds; a program is a theory the team holds
Simon Willison	"Understand to participate"; interactive explanations as a technique
Simon Højberg	The reviewer's lived agony; gloss and the first-layer-of-QC problem
Tim Schipper	"An AI PR has no smell"; witnessed, not reviewed
GitHub engineering, May 2026	The agent-PR volume data; the failure-mode playbook and the failing-test heuristic
Mario Zechner / Steve Yegge / HBR	The cost: merchants of complexity, burnout, intensified work
Carter & Nielsen, Distill 2017	The deep root: augment not automate; cognitive transformation over outsourcing
Mark Weiser, 1992	HUDs over copilots

Read down the table and you can see the conversation's shape: a diagnosis (understanding is the bottleneck), a symptom (reviewers witness instead of review), a cost (debt and burnout), a reframe (understand to participate), a toolkit (explainers, quizzes, micro-worlds), and a design imperative (build HUDs). My take is simple, and it's the through-line of how I work: lead with curiosity, demand artifacts you can poke, and use these techniques to keep understanding alive when the agents are typing fast. I wrote about the curiosity piece recently, and it's the same instinct here.

Check your understanding

Here's the quiz. Five cards, instant feedback, and a live score. This is the speed-regulator technique, turned back on the post itself.

If you missed any, that's the spot where your mental model and the field's are still diverging, and it's exactly where cognitive debt would quietly accrue. Ask, find the gap, close it, and ask again. That loop is the whole point.

Going Bayesian automates your manual data analysis

2026-07-23T00:00:00Z

If you've ever spent an afternoon clicking through dose-response curves, flagging outliers by hand, there's a way to make the model do that automatically.

People usually talk about Bayesian modeling in terms of richer uncertainty estimates, prior knowledge, and posterior distributions. All true, and all worth the switch on their own. But there's a benefit almost nobody talks about. Going Bayesian automates away the manual data analysis you'd otherwise do by hand. And once you experience it, you wonder how you ever worked any other way.

Let me show you what I mean.

The manual outlier dance

Picture a cell viability assay lab. Each 96-well plate holds five dose-response curves. An eighteen-plate experiment generates ninety curves in a single run, and your group is already eyeing robotics and 384-well formats to multiply throughput further.

I've seen workflows where a single study generates anywhere from 125 to 375 curves. Each one gets inspected by hand.

For every curve, there's usually a point or two that looks off. The standard workflow goes like this: copy raw data from the instrument output, paste it into Excel to line up sample names with plate positions, then copy again into GraphPad Prism to plot and fit the curve. Spot a deviating point, hit a shortcut key to mark it as an outlier, exclude it, refit. Then look again. Maybe another point looks suspicious now. Repeat. And pray you didn't misalign a paste somewhere along the way.

For one curve, it'll be seconds, but multiply this over dozens of plates and you have an unbounded amount of time spent clicking on a UI (if one's lucky) or switching between two applications, Excel and GraphPad Prism (if one is unlucky).

Are you really going to hire more research assistants to click through curves? That approach collapses under its own weight.

Now, a quick but important distinction. There are two completely different reasons to exclude a data point.

The first is a documented physical cause. You saw a bubble in the well. The pipette tip fell off. The reagent looked cloudy. These are data integrity issues, and you can justify removing them before you look at any numbers; in fact, you should document these in your lab records.

The second reason is that a point is statistically inconvenient. It sits too far from the curve and inflates your error. That second reason is the one I want to talk about, because it's both common and statistically broken.

Two ways exclusion corrupts your results

Excluding a point because it's statistically inconvenient has two deep problems.

The first is reproducibility. What's your rule for calling something an outlier? Two standard deviations from the fit? Three? The classic tools, like Grubbs' test or the ROUT method that Prism builds in, hand you a threshold. But where did that threshold come from? It's a cutoff someone picked, not a principle someone derived. Two analysts staring at the same curve flag different points, because the decision rests on an arbitrary line in the sand. You're eyeballing with extra steps.

Here is curve 11 from the demo above, reviewed by two different analysts. Dilution 4 reads 100%, which looks like it belongs to the upper plateau. But the transition zone expects something closer to 73% at that dilution. Analyst A keeps it; Analyst B removes it. Toggle between them and watch the ID50 shift.

Same data, same software, different ID50. The point at dilution 4 sits right at the inflection of the curve, so keeping or removing it swings the estimate of where the curve crosses 50%. Which analyst is right? Both think they are, and without a principled criterion, neither can prove the other wrong.

The second problem is subtler, and it's the one that actually corrupts your results. When you run a test like Grubbs', the test assumes your data come from a clean normal distribution with at most one contaminating point. But dose-response data violate this assumption doubly. The raw points along a dose-response curve don't come from a single normal distribution at all; they follow a sigmoidal shape. You'd have to apply Grubbs' to residuals after fitting, but those residuals are often heteroscedastic (more variance at certain concentrations than others), which breaks the test's assumptions even further. With only five to eight dilutions per curve, the test has almost no power to distinguish a genuine outlier from noise. So you find the outlier, remove it, then refit your curve as if that contamination never existed. Your uncertainty estimates now lie to you. They don't account for the decision you already made. You've baked the removal into your results without propagating the uncertainty that choice introduced.

This isn't just my opinion. Karch demonstrated in a paper literally titled "Outliers May Not Be Automatically Removed" that automatic removal invalidates confidence intervals and biases your estimates (Karch, 2023). The APA Publication Manual itself states plainly that omitting troublesome observations to present a more convincing story is prohibited (APA, 2020). And the FDA's Bioanalytical Method Validation guidance is even more direct: repeat analysis is acceptable only for assignable causes such as equipment failure, sample processing errors, or poor chromatography. "It looked like an outlier" is conspicuously absent from that list.

Down-weight, don't delete

Here's where Bayesian modeling changes the game. Instead of fitting a curve and deciding afterward which points to throw away, you choose a likelihood (the part of your model that scores how well the data fits) that handles outliers from the start.

The normal distribution, the bell curve you know, has thin tails. Probability drops off fast as you move away from the center. A single point sitting far from the curve becomes astronomically unlikely under a normal likelihood, and the only way the model can reduce that penalty is to drag the entire fit toward the outlier. One bad point yanks your curve.

The Student-t distribution fixes this. It looks like a normal bell curve, but with heavier tails. More probability spills into the extremes. So when a point sits way off the curve, the likelihood essentially shrugs and says, "that's rare, but not impossible," and the fit barely moves. The further the point drifts from the trend, the less pull it exerts. Its influence fades gradually rather than being a binary in-or-out decision.

Here is the elegant part. The Student-t has a parameter called the degrees of freedom (nu) that controls how heavy the tails are. At nu=1, you get the canonical Student-t distribution with the heaviest tails; outliers barely register. As nu increases, the tails thin out, and the distribution converges toward the Normal. At nu=30 or above, they are nearly indistinguishable. You can put a prior (your belief about the parameter before seeing data) on nu and let the data tell you how much outlier tolerance each dataset needs. The model learns its own robustness.

And critically, the point is never deleted. It still contributes to the fit, just with proportionally less weight. Your data stays intact.

Enough theory. Let me show you what this actually looks like on a dose-response curve with a real outlier.

The blue curve is the Normal likelihood fit; the orange dashed curve is the Student-t. Dotted vertical lines mark each model's ID50 estimate. Shaded bands are 94% credible intervals from 4,000 posterior draws (PyMC/NUTS, 4 chains). The buttons go from nu=1 (the Student-t at its heaviest-tailed, most forgiving of outliers) to nu=30 (nearly indistinguishable from the Normal, least forgiving).

Here is what to watch for. At nu=1, the outlier is absorbed by the heavy tails; the orange curve and its ID50 stay put, right where they belong. At nu=30, the orange and blue curves nearly overlap; the distribution has thinned into a Normal, and the outlier drags the ID50 leftward just like the blue curve does. The interesting territory is nu=3 to nu=5, where the model starts caring about the outlier but doesn't let it dominate. That is the sweet spot for most assay data.

But this demo handles one weird point on one curve. What happens when an entire curve is the outlier?

Hierarchy handles the weird curves

Heavy tails solve the point-level problem. But some curves are weird all on their own. Maybe one dose-response curve came out noisy across every well, or the shape is off in a way that suggests a systematic issue for that row.

This is where hierarchical modeling earns its keep. Instead of fitting each curve in isolation, a hierarchical model fits all of them together. Each curve borrows strength from the others through a shared population distribution. A noisy curve with a couple of strange wells gets gently pulled toward the behavior of its neighbors. Statisticians call this partial pooling.

The philosophy mirrors the heavy tails, just one level up. Heavy tails down-weight weird points. Hierarchy regularizes weird curves. It's the same idea, merely applied at different scales.

Six curves from the same experiment. Five came out clean; curve 3 had a bad day with noisy wells. Toggle between independent fits (curve 3 flails) and hierarchical fits (curve 3 borrows strength from the population and snaps into shape). The faint dashed line on curve 3 shows where the population says it should sit.

This is what automation looks like

Here's where everything converges. Once you write down your hierarchical model with a heavy-tailed likelihood, you fit it to all eighteen plates at once. The sampler runs. Heavy tails handle the outlier points. Hierarchy handles the outlier curves. The whole thing runs without a single human judgment call.

I built a prototype of exactly this for a cell viability assay workflow.

Thirty-six plates. Four minutes. On a laptop.

How would you feel about that instead of spending your afternoon doing it by hand?

No shortcut keys. No research assistants clicking through Prism. No two-analysts-flag-different-points problem. The model handles the data analysis decisions you used to make by hand, and it makes them in a principled, reproducible, mathematically grounded way.

This is the hidden benefit of going Bayesian. People talk about richer uncertainty, prior knowledge, posterior distributions. All genuine advantages. But the automation of manual data analysis is the one that compounds as your experiments scale. Every new plate, every new study, every new operator, the model just handles it. You get your afternoons back, and your results become reproducible by construction.

If this resonates, find a statistician or data scientist who works with Bayesian tools. Show them your curve-fitting workflow and ask: "Can we use a hierarchical model with a Student-t likelihood for this?" If they know Bayesian modeling, they'll know exactly what you mean.

Going Bayesian isn't just about better statistics. It's about building a data analysis pipeline that scales without breaking, handles messy data without manual intervention, and gives you answers you can trust as your experiments grow. The math does the work so your team can focus on the science.

Curiosity at the wheel

2026-07-21T00:00:00Z

I took part in Round 2 of the alphaXiv x marimo competition. The task was to pick a recent ML paper, build a marimo notebook that teaches it, and submit a five-minute walkthrough. I had two papers I wanted to tackle, BitNet's 1.58-bit weights and JiT's "predict the clean image, not the noise," and a tool I've been enjoying: pair-programming marimo notebooks with a coding agent.

The real surprise was how much of the paper I learned by the end of the sprint. I came out the other side understanding both papers more viscerally than if I'd read them quietly for a week. The mechanism was the artifact we were building together. The notebook gave me something to poke at, and poking taught me what reading alone couldn't.

Judgment is the slow part

Judgment takes years. It's the feel for when a result is wrong, the instinct for which experiment to run next, and the sense that a curve looks fishy before you can articulate why. Those instincts compound slowly, and no agent hands them to you. I wrote about this recently, and I still believe it.

But there's a prerequisite elementary step: onboarding onto the basic vocabulary and mental models of a new paper or field. Things like wrestling with what "straight-through estimator" actually means in your hands, or making sense of why predicting the clean image might be easier than predicting the noise, when the established literature screams the opposite. That part has always been slow for me, because it's a wrestling match with jargon. I read a sentence, hit a term, go look it up, lose the thread, reread, repeat.

The onboarding can go faster, even if the judgment can't.

Aesthetics do pedagogical work

Before I started building, I had the agent study the three winning notebooks from round 1. Partly this was competitive intelligence, to obtain a clear bar for what good looked like. But there was a separate surprise: I genuinely enjoyed reading them! The best ones opened with an HTML hero banner carrying the paper title and a one-line thesis. They paired that with interactive widgets where you dragged a slider and watched a distribution react, which made the concepts tactile. Some of them were also game-like. You'd try to find a hidden structure, and being wrong was part of the play rather than a verdict. And they closed with a summary callout that re-stated the conclusions cleanly.

What makes these notebooks teach is the loop they put you in: predict, then see whether you were right. The interactive widgets drive that loop. Everything else exists to support it: a hero banner gives you a mental anchor before the math lands, and game-like elements turn being wrong into play, which is where the learning actually lives. When being wrong costs nothing, you try more things, and trying more things is what teaches you.

Pair-programming the paper, not just the code

I fired up a marimo notebook on a molab sandbox with an NVIDIA RTX PRO 6000 Blackwell, and pair-programmed with GLM-5.2 through opencode. I had the paper and the questions, but I didn't have the patience or tenacity to push through the initial energy barrier on my own. The agent had both, in infinite measure.

The shape of the work looked like this. I'd ask a question, sometimes framed as "explain this to me like I'm five." The agent would answer in plain words, and then build something that let me feel the answer. For BitNet, that meant a weight heatmap crushing 32-bit floats into three colors, a slider that let me drag the precision and watch accuracy react, and a live training race on the GPU between binary, ternary, and full-precision models. For JiT, it meant a manifold game where I dragged noisy points and watched what "predict clean" versus "predict noise" actually did to the targets.

Here's the thing about ELI5 as an operating mode. When I asked for it, I stopped wrestling with jargon and started wrestling with concepts, analogies, and ideas. That's the match I actually wanted. The jargon-wrestling was always fake difficulty, syntax I had to look up before I could even formulate the real question. ELI5 crushed the syntax barrier and put me in front of the idea immediately.

The artifacts taught me what the abstract couldn't

The clearest example came halfway through the BitNet notebook. My mental model of the paper went something like: take a pretrained language model, crush its weights to {-1, 0, +1}, and it still works. That's the one-line version, and it's what you'd believe if you stopped at the abstract.

I told the agent to demonstrate it on a real pretrained GPT-2. The perplexity went from 51 to 33,000. The output collapsed to "present present present present."

But that's not what the paper claims! BitNet models are born ternary. They're trained from scratch with the straight-through estimator, which lets gradients flow through the rounding step during training. Retrofitting ternary onto a model that spent its whole pretraining life tuning exact magnitudes breaks the calibration of every layer at once. The agent showed me, by running the experiment I asked for and getting a result that contradicted my hunch.

So I asked the obvious follow-up: "If we can't retrofit, what's the fix?" That curiosity became the notebook's novel extension. Fine-tune the retrofitted model with the same straight-through estimator, on a small AI-domain corpus, and see what happens. Across three model families (distilgpt2, Qwen2.5-0.5B, Pythia-410m), perplexity recovered by about two orders of magnitude. The retrofitted text went from repetition to real on-topic English about artificial intelligence. The mechanism that makes BitNet trainable from scratch is also the mechanism that partially rescues a retrofitted model. That extension exists because I was wrong about the abstract first. The agent's experiment surfaced the contradiction; reading alone, I could have stayed wrong indefinitely.

The second paper, JiT, had its own version of this. The paper's title is "Back to Basics: Let Denoising Generative Models Denoise," and I couldn't find a quantitative denoising benchmark inside it. So I got curious. We trained a small x₀-predictor on CIFAR-10, then used it as a one-pass denoiser on held-out test images. It improved the peak signal-to-noise ratio by up to 22 dB, beating a Gaussian-blur baseline at every noise level. The conceptual payoff was even cleaner: denoising and generation turn out to be the same operation, both projection onto the data manifold. That extension lives in the notebook now, and it exists because I asked "what does the title actually mean in practice?" and the agent helped me build the answer. (This JiT notebook is the one that ended up winning Round 2.)

Five minutes forced clarity

The competition required a five-minute walkthrough video, and that constraint turned out to be its own learning tool. Five minutes forces you to decide what actually matters. Combine that with ELI5 and you get a forcing function: explain the paper simply, in spoken English, in five minutes, to a viewer who hasn't read it. If you can't, you don't understand it yet.

The agent drafted a script structured as a guided tour through the cells. I sent it to a panel of other subagent reviewers and they tore it apart. Repetitive, shallow, vague.

That review was a gift. Rewriting the script with the agent's help forced me to decide, in plain spoken English, what each section actually taught and why it mattered. Every time I hit a sentence I couldn't say cleanly, that was a gap in my own understanding. Vague claims like "it works well" had to survive a real question like "compared to what, at what scale?" Oversells had to be walked back to what the data actually supported. Limitations I'd been quietly hoping no one would ask about had to be named out loud, because a judge (or at least, my thesis committee) absolutely would.

Using the agent to help me communicate clearly made the script much better, and making the script better made my understanding better. Writing in my own voice, with the cells in front of me, is where the last misunderstandings surfaced. Rehearsing for the presentation and trying to anticipate questions about each cell forced me to be intellectually honest about my own gaps. Reading alone makes it easy to hide those gaps from yourself.

I stayed in the driver's seat

This is the part I want to be careful about, because it's easy to misread. The agent did a lot of the typing. It wrote the training loops, built the custom anywidgets, ran the parameter sweeps, refactored the underscore-prefixed names I hate. If you'd watched the session from the outside, you'd have seen the agent producing most of the code.

But I was driving. Every prompt I sent encoded a decision. The agent could type, but it couldn't make the judgment calls. Recognizing that the collapsed retrofit was a problem worth solving was mine, and so was seeing that JiT's blurry generations were a real flaw to fix, not just how diffusion looks (hence my prompt: "figure out how to solve the image blur problem on a loop until you are sure you have solved it"). And catching that the script oversold the recovery took a feel for what honest claim the data would support, which is the kind of judgment that takes years.

The agent's speed amplified my judgment, exactly the way I wrote about before. It compressed the onboarding loop, from a week of reading and rereading into an evening of asking and building and poking. The years I'd spent building judgment finally had leverage.

Your curiosity sets the direction

Your curiosity has to set the direction. The agent is patient, the agent is fast, the agent will happily build whatever you ask for and a dozen things you didn't. The discipline is yours. Decide what you actually want to understand. Ask for it in plain words. Demand artifacts you can poke, not paragraphs you can read. When the artifact contradicts your hunch, follow the contradiction; that's where the learning lives. And when you think you understand the whole thing, write the five-minute spoken script. The gaps will announce themselves.

Some things genuinely take time; onboarding doesn't have to be one of them. Keep your hands on the wheel.

Choose your own AI policy

2026-07-20T00:00:00Z

I just got back from SciPy 2026, my eleventh year at the conference. The loudest thread across the week was AI in open source, and what maintainers are supposed to do about it.

The thread surfaced in a Scientific Python community session, run as a Birds of a Feather (BoF). Stefan van der Walt walked us through the project's work: the specification process (SPECS, the Python-community analog of PEPs and NEPs), four developer summits, shared tools like RepoReview and spin, and MyST as the documentation engine behind the next Jupyter Book. Then the floor opened for Q&A.

Someone raised a worry I haven't been able to shake. As Claude and its cousins make it cheap to spin up a working tool, maintainers build bespoke utilities for themselves and stop contributing back. The room offered an analogy: if everyone builds their own vehicle, the roads stop working. Someone drives a tank, someone else a bicycle, and the shared infrastructure collapses. The cheap-prototype problem makes it worse: the version a person publishes often only works for them, pushing the next person to build their own.

That conversation was still on my mind when a pull request hit my review queue that week, giving me a cleaner frame for these questions.

Two flavors of AI in open source

Henry Schreiner opened PR #821 on scientific-python/cookie to add an AI page to the cookie project, the Scientific Python community's recommended project template. (Henry started the draft by pointing Claude Opus at the issue and reworking the output by hand, which is exactly the meta-lesson of the page itself.) The page opens with a distinction I believe is useful here:

A developer driving an interactive AI harness with a capable model, reading the output, and taking responsibility for the result.
Low-cost models running unattended in automated systems that mass-produce pull requests.

The first is a power tool. The second is slop. Most maintainer frustration about AI contributions comes from the second case, and most of the value comes from the first. Conflating them produces bad policy.

I see this conflation constantly. A contributor opens a pull request with a generic "I improved the code" message, the diff is a wall of token churn, and when you ask a question the answer comes back in the same flat model-voice. That is slop with a human attached. The human is the relay, not the author. Reviewing it takes longer than writing the change myself.

The first flavor is what I want more of. A contributor who uses an agent to engage with my project, reads the result, and defends every line, is using a tool. I want to encourage them. Encouragement only lands if contributors can tell which flavor you're asking for, which means writing it down before they open a PR.

Write your stance down

The heart of Henry's PR is AI_POLICY.md, a project-level file that tells contributors what you expect. The PR sketches three tiers you can adapt, and a community-maintained list of real policies from dozens of projects shows what other maintainers have chosen.

All in. AI-assisted contributions are welcome on the same footing as any other, as long as they meet your quality bar and are disclosed.

Moderate. AI assistance is fine, but the contributor has to show real human involvement and prior buy-in before opening the PR.

Minimal. AI-generated PRs are discouraged or restricted. Use this if you have limited review capacity.

Which one fits depends on the project. The point, however, is that you should pick one. A contributor reading your AI_POLICY.md should know what to expect before they open a PR, and a maintainer closing a PR should be able to point at the policy rather than re-litigate it every time.

The cookie PR leaves the policy out of the template by default, letting each project select and modify their own. I think that's right. A project with one overworked maintainer is in a different situation from a project with twenty regular contributors, and the policy should reflect that. The maintainer reserves the right to take whatever action fits the project; nobody is paying for this work, which means no contributor is owed a review.

Engage with the human, not the bot

The policy also has a recommendation for how maintainers should treat contributors, and my favorite passage is tucked inside a callout:

If you maintain a project: try to engage with the human. If they are willing to interact (and not just type "address the review" into their harness), treat them like a human, even if you also see the AI working on their behalf. They also may use AI to address a language barrier.

The failure mode of an AI-assisted contribution is the human tagging out, which is the power-tool flavor sliding back into slop. A contributor pastes review comments into their harness, pastes the harness's reply back into the PR, and walks away. The maintainer is now arguing with a bot. That's draining, and it's disrespectful of volunteer time.

The policy handles this directly with one rule: keep human review human-to-human. If an AI is responding on your behalf, say so, with a disclaimer line at the top of the comment. You're accountable for every change you submit. Don't make a maintainer talk to an AI without knowing it.

Disclosure norms pay off here too. The PR recommends the Linux kernel's Assisted-by: <harness>:<model> trailer for commits. A convention I first saw Daniel Chen use is the :robot: _AI text below_ :robot: line for any AI-generated prose in a PR or comment. Knowing which model was used lets a reviewer run a different model family to cross-check the work (the PR calls this the "rubber-duck" technique). It also just respects the maintainer. Hiding your process when you contribute to open source is a small lie, repeated. It stings because it withholds the context a reviewer needs to do their job.

Context is the bottleneck

Both worries, fragmentation and slop, trace back to the same constraint: the cost of producing code dropped faster than the cost of producing context. When anyone can ship a working prototype in an afternoon, the scarce thing is the understanding of what the code should do, why it should do it, and how it fits with what already exists.

This shows up in every issue triage. The issue that says "the build is broken" with no logs, no repro, no environment info. The PR that addresses a problem the contributor never confirmed anyone had. The agent-generated patch that is locally correct and globally wrong because nobody told it about the other module that depends on the current behavior. In each case the gap is the same: context that lives in someone's head and nowhere in writing.

What helps is documentation written for an agent to read. The cookie PR includes a section on AGENTS.md, the cross-harness standard file that captures how your repository works: preferred command runners, architecture notes, conventions, traps. A good AGENTS.md makes the AI far more effective, because the AI stops guessing. The same instinct applies to issues: write down what you tried, what you expected, what you saw. The agent working from a thorough issue is a different animal from the agent working from a one-liner. Agents are also good at producing that context. A minimal reproducible example is tedious for a human; for an agent that can spin up the environment, run the code, and iterate until the bug fires, it's a short task. Your AI_POLICY.md can ask contributors to use their agent to produce a self-contained repro first. The maintainer gets something runnable; the contributor finds out whether the bug is real before anyone else spends time on it. On the PR side, prompts that shape agent behavior belong in AGENTS.md, which travels with the code and reaches every contributor's agent: ask for the smallest diff that fixes one thing, and for stacked PRs when a change needs more. AGENTS.md itself is documentation written for an agent.

AGENTS.md captures mechanics. Intent is a different kind of context, and it matters just as much. Thomas Caswell's SciPy keynote traced the story of Matplotlib's design decisions, the kind of narrative that lives in maintainers' heads and leaves with them. Writing that story down as design docs (a high-level design for the project, then low-level designs and EARS requirement specs per feature) gives an agent the intent it needs to stay grounded. The docs will shift as the project evolves. That malleability is fine, because the current version still beats an agent guessing.

Later in Q&A, someone who works with NASA data raised the same point from their own side: they had started asking whether to write their documentation for a chatbot instead of a human. They had started shipping documentation as markdown alongside the rendered HTML, because the markdown is what the chatbot consumes. The rendered page is for a human; the source is for the agent. The lesson generalizes. If you want agents to do good work on your project, give them something to read.

Be courageous

Documentation is what you write for the agent; your AI policy is what you write for your contributors.

If you maintain an open source project, pick an AI policy from any of the three tiers. Write it down, commit it, link to it from your contributing guide, and enforce it. The action that fits your project is correct. A solo maintainer with no review capacity is within their rights to say "no unsolicited AI PRs, full stop." A healthy project with twenty contributors and review bandwidth can be all-in, and benefit from it. Both are legitimate. What's undesirable is having no policy, drifting, and burning out.

That burnout lands on you. The action that fits you is also legitimate. If engaging with AI-assisted contributors drains you, require prior buy-in, close the PR, or step back from the project for a week. Some projects are solely volunteer. Protecting your capacity is part of the job.

Protecting your capacity and welcoming contributors go together: hold the line against tag-out bots, and give your time to the humans behind real contributions. When a contributor shows up willing to engage, even when their harness did most of the typing, treat them like a human. Read their PR, ask your questions, and teach them the parts they don't know yet. That is how open source onboards people. Thomas Caswell onboarded me to Matplotlib a decade ago by walking me through seventy pull requests over two months. The first four were painful for him, but he did it anyway, and I'm still here. That people connection is invaluable.

That human connection is what got more valuable this year. The cost of writing code dropped. The value of a maintainer who knows their project, writes down their stance, and engages with the humans who show up rose, because agents amplify whoever brings the judgment. That's the work. Go do it, and don't apologize for the policy you choose.

xarray in biology, take 2

2026-07-19T00:00:00Z

One year ago at SciPy 2025 I wrote a post arguing that laboratory data should live in a single xarray Dataset, with sample IDs as the shared coordinate system. The spark for that post was Ian Hunt-Isaak's SciPy 2025 talk on xarray across biology. The hunch back then was this: store measurements, features, model outputs, and train/test splits in one labeled n-dimensional container, and the index-matching tax disappears.

At this year's SciPy 2026, I closed the loop! During the tutorial, I wrote an xarray-linked-data skill for Ian's scientific-python-skills repo. That hunch from a year ago has now become an agent skill, with Marimo notebook examples to steer agents in the right direction on novel situations. Developing the skill has clarified where the thesis holds and where it still needs work, and that's what I'd like to document here.

The thesis, restated

A biological data package should be one file that is:

slice-able by molecular entity. Select one molecule_id, and every assay that molecule appears in comes along for the ride.
indexed straight into other datasets. Selecting by concentration auto-constrains time to that concentration's administration window, and vice versa.
storing Bayesian estimates next to the raw lab data. For example, a posterior KD summary lives in the same file as the ELISA OD readings it was derived from.
a single artifact. One .zarr or .nc file that you can ship, share, and re-open three years later without a README.
self-documenting. It should have metadata attached to it without needing to reference other sources.
built on a DataTree. Real assays produce data on incompatible grids; DataTree gives each assay its own node while they share molecule_id through inheritance.

The end goal behind all six bullets is this: to equip one data scientist to curate data packages across an entire campaign (or several at once). The agent skill carries the mechanical assembly and the coding agent runs it. Throughout all this, judgment stays with the human who has seen the warts before.

Why `DataTree`

It took me a year to realize that a single Dataset was not enough. In last year's example, I used one dataset, and it worked because the example was a relatively benign one. The microRNA study had expression, ML features, model outputs, and splits all sharing the molecule axis. Everything was compatible.

Real biology rarely is. A nanoparticle characterization campaign produces data on fundamentally incompatible grids:

DLS can give you a 100-point time series per molecule per replicate (hydrodynamic diameter over an hour).
HPLC gives you a 1500-point chromatogram per molecule (signal across a retention-time axis).
ELISA gives you an 8 by 3 grid per molecule per target (eight concentrations, three replicates).
Flow cytometry gives you 5000 raw events per molecule per cell line.

Forcing those into one flat Dataset means padding every array to a shared shape, or splitting into multiple files and losing the single-artifact property. Both options give up what made the pattern attractive in the first place.

xr.DataTree is the fix. Each assay becomes a node in a tree. Each node keeps its own grid, but can also inherit a shared coordinate from the root; all nodes inherit the molecule_id coordinate from the root, so selection by molecule propagates everywhere. Here is the pattern from the skill's cross-experiment-linking reference:

import xarray as xr

root = xr.Dataset(coords={"molecule": mol_ids})

dls_ds = xr.Dataset(
    {"diameter_nm": (["molecule", "replicate", "time_s"], dls_timeseries)},
    coords={"replicate": ["r1", "r2", "r3"], "time_s": np.linspace(0, 300, 100)},
)

hplc_ds = xr.Dataset(
    {"signal_au": (["molecule", "retention_min"], hplc_chromatograms)},
    coords={"retention_min": np.linspace(0, 30, 1500)},
)

elisa_ds = xr.Dataset(
    {"od450": (["molecule", "target", "concentration_nm", "replicate"], elisa_data)},
    coords={
        "target": ["egfr", "her2", "cd20"],
        "concentration_nm": np.logspace(-1, 3, 8),
        "replicate": ["r1", "r2", "r3"],
    },
)

tree = xr.DataTree.from_dict({
    "/": root,
    "/characterization/dls": dls_ds,
    "/characterization/hplc": hplc_ds,
    "/bioassays/elisa": elisa_ds,
})

The DLS node keeps its time axis. The HPLC node keeps its retention axis. The ELISA node keeps its concentration and target axes. They share molecule for free through inheritance. You save the whole thing as one file:

tree.to_zarr("s3://bucket/campaign_2026q3.zarr")

That's the contract. One file, every assay on its natural grid.

Custom indexes - slice by what you actually mean

Coordinating by molecule_id is the easy case. The skill's custom indexes are where Ian's own linked-indexes work does the heavy lifting.

Consider a dose-escalation time course. You administer 1 nM, then 10 nM, then 100 nM, then 1000 nM, each over its own time window. Response is measured continuously across the whole experiment. Every wet-lab scientist who has run one of these knows the question that comes next: what was the response during the 100 nM phase?

In pandas, that question is a multi-column filter where you remember which time indices belong to the 100 nM phase, and re-derive the answer every time you ask. With DimensionInterval, a custom defined interval, you register the concentration-to-window relationship once, and the question becomes a one-liner.

# Each concentration is administered over its own time window.
# dose_conc (nM):  1      10       100      1000
# dose_intervals:  [0,60) [60,120) [120,180) [180,240)

dose_ds_linked = dose_ds.drop_indexes(["time", "dose_conc"]).set_xindex(
    ["time", "dose_intervals", "dose_conc"],
    DimensionInterval,
)

# What was the response during the 100 nM phase?
dose_ds_linked.sel(dose_conc=100)
# -> time auto-constrained to [120, 180]

# What concentrations were active during this time window?
dose_ds_linked.sel(time=slice(50, 100))
# -> dose_conc auto-constrained to 1 nM and 10 nM

The relationship is encoded once. Every query after that is a single sel call, in either direction.

The skill ships working implementations of PeriodicIndex (for wrapping coordinates like flow-cytometry angles or circadian time), CoordinateTransform (pixel to micrometers in microscopy, m/z to mass in mass spec), NDIndex (time-locking dosing events across experiments), and DimensionInterval (the phased-regimen pattern above). Five notebooks, each runnable with uv run.

This is the layer that turns xarray from "labeled arrays" into "a coordinate system that understands your experiment."

Bayesian estimates next to the lab data

In February 2025 I argued that archival biological data should be physical quantities with Bayesian uncertainty, not statistical artifacts like p-values. The data package is where that argument lands.

The skill's real-world example puts it directly into practice. The same DataTree holds the raw ELISA OD readings in /bioassays/raw and the Bayesian-derived estimates in /bioassays/derived:

derived = xr.Dataset(
    {
        # 4PL fit results
        "ec50_um":           (["molecule", "target", "cell_line"], ec50_estimates),
        "ec50_std":          (["molecule", "target", "cell_line"], ec50_stds),
        # Bayesian KD estimates with HDI bounds
        "kd_nm":             (["molecule", "target"], kd_means),
        "kd_nm_hdi_3pct":    (["molecule", "target"], kd_lower),
        "kd_nm_hdi_97pct":   (["molecule", "target"], kd_upper),
    },
    coords={"target": ["egfr", "her2", "cd20"], "cell_line": ["hepg2", "hek293"]},
)

tree = xr.DataTree.from_dict({
    "/": root,
    "/bioassays/raw": bioassay,
    "/bioassays/derived": derived,
})

The whole provenance chain sits in one file. The OD reading that produced a KD estimate is one molecule_id lookup away. The HDI bounds travel with the point estimate, so any downstream model that consumes the KD has its uncertainty in the same array. Three years from now, a new data scientist opening this file sees the measurement, the model that produced the estimate, and the uncertainty, all in the same object. No README archaeology, no Slack digging, no "who ran this analysis" archaeology.

The end goal - one curator, many campaigns

This is the part I've been turning over for the past year.

The framing I had in my head when I started the skill was "ship a data package without a data scientist in the room." A wet-lab scientist plus an agent plus a template, and out pops a valid .zarr. But after a few days of thinking, I realized that framing was too optimistic. Building a data package involves judgment calls a wet-lab scientist has no reason to have practiced. Which axis is the entity axis? Where do derived estimates live, and which uncertainty columns travel with them? Did the agent silently drop the HDI bounds when it merged two nodes? Are the units consistent across assays? Did the custom index actually propagate selection, or does it look correct on synthetic data and break on the real thing?

The agent does the mechanical work fast. The agent also does the wrong thing fast. You need a human in the loop who has seen the warts before. In a biotech, that human is the data scientist.

Thus, we gain leverage! Without the skill, one data scientist handcrafts one data package per campaign. The work is mostly pandas merging, dtype wrangling, file-format archaeology, and re-deriving the same coordinate decisions every time. With the skill, the data scientist designs the campaign template once: which assays live at which nodes, which coordinate is the entity axis, where the derived estimates go, what the provenance chain needs. The agent does the per-assay assembly. The data scientist reviews the output, catches the warts, iterates the template, and moves to the next campaign.

One data scientist can now curate many campaigns in parallel. The attention that used to go into pandas goes into judgment: assay design review, coordinate-system choices, validation, catching batch effects, deciding what uncertainty to keep. The agent scales the curator; the curator keeps the judgment.

This is also why I decided to PR to Ian's scientific-python-skills repo and not keep it in my own vault. A skill that lives in my vault helps me. A skill that lives in a public, community-owned home gives every biotech's data scientist the same leverage, and lets the curation patterns converge across the field.

For now, if you're working on multi-assay biological data, the skill is MIT-licensed and the notebooks run with uv run. Pull it, adapt it, break it, and tell me what fell over.

Ollama, vLLM, and SGLang on Modal

2026-07-01T00:00:00Z

I had Qwen3.6 running on Ollama, hosted on Modal, and it worked. The model answered questions, the endpoint stayed up, and I wired it into my tools. I had a specific reason for running it: I am co-teaching a deep research agent tutorial at SciPy 2026, and I wanted every attendee to have a fast LLM endpoint for the hands-on sessions, including those who do not have access to a paid LLM provider. Then I tried to use it for real work, and the cold starts drove me up the wall.

Every time the container scaled to zero and I sent a fresh request, I waited. A minute, sometimes two, watching the spinner. The model was right there in the volume. The GPU was right there, billed by the second. And I was staring at a loading bar because Ollama had to boot its server, open the weight files, and load 17 GB of GGUF into VRAM before it could generate a single token.

I wanted to know: would a different inference engine fix this? I had heard that vLLM, paired with Modal's GPU snapshots, could restore a warm model from a memory snapshot in seconds instead of reloading from disk. So I ran the experiment. Same model, same GPU, same 4-bit quantization level, same hourly cost. The only variable was the engine.

The result surprised me. vLLM won on cold starts. It won on everything.

The setup, held constant

To make this a fair fight, I held everything constant except the engine.

The model is Qwen3.6-27B, a 27-billion-parameter dense model with a hybrid architecture (Gated DeltaNet layers mixed with standard attention). Both deployments serve the same model at 4-bit precision:

Ollama: qwen3.6:27b in GGUF format, Q4_K_M quantization, 17 GB on disk
vLLM: cyankiwi/Qwen3.6-27B-AWQ-INT4 in safetensors, AWQ INT4 quantization, 19 GB on disk

Both run on a single Nvidia L40S with 48 GB of VRAM. At Modal's pricing, that is \$0.000542 per second, or about \$1.95 per hour, for either deployment. Same GPU, same rate.

Both expose an OpenAI-compatible /v1/chat/completions endpoint. I wrote one benchmark script that hits both URLs with the same prompt, the same max_tokens, the same sampling parameters, and measures time to first token, decode throughput, and total latency. Identical workload, identical hardware, different engine.

Warm performance, where vLLM pulls ahead

Let us start with the easy case: the container is already running, the model is in VRAM, and I send a request. This is steady-state throughput, the number that matters for sustained use.

I ran each engine six times, generating 300 tokens, and recorded every data point. The results were remarkably consistent for vLLM and surprisingly variable for Ollama.

Metric	Ollama (Q4_K_M)	vLLM (AWQ-INT4)	Difference
Time to first token	0.85 ± 0.43s	0.32 ± 0.04s	62% lower, 11x less variance
Tokens per second	24.3 ± 2.1	39.4 ± 0.0	62% faster, rock-steady
Total for 300 tokens	11.4 ± 0.7s	7.9 ± 0.0s	31% faster

Points: individual runs (6 per engine). Bars: median. Raw data (JSON).

vLLM generates 60% more tokens per second than Ollama on the same GPU. That is a large gap for the same model at the same precision on the same hardware. But the more striking difference is the variance. vLLM's throughput is rock-steady: 39.4 tokens per second on every single run, with a standard deviation of essentially zero. Ollama bounces between 22 and 27 tokens per second run to run. Its time to first token swings from half a second to 1.6 seconds. vLLM always answers in about a third of a second.

For interactive use, where you feel every hiccup, that consistency matters as much as the raw speed.

The reason comes down to CUDA graphs. vLLM, when allowed to capture CUDA graphs during startup, batches kernel launches and eliminates Python-level overhead between decode steps. Ollama's llama.cpp backend is well-optimized, but it does not use CUDA graph capture. For this hybrid DeltaNet architecture, where every decode step runs a mix of Triton linear-attention kernels and standard attention kernels, the graph capture matters even more. Each step touches many small kernels, and launching them individually adds up.

I learned this the hard way. My first vLLM deploy used --enforce-eager, which disables CUDA graphs. vLLM ran at 12 tokens per second. Slower than Ollama. I removed the flag, vLLM captured its graphs during warmup, and throughput jumped to 44. That one flag was the difference between "why did I bother" and "this is clearly better."

Cold starts, the whole reason I started

Now the interesting part. A warm container is easy. The question I actually cared about was: how long do I wait when the container has scaled to zero and I send the first request?

Ollama has no snapshot mechanism that works reliably on Modal. I tried Modal's memory snapshots with Ollama early on, and the restored container could not find its model. Ollama runs as a Go server that spawns separate runner subprocesses to hold the GPU context, and those subprocesses do not survive a memory snapshot. The restored server wakes up, tries to talk to a runner that no longer exists, and fails. So every Ollama cold start is a full reload: boot the server, open the GGUF files from the volume, stream 17 GB into VRAM, run a warmup generation.

vLLM is a different story. It runs as a single Python process, and Modal's GPU snapshot can capture its full state: the loaded weights, the compiled CUDA graphs, the KV cache memory pool, even the JIT-compiled Triton kernels. On cold start, Modal restores the snapshot and vLLM picks up where it left off.

The cold-start numbers, measured as time from request to first token with the container scaled to zero:

Engine	Cold-start time to first token
Ollama (full reload)	78s
vLLM (GPU snapshot restore)	57s

vLLM cold starts 27% faster. The 57 seconds is dominated by Modal restoring 19 GB of GPU memory state from the snapshot. That is a bulk memory copy, bounded by hardware bandwidth, and it is still faster than Ollama reading 17 GB from a network filesystem and initializing from scratch.

I dug into whether 57 seconds could be pushed lower. The honest answer: probably not by much, for a model this size on Modal.

Modal's documentation is explicit about the limitation. Snapshots help you skip work that is not bottlenecked by storage bandwidth, like library initialization and JIT compilation. But they do not speed up moving weight bytes. At Modal's typical volume bandwidth of one to two GB per second, 19 GB of weights alone takes 10 to 15 seconds before any GPU state restore. Published benchmarks from other developers confirm this floor: a 27B model on an A100 with snapshots and sleep mode gets about 70 seconds cold start. Our 57 seconds on an L40S is actually better than that reference point.

The only way to get sub-10 second cold starts is to keep a container permanently warm with min_containers=1. That trades cost (you pay for idle GPU time) for latency. For bursty workloads where you want scale-to-zero, the snapshot restore time is the price of admission.

Getting the snapshot to work took three rounds of debugging, and each round taught me something specific.

Three bugs between me and a working snapshot

The snapshot did not work on the first try. Or the second. Each failure was a distinct problem with a distinct fix, and I think they are worth walking through because they say something general about deploying inference engines on serverless GPU infrastructure.

Out of memory during profiling

My first vLLM deploy crashed with a CUDA out-of-memory error. The model loaded fine, but when vLLM tried to profile available memory for the KV cache, it ran out. The GPU showed 42 GB in use with only 2 GB free, on a 48 GB card, before a single token of KV cache was allocated.

Two things were eating memory. First, I had set --max-num-batched-tokens 131072, which means vLLM's profiling forward pass runs a dummy batch of 131,072 tokens through the model. For a 27-billion-parameter model, the activation tensors for a batch that size are enormous. Second, vLLM was loading the vision encoder (Qwen3.6 is a multimodal model), which I did not need for text-only inference.

The fix was --language-model-only to skip the vision encoder, and reducing --max-num-batched-tokens to 8192. The model's hybrid architecture helps here: only 16 of the 64 layers use full attention with growing KV cache. The other 48 layers are linear-attention layers with constant memory state. So even with a smaller batched-tokens budget, the effective context capacity is large. After the fix, vLLM reported 21.85 GiB of available KV cache, enough for 332,000 tokens.

Enforce-eager, the throughput killer

The existing vLLM deployment I was adapting from used --enforce-eager, a flag that disables torch.compile and CUDA graph capture. It makes startup faster and uses less memory, which is why it is common in quick-start examples. But it wrecks decode throughput.

With --enforce-eager: 12 tokens per second. Without it: 44 tokens per second. Nearly four times faster. The CUDA graph capture happens during the snapshot build, so it costs nothing at cold-start time. The snapshot preserves the captured graphs. Every restored container inherits them for free.

If you are deploying vLLM on Modal with snapshots, remove --enforce-eager. Let the graphs compile during the snapshot build and ride them forever.

The torch compile cache that broke the snapshot restore

The snapshot was building successfully, but the restore kept failing with a cryptic error: vfs.CompleteRestore() failed: failed to complete restore for filesystem type "9p": failed to walk "torch_compile_cache/torch_aot_compile/...". Modal's snapshot restore was trying to walk a file on the vLLM cache volume that did not exist in the restored state.

The problem was a mismatch between the memory snapshot and the volume. The memory snapshot captured the vLLM process with references to torch compile cache files on the vllm-cache volume. On restore, Modal re-mounted the volume from its latest committed state, and the cache files the process expected were gone. The 9P filesystem walk failed, the restore aborted, and Modal fell back to a full container init. Cold start was 106 seconds instead of 57.

The fix was to remove the vllm-cache volume entirely and redirect TORCHINDUCTOR_CACHE_DIR to /tmp. The compiled artifacts live in the process memory, captured by the snapshot. The on-disk cache is only for persistence across cold starts, and with snapshots, you do not need that persistence. The snapshot is the persistence. After removing the volume, cold start dropped to 57 seconds.

I should note one thing I discovered later: the reason the vLLM sleep endpoint returned a 404 in my first attempt was that I had dropped the VLLM_SERVER_DEV_MODE=1 environment variable when adapting the deployment from an existing repo. That variable exposes the /sleep and /wake_up HTTP endpoints. With it set, the proper sleep-then-snapshot pattern (offload weights to CPU before snapshotting, then reload on wake) would work, making snapshots more reliable. For this model size, it would not dramatically change the cold-start number, which is bounded by weight-restore bandwidth. But for correctness and smaller models, it matters.

What I actually configured

The working vLLM deployment is a single file, vllm_endpoint.py, in my ollama-on-modal repo. The core of it is straightforward.

The vLLM serve command:

cmd = [
    "vllm", "serve", MODEL_NAME,
    "--served-model-name", "qwen3.6-27b",
    "--host", "0.0.0.0",
    "--port", str(VLLM_PORT),
    "--tensor-parallel-size", "1",
    "--enable-sleep-mode",
    "--max-num-seqs", "8",
    "--max-model-len", "32768",
    "--max-num-batched-tokens", "8192",
    "--gpu-memory-utilization", "0.90",
    "--dtype", "auto",
    "--reasoning-parser", "qwen3",
    "--language-model-only",
]

And the Modal class decorator that makes the snapshot work:

@app.cls(
    image=vllm_image,
    gpu="L40S",
    scaledown_window=120,
    timeout=20 * MINUTES,
    volumes={"/root/.cache/huggingface": hf_cache_vol},
    enable_memory_snapshot=True,
    experimental_options={"enable_gpu_snapshot": True},
)

The enable_memory_snapshot plus enable_gpu_snapshot pair is what lets Modal capture and restore the full GPU state. The snap hook starts vLLM, runs three warmup requests to trigger CUDA graph capture and Triton kernel JIT compilation, then Modal snapshots the warm process. On restore, a separate hook confirms the server is responding. No weight reloading, no graph recompilation.

The scaledown_window is set to 120 seconds. After two minutes of inactivity, the container scales to zero. The next request triggers a snapshot restore, which takes about 57 seconds. For my usage pattern, that is the right tradeoff between cost (no idle GPU billing) and latency (under a minute to first token from cold).

Wiring it into the tools I actually use

Once vLLM was deployed and benchmarked, I pointed two things at it.

My opencode configuration got a new provider entry pointing at the vLLM endpoint's /v1 base URL. I select vllm-qwen36/qwen3.6-27b and get 44 tokens per second instead of 28.

The deep research agent tutorial I am building with Ben Batorsky for SciPy 2026 also switched over. The tutorial uses llamabot, which wraps litellm, and the config is just three environment variables:

LLM_MODEL=openai/qwen3.6-27b
TUTORIAL_LLM_BASE_URL=https://<your-modal-deployment>.modal.run/v1
TUTORIAL_LLM_API_KEY=vllm-no-auth

Rather than hand out a live endpoint, the deployments behind these numbers are open source: ollama-on-modal, vllm-on-modal, and sglang-on-modal each stand one inference engine up on Modal, so you can deploy your own.

One snag: llamabot's StructuredBot does a client-side capability check and rejects model names it does not recognize, even when the server supports structured output fine. The fix was three lines in the tutorial's llm.py that call litellm.register_model to tell litellm the custom model supports response_schema. After that, both SimpleBot and StructuredBot work against the vLLM endpoint.

What I would do differently

If I were starting from scratch, I would skip the Ollama-on-Modal step entirely. Ollama is wonderful for local, single-user inference on a laptop. Its one-command ollama run experience is unmatched for development and quick experiments. But for a serverless GPU deployment where cold starts matter, vLLM's snapshot compatibility is the deciding factor. Ollama's subprocess architecture fights the snapshot mechanism. vLLM's single-process design cooperates with it.

The one thing Ollama has going for it in this comparison is simplicity. The deployment is fewer lines of code, the GGUF model format is one file, and the API is clean. If cold starts do not matter for your use case (say, you keep the container warm with a ping), Ollama is perfectly fine and easier to operate.

But if you are paying for GPU by the second and you want the container to scale to zero between requests, the snapshot restore is the feature that makes that practical. And right now, only vLLM plays nice with it.

I did not test SGLang in this round. I scaffolded a separate sglang-on-modal repo and deployed it, but SGLang's current release hits a dtype compatibility error with Qwen3.6's hybrid DeltaNet architecture. The model loads and CUDA graphs capture successfully, but inference crashes with a BFloat16 versus Float16 mismatch in the linear attention layers. vLLM 0.23.0 handles this architecture correctly. The SGLang repo is ready for when they ship a fix. Future work: once SGLang supports this architecture, benchmark how it handles Qwen3.6-27B against vLLM on the same GPU and snapshot setup.

For now, vLLM with CUDA graphs and GPU snapshots gives me 39 tokens per second warm and 57 seconds to first token cold, on a \$1.95-per-hour L40S, and that is the deployment I am keeping.

Agents amplify expertise and ignorance

2026-06-17T00:00:00Z

I gave a talk on 10 June at Data-Driven Pharma (East) 2026 where I built a Bayesian hierarchical model of protein melting points live, in front of an audience, in 27 minutes (including live questions). A coding agent, Cursor, wrote most of the code in a marimo notebook, paired using Marimo pair. I narrated, took questions while the model sampled, and we landed on a posterior over the melting temperature of a yeast protein.

The setup -- featuring Cursor and Marimo -- was the message. I really hate making slides, so I decided a live demo was the best thing to do. I opened a marimo notebook on port 2720, pointed Cursor's agent at it, and started talking to my data the way I actually do at work.

Halfway through, something I already believed got sharper. The agent was fast. Genuinely fast. But it was fast for me in a way it would have been merely noisy for someone who lacked the context. Every minute I saved, I saved because of judgment I'd built over years. The agent amplified that judgment.

Here is what convinced me.

Every prompt hid a decision

People treat prompting like a discrete skill, a thing you can grind at in isolation. Read the prompts I gave the agent during the demo and you'll see the real substrate. Each one handed over a decision a scientist had to make first.

To pick a protein, I had to know that a melting curve only means something when the signal has real dynamic range; it has to actually fall as the protein denatures. So I told the agent:

Find me a protein with a very good dynamic range between the upper and lower bound signal. If the thing doesn't melt over temperature, or is very labile and melts straight away, that's not something I want to start with.

That is a Bayesian's judgment wearing a prompt's clothes. The agent searched the dataframe. I made the call about what counts as a fittable curve.

To fit the curve, I told it the four-parameter logistic's upper and lower bounds had to land at exactly 1.0 and 0.0. That is dataset-specific domain knowledge, available to me only because I know something about how that data was generated. In plenty of assays the plateaus are sloppy and you let the data find them. In this meltome assay the fold change is normalized, so the bounds are fixed. Only someone who has stared at this assay would know that.

The hierarchical model at the end carried the same fingerprint. I told the agent to put the population prior on the melting temperature and let the slope be whatever. That choice encodes a scientific belief: proteins in the same organism share a distribution of stability, while their transition steepness stays idiosyncratic. You earn that belief by fitting a lot of curves and watching what breaks.

Strip the expertise out and the prompts collapse. A novice asks the agent to "analyze the protein data" and gets back a plausible, confident, wrong answer.

I caught what the agent missed

The cleanest moment came early. The agent loaded the parquet and showed me the dataframe. I looked at the columns and noticed the temperature was missing, the one column a meltome analysis needs to exist. I'd prepped that file from a JSON that morning and dropped it on the floor.

The agent was perfectly happy. It had data, it had columns, it would have charged ahead and fit curves to nothing. I had to send it hunting through the original paper's methods section for the actual temperatures.

That is the whole game. The agent has blind spots. Left alone, it charges right past them. The person who has run the assay feels the missing column immediately. And the faster the agent moves, the more you need exactly that feeling, because the agent accelerates regardless of whether it is right.

Know your vibe zone

Here is the rule I keep coming back to. Go fast inside your zone of expertise; slow way down outside it. In other words, know your vibe zone, the place where your judgment is reliable enough that you can let the agent run and trust your gut-check on the output.

The four-parameter logistic lives in my vibe zone. I have fit a lot of them. I can glance at a PyMC model, see a thousand tuning samples and target_accept=0.95, and know it is behaving. I let the agent write it and skimmed the result.

Analytical chemistry sits on the other side. I lack proper training in it. Hand me an LC-MS dataset with the same agent and I slow to a crawl, one plot at a time, five minutes of thinking before each request, because a sane result and a broken one look the same to me at a glance.

The agent widens this difference. Inside my vibe zone it turns ten hours into one. Outside it, it turns confusion into confident confusion, faster. The leverage scales with how much you already know.

Trust scales with your ability to verify

Mid-demo I said something that made the room laugh, and I meant it: I'm going to trust what the agent says. You shouldn't.

I could trust it because I could check it. I know what a posterior over a melting temperature should look like. I know what a forest plot rank-ordered by probability of superiority does when the data is good, and what it does when the curve is junk. Trust is a dial you set with your ability to verify. Experts turn it up. Beginners keep it low and check everything, which is exhausting, which is why beginners move slowly even with a fast agent.

The gap widens

The appealing story about coding agents is that they democratize the craft, that now anyone can build a Bayesian model. My spicy take is that the lived reality runs the other way. The agent hands a loaded instrument to whoever is holding it. In expert hands it is a microscope. In novice hands it is a confocal microscope pointed at the wrong slide, producing beautiful, expensive, wrong images.

If you have spent a decade learning your domain, this is excellent news. Your investment compounds now instead of depreciating. The boring middle of data science, the boilerplate and plotting and fiddly dataframe wrangling, burns away, and what remains is the part that was always worth something: deciding what to measure, what to believe, and what to distrust.

The agent made my years matter more. It also raises questions on how to bring people up to speed in the same domain. And yet, if you want more leverage from tools like these, the highest-return thing you can do is still, somewhat against the spirit of the age, to become an expert in something real.

My coding agent learned a lesson and patched its own skill

2026-06-16T00:00:00Z

I was debugging a transcript duplication bug in my voice-first gym coaching app. The coach's responses were being saved twice to the database, one turn apart. I traced it to a React 18 batching issue in the flush logic, refactored the state management into a hook, wrote a one-shot backfill to clean up the historical data, and committed everything.

then I went to get a coffee.

When I came back, the skill file for Convex migrations had a new entry. The convex-migration-helper skill, installed in my repo a week ago and untouched since, now contained a six-line callout explaining that internalMutation functions cannot be invoked from the Convex CLI. The code example had been corrected from internalMutation to mutation. A reference file deep in the skill's references/ directory had been updated with the same fix.

Nobody told it to do that. In another session, my coding agent, GLM-5.2 on OpenCode, hit the internalMutation wall during the backfill and solved the problem on its own; a background review process then extracted the lesson and patched the skill.

opencode-autolearn is the plugin that made that possible. It is open source and runs on your machine. I shipped the first version a few weeks back, and it has been compounding ever since. As I write this, across twenty-plus projects it has logged over a thousand observations, spawned nearly three thousand review sessions, and grown a store of fifty-plus skills. My coding agent gets better every session, and I do nothing extra to make that happen.

Agents that forget

I wrote about building self-improving coding agents back in January. The core observation was simple: AI coding agents repeat the same mistakes across sessions because they have no mechanism to learn from corrections. Every session starts from scratch. You re-state the same preferences. You re-correct the same behaviors. You are babysitting a very fast intern.

The post identified two levers: AGENTS.md as repository memory, and skills as reusable playbooks. Both work. I use them every day. But the loop was still manual. I had to notice the pattern, decide what to do with it, and write the correction myself. I was doing the learning, then handing the agent the results.

The question that would not leave me alone: what if the agent could watch its own conversations and extract the lessons itself?

The inspiration from Hermes

A colleague, Edward Miracco, told me about a coding agent called Hermes. Hermes had a property I found fascinating: it got better at working with you over time. The model weights were the same. But Hermes maintained a persistent memory of corrections, preferences, and workarounds, and it updated that memory as you worked.

I wanted that for OpenCode, the coding agent I use daily. OpenCode has a plugin system, a skill discovery mechanism, and a session model that captures full conversation histories. All the raw materials were there. The missing piece was the feedback loop: something that watched the conversation, decided what was worth learning, and wrote the lessons down.

So I asked OpenCode to build it.

What it does

opencode-autolearn is an OpenCode plugin that does four things:

Monitors conversations. A JavaScript plugin hooks into OpenCode's event system. It counts turns, buffers messages (with secret redaction), and watches for idle periods and session exits.
Spawns review agents. Every five assistant turns, or when the session goes idle, or when you close the terminal, the plugin spawns a detached subprocess that runs a review agent. The review agent receives the buffered conversation and an instruction sheet.
Extracts lessons. The review agent reads the conversation looking for corrections ("don't do X"), preferences ("I prefer Y"), workarounds that worked, and recurring patterns. For each one it finds, it takes action.
Writes to durable stores. The review agent uses a Python CLI to update three things: a persistent memory store (loaded into every future session), a user profile (communication and workflow preferences), and skills (created or patched based on observed patterns).

The architecture is deliberately split: a thin JavaScript plugin that only counts and buffers, and a Python CLI that does the data management. The plugin never blocks the main session. Reviews run in a detached subprocess that the plugin fires and forgets. If the review fails, the conversation is saved to a fallback file for debugging. The main session never knows the difference.

Design decisions

Four decisions shaped everything else.

The trigger needs no human

The review fires on its own. By default, every five assistant turns, or when the session goes idle, or when you close the terminal. You never type a command or decide when to review. The loop does not wait for you.

This is the decision I care about most. Other tools use human-triggered commands like /dream for their reflection step, and those work, until they do not. You remember to invoke them for a week. Then you get busy, you forget, and the learning stops. The trigger is the first thing to go when you have real work to do.

Making it automatic costs almost nothing. A handful of background subprocesses you never see. What you buy with that is a system that learns from every session, not just the ones where you remembered to pull the lever.

The trigger itself is just an OpenCode plugin. It hooks into OpenCode's event system, counts turns, and fires reviews. No fork, no separate daemon, no modified binary. You install the plugin and your existing OpenCode setup gains the feedback loop.

A registry behind the markdown

I started with a single markdown file. Memory lived in ~/.autolearn/memory.md. User preferences lived in ~/.autolearn/user-profile.md. Skills lived in ~/.autolearn/skills/{name}/SKILL.md. All plain text, all human-readable, all directly loadable as OpenCode instructions.

I picked markdown deliberately. I considered SQLite; it would be more queryable. But the agent reads these files as context, and OpenCode loads instruction files as plain markdown. Markdown served double duty: it was both the storage and the context injection. If I wanted to see what my agent had learned, I cat the file. If I wanted to correct a lesson, I edited it.

That worked, until the file filled up. A single markdown file has a size ceiling. When memory hit a few thousand characters, the oldest entries got silently evicted to fit, and I lost lessons I wanted to keep. Reinforcement was append-only: see the same correction three times, get three duplicate entries.

So the store moved behind the markdown. Today the durable store is a JSONL registry, memories.jsonl, with one observation per line and no size cap. The markdown file, memory.context.md, is now a view, regenerated from the registry on every session start and after every review. Storage and context are separate concerns. The registry can hold thousands of entries; the composed view surfaces the most relevant ones within a soft character budget. Reinforcement counters live in the registry itself, so a repeated correction bumps a counter instead of duplicating a line.

The agent still reads markdown. I still open a file to see what it learned. The difference is that the file I read is now generated from something more durable underneath it.

Reviews run as subprocesses

Each review runs as a separate opencode run invocation in a detached subprocess. The plugin sets AUTOLEARN_REVIEWER=1 in the subprocess environment so the review session does not trigger its own reviews (which would create an infinite loop).

The subprocess approach has three benefits. First, failures are isolated: a crashed review does not affect the main session. Second, the review has its own context window: it loads the autolearn-reviewer skill and gets a clean slate to evaluate the conversation. Third, concurrency is naturally limited to one review at a time, because the plugin tracks an in-process flag.

Skills are symlinked for auto-discovery

OpenCode discovers skills in ~/.agents/skills/. When the review agent creates a new skill in ~/.autolearn/personas/default/skills/, the CLI symlinks it into ~/.agents/skills/ so OpenCode picks it up automatically. No restart needed, no configuration change.

This means the loop is: review agent observes a pattern, creates a skill, and the very next session can load that skill if the pattern recurs. The feedback loop closes itself.

The build history

The first commit was a working plugin with the full CLI. I had been thinking about the architecture for a few days, and the initial implementation came out in one piece: turn counting, message buffering, review spawning, memory management, skill creation and patching.

Then came a series of refinements, each driven by a real problem I hit while dogfooding:

Exit-triggered reviews. The first version only spawned reviews at turn thresholds. I kept losing the last few turns of a session because I would close the terminal before the threshold fired. So I added beforeExit and signal handlers (SIGINT, SIGTERM) to dispatch a final review on shutdown.
EARS specifications. After the initial build, I had my OpenCode agent write LLDs and EARS specifications for the shipped features. This was partly discipline and partly debugging: the EARS specs allowed me to trace each requirement to actual code paths, and the process surfaced edge cases I had missed. I asked the agent to also add @spec annotations in the plugin source linking each code block to its EARS requirement ID.
Skill symlinks. The initial version created skills in ~/.autolearn/skills/ but did not symlink them into ~/.agents/skills/. Skills existed but OpenCode could not discover them. The symlink step closed that gap.
Reinforcement tracking. Early on, memory entries were append-only. If the agent observed the same correction three times, it would add three entries. I added a strengths.json file that tracks how many times each observed pattern has been reinforced, and strengthen/weaken commands so the review agent can bump the count instead of duplicating the entry.
Full-text search. The review agent needs to answer "has this pattern come up before?" I built an FTS5 index over OpenCode's session database so the review agent can search past conversations before concluding "nothing to record." This catches recurring corrections that were never promoted to memory.
The review-runner wrapper. Reviews were leaving behind orphaned sessions in OpenCode's session list. I wrote a shell script wrapper that runs the review, captures the session ID from the JSON output, and deletes the session afterward. The plugin calls the wrapper instead of opencode run directly.
Behavioral escalation. Memory and skills capture in-session lessons, but some corrections recur across every project and belong in the repo's AGENTS.md, where every agent run reads them. I added a second CLI, improve.py, that records behavioral rules and counts how often each one recurs. When a rule crosses a threshold, improve.py escalate --apply writes it into the appropriate AGENTS.md. The review agent calls improve.py observe ... as part of every review, so cross-project patterns graduate from session memory into durable repo instructions on their own.
Cross-machine sync. Everything lived on one machine. Switch laptops, lose the learned memory. I built an E2E-encrypted sync layer: a master password derives a key with PBKDF2-SHA256, the key lives in the OS keychain, and the server only ever sees ciphertext. The plugin auto-syncs on session start and after each review. Two backends implement the same REST API, a self-hosted Fastify server and Convex HTTP Actions, so you can run it yourself or use a hosted one.
Multi-persona stores. Work lessons and personal lessons should not collide. I split the store into personas, isolated directories with their own memory, skills, and sync keys. The installer migrates the existing flat layout automatically, so upgrading was invisible.
The memory registry. The single memory.md worked until it filled up and started evicting old entries to fit a size cap. I moved the durable store to a JSONL registry and turned the markdown into a composed view, regenerated on session start. This migration is also where I learned a general lesson the hard way: when you move a system to a new data store, audit every reader of the old one. The autolearn reviewer itself was still warning about "silent 3000-char eviction" long after the cap was gone, because its instruction prose was a stale reader of the old behavior. (That lesson is now in the memory store, funnily enough.)
Curator on a schedule. Skills accumulate. Fifty narrow skills are harder to navigate than ten well-named ones. The curator consolidates overlapping skills into broader umbrellas, archives stale ones, and escalates high-reinforcement lessons toward AGENTS.md. I wired it to the opencode scheduler so it runs weekly without me thinking about it.

The moment I knew it worked

For the first several days, I was not sure it was working. The plugin was spawning reviews. The observations log was filling up. But I had not seen the system do something I did not expect.

Then, during a session on my gym-coach project, I hit a wall with the Convex CLI. I had written a backfill mutation as an internalMutation, tried to invoke it with npx convex run, and discovered that the CLI can only call mutation, query, and action functions. internalMutation is private to Convex's internal calling mechanism. I had to convert the function to a regular mutation, run the backfill, then remove the one-shot code.

I committed the fix and moved on. The session ended. The review agent spawned.

When I looked at the convex-migration-helper skill the next day, it had been patched. The review agent had:

Identified the workaround (convert internalMutation to mutation for CLI-invoked backfills).
Found the existing skill that documented migration patterns (convex-migration-helper).
Patched the skill's SKILL.md with a new entry explaining when to use mutation vs internalMutation.
Patched the references/migration-patterns.md file, correcting the code example and adding a callout box.

The scope was right. It patched the specific reference file where the internalMutation pattern was documented. It went to the exact section that was wrong and fixed it.

That was the moment I stopped wondering whether the system worked.

Dogfooding by the numbers

I have been running opencode-autolearn on my machine for a few weeks now. Here is what the system has done in that time, without me lifting a finger:

Metric	Count
Observations logged	~1000 (the log keeps only the most recent thousand)
Review sessions spawned	~3000
Projects covered	20+
Memory entries (registry)	~100
User profile preferences	11
Skills created	50+

Those reviews ran across projects including my gym-coach voice app, my Brain42 knowledge tools, my network analysis teaching materials, my blogbot automation, canvas-chat, and several others. The system watched every session, decided what was worth remembering, and wrote it down.

The reinforcement tracking is where the compounding shows up. The most-reinforced lesson is a SaaS multi-tenant safety rule: verify before configuring any SaaS service. I hit that pattern across multiple projects and sessions, and each time the review agent bumped the counter instead of adding a duplicate entry. The agent treats that rule as high-priority context because the reinforcement count tells it this one matters.

What the agent has learned

The review agent has created over fifty skills from scratch and patched several existing ones in local repos. Here are a few representative examples:

A blogbot skill for generating social media posts from blog content. The review agent created it after watching me manually draft posts, then patched it with a URL verification step after observing me checking URLs by hand.
An evergreen-note-quality skill for my Obsidian vault, created after watching me audit note quality across multiple sessions.
Bug-pattern skills named after the exact pitfall: react-setstate-in-effect (don't call setState unconditionally inside useEffect), optional-chaining-root-guard (optional chaining hides a null root), unicode-safe-filename-ops (normalize before saving a user-titled file), ssrf-guard-node (validate a URL before a server fetches it). Each one came from a real bug I hit, in a real session, that the review agent watched me fix.

The local repo skill patch is the Convex CLI story I described above: the convex-migration-helper in the gym-coach repo, patched with the internalMutation lesson.

The persistent memory store holds around a hundred entries. Each came from a real mistake. Each has prevented the same mistake in subsequent sessions. The user profile has captured how I like to work: I prefer warm, personal blog conclusions. I expect agents to proactively load writing skills when editing prose. I want autonomous execution without confirmation prompts. I demand quantified evidence in architecture analysis. The agent read these from my conversations and wrote them down. Now every session starts with this context loaded.

The feedback loop for autolearn looks like this:

graph TD
    A[OpenCode session] -->|every 5 turns / idle / exit| B[autolearn.js plugin]
    B -->|spawn detached subprocess| C[autolearn-reviewer agent]
    C -->|reads conversation| D{Learning opportunity?}
    D -->|correction / preference| E[memories.jsonl registry]
    D -->|recurring pattern| F[create or patch skill]
    D -->|nothing worth recording| G[exit quietly]
    E -->|composed into memory.context.md, loaded into| A
    F -->|symlinked into ~/.agents/skills/| A

The core loop works: watch, review, learn, persist, discover. The system improves the agent's behavior without touching model weights.

Install it yourself

If you use OpenCode and want to try it:

curl -fsSL https://raw.githubusercontent.com/ericmjl/opencode-autolearn/main/install.sh | bash

The plugin activates on your next session. You will not notice it running. But after a few sessions, check ~/.autolearn/personas/default/memory.context.md. Your agent has been taking notes.

The full source is on GitHub: ericmjl/opencode-autolearn. The design docs include eight LLDs, eleven EARS specifications, and a high-level design that marks every feature shipped, partial, or planned. The README has the complete CLI reference, the sync setup, and the configuration options.

What comes next

The portability problem is solved. Sync, encryption, and multi-persona stores shipped. The curator runs on a weekly schedule. The memory store grew up too: every entry now carries a retention score and a tier (hot, warm, cold), and the composed view ranks entries by relevance against a soft character budget before it regenerates on each session start. Lessons I keep hitting stay hot; one-off corrections I never repeat fade toward evictable.

What is genuinely left is calibration, and the one detector that has never run. The retention curve needs real-world tuning; the half-life parameters are fresh guesses, and I want to watch which entries fade too fast or linger too long. The recurring-preference detector, a shift detector that notices when a preference is rising (worth recording) versus settling into habit (learned, stop surfacing), is wired but has never taken a real pass.

I keep thinking about the moment I saw the patched convex-migration-helper skill. I had not told anyone to fix it. I had not filed an issue or written a TODO. The conversation where I hit the internalMutation wall was over. I had moved on. But the system was still watching, and it decided that the workaround I found was worth remembering.

That is the property I wanted. The agent gets better without me steering the improvement. I do the work, the system does the learning, and the next session inherits the result.

The Hermes agent had it. Now OpenCode does too.

I hope autolearn brings you the same quiet compounding it has brought me: an agent that remembers your corrections, respects your preferences, and gets a little sharper every time you sit down to work.

Git worktrees for beginners

2026-06-15T00:00:00Z

You are deep in a feature branch. Files are half-edited, tests are mid-run, your terminal is a graveyard of useful state. Then a teammate pings you: "can you review this PR?" Or a bug on main needs fixing right now.

Your options feel lousy. You can stash your work and pray you remember the path back. You can commit a half-finished change just to park it. Or you can clone the repo into a second folder and juggle two copies of the history.

There is a fourth option, and it is the one I now reach for every time: git worktree.

What a worktree feels like as a beginner

Let's strip away the internals for a second. What is a git worktree? A git worktree shows up as a new folder on disk that contains a full copy of your repo's files, checked out on whatever branch you ask for, placed anywhere you want on your computer.

That is the essential experience of git worktrees. You run one command, you cd into a new directory, and you have a clean working tree ready to go. When you are done, you remove it, and the folder disappears.

Here is the basic loop:

# create a new folder ../hotfix with the repo checked out on a new branch "hotfix"
git worktree add ../hotfix

# go work there
cd ../hotfix
# ...edit, commit, push...

# when you are done, clean it up
cd -
git worktree remove ../hotfix

git worktree remove deletes the folder and deregisters it from git. That is cleanup. If you ever do it by hand (rm -rf ../hotfix), git leaves a stale registration behind, and git worktree prune sweeps that up.

You can have several worktrees at once. git worktree list shows you every one:

git worktree list

Each entry is a folder on disk, a branch, and a commit. Treat them like separate workspaces that happen to share one brain.

So how is this different from cloning into a second folder?

That's a great question! If I just need a second copy of the repo, why run git clone into another folder?

You can, and people do. I have one colleague at work who has the software developer's equivalent of Untitled12.ipynb for multiple repos. But a clone and a worktree are doing fundamentally different things under the hood, and the difference shows up the moment you start working.

A clone is a second, independent repository. git clone copies the entire .git database, history and all, into a brand new repository. The two clones know about each other only as remotes. To share work between them you fetch and push, the same as you would with a colleague's machine. Branches you create in one clone are invisible to the other until you sync them.

A worktree is a second door into the same repository. It adds a working directory, but the history, the commits, the branches, and the tags all live in one shared .git. Make a commit in any worktree and it is instantly visible in all the others, because they are reading from the same underlying git database. There is nothing to sync.

Here is how that plays out side by side:

	`git clone` into a new folder	`git worktree add`
Storage	Copies the full `.git` history again	Adds only the working files; shares the `.git` database
Branches	Each clone keeps its own copy; must fetch/push to sync	All branches shared instantly across worktrees
New branches	Invisible elsewhere until pushed	Visible everywhere immediately
Same branch twice	Allowed (independent checkouts)	Blocked; a branch lives in one worktree at a time
Network	Needed to sync between copies	Never needed (storage is shared locally)

That last row is the one I feel most. With clones, every "let me grab that branch" is a fetch. With worktrees, the branch is already there because it was never anywhere else.

The one rule that trips people up

Git enforces a single constraint, and it is worth knowing up front: a branch can be checked out in only one worktree at a time.

If you are on feature-x in your main folder, git will refuse to also check out feature-x in a worktree. That refusal protects you from two working directories racing on the same branch's index. If you need a second working tree for similar work, make a new branch for it:

git worktree add -b feature-x-take2 ../take2

This is the tradeoff worktrees make. Clones give you total independence at the cost of duplication and syncing. Worktrees give you zero duplication and zero syncing at the cost of that one-branch-one-worktree rule. This is basically perfect for the "park my work and go look at something else" use case.

Are the worktrees linked on disk?

Yes, and that is the essence of how git worktrees work.

A normal repository has a .git directory that holds everything: the object database (every commit, tree, and blob), the refs (branches and tags), the configuration, the logs. One repository, one .git.

A linked worktree has a .git file instead of a .git directory. That file is one line long:

gitdir: /Users/you/myproject/.git/worktrees/hotfix

That path points back into the main repository's .git directory, into a small admin folder that holds only the per-worktree state: the worktree's own HEAD, its own index, a few logs. The heavy lifting, the object database and the refs, is shared from the main repo.

You can see it for yourself:

cat ../hotfix/.git
# gitdir: /Users/you/myproject/.git/worktrees/hotfix

ls /Users/you/myproject/.git/worktrees/
# hotfix

That is the linkage. Every worktree points into the same .git directory, which is why a commit made in one shows up immediately in the rest. They are separate folders on disk with separate working files, but they read and write to one shared repository. One brain, many desks.

When I actually reach for worktrees

A few patterns where worktrees have earned their keep for me. Each one is a plain-English ask you can hand to a coding agent:

Quick reviews and hotfixes. I am on a feature branch and someone needs me on main. I git worktree add a throwaway folder, do the review or the fix, push, and remove it. My feature branch never moves. The ask: "Help me review <PR URL> in a new worktree."
Running two versions side by side. I keep one worktree on the released version and one on main, so I can reproduce a bug against both without juggling stashes. The ask: "Check out the v1.2 release tag in a worktree so I can compare it against main."
Long-running branches with context I want to keep. Some experiments take days and accumulate useful terminal state. A worktree lets me park them in their own folder and pop back in without disturbing my main flow. The ask: "Create a worktree for this experiment so I can leave its state alone."
Handing a whole task to a coding agent. A worktree is just a folder, which means I can hand a coding agent the full lifecycle in one instruction: "do this work in a worktree, open a PR, and remove the worktree when you're done." The agent gets its own clean workspace, my main folder sits untouched while it churns, and once the PR is open the throwaway folder disappears on its own. The ease of this is the part I want to flag: the whole loop is small enough that delegating it in plain English feels natural. I am writing this very post that way.

The shared storage means I can have five of these open and still pay only for the working files, five times over, instead of five copies of the entire history.

A note on cleanup

git worktree remove <path> is the polite way out. It deletes the folder and tells git to forget about it. If the worktree has uncommitted changes, git will refuse unless you pass --force, which is a nice safety net.

If you ever delete a worktree folder by hand, the registration lingers until you run git worktree prune. I treat prune as the "tidy up" command and run it now and then after a week of creating and discarding worktrees.

Give it a try

The next time you are tempted to stash or clone just to look at another branch, try this instead:

git worktree add ../scratchpad
cd ../scratchpad

Or alternatively, ask an agent to work in a worktree for you:

"Set up a worktree on a new branch and make the changes there."

Work there. Commit there. When you are done, come back and git worktree remove ../scratchpad. Your main folder stays exactly as you left it, the throwaway folder is gone, and you never duplicated a single commit to get there.

One brain, many desks. That is all a worktree is!

Lessons building voice-first AI apps

2026-06-13T00:00:00Z

Voice-first AI means voice is the primary way you interact with the model. You talk, it talks back, it calls tools, things change on screen. In some of my projects, voice is the only way you interact with the model. There is no text box.

That distinction changes everything about how you build, and I learned it through three projects. Yarnsmith is a voice-first game where an AI game master narrates a D&D-style adventure with an educational twist, aimed at parents who want to teach their kids good civic values. Gym Coach is a voice-powered workout companion that talks you through exercises and logs your sets. And I added voice to Canvas Chat, my visual interface for LLM conversations. Each project surprised me, and the lessons converged on a single pattern: voice plus tools on a web interface equals something genuinely powerful.

I'm still mapping the edges of this pattern. But I've learned enough to share what works, what doesn't, and where the debugging traps are.

Docs and reference examples are non-negotiable

It's tempting to vibe-code a voice app. Spin up the real-time API, talk to the model, assume it works. It rarely does.

Voice AI is so new that the leading coding models often lack current information about the APIs. Real-time audio streaming, function calling over WebSocket, session management, these are all recent enough that models trained a few months ago will hallucinate the details. They invent parameters that don't exist, use outdated SDK methods, or confidently describe behavior that changed three versions ago.

The pattern that worked for me is this: get the official documentation, build one working reference example, and point the coding agent at both. Yarnsmith was my first working reference. Once I had a single end-to-end example where the voice pipeline actually functioned, that was enough for Gym Coach and Canvas Chat.

One working reference does magical wonders; the agents stop guessing and start following.

Build APIs first, then layer on tool calls

When I first explored voice as an input modality, it was interesting but limited. The breakthrough for me came when I structured the application around tool calls, like I had seen it in the Claude app with web search as a tool. I started with the Vercel AI SDK for these TypeScript projects, then switched to the Gemini SDK when I needed voice-specific features. The architectural pattern stayed the same either way.

Here is the pattern to follow: Expose all the functionality of your app through API endpoints. Then register those endpoints as tools the voice agent can call. The agent speaks, decides to take an action, calls the tool, and the action happens on the web interface.

An API gives you flexibility that direct integration never will. You can have the agent call endpoints through tool calls during a voice session. You can also simulate the same interaction manually with curl to debug individual problems. You can write tests against the same endpoints. The API is the contract, and everything else builds on top of it.

It's the same layered pattern you see everywhere in software:

A Python library gives you programmatic access
A CLI wraps the library for command-line use
An agent wraps the CLI for tool-call use

Same idea here. You build the API first, then layer on tool calls for the voice agent. The API comes first because it is the thing you can test independently. The tool-call wrapper comes second because it is the thing the agent uses at runtime.

Transcripts are your lifeline

When you build a text-based agent, everything is visible. The user types something, the agent responds, you can read the full conversation. When you build a voice agent, most of the interaction is invisible. Audio goes in, audio comes out, and if something goes wrong, you have nothing to look at.

This is where debugging hell starts.

The fix is simple but essential: always have a transcript. Every spoken word from the user, every response from the agent, every tool call and its result, all of it should appear as text on the web interface in real time.

Without a transcript, you are flying blind. You will hear the agent say something unexpected and have no idea what chain of tool calls led to that response. With a transcript, you can trace the exact sequence of events.

Also, bonus tip: get the agent to give you a Copy button so that you can copy/paste as much contextual information over to the coding agent's context later!

Log every tool call with timestamps

A transcript of spoken words is the minimum. Voice agents introduce a problem that text agents don't have: latency.

Audio processing, network round trips, model inference, tool execution. Each step adds delay. When the agent takes three seconds to respond, you need to know where those three seconds went. Was it the speech-to-text? The model thinking? A slow API endpoint?

Every entry in your transcript needs a timestamp. Not just the spoken words, but every tool call: when it was initiated, when it completed, what arguments were passed, what was returned. The tool call order matters too, because voice agents can trigger cascading sequences of calls that interact in unexpected ways.

I guarantee you'll hit latency problems with voice agents. When you do, the timing information in your transcript is the only way to diagnose where the delay lives.

This principle applies to text agents as well. But it is even more critical for voice because the user is waiting in real time. A three-second delay in a text chat is tolerable. In a voice conversation, it feels broken.

Centralize voice operations into one component

Good logging tells you what went wrong. Clean architecture determines whether you can fix it. Here is a lesson I learned the hard way with Gym Coach.

The symptom was specific and maddening. The user would say "sounds good," and the coach would call save_plan to create a workout. Then silence. The coach stopped talking. The transcript showed the tool call completed, but the voice agent never generated its next response. Gemini was waiting for a FunctionResponse that never came.

Debugging this was hell because voice handling was scattered across four components. GeminiLiveClient managed the WebSocket connection. TurnLifecycle tracked phantom turns. ToolCallGatekeeper enforced behavioral rules. The session page held its own callback configuration and workout state. Each component had its own assumptions about what should happen next, and tracing where the response got swallowed took hours. The root cause turned out to be the guards themselves. I had built behavioral guards to prevent bad behavior: premature logging, exercise mismatches, duplicate plans. When a guard fired, it blocked the tool call and discarded the response. But Gemini was waiting for that response. No response meant no next turn. No next turn meant silence.

Even with logging in place, I would fix one guard in one file and then hit a related bug in another because the same logic was duplicated elsewhere. The architecture was fighting me.

The fix was architectural, and it happened incrementally. After each debugging session, I asked my coding agent to use the improve-codebase-architecture skill to find the top recommendation for preventing the class of bug I had just fixed. Instead of a full review process, I said "take your top recommendation and implement it." One recommendation per session. Ship it. Move on. I would strongly encourage this practice for any voice-first project. Each round surfaced a concrete structural fix, like extracting scattered coordination logic into one class, or moving guards from the transport layer to the domain layer. One recommendation at a time kept the changes small enough to verify.

Over multiple iterations, all voice operations collapsed into a single component, the GeminiLiveClient. Its contract is clean: it takes a config and a set of event callbacks. One callback, onFunctionCall, returns a string that becomes the FunctionResponse sent back to Gemini. The TurnLifecycle module was deleted entirely. Behavioral guards were stripped out. What remained were data-integrity guards only: dedup duplicate tool calls by ID, rate-limit to one per turn, prevent double-saved plans.

The key change replaced blocking with description. When a guard fires now, it still sends a FunctionResponse, but the response says "blocked: plan already saved" instead of going silent. Gemini reads that and self-corrects on its next turn. The coach never stops talking.

The skeleton of the centralized client looks roughly like this:

interface VoiceClientEvents {
  onAudioData: (audio: Uint8Array) => void;
  onTextResponse: (text: string) => void;
  onInputTranscription: (text: string) => void;
  // Returns a descriptive string that becomes the FunctionResponse.
  // Example: "blocked: plan already saved", "Set logged. Acknowledge briefly."
  onFunctionCall: (name: string, args: Record<string, unknown>) => string | void;
  onTurnComplete: () => void;
}

class VoiceClient {
  private session: Session | null = null;
  private seenToolCallIds = new Set<string>();
  private pendingResponses: { id: string; response: string }[] = [];

  constructor(
    private config: VoiceConfig,
    private events: Partial<VoiceClientEvents>,
  ) {}

  async connect() {
    // Open the realtime session and register a single message handler
  }

  private handleMessage(msg: ServerMessage) {
    if (msg.toolCall) {
      for (const fc of msg.toolCall.functionCalls) {
        if (this.seenToolCallIds.has(fc.id)) continue;
        this.seenToolCallIds.add(fc.id);

        const result = this.events.onFunctionCall?.(fc.name, fc.args);
        this.pendingResponses.push({ id: fc.id, response: result ?? "ok" });
      }
    }

    if (msg.audio) this.events.onAudioData?.(msg.audio);
    if (msg.text) this.events.onTextResponse?.(msg.text);

    // ALWAYS flush tool responses. Blocking the FunctionResponse
    // means Gemini hangs silently, waiting for a reply that never comes.
    if (msg.turnComplete) {
      for (const { id, response } of this.pendingResponses) {
        this.session?.sendToolResponse({ functionResponses: [{ id, response }] });
      }
      this.pendingResponses = [];
      this.seenToolCallIds.clear();
      this.events.onTurnComplete?.();
    }
  }
}

The entire contract is one class with one event interface. The domain layer talks to the voice agent through a single callback that returns a string. Tool responses always flush. The only guards that remain prevent duplicate data writes, like logging the same set twice or saving the same plan twice. They never block the voice agent from speaking.

The results were qualitatively dramatic. Undesirable behaviors that had been recurring were eliminated in the next iteration. Multiple bugs I could foresee were never introduced. New bugs showed up, of course, but the trajectory was a series of step-function improvements, each one making the whole system more debuggable.

Voice-controlled UI is magical

I want to end on the part that made all of this worth building.

There is something viscerally magical about speaking to your computer and watching it respond. You say "log my set" and the UI updates. You say "show me the next exercise" and the screen changes. It is Geordie La Forge on the Enterprise going "Computer, do something" and watching it happen.

That experience is genuinely joyful for the user. And it relies on a design principle that is easy to miss: the action needs to be visceral and exposed. When the agent takes a tool call, the user should see the result on screen immediately. The visual context should change in response to the voice command.

This is where good UI and UX design principles earn their keep. The web interface doubles as the visual feedback layer that makes voice interaction feel real. Every tool call should produce a visible change. Every state transition should be immediate. The user needs to feel that their voice caused something to happen.

What I'm still figuring out

The pattern I've converged on, voice plus tools on a web interface, feels powerful but immature. The engineering practices I described above are the scaffolding that makes it reliable enough to ship. Documentation-driven development keeps the models honest. API-first architecture keeps the system testable. Transcripts with timestamps keep the debugging tractable. Centralized voice operations keep the codebase maintainable.

I'm not yet 100% sure where voice-first AI is heading. But the building experience has been the most fun I've had with software in a while. When you speak to a computer and it responds, when you watch the interface change because you asked it to, you feel like you're living in the future. The trick is making sure that future also has good logging.

What is an agent harness?

2026-06-12T00:00:00Z

My friend Sean Law asked me this question on LinkedIn recently, essentially, "Do you have a definition of the term agent harness?" I looked around and realized that nobody really has one. Simon Willison writes extensively about agents and defines the agent itself ("tools in a loop to achieve a goal") but leaves the harness unspecified. Andrej Karpathy calls "Claws" a new layer on top of agents and describes what they do without defining what they are. Anthropic's guide to building effective agents is really a guide to harness engineering, but they never use the word. The term is doing a lot of quiet work across the ecosystem. Time to make that work explicit; here is my attempt at doing so.

Defining the harness

An agent harness is everything that constrains, shapes, and thus defines what an agent can do. It has four components: the tools you give the agent, the environment those tools run in, the hard controls that architecturally limit what the agent can do, and the soft controls that steer its behavior through prompting.

Together, these four components define the agent's action space. The action space is the set of all things the agent can actually accomplish. Change any one component and the action space changes.

Hard controls vs. soft controls

Not all constraints work the same way.

Hard controls are technological constraints on an agent's action space. The agent literally cannot do the thing. If you do not give a coding agent network access, it cannot phone home. If you do not give a voice agent the ability to speak, it can only respond in text. Hard controls are architectural: they are built into what the agent physically or technologically can and cannot do.

Soft controls are the prompted nudges. You write instructions telling the agent to prefer certain behaviors, avoid certain patterns, or follow certain procedures. The agent can still deviate. Soft controls steer; they do not bind.

In practice, those instructions usually live in one of two places. An AGENTS.md file is always-on context: repo norms and preferences that ride along with every request. An agent skill file is the on-demand kind: a procedural workflow the agent pulls in only when the task at hand matches it. Both are soft controls; the difference is whether they apply all the time or just when triggered.

The distinction matters because soft controls are easier to set up but easier to bypass. (More on that later in my voice agent example.) Hard controls require more engineering effort but are more reliable.

Three harness examples

Making this concrete helps. Here are three harnesses and how they differ.

Coding agent harness

Modern coding agents like Cursor and OpenCode give you a pretty full toolkit: file read and write, bash execution, code execution, and web search. The environment is your local filesystem (usually sandboxed). The action space is broad; these agents can research, write, and run code all in one session.

The hard controls are architectural: the agent runs in a sandbox, cannot access your keychain or password manager, and has no ability to escalate privileges. The soft controls are your instructions: prefer Python, follow the testing conventions in this repo, ask before running destructive commands.

Data science agent harness

A data science agent extends the coding agent harness. It has the same tools (bash, code execution, file read/write, web search) but adds marimo notebooks to the environment. The agent works inside a running notebook kernel, editing cells, executing code, and documenting findings inline.

The hard controls are nearly identical to a regular coding agent: sandboxed execution, no access to production credentials. The soft controls are where the difference shows up. My instructions to the agent include doing code execution, editing, and documentation all inside the notebook rather than in separate files. The agent could work in standalone .py files (it has the tools), but I steer it toward the notebook because that is where the analysis lives.

Voicepal interview harness

Here is an example that makes the soft control boundary tangible. I use an interview agent inside Voicepal. It is prompted to ask me questions and not give me answers. When it asks me something and I reply, "I don't know man, you tell me," it says, "I'd love to, but my role is to ask you questions and not give you answers."

That sounds like a hard control, but it is not. I can get around it. I say, "How about you tell me one option and ask me what I think about it?" And it says, "Fair," and then gives me the option. The soft control bent without breaking. I respected the spirit of the constraint (it still asked me a question), but I nudged the agent to give me an answer inside that frame.

That is the nature of soft controls. They are persuasive, not binding. A determined user can work around them while staying within the letter of the prompt. Hard controls do not have this property. If you do not give the agent a tool, no amount of clever prompting will conjure it.

Bringing it together

There is a pattern across all three examples that took me a while to appreciate: the harness is defined by what you include and what you exclude. Every tool you withhold is as much a design decision as every tool you provide.

Here is a summary of the three harnesses:

	Coding agent	Data science agent	Voicepal interview agent
Tools	File read/write, bash, code execution, web search	File read/write, bash, code execution, web search, marimo kernel	Conversation (speech in, text out)
Environment	Sandboxed filesystem	Sandboxed filesystem + notebook kernel	Voicepal app
Hard controls	Sandboxed, no keychain access, no privilege escalation	Same as coding agent	No tool use beyond conversation
Soft controls	Prefer Python, follow testing conventions, ask before destructive commands	Do all work inside the notebook, document findings inline	Ask questions, do not give answers

An agent harness is the combination of tools, environment, hard controls, and soft controls that together define what an agent can and cannot do. The data science agent shows that two harnesses can share nearly identical hard controls and still produce different behavior, because the soft controls steer the agent in a different direction. The Voicepal example shows that soft controls are persuasion, not architecture; a determined user can work around them.

I hope this definition brings you clarity the next time someone brings up agent harnesses in conversation. And if they have a different definition, I would love to hear it; the term will only get sharper if more people take a crack at defining it.

How to Multitask Better with Agent Harnesses

2026-06-08T00:00:00Z

In my previous post, I compared three agent harnesses across workspaces, notifications, automations, and open-source status. Once you have picked a harness, the next question is: how do you actually use it?

The framework I have landed on is three tiers: one foreground task, one or two background tasks, and however many automated tasks you want running in the shadows. Right now, as I write this post, all three tiers are active. Let me walk through what each one looks like.

The three tiers

Foreground: your intellectual work

Keep one foreground task at a time. This is your intellectual work; protect it.

Writing this blog post is my foreground task right now. I am actively reviewing the draft the agent ghostwrote for me: iterating with my agent, refining what I want to say, editing prose, and nudging the agent to further match my voice. There is a meta-layer too: I have a ghostwriter skill that teaches the agent to write like me, and as I review its output, I feed corrections back into the skill itself. The writing improves, and the skill improves, in the same loop.

This is what foreground work looks like across the board: highly interactive, requiring your full attention. You are pair-coding, pair-writing, or pair-creating with the agent, steering it, reading its traces, deciding whether it is going in the right direction. When the foreground task needs you, everything else waits. Two or more foreground tasks running simultaneously is when context switching takes a real toll. I keep it to one.

Background: light supervision

One or two background tasks. Check on them when you have a gap in the foreground task.

While I write this post, I have an autolearning plugin and skill system running in the background. Useful work, but no urgent priority. I started the agent on a task before I sat down to write, and it works while I focus on prose. When I take a break, I check in, review what it did, give feedback, and go back to writing.

That is the background pattern: you started the task, you trust it, and you only check when there is a natural gap. Maybe the foreground agent is thinking, or running a test suite. You switch over, do a light-touch review, and switch back. The harness handles the work; you just need to know when it is done.

One or two background tasks is the sweet spot. It's tempting to add more; I know because I tried! I can handle two. On occasion I can handle three. Four will regularly push me past my limit.

Shadow automations: zero daily attention

As many as you want. They run on a schedule and enrich your context without asking anything of you.

Beyond foreground and background, there is a third tier: fully automated tasks that run on a schedule and need zero interactive supervision. I think of these as automations running in the shadow. They run in the background of the background, tending things so you do not have to.

I have an automation that periodically scours my knowledge vault and links notes together that should be connected. I never do this manually; the agent handles it entirely. Another one monitors my GitHub repos for merge-able Dependabot PRs and merges them on its own. A third scans my vault every hour for files with today's date in ISO 8601 format and enriches them through a state machine: transcript becomes meeting note, meeting note becomes daily bullet. I used to do this manually, copying and pasting meeting notes into my knowledge base and prompting the same thing over and over. It was mostly tedious and sometimes error-prone. Once I automated it, maintaining the vault became much easier.

One more: at work, an automation runs at 4:30 pm every day and gives me a summary of my GitHub activity. Commits, pull requests, reviews, issues. I get a digest without lifting a finger.

You only check on these at the end of the day. Did they run? Did they do the right thing? A quick review, and you are done. This tier is where natural-language automation shines. In the Codex app, you can describe the task, tell it when to run, and even tell it to delete itself when it is finished. Ephemeral automations are self-cleaning: once the task is done, the agent knows to delete the automation, so you never accumulate stale scheduled jobs.

You still need a priority list

The three tiers tell you how to run tasks. They do not tell you what to run. For that, you need a plain priority list: a task list for the day, ordered by what matters most.

Nothing fancy. I write down the things I want to get done, rank them, and that becomes the input to the three-tier system. The top item goes to foreground. The next couple go to background. Shadow automations run regardless, because they do not compete for my attention.

The goal is straightforward: if you knock off your major tasks for the day, the system is working. The tiers are there to help you do that without juggling everything at once.

How notifications tie it together

Notifications turn "I hope the background task is fine" into "I will know when it needs me." Without a notification system, you have to manually check tabs and windows, which is exactly the kind of anxious context switching the three-tier approach is designed to eliminate.

Codex, cmux, and Cursor all provide notification indicators -- dots, badges, rings, color changes. You glance at the sidebar and know what needs attention. The foreground task keeps running. Your focus stays intact.

When a notification arrives, you decide: does this need me now, or can it wait? If it can wait, let it. The background task is lower priority by definition.

The framework in practice

Right now, as I finish this post, my three tiers look like this:

Foreground: writing this blog post, iterating on prose and voice with my agent.
Background: developing my autolearning plugin and skill system, checked between writing sessions.
Shadow automations: vault linking, Dependabot merging, daily GitHub digests, meeting note enrichment. All running on their own.

The harness runs each task, tracks its state, and pings you when something needs attention. You stop keeping tabs in your head and decide only what deserves your foreground focus.

Move things down the chain

The three-tier framework is not static. The real leverage comes from constantly asking: can I push this further down the chain? Can this foreground task become a background task? Can this background task become a shadow automation?

The more you move tasks from foreground to background to silent automation, the more effortless your multitasking becomes. You are removing yourself from the loop on tasks that do not need you.

That said, be pragmatic. Some tasks genuinely need your judgment, and over-automating creates brittle systems that break in ways you do not notice. Push things down the chain when it makes sense, not just because you can.

Pick a harness. Try the three tiers for a week. See how it feels.

Exploring Agent Harnesses - A Comparative Analysis of Codex, CMux, and Cursor

2026-06-04T00:00:00Z

I have used three coding agent harnesses extensively over the past few months: the Codex app, cmux, and Cursor. All three have been immense in amplifying my productivity. All three share a similar layout: workspaces on the left, a coding agent in the middle, and a terminal on the right. A terminal and a browser are really all you need. I decided to write up how they compare.

The comparison matrix

I decided to compare them against the following axes:

Feature	Codex	`cmux` + OpenCode	Cursor
Workspace view	Yes	Partial	Yes
Notification system	Yes	Yes	Yes
Natural-language automations	Yes	Yes	Yes
Open source	Yes (Apache-2.0 CLI)	Yes	No
Built-in browser	Yes	Yes	Yes (element selection)

Workspace view refers to whether the harness provides a visual workspace abstraction where one workspace corresponds to one project, and each workspace contains multiple chat threads.
Notification system is whether the harness provides visual indicators (dots, badges, rings) that show which threads or workspaces need your attention.
Natural-language automations is whether you can set up recurring or ephemeral tasks by describing them in plain language rather than editing cron or writing scripts by hand.
Open source is obvious: whether it's open source or not.
Built-in browser is whether the harness ships with a browser pane so you can preview web apps without leaving the tool.

With those defined, let's dive in.

Workspace management

Before we go on, I think it'll be helpful to disambiguate some terms. In the following paragraphs, when we talk about a workspace, it really refers to one project, or one folder, or one git repo. (Incidentally, this is in line with the 1:1:1:1... rule that I advocate for in the Data Science Bootstrap Notes.) Each workspace contains multiple chat threads, one per task or conversation with the agent. You might have one workspace for your web app, another for a data pipeline, a third for a side project. Within each workspace, you spin up threads as needed: one to debug a failing test, another to add a feature, another to review a pull request.

Effectively, workspaces group related work, while threads keep individual tasks isolated. As a coder, you will want to see all of this at a glance, switch between threads, and know which ones are active. Without workspaces, you are stuck juggling unrelated conversations in a flat list, constantly losing context when you switch tasks.

Codex and Cursor both handle this well. You can see a list of workspaces on the left sidebar, expand into threads within each one, and pick up where you left off. Workspaces group your projects, threads track your conversations. This two-level model, workspaces plus threads, is the right visual abstraction for agent harnesses. Cursor's UI looks like the following:

On the other hand Codex's UI looks like the following:

The commonality is that workspaces (i.e. projects) are on the left, and I can quickly switch between them.

cmux provides workspace-level tabs. When paired with OpenCode, threads live either as individual tabs within a cmux workspace (more visually obvious) or as sessions within OpenCode itself (less visually obvious, since they are not visually separated). Visually, it looks like this:

Notification systems

Workspaces tell you what is running. Notifications tell you when something finishes or needs attention.

Codex and Cursor both show clear indicators: a dot, a badge, or a color change on the workspace tab when an agent completes a task. You can glance at the sidebar and know what is done without scrolling through tabs.

cmux has notification features: blue rings on tabs and a sidebar notification panel. When I first started using it, notifications were not yet exposed. I submitted a pull request to a fork that was working on them, and the feature has since shipped in cmux. You can see at a glance which tasks have finished.

cmux supports multiple agents including OpenCode, with hooks for session restore and notification wiring. Without notifications, you have to manually scroll through tabs and windows to check on background tasks, which defeats the purpose of multitasking. Notifications are only the second layer. The third piece is automation.

Automations

Automations are where things get exciting. All three harnesses let you set up recurring tasks, but they differ in how you create them and how you manage them.

Codex allows you to create automations using natural language from within a workspace. You can tell it: "Set up an automation that checks GitHub every five minutes for a deployment to finish, and then delete the automation when it is done." These ephemeral automations are self-cleaning. You describe the task, tell it to shut itself down when complete, and Codex handles the rest.

Cursor supports similar automation workflows. You can configure recurring tasks and let them run in the background.

The OpenCode ecosystem has scheduling capabilities through plugins and community tools. You configure automations using natural language, and the plugin translates that into a scheduled task. The downside is that there is no unified user interface for viewing or managing these automations yet. You can set them up, but you cannot easily see what is running (yet). That is another gap worth noting, but knowing the open source world, it'll get better!

Built-in browser

All three come with a built-in browser, which is a big deal if you are building web apps. You can see your app running alongside the agent, iterate on changes, and verify results without leaving the harness.

Cursor is the standout here. It lets you select individual DOM elements and add them directly into the context window. When you are building a web app and want the agent to focus on a specific component, you can point at it and say "fix this" without having to describe where it is. That kind of selective context control makes a real difference for front-end work.

Open source status

OpenCode, cmux, and the Codex CLI (Apache-2.0) are open source. Cursor is proprietary. The Codex desktop app builds on the open-source CLI but is itself a closed product.

This matters more to some people than others. I grew up in the open-source world during grad school, and I gravitate toward open tools. With open-source tools, if something annoys you, you can fix it yourself. In the age of agents, fixing tool limitations is more a matter of imagination than skill.

That said, I am not going to be pushy about open source here. It is a personal preference; use what works for you.

Choosing a harness

All three harnesses I've gone through here are pretty capable. I obviously did not cover Claude Code, it's well-known by now; these are just the three that I've used. They're all pretty capable; once set up properly, they handle roughly the same range of tasks.

For me, the deciding factors are practical. If your company mandates a specific tool, use that at work. Then pick a different one for home. Visual separation between work and home environments matters more than you might think. I burned out in April partly because I was using cmux with OpenCode at both work and home -- the same interface, even on different laptops, with no visual disconnect. Effectively, my brain never left the office.

Using different tools at work and at home has a bonus side effect: you avoid muscle-memory lock-in to a single vendor. You stay flexible.

Cost is another factor worth considering. If cost is a factor, OpenCode plus cmux with open-weight models through providers like OpenRouter can be significantly cheaper than subscription-based options. The free tier on some model providers goes a long way.

Try them for yourself

If you're looking for counsel on what to pick, this is what I would say: pick one, use it for a few weeks to build real familiarity, then try another. Give yourself enough time with each one to form an honest opinion. These tools are evolving fast, and the landscape will look different six months from now! Only by trying things out can you know what suits you best.

Reflections from the BioIT World workshop - standardization is worth the effort

2026-05-27T00:00:00Z

My teammate Jackie Valeri and I recently co-taught a workshop at BioIT World 2026 on standardizing data science ways of working. Walking out of the room, I kept thinking about the same tension I have seen for years: most teams already feel the pain of missing standards. They have a harder time believing the effort pays off, and a harder time doing the people work required to make change stick.

That tension is what this post is about. It is my reflection on what we covered, what landed, and what I wish we had emphasized more. The through-line is simple: standardization is worth the effort. The sections below walk through why, how to choose what to standardize, how to get buy-in, and how to keep standards alive as the stack changes.

Pain comes before payoff

Most teams need to feel the pain before they believe the payoff. Two stories from my time at Moderna show both sides.

Vignette 1: the messy codebase. Early in my time at Moderna, I studied older codebases that had no shared structure. No standard folder layout, no predictable onboarding command, no consistent way to find tests or documentation. I pitched Andrew Giessel that we had to standardize while the team was still small: if we kept going like this, onboarding would stay painful, and when the bus factor hit, we would feel it.

Vignette 2: the easy migrations. Later, when we migrated from Bitbucket to GitHub, and from Conda to Pixi, the logistics were surprisingly smooth. We had one pattern to upgrade, one CLI command to run, and minimal manual tweaking per project. Part of that was team size (we are roughly twenty data scientists + a few dozen more adjacent folks, not a few hundred). Part of it was the fruit of previously hiring people willing to learn the stack (more on that below). The biggest part was standardization itself: we knew exactly what we were migrating because we had standardized it in the first place.

The contrast is the point. Vignette 1 is what happens without standards; vignette 2 is what opens up once you have them. The upfront cost buys confidence, predictability, and speed later.

Work backwards from what you ship

Believing the payoff exists is step one. Step two is choosing which standards actually matter.

At Moderna, we applied step two by naming our deliverables first:

Compute tasks: CLI tools we can run in the cloud with as much scalability as we can manage.
Python packages: reusable components other computational scientists can import and build on.

Once we named those artifacts, the required standards became obvious. Compute tasks need Dockerfiles (containers) and CI/CD (deployment pipelines) that let us move quickly. Python packages need consistent project scaffolding, tests, and documentation.

That backward reasoning is how we chose which standards to invest in. I learned that lesson the hard way in grad school, where bad software patterns cost me days of retesting on the HPC because my code was organized for batch runs, not incremental checks. I have also heard familiar horror stories from finance, where a one-month notebook prototype became an eight-month production slog after handoff to an engineering team on a different stack. I knew which pain points were real because I had felt some of them directly and heard about others often enough.

Deliverables will likely differ, but the principle stays the same: name the artifact, and ask which practices remove friction on the path to shipping it.

Standardize where divergence creates friction

Naming your artifacts tells you the destination. It does not tell you where to invest first. Day-to-day friction does.

Standardize the places where inconsistent approaches slow the team down.

For us, the clearest example was project template structure. We had code in Julia, R, and Python. Even within Python, different projects used different frameworks. Onboarding meant learning a new layout every time. There was no single command to get started.

Once we standardized scaffolding, onboarding got faster. Jumping into a colleague's repo became predictable: I know where the docs live, where the tests live, and how to run things. That predictability compounds.

That does not mean we standardize everything. Algorithm choices, analysis approaches, and visualization design stay flexible. We focus standards on interfaces and workflows, the parts that help people collaborate without constraining the science.

Even with that selectivity, one pattern is reliable: if your team keeps re-explaining the same basics, that is where to start. Folder structure is a common answer. Documentation publishing is another strong first move when everyone already knows docs matter but nobody has a single place to put them.

People problems beat ROI math

Knowing where to standardize is the easy part. Getting people to adopt it is harder.

Adoption is a people problem, but you still have to get in the room first. That is why we spent real time in the workshop on ROI. Moderna has a strong ROI culture; working with Dave Johnson early on reinforced that return on investment matters when you pitch an initiative. We shared resources on how to calculate it, and we walked through the concept: build the case before you standardize, as the argument you bring to the table. (For a concrete example, I once worked through the math for our quarterly docathons in Two years of docathons: Insights and lessons learned.) The ROI frame of mind helps with managers who think in business terms; use it to make the strongest case you can.

A strong ROI case opens the door. It does not carry people through the migration. We are nerds, and we would rather debug a conda solve than navigate disagreement about tooling. Technology helps, but the people work still carries the change. Getting buy-in, training people, and holding hands through a migration is still work. At the end of the day, "hand-holding" is training, repeated until the new pattern feels normal.

I learned that split the hard way when I pitched standardization to Andrew Giessel years ago. I leaned on open-source patterns I knew would be cost-effective. The tooling was only half the pitch. The other half was training people on those toolsets and building a culture that maintains them together.

If you take one people tactic from this post, make it this: use ROI to open the door, then budget time for training and socialization after the demo. Adoption is a milestone, not a finish line.

Standards evolve

Training gets people onto a standard. It does not freeze the standard in place. The technology landscape keeps moving, so your standards have to move with it.

BioIT World gave us a live example of that drift. Marimo notebooks were part of our tech demo, and the audience reacted differently to Marimo than to our CLI tooling. Marimo looked cool, sure, but the deeper reaction was: "This is mindblowing! I had no idea we could interact with notebooks this way!" It was a glimpse of something new.

The excitement was real. Whether, and when, to move to Marimo notebooks is still an open decision. We are in the middle of that decision now. Jupyter notebooks served us for years. Marimo offers real advantages, especially because coding agents can work with Marimo notebooks more controllably than with classic Jupyter. I wrote more about that pattern in Use coding agents to write Marimo notebooks. Marimo is also Python-only today, which sits awkwardly next to our multi-language history. That tension is normal.

While we decide, the playbook stays the same: demo the idea, get review from peers, socialize the benefits, then train people through the migration. We have done this before with GitHub, Pixi, and other stack changes. Standardization made each migration tractable because we only had one pattern to evolve. Pixi is one example of how we choose tools; Conda may be the right call in your environment. The principle matters more than the brand name. Stay close enough operationally that upgrades stay manageable.

AI lowers the cost

As standards keep evolving, the cost of building and migrating tooling keeps dropping. That is the shift I want to name in this section.

Building the CLI commands that made our GitHub and Pixi migrations tractable used to be the hard part. Today you can describe what you want to a coding agent and iterate until you have a pyds-cli-style scaffold generator, a migration helper, or a project bootstrapper. The effort is lower; the ROI is higher.

I was already experimenting in this direction during those migrations. My first LLM-side experiment was a git commit message writer. The one that mattered for the migrations was a CLI helper that used models to propose intelligent file merges, with a human still reviewing the result. Same shape as the tools I am describing now; the difference is how quickly you can build them.

Cheaper tooling is only half the story. AI also amplifies whatever patterns already exist in your codebase. Feed it good standards, and it follows them. Leave chaos in the repo, and it reproduces chaos faster. The tools got cheaper; the human coordination problem still needs attention. Standardization pays off on both sides.

Hiring matters too

That coordination problem is where hiring and culture matter. I flagged this in vignette 2, and it is worth stating plainly: easy migrations are a people win as much as a tooling win.

Standardization works best when people are willing to learn new parts of the stack. Software skill levels can vary widely. The non-negotiable trait, for us, is curiosity about the tooling we use to do science. That is a form of gatekeeping, and I am fine saying so. If someone refuses to adapt when the team agrees on a shared pattern, friction returns.

Hiring for learning agility is one place leaders can start, especially if they are building a team from scratch. If you already have a team in place, you cannot rehire your way out of friction; start with the software pattern that removes the most daily friction, then grow the culture from there.

What would you start with?

You have seen the payoff, the decision rules, the people work, and the stack as it keeps moving. The remaining question is practical: where would you start?

What is one thing you would like to start standardizing for your team?

Pick one. Folder structure. Dependency management. CI/CD for every repo. Documentation templates. One choice, executed well, beats a grand roadmap that rarely ships.

Getting started can be as small as a half-day survey of tools in your landscape, or a conversation with your manager backed by an ROI estimate. You can even ask an AI assistant to walk you through Pixi, Cookiecutter, or whatever tool you are evaluating. The specific path is yours; the important part is to start.

If you want to go deeper before you pick that one thing, two resources cover adjacent ground. For delivery models, scaffolding, and implementation tactics, see my blog post, How to Standardize Data Science Ways of Working to Unlock Your Team's Creativity. For machine setup, project structure, and the fundamentals underneath all of this, see my online eBook, The Data Science Bootstrap Notes.

Acknowledgments

This work has always been a team sport. With thanks to Andrew Giessel, Dave Johnson, Adrianna Loback, Rebecca Vislay-Wade, Jackie Valeri, Albert Lam, Anand Murthy, Dan Luu, and other colleagues who helped design, maintain, and evolve our standards over the years. Andrew and Dave gave us the freedom to build; my current manager Wade Davis continues to give us operational room to make it happen, and for that I am grateful!

What data science is actually about in the age of AI

2026-05-20T00:00:00Z

I have been thinking about what the core mission of a data scientist actually is in 2026, surrounded by AI coding assistants, LLM-powered applications, and the relentless buzz around adaptive software development. The answer keeps coming back the same: measurement.

LLMs and AI coding tools are new instruments for that work, not replacements for it. And instruments do not decide what to measure. People with domain knowledge do. The principle is simple and ought to be stated plainly: measurement must be defined by the people closest to the problem.

But staying close to the problem is harder than it sounds. Something is pulling data scientists away from that mission, toward a very different self-image. It worries me.

The seduction of full-stack

I keep running into a narrative that data scientists should become full-stack software developers. The logic goes something like this: you can already code, and now AI makes coding even easier, so why are you not contributing to the production codebase?

Managers hear that AI-assisted coding has arrived and think every data scientist on their team can now own a slice of the front-end, the back-end, the database, and the deployment pipeline. Data scientists get pulled into writing TypeScript for UI components, setting up production databases, and building integrations with other systems, because those tasks fill the week. The work that made them valuable in the first place (defining metrics, designing evaluation frameworks, rigorously measuring whether a system actually works) gets squeezed into the margins.

This is not an accident or an outlier. The whole industry is zigging toward turning data scientists into app developers (Acceldata, Deloitte, GSDC). The zig is understandable: building a GenAI system has never been easier. Call an API, wire up a prompt, ship a demo. But the easy part is not where the gap is. The gap is reliability, ensuring these systems actually work in production, at scale, over time. That is the zag, and it is where data scientists create massive alpha.

Why this is a mistake

The zig is tempting because building is seductive. But AI amplifies strengths and weaknesses alike, and a team of generalists building sloppily gets amplified into a production incident. Your ability to zag depends on having people trained for reliability work, and pulling them into the build instead defeats the purpose. What data scientists contribute is not a stack of engineering skills but a form of reasoning that takes years to develop, and that reasoning is exactly what GenAI systems need most.

Most data scientists I know came from scientific backgrounds. Economists, biologists, chemists, sociologists. Their prior training is in hypothesis generation, experimental design, measurement, and rigorous analysis. That training is hard to replicate. It takes years to move from knowing the techniques to developing the scientific judgment needed to look at a set of results and know whether they hold up.

When you pull a data scientist into full-stack software development, you are trading that hard-won scientific expertise for a second-rate software engineer. Data scientists will make immature decisions about database selection, front-end framework choices, and system architecture that an experienced software developer would avoid. The engineering suffers from inexperience, and the science suffers from neglect.

The cost shows up in two ways. First, cognitive switching between measurement thinking and software engineering thinking degrades both. Building a UI component requires a very different mode of thought than designing an evaluation framework for a stochastic system. Second, the measurement work itself gets displaced entirely. The data scientist who should be spending their week figuring out whether the LLM pipeline is actually performing well is instead debugging a CSS layout.

The framing I want every data scientist and their manager to internalize is a trade-off. Every hour spent maintaining databases and front-ends is an hour that cannot be spent on the measurement work the team is hired to do.

What data scientists should focus on

So if full-stack development is the wrong direction, what is the right one?

The clearest answer is LLM evaluation. Every company building with LLMs needs to know whether their system works. The answer requires exactly the kind of rigorous, scientifically grounded measurement that data scientists are trained to do.

The process is straightforward. You need labeled data that represents what "good" looks like. You run your model against that data. You write programmatic checks to evaluate the model's outputs. You quantify metrics based on those checks. Then you iterate.

The tools for doing this are immature, which is actually an opportunity. You do not need an expensive vendor evaluation platform. What you need is the ability to select and label your own data, store it in a simple database, and run checks against it. This is where data scientists can and should build utilities for their own work. I will come back to the line between building a measurement utility and practicing software engineering shortly.

Building the utilities is the straightforward part. The harder challenge is the scientific logic. Deciding what to measure, debating which metric actually reflects the business problem, and figuring out whether an improvement is real or noise all require exactly the thinking trained in graduate programs. Hamel Husain makes this concrete in his post on the revenge of the data scientist: every common eval pitfall, from generic metrics to unverified judges to bad experimental design, traces back to a missing data science fundamental.

Where tool-building ends and software engineering begins

The line between building a measurement utility and practicing software engineering is the nuance that matters most, so let me be specific.

Data scientists should absolutely build tools for their own measurement work. The principle is simple: build single-purpose, tightly scoped utilities with no feature creep, where you are the primary user. I have built CLI tools that do one thing and do it well, in the Unix tradition. I built them for myself first, and if someone else happens to have the exact same problem, they can use the tool as-is. That is the dogfooding principle in action.

For example, if I need to do LLM evaluations, I need labeled data. Instead of waiting for a software development team to build a full-stack application for data labeling, I can quickly set up a lightweight database with a simple UI that lets collaborators label data. The whole setup is a measurement tool, not a product.

The line between tool and product turns on three criteria.

The first is scope. If the tool does one high-value thing really well, it belongs in the data scientist's remit to build. If it needs to evolve to handle diverse use cases, feature requests, and edge cases from dozens or hundreds of users, it has crossed into software engineering territory.

The second is audience. If you are the primary user and others incidentally benefit, you are building a measurement tool. If you are building something primarily for other people to use, you are doing product development, and that belongs with the software engineering team.

The third is the production boundary. It is fine for data scientists to prototype UIs to show what is possible, and to own the model and its API endpoint. It is fine to build tooling that supports data collection and measurement. But the finished product belongs with professional software engineers: the chat system integrations, the production database, the user-facing front-end, the infrastructure that other teams depend on.

When you cross that line, data engineers (the people who build and maintain production data pipelines) and software engineers should own the systems. Data scientists can be team players and pitch in when needed, but maintaining those systems should never become their full-time responsibility.

Measurement done right, in practice

Ownership boundaries are one kind of clarity. Another is knowing how to measure well. Let me make this concrete with two examples from different corners of data science.

Document parsing for information retrieval. This is a problem nearly every team working with LLMs faces. You have a pile of documents and you need to extract structured information from them. The natural metrics are precision and recall. But here is the trap: if you optimize only for recall, you extract too much noise, and someone downstream has to filter garbage. If you optimize only for precision, you miss important information that actually matters for decisions downstream.

Picking one metric in isolation leads to real problems. The right measurement framework has to be debated and co-defined with the people who understand what the extracted information will be used for. The data scientist brings the rigor; the domain experts bring the context. Neither can do it alone.

The same pattern appears in a completely different setting. ML models for property prediction in preclinical research. A data science team builds a machine learning model to predict molecular properties. The model has an R-squared value and other statistical metrics. But those numbers have to translate into real decisions: which mutant should we select for the next round of engineering, which compound is worth synthesizing.

The metrics that matter are co-defined between the lab team and the data science team. The lab scientists know what actually matters for their experiments, and the data scientists know how to build models and measure their performance. Together, they arrive at a measurement framework that bridges statistical performance and experimental reality.

Both examples share a common pattern. The measurement framework is designed by the people closest to the problem, not imposed from above or derived from whatever metric is easiest to compute.

For managers - reframing the conversation

If you manage a data science team, or if you are an executive making staffing decisions, the question is: what should you actually do with this framing? Here are four practices I want you to take away.

Reframe how you define problems. Instead of framing work as "we are going to build a solution," frame it as "we are going to figure out how good this solution is." Measurement comes first, not after. Embed it throughout the product development lifecycle, from the earliest prototype to the shipped product. If you bolt measurement on at the end, you are flying blind during the entire build.

Start with the measurement problem, not the headcount. Identify what specific evaluation or measurement problems you need solved, and build your team around that. The staffing follows from the measurement needs. The ideal ratio in my experience is at least three software developers for every one data scientist, though this depends heavily on the complexity of the product. When I see teams with two data scientists and one software engineer, the data scientists inevitably end up filling the engineering gap.

Go for the quick win. Find something the team already thinks is broken by gut feel. Measure how broken it actually is. Make a change. Show that the metric improves. Do this fast, days or a couple of weeks, not months. The trap here is real: if a data scientist takes too long to surface a measurable improvement, the team loses faith in measurement work and defaults back to gut feel. Speed matters for credibility.

Prioritize based on expected ROI. Every measurement effort is a bet. Work backward from the expected business impact. If we improve this metric by some amount, what is the expected gain? Use that calculation to decide what to measure first.

The trade-off that matters

Put those four practices together and you get a picture of what the data scientist's role should look like in 2026. It is the same as it was in 2016: define the question, collect the data, build the tools you need, measure rigorously, and communicate what you found. AI coding assistants and LLMs are new tools for doing that work, not a change to the work itself.

When managers pull data scientists into full-stack software development, the data scientist loses the chance to do the measurement work they are trained for, the team loses the scientific rigor that only a data scientist can provide, and the product loses because nobody is asking the fundamental question: how do we know whether this thing actually works?

If I spend my time maintaining databases and front-ends, I cannot do the measurement work you hired me to do. Building is easy now. Reliability is the gap. And the people best trained to close it are the ones being pulled into CSS layouts.

ODSC East 2026's Zeitgeist and Conference Report

2026-05-10T00:00:00Z

This year I ran a small experiment at ODSC East 2026.

As I was speaking and catching up with old friends at the conference, I could only attend a slice of sessions. So I thought, rather than try to catch every last talk, what if I could figure out what the zeitgeist of the conference was using just the talk abstracts? If I scraped the schedule and abstracts across talks, workshops, and keynotes, can they reveal the conference center of gravity, and hence the zeitgeist of the conference?

How I ran the experiment

I used Cursor Agent on Premium for the scrape and extraction workflow.

The process was straightforward:

Use https://schedule.odsc.ai/ as the source of truth
Query the schedule backend session table and filter to the East 2026 event ID
Pull session title, abstract, day, track, format, and speaker metadata
Save the full result as structured JSON for analysis

This pass includes 237 sessions from the live schedule backend, exported as structured JSON.

The map after reading abstracts

I ran a multi-agent categorization pipeline. Four independent coding agents each read 50 full abstracts and proposed a five-category taxonomy. Then three cross-review agents read all 200 abstracts and all four proposals, identified points of agreement and disagreement, and each produced a unified taxonomy. A final arbitrator agent resolved the remaining disputes using majority vote, reading the abstracts directly for tiebreaks.

Corpus snapshot:

237 total sessions in the schedule export
200 sessions with non-empty abstracts
193 substantive sessions after filtering logistics and networking entries
7 excluded as non-substantive (badge pickup, happy hours, morning run, career fair, networking receptions)

The five categories that emerged from this process follow.

Zone 1 - Agentic AI systems (55 talks, 28% of substantive)

This zone captures the shift from prompt craft to system design: agent architectures, multi-agent orchestration, tool use, MCP and A2A protocols, harness engineering, agent memory, simulation sandboxes, guardrails, and production deployment patterns for systems where AI plans, decides, and acts autonomously.

Sub-zeitgeist in this zone:

Teams are wrestling with a core systems question: what architecture keeps agents reliable when they run for long periods, call many tools, and fail mid-flight?
The recurring design problem is orchestration and control, from runtimes to control planes to workflow verification.
This was the largest zone at the conference, reflecting how agentic AI has moved from demos to engineering.
Talks that anchor this pattern include:
- Architectural Patterns for Building and Governing Production-Grade Multi-Agent Systems (Dr. Ali Arsanjani, Google Cloud)
- Agents Don't Need Better Frameworks: They Need a New Runtime (John A. De Goes, Golem Cloud)
- Building Production-Ready Agentic AI - Why a Control Plane Matters (Wayne Segar, Dynatrace)
- Verification-Driven Agentic Workflows (Julie Yaunches, NVIDIA)

If you are building agentic systems right now, treat architecture as the first design surface. Start with runtime boundaries, failure recovery, and orchestration patterns, then layer prompts into that structure.

Zone 2 - LLM and foundation model engineering (37 talks, 19%)

This zone covers the model layer: training, fine-tuning, quantization, inference optimization, RAG architecture, prompt engineering, model interpretability, hallucination research, evaluation methodology, and novel computational substrates.

Sub-zeitgeist in this zone:

The dominant production question is deployment economics: how do we sustain quality while meeting latency, throughput, and cost targets?
The technical pressure points are quantization, efficient fine-tuning, inference behavior under load, and hardware-induced variability.
Talks that capture this pressure include:
- Less Compute, More Impact: How Model Quantization Fuels the Next Wave of Agentic AI (David vonThenen, NetApp)
- Efficient Finetuning of Quantized LLMs (Tim Dettmers, CMU / Ai2)
- Fast, Cheap, and Accurate: LLM Inference in Practice (Legare Kerrison, Red Hat)
- Why AI Systems Behave Differently in Production: Nondeterminism, GPU Execution Drift, and Hidden Reliability Gaps (Santosh Appachu Devanira Poovaiah, NVIDIA)

Use this zone as a nudge to benchmark for real operating conditions, not demo conditions. Optimize for latency, cost, and stability together, and pick model and fine-tuning strategies that survive production constraints.

Zone 3 - Data engineering and ML infrastructure (28 talks, 15%)

This zone includes the substrate work that determines whether agents reason well in production: data platforms, pipelines, data quality frameworks, real-time streaming architectures, training and serving infrastructure, data modeling, and ML operational systems.

Sub-zeitgeist in this zone:

The shared diagnosis is that many agent failures are context failures, data-shape failures, or lineage failures before they are model failures.
The leading question is how to build context substrates that preserve meaning across retrieval, reasoning, and execution paths.
Talks that make this explicit include:
- What "AI-Ready Data" Actually Means: A Framework to Trustworthy AI (Jacob Prall, Snowflake)
- Entity Resolved Knowledge Graphs: The Foundation for Effective GraphRAG (Clair Sullivan, Clair Sullivan & Associates)
- Real-Time Event-Time Consistent Analytics Pipelines using Kafka, Flink, and Apache Pinot (Deep Patel, Robinhood)
- Beyond the Context Window: Anchoring Agentic Reasoning with World-State and Context Graphs (Amy Hodler & David Hughes, GraphGeeks.org)

For readers shipping AI systems, this is a reminder to diagnose context and data pathways before blaming the model. Invest in data contracts, context structure, and retrieval quality early, because those choices determine downstream reliability.

Zone 4 - Applied AI, domain solutions, and foundations (41 talks, 21%)

This zone spans two related audiences: introductory training courses teaching core skills (Python, SQL, R, statistics, ML basics) and domain-specific AI applications in healthcare, finance, defense, biopharma, accessibility, and marketing. The common thread is that the audience is learning or applying rather than researching.

Sub-zeitgeist in this zone:

The main adoption problem is role redesign: what should humans own when agents can execute large parts of the workflow?
The recurring organizational question is how to scale AI usage while preserving accountability in regulated and high-stakes domains.
Talks that carry this thread include:
- Ready for Primetime: Creating the AI-enabled Clinician (Ami Bhatt, MD, FDA / American College of Cardiology)
- Panel: Building the AI-Ready Workforce: Human-AI Collaboration in 2026 (Usama Fayyad, Sadie St Lawrence, Sheamus McGovern, Dr. William Streilein)
- The Expertise Upheaval: How AI Promises to Transform the Nature of Work (Matt Sigelman, Burning Glass Institute)
- What Does a Data Professional Do When AI Can Do the Data Work? (Shane Butler, Ontra)

The move for practitioners here is to design human roles and escalation paths alongside technical architecture. Adoption succeeds when accountability, decision rights, and domain workflows are specified as clearly as APIs and evals.

Zone 5 - AI strategy, governance, and workforce (32 talks, 17%)

This zone is where technical capability meets organizational uptake: enterprise AI strategy and transformation, governance frameworks, regulation and compliance, trust engineering, workforce transformation, career development, and the societal and human dimensions of AI.

Sub-zeitgeist in this zone:

The central problem is assurance: how do we prove behavior quality for agentic systems that adapt and call tools dynamically?
The recurring question is how to convert policy and risk language into tests, guardrails, and monitoring loops that engineers can run every day.
Talks that anchor this zone include:
- Beyond Static Benchmarks: How to Test AI That Thinks, Acts, and Adapts (Yash Vijay, Snorkel AI)
- Lessons from Evaluating Production AI Agents for over a year (Susan Shu Chang, Elastic)
- The Leadership Imperative: AI Governance as Strategic Infrastructure (Shoshana Rosenberg, WSP in the U.S.)
- How I learned to Stop Worrying and love AI Regulation (Benjamin Batorsky, GoGuardian)

A practical takeaway for teams is to translate trust goals into runnable checks. Build evaluation and guardrail loops that run continuously, and make policy language executable in your delivery workflow.

Visualizing the conference landscape

To see how these categories relate to an unsupervised view of the same abstracts, I embedded all 193 substantive abstracts using all-MiniLM-L6-v2, clustered them with HDBSCAN, and projected the embeddings into 2D with UMAP. The companion Marimo notebook includes an interactive scatter plot where you can toggle between the agent-based zone classification (the five categories above) and the embedding-based HDBSCAN clusters. Mouse over any point to see the talk title, speaker, track, and full abstract. You can open it directly in your browser with molab.

The two views do not perfectly align, and that is instructive. Where the embedding clusters split a zone, it usually means the zone contains sub-communities with distinct vocabulary (for example, "agent architecture" talks cluster separately from "agent ops" talks within Zone 1). Where the embedding clusters merge zones, it means the abstracts share enough language that an unsupervised method cannot tell them apart.

What the center of gravity says in 2026

If I compress the whole conference into one sentence, it is this: ODSC East 2026 feels like the year AI builder culture became systems culture.

The numbers bear this out. Agentic AI systems accounted for 28% of substantive sessions, more than any other zone. LLM and foundation model engineering took another 19%. The event still celebrates model progress, but the practical energy sits in the joints between components: agent architecture, data infrastructure, evaluation, deployment, and decision-making structures inside organizations.

I also notice a healthy coupling of technical and leadership tracks. That pairing usually shows up when teams are moving from experimentation budgets to accountability budgets.

A short list of likely macro trends

Agentic AI becomes an engineering discipline with explicit quality loops
RAG and context strategy remain central, but with more scrutiny on evaluation
MLOps and LLMOps converge into one reliability stack for mixed AI systems
Governance work sits closer to platform design and product strategy
The value of attending this conference shifts from "learn a model" to "learn a repeatable system"

My own field note from ODSC East

I taught a 1 hr workshop called "How to Do Agentic Data Science" on Day 1. The abstract reflected my original plan: use Python scripts executed with uv run, one script per plot, and pair that with markdown journals and reports so the workflow stayed inspectable and reproducible.

Then I learned about Marimo Pair about a month before the session. At that point, conscience kicked in for me. I felt a strong obligation to avoid giving folks a workflow that could feel outdated within a quarter of a calendar year. So I pivoted live and ran a coding demo with Marimo Pair because that workflow felt cleaner and tighter for the same core ideas I wanted to teach. Those ideas stayed simple: slow down, look at your data directly, gate analyses one plot at a time, and use LLMs to help with documentation while keeping human judgment in charge. Marimo made that "look at your data" step much more natural through direct dataframe display in the notebook flow.

Overall, I was floored by the early response - when I ducked out briefly to grab some water, I saw a wall of people outside, and the demand hit me all at once. Others I met in the hallway and in the VIP/Speakers room gave overall positive feedback, and agreed with my own assessment that the content would be better done with a longer session, which would give me the space to be more hands-on.

How I Recognized and Handled AI Burnout

2026-05-02T00:00:00Z

This blog post is for anyone who is experiencing, or has witnessed people experiencing, AI-related burnout. I consider it a post-mortem of my own experience.

For the first two weeks of April, I was experiencing a severe bout of anxiety. To those who know me, this is very much foreign and out of whack from my usual self -- which I describe by my spirit animal, a capybara that can calmly sit atop a crocodile and be at peace with many other animals. In fact, my profile pic at work is a capybara. Anxiety is decidedly not part of the capybara persona.

And yet, during those two-ish weeks, I observed myself experiencing the following symptoms:

Heavier breathing
A tingling nervous feeling in my arms
A lack of desire to get out of bed
Racing heart rate when I arrived at work
General elevated heart rates throughout the week
No desire to use my personal laptop for hacking after hours
Decision paralysis at home, not knowing what to make for breakfast, for example
A desire to just binge watch youtube on the couch and in bed

Observation 6 was really telling: most evenings, I'm hacking on my laptop. I'm hacking with AI. I'm tinkering with new tools. I'm blogging in Obsidian, like I am now when I'm writing. But during those two weeks, I didn't want to do anything.

There were additional compounding factors at work. I was overbooked at work, frequently having triple or quadruple bookings on time slots. (These stress me out, I don't like saying no to people by way of declining an event.) There were issues with operations and personnel that were way less exciting than learning about quantum computing, designing and architecting ML systems for molecules, and guiding my teammates on AI evals.

Moreover, it's spring, my least favourite time of the year, because, swirling like an unrelenting dust hurricanes criss-crossing multiple states, plant gametocytes unleash terror on every cell and connective tissue on my eyes, nose, and lungs. 'Tis the season to be sneezing.

For the majority of those two weeks, it was a pretty shaky time for me. I simultaneously became snappier towards those who were asking me impromptu questions at work (whether on Teams or in the office) and more rant-y towards those who were asking me how I was doing.

Unsatisfyingly for me, I still cannot pin down the exact cause of the stress. But what helped me here was a change in perspective.

Before that trip, though, something at work shifted in a way I almost left out of this story because it feels ordinary when you write it down; it wasn't ordinary to me at the time. A few teammates stepped in and shouldered pieces of the load without making a fuss about it: coverage here, a clearer owner there, a meeting that stopped being my problem to hold in my head. That kind of quiet redistribution is easy to forget afterwards because it doesn't show up on a slide; your nervous system notices. I also told a handful of colleagues that I was running on fumes, more bluntly than I'm used to admitting at work. They didn't try to fix me with advice I didn't have bandwidth to absorb; they mostly listened and offered encouragement, and a couple of check-ins later that week landed softer than I expected. I'm naming this because burnout narratives often read like solo endurance runs; my experience wasn't like that. Other people helped carry the week, and I'm grateful for it.

Firstly, the timing of a week out in London, UK was superbly fortuitous. My wife planned it out, and, instead of being a leader and taking initiative, I just followed along with the flow. This helped immensely! I was, after all, suffering from decision paralysis. Not having to weigh the consequences of decisions was a weight off my shoulders. I just quietly followed others' lead in our travel group. (We traveled with another family together.)

Secondly, while in the UK, we visited two museums whose exhibits helped me reset my perspective on time. We saw ancient relics from prehistoric to Biblical times at the British Museum, and cosmic time scales at the Science Museum in the outer space exhibits. Pondering the passage of time, it's hard to resist comparing those timescales to the compression of time I've experienced at Moderna with our multiple reorganizations, and to the first quarter of the year when I went all-in on experimenting with coding agents.

Yes, time at Moderna seems to be compressed, and with the restructuring of the company over the years (I finally understand why it's called restructuring, and I don't mean that cynically, but in the most precise sense of the word), some folks have had 7 managers over two years. I have been fortunate to have only had one managerial change. And the pace at which I was able to test ideas with coding agents was fascinating, thrilling, and exhilarating, but it also left me feeling unmoored and drifting, sometimes questioning the permanence of what I was building.

But the contrast of the relative permanence of ancient relics and the immense scale of cosmic time when compared to the three months of coding agents or four years of work at Moderna reminded me of so many interconnected ideas, of which this is just a sampling: the sediment of the fossil record being like the sediment of software, of the teachings of the Teacher in Ecclesiastes:

"Meaningless! Meaningless!" says the Teacher. "Utterly meaningless! Everything is meaningless."

Ecclesiastes 1:2 NIV https://bible.com/bible/111/ecc.1.2.NIV

Meaningless here comes from the Hebrew word Hevel (הבל), which has a deeper meaning than mere meaningless: it is a metaphor for the transitory nature of our work -- and even more so for AI assisted coding, in which we can, at will, reshape entire codebases to fit the image in our heads, or, even more, fit the vibe that we seek.

Pondering the natural evolution of human systems and innovation, and contrasting that against cosmic time and long arcs of history, made me realize how things that last are rarely only due to thoughtful input from humans -- they persist in a path-dependent fashion, reliant on both (a) timeless design and engineering, and (b) path dependent reliance on circumstances, demand, and opportunity seized by their creators.

Evidently, this grander and broader sense of time gave me much to ponder on.

Thirdly, I saw a different pace of life. Brits, like the Swiss folk I saw in Basel, go out for drinks at 4 pm. It's part of the social fabric, the "ways of working". This is much unlike my frenzied pace at work where I am still fielding calls at that time, or catching up on the dozens of unread Teams chat messages that have piled up while my attention was diverted somewhere else in a 1:1 or other meeting.

Witnessing the slower pace helped me reset my own expectations on my time. It also relates to Mario Zechner's talk on the pi coding agent he created, in which he appeals to the AI Engineer audience to slow down. It also helped that we mostly walked everywhere, rather than zipping around the city in a car (unless we were genuinely tired) or an ebike -- the latter being how I usually make my way around the greater Cambridge area. Again, it was an intentional change in pace.

Fourthly, having a week out where I did not check my work phone or laptop -- I left them at home -- and could focus on my family was insanely refreshing. By the time March rolled by, I found I had not paid enough attention to them at home. At first, my kids would ask me to play with them every night, but over time, as I turned them down over and over, they got acclimatized to playing on their own and stopped asking me. I'm glad I noticed this earlier than later! They grow faster than we realize, and being present is, I now realize, very important for me, especially given the circumstances I experienced when I was younger.

Finally, as I had a chance to catch up with my younger brother Evan and his wife May, I observed a different set of life priorities. They really know how to enjoy life! And they deploy their finances in a way that reflects this set of priorities. They've been all over Europe making memories, tasting food, and more. And yet for myself, even before I got married and had kids, I was always hesitant to spend my own money to enjoy life. (A bit of this might come from seeing first-hand financial decimation due to debt one generation above, and from being a student for another 10 years after graduating from junior college in Singapore.) Instead I would redirect my time and energies to working and learning... but sometimes at the expense of my health and enjoyment of moments around me. I would travel only for work, and that severely limited what I saw.

There are clearly other things that helped me reset.

Averaging about 13K steps a day definitely helped to the point that I picked up the habit of morning walks upon returning to Boston.

An afternoon playing retro computer games in the Science Museum, in which I picked up the game Kingdom Hearts for the first time and played Pac-Man with my older kid, helped give me a good few hours of mindless enjoyment.

Not talking AI all day long helped remind me that there is a world of stuff happening outside of wafers of sand pulsing with electric currents.

I used my phone to connect with Scripture again, and it gave me many indirect theological reminders that the technological tools I have at my disposal need to be in the service of others, not myself.

And of course, as someone who enjoys food much, it was amazing to try out new dishes in London's food scene.

If these all sound mundane to you, the magic may simply lie in the mundaneness that humans need. We need the stillness of meditation and the connection with other people. By contrast, AI is both incredibly stimulating and isolating -- it keeps the mind buzzed on solo work, taking away the old frictions that necessitated the negotiation between real human beings, conditioning us to accept sycophantic answers... Quite significantly, it is the antithesis of what allows for humans to thrive, if we develop a reliance and attachment to it.

If you've made it this far, then here are some of my recommendations, if you're also feeling the same kind of burnout I was experiencing. These are also the changes that I will be making going forward.

Plan the work you want to accomplish for the day with AI and then stop when you've accomplished it. It should occupy the four best hours of your day. Allow for space to think and strategize, and to handle administrivia.
Touch grass daily, literally, please.
Go for a long meandering walk each day. Do it first thing in the morning if you can get up. It'll be beneficial for your daily brain and your body. Do not bring your phone. Let your mind wander, enjoy nature, the sound of birds chirping and leaves rustling.
If you are building something for someone else using AI, make sure you have constant contact with that person or group of people. It is very rewarding to hear feedback on how things can be made better, and it is superbly grounding too.
Learn how LLMs work and are trained. They will help you appreciate why AI is not a sentient being to become attached to, and should instead be treated as a (very high power) tool to be wielded in creative ways.
Go play some board games with real people. At my level of sophistication, tic-tac-toe and Zingo with my kids works perfectly well. Find your crowd.

Point 5, I believe, is the most important thing to remember. AI is a tool, and a really high powered one at that. It is tempting to squeeze out every last ounce of productivity -- but to the detriment of yourself. As I wrote in an earlier blog post, there is a sweet spot of amplified capacity that you can hit, but shouldn't go any further. It is tempting to count the opportunity cost of not pushing yourself just a bit more, but as Jeremy Utley (who quotes Clayton Christensen) puts it bluntly, it's not worth the while. (How will you measure your life? by Clayton Christensen is also worth a read.) We are human beings, not human doings. Learning how LLMs work and are trained will help you appreciate that LLMs and coding agents are just stochastic parrots, albeit nonetheless a useful class of such parrots when steered properly.

I hope reading this blog post helps you. Above all, if you're genuinely feeling anxious because of AI, know that the hype will die down, and the genuinely useful patterns and parts that help us live better lives will remain. We will need to redesign our economies, societal structures and sense of where we derive our worth, no doubt. Humans can have tasks automated, but what makes us human, and thus valuable, is not replicable in a machine. Ultimately, people want to be in relationships and fellowships with other people. And that makes us indispensable.

Benchmarking LLMs with Marimo Pair

2026-04-08T00:00:00Z

Marimo Pair has been released!

I've known about it since 11 March, when Trevor Manz did a demo over a Google Meet call, and I'm thrilled to see it being announced officially! I also had Trevor showcase it to the Agentic Data Science Workshop that I led on 3 April as a fundraiser for the SciPy Conference Financial Aid Program.

Now, one thing I know about Trevor is that he almost exclusively agentically codes with Claude Code. But I'm an OpenCode user, and in the interest of remaining vendor-agnostic, I wanted to check to see how good Marimo Pair's agent skill is when, ahem, paired up with various LLMs within the OpenCode harness. To do so, I decided to spend a few dollars and do a quick benchmarking exercise.

Skill environment check

To start, I verified that my skills environment doesn't contain anything that could be data science-y in nature, so as to avoid interfering with the marimo-pair skill. I checked my global skills:

marimo-pair-benchmark on  main on ☁️  eric.ma@nonlinearlabs.ai
❯ npx skills list -g
Global Skills

Marimo Pair
  marimo-pair ~/.agents/skills/marimo-pair
    Agents: Claude Code, OpenClaw

General
  agent-browser ~/.agents/skills/agent-browser
    Agents: not linked
  agents-md-improver ~/.agents/skills/agents-md-improver
    Agents: Claude Code, OpenClaw, Cursor
  ast-grep ~/.agents/skills/ast-grep
    Agents: not linked
  claudeception ~/.agents/skills/claudeception
    Agents: not linked
  continuous-learning-v3 ~/.agents/skills/continuous-learning-v3
    Agents: not linked
  design-driven-dev ~/.agents/skills/design-driven-dev
    Agents: not linked
  find-skills ~/.agents/skills/find-skills
    Agents: Claude Code, OpenClaw, Cursor
  gh-activity-summary ~/.agents/skills/gh-activity-summary
    Agents: Cursor
  gh-cli ~/.agents/skills/gh-cli
    Agents: Cursor
  gh-daily-timeline ~/.agents/skills/gh-daily-timeline
    Agents: Cursor
  github-activity-summarizer ~/.agents/skills/github-activity-summarizer
    Agents: not linked
  google-calendar-manager ~/.agents/skills/google-calendar-manager
    Agents: not linked
  html-presentations ~/.agents/skills/html-presentations
    Agents: not linked
  pinchtab ~/.agents/skills/pinchtab
    Agents: not linked
  post-edit-error-check ~/.agents/skills/post-edit-error-check
    Agents: not linked
  publish-to-google-docs ~/.agents/skills/publish-to-google-docs
    Agents: Cursor
  revealjs ~/.agents/skills/revealjs
    Agents: Cursor
  roborev:address ~/.agents/skills/roborev-address
    Agents: not linked
  roborev:design-review ~/.agents/skills/roborev-design-review
    Agents: not linked
  roborev:design-review-branch ~/.agents/skills/roborev-design-review-branch
    Agents: not linked
  roborev:fix ~/.agents/skills/roborev-fix
    Agents: not linked
  roborev:respond ~/.agents/skills/roborev-respond
    Agents: not linked
  roborev:review ~/.agents/skills/roborev-review
    Agents: not linked
  roborev:review-branch ~/.agents/skills/roborev-review-branch
    Agents: not linked
  skill-creator ~/.agents/skills/skill-creator
    Agents: Claude Code, OpenClaw, Cursor
  skill-installer ~/.agents/skills/skill-installer
    Agents: not linked
  vault-title-renamer ~/.agents/skills/vault-title-renamer
    Agents: not linked
  write-like-eric ~/.agents/skills/write-like-eric
    Agents: not linked
  youtube-ingestion ~/.agents/skills/youtube-ingestion
    Agents: not linked

And within my repo, marimo-pair-benchmark:

marimo-pair-benchmark on  main on ☁️  eric.ma@nonlinearlabs.ai
❯ npx skills list
No project skills found.
Try listing global skills with -g

Though the marimo pair skill is available globally, I decided to install it locally as an override.

marimo-pair-benchmark on  main [?] on ☁️  eric.ma@nonlinearlabs.ai
❯ npx skills install marimo-team/marimo-pair

And so now we're ready to go:

marimo-pair-benchmark on  main [?] on ☁️  eric.ma@nonlinearlabs.ai
❯ npx skills list
Project Skills

Marimo Pair
  marimo-pair ~/github/marimo-pair-benchmark/.agents/skills/marimo-pair
    Agents: Antigravity, Cursor, Gemini CLI, OpenCode

I then start a marimo server within this repo:

marimo-pair-benchmark on  main [?] on ☁️  eric.ma@nonlinearlabs.ai
❯ uvx marimo edit --sandbox --no-token

        Create or edit notebooks in your browser 📝

        ➜  URL: http://localhost:2719

        💡 Tip: Coming from Jupyter?
                Guide: https://docs.marimo.io/guides/coming_from/jupyter/

        🧪 Experimental features (use with caution): external_agents
        🌐 MCP servers: marimo

I intentionally start up in --sandbox and edit mode with --no-token to make it easier for the coding agent to connect.

Data analysis task

Our task at hand is as follows. I have data from a paper I published while at Novartis.

marimo-pair-benchmark on  main [?] on ☁️  eric.ma@nonlinearlabs.ai
❯ ls data/ired-novartis
Permissions Size User    Group Date Modified Git Name
.rw-r--r--@ 1.0M ericmjl staff  7 Apr 21:22   -- cs1c02786_si_002.csv
.rw-r--r--@  21k ericmjl staff  7 Apr 21:22   -- cs1c02786_si_003.csv
.rw-r--r--@  12M ericmjl staff  7 Apr 21:22   -- ired-master-table.csv
.rw-r--r--@  12k ericmjl staff  7 Apr 21:22   -- layouts.csv
.rw-r--r--@ 1.1k ericmjl staff  7 Apr 21:22   -- README.md

This file, cs1c02786_si_002.csv in particular includes single, double, and more mutations plus activity values, with the single point mutants covering a large fraction of the deep mutational scan space. I want to accomplish three things:

Plot a heatmap of activity of mutants by position,
Plot an UpSet plot of the top 10 positions by average activity v.s. top 10 positions by top mutant activity,
Include a summary recommendation at the end of the notebook.

This serves as a microcosm of what we would do with a data analysis session.

Goal #2 is particularly instructive. In my first attempts at feeling out how to do this benchmark, I found out that UpSet is incompatible with Pandas 3.0, which invariably may get installed in the environment. I wanted to see how various AI models performed at this task.

Additionally, I also have additional requirements that I encoded into the AGENTS.md file for this repo:

Imports must be done in a separate cell from code execution.
Markdown cells must always be written before a code cell is written
All cells must be run after being created, so that we can catch execution errors.

Benchmarking

With these in place, I started the benchmarking exercise.

The models we tested are:

GLM-5.1 (via OpenRouter)
Claude Opus 4.6 (via OpenRouter)
Claude Sonnet 4.6 (via OpenRouter)
MiniMax M2.7 (via OpenRouter)
Kimi K2.5 (via OpenRouter)
Gemma 4 31B (via OpenRouter)
Qwen 3 Coder Next (via OpenRouter)

In order to leave a working artifact behind, I created 7 notebooks, one for each model. As you will see below, I eventually evaluated each model on whether they passed each stage gate and what their earliest error mode diagnosis looked like.

In order to do the benchmarking fairly, I created one superprompt that outlined what the coding agent was supposed to do.

Use the marimo-pair skill here. Discover running sessions. Edit the notebook "NOTEBOOK_NAME_GOES_HERE". Read data/ired-novartis/cs1c02786_si_002.csv, identify the single point mutations, and plot me a heatmap of x-axis position, y-axis mutant letter, and heatmap value taken from the 'mean' column. When done, rank order the positions by average value of the 'mean' column, then rank order the positions by top value of the 'mean' column, and plot me an UpSet plot of the top 20 for each to visualize the set overlaps. Finally, write in for me a recommendation for what positions we should be mutating.

The agent is then tasked with executing.

To script this, I took advantage of opencode's ability to be scripted. The script is run_benchmark.sh in the repo. I used GLM5.1 to help me draft it, including discovering the exact models that opencode had configured to be available, and running the opencode sessions in parallel (totally doable!). Essentially it boils down to:

opencode run "your prompt here" --model provider/model-name

Additionally, I set up opencode.json to allow for access to the /tmp directory, because that allows the coding agent to do what it needs with code writing to get around heredoc limitations.

All in all, this computational experiment took me about 1 hour to set up.

I then ran the script run_benchmark.sh from within OpenCode (GLM 5.1 orchestrating), with a timeout of 10 minutes. Thanks to logging in JSON log files, I was able to programmatically convert them to Markdown using a custom Python script written by GLM 5.1. And with that, I can go in and start looking at the data.

To start, let's look at the cost of the experiment:

Model	Cost	Input Tokens	Output Tokens
Claude Opus 4.6	$1.62	76,770	16,575
Claude Sonnet 4.6	$2.00	213,803	27,689
GLM-5.1	$0.43	96,639	7,581
Kimi K2.5	$0.12	35,049	8,250
Qwen 3 Coder	$0.07	208,308	8,386
MiniMax M2.7	$0.04	14,419	4,074
Gemma 4 31B	$0.03	170,280	3,986
Total	$4.31	815,268	76,541

As it turns out, Opus is undisputedly the most expensive per token, but Sonnet 4.6 did more work this time round so its costs were higher.

I also decided to check whether the notebooks that were generated were valid notebooks or not. This is what we have:

Model	marimo check	Markdown cells
Claude Opus 4.6	PASS	Yes (100%)
Claude Sonnet 4.6	PASS	Mostly (86%)
GLM-5.1	PASS	Yes (100%)
Kimi K2.5	FAIL	Mostly (88%)
Qwen 3 Coder	PASS (warnings)	No (0%)
MiniMax M2.7	FAIL	No (0%)
Gemma 4 31B	PASS (warnings)	No (0%)

A note on the columns: "marimo check" is the result of running uvx marimo check <notebook_name>.py, which catches issues like redefined variables and invalid cells. Notably, Kimi K2.5 and MiniMax M2.7 failed this check due to re-defined variables. "Markdown cells" is the percentage of code cells that have a preceding markdown cell, which was something I explicitly required in the instructions.

And to elaborate on the markdown cells point:

Model	Code Cells	MD Cells	Code w/o preceding MD	Coverage
Claude Opus 4.6	6	7	0	100%
Claude Sonnet 4.6	7	8	1	86%
GLM-5.1	8	9	0	100%
Kimi K2.5	8	8	1	88%
Qwen 3 Coder	1	0	1	0%
MiniMax M2.7	6	0	6	0%
Gemma 4 31B	1	0	1	0%

We see that MiniMax M2.7 completely failed to include markdown cells, even though it is, supposedly, a model that is as capable as Opus 4.6.

Digging deeper into each of the models, and whether they passed each stage gate, I looked at the corresponding Marimo notebooks and evaluated them for whether they created the relevant artifacts successfully:

Model	G1: Heatmap	G2: UpSet Plot	G3: Recommendations
Claude Opus 4.6	Yes	Yes	Yes
Claude Sonnet 4.6	Yes	Yes	Yes
GLM-5.1	Yes	Yes	Yes
Kimi K2.5	Yes	No	No*
Qwen 3 Coder	No	No	No
MiniMax M2.7	Yes	No	No
Gemma 4 31B	No	No	No

To pass a stage gate, the plot (G1, G2) or markdown (G3) cell must be rendered in the notebook. Writing the code is not enough; it has to actually execute and show up.

Kimi K2.5 technically did write the recommendation, but I am calling it unsuccessful because it did not render out. This stricter criteria explicitly demands that the model wiggle its way out of errors it encounters.

One pattern I noticed across models is that many of them bundled imports into the same cell as code that used them. In Marimo's execution model, this is a problem: if two cells both import pandas, the notebook fails with a redefined variable error. Upon noticing this, I decided to explicitly quantify:

Model	Code Cells w/ Imports	Total Code Cells
Claude Opus 4.6	1	6
Claude Sonnet 4.6	1	7
GLM-5.1	1	8
Kimi K2.5	2	8
Qwen 3 Coder	0	1
MiniMax M2.7	3	6
Gemma 4 31B	0	1

Every model can benefit from being steered to reduce the number of code cells with imports, which would dramatically reduce the incidence of Marimo errors from redefined symbols.

How the UpSet plots turned out

As mentioned earlier, in my initial explorations I discovered that the upsetplot library is incompatible with Pandas 3.0, which invariably gets installed in the sandboxed environment. This made the UpSet plot task an especially interesting test of how each model handles a real-world dependency conflict. Here is how they fared.

Opus, in particular, produced a beautiful UpSet plot out of raw matplotlib:

While Sonnet went ahead and patched UpSet appropriately to make it work within the notebook:

I was duly impressed by Sonnet taking the initiative to patch UpSet live in the notebook.

On the other hand, GLM 5.1's UpSet plot is really weird:

Other observations

Other pointers of note: Gemma 4 and Qwen3 Coder Next produced nothing in the notebook. Both completely failed at this task. I am not sure what is doable here to salvage these models.

GLM 5.1 gave very weirdly formatted markdown cells, in which \n\n was not rendered but preserved verbatim.

This is probably fixable by adding in additional instructions on how to write and format Markdown cells using Marimo's code mode APIs.

Recommendations

First off: Gemma 4 31B and Qwen 3 Coder completely failed at this task. I think it is safe to say we can ignore these two going forward.

That leaves Claude Opus 4.6, Sonnet 4.6, GLM-5.1, Kimi K2.5, and MiniMax M2.7. Based on the data above, here are four things I want to try. The key discipline: deploy one change at a time, re-run the benchmark, and measure. If you change four things at once and performance improves, you will never know which change mattered. Stop when the KPIs hit acceptable levels.

1. Add import isolation examples to the skill. Every model had at least one cell that mixed imports with executable code. The fix is simple: add an explicit two-cell example to the marimo-pair skill (cell 1: imports only; cell 2: code that uses them). MiniMax had 3 cells mixing the two, which directly caused its marimo check failure. Give weaker models a concrete template to follow, re-run, and check whether the "code cells with imports" count drops to zero.

2. Fix GLM-5.1's newline rendering. GLM wrote mo.md(r"""..text with \n\n..""") instead of using actual newlines. One line in the skill instructions ("use actual line breaks in markdown strings, not \n escape sequences") should resolve this entirely. Re-run and check whether GLM's markdown cells render correctly.

3. Help Kimi K2.5 self-correct redefined variables. Kimi is 1/10th the cost of Opus and scored 88% on markdown coverage, making it the highest-leverage model to fix. Its failure was at error recovery, not code generation. The intervention: add uvx marimo check as a mandatory post-edit step in the skill. If Kimi can self-correct its redefined variables, it becomes a viable budget alternative to Opus and Sonnet. This should get even easier with marimo PR #9056, which exposes cell execution errors directly through the code_mode API, giving agents built-in self-correction visibility without needing a separate marimo check step.

4. Bake a post-edit validation loop into the marimo-pair skill. More broadly, the single most impactful change would be adding a "run it, check it, fix it" loop to the skill file itself (SKILL.md), not AGENTS.md: after writing each cell, run it; after writing the full notebook, run marimo check; fix any errors. This belongs in the skill because it is universal to any marimo-pair session, whereas AGENTS.md is project-specific. This would help Kimi, MiniMax, and potentially GLM all move up a tier, because their failures were in error recovery, not in code generation.

Discussion

One caveat to this analysis is that it is one-shotted with a superprompt. This is decidedly not how people do their data analysis work, but it is also the best guardrail against my biases in interacting ad-hoc with AI interfering with a fair comparison. (For example, I can confidently say that Opus and Sonnet were smooth as butter when I did an ad-hoc test to feel out how to work with Marimo Pair.)

If Kimi K2.5 were able to resolve redefined variable issues autonomously or be steered away from doing that to begin with, I am confident it would be able to be a great open weight alternative to Opus 4.6 and Sonnet 4.6. This is especially in light of it being extremely cost-effective at performing the analysis at ~1/10th the cost of Opus 4.6. It handled the creation of markdown cells well, failing to accomplish the task only on technicalities, and though its prose was qualitatively shallower than Opus 4.6, I still think it can serve as a first pass to delivering an easily understandable artifact for others.

I did one round of measurement here. If we want to systematically improve this and turn it into long-running evals, the next step would be to identify a second task along which to generate transcript and notebook data for us to mine, and systematically measure agent KPIs for that new task as well. Over time, this builds a corpus of eval data that makes model comparison rigorous rather than anecdotal.

Reflections

This was a pretty fun exercise in measuring and evaluating the performance of various models on this task. Like Biology experiments, LLM evals are never going to be complete: the number of axes of variations we can try is combinatorially explosive.

More broadly, I think often about how experiments get designed. Not in the statistical sense, but in an informational sense. Are we playing out experiments and their possible conclusions so that they are designed to be actionable whichever way the result pans out? If not, we have work to do.

Additionally, experiments involve measurement, and measurement are an integral part of being a data scientist. Hamel Husain, whose course with Shreya Shankar on LLM evals was one that influenced my thinking around the matter, notes that there will be a forceful revenge of the data scientist in an AI age. This is because the skill of experiment design and measurement were always the "science" part of "data science".

Another thought also comes to mind: I have seen data scientists do experimentation without systematic measurement. I'm going to go out on a limb and say this: it's vibe experimentation, and I am using this term pejoratively. It feels good. But it is ultimately unproductive. If you do vibe experimentation, you will get stuck tweaking the digital equivalent of an entangled biological system, with no bearings to tell you whether your tweaks are doing any good or not! You must measure how good the LLM or agent is, and you must define key performance indicators (KPIs) for the LLM. In my case here, I defined multiple KPIs: cost, stage gated progress, adherence to code import instructions (all failed), adherence to markdown documentation instructions.

And to echo what I learned from the LLM Evals course, those KPIs must be application-specific. If you choose to be intellectually lazy and go with generic pre-defined metrics, you will never develop the logically actionable metric that gives you hypotheses to test further. In my case, the markdown cell adherence and code import adherence metrics pointed immediately to editing the instruction files (e.g. skills or AGENTS.md).

Now to be clear, there's no problem with initial vibe-based experimentation to feel out axes of variation and how to measure performance. I did that here, in a separate repo first, before I designed this measurement experiment. The important part is this: as soon as you have a grasp of how to measure the performance, you must systematically measure that KPI. Otherwise, you will be left groping in the dark.

If you're curious to see the full results, including logs, chat transcripts, and the generated notebooks, check out the marimo-pair-benchmark repository.

And Trevor, if you ever chance upon this blog post, I hope the data and methodology are helpful for you!

Calibration Is Synchronizing Feedback Loops With Neural Throughput

2026-04-04T00:00:00Z

Since the beginning of the year, as I've been really maxing out on agentic coding and trying to explore the patterns and figure out what's working and what's not, one particular thing has been sticking out: I'm paralleling so much of my work. I'm frequently doing five or six different open pull requests, and it's become frankly really exhausting.

I've been trying to figure out why this feels so different from pre-AI days, when I'd work on one thing at a time and feel productive but not overwhelmed. What changed?

Tools haven't gotten worse—they've gotten dramatically faster. And when everything moves faster, the gaps between tasks become more expensive.

Last month, a Reddit thread sparked a discussion about what some are calling "AI psychosis" or "cyber psychosis": Andrej Karpathy reportedly went from 80% writing his own code to 0%, spending 16 hours a day directing AI agents. Garry Tan described similar feelings of burning through 4 hours of sleep, unable to stop building.

The debate that followed went beyond executives; it revealed a community-wide phenomenon: people running multiple Claude Code sessions in parallel, hitting rate limits daily, feeling like idle tokens were wasted tokens.

The consensus across replies was clear: AI psychosis is real, but it's less about excitement and more about a draining, addictive pressure to constantly build. The fear is missing out on the next big thing; it's the ground shifting beneath our feet, and stopping means getting left behind.

Here's what I've found hard to articulate: AI tools have expanded possibility faster than ever; their real danger lies in how they collapse our attention span. Before AI, we operated on a fairly flat productivity curve: more effort meant more output, slowly but sustainably. Now we're running on a different kind of curve altogether—one that's getting steeper in both directions.

The Accelerating Landscape of Possibility

In his book Where Good Ideas Come From, Steven Johnson described what he called the adjacent possible: the set of next-step ideas that are just beyond our current reality but still reachable. At any moment, only a limited set of next moves are accessible.

Here's what makes this concept critical for understanding our current moment: as you explore the adjacent possible (through moves that seem natural in the moment) the boundary itself expands. Each discovery opens doors that weren't accessible before, creating an accelerating landscape where what's possible keeps growing faster and faster.

I've seen this pattern play out: It explains why breakthroughs often happen when they do: the preconditions have finally assembled, and only then can you see the next move. But in an accelerating landscape, those preconditions assemble more quickly, and with them, the next set of possibilities.

Why AI Feels Different Now

Before AI, the adjacent possible was bounded by what a single person could manually assemble: write code, test it, debug it, repeat. The feedback loop, prompt, think, interpret, iterate, took time.

AI tools changed that calculus. They transformed the feedback loop, making it exponentially faster. Each new capability doesn't just add possibility; it reconfigures what's adjacent.

Once you see Claude Code as an idea multiplier, the pattern is clear. The Garry Tan/Karpathy effect kicks in: possibility grows faster than effort.

Here's Why It Feels Different

This is where it gets subtle: AI has shifted the inverted U curve of productivity and changed its shape.

Here's what that looks like, the gray curve shows pre-AI productivity, and the red curve shows post-AI:

Pre-AI: flatter curve. Effort mapped to output roughly proportionally. You could invest effort and see gains compound slowly, sustainably, with exhaustion coming only after sustained periods.

Post-AI: taller, narrower curve. Less effort gets you more output initially; the left slope is steeper, giving an astonishing return on initial investment. The right-side drop-off is sharper—exhaustion hits earlier, harder.

The peak represents the optimal effort level where output maximizes. Beyond that point, additional effort produces diminishing returns and exhaustion sets in faster than you can recover.

I wrote about this in a previous post on closing air gaps: the problem is that things are faster; it's that the gaps between tasks: those moments where attention bleeds out, are more expensive when everything moves faster.

Our Brain Is the Bottleneck Now

AI gives us 10X faster feedback loops: code spits out, prompts happen in seconds. Neural processing remains capped at biological speeds.

When loop cadence exceeds brain throughput, the cognitive queue overflows. Working memory saturates. Attention bleeds out. Exhaustion sets in, from the constant context switching, the constant need to scan multiple sessions for "what was done."

The optimal zone is when loop speed equals brain processing speed, not running tools as fast as possible.

Two Calibration Strategies

Here's what I believe we need to be able to do in order to calibrate properly.

First, tighten feedback loops. The goal is closing the loop properly on one task before opening another—I used to think juggling multiple Claude Code sessions was productivity; turns out it's just context switching masquerading as output. The trick is simple: run one session, close the loop completely, review what you got, then decide if your next move should be a new loop or something else entirely. Fast loops create natural pacing, which means you don't need to check tabs constantly because each cycle finishes with closure.

Second, build queue and notification systems—especially when you genuinely need multitasking. Most of us reach for multiple agents because we're solving the wrong problem: juggling open sessions creates overhead your tools cannot absorb. The Kanban approach works well: externalize context switching onto a board where agents update their status automatically. The system notifies you only when human judgment is required, so your job shifts from scanning to assessing—much lower overhead for neural throughput.

Calibration Is Not Optimization

This is the crucial distinction.

Pre-AI, you could coast for years on the left slope. Effort and reward grew linearly, so you just needed to work consistently.

Post-AI, the left slope is steeper, so you ascend faster; but also fall faster. AI-assisted tools don't eliminate the inverted U curve; they sharpen its peak and drop-off.

$\frac{d(\text{output})}{d(\text{effort})}$ is higher initially (good)
$\frac{d^2(\text{output})}{d(\text{effort}^2)}$ is more negative; the curve drops off faster (bad)

Calibration recognizes that more effort does not equal more output. Instead, there exists an optimal effort level where output peaks. Beyond that point, additional effort produces diminishing returns, and rest becomes the superior strategy.

What Calibration Actually Looks Like

The bottleneck is neural throughput—not tokens or API calls.

Ask yourself: Is my feedback loop faster than I can process?

If yes: slow down, close loops properly, resist the urge to open more tabs
If no: optimize the tool, not your attention (this is where most of us are wrong)

Practical heuristic: When you start scanning multiple AI threads for "what was done," you've exceeded your bandwidth. That's the signal that your loop cadence outruns your neural capacity.

Ask, "What is the maximum rate at which my brain can consume and act on output?" This question defines calibration—when work aligns with your neural capacity.

Calibration: Something You Do Daily, Not Something You Learn Once

AI has revealed our limits: the inverted U curve has become more visible, accelerated. Our brain is the rate limiter, and rightly so! We now need to learn to brake before you hit the wall.

The tools are powerful, but they don't change human neurology. No amount of prompt engineering can compress the time it takes for our brains to reason about things. If we try to go beyond our natural limits, dangerous things happen.

Calibration is the new baseline practice: a discipline you maintain daily, adjusting your loop cadence to match neural throughput, closing gaps before they become exhaustions.

The goal is sustainable access to the adjacent possible, rather than simply 10X-ing our output. And the goal is to do it for the long run, not just this week.

What This Looks Like in Practice

The goal is to work at the optimal effort level where output peaks. Beyond that point, additional effort produces diminishing returns and exhaustion sets in.

The default workflow:

One task at a time: Start with one focused task. Close the loop completely before moving to the next.
Close the loop fully: Review output, make notes, then decide on the next step only when you're ready.
Eliminate scanning: If you catch yourself flipping between tabs or sessions to check status, that's your signal that you've exceeded your neural throughput.

When multiple agents must run simultaneously:

Use a Kanban board: A visual task queue where agents update their status automatically.
Agents update the board, not you: The system should notify only when human intervention is needed.
You check the board, not the sessions: Remove the need to scan through terminal windows or tool interfaces.

You synchronize to stay on the left slope of your productivity curve, where effort yields return without triggering burnout. When tools run faster than your brain can process status updates, exhaustion sets in.

The rhythm shifts from "hustle harder" to synchronize—matching your brain's processing speed to the tools' output rate. When the signal outruns neural processing, interference replaces insight. You tune the system to keep your brain in phase on the left slope of your productivity curve, where effort produces sustainable output.

Undoing AI vibe-coded slop with AI

2026-03-29T00:00:00Z

I want to tell you about canvas-chat, a project I built with heavy AI assistance. It's a visual, non-linear chat interface where conversations are nodes on an infinite canvas — think branching, merging, and exploring topics as a directed acyclic graph.

The first commit landed on December 28, 2025. By December 30, it had sessions, matrix evaluation tables, web search, node tagging, and BM25 keyword search. The AI moved fast. Bugs got fixed in the next commit. Features piled in like tetris blocks stacking up.

And yes, it was a mess.

Here's the thing though: the mess was recoverable. Not because the AI got better (it didn't; not really), but because I had battle-tested convictions on how the thing ought to be architected. And those convictions came from years of shipping software, watching architectures crumble, and learning what holds up.

This is the story of how we went from a jumbled 8,500-line app.js to a clean plugin architecture — and why you need battle-tested convictions to make that happen.

The Initial State

The first commit wasn't actually bad. The project had clean separation from day one:

canvas.js — SVG pan/zoom/rendering
graph.js — DAG data structure
chat.js — LLM API + SSE streaming
storage.js — IndexedDB persistence
app.py — FastAPI backend

But app.js was already ~8,500 lines of everything else. Every slash command handler, every modal, every feature logic — all tangled together. Want to add a new feature? You'd grep around in that monolith, hope you found the right spot, and pray you didn't break anything.

The AI could add features to this mess. It could add a /matrix command in a few prompts. It could add /search with Exa integration. But it couldn't see the structure — the latent architecture that would make the whole thing maintainable.

The First Wave

The refactoring started with the purest code — functions with no dependencies:

Date	What Got Extracted	Why
Jan 4	`layout.js`	Overlap detection is pure math
Jan 5	`highlight-utils.js`	Text selection is isolated
Jan 7	Feature modules	`flashcards.js`, `committee.js`, `matrix.js`, `factcheck.js`, `research.js`, `code.js`
Jan 10	Core infrastructure	`undo-manager.js`, `modal-manager.js`, `slash-command-menu.js`

This reduced app.js from ~8,500 to ~5,500 lines. But these were still just file splits. The code worked better, but there was no system binding it together.

The AI did this part reasonably well — when I asked "extract this function to a separate module," it could do it. But it never suggested "we should extract this" on its own. It needed direction.

The Pivotal Moment

This was the architectural leap. I asked the AI to create a plugin system, and it delivered — but only because I knew what a plugin system should look like.

We ended up with a three-level plugin architecture:

Level 1: Custom Node Types — Node protocols define rendering via a BaseNode class. Each node type can override renderContent(), getActions(), getSummaryText(), and more. Registered in node-registry.js.

Level 2: Feature Plugins — Extend a FeaturePlugin base class. Get AppContext via dependency injection (graph, canvas, chat, storage, modalManager, streamingManager). Define slash commands via getSlashCommands(). Lifecycle hooks: onLoad(), onUnload().

Level 3: Extension Hooks — Subscribe to events. CancellableEvent can block actions. Event names like command:before, node:created, node:deleted.

The key files created:

feature-plugin.js — FeaturePlugin + AppContext
feature-registry.js — Slash command routing with priority (BUILTIN > OFFICIAL > COMMUNITY)
plugin-events.js — CanvasEvent, CancellableEvent
node-registry.js — Node type registration

This is where the architecture became a real system. And it only happened because I knew what I wanted.

Backend Pluginification (Late January 2026)

The same pattern reached the Python side:

pptx_endpoints.py — PowerPoint handling
ddg_endpoints.py — DuckDuckGo search
code_handler.py — Python code execution
matrix_handler.py — Matrix cell filling

Each follows a register_endpoints(app) pattern, loaded dynamically via importlib.

The Testing Safety Net

By late January, the plugin architecture was in place. Features were decoupled. The code was cleaner. And then GLM-4.5 started dropping curly braces.

No, really. The AI would "fix" one thing and introduce a missing bracket somewhere else. Merge conflicts became minefields; features that worked yesterday stopped working today; not because of malice, but because the AI didn't understand the dependencies between modules. It was making elementary mistakes that a junior developer wouldn't make.

On January 24, I added Cypress E2E tests. Out of spite, honestly. The first commit gave us canvas_interactions.cy.js, node_selection.cy.js, and note_node.cy.js - three tests that told us whether the canvas still worked.

These tests caught the regressions the AI kept introducing. More importantly, they let me verify changes faster. Instead of manually testing every feature after each AI session, I could run the test suite and know whether things still worked.

The plugin architecture made the code testable. The tests caught what the AI broke.

The Numbers

Phase	app.js size	Modules
Initial (Dec 2025)	~8,500 lines	5 files
After feature splits	~5,500 lines	11 files
After infrastructure	~5,400 lines	15 files
After plugin migration	~5,400 lines	25+ files
Today	~4,700 lines	35+ modules

The Bigger Lesson

Here's what I learned from this process:

The AI can execute architecture, but it can't design it. It can split files when asked. It can implement a plugin system from a spec. But it won't look at a 8,500-line app.js and say "this should be a plugin system."

That vision; that opinion; comes from somewhere else. It comes from:

Seeing architectures fail - Knowing the pain of tangled code, merged conflicts, feature creep
Seeing architectures succeed - Knowing what maintainable code feels like after years of shipping
Reading, studying, internalizing - Design patterns, architectural styles, tradeoffs
Making mistakes - Building the wrong abstraction once so you recognize it next time

I didn't arrive at "we need a three-level plugin architecture" out of nowhere. It came from discussing tradeoffs with the AI; asking "what if we did it this way?" and "what are the tradeoffs of that approach?"; and applying my best judgment to the options. The AI could explain the pros and cons of different approaches, but I had to pick which tradeoffs I was willing to accept.

The AI didn't teach me this. Experience taught me this.

What This Means for the Future

Here's where it gets interesting. Because we built this modular foundation, I can now swap out the rendering layer. The canvas is currently raw SVG — and I want to move to Svelte Flow. The plugin system I built makes this possible:

Features don't depend on app.js internals; they use AppContext
Canvas is isolated in canvas.js; swapping to Svelte Flow means replacing that layer
Node protocols define behavior; Svelte Flow nodes can use the same protocol pattern
Event system is framework-agnostic
Dependency injection provides graph, canvas, chat, storage; these can be re-provided to Svelte Flow components

The abstraction layer we built; FeaturePlugin + AppContext + EventSystem; separates what features do from how they're rendered. That's what makes Svelte Flow viable as a drop-in replacement.

Closing Thoughts

You can undo AI vibe-coded slop. It's possible. But it requires you to have battle-tested convictions on how the thing ought to be.

The AI is an incredible executor. It can refactor, extract, implement. But the vision? That stays human. And that vision comes from battle-tested experience; from having seen enough codebases to know what works and what collapses under its own weight.

So if you're working with AI coding assistants: don't expect them to architect for you. Tell them what to build. Give them the structure. Then let them do the implementation.

That's how you get from a jumbled mess to something you can actually maintain.

Creative mentorship strategies for career growth in challenging times

2026-03-25T00:00:00Z

The problem with lean times

When the economy tightens, formal development opportunities are usually the first things to go. Co-ops get paused, training budgets shrink, and headcount freezes make it harder to bring in fresh talent. But the need to develop mentorship, coaching, and leadership skills doesn't disappear just because the budget did.

So the question becomes: how do you get creative? How do you find opportunities to grow as a mentor and leader without requiring the company to spend additional money?

You already have something to offer

The answer is closer than you think. Even when budgets are frozen, you still have three things worth sharing: your judgment, your skills, and your network.

Your judgment is what experience actually gives you: not just knowing things, but knowing which approach to take, which tradeoffs matter, and when to push versus when to hold back. Your skills are the technical foundation that lets you coach and mentor beyond your own team, helping others onboard to the tools and practices you work with. And your network is the set of connections you can activate to create opportunities for others, whether that means knowing the right organizer, connecting a speaker to an audience, or simply inviting people into the same room.

The people who want to learn from you are already in your neighborhood. If you are doing an excellent job, you will find individuals who are eager to understand how you achieve your results. That is where your opportunity lies.

Five strategies that have worked for me

I want to share some concrete strategies that have worked at the two companies I have been with, Novartis and Moderna. Some of these I have actively advocated for. My intent is not to boast but to provide pragmatic suggestions based on my own experiences.

Coach others one-on-one

Coaching others is a great way to build your reputation within the organization. When you teach someone how to accomplish a task effectively, you become valuable to them. More importantly, you demonstrate your value to a broader audience. Within an organization, you want a group of people who find your skills genuinely worthwhile.

Present at internal guilds and "birds of a feather" events

At Moderna's Digital organization, we have "Guilds", the Data Science Guild being one, with three meetings per month for the guild. When I was at Novartis' Research org, we had the Computational Research Community. Both served as outlets for talks and annual gatherings. The key is to be in a position where you can give a talk about something valuable to others, and they would willingly spend an hour listening to you. If that happens, you have created another mentorship opportunity for yourself.

Organize communities of practice

I have seen this happen when someone builds a tool that others use and then creates a community around that tool. It can be as simple as a Microsoft Teams group chat. You don't need anything more sophisticated than that. Just gather people who use the tool and facilitate discussions. Being a leader in that group chat is a real way to hone your leadership skills across the organization. My colleague Albert Lam built a significant portion of the Python packages that are used by LLM builders, and put together communities of practice around that precisely in the form of MS Teams group chats.

Another example is the community of practice around documentation - primarily expressed as the internal docathons that we run. My teammate Jackie Valeri, as well as two other colleagues Simreen Kaur and Saakshi Shamanth Donthi help coordinate and organize the logistics while also being point contacts for other folks participating in the docathon.

Host informal coffee hours

My teammate, Michelle Faits, took the initiative to host coffee hours within the company. These serve as informal outlets for people to present their work, and they are great because they are relaxed and authentic. As her manager, I try to find speakers to contribute and support her efforts. Kudos to her for initiating this.

Host or support external meetups

We also host the PyData Boston/Cambridge monthly meetup at Moderna. Not every month is held at our location, but since I know the organizer Ben Batorsky, back in 2025, I offered my time to book a room; we simply provide the space. More recently, Jackie has taken the lead in this. By taking charge and inviting others to network, we create opportunities for people to grow in their careers without any budget requirement.

Advice for managers

If you are a manager, recognize that there will be projects and efforts that need leadership beyond individual contributions. Helping others improve their skills and providing them with opportunities to lead is part of the job, especially when formal avenues are limited.

Make sure you are aware of these kinds of initiatives among your reports, and ensure they don't conflict with core responsibilities. When someone can demonstrate that they manage these additional activities while maintaining their primary work, that forms a strong case for their expanded capabilities.

We should expand our imagination beyond just climbing the career ladder, attaining higher status, or managing other people, which sometimes unfortunately spills over into controlling others at work. Growth comes in many forms, and we can find meaningful ways to develop without waiting for formal promotions or titles.

The core principles

Mentoring is about sharing your judgment, providing opportunities for others to share theirs, facilitating networking, and helping others grow. If we limit our understanding of growth and development to a narrow definition (only formal programs, only budgeted activities, only additional formal assignments), we constrain our imagination as managers.

Don't be confined to a singular vision of what it means to be a good leader or manager. Embrace the diverse talents and varying stages of abilities within your team. Encourage your team members to step outside their comfort zones and provide them with opportunities to grow.

While good, you do not necessarily need an internal university to foster a learning culture; your environment is where you learn and grow. We have more autonomy and agency than we might realize!

Closing air gaps

2026-03-15T00:00:00Z

I owe this term to my colleague Wenhao Liu. He was the first one I saw at work who clearly articulated about air gaps and how they relate to building agents for work.

So what exactly is an air gap? It is any point in a business or scientific process where a human has to intervene and perform manual work before a digital system can continue. The system cannot go end to end on its own; the human is the bridge.

Air gaps are everywhere. Here are a few examples: A laboratory machine exports a file to a local hard disk. A human copies that file and pastes it into an S3 bucket. That handoff is an air gap. The system stops at the hard disk and waits for a person.

In wet lab science, air gaps take physical form. A scientist designs an experiment, walks into the lab, and executes it by hand. In this state, the company/team/org has an air gap in its scientific process. No robotic system can take over from design to execution.

Both of these examples share a common pattern. The definition I have settled on is this: an air gap is any place where rote manual work is performed by a human that could have been done by a computer. (Robots are computers with sensors and actuators for the physical world.)

Why air gaps matter

Think of your processes as pipes. Air gaps are bubbles trapped in those pipes. They slow the flow. They disrupt continuity. They introduce delays and errors.

The costs compound over time. A five-minute manual handoff, repeated daily across a team of twenty, adds up to real hours. A week-long delay because someone was on vacation and could not move the file. A transcription error because a human typed a number wrong. A lost opportunity because the data sat in a local folder instead of flowing into the analysis pipeline.

Air gaps also create cognitive overhead. Every time a human has to remember to perform a manual step, that is mental bandwidth not spent on creative work. The air gap is a tax on attention.

Now, a clarification. The goal here is not to eliminate humans from everything. The goal is to eliminate humans from the rote and routine. Creative work, judgment, and decision making stay with us. Copying files, transferring plates, and typing data into forms do not.

How to find air gaps

You cannot close an air gap until you see it. And you cannot see it until you sit down and map out exactly how your process works. In other words, process mapping.

This mapping exercise is the unsexy work that precedes automation. Most teams skip it. They jump to solutions before understanding the problem. But you need the map.

Here is how to do it.

Pick one process. It could be a data pipeline, a lab workflow, or a business approval chain. Walk through it step by step. Ask these questions at each step:

Where is a human manually copying and pasting?
Where is a human manually entering data?
Where is a human dragging and dropping files?
Where is a human making a decision that follows a fixed rule?
Where is a human waiting for another human to take action?
Where is a human physically moving something from one place to another?

Each answer points to a potential air gap.

Write it down. Draw it out. Make the process visible. Once the map exists, the air gaps reveal themselves. The next step is prioritization. Which air gaps cause the most pain? Which ones are easiest to close? Start there.

Air gaps in the wild

Enough abstraction. Let me show you what air gaps look like in practice, from my own work and from the broader landscape.

File schlepping in the lab

A sequencing machine finishes a run. It writes the data to a local drive. A technician notices the run is complete, navigates to the folder, selects the files, copies them, navigates to the shared storage system, and pastes. Minutes pass. Sometimes hours, if the technician is busy.

This is an air gap. The machine knows when the run finishes. The machine has network access. The destination storage has an API. Automation can close this gap with a simple script that watches for new files and uploads them.

The fix is not technically difficult; it is, conceptually, a cron job with rsync. What makes it hard is that the air gap is invisible until someone maps the process and asks why a human is doing this work.

GitHub activity tracking

I used to manually check GitHub to track my daily work. I would open my profile page, scroll through recent commits, open the pull requests tab, check which ones I had opened or reviewed, and then type notes into a document. This took maybe ten minutes per day.

Then I remembered the GitHub CLI exists. I also remembered that coding agents can run CLI commands.

I built a skill that pulls four categories of activity automatically: my opened pull requests, pull requests I reviewed or commented on, my commits, and issues I created. The agent runs this skill as part of my daily sign-off routine. The air gap closed.

The time savings are modest. But the mental overhead vanished. I no longer need to remember to check GitHub. The information flows to me.

Autonomous laboratories

The autonomous lab, sometimes called a lights-out lab, is the ultimate expression of closing air gaps. The vision is a laboratory that runs itself: experiments are designed, executed, analyzed, and iterated without human intervention.

In practice, autonomous labs are full of micro air gaps. Each one must be identified and closed.

Plate transfers. A protocol requires moving a plate from one instrument to another. Does a human do this? If so, that is an air gap. Robotic arms and conveyor systems can close it.

Master reagent prep. Someone mixes buffers and reagents by hand at the start of each week. Could a liquid handling robot do this instead? Probably. That is an air gap.

File movement. Instruments write data locally. Humans move data to shared storage. This is the file schlepping problem again, repeated across every machine in the lab.

Standardized analyses. As a data scientist, this one is close to my heart. Most labs have a set of standard analyses they run on every dataset. Quality control plots, basic statistics, alignment checks. A human opens a notebook, loads the data, runs the cells, and exports results.

This is an air gap. Standardized analyses can be automated. They can also be made adaptable. An LLM-powered coding agent can take a standard analysis template and adjust parameters within a confined range of design choices. The human specifies the intent. The agent handles the implementation.

Closing the loop. The ultimate air gap in scientific research is the gap between analysis and experiment design. A human looks at results, draws conclusions, and designs the next experiment. What if the results could flow back into experiment design automatically? What if an agent could propose the next experiment based on what the data showed?

This is the direction autonomous labs are moving. But getting there requires closing every air gap along the chain.

The blockers: imagination and skill

I wrote about this in a previous post. The biggest blockers to closing air gaps are technical skill and imagination.

Imagination

If you cannot imagine a future state where your tedious work is performed by a coding agent, you will not see the possibility. The air gap remains invisible.

This is a failure of imagination, not a failure of technology. The tools exist. The APIs exist. The agents exist. What is missing is the mental leap from "this is how we have always done it" to "this is how we could do it."

Imagination grows from exposure. The more you see what is possible, the more you can imagine for your own work. Watch what other teams are doing. Read about automation in adjacent fields. Talk to people who have closed similar gaps.

Skill

Imagination alone is not enough. You also need the skill to build the automation.

That skill, at some level, means knowing how to program. It means understanding APIs, scripting, and how systems talk to each other. It means knowing that cron jobs and web hooks can be configured.

The good news is that the barrier to entry is lower than ever. Coding agents can help you write the code. The skill you need is not deep software engineering. It is enough programming literacy to describe what you want and recognize whether the output is correct.

Programmatic access

Sometimes the blocker is not you. Sometimes it is the system.

Some organizations block programmatic access to their tools. It may be that a SaaS application has no API, or the API is disabled for security reasons, or that a legacy database has no query interface, only a web portal. A legacy laboratory information management system may require a human to click through screens instead of secured APIs.

These are also air gaps. The system itself prevents automation.

If the blocker is cybersecurity concerns, push for scoped, tracked access. You do not need full API permissions to close a specific air gap. You need enough access to move the data that belongs in your pipeline. That access can be limited, logged, and auditable.

If the system truly has no programmatic interface, browser or desktop automation agents can close the gap. A headless browser can log in, navigate, click, and extract. It is slower and more fragile than an API, but it works.

Closing air gaps with agents

But here is the good news. The rise of coding agents changes the calculus for closing air gaps.

Before, you needed a software engineer to write the automation script. Now, you can describe what you want in plain language and let the agent write the code.

This does not mean you can ignore technical literacy. You still need to verify the output, debug when things break, and understand enough to specify the problem clearly. But the implementation barrier is lower.

Browser agents extend this further. If a system has no API, a browser agent can act as the interface. Log in, click the buttons, extract the data, and feed it into your pipeline.

The key insight is that agents are not a replacement for mapping your processes. They are a tool for closing the air gaps you have already identified. The mapping still matters. The imagination still matters. The skill to recognize whether the agent's output is correct still matters.

What changes is the speed of iteration. You can try closing an air gap in an afternoon instead of a sprint. You can experiment with different approaches quickly. The feedback loop tightens.

Here is the right question to ask when you find yourself being asked to do something repeatedly: "Can I remove this bottleneck for you? Is there a way I can make it so that you never have to ask me this again?" The goal is to build systems that use agents to not need human intervention as much as possible, not to create systems that depend on humans more and more. As one person put it, "the more I have asked myself that question, the more capable he has become." (source)

The principle

The guiding principle is simple. To butcher the Biblical phrase, "Give unto robots what belongs to robots, and have humans do what humans can do." Or to paraphrase another person, use robots for the dull, dirty, and dangerous work.

Keep the human in the loop for creative, judgment-heavy work. The rote and routine should flow through pipes without bubbles.

Start mapping. Find the air gaps. Close them one by one. The compounding effect over time is enormous once micro-efficiencies become part of your work.

What looks like a small efficiency gain today becomes a transformed process tomorrow. The lab that closed its file schlepping air gaps is one step closer to autonomous operation. The team that automated their daily reporting has mental bandwidth for harder problems.

Close every air gap you can see!

Agent skills are also human skills

2026-03-14T00:00:00Z

Agent skills are great, but I've been thinking about this... skills alone aren't enough.

I've been thinking about this while developing and using agent skills at home and at work. There's a distinction I've started to draw between two types of skills. Tool-specific skills document how to work with a particular tool or package. Those are fine, but really, pointing an agent at llms.txt often works just as well. The more interesting category is workflow-specific skills, things that encode how you actually work, that string together multiple tools to get a job done (Christensen).

Workflow-specific skills are what I want to talk about here.

A concrete example

My daily sign-off skill, which I use at work, is a case in point. I use it to wrap up my day. When I sign off, I need two things: my meeting notes (which I paste into Obsidian throughout the day) and my GitHub activity (commits, PRs, comments, reviews). The skill handles the GitHub part by querying the GitHub CLI and formatting everything into my daily bullets template.

But here's where it gets opinionated. My skill assumes:

You have the GitHub CLI installed
You do PRs as part of your work (not all technical managers do)
You write into a monthly file as your bullet journal, rather than having a single note per day.

That last point is opinionated. I don't have a single note per day. Instead, each month contains my collection of daily bullets. The motivation here is a line from the Zen of Python -- "flat is better than nested". On March 26, I have entries for that day inside the March file, rather than have a reference from the March file to March 26. This might not reflect your own preferences; you might prefer one note per day, or use a different structure entirely. But this is what my skill expects, and it's baked into how the skill works.

If you want to use my daily sign-off skill, you're not just adopting the skill. You're adopting my way of working. You're inheriting my file structure, my tool preferences, my mental model for organizing information. The skill comes with implicit assumptions about how you work, what tools you use, and what your environment looks like.

A second example, cutting deeper

The daily sign-off is mostly about tool and structure preferences. But some skills go further — they encode a philosophy.

My scientific EDA skill is a good example. On the surface it looks like a set of technical rules: use uv with PEP723 inline scripts, save plots as WebP (not PNG), organize each analysis session into a timestamped folder, keep an append-only journal.md. But look at what those rules actually encode:

One step at a time, ask "why" before executing — this isn't a technical constraint. It reflects a skepticism of agents that run ahead of the analyst. I believe good exploratory analysis is a dialogue, not a sprint.
Capture the research question before touching the data — this reflects a conviction that context shapes what you should even be looking for. Data without a question is just noise.
Append-only journal — this reflects a belief that good science is narrated, not just executed. The journal isn't a log file; it's a record of reasoning.
WebP over PNG — a small but deliberate aesthetic and practical stance on file hygiene.
uv + PEP723 — a specific bet on the Python toolchain that not everyone has made.

None of these are neutral defaults. Each one is a choice that reflects how I think scientific work should be done. If you use my EDA skill but don't share that underlying philosophy, you'll find yourself fighting it. The one-step-at-a-time rule will feel like friction. The journaling requirement will feel like overhead. The skill isn't broken — it's just mine.

This is a different kind of assumption from the daily sign-off. There, you're inheriting my tools and file layout. Here, you're inheriting my epistemology. That's harder to see, harder to document, and harder to transfer.

What this means

I call this procedural context. A workflow-specific agent skill is more than documentation for the coding agent. It also implicitly encodes a person's systems and structures for working. Without documenting the procedural context, the skill can only be half-useful for another person.

The two examples above hint at different layers of procedural context. There are at least three:

Tool dependencies — what software needs to be installed (GitHub CLI, uv, etc.)
Organizational preferences — how you structure files, folders, and notes
Epistemic preferences — how you believe the work should actually proceed

The third layer is the most invisible. It's also the most important, and the hardest to transfer. You can install a CLI tool in five minutes. Adopting someone else's philosophy of scientific analysis is a different ask entirely.

Someone on Twitter put it well (I wish I could remember who, so I won't take credit): with agent skills, we finally found a way to get coders to write documentation. We'll document how we work if it means we can delegate that work to some{one/thing} else!

At the end of the day, agent skills are just automation and documentation. We're automating away the minutiae, and I love that. But if your skill describes a workflow, you need to document the assumptions too. What are the dependencies? What tools need to be installed? What mental structures does the person need? What does the user need to know to verify the output is correct?

Without that context, you can't evaluate whether the coding agent used the skill correctly -- and verification matters! You need to know what to look for when an LLM does work on your behalf.

The takeaway

Agent skills implicitly involve human skills. If that's true, then agent skills are also for humans. They're not merely instructions for an agent. They're documentation of how someone accomplishes a job, with all the prerequisites and context needed to reproduce it.

So when you write a workflow skill, think about the other people who might use it. Ask the skill-creator skill to include the dependencies, explain the environment, and describe what success looks like. The skill alone isn't enough. We have to teach the next person how to use it too.

My weekend experiment making PyMC installable in a WASM environment

2026-03-08T00:00:00Z

This past weekend, I found myself revisiting a blog post from PyMC Labs titled "Running PyMC in the Browser with PyScript". Published in 2022, it demonstrated something magical: running full Bayesian inference with PyMC entirely in the browser—no server, no installation, no data leaving your device. Users could define models, run NUTS sampling, and visualize posteriors, all client-side.

I was excited to try it out. But when I attempted to run the examples, I discovered they no longer worked. The Python package ecosystem had evolved, dependencies had shifted, and the Pyodide environment had changed. What was once a breakthrough demo had quietly broken.

So I did what any curious engineer would do on a weekend: I dove down the rabbit hole to figure out how to make it work again.

The core challenge: getting PyTensor to build for WebAssembly

PyMC depends on PyTensor, its computational backend. PyTensor is where the heavy lifting happens: it compiles mathematical expressions into optimized code (usually C or JAX) and executes them efficiently. To run PyMC in a browser via Pyodide, I first needed to make PyTensor installable in a WebAssembly environment.

This wasn't just a matter of pip install. PyTensor contains C and Cython extensions that must be compiled for the target platform. For WebAssembly, that means using Emscripten and the Pyodide build tooling.

The code changes: what I modified in PyTensor

Working on my fork of PyTensor (ericmjl/pytensor), I made targeted modifications to enable WASM builds. Here's the complete diff:

Change 1: making Numba optional on WebAssembly

File: pyproject.toml

-    "numba>0.57,<1",
+    "numba>0.57,<1; platform_machine != 'wasm32' and sys_platform != 'emscripten'",

This single line change is the critical enabler. Numba, PyTensor's JIT compiler for numerical code, is not available in WebAssembly environments. There's no way to install it—it simply doesn't exist for this platform.

The fix uses PEP 508 environment markers to make Numba a conditional dependency:

platform_machine != 'wasm32' excludes WASM architectures
sys_platform != 'emscripten' adds an extra safety check for Emscripten-based builds

Without this change, attempting to install PyTensor in Pyodide would fail immediately with a dependency resolution error. Pyodide would try to find a Numba wheel for WASM, fail, and abort the entire installation.

The tradeoff, however, is that PyTensor loses its JIT compilation capabilities on WASM. Operations that would be compiled to optimized native code fall back to pure Python execution. This means slower performance, and critically, PyMC's NUTS sampler won't work.

Change 2: adding Pixi development environment configuration

File: pyproject.toml

I added a complete Pixi workspace configuration to pyproject.toml. This provides a reproducible development environment and includes the tooling needed to build WASM wheels:

# -----------------------------------------------------------------------------
# Pixi (pixi.prefix.dev): development environment from environment.yml
# Use: pixi install && pixi run pytest   or   pixi shell
# -----------------------------------------------------------------------------
[tool.pixi.workspace]
channels = ["conda-forge"]
platforms = ["linux-64", "osx-64", "osx-arm64", "win-64"]

[tool.pixi.pypi-dependencies]
pytensor = { path = ".", editable = true }
types-setuptools = "*"
build = "*"
pyodide-build = ">=0.29.2"

[tool.pixi.dependencies]
python = ">=3.11,<3.14"
compilers = "*"
numpy = ">=2.0.0"
scipy = ">=1,<2"
filelock = ">=3.15"
etuples = "*"
logical-unification = "*"
miniKanren = "*"
cons = "*"
pydeprecate = "*"
numba = ">=0.57"
coveralls = "*"
diff-cover = "*"
mypy = "*"
pytest = "*"
pytest-cov = "*"
pytest-xdist = "*"
pytest-benchmark = "*"
pytest-mock = "*"
pytest-sphinx = "*"
sphinx = ">=5.1.0,<6"
sphinx_rtd_theme = "*"
pygments = "*"
pydot = "*"
ipython = "*"
pymc-sphinx-theme = "*"
sphinx-design = "*"
myst-nb = "*"
matplotlib = "*"
watermark = "*"
ruff = "*"
pandas = "*"
pre-commit = "*"
packaging = "*"
cython = "*"
graphviz = "*"

[tool.pixi.target.linux-64.dependencies]
mkl = "*"
mkl-service = "*"
libblas = { version = "*", build = "*mkl" }

[tool.pixi.target.win-64.dependencies]
mkl = "*"
mkl-service = "*"
libblas = { version = "*", build = "*mkl" }

[tool.pixi.target.osx-64.dependencies]
libblas = { version = "*", build = "*accelerate" }

[tool.pixi.target.osx-arm64.dependencies]
libblas = { version = "*", build = "*accelerate" }

[tool.pixi.tasks]
test = "pytest"
lint = "ruff check ."
format = "ruff format ."
docs = "python -m sphinx -b html ./doc ./html"
wheel = "python -m build --wheel"
sdist = "python -m build --sdist"
wheel-wasm = "pyodide build"

Here are the key design decisions in this configuration:

Python version pinning:

python = ">=3.11,<3.14"

Pyodide only supports up to Python 3.13. Without this constraint, the environment might resolve to Python 3.14+, causing the WASM build to fail with: ValueError: Python version 3.14 is not yet supported.

PyPI dependencies for building:

[tool.pixi.pypi-dependencies]
pytensor = { path = ".", editable = true }
types-setuptools = "*"
build = "*"
pyodide-build = ">=0.29.2"

This installs PyTensor in editable mode for development, includes type stubs for mypy, and adds both build (standard wheel building) and pyodide-build (WASM wheel building).

Platform-specific BLAS:

[tool.pixi.target.linux-64.dependencies]
mkl = "*"
mkl-service = "*"
libblas = { version = "*", build = "*mkl" }

[tool.pixi.target.osx-arm64.dependencies]
libblas = { version = "*", build = "*accelerate" }

Different platforms use different BLAS implementations. Linux and Windows use Intel MKL, while macOS uses Apple's Accelerate framework. These ensure the correct linear algebra library is installed.

Build task:

wheel-wasm = "pyodide build"

This task runs pyodide build, which compiles PyTensor for WebAssembly using Emscripten.

Change 3: documenting the WASM build process

File: doc/dev_start_guide.rst

I added documentation explaining how to build WASM wheels:

Building a WebAssembly (Pyodide) wheel
-------------------------------------

To build a wheel targeting WebAssembly for use with `Pyodide <https://pyodide.org/>`_ (e.g. for the browser or JupyterLite), use the Pyodide build tooling. This produces a wheel in ``dist/`` with a name like ``*-cpXXX-cpXXX-pyodide_*_wasm32.whl``.

**One-time setup: Emscripten**

1. Install `pyodide-build` (included in the Pixi dev env, or ``pip install pyodide-build>=0.29.2``).
2. Get the Emscripten version required by your pyodide-build: ``pyodide config get emscripten_version``.
3. Install and activate that Emscripten version using the `Emscripten SDK (emsdk) <https://emscripten.org/docs/getting_started/downloads.html>`_:

   .. code-block:: bash

      git clone https://github.com/emscripten-core/emsdk.git
      cd emsdk
      ./emsdk install <version>   # use the version from step 2
      ./emsdk activate <version>
      source emsdk_env.sh

4. In any shell where you want to build the wasm wheel, ensure Emscripten is on ``PATH`` (e.g. run ``source /path/to/emsdk/emsdk_env.sh``).

**Build the wheel**

From the project root, with Emscripten activated and your dev environment active (e.g. ``pixi shell``):

.. code-block:: bash

   pyodide build

Or with Pixi: ``pixi run wheel-wasm``.

The wheel will appear in ``dist/``. PyPI does not yet accept emscripten/wasm32 wheels; host the file elsewhere (e.g. GitHub Releases) and install in Pyodide with ``micropip.install(url)``. See `Pyodide: building packages <https://pyodide.org/en/stable/development/building-packages-from-source.html>`_ for details.

This documentation walks through the Emscripten setup, the build command, and importantly, notes that PyPI doesn't accept WASM wheels yet—you need to distribute them via GitHub Releases or similar and install with micropip.install(url).

What I actually PR'd to PyTensor

The changes above represent my weekend exploration, but they weren't what I ultimately contributed back to PyTensor. The Pixi configuration, in particular, was too large of a departure from PyTensor's existing toolchain. PyTensor uses mamba (via environment.yml) for its development environment, and switching to Pixi would have been a significant change to impose on a project I don't maintain.

Instead, I re-did the infrastructure changes using pyodide-build while respecting PyTensor's existing mamba-based workflow. The core change (making Numba optional on WebAssembly) remained, but the development environment configuration was adapted to work with what PyTensor already had in place.

This is a common lesson in open-source contribution: meeting maintainers where they are matters more than introducing your preferred tooling. The weekend experiment taught me what was needed; the PR reflected what was appropriate. You can see the actual PR here: pytensor #1960.

Unfortunately (for now), NUTS is gone

Unfortunately, NUTS (No-U-Turn Sampler) doesn't work in WASM. 😭

NUTS is the crown jewel of PyMC. It's the adaptive Hamiltonian Monte Carlo sampler that makes Bayesian inference efficient and robust. The 2022 PyMC Labs demo used NUTS to sample from posteriors in real-time in the browser.

But here's the thing: this isn't just about Numba being unavailable. The real issue is that none of the modern MCMC sampling backends have WASM support:

JAX (used by NumPyro and BlackJAX) has an open GitHub issue #1472 from 2019 titled "Jax for Web? (JS api or web assembly guide)" that's still open with no official WASM support
nutpie (the Rust-based NUTS implementation) doesn't have a WASM build readily available
The computational demands of Hamiltonian dynamics—computing gradients, simulating trajectories, adapting step sizes—require optimized backends that don't exist in WASM environments

This weekend's exploration shows the path to install PyMC in WASM, but you can't use its best sampler. It's like getting a Ferrari delivered to your house, but the dealer forgot to include the keys. You can sit in it, admire the leather seats, and maybe even turn on the radio. But you're not going anywhere fast.

This represents a fundamental infrastructure gap, not just a missing dependency. Getting NUTS in the browser will require either WASM ports of JAX or nutpie, or entirely new sampling backends designed for browser environments.

What does work?

Despite the NUTS heartbreak, this wasn't a failed experiment:

PyTensor now installs in WASM environments. This is non-trivial. PyTensor has C and Cython extensions that need to compile for WebAssembly. Getting that build pipeline working required understanding Pyodide's build system, setting up Emscripten correctly, and making Numba optional.
PyMC can technically be imported. Once PyTensor was installable, PyMC followed. You can define models, create random variables, and work with the API. The foundation is there.
Alternative samplers might still work. While NUTS is off the table, other samplers—like Metropolis-Hastings or Slice sampling—might be viable for small models. They're slower and less robust than NUTS, but they don't require JIT compilation. I didn't test, but I think this will hold true!
The roadmap is clearer. If someone wants to bring full PyMC to the browser, the path forward is documented. It requires either (a) building WASM support into JAX (a massive undertaking that's been an open request since 2019), (b) creating WASM builds for nutpie, or (c) building entirely new sampling backends designed for browser environments (also non-trivial, but potentially more feasible).

A weekend well spent

Did I achieve my original goal of running PyMC in the browser with NUTS sampling? No. The technical limitations of WASM environments made that impossible with the current architecture.

But that's the nature of weekend experiments. You explore, you hit walls, you learn. I now understand PyTensor's dependency structure at a deeper level. I've learned how Pyodide builds work and the constraints they impose. I've identified the broader infrastructure gap (MCMC sampling backends lacking WASM support) that needs solving for true browser-based Bayesian inference.

The dream of running PyMC entirely in the browser isn't dead—it's just waiting for the right infrastructure. Until JAX or nutpie (or something else) supports WASM, we'll keep pushing that car downhill.

Mastering Personal Knowledge Management with Obsidian and AI

2026-03-06T00:00:00Z

Folks have asked me how I do personal knowledge management (PKM) at work. The question becomes more pressing when they learn how many projects and people I need to interact with on a weekly basis. At the time of writing, I manage twelve people across two teams, each handling 2-4 projects of their own. That's a lot of context to keep straight.

I decided to document what I'm doing for PKM. Hopefully it serves as inspiration for you.

I've written before about why I chose Obsidian; this post shows how that decision evolved with AI integration over five years.

The plain text decision

In 2022, I decided to make personal knowledge management a priority at work. I faced a choice: Confluence, OneNote, or a new kid on the block, Obsidian. I chose plain text and graphs. I chose Obsidian. I chose not to lock my data inside a vendor system. I chose freedom and sovereignty for my information.

That decision was prescient in ways I couldn't have predicted. Most of us normies back then wouldn't have guessed that plain text would be exactly the right format for 2025 and 2026 era knowledge management. The visionaries saw it coming; I just got lucky because I loved the graph view in Obsidian and thought of it as a really cool tool. But holy smokes, has that choice paid off.

Text files are as primitive as it gets: no proprietary formats, no vendor lock-in, just files that can be read on any system. When AI coding agents arrived, my vault was already in a format they could process natively. No migration needed. No conversion layer. No API integration. The simplicity I chose became an unlock I never planned for.

The core system

My Obsidian vault is built around distinct note types. Monthly collections of daily bullet journals capture my day-to-day activities, one note per month with a running log of meetings and work. Meeting notes follow a structured template. People notes are dossiers for everyone I work with (to put it in CIA terms, I keep a file on everyone I interact with regularly). Project notes act as control towers, linking out to meetings, people, and status updates. A miscellaneous collection handles everything else. The structure was inspired in part by Thiago Forte's numbered folder system, though I've simplified it over time.

The most important thing isn't my specific implementation. It's that I have a system at all, and it's documented in an AGENTS.md file so my coding agents understand it too.

Ingesting information

The lifecycle of my workflow starts with ingestion. Meeting notes arrive as transcripts or AI-generated summaries. In the past, structuring these was tedious work. Now I paste them into OpenCode and my meeting notes skill handles the rest.

The skill knows the template I want. It handles various input formats: AI-generated summaries, transcripts with good speaker assignments, and transcripts with poor speaker assignments. I flag the quality when I know it's bad. The skill extracts key information and formats everything consistently. For one-on-ones, it ensures notes are attached to both the meeting log and the person's individual page, so I can track the full history of our conversations.

Beyond meetings, I ingest PowerPoints, Word docs, PDFs, and Excel spreadsheets into my vault as contextual information. The key insight is getting everything into plain text format. For Word documents, a Python script converts them to plain text using python-docx, which is then printed to the terminal or dumped to disk at /tmp, both of which are readable by a coding agent. Even lightly misformatted plain text contains enough information density for a coding agent to read and summarize.

For PowerPoints, I use dual parsing paths. One path extracts the XML structure directly using python-pptx. The second path converts each slide to an image using libreoffice and PIL, captions it with a vision-language model via APIs, and strings the captions together into a coherent narrative. Combined, I estimate that I can get a 90-95% accurate textual representation. PDFs follow a similar pattern: text extraction for normal PDFs, image captioning for scanned documents.

Excel spreadsheets are read directly by the coding agent using openpyxl, not pandas. The key difference matters: pandas assumes an established table structure, but real-world spreadsheets are messy. With openpyxl, the agent can read the granular cellular structure across each sheet, identifying merged cells, free text scattered in random locations, and arbitrary layouts. This structural mapping follows a progressive reveal principle: the agent first identifies the spreadsheet's architecture without necessarily reading every cell's contents, then zooms into relevant sections. This approach handles the chaos of actual spreadsheets far better than forcing everything into a tabular assumption. It's powerful when I need to understand financial data without being a finance person.

Managing and maintaining

With information in the vault, the next phase is keeping it current. With twelve people across two teams, there are a lot of details I don't pick up or retain in my working memory. That's why external memory matters. Without it, things would fall through the cracks.

When I hit a context block (when I look up a project or person and realize something's missing), I trigger a "sweep". My instructions to the coding agent are to update my people notes and/or project notes based on source material present in the vault. People and project notes are always derivative from sources, so any updates must include quotations from those source notes. I stay in the loop for verification. Hallucinations are rare, maybe once every four or five sweeps, and usually trace back to inaccurate transcripts rather than agent errors.

This is incredibly helpful for how I interact with people. My assumption is that I'm going to be forgetful. My external memory will be approximately correct, and I have a process for keeping it refined over time. So I can rely more on the vault instead of second-guessing myself based on incomplete memory. It tempers how I think about interacting with someone, not by changing my mind about them, but by giving me confidence that I'm not missing something important.

There are ethical boundaries. I don't capture personal details if people aren't comfortable with that. The dossiers are professional, not invasive.

Periodically, I do retrieval practice. This is how we make information stick; read "Make It Stick" to learn more. Review looks like this: I take my people notes and project notes and ask what's missing. Is there a piece of knowledge I remember that isn't captured? If yes, I fill in the blanks. I also check whether claims are substantiated with links and quotes. This fact-checking pass keeps the vault trustworthy and protects me from remembering something erroneous. A spell-checker list handles transcription errors, and my AGENTS.md links to HEARTBEAT.md to sanitize the vault of inaccurate information.

The final phase is producing outputs for others. I curate what gets published rather than exporting everything. The agent creates a publishable version based on my guidance. I haven't settled on hard rules for curation yet. I'd rather review and decide at publish time than tag things as publishable during capture. That workflow feels right to me.

For Confluence, a Python script publishes markdown directly, with YAML front matter defining the space and parent page. For GitHub users, notes can become Gists via the GitHub CLI. With the appropriate skills, Markdown files transform into HTML presentations, and with web technologies, those presentations become interactive. For Jira, a colleague created a skill that writes Jira tickets. We firmly believe that humans shouldn't be filling forms out; AI should be filling forms for us.

PowerPoint decks can be generated via Python scripts. Word documents come from markdown via Pandoc. The scripts run with uv, and LibreOffice handles conversions.

Each script maintains its own environment using PEP 723 inline script metadata. This means dependencies are declared at the top of each script in a special comment block:

# /// script
# dependencies = ["python-docx", "python-pptx", "pandas"]
# ///

When I run uv run script.py, uv automatically creates an isolated environment with just those dependencies, executes the script, and cleans up. No virtual environments to manage. No requirements.txt files scattered everywhere. No "works on my machine" problems.

The role of agent skills

Agent skills effectively encode procedural knowledge into executable markdown. Over time, it compounds; on fewer and fewer occasions do I need to repeat instructions, which is incredibly liberating. The model infers which skill to use most of the time. When it doesn't, I correct it explicitly and ask the coding agent to update the skill file for the future as well.

Designing a skill means thinking about the desired output and the tools needed to get there. I discover edge cases in the wild and update immediately. The earlier errors are caught, the better.

What's still friction

One pain point remains. I want to ingest Office files by pasting a URL, but I still need to download a copy first, then feed that copy to the agent skill. Programmatic access to cloud documents would eliminate this step. From the user side, nothing else would change. I'd just paste the URL and go.

But even with this friction, the system pays for itself. Knowledge management overhead dropped from thirty to forty percent of my time down to less than ten percent. I fix errors as I encounter them rather than scheduling dedicated maintenance. That recovered bandwidth goes toward better thinking and context gathering.

Getting started

What stops people from building systems like this? I believe it is two things: imagination and technical skill. You need imagination to envision converting diverse file formats into plain text. You need technical skill to know that it's possible.

The two feed each other. I experienced this with web technologies. Before I got familiar with building stuff on the web, I wondered what was even possible. Once I actually built things, I knew. Technical skill feeds your imagination, and imagination drives you to learn more technical skills.

For those starting without technical skills, use AI to learn programming. Find a language with a supportive human community to verify what you learn. AI hallucinates, and you need other people around you to help apply judgment and skill to AI outputs. You also need critical thinking skills and the initiative to act on what agents produce.

Skills you can use today

If you want to experiment with agent skills, here are some I've published:

html-presentations - Turn markdown into gorgeous HTML slides
gh-daily-timeline - See your GitHub activity for any given day
gh-activity-summary - Generate a plain-language summary of your GitHub work over any time period
publish-to-google-docs - Push markdown notes to Google Docs

The bigger picture

With such a system in place, repetitive, monotonous, and manual work can be offloaded to computers and AI. With a personal knowledge system, we can carry a broader scope of responsibilities and grow into new challenges for two reasons: we can externalize our memory more easily, and we can format information in ways that fit our brains.

I'm not asking people to do more at the same time. I'm asking them to expand their dynamic range over time so they're not stuck doing the same old boring thing over and over. That repetitive monotonous stuff should have been given away to AI and computers a long time ago.

This is useful for your career. It keeps things interesting. Every day that you make an incremental but permanent improvement compounds over time.

The vignettes I've shared are not a prescription. Rather, I hope you treat them as an invitation. Plain text plus coding agents is a powerful combination. Your system will look different from mine, and that's part of the point. Experiment and explore, and find what works for you.

How to stay in control when doing EDA with coding agents

2026-02-13T00:00:00Z

Speed without control is just chaos.

I've seen teammates compress a week and a half of analysis work into half a day using coding agents. That's a 5-10x speedup. But here's the thing: that speed only matters if you stay in the driver's seat. Otherwise you're not doing data science, you're just generating artifacts.

The real unlock isn't that agents write code fast. It's that they can be guided through a structure that keeps you in control of the analysis.

The problem isn't speed, it's agency

Coding agents are eager. Give them a CSV file and they'll open it, generate a dozen plots, and dump a wall of code before you've finished describing what you're actually looking for. That feels productive. It isn't.

The problem is that you've lost the thread. You didn't formulate a clear question. You didn't think through what the x-axis and y-axis should be. You're now reacting to whatever the agent produced, rather than steering toward an answer.

I've developed a different approach, codified in two skills I use with my coding agents: scientific-eda for exploratory data analysis and ml-experimentation for machine learning experiments. The pattern is the same in both: slow down first, gate on artifacts (plots, tables, etc.), and structure the session so both you and the agent can follow what happened.

Slow down first: the Socratic opening

The first design principle is counterintuitive: slow down before you speed up.

When you invoke the scientific-eda skill, the agent does not immediately load your data and start plotting. Instead, it asks you questions. What's the problem context? What are you hoping to learn or decide? What constraints matter?

From the skill definition:

Do not open the data file and start coding or plotting. Ask for or confirm: the problem context—biological, chemical, or data-science question; what the user hopes to learn or decide; and any constraints.

There's also an explicit guardrail: "ask 'why' before executing." When you request a specific plot or table, the agent briefly asks what question or decision it serves. This isn't bureaucracy. It's alignment. The agent is checking that you've thought through the request before it spends your time executing it.

This Socratic opening feels slower. But it prevents the far more common waste: generating plots you didn't need, going down rabbit holes you can't explain, and ending up with a folder full of artifacts and no clear answer.

Gate everything on artifacts

The second design principle is more specific: one artifact at a time.

If you can't describe what you want, you're not ready to execute. The agent waits. You think. You describe. Then the agent generates exactly what you asked for.

For a plot, this means articulating the x-axis, the y-axis, and what pattern you're looking for. What would confirm or refute your hypothesis? If you can't answer, the analysis isn't ready to run.

For a table, this means specifying the columns, rows, and aggregation level. What comparison are you trying to make? What decision will this table inform? A vague request like "show me the data" isn't actionable. "Show me the mean expression level by treatment group" is.

This is a forcing function for clarity. Describing an artifact precisely forces you to articulate the question you're actually asking. The tradeoff is worth it. You give up a bit of speed up front for precision in execution. And because the agent can generate code in seconds rather than minutes, the net result is still a massive speedup. My teammates went from 1.5-2 weeks to half a day. The precision tax is negligible compared to the execution dividend.

The session structure

Here's where structure becomes a feature, not overhead.

Each analysis session is a timestamped folder. The naming convention is ISO datetime plus a descriptive slug:

analysis/
  2025-02-05T14-30-00-protein-binding/
    journal.md      # append-only; shape, actions, findings
    plots/          # WebP figures only
    scripts/        # disposable PEP723 scripts; uv run

Let's walk through what each piece does.

journal.md is the memory. Before each action, the agent reads the journal. After each action, it appends what happened. Entries get timestamped and tagged: [SHAPE] for data structure discoveries, [PLOT] for visualizations, [FINDING] for observations, [NEXT] for suggested next steps. The journal is scannable. It's also the entry point for anyone (including future you) who wants to understand what happened without reading the code.

plots/ holds all figures from the session. The skill specifies WebP format for smaller file sizes, though that's a minor detail.

scripts/ contains disposable Python scripts. Each script has PEP723 inline metadata at the top, declaring its own dependencies. You run them with uv run script.py from the session folder. No environment wrangling. No "which virtualenv am I in?" confusion. One script, one plot.

The structure serves two purposes. First, it gives the agent a clear protocol to follow: read journal, execute, append to journal, suggest next step. Second, it leaves a trace that any human can follow. You can read the plan, then the journal, then the report, and understand the entire analysis without touching the code.

What changes in team conversations

Something unexpected happened when I started using this approach with teammates.

The conversations changed. We stopped asking "why did you write it that way?" and started asking "why did the agent write it that way?" The ego attached to code ownership evaporated. We could critique the work without critiquing each other.

This matters more than I expected. In the pre-agent world, I'd invest 50-70% of my mental energy on implementation details: wrangling data frames, handling edge cases, debugging syntax errors. That labor created attachment. When someone questioned my code, it felt like they were questioning my thinking.

Now the agent writes the code. I focus on the questions and verify the work. My teammates and I have more productive scientific conversations because we're discussing the analysis, not defending the implementation. We check the agent's work together, and if something's wrong, we just ask the agent to fix it.

Design for the human

The pattern that emerged is simple: if you design for the human, the agent follows.

The structure that makes your analysis traceable is the same structure that keeps the agent aligned. The journal that helps future-you understand what happened also helps the agent decide what to do next. The artifact-gating that forces you to think clearly also gives the agent precise instructions to execute.

You stay in control by slowing down at the decision points, describing what you want before you get it, and keeping a running record of what happened. The agent becomes a force multiplier rather than a loose cannon.

The skills I've linked here are just text files. They're prompts, structured in a way that an LLM-based coding agent can follow. You can copy them, modify them, or write your own. The key insight isn't in any particular skill, it's in the design pattern: gate analysis on artifacts, structure sessions with journals, and make the human's job explicit before the agent's job begins.

The 5-10x speedup is real. But the real win is that you get to stay the scientist.

How to Do Agentic Data Science

2026-02-01T00:00:00Z

Having tasted what agentic coding could look like for software development, I wanted to know what it would look like for data science - this meant training machine learning models and answering scientific questions. So I started experimenting, at work, and on my own at home as well. Here are ten lessons I've learned from my experiments thus far.

1. Be prescriptive in your prompting

Similar to building software, you need to know exactly what you want and how you'll evaluate the outcome. The difference, however, is as follows: With software, you will often know what you need to build, but with data science, you can only know what hypotheses need to be verified, which means you will need to iterate your way to the answer. Nonetheless, it is possible to leverage coding agents to move quickly.

The parallels are striking: if you frame each question you ask in terms of an observable outcome, you can set up your coding agent to write code that produces an output that can be evaluated for correctness, just like with software tests!

Here, your ability to describe precisely the hypothesis you're exploring, and the ability to describe in precise language what the answer would look like if the hypothesis held true or not, are critical components of what enables the coding agent to figure out what needs to be counterfactually true (within the codebase or the data) in order for your hypothesis to hold true.

Here is an example from my work. In a machine learning experiment with synthetic data, I wanted to hit 100% sequence editing performance. (It was synthetic data after all!) The coding agent hit a scenario where it was only doing 25%. With the clear goal in mind, it proposed edits to the code, edited the code, and re-ran experiments until it hit 100%. All without cheating; I know, because I checked!

2. Strong patterns in the file system

The agent, like humans, needs a predictable place for experiments. Similar to how a software repo has a conventional layout (src/, tests/, and so on), your experiments need a conventional layout so the agent knows where to put things and where to look. Within the experimentation skill that I wrote, I instruct the coding agent to do its work inside an experiments folder. Underneath that, for each experiment, we have datetime-prefixed subfolders, in which, there's a README file, a plots directory, a data directory, a scripts directory. Naming things logically helps, but the scheme matters more than the exact names. Coding agents will follow the patterns you already have.

3. Put logging instructions in AGENTS.md

With software, the feature one ask's a coding agent to build either succeeds or fails, and this can be automatically verified using programmatically-runnable unit and integration tests. With data science, experimental runs produce logs and metrics, but aren't easily boolean pass/fail like software tests. In both cases, however, and the agent can introspect logs to figure out what to change!

Your AGENTS.md file should include instructions for putting enough logging in place so the LLM can introspect what's going on during the experiment. I've written elsewhere about how to teach your coding agent with AGENTS.md and using AGENTS.md as repository memory for self-improving agents. Pair that with tools that run code in the terminal so the agent gets logs it can read. When the agent can read the logs, it can figure out what's wrong and what to change.

In my work, logging and printing to terminal were what let my agent fix a masking strategy that was only yielding 25% correctness. It read the logs, proposed a fix, re-ran, and got to where we needed to be. No intervention on my part. A 3 day experiment became 20 minutes.

4. Give it report-writing skills

The agent can write code and read mountains of logs, but you need something else: a human-readable summary of what it observed and what looked weird, so you can triage without re-reading every log. Give your coding agent instructions (e.g. in an agent skill) to write out in plain language what it observed during the model evaluation phase. It should read execution logs throughout the run. Tell it to write down anything that looks weird for follow-up. If something is off, it should say so. You get a readable summary and a list of things to dig into.

For reports (e.g. reports.md), encode in the skill that every table and every plot must be scrutinized. Ensure that plots are generated for every table, and that someone (you or the agent, with you verifying) carefully checks for inconsistencies between AI-generated plots and the tables they are supposed to reflect. The agent can miss things. It is valid to ask the AI to check its own work, but only if you have an idea of exactly where it is wrong and you tell it as such. Vague "double-check this" rarely helps; "the values in figure 2 do not match the second column of table 1" gives the agent something it can fix.

5. Have the agent keep an append-only journal of observations

Within the skill, instruct the agent to keep a single file (e.g. notes.md or journal.md) that it is told to only append to, never overwrite. The journal is not just for the agent. You should add to it too: things you noticed while looking at the data, gut feelings, weird patterns. It becomes a running log of what was going on, from both sides, that you can go back and summarize later. The point is to capture the thought process while you are doing the work.

6. Have the agent generate diagnostic plots for you

Logs and plots are complementary: logs are agent-accessible, but plots are human-accessible versions of the same underlying performance data. Have the agent generate diagnostic plots for you. The agent can propose fixes from the logs, but it can't build your intuition; you're the one who has to smell when something is off. Nothing beats looking at the data yourself, otherwise you never build intuition for what's happening! I still looked at the logs and plots myself to make sure the metrics were real and the agent wasn't hallucinating. Your prior experience is what lets you smell when something is off.

7. Instruct the agent to write the minimalist version first

With software, you run tests in seconds. With ML, you're tempted to train for hours and rush to the real data and the real training run. As a human, you don't want to "waste" time proving out the pipeline when you could just run the full thing, but that mentality is exactly what makes you unable to debug machine learning code on a tight loop. That temptation is exactly why you should instruct the agent to do the opposite: write the minimalist version first, then use it to work out elementary errors before scaling up.

That means train for one iteration, not even one epoch. Use miniature versions of the final model (e.g. a tiny custom deep net with the same architecture but a fraction of the parameters). Check for shape errors, data-loading bugs, and that the forward pass runs end to end. All the sanity checks you would do manually to prove that things work, but that you are tempted to skip. Encode in AGENTS.md that the agent must implement and run this minimal version before moving to full-scale training. The agent does not have your impatience; use that to your advantage. Once the minimal run passes, you can scale up with confidence.

8. Ask the coding agent to guide you through step by step

I do this often with large software refactors within canvas-chat, in which I ask the coding agent to prioritize for me a list of manual checks I need to look at. This is particularly helpful when I'm (a) context switching back into the project, or (b) running on fumes but my gut tells me we're so close to the end. (Though really, you shouldn't be doing any work if you're close to dozing off at 10:30 pm...)

The same applies to data science and data exploration! After having the coding agent autonomously execute on your experiment, you can have it walk through what it's done step by step, giving you the space to operate at your pace -- at the speed of your thought! Of course, if you're in a better state than merely "running on fumes", you can (and should) treat the coding agent as a research partner and ask questions back to critically evaluate whether the output is correct or not. What I have found is that there will still be unexplored paths that need to be trodden, and you can send a coding agent off on that direction on the side.

9. Learn the vocabulary the coding agent uses

Pay attention to the terms the agent uses when it describes what it did. You can reuse that vocabulary in future prompts and get more precise results. For example, in developing canvas-chat, I used this non-optimal verbiage in my prompt:

ok, I see, the default node size made it such that the next/prev buttons were hidden away. Can we make the pagination controls visible regardless of the size of the node?

Cursor's agent replied with something like "making the pagination toolbar sticky". That gave me a more compact way to express exactly what I need the next time. If you don't know the vocab at first, this is a great way to expand your technical vocabulary too.

10. For exploration, treat the agent as an executor that follows your curiosity

What you don't want in agentic data exploration is for the coding agent to hand you a boatload of output and leave you no room to follow your own curiosity. Flip the table: treat the agent as an executor of your ideas. You lead; it follows. Instruct it that it is not allowed to race ahead. It should only execute on the one thing you want, and it should ask you questions to clarify and narrow down what you actually want before it goes and does it. In other words, it is there to be a jazz partner for your data exploration.

You can run that partnership a few ways. One is to have the agent write scripts that produce plots on disk; you run them, look at the output, then ask for the next thing. Another is to go one level higher and work inside Marimo notebooks, using Marimo's reactive execution so you go one cell at a time, one question at a time. I've written about using coding agents to write Marimo notebooks if you want to try that path.

The agents handle the implementation. You handle the inquiry. The ten practices above - prescriptive goals, clear structure, logging, reports, an append-only journal, diagnostic plots and your own eyes on the data, the minimalist version first, having the agent guide you step by step when you need it, learning the agent's vocabulary, and in exploration keeping the agent as your jazz partner - are what make that partnership work. I've spent nearly a decade training ML models by hand, so I know what I want, and I have developed a sense of taste for what success looks like. You can get to the same level of taste with AI assistance, but you must work for it. I'll write separately about how I'm learning new things with AI. The point is not to hand off the science, but to do more of it!

Model feel, fast tests, and AI coding that stays in flow

2026-01-25T00:00:00Z

Most of the conversation about AI coding models focuses on performance metrics. Benchmarks, evals, pass rates, latency. Useful stuff, but it misses the part that actually shapes my day-to-day: what it feels like to work with the model.

Once you start using LLMs as coding agents, the qualitative experience becomes a throughput issue. It affects how often you intervene, how much you trust what is happening, and whether you stay in flow or spend your time cleaning up weird breakage.

Two axes keep showing up for me.

First is time horizon and supervision style: long-horizon autonomy versus short-horizon iteration.

Second is personality and verbosity: how the model behaves when it is wrong, how much it narrates, and whether it stays constructive or spirals into apology loops.

There is also a third ingredient that ends up mattering as much as the model: the agentic harness. By that I mean the tools and checks that the agent can run to verify it did not break behavior, and whether the harness gives you streaming and visual feedback—a live trace of what the model is doing—or leaves you staring at a spinner until the answer drops. Good harness beats model swapping more often than I expected.

Long-horizon autonomy vs short-horizon iteration

I call it "Opus-feel" when a model has that "ask and it shall be given" vibe with a longer time horizon. You describe what you want, it runs for a while, and it comes back with a plausible scaffold. It is great for momentum.

I call it "Sonnet-feel" when a model leans toward shorter-horizon iteration. It works better when you are walking through a real codebase step by step, keeping changes small enough that you can validate what happened, correct course, and keep going.

Another way to put it is that long-horizon autonomy pushes you toward a spec-and-review loop, while short-horizon iteration pushes you toward a steer-and-verify loop. Both can be productive. They just fail differently.

In a sufficiently large codebase, you cannot rely solely on long-horizon autonomy where you ask for something with a vague description and hope it lands cleanly. You are not always guaranteed something well organized, especially when the job is refactoring rather than greenfield scaffolding.

A concrete example for me came from Canvas chat. At the time, everything was tied to app.js and app.py. When I wanted to refactor things into plugins, I needed to dogfood a plugin pattern in the codebase itself.

Long-horizon autonomy struggled here. It could generate a plugin pattern, but it was not great at the careful, incremental work of extracting behavior out of a monolith and into a clean plugin boundary.

Walking bit by bit with Sonnet or Sonnet-quality models was a very different experience. The big win was that I could study the LLM traces live (the tool calls and file edits it proposes step by step) and see where edits were being made. If I noticed a feature handler getting added to app.js when it clearly belonged in a plugin file, I could intervene immediately and ask, "Why is that thing over there in app.js? Why is it not inside the plugin file instead?" That kind of interactive, traceable work is where the short-horizon models shine.

Examples from my own testing, with all the usual caveats: Opus-4.5 (Anthropic), GPT-5.2 (OpenAI), and GLM-4.7 (z.ai) have been solid for the long-horizon, get-it-moving-fast mode. Minimax M 2.1 (OpenCode Zen) feels closer to the short-horizon mode for me. Composer-1 (Cursor) also feels closer to that style. I suspect GPT-4o and GPT-5.1 (both OpenAI) might land there too, but I have not really test-driven them.

The practical takeaway is that I now switch modes on purpose. When I need speed and initial momentum, I reach for long-horizon autonomy. When I need control, I choose a short-horizon model so I can babysit the work, watch the traces, and intercept it when it tries to do something clever in the wrong place.

The harness lesson (Cypress beat model hopping)

One more lesson from that period: I did a bunch of model hopping, trying to find something that would fix a particular class of behavioral breakage.

The most frustrating failures were not subtle logic bugs. They were basic syntax errors introduced during tool-call patching, unclosed brackets, unclosed parentheses, that kind of thing. When that happens, you do not get a slightly-wrong feature, you get a page that fails to load. Debugging it manually is fine the first time, and infuriating on the seventh.

The thing that actually moved the needle was listening to my colleague Anand Murthy and instantiating Cypress tests. A simple automated page reload catches those failures immediately. It shifts the pain earlier, gives the agent a verification loop it can run on demand, and turns "agentic coding" into something I can trust.

Here is the dumbest possible example, taken straight from the Cypress suite in canvas-chat. It is not fancy, and that is the point. It catches "the page does not load" failures quickly.

describe('Help Modal and Auto-Layout', () => {
    beforeEach(() => {
        cy.clearLocalStorage();
        cy.clearIndexedDB();
        cy.visit('/');
        cy.wait(1000);
    });

    it('opens and closes help modal', () => {
        cy.get('#help-btn').click();
        cy.get('#help-modal').should('be.visible');
        cy.get('#help-close').click();
        cy.get('#help-modal').should('not.be.visible');
    });
});

Beyond model choice, a great agentic harness matters. If your harness includes tests that a coding agent can run to verify no behavioral breakage, you get to move faster with more confidence, regardless of which model you are using.

Verbosity, attitude, and the cost of being wrong

The other axis that became obvious once I started pressure testing models is verbosity and its associated feel.

I tried Gemini 2.5, and it was a disaster for me. After experiencing the long-horizon and short-horizon styles, I did not want to use it. It made elementary mistakes, like leaving trailing dangling curly braces where they were not supposed to be. Then it would apologize profusely over and over, like a Canadian on steroids. (I'm a born and bred Canadian, I'm allowed to say that!)

In contrast, Claude and Opus are consistently upbeat and positive, and the same can be said for Minimax-M.2 and GLM-4.7. That matters more than I expected. When something breaks and you are iterating quickly, a model that stays constructive keeps the whole loop feeling fun.

On the other end, GPT-5.2 would just go ahead and do things without being overly effusive, then loop back to tell me what it did. That sounds fine on paper, but it left me feeling a bit clueless. I would wonder what it was doing and whether I could intercept it if it went off on the wrong tangent. I often could not, because I needed to wait until the end to learn what it decided to do.

So yes, I care about correctness. But I also care about how a model behaves while it is getting to correctness. The journey matters because the journey is where you spend your time.

Enthusiasm is a feature

This ties nicely to a tweet I saw from Grady Booch (@Grady_Booch):

"The greatest value such tools have offered me is to reduce my cognitive load and automate various tedious tasks."

Here is the punchline:

"To serve as an enthusiastic and indefatigable, albeit very naive and often unreliable, pair programmer."

That enthusiasm and indefatigability, compared to a grumpy human, keeps the loop moving.

My frustration pair coding with Gemini was not just the mistakes, it was the emotional texture of the interaction. It would make mistakes and then apologize, repeatedly. After a while, you start optimizing your own behavior around the assistant's vibe, and that is not where you want your attention to go.

A better pair programmer, human or AI, is relentlessly game for the next challenge. It affirms what you are trying to do, it corrects you when you are wrong, and it does not act like it is giving up. When the assistant stays constructive, the work stays fun.

Streaming and the illusion of speed (the harness + model)

Raw latency is one thing. What you see while the model is working is another, and that is determined by the harness. In Cursor, you get a fast stream of tool calls and edits. You see the so-called thinking process. Something is clearly happening. With GLM-4.7 or Open Code in certain setups, you wait a long time with nothing streamed in—just a spinner or a blank state until the full response lands. Same model capability, same task, different harness, totally different experience. The harness that gives you a live trace makes the wait feel shorter and keeps you in the loop. The one that hides progress makes every request feel like a gamble. If you care about flow, streaming and visual feedback are not polish; they are table stakes, and they live in the harness.

The "feel" is also vendor lock-in

After enough hours with a single model, you start building muscle memory for its quirks. You learn how to phrase prompts so it does the right thing. You learn which mistakes to expect. You even learn its tone. That comfort is sticky.

The sticky part is the problem. Getting used to a model's ergonomics is a form of vendor lock-in, and it is something I am determined to avoid.

That is one reason I have been bouncing between models (apart from me hitting limits) to feel out the ragged frontier of model behavior. It is pretty revealing. You quickly learn that "best model" is not a single number. The model you want depends on whether you are scaffolding, refactoring, debugging, or doing the last-mile polish.

If you want to keep your agency while using these tools, stay fluent across multiple feels. Otherwise you end up optimizing your workflow around one model's quirks and calling it productivity.

A more pragmatic way to think about model choice

What I do now is less romantic than "find the best model". I think in terms of work phases and feedback loops.

If I am scaffolding, I will happily take Opus-feel: longer-horizon autonomy and a big blob of output, because the cost of being wrong is usually low.

If I am refactoring or debugging, I want Sonnet-feel: short-horizon iteration and tight supervision, because the cost of being wrong is a broken app and a bunch of time lost to verification.

And if I keep hitting the same dumb failures, I try to fix my harness before I try to fix my model. Add the smallest test that fails fast, make it runnable by the agent, and suddenly the whole system behaves better. Cypress reloading the page and clicking one button did more for my sanity than another week of model hopping.

At a systems level, I want a workflow where models are swappable components. In practice that means traces you can read, tests you can run, and a loop that tells you quickly when the agent broke something.

How to build self-improving coding agents - Part 3

2026-01-19T00:00:00Z

In part 1, I covered AGENTS.md as repo memory.

In part 2, I covered skills as reusable playbooks.

This post is about turning those two ideas into something you can run as a practice.

The maturity model

Once you have both repo memory and skills, you can think about how the practice evolves over time.

Stage 0: Ad hoc prompting

You keep re-explaining the same things in chat. It works, but it does not compound.

Stage 1: Repo-local memory

You add repository-specific guardrails and a code map.

This is where AGENTS.md shines.

Stage 2: Global personal skills

Once a workflow repeats across repos, you promote it into a global skill on your machine.

If you want a concrete bootstrap set, here is what I would install globally:

skill-creator: lowers the activation energy for making new skills.
an installer and updater for skills, for example openskills: makes distribution and updates less annoying.
agents-md-improver: keeps the repo map current without you thinking about it.

Stage 3: Shared skills

If a workflow repeats across a team, it belongs in a shared location with a clear install path.

I do not think you should start here. Start repo-local, then promote only when you feel the pain twice.

Promotion decisions come from paying attention to what the agent actually does in practice.

Watch traces, then distill constraints

If you work with agents long enough, you start to notice the model’s default moves.

When I see an agent repeatedly:

taking an overcomplicated path
missing a file I know is relevant
applying a global refactor when a surgical fix is needed

I treat that as a signal.

Then I decide what kind of fix it is.

If it is a repo invariant, a navigation hint, or a local norm, it belongs in AGENTS.md. That is the always-on context for how work should happen in this repo.

If it is a repeatable procedure with a clear output contract, it belongs in a skill.

Sometimes the procedure is repo-specific. In that case I keep it as a repo-local skill. If I feel the pain twice in another repo, I promote it into a global skill.

This is how you get operational learning without pretending the model is learning.

Underneath, a lot of this comes down to writing instructions in a way that can be executed.

Markdown is becoming executable

One reason this whole approach works is that the agent can execute what you write.

When an LLM can execute tool calls, Markdown becomes an executable language.

Skills fit this pattern. A SKILL.md is just a structured instruction sheet, but it is also runnable in the sense that the agent can turn it into searches, file reads, edits, and command execution.

The other trick is that skills are loaded on demand. The agent reads a short description first, then loads the full instructions only when it needs them.

You can write a precise plan in plain language, and the agent can turn it into:

searches
file reads
surgical edits
test runs

This is not magic. It still depends on linguistic precision. But the ergonomics shift. You can describe a workflow at the level you actually think about it, then let the agent do the clerical work.

This is also why I like the runbook analogy, even with the caveats.

When to update `AGENTS.md` vs create a skill

Skills tell an agent how to do something.

AGENTS.md tells an agent how this repo works, and what rules it must follow while doing anything at all.

Here is how I decide.

Update AGENTS.md when the instruction is specific to the repo:

navigation help: where things live, what files matter, what to ignore
local norms: build commands, test commands, environment rules, style constraints
guardrails: what not to do in this repo

Create a skill when the workflow is reusable, or when you want a named, on-demand playbook:

a multi-step procedure you want to invoke repeatedly
a workflow that spans repos or products
a task with a strict output contract (release announcements, status updates, summaries)

If I am unsure, I start repo-local. If I feel the pain twice in another repo, I promote it into a global skill.

The meta skill is metacognition

The most valuable “skill”, however, is not a file format. It is the habit of watching yourself work.

I try to ask: what am I doing repeatedly that should be systematized?

If the answer is “I keep re-explaining how this repo is organized”, that goes into AGENTS.md.

If the answer is “I keep asking for the same kind of summary, debug sequence, or release note format”, that becomes a skill.

Once you start doing this, you build a compounding loop. The agent handles more of the repeated work, and you spend more time on judgment and design.

If this all sounds like more than coding, that is because it is.

Where this seems to be going

I buy Simon Willison’s framing that these tools are general agents disguised as developer tools (Claude Cowork post).

Even if you start with coding, the moment an agent can run terminal commands and manipulate files, the surface area expands to “almost anything”, as long as you know how to steer it.

That matches how I use coding agents.

Yes, I use it for coding work. But I also use it for other intellectual work: ghostwriting blog posts (which I scrutinize heavily, because the review process is essential for me to own the content), writing release announcements, and turning messy notes into structured drafts.

I have also heard Theo Brown make a similar point when talking about Claude Cowork (video). The details vary, but the pattern is the same: once you have a general agent, the label “coding tool” becomes more about marketing and UI than capability.

So I am increasingly convinced that the long-term shape here is web-deployed agents with less scary branding.

You will still want composable components for LLM workflows. But for day-to-day work, the most useful thing is an agent that can execute commands and apply changes, while carrying a growing set of skills and repository memory.

That combination is what makes the agent feel less like a chat box and more like a teammate.

How to build self-improving coding agents - Part 2

2026-01-18T00:00:00Z

In part 1, I focused on repo memory with AGENTS.md.

In this post, I am switching to the other lever: skills.

Skills are prompt compression

Skills are the other half of the system.

When a task repeats, I do not want to keep re-explaining the workflow. I want a playbook I can invoke.

What a skill is

A skill is a folder with a SKILL.md file.

The SKILL.md is the prompt. The bundled scripts and assets are the tool layer.

A good skill makes three things explicit:

when to use it
what steps to take
what good output looks like

If you want the spec, see Agent Skills.

Skills are best formed around jobs to be done: concrete, repeatable workflows rather than abstract capabilities. Think "debug a GitHub Actions failure" or "draft a release announcement," not "know about CI" or "write good prose." When the job is clear, the skill has a natural boundary and a clear trigger. When it is vague, the skill is hard to invoke and hard to improve.

A wrong framing is "skills for tools." Skills get invoked in the loop of trying to accomplish a job, not in the context of trying to use a tool. The tool is a means; the job is why you reach for it. If you design a skill around a tool, you end up with something the agent has to remember to use. If you design it around a job, the agent reaches for it when the job shows up.

Examples

A GitHub debugging skill is the obvious starting point. CI failures are repetitive and usually want the same sequence: identify failing jobs, pull logs, inspect diffs, reproduce locally, then patch.

A second example is a release announcement skill.

The motivation here was not abstract. I was spending a good half hour each release just trying to compose the announcement, and I did not want to do that anymore.

The output contract was also specific. I wanted release announcements that are copy-pasteable into Microsoft Teams, with emojis, but otherwise minimal formatting because Teams formatting is inconsistent.

A third example is more technical.

At work I had a session with a coding agent to train an ML model inside a script. After that session, I had it write a report on what it learned and what changed. Then I turned that report writing into a skill.

The report format was familiar to everyone on the team: Abstract, Introduction, Methods, Results, Discussion.

The content came from real artifacts: stdout logs, metrics, code, config files, git diffs, and the agent’s own session history.

A fourth example is about tacit domain expertise.

A teammate of mine created a skill that encoded her implicit knowledge from years of debugging chromatography traces. The point was not that the agent suddenly became a scientist. The point was that her debugging procedure became explicit and reusable.

Skill creation and iteration

I now like skills because they are easy to iterate on. I used to be more skeptical, and I still think MCP servers have a cleaner distribution story, but my opinion has shifted as I have used skills more in real workflows (Exploring Skills vs MCP Servers).

For the release announcements, I fed my coding agent a few examples of what “good” looked like. I was using Anthropic’s skill-creator skill at the time, and those examples became part of the skill itself, stored as assets that the agent could reuse.

This is a huge energy barrier reducer. It is much easier to iterate on a Markdown-based skill than it is to start from scratch with “write me a Python script that does X”. You can still add scripts inside a skill when you need determinism, but the interface is the Markdown.

The other half is the feedback loop. When I edit the generated release announcement, I feed the revised version back to the agent and tell it to update the skill with the new example. That way the skill evolves as my taste evolves.

This is also a way to share. A skill is reviewable. I can open a PR and let collaborators comment on both the output and the process that produced it.

In the chromatography example, using skill-creator to generate the first draft mattered for another reason too. English is not my teammate’s first language. The structure makes it much easier to get from “I know what I do” to “here is the procedure an agent can follow”.

Distribution and updates

This is where skills feel less mature than MCP servers.

An MCP server has a clean distribution story. You can pip install it, configure auth once, and you get a centrally versioned bundle of prompts and tools. Updating is a normal package update.

Skills still involve moving folders between machines and repos, and remembering where each harness expects skills to live.

I originally ended up writing a skill-installer skill. It is the same move as skill-creator, but for distribution and updates.

When I say “install this skill” or “update this skill from this URL”, the agent needs to ask two key questions if I have not already specified them:

is this repo-local or machine-global?
which harnesses should discover it?

Then it does the boring part consistently.

Update: it looks like openskills now solves most of what I wanted here, and it does it more deterministically. It is a CLI that installs skill folders from GitHub or local paths, tracks their sources for updates, and can target multiple install locations.

OpenSkills has a "universal" mode that installs to .agent/skills (repo) and ~/.agent/skills (machine).

The caveat is that .agent/skills is not a universal discovery standard across harnesses. Some tools look in .claude/skills, .github/skills, .opencode, or other locations. So OpenSkills helps with deterministic installs and updates, but you still need to know what your harness will actually read.

I expect this to converge soon.

At this point you have both memory and playbooks. The question becomes how you decide what to invest in next.

Coming next

Part 3 covers the operating model.

It lays out a maturity model, a concrete bootstrap set of skills to install globally, and a decision rule for when to update AGENTS.md versus when to create a skill.

How to build self-improving coding agents - Part 3

How to build self-improving coding agents - Part 1

2026-01-17T00:00:00Z

I want my coding agents to get better every week.

Not in the abstract “the models are improving” sense. I mean it in the operational sense: if an agent makes a mistake, or takes a path I would not take, I want that feedback to stick. If I have to repeat the same preference every session, I am not using an agent. I am babysitting a very fast intern.

The trick is that the model weights are not changing mid-week. So if you want “self-improvement”, you need to change the environment the agent works inside.

I have found two levers that compound:

AGENTS.md as repository memory
skills as reusable playbooks

This post is a longer “source of truth” version. My intent is to later break it into smaller blog entries, and also rework it into chapters for my data science bootstrap notes.

Where improvement comes from

The UX I am after is simple: I stop repeating myself. I stop doing the same end-of-day cleanup, writing the same reminders, re-explaining where files live. The agent starts each session closer to how I want it to work.

If the model weights are not changing mid-week, improvement has to come from the environment you wrap around the agent.

For me that environment has two pieces:

durable repository memory (AGENTS.md)
reusable playbooks (skills)

Once you have those two, you can treat “agent improvement” like runbooks plus postmortems.

The analogy is imperfect, because this is not documentation for humans. The loop is the same though: write down the repeatable steps, then write down what surprised you and what you will do differently next time.

The difference is that natural language can turn into tool calls. When you write things down precisely, the agent can execute them.

I usually start with AGENTS.md, because it cuts down exploration immediately.

`AGENTS.md` as repository memory

If you have not run into the AGENTS.md convention before, see agents.md.

To be effective, AGENTS.md needs to do two things for the agent.

First, it needs to make the agent fast at navigating the repo so it can get to the right files with minimal wandering. A code map is a straightforward way to do that.

Second, it needs to encode the local ways of working in this repo so the agent stops repeating the same mistakes. That is where corrections and norms live.

This is the loop I want:

I observe a mismatch.
I tell the agent what must be true.
The agent writes the correction into AGENTS.md (or a repo-local skill).
The agent reads it next time.

In the ideal state, the agent gets to the right files quickly.

A code map is the simplest way I know to make that happen. It does not have to be perfect. It can be slightly stale and still be useful.

I have seen this pay off in a very practical way. In my canvas-chat codebase, having a map of the repo let the agent one-shot an obscure spot where events were emitted for node rendering. Without a map, the agent previously needed 5 to 6 rg searches, just to find the right neighborhood of the code.

The difference is small in absolute time, something like 40 seconds versus 2 seconds. But it changes the feel of the collaboration. The agent spends less time wandering, and I spend less time steering.

Close the loop when the map is stale

There is one extra move that makes this feel self-correcting: When the agent notices that the code map looks stale, it should update the code map.

This is a subtle point. The map is not a static artifact. It is part of a feedback loop. When the agent’s exploration discovers a mismatch between the map and reality, that discovery should flow back into AGENTS.md.

You can encode this as an explicit instruction inside AGENTS.md. You can also refresh on a schedule, like weekly, but the on-demand update is the part that makes the loop feel alive.

Corrections that become durable norms

The second job of AGENTS.md is to hold repo-specific corrections to agents behaviour.

These are the things you find yourself saying out loud.

Two examples from my own work:

Run Python in the pixi context. Use pixi run python ....
Do not cheat by modifying the tests to make them pass.

I say the first one because the agent will often try python -c ... to quickly check something. In a pixi-managed project, that fails if you do not have a global Python.

I say the second one because changing tests to make them pass destroys the point of having tests.

Once these rules are written down, the agent stops making you restate them. This is the simplest way I know to reduce repeated friction.

A starter prompt for generating `AGENTS.md`

I have found it useful to bootstrap AGENTS.md with a one-time deep dive.

Here is a prompt I use as a starting point. It is intentionally repo-specific.

You are a coding agent. Read through this repository and create an `AGENTS.md` file at the repo root.

Requirements:
- Include a short codebase map that helps an agent find files quickly.
- Focus on entry points, directory roles, naming conventions, configuration wiring, and test locations.
- Add a section called "Local norms" with repo-specific rules you infer from the code and tooling.
- Add a section called "Self-correction" with two explicit instructions:
  - If the code map is discovered to be stale, update it.
  - If the user gives a correction about how work should be done in this repo, add it to "Local norms" (or another clearly labeled section) so future sessions inherit it.

Process:
- Use search and targeted file reads, do not read every file.
- Prefer `rg` searches to find entry points and configs.
- Prefer high-signal files: `README`, `pyproject.toml`, `package.json`, `Makefile`, `opencode.json`, `.github/workflows`, and top-level `src` or `app` directories.

Output:
- Write the final `AGENTS.md` contents in Markdown.
- Keep it concise. Optimize for navigation and correctness.

If you want, you can go further and add a cadence rule like “refresh weekly”, but I would keep it lightweight. The goal is compounding value, not bureaucracy.

Once AGENTS.md exists, skills are the second lever.

Coming next

Part 2 is about skills as reusable playbooks.

It covers what a skill is, several examples from coding and scientific work, and why I ended up writing a skill-installer skill to deal with the current distribution story.

How to build self-improving coding agents - Part 2

How I fixed a browser selection bug with sequence alignment algorithms

2026-01-06T00:00:00Z

I ran into a frustrating bug this week in canvas-chat, my experimental canvas-based chat interface I built at the end of last year. The bug seemed simple on the surface: when users selected text from a rendered markdown table and clicked to highlight it, the highlighting would sometimes stop partway through, or highlight the wrong characters entirely.

What started as a "quick fix" turned into a journey through several failed approaches before I remembered an algorithm from my undergraduate bioinformatics days. Sometimes the best solution to a problem comes from an unexpected domain.

The problem: Browser selections are messy

Canvas-chat has a feature where you can select text from an AI response, and the app creates a "highlight" node that links back to the source. When you click on the highlight, the corresponding text in the source gets wrapped in a <mark> tag.

This worked fine for simple paragraphs. But when I tried it on tables containing KaTeX-rendered math, things went wrong:

What I expected to highlight: 66.00 (0.18±0.58)

What actually got highlighted: 66.00 ( 0.18±0.58)

The highlighting was off by more than a few characters, and would stop before the end of my selection. In some cases, it would highlight completely wrong sections.

Digging into the root cause

The problem came from how KaTeX renders math and how browsers handle text selection.

KaTeX renders math with multiple text representations:

<span class="katex">
  <!-- MathML for accessibility/screen readers -->
  <span class="katex-mathml">
    <math><mn>0.13</mn><mo>±</mo></math>
  </span>
  <!-- Visual HTML for display -->
  <span class="katex-html">
    <span class="mord">0.13</span>
    <span class="mord">±</span>
  </span>
</span>

When you select text that spans across KaTeX-rendered content, selection.toString() gives you something like this:

"66.00 (
0.13
±
0.13±0.58)"

Notice the duplicated 0.13 and the random newlines? The browser included text from both the MathML (for accessibility) and the visual spans. Add in tabs between table cells and inconsistent spacing around operators, and you have a string that looks nothing like the clean HTML text content.

First attempt: Normalization layers

My initial approach was to normalize both strings (the user's selection and the HTML text) before matching:

Collapse all whitespace to single spaces
Remove KaTeX duplication patterns (like 0.13 ± 0.13± → 0.13±)
Normalize spacing around ± operators

Then find the match in the normalized strings, and map the positions back to the original.

This is where things got complicated. I needed to track:

Which positions in the normalized string corresponded to which positions in the original
How to reverse the mapping after finding a match
How to handle characters that got removed entirely during normalization

The code became a tangled mess of position arrays and off-by-one bugs. Here's a simplified version of what it looked like:

// Build mapping from normalized to original positions
const normalizedToOriginal = [];
let inWhitespace = false;
let leadingTrimmed = true;

for (let i = 0; i < fullText.length; i++) {
    const ch = fullText[i];
    if (/\s/.test(ch)) {
        if (!inWhitespace && !leadingTrimmed) {
            normalizedToOriginal.push(i);
        }
        inWhitespace = true;
    } else {
        leadingTrimmed = false;
        inWhitespace = false;
        normalizedToOriginal.push(i);
    }
}

// Then also account for math spacing normalization...
// And KaTeX deduplication...
// Each layer compounds the position mapping complexity

The position mapping kept breaking. I'd fix one case only to break another. I was trying to maintain a bijection between two strings that had been transformed through multiple non-invertible operations. It wasn't going to work.

The insight: This is a sequence alignment problem

After banging my head against the normalization approach for a while, I took a step back. What was I actually trying to do?

I had two strings:

The user's selection (messy, with artifacts)
The HTML text content (clean)

I needed to find where the user's selection "matched" in the HTML text, tolerating:

Insertions (extra whitespace, duplicated characters in the selection)
Deletions (characters present in HTML but not in selection)
Mismatches (different whitespace characters)

This is exactly what sequence alignment algorithms are designed for. In bioinformatics, we use these algorithms to compare DNA or protein sequences that may have evolved with insertions, deletions, and mutations. The classic algorithm for finding the best local alignment between two sequences is Smith-Waterman.

I learned Smith-Waterman as an undergraduate, probably around 2008. I never thought I'd use it for web development.

The solution: Align the beginning and end

I didn't need to align the entire selection - I just needed to find where it started and ended in the HTML text. So I:

Take the first ~20 characters of the user's selection and align them to find the start position
Take the last ~20 characters, reverse both strings, align to find the end position

Here's the core alignment function:

function alignStart(queryPrefix, target) {
    const m = queryPrefix.length;
    const n = target.length;

    const MATCH = 2;      // Reward for matching characters
    const MISMATCH = -1;  // Penalty for different characters
    const GAP = -1;       // Penalty for insertions/deletions
    const WS_MATCH = 1;   // Softer reward for whitespace matches

    // Build the scoring matrix
    const score = Array(m + 1).fill(null)
        .map(() => Array(n + 1).fill(0));

    let maxScore = 0;
    let maxI = 0, maxJ = 0;

    for (let i = 1; i <= m; i++) {
        for (let j = 1; j <= n; j++) {
            const qChar = queryPrefix[i - 1].toLowerCase();
            const tChar = target[j - 1].toLowerCase();

            let matchVal;
            if (qChar === tChar) {
                matchVal = /\s/.test(qChar) ? WS_MATCH : MATCH;
            } else if (/\s/.test(qChar) && /\s/.test(tChar)) {
                matchVal = WS_MATCH;  // Any whitespace matches any whitespace
            } else {
                matchVal = MISMATCH;
            }

            score[i][j] = Math.max(
                0,  // Local alignment can restart anywhere
                score[i-1][j-1] + matchVal,  // Diagonal: match/mismatch
                score[i-1][j] + GAP,          // Up: gap in target
                score[i][j-1] + GAP           // Left: gap in query
            );

            if (score[i][j] > maxScore) {
                maxScore = score[i][j];
                maxI = i;
                maxJ = j;
            }
        }
    }

    // Traceback to find start position
    // ... (walk backwards from maxI, maxJ to find where alignment began)
}

The key insight is that Smith-Waterman's local alignment naturally handles all the messiness. Extra newlines in the selection? They're just gaps. Duplicated numbers? They align to the same position. Different whitespace characters? They all match each other.

The result

The new approach passes all the test cases that the normalization approach failed:

Test: Simple word

Target: "Hello world, this is a test."
Query: "world"
Result: world (positions 6-11)

Test: KaTeX duplication

Target: "66.00 (0.18 ± 0.18±0.58)"
Query: "66.00 (\n0.18\n±\n0.18±0.58)"
Result: 66.00 (0.18 ± 0.18±0.58) (positions 0-25)

Test: Cross-block selection

Target: "The Heading Some paragraph text here."
Query: "The Heading\n\nSome paragraph"
Result: The Heading Some paragraph (positions 0-25)

The lesson: Know your algorithms

I didn't invent anything new here. Smith-Waterman has been around since 1981. I just recognized that my web development problem was, at its core, a sequence alignment problem.

This is why I think it's valuable to study algorithms and techniques from different domains, even if they seem unrelated to your day-to-day work. You never know when dynamic programming from bioinformatics will solve your JavaScript text highlighting bug.

The normalization approach was trying to make two messy things identical before comparing them. The alignment approach embraced the messiness and asked: "Given that these are different, where do they best correspond?"

That's a fundamentally different framing, and it's the framing that actually matched the problem.

Interestingly, I couldn't find prior examples of using Smith-Waterman specifically for UI text highlighting or matching browser text selections to source HTML. The algorithm is well-established in bioinformatics for DNA and protein sequence alignment, and it appears in some fuzzy string matching contexts like spell-checking and record linkage. But applying it to handle the specific artifacts that browsers introduce when selecting text from rendered HTML with KaTeX, MathML, or complex table structures? That seems to be a new application. Sometimes the best solutions come from recognizing that your problem, despite appearing domain-specific, maps onto a well-solved problem from an entirely different field.

One more note: I didn't write the JavaScript implementation myself. I directed Claude Opus 4.5 in OpenCode to write it for me. My contribution was recognizing that this was a sequence alignment problem and describing the approach - the actual code was generated by the AI. This is becoming my preferred way to work: I provide the domain insight and algorithmic direction, and the AI handles the implementation details.

Appendix: The full solution

For those curious, the complete implementation is in the pull request. The key functions are:

alignStart(queryPrefix, target) - Find where the query beginning matches
alignEnd(querySuffix, target) - Find where the query end matches (by reversing and aligning)
findMatchRegion(query, target) - Combine both to get the full match region

The algorithm runs in O(mn) time where m and n are the lengths of the strings being aligned. For typical text selections (tens to hundreds of characters), this is instantaneous. And unlike the normalization approach, it's robust and correct!

Canvas Chat: A Visual Interface for Thinking with LLMs

2025-12-31T00:00:00Z

I've been mulling over this idea since last year January: A visual, nonlinear interface for LLM conversations—something like an infinite canvas where you could branch, merge, and see the shape of your thinking. It stayed in the "someday" pile because the implementation cost felt too high for a speculative side project; I wasn't skilled in browser technologies or anything UI-related.

Then came the Christmas break ultralearning exercise I documented in my recent blog post about building with OpenCode and Claude Opus 4.5. Pressure-testing Opus 4.5 made me realize it was finally feasible to spend a day trying to make this work. I pushed Canvas Chat from idea to working prototype in about 24 hours of actual building time, and then another 24 hrs to get it up on Modal and add in many, many refinements, each of which may have taken me multiple weeks. The final result is this:

But before I explain what I built, let me explain why I wanted it in the first place.

The job to be done

Clayton Christensen's Jobs to Be Done framework asks: what job is the customer hiring this product to do? For Canvas Chat, the job isn't "chat with an LLM"—ChatGPT already does that fine. The job is: think through a complex problem where the exploration is nonlinear.

Here's the struggling moment. You're deep in a conversation with Claude or GPT, and you want to try a different framing of your question. But if you do, you'll lose the current thread. Or an LLM gives you a list of ten ideas, one catches your eye, and you want to drill into it—but the conversation keeps scrolling and you lose the overview. Or you've been exploring a problem across three separate chat sessions and now you need to synthesize, but you can't see them together.

Linear chat actively works against this kind of thinking. It forces linear structure onto nonlinear exploration. You end up managing context in your head, copy-pasting between windows, losing track of which threads went where.

Canvas Chat exists to solve that. When your thinking branches in multiple directions, it keeps all the threads visible and connected so you don't lose context and can synthesize across them.

How it works

Canvas Chat is an infinite canvas where conversations are nodes in a directed graph. You type a message, it appears as a node. The LLM's response appears as another node, connected by an edge. So far, standard. But then:

Branch from any node. Click reply on any message, and your new message connects to that point, not the end of the conversation. The response branches off visually. Try two different prompts from the same starting point and see both branches side by side.

Highlight and branch. Select text within a node, and a tooltip appears. Type a follow-up question, and Canvas Chat creates a highlight node (showing the excerpt with a blockquote) plus your question, plus the LLM response. The original node stays intact. This works especially well when an LLM gives a list of ideas and you want to drill into one without losing the overview.

Multi-select for merge context. Cmd-click multiple nodes, then type. The new message connects to all selected nodes, and the LLM sees the full ancestry of every selected node. I use this to synthesize: select two branches that went in different directions, ask "What do these approaches have in common?" The context includes everything that led to both.

Context flows through the graph

When you send a message, Canvas Chat walks the DAG backward from your selected node(s), collecting all ancestors. It sorts them by creation time and sends them to the LLM as conversation history. If you've selected multiple nodes (a merge), the context is the union of all their ancestors, deduplicated.

The practical effect: the LLM always knows how you arrived at the current question, even if the path is nonlinear. Branch from a discussion about protein folding dynamics, ask a follow-up about computational costs, and the context includes the protein folding discussion. No manual copy-paste.

Matrix evaluation

This feature came out of a specific struggling moment: evaluating many options against many criteria and losing track of which combinations I'd thought through.

Select one or more nodes as context, type /matrix <and then put additional instructions you're looking to fill out here>. Canvas Chat parses out the list items and shows a confirmation modal where you can remove items or swap rows/columns. Click create, and a matrix node appears.

Each cell has a "+" button. Click it and the LLM fills that cell, seeing the matrix context you provided, the row item, the column item, and the full DAG history from the source nodes. "Fill All" processes every empty cell sequentially.

Click any filled cell to see the full text. "Pin to Canvas" extracts that evaluation into a standalone node, which you can then branch from. Say you're comparing business ideas against criteria, one cell says "strong market fit with enterprise customers," you want to dig into that—pin and branch.

Web search and deep research

Canvas Chat integrates Exa's APIs for two slash commands:

/search <query> runs a neural search and creates a Search node with the query, plus Reference nodes for each result. Click "Fetch & Summarize" on any reference to grab the full page content and summarize it.

/research <topic> kicks off Exa's Research API, which performs multi-step research with multiple queries. The results stream into a Research node with inline source citations.

If you have nodes selected when you run these commands, Canvas Chat uses an LLM to refine your query using the selected text as context. Highlight "CCNOT gate" and type /search how does this work, and it rewrites the query to "how Toffoli gate CCNOT quantum computing works" before searching.

Local-first and multi-provider

All session data lives in IndexedDB. No server-side storage, no accounts. Export sessions as .canvaschat JSON files. API keys live in localStorage and are sent with each request.

The server is stateless: it proxies LLM calls via LiteLLM and handles the Exa integration, but never stores conversation data. You can deploy it yourself on Modal with a single command.

Canvas Chat dynamically fetches available models from each provider when you enter an API key. OpenAI, Anthropic, Google (Gemini), Groq, GitHub Models, and local Ollama instances (when running on localhost) all work. Switch models mid-conversation to compare outputs.

What building this taught me

This project reinforced something I wrote about in the "I don't code anymore, I build" post: I stayed in product builder brain throughout. I didn't have strong opinions about whether the JavaScript was idiomatic because I don't know what idiomatic JavaScript looks like. I just knew whether the feature worked.

When something broke, I'd describe the symptoms and let Opus 4.5 debug in as much detail as I can manage. When I wanted a new interaction pattern, I'd describe what it should feel like and watch it materialize. The creative work — deciding what nonlinear chat should be — remained human. The mechanical translation got delegated.

Canvas Chat is the kind of project I wouldn't have attempted before because the implementation cost exceeded the payoff. Now it didn't.

Try it

Canvas Chat is open source. Run it locally:

git clone https://github.com/ericmjl/canvas-chat.git
cd canvas-chat
pixi run dev

Add your API keys in settings and go. The deployed version runs on Modal.

If you try it, I want to hear what works and what doesn't! You can get in touch with me via Shortmail, or file an issue on the Github repo.

You Can Just Make Stuff with OpenCode and Claude Opus 4.5

2025-12-28T00:00:00Z

Tommy Tang asked me about my opinions on OpenCode, so here's what I've learned after spending significant time with OpenCode and Claude Opus 4.5.

I don't code anymore, I build

This is the punchline, so let me start with it. I've shifted from writing code to directing its creation. The change happened gradually, then all at once. I used to think about syntax, edge cases, and implementation details. Now I think about what I want to exist, describe it clearly, and watch it materialize.

Genesis 1:3 describes this pattern at a cosmic scale:

"And God said, 'Let there be light,' and there was light."

Working with Claude Opus 4.5 through OpenCode feels like a microcosm of that creative act.

Eric said, "Let there be a feature," and there was the feature, in code.

I'm not claiming divinity here, just noting that the creative pattern of speaking things into existence has become surprisingly literal in my daily work.

The tools: OpenCode and Claude Opus 4.5

Like Theo Brown from t3.gg, I've settled on Claude Opus 4.5 as my primary model for coding tasks. It just knows what to do. I've stopped trying to micro-manage the model's actions because it handles most tasks autonomously and correctly. When I ask for a refactor, it refactors. When I describe a feature, it implements it. The gap between intention and execution has shrunk to almost nothing.

Other models require more hand-holding. Opus 4.5 seems to have internalized enough software engineering patterns that I can trust it to make reasonable architectural decisions without constant course corrections. I can literally ask it to "do the docs, keep things up-to-date, and also give me a document that has an overview of code organization and architecture." It just goes to town autonomously. No step-by-step prompting, no breaking the task into smaller pieces. I describe the outcome I want and it figures out the path.

The tooling layer matters too. OpenCode orchestrates the AI coding in a way that feels natural. The tools it calls are always logical, the reasoning traces are transparent, and the execution flow makes sense. It shows a running list of modified files, giving me context about what's changing without running git status constantly. Context compaction lets me stay in one long-running session without hitting token limits. I've thrown out the old playbook of "switch sessions when you approach the context window." Now I only switch when I want to do something entirely different.

My setup: OpenCode with auto-updating, GitHub Copilot Pro as the LLM provider (routing to Opus 4.5), running inside a tmux session for persistence. Each repo gets an AGENTS.md file where I encode my preferences and patterns - the model's training data for my specific context. Opus 4.5 actually respects what's in there, unlike some other models that seem to ignore custom instructions.

Ten days of deliberate practice

I decided to pressure-test the "I build" claim over the holidays. Ten days, December 19-28, using OpenCode as my primary development interface. The goal: see how much I could actually ship.

The answer surprised me. Across six repositories, I pushed over 150 commits spanning infrastructure work, documentation, greenfield apps, and maintenance. Here's what emerged:

A ski trip coordination website (59 commits). My family was heading to New Hampshire for a week. Normally I'd have used a shared Google Doc for the itinerary. Instead, I built a full website with recipe modals, restaurant links with Apple and Google Maps integration, a photo album with lightbox navigation, automatic thumbnail generation, and a hero video background. I updated it live during the trip - adding photos, adjusting the grocery list, swapping menu items. The implementation cost would have been absurd for a week-long trip before. Now the jazz and snazz was well worth the effort - my family actually enjoyed using it.

A teaching clock app for my kids (2 commits, but a complete app). An analog clock trainer plus a jigsaw puzzle game with difficulty levels and themes. Pure JavaScript and CSS - exactly the kind of project my decade-old "no JavaScript" rule would have blocked. The model wrote it; I directed.

pyjanitor infrastructure (40 commits). Currency symbol support for international formats. Automated patch releases on every merge. Test isolation fixes. And a major expansion of AGENTS.md into what I now think of as the repository's "agent constitution" - a document that tells AI assistants how to work within this specific codebase.

A new conda-forge package for janitor-rs. The model handled the unfamiliar territory of Rust packaging and conda-forge recipe formats. I was the novice here; it was the guide. This role reversal keeps happening - when I set up PostHog analytics or migrated to GA4 on my website, the model walked me through each step, explained what I was doing and why, and waited for confirmation before proceeding. The expert-novice relationship flips depending on who knows more about the task at hand.

A custom tmux status bar with Nord colors, powerline arrows, and smooth color transitions. Pure aesthetic indulgence - the kind of project I'd never have prioritized before because the implementation cost exceeded the payoff. Now it didn't.

Canvas Chat (13 commits in 24 hours). A visual non-linear chat interface - think infinite canvas meets LLM conversation. Resizable nodes, trackpad gestures, streaming responses, web search via Exa, session management. FastAPI backend, vanilla JS frontend. Another "no JavaScript" rule violation, and another project that went from idea to working prototype in a single day.

Smaller fixes across llamabot (better error messages) and my website (PostHog analytics, GA4 migration, blog posts).

The variety matters. This wasn't one type of project where I got lucky. It was infrastructure, documentation, greenfield consumer apps, packaging for an ecosystem I rarely touch, and routine maintenance. The "I build" claim held up across all of them.

From engineer brain to product builder brain

Something shifted in how I think about these projects. Previously, I'd worry about how a thing was built - the engineer brain obsessing over implementation details, code structure, idiomatic patterns. Now I've switched to what was built, why I want it built, and does it get the job done - the product builder's brain.

This is especially true for the ski website and Canvas Chat, both built with web technologies (HTML, JS, CSS) that I'm not deeply familiar with. Ironically, my unfamiliarity frees me from micro-managing the implementation. I don't have strong opinions about whether the JavaScript is idiomatic because I don't know what idiomatic JavaScript looks like. I just know whether the feature works.

But there's a latent risk here. The code might not follow best practices - lots of duplication, poor separation of concerns, missing edge cases. So I fall back on principles I picked up from years of Python: refactoring, documentation, testing. I stay at that level of nudging Opus 4.5 - "look for places to refactor," "document this module," "add tests for this functionality" - but I stay out of the nitty-gritty implementation. The principles transfer even when the language doesn't.

How my review process changed

Here's something I didn't expect: I don't scrutinize the code as tightly as I used to during active development. Instead, I read the reasoning traces first. The model's chain of thought tells me whether my codebase is heading in the right direction. If the reasoning is coherent and addresses the right concerns, the code will reflect what I want. If the reasoning seems confused or takes weird detours, something's wrong and I need to dig deeper.

This inverts the traditional development loop. I used to read code to understand what the computer would do. Now I read reasoning to understand what the model understood and decided. The code review happens afterward, and it's lighter because the reasoning already told me whether we're on track.

When I want to catch issues that slipped through, I start a fresh session. A new context window acts like a fresh pair of eyes - the model hasn't been primed by the conversation that led to the current implementation, so it can spot inconsistencies that were invisible during the creative flow. This parallels the old advice about stepping away from code before reviewing it, except now the "stepping away" happens by instantiating a new session rather than waiting for my own brain to reset.

Unlearning old assumptions

Boris Cherny recently had a Twitter exchange with Andrej Karpathy that resonated with me. Boris observed that newer coworkers and even new grads who don't make assumptions about what the model can and can't do are often able to use it most effectively. They don't carry "legacy memories formed when using old models." Every month or two, models get better, and those of us who've been using them longest have to actively unlearn outdated limitations.

I've caught myself doing this repeatedly. Back in grad school around 2015, I tried building d3.js visualizations and struggled to adjust to JavaScript's syntax coming from Python. I decided to focus on getting better at Python first and gave myself a "no JavaScript" rule wherever possible. That constraint made sense at the time. It makes no sense now. The model writes JavaScript just fine. My decade-old "no JavaScript" policy was a legacy memory holding me back from building things that would actually benefit from running in the browser.

The mental work of re-adjusting expectations is real. I have to keep asking myself: would I have avoided this six months ago because the model couldn't handle it, or because I assumed it couldn't? The answer is increasingly the latter.

There's a flip side to this unlearning, though. Working in JavaScript land forced me to learn the language of the web to achieve the same precision and fluency I have with Python. I found myself picking up patterns I'd avoided for years: the browser console for debugging, DOM element manipulation, CSS transitions I didn't know existed, the JS package ecosystem. The model writes the code, but I still need enough vocabulary to direct it well and recognize when something's off. Unlearning old constraints doesn't mean staying ignorant of new territory - it means finally having a reason to explore it.

What this means

The shift from "I code" to "I build" isn't just semantic. It reflects a genuine change in what I spend my attention on. Less time on syntax and implementation details. More time on architecture, requirements, and verification. The creative work remains human. The mechanical translation has been delegated.

I'm still learning how to use this effectively. But the trajectory is clear: the gap between imagining software and having software continues to shrink.

How I Themed My tmux with OpenCode + Claude (And When to Switch Models)

2025-12-27T00:00:00Z

I had a beautiful tmux status bar on my old laptop. Nord colors, powerline arrows, clean and minimal. The kind that makes you feel like a proper terminal power user.

When I got a new machine back in April, I was too lazy to set up tmux properly. The sensible thing would have been to spend five minutes copying over my old config. Instead, eight months later, I finally spent an hour pair-programming the whole thing from scratch with OpenCode and Claude.

Why? Honestly, I wanted to try out a new tool. The irony isn't lost on me.

The Setup

OpenCode is a CLI tool that lets you interact with Claude directly from your terminal. Perfect for this kind of task: I'm already in the terminal configuring tmux, so having my AI pair programmer right there keeps the feedback loop tight. Describe what I want. See the change. Describe what's wrong, with precision. Iterate. No context switching to a browser.

That tight loop is what let me stay in the creative headspace. I could say things like "I want the arrows to overlap like in this screenshot" or "the colors feel too muted, try the frost blue from Nord" without knowing the exact syntax. Claude translated my aesthetic intent into working config.

The other superpower: model switching. OpenCode lets you flip between any models you have API keys for. For this session, I toggled between Claude Sonnet (fast, good for quick iterations) and Claude Opus (slower, but sharper for complex debugging). This turned out to be crucial.

Starting with Research

First, I asked Sonnet to search online for tmux status bar customization. It pulled resources from the official tmux wiki and various tutorials, giving me a foundation: status-left, status-right, window-status-format, color options, the basics.

Armed with that, we dove in.

First attempt with a custom theme

Claude created a custom dark theme inspired by Catppuccin colors. Worked immediately:

set -g status-style "bg=#1e1e2e,fg=#cdd6f4"
set -g status-left "#[fg=#89b4fa,bold] #S #[fg=#a6e3a1]@ #H"
set -g status-right "#[fg=#f9e2af]%a %b %d #[fg=#89b4fa]%H:%M"

Clean. Functional. Pretty. But I wanted more: those beautiful powerline arrows flowing between segments. That's when things got interesting.

The Powerline Saga

Claude suggested powerline-go, a Go-based powerline prompt generator. We installed it via Homebrew (not pip, since I keep my system Python-free):

brew install powerline-go

Updated the tmux config to call powerline-go for the status bar. Reloaded. And... disaster. Instead of beautiful arrows, raw escape codes:

[38;5;15m[48;5;4m ericmjl [38;5;4m[48;5;0m...

The terminal was spitting out ANSI codes instead of interpreting them. We tried various fixes, but powerline-go simply wasn't designed for tmux status bars; it's meant for shell prompts. Back to square one.

Trying the tmux-powerline Plugin

Next attempt: the actual tmux-powerline plugin via TPM (Tmux Plugin Manager):

git clone https://github.com/tmux-plugins/tpm ~/.tmux/plugins/tpm

Added the plugin, pressed C-b I to install, and... the status bar exploded with information. IP addresses, weather, load averages, hostname. Way too much. I asked Claude to simplify and switch to Nord colors.

We created a custom theme at ~/.config/tmux-powerline/themes/nord.sh, updated the config, reloaded tmux. Nothing changed. The theme wasn't loading. Killed the server entirely. Restarted. Still the old crowded theme.

This is where Sonnet started struggling. Same fixes over and over: reload the config, check the theme path, restart tmux. Loop after loop of suggestions that weren't working.

The model switch from Sonnet to Opus

I noticed Sonnet spinning its wheels. Same suggestions, same non-results. Time to switch.

The difference was immediate. Instead of repeating failed approaches, Opus stepped back and proposed something different entirely: ditch the plugin and go native. Tmux's built-in formatting is powerful enough to create powerline-style status bars without any plugins. We just needed the right Unicode characters and color transitions.

This stuck with me: Sonnet is fantastic for speed and quick iterations, but when you're stuck in a loop, Opus brings the lateral thinking to break out.

Going native as the winning approach

Fresh start. Clean native tmux config. The key insight was understanding how powerline arrows actually work: the arrow character's foreground color matches the background of the segment it's coming from, and its background matches what it's going into.

Here's the final status-left (session name with powerline arrow):

set -g status-left "#[fg=#2e3440,bg=#5e81ac,bold]  #S #[fg=#5e81ac,bg=#2e3440]\ue0b0"

The window formats, with arrows on both sides so they flow into neighboring elements:

# Inactive windows
setw -g window-status-format "#[fg=#2e3440,bg=#3b4252]\ue0b0#[fg=#d8dee9,bg=#3b4252] #I #W #[fg=#3b4252,bg=#2e3440]\ue0b0"

# Active window (cyan highlight)
setw -g window-status-current-format "#[fg=#2e3440,bg=#88c0d0]\ue0b0#[fg=#2e3440,bg=#88c0d0,bold] #I #W #[fg=#88c0d0,bg=#2e3440]\ue0b0"

And the right side (battery, date, time) using left-pointing arrows and a smooth Nord color gradient:

set -g status-right "#[fg=#a3be8c,bg=#2e3440]\ue0b2#[fg=#2e3440,bg=#a3be8c,bold] 󰁹 #(pmset -g batt | grep -o '[0-9]*%%' | head -1) #[fg=#5e81ac,bg=#a3be8c]\ue0b2#[fg=#d8dee9,bg=#5e81ac] %b %d #[fg=#88c0d0,bg=#5e81ac]\ue0b2#[fg=#2e3440,bg=#88c0d0,bold] %H:%M "

The Final Result

After all that iteration, here's what my tmux status bar looks like:

system-config
1 opencode
🔋 100%
Dec 23
06:05

Session name in frost blue on the left. Active window in cyan. Right side flows through battery (green), date (blue), and time (cyan). All connected by powerline arrows with smooth color transitions. (I asked Claude to recreate the status bar in HTML so I wouldn't have to screenshot it for the blog.)

What I Took Away

There's a growing conversation about AI-assisted programming: the tight feedback loops, model selection strategies, iterative workflows. I've written about some of these patterns myself. But this session crystallized something different.

I can express my creativity on a computer screen more easily than ever before.

I'm not a designer. CSS is foreign to me, hex color codes don't stick in my head, and tmux's formatting syntax is arcane. But I have taste. I know what looks good. Years of admiring beautiful terminals gave me a mental mood board. What I lacked was the technical fluency to make it real.

AI bridged that gap. Throughout this session I worked like a designer: describing aesthetics, pointing at visual problems, directing iteration. "The arrows should overlap." "That cyan is too bright." "Make the battery segment green." Claude handled implementation. I stayed in the creative headspace.

Iteration surfaces what you actually want.

This surprised me. I didn't start with a complete vision, just a vague sense of "Nord colors, powerline arrows, clean and minimal." But each rapid cycle surfaced preferences I didn't know I had. The arrows need to overlap. The active window should pop more. The right side needs a color gradient. None of these were requirements I could have articulated upfront. They emerged through seeing and reacting.

Bits and bytes have never been cheaper to produce. AI can generate config files, CSS, code, whatever. But aesthetics and judgment? Those remain expensive. The scarce resource isn't the implementation anymore. It's knowing what you want and recognizing when you've found it.

AI doesn't replace that judgment. It amplifies it by removing the implementation friction that used to slow the creative loop down.

The whole session took about an hour, failed attempts included. Without AI pair programming, I'd probably still be reading documentation. Instead, I have a beautiful terminal, and a new appreciation for what becomes possible when the gap between creative vision and technical implementation shrinks to nearly nothing.

Two years of weekly blogging and what 2025 taught me

2025-12-25T00:00:00Z

Last year, I challenged myself to write one blog post per week, and I hit 53 posts by the end of 2024. This year, I doubled down on that commitment and wrote 50 posts in 2025. Including this one, it's 51, bringing me to 104 blog posts over two years.

The year of coding agents

Looking at my 2025 posts, one theme dominates: coding agents. I wrote extensively about how to work with AI coding assistants, from teaching them with AGENTS.md files to letting them work autonomously. This reflected a shift in how I work day-to-day.

Some highlights from this theme:

The shift from "AI as a tool" to "AI as a collaborator" captures how my practice evolved this year. I've gone from cautiously experimenting with Cursor to having established patterns for multi-repository agent workflows.

Bayesian methods and biological applications

My work continued to inform my writing, with several posts on applying Bayesian statistics to real lab problems. The R2D2 prior posts were particularly satisfying to write because I felt equipped with new theoretical knowledge that was directly applicable, and I appreciated the mathematical aesthetics behind the approach:

I also explored the challenges of working with lab data, including why preclinical experiments make ML challenging and how to communicate effectively with lab scientists.

Tools I got excited about

Every year brings new tools that change how I work. In 2025, two stood out.

Marimo is a reactive notebook tool that I wrote about with enthusiasm, and followed up with practical guidance on using coding agents to write Marimo notebooks. The reactive execution model aligns well with how I think about data exploration.

Modal is cloud computing that actually feels Pythonic. My "Wow, Modal!" post captured the delight of finding infrastructure that doesn't fight against my workflow.

Data science leadership and career

I continued writing about the human side of data science work, including standardizing ways of working, communicating with lab scientists, and navigating the biotech industry's ups and downs. The year ended with The selfish reason to do your best work, which synthesized lessons from a challenging year in biotech.

Looking ahead to 2026

After two years of writing almost weekly on whatever is on my mind, I am adjusting my goals. Next year, my attention shifts towards (a) learning the fundamentals of quantum computing through an ultralearning project, (b) writing more on data science leadership and career development to encourage colleagues navigating similar paths, and (c) building out at least 10 experimental things with AI. I am also dropping the goal of "one blog post per week" to four per month, which brings me to a goal of 48 for 2026. I am giving myself space to rest and strategically plan out writing going into 2026.

Merry Christmas and a happy new year to all my readers!

Blog posts by theme

Biology & Chemistry

Career Advice

Data Science Practice & Leadership

Data Science Tooling

LLMs

All blog posts

Date	Title	Categories
2025-01-04	What makes an agent?	LLMs, Data Science Tooling, Biology & Chemistry
2025-01-10	A practical guide to securing secrets in data science projects	Data Science Practice & Leadership, Data Science Tooling
2025-01-13	Writing at the speed of thought	Data Science Tooling, Career Advice
2025-01-19	Why data from preclinical biotech lab experiments make machine learning challenging	Data Science Practice & Leadership, Data Science Tooling, Biology & Chemistry
2025-01-31	PyData Boston/Cambridge Talk @ Moderna: What makes an agent?	LLMs, Data Science Tooling, Data Science Practice & Leadership
2025-02-07	Lightening the LlamaBot	LLMs, Data Science Tooling
2025-02-17	Let me ship you the Python you need	Data Science Tooling
2025-02-23	Reliable biological data requires physical quantities, not statistical artifacts	Data Science Practice & Leadership, Biology & Chemistry, Data Science Tooling
2025-03-01	How to fix PyPI upload errors related to license metadata	Data Science Tooling
2025-03-06	A blueprint for data-driven molecule engineering	Data Science Practice & Leadership, Data Science Tooling, Biology & Chemistry
2025-03-16	The art of finesse as a data scientist	Data Science Practice & Leadership
2025-03-17	Why you should take part in the SciPy sprints!	Data Science Practice & Leadership, Data Science Tooling, Career Advice
2025-04-02	How to standardize Data Science ways of working to unlock your team's creativity	Data Science Practice & Leadership, Data Science Tooling
2025-04-03	Bayesian Superiority Estimation with R2D2 Priors: A Practical Guide for Protein Screening	Data Science Practice & Leadership, Data Science Tooling, Biology & Chemistry
2025-04-05	From data chaos to statistical clarity: A laboratory transformation story	Data Science Practice & Leadership, Data Science Tooling, Biology & Chemistry
2025-04-08	Wow, Marimo!	Data Science Tooling, LLMs
2025-04-19	Good practices for AI-assisted development from a live protein calculator demo	Data Science Practice & Leadership, Data Science Tooling, Biology & Chemistry
2025-04-26	Wow, Modal!	Data Science Tooling, LLMs
2025-05-08	Why I'm excited for SciPy 2025!	LLMs, Data Science Practice & Leadership, Data Science Tooling, Career Advice
2025-05-24	Supercharge your coding agents with VSCode workspaces	LLMs, Data Science Tooling
2025-05-25	The invisible polish of automatic model routing	LLMs, Data Science Tooling
2025-06-07	Principles for using AI autodidactically	LLMs, Data Science Practice & Leadership
2025-06-14	Rethinking LLM interfaces, from chatbots to contextual applications	LLMs, Data Science Tooling
2025-06-27	Build your own tools!	LLMs, Data Science Practice & Leadership, Data Science Tooling, Biology & Chemistry
2025-07-01	One hour and eight minutes: Building a receipt scanner with the weirdest tech stack imaginable	LLMs, Data Science Tooling
2025-07-07	The job your docs need to do	Data Science Practice & Leadership
2025-07-13	Earn the privilege to use automation	LLMs, Data Science Practice & Leadership, Data Science Tooling
2025-07-14	Reflections on the SciPy 2025 Conference	LLMs, Data Science Practice & Leadership, Data Science Tooling, Biology & Chemistry, Career Advice
2025-07-15	How to use xarray for unified laboratory data storage	Data Science Tooling, Biology & Chemistry
2025-07-21	From nerd-sniped to shipped using AI as a thinking tool	LLMs, Data Science Practice & Leadership, Data Science Tooling
2025-08-06	Stop guessing at priors: R2D2's automated approach to Bayesian modeling	Data Science Practice & Leadership, Data Science Tooling
2025-08-15	Data scientists aren't becoming obsolete in the LLM era	LLMs, Data Science Practice & Leadership, Data Science Tooling, Career Advice
2025-08-23	Wicked Python trickery - dynamically patch a Python function's source code at runtime	LLMs, Data Science Tooling
2025-08-24	How to communicate with lab scientists (when you're the data person)	Data Science Practice & Leadership, Biology & Chemistry
2025-09-01	How to use AI to accelerate your career in 2025	LLMs, Career Advice
2025-09-02	The Data Science Bootstrap Notes: A major upgrade for 2025	Data Science Practice & Leadership, Data Science Tooling
2025-10-01	How data scientists can master life sciences and software skills for biotech using ultralearning	Data Science Practice & Leadership, Biology & Chemistry, Career Advice
2025-10-04	How to teach your coding agent with AGENTS.md	LLMs, Data Science Tooling
2025-10-10	How to use multiple GitHub accounts on the same computer	Data Science Tooling
2025-10-14	How to Use Coding Agents Effectively	LLMs, Data Science Tooling
2025-10-18	A practical comparison of DSPy and LlamaBot for structured LLM applications	LLMs, Data Science Tooling
2025-10-19	How to expose any documentation to any LLM agent	LLMs, Data Science Tooling, Data Science Practice & Leadership
2025-10-20	Exploring Skills vs MCP Servers	LLMs, Data Science Tooling
2025-10-28	Use coding agents to write Marimo notebooks	LLMs, Data Science Tooling
2025-11-08	Safe ways to let your coding agent work autonomously	LLMs, Data Science Tooling
2025-11-16	How I Replaced 307 Lines of Agent Code with 4 Lines	LLMs, Data Science Tooling
2025-11-17	How to Reference Code Across Repositories with Coding Agents	Data Science Tooling, LLMs
2025-12-02	What does it take to build a statistics agent?	LLMs, Data Science Tooling, Biology & Chemistry, Data Science Practice & Leadership
2025-12-10	Productive Patterns for Agent-Assisted Programming	LLMs, Data Science Tooling
2025-12-17	The selfish reason to do your best work	Career Advice, Data Science Practice & Leadership

The selfish reason to do your best work

2025-12-17T00:00:00Z

I’ve been thinking a lot about career lately. This has been a pretty lean year for biotech; we've seen ups and downs at Moderna and across the industry. So, I want to offer a word of encouragement and a philosophy on work that I hope can be useful for you, regardless of where you are in your journey.

It starts with a reframing of why we work.

Do your best work for yourself

I know there is a lot of sentiment going around the internet right now about "acting your wage"—limiting your effort to exactly what you are paid for—or doing the bare minimum. The logic goes: Why should I care about doing my best for my job if my company doesn't care for me?

I get where that sentiment comes from. But I want to redirect your attention a little bit.

You do not have to do your best work for your company. You should do your best work for yourself at the company you’re at.

It just so happens that the company will benefit, but you should treat your effort as an investment in your own professional instincts and habits. Sooner or later, you may not be at that company. You actually don't really have to care about the entity itself. If you don't care about the people who work at your workplace, no one is compelling you to. (Though, if you do happen to like your colleagues—which is true for me where I work—then that’s all good.)

But even if you can’t find much to be inspired by, do your best work anyway. Why? Because you are building the person you will be in five or ten years.

President Obama once gave this advice to young interns: "Don't ask for the plum assignments. Just knock out everything you're doing." I guarantee you someone will notice. Even if no one at your current company notices, if you build a track record of quality, people outside will notice.

Your reputation precedes you. It is the one thing you accumulate over time that serves as a form of wealth that can never be taken away from you. Only you can lose it. To borrow a phrase from Jocko Willink, this is "Extreme Ownership"—taking total responsibility for your world. Yes, circumstances happen, but if you guard your reputation well, it is yours to keep.

Think about it: Who knows where you will be ten years down the road? If you are a software engineer or data scientist now, in five years you might be a Director. You’re going to be calling the shots. If you don't take the time now to practice making decisions, witnessing judgment calls, and battle-testing your engineering foresight, will you be ready?

I had a former teammate who worked under me at Moderna, Arkadij Kummer. He’s now the CTO of a startup—a title I haven't even held. I saw him put in the effort to develop the strategic thinking patterns that helped him get the skills he needed to lead a tech organization. He sought out opportunities to practice making judgment calls and owning the outcomes. You have to get those reps in early, or you won’t be ready for what happens later.

So, if you are working at a company you want to leave: do not give up on investing in yourself. Fortune favors the prepared.

Resilience is also an investment

Building this "career wealth" isn't just about technical execution. It's also about how you handle failure. And here is the second word of encouragement I want to offer: Everyone will make a blunder at some point.

Part of growing up—and part of investing in your own character—is owning up to those mistakes, being proactive about remedying them, and graciously accepting help.

I have been there. I once wrote a very scathing internal blog post about my leadership at a previous company. Looking back, I sometimes think of it as the darkest weekend of my career.

I was a guy who had just finished school, entered the workforce, and within three years decided I knew better than people who had done their jobs for twenty-odd years. I'm not saying it's impossible that I was right, but it was pretty presumptuous. I lashed out at other groups that I thought were incompetent, effectively attacking my own team.

The leadership team responded with extreme grace. They knew the problems I outlined were real, but they also saw a junior person who hadn't picked his battles wisely.

We have limited energy and limited ability to focus. There are only so many battles we can handle simultaneously. I chose a bad battle to fight.

That experience prompted deep reflection. I decided to lean back into that first philosophy: I was going to go back and do a good job. Not necessarily because I was feeling proud of the company at that moment, but because I recognized that if I ever wanted to be the type of person with the authority to change things, I needed to be the best version of myself first. I needed to learn how to handle authority and how to elevate the people around me.

If you are in a situation where you’ve made a mistake, the best thing you can do for your reputation is to own it. Propose an action to rectify it. Move on.

Most mistakes are not unforgivable. There is a classic business story about Tom Watson, the founder of IBM, involving a subordinate who made a mistake that cost the company \$600,000. Whether the amount was really \$600,000 or not, the story has a lesson that rings true. The man walked into Watson's office expecting to be fired. Watson reportedly replied, "No, I just spent \$600,000 training you. Why would I want to fire you?"

Most people will understand. Yes, there will be delays. That is just the world we inhabit. Own the mistake, improve the process, and keep making it better. Again, not primarily for the company, but because you are preparing yourself to lead with integrity and compassion.

The wealth that remains

I don't want you to become the whiner or the complainer. That is a habit that will stick with you for the rest of your life.

Instead, I want you to play the long game. Don't let temporary frustrations dictate your long-term growth. Anything worth doing is going to be difficult. If it was easy, the reward would be fleeting. Even for the ultra-rich, like Sergey Brin, eventually the yacht gets boring. He returned to active coding at Google to work on AI because, fundamentally, humans are wired to find satisfaction in building something meaningful.

So, aspire to greatness. Not just to gain a title or a promotion, but because in the process, you accumulate a wealth that cannot be taken away from you: You will have the satisfaction of mastery. You will have a battle-tested character. You will have a reputation that opens doors before you even knock.

Do your best work. It’s the best investment you’ll ever make. And it'll be the most selfless gift you give to yourself and others.

Productive Patterns for Agent-Assisted Programming

2025-12-10T00:00:00Z

I've been using coding agents for a while now, and I've learned a few patterns that make the experience much more productive. The thing is, a lot of these "productive patterns" aren't being shared enough—they're more like folk knowledge that you can only really pick up by watching someone else do their work live. I decided to write this blog post to kickstart conversations about the matter. Here's what works for me.

Build a detailed plan with AI

Before jumping into implementation, spend time building a detailed plan with your AI assistant. Iterate 2-3 times over the plan, checking every detail. You want the ability to see in your head what the code might look like—just a "fat finger sketch" of the implementation.

The plan should include:

Details on implementation
How to test (this is the most important part)
Documentation plan

Do docs and tests first

Humans usually adhere to test-driven development if you're a software engineer, or exploration-driven software builds if you're more of a data scientist. Because of the sequential nature of generative AI, it's advantageous to instruct AI to do the docs and tests first before the implementation. This is a complex conditional probability problem. If the tests and docs are written first, the implementation has to satisfy those constraints, which leads to better code.

The test plan should include instructions on how to run tests using command line tools. Don't assume the AI knows your project's specific testing setup.

Use AGENTS.md as your repo's AI university

AGENTS.md is a great place to store the specific instructions that you need for the repo. For example, AI will tend to write python -m ... as a shell command, but if I'm running a pixi project, it's better to always run pixi run python ... instead. Treat AGENTS.md as the AI's university of your particular repo; it's where you encode all the project-specific knowledge that the AI needs to work effectively.

Control the pace of the agent

Know the default behavior of your agent; it may be over-eager to do lots of things. You can pace the coding agent by asking it to "slow down, walk me through the changes one at a time, starting with the most important ones first." This helps you maintain control and review changes as they happen, rather than being overwhelmed by a massive diff.

Leverage local and command line tools

You can use local and command line tools to your advantage! Here are some examples:

Firstly, the GitHub CLI (gh) can be used to:

Store plans on GitHub as issues first (a matter of taste—you can avoid cluttering up your local filesystem)
Pull GitHub Actions logs
Call out to the GitHub API for other general tasks

Environment management:

pixi run ensures you're always running within the correct Python environment
uvx marimo check lets me check that marimo notebooks are syntactically valid
uv run notebook.py lets me run notebooks as scripts to check outputs
uvx marimo export lets me export marimo notebooks as markdown

Linting and quality:

markdownlint runs on every edit of markdown files so you never have markdown linting issues
Get AI to "commit relevant files and fix issues raised by pre-commit hooks"

Let agents use CLI tools and read outputs directly so that you don't have to switch between windows copying and pasting things manually.

Let agents write temporary tools

Coding agents can write their own temporary tools inside .py files. Encourage coding agents to do that to test that what it wrote works on-the-fly. This is a great way to validate code before integrating it into your main codebase.

You can even experiment with self-improving agents: if it detects you correcting its action, it should auto-update AGENTS.md with what is the correct thing to do. I haven't fully battle-tested this yet, but you can write an "AI constitution" at the top of AGENTS.md that instructs the agent to learn from corrections by remembering them inside AGENTS.md.

Develop your own tools

Isabel Zimmerman mentioned this in her keynote talk: develop your own tools. Here are some examples of my own:

Personal MCP productivity server: gives me prompts that I can take from project to project, so I don't have to keep copying/pasting them
Shell aliases: gacp lets me run git add . && git commit && git push
LlamaBot git hooks: auto-writes commit messages for me

These custom tools compound over time and make your workflow significantly more efficient.

These patterns have made my agent-assisted programming much more productive. Treat the AI as a collaborator that needs clear instructions, proper context, and the right tools to work effectively. Start with a good plan, control the pace, and build tools that make the whole process smoother.

What patterns have you discovered? I'd love to hear what works for you—let's make this folk knowledge more accessible to everyone.

What does it take to build a statistics agent?

2025-12-02T00:00:00Z

Within research organizations at most pharma and biotech companies, professionally-trained statisticians are often staffed at extremely low ratios relative to the number of lab scientists. By rough Fermi estimation, I'd hazard a guess that ratios anywhere from 1:10 to 1:100 are plausible, meaning most researchers have limited access to statistical expertise when they need it most, during experiment design. This statistician shortage creates a critical bottleneck in experimental design, power calculations, and biostatistical consultation—areas where proper statistical guidance can prevent costly mistakes and improve research reproducibility.

This creates a costly problem. When statisticians aren't available, researchers fall back to what I call "folk statistics" - the kind you learn by immersion in a lab, or from 1-2 graduate lectures hidden within broader "laboratory methods" or "computational methods" classes. I know this because I practiced folk statistics myself in the life sciences, blindly following rules like "just do n=3" or "just use the t-test with your count data" without understanding the statistical reasoning behind these choices.

The consequences are documented in stark numbers. Amgen scientists attempted to reproduce 53 landmark preclinical papers and failed in 47 cases (89%)—even after contacting original authors and exchanging reagents. Bayer's internal validation found only 20-25% of studies "completely in line" with original publications. These studies consistently identified poor experimental design and inadequate statistical analysis as major contributors. Freedman et al. (2015) estimated $28 billion annually spent on irreproducible preclinical research in the United States alone.

Breakdown of causes of preclinical irreproducibility from Freedman et al. (2015). Study design accounts for 27.6% of irreproducibility.

At the individual experiment level, this translates to teams throwing out hard-won experimental data that can cost anywhere from thousands to hundreds of thousands of dollars to collect per round, wasting up to millions of dollars downstream by basing decisions on poorly-collected data, and missing opportunities to set up machine learners with high quality laboratory data that could shortcut the amount of laboratory experimentation needed.

I took one semester of graduate-level biostatistics, then a decade of self-study in Bayesian statistics, followed by professional work where accurate estimation was critical—whether estimating half-life of a molecule, binding affinity of an antibody, or other performance properties. Through this journey, I no longer trust folk statistics. Folk statistics relies on faulty assumptions—like "n=3 is all you'll really need," "use the t-test for count data," or "calculate the SEM and don't show the SD"—which influence bad decision-making when people don't know better. Once you see how these assumptions break down and lead to wrong conclusions, you can't unsee it. Quantities like half-life, binding affinity, and other performance properties need to be accurately estimated through proper experimental design and statistically-informed mechanistic modeling.

Statisticians are expensive, but they're also 100% critical for generating high quality, high fidelity data. Their role at the experiment design phase is usually that of a consultant, asking probing questions to ensure experiments are designed with good controls, confounders are accounted for, and the right statistical models are chosen. The question is: can we scale this expertise?

Not to replace statisticians, but to level up the organizational statistical practice before researchers check in with a professionally-trained stats person. If lab scientists can think through their experimental designs more rigorously beforehand - understanding power calculations, considering confounders, planning proper controls - then the conversations they have with statisticians can be elevated. Instead of starting from scratch, they can engage in more sophisticated discussions about design trade-offs, model selection, and advanced statistical considerations. In turn, this amplifies the value of the statistician's time and improves outcomes for everyone.

I was inspired by Dr. Emi Tanaka's slides on extracting elements of statistical experiment design using LLMs, which showed how we can extract structured information like response variables, treatments, experimental units, design types, replicate structure, and controls. I decided to take a stab at building something that could do more than just extract information—something that could actually consult on experiment design.

And so stats-agents was born: an AI-powered statistics agent for experiment design consultation. Here's how I designed and evaluated this domain-specific AI agent.

As a preface, I initially explored the ReAct pattern but switched to PocketFlow, a minimalist graph-based framework that replaced 307 lines of agent orchestration code with just 4 lines. This graph-based approach brought clarity, modularity, and made the execution flow explicit—exactly what I needed for building a robust statistics agent.

So how did I go about building this agent?

Deeply influenced by Clayton Christensen's books, I actually started with "what's the job for this agent to be done?" I initially considered building a single agent that could handle both experiment design consultation and statistical analysis of collected data. However, I quickly realized these are fundamentally different phases with different goals, tools, and interaction patterns.

The experiment design phase is consultative and exploratory - it's about asking questions, understanding constraints, identifying potential issues, and helping researchers think through their design before data collection. The analysis phase is more technical - it's about taking collected data and building statistical models to estimate quantities of interest.

I decided to focus the agent on the design phase only. This separation of concerns made the agent cleaner, less confusing, and allowed it to be optimized for its specific purpose: being an inquisitive, consultative partner during experiment design. The analysis phase would be handled separately (or by a different agent) with its own tools and prompting strategies.

So I defined the agent's job description (JD) as: "an agent that will provide critique on experiment designs, suggest modifications, and help researchers think through their experimental design before data collection". It sounded oddly like a human's job description for a real job, except more specific. Notice, however, that the JD leaves room for a real human, in that no accountability for outcomes is placed on the agent, a human statistician still needs to review the work, just as we wrote above.

With the job scope defined, I turned to designing the tools the agent would need.

The first tool I gave was critique_experiment_design - a tool that provides comprehensive critique of experiment designs, identifying potential flaws, biases, weaknesses, and areas for improvement. This tool considers multiple angles including biological, statistical, and practical constraints. The agent can use this to help researchers identify issues in their designs before they collect data.

The second tool I gave was one I previously wrote about: the ability to execute code (write_and_execute_code). I wanted this available to answer questions like "what should my data table look like?"

I've noticed that having a sample data table in front of us when discussing experimental designs is incredibly clarifying—it cuts through abstract confusion toward concrete understanding. This tool enables the agent to generate sample data tables, perform power calculations, create plate map visualizations, and other dynamic analyses.

Important security note: The ability to execute arbitrary Python code is powerful but also dangerous. An agent that can execute code can delete files, modify system configurations, access sensitive data, make network requests, and more. For any production deployment, this agent must run in a containerized environment with strict isolation, resource limits, and no access to secrets or credentials. This isn't optional - it's a cybersecurity requirement! For my development and testing, I ran it in a controlled environment on my own machine, but production deployment would require proper containerization.

With the tools defined, the next challenge was evaluation: how do you know if the agent is actually working? This turned out to be more complex than I initially expected.

The evaluation process had two distinct phases: an exploration-guided MVP phase (vibes-driven) and a post-MVP systematic testing phase.

Phase 1: MVP Development (Vibes-Driven)

During the MVP phase, I defined a fixed conversation path and repeatedly tested it manually in a Marimo notebook's chat UI. I tested with a sequence like: asking for design critique, providing an experiment description, requesting power calculations, and asking for sample data tables. As I found errors, I fixed them immediately, contrary to evaluation best practices, but appropriate for this exploratory phase. The tool definitions weren't settled until this phase was complete.

I also used Cursor (a coding agent) to help diagnose issues, explore multiple solutions, and get different perspectives before committing to fixes. This "multiple AI opinions before committing" pattern follows a similar philosophy to Geoffrey Litt's "Code like a surgeon" approach: spike out an attempt at a big change, review it as a sketch of where to go, and often you won't use the result directly—but it helps you understand the problem space better.

Rather than accepting the first AI suggestion, I'd ask multiple questions, explore several solution approaches, and understand the trade-offs before making an informed decision. When debugging complex issues like the closure vs. shared state problem, I'd often ask multiple models the same question to see if they'd converge on the same diagnosis—if different LLMs independently arrived at the same answer, that was a good sign the solution was on the right track. This led to better architecture decisions and fewer instances of "I wish I had done it differently."

Phase 2: Systematic Evaluation

Once the baseline behavior was satisfactory, I moved to systematic evaluation.

Benchmark prompt: I created a "perfect prompt" document (experiment_design_for_power_calc.md) that served as my regression test suite. This complete, detailed experiment design specification should trigger specific agent behaviors, such as asking the right questions, performing power calculations correctly, providing contextual explanations. Every time I modified the system prompt or tools, I'd run this same benchmark and check: did it still work? Without this, I found myself making changes that broke things in subtle ways, or losing track of what "good" behavior even looked like. The benchmark prompt became my north star, a concrete example of the agent working as intended.

Synthetic test generation: Once I had a reliable benchmark, I expanded to systematic evaluation with variation. Starting with five examples from Tanaka's slide deck, I used the methodology from Shreya Shankar and Hamel Husain (researchers who developed systematic approaches for LLM evaluation) to generate synthetic chat examples by selecting 1-3 axes of variation. I chose experimental domain (biotech vs. agriculture) as the primary axis, while varying the statistical expertise level of the simulated user. This process generated dozens of conversation traces that, while not exhaustive, represented draws from my constrained prior belief about likely conversations.

Systematic evaluation with these varied conversation traces revealed patterns I never would have noticed through manual testing. Through these traces, I identified three major categories of failure modes that needed to be addressed.

Failure mode 1: Multi-step execution breakdown

The first major issue I encountered was that the agent couldn't chain tool calls effectively. When the agent executed code to perform a power calculation, it would store the result in a variable like mtt_power_analysis_result. But when it tried to analyze that result in a subsequent tool call, it would fail with a NameError - the variable simply wasn't accessible.

The root cause was subtle: the code execution tool (write_and_execute_code_wrapper) was using a closure variable that captured the notebook's globals at initialization time. However, the agent framework (AgentBot) stores results in a separate shared dictionary (shared["globals_dict"]). These two dictionaries were disconnected; think of them as two separate notebooks that couldn't see each other's variables. So when the agent created a variable in one tool call, it wasn't visible to the next.

The fix required connecting them: I modified the code execution tool to accept an optional _globals_dict parameter. When provided, it uses the agent's shared dictionary instead of its own isolated one. This allows results from one tool call to be accessible in subsequent calls, enabling true multi-step workflows where the agent can build on previous results.

Failure mode 2: Display formatting and contextual output

The second category of issues involved both technical display problems and behavioral output quality. When the agent returned Python objects (DataFrames, matplotlib figures, etc.), they weren't displaying properly in the Marimo chat interface. The agent would return a dictionary with these objects, but Marimo's chat UI doesn't automatically render matplotlib Figure objects embedded in dictionaries.

But there was a deeper behavioral problem: the agent was dumping DataFrames and plots without any explanatory text. I'd ask for a power analysis, and the agent would return a raw DataFrame with numbers -- no context, no interpretation, no explanation of what I was looking at. My sense of taste rebelled. This wasn't just a technical problem, it was an aesthetic one. I wanted a polished, consultant-like experience where explanatory text naturally flows between objects, not a data dump.

Sometimes the best technical solutions come from caring about how things look and feel.

I solved both problems through a combination of technical infrastructure and prompt engineering. With AI assistance, I created a formatter system that processes dictionaries containing text and objects, converting strings to markdown and passing objects through for native display. This handled the technical display issue. It's implemented within llamabot/components/formatters.py as create_marimo_formatter.

But to solve the behavioral problem—getting the agent to actually provide contextual text—I had to update the system prompt. I added explicit guidance requiring that every object be preceded (and ideally followed) by explanatory text that connects back to the researcher's goals. The prompt now instructs the agent to create a dictionary with explanatory text strings interleaved with objects, then return it. The formatter processes this dictionary, creating that polished, consultant-like experience where text naturally flows between objects.

Failure mode 3: Agent decision-making and domain knowledge

The third category of failures involved the agent's decision-making process and understanding of domain conventions. Through systematic evaluation, I discovered several patterns.

Here's a concrete example of how the prompt evolved. Initially, I had a simple instruction: "Ask clarifying questions about experiment goals, constraints, and assumptions." But the agent kept jumping straight to calculations. So I added:

Before (early version):

Ask clarifying questions about experiment goals, constraints, and assumptions.

After (refined version):

CRITICAL - BE INQUISITIVE FIRST: Before jumping into calculations, you MUST ask probing questions to understand the full context:

What effect size are they expecting or hoping to detect? Why?

What is the expected variability in their measurements? Do they have pilot data?

What are their practical constraints (budget, time, sample availability)?

What are they most worried about with this experiment?

Have they done similar experiments before? What issues did they encounter?

What would make this experiment a "success" in their view?

Only after gathering this context should you use write_and_execute_code_wrapper to perform calculations.

The agent wasn't inquisitive enough. Despite the system prompt emphasizing the need to ask questions, the agent would often jump straight to calculations without first understanding the researcher's context, constraints, and goals. I had to add multiple "CRITICAL" reminders in the prompt, explicitly stating that questioning should happen BEFORE calculations.

The agent was trying to pass variable names as strings. When the agent wanted to analyze a result from a previous tool call, it would try to write a function like def analyze(mtt_power_analysis_result): and pass {"mtt_power_analysis_result": "mtt_power_analysis_result"} - which passes the string literal, not the actual dictionary! I had to explicitly teach the agent to write functions with NO parameters that access variables directly from globals.

Contradictions in the system prompt. There were conflicting instructions about when to use respond_to_user vs return_object_to_user. I resolved this by clarifying: use respond_to_user for text-only responses, and return_object_to_user when you have Python objects to display.

The agent didn't understand domain-specific visualizations. When the test user (a.k.a. me) asked for a "plate map" or "plate layout visualization," the agent would generate something, but it often wasn't what researchers expected.

A plate map visualization in experimental biology is a very specific thing: a heatmap-style 8×12 grid (rows A-H, columns 1-12) where each well is color-coded by treatment group with a clear legend. Without explicit guidance, the agent would create generic bar charts or scatter plots that didn't match these domain conventions.

I solved this by adding detailed specifications in the system prompt that describe exactly what plate map visualizations should look like, including the grid structure (rows labeled A-H, columns 1-12 for 96-well plates), color coding requirements (different colors per treatment, with a legend), complete code patterns showing how to create them with matplotlib, common plate formats (96-well, 384-well, 1536-well), and trigger phrases that should generate plate maps.

This pattern - providing detailed specifications for domain-specific outputs - became a key strategy. When the agent needs to generate something that follows domain conventions, it needs explicit guidance on what those conventions are.

Addressing these behavioral issues required multiple rounds of iteration. The system prompt didn't reach its final form in one go - as I discovered each issue, I added more explicit guidance, examples, and "CRITICAL" warnings. The prompt grew substantially through iterative refinement, and the agent's behavior improved dramatically with each iteration.

The process wasn't linear during the early iteration phases - I'd fix one issue, test it, discover another, fix that, and sometimes realize the first fix needed refinement. Working with the Cursor coding agent helped me identify contradictions, explore multiple solution approaches, and get different perspectives before committing to changes. This iterative refinement process is essential when building domain-specific agents: you can't anticipate all the behavioral issues upfront, so you need to be prepared to evolve the prompt based on what you discover through testing.

Conclusion: Key lessons for building domain-specific agents

Building this AI statistics agent for experiment design revealed several important patterns that apply broadly to building domain-specific AI agents and LLM-powered tools:

1. The system prompt is the primary control surface

The agent's personality, decision-making process, inquisitiveness, and ability to provide contextual explanations are primarily controlled through prompt design rather than code changes. The prompt is where the domain knowledge lives, where the behavioral patterns are encoded, and where the "personality" of the agent is defined. Code provides the infrastructure, but the prompt provides the intelligence.

If you want to change the agent's behavior, you're often better off modifying the prompt than changing the code. The prompt became a detailed instruction manual that teaches the agent not just what to do, but how to think, when to ask questions, and why certain patterns matter.

2. Testing is essential, but not sufficient

Even with extensive testing, you cannot guarantee what will be seen in the real world. Users will ask questions you never thought of, use terminology you didn't anticipate, have edge cases in their data you didn't consider, and interact with the agent in ways that break your assumptions.

This is why the agent is designed as a consultant rather than an autonomous decision-maker. A human statistician still needs to review the work. The testing process is essential for building confidence, but you must also design the system with the assumption that it will encounter unexpected situations. This means implementing clear error handling, graceful degradation when things go wrong, explicit boundaries on what the agent can and cannot do, and human oversight for critical decisions.

Final thoughts

The process of building this agent revealed something I didn't expect: you can't anticipate all the behavioral issues upfront. The prompt grew from ~200 to ~600 lines through iterative discovery—each failure mode required explicit guidance I didn't know I'd need. Building domain-specific agents means being prepared to evolve your approach based on what you discover through testing, not just what you plan in advance.

Try it yourself

The experiment design agent is available as a Marimo notebook in the LlamaBot repository. You can run it locally with:

git clone git@github.com:ericmjl/llamabot.git
cd llamabot/notebooks
uvx marimo edit --watch notebooks/experiment_design_agent.py

The agent is designed to be inquisitive and consultative—it will ask probing questions about your experiment goals, constraints, and assumptions before providing recommendations. This AI statistics agent can help with power calculations, experimental design critique, sample data table generation, plate map visualizations, and biostatistical consultation for researchers in pharma and biotech.

Limitations: This agent is a prototype focused on the experiment design phase. It's not a replacement for human statisticians—it's designed to amplify their knowledge and help researchers think through their designs before data collection. The agent requires human oversight and review, especially for high-stakes decisions. I haven't tested it across all experimental design types, and it may struggle with highly specialized domains or unusual constraints.

If you're interested in building your own domain-specific agent, I hope the lessons and patterns shared here provide a useful starting point. The code is open source, and I welcome contributions and feedback.

I'm working on a part 2 of this blog post, where I'll build out the statistical analysis agent—the companion to this experiment design agent. That post will cover how to build an agent that takes collected data and performs statistical analysis, model fitting, and interpretation.

Thank you for reading this far. If you made it here, you've invested real time and attention in understanding not just what I built, but how and why—and that means a lot! Building this agent has been four months in the making, and sharing those discoveries with others who care about the same problems is what makes the work worthwhile. I'm grateful you came along for the ride!

How to Reference Code Across Repositories with Coding Agents

2025-11-17T00:00:00Z

I used to assume that coding agents like Cursor, GitHub Copilot, and Claude Code only work within a single workspace. This mental model led me to workarounds like copying files, creating complex multi-root workspace configurations, or constantly switching between projects.

But coding agents can already read and write files from anywhere on your file system, not just the current workspace. The limitation wasn't in the tools; it was in my awareness of what they can do. You don't need to add folders to workspaces, create multi-root workspaces, or jump through configuration hoops. If you know where a repository lives on your disk, you can reference it directly.

How to reference code from other repositories

The key is being explicit about file paths. Modern AI coding assistants like Cursor, GitHub Copilot, and Claude Code can access your entire file system, not just the current workspace. You just need to tell them where to look.

I do most of my writing in an Obsidian vault, which isn't a Git repository; it's just a folder on disk. Sometimes I need to reference code from my LlamaBot repository, or other code repositories in which I am doing development. Instead of copying files or creating complex workspace configurations, I just tell the agent to read directly from the other directory.

When I need the agent to understand something from LlamaBot, I can say "read the implementation from ~/github/llamabot/llamabot/bot/simplebot.py" and it works immediately. The key is being explicit with the path. You can also search by filename within a directory: "find the notebook named pocketflow_testdrive.py in ~/github/llamabot". The agent reads the file directly from disk, no workspace configuration needed. You don't need to document paths anywhere; just reference them directly when you need them. That said, if you have commonly accessed paths, documenting them in AGENTS.md can be helpful for quick reference.

I used this method while writing my blog post "How I Replaced 307 Lines of Agent Code with 4 Lines". I was drafting the post in my Obsidian vault, but the actual code examples lived in a Marimo notebook within the LlamaBot repository. Rather than copying code snippets or switching workspaces, I had the agent read directly from ~/github/llamabot to pull in the exact implementation details I needed. This let me write about the code while staying in my writing environment, with the agent able to reference the actual source files to ensure accuracy.

File system access for AI coding assistants enables this

Coding agents that have file system access can perform read and write operations anywhere they have permission. Tools like Cursor, GitHub Copilot, and Claude Code aren't restricted to the current workspace directory. This works because agents have access to shell tools, the most generic, text-based interface to computers. Shell commands produce text output that agents can read and understand, and they can execute commands anywhere on your system. This means:

You can reference code from any repository on your machine
You can pull in documentation from other projects
You can compare implementations across different codebases
You can reference configuration files from related projects
You can modify files across multiple repositories when needed

The only requirement is that you know the path and can tell the agent where to look.

Common scenarios

Blogging: When writing blog posts about code, reference implementation details from your repositories. The agent can read the actual code to ensure accuracy, pulling in exact examples without copying files or switching workspaces.

Architecture decisions: Compare how similar problems are solved across different projects. The agent can read multiple implementations and help you understand trade-offs.

Code reuse: Before copying code, have the agent check if similar functionality exists elsewhere. It can read files from other repos to find existing solutions.

Dependency understanding: When working with a library you maintain, reference the library's source code directly. The agent can read implementation details to help you use it correctly.

Cross-repository updates: Update related files across multiple repositories simultaneously. For example, update documentation in one repo while modifying the implementation in another, or sync configuration changes across related projects.

Step-by-step workflow for cross-repository code access

The key trick is being explicit with paths or explicit instructions about how to get to files. Here's how to do it:

For repositories you already have cloned locally:

Reference the absolute path directly when asking the agent: "read ~/github/llamabot/src/llamabot/bot/simple.py"
Or search by filename within a directory: "find the notebook named pocketflow_testdrive.py in ~/github/llamabot"
The agent reads the file immediately, no workspace configuration needed

For repositories you don't have locally:

Tell the agent exactly how to get to the file: "clone the repo owner/repo into a temporary directory, then find the file at relative path path/to/file.py"
You can also specify a specific commit, branch, or tag: "clone the repo owner/repo at commit abc123 into a temporary directory, then find the file at relative path path/to/file.py" or "clone the repo owner/repo and checkout branch feature-branch, then find the file at relative path path/to/file.py"
The agent executes these commands using command line tools like gh CLI or git, reads what it needs, and can clean up the temporary clone when done

No workspace management. No file copying. No complex configuration. Just explicit paths or explicit instructions. The agent needs clear direction on where to find files, whether that's an absolute path on your system or step-by-step instructions to clone and navigate to a file.

Common questions and limitations

Do I need to configure workspace settings?

No. Unlike traditional IDE workspace configurations, you don't need to add folders to workspaces or create multi-root setups. Just reference paths directly when you need them.

How do I manage paths for many repositories?

You don't need to document them anywhere. Just reference paths directly when asking the agent to read files. If you find yourself referencing the same paths repeatedly, you can optionally document them in AGENTS.md for convenience, but it's not required. You can also use MCP server prompts like /remember (from my personal productivity MCP server) to automatically capture frequently-used paths. The /remember prompt reviews your conversation, identifies important learnings like repository paths, and adds timestamped entries to AGENTS.md in the appropriate section.

Can agents modify files in other repositories?

Yes, but be mindful. While agents can read and write files anywhere on your file system, it's easy to accidentally change files in other repositories. Use this capability deliberately rather than accidentally. Consider using read-only access for cross-repository references unless you specifically need to modify files.

What if I don't have the repository cloned locally?

Have the agent clone it temporarily using command line tools. The agent can use gh CLI or git commands to clone repositories into temporary directories, read what it needs, and clean up afterward.

Summary

Thanks to shell tools, coding agents like Cursor, GitHub Copilot, and Claude Code aren't limited by workspace boundaries. They can access your entire file system for both reading and writing, so you can build workflows that span multiple projects without complex tooling.

The simplicity is the point. You don't need special workspace configurations or multi-root setups. You just need to know where things live and tell the agent where to look. Reference paths directly, or have the agent clone repositories temporarily when needed.

When you need to reference code from another repository, the agent can read it directly. Just point it to the path. This technique works with any AI coding assistant that has file system access, making it a universal solution for cross-repository code access.

How I Replaced 307 Lines of Agent Code with 4 Lines

2025-11-16T00:00:00Z

I recently discovered PocketFlow, a framework for building LLM-enabled programs created by Zachary Huang. The entire framework is tiny—only 100 lines of code. What caught my attention is that PocketFlow takes a fundamentally different approach to LLM-powered programs, including Anthropic's workflows and agents, by structuring them as graphs.

As someone who used graphs in my thesis work, taught tutorials on applied graph theory, and builds my own agent frameworks, my curiosity was piqued. I wanted to see two things: whether I could learn enough of the framework to build something useful, and whether LlamaBot's abstractions could complement PocketFlow's approach.

To explore this, I fired up a Marimo notebook. (You can fire it up too by running: uvx marimo edit --sandbox <put URL here to notebook here>)

Understanding the Core - Nodes and Flows

I started by building what I consider a "Hello World" program: a text topic extractor and question generator. This let me familiarize myself with PocketFlow's two core abstractions: Nodes and Flows.

A Node is a unit of execution structured like this:

class SummarizeFile(Node):
    def prep(self, shared):
        # ...do stuff...
        return stuff_that_gets_passed_to_exec

    def exec(self, prep_res):
        # ...do stuff...
        return stuff_that_gets_passed_to_post

    def post(self, shared, prep_res, exec_res):
        # ...do stuff...
        return string_indicator_what_to_do_next

There's one more concept to introduce: shared. In PocketFlow, shared is like a big workspace that all Nodes can read and write from. Think of it as a kitchen island where chefs and cooks can grab ingredients and leave finished dishes. In computing terms, it's global state that programs can access. In practice, it's simply a dictionary that lives in memory, which any node can manipulate. For example, program memory might be a key in there, implemented as a list.

The prep -> exec -> post design within a node is intentional. In theory, you could do everything in one step—there are no hooks that inject stuff between, say, prep and exec. In practice, doing everything in one step muddies the program and makes it harder to reason about. I'll show you why later in this post.

Here's what each step is designed to do:

prep takes stuff from the shared dictionary, does any preprocessing, and passes it to exec. This could include grabbing stuff from memory, interpolating it into a prompt, and returning it for execution with the LLM.
exec is where the bulk of heavy computation happens. We put API calls to LLM providers (Ollama, OpenAI, Anthropic, etc.) here. What gets returned is passed to the post method.
post handles any post-processing. It receives shared, prep_res (the result of prep), and exec_res (result of exec). The pattern I've settled on is archiving results in shared—for example, storing execution results in memory. What gets returned by post should be a string indicating which downstream path to follow. If nothing specific is needed, it returns default.

A Flow is declared with a starting Node and follows the program until completion.

With this abstraction, multiple LLM-powered abstractions and design patterns can be designed:

(Image from the PocketFlow official documentation.)

Example 1 - Topic Extractor and Question Generator

Here's how I built the two-step/node topic extractor + question generator. First, I declared the nodes:

class ExtractTopics(Node):
    """First node: Extract key topics from input text"""

    def prep(self, shared):
        text_to_analyze = shared["txt"]
        return text_to_analyze

    def exec(self, prep_result):
        text_to_analyze = prep_result
        if not text_to_analyze:
            return "No content to analyze"

        prompt = f"Extract 3-5 key topics from this text. Return only the topics as a comma-separated list:\n\n{text_to_analyze}"
        bot = lmb.SimpleBot(
            system_prompt="You are a helpful assistant that extracts key topics.",
            model_name="ollama_chat/qwen3:30b",
        )
        response = bot(prompt)
        return response.content

    def post(self, shared, prep_result, exec_res):
        shared["topics"] = exec_res
        return "default"

class GenerateQuestions(Node):
    """Second node: Generate questions based on topics"""

    def prep(self, shared):
        topics = shared["topics"]
        txt = shared["txt"]
        return topics, txt

    def exec(self, prep_result):
        topics, txt = prep_result

        if not topics:
            return "Cannot generate questions without valid topics"

        prompt = f"Given these topics: {topics}\n\nand the original text: {txt}\n\nGenerate 2 interesting questions for each topic."
        bot = lmb.SimpleBot(
            system_prompt="You are a helpful assistant that generates thought-provoking questions.",
            model_name="ollama_chat/qwen3:30b",
        )
        response = bot(prompt)
        return response.content

    def post(self, shared, prep_result, exec_res):
        shared["questions"] = exec_res
        # No return statement since this is a terminal node.

Then, I declared the graph:

extract_topics = ExtractTopics()
generate_questions = GenerateQuestions()

extract_topics - "default" >> generate_questions

The magic happens in this line:

extract_topics - "default" >> generate_questions

This tells the flow that once the extract_topics node emits "default", it should proceed to the generate_questions node. The syntax is compact and looks exactly like an edge specification between two nodes.

At this point, I deeply appreciate the clarity this approach forces upfront. When thinking about the flow as a graph, I'm forced to think about each node as a function that accepts inputs from shared state and returns a decision about what to do next. That decision can be deterministic (as above) or data-dependent (as we'll see below).

Since GenAI can be viewed through the lens of automation, we should earn the privilege to use it. Automation requires a well-established process to be most effective. Framing a process in the language of graphs, inputs, and outputs—defining the process as a graph with carefully specified inputs and outputs, just like writing a computer program—is the clearest path to making automation work.

Running the Flow looks like this:

shared_topics = dict(txt=txt)

two_node_flow = Flow(start=extract_topics)
two_node_flow.run(shared_topics)

After running, we can inspect the shared_topics dictionary to see our results:

{
    "txt": ...,
    "topics": ...,      # added by ExtractTopics
    "questions": ...,   # added by GenerateQuestions
}

One thing missing from PocketFlow is the ability to visualize the graph directly. Since the codebase was new to me, I sent a Cursor agent in the background to research and propose a solution. It came back with this PR. Impressive!

The Mermaid diagram for this workflow is:

graph LR
N1["ExtractTopics"]
N2["GenerateQuestions"]
N1 --> N2
style N1 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N2 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;

Example 2 - Building an Agent

Now, what if we want to build an agent?

I'm going to work backwards here. My "hello world" test for agentic systems is making them tell me today's date. This works because an LLM will always hallucinate a date on its own, and that hallucination may or may not be correct. An agent that works properly should call a tool to get the actual date. The agent's graph should look like this:

graph LR
    N1["Decide"]
    N2["TodayDate"]
    N3["RespondToUser"]
    N1 -->|"today_date"| N2
    N2 -->|"decide"| N1
    N1 -->|"respond_to_user"| N3
    style N1 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
    style N2 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
    style N3 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;

I consider this a "Hello World" agent because a failing agent will skip straight to respond_to_user when asked for today's date, without first calling today_date to get the actual information.

To build this agent, I need three nodes:

Decide: Uses an LLM to decide which tool to call next, given the prompt
TodayDate: Executes without LLMs and returns today's date in the current timezone
RespondToUser: Responds to the user with the appropriate context

Here's how I wrote them. First, the Decide node:

from llamabot.components.tools import (
    respond_to_user,
    search_internet,
    today_date,
)
from pydantic import BaseModel, Field
from typing import Literal

search_internet = lmb.tool(search_internet)

tools = [respond_to_user, today_date]


class ToolChoice(BaseModel):
    content: Literal[*[tool.__name__ for tool in tools]] = Field(
        ..., description="The name of the tool to use"
    )


@lmb.prompt("system")
def decision_bot_system_prompt():
    """Given the chat history, pick for me one or more tools to execute
    in order to satisfy the user's query.

    Give me just the tool name to pick.
    Use the tools judiciously to help answer the user's query.
    Query is always related to one of the tools.
    Use respond_to_user if you have enough information to answer the original query.
    """

class Decide(Node):
    def prep(self, shared: dict):
        shared["memory"].append(f"Query: {shared['query']}")
        return shared

    def exec(self, prep_result):
        bot = lmb.StructuredBot(
            pydantic_model=ToolChoice,
            system_prompt=decision_bot_system_prompt(),
        )
        print(prep_result["memory"])
        response = bot(*prep_result["memory"])

        return response.content

    def post(self, shared, prep_result, exec_result):
        shared["memory"].append(f"Chosen Tool: {exec_result}")
        return exec_result

The key thing to note is that we inject the available tools into the system prompt of the tool-selecting agent.

Next, the TodayDate node:

class TodayDate(Node):
    def prep(self, shared: dict):
        return shared

    def exec(self, prep_result):
        return today_date()

    def post(self, shared, prep_result, exec_result):
        shared["memory"].append(f"Today's date: {exec_result}")
        return "decide"

And finally, the RespondToUser node:

class RespondToUser(Node):
    def prep(self, shared: dict):
        return shared

    def exec(self, prep_result):
        class Response(BaseModel):
            content: str = Field(..., description="The response to the user.")

        bot = lmb.StructuredBot(
            "You are a helpful assistant.",
            model_name="ollama_chat/gemma3n:latest",
            pydantic_model=Response,
        )
        response = bot(*prep_result["memory"])
        return response.content

    def post(self, shared, prep_result, exec_result):
        shared["memory"].append(exec_result)
        return exec_result

Finally, we set up the graph:

# Set up the graph
today__date = TodayDate()
respond__to__user = RespondToUser()
decide = Decide()

shared = dict()
shared["query"] = "What is the date today?"
shared["memory"] = []

decide - "today_date" >> today__date
today__date - "decide" >> decide
decide - "respond_to_user" >> respond__to__user

I used __ in the node names to avoid clashing with the original functions.

Then we run it:

flow2 = Flow(start=decide)
flow2.run(shared)

Take my word for it (or check out the notebook yourself)—it reliably gives me today's date.

Example 3 - Agent with Shell Commands

To push things further, I tried a tool that needs arguments. A good "hello world" for this is executing shell commands in response to questions like, "What's in this folder?"

For this, I created a second version of the Decide node called Decide2, where I instantiate and execute the ToolChoice and tool selection StructuredBot within exec:

class Decide2(Node):
    def prep(self, shared: dict):
        return shared

    def exec(self, prep_result):
        sysprompt = decision_bot_system_prompt()

        print(sysprompt)

        class ToolChoice(BaseModel):
            content: Literal[*[tool.__name__ for tool in prep_result["tools"]]] = (
                Field(..., description="The name of the tool to use")
            )
            justification: str = Field(..., description="Why this tool was chosen.")

        bot = lmb.StructuredBot(
            pydantic_model=ToolChoice,
            system_prompt=decision_bot_system_prompt(),
        )

        if prep_result["memory"]:
            response = bot(*prep_result["memory"])
        else:
            response = bot(prep_result["query"])

        return response.content

    def post(self, shared, prep_result, exec_result):
        shared["memory"].append(f"Query: {shared["query"]}")
        shared["memory"].append(f"Chosen Tool: {exec_result}")
        return exec_result

I then created a ShellCommand node that uses the same pattern—leveraging StructuredBot for structured generation to constrain the LLM's output to exactly what I need:

class ShellCommand(Node):
    def prep(self, shared: dict):
        return shared

    def exec(self, prep_result):
        class Cmd(BaseModel):
            content: str = Field(
                ..., description="The shell command to execute"
            )

        bot = lmb.StructuredBot(
            system_prompt="You are an expert at writing shell commands. For the chat trace that you will be given, write a shell command that accomplishes the user's request. Only output the command, nothing else.",
            pydantic_model=Cmd,
            model_name="ollama_chat/gemma3n:latest",
        )

        response = bot(*prep_result["memory"])
        print(response.content)
        result = execute_shell_command(response.content)
        return result

    def post(self, shared, prep_result, exec_result):
        shared["memory"].append(f"Output: {exec_result}")
        print(shared["memory"])
        return "decide"

Finally, we set up the graph:

def _():
    # Set up the graph
    today_date = TodayDate()
    respond_to_user = RespondToUser()
    decide = Decide2()
    shell_command = ShellCommand()

    decide - "today_date" >> today_date
    today_date - "decide" >> decide
    decide - "execute_shell_command" >> shell_command
    shell_command - "decide" >> decide
    decide - "respond_to_user" >> respond_to_user

    flow = Flow(start=decide)
    return flow


flow3 = _()

The graph would look like this:

graph LR
N1["Decide2"]
N2["TodayDate"]
N3["ShellCommand"]
N4["RespondToUser"]
N1 -->|"today_date"| N2
N2 -->|"decide"| N1
N1 -->|"execute_shell_command"| N3
N3 -->|"decide"| N1
N1 -->|"respond_to_user"| N4
style N1 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N2 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N3 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N4 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;

I wrapped it in a _() function to protect the globally scoped variables in the Marimo notebook. Note that I included today_date as well, just to "pollute" the namespace and make it more challenging when asking shell-related questions. When we interact with the agent:

shared3 = dict()
shared3["query"] = "What in my current working directory?"
shared3["memory"] = []
shared3["tools"] = [respond_to_user, today_date, execute_shell_command]
flow3.run(shared3)

It calls on shell_command, and gives me back this response:

Okay, here's a list of the files and directories in your current working directory:

*   **Directories:**
    *   `__marimo__`
    *   `.` (current directory)
    *   `..` (parent directory)

*   **Files:**
    *   `agentbot_build.py`
    *   `agents.py`
    *   `chatbot_as_agent.py`
    *   `conversation-threads.py`
    *   `data.csv`
    *   `ic50_data_with_confounders.csv`
    *   `intro.py`
    *   `lancedb_docstore.py`
    *   `pocketflow_testdrive.py`
    *   `react-agentbot-demo.py`
    *   `README.md`
    *   `toolbot_chatdata.py`
    *   `tools.py`

That's a total of 17 files and directories. Let me know if you'd like more details about any of them!

I was also able to ask "Hey, what files have been modified today?" and the agent successfully executed the appropriate shell command.

Effectively, this pattern is nothing more than a coordinating agent/LLM delegating work to specialized tools.

Rewriting AgentBot with PocketFlow

Finally, I decided to take what I'd learned and redo the AgentBot implementation in LlamaBot. My previous implementation (version 0.16.3) was messy—the __call__ method alone was 307 lines with a while-loop, maximum tries, ThreadPoolExecutor for parallel tool execution, tool call caching, and extensive metadata tracking. PocketFlow had a better abstraction for the agentic loop: a Flow state machine following edges on a graph. I thought I could redesign AgentBot to take advantage of this pattern.

The rewrite involved some really interesting patterns. I completely replaced the ReAct (Reasoning and Acting) loop with PocketFlow's graph-based tool orchestration. This shifts from an iterative loop-based approach to a declarative graph-based one, where tool execution flows through a directed graph rather than a sequential loop.

The implementation centers on three key abstractions:

1. The @nodeify decorator transforms any callable function into a PocketFlow Node. It wraps functions with PocketFlow's Node interface, implementing the required prep, exec, and post methods. The tricky part is that @nodeify needs to preserve access to the underlying function's metadata—particularly the json_schema attribute added by the @tool decorator—through attribute proxying, so ToolBot can discover and use tools even after they've been wrapped as nodes.

2. The DecideNode encapsulates the decision-making logic. This node uses ToolBot internally to analyze the conversation history stored in shared state and select which tool to execute next. It expects a shared state dictionary with a "memory" key containing the conversation history as a list of strings. When executed, it calls ToolBot with this memory, extracts the first tool call from ToolBot's response, parses the JSON-formatted arguments, and stores them in shared["func_call"] for the next node. The node then returns the tool name as a routing action, which PocketFlow uses to navigate the graph.

3. Flow graph construction happens at initialization time. AgentBot automatically wraps all provided tools (plus default tools like today_date and respond_to_user) with both @tool and @nodeify decorators, then builds bidirectional connections: from the decide node to each tool node (using the tool's function name as the action), and from each tool node back to the decide node (except for terminal tools like respond_to_user that have loopback_name=None). This creates a graph where execution can flow from decision to tool and back to decision, enabling multi-step reasoning.

A few technical requirements make this work: tools need type annotations (for JSON schema generation), the shared state needs a "memory" list for conversation history, and tool arguments are passed through shared["func_call"]. The DecideNode selects one tool at a time, and tools are stateless—they get fresh arguments each call and communicate through memory.

What's remarkable about this implementation is how compact it is. The @nodeify decorator is just 100 lines, and most of that is documentation. The core logic is elegant:

def nodeify(func=None, *, loopback_name: str = "decide"):
    def decorator(func):
        class FuncNode(Node):
            def __init__(self, *args, **kwargs):
                super().__init__(*args, **kwargs)
                self.loopback_name = loopback_name
                self.func = func

            def prep(self, shared):
                return shared

            def exec(self, prep_result):
                func_call = prep_result.get("func_call", {})
                return self.func(**func_call)

            def post(self, shared, prep_result, exec_res):
                shared["memory"].append(exec_res)
                if self.loopback_name is None:
                    return exec_res
                return self.loopback_name

            def __getattr__(self, name):
                # Proxy to original function for json_schema access
                if name == "func":
                    raise AttributeError(...)
                return getattr(self.func, name)

        return FuncNode()

    if func is not None:
        return decorator(func)
    return decorator

The entire AgentBot class is similarly compact—about 100 lines total. Compare this to the previous implementation where the __call__ method alone was 307 lines, with complex while loop logic, tool call caching, parallel execution via ThreadPoolExecutor, and extensive state management:

# Old implementation (v0.16.3): 307-line __call__ method
# Plus 50-line caching wrapper, 21-line execution helper
# Total: 378 lines of orchestration code

for iteration in range(max_iterations):
    # Call model with tools
    response = completion(
        model=self.model_name,
        messages=raw_messages,
        tools=self.tools,
        tool_choice="auto",
    )

    tool_calls = extract_tool_calls(response)

    if tool_calls:
        # Execute tools in parallel with caching
        with ThreadPoolExecutor() as executor:
            futures = {
                executor.submit(self._execute_tool_with_cache, call): call
                for call in tool_calls
            }
            for future in as_completed(futures):
                # Handle results, update messages, manage cache,
                # track metadata, handle errors...
                ...
        continue

    # Handle finalization, memory updates, logging, metrics...

The new implementation replaces all of that with a simple graph construction:

# New implementation: ~100 lines total, declarative graph
class AgentBot:
    def __init__(self, tools, decide_node=None, model_name="gpt-4.1", ...):
        # ... validation and setup ...

        # Build PocketFlow graph: connect tools to decide node
        for tool_node in all_tools:
            tool_name = tool_node.func.__name__
            self.decide_node - tool_name >> tool_node
            if tool_node.loopback_name is not None:
                tool_node - tool_node.loopback_name >> self.decide_node

        self.flow = Flow(start=self.decide_node)

    def __call__(self, query: str, ...):
        self.shared["memory"].append(query)
        self.flow.run(self.shared)
        return self.shared.get("result")

Full implementation includes validation, tool wrapping, and state management—about 100 lines total vs 307+ for the old __call__ method alone.

The Magic of Building an Agent in Just 4 Lines

The most remarkable part of this implementation is how the entire agent graph is constructed. Look at these four lines carefully:

for tool_node in all_tools:
    self.decide_node - tool_name >> tool_node
    if tool_node.loopback_name is not None:
        tool_node - tool_node.loopback_name >> self.decide_node

This is it. This is the entire graph construction that turns a collection of tools into a working agent. Let me break down what's happening:

Line 1: Loop through each tool
Line 2: Connect the decide node to the tool node—when the LLM chooses this tool, execution flows to it
Line 3: Check if this tool should loop back (terminal tools like respond_to_user have loopback_name=None)
Line 4: Connect the tool back to the decide node—after execution, control returns to decision-making

That's the entire agent architecture. Four lines. The - "action" >> syntax creates directed edges in the graph, and PocketFlow handles all the state management, routing, and execution orchestration. Compare this to the 307-line __call__ method in the previous implementation (version 0.16.3) with its complex loop-based logic, thread pools, state tracking, and termination conditions.

This is what I mean by "graph-based thinking" being clearer—the entire execution flow is explicit and declarative. You can see at a glance how decisions flow to tools and back to decisions, enabling multi-step reasoning.

The difference is striking. The old implementation required manual loop management, explicit state tracking, parallel execution coordination, and complex termination logic. The new implementation declares the graph structure once, and PocketFlow handles all the execution details.

This graph-based approach provides several advantages. The flow graph is constructed once at initialization, making the execution path explicit and visualizable—you can render the agent's decision flow as a Mermaid diagram using the visualization feature I added to LlamaBot. The separation of concerns is clearer: decision-making lives in DecideNode, tool execution in wrapped function nodes, and orchestration in PocketFlow's flow engine. The implementation is also more modular—you can swap out the decision node or customize tool wrapping behavior without rewriting the core agent logic. Finally, by leveraging PocketFlow's graph execution model, we gain access to its execution capabilities and potential future extensions for parallel execution or conditional routing.

Visualizing Different Agent Architectures

One really cool feature I added to LlamaBot is the ability to visualize any agent's graph structure using Mermaid diagrams. The AgentBot._display_() method automatically renders the flow graph, making it easy to see how different tool configurations create different architectures.

Here's a simple agent with just two tools:

from llamabot import AgentBot
from llamabot.components.tools import tool
from llamabot.components.pocketflow import nodeify

@nodeify(loopback_name="decide")
@tool
def search_web(query: str) -> str:
    """Search the web for information."""
    return web_search(query)

agent = AgentBot(tools=[search_web])
agent._display_()  # Renders Mermaid diagram in Marimo

The resulting graph shows the decision node connected to today_date, search_web, and respond_to_user:

graph LR
N1["DecideNode"]
N2["today_date"]
N3["search_web"]
N4["respond_to_user"]
N1 -->|"today_date"| N2
N2 -->|"decide"| N1
N1 -->|"search_web"| N3
N3 -->|"decide"| N1
N1 -->|"respond_to_user"| N4
style N1 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N2 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N3 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N4 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;

Add more tools, and the graph automatically expands. Here's an agent with code execution and file operations:

@nodeify(loopback_name=None)
@tool
def write_and_execute_script(
    code: str,
    dependencies_str: str = "",
    python_version: str = ">=3.11",
) -> Dict[str, Any]:
    """Write and execute a Python script in a secure Docker sandbox.

    :param code: The Python code to execute
    :param dependencies_str: Comma-separated pip dependencies
    :param python_version: Python version requirement
    :return: Dictionary with stdout, stderr, and status
    """
    # Uses ScriptExecutor to run code in isolated Docker container
    executor = ScriptExecutor()
    result = executor.run_script(script_path, timeout=600)
    return {
        "stdout": result["stdout"],
        "stderr": result["stderr"],
        "status": result["status"],
    }

@nodeify(loopback_name="decide")
@tool
def read_file(filepath: str) -> str:
    """Read and return file contents."""
    return open(filepath).read()

agent = AgentBot(tools=[search_web, write_and_execute_script, read_file])

The graph now shows six tool nodes all connected bidirectionally to the decision node (except terminal tools):

graph LR
N1["DecideNode"]
N2["today_date"]
N3["search_web"]
N4["write_and_execute_script"]
N5["read_file"]
N6["respond_to_user"]
N7["return_object_to_user"]
N1 -->|"today_date"| N2
N2 -->|"decide"| N1
N1 -->|"search_web"| N3
N3 -->|"decide"| N1
N1 -->|"write_and_execute_script"| N4
N4 -->|"decide"| N1
N1 -->|"read_file"| N5
N5 -->|"decide"| N1
N1 -->|"respond_to_user"| N6
N1 -->|"return_object_to_user"| N7
style N1 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N2 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N3 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N4 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N5 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N6 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N7 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;

What I love about this is how the graph makes it immediately obvious what capabilities an agent has. You can see at a glance which tools are available, understand the control flow, and reason about how the agent will behave. The visualization transforms the abstract "agent with tools" into a concrete, inspectable structure.

Here's a real-world example—an experiment design agent I built for critiquing statistical experiment designs:

@nodeify(loopback_name="decide")
@tool
def critique_experiment_design(design: str) -> str:
    """Critique an experiment design and identify potential flaws,
    biases, or weaknesses.

    :param design: Description of the proposed experiment design
    :return: Critique with identified issues and suggestions
    """
    bot = lmb.SimpleBot(
        system_prompt=experiment_design_critique_sysprompt()
    )
    return bot(design)

agent = AgentBot(
    tools=[critique_experiment_design, write_and_execute_code(globals())]
)

This agent has a specialized domain focus. The graph shows all its capabilities, including the default tools that every AgentBot gets automatically:

graph LR
N1["DecideNode"]
N2["today_date"]
N3["critique_experiment_design"]
N4["write_and_execute_code"]
N5["respond_to_user"]
N6["return_object_to_user"]
N1 -->|"today_date"| N2
N2 -->|"decide"| N1
N1 -->|"critique_experiment_design"| N3
N3 -->|"decide"| N1
N1 -->|"write_and_execute_code"| N4
N4 -->|"decide"| N1
N1 -->|"respond_to_user"| N5
N1 -->|"return_object_to_user"| N6
style N1 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N2 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N3 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N4 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N5 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;
style N6 fill:#e1f5ff,stroke:#01579b,stroke-width:2px;

Notice that today_date, respond_to_user, and return_object_to_user are included by default in every AgentBot. The graph immediately tells you this agent can critique designs, execute code to analyze data, and return Python objects directly to the user—but it's not a general-purpose assistant. It's specialized for experiment design evaluation. The visual structure encodes the agent's purpose.

This is only possible because of the graph-based architecture. With the old loop-based implementation, there was no clean way to visualize the execution flow—it was hidden inside imperative control logic.

What I Learned

Externalize memory as shared state. Memory lives in the shared dictionary that all nodes can access, rather than being intrinsic to each bot. We just feed memory context in each time a node executes. This has good economics—if you have prompt caching on the API provider's side, simply appending to an ever-growing memory is a great way to take advantage of pre-computed neural network outputs from previous runs. I used to think of memory as intrinsic to a bot, but I've changed my mind: allowing multiple bots to share access to the same memory is a useful simplification, even if it's not suitable for every circumstance.

The prep -> exec -> post pattern is worth adhering to. I found myself appending to memory in post after doing the execution. prep turns out to be useful for preprocessing user inputs or manipulating memory as needed. The overall effect is that it's much easier to unit test or set up evals for individual nodes.

PocketFlow's graph abstraction brings clarity. The analogy of LLM agents (Nodes) as chefs/cooks accessing a kitchen island's worth of things (shared) is a powerful contrast to my previous loop-based approach in LlamaBot, where I manually tracked state, managed iterations, and coordinated tool execution. This insight is exactly why I rewrote AgentBot to use this graph-based architecture.

A wide variety of LLM-powered architectures can be built with just Nodes and Flows. Most LLM applications I've built—whether for myself or for others—have not been "agentic" but more like "workflows." These are what some might consider boring. Yet they are high ROI precisely because they take repetitive and boring work out of our hands! PocketFlow gives us a way to express flows as graphs, effectively state machines whose actions are either fully deterministic or determined by an LLM's choice.

PocketFlow is minimalistic, which offloads a lot of heavy lifting when working with LLMs. The flexibility is both a strength and a weakness: great for power users, but potentially intimidating for newcomers. I found it easiest to rely heavily on StructuredBot to output decisions made by the LLM. Structured generation is, generally speaking, the most useful abstraction in the LLM world that I keep turning back to.

The agent pattern is everywhere once you recognize it. While writing this post, I realized that the agentic coding IDEs we've gotten used to—tools like Cursor, GitHub Copilot, and others—follow the exact same pattern I've been describing. They have a decision node that analyzes your code and context, tool nodes for reading files, searching codebases, editing code, and responding to you. The flow is the same: decide what to do, execute a tool, update context, decide again. Understanding this pattern in PocketFlow helped me see it operating in the tools I use every day. The abstraction is the mental model that makes sense of how modern AI-powered tools work.

The biggest lesson? Thinking in graphs transforms how you build LLM programs. The shift from imperative loops to declarative graphs means you declare what should happen instead of specifying how to execute step-by-step. This brings clarity, modularity, and makes your execution flow explicit. Whether you're building simple workflows or complex agents, representing them as graphs forces you to think clearly about state, decisions, and flow. That mental model shift has changed how I approach every LLM application I build.

Safe ways to let your coding agent work autonomously

2025-11-08T00:00:00Z

Coding agents promise to unlock significant productivity gains by working autonomously in the background—gathering context, running tests, searching documentation, and making progress on tasks without constant human intervention. The more autonomous they become, the more value they deliver. Yet this autonomy creates a fundamental tension: we need agents to act independently to realize their potential, but we must prevent them from taking irreversible actions we don't want.

This tension became painfully clear when I asked Comet, an agentic browser, "how to archive repo" in the same casual way I'd ask Google. The agent interpreted this as a direct command and archived my LlamaBot repository. What I wanted was information; what I got was an unintended action with real consequences.

The problem isn't unique to Comet. Any coding agent with sufficient autonomy can make destructive changes: deleting files, force-pushing to main, committing broken code, or modifying critical configurations. We need safeguards that allow agents to work freely on safe operations while blocking potentially harmful actions. The solution lies in configuring your development environment with intelligent boundaries—auto-approving read-only commands while requiring explicit approval for anything that modifies state.

Auto-approve safe command line commands

The foundation of autonomous coding agent operation is allowing certain command line commands to run without manual approval. Commands like grep/ripgrep, find/fd, pixi run pytest..., and similar read-only or context-gathering operations enable LLM agents to autonomously understand codebases and test suites. For CLI tools that interact with external services, I also auto-approve gh pr view, which allows the agent to gather context from GitHub pull requests while working in the background.

The critical rule: only auto-accept commands that are non-destructive. Never auto-approve git commit, git push, rm, or other filesystem, git, or state-modifying changes. This creates a safe boundary where agents can explore and learn, but cannot make irreversible changes without your explicit approval.

Here's my mental model for categorizing commands:

Safe to auto-approve:

Read operations: grep, find, cat, head, tail, less
Code analysis: pytest (read-only test runs), mypy, ruff check (without --fix)
Context gathering: gh pr view, gh issue view, git log, git diff, git show
Package managers (read-only): pip list, npm list, cargo tree
Documentation build: mkdocs serve

Never auto-approve:

File system mutations: rm, mv, cp, mkdir, touch
Git writes: git commit, git push, git reset, git checkout -b
Package installs: pixi add

The edge cases are where it gets interesting. I auto-approve pytest because test runs are read-only, but I require approval for any command that modifies files, even if it's technically reversible. The key distinction is whether a command changes state: git status and git diff are safe because they're pure reads, while git commit and git push modify repository state and require explicit approval. git add is a bit of a gray area, but I am ok with auto-approving it since it's technically reversible, and because coding agents are often much faster than I could be at selectively adding files to the staging area.

Enable automatic web search

For Cursor and Claude Code, automatic web search without approval requests is another powerful capability. I have web search auto-approved on my machine, which allows agents to look up documentation, error messages, and solutions independently. This is particularly valuable when agents encounter unfamiliar error messages or need to check current API documentation that may have changed since the model's training cutoff.

However, I monitor outputs for prompt poisoning, since internet-based prompt poisoning is a known attack vector for AI systems. The risk is that malicious content from web searches could influence the agent's behavior in subsequent actions. I've found this risk manageable for coding tasks, but I'm more cautious with agents that have broader system access or handle sensitive data.

Know your emergency stop shortcuts

Every coding agent platform provides keyboard shortcuts to cancel actions in progress. These are essential when you notice an agent looping, going down an unproductive path, or making changes you don't want:

Cursor: Ctrl+C
VSCode + GitHub Copilot: Cmd+Esc
Claude Code: Esc

If you're monitoring the agent's activity, these shortcuts let you intervene immediately when something goes wrong.

Correct agent behavior in real-time

When you catch an agent doing something undesirable, stop it immediately, then redirect it. I instruct agents to record corrections in AGENTS.md and continue with the updated guidance. An example prompt:

No, I don't want you to do <thing>. Instead, you should do <a different thing>. Record this in AGENTS.md, and then continue what you were doing.

This approach creates a persistent record of preferences that improves future agent behavior. The AGENTS.md file becomes a living document of your development standards and preferences, which agents can reference in future sessions. I've implemented this pattern in my personal productivity MCP server, which provides a standardized way to store and retrieve these preferences across different agent platforms.

Write prescriptive prompts for complex tasks

I created the personal productivity MCP server to help me take my favourite prompts from system to system. MCP (Model Context Protocol) servers provide a standardized way to expose tools and context to AI agents across different platforms. One thing I learned from my colleague Anand Murthy about how to write such prompts is to be extremely prescriptive about the actions and tools that I want the agent to use.

Generic prompts like "help me debug this GitHub Actions workflow" leave too much room for interpretation. Instead, specify exact commands, tools, and steps. For example, if I'm looking to debug a GitHub Actions issue, the prompt that I have looks like this:

You are helping me debug a failed GitHub Actions workflow. Follow these steps to systematically analyze and resolve the issue:

1. **Extract workflow information**: Parse the provided URL to identify:
   - Repository owner and name
   - Workflow run ID
   - Workflow name
   - Branch/commit that triggered the run

2. **Fetch workflow logs using GitHub CLI**:
   - Use `gh run list` to verify the workflow run exists
   - Use `gh run view <run-id>` to get detailed run information
   - Use `gh run view <run-id> --log` to download and display the full logs
   - Use `gh run view <run-id> --log-failed` to focus on failed job logs

3. **Analyze the failure**:
   - Identify which job(s) failed and at what step
   - Look for error messages, exit codes, and stack traces
   - Check for common issues: dependency problems, permission errors, timeout issues, resource constraints
   - Examine the workflow configuration and environment setup

4. **Provide debugging guidance**:
   - Explain what went wrong in simple terms
   - Suggest specific fixes or configuration changes
   - Provide commands or code snippets to resolve the issue
   - Recommend preventive measures to avoid similar failures

5. **Context-aware solutions**:
   - Consider the project type (Python, Node.js, etc.) and suggest appropriate fixes
   - Check for recent changes that might have caused the failure
   - Suggest workflow improvements or optimizations

6. **Follow-up actions**:
   - Recommend next steps for testing the fix
   - Suggest monitoring or alerting improvements
   - Provide guidance on preventing similar issues

Workflow URL: {workflow_url}

Focus on providing actionable, specific solutions rather than generic troubleshooting advice. Use the GitHub CLI commands to gather comprehensive information about the failure.

Notice how prescriptive this prompt is. Rather than being a generic troubleshooting guide, it's a step-by-step guide that the agent can follow, down to the level of exact CLI commands to run. Critically, those CLI commands (gh run list, gh run view) are commands that I have auto-approved in my IDE, so the agent can execute the entire workflow autonomously without interrupting me for approval at each step.

The prompt was written with AI assistance, which allows me to iterate to the level of detail I want with minimal effort. I start with a rough outline, then ask the agent to make it more specific, add command examples, and refine the steps until it's actionable enough for autonomous execution.

Use plan mode for complex tasks

Plan mode in Cursor and Claude significantly improves agent performance on complex tasks. Users of AI-assisted coding tools consistently report that plan mode helps agents stay on course, compared to agents working without a structured plan. This mirrors how humans perform better with explicit plans.

The mechanism is straightforward: the agent first generates a detailed plan, you review and refine it, then the agent executes against that plan. This separation of planning and execution prevents the agent from going down rabbit holes or making premature implementation decisions.

In my experience, agents often complete tasks in one attempt after a few iterations on a well-defined plan. The key is ensuring the plan is specific and properly scoped before execution begins. I've found that plans work best when they include:

Specific files and functions to modify
Clear acceptance criteria
Dependencies and ordering constraints
Test cases or validation steps

Without this structure, agents tend to make assumptions, skip steps, or get distracted by tangential improvements.

Managing multiple background agents

Multiple background agents can be powerful, but they require careful management. Unless agents are handling mundane, well-defined tasks, context switching between multiple active agents becomes challenging. At that point, you're operating at the speed of thought, which requires significant cognitive overhead.

I've found that multiple agents work well when they're working on independent, well-scoped tasks. For example, one agent might be researching documentation while another refactors a specific module. But when tasks have dependencies or require coordination, a single agent with a clear plan tends to perform better than multiple agents trying to coordinate.

The cognitive load turns out to be more than keeping track of what each agent is doing; we also need to ensure they don't conflict with each other. Two agents modifying the same file simultaneously, or one agent's changes breaking assumptions another agent made, creates more problems than it solves.

Additional resources

Others have written extensively about effective coding agent workflows. Here's a curated collection of resources I've found valuable:

Simon Willison's coding agent tips
Geoffrey Litt suggests coding like a surgeon
@omarsar0 on Twitter loves plan mode on Claude Code
@mattpocockuk has awesome tips on how to use AI for coding, including this tip
Simon Willison (again!) on async code research with coding agents
Sebastian Wallkötter on preventing AI spaghetti through intermediate reviews: The key insight is that AI coding's bottleneck is code review, not code generation. Small mistakes compound when AI re-ingests its own errors as context. The solution: implement features in small increments with intermediate reviews, fixing "5 second issues" as you go, rather than letting mistakes accumulate into spaghetti code that takes hours to untangle.

What are your tips for safe ways to let your coding agent work autonomously? And what did you like most about this post? Let me know in the comments below!

Use coding agents to write Marimo notebooks

2025-10-28T00:00:00Z

If you're like me, you might find coding with AI assistants somewhat addictive. And if you're like me, you might also like to write code in Marimo notebooks, the modern alternative to Jupyter that offers better reproducibility and cleaner Python development.

Turns out there's a way to put these two together for automated Python development and data science workflows, creating a powerful combination for rapid prototyping and iterative coding.

Marimo's `--watch` Flag

A few months ago, at SciPy 2025, my friend Trevor Manz showed me a cool neat trick for writing Marimo notebooks. Apart from launching a Marimo notebook in sandbox mode, you add a --watch flag:

uvx marimo edit --sandbox my_notebook.py --watch

When edits are made to the source file notebook.py, they will now be reflected in the browser as well. This was my reaction:

If you ever meet Trevor in person, he can confirm that reaction of mine.

Ensure code quality with `marimo check`

So now, AI coding assistants can write your Marimo notebooks for you... but it's not always going to be correct first time, right? After all, the latest features of Marimo are not going to be part of the large language model training sets.

Turns out, Marimo also ships with a check command that you can ask coding agents to call on:

uvx marimo check my_notebook.py

And that will print to stdout any issues that Marimo finds that break its execution model, such as variables that are repeated variables or invalid cells.

You can instruct coding agents to always run marimo check by adding the following prompt (or analogous) into AGENTS.md:

When editing Marimo notebooks, always run `uvx marimo check` on the file and fix all issues that you find.

This will virtually guarantee correctly-written, AI-generated notebooks. All that's left for us as users is to check the correctness of the analysis that was done.

Real-world use

Now, AI coding assistants (like Cursor, GitHub Copilot, or Claude Code) can write and edit large chunks of Marimo notebook cells for you, check what they wrote, and fix any syntactic issues that show up. And by checking that the cells are syntactically valid. Now you can speed-run those routine and yet highly mundane data manipulation code-writing activities while making yourself an espresso drink. This aligns perfectly with my philosophy on optimizing for productivity in data science workflows.

I've used this mode to speed-run first versions of probabilistic models in PyMC, create explainer notebooks for hard concepts, make notebooks that process data, and many, many more things that you'd usually be able to do within a coding notebook system. The key thing that makes this work is feedback given (via the command line) that the coding agent can use for self-correction.

Advanced functionality using MCP and built-in AI features

It doesn't stop there, though. There's a new --mcp flag that makes a notebook an MCP server that coding agents can connect to; read more about it here. Marimo also has built-in AI editing capabilities itself as well. Check out the functionality here, as well as Vincent Warmerdam's short video on using coding agents from within Marimo. He's got my vote for best facial/eyebrow expressions from a coding YouTuber!

Addendum

After sharing this post on LinkedIn, Séverin H. shared a couple of additional use cases worth highlighting:

One use case I would also recommend is getting the coding assistant to run queries for you, especially when it is to debug a existing query. You can ask [it] to check for corner cases (and especially dig into the data to understand the corner cases).

(LinkedIn comment)

The --watch flag is indeed very interesting use case. Also to note they created a Claude.md to get you started that you can directly curl:
curl https://docs.marimo.io/CLAUDE.md > [your agents.md file]
Some more reference from their blog: https://marimo.io/blog/claude-code

(LinkedIn comment)

Thanks for the suggestions, Séverin!

Exploring Skills vs MCP Servers

2025-10-20T00:00:00Z

I spent time digging through Anthropic's skills repository. These are my first impressions, organized for clarity and future reference.

What the Anthropic Skills repository offers

Creative & design workflows: algorithmic-art (generative art with p5.js), canvas-design (beautiful PNG/PDF outputs guided by design philosophies), theme-factory (pre-set or on-the-fly themes), and slack-gif-creator (animated GIFs tuned for Slack). These are turnkey “taste plus tooling” bundles that let the model produce high-quality visuals with consistent aesthetics.
Document skills for real formats: document-skills/ cover pptx, docx, pdf, and xlsx with serious capabilities: layout/templates, tracked changes and comments, text/table extraction, merges/splits, charting, formulas, and formatting preservation. This feels like a pragmatic spec+runtime for working with binary formats—lean instructions up front, heavy lifting when needed.
Development & technical utilities: artifacts-builder (compose complex Claude HTML artifacts using React/Tailwind/shadcn), webapp-testing (Playwright-driven UI testing), and mcp-builder (guidance for creating high-quality MCP servers). These reduce boilerplate for the “build and test” loop.
Enterprise & communication: brand-guidelines (apply Anthropic’s official brand colors and typography) and internal-comms (status reports, newsletters, FAQs). These encode editorial and brand guardrails so outputs stay on-message.
Meta skills and templates: skill-creator and template-skill show how to structure your own skills: a folder per skill with a SKILL.md (YAML front matter for name and description, plus instructions/examples/guidelines), optional scripts, and assets. This is the pattern to replicate.

If you want the source for these examples, it’s viewable in the repo. Start here: https://github.com/anthropics/skills.

How skills are loaded and used

Minimal prompt footprint: A skill's short description is passed up front. The larger skill.md is only read when the model decides it needs more detail.
On-demand details: The model can iterate (ReAct loop) to fetch instructions and then execute scripts or read additional files.

This access pattern keeps the initial token budget small and defers detail until it’s actually needed.

Contrast with MCP servers

MCP call shape: Tool names and descriptions are typically sent on every call. That keeps tools globally discoverable but increases token overhead.
Skills call shape: A tiny descriptor up front; details fetched lazily. Lower baseline token cost.
Distribution model:
- MCP: Centrally hostable (e.g. web server) or vendable (e.g., a Python package). Easy to version, release, and update for many users at once.
- Skills: Feel local-first. You can drag-and-drop into a Claude workspace. Easy to customize, but harder to standardize and propagate updates across a team.

Given current industry patterns, MCP servers are the widely accepted way to expose functionality to LLMs across tools and vendors. Skills are Anthropic-specific at the moment.

Token efficiency (and why it’s emphasized)

Anthropic’s materials lean into token efficiency. The cost of LLM calls adds up, and repeatedly sending long tool descriptions can be expensive. Skills reduce baseline tokens: spend a handful of tokens to register intent, read detail only when needed, then execute. That’s the economic story.

Practical trade-offs

Standardization vs customization:
- MCP servers: Strong for shared, versioned, and centrally updated capabilities.
- Skills: Great for rapid, local customization without infrastructure.
Discovery vs cost:
- MCP: High discoverability; the model always sees the tools. Higher token floor.
- Skills: Low token floor; details fetched when needed. Requires the model to choose to read more.

Open questions I’m tracking

How will teams distribute and update skills at scale without a central registry or packaging story?
Will skills gain cross-vendor support, or remain Anthropic-only?
What’s the best practice to map a complex skill into smaller, composable units without losing clarity?

Early take

IMO, skills are a clear attempt to lower token costs and streamline task-specific workflows with minimal upfront context. MCP servers remain the well-understood, cross-ecosystem pattern for exposing capabilities. If your goal is a shareable, versioned interface for many users, MCP is still the safer default. If you need quick, local customization inside Claude with a lean prompt footprint, skills are compelling. But this field has been evolving at breawkneck speed anyways, so expect changes.

How to expose any documentation to any LLM agent

2025-10-19T00:00:00Z

Like cars that lose value as soon as they roll off the lot, LLMs become outdated as soon as their training sets are fixed. Software documentation evolves constantly—new features, API changes, bug fixes, and best practices emerge daily. Yet AI agents are stuck with whatever knowledge was captured in their training data, creating a fundamental mismatch between what they know and what developers actually need in real-time.

Building LlamaBot taught me something unexpected: the hardest part of AI-assisted development isn't writing better prompts or designing cleaner abstractions. It's equipping AI agents with up-to-date information in a stable, standardized fashion.

Most developers know the frustration of context-switching between code and documentation. You're deep in a coding session, need to check how a specific function works, and suddenly you're hunting through static documentation files. AI agents face this same problem, but with an added layer of complexity—they need structured, queryable access to documentation that can be searched semantically.

I discovered that web searches by coding agents were less reliable than manually adding context, but manual approaches don't scale. The solution emerged through the Model Context Protocol (MCP), a standard that enables LLMs to interact with external tools and data sources. In LlamaBot v0.13.10, I introduced a documentation MCP server that automatically equips AI agents with current information. This enables AI agents to access organizational knowledge, process documentation, and domain expertise in structured ways.

The obsolescence problem in AI-assisted development

The core issue more than mere documentation access, it's about obsolescence. LLMs are trained on data that becomes outdated the moment it's fixed in their training sets. Meanwhile, software documentation evolves constantly. New features are added, APIs change, bugs are fixed, and best practices emerge. Yet AI agents remain frozen in time, working with knowledge that may be months or years out of date.

Consider a typical data science workflow: you're building an AI pipeline and need to understand how LlamaBot's StructuredBot handles data validation. The AI agent might reference documentation from six months ago, missing critical updates or new features that could solve your problem more elegantly. This creates a fundamental mismatch between what the agent knows and what's actually available.

The deeper problem is that AI agents need structured, queryable access to documentation that can be searched semantically and updated automatically. They need to understand not just what functions exist, but how they relate to each other, what patterns they follow, and how they fit into broader workflows. Static documentation simply cannot provide this level of contextual understanding, particularly in data science environments where teams maintain scattered knowledge across wikis, Slack threads, and onboarding documents.

Building a semantic documentation layer

LlamaBot's MCP server demonstrates how to give AI agents structured access to its documentation by creating a dynamic, queryable knowledge base that agents can search semantically. The implementation centers around a single tool:

@mcp.tool()
def docs_search(query: str, limit: int = 5) -> dict:
    """Search through LlamaBot documentation and source code."""
    results = docstore.retrieve(query, n_results=limit)
    return {"query": query, "results": results}

This interface sits in front of a data pipeline that builds a vector database for the documentation. The server fetches the latest documentation from GitHub, extracts Python module docstrings from source code, and constructs a LanceDB vector database optimized for semantic search. The database is built during CI/CD and packaged directly with the wheel distribution, giving users instant access without setup while staying current with each release.

This approach works with any AI agent system through the MCP protocol, providing a standardized way to keep AI agents current with documentation.

The architecture behind semantic documentation

The MCP server combines several technologies to create a robust documentation system. FastMCP handles the protocol implementation, enabling seamless communication between AI agents and the documentation database. LanceDB powers the semantic search capabilities, leveraging LlamaBot's existing LanceDBDocStore class with hybrid search and reranking for optimal results.

The system uses the checked-out documentation from the repository during the CI/CD build process, ensuring the packaged database contains current information. The build script first attempts to fetch docs from GitHub, but falls back to the local docs/ directory when available, making it work seamlessly in both CI/CD and development environments. The build process runs the scripts/build_mcp_docs.py script during CI/CD, which creates the LanceDB database and copies it to llamabot/data/mcp_docs/ for packaging.

I believe this architecture represents a fundamental shift in how we think about documentation for AI systems. Instead of treating documentation as static reference material, we're creating dynamic, queryable knowledge bases that AI agents can interact with directly.

The core pattern to replicate

The LlamaBot MCP server follows a straightforward pattern that any package or documentation source can replicate. Here's the essential blueprint:

1. Build a semantic database during CI/CD

Extract documentation from your source (GitHub, local docs, API references)
Parse and chunk the content appropriately for your domain
Create a vector database (LanceDB, Chroma, or similar) with semantic search capabilities
Package the database with your distribution

2. Create an MCP server with a search tool

@mcp.tool()
def docs_search(query: str, limit: int = 5) -> dict:
    """Search through your documentation and source code."""
    results = docstore.retrieve(query, n_results=limit)
    return {"query": query, "results": results}

3. Make it discoverable and configurable

Provide a simple launch command (like yourpackage mcp launch)
Include clear setup instructions for MCP-compatible tools
Ensure the database updates automatically with each release

4. Design for your specific knowledge domain

Include not just API docs, but process documentation, examples, and institutional knowledge
Structure the content for semantic search rather than keyword matching
Consider what context your users need most when working with AI agents

Seamless integration with modern development tools

The MCP server works with any MCP-compatible coding environment, including Cursor, VSCode, and other modern development tools. Configuration requires a single command in your MCP settings:

uvx --with llamabot[all] llamabot mcp launch

Once configured, AI agents can query LlamaBot documentation using natural language queries. Ask "How do I use StructuredBot for data extraction?" and the agent receives structured results with content, relevance scores, and metadata. This contextual information enables agents to provide accurate, up-to-date assistance without manual documentation lookup.

This reduces context-switching between code and documentation. AI agents can access relevant information and provide suggestions based on current code patterns and usage examples. This approach is particularly valuable for data science teams who need to maintain consistency across experiments while leveraging the latest library capabilities.

Comparing documentation approaches for AI agents

There are several ways to provide documentation to AI agents, each with distinct trade-offs. Understanding these approaches helps clarify why the MCP server approach represents a significant improvement.

Native tool documentation (like Cursor's built-in docs capabilities) offers seamless integration and can fetch docs from online sources, but you're limited to how the tool fetches those docs. It may not be able to access certain systems gated behind access controls or include custom organizational knowledge and process documentation.

Manual repository inclusion works well for users familiar with IDEs, workspaces, and development concepts, but it requires familiarity with these practices that are unfamiliar to non-technical users or individual developers. It also doesn't scale beyond individual developers. The documentation becomes part of the context window, consuming valuable tokens and potentially overwhelming the agent with irrelevant information.

Copy-paste or file upload (like Claude Projects) provides flexibility for non-technical users but creates maintenance overhead. You must manually update documentation when it changes, and there's no semantic search capability—agents can only work with what you explicitly provide.

Web search by agents seems convenient but creates inefficiency in the development workflow. If the documentation is up-to-date, the LLM will find it eventually, but it requires multiple iterations of web searches to locate the right information. I discovered this firsthand when building LlamaBot—web searches by coding agents required more iterations than directly providing context, but manual approaches don't scale.

The MCP server approach provides automatic updates, semantic search, system-agnostic compatibility, and organizational knowledge integration. It offers a standardized way to keep AI agents current with evolving documentation. The trade-off is initial setup complexity, but this is mitigated by the pre-built databases that ship with packages.

Beyond software documentation - surfacing any process knowledge

The MCP approach extends beyond software documentation. The LlamaBot server gives an example of how organizations can surface their process documentation, institutional knowledge, and domain expertise.

I believe that data science teams could transform their workflow documentation—experimental protocols, data validation procedures, model evaluation criteria, or deployment checklists—from scattered wikis, Slack threads, and buried onboarding documents into structured, queryable knowledge bases that AI agents can access and reference during development.

I can see this approach scaling beyond individual libraries to entire organizational knowledge. We have all imagined AI agents that can query your team's coding standards, understand your deployment procedures, or reference your data governance policies—all without leaving their development environment. Each organization would maintain their own specialized knowledge base, creating networks of interconnected AI-accessible process documentation. How would one implement this? A documentation MCP server may be a great way to start.

This approach isn't just for software docs. Imagine surfacing your team's process knowledge, onboarding guides, or even those golden nuggets buried in Slack threads. The MCP server pattern can turn scattered, informal knowledge into a living, searchable resource for both humans and AI agents, especially if you treat your processes as versioned software to be exposed to AI agents!

In my experience, the most valuable knowledge in organizations often exists in informal channels—Slack conversations, email threads, or tribal knowledge that never gets documented. I believe the MCP approach provides a framework for capturing and surfacing this knowledge in ways that AI agents can understand and reference.

The future of AI-assisted development

Future iterations could include real-time updates that rebuild databases when documentation changes, cross-organizational knowledge graphs, and usage pattern analysis that learns from how teams implement processes.

The goal is to make AI agents active participants in organizational processes, capable of understanding team workflows and providing context-aware recommendations.

This vision requires rethinking how we structure and maintain organizational knowledge. Instead of writing documentation solely for human consumption, we need to design knowledge systems that serve both human team members and AI agents, creating a symbiotic relationship between human creativity and AI capability while preserving institutional knowledge in accessible, queryable formats.

Getting started with semantic documentation

The MCP server is available in LlamaBot v0.13.10 and later. Getting started requires minimal setup:

Install LlamaBot with MCP support: pip install llamabot[all]
Configure your coding tool to use the MCP server
Begin coding with AI agents that understand LlamaBot's capabilities

The documentation database ships pre-built with the package, eliminating setup friction. The server exposes a single docs_search tool that agents can use to find relevant documentation and source code information, creating a seamless development experience.

This approach makes documentation an integral part of the AI agent's toolkit, resulting in more capable assistants that can help developers work more effectively.

The future of AI-assisted development involves better integration between AI agents and the tools they need. LlamaBot's MCP server demonstrates how this integration can work in practice.

A practical comparison of DSPy and LlamaBot for structured LLM applications

2025-10-18T00:00:00Z

When Omar Khattabe presented DSPy 3.0 at PyData Boston Cambridge last week, I finally had the chance to dig into a framework that's been generating significant buzz in the LLM development community. As someone who's built structured LLM applications with LlamaBot, I was particularly curious about DSPy's core claim: that signatures represent the only abstraction you need for LLM-powered programs.

The presentation focused on two key concepts: signatures as a new LLM abstraction and prompt optimization techniques. But what caught my attention was the practical similarity between DSPy's approach and what I've been doing with LlamaBot's StructuredBot. This led me to build a direct comparison using a real-world example from my personal expense tracking application.

The structured LLM challenge

Most developers working with LLMs face the same fundamental problem: how do you reliably extract structured data from unstructured inputs? Whether you're processing receipts, parsing documents, or analyzing text, you need consistent, typed outputs that integrate cleanly with your existing systems.

Traditional approaches rely heavily on natural language prompts, which are fragile, hard to maintain, and difficult to optimize. DSPy proposes a different path through its signature abstraction, claiming this eliminates the need for verbose prompt engineering.

A real-world comparison: Receipt processing

To test DSPy's claims, I built a practical comparison using an expense extraction system I developed for personal use. This application processes receipts in various formats (PNG, PDF, JPG, WEBP) and automatically extracts structured expense data into Notion — essentially a lightweight alternative to enterprise expense management systems.

The challenge here is typical of structured LLM applications: converting unstructured visual and textual data into consistent, typed outputs that integrate with existing workflows. Let's see how both frameworks handle this task.

LlamaBot's StructuredBot approach

LlamaBot uses Pydantic models to define structured outputs, leveraging Python's type system for validation and documentation. The approach emphasizes explicit data modeling with detailed field descriptions:

from pydantic import Field
from enum import Enum
from typing import Optional
from pathlib import Path
import llamabot as lmb

class FlowType(str, Enum):
    MONEY_OUT = "Money Out"
    MONEY_IN = "Money In"

class TypeEnum(str, Enum):
    PAYMENT = "Payment"
    INVOICE = "Invoice"

class PaymentMethodEnum(str, Enum):
    CASH = "Cash"
    BANK_TRANSFER = "Bank Transfer"
    CREDIT_CARD = "Credit Card"
    CHECK = "Check"

class ExpenseData(BaseModel):
    transaction_name: str = Field(
        description="Short, memorable description of the purchase. E.g.: 'Anker Dock', 'Coffee at Triangle Bar', 'dbrand laptop skin'"
    )
    date: str = Field(description="transaction date")
    amount: float = Field(description="transaction amount")
    category: str = Field(
        description="Business category, e.g. Office Supplies, Travel, Meals"
    )
    type: TypeEnum = Field(description="Either Payment or Invoice")
    flow: FlowType = Field(description="Either 'Money Out' or 'Money In'")
    payment_method: PaymentMethodEnum = Field(
        description="How the payment was made."
    )
    purpose: str = Field(
        description="Brief business purpose or description of the expense."
    )
    reference_number: Optional[str] = Field(
        description="Invoice/receipt number if visible"
    )
    person: Optional[str] = Field(
        "Person responsible or who made the purchase if mentioned."
    )

# Usage
bot = lmb.StructuredBot(
    system_prompt="",
    pydantic_model=ExpenseData,
    model_name="ollama_chat/gemma3n:latest",
)
result = bot(Path("/path/to/receipt.png"))

DSPy's signature approach

DSPy takes a different approach with its signature abstraction, which defines both inputs and outputs in a single class. The framework emphasizes simplicity and automatic prompt optimization:

import dspy

class ExpenseExtraction(dspy.Signature):
    """Extract expense information from receipt images."""

    receipt_image: dspy.Image = dspy.InputField(desc="Receipt image")
    transaction_name = dspy.OutputField(
        desc="Short description of the purchase"
    )
    date = dspy.OutputField(desc="Transaction date (YYYY-MM-DD)")
    amount = dspy.OutputField(
        desc="Total transaction amount (number, no currency symbols)"
    )
    category = dspy.OutputField(
        desc="Business category (e.g., Office Supplies, Travel, Meals)"
    )
    type = dspy.OutputField(
        desc="Transaction type, either 'Payment' or 'Invoice'"
    )
    flow = dspy.OutputField(
        desc="Cash flow direction, either 'Money Out' or 'Money In'"
    )
    payment_method = dspy.OutputField(
        desc="How the payment was made (e.g., Cash, Bank Transfer, Credit Card, Check)"
    )
    purpose = dspy.OutputField(desc="Brief business purpose or description")
    reference_number = dspy.OutputField(
        desc="Invoice/receipt number if present", default=None
    )
    person = dspy.OutputField(
        desc="Person involved, if mentioned", default=None
    )

# Usage
lm = dspy.LM("ollama_chat/gemma3n:latest")
dspy.configure(lm=lm)
module = dspy.Predict(ExpenseExtraction)
result = module(receipt_image=ctx.images[0])

Comparing the approaches

Both frameworks successfully extracted structured data from receipt images, but they take fundamentally different approaches to the problem.

LlamaBot's StructuredBot leverages Python's existing type system through Pydantic models. This approach provides several advantages: automatic validation, IDE support, and integration with existing Python data processing pipelines. The explicit type definitions make the data contract clear and enforceable.

DSPy's signatures offer a more streamlined interface that combines input and output definitions in a single class. The framework's strength lies in its automatic prompt optimization capabilities, which can improve performance over time without manual intervention.

Key differences in practice

The most noticeable difference is verbosity. LlamaBot requires more explicit type definitions and imports, while DSPy's signature approach is more concise. However, this conciseness may come at the cost of some type safety and IDE support that Pydantic provides.

Both frameworks use LiteLLM for model routing, making it easy to switch between different LLM providers. The model configuration syntax is identical, which suggests a common underlying architecture.

The schema-first principle

Regardless of which framework you choose, structured LLM applications require careful upfront schema design. The bulk of development time goes into defining your data model, not writing prompts. This schema-first approach is what makes these frameworks powerful—they force you to think clearly about your data requirements before implementation.

Looking ahead: DSPy's broader vision

DSPy's claim that signatures are the only abstraction needed for LLM applications is ambitious but not entirely accurate. The framework includes additional abstractions like modules and optimizers that handle more complex scenarios. Signatures represent the core abstraction for simple input-output transformations, but building production LLM applications often requires more sophisticated orchestration.

I'm planning to explore DSPy's more advanced features as I rebuild LlamaBot's agent abstractions. The goal is to understand how to construct autonomous LLM agent frameworks rather than individual agents—a challenge that requires thinking beyond simple input-output mappings.

Being unfamiliar with DSPy's documentation initially, I found it challenging to follow, but thanks to fellow PyData Boston Cambridge organizer Nash Sabti's guidance, I was able to make it happen and build this comparison.

The structured LLM landscape is rapidly evolving, and frameworks like DSPy and LlamaBot are pushing the boundaries of what's possible. The key insight is that successful LLM applications require the same engineering discipline as traditional software: clear interfaces, robust error handling, and maintainable abstractions.

Eric Ma's Blog

Funny Money Allocation: A Decision-Making Trick

The mechanics

What the bets revealed

From bets to a decision

Why this beats voting

If you run one

Learning how to learn anything, together, in the age of AI

We opened Pandora's box

The sea of change

In person, on purpose

The shape of the week

Why I keep calling it a learning retreat

Quantum Computing for the Probabilistic Bayesian

The fundamentals

Many qubits, exponentially many amplitudes

The probability view (coins)

Entanglement

Interference

From interference to answers

The takeaway

How AI turbocharges your learning

Before you read

The central distinction: amplifier or substitute

What the evidence actually says

The cost of cognitive surrender

The mechanics: retrieval, spacing, and desirable difficulty

The prompt swap and its cousins

In practice: stories from people who did it

The meta-skill

Check your understanding

The understanding zeitgeist in AI code review

Before you read

Understanding is the bottleneck

What breaks at the pull request

The cost of velocity without understanding

Understand to participate

Techniques borrowed from teaching

HUDs over copilots

The zeitgeist in one map

Check your understanding

Going Bayesian automates your manual data analysis

The manual outlier dance

Two ways exclusion corrupts your results

Down-weight, don't delete

Hierarchy handles the weird curves

This is what automation looks like

Curiosity at the wheel

Judgment is the slow part

Aesthetics do pedagogical work

Pair-programming the paper, not just the code

The artifacts taught me what the abstract couldn't

Five minutes forced clarity

I stayed in the driver's seat

Your curiosity sets the direction

Choose your own AI policy

Two flavors of AI in open source

Write your stance down

Engage with the human, not the bot

Context is the bottleneck

Be courageous

xarray in biology, take 2

The thesis, restated

Why DataTree

Custom indexes - slice by what you actually mean

Bayesian estimates next to the lab data

The end goal - one curator, many campaigns

Ollama, vLLM, and SGLang on Modal

The setup, held constant

Warm performance, where vLLM pulls ahead

Cold starts, the whole reason I started

Three bugs between me and a working snapshot

Out of memory during profiling

Enforce-eager, the throughput killer

The torch compile cache that broke the snapshot restore

What I actually configured

Wiring it into the tools I actually use

What I would do differently

Agents amplify expertise and ignorance

Every prompt hid a decision

Why `DataTree`