Eric J Ma's Website

Swapping the judge in a judgment language

written by Eric J. Ma on 2026-09-26 | tags: python programming ai agents creativity inference open source decision models machine learning


In this post, I share how I forked Seems, a Python-flavored judgment language, into seems-laya, swapping the hosted judge for an open-weight model running on my own machine. I walk through the one-file swap, getting judgment syntax into plain Python imports and marimo notebooks, and the concurrency, calibration, namespace, and bytecode-cache surprises along the way, most of them fixed by my coding agent. I also wrestle honestly with whether the work was creative at all. If I can't tell the difference between interpolation and invention, how would any of us know when a machine crosses that line?

In late September, @kavehmz published Seems, a programming language that is Python plus a few words. In a Seems program, a condition can be plain English, and a decision model answers it with a probability:

if ticket.amount > 500 and ticket.text asks for a refund:
    send_to_manager(ticket)
unsure:
    send_to_human(ticket)

ticket.amount > 500 runs in Python. ticket.text asks for a refund goes to a decision model. TypeSafe's Jev reads the text and returns a probability. Below 0.25 the judgment is no, above 0.75 it's yes, and in between it's unsure. unsure: is a real branch of if, so a program that isn't sure enough routes to a human instead of acting on a guess.

Under the hood, there is no AI in the language itself. A Seems program is a .seems file, and a translator rewrites it into plain Python using Python's own tokenizer and a fixed list of judgment verbs (seems, sounds, asks, contradicts, and a few more). The rule is mechanical: an expression, then a verb, then English up to the next comma or colon. The expression becomes the state, the English becomes the question, and the whole judgment becomes a call into a small runtime that batches questions and caches answers. Everything else in the file passes through byte for byte, line numbers untouched. The one thing the translator cannot do locally is answer: judgments leave as a single HTTP call to TypeSafe's hosted Jev endpoint, billed to an API key.

The project ships as a Docker container. There is no wheel to install. The import hook that understands .seems files only switches on inside the project's own entry points, so a regular Python session has no path to any of it: import has never heard of the .seems suffix, and a script you run directly bypasses the import system entirely. Hence the compose file, the playground, the demo desk app, and a README whose setup section is one command, docker compose up --build -d.

I loved it! And I had two questions the README couldn't answer for me. Can this work outside its Docker container, in a regular Python session? And the one I cared about more, the reason I built anything at all: every judgment in this language is billed to a closed hosted model. Could the whole idea run on open weights instead, on my own machine, with no API key and no meter running?

So I forked the idea and the code. The result is seems-laya, where every judgment is answered by Laya, Convai Innovations' open-weight decision model. The language is kavehmz's; I only swapped the judge. He added an MIT license the same week, which unblocks a PyPI release for the variant.

The swap was one seam

The architecture @kavehmz shipped made this easier than it sounds. Every judgment in Seems funnels through a single client call: give it a state and a list of questions, and it returns an answer with a probability for each one. Jev and Laya, rivals launched three days apart, turned out to ask questions in an identical format (noul for yes or no, choice for picking among options, score for rating on a scale, each with instructions and optional criteria). Everything around the questions differs: Jev wants an API key and bills per token; Laya runs locally for free. So I only had to rewrite one file. seems/client.py used to post to TypeSafe's endpoint; now it loads a Laya checkpoint in-process, or posts to a Laya server on your network if you'd rather keep the model on a GPU box.

Three doors into plain Python

seems-laya ships as an import hook. The wheel installs a startup file, so any plain .py file on your machine can contain judgment syntax and a normal import runs it through the translator. You never create a .seems file: write an ordinary .py file, judgment syntax included, and the hook translates the files that carry judgment verbs. Original .seems files still load, so programs written for the language as published keep working, but nothing generates them either. Flask apps, workers, pytest files: anything that imports your package gets judgments. Two boundaries needed extra work. CPython compiles the file you run directly below the import system, so scripts go through a launcher, seems run script.py. And marimo compiles notebook cells itself, so the project ships a wrapper, seems-marimo, that teaches marimo's compiler the same trick. A cell can now hold the if / unsure: branch directly, reactive over widgets.

What broke

  • Concurrency. Laya crashed the process under simultaneous requests on macOS. Turns out, it was due to an MPS command-buffer race. The runtime already batches same-state questions into one request, so serializing inference cost almost nothing. A hosted API parallelizes for you; a local model asks you to serialize.
  • Calibration. Example 06 in the original repo now raises Unsure where Jev was confident enough to act. In it, intent is one of leave, complain scores 0.64 under Laya against the 0.75 bar, and the example has no unsure: branch there. So the program stops with an Unsure error instead of acting on the 0.64 answer. The base Laya checkpoint spreads probability more diffusely than Jev, and programs written against Jev's calibration can need explicit unsure: branches under Laya.
  • Namespaces. My first version injected the runtime's names into every imported module. Torch's config machinery asserts that its own module namespaces are clean, and the process died on import. The fix was to inject names only where the compiled code actually references them.
  • The bytecode cache. Translated code validated against untouched source files, so a warm cache would keep running judgment syntax even after you turned the hook off. Translated files never write bytecode now.

Here is the performance of seems-laya. Warm, the six example programs finish in 0.2 to 0.9 seconds; upstream measured 1 to 4 seconds against the hosted API. Inference runs on my machine, so the marginal cost is $0.00, and the answers are deterministic.

Was any of this actually creative?

I've been writing Python for many years. I maintain libraries, I teach data science, Python pays my salary. And almost nothing in this project was within my range of knowledge. When the notebook kernel died with a Metal command-buffer assertion, I hadn't finished reading the traceback before the agent had serialized the client. When torch's config machinery choked on an injected name, the agent moved the injection where third-party introspection wouldn't see it. When marimo's kernel turned out to be a spawned subprocess, the fix was an environment variable and a startup file we already had. All told, GLM-5.3-Flash running in Pi went from kavehmz's untouched tree to the working notebook in about 90 minutes and five commits.

If I had hired a human engineer with that exact combination of knowledge, I would call the engineer creative. But I have no way to audit the model's reading. Every fix it made might be interpolation over some GitHub thread I've never read; my sense that it was creative could just measure how small my corpus is. There's a company-scale version of the same question: Laya shipped three days after Jev, and Laya's author has said in public that the architecture is his own earlier work, relaunched by a better-funded company. When everyone reads the same papers and builds with the same tools, two teams can pull the same thread and each experience the pull as invention.

So, will AI be truly creative, at least in the sense of going beyond its own training set boundaries? Most of what I can find says not yet. Demis Hassabis proposed the cleanest test: train a model on everything known before 1900 and ask it to derive relativity, the way Einstein did. The attempts so far mostly fail. One experiment found that a searcher trained on pre-1905 physics was over a million times less likely to propose the Lorentz factor than a blind one; everything it had read pointed at Newton, and the right answer violated everything it knew. And yet, in 2026, one paper showed that in a setting where interpolation provably scores zero, a transformer learns the hidden rule anyway, and other teams have coaxed frontier models into original proofs of open conjectures. I can't tell the difference from my chair. Maybe creativity was always interpolation over a corpus, with taste deciding what to combine; maybe the model is doing exactly that with a corpus the size of civilization. Or maybe creativity is in the eye of the beholder, expanding the beholder's sense of what's possible. I'll believe it the day a machine combines ideas into something that works, and nobody can trace where it came from: not to a paper, not to a repository, not to anything in its training data.


Cite this blog post:
@article{
    ericmjl-2026-swapping-the-judge-in-a-judgment-language,
    author = {Eric J. Ma},
    title = {Swapping the judge in a judgment language},
    year = {2026},
    month = {09},
    day = {26},
    howpublished = {\url{https://ericmjl.github.io}},
    journal = {Eric J. Ma's Blog},
    url = {https://ericmjl.github.io/blog/2026/9/26/swapping-the-judge-in-a-judgment-language},
}
  

I send out a newsletter with tips and tools for data scientists. Come check it out at Substack.

I'm co-teaching a one-week retreat on how to learn anything with AI with Daniel Chen, February 2027.

If you would like to sponsor the coffee that goes into making my posts, please consider GitHub Sponsors!