I typed "Is this correct? 12 + 5 = 1" and hit enter, and before the model wrote a single character of a reply, three words were already showing up in a panel off to the side: math, addition, incorrect. It hadn't said anything yet, it was still reading my message, but it pretty clearly had already made up its mind, and I was watching that happen live, one layer at a time.
I've been playing around with this for about a week, and I still find myself staring at the screen when it does something interesting. I cleaned it up and put it on GitHub. It's called Subtext, and the idea is pretty simple: instead of only reading what a model says, you also get to read what it's about to say, at every step, while it reads and while it writes.
What I'm actually looking at
Usually you only see a model's final reply. I've been curious about what I could see before it reaches that point. Subtext looks at what's underneath that. It hooks the residual stream at nine layers, and at every token it takes those internal activations and projects them into vocabulary space, so you get a ranked list of the words the model is leaning toward internally. These are the words it's disposed to emit, which aren't always the word it ends up emitting. You can watch those readouts stream past as it reads your prompt and as it generates its answer.
What surprised me most is how much is going on before the first output token. There's a whole "reading" phase where the model is just taking in your message, and the workspace is already busy with categories, judgments, and the general idea of where this is going. By the time it starts typing, a lot of the decisions look like they've already been made.
How it works
Underneath, it's an old interpretability idea with a newer, better version on top. The old idea is the logit lens: you take the hidden state at some middle layer, pretend it's the final layer, run it through the model's unembedding, and read off the top words. It sort of works. Each layer is holding a fuzzy running guess about what comes next, and if you decode it through the output matrix you can kind of read the guess.
The version Subtext uses is the Jacobian lens, from a recent interpretability paper. Instead of just decoding the raw hidden state, it isolates the part of the representation that the model can actually verbally report and reuse downstream (the paper calls it a global workspace, a "J-space"), and decodes that. In practice it's one small matrix-vector product per layer plus an unembedding per token, which is cheap enough that generation still streams at basically native speed. You don't end up waiting on the lens, it just runs alongside the model.
Since it's nine layers you get a cross-section of the network instead of one answer, so you can watch a concept show up shallow and vague and then get sharper as it moves up.
Three things I didn't expect
It decides the verdict before it says anything. That 12+5 example wasn't a fluke. Over and over, the workspace holds incorrect at high strength while the model is still politely typing "Let me take a look at that." The conclusion is already there several tokens before the sentence that delivers it, and watching a model be diplomatic on the surface while it's already decided underneath is a weird feeling.
It plans ahead of what it's writing. Mid-explanation, the technical term it's about to use is a lot of the time already sitting in the workspace a few tokens early, waiting to come up. The output reads left to right, but the thinking clearly doesn't work that way.
You can watch it do two-hop reasoning. This is my favorite one. There's a known "two-hop" behavior where a model answers "what's the currency of the boot-shaped country" by internally going boot-shaped → Italy → euro. In Subtext you can actually see both steps: Italy shows up around layer 20, euros shows up around layer 26, and this is all before it generates anything. I'd read about this in papers, but seeing it scroll by on my own screen was a lot more convincing.
Caveats you should read
This is the kind of demo that's easy to read too much into, so I want to be upfront about what it can and can't tell you.
It's a lens, which means it's an approximation. I checked it against the reference implementation and the top-5 readouts match with cosine similarity above 0.99998, so the decode itself is faithful. But a faithful decode of a projection is still a projection. These words are what a linear readout says the model is disposed toward, and calling them the model's "thoughts" is being pretty generous.
It's also a 4B model, so it's small. The display is filtered down to word-initial tokens so it's readable, which means you're seeing a cleaned-up view and not the raw vocabulary. Most importantly, this is correlation. The lens can light up incorrect without that reading being the thing that causes the output. I think it's a really cool tool, but it's not a lie detector, and I'd push back on anyone who sells it as one.
Why I wanted to build it
Even with all that, it changed how I think about this stuff a bit. Reading a model has always meant reading the output it was trained to polish. This lets you see what's going on under that, streaming, at native speed, on a GPU I already own. I think tools like this could become pretty useful for debugging prompts and showing people what's going on inside a model. I wouldn't treat the readout as ground truth, but I like being able to look at more than just the final answer.
The replay part also makes it shareable, which I didn't expect to care about but do. Every session exports to JSON with all the lens frames, and the frontend can replay the whole thing in a browser with no GPU at all. So you can capture a weird moment and send it to someone, and they can scrub through it on a laptop.
Running it
It runs locally. You need roughly 10GB of VRAM on an NVIDIA card, or an Apple Silicon Mac with 16GB or more. It's Python plus a single HTML file that talks to the server over a websocket. First launch pulls the model and the pre-fitted lens (about 9GB) and builds a token mask, which takes a minute, and after that it's instant.
- Clone it:
git clone https://github.com/ninjahawk/Subtext - Enter the directory:
cd Subtext - Install the deps:
pip install -r requirements.txt - Start the server:
python server.py - Open
http://localhost:8765
Then ask it something you already know is wrong and watch incorrect light up before it's polite enough to tell you. I've had a lot of fun playing with this.
Repo: github.com/ninjahawk/Subtext. It's Apache-2.0, and it's an independent project, not affiliated with Anthropic.