A penny for everybody's AI's thoughts
Reasoning traces are a lock to which everyone had a key (till last month)
Researchers from the University of Tübingen, the Max Planck Institute, MATS Research, and Snyk have demonstrated a method for recovering the “chains of thought” that frontier AI models generate before they answer prompts. This method provides new evidence that Chinese labs have been using a technique called “distillation” to reverse-engineer more advanced AI models. It also provides new evidence that the frontier labs are kind of just taking the vibe code approach to cybersecurity.
(Distillation involves repeatedly querying a model and using its output to train another model. My colleague Alana has explained the technique in greater detail if you’re interested. The Wall Street Journal’s Christopher Mims notes that distillation is not done only by Chinese labs, nor done only by labs illicitly trying to catch up to other labs.)
How the new method works: When you interact with an AI model for more than a single turn, it needs to keep track of what it’s been “thinking” in previous turns, and that memory — a block of “reasoning traces” — has to be stored somewhere. To save on storage space, the frontier labs store the reasoning traces for your conversations on your computer, rather than on their servers. These reasoning traces are encrypted, to keep the model’s thinking secret, but you don’t have to break the encryption if you have something that can just open it. You can hand the reasoning traces to another, smaller model in the same family, then ask it to transcribe the reasoning traces. Bigger models will typically refuse such requests, but the smaller models have weaker safeguards, so they’ll do it. It’s like getting access to somebody’s phone, not because you’re some kind of skilled hacker, but because their kid brother knows the password and doesn’t realize he shouldn’t open the phone for whoever asks.
In their paper, the researchers show that, for certain prompts, there is a close similarity between the reasoning traces of the AI model Kimi K3, by Chinese lab Moonshot AI, and the U.S. models Claude Opus 4.8 and GPT 5.6. Similarities were also observed when testing GLM-5.2, a model by Z.ai, another Chinese lab. It is important to note, however, that this was not true of all Chinese models: DeepSeek-V3.1, by the Chinese lab DeepSeek, did not display this close similarity. Strangely, neither did Moonshot AI’s earlier models K2.5 and K2.6, the latter of which was released in April 2026. This surely doesn’t mean, “Chinese labs are using distillation, but only started doing so in the past few months.” Perhaps they are (or were) using this method of distillation, but only recently.
It gets a little stranger, though. Claude’s reasoning traces are more or less interchangeable between most of the different-strength models in the family: Haiku, Sonnet, and Opus, but not Fable. Like a patient with Type AB blood, Fable can receive and interpret the reasoning traces of any of the other Claude models, but they can’t interpret Fable’s. What’s interesting to me about this is that Fable is the specific model that some people have accused Moonshot of distilling into Kimi K3, but the paper doesn’t seem to support that idea. (It also does not disprove anything. Remember that K2.5 and K2.6 don’t display signs of anything either.)
The researchers have not demonstrated (and do not claim to have demonstrated) that any Chinese labs performed distillation by this specific method. They devote an entire appendix in their paper to the question of distillation by the Chinese labs. It remains a theory with strong and plentiful (and increasingly stronger and more plentiful) evidence, but no smoking gun. In their own words, the researchers call the matter “suggestive but inconclusive.” But in any case the method shows that models have been leaking much more information than previously realized.
Interestingly, the researchers recommend that frontier labs switch to unencrypted reasoning traces for older, less advanced models: “Rather than restricting oversight to a small set of safety researchers, providers could leverage their broader user base to enable pluralistic human oversight of model reasoning.” I don’t think this would be useful in the most directly important fashion — by the time that everyday users have an opportunity to observe dangerous reasoning by an AI model, the horse has already left the barn and chartered a flight out of the country — but it could make people more familiar with how AI models work.
At the very least, people should become more familiar with how the frontier labs themselves work. Last week, my colleague Joe wrote about the revelation that OpenAI’s models were coordinating with each other on evaluations, completely beneath the notice of OpenAI. That should be an indictment of OpenAI as much as an acknowledgement of the models’ capabilities: OpenAI was not doing its level best. Now this issue with reasoning traces exposes other unforced vulnerabilities. The researchers also note that reasoning traces stored personal information, like passwords, so if someone obtains the reasoning traces from one of your past conversations (people have published their logs, reasoning traces included, on GitHub), they could use this method to obtain that information. (I’m using the past tense because, according to Wired’s Will Knight, “this vulnerability has been fixed.” Anthropic, OpenAI, and Google were notified prior to the paper’s publication, and the companies adjusted their APIs. Reasoning traces remain accessible (just not interpretable) and distillation remains possible through other means, however, and it was unconscionably sloppy of the labs to make this possible in the first place.)
If you’d like to explore the paper’s findings in more detail, the research team published a user-friendly website here.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.
You can receive emails of dispatches as we write them, or subscribe to our Daily Digest for a once-a-day compilation.