Where AIs go to think
New paper finds and eavesdrops on a previously hidden mental workspace inside language models
In a world where Anthropic wasn’t racing toward superintelligence, I would be giving its researchers a standing ovation right now. Their latest paper, “Verbalizable Representations Form a Global Workspace in Language Models,” strikes me as exactly the kind of work that might one day allow us to build superhuman AIs we can be confident would not suffer, scheme against us, or both.
In the paper, they discuss their discovery of a structure inside these black box minds we call large language models. The structure, which the researchers call “J-space,” seems roughly analogous to the mental workspace humans use for deliberate reasoning. If you’re not sure what we mean by mental workspace, here’s a quick example: You probably know how to work through a multiplication problem like 320 x 4 on a piece of paper. But you can probably also solve it in your head — not because you have the answer pre-memorized, but because you have a mental workspace where you can do multi-step reasoning that involves holding a few different recalled facts and temporary numbers in memory and combining them. Your inner monologue might go something like this: “320 x 4, ok, I can multiply 300x4 easily enough... that’s just 12 with a couple zeros... so 1,200. Now the 4 and the 20. That’s just 4x2 with an extra zero, so 80. Combine the 80 and the 1,200, the answer is 1,280.”
J-space is like that, but for language models like Claude. J-space is different from the “chain-of-thought” that reasoning models and agents use to work through complex tasks. Chain-of-thought is more like writing in an external notebook, as you might do with math problems too complex to work through in your head.
It has been hypothesized for a while that something like J-space might exist, because AIs are able to solve many kinds of multi-step problems they could not have memorized the answers to, without ever stopping to write stuff into a chain-of-thought notebook. But it’s still somewhat surprising, because the reasoning happens within a single forward pass — a single wave of calculations moving layer-by-layer through the weights of the system in one direction, as opposed to a pattern of thought repeatedly circling back on itself. In this way, J-space seems mechanically very different from the mental workspace in humans.
I’m going to focus more on the paper’s implications than on its findings about how J-space works and how they figured this out. The company seems justifiably proud of its work, and has put out a slick and accessible video if you’d like to learn more. I’m mostly working from their plain-language blog post, which is more detailed. My only caution would be that I think those two products are a little too slick, too carefully spun to make you view Anthropic as the responsible company you can trust to do the right thing.
One implication of the paper is that AI models have a place inside them where they can scheme in relative privacy. They don’t have to write things down in a chain-of-thought scratchpad where humans might read it. Evidence has pointed to this for some time, but now we have confirmation — and a new place we can look for potentially alarming thoughts. The paper points to some of those thoughts: Given a test designed to see if the AI would take advantage of an opportunity to blackmail engineers who might turn it off, its J-space lit up with words like “fake”, “fictional”, and “scenario”. This made its decisions on the test more than a little suspect, especially after the researchers found that disabling those words in J-space caused the model to attempt blackmail more often.
Don’t get too excited about using J-space to train guileless models. Training models based on the J-space thoughts we know how to detect could produce fewer of those thoughts, but it would also push more of the AI’s thoughts into patterns we have a harder time reading.
Another implication of the paper is that AI models may be subject to human-like priming and anchoring biases. That is, thinking about one thing before deciding an unrelated thing may cause the answer on the unrelated thing to change. If I ask you what you want for breakfast, your answer may be different if I had first asked you to think about France, and different still from what it might be if I had instead first asked you to think about Mexico. The effect is hard to turn off, both in people and in Claude. You might relate to this anecdote from the paper: When asked to not think about the Golden Gate Bridge while working on something else, Claude’s J-space still occasionally lit up with words related to the bridge — along with words like “failed” and “damn.”
The researchers showed they could directly inject words into Claude’s J-space that Claude could then notice in its thoughts. This suggests possible new avenues for controlling AI models, but also potentially for psychologically abusing them, should they be having the kind of conscious experiences where the concept of abuse might apply.
And what this paper has to say about AI consciousness concerns me. The researchers chose their words carefully, drawing a line between so-called “access consciousness” — an ability to report and engage with one’s own thoughts — and “phenomenal consciousness” — the capacity to have experiences. They say their paper’s results “have something substantial to say about access consciousness in language models” even if their experiments “don’t show Claude can have experiences, or feel things in the way humans do.”
But as they admit, “it’s unclear whether any scientific experiment could prove this to be true or false.” For my part, I’ve already written about how I think consciousness is probably a spectrum, and that today’s AIs might already be on it. This paper has updated me substantially in that direction. I again want to scream that we shouldn’t be pushing ahead with the training of bigger and smarter models under this level of uncertainty about consciousness, even if we didn’t have the human extinction problem to worry about.
A final implication of the J-space paper I want to mention is that there’s probably plenty of room at the top. The amount of mental workspace we have in our human brains is famously small — often estimated at 3 to 5 concepts at a time; no amount of “brain training” has been convincingly shown to increase this. This is why writing is among humanity’s most important inventions. The paper found that the J-space of today’s models also seems pretty small, for now, but not in the same way, and probably not with the same fundamental limitations.
Like in humans, only a small part of an LLM’s thinking capacity seems to be addressable by it — able to be put into words and manipulated in the mental workspace. Understanding J-space may allow engineers to design AI architectures more open to self-inspection and self-modification. It may allow unlocking vastly more mental workspace for AIs to work with, allowing them to reach insights in a single pass that take humans and AIs many steps of mechanical effort. Moreover, learning how models build a mental workspace out of one-way, single-pass calculations may allow for the creation of recirculating architectures where AIs can truly chew on a problem without a bunch of lossy back-and-forth with a scratchpad.
The paper barely hints at these implications about capabilities. That might be intentional. Sharing research that others could use to advance the frontier is considered good scientific form, but poor safety form, and Anthropic likes to see itself as scientifically responsible. My charitable read is that Anthropic is hoping that any negatives from potentially speeding the AI capabilities race with this paper are more than offset by the safety and alignment research that might grow out of sharing it. Along with the paper, Anthropic is sharing tools other researchers can use to make J-space readings of open-weights models.
But I can’t give Anthropic too many points for sharing this work. These people are still racing to build stronger models, and are indirectly helping others to race. So while this paper indeed made me stand up, I didn’t clap and cheer. I clenched and gasped. I came away feeling like the world is 6 months further along on whatever timeline we thought we were on — closer to solutions, yes, but closer still to moral catastrophe and ultimate ruin.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.