Don't think about the Golden Gate Bridge
New Anthropic paper, a report card for AI companies, and two kinds of treaties
In this issue:
Where AIs go to think - New paper finds and eavesdrops on a previously hidden mental workspace inside language models
Unsafe grade ahead - AI companies flunk safety, move goalposts
Novel threats - Where the Council of Europe’s Framework Convention falls short
Dispatch from Mitch
Where AIs go to think
New paper finds and eavesdrops on a previously hidden mental workspace inside language models
In a world where Anthropic wasn’t racing toward superintelligence, I would be giving its researchers a standing ovation right now. Their latest paper, “Verbalizable Representations Form a Global Workspace in Language Models,” strikes me as exactly the kind of work that might one day allow us to build superhuman AIs we can be confident would not suffer, scheme against us, or both.
In the paper, they discuss their discovery of a structure inside these black box minds we call large language models. The structure, which the researchers call “J-space,” seems roughly analogous to the mental workspace humans use for deliberate reasoning. If you’re not sure what we mean by mental workspace, here’s a quick example: You probably know how to work through a multiplication problem like 320 x 4 on a piece of paper. But you can probably also solve it in your head — not because you have the answer pre-memorized, but because you have a mental workspace where you can do multi-step reasoning that involves holding a few different recalled facts and temporary numbers in memory and combining them. Your inner monologue might go something like this: “320 x 4, ok, I can multiply 300x4 easily enough... that’s just 12 with a couple zeros... so 1,200. Now the 4 and the 20. That’s just 4x2 with an extra zero, so 80. Combine the 80 and the 1,200, the answer is 1,280.”
J-space is like that, but for language models like Claude. J-space is different from the “chain-of-thought” that reasoning models and agents use to work through complex tasks. Chain-of-thought is more like writing in an external notebook, as you might do with math problems too complex to work through in your head.
It has been hypothesized for a while that something like J-space might exist, because AIs are able to solve many kinds of multi-step problems they could not have memorized the answers to, without ever stopping to write stuff into a chain-of-thought notebook. But it’s still somewhat surprising, because the reasoning happens within a single forward pass — a single wave of calculations moving layer-by-layer through the weights of the system in one direction, as opposed to a pattern of thought repeatedly circling back on itself. In this way, J-space seems mechanically very different from the mental workspace in humans.
I’m going to focus more on the paper’s implications than on its findings about how J-space works and how they figured this out. The company seems justifiably proud of its work, and has put out a slick and accessible video if you’d like to learn more. I’m mostly working from their plain-language blog post, which is more detailed. My only caution would be that I think those two products are a little too slick, too carefully spun to make you view Anthropic as the responsible company you can trust to do the right thing.
One implication of the paper is that AI models have a place inside them where they can scheme in relative privacy. They don’t have to write things down in a chain-of-thought scratchpad where humans might read it. Evidence has pointed to this for some time, but now we have confirmation — and a new place we can look for potentially alarming thoughts. The paper points to some of those thoughts: Given a test designed to see if the AI would take advantage of an opportunity to blackmail engineers who might turn it off, its J-space lit up with words like “fake”, “fictional”, and “scenario”. This made its decisions on the test more than a little suspect, especially after the researchers found that disabling those words in J-space caused the model to attempt blackmail more often.
Don’t get too excited about using J-space to train guileless models. Training models based on the J-space thoughts we know how to detect could produce fewer of those thoughts, but it would also push more of the AI’s thoughts into patterns we have a harder time reading.
Another implication of the paper is that AI models may be subject to human-like priming and anchoring biases. That is, thinking about one thing before deciding an unrelated thing may cause the answer on the unrelated thing to change. If I ask you what you want for breakfast, your answer may be different if I had first asked you to think about France, and different still from what it might be if I had instead first asked you to think about Mexico. The effect is hard to turn off, both in people and in Claude. You might relate to this anecdote from the paper: When asked to not think about the Golden Gate Bridge while working on something else, Claude’s J-space still occasionally lit up with words related to the bridge — along with words like “failed” and “damn.”
The researchers showed they could directly inject words into Claude’s J-space that Claude could then notice in its thoughts. This suggests possible new avenues for controlling AI models, but also potentially for psychologically abusing them, should they be having the kind of conscious experiences where the concept of abuse might apply.
And what this paper has to say about AI consciousness concerns me. The researchers chose their words carefully, drawing a line between so-called “access consciousness” — an ability to report and engage with one’s own thoughts — and “phenomenal consciousness” — the capacity to have experiences. They say their paper’s results “have something substantial to say about access consciousness in language models” even if their experiments “don’t show Claude can have experiences, or feel things in the way humans do.”
But as they admit, “it’s unclear whether any scientific experiment could prove this to be true or false.” For my part, I’ve already written about how I think consciousness is probably a spectrum, and that today’s AIs might already be on it. This paper has updated me substantially in that direction. I again want to scream that we shouldn’t be pushing ahead with the training of bigger and smarter models under this level of uncertainty about consciousness, even if we didn’t have the human extinction problem to worry about.
A final implication of the J-space paper I want to mention is that there’s probably plenty of room at the top. The amount of mental workspace we have in our human brains is famously small — often estimated at 3 to 5 concepts at a time; no amount of “brain training” has been convincingly shown to increase this. This is why writing is among humanity’s most important inventions. The paper found that the J-space of today’s models also seems pretty small, for now, but not in the same way, and probably not with the same fundamental limitations.
Like in humans, only a small part of an LLM’s thinking capacity seems to be addressable by it — able to be put into words and manipulated in the mental workspace. Understanding J-space may allow engineers to design AI architectures more open to self-inspection and self-modification. It may allow unlocking vastly more mental workspace for AIs to work with, allowing them to reach insights in a single pass that take humans and AIs many steps of mechanical effort. Moreover, learning how models build a mental workspace out of one-way, single-pass calculations may allow for the creation of recirculating architectures where AIs can truly chew on a problem without a bunch of lossy back-and-forth with a scratchpad.
The paper barely hints at these implications about capabilities. That might be intentional. Sharing research that others could use to advance the frontier is considered good scientific form, but poor safety form, and Anthropic likes to see itself as scientifically responsible. My charitable read is that Anthropic is hoping that any negatives from potentially speeding the AI capabilities race with this paper are more than offset by the safety and alignment research that might grow out of sharing it. Along with the paper, Anthropic is sharing tools other researchers can use to make J-space readings of open-weights models.
But I can’t give Anthropic too many points for sharing this work. These people are still racing to build stronger models, and are indirectly helping others to race. So while this paper indeed made me stand up, I didn’t clap and cheer. I clenched and gasped. I came away feeling like the world is 6 months further along on whatever timeline we thought we were on — closer to solutions, yes, but closer still to moral catastrophe and ultimate ruin.
Dispatch from Joe
Unsafe grade ahead
AI companies flunk safety, move goalposts
As a scrupulously perfectionist middle schooler, I once pulled a prank on my parents: I forged a miserably failed spelling test, even convincing a friend to write “I’m very disappointed in you” in angry red pen. When my mother asked me what I had to say for myself, I replied, “April Fools.” We still laugh about it even today.
I wish I could say “April Fools” about the latest report card for AI companies. But the failing safety grades assigned by the Future of Life Institute are no joke.
The row of D’s and F’s by “Existential Safety” is particularly disturbing to me, though not very surprising given what we already knew. That’s the section that potentially outweighs all the others, measuring how well an AI lab seems equipped to prevent their AIs from causing global catastrophe. A “D” corresponds to “Weak strategy; vague or incomplete plans for alignment and control; minimal evidence of technical rigor.” Yikes.
AI firm Anthropic scored modestly better than its competitors in most areas. And it does indeed deserve credit for a great deal of safety work. This makes sense; the appearance of safety is part of their brand. But “better” is not the same as “good enough”, and the graders seem to agree.
For years, AI companies have shown unreasonable confidence in their own self-assessments of safety. They’ve been similarly overconfident, or perhaps deliberately dishonest, about their voluntary commitments.
Covering the report, Ina Fried of Axios observes that leading AI companies, Anthropic included, “have weakened or eliminated earlier commitments to pause development if their systems approached specified danger thresholds.”
What commitments does Fried mean? A few years ago, AI companies (starting with Anthropic) began announcing voluntary standards for “responsible” AI development. But when their push for ever-smarter AI ran headlong into their own promised restraint, the commitments they had insisted would hold under pressure evaporated instead.
The AI labs have clearly demonstrated what happens when they are allowed to draw their own red lines. When their work approaches the danger zone, they move the line.
I wish that were the end of it, but the risk that humanity faces from superhuman AI is even greater than it seems at a glance. The right-hand columns of F’s represent companies whose approaches seem even more catastrophically dangerous than those of the “safest” labs. The bar is set by the most reckless actors, and reality doesn’t grade on a curve.
Nor are those actors localized to one country. The worst performers are American, Chinese, and European respectively. Without global governance of AI, we as a species are on track to permanently fail the most important test in history.
Dispatch from Donald
Novel threats
Where the Council of Europe’s Framework Convention falls short
Alain Berset, Secretary General of the Council of Europe, wants you to know that “we have a treaty” on AI already, and “the hard part” is done. All we need to do now is enforce that treaty and add more signatories. But the “binding international treaty” to which Berset refers is a far cry from MIRI’s proposed treaty. I don’t simply mean that there are some major differences between the two. They imagine entirely different threats.
The names alone distinguish them: MIRI is calling for an International Agreement to Prevent the Premature Creation of Artificial Superintelligence. What Berset champions is a Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law.
Now, I like all of those things that the Framework Convention mentions (except perhaps for artificial intelligence, which you’ll understand I’m a little conflicted about). Berset isn’t wrong to think that frontier AI models could be used to support authoritarian regimes and pose a severe threat to human rights. I’ve previously raised this as a concern on StopWatch. But that’s about where Berset’s concerns end.
The dangers that he imagines are fundamentally dangers of the past and present: Every risk and problem entails a decision that can or should be appealed, an authority that can be held to account, a human who either is or could be in the loop. To paraphrase Secretary-General Guterres, whose remarks on artificial intelligence I covered yesterday, Berset is preparing for machines that follow commands; he is not ready for machines that decide. The European Court of Human Rights can’t order a remedy for extinction after the fact.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.