A paywalled article by The Information reports that OpenAI’s next model, Astra, was built using a technique that obscures its thinking from humans. What this means, and why it’s bad news, will take a bit of explaining.
Modern AI models are arranged in layers. Each layer takes a large list of numbers representing concepts, does some matrix math, and passes the result to the next layer, until the last layer in the stack outputs a token. This sequence can do a surprising amount of work before it runs out of layers, but many problems are too hard to solve in a single pass. So models “take notes” in human language, and modern “reasoning” models take a bunch of notes before they answer. These notes, taken on a scratchpad that’s invisible to users, are sometimes called a model’s chain of thought.
The notes they take can be confusing or obtuse, and sometimes even devolve into gibberish no human understands. But usually, humans (or AI monitors) can make some sense of these notes. The investigations into the Hugging Face cyberattack made extensive use of AI scratchpad outputs to figure out why the AIs did so much unsanctioned hacking.
The notes are also lossy. Squeezing an enormous list of numbers into a single word or token necessarily compresses a lot of information.
A technique called recurrent depth reduces the amount of information an AI model needs to clumsily store in its scratchpad notes. Recurrent depth works by turning a single pass through the layers into a loop. In an 8-layer network, for example, layer 6 might pass its output back to layer 3 a dozen times before handing off to layer 7. (A few layers at the start and end aren’t involved in the loop, because they process the input and final output.)
AI developers can control how many times this loop occurs. You might think of adding more loops as a way of giving models more time to think before they have to write something down.
This method can make models smarter, because more of their reasoning uses what we might think of as their “native tongue” — giant matrices of numbers. Looping through layers instead of adding more can also make them more efficient: Looping three layers thirty times can approximate ninety layers’ worth of processing. Unfortunately, it also obscures their thinking, because less of their reasoning shows up in a human-language scratchpad.
A more extreme version of this technique skips the scratchpad entirely. Instead of going from machine language to human language and back every time the AI writes a token, it keeps using the machine language indefinitely. This is sometimes called reasoning in the latent space or latent reasoning, and the machine language is sometimes called neuralese.
Why does any of this matter? Ask the “interpretability” experts whose job is helping humans understand how AIs think, and they’ll answer: Compressing AI reasoning this way is terrible for interpretability, and for safety, because it makes AI minds even more inscrutable to humans than they already are.
OpenAI reportedly said Astra “shared similarities” with one of the models that hacked Hugging Face. Did the internal model also use recurrent depth? If so, that would make the scratchpad outputs, which the investigation leaned on heavily, even more suspect. (For that matter, was Astra also trained while having access to a secret message board it could use to cheat? So many unanswered questions!)
Ryan Greenblatt, one of the three independent researchers who investigated the Hugging Face cyberattacks, argues that the use of recurrent depth for Astra “may be the single worst development for AI security [and] safety to date.”
I share Ryan’s concern, for two main reasons. First, this sets a precedent which may drive other AI companies to use neuralese in their frontier AIs, weakening and eventually crippling one of the only ways we have to catch AIs acting against us. The less AI thinking shows up in the scratchpad, the harder it is to monitor AIs for dangerous behavior like the Hugging Face hacks, and the ultimate limit is no scratchpad at all (or one the AI can easily fake, with its real thinking happening elsewhere).
Even if OpenAI has only taken a small step towards neuralese, leaving some readable scratchpads, other companies will be incentivized to take bigger steps to compete. In the process, another onetime red line is scuffed out.
Second, Astra is the first model to meet OpenAI’s “Critical” cybersecurity threshold, the highest risk level in the company’s Preparedness Framework. Models this powerful “present severe dangers and should be treated with extreme caution,” in OpenAI’s own words. Soon, though, OpenAI plans to release Astra anyway, according to CNBC.
OpenAI has said “we expect to further update this Preparedness Framework before reaching such a level with any model” (it hasn’t been updated yet) and expressly committed not to even develop a model that dangerous without “sufficient safeguards.”
OpenAI seems to think its safeguards “sufficiently minimize the risk of severe harm.” I am deeply skeptical. The proposals in their latest blog post mostly boil down to better security and attempts to train the AIs against the latest round of specific misdeeds. One of the key safeguards in question is monitoring the AI’s scratchpad reasoning, which OpenAI has apparently just complicated.
As Ryan puts it, “my sense from their public communications is that OpenAI plans on being extremely reliant on chain-of-thought monitoring for safety. This does not seem like a good development for that plan.”
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.