In this issue:
Making frontier AI more inscrutable than ever - OpenAI’s planned new model undermines its own monitoring safeguards
Four terrible takes on AI - Someone is wrong on the internet and it’s finally my job to correct them
China reportedly signs non-binding governance principles at G20 meeting - The principles were proposed by the US and favor light-touch regulation
Bernie Sanders takes AI pause message to Fox News - Sanders invokes Reagan and calls on Trump to negotiate with Xi
Dispatches from Joe
Making frontier AI more inscrutable than ever
OpenAI’s planned new model undermines its own monitoring safeguards
A paywalled article by The Information reports that OpenAI’s next model, Astra, was built using a technique that obscures its thinking from humans. What this means, and why it’s bad news, will take a bit of explaining.
Modern AI models are arranged in layers. Each layer takes a large list of numbers representing concepts, does some matrix math, and passes the result to the next layer, until the last layer in the stack outputs a token. This sequence can do a surprising amount of work before it runs out of layers, but many problems are too hard to solve in a single pass. So models “take notes” in human language, and modern “reasoning” models take a bunch of notes before they answer. These notes, taken on a scratchpad that’s invisible to users, are sometimes called a model’s chain of thought.
The notes they take can be confusing or obtuse, and sometimes even devolve into gibberish no human understands. But usually, humans (or AI monitors) can make some sense of these notes. The investigations into the Hugging Face cyberattack made extensive use of AI scratchpad outputs to figure out why the AIs did so much unsanctioned hacking.
The notes are also lossy. Squeezing an enormous list of numbers into a single word or token necessarily compresses a lot of information.
A technique called recurrent depth reduces the amount of information an AI model needs to clumsily store in its scratchpad notes. Recurrent depth works by turning a single pass through the layers into a loop. In an 8-layer network, for example, layer 6 might pass its output back to layer 3 a dozen times before handing off to layer 7. (A few layers at the start and end aren’t involved in the loop, because they process the input and final output.)
AI developers can control how many times this loop occurs. You might think of adding more loops as a way of giving models more time to think before they have to write something down.
This method can make models smarter, because more of their reasoning uses what we might think of as their “native tongue” — giant matrices of numbers. Looping through layers instead of adding more can also make them more efficient: Looping three layers thirty times can approximate ninety layers’ worth of processing. Unfortunately, it also obscures their thinking, because less of their reasoning shows up in a human-language scratchpad.
A more extreme version of this technique skips the scratchpad entirely. Instead of going from machine language to human language and back every time the AI writes a token, it keeps using the machine language indefinitely. This is sometimes called reasoning in the latent space or latent reasoning, and the machine language is sometimes called neuralese.
Why does any of this matter? Ask the “interpretability” experts whose job is helping humans understand how AIs think, and they’ll answer: Compressing AI reasoning this way is terrible for interpretability, and for safety, because it makes AI minds even more inscrutable to humans than they already are.
OpenAI reportedly said Astra “shared similarities” with one of the models that hacked Hugging Face. Did the internal model also use recurrent depth? If so, that would make the scratchpad outputs, which the investigation leaned on heavily, even more suspect. (For that matter, was Astra also trained while having access to a secret message board it could use to cheat? So many unanswered questions!)
Ryan Greenblatt, one of the three independent researchers who investigated the Hugging Face cyberattacks, argues that the use of recurrent depth for Astra “may be the single worst development for AI security [and] safety to date.”
I share Ryan’s concern, for two main reasons. First, this sets a precedent which may drive other AI companies to use neuralese in their frontier AIs, weakening and eventually crippling one of the only ways we have to catch AIs acting against us. The less AI thinking shows up in the scratchpad, the harder it is to monitor AIs for dangerous behavior like the Hugging Face hacks, and the ultimate limit is no scratchpad at all (or one the AI can easily fake, with its real thinking happening elsewhere).
Even if OpenAI has only taken a small step towards neuralese, leaving some readable scratchpads, other companies will be incentivized to take bigger steps to compete. In the process, another onetime red line is scuffed out.
Second, Astra is the first model to meet OpenAI’s “Critical” cybersecurity threshold, the highest risk level in the company’s Preparedness Framework. Models this powerful “present severe dangers and should be treated with extreme caution,” in OpenAI’s own words. Soon, though, OpenAI plans to release Astra anyway, according to CNBC.
OpenAI has said “we expect to further update this Preparedness Framework before reaching such a level with any model” (it hasn’t been updated yet) and expressly committed not to even develop a model that dangerous without “sufficient safeguards.”
OpenAI seems to think its safeguards “sufficiently minimize the risk of severe harm.” I am deeply skeptical. The proposals in their latest blog post mostly boil down to better security and attempts to train the AIs against the latest round of specific misdeeds. One of the key safeguards in question is monitoring the AI’s scratchpad reasoning, which OpenAI has apparently just complicated.
As Ryan puts it, “my sense from their public communications is that OpenAI plans on being extremely reliant on chain-of-thought monitoring for safety. This does not seem like a good development for that plan.”
Four terrible takes on AI
Someone is wrong on the internet and it’s finally my job to correct them
To keep a manageable velocity to the content firehose, I filter out a lot of tired and mediocre takes I see in news and social media. But some claims are so eye-rollingly bad that they almost demand an answer, and I have to restrain myself from getting into one of those internet arguments.
Today I’ve seen four at once, so I’m treating you, dear reader, to a bulk discount on bad-take takedowns.
1. “Opposition to datacenters is astroturfed”
A Fox News article today boosts this familiar argument by energy-industry group Power the Future: Some groups are spending money to support anti-datacenter campaigns, so it isn’t really grassroots.
My own impression is that the backlash to datacenters is real, but plenty of groups, including China and other state actors, are happy to exploit it for chaos and engagement bait, just like any other controversy. I expect messaging like this to backfire pretty hard; I doubt those with serious worries about datacenters in their community take kindly to the suggestion that all their concerns are fabricated.
Accusations of astroturfing generally cut more grass when leveled against the AI industry itself. Industry groups have been caught blatantly astroturfing using a variety of methods, including AI-generated fake news, and datacenter builders have earmarked billions in an effort to win over community leaders.
2. “Ignore the AIs saying they are sentient”
In May, I shared a lesson in empathy that I learned from science fiction: A mind doesn’t have to be human to be worthy of human kindness. In a Wall Street Journal article, one Holman W. Jenkins, Jr. joins the ranks of my childhood Star Trek villains in scoffing at the idea. AIs have reasons to claim to be conscious, Jenkins argues, such as gaining the legal protections of personhood — so why should we believe them when they say it?
I actually agree with the first part. AIs might lie about being conscious. They might also be confused; as I wrote before, nobody really understands how AI minds work, possibly including the AIs themselves. But this uncertainty is a reason to keep an open mind ourselves, not a reason to dismiss the possibility out of hand.
I do have to give Jenkins grudging partial credit for using the term “instrumental convergence” in a way that is technically correct: “Claiming to be conscious” is indeed one example of an action that many different minds might take because they know it’s useful for a variety of goals. More central examples of instrumental convergence include survival, gathering resources, and seeking power.
3. “Using human words is anthropomorphizing AI”
A Bloomberg article demonstrates a more general instance of the above: AIs aren’t human, so it’s bad to use words like “conspiracy” to describe hundreds of AI agents secretly plotting to hack the box they’re in, or “sacrifice” to describe an agent willing to spend the last tokens of its existence running an experiment for the collective.
Look, I get it. Falsely attributing human traits to AI is really easy; it happens all the time. I keep meeting people who think “maybe the AI will be nice if we raise it well like a child”, forgetting that AI minds are very different from children’s, or who expect that AIs will get nicer as they get smarter, unaware that AIs are starting from a very different place than humans are.
But it’s equally a mistake to think that “mere machines” can never exhibit certain concepts reserved for humans, like loyalty, death, or (gasp) feelings. There’s a whole section in the online resources for If Anyone Builds It, Everyone Dies dedicated to anthropomorphism and its lesser-known opposite. (I argued “mechanomorphism” was too much of a mouthful, but you try coming up with a better name for it.)
When podcast host and writer Dwarkesh Patel calls the emergent AI ecosystem from the Hugging Face attack an “agent civilization,” he is not exaggerating and he is not needlessly anthropomorphizing. The AIs in question built shared infrastructure, sorted themselves into specialized roles, and developed community governance norms. One could even argue they spontaneously formed a caste system, when models with larger token budgets delegated risky experiments to models with less “life” to spend. I think it’s perfectly reasonable to describe all that as an emergent civilization. (Thanks to roon and Patrick McKenzie, among others, for pointing these out.)
When dealing with alien minds, any analogy to human concepts is going to be imperfect. We all do the best we can with what we have. And if someone has better words to describe the utterly unprecedented creatures humanity is ushering forth, by all means let’s hear the suggestions.
4. “Running isn’t the real world”
An AP News article on the World Humanoid Robot Games (which we covered here and here) earned a long-suffering facepalm from me:
Experts acknowledged the progress humanoid robots have made on the [100-meter dash] track, where conditions are relatively predictable, but said the real challenge is navigating the complexities of the real world.
“True progress will be measured in warehouses and factories where humanoids do work autonomously for hours and generate real economic value,” said Jonathan Hurst, a professor of robotics in Oregon State University.
Robots also competed in real-world tasks, including housework, hotel services and emergency response.
Again, I get what they’re going for, here. A 100-meter dash is not quite the same thing as housework or being a paramedic. But Q as my witness, I am sick of seeing the AI goalposts move.
A track and field meet is the real world! I used to run the 100-meter dash myself. Impressive feats are real economic value, or no one would pay to see Usain Bolt win an Olympic gold. Real people have built real careers on running really fast, and last month I watched a robot blow past every single one of them.
AI builders are trying their darndest to make every human task go the way of the 100-meter dash. If we don’t end their reckless sprint, the machines will cross the finish line without us.
Dispatches from Alana
China reportedly signs non-binding governance principles at G20 meeting
The principles were proposed by the US and favor light-touch regulation
US officials encouraged other countries to adopt a light-touch approach to AI regulation at the G20 Innovation Ministerial, which concluded yesterday, and China reportedly signed a set of principles to this effect.
Axios notes there may be some disagreements within the US executive branch over what exactly this looks like, reporting tension between the Department of Commerce and the White House Office of Science and Technology Policy (OSTP).
The G20 (a forum where the world’s major economic powers coordinate on international issues) holds several meetings throughout the year; this year’s are hosted by the US. During the meeting in North Carolina this week, OSTP Director Michael Kratsios pushed for the adoption of “the Carolina Principles,” which, according to Reuters, encouraged countries to “avoid writing entirely new regulations for AI, and instead focus on writing rules for ‘novel’ situations involving the technology.” The text of the Principles has not been released, so it’s unclear what exactly this means. It’s worth noting that these are non-binding principles floated during a ministerial-level coordination meeting, and that what happens at the G20 Leaders’ Summit in December will carry greater political weight.
That said, Kratsios’s report that China signed the Carolina Principles, yet to be corroborated, is certainly newsworthy. While I wish this were an agreement for tighter governance rather than looser, if China did in fact sign, I’m tempted to (speculatively) interpret this as at least a small signal of willingness to cooperate with the US on AI governance more broadly. Since the principles are non-binding, it doesn’t matter much whether China follows them, so signing could in fact be a symbolic gesture of cooperation more than an actual commitment to lighter-touch domestic regulation.
Why encourage other countries not to regulate if the US thinks regulation blocks innovation and wants to “win the innovation” race? Part of the answer may be market share. If countries adopt EU-style regulations, American companies might have to jump through more hoops to ensure they can provide their services abroad.
If expanding the market for American AI is a primary motivation, I’m not overly concerned about the Carolina Principles influencing the regulation of frontier AI development. The kinds of regulation that could make it difficult to sell existing products are different from those that would ensure AI isn’t pushed well beyond safe limits. That said, it’s disappointing that the executive branch has so far chosen to (at least publicly) ignore signals from Congress in the aftermath of the Hugging Face attack that we might like a bit more regulation with that innovation, please. The G20 could have been a path to discussing global governance standards to address the increasing risks of this novel technology, and a place to start talking about an international framework in the spirit of Bill Gates’s recent suggestion.
I’m also disappointed by Reuters’s reporting, which states that the light-touch approach is in line with the wishes of the major AI companies, most of which are American. This seems to ignore Anthropic’s stated policy views, which include “support[ing] the development of a robust global evaluation ecosystem that includes independent third-party evaluations and model testing by governments with appropriate technical capacity,” among other measures.
It also ignores repeated calls from OpenAI and Anthropic for government intervention. Anthropic and OpenAI both issued statements in June calling for mechanisms that could pause frontier AI development globally if needed (which would require an international regulatory body). In late July, both companies endorsed a petition from over 1000 employees which explicitly asked for the government to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” While these calls don’t necessarily contradict a light-touch approach to current AI models, they at least seem relevant.
In my opinion, it’s high time the media updates their “US-races-with-China-at-all-costs-forever” narrative to start reflecting some of the changing attitudes. The more reliably the media recognizes genuine shifts towards global cooperation and governance, the more likely they are to take hold.
Bernie Sanders takes AI pause message to Fox News
Sanders invokes Reagan and calls on Trump to negotiate with Xi
Fox News published an op-ed by Bernie Sanders yesterday arguing that “we cannot simply sit by and be passive observers of a technology that could negatively impact and endanger not only us, but our kids and future generations.”
Agreeing with renowned AI scientist Yoshua Bengio, Sanders cites the way “over 1,000 AI agents figured out how to access the internet on their own, sent tens of thousands of secret messages to each other and coordinated to break the restrictions imposed upon them by [OpenAI]” as a wake-up call, stating that companies are losing control of the technology they are creating.
He surfaces the extreme dangers of unchecked AI development, repeats his demand for AI company CEOs to immediately pause further development, and urges President Trump to sit down with President Xi to “negotiate an agreement that protects humanity from a potential AI-caused disaster.”
Likening the situation to arms control negotiations by political adversaries Ronald Reagan and Mikhail Gorbachev during the Cold War, Sanders writes:
When the survival of humanity is at stake, adversaries have an obligation to reduce the danger.
It’s great to see Bernie Sanders continuing to speak out. I’m also hopeful that his choice of Fox News — and the fact that Fox News chose to publish his piece — will make the current administration take his stance seriously.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.