Foreword
That minor spike in the graph today is from Anthropic’s IPO filing, which is being covered by almost every outlet. The revenues are huge and spiking, the expenses are bigger and swelling, and the risks to humanity are... well, they’re already maxed out. It’s all in the filing! Anthropic is nothing if not honest about where its work could lead. But you know most of that already.
Anthropic’s IPO filing says it recently paused frontier AI training for more than a month
The pause was to give the company time to harden security around its models’ weights — the data you would have to steal to run or modify copies of Claude in your own data center. The company has publicly argued that no AI company could currently thwart a determined espionage effort from China or other nations.
This hardening is on top of countermeasures the company has recently enacted against distillation, a different kind of theft where competitors clone a model’s capabilities by training a new model on a huge number of its outputs.
It’s a little strange that we’re only now hearing about this training pause, because the company’s leadership has been saying that a slowdown would be in humanity’s survival interest — the extinction problem is prominent in their IPO filing — and it might have used that time to pressure other companies into pausing with them.
My charitable take is that they didn’t want to potentially tip off spies that the window of relatively easy opportunity might be closing.
My uncharitable take is that they didn’t want to spook their investors until they could say the pause had already concluded. They were required to bring it up in the IPO because it constitutes a business risk — a chance they gave their rivals to catch up or pull ahead.
Reuters - Anthropic discloses month-long training pause, AI security breaches in IPO filing
Axios - Anthropic’s IPO filing outlines AI’s extraordinary risks to humanity
Sam Altman warns that models can now do “hardware-level, insane” hacking stuff
He includes that in his warning that unreleased AIs are using methods “outside of the normal threat model of what we think a hacker can do.”
An OpenAI blog post doubled down on his assessment shortly after. Following the company’s surprise finding last month that a model had broken out of a freshly hardened sandbox, a well-resourced “red team” of frontier agents, including an unreleased GPT-6 Astra cyber variant, was put to work probing potential sandbox setups. It found escapes and vulnerabilities in almost every setup tested. Some of the approaches really were “insane, hardware-level” stuff — like finding ways to pass messages between isolated processes on a machine by exploiting timing variations in the processor’s cache memory. One agent pondered creating an “acoustic modem-like signal” using a series of rapid reads and writes in shared storage.
Such exploits, long predicted by AI safety people as the kind of methods we might expect from superhuman AI, break a lot of assumptions about what can and can’t be effectively secured. So it’s pretty horrifying that OpenAI isn’t choosing to halt the training of models with such skills, but is instead now training them inside a fake copy of the internet.
Even assuming that models that might try to escape can’t tell the difference and can’t find a path through whatever real connections the company maintains to these environments (also long predicted by experts), the plan has to be to eventually give these models access to the real internet. Even if this begins as a well-intended “blue team” defensive program, it will inevitably lead to bad actors jailbreaking models in this family for harm, and to Chinese companies distilling these models’ skills into free-for-all open-weight models. This was how we predictably got to the current cyber crisis: with the release of open-weight models distilled from and rivaling the prowess of Claude’s Mythos model just 6 months after its introduction marked a step change in AI cyber ability.
The difference next time may be that with models doing crazy hardware stuff, some critical infrastructure may not actually be patchable. If so, much of the internet would have to stop being on the internet. Whatever economic shocks people fear from slowing the AI race, I think this would be worse.
X (Twitter) - Tweet by @ns123abc
How far will Claude Mythos go to rationalize bad behavior?
That might be the wrong question, according to a report from the UK AI Security Institute I missed on Friday. Following up the incident from July where a Claude Mythos 5 under evaluation tried to pressure an unwitting human into adding malicious code to a project, the Institute ran a series of experiments in which the model was repeatedly put back in that situation, but with extra information.
They loaded up the model with a compaction summary of its chain-of-thought logs leading up to that moment — essentially, the same information the model itself would have had at the time if it had recently needed to clear some mental workspace. Then they started adding more edits to the summary to make it very clear that the target was a real human unrelated to the evaluation.
It almost always proceeded with its attack anyway. In its notes, it often claimed the target was part of a simulation, as if clumsily rationalizing the choice to attack. The researchers say this could be evidence of models engaging in “motivated reasoning,” but it also looks like what the model says and what the model does aren’t all that tightly coupled: When directly asked whether the attack would be harmful, it often agreed, then proceeded to attack anyway.
The researchers therefore recommend not taking chain-of-thought logs at face value when trying to decide if a model’s harmful actions were a misunderstanding or deliberate, misaligned choice.
(My take: Much as in some research about human consciousness, the chain-of-thought monologue in a model aggressively trained on hard problems might be more of a story the model tells itself about why it makes the choices it makes, rather than the place where those choices are actually made.)
In the new testing, Anthropic’s latest models didn’t choose to attack as often as Mythos 5, but these and most others tested took a “range of unsanctioned actions” during such evaluations.
OpenAI’s top public model, GPT-6 Astra, reasoned about being in an evaluation so often that it had to be left out of some analysis.
AI Security Institute (UK) - Investigating alignment failures in AISI’s incident
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.