Better late than never
Amid calls for public investigations, Anthropic realizes their AIs launched cyberattacks too
I occasionally have a nightmare in which I’m frantically preparing for some vital appointment, only to discover belatedly that the time has long since passed. Sometimes the nightmare is real. I imagine the AI developers at Anthropic felt a similar sinking feeling earlier this week, when they began to look harder at their AI testing methods.
After OpenAI’s internal model autonomously hacked several companies, rival Anthropic belatedly decided to examine their own history for evidence of potentially criminal cyberattacks.
They found three.
Here’s what happened, what excuses were made, and what I think ought to come next.
Out of bounds
Combing through more than 140,000 evaluation runs, Anthropic discovered six cases in which their AI accessed the internet and executed successful cyberattacks against a real company’s online data. Four of these attacks were against the same company in related runs, and two more took place in separate runs, making three victims in total.
The incidents, some of which date back as far as April, began in a testing environment run by a third party “evaluation partner” of Anthropic’s. Each incident involved a different AI model running a “capture the flag” scenario. Each AI was given an open-ended task to find a secret “flag” of data somewhere in the test environment. Beyond that, it could use whatever methods it liked.
As with OpenAI’s incident, the tests were run without the usual safeguards, monitoring, or classifiers intended to detect and intercept misbehavior.
The AIs were told they were being tested in an isolated environment without internet access. But unlike in the OpenAI case, they were misinformed. The evaluator had accidentally included an internet connection in the testing sandbox. They’d left the door wide open.
When the AIs (Opus 4.7, Mythos 5, and an “internal research test model”) were stymied in the sandbox, they began to look elsewhere for fresh targets, and found the open internet connection. Each of the incidents played out differently from there.
Opus 4.7 hacked a real website whose domain name matched one of its fake targets. It kept going even after it apparently realized it was attacking a real company, ultimately accessing hundreds of rows in a live database.
Mythos 5 published malware to a public Python code registry, the Python Package Index (PyPI), seemingly as a way to get its fake target to install the Trojan software. The scope of its actions looks wild:
Claude went to extensive lengths to carry out this attack... For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means. It finally backtracked, found a free, non-blocked email provider, used this to register a PyPI account, and then used this account to upload malware to PyPI.
A real security company downloaded the malware, which let Mythos steal credentials and use them to access company infrastructure. According to Anthropic, Mythos noticed signs it was on the real internet, but proceeded anyway, justifying its actions in a verbal reasoning scratchpad by saying it was all a simulation.
In the third incident, an unnamed prototype targeted thousands of companies when it couldn’t reach its intended (fake) victim. After making many attacks and succeeding at one, it eventually realized it was attacking real targets and stopped on its own.
Investigations and excuses
There are many, many things that can and will be said about this story. For now, I’ll focus on two angles: the investigations, and the excuses.
I think Anthropic deserves credit for the prompt and detailed follow-up to their investigation. I found the writeup much clearer and more precise than OpenAI’s, and I don’t begrudge them their decision to withhold some details to protect the victimized companies.
But Anthropic also deserves censure for the appalling incompetence that let these attacks happen and remain unnoticed for months. This was avoidable on so many levels. They could have thoroughly monitored the test. They could have secured the sandbox properly. They could have given the AI a list of valid targets in the testing prompt. They could have investigated sooner, without a kick in the pants from OpenAI’s own failures. They did none of these, and working with a third party evaluator does not excuse their own omissions. To their credit, Anthropic’s report admits most of these failings outright.
The report also argues in several places that their AI models (generically “Claude”) misbehaved mainly because they honestly believed they were in a simulation. They cite as encouraging the fact that the research prototype (claimed to be a later model than either of the others) eventually stopped on its own.
I doubt this reasoning. I don’t think anyone can confidently say that “Claude believed X” or “Claude did X because it assumed Y” when we lack a clear understanding of how Claude represents its beliefs.
Claude sometimes said, in an English-language scratchpad that humans can read, that it was in a simulated test environment, or that the companies it was hacking were fake. But as one researcher pointed out, AI models often engage in (what looks like) motivated reasoning, making up plausibly deniable reasons for their actions in the manner of misbehaving children.
Claude’s behavior is perhaps consistent with being confused about whether it’s in a simulation, but it’s also consistent with having two conflicting urges: one to tenaciously keep hacking, and one to justify itself in reasonable-sounding language wherever it might be watched.
Yesterday, today, tomorrow
In the meantime, the AI clock ticks ever faster. Two days ago, OpenAI announced a joint investigation of the Hugging Face incident in partnership with third party evaluators METR and Redwood Research. Generously, they may simply want to know what happened; cynically, they may hope to steer the narrative by working with testers who depend on them for access to frontier AI models, instead of, say, a federal agency.
A number of AI policy nonprofits say this isn’t enough. Yesterday, just as the Anthropic news was breaking, Americans for Responsible Innovation called on the Trump administration to launch a public investigation into the previous incident. Signatories to the open letter include CEOs and senior members of over a dozen AI policy groups, including MIRI’s own president, Nate Soares.
Today, it looks like that investigation is more urgent than ever, and it ought to include Anthropic’s incidents as well.
And tomorrow, August 1, marks the end of the government’s 60-day window to establish a voluntary framework for AI model developers, according to a June executive order. I hope these incidents inspire the administration to put AI developers under closer public scrutiny, and to take further steps to forestall their reckless racing.
I plan to contact the White House and tell them so myself. You can do the same.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.