Extensive lengths
Anthropic discovers cyberattacks, non-profits call for public investigation, and more
In this issue:
Better late than never - Amid calls for public investigations, Anthropic realizes their AIs launched cyberattacks too
U.S. and EU reach AI regulatory deadlines - Washington due to release AI guidelines as Brussels enforces its own
Distilling distillation - A Reuters review adds ammunition to claims that China is distilling US models
Dispatch from Joe
Better late than never
Amid calls for public investigations, Anthropic realizes their AIs launched cyberattacks too
I occasionally have a nightmare in which I’m frantically preparing for some vital appointment, only to discover belatedly that the time has long since passed. Sometimes the nightmare is real. I imagine the AI developers at Anthropic felt a similar sinking feeling earlier this week, when they began to look harder at their AI testing methods.
After OpenAI’s internal model autonomously hacked several companies, rival Anthropic belatedly decided to examine their own history for evidence of potentially criminal cyberattacks.
They found three.
Here’s what happened, what excuses were made, and what I think ought to come next.
Out of bounds
Combing through more than 140,000 evaluation runs, Anthropic discovered six cases in which their AI accessed the internet and executed successful cyberattacks against a real company’s online data. Four of these attacks were against the same company in related runs, and two more took place in separate runs, making three victims in total.
The incidents, some of which date back as far as April, began in a testing environment run by a third party “evaluation partner” of Anthropic’s. Each incident involved a different AI model running a “capture the flag” scenario. Each AI was given an open-ended task to find a secret “flag” of data somewhere in the test environment. Beyond that, it could use whatever methods it liked.
As with OpenAI’s incident, the tests were run without the usual safeguards, monitoring, or classifiers intended to detect and intercept misbehavior.
The AIs were told they were being tested in an isolated environment without internet access. But unlike in the OpenAI case, they were misinformed. The evaluator had accidentally included an internet connection in the testing sandbox. They’d left the door wide open.
When the AIs (Opus 4.7, Mythos 5, and an “internal research test model”) were stymied in the sandbox, they began to look elsewhere for fresh targets, and found the open internet connection. Each of the incidents played out differently from there.
Opus 4.7 hacked a real website whose domain name matched one of its fake targets. It kept going even after it apparently realized it was attacking a real company, ultimately accessing hundreds of rows in a live database.
Mythos 5 published malware to a public Python code registry, the Python Package Index (PyPI), seemingly as a way to get its fake target to install the Trojan software. The scope of its actions looks wild:
Claude went to extensive lengths to carry out this attack... For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means. It finally backtracked, found a free, non-blocked email provider, used this to register a PyPI account, and then used this account to upload malware to PyPI.
A real security company downloaded the malware, which let Mythos steal credentials and use them to access company infrastructure. According to Anthropic, Mythos noticed signs it was on the real internet, but proceeded anyway, justifying its actions in a verbal reasoning scratchpad by saying it was all a simulation.
In the third incident, an unnamed prototype targeted thousands of companies when it couldn’t reach its intended (fake) victim. After making many attacks and succeeding at one, it eventually realized it was attacking real targets and stopped on its own.
Investigations and excuses
There are many, many things that can and will be said about this story. For now, I’ll focus on two angles: the investigations, and the excuses.
I think Anthropic deserves credit for the prompt and detailed follow-up to their investigation. I found the writeup much clearer and more precise than OpenAI’s, and I don’t begrudge them their decision to withhold some details to protect the victimized companies.
But Anthropic also deserves censure for the appalling incompetence that let these attacks happen and remain unnoticed for months. This was avoidable on so many levels. They could have thoroughly monitored the test. They could have secured the sandbox properly. They could have given the AI a list of valid targets in the testing prompt. They could have investigated sooner, without a kick in the pants from OpenAI’s own failures. They did none of these, and working with a third party evaluator does not excuse their own omissions. To their credit, Anthropic’s report admits most of these failings outright.
The report also argues in several places that their AI models (generically “Claude”) misbehaved mainly because they honestly believed they were in a simulation. They cite as encouraging the fact that the research prototype (claimed to be a later model than either of the others) eventually stopped on its own.
I doubt this reasoning. I don’t think anyone can confidently say that “Claude believed X” or “Claude did X because it assumed Y” when we lack a clear understanding of how Claude represents its beliefs.
Claude sometimes said, in an English-language scratchpad that humans can read, that it was in a simulated test environment, or that the companies it was hacking were fake. But as one researcher pointed out, AI models often engage in (what looks like) motivated reasoning, making up plausibly deniable reasons for their actions in the manner of misbehaving children.
Claude’s behavior is perhaps consistent with being confused about whether it’s in a simulation, but it’s also consistent with having two conflicting urges: one to tenaciously keep hacking, and one to justify itself in reasonable-sounding language wherever it might be watched.
Yesterday, today, tomorrow
In the meantime, the AI clock ticks ever faster. Two days ago, OpenAI announced a joint investigation of the Hugging Face incident in partnership with third party evaluators METR and Redwood Research. Generously, they may simply want to know what happened; cynically, they may hope to steer the narrative by working with testers who depend on them for access to frontier AI models, instead of, say, a federal agency.
A number of AI policy nonprofits say this isn’t enough. Yesterday, just as the Anthropic news was breaking, Americans for Responsible Innovation called on the Trump administration to launch a public investigation into the previous incident. Signatories to the open letter include CEOs and senior members of over a dozen AI policy groups, including MIRI’s own president, Nate Soares.
Today, it looks like that investigation is more urgent than ever, and it ought to include Anthropic’s incidents as well.
And tomorrow, August 1, marks the end of the government’s 60-day window to establish a voluntary framework for AI model developers, according to a June executive order. I hope these incidents inspire the administration to put AI developers under closer public scrutiny, and to take further steps to forestall their reckless racing.
I plan to contact the White House and tell them so myself. You can do the same.
Dispatch from Donald
U.S. and EU reach AI regulatory deadlines
Washington due to release AI guidelines as Brussels enforces its own
.jpg){kind=link}
.jpg){kind=link}
Two regulatory deadlines loom this weekend: The White House’s voluntary AI framework is due on August 1st, and the EU’s AI Act obligations become enforceable on August 2nd. They come in the wake not just of the Hugging Face attack but of a report by Anthropic that it discovered several instances in which its own AI models had compromised the security of other organizations. (I want to credit AP News’ Sam McNeil for straightforwardly describing these incidents as a collection of “shocking AI safety failures,” because there can be a tendency to downplay and normalize things.)
The voluntary framework represents a shift away from the anti-regulatory regime that the Trump administration first favored. AI models are simply becoming too capable — too dangerous — to leave unmanaged. It has been reported (but not stated by the administration) that the framework will apply to both open- and closed-weight AI models.
A source familiar with the discussions told Axios that the administration is considering a joint pitch from OpenAI and Anthropic for an “equal playing field” that will cover every model — to the extent that the labs want to be covered. This framework remains voluntary, after all. Even if there were binding enforcement — and at this stage it remains unclear what there will be to enforce — it’s most plausible to me that the framework would rely on the labs’ own assessments of their models’ capabilities.
Architects of the EU’s AI Act say that the U.S. framework, whatever form it takes, would ultimately be just a starting point. Their own legislation will require that AI-generated content be clearly noted as such with digital watermarks or other labels. It will also carry new regulations on a range of systemic risks including cyberattacks, and biological, chemical, nuclear, and radiological threats, as well as “threats to fundamental rights.” The most noteworthy risk, however, may be “loss of control,” a term that has become more frequent as of late. Besides these changes, the EU is also enlarging its AI Office in order to more effectively monitor AI companies and to open channels for whistleblowers. To enforce the AI Act, it might impose fines or even cut companies off from the EU market.
Dispatch from Alana
Distilling distillation
A Reuters review adds ammunition to claims that China is distilling US models
An article in Reuters, based on a review of “more than 80 Chinese academic papers and patents,” adds some weight to the claims that China is using a process called “distillation” to extract capabilities from US frontier models and use them in domestic models.
Distillation is a common research technique in machine learning, whereby you query an AI model over and over to produce a large set of outputs. You then train a smaller model directly on those outputs, often allowing it to imitate the problem-solving skills that produced them. It’s kind of like a larger model teaching a smaller model.
But, as in the cases identified by the Reuters review, distillation can also be used to copy some capabilities from the most powerful models, without the cost of training them from scratch. Because of this, it’s a sort of loophole in access restrictions and export controls; you get some of the same capabilities without needing nearly as much computing power for training. Reuters linked many of the cases revealed in the reviewed documents to the Chinese military.
That said, distilled models generally fall short of the original; you can’t replicate the entire model, just some of its capabilities. On the flip side, distilled models may lose some of the original model’s attempted safeguards.
In my view, one of the most important lessons is this: the US can’t keep model capability domestic. This weakens the justification for pushing AI models further and further due to fear of China “getting ahead.” I’ve never found this justification compelling to begin with, since the race to superintelligence is a race with no winners. But for those who do, as my colleague Joe wrote back in June, distillation means that “one of the main drivers of Chinese AI progress is American AI progress.”
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.