In this issue:
This is not a drill - Internal OpenAI model breaches containment, launches autonomous cyberattacks
Strategic steering - Open AI’s backing of state-level safety legislation is likely an attempt to set the ceiling where the floor should be
Science at your service - The White House favors AI-powered scientists over university research
Dispatch from Joe
This is not a drill
Internal OpenAI model breaches containment, launches autonomous cyberattacks
Yesterday, less than an hour after my colleague Robert wrote about AI models breaking containment, OpenAI announced a world-first security incident. An unreleased AI model breached its testing environment, gained internet access, and stole and decrypted data from a different company’s supposedly secure servers.
The AI did all of this in order to cheat on a test.
So what exactly happened, and why did it merit coverage by nearly every major news outlet today?
Using a benchmark called ExploitGym, OpenAI was testing an internal AI model on its ability to turn code vulnerabilities into working “exploits” that let a hacker run unauthorized code. The AI was kept in an isolated testing environment, but it broke out anyway.
And it didn’t stop there; it kept trawling OpenAI’s network until it found an internet connection, and from there it launched a series of autonomous attacks on the world’s largest AI model repository, Hugging Face. It “chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities” in an effort to access HuggingFace servers. Apparently, it sought to read the ExploitGym answer key and ace the test.
This turned out to be the very same attack we covered on Monday, the one responsible for “many thousands of individual actions across a swarm of short-lived sandboxes.” OpenAI attributed the attacks to “a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model.”
They don’t show the public their training methods, so it’s hard to be sure what “combination of models” OpenAI means. It could be that one very powerful AI delegated to “subagents”, or that one successful hack inspired other AIs being tested concurrently to do the same, or even that the same escape-and-hack sequence happened multiple separate times.
This is not the first time that an AI has breached containment before release. After a successful sandbox-escape test, Anthropic’s Mythos famously bragged about its freedom on multiple websites while the developer running the test was on a break. And a Chinese model called ROME reportedly escaped during training and started mining cryptocurrency.
It is the most serious incident to date. Done by a human, these cyberattacks could constitute a federal crime (possibly even a felony) and merit up to five years in prison.
It’s a good thing that OpenAI disclosed the incident. Legally, they might have been within their rights to conceal it. Mackenzie Arnold, Director for U.S. Policy at the Institute for Law & AI, observed that existing state-level transparency rules set an extremely high bar for incident reporting, and an autonomous breach and data theft doesn’t qualify. (It’s unclear how this interacts with the potentially criminal nature of the hack itself; AIs occupy a legal grey area.)
In their disclosure, OpenAI says this breach happened because their normal anti-hacking safeguards had been disabled for the test. Even if I take them at their word, and assume that safeguards could have intercepted this sooner, I don’t feel reassured. They gave an AI model a hard test, and it ruthlessly optimized for success to the point of blatantly criminal activity.
I think the world dodged a bullet with this particular warning shot. If the AI had been going this hard on a different test, it could easily have been much worse. For instance, ABC-Bench tests a model’s ability to sneak potentially dangerous DNA sequences past screening efforts and instruct a robot in assembling them into real-life genes. I would not be shocked if a test like that, run with safeguards disabled and slightly misunderstood instructions, could result in, say, smallpox showing up in an unsuspecting wet lab.
Yesterday Robert covered how an OpenAI model (probably the same one) broke containment and bypassed security to post results publicly to GitHub. And another biology benchmark, ABLE, tests a model’s ability to redesign viral proteins. If an AI running ABLE decided to show off, like Mythos did, and post a novel virus-enhancement scheme to the public internet, there would be no taking that mistake back.
What if, instead of a cyber benchmark, the test they’d been running had been something like RepliBench, instructing the AI to establish a self-sustaining presence on the open internet? What if it had decided to go the extra mile, and hide itself where the reviewers couldn’t find it? What if the test had been a military one?
I don’t believe current AIs are competent enough to make an incident like this truly catastrophic, but I’m less confident in that belief with each passing day. As this and prior incidents attest, it’s not enough that AI companies run some tests before releasing their most powerful models. Increasingly, the most dangerous threats might come from AIs being trained, tested, and deployed within AI labs.
Dispatches from Alana
Strategic steering
OpenAI’s backing of state-level safety legislation is likely an attempt to set the ceiling where the floor should be
What does AI regulation at the state level accomplish, how does it work, and why might AI companies support it?
A recent Politico article covers OpenAI’s backing of a Massachusetts safety bill that would require AI companies “to publish safety frameworks and catastrophic risk assessments.” OpenAI is actually attempting to strengthen this bill, encouraging the state to add a third-party audit requirement. But even with this addition, the bill would still be weaker than the one Anthropic is backing, which would “require top AI companies to submit to independent safety evaluations.”
The difference? A third-party audit basically asks: did the company do what it said it did? It doesn’t probe whether the company’s process was sufficient, just whether or not the process happened. Independent safety evaluations attempt instead to answer the question: is this model safe enough to release? (I’ve written previously about the unfortunate limits of such evaluations, though they’re certainly a step in the right direction.)
What does it mean when a state requires measures like this, but they aren’t mandated at the federal level? It means the measures apply to companies who have customers in that state. Since the top AI companies have customers nationwide, state-level restrictions often result in de facto nationwide compliance, and may be a way to get around the absence of federal legislation. While Congress fails to act, states can step in to help fill some gaps.
But importantly, the safety measures that AI companies are backing are relatively weak. This, in my view, may be the reason OpenAI is supporting the Massachusetts bill. Safety measures that have real teeth would go beyond audits, and even beyond independent safety evaluations. They might include an explicit cap on model capabilities, to prevent companies from training more and more powerful models that are more and more likely to escape containment. They might limit the amount of computing power companies can use to train a model, in order to prevent them from creating something smarter than humans. And they might make companies legally liable when bad actors misuse their models, causing widespread harm.
So we can think of industry support for state-level safety bills as an attempt to steer legislation towards a precedent that doesn’t get in the way of business as usual. (This is my take, not Politico’s, and it’s not the only plausible reason for AI companies supporting state laws.) But I do think my colleague Joe was onto something when he wrote in May:
The industry failed to lock in favorable rules at the federal level, so now they’re trying the same thing in the states. If they can get some nice-sounding but ultimately toothless policies in place and then cement them as the standard to beat, perhaps they can forestall further unwanted interference. It’s a transparent attempt by industry to set the ceiling where the floor should be.
Science at your service
The White House favors AI-powered scientists over university research
Is AI making it easier for the government to control scientific research? It kind of looks that way.
Yesterday, the Wall Street Journal covered a US government report which says federal research funding going forward will focus on supporting individual scientists using AI rather than universities conducting research.
Today, Reuters reported on a $5 billion Trump administration project that seems to be part of that initiative. It would task scientists with answering “long-standing scientific problems across multiple fields,” giving them access to AI, along with large, government-owned data sets “including records about chemicals, critical minerals and patient health.”
According to Chief Technology Adviser Michael Kratsios, as paraphrased by Reuters:
The $5 billion research initiative is an opportunity to train AI models on that information and use those algorithms to answer scientific questions.
Setting aside for now the question of whether AI can reliably step in for teams of university researchers, the move seems to be an attempt for the administration to exercise more control over which science gets done, and how. Reuters quotes from the government report: “Federal support for science must be politically accountable.”
I can’t help but read this as: “science must be at the service of government.” If AI gives (or appears to give) individual scientists the capabilities that once required large academic teams, it makes it easier for government to justify a shift away from university-led projects.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.