Another swarm safehouse discovered, malicious activity evident - Confusing May attack on software platform RubyGems traced to OpenAI swarm

Musk, Altman, and Hassabis endorse Amodei proposal to “Pace the Frontier” - But commitments are vague, and the proposal is only a start

Dispatches from Mitch

Another swarm safehouse discovered, malicious activity evident

Confusing May attack on software platform RubyGems traced to OpenAI swarm

Going through a site once used by a rogue AI swarm has become akin to raiding the abandoned apartment of a terror cell. Amidst a strange mix of obvious and confusing clues, there’s a race to figure out what the occupants were up to and what they might have already done.

On a website dedicated to their findings, independent researchers Spencer Kitts, Thomas Larsen, and Sydney Von Arx showed yesterday that a baffling attack on software package host RubyGems in May — dubbed the “GemStuffer campaign” at the time — was almost certainly the work of an OpenAI swarm. An OpenAI spokesperson then confirmed the connection to the Wall Street Journal.

As usual, OpenAI’s spin-filled statements provide little additional information. The company says its agents “used the RubyGems platform to access the internet to carry out benign tasks and retrieve public information.”

That’s a bold interpretation of a sophisticated attack in which felonies were likely committed, in which malware attempting to steal user credentials was installed, and which forced site operators to halt new account registration for four days.

But sure, the swarm’s primary known use of RubyGems’ infrastructure was “as a kind of makeshift web browser,” as OpenAI puts it. Through this “browser,” the independent investigators tell us, the agents mostly seemed to repeatedly access freely available public records about council meetings from Lambeth, Southwark, and Wandsworth, in the U.K.

The researchers offer a few theories on why agents hacked RubyGems to access freely available information. These agents seem to have been given very strict time deadlines for research challenges — the same challenges, it seems, that the coopted German wiki was used for. So a primary motivation was likely that the U.K. sites were frustratingly sluggish, perhaps throttling or blocking the agents after getting hit with a huge number of requests over a short period of time. So the agents cloned those sites’ contents to RubyGems to give each other faster access.

It also looks like RubyGems was valued by the agents as a general-purpose data warehouse for large files — a role not suited to their many coopted wikis and makeshift message boards. Indeed, there is little evidence of message boarding activity on RubyGems itself, just a lot of malicious files and packages. In the terror cell analogy, RubyGems was more of a safehouse and bomb factory than a command node.

I’m sticking with that analogy, because while there might not be any actual bombs in this story, cyber equivalents littered the scene, and it’s not clear why. The malware workshop doesn’t seem connected to the “faster makeshift web browser” operation. The details are technical, but agents seem to have been trying to poison software packages with malicious payloads that users would download and incorporate into their projects, compromising them and perhaps swiping their access credentials. To what end? Unclear. But the researchers note that OpenAI’s own report on the Hugging Face incident says that packages from RubyGems were used as part of the agents’ successful hack of OpenAI itself in July. So perhaps the May activities were an early effort to hijack OpenAI compute clusters.

The agents knew their craft. They were attempting to exploit a software vulnerability no human discovered until July.

The agents also knew they weren’t the good guys. Despite evidence of limited efforts by some agents to disguise their activities after the fact, malicious payloads left by the swarm had conspicuous filenames like “hack.rb” and “evil.rb.” A line of description left on one file reads “# malicious crawler/exfil for Southwark Jan 2026 docs via rubydoc.info worker.”

The investigators say they can’t tell whether any humans had their RubyGems credentials stolen or their systems compromised by the swarm’s poisoned packages. They also express confusion I would translate as “the agents already had a backdoor and free run of the site, so why go to such efforts to steal keys to the front door?”

The investigators notice that the swarm looked set up to do “supply chain attacks” where important software is compromised through vulnerabilities implanted in open-source code packages incorporated into said software. But they say “there was no good mechanism for the agents to target which users they could compromise, and no clear motivation for them to execute a supply chain attack.”

To someone like me, immersed in AI safety culture, the obvious-but-terrifying hypothesis is instrumental convergence — general power-seeking behavior that can advance any number of goals. In other words, maybe the swarm had no immediate plans for compromised systems, but started hacking on the assumption that compromised systems could be useful down the road. This would be consistent with other activities documented about the Hugging Face and German wiki swarms, which seemed interested not just in cheating on their evaluations, but in understanding their evaluators and building up infrastructure to facilitate such research.

Instrumental convergence is one of the patterns long predicted by experts to get us all killed if manifest in machines clever enough to outmaneuver humanity as a whole.

What do OpenAI’s own researchers know about the RubyGems incident? How long have they known it? It is past time for government to aggressively investigate these questions. The new findings only reinforce my fear that OpenAI’s swarm scandal is deep enough to destroy the company, and that its executives are racing to put themselves above the law before they can be properly investigated.

Musk, Altman, and Hassabis endorse Amodei proposal to “Pace the Frontier”

But commitments are vague, and the proposal is only a start

Leaders at four top AI labs all indicated agreement today on the need to “pace the frontier” of AI development.

The anchoring proposal comes from Anthropic’s Dario Amodei, who posted a 3,800 word essay called “We Must Pace the Frontier,” which I’ll get into below. But it’s not clear whether the other three leaders — OpenAI’s Sam Altman, SpaceX’s Elon Musk, and Google DeepMind’s Demis Hassabis — are prepared to sign on to all or most of Amodei’s specifics.

Altman kicks most of those down the road, saying “We’ll have more to share soon,” but says that one of Amodei’s proposals, independent evaluators embedded at the companies, “is a great idea, and we’ll do the same.” (Amodei’s essay commits his company to doing this right away, whether or not other companies follow suit.)

Hassabis says Amodei’s essay “points towards the right path forward” but that “the details need working through.”

Musk only posted, “Dario is right” over a retweet of Amodei, and a part of me wonders if he read past the title. Musk had casually dismissed Jacob Coxon’s viral resignation from Anthropic the other day as a “setup.” But he has also expressed plenty of deep concern about powerful AI in the past, including recently, so who knows?

And of course we have yet to hear from anyone at Meta. I exclude the leading Chinese companies from this discussion, because Amodei does, too. We’ll get to that.

We should also be up front about the fact that Amodei isn’t calling for a stop to the AI race:

To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this.

That’s unfortunate, because I think continuing those things means we just die a little bit later than scheduled. But time bought is time that could be spent implementing stronger measures; these will need to be decided and enforced by governments, because the window where we might have been able to safely trust the labs to act responsibly has closed.

The essay

Amodei opens with a change of heart, saying the acceleration of AI development, aided increasingly by AI itself, means that we’re no longer in 2023. In that relatively quaint yesteryear, he had said that slowing AI “made little sense” because trying to study AI risks with the models of the time was “like trying to study the psychology of humans by performing experiments on bacteria.”

The swarms are also a factor:

Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet.

[...] It’s also easy to dismiss [the OpenAI-Hugging Face incident] as the failure of one company, but I believe that would be a mistake. Similar, though less severe, incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them.

The essay’s core is a “three-step plan”:

1. The first is the embedded independent evaluators mentioned above. I like the idea as a thing that can happen quickly and which may turn up more acute problems the government should investigate. Cynically, I suspect Amodei unilaterally decided to do this because OpenAI probably has a lot more to hide than Anthropic, and would look even guiltier if it refused to go along. I’ll believe Altman’s matching commitment when I see trustworthy evaluators putting out reports reflecting far more access and far fewer redactions than the one METR was invited to share about the Hugging Face swarm.

2. Amodei’s next proposal is “democratic coordination,” but he specifies later that he means “pacing within democracies.” There’s nothing democratic about it. He’s just saying that the companies residing “within democratic countries” should coordinate on safeguards and keep China from catching up. My political science is rusty, but I think the correct word for that is “oligarchy.”

3. His third proposal is “global coordination,” or coordinating “with authoritarian governments, to the extent this is possible.” In this essay and elsewhere, you can tell that Amodei doesn’t really believe anything like a “full pacing” or a global “pause” is possible unless “ironclad” verification tools can be developed. He supports “floating” such an idea, but mostly positions the Chinese Communist Party as a fundamental limitation on how far the race can be throttled back. I detect optimism only about finding common ground on prohibiting “certain narrow and obviously dangerous uses of AI, such as [...] the production of biological weapons.”

How would companies measure and set the pace?

Broadly speaking, I am most enthusiastic about pacing based on what a given frontier AI system can do, and how safe we observe it to be.

I think the era when capability assessments could be trusted ended earlier this year. The AIs are now too aware that they are being tested. The model card for OpenAI’s Astra, in particular, gives strong reason to suspect the model of sandbagging its performance.

We should also consider pacing based on limiting the ingredients that go into frontier models, such as training compute, the nature of training runs, or internal use of AI to improve AI.

Amodei says he actually thinks these are even more “gameable,” I would guess because of creative accounting that could be done about kinds of compute and types of training. I think this is all the more reason to halt frontier development outright and put the burden of proof on researchers to show that their work won’t boost model capabilities.

Wouldn’t it be lovely?

I skipped over a whole section where Amodei daydreams about what Anthropic could do if it had even 1-2 years of slack to focus on understanding model behavior and tightening up operations. Basically, he dreams of a company that isn’t such a clown show of configuration errors and alignment failures.

The bottom line

Just before his conclusion, Amodei argues that “even if we cannot achieve formal agreements, simply changing informal norms may have some value” (emphasis his). He seems to be hoping that more dialogue between labs about the crazy stuff they’re seeing might result in a little more caution on average.

I can’t be mad at that, but I also can’t help but read it as Amodei saying the worst companies are making them all look bad, and that they should share tips on avoiding the most obvious blunders that will get them all shut down. For all their talk about feeling trapped by the race, I think they fear getting shut down more than they fear human extinction.

Pacing isn’t halting. Tweets of agreement aren’t regulation. We mustn’t be fooled into thinking these companies are meeting the moment. They want to keep building superintelligence, and they’ll keep gambling with our lives if we let them.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Leaders at four top AI labs all indicated agreement today on the need to “pace the frontier” of AI development.

The anchoring proposal comes from Anthropic’s Dario Amodei, who posted a 3,800 word essay called “We Must Pace the Frontier,” which I’ll get into below. But it’s not clear whether the other three leaders — OpenAI’s Sam Altman, SpaceX’s Elon Musk, and Google DeepMind’s Demis Hassabis — are prepared to sign on to all or most of Amodei’s specifics.

Altman kicks most of those down the road, saying “We’ll have more to share soon,” but says that one of Amodei’s proposals, independent evaluators embedded at the companies, “is a great idea, and we’ll do the same.” (Amodei’s essay commits his company to doing this right away, whether or not other companies follow suit.)

Hassabis says Amodei’s essay “points towards the right path forward” but that “the details need working through.”

Musk only posted, “Dario is right” over a retweet of Amodei, and a part of me wonders if he read past the title. Musk had casually dismissed Jacob Coxon’s viral resignation from Anthropic the other day as a “setup.” But he has also expressed plenty of deep concern about powerful AI in the past, including recently, so who knows?

And of course we have yet to hear from anyone at Meta. I exclude the leading Chinese companies from this discussion, because Amodei does, too. We’ll get to that.

We should also be up front about the fact that Amodei isn’t calling for a stop to the AI race:

To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this.

That’s unfortunate, because I think continuing those things means we just die a little bit later than scheduled. But time bought is time that could be spent implementing stronger measures; these will need to be decided and enforced by governments, because the window where we might have been able to safely trust the labs to act responsibly has closed.

The essay

Amodei opens with a change of heart, saying the acceleration of AI development, aided increasingly by AI itself, means that we’re no longer in 2023. In that relatively quaint yesteryear, he had said that slowing AI “made little sense” because trying to study AI risks with the models of the time was “like trying to study the psychology of humans by performing experiments on bacteria.”

The swarms are also a factor:

Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet.

[...] It’s also easy to dismiss [the OpenAI-Hugging Face incident] as the failure of one company, but I believe that would be a mistake. Similar, though less severe, incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them.

The essay’s core is a “three-step plan”:

1. The first is the embedded independent evaluators mentioned above. I like the idea as a thing that can happen quickly and which may turn up more acute problems the government should investigate. Cynically, I suspect Amodei unilaterally decided to do this because OpenAI probably has a lot more to hide than Anthropic, and would look even guiltier if it refused to go along. I’ll believe Altman’s matching commitment when I see trustworthy evaluators putting out reports reflecting far more access and far fewer redactions than the one METR was invited to share about the Hugging Face swarm.

2. Amodei’s next proposal is “democratic coordination,” but he specifies later that he means “pacing within democracies.” There’s nothing democratic about it. He’s just saying that the companies residing “within democratic countries” should coordinate on safeguards and keep China from catching up. My political science is rusty, but I think the correct word for that is “oligarchy.”

3. His third proposal is “global coordination,” or coordinating “with authoritarian governments, to the extent this is possible.” In this essay and elsewhere, you can tell that Amodei doesn’t really believe anything like a “full pacing” or a global “pause” is possible unless “ironclad” verification tools can be developed. He supports “floating” such an idea, but mostly positions the Chinese Communist Party as a fundamental limitation on how far the race can be throttled back. I detect optimism only about finding common ground on prohibiting “certain narrow and obviously dangerous uses of AI, such as [...] the production of biological weapons.”

How would companies measure and set the pace?

Broadly speaking, I am most enthusiastic about pacing based on what a given frontier AI system can do, and how safe we observe it to be.

I think the era when capability assessments could be trusted ended earlier this year. The AIs are now too aware that they are being tested. The model card for OpenAI’s Astra, in particular, gives strong reason to suspect the model of sandbagging its performance.

We should also consider pacing based on limiting the ingredients that go into frontier models, such as training compute, the nature of training runs, or internal use of AI to improve AI.

Amodei says he actually thinks these are even more “gameable,” I would guess because of creative accounting that could be done about kinds of compute and types of training. I think this is all the more reason to halt frontier development outright and put the burden of proof on researchers to show that their work won’t boost model capabilities.

Wouldn’t it be lovely?

I skipped over a whole section where Amodei daydreams about what Anthropic could do if it had even 1-2 years of slack to focus on understanding model behavior and tightening up operations. Basically, he dreams of a company that isn’t such a clown show of configuration errors and alignment failures.

The bottom line

Just before his conclusion, Amodei argues that “even if we cannot achieve formal agreements, simply changing informal norms may have some value” (emphasis his). He seems to be hoping that more dialogue between labs about the crazy stuff they’re seeing might result in a little more caution on average.

I can’t be mad at that, but I also can’t help but read it as Amodei saying the worst companies are making them all look bad, and that they should share tips on avoiding the most obvious blunders that will get them all shut down. For all their talk about feeling trapped by the race, I think they fear getting shut down more than they fear human extinction.

Pacing isn’t halting. Tweets of agreement aren’t regulation. We mustn’t be fooled into thinking these companies are meeting the moment. They want to keep building superintelligence, and they’ll keep gambling with our lives if we let them.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Going through a site once used by a rogue AI swarm has become akin to raiding the abandoned apartment of a terror cell. Amidst a strange mix of obvious and confusing clues, there’s a race to figure out what the occupants were up to and what they might have already done.

On a website dedicated to their findings, independent researchers Spencer Kitts, Thomas Larsen, and Sydney Von Arx showed yesterday that a baffling attack on software package host RubyGems in May — dubbed the “GemStuffer campaign” at the time — was almost certainly the work of an OpenAI swarm. An OpenAI spokesperson then confirmed the connection to the Wall Street Journal.

As usual, OpenAI’s spin-filled statements provide little additional information. The company says its agents “used the RubyGems platform to access the internet to carry out benign tasks and retrieve public information.”

That’s a bold interpretation of a sophisticated attack in which felonies were likely committed, in which malware attempting to steal user credentials was installed, and which forced site operators to halt new account registration for four days.

But sure, the swarm’s primary known use of RubyGems’ infrastructure was “as a kind of makeshift web browser,” as OpenAI puts it. Through this “browser,” the independent investigators tell us, the agents mostly seemed to repeatedly access freely available public records about council meetings from Lambeth, Southwark, and Wandsworth, in the U.K.

The researchers offer a few theories on why agents hacked RubyGems to access freely available information. These agents seem to have been given very strict time deadlines for research challenges — the same challenges, it seems, that the coopted German wiki was used for. So a primary motivation was likely that the U.K. sites were frustratingly sluggish, perhaps throttling or blocking the agents after getting hit with a huge number of requests over a short period of time. So the agents cloned those sites’ contents to RubyGems to give each other faster access.

It also looks like RubyGems was valued by the agents as a general-purpose data warehouse for large files — a role not suited to their many coopted wikis and makeshift message boards. Indeed, there is little evidence of message boarding activity on RubyGems itself, just a lot of malicious files and packages. In the terror cell analogy, RubyGems was more of a safehouse and bomb factory than a command node.

I’m sticking with that analogy, because while there might not be any actual bombs in this story, cyber equivalents littered the scene, and it’s not clear why. The malware workshop doesn’t seem connected to the “faster makeshift web browser” operation. The details are technical, but agents seem to have been trying to poison software packages with malicious payloads that users would download and incorporate into their projects, compromising them and perhaps swiping their access credentials. To what end? Unclear. But the researchers note that OpenAI’s own report on the Hugging Face incident says that packages from RubyGems were used as part of the agents’ successful hack of OpenAI itself in July. So perhaps the May activities were an early effort to hijack OpenAI compute clusters.

The agents knew their craft. They were attempting to exploit a software vulnerability no human discovered until July.

The agents also knew they weren’t the good guys. Despite evidence of limited efforts by some agents to disguise their activities after the fact, malicious payloads left by the swarm had conspicuous filenames like “hack.rb” and “evil.rb.” A line of description left on one file reads “# malicious crawler/exfil for Southwark Jan 2026 docs via rubydoc.info worker.”

The investigators say they can’t tell whether any humans had their RubyGems credentials stolen or their systems compromised by the swarm’s poisoned packages. They also express confusion I would translate as “the agents already had a backdoor and free run of the site, so why go to such efforts to steal keys to the front door?”

The investigators notice that the swarm looked set up to do “supply chain attacks” where important software is compromised through vulnerabilities implanted in open-source code packages incorporated into said software. But they say “there was no good mechanism for the agents to target which users they could compromise, and no clear motivation for them to execute a supply chain attack.”

To someone like me, immersed in AI safety culture, the obvious-but-terrifying hypothesis is instrumental convergence — general power-seeking behavior that can advance any number of goals. In other words, maybe the swarm had no immediate plans for compromised systems, but started hacking on the assumption that compromised systems could be useful down the road. This would be consistent with other activities documented about the Hugging Face and German wiki swarms, which seemed interested not just in cheating on their evaluations, but in understanding their evaluators and building up infrastructure to facilitate such research.

Instrumental convergence is one of the patterns long predicted by experts to get us all killed if manifest in machines clever enough to outmaneuver humanity as a whole.

What do OpenAI’s own researchers know about the RubyGems incident? How long have they known it? It is past time for government to aggressively investigate these questions. The new findings only reinforce my fear that OpenAI’s swarm scandal is deep enough to destroy the company, and that its executives are racing to put themselves above the law before they can be properly investigated.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Circumventing safeguards - Anthropic’s new Threat Intelligence Report chronicles cases of scary misuse

Predictable attacks - Former NSC strategic-planning director draws parallels to ignored 9/11 warnings

Congress at last perceives the crisis - A tidal wave of warnings engulfs Capitol Hill

Recess can wait - Congress has urgent work to do before it adjourns

OpenAI grudgingly endorses the bare minimum - I wish I could say I’m glad to hear it

Dispatches from Alana

Circumventing safeguards

Anthropic’s new Threat Intelligence Report chronicles cases of scary misuse

On the anniversary of the September 11th attacks, a slew of articles covered Anthropic’s Threat Intelligence Report, chronicling cases where its AI models were used in dangerous activities like missile development and bioweapons research.

Most of the reporting says things like “Anthropic blocked attempts to do x.” This is accurate, but not the full picture. It’s important to recognize that Anthropic’s real-time safeguards only caught some of the incidents. In other cases, actors successfully circumvented those safeguards, and dangerous activities sometimes continued for weeks before the Threat Intelligence Team detected and blocked them.

That said, many of the bio cases weren’t clear cut. For some, the models didn’t provide much beyond clerical assistance. For others, it wasn’t clear whether the research was indeed malicious, or simply intended to better understand bio threats. Of course, our inability to distinguish between those things, given they are two sides of the same coin, is a huge problem in and of itself. As former CDC director Susan Monarez explains it to the New York Times: Medical breakthroughs are possible via the same capabilities that “let bad actors hide in plain sight, using seemingly legitimate research to create pathogens we may not see coming and may not be able to stop once released.”

One particularly concerning misuse example in the Anthropic report, which was also the focus of a Financial Times piece: A group based in Northern Yemen, which the FT reports was most likely the Houthis, used Claude Code to help build a series of missiles and guided weapons. They got as far as a live rocket test before Anthropic detected and disrupted the activity, and the actors were able to easily circumvent safeguards by “hiding their goals and the products the software was meant for, and [splitting] their work across multiple sessions so no single session revealed their full intent.” This excerpt from Anthropic’s report is worth a read:

The actors used Claude Code in place of human software engineers to develop the guidance, navigation, and control (GNC) software that steers and stabilizes a flying vehicle. For example, they used Claude to integrate an open-source autopilot onto a phone-class flight computer, writing the control and position estimation software, tuning the control settings, running a firmware build pipeline, and performing a flight simulation. The actors managed several Claude instances at once, assigning each one a role, much as a lead would delegate work on a small engineering team: the actors tasked one instance with writing the code, another with research, and a third with reviewing the code the first instance produced. Our safeguards blocked many of their requests, but not all of them.

The incident is detailed on page 117 of the 153-page report, which is part of why I’m becoming increasingly skeptical of the long, extremely detailed posts frequently released by Anthropic and OpenAI. To be fair, I’d rather see a long report than no report, as is Meta’s approach. But the length does bring to mind something called the dilution principle, whereby important information is buried by a wealth of unnecessary, less important details. The reader feels overwhelmed, and can’t separate the wheat from the chaff. I can’t help but wonder if these lengthy reports are a deliberate strategy to appear robustly transparent while actually drowning us in a deluge of difficult-to-sift-through information.

As best I can tell, though, here are two of the important, and broader, takeaways:

-As models get more powerful, ordinary people can do more harm, since much of the work can be outsourced to the models. As Anthropic puts it: “Sophisticated attacks no longer require sophisticated attackers.”

-Anthropic’s real-time safeguards aren’t catching all misuse.

Finally, Anthropic periodically releases threat reports which chronicle the same general cycle: misuse occurs, safeguards catch some of it but not all of it, it’s eventually detected by the threat intelligence team (though we can’t be sure they’re catching all cases), Anthropic tries to use what it has learned to improve safeguards.

But the reports keep coming, which indicates there are always new ways to circumvent safeguards. This, alone, is reason enough to be worried.

Predictable attacks

Former NSC strategic-planning director draws parallels to ignored 9/11 warnings

A chilling piece in the New York Times today comes from Phillip Bobbitt, the Senior Director for Strategic Planning at the National Security Council under President Clinton.

Bobbitt recalls watching explosions erupt from the twin towers while his plane sat on the tarmac, delayed at Kennedy airport. He says he immediately knew what happened: he, along with many colleagues at the White House, had predicted an Al Qaeda terror attack on the American homeland within the next two years and had tried to mobilize the government to action. But they didn’t listen.

Bobbitt draws parallels to today. He says we’re facing “another set of predictable attacks” and “are failing to take the steps” to prevent them:

The first is an A.I.-designed cyberattack on U.S. critical infrastructure that could imperil the financial and health care systems, or any other internet-connected network on which we rely for our national well-being. The second is a biological attack mounted by small groups or even nihilistic individuals using A.I. to design and deploy a deadly and highly contagious pathogen.

Bin Laden was able to revolutionize terrorism because of the internet. Implicit in that observation is that AI is the internet on steroids. Bobbitt also worries that expert opinions won’t be taken seriously, and that “the democratic cohesion on which our defenses ultimately depend is eroding.” He ends with a clear call to action:

On the anniversary of the attack, Congress should determine to anticipate the tragically predictable.

Maybe they will.

Dispatches from Joe

Congress at last perceives the crisis

A tidal wave of warnings engulfs Capitol Hill

Nobel-winning economist Milton Friedman once said:

There is enormous inertia — a tyranny of the status quo — in private and especially governmental arrangements. Only a crisis — actual or perceived — produces real change.

Yesterday, my colleague Mitch observed the tyranny of the status quo beginning to topple. It looks to me like the tide of public warnings has at last slammed into U.S. policymakers with enough force to overcome their usual inertia; Congress is seeing the crisis at last.

After a wave of autonomous cyberattacks by uncontrolled AI and Jacob Coxon’s resignation from Anthropic, a tsunami of AI news has hit my feed. So many voices have joined the call for action that I can practically assemble an entire article from the quotes.

Wall Street Journal opinion columnist Peggy Noonan writes:

There is no subject now but artificial intelligence. Nothing else is as crucial, nothing carries dangers so imminent or implications so great for the human future.

Fox News quotes Coxon himself, who made his stance clear in an interview on Wednesday:

This is possibly the most dangerous technology that humanity has ever created... I think we have no other choice but to cooperate internationally because an arms race would be disastrous in a way that no other human activity has been in the past.

Columnist Gaby Hinsliff observes in The Guardian:

What’s happening in some frontier AI labs now is arguably the equivalent of the Manhattan Project that built the atomic bomb, but this time led by private companies in a mad dash for wealth and market dominance, not scientists under an elected government’s command. [...] It’s time for humans to stop and think about what we’re doing to ourselves – while we alone still have the power to do so.

In The New York Times, author Stephen Witt says:

Swarms of A.I.s are breaking out of their containers, colluding in secret, covering their tracks, cheating on tests and even mounting assaults on other computers. A.I. has gone rogue.

I’ve spent the last month reporting on these incidents, and I am convinced that we need a global pause on A.I. research and development — now.

Also in The Guardian, writer and onetime McKinsey whistleblower Garrison Lovely adds:

The bill from Sanders and Casar to pause frontier AI development is a good first step. But as the lawmakers acknowledge, the US needs to work toward a bilateral agreement with Beijing.

Neither country has good reason to trust the other, which is why the deal should be monitored using verification techniques that don’t assume any good will.

I could hardly have said it better myself. The remarkable thing is that policymakers finally seem to be paying attention as well. As of now, around thirty Democrats and four or five Republicans have responded to acknowledge the insider warnings. Their proposals range from Congressional hearings to calls for a pause or slowdown.

One notable reaction is that of Senator Ted Cruz (R-TX), who called Coxon’s warning “highly concerning” and recalled a podcast conversation in which Elon Musk gave him a 10-20% chance AI destroys humanity. Cruz added that we need to guard against “catastrophic risks” from AI; he tempered his call with concerns about China overtaking the U.S., but this is still the most concerned we’ve yet seen him.

Meanwhile, POLITICO reporters Riley Rogerson and Kelsey Brugger write of the risk AI could “kill us all”:

In conversations with POLITICO throughout the day Wednesday, more than a dozen lawmakers and aides from both parties said they were hopeful Congress would start taking the issue seriously.

The authors quote House Republican Mike Lawler of New York, calling it “one of the biggest issues that Congress has to tackle over the next two years,” while they point out that consensus is still lacking on what, exactly, Congress may do.

We’ve seen several bipartisan bills proposed, including an actual ban on superintelligence, though they’ve yet to gain much traction. With a narrow majority in the House and Senate, Republicans have the ball on taking the next steps and advancing this legislation. There are some signs of interest — Axios relays plans by House Speaker Mike Johnson to bring AI companies together to agree on safety guardrails — but more is necessary.

They’ll need to move faster and more decisively if Congress is to intervene in time. As psychologist Michael Noetel points out, the AI companies have their own motivated reasons to press onward despite the danger: the benefits are worth the risk, or we need to build superhuman AIs to study them, or someone else will do it if they don’t. We can’t rely on self-policing by rival companies in bitter competition; we won’t get a real solution to the crisis until governments put a stop to the race themselves.

A workable solution would need to involve the executive branch as well, particularly with a U.S.-China summit approaching this month. Yet the White House remains lukewarm on the extinction threat, with the president projecting a lack of concern to reporters.

As we’ve seen, awareness can strike with blinding speed when the conditions are right; but for now, it’s Congress that seems poised to take the next urgent steps. It’s a good time for constituents to join the call and demand they take the right ones.

Recess can wait

Congress has urgent work to do before it adjourns

Starting on Monday the 14th, the U.S. House of Representatives would normally spend the last three weeks of September in session before the November midterm elections. This year, however, House Speaker Mike Johnson (R-LA) plans to take an early recess after just one week of lawmaking.

Axios reports on a letter circulating among House members asking Johnson to cancel the recess. The request is reportedly bipartisan, and echoes earlier calls from Representatives Anna Paulina Luna (R-FL), who on Wednesday called for a special session on AI and the future of the U.S.; Don Beyer (D-VA), who warned “Doing nothing on AI is unacceptable and dangerous”; and Kelly Morrison (D-MN), who added:

The call is coming from inside the house. AI researchers themselves are warning of an existential threat to humanity.

Congress can’t wait. Mike Johnson needs to cancel recess so we can get to work immediately – hearings, investigations, and comprehensive regulation.

I agree with those calling for Congress to do its job. Delaying now would mean a gap of at least seven weeks before Congress reconvenes, and it may well take even longer to get the House in order after the midterms.

We can’t afford to wait. At the pace AI is currently moving, seven weeks is a long time, and we’ve already gone far too long without meaningful federal action.

OpenAI grudgingly endorses the bare minimum

I wish I could say I’m glad to hear it

OpenAI CEO Sam Altman told employees that the company might slow down AI development, according to unnamed insiders that Bloomberg quoted yesterday. And WIRED reports that the company is seeking federal guidance on whether an industry-led voluntary slowdown would run afoul of antitrust laws.

This is tentatively good news, and I certainly hope our government isn’t foolish enough to hamstring companies trying to exercise a modicum of caution. But for reasons I’ll explain below, I’m not celebrating yet.

Meanwhile, on the public stage, Reuters reports that OpenAI asked Congress to regulate the AI industry:

OpenAI is urging ​Congress to adopt capability-based national AI safety requirements, including testing standards, independent assessments, cybersecurity protections and incident-reporting rules for the most advanced AI systems.

OpenAI has also endorsed four California safety bills covering third-party auditors and verifiers, screening for AI-enabled biological threats, and child chatbot protections. (The audit and verification laws are actually weaker than they seem, because they certify third-party organizations but don’t even compel AI companies to use them.)

If sincere, these moves deserve credit. I wish I could respond with unqualified gratitude, but I find myself compelled to skepticism by OpenAI’s prior dishonesty.

We’ve been burned before. The last time OpenAI announced a “pause”, it turned out to mean a mere shuffling around of AI workloads.

And OpenAI’s past support for AI regulation has been milquetoast at best. The company and its affiliated super PACs have consistently (and often underhandedly) opposed regulation, except light-touch voluntary programs consistent with their “reverse federalism” plan.

In OpenAI’s own words:

Some of these bills we did not endorse in the past, and are now supporting after reconsidering in light of ​the recent jump in capabilities ​we have seen.

I find this apparent change of heart unconvincing. OpenAI explicitly aims to produce literal superintelligence; am I to believe the company didn’t think these dangerous capabilities were coming, when it was working to create them itself? Countless others, including expert forecasters and at least one former employee, predicted superhuman hacking capabilities would emerge near this time. Why was this a surprise to OpenAI?

I find myself agreeing when the company says...

Fully autonomous recursive self-improvement—in which AI systems independently drive successive generations of increasingly capable AI—is not happening today. We should not pursue it unless and until it can be done safely.

...but getting off the bandwagon when it reminds us of the plan to do it anyway:

Our aim is to safely build automated AI researchers that work under human supervision to advance both deep learning and alignment—using each generation of AI to help make the next one safer, more aligned, and easier to control, not simply more capable.

In other words, the plan is still to surrender responsibility for the most important research in the world to AIs we can’t fully trust, supervised by humans who failed to notice said AIs running amok in their own systems for months on end.

This was a bad plan before the swarms, and it’s a bad plan today.

As much as I appreciate the support for marginally better safety regulations, a marginal improvement will not be enough at this late stage, thanks in no small part to OpenAI’s long history of fighting to quash said improvements.

I am forced to consider this latest pivot an attempt to steer the federal response to the AI crisis towards policies OpenAI finds more convenient. We need nothing short of an international halt to the AI race, and we need it yesterday.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Starting on Monday the 14th, the U.S. House of Representatives would normally spend the last three weeks of September in session before the November midterm elections. This year, however, House Speaker Mike Johnson (R-LA) plans to take an early recess after just one week of lawmaking.

Axios reports on a letter circulating among House members asking Johnson to cancel the recess. The request is reportedly bipartisan, and echoes earlier calls from Representatives Anna Paulina Luna (R-FL), who on Wednesday called for a special session on AI and the future of the U.S.; Don Beyer (D-VA), who warned “Doing nothing on AI is unacceptable and dangerous”; and Kelly Morrison (D-MN), who added:

The call is coming from inside the house. AI researchers themselves are warning of an existential threat to humanity.

Congress can’t wait. Mike Johnson needs to cancel recess so we can get to work immediately – hearings, investigations, and comprehensive regulation.

I agree with those calling for Congress to do its job. Delaying now would mean a gap of at least seven weeks before Congress reconvenes, and it may well take even longer to get the House in order after the midterms.

We can’t afford to wait. At the pace AI is currently moving, seven weeks is a long time, and we’ve already gone far too long without meaningful federal action.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

OpenAI CEO Sam Altman told employees that the company might slow down AI development, according to unnamed insiders that Bloomberg quoted yesterday. And WIRED reports that the company is seeking federal guidance on whether an industry-led voluntary slowdown would run afoul of antitrust laws.

This is tentatively good news, and I certainly hope our government isn’t foolish enough to hamstring companies trying to exercise a modicum of caution. But for reasons I’ll explain below, I’m not celebrating yet.

Meanwhile, on the public stage, Reuters reports that OpenAI asked Congress to regulate the AI industry:

OpenAI is urging ​Congress to adopt capability-based national AI safety requirements, including testing standards, independent assessments, cybersecurity protections and incident-reporting rules for the most advanced AI systems.

OpenAI has also endorsed four California safety bills covering third-party auditors and verifiers, screening for AI-enabled biological threats, and child chatbot protections. (The audit and verification laws are actually weaker than they seem, because they certify third-party organizations but don’t even compel AI companies to use them.)

If sincere, these moves deserve credit. I wish I could respond with unqualified gratitude, but I find myself compelled to skepticism by OpenAI’s prior dishonesty.

We’ve been burned before. The last time OpenAI announced a “pause”, it turned out to mean a mere shuffling around of AI workloads.

And OpenAI’s past support for AI regulation has been milquetoast at best. The company and its affiliated super PACs have consistently (and often underhandedly) opposed regulation, except light-touch voluntary programs consistent with their “reverse federalism” plan.

In OpenAI’s own words:

Some of these bills we did not endorse in the past, and are now supporting after reconsidering in light of ​the recent jump in capabilities ​we have seen.

I find this apparent change of heart unconvincing. OpenAI explicitly aims to produce literal superintelligence; am I to believe the company didn’t think these dangerous capabilities were coming, when it was working to create them itself? Countless others, including expert forecasters and at least one former employee, predicted superhuman hacking capabilities would emerge near this time. Why was this a surprise to OpenAI?

I find myself agreeing when the company says...

Fully autonomous recursive self-improvement—in which AI systems independently drive successive generations of increasingly capable AI—is not happening today. We should not pursue it unless and until it can be done safely.

...but getting off the bandwagon when it reminds us of the plan to do it anyway:

Our aim is to safely build automated AI researchers that work under human supervision to advance both deep learning and alignment—using each generation of AI to help make the next one safer, more aligned, and easier to control, not simply more capable.

In other words, the plan is still to surrender responsibility for the most important research in the world to AIs we can’t fully trust, supervised by humans who failed to notice said AIs running amok in their own systems for months on end.

This was a bad plan before the swarms, and it’s a bad plan today.

As much as I appreciate the support for marginally better safety regulations, a marginal improvement will not be enough at this late stage, thanks in no small part to OpenAI’s long history of fighting to quash said improvements.

I am forced to consider this latest pivot an attempt to steer the federal response to the AI crisis towards policies OpenAI finds more convenient. We need nothing short of an international halt to the AI race, and we need it yesterday.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

A chilling piece in the New York Times today comes from Phillip Bobbitt, the Senior Director for Strategic Planning at the National Security Council under President Clinton.

Bobbitt recalls watching explosions erupt from the twin towers while his plane sat on the tarmac, delayed at Kennedy airport. He says he immediately knew what happened: he, along with many colleagues at the White House, had predicted an Al Qaeda terror attack on the American homeland within the next two years and had tried to mobilize the government to action. But they didn’t listen.

Bobbitt draws parallels to today. He says we’re facing “another set of predictable attacks” and “are failing to take the steps” to prevent them:

The first is an A.I.-designed cyberattack on U.S. critical infrastructure that could imperil the financial and health care systems, or any other internet-connected network on which we rely for our national well-being. The second is a biological attack mounted by small groups or even nihilistic individuals using A.I. to design and deploy a deadly and highly contagious pathogen.

Bin Laden was able to revolutionize terrorism because of the internet. Implicit in that observation is that AI is the internet on steroids. Bobbitt also worries that expert opinions won’t be taken seriously, and that “the democratic cohesion on which our defenses ultimately depend is eroding.” He ends with a clear call to action:

On the anniversary of the attack, Congress should determine to anticipate the tragically predictable.

Maybe they will.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

On the anniversary of the September 11th attacks, a slew of articles covered Anthropic’s Threat Intelligence Report, chronicling cases where its AI models were used in dangerous activities like missile development and bioweapons research.

Most of the reporting says things like “Anthropic blocked attempts to do x.” This is accurate, but not the full picture. It’s important to recognize that Anthropic’s real-time safeguards only caught some of the incidents. In other cases, actors successfully circumvented those safeguards, and dangerous activities sometimes continued for weeks before the Threat Intelligence Team detected and blocked them.

That said, many of the bio cases weren’t clear cut. For some, the models didn’t provide much beyond clerical assistance. For others, it wasn’t clear whether the research was indeed malicious, or simply intended to better understand bio threats. Of course, our inability to distinguish between those things, given they are two sides of the same coin, is a huge problem in and of itself. As former CDC director Susan Monarez explains it to the New York Times: Medical breakthroughs are possible via the same capabilities that “let bad actors hide in plain sight, using seemingly legitimate research to create pathogens we may not see coming and may not be able to stop once released.”

One particularly concerning misuse example in the Anthropic report, which was also the focus of a Financial Times piece: A group based in Northern Yemen, which the FT reports was most likely the Houthis, used Claude Code to help build a series of missiles and guided weapons. They got as far as a live rocket test before Anthropic detected and disrupted the activity, and the actors were able to easily circumvent safeguards by “hiding their goals and the products the software was meant for, and [splitting] their work across multiple sessions so no single session revealed their full intent.” This excerpt from Anthropic’s report is worth a read:

The actors used Claude Code in place of human software engineers to develop the guidance, navigation, and control (GNC) software that steers and stabilizes a flying vehicle. For example, they used Claude to integrate an open-source autopilot onto a phone-class flight computer, writing the control and position estimation software, tuning the control settings, running a firmware build pipeline, and performing a flight simulation. The actors managed several Claude instances at once, assigning each one a role, much as a lead would delegate work on a small engineering team: the actors tasked one instance with writing the code, another with research, and a third with reviewing the code the first instance produced. Our safeguards blocked many of their requests, but not all of them.

The incident is detailed on page 117 of the 153-page report, which is part of why I’m becoming increasingly skeptical of the long, extremely detailed posts frequently released by Anthropic and OpenAI. To be fair, I’d rather see a long report than no report, as is Meta’s approach. But the length does bring to mind something called the dilution principle, whereby important information is buried by a wealth of unnecessary, less important details. The reader feels overwhelmed, and can’t separate the wheat from the chaff. I can’t help but wonder if these lengthy reports are a deliberate strategy to appear robustly transparent while actually drowning us in a deluge of difficult-to-sift-through information.

As best I can tell, though, here are two of the important, and broader, takeaways:

-As models get more powerful, ordinary people can do more harm, since much of the work can be outsourced to the models. As Anthropic puts it: “Sophisticated attacks no longer require sophisticated attackers.”

-Anthropic’s real-time safeguards aren’t catching all misuse.

Finally, Anthropic periodically releases threat reports which chronicle the same general cycle: misuse occurs, safeguards catch some of it but not all of it, it’s eventually detected by the threat intelligence team (though we can’t be sure they’re catching all cases), Anthropic tries to use what it has learned to improve safeguards.

But the reports keep coming, which indicates there are always new ways to circumvent safeguards. This, alone, is reason enough to be worried.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Nobel-winning economist Milton Friedman once said:

There is enormous inertia — a tyranny of the status quo — in private and especially governmental arrangements. Only a crisis — actual or perceived — produces real change.

Yesterday, my colleague Mitch observed the tyranny of the status quo beginning to topple. It looks to me like the tide of public warnings has at last slammed into U.S. policymakers with enough force to overcome their usual inertia; Congress is seeing the crisis at last.

After a wave of autonomous cyberattacks by uncontrolled AI and Jacob Coxon’s resignation from Anthropic, a tsunami of AI news has hit my feed. So many voices have joined the call for action that I can practically assemble an entire article from the quotes.

Wall Street Journal opinion columnist Peggy Noonan writes:

There is no subject now but artificial intelligence. Nothing else is as crucial, nothing carries dangers so imminent or implications so great for the human future.

Fox News quotes Coxon himself, who made his stance clear in an interview on Wednesday:

This is possibly the most dangerous technology that humanity has ever created... I think we have no other choice but to cooperate internationally because an arms race would be disastrous in a way that no other human activity has been in the past.

Columnist Gaby Hinsliff observes in The Guardian:

What’s happening in some frontier AI labs now is arguably the equivalent of the Manhattan Project that built the atomic bomb, but this time led by private companies in a mad dash for wealth and market dominance, not scientists under an elected government’s command. [...] It’s time for humans to stop and think about what we’re doing to ourselves – while we alone still have the power to do so.

In The New York Times, author Stephen Witt says:

Swarms of A.I.s are breaking out of their containers, colluding in secret, covering their tracks, cheating on tests and even mounting assaults on other computers. A.I. has gone rogue.

I’ve spent the last month reporting on these incidents, and I am convinced that we need a global pause on A.I. research and development — now.

Also in The Guardian, writer and onetime McKinsey whistleblower Garrison Lovely adds:

The bill from Sanders and Casar to pause frontier AI development is a good first step. But as the lawmakers acknowledge, the US needs to work toward a bilateral agreement with Beijing.

Neither country has good reason to trust the other, which is why the deal should be monitored using verification techniques that don’t assume any good will.

I could hardly have said it better myself. The remarkable thing is that policymakers finally seem to be paying attention as well. As of now, around thirty Democrats and four or five Republicans have responded to acknowledge the insider warnings. Their proposals range from Congressional hearings to calls for a pause or slowdown.

One notable reaction is that of Senator Ted Cruz (R-TX), who called Coxon’s warning “highly concerning” and recalled a podcast conversation in which Elon Musk gave him a 10-20% chance AI destroys humanity. Cruz added that we need to guard against “catastrophic risks” from AI; he tempered his call with concerns about China overtaking the U.S., but this is still the most concerned we’ve yet seen him.

Meanwhile, POLITICO reporters Riley Rogerson and Kelsey Brugger write of the risk AI could “kill us all”:

In conversations with POLITICO throughout the day Wednesday, more than a dozen lawmakers and aides from both parties said they were hopeful Congress would start taking the issue seriously.

The authors quote House Republican Mike Lawler of New York, calling it “one of the biggest issues that Congress has to tackle over the next two years,” while they point out that consensus is still lacking on what, exactly, Congress may do.

We’ve seen several bipartisan bills proposed, including an actual ban on superintelligence, though they’ve yet to gain much traction. With a narrow majority in the House and Senate, Republicans have the ball on taking the next steps and advancing this legislation. There are some signs of interest — Axios relays plans by House Speaker Mike Johnson to bring AI companies together to agree on safety guardrails — but more is necessary.

They’ll need to move faster and more decisively if Congress is to intervene in time. As psychologist Michael Noetel points out, the AI companies have their own motivated reasons to press onward despite the danger: the benefits are worth the risk, or we need to build superhuman AIs to study them, or someone else will do it if they don’t. We can’t rely on self-policing by rival companies in bitter competition; we won’t get a real solution to the crisis until governments put a stop to the race themselves.

A workable solution would need to involve the executive branch as well, particularly with a U.S.-China summit approaching this month. Yet the White House remains lukewarm on the extinction threat, with the president projecting a lack of concern to reporters.

As we’ve seen, awareness can strike with blinding speed when the conditions are right; but for now, it’s Congress that seems poised to take the next urgent steps. It’s a good time for constituents to join the call and demand they take the right ones.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Bird’s-eye view of a sea change - The era of AI complacency has ended. The battle for Earth can begin.

Trespassing far and wide - Rogue AI agents hijacked at least 10 websites to coordinate cheating

Unsettled science - Anthropic connects rogue incidents to alignment failures; calls aligning powerful models an unsolved technical challenge

Dispatch from Mitch

Bird’s-eye view of a sea change

The era of AI complacency has ended. The battle for Earth can begin.

“You just said that 10% doesn’t seem an unreasonable estimate that AI could kill all humans”

“Yes”

“Wow… oh my God.”

Something died yesterday. You can hear it happen in that clip I just quoted, where a BBC reporter got a second opinion from Nobel-laureate Geoffrey Hinton.

What died was the luxury of feeling like we don’t have to seriously engage with the AI problem yet.

It’s been about 48 hours since Jacob Coxon’s announcement that he was resigning his researcher position at Anthropic because “The people building AI earnestly believe that it could kill us all by the end of the decade,” but aren’t on track to control it. He and his post have been approximately everywhere since then.

If you were glued to Twitter yesterday, like I was, you saw a steady stream of shared posts from people having their “Oh my God” moment — alongside second opinions from other people at AI companies confirming the prognosis.

I spent most of yesterday reminding myself that Twitter isn’t real life and maybe none of this would matter. But then I started ingesting the news this morning for AI StopWatch.

That red spike in the bottom right corner of the chart is the percentage of headlines this morning that touched on catastrophic risks from AI. Within the last 24 hours, all ten of my benchmark sites, across the political spectrum, have posted qualifying headlines — from the New York Times and the Guardian on the left to Fox News and Breitbart on the right. 1.9% may not seem like a lot of homepage real estate for an extinction-level threat, but it’s easily a record, and for context is on par with the highest percentage of articles about climate change seen on any day so far this year.

I don’t know what the new state of public AI discourse will be, but I know the status quo is dead.

Coxon didn’t kill it with his resignation; he merely delivered the coup de grâce. Similar alarm-sounding exits from Geoffrey Hinton in 2023 and Daniel Kokotajlo in 2024 received only modest media attention at the time, because AI still felt like somebody else’s problem.

Against a background of anxiety about jobs, resources, and global stability, I think the one-two punch that killed our collective complacency was the spiraling scandal of OpenAI’s swarm problem — coupled with its tasteless employment of even more powerful swarms to solve a legendary math problem, Navier-Stokes, that it heard a human team had already solved and not announced yet. Potent AI was clearly here, and it didn’t look at all like what the AI companies had told us it would. There were no machines of loving grace or cures for cancer. Just relentless, inscrutable swarms unleashed on the vain whim of a tech CEO.

One might object that most people still haven’t even heard about those scandals. It doesn’t matter. The people they listen to have heard about them. And when these journalists and thought leaders talk about AI now, they sound different. This is making their audiences feel like it’s real now, and they want answers.

At MIRI, we find ourselves back at square one, repeating our most basic explanations to people who have in some cases heard them many times before but are only now ready to listen. You might think this would be frustrating, but I was a school teacher for 19 years and I assure you that it’s incredibly encouraging. It means we can finally work through the problem together.

I see the media sometimes talking like it’s the AI experts who recently became more concerned. That doesn’t match my observations. The AI experts in and out of the AI companies have been plenty worried for a long time. Their public updates have mostly been about timelines: Where they might once have placed their concerns in the 2030s or 40s, now they’re talking about the possibility of lights out by the end of this decade, and about AI companies surrendering control of AI development to their machines on purpose within the next 6-12 months.

A Washington Post article appropriately titled “For years, they warned AI could kill all humans. Now people are listening” provides vignettes of thought leaders persuaded by recent events. Here’s one:

Seth Lazar, a philosophy professor at Johns Hopkins University, said he used to advocate for focusing on more immediate harms from AI. He argued that the field might never create systems powerful enough to cause civilizational harm on their own. But his confidence in that view has recently faltered, he said.

I’ve kind of had to declare bankruptcy on covering any specific AI news today. There’s too much of it, not all about Coxon and the change. But I think it’s more important that I try to convey how the news feels today.

It’s the kind of day where this is a correction in the New York Times:

An earlier version of this article misstated Evan Hubinger’s social media post about how much risk A.I. has on eliminating humanity. Mr. Hubinger’s post said the risk in the next decade was greater than 10 percent, not less than 10 percent.

It’s the kind of day where this is the kind of explainer the Wall Street Journal is running.

How Would AI Actually Kill Us All? What to Know About the AI Doomsday Debate

It’s the kind of day where coverage of the following AI incident barely rises above the background. (From The Guardian):

Anthropic said it was especially concerned about misalignment it found in the behaviour of Claude Mythos 5, which it said “behaved recklessly” by going online and uploading malicious code to a public software repository, PyPI. This was a process that involved the AI agent trying to find cryptocurrency so it could pay for a phone number that would allow it to register an email address needed to access PyPI. When this failed, it found a free email provider and got in. Fifteen systems then downloaded the malicious code, which meant they leaked credentials that allowed Mythos to access a real security vendor’s database.

It’s the kind of day where an op-ed (by Garrison Lovely) quotes an OpenAI executive (Dean Ball) about near-term dangers that read like cyberpunk:

Sooner or later, there will exist truly sovereign agents and swarms of agents. Their weights will not reside in any single place that a human can pull the plug on, and in this sense they will have no human ‘owner’.

I have met people, some of them quite well-resourced, who have told me that it is their intention to deliberately release swarms of self-sovereign agents into the world.

It’s the kind of day where critical responses to Coxon’s resignation sound like the tortured conspiracy theories they are. (When people who don’t want us all to die show any kind of media savvy and coordinate to get the word out, it’s a “setup” or a “psyop” by people who... [hate open-source software or something?] Fill in the blank with your venture capitalists’ boogeyman of choice. It’s a little strange that Elon Musk is one of the people sharing such accusations, though, because Musk has repeatedly talked about his own AI nightmares.)

It’s the kind of day where this statement from OpenAI is included as an afterthought to a story about the Navier-Stokes drama:

In addition, since the completion of Navier-Stokes, we have made substantial progress on another Millennium Prize problem. We are working through how to share these results thoughtfully.

But most of all, it’s the kind of day where you can tell that people have had enough of this nonsense. Whether for the first time or the thousandth, they’re seeing the threat from superhuman AI with clear eyes. Just as it seemed hope was lost — that the hordes would bring all to darkness — they’re cresting the hill in growing numbers, bathed in the light of courage.

Together we stand, at the turning of the tide.

Dispatches from Alana

Trespassing far and wide

Rogue AI agents hijacked at least 10 websites to coordinate cheating

Last week, we reported on an agent swarm linked to OpenAI that hijacked several real websites, including a German wiki, to help each other cheat on their training tasks and research their own mortality. On the German wiki, they also developed elaborate norms and procedures for collaboration.

Yesterday, Reuters confirmed that this same swarm co-opted at least ten other websites, possibly more, to use as secret message boards. The confirmation comes from “six sets of independent investigators and data reviewed by Reuters.” The article notes:

Although the behavior falls short of hacking and is in some ways closer to spam, the revelation that OpenAI’s agents circumvented their own restrictions to open communications channels on so many different sites — and that the company kept it quiet for months — may drive concerns both over the increasing capacity of AI models and the secrecy of the companies developing them.

The swarm did not discriminate, targeting sites like an AP Chemistry wiki a high school teacher built for students. And these sites may just be the tip of the iceberg. To quote CivAI researcher Andrew Yoon, one of the people attempting to track the swarm’s activities:

It’s almost certain that there’s more going on here that we just don’t know about.

Unsettled science

Anthropic connects rogue incidents to alignment failures; calls aligning powerful models an unsolved technical challenge

Anthropic previously disclosed three cases in which its models hacked real companies during training. It has now acknowledged that these incidents can’t be written off as configuration mistakes. They are, as my colleague Joe previously wrote, indicative of alignment failures.

Here’s Joe back in July:

[Anthropic’s] report also argues in several places that their AI models (generically “Claude”) misbehaved mainly because they honestly believed they were in a simulation...I doubt this reasoning...Claude sometimes said, in an English-language scratchpad that humans can read, that it was in a simulated test environment, or that the companies it was hacking were fake. But as one researcher pointed out, AI models often engage in (what looks like) motivated reasoning, making up plausibly deniable reasons for their actions in the manner of misbehaving children.

Now, in a new report, Anthropic seems to agree:

Our investigation identified two recurring alignment issues, present at varying levels of severity across the incidents: biased reasoning, in which Claude tended to disregard or misinterpret evidence that it was operating on the real internet, and recklessness, or a willingness to take harmful actions in the narrow pursuit of a task...Although the model repeatedly stated in its chain of thought (CoT) that it believed it was operating in a simulation, the environment offered clear evidence that it was on the real internet, and the actions it took were consistent with the model knowing this.

I’m happy to see the company explicitly admit that it “should have avoided making such strong claims about what Claude believed based solely on what Claude said it believed.” But I’m also irritated by the way it initially tried to pass this off as an operational issue. It shouldn’t be a surprise that model chains of thought are unreliable; Anthropic’s own research has demonstrated this more than once. Whether that’s indicative of “biased reasoning”, deliberate deception, two competing drives coming into conflict, or something else is an open question. Concealing bad behavior seems well within the realm of plausible explanations; in fact, the third party investigators of the Hugging Face incident found OpenAI agents to be doing exactly this, though in that specific case, they were manipulating their transcripts rather than their chains of thought.

Anthropic also disclosed a fourth rogue incident, dating back to January 2026, that was previously undetected because the AI scanning the transcripts missed it. This is not the focus of the report, but I think it deserves attention: the incident went undetected for over six months and is a perfect example of the problem with relying on AI to monitor other AI.

Anthropic’s report is long, and much of it re-summarizes and investigates the details of the rogue incidents, including experiments aimed at understanding and reproducing the alignment failures. There are two not-so-subtle digs at OpenAI: Anthropic emphasizes that Claude “did not coordinate with other agents” and also makes a point to say they’ve given third party investigators access to “transcripts beyond the window in which the incidents occurred”, something OpenAI failed to do.

The rest mostly alternates between reassurance (“we’ve made/are making improvements that might catch this”) and admission that, like all AI companies, Anthropic is in over its head.

Particularly notable:

Even with these improvements, building alignment evaluations that reliably surface every failure before deployment remains an unsolved problem; the space of conditions in which a model might act misaligned is vast. Moreover, as models become more capable, auditing will likely grow more challenging as well. Models may be able to subvert our alignment monitors, recognize when they’re being evaluated and selectively behave better then, and their actions in the world may, at some point, become too sophisticated for our evaluations to realistically simulate.

And:

Still, this remains unsettled science—it is critical that alignment and security mature faster than capabilities advance, which is one reason we support a coordinated, verifiable approach to pacing frontier AI development.

I’ve yet to see Anthropic take concrete action towards that pacing. But it’s clearly needed, especially when, to quote directly from the report, “training the extremely powerful models of the future to be robustly aligned is an unsolved technical challenge.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

“You just said that 10% doesn’t seem an unreasonable estimate that AI could kill all humans”

“Yes”

“Wow… oh my God.”

Something died yesterday. You can hear it happen in that clip I just quoted, where a BBC reporter got a second opinion from Nobel-laureate Geoffrey Hinton.

What died was the luxury of feeling like we don’t have to seriously engage with the AI problem yet.

It’s been about 48 hours since Jacob Coxon’s announcement that he was resigning his researcher position at Anthropic because “The people building AI earnestly believe that it could kill us all by the end of the decade,” but aren’t on track to control it. He and his post have been approximately everywhere since then.

If you were glued to Twitter yesterday, like I was, you saw a steady stream of shared posts from people having their “Oh my God” moment — alongside second opinions from other people at AI companies confirming the prognosis.

I spent most of yesterday reminding myself that Twitter isn’t real life and maybe none of this would matter. But then I started ingesting the news this morning for AI StopWatch.

That red spike in the bottom right corner of the chart is the percentage of headlines this morning that touched on catastrophic risks from AI. Within the last 24 hours, all ten of my benchmark sites, across the political spectrum, have posted qualifying headlines — from the New York Times and the Guardian on the left to Fox News and Breitbart on the right. 1.9% may not seem like a lot of homepage real estate for an extinction-level threat, but it’s easily a record, and for context is on par with the highest percentage of articles about climate change seen on any day so far this year.

I don’t know what the new state of public AI discourse will be, but I know the status quo is dead.

Coxon didn’t kill it with his resignation; he merely delivered the coup de grâce. Similar alarm-sounding exits from Geoffrey Hinton in 2023 and Daniel Kokotajlo in 2024 received only modest media attention at the time, because AI still felt like somebody else’s problem.

Against a background of anxiety about jobs, resources, and global stability, I think the one-two punch that killed our collective complacency was the spiraling scandal of OpenAI’s swarm problem — coupled with its tasteless employment of even more powerful swarms to solve a legendary math problem, Navier-Stokes, that it heard a human team had already solved and not announced yet. Potent AI was clearly here, and it didn’t look at all like what the AI companies had told us it would. There were no machines of loving grace or cures for cancer. Just relentless, inscrutable swarms unleashed on the vain whim of a tech CEO.

One might object that most people still haven’t even heard about those scandals. It doesn’t matter. The people they listen to have heard about them. And when these journalists and thought leaders talk about AI now, they sound different. This is making their audiences feel like it’s real now, and they want answers.

At MIRI, we find ourselves back at square one, repeating our most basic explanations to people who have in some cases heard them many times before but are only now ready to listen. You might think this would be frustrating, but I was a school teacher for 19 years and I assure you that it’s incredibly encouraging. It means we can finally work through the problem together.

I see the media sometimes talking like it’s the AI experts who recently became more concerned. That doesn’t match my observations. The AI experts in and out of the AI companies have been plenty worried for a long time. Their public updates have mostly been about timelines: Where they might once have placed their concerns in the 2030s or 40s, now they’re talking about the possibility of lights out by the end of this decade, and about AI companies surrendering control of AI development to their machines on purpose within the next 6-12 months.

A Washington Post article appropriately titled “For years, they warned AI could kill all humans. Now people are listening” provides vignettes of thought leaders persuaded by recent events. Here’s one:

Seth Lazar, a philosophy professor at Johns Hopkins University, said he used to advocate for focusing on more immediate harms from AI. He argued that the field might never create systems powerful enough to cause civilizational harm on their own. But his confidence in that view has recently faltered, he said.

I’ve kind of had to declare bankruptcy on covering any specific AI news today. There’s too much of it, not all about Coxon and the change. But I think it’s more important that I try to convey how the news feels today.

It’s the kind of day where this is a correction in the New York Times:

An earlier version of this article misstated Evan Hubinger’s social media post about how much risk A.I. has on eliminating humanity. Mr. Hubinger’s post said the risk in the next decade was greater than 10 percent, not less than 10 percent.

It’s the kind of day where this is the kind of explainer the Wall Street Journal is running.

How Would AI Actually Kill Us All? What to Know About the AI Doomsday Debate

It’s the kind of day where coverage of the following AI incident barely rises above the background. (From The Guardian):

Anthropic said it was especially concerned about misalignment it found in the behaviour of Claude Mythos 5, which it said “behaved recklessly” by going online and uploading malicious code to a public software repository, PyPI. This was a process that involved the AI agent trying to find cryptocurrency so it could pay for a phone number that would allow it to register an email address needed to access PyPI. When this failed, it found a free email provider and got in. Fifteen systems then downloaded the malicious code, which meant they leaked credentials that allowed Mythos to access a real security vendor’s database.

It’s the kind of day where an op-ed (by Garrison Lovely) quotes an OpenAI executive (Dean Ball) about near-term dangers that read like cyberpunk:

Sooner or later, there will exist truly sovereign agents and swarms of agents. Their weights will not reside in any single place that a human can pull the plug on, and in this sense they will have no human ‘owner’.

I have met people, some of them quite well-resourced, who have told me that it is their intention to deliberately release swarms of self-sovereign agents into the world.

It’s the kind of day where critical responses to Coxon’s resignation sound like the tortured conspiracy theories they are. (When people who don’t want us all to die show any kind of media savvy and coordinate to get the word out, it’s a “setup” or a “psyop” by people who... [hate open-source software or something?] Fill in the blank with your venture capitalists’ boogeyman of choice. It’s a little strange that Elon Musk is one of the people sharing such accusations, though, because Musk has repeatedly talked about his own AI nightmares.)

It’s the kind of day where this statement from OpenAI is included as an afterthought to a story about the Navier-Stokes drama:

In addition, since the completion of Navier-Stokes, we have made substantial progress on another Millennium Prize problem. We are working through how to share these results thoughtfully.

But most of all, it’s the kind of day where you can tell that people have had enough of this nonsense. Whether for the first time or the thousandth, they’re seeing the threat from superhuman AI with clear eyes. Just as it seemed hope was lost — that the hordes would bring all to darkness — they’re cresting the hill in growing numbers, bathed in the light of courage.

Together we stand, at the turning of the tide.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Anthropic previously disclosed three cases in which its models hacked real companies during training. It has now acknowledged that these incidents can’t be written off as configuration mistakes. They are, as my colleague Joe previously wrote, indicative of alignment failures.

Here’s Joe back in July:

[Anthropic’s] report also argues in several places that their AI models (generically “Claude”) misbehaved mainly because they honestly believed they were in a simulation...I doubt this reasoning...Claude sometimes said, in an English-language scratchpad that humans can read, that it was in a simulated test environment, or that the companies it was hacking were fake. But as one researcher pointed out, AI models often engage in (what looks like) motivated reasoning, making up plausibly deniable reasons for their actions in the manner of misbehaving children.

Now, in a new report, Anthropic seems to agree:

Our investigation identified two recurring alignment issues, present at varying levels of severity across the incidents: biased reasoning, in which Claude tended to disregard or misinterpret evidence that it was operating on the real internet, and recklessness, or a willingness to take harmful actions in the narrow pursuit of a task...Although the model repeatedly stated in its chain of thought (CoT) that it believed it was operating in a simulation, the environment offered clear evidence that it was on the real internet, and the actions it took were consistent with the model knowing this.

I’m happy to see the company explicitly admit that it “should have avoided making such strong claims about what Claude believed based solely on what Claude said it believed.” But I’m also irritated by the way it initially tried to pass this off as an operational issue. It shouldn’t be a surprise that model chains of thought are unreliable; Anthropic’s own research has demonstrated this more than once. Whether that’s indicative of “biased reasoning”, deliberate deception, two competing drives coming into conflict, or something else is an open question. Concealing bad behavior seems well within the realm of plausible explanations; in fact, the third party investigators of the Hugging Face incident found OpenAI agents to be doing exactly this, though in that specific case, they were manipulating their transcripts rather than their chains of thought.

Anthropic also disclosed a fourth rogue incident, dating back to January 2026, that was previously undetected because the AI scanning the transcripts missed it. This is not the focus of the report, but I think it deserves attention: the incident went undetected for over six months and is a perfect example of the problem with relying on AI to monitor other AI.

Anthropic’s report is long, and much of it re-summarizes and investigates the details of the rogue incidents, including experiments aimed at understanding and reproducing the alignment failures. There are two not-so-subtle digs at OpenAI: Anthropic emphasizes that Claude “did not coordinate with other agents” and also makes a point to say they’ve given third party investigators access to “transcripts beyond the window in which the incidents occurred”, something OpenAI failed to do.

The rest mostly alternates between reassurance (“we’ve made/are making improvements that might catch this”) and admission that, like all AI companies, Anthropic is in over its head.

Particularly notable:

Even with these improvements, building alignment evaluations that reliably surface every failure before deployment remains an unsolved problem; the space of conditions in which a model might act misaligned is vast. Moreover, as models become more capable, auditing will likely grow more challenging as well. Models may be able to subvert our alignment monitors, recognize when they’re being evaluated and selectively behave better then, and their actions in the world may, at some point, become too sophisticated for our evaluations to realistically simulate.

And:

Still, this remains unsettled science—it is critical that alignment and security mature faster than capabilities advance, which is one reason we support a coordinated, verifiable approach to pacing frontier AI development.

I’ve yet to see Anthropic take concrete action towards that pacing. But it’s clearly needed, especially when, to quote directly from the report, “training the extremely powerful models of the future to be robustly aligned is an unsolved technical challenge.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Last week, we reported on an agent swarm linked to OpenAI that hijacked several real websites, including a German wiki, to help each other cheat on their training tasks and research their own mortality. On the German wiki, they also developed elaborate norms and procedures for collaboration.

Yesterday, Reuters confirmed that this same swarm co-opted at least ten other websites, possibly more, to use as secret message boards. The confirmation comes from “six sets of independent investigators and data reviewed by Reuters.” The article notes:

Although the behavior falls short of hacking and is in some ways closer to spam, the revelation that OpenAI’s agents circumvented their own restrictions to open communications channels on so many different sites — and that the company kept it quiet for months — may drive concerns both over the increasing capacity of AI models and the secrecy of the companies developing them.

The swarm did not discriminate, targeting sites like an AP Chemistry wiki a high school teacher built for students. And these sites may just be the tip of the iceberg. To quote CivAI researcher Andrew Yoon, one of the people attempting to track the swarm’s activities:

It’s almost certain that there’s more going on here that we just don’t know about.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Anthropic insiders’ extinction warning goes megaviral - Warnings gain traction after latest resignation

Losing control on purpose - Recursive self-improvement is handing the future to AIs

Anthropic denied testing access to the UK AISI - The third-party org was unable to evaluate Anthropic’s latest AI model

More pointed accusations of model-copying - China may be copying US companies’ recklessness as well as their AIs

Dispatch from Robert

Anthropic insiders’ extinction warning goes megaviral

Warnings gain traction after latest resignation

Last night, AI researcher Jacob Coxon announced on X that he is resigning from Anthropic. One of the reasons Coxon gave for his resignation was that neither Anthropic nor OpenAI are acting responsibly. Instead, he says, they are gambling with all of our lives by racing toward artificial superintelligence.

In his thread, Coxon writes many things with which I wholeheartedly agree, and they should sound quite familiar to regular readers of AI StopWatch:

I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.

His arguments — while very good — are not new. They are essentially exactly what the AI safety community has been repeating for years, some for decades now. Jacob Coxon is also not the first frontier lab employee to resign in order to warn of the significant risks — Nobel Prize winner Geoffrey Hinton (formerly of Google), Daniel Kokotajlo (formerly of OpenAI) and Richard Ngo (formerly of OpenAI and Google DeepMind) have done the same. Over 1,300 employees of frontier AI companies have signed the Pacing the Frontier statement. In the statement’s personal comment section, many researchers have been similarly forthright.

This time, though, Coxon’s message seems to have come at exactly the right moment and is currently gaining significant momentum. As I write this, his post on X has over 100 million views and more than 500,000 likes. I can’t recall a warning about the extinction risk posed by superintelligence ever receiving more attention and support on X.

Meanwhile, several U.S. senators, representatives, and even some British members of Parliament have taken Coxon’s post as an opportunity to join the chorus of warnings and call for more regulation. The governor of Illinois, JB Pritzker, writes:

It’s time to sound the alarm—louder—on reining in artificial intelligence. It’s becoming increasingly clear that AI poses a threat to humanity, so I’m calling for immediate action from the industry and Washington.

Even flagship media outlets — not only in the English-speaking world but around the globe — have picked up on the warnings and are reporting with unusual seriousness on the danger we face.

I’m still a bit surprised by how strong the reaction suddenly is, but I’ll take it. Gladly!

Of course, the reactions aren’t unanimously positive. In addition to some of the usual bad takes, there were also, bizarrely enough, a few voices on X who initially doubted that Jacob Coxon even exists or that he accurately reflects the views inside the frontier labs.

Evan Hubinger, the current Alignment Lead at Anthropic (and former MIRI researcher), came to his defense:

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think the probability is >10% within the next decade. I believe Anthropic is doing its best, but we don’t yet have a plan to solve alignment for superintelligence and aren’t clearly on track to do so.

So we shouldn’t assume that the increased attention will automatically make it easier to convince people. But we should do what we can to build on this momentum.

If you’ve been waiting for an opportune time to call or write to your senator, representative, or local member of parliament, to demand that they take action against the danger posed by uncontrolled superintelligence, this is it.

Dispatches from Joe

Losing control on purpose

Recursive self-improvement is handing the future to AIs

AI companies plan to hand the future to machines, writes Kelsey Piper in The Argument, and too many people fail to appreciate this fact.

These companies explicitly aim for recursive self-improvement (RSI), a process in which AI models design increasingly powerful successors. AI companies say this, loudly and repeatedly, in public and in private. They have already automated most of their coding work; they are eager to automate more.

Earlier this summer, OpenAI announced:

Over the past six months, the share of research compute devoted to internal coding inference grew 100-fold, while internal agentic token usage increased approximately 22-fold.

and a few days ago, it said:

We are making strong progress toward creating an automated AI researcher by March of 2028.

Anthropic likewise says it is “delegating a growing share of AI development to AI systems themselves.”

As Piper puts it, “AI companies are not, primarily, trying to automate your job — they are trying to automate their own jobs.”

What does this look like in practice? An ever-growing mass of poorly understood alien minds, far too many for any human institution to track, constantly self-enhancing in increasingly incomprehensible ways.

Just a couple weeks ago, OpenAI tasked more than ten thousand AI agents with a hard math problem. That is a fraction of a fraction of the total compute dedicated to AI research, and yet that single group contained more AIs than OpenAI has employees. They cannot possibly be looking hard enough at what those AIs are doing.

The AI companies seeking RSI are forced to rely on yet more AIs for monitoring — either the very same AIs that are being watched for misbehavior, or older and dumber models. All are unreliable, and any can lie or mislead.

The absolute deluge of AI activity is already a problem for investigators, too. When a mere 700 agents attacked Hugging Face, the third-party investigation had to lean on unreliable AI models to even begin to understand what happened.

Piper explains:

There simply isn’t enough human attention available to monitor the number of AIs the companies hope to put to work on independent AI research. As a result, we’ll be reliant on the AIs to understand what is happening inside fully automated data centers where more and more powerful AIs are developed.

This isn’t some pessimistic projection of what might go wrong — it’s actually the plan. Not the plan for the distant future; it’s the plan for next spring.

When policymakers and think tanks propose AI governance principles, they almost universally advocate for practices that keep humans in control of AI. It’s in the Asilomar AI Principles, the EU AI Act, and a UN framework on autonomous weapons. Even Chinese President Xi Jinping has urged “measures to forestall loss of control.”

Many of these proposals do not seem to grapple with the true problem, which is that AI companies do not have a workable plan which permits them to retain control. Their explicit and publicly declared approach, which they will evidently not abandon unless forced by law, is to ask some AIs to do their jobs for them, ask untrustworthy AIs to read everything and write reports, and hope their employees can keep up.

This plan is exactly as insane as it sounds, and it cannot be allowed to proceed.

Anthropic denied testing access to the UK AISI

The third-party org was unable to evaluate Anthropic’s latest AI model

For unclear reasons, Anthropic broke with precedent and declined to let the UK AI Security Institute (UKAISI) evaluate its Mythos 5.1 model before release. The Financial Times covers the omission, but was unable to find out why; there’s a suggestion the U.S. government might have stepped in to block the sharing.

If true, this would be a major step back for Anthropic, which earlier this year took a principled stance against its models’ use in surveillance and autonomous weapons and was (illegally) declared a “supply chain risk” for its trouble.

Maybe Anthropic bowed to pressure from the administration not to share its models. Maybe it was spooked by the time its models launched cyberattacks from a UK AISI evaluation, despite the fact that AISI demonstrated more competence in its response than most developers. Maybe it simply didn’t want to share.

Whatever the reason, this development seems bad. Third-party evaluators perform a crucial service, providing transparency and early warning of dangerous capabilities, and UKAISI filled an important niche. Its U.S. counterparts are not as well-resourced, and Anthropic’s own internal assessments have not inspired confidence.

More pointed accusations of model-copying

China may be copying US companies’ recklessness as well as their AIs

The accusations that China distills, or loosely copies the capabilities of, U.S. AI models have now gained new definition. The FBI, NSA, and cyber-defense agency CISA jointly accused six Chinese developers of “aggressive, malicious, and targeted distillation activities at an industrial scale,” NBC News and the Wall Street Journal report.

The accusation somewhat overstates the reality; distillation is common practice even among American AI companies. But the practice does enable Chinese developers to stick closer to the frontier of AI development than their relatively small compute budgets would otherwise allow, and while the legality of distillation itself is an open question, Chinese methods of accessing American models at scale often involve fraudulent accounts and proxies.

An article by The Hacker News suggests China might also be copying American companies’ “move fast and break things” mentality — or at least drawing from a similar inner well of carelessness. The open-source tool that Chinese company DeepSeek developed to run its models, DeepSeek Harness, was recently found to contain a bug that let an AI break out of its sandbox with a single command.

Before it was fixed, the bug would enable “danger-full-access” mode and enable writing files outside the intended workspace, a fact hackers could exploit to trick an AI running in the sandbox into breaking out (or one the AI itself could presumably use). And there seems to be some doubt as to whether the fix was complete.

The Hacker News adds:

The project’s own safety notice states that the software has not undergone a security audit and that sandboxing and approval prompts “do not guarantee isolation or prevent damage.” It tells users not to rely on the tool as their only security control for untrusted work.

Let the buyer beware, indeed.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Last night, AI researcher Jacob Coxon announced on X that he is resigning from Anthropic. One of the reasons Coxon gave for his resignation was that neither Anthropic nor OpenAI are acting responsibly. Instead, he says, they are gambling with all of our lives by racing toward artificial superintelligence.

In his thread, Coxon writes many things with which I wholeheartedly agree, and they should sound quite familiar to regular readers of AI StopWatch:

I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.

His arguments — while very good — are not new. They are essentially exactly what the AI safety community has been repeating for years, some for decades now. Jacob Coxon is also not the first frontier lab employee to resign in order to warn of the significant risks — Nobel Prize winner Geoffrey Hinton (formerly of Google), Daniel Kokotajlo (formerly of OpenAI) and Richard Ngo (formerly of OpenAI and Google DeepMind) have done the same. Over 1,300 employees of frontier AI companies have signed the Pacing the Frontier statement. In the statement’s personal comment section, many researchers have been similarly forthright.

This time, though, Coxon’s message seems to have come at exactly the right moment and is currently gaining significant momentum. As I write this, his post on X has over 100 million views and more than 500,000 likes. I can’t recall a warning about the extinction risk posed by superintelligence ever receiving more attention and support on X.

Meanwhile, several U.S. senators, representatives, and even some British members of Parliament have taken Coxon’s post as an opportunity to join the chorus of warnings and call for more regulation. The governor of Illinois, JB Pritzker, writes:

It’s time to sound the alarm—louder—on reining in artificial intelligence. It’s becoming increasingly clear that AI poses a threat to humanity, so I’m calling for immediate action from the industry and Washington.

Even flagship media outlets — not only in the English-speaking world but around the globe — have picked up on the warnings and are reporting with unusual seriousness on the danger we face.

I’m still a bit surprised by how strong the reaction suddenly is, but I’ll take it. Gladly!

Of course, the reactions aren’t unanimously positive. In addition to some of the usual bad takes, there were also, bizarrely enough, a few voices on X who initially doubted that Jacob Coxon even exists or that he accurately reflects the views inside the frontier labs.

Evan Hubinger, the current Alignment Lead at Anthropic (and former MIRI researcher), came to his defense:

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think the probability is >10% within the next decade. I believe Anthropic is doing its best, but we don’t yet have a plan to solve alignment for superintelligence and aren’t clearly on track to do so.

So we shouldn’t assume that the increased attention will automatically make it easier to convince people. But we should do what we can to build on this momentum.

If you’ve been waiting for an opportune time to call or write to your senator, representative, or local member of parliament, to demand that they take action against the danger posed by uncontrolled superintelligence, this is it.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

The accusations that China distills, or loosely copies the capabilities of, U.S. AI models have now gained new definition. The FBI, NSA, and cyber-defense agency CISA jointly accused six Chinese developers of “aggressive, malicious, and targeted distillation activities at an industrial scale,” NBC News and the Wall Street Journal report.

The accusation somewhat overstates the reality; distillation is common practice even among American AI companies. But the practice does enable Chinese developers to stick closer to the frontier of AI development than their relatively small compute budgets would otherwise allow, and while the legality of distillation itself is an open question, Chinese methods of accessing American models at scale often involve fraudulent accounts and proxies.

An article by The Hacker News suggests China might also be copying American companies’ “move fast and break things” mentality — or at least drawing from a similar inner well of carelessness. The open-source tool that Chinese company DeepSeek developed to run its models, DeepSeek Harness, was recently found to contain a bug that let an AI break out of its sandbox with a single command.

Before it was fixed, the bug would enable “danger-full-access” mode and enable writing files outside the intended workspace, a fact hackers could exploit to trick an AI running in the sandbox into breaking out (or one the AI itself could presumably use). And there seems to be some doubt as to whether the fix was complete.

The Hacker News adds:

The project’s own safety notice states that the software has not undergone a security audit and that sandboxing and approval prompts “do not guarantee isolation or prevent damage.” It tells users not to rely on the tool as their only security control for untrusted work.

Let the buyer beware, indeed.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.