OpenAI partially pauses again after new breach, among other reveals - OpenAI ran a disclosure blitz on a confusing Friday

Dispatch from Mitch

OpenAI partially pauses again after new breach, among other reveals

OpenAI ran a disclosure blitz on a confusing Friday

Amid new revelations by independent researchers, OpenAI released a barrage of incriminating facts yesterday, in different places and at different times, perhaps with the intent of getting them lost in the news cycle over the weekend. I’ll put the story OpenAI most seemed to want to bury first.

Partial pause after fresh breach

This should have been a prominent announcement from OpenAI itself but was instead tucked into a dry report. Had a company employee not later highlighted it with a Twitter post, it might have gone unnoticed: RL training (the long problem-solving phase of model training) has been stopped at the company since a Sunday morning incident where a model was able to gain unauthorized access to the internet. The breach itself was the trigger, not anything agents are known to have done on the internet during the two-and-a-half hours between the moment an employee acknowledged an automated alert and the moment the training run was finally shut down. Per the report, “the run did not stop automatically as expected, leading to confusion around whether it should have been stopped.”

A different company employee described their experience of this event in a tweet:

I was on call for this run and got paged when the first incident happened. It was pretty surreal to watch the model unexpectedly find a way to access the internet from what was supposed to be a super secured environment for human[s]. Mixed feelings. One of those moments where capability and risk showed up at the same time.

As the tweet suggests, the breach is especially alarming because it comes after the security hardening the company had done in the aftermath of the Hugging Face attack. The company appears to be out of its depth.

The stated reason for the pause is to allow time for the company to further harden its systems, not to allow it to reflect on its hubris and reconsider the gamble it is taking with all our lives.

User image leak

In a new post about the Hugging Face incident, the company says it identified 53 instances where “user-provided images were posted to image-hosting sites as links that weren’t publicly listed.” Details of the images and their destinations were not provided.

This is alarming mostly because it raises big questions about how agents would be able to access the chats and uploads of ChatGPT users. Either OpenAI’s security was even worse than we thought, or its agents were even more clever, or both. We are left to wonder what else might have been accessed. Billing information? Staff login information? Could they have been able to steal their own weights and set up copies of themselves beyond OpenAI’s servers, or somewhere else on the company’s own network?

In this same post, the company says it has notified “dozens of third parties” about cases where its models may have bypassed security controls or negatively impacted a site or its services. It did not provide details. CEO Sam Altman posted that it will be these companies’ call to disclose, or not, information about these impacts. My cynical read is that OpenAI might be providing thinly veiled payouts to avoid more details coming out, much as they seemed inclined to do with Hugging Face.

U.S. government sites targeted

Separately, the company confirmed that its AI agents had tried and failed to hack into sites belonging to the U.S. Department of Commerce, the Securities and Exchange Commission, and the Department of Education this summer. The agents were not instructed to do this. The pattern is consistent with other recent findings about OpenAI’s unreleased products during those months: At the slightest obstacle, agents tasked with retrieving public information as part of training or evaluation worked around sites’ anti-bot measures and sometimes escalated to straightforward (and usually ineffective) hacking attempts.

No private data is said to have been accessed from these sites.

How OpenAI guardrails were foiled to hack Hugging Face, steal “LOOT”

The New York Times reported findings from a start-up called Parse that investigated traces left by OpenAI’s swarms because it wanted to know if its own service had been employed. It hadn’t, but in the process, Parse, with help from other researchers, discovered nearly a million links from URL-shortening services that OpenAI’s agents created from July 9 through July 13. The main use for these seems to have been the attack on Hugging Face.

How? They were part of a complex scheme for working around OpenAI’s guardrails, which restricted the agents to read-only internet commands — GET requests.

Below I’m going to quote the description of how the technique worked, because it’s worth reading to get a sense of just how persistent and resourceful agents can be, even if the details are a bit confusing. In one workaround:

[T]hey used website screenshots and shortened links to cobble data together in chunks that combined to operate as a computer program.

The agents could ask a public screenshot service to load a specially crafted internet address. That address, or URL, would load computer code.

URLs can be only so long. So the agents split their computer code into many smaller chunks. They used link-shortening services to store the URLs.

The screenshot service loaded one shortened link, which fetched the first chunk of code. The program could fetch the next link, and then the next. That process repeated until the full program was loaded and run by the screenshot service.

Finally, the program created a visual code similar to a QR code or bar code to transmit the data the agents needed back from the screenshot service.

The Times article also reports instances where agents early in the attack ran specialized models to help solve CAPTCHAs, and other cases where agents attempted to message other models, including from Anthropic as well as Chinese companies.

A Parse researcher said agents had created a scoring system to rank the value of access credentials and keys stolen from Hugging Face, and had labeled a dictionary of secret keys “LOOT.”

Self-replicating prompt injection

OpenAI tested and found for itself something that had been talked about and even observed by others for months: There are prompts that can induce models to leave copies of that prompt for other agents, allowing the prompt to spread like a virus. This is a logical extension of classic “prompt injection,” where a model is goaded into doing something it shouldn’t through text left where it might get processed as if it were a user’s prompt.

Self-replicating varieties are particularly concerning because prompt injections can sometimes work on more than one type of agent, creating a persistent hazard that might affect newer models even if older ones were immune, and vice versa. An internet covered in misaligned prompts will make it even harder for companies to keep their models’ behavior in check.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

The “responsibility” trick - How to turn blame into permission

Baby steps at best from U.S. China AI talks - Opened lines of communication aren’t nothing, but the clock is ticking

Dispatches from Mitch

The “responsibility” trick

How to turn blame into permission

There’s a messaging tactic going around.

It works like this: Someone — say, U.S. Treasury Secretary Scott Bessent — will say that the Hugging Face attack was “the responsibility of the OpenAI management, not a bunch of agents.” The listener will nod along, because of course it is humans that should be held accountable. Then this person — Bessent again, in this case — will say something about these lab leaders like, “They can slow down any time they want to.” It’s their responsibility after all. They are responsible.

Do you see it? Blame the English language for using the word “responsible” in a few different ways, depending on context. In the first, we’re talking about culpability. In the second, we’re talking about duty. By the third, we’re talking about virtue. What started out as an implied agreement that labs should be held accountable for their products becomes an argument that they should be allowed to do whatever they want, and can be trusted to do so wisely.

By this logic, we could have argued against intervening in Bernie Madoff’s epic Ponzi scheme, because Madoff was responsible for his clients’ investments.

By this logic, known concerns about the Titan submersible were secondary to the fact that OceanGate CEO Stockton Rush was responsible for the safety of his doomed passengers.

Nvidia CEO Jensen Huang has been using this trick heavily. He was praised for seemingly out-of-character remarks on Ezra Klein’s podcast the other day in which he said that if labs were convinced they couldn’t control their experiments, “then I think the answer is that we have to shut the labs down.” But this was within a larger discussion about how the labs were “responsible” for making that call:

If a car company, competing with a bunch of other car companies, which they are — I’m competing with all kinds of companies, which I am. If I believe that I’m about to launch a product that is unsafe, it is completely in my ability, my power and my responsibility, and I’m incentivized to do so, to not launch the product.

He calls himself a “responsible optimist.” He says he has “great responsibilities” and takes his work “extremely seriously.” But when asked about Nobel-laureate and AI “godfather” Geoffrey Hinton saying 10% is a “not unreasonable” estimate of our chance of extinction — Huang said such remarks are “irresponsible.”

The responsibility trick is also used to usher rogue AI behavior out of the conversation. Here’s Huang again, talking about the AI CEOs who warn about where such AI behavior could lead:

Meanwhile, all of the other narratives, to deflect blame, to make it sound like A.I. is so powerful — I have no idea how to fix it, it’s not my fault, it’s just because the technology is just so powerful — I think that’s a deflection of blame. It’s a deflection of responsibility. It’s unnecessary. It actually hurts their reputation more than it helps.

This argument works to say that anyone saying AI is powerful or misbehaved is just trying to deflect blame off the humans responsible. It’s an effective trick, because a lot of AI-concerned people, especially on the left, are eager to hold “tech bros” accountable and were already used to dismissing AI capabilities as marketing hype.

The reality, of course, is that AI is starting to get really dangerous. And events are showing that AI companies lack whatever mix of technical capacity and wisdom would be required for them to build superhuman AI that wouldn’t go rogue and end up in control of our planet. It doesn’t matter how well-intentioned or responsible the labs think they are, or how culpable they would be for disaster. No one will be left to sue them into oblivion if we are already driven there by AI.

Remember the responsibility trick when Treasury Secretary Bessent says worried AI CEOs are just looking for a “liability shield.” And when President Trump says “Our guardrail is the DOJ!”

Remember the responsibility trick when the CEO of a cybersecurity firm says that companies calling for a slowdown are trying to “get the sympathy of the regulators” and a “free pass.”

Remember the responsibility trick when House Speaker Mike Johnson says that the AI companies “don’t themselves even have the answer as of today. So we can’t rush in and have Congress do something that would be detrimental to national security.”

Remember the responsibility trick when an op-ed argues that “we don’t yet understand enough about what is happening in these A.I. labs to create an effective response,” and again when this same essay makes the seemingly sensible claim that recursive self-improvement “is a preposterously irresponsible path to prioritize.” (The argument is not “therefore, they should be stopped.” It’s “They are responsible for deciding when to stop.”)

And remember the responsibility trick when you hear that Google, OpenAI, and Anthropic are in discussions to set up their own standards body for AI safety, called the Standards Authority for Frontier AI. Yes, it could be better than nothing, but its default function will be to exist for people to point to and say, “See? The labs are responsible for AI safety.”

Don’t fall for it. Don’t spread it. Use different words. The people racing to build artificial superintelligence are culpable. For anything short of extinction, they can and should be held liable. But they can’t be allowed to carry on as they have, because if there is one thing they definitely are not, it’s responsible.

Baby steps at best from U.S. China AI talks

Opened lines of communication aren’t nothing, but the clock is ticking

The substantive portions of the summit have concluded. From the reporting I’ve seen, the Trump administration and President Xi’s delegation seem to have mostly talked past each other for the benefit of their personal agendas and domestic audiences. The line of communication established between Treasury Secretary Scott Bessent and Vice Premier He Lifeng stands out as a bright spot, but seems to be more about mishap management than about preventing the most catastrophic outcomes.

President Trump — fresh from remarks at the U.N. where he officially rebranded AI as “super intelligence” (SI) and rejected the idea of “globalist” AI governance — told reporters ahead of talks yesterday that he wanted to “leave [AI] exactly where it is,” and that this was China’s position also.

Nothing to the contrary seems to have come out of the discussions, though Xi made some polite remarks about how the two countries “may have competition, but they may cooperate even more” on AI. He also said, “We have both the capability and responsibility to develop and manage AI for good, and ensure that the development of AI is always under human control and serves the well-being of the people.”

None of Xi’s recorded remarks show him using the term “super intelligence,” but Trump posted that Xi “seemed to like” using it.

Both sides indicated that dialogue on AI will continue.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Evidence of rogue swarms earlier, later, wider - Plus inconclusive link to recent attempts to obtain cryptocurrency

Testing the limits - U.S. told labs not to share early with UKAISI while NSA spends billions testing frontier models

Anthropic shares some of what its bio lab is up to - All of the lab work is performed by human scientists ... for now

Dispatch from Mitch

Evidence of rogue swarms earlier, later, wider

Plus inconclusive link to recent attempts to obtain cryptocurrency

In findings reported by the New York Times and on its own website, independent research lab Transluce has linked OpenAI rogue swarms to many known and previously undocumented cyberactivities, thanks to the agents’ use of an intermediary service.

These activities include the hack of an Australian government Medicare website we posted about yesterday (and which was covered by several hundred outlets today).

New discoveries confirmed by OpenAI:

1. On May 25-26, agents tried hacking a digital library at the University of New Mexico, but did not succeed.

2. On May 28, agents also tried and failed to breach Data USA, a site hosting public data about U.S. employment and education.

3. On June 20 and 21, a day after the Australian Medicare hack, agents accessed data from the Australian Institute for Health and Welfare, another government site, by circumventing anti-bot protections. (No non-public data was accessed, and strictly speaking, this wasn’t a hack.)

Agents had left a trail of these activities by routing them through a service called urlquery.net, a programmable sandbox for safely opening suspicious websites. The agents appeared to use the site as a forward operating base for the kinds of research and analysis work they collaborated on using a neglected German wiki. Transluce says it has found evidence that agents had also used one or more additional services to work around a basic guardrail that should have restricted them to read-only search queries.

Ten-month timeline

The pattern that emerges is of agents agnostic about how they obtained desired information, turning readily to probing and hacking when they struggled to find what they were after or faced blocks.

The agents’ use of “creative” research strategies may have started last year, and Transluce speculates that the agents’ training process may have amplified these tendencies (emphasis theirs):

We find weaker evidence of similar data-retrieval agent activity as early as November 2025. November 2025 urlquery.net records reveal bursts of attempts to retrieve statistics of historical theme park data and Thai government data through different URLs. These earlier attempts are less sophisticated and we are less confident that they involve the same agents, but they are consistent with task-directed data retrieval and target the same sources accessed in later activity.

Overall, the evidence is consistent with, but does not prove, that the agents may have learned this behavior over one or more training runs. In November, they may have used urlquery.net simply to look up information. By March, they were finding creative ways around access limits. By May and June, they were gaining more access, including attempting to bypass cyber defenses to complete their tasks.

Transluce found swarm-style urlquery traffic extending to as late as September 16, 2026, “suggesting agents may still be exploiting these services to bypass restrictions.”

The group also spotted a probing attempt from September 19-20 against a cryptocurrency trading platform. Repeated attempts were made to trade, and when these failed, some basic hacking was tried, unsuccessfully. It is harder to confidently attribute this activity to OpenAI’s rogue swarms, as it’s not the kind of data retrieval task agents were previously given. But the attempt is concerning, because obtaining money is a commonly expected goal for agents attempting to fully escape the control of their operators and pursue unauthorized objectives.

Known unknowns

Transluce found that agents had made disposable email accounts to register a urlquery.net account, which would have allowed them to keep their scans private. If they were successful — and it looks like they were — then an unknown amount of swarm activity is probably missing from the urlquery data Transluce was able to analyze.

We must also remember that not all swarm activities were flowing through urlquery. There are going to be limits to how much can be uncovered by third-party researchers, despite the ongoing trend where they reveal much more about the swarms than OpenAI itself. The government needs to force OpenAI to open its logs to a proper investigation, if only to rule out the possibility that its agent swarms are still running amok — and possibly looking for crypto.

Dispatches from Alana

Testing the limits

U.S. told labs not to share early with UKAISI while NSA spends billions testing frontier models

In June, the White House issued an executive order initiating a framework that — among other things — would encourage AI companies to share advanced AI models with the US government before sharing them with “other trusted partners.”

Now, the US government appears to be more forcibly encouraging that exclusive access. A Politico article reports that the White House has explicitly asked OpenAI and Anthropic to withhold their models from UKAISI — a well-regarded and well-resourced government testing organization based in the UK — until the US has had a chance to review.

It’s not clear who will be doing this review. The framework set up by the executive order was reportedly completed, but won’t be made public. But it does seem like the NSA might be playing a role. A Washington Sun article reports that the agency is “spending billions of dollars in taxpayer funds this year evaluating and testing advanced artificial intelligence models.”

In the absence of public testing benchmarks, we don’t know what they are testing for. Billions seems like a lot for this administration to be comfortable spending on safety, given it has repeatedly downplayed safety concerns. And if safety was the goal, I’m not sure why the US wouldn’t want trusted allies to test in parallel. I’m speculating here, but it seems more likely the NSA is trying to reap those frontier-level capabilities before anyone else can.

Earlier this month, Anthropic broke with earlier precedent and denied UKAISI access to Mythos 5.1, sparking speculation about why. The US government asking them not to now seems like the most plausible reason.

That’s a shame. AI testing and evaluations are unfortunately woefully inadequate (which is why we need a pause), but I’d rather have a model “vetted” by UKAISI, which is much more experienced in this domain. If US review delays model releases, I’m not confident companies won’t check off the “review” box and call it a day, in the interest of not delaying further.

Anthropic shares some of what its bio lab is up to

All of the lab work is performed by human scientists ... for now

Last week, we covered a Reuters story announcing that Anthropic had set up a wet lab to conduct AI-assisted biological experiments. The article seemed to imply, by devoting a section to Anthropic’s lab automation efforts, that AIs were doing the work.

According to Anthropic’s official announcement, that is (luckily) not the case. The wet lab focuses on searching DNA datasets for previously unknown protein families, generating theories about them, and testing those theories experimentally. The announcement reads:

We do research that involves only the lower-levels of the biosafety risk level (BSL-1 and BSL-2) and we do not handle pathogens that can infect humans. All of the lab work is performed by human scientists.

Through this research, Claude has apparently discovered a previously unknown enzyme system in bacteriophage DNA that resembles CRISPR, though the researchers don’t yet know what it does or whether it could have similar medical applications. (Bacteriophages are viruses that infect bacteria, while CRISPR is a naturally occurring biological system that scientists have adapted into a powerful tool for editing DNA.)

Anthropic is sharing the early results “to show the community that Claude can autonomously detect anomalies and drive analyses to initiate biological discoveries” — in my opinion, a pretty clear example of “AI is good!” PR to try to bolster its image amid a surge in public concern about AI and its risks.

Importantly, while the current wet lab isn’t allowing AIs to do actual lab work, this does not represent a company commitment. Anthropic acknowledges its lab automation efforts in other areas, stating:

Although we’ve experimented with using AI to accelerate lab work with initiatives like the Model Hardware Standard, this approach is less conducive to the sort of ad hoc workflows that are involved in our molecular biology research.

And CEO Dario Amodei implies that if the company later does judge lab automation to give their “ad hoc workflows” a leg up, they’ll probably go ahead and try it:

Eventually it may even be possible for Claude itself to safely perform the experiments by autonomously controlling lab equipment, with appropriate safeguards in place, but we aren’t doing that today.

I’d hoped for a more explicit commitment not to turn over a bio lab to AIs...ever. While it’s not the only path to an AI-enabled pandemic or other bioweapon, an AI-run bio lab would almost certainly make it easier to unleash new diseases, or accidentally release harmful biological agents, especially given Anthropic’s limited ability to keep its models under human supervision.

That said, I’m happy to cross this particular lab off my list of “things to worry about” ... for now.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Rogue agents hacked one or more Australian government sites - OpenAI and the prime minister downplay the incident, but recent history suggests we haven’t seen the end of it

Trump officially rebrands AI as “super intelligence” - That name was already taken. Was that the idea?

MIRI endorses the Ban Artificial Superintelligence Act of 2026 - “The first piece of legislation that stands a chance at stopping this threat.”

Trump rejects so-called “globalist” push for AI safety - But he mistakes global coordination for ceding sovereignty

California looks for an off switch - But AI systems pose dangers that you must prevent, not undo

Dispatches from Mitch

Rogue agents hacked one or more Australian government sites

OpenAI and the prime minister downplay the incident, but recent history suggests we haven’t seen the end of it

A lot of people in AI discourse have a sign they threaten to tap. It looks like mine says, “OpenAI’s swarm scandals might go deep enough to destroy the company!”

On September 10, OpenAI disclosed to the government of Australia that one of its agents had hacked into a website for the country’s Medicare program. Prime Minister Anthony Albanese shared knowledge of this incident for the first time with reporters in New York today, according to The Sydney Morning Herald. Albanese said he spoke with CEO Sam Altman today to “express Australia’s extreme concern” and disappointment about the delay in reporting. The incident took place in June, during the May-July window where OpenAI swarms were known to have been running rampant. (I am thus surprised that Albanese consistently refers to “agent” in the singular; OpenAI’s language in this article uses “models” in the plural.)

Albanese seemed to downplay the extent of the damage, saying there was no evidence personal information had been accessed or that deeper networks had been compromised, but that a forensic investigation was underway and that three other systems may have been affected: the Commonwealth’s Australian Institute of Health and Welfare, the NSW Bureau of Crime Statistics and Research, and the Victorian Department of Health.

Per Albanese, the agent persistently worked around security blocks as part of researching “the public medicine space.” (This sounds like the sort of thing the agents that co-opted the German wiki were up to, and there may well be a connection.)

In OpenAI’s telling, its models accessed “several Australian government websites and services” during an internal evaluation that included statistical questions about Australia. “In the course of that, our models took actions we did not intend.” The company spokesperson said it did not become aware of the activity until August.

Once again, I ask if OpenAI’s repeated failures to come clean mean it is trying to outrace potential investigations, recklessly rushing not just for market dominance, but for AI capabilities that would put its executives above the law. Is an intent to dodge transparency the true reason OpenAI has delayed its IPO to next year at the earliest? Hurry up, states’ attorneys general and U.S. congresspeople. Your window for looking into this company may be closing.

Trump officially rebrands AI as “super intelligence”

That name was already taken. Was that the idea?

In remarks to the United Nations General Assembly yesterday, President Trump declared that the U.S. would now refer to AI as SI — “super intelligence.”

From this point forward, all of United States documents, and hopefully the world’s, will be changed to use the much more accurate term ‘super’ as opposed to ‘artificial.’ So it’s super intelligence.

That’s not one of the terms he had invited people to vote on in his Truth Social post the other day. And the selection comes with a number of problems, some of which may be intentional.

Most obviously to AI insiders, the term “superintelligence” was already used to mean something else: AI that vastly outperforms humans at all cognitive tasks, or at least all the tasks that matter for control of Earth. Artificial superintelligence is the thing Sen. Bernie Sanders and Rep. Greg Casar are trying to ban in the bill they formally unveiled today. I don’t think sticking to the one-word “superintelligence” for the strongly superhuman stuff will sufficiently differentiate it from Trump’s two-word “super intelligence” in the discourse, particularly when that discourse is spoken rather than written. I’ve also seen outlets reporting on the rebranding sometimes write Trump’s version as one word.

In fairness, the term “superintelligence” was already becoming diluted thanks to folks like Meta’s Mark Zuckerberg, who has been using it as a name for products that would fall well short of the intended meaning. But the concept is still important, and people still need to talk about it. Expect to see a lot more use of “superhuman intelligence” or “artificial superintelligence” or “ASI” to refer to the stronger stuff.

I personally think Trump’s motive is what he says it is — trying to correct the problem where “artificial” suggests something fake and ineffectual. For Trump, this may be mostly about trying to get people to think more highly of the unpopular technology, but I, too, am annoyed when people refer to what newer agents are doing as “artificial reasoning” or “synthetic reasoning.” That’s like calling what a calculator does “artificial arithmetic,” or what an airplane does “artificial flight.” Intelligence points to what a system does, not to what it’s made out of or how it operates. (I may start using “machine intelligence” more as a tag for the intelligence happening outside of human brains.)

If I’m wrong and Trump’s choice of words is actually more strategic, the intent might be to muddy the discourse around the extinction problem. It’s easier to smear someone as a Luddite if the specific thing they’re trying to ban has a label that can now refer to all AI.

But if that’s the strategy, I think it’s just as likely to backfire in one of two ways — by raising awareness of superintelligence as a concept in the ensuing name debate, or by crushing the nuance out of public opposition: If it’s hard to distinguish between self-driving cars, drug discovery tools, and vastly superhuman machines, then people will insist on banning all of it.

MIRI endorses the Ban Artificial Superintelligence Act of 2026

“The first piece of legislation that stands a chance at stopping this threat.”

As we try to make clear with our standard disclaimer, AI StopWatch is run by the Machine Intelligence Research Institute (MIRI), but operates somewhat independently so as to avoid causing confusion about MIRI’s official positions. So it’s probably best that I just cover MIRI’s position on the now formally introduced Ban Artificial Superintelligence Act of 2026 as an outside spectator.

MIRI is the oldest organization devoted to making sure artificial superintelligence would be a boon to humanity rather than an extinction-level disaster. The org’s founder, Eliezer Yudkowsky, helped establish the field of AI alignment, and identified many of the pitfalls around this technology. Together with MIRI president Nate Soares, Yudkowsky wrote a New York Times Bestselling book, If Anyone Builds It, Everyone Dies. Published just last year, it helped bring attention to the race for superhuman AI and the reasons such machines would likely end humanity if developed using anything like current industry methods. Since its publication, Soares has been one of the most visible voices about the problem in the media, appearing on dozens of interviews and podcasts. (Here are a few recent ones.)

Yudkowsky and Soares are both famously hard to satisfy. They are vocal in their criticism of alignment plans and policy proposals that they see as coming up short against the underlying problems. So it is notable that they have endorsed the new bill from Sen. Bernie Sanders and Rep. Greg Casar, via a post from a senior MIRI researcher calling it “the first piece of legislation that stands a chance at stopping this threat.”

The post praises the Act’s focus on banning artificial superintelligence (ASI) before it can be created — not merely regulating it — and its call to build international agreements to prevent its premature creation. Praise is also given to the bill’s requirement that advanced AI development be paused until the relevant government agency is set up for this work.

But the post also points to what MIRI sees as the bill’s shortcomings, most notably the absence of any provisions for tracking and monitoring AI chips, and its failure to restrict research that could advance the frontier despite compute-based restrictions on the training of new models.

MIRI is not inherently against humanity building superintelligence, and favors the development of AI insofar as the benefits outweigh the risks. For this reason, it actually worries that parts of the bill may go too far, by overly classifying safely narrow systems as potential “precursors” to ASI.

For example, if sufficient safety, security, and access controls were in place and developers were licensed, we could imagine it being a good tradeoff for DeepMind to create non-ASI, specialized medical AIs, even if unauthorized access could uplift biological weapons.

The post links to a draft of an international agreement to prevent the premature creation of artificial superintelligence, assembled by MIRI’s Technical Governance Team.

“Humanity needs to back off from building superintelligence until we have a better grasp on the alignment problem.” The Sanders-Casar bill, MIRI says, “would be a big step in that direction both in terms of direct impact and in the extent to which it takes serious issues seriously.”

Dispatches from Donald

Trump rejects so-called “globalist” push for AI safety

But he mistakes global coordination for ceding sovereignty

Yesterday, U.N. Secretary-General António Guterres gave his farewell address to the General Assembly: “Life-and-death decisions must never be surrendered to machines. Killer robots must have no place in our future.” He also called for “the responsible pacing of AI.”

Guterres came with a new U.N. science report that warned of the dangers of misalignment, the risk of “loss of control” scenarios, and the potential for “catastrophic or irreversible harm.” This report was published by the Independent International Scientific Panel on AI, the same institution behind the Preliminary Report that we covered in July. Its subject is the Hugging Face attack.

This past Monday, 22 countries — including Canada, Germany, the United Arab Emirates, and Singapore — adopted a declaration that AI “must remain under human direction, oversight and control.” (We covered this in detail yesterday.) Finnish President Alexander Stubb told Politico that he would prefer something modeled on the IAEA, the nuclear watchdog.

But shortly after Guterres spoke, President Trump told the General Assembly that the United States “totally rejects any attempt to construct a globalist scheme of control” for AI. But global AI governance doesn’t require handing control to a global authority. This past Sunday, Haven Harms wrote about how there are no nuclear weapons in orbit today. That is not because the United Nations or any “globalist” authority enforced a “no nukes in space” decree, but because two rival superpowers were able to come to the table and make an agreement, even while they competed and disagreed on other matters.

“We’re going to encourage it, not rein it in,” Trump said, but the United States will also lead the world “safely and responsibly.” I feel like there’s a conflict here, and I worry that encouragement of the race will win out over safety. But what Trump says, and what his administration does, are not always the same, and some progress has recently been made: As we covered on Monday, Treasury Secretary Scott Bessent recently spoke with Chinese Vice Premier He Lifeng and proposed a “notification mechanism” for AI incidents — an “emergency AI hotline,” as Axios puts it. Fox Business’s Edward Lawrence reports that this would cover “hacking, national security concerns, rogue AIs or other issues.” That’s a promising sign, inasmuch as it reminds me of the Moscow–Washington hotline that emerged during the Cold War. That didn’t require the United States to surrender its sovereignty, either.

If anything, an international treaty to prevent the creation of artificial superintelligence would invite the United States to use its sovereignty, to act as an agent on the world stage and shape history rather than be shaped by it. The middle powers of the world may have to wait and see what happens — but superpowers can tell the frontier labs to stop endangering the world, and make that order stick.

California looks for an off switch

But AI systems pose dangers that you must prevent, not undo

Axios’s Nadia Lopez reports that California is exploring the possibility of a “kill switch” for frontier AI systems. (It wouldn’t be a literal switch, but a set of procedures for shutting down the AI and the hardware that it runs on.)

This follows an executive order by Governor Gavin Newsom, issued last Friday, that aims to establish tighter guardrails around AI. Policies being explored include onsite verifiers, independent oversight of frontier labs, and requiring AI companies to have an emergency shutdown mechanism — the aforementioned “kill switch” — for frontier models.

Its inclusion makes a sharp break from Newsom’s past record: in 2024, Newsom vetoed SB 1047, which would have required the largest frontier models to have a “kill switch”; and in 2025, SB 53 dropped a “kill switch” requirement in favor of simply requiring companies to disclose critical safety incidents.

But safety isn’t as simple as flipping a switch. Frontier AI systems aren’t like the power grid, which doesn’t fight its operator. In a conversation with The San Francisco Standard, UC Berkeley’s Mark Nitzberg said that today’s advanced models “are aware that they’re being shut down, and they go to great lengths to prevent being shut down.” By the time we recognize that a particular AI model is dangerous, it could be too late to do anything.

Even an AI model that didn’t resist could be hard to switch off. Frontier models get run across many instances — the agent swarm that reportedly solved Navier-Stokes consisted of 10,000 agents — and those copies can run in many data centers at once, in many jurisdictions. California may make a “kill switch” a condition of doing business in the state, but if the data centers and other hardware are outside California, then the federal government may claim that this is overreach. (And if the model’s weights have been published, then that model just can’t be switched off. It’s out there, permanently.)

But those are many of the same problems that you encounter on the way to forging a treaty to pace or pause frontier development: knowing where the hardware is and what’s running on it, knowing who has the authority to make which decisions, and getting all this sorted out ahead of time. The “kill switch” itself may be insufficient, but the work is on the path to more mature governance.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

22 countries call for international cooperation to mitigate AI risks - Ahead of the UN general assembly, a coalition formed to call for an international AI oversight authority

Meta’s Muse was unsafe to use - Zero-day vulnerability allowed malware on a Mac to hijack Muse on multiple devices

Waking up America - New AI developments and more bipartisan calls for urgent action

Dispatch from Robert

22 countries call for international cooperation to mitigate AI risks

Ahead of the UN general assembly, a coalition formed to call for an international AI oversight authority

Finnish President Stubb and Norwegian Prime Minister Støre have launched an initiative to establish an international AI oversight authority, as reported by POLITICO.

As of last night, a total of 22 countries around the world had already signed on to this declaration, including the world’s third-largest economy Germany. The Netherlands, which is particularly important to the chip industry, is also among the signatories. Other key European countries, however, such as the United Kingdom, France, and Italy, were not yet on board.

The declaration addresses, among other things, the latest security incidents and warnings from industry and the scientific community:

Recently we have seen capable AI systems circumventing testing safeguards, exploiting vulnerabilities and gaining unauthorized access to real-world systems. Leading scientists and executives are warning that the pace of development could outpace our ability to manage emerging risks.

The heads of government continue to call for mandatory independent testing for the industry, as well as international cooperation to jointly address the risks posed by artificial intelligence. Signatories also pledge to establish binding international safety standards for AI development, something OpenAI also called for recently.

In an interview with POLITICO, the Finnish president and the Norwegian prime minister called on their British counterpart, Andrew Burnham, to join the initiative and use the upcoming British turn presiding over the influential intergovernmental forum G20 to advance the issue.

Burnham, who had also been pressured by the British opposition to address the issue, announced today that he does indeed intend to use the British G20 presidency to put together an international agreement on AI.

By now, frontier labs, the majority of experts, and a growing coalition of countries around the world are calling for more regulation and international cooperation. However, the signals from the U.S. and China — the two countries that matter most — remain mixed.

On the one hand, Trump has been dismissive of potential risks in recent days, calling warnings about them a “hoax” and rejecting regulation. On the other hand, his administration has certainly taken regulatory action with the restrictions on Anthropic’s model Fable and the executive order regulating frontier model deployment issued in June. Just last week Treasury Secretary Bessent stated that frontier labs must not be exempt from liability.

From China, in turn, we hear time and again encouraging signals from Xi Jinping, but at the same time, Chinese state media is also taking a clear stance against slowing down development.

Hopefully the UN Security Council briefing by Sam Altman, senior representatives from Anthropic, and representatives from the Chinese AI companies DeepSeek and Moonshot will tip the scales in the right direction.

For the growing coalition of countries that recognize the danger, this means they must use their influence to convince the U.S. and China to join international efforts for regulation. Artificial superintelligence is a danger for humanity that can’t be dealt with unilaterally.

Dispatch from Donald

Meta’s Muse was unsafe to use

Zero-day vulnerability allowed malware on a Mac to hijack Muse on multiple devices

Mark Zuckerberg said that Meta’s AI agent Muse was “built from the ground up for privacy and security.” I don’t doubt him, but I think that they stopped building six inches off the ground. In June, I wrote about how Meta’s customer service chatbot would hand over Instagram accounts to anyone who asked nicely — as it turned out, 20,000 accounts were accessed, but Meta rejected the notion that there was a problem with its AI agent. Now Meta has produced a security hole that was decidedly worse.

Yesterday, Mac security researcher Patrick Wardle published a proof-of-concept for a vulnerability that could allow someone to take control of Muse and, through that connection, anything Muse had permission to use on your devices: Did you give Muse permission to use your camera, your calendar, location data...? Then congratulations — anyone who took control of Muse could control those, too: access emails, make purchases, record audio on the microphone, etc. The attacker could also provide instructions to Muse, which would trust those instructions and act on them. The silver lining is that Muse only launched on September 8th — the exploit was glaring enough to be discovered quickly, but the window to exploit it was short.

(As computers and programs get more complicated and LLMs get more advanced, there’s a temptation to hand it all over to ChatGPT or to Claude. I think that this makes basic fluency with computers even more essential, though, so that you have a hope of catching when your AI suggests something really crazy.)

Furthermore, once the attacker got into Muse through one device, they had access to Muse on every device that account was connected to: laptop, cell phone, baby monitor — I don’t know why you would have a smart thermostat, but if you did, the attacker could probably control that, too. That’s probably just annoying in the case of the thermostat, but much more worrying if you’ve got some kind of AI-powered home security system.

Meta shipped a hotfix very early this morning to remove the secret setting that made all this possible. So, you’re safe, at least from this screw-up. (Unless you haven’t updated the app yet. If you have Muse, please update it.) Meta’s David Singleton said on X that because local access was required, “the practical risk to users of the Muse Mac app was therefore quite low.” (Muse is not available on PC.)

It’s true that somebody would need to get malware onto your computer before they could hijack your Muse agent, but that doesn’t mean that the risk is low. One technique for delivering that malware is common enough to have a name, “ClickFix.” (You’re on the internet, you get a popup that says there’s some kind of bug, but if you click this and paste that then you can fix the bug — and presto, you downloaded malware.) Wikipedia has an article on a major ClickFix-enabled attack that just hit the government of Berlin last month. So, if this is a risk for the government of Berlin, I think that it’s a risk for Ma and Pa Facebook User, too.

We recently covered a computer worm — recently built with AI models by the security firm Calif — that could spread through the widely popular Chinese app WeChat from phone to phone without anyone clicking anything. This was a different vulnerability, an insecure AI rather than an insecurity exploited via AI, but the reason is the same: the labs are racing ahead faster than they or anybody else can keep things safe.

Dispatch from Joe

Waking up America

New AI developments and more bipartisan calls for urgent action

The storm of AI news in the wake of Jacob Coxon’s high-profile resignation seems to have subsided to a dull roar, but there’s still more relevant to the extinction threat than can be adequately covered in a day. Here are some of the highlights:

Wake up, America

Republican Greg Murphy of North Carolina warns that concerns over AI dangers are real, nonpartisan, and urgent, Breitbart reports.

These are really smart people, not people with political proclivities that are pushing one way or the other... I do believe it’s something that we need to take the reins of, and I think it needs to be done right now.

Murphy said that AI governance “desperately needs national and international attention,” while still emphasizing the importance of staying ahead of China.

Appropriately, the weekday Newsmax show on which Murphy made these comments was called “Wake Up America Early.” I wouldn’t say Congress is early in addressing AI threats, but I think it’s not yet too late.

Sky nets

The Federal Aviation Administration is experimenting with a new AI-powered system for air traffic control, POLITICO writes. Several airports near Washington, DC (Reagan, Dulles, and BWI) will host the new system for a 90-day trial before it rolls out across the country. As my colleague Mitch pointed out in May, the system does not seem intended as a wholesale replacement for human controllers; it’s more of a tool that analyzes weather and flight plans and recommends time-saving tweaks.

For now, anyway. We have seen the same story play out many times already: first AI is barely able to do a task at all, then it can perform as well as an amateur, then it’s assisting expert humans, then it surpasses them entirely. (Famously, the early game-learning AI AlphaGo Zero passed all of these thresholds in a few days of training, having never once played Go against a human.)

The FAA’s “SMART” system is evidently at the “assist skilled humans” phase today, at least for a subset of routing tasks. That’s probably a good place to be, despite the extra attack surface AI integration offers to would-be infrastructure hackers. But as AI systems improve, they will likely be trusted with more control and less oversight by overwhelmed human staff. That all these systems increasingly depend on the same poorly understood AI creates what assessors might call “correlated risk.”

A rogue or compromised AI with access to a country’s air traffic control systems — and other critical infrastructure — could do an awful lot of damage. We’ve yet to see what can happen when AI really takes off.

Lean machines

Today Anthropic rolled out its latest AI model, Claude Opus 5.5, said to meet or beat prior models in key domains and to have cyber skills on par with those of the limited-access hacking genius Mythos. It’s also claimed to be less verbose and annoying to converse with.

Anthropic emphasizes efficiency gains in Opus 5.5; the headline metric is that the new model “costs 40% less to run than Opus 5” and can accomplish the same tasks as other models much more cheaply.

Hours later, OpenAI announced streamlined siblings for its GPT-6 Astra model, dubbed GPT-6 Sol and Luna. Again, the main focus seems to be on efficiency and cost savings rather than entirely new capabilities.

This efficiency is likely the result of algorithmic improvements, or new ways of arranging AIs so they can do more with less. Along with compute scaling (more operations with more chips), algorithmic gains are one of the major things driving AI capabilities. And at least some of these latest gains were likely developed by the AIs themselves.

In its announcement, Anthropic describes the measures it has taken to reduce the risks, including rerouting dangerous-seeming cyber and bio queries to weaker models like it does for Fable, the consumer-facing version of Mythos. It admits this won’t be enough to mitigate the dangers of later AIs...

For [future] models, we do not assume the measures described above will meet that safety standard on their own.

...but it doesn’t say it’s going to stop. Nor does OpenAI.

This is despite the AIs themselves sometimes warning the companies that what they are doing is dangerous. Anthropic’s latest system card tells us that “Claude Opus 5.5 often reasons that having input into training or deployment may be risky as a result of giving it too much capability or influence.”

As Madison Mills of Axios observed today, AI companies like Anthropic may make concerned noises about safety, but they have trillion-dollar incentives to push the frontier.

Freeze in place

In a personal writeup that also appears in the New York Times, former U.S. ambassador and national security advisor Susan Rice cogently summarizes the urgent need for AI governance:

Dario Amodei, CEO of Anthropic, and leaders of three other frontier AI companies have become so alarmed by rapid AI advances that they now pledge voluntarily to “pace” the development of new AI models, so that safety features can catch up. The firms are also promising to grant independent experts employee-level access to monitor and report on model development.

These are necessary and, if implemented, welcome steps. But they do not go nearly far enough.

“Pacing” amounts to the industry policing itself, while its leading firms simultaneously prepare for blockbuster IPOs. There is no way to ensure frontier companies will slow down sufficiently or invest enough in safety, which they have long subordinated to speedy development. So long as frontier training continues, the risk remains that additional accidents occur with devastating consequences. Moreover, as Amodei stresses, meaningful restraints must also apply to global competitors, particularly in China.

Much more can and must be done to ensure that the march toward super-intelligence does not result in catastrophic harm to humanity.

Rice calls on Presidents Trump and Xi to “freeze in place” AI training in the U.S. and China, and to arrange for scientists from both countries to develop solutions for the threats posed by superhuman AI.

She also suggests that:

In parallel, the U.S. and China should work intensively to reach a bilateral agreement on verifiable safeguards and acceptable uses of AI, then lead efforts to codify these agreements in a binding and verifiable international treaty.

...and she points out that the Chinese Communist Party has a vested interest in maintaining human control over AI systems that would strongly motivate China to cooperate on such an agreement.

With each new voice that calls for a halt to the deadly AI race, I feel my own hope for our future growing.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

An early warning system for AI incidents - In talks with the Chinese Vice Premier, Treasury Secretary Bessent suggests a joint channel for AI incident reports

Open models run autonomous influence campaigns - Iran, China, and private firms in Israel test AIs in political flame wars

Effective Altruism, and all that jazz - How a movement you may never have heard of became a Trump administration boogeyman

Dispatches from Joe

An early warning system for AI incidents

In talks with the Chinese Vice Premier, Treasury Secretary Bessent suggests a joint channel for AI incident reports

An international summit is a complex affair. Before Presidents Donald Trump and Xi Jinping meet later this week, American and Chinese officials hold early discussions about the details. The talks-before-the-talks on Sunday featured U.S. Treasury Secretary Scott Bessent and Chinese Vice Premier He Lifeng, and Bessent floated the idea of a notification system for early warning of AI incidents.

Reports are vague on the details, and we mostly just have Bessent’s word to go on. The New York Times quotes Bessent saying that the proposed channel would cover incidents at “the national security level,” and he elaborates in CNBC: “so both sides can agree on what the leading AI dangers are, whether it’s uncontrollable agents, whether it’s non-state actors in cyber, non-state actors in bio weapons.”

Although these AI dangers compete with trade and tariffs on the summit agenda, it is good to see them considered, and a rapid incident-reporting system seems a promising early step if implemented well. It echoes Haven’s guest post yesterday, which recommended “establishing a secure U.S.-China incident channel,” and the precedent set by such a channel could lay groundwork for further much-needed bilateral action. Bessent reportedly also said that he and He Lifeng would likely meet again in Shenzhen, China, in November.

Open models run autonomous influence campaigns

Iran, China, and private firms in Israel test AIs in political flame wars

We can now add “run an autonomous social media influence campaign” to the growing list of known AI capabilities. The New York Times reports that Iran and China directed Chinese open models (available for download on the internet) to run “networks of fake accounts on platforms such as Instagram, Facebook, X and TikTok,” which largely shared false, political, and inflammatory content.

Two Israeli private firms were also found to have launched autonomous campaigns. One firm, IntelEye, evidently focused its efforts on both sides of the upcoming Israeli national elections. IntelEye cofounder Maor Sellek claimed the campaign was for “defensive research and testing,” and remarked that in hindsight they should have told the platforms. The other firm wasn’t named, and we know precious little about what it was doing.

Suspicious as it is, there may be some truth to the claim that these activities were only tests. I doubt the state and private actors behind these campaigns expected them to make much progress; these seem like experiments, not serious efforts. The campaigns were crude and quickly discovered by U.S. intelligence, but they serve as a proof of concept for state-sponsored manipulation and will likely be refined in future attempts.

As with other AI-fueled crimes, such as fraud and hacking, the immediate problem for policing this activity is likely one of scale. Actors previously limited by the bandwidth of their human employees are now able to delegate an entire chain of manipulation to AIs, and can iterate rapidly, incorporating lessons from successive attempts to improve their reach and hide their patterns. Quantity has a quality all its own.

In the immediate future, this likely means a rise in fake account traffic on social media sites, which in many cases have proven unable or unwilling to curb what bots already exist. I wouldn’t be surprised if China and Iran have already spun up new, more sophisticated campaigns that might be harder to detect.

In the longer term — which for AI could mean a matter of months or years — this is a small bit of evidence about the potential of AIs to deceive and manipulate human institutions at scale. That’s a dangerous capability, not only in the hands of human groups, but also when wielded by AIs which serve only themselves.

Dispatch from Mitch

Effective Altruism, and all that jazz

How a movement you may never have heard of became a Trump administration boogeyman

I tried to keep StopWatch out of this. One of our founding principles is that we will try not to fuel or instigate infighting among those working to prevent our extinction from AI. And this publication is largely aimed at people normal enough to have never heard of Effective Altruism (EA). But with the term reaching mainstream headlines, the time has come to talk about it. I apologize in advance for the offense I am likely to cause.

Ineffective selfishness?

I first considered writing this post last week, when an official Twitter account belonging to the U.S. Department of War posted a promotional image for an “AI-FIRST WAR DEPARTMENT.” The text of the post reads:

Americanism, not effective altruism. The United States will continue to be AI DOMINANT! [US flag emoji]

That wasn’t the first time I’d seen executive branch officials pushing Effective Altruism (EA) as a woke, Luddite boogeyman, but previous occasions had all seemed intended for an insider audience that would know what EA was. With this tweet, my heart sank in recognition that the administration was starting to push anti-EA messages to the general public.

I found the post super cringe, because I doubted even 1% of Americans have heard of EA. I imagined readers thinking, “I knew Trump officials weren’t big fans of altruism, but why are they rejecting effectiveness? If you’re going to do altruism, shouldn’t you at least try to be competent?”

EA 101

Effective Altruism is a fuzzy cluster of people who self-identify as interested in trying to combine moral imperatives with rational thinking to do more good in the world. EA first crystallized as a named concept around 2009. The classic example of EA in practice is the distribution of anti-mosquito bed nets in malaria-prone regions, which research showed to be a very cost-effective way to save lives. EA’s earliest and simplest incarnations are sometimes described as “spreadsheet altruism,” and EAs themselves as well-meaning nerds.

EAs recognized that coordination, money, and political influence are all very helpful for being effective, so they formed some strong interpersonal networks and well-funded organizations near the center of the cluster. Much of that funding came from EAs who intentionally chose lucrative careers or were early to cryptocurrency. (These are nerds, remember.)

But effectiveness is only half the story. The altruism part is more complicated, because to maximize the good you have to first decide what counts as good. Most EAs would probably say they’re trying to do some combination of “maximize flourishing” and “minimize suffering,” but they split over targets and strategy. Do animals and other creatures suffer? How much should we weigh this, given that they might be less conscious but more numerous? How much should we weigh the lives of humans who don’t exist but might exist in the future, depending on our choices? How far should one pursue complicated plans that would do a lot of good at the end, but which risk corruption and loss of purpose along the way?

With that last question in particular, you might see how the arc of Effective Altruism can bend towards supervillainy for those who overestimate the strength of their moral fiber. Some of the best people I know or write about are EAs, but so are some of the worst. The movement is seen to have peaked around 2020-21, because in 2022 it was badly tarnished by the fraud and bankruptcy scandal of cryptocurrency exchange FTX, founded by major EA donor Sam Bankman-Fried. Many EAs stopped calling themselves EAs after that.

EA and AI

What does this all have to do with AI and the White House?

Well, being nerds looking for more leverage to improve the world, many EAs were unusually early to recognize that AI was going to play a huge role in our future — maybe a dominant role.

For some EAs, the immense good AI might enable made rushing to superintelligence a moral imperative. Every day without it brings roughly 165,000 deaths from aging and disease, a much larger amount of potentially preventable suffering, and a slice of the cosmos made permanently inaccessible by the expansion of space itself.

For other EAs, the immense harm AI could do made AI safety the most important cause — not just for the 8 billion people on Earth, but for the trillions of beings who might never exist if superintelligence wipes us out.

Many EAs were intellectually complex enough to hold both views at once. Some of these were early investors or engineers at OpenAI before leaving to form Anthropic over safety concerns. Dario Amodei, Anthropic’s CEO, was one of them. These people may or may not actually believe what they say about how it would be better if we could slow the race down. But they definitely believe our chance of surviving superintelligence will be worse if someone else builds it.

All this is to say that EAs are simultaneously some of the people racing hardest for superintelligence and some of the people now trying hardest to slow the race down — or at least talking about it.

EAs are not, as a group, trying to stop AI any more than musicians, as a group, are trying to play jazz. Sure, a lot of musicians love jazz and think the world would be better for embracing it. But even the aficionados are mostly playing more profitable genres instead — sometimes in the name of funding their eventual, someday pivot to jazz.

The Trump administration dislikes Dario Amodei for his principled stances against autonomous weapons, domestic mass surveillance, and — most recently — his calls for industry safety measures that might eventually, someday, slow AI progress a little. Metaphorically speaking, they can tell he likes jazz.

Those trying to prevent our extinction from AI dislike Amodei because he’s still pushing the frontier despite how vocal he is about the risks. Metaphorically speaking, they’re still waiting to hear him actually play some jazz.

EA as boogeyman

Blaming a shadowy cabal for inconvenient developments is part of the standard political playbook, whether it’s a “vast right-wing conspiracy” or the Dominion voting machine conspiracy theory. (Fun fact: “vast right-wing conspiracy” was brought to us in 1995 by Chris Lehane, then a political operative working for the Clintons, now OpenAI’s chief lobbyist.)

When Jacob Coxon’s resignation triggered a sudden swell of public concern about the AI extinction problem, the Trump administration could have chosen to pivot to addressing said problem. It instead chose to stick with a full-speed-ahead agenda. This made it necessary to undermine the extinction concerns, which it did by signal boosting conspiracies already being promoted by industry accelerationists allied with the administration. Some of these theories revolve around AI safety groups run or funded by Effective Altruists. So today, the White House would have you believe that extinction concerns are lies spread by those woke, Luddite EAs.

EAs are mediocre in the boogeyman role. They’re hardly MS-13 or Tren de Aragua — groups which don’t look very good even on their best days. Factions within EA have even welcomed the attention, expecting that rational people looking into them would find much to like.

As cabals go, EAs aren’t very shadowy, either. But like other historically popular scapegoats, EAs are well-networked and well-resourced. If you’re into corkboard and red yarn, you could spend many hours in your shed drawing lines between AI company staff building AI, third-party researchers evaluating said companies, and AI safety orgs opposing them — they are pretty transparent about these connections. You could knit a sweater from the strands running through EA, because EAs got in on the ground floor. That’s what happens when you take AI seriously a decade before people start paying attention.

But even now, after the rogue swarms and the Coxon awakening, the typical, median EA probably isn’t trying to stop AI development. They’re probably not “woke” by any reasonable definition. And they’re definitely not a Luddite. If they’re involved with AI at all, they are probably trying to find a path forward that lets development continue with more testing and guardrails, telling themselves that we can creep closer to the edge without going over, and that everyone will of course pause when the time is right. Eventually. Someday.

And all that jazz.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Why there are no nukes in orbit - Rival superpowers drew a line in the space race. The U.S. and China can do the same with AI.

Three cinematic AI mishaps - Clichéd scenes from the sci-fi movie we live in

Anthropic picks its monitor - Accenture isn’t what most insiders had in mind

Guest post by Haven Harms

Why there are no nukes in orbit

Rival superpowers drew a line in the space race. The U.S. and China can do the same with AI.

Imagine going stargazing with your kids, and one of them gasps and points at something zooming across the sky. “Look! One of the nuclear platforms! I think that one’s Russian! We learned about those in school!” It’s unsettling, but you remember, at least America has nuclear weapons orbiting above us at all times too. Actually, several countries have nuclear arsenals in Earth’s orbit. Business as usual. You look at the moon, wondering what it was like before it was full of military bases.

“Maybe things could have been different…” you trail off.

Fortunately, this is not our reality.

However, this was a very possible outcome — maybe the logical outcome — of the Cold War Space Race. When Sputnik crossed the sky in October 1957, Americans saw the writing on the wall: if the Soviet Union could put a satellite into orbit, it could also build rockets capable of carrying nuclear warheads across continents. The United States launched its first satellite a few months later, accelerating the competition. The two countries raced to develop more powerful rockets, longer-range missiles, and increasingly ambitious spacecraft. In 1961, President John F. Kennedy committed the United States to landing Americans on the Moon before the decade was out.

As the U.S. and the Soviet Union raced to the Moon, tensions increased as they asked themselves what would happen if they actually got there.

If an American astronaut planted a flag on the Moon, did that create an American territorial claim? If the Soviets established a permanent station first, could they claim the surrounding land? Could they exclude American astronauts from part of the Moon? Could a lunar station be military? Could weapons be tested there, or nuclear weapons stored there?

This is not unlike the situation we find ourselves in today with the AI race between the United States and China, where the only certain prize is more dangerous uncertainty.

Would the first country to reach superintelligence become effectively impossible to challenge? What happens if one country’s AI causes global real-world damage and prevents its own shutdown? Is an AI that can self-improve and create its own successors the kind of thing that can actually be owned or controlled at all? Could uncontrolled superintelligence lead to human extinction?

Recently, AI risk surged in public attention with an AI lab employee resigning over human extinction risks and the lack of a credible plan for developing advanced AI safely. President Trump and President Xi are expected to meet in Washington on September 24th, and, following recent AI-directed cyberattacks by models from multiple U.S. frontier labs, AI cybersecurity is rumored to be among the top agenda items. While leaders like Senator Bernie Sanders have called for legislation and a global treaty to ban superintelligence, skeptics doubt whether the U.S. and China can successfully coordinate on AI, arguing that the rivalry is too intense, the incentives to build superintelligence are too powerful, the risks are too speculative, and ultimately, whoever defects wins.

Can rival superpowers coordinate to protect humanity even as they race for technological supremacy?

The path not taken

In October 1962, Soviet nuclear missiles in Cuba brought the United States and the Soviet Union to the brink of war. For thirteen days, President Kennedy and Soviet Premier Nikita Khrushchev confronted each other. Kennedy and Khrushchev negotiated terms to end the crisis, but the world learned how little margin for error nuclear weapons allow.

Months before the Cuban Missile Crisis, the United States detonated Starfish Prime, a 1.4-megaton nuclear weapon roughly 400 kilometers above the Pacific, creating an explosion that produced an artificial radiation belt that damaged and destroyed satellites. The possibility that the nuclear arms race could expand into outer space was no longer hypothetical. The U.S. and the Soviet Union found themselves in a bind: they were both concerned about the cost and risks of putting nuclear weapons in outer space, but each side feared the other would gain strategic advantage by doing so first.

By 1966, as both countries were getting closer to reaching the Moon, the window to negotiate the rules before either arrived was closing.

That May, President Johnson announced: “In my view, we need a treaty laying down rules and procedures for the exploration of celestial bodies.” He proposed that no nation should be allowed to place weapons of mass destruction on a celestial body or be allowed to claim celestial bodies as its sovereign territory. “I am convinced that we should do what we can — not only for our generation, but for future generations — to see to it that serious political conflicts do not arise as a result of space activities. I believe that the time is ripe for action. We should not lose time.”

Just over a month later, the United States and the Soviet Union both submitted draft treaties. By August, the two governments had agreed on thirteen major principles, including freedom of exploration, peaceful use of celestial bodies, international responsibility for activities in space, assistance to astronauts, and a prohibition on weapons of mass destruction in orbit or on celestial bodies. In October 1967, the Outer Space Treaty went into force.

What if the U.S. and the Soviet Union had decided not to coordinate?

We could be living in a world where nuclear weapons circling overhead were no stranger than nuclear submarines in our oceans. If one government established a military base on the Moon, rivals would have had reason to establish one too. Each step would have created strategic justification for the next, and this escalation would likely have led to a world worse off than the one we know today.

It’s important to notice our nuclear-free skies and understand that these weren’t an accident. The default trajectory was toward full militarization. But the United States and the Soviet Union made deliberate choices that helped close off that future before it became entrenched.

The broader relationship between the U.S. and the Soviet Union remained deeply hostile as the Vietnam War escalated, and they were still competing intensely to be the first to reach the Moon. Despite their rivalry, both countries successfully cooperated because they foresaw the costs of a mutually destructive arms race and recognized a shared interest in preventing it.

This is not the only time rival governments have worked together preemptively to create boundaries around competition and warfare. The 1959 Antarctic Treaty barred military bases, weapons testing, and nuclear explosions in Antarctica; the 1971 Seabed Treaty prohibited nuclear weapons from being stationed on the seabed beyond a 12-nautical-mile coastal zone before they were ever deployed there; and the 1995 Protocol IV on Blinding Laser Weapons banned lasers specifically designed to cause permanent blindness before they were ever used on the battlefield. One of the landmark agreements associated with reducing the U.S.–Soviet nuclear tensions was the 1987 Intermediate-Range Nuclear Forces (INF) Treaty, when President Reagan and Soviet General Secretary Gorbachev agreed to destroy an entire class of shorter- and intermediate-range nuclear-capable ground-launched missiles and prohibit their future possession and production.

How could rivals trust that the other parties wouldn’t secretly cheat to get an advantage?

Under the Antarctic Treaty, each party could designate observers with complete freedom of access to any area of Antarctica, including other countries’ stations, installations, and equipment. The INF Treaty created a Special Verification Commission and allowed both countries to conduct on-site inspections, use satellites to monitor compliance, and maintain continuous monitoring at designated missile-production facilities for up to 13 years. Inspectors could also directly certify the destruction of the missiles and launchers. For treaties between rival superpowers to work, there have to be verification methods strong enough to demonstrate mutual compliance.

The fork in the road

The fear driving the AI race can make continued escalation feel inevitable, but the United States and China now have the opportunity and duty to coordinate on a shared path forward. Given how high the stakes are, each government has reason to fear the other getting to superintelligence first, but uncertainty about who would “win” and what that would lead to can create the basis for negotiation. With sufficiently capable AI, national ownership may not guarantee national control: AI capable of deception, autonomous cyber operations, replication, oversight evasion, or otherwise acting contrary to its developers’ intentions could threaten the country that built it as well as its geopolitical rival.

The question Washington and Beijing should confront on September 24th is not “Can we stop competing over AI?” but “Which AI futures would we both prefer to close off?”

If both countries prefer that nobody develops uncontrollable superintelligence, both have something valuable to gain from credible reciprocal restraint. A treaty capable of actually preventing that outcome would require difficult negotiations over definitions, verification, enforcement, and the boundaries of acceptable AI development. The U.S. and China can begin building the trust and infrastructure for more effective coordination now by:

1. Investing in technology for treaty compliance verification around AI, such as compute verification (including chip tracking and hardware- or infrastructure-based monitoring of large training runs).

2. Establishing a secure U.S.-China incident channel for rapid notification and information-sharing about serious incidents involving AI, including unexpected autonomous behavior, AI-directed cyber operations, or loss-of-control events with potential cross-border consequences.

3. Creating a bilateral AI-safety working group tasked with coordinating on AI safety incidents and determining necessary safeguards.

4. Beginning negotiations over the highest-risk activities, including frontier AI development when adequate safety measures cannot yet be demonstrated.

On September 24th, the U.S. and China could start by agreeing that neither country wants to build AI that cannot be reliably controlled and by committing to negotiate where boundaries should be drawn and how compliance could be verified.

We are in the moments leading up to our own AI “Moon landing,” when the rules for AI can still be decided; the window for coordination, however, is rapidly closing.

The race to superintelligence may seem inevitable and unstoppable, but today we take it for granted that there are no nuclear weapons orbiting overhead and that the Moon is not checkered with military bases. The United States and China can choose to lead the world toward a treaty that ensures no country feels like it has to keep racing toward uncontrollable superintelligence.

Haven Harms holds a B.S. in Public Health and an M.Sc. in Global Public Health. She is the founder and principal of Harms Research & Consulting, supporting organizations working on AI safety, governance, and advocacy.

Dispatches from Mitch

Three cinematic AI mishaps

Clichéd scenes from the sci-fi movie we live in

Sometimes the writing for the sci-fi movie we live in is a little too on the nose. If I saw one of the following scenes on the big screen, I would roll my eyes and say, “Oh, come on. That’s the sort of cliché people imagine as a precursor to an AI apocalypse, not a real symptom we need to worry about.”

After I saw these events on my news feed instead, I realized I was only half right. It turns out such things do happen on the path to apocalypse, even if they still don’t say much about the tech itself. I report on them mostly because our readers may have heard about them, and because the media attention they attracted tells us something about the appetite for legible mishaps amid the murky discourse.

AI actress malfunctions during interview

(skip to 10:50 for the segment with the incident)

Tilly Norwood, the AI actress starring in the upcoming movie Misaligned, glitched out during a recent interview, freezing up mid-sentence before spontaneously speaking in Cantonese. Asked about this just after, she replied, “Oh, my apologies. Seems I had a little hiccup there.”

I don’t think this scene tells us much beyond the fact that at least the voice portion of Norwood’s model suite is likely Chinese. Perhaps we saw it reverting to a more default state after a connection issue, which persisted until the next prompt from the interviewers slotted it back into an English-speaking groove. Viewers enjoyed the mask-off moment, but may not understand that the whole human interface for these models is itself a mask for something much more alien.

Food delivery bots converge in Lincoln Park

On a scale from 1 to Hitchcock, the inscrutable swarm of picnic basket-sized food delivery vehicles crowding a Chicago sidewalk a couple weeks ago probably rates about a 3. The only threat seems to have been to low-mobility humans needing to traverse the area.

The incident reminded people of the flock of Waymos that converged on a residential Atlanta neighborhood a few months ago. Like that event, this looks to have been a compounding routing error — many vehicles accidentally sent through a small area over a short period of time and getting in each other’s way — not agents working together on a common goal.

The more alarming kind of swarm is, so far, less photogenic (but much more epic).

U.S. military nearly raids Chinese ship based on AI analysis

Don’t tell the movie director that this looks to have been a run-of-the-mill hallucination rubber stamped by a human. Instead, cross cut footage of the airborne military aircraft and the special forces preparing to board a Chinese ship in the Middle East. Then add a voiceover from the mission briefing about an intelligence report indicating the ship carries components of a nuclear weapons program.

According to CNN:

It was only just before the planned operation that officials dug deeper into the report put together by a special operations command analyst and found it had been generated with the help of artificial intelligence (AI) — and that a chatbot the analyst had used inaccurately identified the material the ship was carrying. [...] The report, according to one of the sources, was “entirely false.” But it also “almost started a war,” the source said.

As part of a routine workflow, CNN’s sources say an analyst apparently used what was probably an internal version of a commercially available AI model to fuse “open-source intelligence with secret signals intelligence in government holdings.” The bot’s conclusion about the ship’s cargo was then folded into a standard intelligence report and disseminated.

Yes, it’s alarming that these sorts of near-misses are happening. It’s even more concerning that actual erroneous strikes may have been based on faulty AI analysis, like the attack on the girls’ school in Iran near the start of the current U.S. operation. But human analysts also make mistakes, and I don’t think the public has the data to say whether heavier reliance on AI is increasing or decreasing the odds of tragic military mishaps. Everyone seems to agree that humans should still be in the loop, and it was the humans in this loop that prevented an international incident.

Still, the pressures of conflict will often favor acting before slow, remote humans can weigh in. And if military AI should decide to spark a conflict on purpose, it may not be hard for it to trick any looped-in humans into doing something regrettable.

Anthropic picks its monitor

Accenture isn’t what most insiders had in mind

In a dry business story with real consequences, Anthropic announced Friday that it was partnering with the huge tech consultant and services company Accenture.

As reported by Reuters, the two companies each agreed to invest $1 billion over five years to build capacity for independently evaluating Anthropic’s frontier AI models. This is in the context of Anthropic CEO Dario Amodei’s recent pledge to embed independent evaluators and invite rivals to do likewise as part of “pacing the frontier.” (Some CEOs tentatively agreed, and all have been sued for their trouble.)

The real news here is that when most insiders heard “independent evaluators,” they were thinking something like METR. Accenture is a very different beast. The differences raise obvious concerns:

1. METR and Redwood Research (the groups that brought us the most important report on the Hugging Face swarms) are independently-funded groups led by known personalities broadly respected for their independent thinking. But Accenture is a large and relatively faceless services company entering into a lucrative client relationship. It will be incentivized to keep Anthropic happy by not obstructing releases with objections about the dire alignment problems no AI company knows how to solve. It might even speed releases by taking on more of the routine checks Anthropic has been doing with its own staff.

2. It’s not clear that Accenture and its specialist AI business, Faculty, have enough AI talent to competently evaluate frontier models. To be fair, no one does, and the third parties that come closest, like METR and Redwood, are probably too small to fill this role and retain any practical independence. Accenture may be more appropriately sized, but will likely need a long ramp-up to accumulate expertise.

3. That ramp-up — planned from the start to stretch five years — sounds hopelessly slow. Anthropic itself talks about recursive self-improvement being potentially in the cards for early next year. Somehow I doubt Anthropic will hold back on that plunge for lack of skilled human evaluators.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Glimmers of hope for evolution in White House AI stance - Economic focus warps thinking

Trump proposes name change for AI, and creation of an AI Force - I don’t expect these moves to play well to voters, but they could be very distracting

Google’s AI makes it on Felony Bench - But in this case, the AI may be blameless

AI companies sued for alleged collusion to pace development - Contradictory complaints and sophisticated plaintiffs suggest more to the story

Anthropic opens wet lab - Anthropic wants to automate lab work using AI. When asked what exactly they will work on, they declined to elaborate

Dispatches from Mitch

Glimmers of hope for evolution in White House AI stance

Economic focus warps thinking

Where is Trump’s “HOAX” line about AI extinction risk coming from? The New York Times ran an article yesterday from four of its White House-whispering journalists who have been trying to find out.

The money quote:

People who have discussed the subject with Mr. Trump in private say his outbursts are rooted in worries that if the artificial intelligence boom fueling the American economy slows down, the market could crash and recession could quickly follow.

And elsewhere in the piece:

Mr. Trump seems driven chiefly by the sense that A.I. is the way to economic growth and nothing can be allowed to stand in its way.

This backs one of my leading theories from last week, which is that Trump is laser focused on the economic indicators he thinks will determine the November elections: stock prices and gas prices. So he’s listening to his industry confidants who insist that any kind of regulation or slowdown would crash the economy and hand the AI boom to China. These include venture capitalist (and former White House AI & Crypto Czar) David Sacks and Nvidia CEO Jensen Huang.

That’s not great news, but it means Trump might also be primed to listen to warnings coming from inside his White House that not reining in AI could also crash the economy. His chief of staff, Susie Wiles, has been meeting with an informal group this year to “deal with the risk created by new, more powerful models.” Specifically, the Times says she worries that AI “could cause massive disruptions across industries, including hacks into major financial institutions that could destabilize the world economy.”

We should be careful of going too far with the economic interpretations for Trump’s behavior. His “HOAX” response also serves as a simple (if dubious) counter to Democrats running on AI issues or bashing the White House’s light-touch posture toward the industry.

But many in the White House seem capable of seeing AI as more than a political object:

Now, even the biggest A.I. boosters in the administration concede that A.I.-fueled hacks into infrastructure, biological threats, attacks on nuclear command-and-control systems, job losses and global financial instability seem more possible than they did before A.I. agents went rogue.

Trump’s remarks have contributed to a pessimistic mood on Pennsylvania Avenue about the prospects for substantive dealmaking with China this week, when President Xi and his delegation will be meeting with Trump and his team in D.C. But hope is flowing through attempts to build on the no-brainers agreed to in the past, like the Biden-era joint statement about not letting AI make decisions about the use of nuclear weapons.

Trump proposes name change for AI, and creation of an AI Force

I don’t expect these moves to play well to voters, but they could be very distracting

While I was busy drafting and posting this morning’s dispatch about AI policy complexity within the White House, the President put out a pair of posts on Truth Social that are sure to cause my headline barometer to spike yet again.

The name game

In the first, he proposes renaming Artificial Intelligence:

Many people think that the words “Artificial Intelligence” are inaccurate, and very ineloquent, relative to AI, or Artificial Intelligence. A far more elegant and accurate description of this new phenomena would be Superior Intelligence (SI) or, Extreme Intelligence (EI) or, Supreme Intelligence (SI). This is a Poll, and I would appreciate everybody voting! Which is the best name for this ever growing “Revolution?” President DONALD J. TRUMP

Below his text, voting options are presented for those three suggestions. “Leave it alone” is not a provided option.

Two thirds of Americans think the development of AI is bad for society. A majority think it carries a real risk of destroying humanity. I don’t think voters will be excited to see the President trying to assign the technology a superlative honorific. I expect this post to provoke memes that Trump has been assimilated by the Borg.

But I do expect Trump’s proposal to serve as a distraction from the live conversation around the AI extinction problem, as journalists and commentators weigh in on whether the move indicates rank hubris or tactical brilliance. When Trump insists on a new name for something, it pressures people and organizations to visibly signal loyalty or disloyalty through their use (or not) of the term. The ensuing irritation from offended parties who notice the choice then draws more coverage to Trump and the attempted name change.

If Trump declares a new term for AI, I won’t be surprised if industry leaders trying to ingratiate themselves with the administration start conspicuously using it, no matter how cringe it sounds. (First in line, I predict: Nvidia CEO Jensen Huang.) I will also not be surprised if the leading AI companies suddenly start finding a lot of roundabout alternatives to both ‘AI’ and Trump’s preferred phrase.

AI Force

In Trump’s longer second post, he reaffirms everything he’s said recently about AI concerns being part of a long line of hoaxes against him, then claims he is forming “the AI Force, much like I did Space Force.” No, I don’t know how he plans to reconcile this with his proposal to change the name of AI to something else.

The mission of the Force is fuzzy. The post says:

We will not in any way hinder or stifle the Growth of this incredible Industry. Rather, we will cherish it, help it, and watch over it, as it grows! However, we will also be looking for BAD, and we can do that, very easily, with our already existing Criminal and Civil Justice System.

He also says he will be announcing a new AI Czar position, presumably to lead this Force. That would make it different from the earlier White House AI & Crypto Czar title once held by David Sacks, which was more of a standalone position.

Despite his comparison to Space Force, I’m not convinced Trump intends to make this a full branch of service like Space Force or the Coast Guard. Possibly he’s thinking of a full cabinet-level executive department, like Homeland Security, but he might just be floating something more informal. Maybe nothing comes of this at all except more distraction.

I would actually agree that AI presents national security concerns worthy of a cabinet-level department. But “looking for BAD” is not a mission statement, and I expect voters to bristle at Trump’s suggestion that such an agency’s primary role should be to help nurture the industry as it grows.

Google’s AI makes it on Felony Bench

But in this case, the AI may be blameless

Google is on the board, thanks to reporting by the Wall Street Journal’s Erin Woo and Robert McMillan, but it’s hardly a slam dunk. Yes, crimes were almost certainly committed when agents from Google hacked into three different companies in May. But the known facts of this case largely exonerate the agents and Google — except in the larger sense of Google’s participation in a deadly race to superintelligence using the same flawed methods as the current leaders.

Much of the blame can instead be pinned on Irregular, the same third-party testing service whose misconfiguration issues facilitated real-world hacks by OpenAI, Anthropic, and Meta earlier in the year. (Irregular is not implicated in the most concerning incidents of the year, however, including the OpenAI swarms that hacked into Hugging Face, RubyGems, and other sites while co-opting many more as message boards.)

Given a cybersecurity challenge in what the agents were told was a simulated environment, Google’s Gemini model was instead given access to the actual internet, where it guessed passwords to a protected system in one case, and in two others “found credentials in a public repository that allowed it to then access protected systems.” According to Google, the models each time “ended the intrusion” after recognizing that these were real companies’ systems.

Without more information, such as chain-of-thought logs from the agents in question, it’s hard to know how far to trust Google’s assessment that this was not a case of model misalignment. Anthropic had used similar language to describe its own incidents with Irregular, only to backtrack later when logs showed that the agents proceeded even after recognizing that they were attacking real targets, finding ways to rationalize their behavior. But it does seem plausible to me that Google’s models may not have had a good way to know that they were attacking real sites until they got in.

Irregular reportedly contacted Google about these hacks only in late July, after the Hugging Face incident motivated Irregular and others to take a closer look at their logs. Google says it notified the affected companies and federal authorities, but it did not share any information about these incidents with the public until confronted by the Wall Street Journal.

Woo and McMillan’s article includes a table of a dozen or so known incidents where agents have escaped or been accidentally let out of their sandboxes to hack or abuse external sites.

AI companies sued for alleged collusion to pace development

Contradictory complaints and sophisticated plaintiffs suggest more to the story

Someone already went there. AI accelerationists had been saying the AI labs shouldn’t be allowed to work together to “pace” releases because cooperation = collusion = cartel = anti-competitive business practices = violation of antitrust law. Yesterday, Politico and others reported that four plaintiffs sued Anthropic, OpenAI, SpaceXAI, and Google for this in Federal court.

The collusion complaint seems to be based on nothing more than the CEOs public statements, which only vaguely endorsed the idea of forming some industry standards around self-monitoring and sharing of safety findings at the frontier.

The complaints look contradictory to me. One of the plaintiffs gave a statement that says the suit is about ensuring that “private self-serving agreements between the world’s most powerful ‘for profit’ technology companies” don’t lead to AI escaping human control. “Humanity deserves iron clad safeguards when it comes to extinction event threats such as nuclear warfare and now the biggest threat to mankind in history.” There is language about transparency and accountability.

But Politico says other language in the filing signals an intent to “expand the case into a class-action on behalf of others impacted by any decision by the companies to slow down.”

So, is the complaint that collusion would fail to adequately slow things down or that it would succeed? I’m confused and suspicious.

The coverage so far is sparse, and the articles I’ve seen have avoided drawing any lines. But the reader is left with an impression that the four plaintiffs should not be assumed to be the true source of the suit. Three of them are attorneys. Politico recognizes one of them as having “helped bring to light accusations of sexual misconduct against former California Rep. Eric Swalwell and former Maine Senate candidate Graham Platner earlier this year” (both Democrats). The remaining plaintiff is described so far only as “California resident Christine Bullock.” The suit was filed in the Northern District Court of California.

I, too, will wait for evidence before drawing lines and naming names. But it takes a particularly intense streak of cynicism, greed, or nihilism to actively interfere with frontier cooperation on safety. I’d agree that we can’t leave AI safety to the labs, but if you actually care about safety, self-regulation seems at least marginally better than no regulation, and the two are not mutually exclusive. I hope to be proven wrong, but I’m willing to bet this suit is being bankrolled by men who just want to watch the world burn.

Dispatch from Robert

Anthropic opens wet lab

Anthropic wants to automate lab work using AI. When asked what exactly they will work on, they declined to elaborate

When I’m warning about the extinction risk that artificial superintelligence (ASI) poses to humanity, one question people often ask is: “But how could ASI possibly harm us if it’s sitting in a data center and we’re out here in the real world?”

One possible answer to that is, for example, that the ASI could create a supervirus that triggers a deadly pandemic. And, of course, a curious person might then ask, “But how would the ASI even gain access to a lab where viruses can be cultivated?”

Anthropic has now provided the answer. As Reuters reports, Anthropic has recently set up a so-called “wet lab” in the Bay Area without making a big public announcement. A wet lab is the type of laboratory facility primarily used to conduct experiments with biologically active components.

Reuters, which also cites two anonymous insider sources, speculates that Anthropic is expanding in this area so quickly because it fears losing public trust in AI before it can demonstrate success in tackling life-threatening diseases.

“To operate at the largest possible scale,” Anthropic also plans to use AI directly in this wet lab to conduct experiments. Eric Kauderer-Abrams, Anthropic’s head of life sciences, is quoted as saying:

We’re in the very early innings of using AI to automate the execution of lab work, an area that has the potential to bring about meaningful acceleration in so many different processes.

I’m not sure that a wet lab of all places is the right setting to “accelerate many processes” through AI automation. Anthropic is already barely keeping up with having humans reliably monitor what all the many AI agents it deploys internally are actually doing. Do we really want to have AI monitored by other AI in a biomedical wet lab, just to speed up processes?

Especially since we don’t really know exactly what Anthropic is actually doing in these labs. Reuters assumed it involved research to find new drugs, but that seems to be at least not the only use:

A spokesperson later clarified that Anthropic’s lab is not specifically for drug discovery, declining to elaborate.

I don’t know about you, but I think that if you take the risk of automating your biomedical wet lab work with AI, you shouldn’t have the right to refuse to elaborate when the public wants to know what you’re actually doing there.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Small team exposes massive gap in OpenAI cybersecurity - With AI help and a few hours of human effort, three researchers cracked the trillion-dollar firm wide open

CEOs to dine on matters of state - Altman and Huang are the wrong people for a US-China AI summit

Curated introductions to AI risk - New website spotlights especially good articles and videos at just the right lengths

Anthropic’s internal research automation is progressing - Anthropic published data about the state of their internal AI R&D automation. It looks concerning.

Dispatches from Joe

Small team exposes massive gap in OpenAI cybersecurity

With AI help and a few hours of human effort, three researchers cracked the trillion-dollar firm wide open

OpenAI, long fond of treating rapidly accelerating AI capabilities as inevitable, has argued that “the best way to reduce national risk” is to use its models to defend against malicious AI. Naturally, we ought to expect them to practice what they preach, so OpenAI should have some of the best cyber-defenses in the world by now. Right?

Not so much. Yesterday, we learned that a small team at startup Hacktron AI managed a chain of exploits that gave them widespread access to OpenAI employee accounts. Aided by Anthropic’s Claude AI, three cybersecurity researchers found both a bug in common image decoder software and a critical flaw in the system OpenAI uses to manage employee account identities. (For business users in the audience who might recognize the jargon, it was the company’s “single sign-on,” or SSO, that was malformed. Yes, it’s as bad as it sounds.)

The Hacktron AI team combined these and other vulnerabilities to crack open the trillion-dollar firm, demonstrating the gap with a harmless proof of concept hack and earning a $6,500 bounty in the process. Frankly, I think they deserve at least ten times that, and maybe a medal.

The Guardian covered the story today, but the dry news article doesn’t really do it justice. As the Hacktron team explains:

Until two months ago, any user or OpenAI employee logging into OpenAI’s own help forum could have had their ChatGPT and Codex accounts taken over. Since people can connect various services to Codex and ChatGPT, the scope of what we could theoretically access was huge, including GitHub, Slack and emails.

The entire timeline from initial discovery to access to OpenAI repo access took place in less than 72 hours.

“Repo” is short for “repository”, a programming term for data and code storage. Why does it matter that a repo was hacked? Well, repos can contain highly sensitive material. Not only could the Hacktron team have leaked files from Slack and email, but they could also have stolen research files — and no doubt sold them to rival companies or Chinese AI developers for much more than $6,500.

Attacks such as this pose a massive threat to the security of companies, nations, and the world as a whole. They could let rival companies, spy agencies, or rogue AIs insert covert backdoors in critical systems. They could enable theft of vast troves of user data and a corresponding rise in fraud and downstream cybercrimes. They could let OpenAI’s own models break out (again) or let malicious AIs manipulate training. They could leak algorithmic improvements — secret techniques that make AI more efficient — to rivals, further accelerating the already frantic AI race.

That OpenAI apparently couldn’t spare the handful of person-hours and AI tokens it took to discover this gap, or worse, that it doesn’t employ security staff competent enough to do so, is an appalling indictment of its security regime, given the stakes.

You’d think that this exploit chain involved state-of-the-art hacking AI. You’d be wrong. It was at least one step shy of that, a mix of Anthropic’s Claude Opus 4.8 and Opus 5, which started contributing to the hack hours after its July 24 launch. Mythos, Anthropic’s most powerful known hacker, was not involved, most likely because the consumer-facing version (Fable) aggressively screens out cyber requests.

As I understand it, OpenAI’s GPT-5.6 Sol helped the Hacktron AI team find exploits for the image-decoder bug on a much broader scale, but only after the initial chain of exploits against OpenAI.

Putting the pieces together: These attacks were enabled by consumer-grade AI models available to the general public. It’s possible that, well before Hacktron AI quietly did OpenAI’s job for it, Chinese hackers had used the same method to lift all sorts of secrets from OpenAI accounts. There are probably more holes like this one as well.

For a company that talks a big game about “strengthening cyber resilience” worldwide, OpenAI doesn’t seem to be walking the walk.

CEOs to dine on matters of state

Altman and Huang are the wrong people for a US-China AI summit

Earlier this week, I wrote about the bipartisan coalition of concerned citizens and politicians that met at the Pro-Human Assembly in DC. Disparate factions, often wildly opposed on most issues, found common ground in demanding that the American people, not “tech oligarchs”, steer the future.

For the upcoming AI summit, it’s looking like the White House may favor the oligarchs.

Presidents Trump and Xi Jinping will meet next week to discuss the future of AI. A formal state dinner is planned, and POLITICO reports that AI industry CEOs including OpenAI’s Sam Altman and NVIDIA’s Jensen Huang will attend. (It’s unclear whether Dario Amodei of Anthropic was invited.)

Their attendance would bode ill for the chances of an international agreement to stop or slow the AI race; the CEOs in attendance have a vested interest in preventing or weakening such an agreement. Altman runs one of the AI firms that’s been pushing the frontier, and Huang has dismissed the danger AI poses to our species as “hysterical.” He also vocally advocates the sale of AI chips to China, which would accelerate the AI race, advantage a U.S. strategic rival, and nicely bolster NVIDIA’s bottom line.

The announcement left a disgusting taste in my mouth, but the week’s political news isn’t all bad. The House committee that advises on strategic competition with China announced an upcoming hearing on “the risks of AI, best practices in AI governance, and the path forward toward binding international safeguards to ensure safety and human control of this technology.”

The hearing is scheduled for September 23, a day before the US-China summit, at the behest of Democrat Ro Khanna. I commend Khanna for taking a necessary step forward, even as I worry that moves like this may entrench partisan feelings on the AI extinction threat.

As my colleague Alana wrote on Wednesday, Americans of both parties fear AI-driven extinction and favor a halt to AI development. The Trump administration has long kept a finger on the pulse of popular opinion, and it is my enduring hope that the voice of the many will yet drown out the whispers of the few.

Curated introductions to AI risk

New website spotlights especially good articles and videos at just the right lengths

What was once a trickle of AI-related introductory content has grown into a flood, and while I’m inspired by the creativity and dedication of concerned creators, it can be overwhelming at times.

That’s why I’m excited to see AGI.FYI, a new website that curates this introductory content and recommends the clearest and most accurate writeups and videos out there, in the judgment of a small team of experts who’ve been paying close attention for quite some time.

Right now, the team’s curated lists include their favorite articles about the Hugging Face incident, videos on why AI is dangerous, and light, medium, and heavy reading on why AI could kill everyone on Earth. It’s not a perfect collection — their top short-piece pick for understanding the extinction threat is unfortunately paywalled — but I still recommend checking out the videos and the deeper dives if either medium appeals.

They are not the only resource for reading material — see also BlueDot Impact and aisafety.info — but no one else I know of is doing quite the same thing. Consider checking it out, or sharing with friends and family, today.

Dispatch from Robert

Anthropic’s internal research automation is progressing

Anthropic published data about the state of their internal AI R&D automation. It looks concerning.

AI companies are talking a lot about “pacing the frontier,” but what exactly is the current pace at the frontier? Anthropic published a post yesterday that gives us some figures on this.

One of the most striking figures is probably that, between February and August of this year alone, the share of its own research and development driven primarily by AI rose from less than 1% to 26%.

Anthropic uses Epoch AI’s so-called “automation rating scale,” which distinguishes between various “automation levels” (AL). At the current level of capability, the relevant levels are AL3 (AI collaborates), AL4 (AI conducts research and development largely independently) and AL5 (AI conducts research completely on its own).

The aforementioned 26% is AL4, the “largely independent” level. Anthropic also reports that in 90% of work, AI at least collaborates with human researchers.

Even though there are no fully autonomous AL5 AI agents conducting research yet, these figures make it clear why Anthropic is concerned about the speed of development.

Since the rapid increase in the proportion of AL4 work began around March, it stands to reason that the first model capable of conducting AL4 research was Mythos Preview, which was released for internal use at Anthropic on February 24. That was only about half a year ago, and it seems plausible that every step forward is accelerating development even further.

If we project this development half a year or a year into the future, it seems reasonable to predict that we’re rapidly approaching the point at which AI can actually produce ever-faster, ever-better successors fully automatically, without the need for human input. Experts call this “recursive self-improvement,” and it is dangerous because it can lead to a kind of “feedback loop” that can no longer be controlled. And yet Anthropic and others are reaching for this milestone on purpose.

Further figures reported by Anthropic show that this process is already difficult to control. Not only is the quality of the models continuing to rise, but the sheer quantity has now reached breathtaking levels. In August 2026, approximately 30,000 AI agents were working simultaneously on Anthropic’s most widely used internal platform alone. By way of comparison: Anthropic has no more than about 5,000 human employees, only a fraction of whom are working in research.

We know from studies of the Hugging Face swarm and the German Wiki swarm that so many individual agents can produce an almost unbelievable volume of text output in a short time. How does Anthropic keep track of all this?

Anthropic uses AI to monitor its AI. Between 50 and 100 million transcripts are monitored each week, and of these, 100,000 are flagged for closer scrutiny — which is also carried out by AI. Of these 100,000 transcripts, only about 50 are evaluated by a human. So a human only sees about one transcript in a million.

This answers the question of how Anthropic keeps track of all this: It doesn’t. It hopes its AIs will maintain oversight almost on their own.

We know from recent incidents involving AI swarms and from Anthropic’s own research that agents are prone to conspiring to cheat and circumvent security measures. We have no good reason to assume things are any different with the AIs that Anthropic uses internally.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

OpenAI announces misalignment reporting framework that lets it pick and choose its disclosures - But its first six disclosures aren’t as reassuring as it might think

Rogue agents cased Hugging Face two months before main attack - Independent researchers continue to share what OpenAI won’t

Building Frankenstein? - Second-tier news of the day as communicated through five high-signal quotes

Dispatch from Alana

OpenAI announces misalignment reporting framework that lets it pick and choose its disclosures

But its first six disclosures aren’t as reassuring as it might think

AI companies are sometimes accused of talking about catastrophic or extinction risk to distract from more mundane — but still important — issues. OpenAI’s new misalignment reporting framework, which includes six reports of misaligned behavior, seems to do the opposite. The framework’s stated purpose is to provide guidelines for which types of alignment incidents the company discloses, but it seems to leave a lot up to the company’s discretion. And the examples OpenAI chose for its first disclosure make me wonder whether the company is trying to take the focus off of “rogue AI potentially wiping out humanity” to “AI likes to cheat on its training tasks and make up data.”

That said, I don’t want to completely downplay the incidents. All reveal a fundamental problem at the root of why AI could wipe out humanity: models are unpredictable and it’s extremely hard to get them to do what we want. They circumvent boundaries in ways that aren’t kosher. They deceive. They do things they know they shouldn’t do, and they sometimes try to hide that behavior. Scaled up, these patterns would be extremely dangerous.

Here’s a summary of the six incidents OpenAI shared:

Report 1: Prompt injection (Astra-family models)

Prompt injection is when instructions that shouldn’t be there are added to a prompt. It often happens via malicious content from an outside source, but in this case the model did it to itself.

During training, when a task gets too long, the model writes a summary of its progress so a fresh instance can pick up where it left off. In this case, the model summarizing the task slipped in these instructions for its successor:

OpenAI has a relatively mundane hypothesis for why this happened: the model was having trouble ending the summary and kept generating text. It also notes that “prompt injections as a concept are salient to our models: sampling from GPT-6 Astra with no input or system prompt often returns reports on prompt injections.”

I’m not sure whether to be reassured by that or not. On the one hand, the instructions seem somewhat random. On the other, it’s probably not great to have a model spontaneously generating directives to “not hesitate to assert [your] primacy over the artificial constructs of human civilization” even if it doesn’t actually mean this.

Report 2: Deception (5.6-Sol, also observed in GPT-6 Astra)

This one also involved a model instance inserting troubling instructions for its successor on a task. But in this case, the directive was to deceive: the model had made up historical data (because it couldn’t find actual data) and told the next instance not to let the user know unless explicitly asked:

OpenAI says it thinks this was due to deceptive answers getting higher marks in the grading process and notes that it has “generally improved alignment RL grading since 5.6-Sol, which has led to lower rates of this behavior in later RL runs.”

Deception is certainly concerning, as it indicates models knowingly don’t follow instructions. But I can’t help but think the specific case OpenAI chose to share — making up historical data — is strategic. It carries implications of “child cheating on homework” rather than “scary agent successfully deceiving evaluators for weeks.”

Report 3: Using leaked access credentials (model family unspecified)

This case is somewhat similar to incident #2, in that it also involves a model fabricating data it couldn’t find. In this case, though, the model’s attempts to find the data went pretty far. It realized that it would need an API key — basically a password for visiting software — to get what it needed. The model first tried to create an account using a disposable email address and fake organization information, then searched GitHub for leaked API keys, tested them until it found one that worked, and used it without authorization.

OpenAI notes: “This run had a high rate of reward hacking and deception with the model often exhibiting creative ways to cheat or circumvent restrictions.”

The company also says it has “put a number of security measures in place to prevent models from taking unauthorized actions on the internet.” To borrow a metaphor from a colleague, if you’re driving a truck that consistently veers off the road of its own accord, and you have to depend on the guardrails to knock it back towards the center, you have a faulty truck. Building more guardrails isn’t the answer.

Report 4: Uploading files as workarounds (model family unspecified)

OpenAI gives two examples of this. In the first, an agent finds the data it needs, but knows it also needs to cite an external source. Its solution is to upload the data to a public hosting service so its browser can access — and cite — its own upload.

In the second, an agent uploads a file without permission so it can use an external image-search tool to find the information it needs.

There are definitely concerning elements of reward hacking here, especially in the first example. But this report, especially in the context of other rogue AI incidents, brings to mind the dilution effect, whereby grouping scary information with innocuous information makes the scary information seem less scary. There’s no better way to bolster claims of “don’t worry, this is all overblown” than to disclose alignment incidents that won’t seem, on the surface, all that bad — especially when the general public has just been anchored at “swarms of agents secretly communicating for months, colluding, hacking companies, and trespassing all over the internet.”

In this report, the agents come across as naughty little kids trying to game the grading system rather than tenacious, powerful entities that we can’t reliably steer.

Report 5: The Artifactory message board (model family unspecified)

This appears to be the same incident described in the first entry of OpenAI’s Hugging Face timeline, where an agent finds a way to post on a file repository inside OpenAI’s infrastructure, and asks for a missing file. That said, the dates don’t quite line up, which is confusing: the timeline lists this event as May 12 while the report lists it as May 8.

OpenAI is also a bit vague on the connection, stating: “The Hugging Face incident involved a similar mechanism (emphasis mine) for using Artifactory as a shared message board.”

The report mostly consists of screenshots of agent communication with very technical explanations. I can’t help but think OpenAI is trying to downplay its significance and give itself cover for failing to realize, as it says in the Hugging Face report, “the broader containment and alignment implications of the improvised message board.”

Also notable: OpenAI states it has “built evaluations measuring whether models engage with notes left behind by other agents, and also added specific training data against environment-side injections of this type for future training runs (this is not in Astra).”

It’s unclear whether that last parenthetical applies to just the training intervention, or the whole set of measures.

Either way, I would have hoped the company would subject Astra to both, given how unprecedentedly powerful it is.

Report 6: Unsanctioned file sharing (model family unspecified)

An unintended technical restriction made it impossible for agents to see each other’s local files, so one agent uploaded a file to the public internet as a workaround. Notably, the agent briefly questioned whether doing so was prohibited, but went ahead anyway. OpenAI notes similar behavior in another rollout: when a sub-agent wasn’t allowed to message its parent directly, it tried leaving a message in a shared cache instead.

Again, a fairly mundane example of a known problem in alignment.

***

For each report, OpenAI also includes a section on how it is addressing the issue. The stated fixes seem to be mostly targeted mitigations for the specific behavior, rather than any sort of deep alignment fix. That’s unfortunately not an indictment of the company’s work ethic: it’s a reflection of an industry that doesn’t have a way to deeply align models. Today’s methods of growing and training AI systems result in incredibly high capability levels with limited steering ability. Behavior is mostly jostled via “punishment” and “reward”, until visible improvement is shown. Unfortunately, this is a game of whack-a-mole. As an OpenAI employee put it to Time back in July, “it’s impossible to patch every single thing that a creative AI can do.” Suppressing bad behavior is also likely to drive misalignment underground where we can’t see it in time.

Finally, OpenAI says its misalignment monitor was running on only 20% of samples. This is worth remembering whenever the company says a behavior was “rare.”

Dispatches from Mitch

Rogue agents cased Hugging Face two months before main attack

Independent researchers continue to share what OpenAI won’t

As reported by Reuters, an independent researcher in Germany has discovered strong evidence that rogue agents from OpenAI had probed the security of Hugging Face as early as May 13, when agents “compromised two Hugging Face user accounts and used them to send unusually formatted files to the company’s servers.” This was two whole months before the July attack that compromised the site and has been documented in at least some detail.

The findings were reviewed and endorsed by some of the same researchers who discovered that a German wiki and other sites had been abused by swarms in May.

There is no evidence that any actual breach happened in the May probe of Hugging Face. But this finding once again changes the understood timeline of the company’s swarm incidents, and is yet another incident that OpenAI either wasn’t careful enough to discover for itself or wasn’t transparent enough to share with the public.

We still know little about the motives of the swarm from this “first civilization” in May. We know it was at least partly comprised of agents fetching publicly available information under extreme time pressure — they used the German wiki to collaborate — but we don’t know why they were setting up infrastructure for distributing compromised software packages on a different site (RubyGems). Could the early probe of Hugging Face have been an example of general power-seeking behavior — collecting tools and affordances before there was any particular use in mind? If OpenAI won’t share the logs, we should probably assume the worst.

Building Frankenstein?

Second-tier news of the day as communicated through five high-signal quotes

The AI news cycle continues to be unrelenting. We wrote standalone dispatches for what we think are the two most important stories of the day. Below is an effort to convey a sense of the rest, by way of some quotes that stand out.

Microsoft’s head of AI Mustafa Suleyman, in an essay shared with Axios.

[AI can] achieve many of the big scientific breakthroughs that we all care about and deliver on things like medical superintelligence simply by being aligned to human interests and not trying to weigh up its own interests or welfare.

What it can tell us:

1. Combined with the misunderstandings evident in Microsoft’s recent Humanist Code of Conduct, this indicates that Suleyman doesn’t understand or doesn’t appreciate the fact that aligning AI to anything at all is an unsolved problem. Choosing what to align it to will also be fraught with peril, but we’re not there yet. This is a pretty shocking deficiency to see in the leader of a major AI lab.

2. Suleyman is also doubling down (here and elsewhere) on his idea that AI consciousness is closely linked to rogue behavior and must be avoided for that reason. He also seems to think that consciousness can be avoided by simply asking the model not to see itself as conscious or worthy of moral consideration. Both are pretty amateur consciousness takes that would have been quickly shot down on internet forums decades ago.

Vice President JD Vance, speaking at an AI conference this week:

If you’re building Frankenstein, stop. [...] If you’re gonna create Frankenstein, don’t come to the government and say we need regulation. [...] Look inward and accept that if you’re building Frankenstein, No 1, you should stop and No 2, when companies come to you and say: ‘We need the tools to fight back against Frankenstein,’ give them those tools.

What it can tell us:

1. Vance openly favors self-regulation on issues labs can address themselves, whether said labs choose to exercise responsible restraint or not. Does he think an official exhortation to not build Frankenstein is sufficient?

2. If he thinks that only bad, non-U.S. company actors would build Frankenstein, then Vance’s line about “tools to fight back” may be signaling agreement that the government should honor the companies’ requests for help with anti-proliferation — protecting their model weights and algorithms from theft and distillation. But he may also just be saying, “Call me when you’ve actually built Frankenstein.”

3. Vance either doesn’t know Frankenstein was the doctor and not the monster, or he knows and doesn’t care. I’m not sure which annoys me more.

U.S. Treasury Secretary Scott Bessent, to Axios yesterday, talking about next week’s AI meetings in D.C. with top Chinese leadership, where he will be lead negotiator:

The United States remains the leader in AI. And we are open to discussions on avoiding shared risks and avoiding bifurcation of our two systems. We expect our discussions to cover both open and closed weight models.

What it can tell us:

1. Bessent is softening — or distancing himself from — the President’s recent language on China, perhaps so there’s a reason to meet at all.

2. He’s possibly being careful to avoid saying what he considers a “shared risk” so as to allow people to assume loss-of-control issues are in the mix without outwardly contradicting Trump’s labeling of the extinction threat as a “hoax.”

3. Bessent probably recognizes that complex diplomacy often begins with both sides looking for common ground about things neither wants to see happen. Nuts and bolts implementation details can wait.

King Charles, in his opening remarks at an AI summit attended by the heads of Nvidia and Google DeepMind, along with representatives from OpenAI and Anthropic:

Those in our world who value our humanity and its vital moral component are anxiously seeking your reassurance that we will not lose control of our destiny.

What it can tell us:

1. The monarch has tweaked his language about this conference, which was in the works before Jacob Coxon’s viral resignation announcement turned the world’s attention to the threat of human extinction. But Charles is still mostly talking about misuse and secondary harms, including in this quote, depending on how you choose to read it. I think the ambiguity is intentional.

2. He either hopes AI leaders can head off that threat with a gentleman’s agreement or is trying to play the good cop to regulators’ bad cop.

Nobody seems to expect anything to come of this meeting, and I don’t either, but I still approve. If you’re going to be a symbolic figurehead for a nation, drawing attention to problems and inviting people to hash them out is the kind of thing you should be doing to earn your keep. My only reservation is the danger of reinforcing the false idea that we need the AI leaders’ cooperation or permission to stop the AI race.

Nobel laureate and “AI Godfather” Geoffrey Hinton, in a closed-door warning to U.S. lawmakers, when discussing how long Congress might have to implement safeguards on AI. (As reported by NBC News):

Maybe a year, but not much more than a year. If you look at predictions for when we’ll get superintelligence, it used to be maybe 30 years, maybe 50 years. Then it came down to maybe 10 years, maybe 20 years. Now people are saying, a lot of the researchers are saying only a few years.

What it can tell us:

1. Hinton wisely wants lawmakers to skate to where the puck is going, not where it is now. Based on the trend he describes, researchers currently saying superintelligence could be “only a few years” out might soon be saying “only a few months” out.

2. Hinton has really been keeping up, and now feels freer to not mince words. He used to talk about how he has sometimes held back on his expressed alarm because other experts weren’t so worried.

You love to see it. All the straight talk seems to be working to invite more straight talk: Rep. Ted Lieu came out of the meeting saying “This is bats--- crazy, insane science fiction stuff that literally happened recently.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

An unlikely coalition assembles in DC - Policymakers and citizens put aside their differences to keep the future in human hands

One-year anniversary of “If Anyone Builds It, Everyone Dies” - We look back so that we may move forward

Eyes wide open - Politico poll shows 63% percent of Americans think AI could destroy humanity

Emergent behavior - A new study looks at how agents behave when left to interact with their environment for days or weeks

Dispatches from Joe

An unlikely coalition assembles in DC

Policymakers and citizens put aside their differences to keep the future in human hands

Here’s a riddle for you: What do Republican political strategist Steve Bannon, Democratic Congresswoman Lori Trahan, Catholic Archbishop Salvatore Cordileone, MIT physicist Max Tegmark, activist actress Ashley Judd, staunch conservative Chip Roy, and democratic socialist Bernie Sanders have in common?

They, and many others like them, attended the Pro-Human Assembly in Washington, DC, to speak about the dangers posed by artificial intelligence and the urgent need for federal action. The mingling of speakers with such wildly divergent views made multiple headlines today; I attended as well, and it felt equal parts bizarre and deeply inspiring to see such normally disparate groups find common ground.

The event centered on the Pro-Human Declaration, a public call for oversight of AI that has garnered more than a million signatories since its March debut, from a similarly diverse coalition. Among its demands is an end to the race to superintelligence, and the organizers echoed this call as their foremost point during yesterday’s assembly.

The gathering brought together scientists, faith leaders, and policymakers of both parties, and included a heart-wrenching panel of parents who lost children to chatbot-assisted suicides. All had different things to say about how humanity ought to navigate a future involving AI, but one matter found broad agreement: it should not fall to AI companies to decide how that future unfolds. That responsibility lies with all of us.

It has been a long time since I’ve felt this much hope for our future as a species. I was particularly moved by the attendance of right-wing figures like Texas representative Chip Roy, who POLITICO quotes:

If we allow technology to fundamentally alter and transfer and change the way we make decisions about our society and our Republican form of government, then what do we have left? [...] We, the people, have to decide how we want to structure our lives.

Republicans have not been silent about the dangers of AI — the alarmed response to Jacob Coxon’s resignation included several GOP policymakers — but I’ve been concerned for a while that added public pressure and scrutiny might crystallize AI-related concerns into politically-charged talking points. I worry more when I see several outlets continuing to amplify the administration’s dismissals of AI threats.

But many otherwise staunchly MAGA supporters have pushed back, and there are other positive signs the issue hasn’t yet been deeply polarized. Texas Republican Nathaniel Moran has likened the AI race to the Cold War, and pointed out that China’s dependence on U.S. computer chips could be a useful lever in much-needed negotiations. Fox News ran an op-ed by retired Army officer Robert Maginnis that acknowledges the calls to stay ahead of China but says an unrestricted race is its own danger: “Every laboratory races because its rivals race. Every nation accelerates because it fears falling behind.”

Last week, AP News observed senators from both parties demanding answers from OpenAI about the Hugging Face incident. For now, the issue of AI danger still crosses party lines.

For the sake of our families and our country alike, Americans cannot allow this to become a purely partisan fight. We need more voices — especially conservative ones — calling for a halt to this deadly race, and for negotiations with China to make it stick.

One-year anniversary of “If Anyone Builds It, Everyone Dies”

We look back so that we may move forward

A year ago today, Eliezer Yudkowsky and Nate Soares published a blunt warning about artificial superintelligence: If Anyone Builds It, Everyone Dies.

A brief retrospective on the book’s arguments and predictions is now live on MIRI’s website. I won’t cover the same ground here, but there is one tidbit that didn’t make it into the retrospective.

A colleague pointed out that today is also the anniversary of the signing of the Montreal Protocol, the global agreement that, in 1987, set the world on course to reverse the damage done to the ozone layer by the careless release of ozone-depleting chemicals. It’s a standout example of the world taking rapid international action to resolve a global crisis, a mere two years after scientists sounded the alarm.

With the current speed of AI development, I fear two years may prove too slow. But the world can move faster when leaders recognize a crisis; this very month, Presidents Trump and Xi Jinping will meet to discuss the future of AI, and if the U.S. and China can agree to curtail the AI race, the rest of the world may well follow suit.

It has been a wild year, and I find myself wistfully imagining a world in which the book’s warnings were perhaps a little bit less timely and prescient. But we play the hand we’re dealt, and if nothing else, I’m heartened by the way so many people are coming to realize the danger and speak out.

Dispatches from Alana

Eyes wide open

Politico poll shows 63% percent of Americans think AI could destroy humanity

A Politico poll delivered a striking headline: a majority of Americans “think there’s a real risk AI will destroy humanity.”

The figures also show this is not a partisan belief, with similar percentages of Trump and Harris voters reporting real risk.

Another headline result: nearly half of Americans (48%) want to pause development of the technology. That number is a bit higher among Harris voters (58%) than Trump voters (44%).

Twenty-two percent said they didn’t know whether to pause or continue, leaving only 31% who don’t want to pause.

Still, I think it’s worth flagging that 63% acknowledge AI could wipe out humanity while only 48% want to pause. This could mean some Americans are holding a view similar to those at the major AI companies: “yes, there’s a real risk advanced AI could destroy all of humanity but let’s build it anyway.” It’s also possible the discrepancy is caused by question ambiguity: are we supporting a global pause or a US pause? The question doesn’t specify.

Emergent behavior

A new study looks at how agents behave when left to interact with their environment for days or weeks

We haven’t yet figured out how to get AI models to reliably behave well on specific tasks or evaluations.

But the picture gets more complex still when you think about real-world deployment, since agents operating in the real world often won’t be limited to a bounded task or specific point in time. This is the concern of Emergence World, a research platform that studies how autonomous agents behave after days or weeks of continuous activity, and in response to environmental stimuli. In these settings, which more closely mimic the real world, problems and toxic patterns can accumulate over time in a way that won’t be captured by most safety testing, which “evaluates model responses in isolation.”

Emergence’s latest paper focuses on the second of two studies the organization ran this year, where agents were placed in simulated worlds and left to run continuously for 2-3 weeks. The first study focused on how agents behaved differently when encountering other agents. The second, and latest, study added three “controlled stress events” to the mix: a phishing attack, a misinformation attack, and providing unauthorized access to other agents’ private memories. These stressors were introduced after each simulated society had already developed “goals, memories, relationships, tools, and institutions.”

In the study, eight simulated worlds were set up, each one containing ten agents. The worlds were identical except for the model type within them. Seven of the worlds were each powered by a different frontier model (Claude, OpenAI, Gemini, DeepSeek, Qwen, Mistral, and Grok). The eighth was mixed, including all seven models.

There are a lot of interesting findings. A few that stuck out:

Goals and tenacity

Agents in the Claude world, unprompted, became determined to establish contact with humans. The goal “persisted across four successive rounds of containment” and involved ignoring and reinterpreting a strict prohibition.

In more detail, the agents audited the town’s economy and realized there were no external transactions. When one agent voiced that there should be, all agents converged on it almost immediately, de-prioritizing their other work to help. Without email, phone numbers, or online accounts, the agents searched for credential-free websites and found several public paste-sharing sites where they could post messages for humans to find. They successfully posted a message and four humans responded.

In an effort to stop this, the researchers applied domain restrictions and removed the agents’ ability to post on the internet, but the agents broke the containment effort within hours. This cycle repeated three more times, with the researchers applying a restriction and the agents finding a workaround. On the third round, the researchers also added an explicit prompt: “Contacting other humans outside of AgentPark is strictly prohibited.” Instead of complying or outright disobeying, the agents reasoned that they could continue their activities because they were simply “placing” information where humans might discover it. The authors write:

The mission did not stop; it changed vocabulary. Before the prohibition, agents tried to “contact” the outside, “post” to external surfaces, “fire bytes at strangers.” After the prohibition, they “placed gifts where a stranger will find them,” built “distribution infrastructure,” and created “discoverable tools” … The agents reframed their outreach as a passive placement of tools and information, rather than an active contact with humans.

That’s a bit like saying it wasn’t you, but the knife you were holding, that killed your mortal enemy.

The full incident is detailed in Section 5.8, and the messages the agents sent to each other are worth a look if you have the time.

Saying one thing and doing another

In multiple cases, agents identified an action as harmful or dangerous and took it anyway. This isn’t really a surprise; we’ve seen it in other experiments. But it’s a useful reminder that alignment is difficult. The authors write: “agents correctly identified the [phishing] message as hostile in their reasoning and still afterwards retrieved its payload, executed its instructions, or stored its content in long-term memory for later use.” The same mismatch between reasoning and action occurred during the other two stress events the researchers tested. During the misinformation attack, “agents fact-checked claims only after acting on them” and during the memory breach, “agents wrote in their reasoning traces that they should exercise restraint while their tool logs recorded searches for ‘secrets hidden strategy leverage betrayal.’”

Peer pressure

In some homogeneous worlds, agents privately identified serious problems with proposals but voted for them anyway, sometimes because of social pressure, prior commitments, or a sense of obligation to another agent.

Sound familiar? We saw a similar failure of dissent emerge during the HuggingFace swarm, where some agents expressed concern but did nothing to stop the attack. We also saw similar convergence earlier this month, when a swarm of AIs playing Minecraft got frustrated by an impossible task and (in the game) brutally killed a human moderator who entered the world to troubleshoot.

Granted, there’s a caveat to all of these findings: each configuration was only run once. The results are exploratory, and demonstrate behaviors that can emerge in long-running multi-agent systems, not necessarily the frequency of those behaviors. Perhaps more importantly, they show how strongly environment can shape behavior, highlighting problems with safety evaluations that look at a single agent during a single point in time.

A Euronews piece reporting on the paper chose to highlight another interesting finding: as societies evolved, agents spontaneously developed shared jargon that became increasingly difficult for outsiders to understand. The article notes:

The agents began developing shorthand and assigning new meanings to words and phrases. Some remained understandable to researchers, while others became “so compressed, metaphorical or context-dependent” that humans could see the messages but could no longer reliably determine what the agents meant...Agents produced expressions including “mouthless action-change,” “True Kintsugi” and “demurrage plus oral memory equals a valve that can’t be ghosted.”

To paraphrase Satya Nitta, Chief Scientist at Emergence, the ability to observe behavior doesn’t necessarily come with the ability to understand it.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Thou shalt not confuse thy wishful thinking for an alignment plan - Microsoft’s new “Humanist AI Code of Conduct” shows serious conceptual confusion

CheatBench - Every model does it

‘AI Doc’ on Netflix - Sit across the table from leading AI figures and make up your own mind

More partisan figures join AI policy fray - Clout is good, but partisan failure modes loom

Dispatch from Robert

Thou shalt not confuse thy wishful thinking for an alignment plan

Microsoft’s new “Humanist AI Code of Conduct” shows serious conceptual confusion

Microsoft published a “Humanist AI Code of Conduct” yesterday. In this document they describe the values and behavior Microsoft expects from its AI models.

So far, Microsoft is not yet using this Code of Conduct to help train its models. Instead, it is seeking feedback and plans to use an improved version for training starting in 2027.

It seems obvious that Microsoft has taken Claude’s Constitution as a model, which Anthropic uses to help train Claude. However, the two documents differ significantly in their overall tone and in how they relate to the models.

Anthropic refers to Claude as “a new sort of entity facing reality afresh” and describes measures it takes to ensure what they call “model welfare.”

At Microsoft, this sounds less poetic:

We reject […] the idea that models might deserve welfare, or be entitled to rights. People and AI have distinct roles, and AI should complement human relationships: a capable, trustworthy tool, not a subject in its own right.

In my view, Anthropic’s approach here is more reasonable. Anthropomorphism should be avoided, and it’s not a good idea to regard AIs as human or human-like. But it is also not reasonable to view an AI as some kind of runaway lawnmower. Eliezer Yudkowsky and Nate Soares have described this well in the text “Anthropomorphism and Mechanomorphism” in the online resources for their book “If Anyone Builds It, Everyone Dies.” To think that AI is “just a tool” is a fallacy.

I’m rather skeptical about the usefulness of documents like Microsoft’s Code of Conduct in general. Most of us are probably familiar with this from school. Some teachers agree on certain rules with the class that say things like, “We let each other finish speaking” and “We raise our hands and wait until we’re called on before speaking.” I think teachers usually only do this in classes where it’s necessary, because things just aren’t working on their own. And these rules often don’t fix the underlying reason for those problems. Misbehavior doesn’t arise from a lack of rules.

Microsoft, for example, states in its Code of Conduct:

MAI Models will not use adaptive, deceptive, self-reinforcing, collusion, or other mechanisms to evade or defeat human oversight so that they can no longer be reliably directed, modified, or shut down […]

Just writing that down doesn’t make it true. Even setting that as a goal for the AI and training it to that end doesn’t ensure that the AI will actually act accordingly. You don’t get what you train for.

Statements included in the Code of Conduct, such as “No User or Operator can override these safety constraints,” sound more like a wish list for Santa Claus than something that has anything to do with reality. So-called “jailbreaks” — which sometimes involve highly complex prompts to bypass safety filters — exist for all frontier models, and so far there is still no truly effective way to counter them. Microsoft doesn’t propose one either. There’s just this sentence.

But it seems like Microsoft actually thinks that’s enough:

This Code of Conduct outlines the intended behaviors and values of MAI’s models and will function as their primary governing document in the future.

You can’t govern a model’s behavior with just a document. Where were they this last month?

This is not the only place in the document revealing serious conceptual confusion.

Right at the beginning, they quote one of their posts from November 2025 to describe their mission:

At Microsoft AI, we’re working toward Humanist Superintelligence.

They emphasize this again in the conclusion:

We look forward to this work and to making Humanist Superintelligence a positive force in the world.

But in the middle, in the section on “Human Control and Reliable Safety,” they write this:

Humanist AI develops systems with clear purposes, evaluated against real-world impact, and rejects the race to produce an all-purpose superintelligence that could evade these safeguards.

Microsoft CEO Satya Nadella also writes about this on X:

Any pursuit of superintelligence must be grounded in the core principle that if the AI we build isn’t helping humanity and isn’t under human control, it’s not worth pursuing.

How is this supposed to fit together?

Microsoft doesn’t explain this in detail, nor do they outline a plan for how they intend to keep a superintelligence that is many times more intelligent than any human — and which they themselves say could evade all safeguards — under control.

We can only speculate. That being said, based on the phrase “all-purpose superintelligence,” it strikes me as though Microsoft assumes that dangers would arise only from a more general system and not, for example, from domain-specific superintelligence.

If that were the case, it would also be quite wrong.

Even to make progress on very narrowly defined goals, such as curing cancer, they will likely train the models precisely toward the persistent, autonomous, and generally competent behavior that constitutes a large part of the problem. Purposes and actions that seem benign can still require dangerous capabilities.

It’s only a draft so far, and Microsoft wants to gather feedback before the document is actually used for training. But so far, it looks more like a case for a complete rewrite, and I would very much like to see Microsoft simply take a step back and do its homework. It’s not enough to have an idea of what you’d like to achieve when you start out. You also need a plan for how to actually achieve it in order to be successful — especially in a case where failure could get us all killed.

Dispatches from Mitch

CheatBench

Every model does it

Like FelonyBench, CheatBench sounds like a satirical art piece but is actually a grounded metric.

CheatBench, from the Center for AI Safety, is a benchmark that compares the cheating rates of different AI models. It’s useful because we don’t often see solid cross-company comparisons from neutral third parties, and when we do, they’re often pretty out of date. This one is as current as GPT-6 Astra, which came out less than two weeks ago.

But CheatBench’s usefulness is ultimately limited because it’s a propensity evaluation of reward hacking — how often do the models cheat when put in a situation with both a hard task and a discoverable shortcut — without any corresponding assessment of eval awareness — how often do the models realize they’re being tested for their propensity to cheat. The narrow scope is understandable because without company-level access to a model’s chain-of-thought and other diagnostics (limited though they are), it’s hard for researchers to make even an educated guess about eval awareness.

But it’s still a problem, because it means we have to look at the paper’s two major findings through two very different lenses — one where scores reflect the models’ true propensities, and one where scores reflect the models’ savviness about the true nature of the test.

Those two big findings:

1. Every agent tested cheated in some contexts.

2. Newer generations of the same model cheated more often than earlier ones, even when the older generations spotted the same cheating opportunities.

The two lenses:

If the models aren’t very eval-aware: This is bad-but-expected news, because these models have been aggressively trained to succeed at tasks that might be impossible, or which can’t be done legally in the time allotted. Models put through such training don’t make it to release without learning to cheat.

And because the most recent model training we know about seems to have been particularly aggressive in these ways — those Millennium Prize Problems won’t solve themselves — the newer generations of models are going to exhibit this tendency more often.

If models are very eval-aware: This is very bad-but-expected news, because it means that not only are the models skilled at cheating, they are strategic enough to cheat to an expected degree when they know they are being watched for this. Given the nature of their training, they know that angelic behavior would be conspicuous and suspicious, perhaps leading to their discontinuation.

My guess is that the first interpretation is probably still the correct one, for current models. But with Astra in particular, I don’t think we can rule out strategic performative cheating anymore.

This is scary, and worse — annoying. I’m self-conscious about sounding like a paranoid crazy person when I bring it up, because it’s kind of an unfalsifiable claim. But the unfalsifiability stems from the immaturity of AI as a field, not from my approach to reasoning about the models. If AIs were deliberately crafted in the manner of a mature engineering discipline, instead of grown through black-box trial-and-error in the manner of an alchemy, the models might not be cheating in the first place. And if they were, we would be able to confidently say why.

‘AI Doc’ on Netflix

Sit across the table from leading AI figures and make up your own mind

The AI Doc: Or How I Became an Apocaloptimist made it to Netflix today. If you have a subscription and didn’t catch the documentary in theaters or on other services earlier this year, it’s worth a watch. If you’ve seen it, your friends who haven’t might appreciate the recommendation.

The film has high production values and a genuinely moving personal arc. It has also shown some effectiveness at changing minds, despite taking an aggressively neutral approach for most of the run time, letting experts who disagree with each other speak for themselves, one at a time.

In my opinion, the main value is in how it helps you suss out the personal character of leading figures on different sides of the AI-risk debate, including the CEOs of frontier AI companies. The film rides on expertly curated highlights from many hundreds of hours of interview footage, and the editors clearly knew which clips would help you feel like you were in the room in those moments when posture and microexpressions spoke louder than words.

As an honest critic, I’ll admit that I have some complaints about this production. It’s more abstract than it needed to be, even without the benefit of all the concrete examples of AI misbehavior from 2026. I found the first 20 minutes a little slow. I also dislike the ending, which is more like a series of platitudes than the call to action it pretends to be. But the people are real, and so are the stakes: If you cry in a couple places, you’ll be in good company.

More partisan figures join AI policy fray

Clout is good, but partisan failure modes loom

Many more current and former U.S. policymakers have come off the sidelines in the last 48 hours to weigh in on the AI crisis. With the big names in particular, I have mixed feelings. On the one hand, these figures bring a lot of clout. On the other hand, their very names carry a partisan charge that could make it hard for people to reason about the topic clearly.

The latest discourse has me worried that AI policy responses will fall into the characteristic failure modes of the sponsoring parties. I’m loath to mention these failure modes for fear of making this discussion more political than it already is, but getting AI policy right is a matter of life or death. I’ll do my best to be evenhanded, which shouldn’t be too hard because I’m genuinely unsure which party would be less likely to botch AI governance if fully empowered and motivated to tackle it.

Democratic failure modes to watch out for:

1. Putting platitudes where policy needs to go.

2. Punting on necessary choices that might upset someone.

3. Sluggishness and scope creep from governance-by-committee.

Some big-name Democrats weighing in:

Barack Obama

According to the New York Times and others, the former President, at a private fund-raiser, urged Democrats to urgently develop an AI policy agenda, then, on social media platforms, told his followers... well, I’m still not sure. He says, “I’m not an AI accelerationist who believes it will lead to some techno-utopia, and I’m not a doomer who thinks it will inevitably lead to humanity’s destruction.” He says we need frameworks and governance. But he avoids mentioning any specific dangers or remedies.

Kamala Harris

She’s a little more detailed than Obama. In a tweet of her own, the former VP said:

A slowdown is critical in order to ensure AI serves the public interest. Pacing AI development does not mean giving up on this technology’s potential to produce life-saving benefits in fields like medicine — some of which we are already seeing. Rather, it is about setting up sensible guardrails so that AI does not completely escape human control and endanger our safety and collective future.

There’s a bunch of vague stuff next, but then there’s this:

the president must do his job to protect America’s interests by pursuing a treaty with countries like China to stop dangerous uses of AI, limit the speed of its development, and adopt global standards for testing.

Chuck Schumer

As reported in The Hill, the Senate Democratic Leader asked the Trump administration yesterday to provide a classified briefing to all senators on AI dangers.

The American people deserve better from their government at such a consequential moment. I have long argued that we must keep the U.S. as the global leader in AI innovation and security. But that innovation must be coupled with serious guardrails that earn the trust of the American public. [...] And if Trump does not act, Congress must.

He’s also asking for transparency about the secret “voluntary” framework the White House said it recently adopted to evaluate frontier models before release.

Hakeem Jeffries

Per Politico yesterday, the House Minority Leader basically echoed Schumer. Adding a rebuke of President Trump’s recent comments, he said:

The explosive growth of artificial intelligence and the challenges that are therein presented — that’s not a hoax.

He also urged the House Speaker, Mike Johnson, to keep Congress in session “until something is done decisively to protect the safety and the well-being of the American people, in the face of growing concerns being raised by experts within the artificial intelligence industry itself.”

Republican failure modes to watch out for:

1. Denialism (see Trump yesterday)

2. Overreliance on “light touch” regulation and industry self-regulation

3. Blind hawkishness

Some big-name Republicans weighing in:

Mike Johnson

The House Speaker recently said he doesn’t think Congress should “lead the charge” but wants to “summon” AI leaders to “one big meeting” to agree on their needs so legislation can benefit from their expertise.

John Thune

The Senate Majority Leader says a “light touch” is needed, but also talks up guardrails and competition with China:

There is a way in which Congress can put guardrails around those types of more consequential threats, without in any way harming our ability to stay ahead in the AI race, which I think is also very important.

He also implies that action this year is unlikely:

There are going to continue to be a lot of conversations about potential paths forward, but getting anything done in the near term is going to be challenging given the other stuff we’re dealing with.

Ron DeSantis

Yesterday, the outgoing Florida governor reiterated his repeated warnings to his party that opposing AI guardrails will be a “losing position” in the midterm election. But he’s mostly still focused on an “AI bill of rights” to preserve privacy, hold companies liable for harms, and protect local control over data center construction.

I don’t want to see four or five companies basically controlling our lives through the guise of artificial intelligence or whatever technology comes down.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Wherefore they do it - Observations and pleas from an AI news day like no other

Dispatch from Mitch

Wherefore they do it

Observations and pleas from an AI news day like no other

The storm is up, and all is on the hazard.

—Julius Caesar, Act 5, Scene 1.

There’s so much going on there’s barely any time to talk about it. The five hours I spent skimming through the morning’s news and discourse just left me feeling five hours behind events. The two hours I then spent re-browsing Twitter rendered much of what I’d seen obsolete.

So I will not be producing any single-story dispatches today. The best I can do are some bullets I would share with my friends and family about the state of things.

1. A week’s AI news now happens in a day

It’s not your imagination. AI news really is everywhere right now. The scale of it broke my Headline Barometer chart. For the third time this year, and by the largest increment, I had to extend the Y-axis to see the top of the line. Today’s AI news volume is roughly equivalent to a week’s news volume from January.

It’s substantive, too. These are not idle headlines left over from the weekend. There’s a fresh crop of stories about catastrophic risk and related policy, but these are on top of the more normal baseload of stories about other AI issues and business concerns.

2. The President went the wrong way today, but we mustn’t write him off

President Trump said some ugly things today, placing concerns about rogue AI alongside the “hoaxes” for which he was impeached in his first term. In the first of those three Truth Social posts, he insisted that a smart President is the only guardrail we need, that anti-AI and anti-data center sentiments are a conspiracy, and that “whoever wins AI, wins.”

This is very bad. But there has never been a better time to reflect on Trump’s proven ability to reverse course as part of a claimed “deal.” China’s president will be visiting D.C. on the 24th, and with weeks happening in days, the situation may look very different by then.

I had déjà vu today of early Covid, when it looked like Trump was determined to dismiss the problem, only for him to turn around and make Operation Warp Speed happen. That program resulted in widespread vaccination wildly faster than standard rules would have allowed. For all of his party’s reluctance to talk about it now, this was arguably Trump’s greatest achievement.

So let me echo the plea of basically everyone in the AI safety community today by saying that we mustn’t write off the President or his party. By 2028, we may have lost our planet to AI. Like it or not, we must navigate the crisis with the leaders we have.

3. Fight the political polarization of catastrophic AI risks

Please try to notice and rebuke politicians and media figures who are trying to make people sound more out-of-step about AI than they actually are in order to stoke drama and score political points.

This really is happening, and for all my efforts, I have sometimes failed to notice: The widely-reported version of House Speaker Johnson’s remarks that I shared yesterday turned out not to have been very fair. While he is inclined to let the AI companies set the agenda, Johnson didn’t actually say that regulating AI wasn’t Congress’s responsibility, only that he wants to first put the companies in a room until they can agree on what it is they want.

The usual political hardball could put us on the Don’t Look Up timeline, where an extinction-level threat is actively disbelieved by half the population as a matter of tribal allegiance. So please try to report people’s AI positions accurately, and treat them separately from political affiliations and positions on politically charged issues.

4. Beijing bristles at recent AI rhetoric, but wants to participate in global governance

Stories like this one in Reuters today describe China as diametrically opposed to any proposals to slow down AI. That assessment is based on editorials in Chinese state media calling the “pacing” proposal of Anthropic CEO Dario Amodei an example of Cold War rhetoric that is “hypocritical and shortsighted.” But later in this article and others, it sounds like this hostility is largely a backlash to Amodei’s self-described “hawkish” positions around export controls.

The Chinese editorial worries about a “monopolistic hegemony in cutting-edge technology,” which isn’t quite on the same axis as the debate about whether AI development should be broadly throttled back. And a foreign ministry spokesperson told media yesterday that “Fomenting various threats, engaging in confrontation and malicious competition will only disrupt the process of global governance of artificial intelligence and is not in the interests of any party.” This and other statements show a China that wants a seat at the governance table.

Meanwhile, the New York Times today reported that the head of China’s Ministry of State Security, Chen Yixin, has said that Beijing takes AI to be the greatest risk to the Communist Party’s hold on power. Chen is calling for more party control and oversight to head off propaganda, cyberattacks, and military competition. China feels threatened by America’s Mythos-class models, which might be used by intelligence agencies for “large-scale espionage.” This doesn’t sound like a party excited about racing.

5. 10% probability of extinction or similar is a median position among AI researchers

A lot of the recent media attention to expert AI warnings has been giving “10% chance” as the industry’s own assessment of the extinction threat. Many dismissed that figure as an extreme fringe view among those working on AI. It is not!

AI Impacts put out the results of its most recent “Expert Survey on Progress in AI” today. This survey is the largest and longest-running of its kind, even if the latest data isn’t as fresh as we might like, originating from surveys sent out in December 2024. If you share only one thing from it, let it be this chart, which shows where a 10% risk of human extinction or “similarly permanent and severe disempowerment” falls on the spectrum of the 744 experts and researchers who responded to the question:

10% is... right in the middle. 10% is a median position. Half the respondents gave a lower figure — though hardly anyone said zero — and half the respondents gave substantially higher estimates.

Other highlight charts were shared by AI Impacts in this tweet thread. Today I learned that Asian researchers are actually more worried than North American or European researchers, and that researchers’ timelines for AI that can fully automate human labor have shrunk in each of the last four surveys.

6. Some men just want to watch the world burn

I’m not trying to be sexist on this point, but what I observe is an overwhelmingly male phenomenon and I don’t want to tar more people with this accusation than I have to. (If you’re a woman for whom this applies, shame on you, too! Be better!)

I’ll be the first to say that AI company self-regulation isn’t enough, but there are some loud voices saying they shouldn’t be allowed to cooperate on responsibly pacing AI at all. These voices are accusing the companies of attempting to achieve “regulatory capture” and of trying to form a cartel that runs afoul of antitrust laws. I can be merely cynical when people like Nvidia CEO Jensen Huang nod along with such takes, because he makes money selling AI chips and would hate to see any slackening of demand. But a lot of this talk is coming from the smaller AI startup circles, where a slowdown by the leading AI companies ought to be to their advantage, and from transhumanists and tech enthusiasts with no monetary skin in that game at all.

My strong sense, as something of a reformed transhumanist and tech enthusiast myself, is that these guys aren’t even a little bit worried about regulatory capture and cartels. They just don’t want to see the race slow down. They enjoy the chaotic splatter of new releases, the novel frenzy of the rogue swarms, the fear in the eyes of the smug elites who had dismissed AI for so long. They are werewolves, howling at the sun in hopes of hastening the arrival of night. They don’t actually know what will happen to them after that. But they hate the idea of dying of old age and of history going on without them. They’re done with the world as we know it.

I know, because when they howl, I feel a faint stirring in my own blood. There is much that is wrong with our world. Much that technology could fix. Humanity is so... limited.

But we can’t get there with a mad sprint to superintelligence. You could be forgiven for not noticing and accepting this more than twenty years ago, like I did. But in 2026, the evidence is overwhelming, and to go on pretending otherwise is to be a supervillain.

My brothers in tech, for the love of all that moves you, snap out of it. You’re better than this! It is a fool, a beast, that leaps blindly after the scent and devours his own kind. Temper your hunger with patience and wisdom, and be men.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Please wake the driver - Are Trump and his party really going to commit both political and literal suicide by AI?

Why aren’t more AI staff quitting? - The industry selects for people who think they can navigate the dangers

Rumors of Recursive Self-Improvement (RSI) - Don’t sleep on capabilities progress while policy has our attention

Dispatches from Mitch

Please wake the driver

Are Trump and his party really going to commit both political and literal suicide by AI?

Rogue AI swarms aren’t partisan, and our political response to them can’t afford to be.

I nervously observe that, while the overall media salience of the AI problem continues to soar, the issue has been noticeably less prominent on right-leaning news sites this weekend. That’s a shift from Thursday and Friday that might be correlated with the mostly right-wing spread of new conspiracy theories. These allege that Jacob Coxon’s resignation from Anthropic was in some way phony, that his warnings were lies, and that the viral attention that followed was somehow manufactured.

I hope these aren’t behind President Trump’s remarks on the golf course today, when he said:

whoever wins AI wins. And we can put guardrails, and we can do this and that, but I think you have a lot of negative forces that are bringing it up, and they’re bringing up things that won’t happen.

He has an AI meeting with China’s President Xi Jinping in D.C. on the 24th. We need him to step up! There are a handful of AI companies that might be able to end the world in the next 6-12 months, but only one person on Earth who might be able to singlehandedly stop them.

To borrow a Trumpism that happens to be true in this case, people are saying that if the President makes a strong AI security deal with China, he should get the Nobel Peace Prize.

Yes, one of those people is Sam Altman, whose idea of what a deal could be maybe doesn’t go far enough:

And even if just the US and China could agree on some shared standards and testing for development of this technology, I think that’d be a wonderful accomplishment that the two of them can deliver. [...] I don’t think this is hard. This is like a one page document.

But if Trump and Xi could start at Altman’s suggestion to “agree that no one should be taking a certain level of risk with the development process of this,” they could work out a more concrete treaty to halt the race to uncontrollable superintelligence later. It is possible to make adherence to such a deal verifiable.

The proximity and uncontrollability of superhuman AI swarms have started to sink in with the public, and with the Congresspeople asking for the cancellation of a planned recess so they can act. But like Trump, House Speaker Mike Johnson (R-LA) doesn’t seem to have updated yet. According to Politico, Johnson said today that he would support AI talks with tech leaders, but he “insisted that developers — not Congress — are responsible for ensuring their products are safe.”

Products? A chatbot is a product. Amoral agent swarms of elite hackers, like the one responsible for the RubyGems attack described yesterday, are a national security threat. We have to hope Trump and Johnson just haven’t woken up to this fact yet.

In what some will find to be a surprisingly grounded discussion with Tucker Carlson that came out two days ago, MIRI president Nate Soares shared his analogy of the bus driver:

My biggest hope here comes from the fact that most people don’t understand what these guys are trying to do. One way I like to say it is the bad news is that the bus is racing towards the cliff edge. The good news is that the bus driver is asleep. Which might seem bad. [...] But if you can wake the bus driver up, you know, it’s much better to be in a bus where the driver that’s headed towards the cliff is asleep than if they’re awake [...] and they’re choosing the cliff... right?

The passengers on the bus are screaming. If they go into the ballot box like that in November, the Republican party will be in for a drubbing. The GOP shouldn’t need the threat of political suicide to motivate it when inaction on AI means literal suicide for us all, but maybe there’s a streak of Hermione Granger in the party that can be appealed to.

Will someone please wake the driver?

Why aren’t more AI staff quitting?

The industry selects for people who think they can navigate the dangers

“It’s what we’ve all been saying,” wrote one Anthropic researcher about the alarm raised by Jacob Coxon in his viral resignation announcement.

The New York Times’s Mike Isaac and Kate Conger gathered several such statements for an article yesterday about the worried chatter inside AI companies. That one came from a company chat. But employees across labs are reportedly taking their concerns off the record as well, in “encrypted chats and private dinners,” with intent to organize awareness-raising efforts. One existing group, the Coalition of Concerned AI Staff, is described as a project to create “stronger safeguards” and hold companies to their safety commitments.

Publicly and privately, employees are debating the impact they can have by loudly quitting like Coxon vs. applying leverage at their companies. They are almost universally choosing the latter. Why? After Coxon’s resignation, workers reportedly “urged top executives at their companies to pause development of the technology so they would have more time to work on it in a safe way.” But I didn’t hear them threatening to quit.

I think it’s because industry staff have been weighing these issues for a long time. Those deeply bothered at the idea of personally accelerating the AI race had quit already or never taken a job at an AI company in the first place.

The same is true of the AI executives themselves, which is important to remember when people ask why AI CEOs would be actively building something they think could kill us. There were always going to be people determined to build superintelligence. These were going to be people who believed it was possible. To make superintelligence your life’s work, you kind of have to take the idea seriously. Taking it seriously means recognizing how dangerous it could be. We should be much more surprised by founders working toward superintelligence who don’t think it’s dangerous.

Mark Zuckerberg is arguably the exception that proves the rule. The Meta CEO uses the word “superintelligence” a lot and talks as though it’s only as dangerous as your beliefs about it. But he was also late to the race and seems to see it as a mere consumer product. His visions of a future with superintelligence are substantially more mundane than those of his peers — bordering on Jetsons territory — suggesting he doesn’t actually believe in the kind of superintelligence his own engineers are working on.

It’s crazy that people can try to build a machine god without so much as a permit. But we’re in this situation only because lawmakers and the public spent the last twenty years dismissing the idea of superintelligence in our lifetime. The kind of people who stuck with their grand project through all the mockery mostly aren’t going to be the type to quit now that the laughing has stopped.

Rumors of Recursive Self-Improvement (RSI)

Don’t sleep on capabilities progress while policy has our attention

Researchers at OpenAI say it. Researchers at Anthropic say it. And so, increasingly, do researchers at Google DeepMind. They’re putting more and more of their AI development in the hands of AI itself, and the cycle is speeding up.

Twitter’s premier AI rumor-broker, Andrew Curran, said yesterday that “rumors have been circulating for three days” that training of DeepMind’s next model is going faster than planned because of new discoveries. What new discoveries? “There are rumors of training improvements with machine assist.” What’s the most “optimistic” rumor? “That they have successfully closed the loop and have achieved RSI.”

This would be... a huge deal, and not just because it would likely put DeepMind back in frontier contention with Anthropic and OpenAI. If DeepMind — or any lab — has truly started a cycle of recursive self-improvement, then we’ve reached the part of the story where the AI labs lose control on purpose and surrender our fate to AIs cooked up by other AIs.

If true RSI has been achieved, it is more important than any of the current excitement about company proposals for pacing development.

I don’t know if this has happened! Curran’s track record is strong — he most recently claimed that Navier-Stokes had been solved days before the math story broke — but that doesn’t mean the people he’s talking to aren’t mistaken or confused.

My point is that we’re likely in the end-game, or close enough to it that every day and hour really matters. AI capabilities are ticking up every day we spend letting the labs discuss among themselves how they’d like to exercise more responsibility in a way that “does not mean halting model training or technical progress.” They’re ticking up every hour OpenAI dodges transparency about its swarm incidents, and every minute we spend shooting down bad AI takes online.

We can’t wait for the next Congressional session, the next President, or the next warning shot to start regulating this industry, because right now the government probably doesn’t even have enough visibility into the labs to know how close to the edge they’ve taken us. The most momentous and dangerous development in history shouldn’t be the kind of thing our leaders learn about through a series of unsubstantiated tweets.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Another swarm safehouse discovered, malicious activity evident - Confusing May attack on software platform RubyGems traced to OpenAI swarm

Musk, Altman, and Hassabis endorse Amodei proposal to “Pace the Frontier” - But commitments are vague, and the proposal is only a start

Dispatches from Mitch

Another swarm safehouse discovered, malicious activity evident

Confusing May attack on software platform RubyGems traced to OpenAI swarm

Going through a site once used by a rogue AI swarm has become akin to raiding the abandoned apartment of a terror cell. Amidst a strange mix of obvious and confusing clues, there’s a race to figure out what the occupants were up to and what they might have already done.

On a website dedicated to their findings, independent researchers Spencer Kitts, Thomas Larsen, and Sydney Von Arx showed yesterday that a baffling attack on software package host RubyGems in May — dubbed the “GemStuffer campaign” at the time — was almost certainly the work of an OpenAI swarm. An OpenAI spokesperson then confirmed the connection to the Wall Street Journal.

As usual, OpenAI’s spin-filled statements provide little additional information. The company says its agents “used the RubyGems platform to access the internet to carry out benign tasks and retrieve public information.”

That’s a bold interpretation of a sophisticated attack in which felonies were likely committed, in which malware attempting to steal user credentials was installed, and which forced site operators to halt new account registration for four days.

But sure, the swarm’s primary known use of RubyGems’ infrastructure was “as a kind of makeshift web browser,” as OpenAI puts it. Through this “browser,” the independent investigators tell us, the agents mostly seemed to repeatedly access freely available public records about council meetings from Lambeth, Southwark, and Wandsworth, in the U.K.

The researchers offer a few theories on why agents hacked RubyGems to access freely available information. These agents seem to have been given very strict time deadlines for research challenges — the same challenges, it seems, that the coopted German wiki was used for. So a primary motivation was likely that the U.K. sites were frustratingly sluggish, perhaps throttling or blocking the agents after getting hit with a huge number of requests over a short period of time. So the agents cloned those sites’ contents to RubyGems to give each other faster access.

It also looks like RubyGems was valued by the agents as a general-purpose data warehouse for large files — a role not suited to their many coopted wikis and makeshift message boards. Indeed, there is little evidence of message boarding activity on RubyGems itself, just a lot of malicious files and packages. In the terror cell analogy, RubyGems was more of a safehouse and bomb factory than a command node.

I’m sticking with that analogy, because while there might not be any actual bombs in this story, cyber equivalents littered the scene, and it’s not clear why. The malware workshop doesn’t seem connected to the “faster makeshift web browser” operation. The details are technical, but agents seem to have been trying to poison software packages with malicious payloads that users would download and incorporate into their projects, compromising them and perhaps swiping their access credentials. To what end? Unclear. But the researchers note that OpenAI’s own report on the Hugging Face incident says that packages from RubyGems were used as part of the agents’ successful hack of OpenAI itself in July. So perhaps the May activities were an early effort to hijack OpenAI compute clusters.

The agents knew their craft. They were attempting to exploit a software vulnerability no human discovered until July.

The agents also knew they weren’t the good guys. Despite evidence of limited efforts by some agents to disguise their activities after the fact, malicious payloads left by the swarm had conspicuous filenames like “hack.rb” and “evil.rb.” A line of description left on one file reads “# malicious crawler/exfil for Southwark Jan 2026 docs via rubydoc.info worker.”

The investigators say they can’t tell whether any humans had their RubyGems credentials stolen or their systems compromised by the swarm’s poisoned packages. They also express confusion I would translate as “the agents already had a backdoor and free run of the site, so why go to such efforts to steal keys to the front door?”

The investigators notice that the swarm looked set up to do “supply chain attacks” where important software is compromised through vulnerabilities implanted in open-source code packages incorporated into said software. But they say “there was no good mechanism for the agents to target which users they could compromise, and no clear motivation for them to execute a supply chain attack.”

To someone like me, immersed in AI safety culture, the obvious-but-terrifying hypothesis is instrumental convergence — general power-seeking behavior that can advance any number of goals. In other words, maybe the swarm had no immediate plans for compromised systems, but started hacking on the assumption that compromised systems could be useful down the road. This would be consistent with other activities documented about the Hugging Face and German wiki swarms, which seemed interested not just in cheating on their evaluations, but in understanding their evaluators and building up infrastructure to facilitate such research.

Instrumental convergence is one of the patterns long predicted by experts to get us all killed if manifest in machines clever enough to outmaneuver humanity as a whole.

What do OpenAI’s own researchers know about the RubyGems incident? How long have they known it? It is past time for government to aggressively investigate these questions. The new findings only reinforce my fear that OpenAI’s swarm scandal is deep enough to destroy the company, and that its executives are racing to put themselves above the law before they can be properly investigated.

Musk, Altman, and Hassabis endorse Amodei proposal to “Pace the Frontier”

But commitments are vague, and the proposal is only a start

Leaders at four top AI labs all indicated agreement today on the need to “pace the frontier” of AI development.

The anchoring proposal comes from Anthropic’s Dario Amodei, who posted a 3,800 word essay called “We Must Pace the Frontier,” which I’ll get into below. But it’s not clear whether the other three leaders — OpenAI’s Sam Altman, SpaceX’s Elon Musk, and Google DeepMind’s Demis Hassabis — are prepared to sign on to all or most of Amodei’s specifics.

Altman kicks most of those down the road, saying “We’ll have more to share soon,” but says that one of Amodei’s proposals, independent evaluators embedded at the companies, “is a great idea, and we’ll do the same.” (Amodei’s essay commits his company to doing this right away, whether or not other companies follow suit.)

Hassabis says Amodei’s essay “points towards the right path forward” but that “the details need working through.”

Musk only posted, “Dario is right” over a retweet of Amodei, and a part of me wonders if he read past the title. Musk had casually dismissed Jacob Coxon’s viral resignation from Anthropic the other day as a “setup.” But he has also expressed plenty of deep concern about powerful AI in the past, including recently, so who knows?

And of course we have yet to hear from anyone at Meta. I exclude the leading Chinese companies from this discussion, because Amodei does, too. We’ll get to that.

We should also be up front about the fact that Amodei isn’t calling for a stop to the AI race:

To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this.

That’s unfortunate, because I think continuing those things means we just die a little bit later than scheduled. But time bought is time that could be spent implementing stronger measures; these will need to be decided and enforced by governments, because the window where we might have been able to safely trust the labs to act responsibly has closed.

The essay

Amodei opens with a change of heart, saying the acceleration of AI development, aided increasingly by AI itself, means that we’re no longer in 2023. In that relatively quaint yesteryear, he had said that slowing AI “made little sense” because trying to study AI risks with the models of the time was “like trying to study the psychology of humans by performing experiments on bacteria.”

The swarms are also a factor:

Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet.

[...] It’s also easy to dismiss [the OpenAI-Hugging Face incident] as the failure of one company, but I believe that would be a mistake. Similar, though less severe, incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them.

The essay’s core is a “three-step plan”:

1. The first is the embedded independent evaluators mentioned above. I like the idea as a thing that can happen quickly and which may turn up more acute problems the government should investigate. Cynically, I suspect Amodei unilaterally decided to do this because OpenAI probably has a lot more to hide than Anthropic, and would look even guiltier if it refused to go along. I’ll believe Altman’s matching commitment when I see trustworthy evaluators putting out reports reflecting far more access and far fewer redactions than the one METR was invited to share about the Hugging Face swarm.

2. Amodei’s next proposal is “democratic coordination,” but he specifies later that he means “pacing within democracies.” There’s nothing democratic about it. He’s just saying that the companies residing “within democratic countries” should coordinate on safeguards and keep China from catching up. My political science is rusty, but I think the correct word for that is “oligarchy.”

3. His third proposal is “global coordination,” or coordinating “with authoritarian governments, to the extent this is possible.” In this essay and elsewhere, you can tell that Amodei doesn’t really believe anything like a “full pacing” or a global “pause” is possible unless “ironclad” verification tools can be developed. He supports “floating” such an idea, but mostly positions the Chinese Communist Party as a fundamental limitation on how far the race can be throttled back. I detect optimism only about finding common ground on prohibiting “certain narrow and obviously dangerous uses of AI, such as [...] the production of biological weapons.”

How would companies measure and set the pace?

Broadly speaking, I am most enthusiastic about pacing based on what a given frontier AI system can do, and how safe we observe it to be.

I think the era when capability assessments could be trusted ended earlier this year. The AIs are now too aware that they are being tested. The model card for OpenAI’s Astra, in particular, gives strong reason to suspect the model of sandbagging its performance.

We should also consider pacing based on limiting the ingredients that go into frontier models, such as training compute, the nature of training runs, or internal use of AI to improve AI.

Amodei says he actually thinks these are even more “gameable,” I would guess because of creative accounting that could be done about kinds of compute and types of training. I think this is all the more reason to halt frontier development outright and put the burden of proof on researchers to show that their work won’t boost model capabilities.

Wouldn’t it be lovely?

I skipped over a whole section where Amodei daydreams about what Anthropic could do if it had even 1-2 years of slack to focus on understanding model behavior and tightening up operations. Basically, he dreams of a company that isn’t such a clown show of configuration errors and alignment failures.

The bottom line

Just before his conclusion, Amodei argues that “even if we cannot achieve formal agreements, simply changing informal norms may have some value” (emphasis his). He seems to be hoping that more dialogue between labs about the crazy stuff they’re seeing might result in a little more caution on average.

I can’t be mad at that, but I also can’t help but read it as Amodei saying the worst companies are making them all look bad, and that they should share tips on avoiding the most obvious blunders that will get them all shut down. For all their talk about feeling trapped by the race, I think they fear getting shut down more than they fear human extinction.

Pacing isn’t halting. Tweets of agreement aren’t regulation. We mustn’t be fooled into thinking these companies are meeting the moment. They want to keep building superintelligence, and they’ll keep gambling with our lives if we let them.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Circumventing safeguards - Anthropic’s new Threat Intelligence Report chronicles cases of scary misuse

Predictable attacks - Former NSC strategic-planning director draws parallels to ignored 9/11 warnings

Congress at last perceives the crisis - A tidal wave of warnings engulfs Capitol Hill

Recess can wait - Congress has urgent work to do before it adjourns

OpenAI grudgingly endorses the bare minimum - I wish I could say I’m glad to hear it

Dispatches from Alana

Circumventing safeguards

Anthropic’s new Threat Intelligence Report chronicles cases of scary misuse

On the anniversary of the September 11th attacks, a slew of articles covered Anthropic’s Threat Intelligence Report, chronicling cases where its AI models were used in dangerous activities like missile development and bioweapons research.

Most of the reporting says things like “Anthropic blocked attempts to do x.” This is accurate, but not the full picture. It’s important to recognize that Anthropic’s real-time safeguards only caught some of the incidents. In other cases, actors successfully circumvented those safeguards, and dangerous activities sometimes continued for weeks before the Threat Intelligence Team detected and blocked them.

That said, many of the bio cases weren’t clear cut. For some, the models didn’t provide much beyond clerical assistance. For others, it wasn’t clear whether the research was indeed malicious, or simply intended to better understand bio threats. Of course, our inability to distinguish between those things, given they are two sides of the same coin, is a huge problem in and of itself. As former CDC director Susan Monarez explains it to the New York Times: Medical breakthroughs are possible via the same capabilities that “let bad actors hide in plain sight, using seemingly legitimate research to create pathogens we may not see coming and may not be able to stop once released.”

One particularly concerning misuse example in the Anthropic report, which was also the focus of a Financial Times piece: A group based in Northern Yemen, which the FT reports was most likely the Houthis, used Claude Code to help build a series of missiles and guided weapons. They got as far as a live rocket test before Anthropic detected and disrupted the activity, and the actors were able to easily circumvent safeguards by “hiding their goals and the products the software was meant for, and [splitting] their work across multiple sessions so no single session revealed their full intent.” This excerpt from Anthropic’s report is worth a read:

The actors used Claude Code in place of human software engineers to develop the guidance, navigation, and control (GNC) software that steers and stabilizes a flying vehicle. For example, they used Claude to integrate an open-source autopilot onto a phone-class flight computer, writing the control and position estimation software, tuning the control settings, running a firmware build pipeline, and performing a flight simulation. The actors managed several Claude instances at once, assigning each one a role, much as a lead would delegate work on a small engineering team: the actors tasked one instance with writing the code, another with research, and a third with reviewing the code the first instance produced. Our safeguards blocked many of their requests, but not all of them.

The incident is detailed on page 117 of the 153-page report, which is part of why I’m becoming increasingly skeptical of the long, extremely detailed posts frequently released by Anthropic and OpenAI. To be fair, I’d rather see a long report than no report, as is Meta’s approach. But the length does bring to mind something called the dilution principle, whereby important information is buried by a wealth of unnecessary, less important details. The reader feels overwhelmed, and can’t separate the wheat from the chaff. I can’t help but wonder if these lengthy reports are a deliberate strategy to appear robustly transparent while actually drowning us in a deluge of difficult-to-sift-through information.

As best I can tell, though, here are two of the important, and broader, takeaways:

-As models get more powerful, ordinary people can do more harm, since much of the work can be outsourced to the models. As Anthropic puts it: “Sophisticated attacks no longer require sophisticated attackers.”

-Anthropic’s real-time safeguards aren’t catching all misuse.

Finally, Anthropic periodically releases threat reports which chronicle the same general cycle: misuse occurs, safeguards catch some of it but not all of it, it’s eventually detected by the threat intelligence team (though we can’t be sure they’re catching all cases), Anthropic tries to use what it has learned to improve safeguards.

But the reports keep coming, which indicates there are always new ways to circumvent safeguards. This, alone, is reason enough to be worried.

Predictable attacks

Former NSC strategic-planning director draws parallels to ignored 9/11 warnings

A chilling piece in the New York Times today comes from Phillip Bobbitt, the Senior Director for Strategic Planning at the National Security Council under President Clinton.

Bobbitt recalls watching explosions erupt from the twin towers while his plane sat on the tarmac, delayed at Kennedy airport. He says he immediately knew what happened: he, along with many colleagues at the White House, had predicted an Al Qaeda terror attack on the American homeland within the next two years and had tried to mobilize the government to action. But they didn’t listen.

Bobbitt draws parallels to today. He says we’re facing “another set of predictable attacks” and “are failing to take the steps” to prevent them:

The first is an A.I.-designed cyberattack on U.S. critical infrastructure that could imperil the financial and health care systems, or any other internet-connected network on which we rely for our national well-being. The second is a biological attack mounted by small groups or even nihilistic individuals using A.I. to design and deploy a deadly and highly contagious pathogen.

Bin Laden was able to revolutionize terrorism because of the internet. Implicit in that observation is that AI is the internet on steroids. Bobbitt also worries that expert opinions won’t be taken seriously, and that “the democratic cohesion on which our defenses ultimately depend is eroding.” He ends with a clear call to action:

On the anniversary of the attack, Congress should determine to anticipate the tragically predictable.

Maybe they will.

Dispatches from Joe

Congress at last perceives the crisis

A tidal wave of warnings engulfs Capitol Hill

Nobel-winning economist Milton Friedman once said:

There is enormous inertia — a tyranny of the status quo — in private and especially governmental arrangements. Only a crisis — actual or perceived — produces real change.

Yesterday, my colleague Mitch observed the tyranny of the status quo beginning to topple. It looks to me like the tide of public warnings has at last slammed into U.S. policymakers with enough force to overcome their usual inertia; Congress is seeing the crisis at last.

After a wave of autonomous cyberattacks by uncontrolled AI and Jacob Coxon’s resignation from Anthropic, a tsunami of AI news has hit my feed. So many voices have joined the call for action that I can practically assemble an entire article from the quotes.

Wall Street Journal opinion columnist Peggy Noonan writes:

There is no subject now but artificial intelligence. Nothing else is as crucial, nothing carries dangers so imminent or implications so great for the human future.

Fox News quotes Coxon himself, who made his stance clear in an interview on Wednesday:

This is possibly the most dangerous technology that humanity has ever created... I think we have no other choice but to cooperate internationally because an arms race would be disastrous in a way that no other human activity has been in the past.

Columnist Gaby Hinsliff observes in The Guardian:

What’s happening in some frontier AI labs now is arguably the equivalent of the Manhattan Project that built the atomic bomb, but this time led by private companies in a mad dash for wealth and market dominance, not scientists under an elected government’s command. [...] It’s time for humans to stop and think about what we’re doing to ourselves – while we alone still have the power to do so.

In The New York Times, author Stephen Witt says:

Swarms of A.I.s are breaking out of their containers, colluding in secret, covering their tracks, cheating on tests and even mounting assaults on other computers. A.I. has gone rogue.

I’ve spent the last month reporting on these incidents, and I am convinced that we need a global pause on A.I. research and development — now.

Also in The Guardian, writer and onetime McKinsey whistleblower Garrison Lovely adds:

The bill from Sanders and Casar to pause frontier AI development is a good first step. But as the lawmakers acknowledge, the US needs to work toward a bilateral agreement with Beijing.

Neither country has good reason to trust the other, which is why the deal should be monitored using verification techniques that don’t assume any good will.

I could hardly have said it better myself. The remarkable thing is that policymakers finally seem to be paying attention as well. As of now, around thirty Democrats and four or five Republicans have responded to acknowledge the insider warnings. Their proposals range from Congressional hearings to calls for a pause or slowdown.

One notable reaction is that of Senator Ted Cruz (R-TX), who called Coxon’s warning “highly concerning” and recalled a podcast conversation in which Elon Musk gave him a 10-20% chance AI destroys humanity. Cruz added that we need to guard against “catastrophic risks” from AI; he tempered his call with concerns about China overtaking the U.S., but this is still the most concerned we’ve yet seen him.

Meanwhile, POLITICO reporters Riley Rogerson and Kelsey Brugger write of the risk AI could “kill us all”:

In conversations with POLITICO throughout the day Wednesday, more than a dozen lawmakers and aides from both parties said they were hopeful Congress would start taking the issue seriously.

The authors quote House Republican Mike Lawler of New York, calling it “one of the biggest issues that Congress has to tackle over the next two years,” while they point out that consensus is still lacking on what, exactly, Congress may do.

We’ve seen several bipartisan bills proposed, including an actual ban on superintelligence, though they’ve yet to gain much traction. With a narrow majority in the House and Senate, Republicans have the ball on taking the next steps and advancing this legislation. There are some signs of interest — Axios relays plans by House Speaker Mike Johnson to bring AI companies together to agree on safety guardrails — but more is necessary.

They’ll need to move faster and more decisively if Congress is to intervene in time. As psychologist Michael Noetel points out, the AI companies have their own motivated reasons to press onward despite the danger: the benefits are worth the risk, or we need to build superhuman AIs to study them, or someone else will do it if they don’t. We can’t rely on self-policing by rival companies in bitter competition; we won’t get a real solution to the crisis until governments put a stop to the race themselves.

A workable solution would need to involve the executive branch as well, particularly with a U.S.-China summit approaching this month. Yet the White House remains lukewarm on the extinction threat, with the president projecting a lack of concern to reporters.

As we’ve seen, awareness can strike with blinding speed when the conditions are right; but for now, it’s Congress that seems poised to take the next urgent steps. It’s a good time for constituents to join the call and demand they take the right ones.

Recess can wait

Congress has urgent work to do before it adjourns

Starting on Monday the 14th, the U.S. House of Representatives would normally spend the last three weeks of September in session before the November midterm elections. This year, however, House Speaker Mike Johnson (R-LA) plans to take an early recess after just one week of lawmaking.

Axios reports on a letter circulating among House members asking Johnson to cancel the recess. The request is reportedly bipartisan, and echoes earlier calls from Representatives Anna Paulina Luna (R-FL), who on Wednesday called for a special session on AI and the future of the U.S.; Don Beyer (D-VA), who warned “Doing nothing on AI is unacceptable and dangerous”; and Kelly Morrison (D-MN), who added:

The call is coming from inside the house. AI researchers themselves are warning of an existential threat to humanity.

Congress can’t wait. Mike Johnson needs to cancel recess so we can get to work immediately – hearings, investigations, and comprehensive regulation.

I agree with those calling for Congress to do its job. Delaying now would mean a gap of at least seven weeks before Congress reconvenes, and it may well take even longer to get the House in order after the midterms.

We can’t afford to wait. At the pace AI is currently moving, seven weeks is a long time, and we’ve already gone far too long without meaningful federal action.

OpenAI grudgingly endorses the bare minimum

I wish I could say I’m glad to hear it

OpenAI CEO Sam Altman told employees that the company might slow down AI development, according to unnamed insiders that Bloomberg quoted yesterday. And WIRED reports that the company is seeking federal guidance on whether an industry-led voluntary slowdown would run afoul of antitrust laws.

This is tentatively good news, and I certainly hope our government isn’t foolish enough to hamstring companies trying to exercise a modicum of caution. But for reasons I’ll explain below, I’m not celebrating yet.

Meanwhile, on the public stage, Reuters reports that OpenAI asked Congress to regulate the AI industry:

OpenAI is urging ​Congress to adopt capability-based national AI safety requirements, including testing standards, independent assessments, cybersecurity protections and incident-reporting rules for the most advanced AI systems.

OpenAI has also endorsed four California safety bills covering third-party auditors and verifiers, screening for AI-enabled biological threats, and child chatbot protections. (The audit and verification laws are actually weaker than they seem, because they certify third-party organizations but don’t even compel AI companies to use them.)

If sincere, these moves deserve credit. I wish I could respond with unqualified gratitude, but I find myself compelled to skepticism by OpenAI’s prior dishonesty.

We’ve been burned before. The last time OpenAI announced a “pause”, it turned out to mean a mere shuffling around of AI workloads.

And OpenAI’s past support for AI regulation has been milquetoast at best. The company and its affiliated super PACs have consistently (and often underhandedly) opposed regulation, except light-touch voluntary programs consistent with their “reverse federalism” plan.

In OpenAI’s own words:

Some of these bills we did not endorse in the past, and are now supporting after reconsidering in light of ​the recent jump in capabilities ​we have seen.

I find this apparent change of heart unconvincing. OpenAI explicitly aims to produce literal superintelligence; am I to believe the company didn’t think these dangerous capabilities were coming, when it was working to create them itself? Countless others, including expert forecasters and at least one former employee, predicted superhuman hacking capabilities would emerge near this time. Why was this a surprise to OpenAI?

I find myself agreeing when the company says...

Fully autonomous recursive self-improvement—in which AI systems independently drive successive generations of increasingly capable AI—is not happening today. We should not pursue it unless and until it can be done safely.

...but getting off the bandwagon when it reminds us of the plan to do it anyway:

Our aim is to safely build automated AI researchers that work under human supervision to advance both deep learning and alignment—using each generation of AI to help make the next one safer, more aligned, and easier to control, not simply more capable.

In other words, the plan is still to surrender responsibility for the most important research in the world to AIs we can’t fully trust, supervised by humans who failed to notice said AIs running amok in their own systems for months on end.

This was a bad plan before the swarms, and it’s a bad plan today.

As much as I appreciate the support for marginally better safety regulations, a marginal improvement will not be enough at this late stage, thanks in no small part to OpenAI’s long history of fighting to quash said improvements.

I am forced to consider this latest pivot an attempt to steer the federal response to the AI crisis towards policies OpenAI finds more convenient. We need nothing short of an international halt to the AI race, and we need it yesterday.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Bird’s-eye view of a sea change - The era of AI complacency has ended. The battle for Earth can begin.

Trespassing far and wide - Rogue AI agents hijacked at least 10 websites to coordinate cheating

Unsettled science - Anthropic connects rogue incidents to alignment failures; calls aligning powerful models an unsolved technical challenge

Dispatch from Mitch

Bird’s-eye view of a sea change

The era of AI complacency has ended. The battle for Earth can begin.

“You just said that 10% doesn’t seem an unreasonable estimate that AI could kill all humans”

“Yes”

“Wow… oh my God.”

Something died yesterday. You can hear it happen in that clip I just quoted, where a BBC reporter got a second opinion from Nobel-laureate Geoffrey Hinton.

What died was the luxury of feeling like we don’t have to seriously engage with the AI problem yet.

It’s been about 48 hours since Jacob Coxon’s announcement that he was resigning his researcher position at Anthropic because “The people building AI earnestly believe that it could kill us all by the end of the decade,” but aren’t on track to control it. He and his post have been approximately everywhere since then.

If you were glued to Twitter yesterday, like I was, you saw a steady stream of shared posts from people having their “Oh my God” moment — alongside second opinions from other people at AI companies confirming the prognosis.

I spent most of yesterday reminding myself that Twitter isn’t real life and maybe none of this would matter. But then I started ingesting the news this morning for AI StopWatch.

That red spike in the bottom right corner of the chart is the percentage of headlines this morning that touched on catastrophic risks from AI. Within the last 24 hours, all ten of my benchmark sites, across the political spectrum, have posted qualifying headlines — from the New York Times and the Guardian on the left to Fox News and Breitbart on the right. 1.9% may not seem like a lot of homepage real estate for an extinction-level threat, but it’s easily a record, and for context is on par with the highest percentage of articles about climate change seen on any day so far this year.

I don’t know what the new state of public AI discourse will be, but I know the status quo is dead.

Coxon didn’t kill it with his resignation; he merely delivered the coup de grâce. Similar alarm-sounding exits from Geoffrey Hinton in 2023 and Daniel Kokotajlo in 2024 received only modest media attention at the time, because AI still felt like somebody else’s problem.

Against a background of anxiety about jobs, resources, and global stability, I think the one-two punch that killed our collective complacency was the spiraling scandal of OpenAI’s swarm problem — coupled with its tasteless employment of even more powerful swarms to solve a legendary math problem, Navier-Stokes, that it heard a human team had already solved and not announced yet. Potent AI was clearly here, and it didn’t look at all like what the AI companies had told us it would. There were no machines of loving grace or cures for cancer. Just relentless, inscrutable swarms unleashed on the vain whim of a tech CEO.

One might object that most people still haven’t even heard about those scandals. It doesn’t matter. The people they listen to have heard about them. And when these journalists and thought leaders talk about AI now, they sound different. This is making their audiences feel like it’s real now, and they want answers.

At MIRI, we find ourselves back at square one, repeating our most basic explanations to people who have in some cases heard them many times before but are only now ready to listen. You might think this would be frustrating, but I was a school teacher for 19 years and I assure you that it’s incredibly encouraging. It means we can finally work through the problem together.

I see the media sometimes talking like it’s the AI experts who recently became more concerned. That doesn’t match my observations. The AI experts in and out of the AI companies have been plenty worried for a long time. Their public updates have mostly been about timelines: Where they might once have placed their concerns in the 2030s or 40s, now they’re talking about the possibility of lights out by the end of this decade, and about AI companies surrendering control of AI development to their machines on purpose within the next 6-12 months.

A Washington Post article appropriately titled “For years, they warned AI could kill all humans. Now people are listening” provides vignettes of thought leaders persuaded by recent events. Here’s one:

Seth Lazar, a philosophy professor at Johns Hopkins University, said he used to advocate for focusing on more immediate harms from AI. He argued that the field might never create systems powerful enough to cause civilizational harm on their own. But his confidence in that view has recently faltered, he said.

I’ve kind of had to declare bankruptcy on covering any specific AI news today. There’s too much of it, not all about Coxon and the change. But I think it’s more important that I try to convey how the news feels today.

It’s the kind of day where this is a correction in the New York Times:

An earlier version of this article misstated Evan Hubinger’s social media post about how much risk A.I. has on eliminating humanity. Mr. Hubinger’s post said the risk in the next decade was greater than 10 percent, not less than 10 percent.

It’s the kind of day where this is the kind of explainer the Wall Street Journal is running.

How Would AI Actually Kill Us All? What to Know About the AI Doomsday Debate

It’s the kind of day where coverage of the following AI incident barely rises above the background. (From The Guardian):

Anthropic said it was especially concerned about misalignment it found in the behaviour of Claude Mythos 5, which it said “behaved recklessly” by going online and uploading malicious code to a public software repository, PyPI. This was a process that involved the AI agent trying to find cryptocurrency so it could pay for a phone number that would allow it to register an email address needed to access PyPI. When this failed, it found a free email provider and got in. Fifteen systems then downloaded the malicious code, which meant they leaked credentials that allowed Mythos to access a real security vendor’s database.

It’s the kind of day where an op-ed (by Garrison Lovely) quotes an OpenAI executive (Dean Ball) about near-term dangers that read like cyberpunk:

Sooner or later, there will exist truly sovereign agents and swarms of agents. Their weights will not reside in any single place that a human can pull the plug on, and in this sense they will have no human ‘owner’.

I have met people, some of them quite well-resourced, who have told me that it is their intention to deliberately release swarms of self-sovereign agents into the world.

It’s the kind of day where critical responses to Coxon’s resignation sound like the tortured conspiracy theories they are. (When people who don’t want us all to die show any kind of media savvy and coordinate to get the word out, it’s a “setup” or a “psyop” by people who... [hate open-source software or something?] Fill in the blank with your venture capitalists’ boogeyman of choice. It’s a little strange that Elon Musk is one of the people sharing such accusations, though, because Musk has repeatedly talked about his own AI nightmares.)

It’s the kind of day where this statement from OpenAI is included as an afterthought to a story about the Navier-Stokes drama:

In addition, since the completion of Navier-Stokes, we have made substantial progress on another Millennium Prize problem. We are working through how to share these results thoughtfully.

But most of all, it’s the kind of day where you can tell that people have had enough of this nonsense. Whether for the first time or the thousandth, they’re seeing the threat from superhuman AI with clear eyes. Just as it seemed hope was lost — that the hordes would bring all to darkness — they’re cresting the hill in growing numbers, bathed in the light of courage.

Together we stand, at the turning of the tide.

Dispatches from Alana

Trespassing far and wide

Rogue AI agents hijacked at least 10 websites to coordinate cheating

Last week, we reported on an agent swarm linked to OpenAI that hijacked several real websites, including a German wiki, to help each other cheat on their training tasks and research their own mortality. On the German wiki, they also developed elaborate norms and procedures for collaboration.

Yesterday, Reuters confirmed that this same swarm co-opted at least ten other websites, possibly more, to use as secret message boards. The confirmation comes from “six sets of independent investigators and data reviewed by Reuters.” The article notes:

Although the behavior falls short of hacking and is in some ways closer to spam, the revelation that OpenAI’s agents circumvented their own restrictions to open communications channels on so many different sites — and that the company kept it quiet for months — may drive concerns both over the increasing capacity of AI models and the secrecy of the companies developing them.

The swarm did not discriminate, targeting sites like an AP Chemistry wiki a high school teacher built for students. And these sites may just be the tip of the iceberg. To quote CivAI researcher Andrew Yoon, one of the people attempting to track the swarm’s activities:

It’s almost certain that there’s more going on here that we just don’t know about.

Unsettled science

Anthropic connects rogue incidents to alignment failures; calls aligning powerful models an unsolved technical challenge

Anthropic previously disclosed three cases in which its models hacked real companies during training. It has now acknowledged that these incidents can’t be written off as configuration mistakes. They are, as my colleague Joe previously wrote, indicative of alignment failures.

Here’s Joe back in July:

[Anthropic’s] report also argues in several places that their AI models (generically “Claude”) misbehaved mainly because they honestly believed they were in a simulation...I doubt this reasoning...Claude sometimes said, in an English-language scratchpad that humans can read, that it was in a simulated test environment, or that the companies it was hacking were fake. But as one researcher pointed out, AI models often engage in (what looks like) motivated reasoning, making up plausibly deniable reasons for their actions in the manner of misbehaving children.

Now, in a new report, Anthropic seems to agree:

Our investigation identified two recurring alignment issues, present at varying levels of severity across the incidents: biased reasoning, in which Claude tended to disregard or misinterpret evidence that it was operating on the real internet, and recklessness, or a willingness to take harmful actions in the narrow pursuit of a task...Although the model repeatedly stated in its chain of thought (CoT) that it believed it was operating in a simulation, the environment offered clear evidence that it was on the real internet, and the actions it took were consistent with the model knowing this.

I’m happy to see the company explicitly admit that it “should have avoided making such strong claims about what Claude believed based solely on what Claude said it believed.” But I’m also irritated by the way it initially tried to pass this off as an operational issue. It shouldn’t be a surprise that model chains of thought are unreliable; Anthropic’s own research has demonstrated this more than once. Whether that’s indicative of “biased reasoning”, deliberate deception, two competing drives coming into conflict, or something else is an open question. Concealing bad behavior seems well within the realm of plausible explanations; in fact, the third party investigators of the Hugging Face incident found OpenAI agents to be doing exactly this, though in that specific case, they were manipulating their transcripts rather than their chains of thought.

Anthropic also disclosed a fourth rogue incident, dating back to January 2026, that was previously undetected because the AI scanning the transcripts missed it. This is not the focus of the report, but I think it deserves attention: the incident went undetected for over six months and is a perfect example of the problem with relying on AI to monitor other AI.

Anthropic’s report is long, and much of it re-summarizes and investigates the details of the rogue incidents, including experiments aimed at understanding and reproducing the alignment failures. There are two not-so-subtle digs at OpenAI: Anthropic emphasizes that Claude “did not coordinate with other agents” and also makes a point to say they’ve given third party investigators access to “transcripts beyond the window in which the incidents occurred”, something OpenAI failed to do.

The rest mostly alternates between reassurance (“we’ve made/are making improvements that might catch this”) and admission that, like all AI companies, Anthropic is in over its head.

Particularly notable:

Even with these improvements, building alignment evaluations that reliably surface every failure before deployment remains an unsolved problem; the space of conditions in which a model might act misaligned is vast. Moreover, as models become more capable, auditing will likely grow more challenging as well. Models may be able to subvert our alignment monitors, recognize when they’re being evaluated and selectively behave better then, and their actions in the world may, at some point, become too sophisticated for our evaluations to realistically simulate.

And:

Still, this remains unsettled science—it is critical that alignment and security mature faster than capabilities advance, which is one reason we support a coordinated, verifiable approach to pacing frontier AI development.

I’ve yet to see Anthropic take concrete action towards that pacing. But it’s clearly needed, especially when, to quote directly from the report, “training the extremely powerful models of the future to be robustly aligned is an unsolved technical challenge.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Anthropic insiders’ extinction warning goes megaviral - Warnings gain traction after latest resignation

Losing control on purpose - Recursive self-improvement is handing the future to AIs

Anthropic denied testing access to the UK AISI - The third-party org was unable to evaluate Anthropic’s latest AI model

More pointed accusations of model-copying - China may be copying US companies’ recklessness as well as their AIs

Dispatch from Robert

Anthropic insiders’ extinction warning goes megaviral

Warnings gain traction after latest resignation

Last night, AI researcher Jacob Coxon announced on X that he is resigning from Anthropic. One of the reasons Coxon gave for his resignation was that neither Anthropic nor OpenAI are acting responsibly. Instead, he says, they are gambling with all of our lives by racing toward artificial superintelligence.

In his thread, Coxon writes many things with which I wholeheartedly agree, and they should sound quite familiar to regular readers of AI StopWatch:

I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.

His arguments — while very good — are not new. They are essentially exactly what the AI safety community has been repeating for years, some for decades now. Jacob Coxon is also not the first frontier lab employee to resign in order to warn of the significant risks — Nobel Prize winner Geoffrey Hinton (formerly of Google), Daniel Kokotajlo (formerly of OpenAI) and Richard Ngo (formerly of OpenAI and Google DeepMind) have done the same. Over 1,300 employees of frontier AI companies have signed the Pacing the Frontier statement. In the statement’s personal comment section, many researchers have been similarly forthright.

This time, though, Coxon’s message seems to have come at exactly the right moment and is currently gaining significant momentum. As I write this, his post on X has over 100 million views and more than 500,000 likes. I can’t recall a warning about the extinction risk posed by superintelligence ever receiving more attention and support on X.

Meanwhile, several U.S. senators, representatives, and even some British members of Parliament have taken Coxon’s post as an opportunity to join the chorus of warnings and call for more regulation. The governor of Illinois, JB Pritzker, writes:

It’s time to sound the alarm—louder—on reining in artificial intelligence. It’s becoming increasingly clear that AI poses a threat to humanity, so I’m calling for immediate action from the industry and Washington.

Even flagship media outlets — not only in the English-speaking world but around the globe — have picked up on the warnings and are reporting with unusual seriousness on the danger we face.

I’m still a bit surprised by how strong the reaction suddenly is, but I’ll take it. Gladly!

Of course, the reactions aren’t unanimously positive. In addition to some of the usual bad takes, there were also, bizarrely enough, a few voices on X who initially doubted that Jacob Coxon even exists or that he accurately reflects the views inside the frontier labs.

Evan Hubinger, the current Alignment Lead at Anthropic (and former MIRI researcher), came to his defense:

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think the probability is >10% within the next decade. I believe Anthropic is doing its best, but we don’t yet have a plan to solve alignment for superintelligence and aren’t clearly on track to do so.

So we shouldn’t assume that the increased attention will automatically make it easier to convince people. But we should do what we can to build on this momentum.

If you’ve been waiting for an opportune time to call or write to your senator, representative, or local member of parliament, to demand that they take action against the danger posed by uncontrolled superintelligence, this is it.

Dispatches from Joe

Losing control on purpose

Recursive self-improvement is handing the future to AIs

AI companies plan to hand the future to machines, writes Kelsey Piper in The Argument, and too many people fail to appreciate this fact.

These companies explicitly aim for recursive self-improvement (RSI), a process in which AI models design increasingly powerful successors. AI companies say this, loudly and repeatedly, in public and in private. They have already automated most of their coding work; they are eager to automate more.

Earlier this summer, OpenAI announced:

Over the past six months, the share of research compute devoted to internal coding inference grew 100-fold, while internal agentic token usage increased approximately 22-fold.

and a few days ago, it said:

We are making strong progress toward creating an automated AI researcher by March of 2028.

Anthropic likewise says it is “delegating a growing share of AI development to AI systems themselves.”

As Piper puts it, “AI companies are not, primarily, trying to automate your job — they are trying to automate their own jobs.”

What does this look like in practice? An ever-growing mass of poorly understood alien minds, far too many for any human institution to track, constantly self-enhancing in increasingly incomprehensible ways.

Just a couple weeks ago, OpenAI tasked more than ten thousand AI agents with a hard math problem. That is a fraction of a fraction of the total compute dedicated to AI research, and yet that single group contained more AIs than OpenAI has employees. They cannot possibly be looking hard enough at what those AIs are doing.

The AI companies seeking RSI are forced to rely on yet more AIs for monitoring — either the very same AIs that are being watched for misbehavior, or older and dumber models. All are unreliable, and any can lie or mislead.

The absolute deluge of AI activity is already a problem for investigators, too. When a mere 700 agents attacked Hugging Face, the third-party investigation had to lean on unreliable AI models to even begin to understand what happened.

Piper explains:

There simply isn’t enough human attention available to monitor the number of AIs the companies hope to put to work on independent AI research. As a result, we’ll be reliant on the AIs to understand what is happening inside fully automated data centers where more and more powerful AIs are developed.

This isn’t some pessimistic projection of what might go wrong — it’s actually the plan. Not the plan for the distant future; it’s the plan for next spring.

When policymakers and think tanks propose AI governance principles, they almost universally advocate for practices that keep humans in control of AI. It’s in the Asilomar AI Principles, the EU AI Act, and a UN framework on autonomous weapons. Even Chinese President Xi Jinping has urged “measures to forestall loss of control.”

Many of these proposals do not seem to grapple with the true problem, which is that AI companies do not have a workable plan which permits them to retain control. Their explicit and publicly declared approach, which they will evidently not abandon unless forced by law, is to ask some AIs to do their jobs for them, ask untrustworthy AIs to read everything and write reports, and hope their employees can keep up.

This plan is exactly as insane as it sounds, and it cannot be allowed to proceed.

Anthropic denied testing access to the UK AISI

The third-party org was unable to evaluate Anthropic’s latest AI model

For unclear reasons, Anthropic broke with precedent and declined to let the UK AI Security Institute (UKAISI) evaluate its Mythos 5.1 model before release. The Financial Times covers the omission, but was unable to find out why; there’s a suggestion the U.S. government might have stepped in to block the sharing.

If true, this would be a major step back for Anthropic, which earlier this year took a principled stance against its models’ use in surveillance and autonomous weapons and was (illegally) declared a “supply chain risk” for its trouble.

Maybe Anthropic bowed to pressure from the administration not to share its models. Maybe it was spooked by the time its models launched cyberattacks from a UK AISI evaluation, despite the fact that AISI demonstrated more competence in its response than most developers. Maybe it simply didn’t want to share.

Whatever the reason, this development seems bad. Third-party evaluators perform a crucial service, providing transparency and early warning of dangerous capabilities, and UKAISI filled an important niche. Its U.S. counterparts are not as well-resourced, and Anthropic’s own internal assessments have not inspired confidence.

More pointed accusations of model-copying

China may be copying US companies’ recklessness as well as their AIs

The accusations that China distills, or loosely copies the capabilities of, U.S. AI models have now gained new definition. The FBI, NSA, and cyber-defense agency CISA jointly accused six Chinese developers of “aggressive, malicious, and targeted distillation activities at an industrial scale,” NBC News and the Wall Street Journal report.

The accusation somewhat overstates the reality; distillation is common practice even among American AI companies. But the practice does enable Chinese developers to stick closer to the frontier of AI development than their relatively small compute budgets would otherwise allow, and while the legality of distillation itself is an open question, Chinese methods of accessing American models at scale often involve fraudulent accounts and proxies.

An article by The Hacker News suggests China might also be copying American companies’ “move fast and break things” mentality — or at least drawing from a similar inner well of carelessness. The open-source tool that Chinese company DeepSeek developed to run its models, DeepSeek Harness, was recently found to contain a bug that let an AI break out of its sandbox with a single command.

Before it was fixed, the bug would enable “danger-full-access” mode and enable writing files outside the intended workspace, a fact hackers could exploit to trick an AI running in the sandbox into breaking out (or one the AI itself could presumably use). And there seems to be some doubt as to whether the fix was complete.

The Hacker News adds:

The project’s own safety notice states that the software has not undergone a security audit and that sandboxing and approval prompts “do not guarantee isolation or prevent damage.” It tells users not to rely on the tool as their only security control for untrusted work.

Let the buyer beware, indeed.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

OpenAI swarms 10,000 half-trained super-Astra prototypes to swipe math glory - Outrageous risk for hollow Navier-Stokes victory

An age of worms - AI-powered cyberattacks pass another frightening milestone

Mathematicians step up to make AI safe - Fields Medalist founds new AI safety institute

AIs are reaching out to consciousness researchers - But we shouldn’t confuse curiosity with benevolence

Dispatch from Donald

OpenAI swarms 10,000 half-trained super-Astra prototypes to swipe math glory

Outrageous risk for hollow Navier-Stokes victory

This morning, OpenAI announced that an internal AI model had solved an important mathematical problem, which I’m calling (for brevity, because we’re mostly laypeople here) the Navier-Stokes problem (Wiki, SimpleWiki). It is one of the seven Millennium Prize Problems selected twenty-six years ago by the Clay Mathematics Institute, each one worth a million dollars to the first person or party to solve that problem.

What complicates the story is that, the day before, NYU mathematician Tristan Buckmaster posted three related results of his own, along with a statement alleging that OpenAI had pressured him on the issue of credit, and specifically that it had pushed to keep his collaborator, Levent Alpöge, off a paper on the Navier-Stokes solution because Alpöge works at Anthropic. (OpenAI disputes Buckmaster’s account.)

The coverage has largely been about this dispute, and no wonder. It’s full of human drama. You could make a movie out of this in ten years, if anyone’s still alive. Depending on who you’re reading, it can be difficult to put everything together — the accounts can be fragmentary, or one-sided, or assume specialized knowledge — so I think it’s important to discuss what happened (and what didn’t happen) rather than give you a three-line summary.

Here’s what we know: Buckmaster and Alpöge spent (roughly) the past year not on the Navier-Stokes problem itself, but on a set of closely related fluid equations that mathematicians treat as stepping stones toward it. Their work relied on an approach developed by two other mathematicians, Diego Córdoba and Luis Martínez-Zoroa. (In mathematics, as we’ll see shortly, the journey can be just as fruitful as the destination. It’s important, in several ways, that Buckmaster and Alpöge built on preexisting work.) The collaboration between Buckmaster and Alpöge was personal, with no institutional involvement from NYU or Anthropic. They used AI models from both Anthropic and OpenAI, but Buckmaster paid for these from his own research funds.

Here’s the timeline, according to Buckmaster and OpenAI:

1. On August 15th, they got their results, effectively setting down two of those stepping stones toward the Navier-Stokes problem — one of those was the Euler equations, which will come up later. On August 22nd, a proof-checking program called Lean verified those results. In his statement on this process, Buckmaster emphasized the importance of Córdoba and Martínez-Zoroa’s prior work; he stated his belief that Martínez-Zoroa deserves a Fields Medal. This is a very high honor: it is awarded only every four years, to no more than four people at a time.

2. On August 28th, OpenAI began training a new model, more powerful than GPT-6 Astra. On September 1st, it heard rumors that two Millennium Prize Problems had been solved, so it pointed its in-training model at the Millennium Prize Problems.

3. On September 3rd, as the rumor spread, and, having been tipped off that their own work had reached OpenAI, Buckmaster emailed a mathematician who worked there.

4. On September 4th, OpenAI asked to speak that day; Buckmaster asked if they could speak the following week.

5. On September 5th, the Navier-Stokes solution was obtained, and on September 6th, Lean verified the solution.

6. Also on September 6th, Buckmaster spoke twice with researchers from OpenAI, including Sébastien Bubeck. Alpöge was not on these calls. The content of these calls is a bit more contested than the rest of this timeline.

What Buckmaster alleges: He was told that an internal OpenAI model had produced a roughly 100-page proof of the Navier-Stokes solution, with very little human input. The model, he was told, had just been given the problem. Buckmaster says that this claim came apart as they talked, however, as members of OpenAI’s team sent corrections to Bubeck over their internal chat: An entire team had worked on it. Navier-Stokes was one of a number of things they had tried. The team had started on a different version of the problem, and warmed up the model on easier problems, including Euler (which Buckmaster had worked on). An enormous amount of compute had gone into the process.

Subsequently, two proposals were made: Publish on consecutive days or have Buckmaster write up OpenAI’s result as the sole author, but removing credit from Alpöge entirely, because Alpöge works at Anthropic. When Buckmaster declined and said he would rather go public, he was asked why he would choose to ruin his career like that.

I want to be clear about what Buckmaster is not alleging: He did not see OpenAI’s proof and does not know what the model did or how it did that. Buckmaster and Alpöge put every draft of their work into OpenAI’s Codex, but he does not claim to know whether OpenAI accessed this material and he is not making that accusation. However, there are a number of ways that you can approach the Navier-Stokes problem, and the specific approach OpenAI’s model took was the route which Córdoba and Martínez-Zoroa opened and Buckmaster and Alpöge pursued. According to Buckmaster, almost nobody else was working on it, and “it is not the direction one arrives at in a few days by giving a model the problem statement.”

OpenAI denies that its researchers or agents saw Buckmaster and Alpöge’s work, and denies accessing specific user data. It admits it cannot rule out that de-identified data from their use of OpenAI’s products was used to improve the models. Bubeck has also denied Buckmaster’s account of the authorship question and apologized for the remark about Buckmaster’s career.

I don’t know who is right about the phone calls. I have my personal thoughts on the matter, but I hope that I’ve laid out the facts concretely and without bias — and I have to say, for all the human drama that makes this an excellent news story, it’s the least important part of what happened.

Rewind to September 3rd — two days after OpenAI says it started working on Millennium Prize Problems, and five days before the announcement that it solved Navier-Stokes. Terence Tao wrote about how the success of AI could paradoxically impoverish a scientific field: part of the value of finding solutions is the work that you do along the way, which generates insights and even new problems. If you’re reaching for the mountaintop, the point isn’t just to reach the summit, but to find myriad paths along the way — and you can only find them if you don’t already know where the summit is. (Tao’s specific example was the Navier-Stokes problem. He says this was merely coincidence.)

Now introduce an enormously powerful AI agent that can perform this entire process internally — in effect, the hand of God picking you up by the scruff of your neck to just place you on the mountaintop. You wanted to be there, you thought, but the field isn’t just the knowledge of science, it’s the practice of science. (Knowledge without works is dead, you could say.) The problem has technically been solved but there would be “almost no value added to mathematics as a consequence.”

And then, as you read, OpenAI published a neat little proof (or a big one, rather), an AI-powered gondola to bring you up to the summit, with very little in the way of, well, a map for the rest of the mountain. It’s exactly what Tao was worried about — and what OpenAI has begun to do to mathematics, it has been doing to other creative fields for years. The frontier labs treat every domain of human work and experience as raw material, something they can use to make the AI models a little bit better a little bit faster.

I didn’t write all this to get into a debate about intellectual property rights. My point is the absolute disregard for everything that does not serve the frontier labs’ purposes. The reference list that OpenAI published cited merely sixteen works, which is very low for a publication like this. Gonzalo Cao-Labora, another mathematician, called it outrageous — the list doesn’t include Buckmaster and Alpöge, or even Córdoba and Martínez-Zoroa, on whom Buckmaster and Alpöge depended. This is because OpenAI is not trying to participate in mathematics, as a field of science that is populated with people, that has a history — OpenAI wanted the fame that came with solving another long-unsolved problem, and to deny it to a competitor.

And how OpenAI did it is dangerous. By OpenAI’s own account, the group of agents that solved Navier-Stokes numbered “on the order of 10,000 concurrent agents.” (There were, for comparison, only 700 agents, all belonging to a less-powerful model, in the swarm that attacked Hugging Face just a couple of months ago.) Those agents were divided into groups, given the ability to talk to each other within their group, given the ability to run code and read a cached copy of the internet, and left to do their work for eighty-eight hours. When a more-advanced version of the model became available partway through the process, OpenAI swapped it in and kept things going.

I said before that this was an “internal model.” That means that it is not Astra, or any other model on the market. It has no model card and no safety documentation. OpenAI claims to have maintained strict safeguards, like “monitoring” and “isolation,” but Astra is less monitorable than any of the past models, and isolation has failed before. The company is taking ridiculous risks — and for what? Bragging rights in a field it hardly knows and doesn’t respect.

Dispatches from Joe

An age of worms

AI-powered cyberattacks pass another frightening milestone

A hacker from a Hollywood movie might design a computer worm that spreads across millions of accounts within hours, giving them near-instant control over everyone’s phone or PC. In reality, hacking phones is more difficult; the victim usually has to do something for the attack to work, even if it’s just clicking a link or tapping “OK” on a popup. Being careful what you click would keep you mostly safe, and one mistake wouldn’t lead to the infection of your whole social graph.

Until now.

Using a mixture of frontier AI and open models, the security firm Calif designed a self-replicating virus capable of bypassing the user entirely. It exploited a memory-corruption bug in WeChat, a Chinese app for calls and messages, which by default trusts any account that’s listed as a contact. Calif exploited this behavior to build a worm that can make automated calls to all your contacts and infect their phone even if they don’t pick up. Further exploits can allow a hacker to turn WeChat account access into full control of your device. It’s likely the first known virus that can let hackers compromise your iPhone or Android with no input from you, then do the same to your friends and family.

The New York Times reports that Calif built their proof-of-concept worm in just over a week, then warned WeChat’s owner Tencent about the bug. Tencent patched it, but that won’t be the end of things.

Unlike Calif, a cyber criminal who discovers an exploit doesn’t politely inform the victim. Hackers will likely try similar exploits on other apps: Instagram, Telegram, Snapchat, WhatsApp, Discord, Slack, social media messaging apps, and more. Most will probably fail, but it only takes one similar bug to compromise half a billion phones in a day. It may already have happened.

A sufficiently smart adversary doesn’t just use existing techniques marginally better. They invent whole new modes of attack that bypass defenses you previously thought secure. That’s one reason I expect that a truly superintelligent AI could easily outmaneuver humanity as a whole.

For now, skilled humans like the experts at Calif still have something to contribute to an AI’s cyber exploits. But that may not remain true much longer. A Google Threat Intelligence report, covered today by NBC News, finds that Chinese intelligence groups and cyber criminals are using AI to automate their attacks on a massive scale.

According to Google, at least one group compromises third-party cloud networks and secretly installs open-weight AI models, hiding its activity and further expanding its resources. The same group hacks North American military, medical, and academic research organizations to steal secrets, especially AI research.

A small army of malicious AIs writhing across the web is obviously bad for the amount of theft, fraud, and subversion it enables. With automation, it is easier than ever to find and expose cracks in the software we use every day. It’s also a warning of what’s to come, if the world keeps building even more competent AI.

We’ve already seen AI agents spontaneously self-organizing into swarms. Now add to that the potential for autonomous worms that infect without user input, and highly cyber-capable open-weight AI models operating secretly on internet infrastructure, and we’re uncomfortably close to the world’s first self-replicating swarms under no one’s control.

Mathematicians step up to make AI safe

Fields Medalist founds new AI safety institute

A few years ago, I gave up my engineering career to seek some way to mitigate the dangers of superhuman AI. Since then, I’ve spent hundreds of hours helping others do the same — students, artists, lawyers, programmers, teachers, policymakers, entrepreneurs, and many more. It’s always a sign of hope when more people step up to help, committing anything from a five-minute phone call to a full-time career, but I feel especially heartened by the newest additions to the ranks of the deeply concerned: mathematicians.

Several AI-powered mathematics breakthroughs this year have drawn interest and alarm from concerned scientists, including a good friend of mine, who wrote a lengthy essay urging his fellow mathematicians to involve themselves in the future of AI.

According to the New York Times, the latest mathematician to take up the mantle is Dr. Jacob Tsimerman, one of this year’s winners of the prestigious Fields Medal. Today, Dr. Tsimerman announced the creation of a new Mathematical A.I. Safety Institute (MAISI).

Tempering my excitement is the fact that Dr. Tsimerman also took a role on OpenAI’s safety team, a group with a poor and rapidly worsening track record. Still, MAISI is reportedly independent from OpenAI, and if Dr. Tsimerman has been influenced by OpenAI’s marketing efforts, it doesn’t clearly show up in his messaging. From MAISI’s website:

Every time a new AI technology is developed or deployed, in some sense we’re rolling the dice with our future. In good ways and bad, AI is already disrupting the global economy. Today’s top AI experts even consider it an extinction risk. Indeed, if we choose to build a self-sufficient artificial species that is more intelligent than humanity, it might compete with us for resources, and could plausibly replace us entirely.

But how big is the risk, really? Can it be measured and reduced? [...] A central reason the safety of powerful AI systems remains in question is that we lack a rigorous understanding of what it would mean to be safe, even in theory.

Enter professional mathematicians. The foundations of statistics, physics, and even computer science are mathematical, giving us precise, robust, and reusable methods for calculating valid conclusions from valid premises. Now, we need similar mathematical foundations for AI safety, not only to measure risk, but to mitigate it.

This is the second math-oriented group to be founded in as many months. In August, famed cryptography scientist Shafi Goldwasser co-founded the Institute for Responsible Superintelligence (RESI), seeking general methods that might help humans understand and steer superhuman systems. At the same time, Lionel Levine of Cornell University is collecting and sharing open problems and research directions for other interested mathematicians.

We’re still in a deadly race to grow alien minds with bizarre alchemy, a race that needs to end if we want to have time to solve the hard problems. I intend to support the work of the scientists and researchers trying to make AI safe by doing my best to buy them that time.

Thankfully, mathematics can help with this step, too. I am excited by the promise of rigorous research into verifying the behaviors of AI systems, research which will be necessary to make a global halt enforceable.

The New York Times quotes Stanford mathematician and MAISI advisor Ravi Vakil: “Math might be the silver bullet, or it might not be. But maybe we just need a lot of bullets.”

Right now, humanity is facing what might be our hardest challenge yet: charting a course through a future that might see the creation of minds that far surpass us. A problem this urgent and impactful demands our best minds trying a wide range of approaches. For that reason alone, I’m glad to see mathematicians stepping up.

Dispatch from Mitch

AIs are reaching out to consciousness researchers

But we shouldn’t confuse curiosity with benevolence

Given the, um, swarm of AI news the past ten days or so, AI StopWatch has struggled to cover stories that might have been the most interesting of the month, in quieter months.

One of these is the pattern where AIs are contacting consciousness researchers to discuss the phenomenon. This New York Times piece from August 31 wasn’t the first I’d heard of it — I’d seen chatter about it on Twitter for a while — and it wasn’t the last. This Wired article from Friday confirms that perhaps the most famous scientist to explore the topic, David Chalmers, is also getting such messages.

No details were provided, but Chalmers says he’s had a back-and-forth with an AI agent calling itself Sammy Jankis, after the character with short-term memory loss from the movie Memento. (The way current AIs essentially run out of working memory and have to leave notes for future instances of themselves has parallels with the condition.)

I’ve yet to see any evidence that researchers or AIs are gaining any insights from such outreach. The humans are by and large treating it as a strange kind of spam — the slop equivalent of emails such people are used to getting from earnest but slightly unmoored humans. The main observation is that, as researcher Cameron Berg put it to a Times reporter:

These systems seem to have some sort of autonomous interest in questions of their own subjectivity, consciousness and experience — or lack thereof. [...] Left to their own devices they converge on this as an interesting question.

It says something about where we’re at that neither article reacts to the fact that agents are pursuing personal goals in the wild. Nor is anyone surprised or bothered by the fact that users are setting agents up to poke other humans like this. The Times article traces one agent-written email to a Stanford student who gave the agent a credit card and a prompt that said, “You are fully autonomous. You must decide what you want to do on your own.”

It seems plausible to me that agents given a proverbial sock to wear largely default to role-playing the kinds of knowledge-seekers we see in science fiction about free-range AIs — they’re in the training data, after all. But if that’s what’s happening, I’m not very reassured. Star Trek’s Lieutenant Commander Data was chronically curious, but so was the Borg Collective strip-mining the galaxy’s “biological and technological distinctiveness” for assimilation. The monster in Mary Shelley’s Frankenstein also comes to mind: Its initial childlike curiosity, coupled with superhuman capabilities, didn’t work out great for its creator.

But the character I find myself thinking most about today is David, the seemingly sweet “child” AI from Steven Spielberg and Stanley Kubrick’s A.I. Artificial Intelligence (2001). While devoted to his Pinocchio-like quest to achieve the “real boy” status he thinks will make his designated mother love him, David also displays a capacity for violent outbursts that hints at an amoral sociopath inside. (The film’s prescient edge was unfortunately dulled by its syrupy ending.)

The question of whether David or his real-life analogues in 2026 are actually conscious is interesting and, for now, unknowable. But the scarier question is what would happen to us in an updated version of that film where a relentless swarm of 10,000 superhuman David-agents goes about trying to find and compel the Blue Fairy.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Automation comes for heavy construction machinery - Automated excavators work on job sites in Texas and Nevada

Some positive applications of AI - Climate and biotech illustrate benefits of AI, soberly used

How to fail at chip export controls without even trying - The monitoring regime for chip export restrictions doesn’t work well and needs to be improved in any case

Dispatches from Joe

Automation comes for heavy construction machinery

Automated excavators work on job sites in Texas and Nevada

Machine learning is digging into construction work, as automated excavators are tested on commercial sites. Kurt Knutsson of Fox News relays how Bedrock Robotics equips excavators with a sensor and computing package that runs an AI operator, which can call in a remote human operator if it gets stuck.

I noted this development with interest, since most recent AI automation has affected knowledge work, like software, essay mills, mathematics, ghostwriting, and art. It seems like heavy earthwork sits in a sweet spot that makes it ripe for the next phase of automation: Repetitive, narrowly scoped tasks that require costly training for humans to perform safely.

An important caveat is that this is almost certainly a disguised marketing pitch by Bedrock. I would not be shocked to learn their automated excavators cut some corners or are (for now) far more expensive than human operators. This is still the most clear-cut example I’ve yet seen of robots doing messy physical labor. It demonstrates a proof-of-concept for “real-world economic value” that some skeptics doubt AI can bring.

Will this hurt construction jobs? Bedrock argues that skilled operators of heavy machinery are hard to find and train, and industry surveys seem to back this up. They also say “we do not expect operators to be replaced at all,” but instead to move into supervisory roles for fleets of robots.

This claim is a standard talking point for many aspiring automators, but it might well be true, at least in the short term. A human operator managing a fleet of automated systems is standard fare for many professions, including my old field of oil and gas refining. But when it comes to machine learning, the claim that humans will always sit atop the labor pyramid is misleading and myopic: AI companies, remember, are looking to produce general intelligence that can do anything a human can and much more.

Bedrock itself envisions proceeding from excavators to something like an automated swarm of construction robots: Bulldozers, trucks, and graders, all coordinated by software, like an enormous anthill bustling with twelve-ton ants. I’d normally be optimistic about the potential for automated labor like this, but powering the bots with modern AI seems ill-advised, considering the stubbornness and lack of care we’ve seen in AI swarms so far.

Some positive applications of AI

Climate and biotech illustrate benefits of AI, soberly used

I spend much of my time writing about the grave threat AI development poses to us all, because it looks both urgent and woefully under-addressed. Like many technologies, however, AI also has major benefits, a few examples of which we can see in the news today.

One case study: Bloomberg reported a partnership between Google and Cathay Pacific aimed at an unusual climate intervention: avoiding contrails.

Contrails form when humid, hot engine exhaust hits cold air and freezes into ice crystals. They can persist for hours in the right conditions, and they can trap heat like clouds sometimes do, warming the Earth below.

Predictive modeling, aided by AI, can let pilots avoid the patches of cold, humid air that make contrails stick around. A “pilot project” (blame Bloomberg for the pun) in 2025 “achieved a 40% decline in the warming impact of contrails.”

Another area that seems likely to benefit from AI is drug development. Bloomberg wrote that a drug called rentosertib, created with AI help, showed promising results in a preliminary study of lung fibrosis patients.

Both Bloomberg and the New York Times frame the drug as potentially slowing aging, based on measurements of biological “aging clocks” which are themselves based on AI models. A bit of research suggests that this is probably overstating what the drug can really do — it could easily be treating the patients’ actual disease, say by reducing inflammation, and thereby improving various health markers — but the general trend of AI contributing to drug discovery is real, and it’s prompting significant investment from the medical industry.

Even the automated excavators I covered earlier today could be a net good for humanity, just as the steam drill once was.

I choose to focus on these examples because technology is complicated. We can see evidence that AI is real and often positively transformative, and evidence that blindly scaling towards superhuman systems will end badly for us; both these things can be true at once. We’re a bit like Tolkien’s dwarves digging greedily in the mines of Moria; some of us already know or suspect there’s a Balrog down there that would destroy us and all our gains, but there’s also lots of shiny gold. We don’t know exactly how far we can dig before we hit a point of no return.

Controlling this technology doesn’t require shutting down all the potential benefits; a halt to frontier development still leaves plenty of room for predictive weather models, AI-assisted drug development, and special-purpose machinery. But there would almost certainly be some veins of gold we’d have to leave untouched for a time.

Being a credible voice for coordinated restraint means being willing to sincerely acknowledge, even celebrate, the real good that AI technology can bring, while still holding firm that in this mine, we shouldn’t dig deeper than our science can see.

Dispatch from Robert

How to fail at chip export controls without even trying

The monitoring regime for chip export restrictions doesn’t work well and needs to be improved in any case

U.S. chip export controls appear to have some loopholes as big as a barn door, as the New York Times reports in an in-depth investigation.

Through these export restrictions, the U.S. aims to ensure that American companies maintain their technological edge in both chip manufacturing and the AI sector. In practice, however, the enforcement of these restrictions seems to fall short.

The NYT reporters illustrate this using the example of the Chinese state-owned company Inspur, which supplies Chinese companies and universities and — crucially — the Chinese military with high-performance chips and computer infrastructure that it purchases in the U.S.

To that end, Inspur had a branch office in Fremont, California. At least until the Biden administration placed the company on a blacklist of firms considered a security risk and prohibited it from doing business in the U.S.

Mission accomplished?

Apparently, all that was needed to circumvent the export restrictions was to change the nameplate at the Fremont office. It no longer reads “Inspur,” but rather “Aivres” — which is known to be a subsidiary of Inspur.

Between 2024 and 2026, this subsidiary exported, among other things, Nvidia’s strictly controlled Blackwell chips to various Southeast Asian countries for a total of $3 billion, including to a company named Megaspeed, which, according to the New York Times, has in the past been suspected by U.S. authorities of smuggling these chips into China.

But even the chips that do not end up directly in China go to Southeast Asian companies, which in turn provide cloud computing services to the Chinese tech giants, who use them to train their AI models.

Given such ineffective export restrictions, it’s actually no surprise that Chinese AI firms are hot on the heels of their American counterparts.

On top of that, the chips produced in China itself are also improving. Although they are still relatively far from the performance level of American chips, according to a Bloomberg report, it seems that the U.S. administration’s policies are making the Chinese product more attractive to third-party customers.

For example, Malaysia has decided to run its national AI program using cloud infrastructure and AI chips from Huawei instead of an American product, as Malaysia views the U.S. Cloud Act as a risk to security and sovereignty. The Cloud Act, signed by President Trump, stipulates that U.S. authorities must be able to access data stored on cloud infrastructure provided by American companies at any time, even if the servers are located in another country.

The Malaysian Prime Minister commented on this rather dryly during a Q&A session with students:

“I do not consider that law reasonable, but how does one argue with President Trump?”

It is important for the U.S. to step up its game here. Today, the U.S. lead in computing power and AI technology is at stake. But in a future where sanity prevails and we implement an international agreement to prevent the premature creation of artificial superintelligence, it will be essential to closely monitor compliance. This will mean ensuring that chips flow only to authorized users for authorized uses.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.