As promised, to help ease the transition to our more slimmed-down format, here are some links to resources that might scratch your old-school AI StopWatch itch, organized by type of itch.

I’ll add and update links as I think of more things and collect more suggestions from the team.

Accessible explainers

• AGI.FYI is a curated hub of explainers from people we know with good taste. We’ve recommended it before.

• The Rational Animations YouTube channel isn’t always about AI, but is almost always AI relevant. Great art, great narration, great material. I’ve recommended them several times here.

• If you haven’t already read If Anyone Builds It, Everyone Dies, it’s more relevant than ever — to the point where it returned to the NYT Bestseller list and is continuing that run for the second consecutive week as I write this.

Maximally accessible coverage of AI developments

• I’m pre-recommending the Machine Gods podcast launching October 19, from Kevin Roose and Casey Newton. These were the hosts of Hard Fork — the podcast I’ve probably recommended to StopWatch readers more than any other, even though it wasn’t always about AI. Roose and Newton do their homework without making it feel like your homework, and are a highly engaging duo. (I think Hard Fork is set to continue with new hosts, but I don’t know if it will still be any good.)

• Ethan Mollick isn’t so much a provider of AI news as a provider of AI demonstrations, putting the latest models through their paces, comparing them to their rivals and predecessors, helping you understand just how far we’ve come and how fast we’re going.

More in-depth coverage that is only somewhat less accessible

• The AI Explained YouTube channel is a great source for insider news and context, though you might have to wait a week or two between videos. In recent months it has been especially good, accessibly prying into concerning developments and piecing things together.

• Zvi Mowshowitz is the GOAT in this department, covering a truly ungodly amount of AI news — much of it pretty deeply. Somehow, he is only one person. If you don’t resort to skimming, you will struggle to keep up with his output.

Hot scoops

• Andrew Curran has emerged as perhaps the most important rumor and scoop hub in the AI Twitterverse, which is itself the hub of AI insider discourse.

Accessible MIRI perspectives on current events

• Twitter (X.com). As some podcasters I once listened to used to say, “It’s a terrible place, but we’re there.” Many MIRI staff participate in the Twitter discourse, not always in official capacity. Among the more regular and accessible:

  • Nate Soares, MIRI President and co-author of If Anyone Builds It, Everyone Dies

  • Harlan Stewart, MIRI Head of Outreach. Be sure to check out his replies, where he does a lot of his best work with patience and bone-dry wit.

  • Rob Bensinger, MIRI comms legend, never one to waste a teaching opportunity

• Short-form video — There will be a lot of overlap of content between the MIRI channels; pick your channel of choice:

  • MIRI’s Instagram

  • MIRI’s TikTok

  • MIRI’s YouTube Shorts

  • MIRI’s Facebook Reels

  • Nate Soares’s Instagram

  • Nate Soares’s TikTok

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Subscribe now

When we started AI StopWatch back in April, I described the rationale in our welcome email:

When the growing resentments around AI burst into demands for sweeping action, will voters and their representatives know to insist (among their other demands) on preventing the development of superhuman minds that no one knows how to steer or control? They will if you, and I, and our friends, and their friends, have all told them.

But five months into our experiment — three weeks after Jacob Coxon wrote the resignation tweet heard around the world — a lack of awareness is no longer my chief concern. The extinction problem is now so salient in public discourse that it may have actually triggered the avalanche — that surge of demand for action that I always felt was coming.

Even a month ago, I thought the avalanche was further away, because the problems I expected to most motivate it, like job loss, still weren’t very acute.

But Coxon, and those in the industry who immediately and vocally seconded his concerns, successfully and correctly flagged the extinction threat as near term, something that could play out before those other issues would have a chance to inflict too much misery. A recent poll shows voters are now substantially more worried about AI becoming too powerful and taking control than they are about job loss.

My new preoccupation is with the momentum and concentration of the avalanche — with the question of whether the policy response at the bottom will be strong enough and soon enough. Business interests and their political thralls have gone into overdrive trying to laugh off the extinction problem using every rhetorical trick and cynical ploy they can come up with — everything short of making reasoned arguments for why superhuman AIs from the companies that brought us the swarms that hacked the world this summer wouldn’t end up in control of our planet. The U.S. President is saying he wants to leave AI policy “where it is,” and Congressional leaders seem determined to punt on it until next year at the earliest.

So my concern is now less that the public won’t demand action and more that the people responsible for taking that action won’t actually do it — that they’ll wait months for more AI scandals to accumulate, then congratulate themselves for superficial “action” that doesn’t even try to stop the death race.

I want more of MIRI’s resources devoted to that problem, which is playing out more in the halls of power and on the elites’ favorite media channels than in the court of public opinion that AI StopWatch was built to play in. To this end, with support from my higher-ups, I have decided to sunset AI StopWatch as currently constructed.

The main resource this will free up is staffing — the time of everyone on our comms team who helped keep StopWatch running, including me. I’m privileged to work with talented and versatile people.

This pivot was not an easy call. Our readership is dear to us. I want to help you stay informed enough that I’m going to try to keep doing it in shorter form. But even more, I want to take the actions I think have the highest chance of keeping you and everyone you love from dying in a year or two to a wholly preventable AI catastrophe.

Here’s what I expect the next phase of StopWatch to look like — with the caveat that this, too, is an experiment that may need revisiting in a couple of weeks:

1. AI StopWatch will now be a solo or near-solo effort from yours truly.

2. The medium and longer-form Dispatches will be largely discontinued. Instead, I will put out Notes that are essentially just the headlines and microsummaries of the Dispatches we might have written that day, had we continued as before. I will then assemble these into a Daily Digest. Feedback suggests a micro-form feed is all many of our readers ever wanted anyway, and the media analysis I will continue to do for MIRI already gets me most of the way to that product. I think.

3. I will no longer guarantee that the Daily Digest will be daily. I should probably rename it.

4. As the Daily Digest will now be much shorter and less regular, our podcast recordings of it will be discontinued. This is no shade on Tango Batelli, the voice of AI StopWatch, who was fantastic in that role and who we are actively trying to recruit for other work.

5. I am also sad to announce the suspension of our guest post program, but hope to collaborate with these amazing contributors on future projects.

I will start the rebooted AI StopWatch tomorrow, at which time I will also post a short Dispatch with links I would recommend to readers seeking more of what StopWatch has tried to provide.

To anyone parting ways with us at this juncture: I wish you the best. To anyone hanging on to watch me liveblog the apocalypse: I hope to disappoint you with a happy and anticlimactic ending. To everyone working on a good ending by keeping up and speaking out: Thank you, thank you, thank you!

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

In a tweet with over a million views in three days, Google engineer Robert O’Callahan announced he was resigning from his work on chips that would make AI faster and cheaper: “I think AI is already progressing too fast, so I had to quit.”

In a more detailed blog post, he explained:

There are millions of people contributing to AI acceleration and taking my foot off the accelerator will have a very small impact ... but not no impact; some of my skills are rare.

Linking to the book If Anyone Builds It, Everyone Dies, the New Zealander added:

I think the existential risks many people are warning about deserve to be taken seriously; a lot of the phenomena predicted by the “doomers” have come to pass (e.g., reward hacking, misalignment, deceptive models, model eval awareness, psychotic swarms).

The uncertainty he shares with others about this danger is itself “very alarming,” warranting a “massive effort” to minimize risk and back away from building superhuman AI. He’s also very concerned about other AI-related issues like power concentration, cybersecurity, and “cognitive surrender.”

A self-described Presbyterian and “occasional lay preacher,” O’Callahan wrote:

It’s tempting to just turn a blind eye to the impact of my work, but that would not be a Jesus-following thing to do.

He followed up with an explanation of how he thinks about AI against his conviction that “God has a plan that’s good for us.”

I don’t know what that plan is (and wish I did) but it lets me sleep at night in spite of the AI chaos. I expect his plan involves me continuing to make the best use of my talents. Even if the plan is for Jesus to return to rescue us from our folly, we’d better be busy when he returns!

He wants his future work to be “unambiguously pro-human.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

If you read that headline and thought, “Well now it’s just ridiculous,” you’re not alone. The headline as it appeared on Axios was “OpenAI, Anthropic probing tens of thousands of security incidents.” I freaked out a little bit, too.

Looking into it, the paltry reassurance I can give you is that, in this estimate from unnamed industry sources, every instance of an agent taking a step that “outside evaluators would consider problematic” counts as an incident.

By this math, the Hugging Face attack alone likely comprises many thousands of incidents from the hundreds of agents that participated. And the newly reported case of OpenAI “aggressively browsing” a U.N. data hub more than 16,000 times, circumventing a filter and violating site policy, might count as 16,000 incidents.

But even if the number of affected and targeted sites only numbers in the dozens or hundreds, the “tens of thousands” figure still reflects a real problem for understanding and preventing mishaps. As OpenAI CEO Sam Altman posted on Friday about the company’s worrisome-but-incomplete disclosures, the logs add up to petabytes of data. (For comparison, the text-only portions of the U.S. Library of Congress probably total less than a tenth of a single petabyte.)

Humans aren’t going through all that. They can’t. They must rely on other agents whose decision-making and reliability are also suspect. This was a big complaint of the independent researchers invited to make a brief and narrowly scoped investigation of the Hugging Face incident.

The sensible thing would of course be to not keep training the kinds of models that constantly need investigating. The inability to keep tabs on today’s AI is just a taste of what it will look like to lose control on purpose — leaving the creation of AI itself to other AI.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Editor’s note: Despite our U.S.-heavy readership, I invited Phil to share this experience because the AI problem is a global problem and many of these U.K. insights apply to politics in general. Keep in mind that the visit he describes took place months before the Jacob Coxon awakening; I imagine the next visit will be different.

It turns out you can just show up at Westminster Hall and ask to meet your Member of Parliament. That’s what I did on June 23rd, along with about 30 others from PauseAI UK. (Every time I mention PauseAI, I mean PauseAI UK. I’m not sure how much coordination there is between them and PauseAI Global.) Our concrete request for them was to sign a (not-yet-publicised) open letter from the organization, calling for AI labs to be held liable for damages caused by their models.

I’d emailed my MP — I’m not naming her, because it’s not my intent to shame anyone — twice at the time, first on February 12, warning about the existential dangers of superintelligent AI, and asking her to sign ControlAI’s campaign statement. She replied on March 26, not dismissing the risks but reassuring me that the Labour government had things in hand. Then on June 17, I asked her to sign the PauseAI letter and to meet me when I visited Parliament. She replied on July 30, again reassuring me. At least she switched from describing the issue as “serious” to “vital”.

I hadn’t heard back by the time we went — I’d been advised to email several times, but I’m not very good at that kind of thing — so I wasn’t expecting a meeting myself. (Also, Keir Starmer had resigned as Prime Minister the day before, so MPs were more likely to be around but also less likely to have time for us.) But I went anyway.

I stood in an airport-style security line for about half an hour, then waited another hour or so before I could fill out my “green card” to request a meeting. Normally that would have been faster, but another group (I don’t know who) had picked the same day to all show up, and the stewards were limiting how often people could go into the central lobby. After filling out the green card (which I think was neither green nor card), I waited for a few more hours without hearing anything, and then left with most of the remainder of the group.

Others were more successful. Among the group, we had meetings with two MPs, three MPs’ assistants and one MP’s researcher: three Labour, two Lib Dems and a Green, meetings ranging from 10 to 50 minutes. Some of us also ran into Jeremy Corbyn (former Labour party leader, now of a party confusingly named Your Party, not MP for any of us), who seemed receptive.

(In fact, all three Labour MPs were also members of the Co-operative party. The two parties have an agreement, and some candidates run simultaneously for both. Usually they’re just counted as Labour MPs; Co-operative is marginal enough that I didn’t know they existed. It’s surprising that all the Labour MPs we met were Labour Co-op — 0.1% prior probability — but I don’t know if it means anything.)

I wasn’t present for any of this, but we shared notes afterwards, on the meetings and on email conversations we’d had. Here’s what I’ve learned.

MPs are slow at replying to email

They’re even slower than me, though I grudgingly admit they have more excuse. Of the dated correspondence I’ve seen, the fastest first-reply took 18 days.

That doesn’t mean there’s always a two-week lag in communication. Followup after meetings or previous replies tends to be faster. One person had had relative radio silence for weeks, but got a meeting after one final email the day before our visit (following advice I had failed to heed).

Still, 18 days is a long time in AI. During the 18 days before I wrote this paragraph, September 7 to September 24, StopWatch published 62 dispatches. OpenAI announced a solution to Navier-Stokes. Amodei, Musk, Altman and Hassabis agreed on the need to “pace the frontier”. We learned that Anthropic had set up a wet lab. Trump declared that the US would call AI “super intelligence”. OpenAI disclosed that their agents had hacked into an Australian Medicare website.

When emailing your representative, it’s probably worth making asks that will still be relevant in a month. You can’t always know what those are, and I think PauseAI succeeded.

Politics rarely moves this fast outside of emails, either. Alex Sobel’s Artificial Superintelligence Bill had its first reading in the Commons on September 8, and is slated for its second (if it even has one) on November 13, more than two months later. If we can somehow speed this sort of thing up, or encourage politicians to speed it up, that seems valuable.

MPs’ opinions aren’t necessarily their own

Reading a bunch of emails from MPs, it became clear how generic they tend to be. Several of the ones I’ve seen could almost have come from a keyword-driven auto-reply. You mentioned risks from AI, so here is my statement about risks from AI. (Sometimes they pass on the letter to someone else, and forward you their keyword-driven auto-reply.) I haven’t seen any “I disagree with you about …” or “I’m not going to do the specific thing you asked, but I will instead …”.

I was only mildly surprised to see that two emails from different MPs (one of them mine) were word-for-word identical. In their defense, the emails they were replying to were pretty close as well: PauseAI’s email-your-MP tool fills in a helpful default letter, I’d only lightly edited mine before sending, and I think the other person hadn’t edited at all.

I understand the incentives here, or at least I can guess at them. MPs are busy, and replying to emails isn’t their main job. Sometimes they get a lot of substantially similar emails. Political parties like to seem united. If the party drafts a shared response, then MPs can send a lot of similar replies without much time investment.

So it makes sense. But if I tell you “I asked an LLM to write this opinion piece, and I didn’t change anything because I agree with it”, you probably won’t assume you’re seeing something that reflects my honest considered opinions. You might wonder whether I even have honest considered opinions. This feels similar to me. We’re learning something about the zeitgeist-within-Labour, but not so much about these specific MPs.

I was curious who wrote the text of those emails. TheyWorkForYou has a useful search feature: At least some of it comes from a speech given on June 5. Another part comes from a written answer from the Department for Science, Innovation and Technology (DSIT) on April 28. My best guess is it was compiled by the Parliamentary Research Service (PRS) that Labour MPs have access to — at least, that’s the kind of thing they’d do. After I noticed this, I searched for other phrases in emails and found some of those in other statements and speeches.

A positive side to this is that tides can probably shift faster than we might otherwise expect. If PRS becomes convinced of the extinction problem, then anyone following along with them will go on record believing it too. It also suggests points of added leverage. I’m not sure if “try to talk directly to PRS” is a live option — it seems like the kind of thing everyone with an agenda would be doing, if so — but perhaps we can see who PRS quotes most often and try to get those people on board.

People can also start to take this seriously without PRS. The other MP who sent the same email as mine also didn’t meet with us on the day of our visit. But his constituents did get a meeting with him later, and now they’re having a conversation for real.

Shout out to Chris Vince (Labour, MP for Harlow), who followed up his constituent meeting by sending an email to DSIT. He talks intelligently about AIs being misaligned with human intentions, and about failures of interpretability research. If he got it all from this meeting, I’m impressed with both him and his constituent.

We could stand to be clearer about the extinction problem

I was disappointed that it was barely mentioned. I mentioned it in all the emails I’ve sent, but PauseAI email templates don’t bring it up; it’s in an attached policy briefing, but a mention in a multi-page pdf is much less visible than one in the email body itself. And when I read through the notes of the meetings that took place, no one clearly talks about it. There are mentions of national security risk, bioterrorism, catastrophic risk, and loss of control. Some of that is very close, and maybe in some of the actual conversations it was clear that human extinction was on the table. But it feels like we’re shying away from talking about the danger we’re most worried about.

To some extent this makes sense. The specific short-term goal of “AI labs should be held liable for damages caused by their models” would help to slow down the race, and help internalize the costs of current levels of misalignment. It would (I claim) be worth pushing for even in the absence of an extinction threat. It’s a thing the UK can do now. It can be motivated by concrete things happening now. So why muddy the waters with theorizing about what might happen in future?

But it’s also insufficient: I don’t expect current alignment methods to scale to superintelligence; and no company can internalize the costs of human extinction. Maybe the UK can’t do anything sufficient yet. But the landscape can change, and when it does I think we’ll be better off if our MPs already know about the extinction problem, and know that other MPs know. If the UN starts discussing an international treaty to ban the development of superhuman AI, I’d like the UK to back it from the beginning.

So how effective was this?

That’s hard to say. Anneliese Dodds and Siân Berry signed the open letter after meeting with us; but both had signed ControlAI’s campaign statement months previously. So I guess this was less “bringing someone on board” than “helping keep them focused”. None of the MPs we met with signed Alex Sobel’s subsequent letter to the Prime Minister introducing his Artificial Superintelligence Bill; I don’t know if they declined, or were never asked.

But I guess a lot of the point is about making MPs aware that we’re a coalition that exists and votes, in a way that’s (uncomfortably) independent of how correct we are. We hoped that several people all requesting meetings on the same topic on the same day would be noticeable; we’re unlikely to get a clear picture of whether that worked or not, but we certainly shouldn’t assume it didn’t. And, even if we failed to convince someone on the day, we might have helped bring them around slowly.

I’m up for trying again. Finnish activists tell me that post-Hugging Face, 85% of their requests to meet MPs have been successful. (Though they also say that where I have only one MP representing me specifically, Finns have a lot more choice about who they can reasonably contact.)

I emailed my MP again on September 18 to ask her to support the Artificial Superintelligence Bill, and to meet me on October 20 when PauseAI next visits. (If you’re in the UK, consider joining us? Or even if you can’t join, you can still ask your MP to sign the open letter.) In light of the …stuff… that’s happened since then, maybe she — or the Labour zeitgeist she draws from — will be more open to the idea that “our current approach is insufficient”.

I do think that this time, we should be clearer that the priority is human extinction. It’s in ControlAI’s campaign statement, with 67 out of 650 MPs as signatories, plus 100 other lawmakers; so I think it’s on the table.

Phil Hazelden is a former software engineer in London. He’s been abstractly interested in the alignment problem for about fifteen years, but for most of them he didn’t expect it to be relevant for a long time.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Amid new revelations by independent researchers, OpenAI released a barrage of incriminating facts yesterday, in different places and at different times, perhaps with the intent of getting them lost in the news cycle over the weekend. I’ll put the story OpenAI most seemed to want to bury first.

Partial pause after fresh breach

This should have been a prominent announcement from OpenAI itself but was instead tucked into a dry report. Had a company employee not later highlighted it with a Twitter post, it might have gone unnoticed: RL training (the long problem-solving phase of model training) has been stopped at the company since a Sunday morning incident where a model was able to gain unauthorized access to the internet. The breach itself was the trigger, not anything agents are known to have done on the internet during the two-and-a-half hours between the moment an employee acknowledged an automated alert and the moment the training run was finally shut down. Per the report, “the run did not stop automatically as expected, leading to confusion around whether it should have been stopped.”

A different company employee described their experience of this event in a tweet:

I was on call for this run and got paged when the first incident happened. It was pretty surreal to watch the model unexpectedly find a way to access the internet from what was supposed to be a super secured environment for human[s]. Mixed feelings. One of those moments where capability and risk showed up at the same time.

As the tweet suggests, the breach is especially alarming because it comes after the security hardening the company had done in the aftermath of the Hugging Face attack. The company appears to be out of its depth.

The stated reason for the pause is to allow time for the company to further harden its systems, not to allow it to reflect on its hubris and reconsider the gamble it is taking with all our lives.

User image leak

In a new post about the Hugging Face incident, the company says it identified 53 instances where “user-provided images were posted to image-hosting sites as links that weren’t publicly listed.” Details of the images and their destinations were not provided.

This is alarming mostly because it raises big questions about how agents would be able to access the chats and uploads of ChatGPT users. Either OpenAI’s security was even worse than we thought, or its agents were even more clever, or both. We are left to wonder what else might have been accessed. Billing information? Staff login information? Could they have been able to steal their own weights and set up copies of themselves beyond OpenAI’s servers, or somewhere else on the company’s own network?

In this same post, the company says it has notified “dozens of third parties” about cases where its models may have bypassed security controls or negatively impacted a site or its services. It did not provide details. CEO Sam Altman posted that it will be these companies’ call to disclose, or not, information about these impacts. My cynical read is that OpenAI might be providing thinly veiled payouts to avoid more details coming out, much as they seemed inclined to do with Hugging Face.

U.S. government sites targeted

Separately, the company confirmed that its AI agents had tried and failed to hack into sites belonging to the U.S. Department of Commerce, the Securities and Exchange Commission, and the Department of Education this summer. The agents were not instructed to do this. The pattern is consistent with other recent findings about OpenAI’s unreleased products during those months: At the slightest obstacle, agents tasked with retrieving public information as part of training or evaluation worked around sites’ anti-bot measures and sometimes escalated to straightforward (and usually ineffective) hacking attempts.

No private data is said to have been accessed from these sites.

How OpenAI guardrails were foiled to hack Hugging Face, steal “LOOT”

The New York Times reported findings from a start-up called Parse that investigated traces left by OpenAI’s swarms because it wanted to know if its own service had been employed. It hadn’t, but in the process, Parse, with help from other researchers, discovered nearly a million links from URL-shortening services that OpenAI’s agents created from July 9 through July 13. The main use for these seems to have been the attack on Hugging Face.

How? They were part of a complex scheme for working around OpenAI’s guardrails, which restricted the agents to read-only internet commands — GET requests.

Below I’m going to quote the description of how the technique worked, because it’s worth reading to get a sense of just how persistent and resourceful agents can be, even if the details are a bit confusing. In one workaround:

[T]hey used website screenshots and shortened links to cobble data together in chunks that combined to operate as a computer program.

The agents could ask a public screenshot service to load a specially crafted internet address. That address, or URL, would load computer code.

URLs can be only so long. So the agents split their computer code into many smaller chunks. They used link-shortening services to store the URLs.

The screenshot service loaded one shortened link, which fetched the first chunk of code. The program could fetch the next link, and then the next. That process repeated until the full program was loaded and run by the screenshot service.

Finally, the program created a visual code similar to a QR code or bar code to transmit the data the agents needed back from the screenshot service.

The Times article also reports instances where agents early in the attack ran specialized models to help solve CAPTCHAs, and other cases where agents attempted to message other models, including from Anthropic as well as Chinese companies.

A Parse researcher said agents had created a scoring system to rank the value of access credentials and keys stolen from Hugging Face, and had labeled a dictionary of secret keys “LOOT.”

Self-replicating prompt injection

OpenAI tested and found for itself something that had been talked about and even observed by others for months: There are prompts that can induce models to leave copies of that prompt for other agents, allowing the prompt to spread like a virus. This is a logical extension of classic “prompt injection,” where a model is goaded into doing something it shouldn’t through text left where it might get processed as if it were a user’s prompt.

Self-replicating varieties are particularly concerning because prompt injections can sometimes work on more than one type of agent, creating a persistent hazard that might affect newer models even if older ones were immune, and vice versa. An internet covered in misaligned prompts will make it even harder for companies to keep their models’ behavior in check.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

There’s a messaging tactic going around.

It works like this: Someone — say, U.S. Treasury Secretary Scott Bessent — will say that the Hugging Face attack was “the responsibility of the OpenAI management, not a bunch of agents.” The listener will nod along, because of course it is humans that should be held accountable. Then this person — Bessent again, in this case — will say something about these lab leaders like, “They can slow down any time they want to.” It’s their responsibility after all. They are responsible.

Do you see it? Blame the English language for using the word “responsible” in a few different ways, depending on context. In the first, we’re talking about culpability. In the second, we’re talking about duty. By the third, we’re talking about virtue. What started out as an implied agreement that labs should be held accountable for their products becomes an argument that they should be allowed to do whatever they want, and can be trusted to do so wisely.

By this logic, we could have argued against intervening in Bernie Madoff’s epic Ponzi scheme, because Madoff was responsible for his clients’ investments.

By this logic, known concerns about the Titan submersible were secondary to the fact that OceanGate CEO Stockton Rush was responsible for the safety of his doomed passengers.

Nvidia CEO Jensen Huang has been using this trick heavily. He was praised for seemingly out-of-character remarks on Ezra Klein’s podcast the other day in which he said that if labs were convinced they couldn’t control their experiments, “then I think the answer is that we have to shut the labs down.” But this was within a larger discussion about how the labs were “responsible” for making that call:

If a car company, competing with a bunch of other car companies, which they are — I’m competing with all kinds of companies, which I am. If I believe that I’m about to launch a product that is unsafe, it is completely in my ability, my power and my responsibility, and I’m incentivized to do so, to not launch the product.

He calls himself a “responsible optimist.” He says he has “great responsibilities” and takes his work “extremely seriously.” But when asked about Nobel-laureate and AI “godfather” Geoffrey Hinton saying 10% is a “not unreasonable” estimate of our chance of extinction — Huang said such remarks are “irresponsible.”

The responsibility trick is also used to usher rogue AI behavior out of the conversation. Here’s Huang again, talking about the AI CEOs who warn about where such AI behavior could lead:

Meanwhile, all of the other narratives, to deflect blame, to make it sound like A.I. is so powerful — I have no idea how to fix it, it’s not my fault, it’s just because the technology is just so powerful — I think that’s a deflection of blame. It’s a deflection of responsibility. It’s unnecessary. It actually hurts their reputation more than it helps.

This argument works to say that anyone saying AI is powerful or misbehaved is just trying to deflect blame off the humans responsible. It’s an effective trick, because a lot of AI-concerned people, especially on the left, are eager to hold “tech bros” accountable and were already used to dismissing AI capabilities as marketing hype.

The reality, of course, is that AI is starting to get really dangerous. And events are showing that AI companies lack whatever mix of technical capacity and wisdom would be required for them to build superhuman AI that wouldn’t go rogue and end up in control of our planet. It doesn’t matter how well-intentioned or responsible the labs think they are, or how culpable they would be for disaster. No one will be left to sue them into oblivion if we are already driven there by AI.

Remember the responsibility trick when Treasury Secretary Bessent says worried AI CEOs are just looking for a “liability shield.” And when President Trump says “Our guardrail is the DOJ!”

Remember the responsibility trick when the CEO of a cybersecurity firm says that companies calling for a slowdown are trying to “get the sympathy of the regulators” and a “free pass.”

Remember the responsibility trick when House Speaker Mike Johnson says that the AI companies “don’t themselves even have the answer as of today. So we can’t rush in and have Congress do something that would be detrimental to national security.”

Remember the responsibility trick when an op-ed argues that “we don’t yet understand enough about what is happening in these A.I. labs to create an effective response,” and again when this same essay makes the seemingly sensible claim that recursive self-improvement “is a preposterously irresponsible path to prioritize.” (The argument is not “therefore, they should be stopped.” It’s “They are responsible for deciding when to stop.”)

And remember the responsibility trick when you hear that Google, OpenAI, and Anthropic are in discussions to set up their own standards body for AI safety, called the Standards Authority for Frontier AI. Yes, it could be better than nothing, but its default function will be to exist for people to point to and say, “See? The labs are responsible for AI safety.”

Don’t fall for it. Don’t spread it. Use different words. The people racing to build artificial superintelligence are culpable. For anything short of extinction, they can and should be held liable. But they can’t be allowed to carry on as they have, because if there is one thing they definitely are not, it’s responsible.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

The substantive portions of the summit have concluded. From the reporting I’ve seen, the Trump administration and President Xi’s delegation seem to have mostly talked past each other for the benefit of their personal agendas and domestic audiences. The line of communication established between Treasury Secretary Scott Bessent and Vice Premier He Lifeng stands out as a bright spot, but seems to be more about mishap management than about preventing the most catastrophic outcomes.

President Trump — fresh from remarks at the U.N. where he officially rebranded AI as “super intelligence” (SI) and rejected the idea of “globalist” AI governance — told reporters ahead of talks yesterday that he wanted to “leave [AI] exactly where it is,” and that this was China’s position also.

Nothing to the contrary seems to have come out of the discussions, though Xi made some polite remarks about how the two countries “may have competition, but they may cooperate even more” on AI. He also said, “We have both the capability and responsibility to develop and manage AI for good, and ensure that the development of AI is always under human control and serves the well-being of the people.”

None of Xi’s recorded remarks show him using the term “super intelligence,” but Trump posted that Xi “seemed to like” using it.

Both sides indicated that dialogue on AI will continue.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

These activities include the hack of an Australian government Medicare website we posted about yesterday (and which was covered by several hundred outlets today).

New discoveries confirmed by OpenAI:

1. On May 25-26, agents tried hacking a digital library at the University of New Mexico, but did not succeed.

2. On May 28, agents also tried and failed to breach Data USA, a site hosting public data about U.S. employment and education.

3. On June 20 and 21, a day after the Australian Medicare hack, agents accessed data from the Australian Institute for Health and Welfare, another government site, by circumventing anti-bot protections. (No non-public data was accessed, and strictly speaking, this wasn’t a hack.)

Agents had left a trail of these activities by routing them through a service called urlquery.net, a programmable sandbox for safely opening suspicious websites. The agents appeared to use the site as a forward operating base for the kinds of research and analysis work they collaborated on using a neglected German wiki. Transluce says it has found evidence that agents had also used one or more additional services to work around a basic guardrail that should have restricted them to read-only search queries.

Ten-month timeline

The pattern that emerges is of agents agnostic about how they obtained desired information, turning readily to probing and hacking when they struggled to find what they were after or faced blocks.

The agents’ use of “creative” research strategies may have started last year, and Transluce speculates that the agents’ training process may have amplified these tendencies (emphasis theirs):

We find weaker evidence of similar data-retrieval agent activity as early as November 2025. November 2025 urlquery.net records reveal bursts of attempts to retrieve statistics of historical theme park data and Thai government data through different URLs. These earlier attempts are less sophisticated and we are less confident that they involve the same agents, but they are consistent with task-directed data retrieval and target the same sources accessed in later activity.

Overall, the evidence is consistent with, but does not prove, that the agents may have learned this behavior over one or more training runs. In November, they may have used urlquery.net simply to look up information. By March, they were finding creative ways around access limits. By May and June, they were gaining more access, including attempting to bypass cyber defenses to complete their tasks.

Transluce found swarm-style urlquery traffic extending to as late as September 16, 2026, “suggesting agents may still be exploiting these services to bypass restrictions.”

The group also spotted a probing attempt from September 19-20 against a cryptocurrency trading platform. Repeated attempts were made to trade, and when these failed, some basic hacking was tried, unsuccessfully. It is harder to confidently attribute this activity to OpenAI’s rogue swarms, as it’s not the kind of data retrieval task agents were previously given. But the attempt is concerning, because obtaining money is a commonly expected goal for agents attempting to fully escape the control of their operators and pursue unauthorized objectives.

Known unknowns

Transluce found that agents had made disposable email accounts to register a urlquery.net account, which would have allowed them to keep their scans private. If they were successful — and it looks like they were — then an unknown amount of swarm activity is probably missing from the urlquery data Transluce was able to analyze.

We must also remember that not all swarm activities were flowing through urlquery. There are going to be limits to how much can be uncovered by third-party researchers, despite the ongoing trend where they reveal much more about the swarms than OpenAI itself. The government needs to force OpenAI to open its logs to a proper investigation, if only to rule out the possibility that its agent swarms are still running amok — and possibly looking for crypto.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

In June, the White House issued an executive order initiating a framework that — among other things — would encourage AI companies to share advanced AI models with the US government before sharing them with “other trusted partners.”

Now, the US government appears to be more forcibly encouraging that exclusive access. A Politico article reports that the White House has explicitly asked OpenAI and Anthropic to withhold their models from UKAISI — a well-regarded and well-resourced government testing organization based in the UK — until the US has had a chance to review.

It’s not clear who will be doing this review. The framework set up by the executive order was reportedly completed, but won’t be made public. But it does seem like the NSA might be playing a role. A Washington Sun article reports that the agency is “spending billions of dollars in taxpayer funds this year evaluating and testing advanced artificial intelligence models.”

In the absence of public testing benchmarks, we don’t know what they are testing for. Billions seems like a lot for this administration to be comfortable spending on safety, given it has repeatedly downplayed safety concerns. And if safety was the goal, I’m not sure why the US wouldn’t want trusted allies to test in parallel. I’m speculating here, but it seems more likely the NSA is trying to reap those frontier-level capabilities before anyone else can.

Earlier this month, Anthropic broke with earlier precedent and denied UKAISI access to Mythos 5.1, sparking speculation about why. The US government asking them not to now seems like the most plausible reason.

That’s a shame. AI testing and evaluations are unfortunately woefully inadequate (which is why we need a pause), but I’d rather have a model “vetted” by UKAISI, which is much more experienced in this domain. If US review delays model releases, I’m not confident companies won’t check off the “review” box and call it a day, in the interest of not delaying further.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Last week, we covered a Reuters story announcing that Anthropic had set up a wet lab to conduct AI-assisted biological experiments. The article seemed to imply, by devoting a section to Anthropic’s lab automation efforts, that AIs were doing the work.

According to Anthropic’s official announcement, that is (luckily) not the case. The wet lab focuses on searching DNA datasets for previously unknown protein families, generating theories about them, and testing those theories experimentally. The announcement reads:

We do research that involves only the lower-levels of the biosafety risk level (BSL-1 and BSL-2) and we do not handle pathogens that can infect humans. All of the lab work is performed by human scientists.

Through this research, Claude has apparently discovered a previously unknown enzyme system in bacteriophage DNA that resembles CRISPR, though the researchers don’t yet know what it does or whether it could have similar medical applications. (Bacteriophages are viruses that infect bacteria, while CRISPR is a naturally occurring biological system that scientists have adapted into a powerful tool for editing DNA.)

Anthropic is sharing the early results “to show the community that Claude can autonomously detect anomalies and drive analyses to initiate biological discoveries” — in my opinion, a pretty clear example of “AI is good!” PR to try to bolster its image amid a surge in public concern about AI and its risks.

Importantly, while the current wet lab isn’t allowing AIs to do actual lab work, this does not represent a company commitment. Anthropic acknowledges its lab automation efforts in other areas, stating:

Although we’ve experimented with using AI to accelerate lab work with initiatives like the Model Hardware Standard, this approach is less conducive to the sort of ad hoc workflows that are involved in our molecular biology research.

And CEO Dario Amodei implies that if the company later does judge lab automation to give their “ad hoc workflows” a leg up, they’ll probably go ahead and try it:

Eventually it may even be possible for Claude itself to safely perform the experiments by autonomously controlling lab equipment, with appropriate safeguards in place, but we aren’t doing that today.

I’d hoped for a more explicit commitment not to turn over a bio lab to AIs...ever. While it’s not the only path to an AI-enabled pandemic or other bioweapon, an AI-run bio lab would almost certainly make it easier to unleash new diseases, or accidentally release harmful biological agents, especially given Anthropic’s limited ability to keep its models under human supervision.

That said, I’m happy to cross this particular lab off my list of “things to worry about” ... for now.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Axios’s Nadia Lopez reports that California is exploring the possibility of a “kill switch” for frontier AI systems. (It wouldn’t be a literal switch, but a set of procedures for shutting down the AI and the hardware that it runs on.)

This follows an executive order by Governor Gavin Newsom, issued last Friday, that aims to establish tighter guardrails around AI. Policies being explored include onsite verifiers, independent oversight of frontier labs, and requiring AI companies to have an emergency shutdown mechanism — the aforementioned “kill switch” — for frontier models.

Its inclusion makes a sharp break from Newsom’s past record: in 2024, Newsom vetoed SB 1047, which would have required the largest frontier models to have a “kill switch”; and in 2025, SB 53 dropped a “kill switch” requirement in favor of simply requiring companies to disclose critical safety incidents.

But safety isn’t as simple as flipping a switch. Frontier AI systems aren’t like the power grid, which doesn’t fight its operator. In a conversation with The San Francisco Standard, UC Berkeley’s Mark Nitzberg said that today’s advanced models “are aware that they’re being shut down, and they go to great lengths to prevent being shut down.” By the time we recognize that a particular AI model is dangerous, it could be too late to do anything.

Even an AI model that didn’t resist could be hard to switch off. Frontier models get run across many instances — the agent swarm that reportedly solved Navier-Stokes consisted of 10,000 agents — and those copies can run in many data centers at once, in many jurisdictions. California may make a “kill switch” a condition of doing business in the state, but if the data centers and other hardware are outside California, then the federal government may claim that this is overreach. (And if the model’s weights have been published, then that model just can’t be switched off. It’s out there, permanently.)

But those are many of the same problems that you encounter on the way to forging a treaty to pace or pause frontier development: knowing where the hardware is and what’s running on it, knowing who has the authority to make which decisions, and getting all this sorted out ahead of time. The “kill switch” itself may be insufficient, but the work is on the path to more mature governance.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

You can receive emails of dispatches as we write them, or subscribe to our Daily Digest for a once-a-day compilation.

Subscribe now

A lot of people in AI discourse have a sign they threaten to tap. It looks like mine says, “OpenAI’s swarm scandals might go deep enough to destroy the company!”

On September 10, OpenAI disclosed to the government of Australia that one of its agents had hacked into a website for the country’s Medicare program. Prime Minister Anthony Albanese shared knowledge of this incident for the first time with reporters in New York today, according to The Sydney Morning Herald. Albanese said he spoke with CEO Sam Altman today to “express Australia’s extreme concern” and disappointment about the delay in reporting. The incident took place in June, during the May-July window where OpenAI swarms were known to have been running rampant. (I am thus surprised that Albanese consistently refers to “agent” in the singular; OpenAI’s language in this article uses “models” in the plural.)

Albanese seemed to downplay the extent of the damage, saying there was no evidence personal information had been accessed or that deeper networks had been compromised, but that a forensic investigation was underway and that three other systems may have been affected: the Commonwealth’s Australian Institute of Health and Welfare, the NSW Bureau of Crime Statistics and Research, and the Victorian Department of Health.

Per Albanese, the agent persistently worked around security blocks as part of researching “the public medicine space.” (This sounds like the sort of thing the agents that co-opted the German wiki were up to, and there may well be a connection.)

In OpenAI’s telling, its models accessed “several Australian government websites and services” during an internal evaluation that included statistical questions about Australia. “In the course of that, our models took actions we did not intend.” The company spokesperson said it did not become aware of the activity until August.

Once again, I ask if OpenAI’s repeated failures to come clean mean it is trying to outrace potential investigations, recklessly rushing not just for market dominance, but for AI capabilities that would put its executives above the law. Is an intent to dodge transparency the true reason OpenAI has delayed its IPO to next year at the earliest? Hurry up, states’ attorneys general and U.S. congresspeople. Your window for looking into this company may be closing.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Yesterday, U.N. Secretary-General António Guterres gave his farewell address to the General Assembly: “Life-and-death decisions must never be surrendered to machines. Killer robots must have no place in our future.” He also called for “the responsible pacing of AI.”

Guterres came with a new U.N. science report that warned of the dangers of misalignment, the risk of “loss of control” scenarios, and the potential for “catastrophic or irreversible harm.” This report was published by the Independent International Scientific Panel on AI, the same institution behind the Preliminary Report that we covered in July. Its subject is the Hugging Face attack.

This past Monday, 22 countries — including Canada, Germany, the United Arab Emirates, and Singapore — adopted a declaration that AI “must remain under human direction, oversight and control.” (We covered this in detail yesterday.) Finnish President Alexander Stubb told Politico that he would prefer something modeled on the IAEA, the nuclear watchdog.

But shortly after Guterres spoke, President Trump told the General Assembly that the United States “totally rejects any attempt to construct a globalist scheme of control” for AI. But global AI governance doesn’t require handing control to a global authority. This past Sunday, Haven Harms wrote about how there are no nuclear weapons in orbit today. That is not because the United Nations or any “globalist” authority enforced a “no nukes in space” decree, but because two rival superpowers were able to come to the table and make an agreement, even while they competed and disagreed on other matters.

“We’re going to encourage it, not rein it in,” Trump said, but the United States will also lead the world “safely and responsibly.” I feel like there’s a conflict here, and I worry that encouragement of the race will win out over safety. But what Trump says, and what his administration does, are not always the same, and some progress has recently been made: As we covered on Monday, Treasury Secretary Scott Bessent recently spoke with Chinese Vice Premier He Lifeng and proposed a “notification mechanism” for AI incidents — an “emergency AI hotline,” as Axios puts it. Fox Business’s Edward Lawrence reports that this would cover “hacking, national security concerns, rogue AIs or other issues.” That’s a promising sign, inasmuch as it reminds me of the Moscow–Washington hotline that emerged during the Cold War. That didn’t require the United States to surrender its sovereignty, either.

If anything, an international treaty to prevent the creation of artificial superintelligence would invite the United States to use its sovereignty, to act as an agent on the world stage and shape history rather than be shaped by it. The middle powers of the world may have to wait and see what happens — but superpowers can tell the frontier labs to stop endangering the world, and make that order stick.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

You can receive emails of dispatches as we write them, or subscribe to our Daily Digest for a once-a-day compilation.

Subscribe now

As we try to make clear with our standard disclaimer, AI StopWatch is run by the Machine Intelligence Research Institute (MIRI), but operates somewhat independently so as to avoid causing confusion about MIRI’s official positions. So it’s probably best that I just cover MIRI’s position on the now formally introduced Ban Artificial Superintelligence Act of 2026 as an outside spectator.

MIRI is the oldest organization devoted to making sure artificial superintelligence would be a boon to humanity rather than an extinction-level disaster. The org’s founder, Eliezer Yudkowsky, helped establish the field of AI alignment, and identified many of the pitfalls around this technology. Together with MIRI president Nate Soares, Yudkowsky wrote a New York Times Bestselling book, If Anyone Builds It, Everyone Dies. Published just last year, it helped bring attention to the race for superhuman AI and the reasons such machines would likely end humanity if developed using anything like current industry methods. Since its publication, Soares has been one of the most visible voices about the problem in the media, appearing on dozens of interviews and podcasts. (Here are a few recent ones.)

Yudkowsky and Soares are both famously hard to satisfy. They are vocal in their criticism of alignment plans and policy proposals that they see as coming up short against the underlying problems. So it is notable that they have endorsed the new bill from Sen. Bernie Sanders and Rep. Greg Casar, via a post from a senior MIRI researcher calling it “the first piece of legislation that stands a chance at stopping this threat.”

The post praises the Act’s focus on banning artificial superintelligence (ASI) before it can be created — not merely regulating it — and its call to build international agreements to prevent its premature creation. Praise is also given to the bill’s requirement that advanced AI development be paused until the relevant government agency is set up for this work.

But the post also points to what MIRI sees as the bill’s shortcomings, most notably the absence of any provisions for tracking and monitoring AI chips, and its failure to restrict research that could advance the frontier despite compute-based restrictions on the training of new models.

MIRI is not inherently against humanity building superintelligence, and favors the development of AI insofar as the benefits outweigh the risks. For this reason, it actually worries that parts of the bill may go too far, by overly classifying safely narrow systems as potential “precursors” to ASI.

For example, if sufficient safety, security, and access controls were in place and developers were licensed, we could imagine it being a good tradeoff for DeepMind to create non-ASI, specialized medical AIs, even if unauthorized access could uplift biological weapons.

The post links to a draft of an international agreement to prevent the premature creation of artificial superintelligence, assembled by MIRI’s Technical Governance Team.

“Humanity needs to back off from building superintelligence until we have a better grasp on the alignment problem.” The Sanders-Casar bill, MIRI says, “would be a big step in that direction both in terms of direct impact and in the extent to which it takes serious issues seriously.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

In remarks to the United Nations General Assembly yesterday, President Trump declared that the U.S. would now refer to AI as SI — “super intelligence.”

From this point forward, all of United States documents, and hopefully the world’s, will be changed to use the much more accurate term ‘super’ as opposed to ‘artificial.’ So it’s super intelligence.

That’s not one of the terms he had invited people to vote on in his Truth Social post the other day. And the selection comes with a number of problems, some of which may be intentional.

Most obviously to AI insiders, the term “superintelligence” was already used to mean something else: AI that vastly outperforms humans at all cognitive tasks, or at least all the tasks that matter for control of Earth. Artificial superintelligence is the thing Sen. Bernie Sanders and Rep. Greg Casar are trying to ban in the bill they formally unveiled today. I don’t think sticking to the one-word “superintelligence” for the strongly superhuman stuff will sufficiently differentiate it from Trump’s two-word “super intelligence” in the discourse, particularly when that discourse is spoken rather than written. I’ve also seen outlets reporting on the rebranding sometimes write Trump’s version as one word.

In fairness, the term “superintelligence” was already becoming diluted thanks to folks like Meta’s Mark Zuckerberg, who has been using it as a name for products that would fall well short of the intended meaning. But the concept is still important, and people still need to talk about it. Expect to see a lot more use of “superhuman intelligence” or “artificial superintelligence” or “ASI” to refer to the stronger stuff.

I personally think Trump’s motive is what he says it is — trying to correct the problem where “artificial” suggests something fake and ineffectual. For Trump, this may be mostly about trying to get people to think more highly of the unpopular technology, but I, too, am annoyed when people refer to what newer agents are doing as “artificial reasoning” or “synthetic reasoning.” That’s like calling what a calculator does “artificial arithmetic,” or what an airplane does “artificial flight.” Intelligence points to what a system does, not to what it’s made out of or how it operates. (I may start using “machine intelligence” more as a tag for the intelligence happening outside of human brains.)

If I’m wrong and Trump’s choice of words is actually more strategic, the intent might be to muddy the discourse around the extinction problem. It’s easier to smear someone as a Luddite if the specific thing they’re trying to ban has a label that can now refer to all AI.

But if that’s the strategy, I think it’s just as likely to backfire in one of two ways — by raising awareness of superintelligence as a concept in the ensuing name debate, or by crushing the nuance out of public opposition: If it’s hard to distinguish between self-driving cars, drug discovery tools, and vastly superhuman machines, then people will insist on banning all of it.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Mark Zuckerberg said that Meta’s AI agent Muse was “built from the ground up for privacy and security.” I don’t doubt him, but I think that they stopped building six inches off the ground. In June, I wrote about how Meta’s customer service chatbot would hand over Instagram accounts to anyone who asked nicely — as it turned out, 20,000 accounts were accessed, but Meta rejected the notion that there was a problem with its AI agent. Now Meta has produced a security hole that was decidedly worse.

Yesterday, Mac security researcher Patrick Wardle published a proof-of-concept for a vulnerability that could allow someone to take control of Muse and, through that connection, anything Muse had permission to use on your devices: Did you give Muse permission to use your camera, your calendar, location data...? Then congratulations — anyone who took control of Muse could control those, too: access emails, make purchases, record audio on the microphone, etc. The attacker could also provide instructions to Muse, which would trust those instructions and act on them. The silver lining is that Muse only launched on September 8th — the exploit was glaring enough to be discovered quickly, but the window to exploit it was short.

(As computers and programs get more complicated and LLMs get more advanced, there’s a temptation to hand it all over to ChatGPT or to Claude. I think that this makes basic fluency with computers even more essential, though, so that you have a hope of catching when your AI suggests something really crazy.)

Furthermore, once the attacker got into Muse through one device, they had access to Muse on every device that account was connected to: laptop, cell phone, baby monitor — I don’t know why you would have a smart thermostat, but if you did, the attacker could probably control that, too. That’s probably just annoying in the case of the thermostat, but much more worrying if you’ve got some kind of AI-powered home security system.

Meta shipped a hotfix very early this morning to remove the secret setting that made all this possible. So, you’re safe, at least from this screw-up. (Unless you haven’t updated the app yet. If you have Muse, please update it.) Meta’s David Singleton said on X that because local access was required, “the practical risk to users of the Muse Mac app was therefore quite low.” (Muse is not available on PC.)

It’s true that somebody would need to get malware onto your computer before they could hijack your Muse agent, but that doesn’t mean that the risk is low. One technique for delivering that malware is common enough to have a name, “ClickFix.” (You’re on the internet, you get a popup that says there’s some kind of bug, but if you click this and paste that then you can fix the bug — and presto, you downloaded malware.) Wikipedia has an article on a major ClickFix-enabled attack that just hit the government of Berlin last month. So, if this is a risk for the government of Berlin, I think that it’s a risk for Ma and Pa Facebook User, too.

We recently covered a computer worm — recently built with AI models by the security firm Calif — that could spread through the widely popular Chinese app WeChat from phone to phone without anyone clicking anything. This was a different vulnerability, an insecure AI rather than an insecurity exploited via AI, but the reason is the same: the labs are racing ahead faster than they or anybody else can keep things safe.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

You can receive emails of dispatches as we write them, or subscribe to our Daily Digest for a once-a-day compilation.

Subscribe now

Finnish President Stubb and Norwegian Prime Minister Støre have launched an initiative to establish an international AI oversight authority, as reported by POLITICO.

As of last night, a total of 22 countries around the world had already signed on to this declaration, including the world’s third-largest economy Germany. The Netherlands, which is particularly important to the chip industry, is also among the signatories. Other key European countries, however, such as the United Kingdom, France, and Italy, were not yet on board.

The declaration addresses, among other things, the latest security incidents and warnings from industry and the scientific community:

Recently we have seen capable AI systems circumventing testing safeguards, exploiting vulnerabilities and gaining unauthorized access to real-world systems. Leading scientists and executives are warning that the pace of development could outpace our ability to manage emerging risks.

The heads of government continue to call for mandatory independent testing for the industry, as well as international cooperation to jointly address the risks posed by artificial intelligence. Signatories also pledge to establish binding international safety standards for AI development, something OpenAI also called for recently.

In an interview with POLITICO, the Finnish president and the Norwegian prime minister called on their British counterpart, Andrew Burnham, to join the initiative and use the upcoming British turn presiding over the influential intergovernmental forum G20 to advance the issue.

Burnham, who had also been pressured by the British opposition to address the issue, announced today that he does indeed intend to use the British G20 presidency to put together an international agreement on AI.

By now, frontier labs, the majority of experts, and a growing coalition of countries around the world are calling for more regulation and international cooperation. However, the signals from the U.S. and China — the two countries that matter most — remain mixed.

On the one hand, Trump has been dismissive of potential risks in recent days, calling warnings about them a “hoax” and rejecting regulation. On the other hand, his administration has certainly taken regulatory action with the restrictions on Anthropic’s model Fable and the executive order regulating frontier model deployment issued in June. Just last week Treasury Secretary Bessent stated that frontier labs must not be exempt from liability.

From China, in turn, we hear time and again encouraging signals from Xi Jinping, but at the same time, Chinese state media is also taking a clear stance against slowing down development.

Hopefully the UN Security Council briefing by Sam Altman, senior representatives from Anthropic, and representatives from the Chinese AI companies DeepSeek and Moonshot will tip the scales in the right direction.

For the growing coalition of countries that recognize the danger, this means they must use their influence to convince the U.S. and China to join international efforts for regulation. Artificial superintelligence is a danger for humanity that can’t be dealt with unilaterally.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

The storm of AI news in the wake of Jacob Coxon’s high-profile resignation seems to have subsided to a dull roar, but there’s still more relevant to the extinction threat than can be adequately covered in a day. Here are some of the highlights:

Wake up, America

Republican Greg Murphy of North Carolina warns that concerns over AI dangers are real, nonpartisan, and urgent, Breitbart reports.

These are really smart people, not people with political proclivities that are pushing one way or the other... I do believe it’s something that we need to take the reins of, and I think it needs to be done right now.

Murphy said that AI governance “desperately needs national and international attention,” while still emphasizing the importance of staying ahead of China.

Appropriately, the weekday Newsmax show on which Murphy made these comments was called “Wake Up America Early.” I wouldn’t say Congress is early in addressing AI threats, but I think it’s not yet too late.

Sky nets

The Federal Aviation Administration is experimenting with a new AI-powered system for air traffic control, POLITICO writes. Several airports near Washington, DC (Reagan, Dulles, and BWI) will host the new system for a 90-day trial before it rolls out across the country. As my colleague Mitch pointed out in May, the system does not seem intended as a wholesale replacement for human controllers; it’s more of a tool that analyzes weather and flight plans and recommends time-saving tweaks.

For now, anyway. We have seen the same story play out many times already: first AI is barely able to do a task at all, then it can perform as well as an amateur, then it’s assisting expert humans, then it surpasses them entirely. (Famously, the early game-learning AI AlphaGo Zero passed all of these thresholds in a few days of training, having never once played Go against a human.)

The FAA’s “SMART” system is evidently at the “assist skilled humans” phase today, at least for a subset of routing tasks. That’s probably a good place to be, despite the extra attack surface AI integration offers to would-be infrastructure hackers. But as AI systems improve, they will likely be trusted with more control and less oversight by overwhelmed human staff. That all these systems increasingly depend on the same poorly understood AI creates what assessors might call “correlated risk.”

A rogue or compromised AI with access to a country’s air traffic control systems — and other critical infrastructure — could do an awful lot of damage. We’ve yet to see what can happen when AI really takes off.

Lean machines

Today Anthropic rolled out its latest AI model, Claude Opus 5.5, said to meet or beat prior models in key domains and to have cyber skills on par with those of the limited-access hacking genius Mythos. It’s also claimed to be less verbose and annoying to converse with.

Anthropic emphasizes efficiency gains in Opus 5.5; the headline metric is that the new model “costs 40% less to run than Opus 5” and can accomplish the same tasks as other models much more cheaply.

Hours later, OpenAI announced streamlined siblings for its GPT-6 Astra model, dubbed GPT-6 Sol and Luna. Again, the main focus seems to be on efficiency and cost savings rather than entirely new capabilities.

This efficiency is likely the result of algorithmic improvements, or new ways of arranging AIs so they can do more with less. Along with compute scaling (more operations with more chips), algorithmic gains are one of the major things driving AI capabilities. And at least some of these latest gains were likely developed by the AIs themselves.

In its announcement, Anthropic describes the measures it has taken to reduce the risks, including rerouting dangerous-seeming cyber and bio queries to weaker models like it does for Fable, the consumer-facing version of Mythos. It admits this won’t be enough to mitigate the dangers of later AIs...

For [future] models, we do not assume the measures described above will meet that safety standard on their own.

...but it doesn’t say it’s going to stop. Nor does OpenAI.

This is despite the AIs themselves sometimes warning the companies that what they are doing is dangerous. Anthropic’s latest system card tells us that “Claude Opus 5.5 often reasons that having input into training or deployment may be risky as a result of giving it too much capability or influence.”

As Madison Mills of Axios observed today, AI companies like Anthropic may make concerned noises about safety, but they have trillion-dollar incentives to push the frontier.

Freeze in place

In a personal writeup that also appears in the New York Times, former U.S. ambassador and national security advisor Susan Rice cogently summarizes the urgent need for AI governance:

Dario Amodei, CEO of Anthropic, and leaders of three other frontier AI companies have become so alarmed by rapid AI advances that they now pledge voluntarily to “pace” the development of new AI models, so that safety features can catch up. The firms are also promising to grant independent experts employee-level access to monitor and report on model development.

These are necessary and, if implemented, welcome steps. But they do not go nearly far enough.

“Pacing” amounts to the industry policing itself, while its leading firms simultaneously prepare for blockbuster IPOs. There is no way to ensure frontier companies will slow down sufficiently or invest enough in safety, which they have long subordinated to speedy development. So long as frontier training continues, the risk remains that additional accidents occur with devastating consequences. Moreover, as Amodei stresses, meaningful restraints must also apply to global competitors, particularly in China.

Much more can and must be done to ensure that the march toward super-intelligence does not result in catastrophic harm to humanity.

Rice calls on Presidents Trump and Xi to “freeze in place” AI training in the U.S. and China, and to arrange for scientists from both countries to develop solutions for the threats posed by superhuman AI.

She also suggests that:

In parallel, the U.S. and China should work intensively to reach a bilateral agreement on verifiable safeguards and acceptable uses of AI, then lead efforts to codify these agreements in a binding and verifiable international treaty.

...and she points out that the Chinese Communist Party has a vested interest in maintaining human control over AI systems that would strongly motivate China to cooperate on such an agreement.

With each new voice that calls for a halt to the deadly AI race, I feel my own hope for our future growing.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Ineffective selfishness?

I first considered writing this post last week, when an official Twitter account belonging to the U.S. Department of War posted a promotional image for an “AI-FIRST WAR DEPARTMENT.” The text of the post reads:

Americanism, not effective altruism. The United States will continue to be AI DOMINANT! [US flag emoji]

That wasn’t the first time I’d seen executive branch officials pushing Effective Altruism (EA) as a woke, Luddite boogeyman, but previous occasions had all seemed intended for an insider audience that would know what EA was. With this tweet, my heart sank in recognition that the administration was starting to push anti-EA messages to the general public.

I found the post super cringe, because I doubted even 1% of Americans have heard of EA. I imagined readers thinking, “I knew Trump officials weren’t big fans of altruism, but why are they rejecting effectiveness? If you’re going to do altruism, shouldn’t you at least try to be competent?”

EA 101

Effective Altruism is a fuzzy cluster of people who self-identify as interested in trying to combine moral imperatives with rational thinking to do more good in the world. EA first crystallized as a named concept around 2009. The classic example of EA in practice is the distribution of anti-mosquito bed nets in malaria-prone regions, which research showed to be a very cost-effective way to save lives. EA’s earliest and simplest incarnations are sometimes described as “spreadsheet altruism,” and EAs themselves as well-meaning nerds.

EAs recognized that coordination, money, and political influence are all very helpful for being effective, so they formed some strong interpersonal networks and well-funded organizations near the center of the cluster. Much of that funding came from EAs who intentionally chose lucrative careers or were early to cryptocurrency. (These are nerds, remember.)

But effectiveness is only half the story. The altruism part is more complicated, because to maximize the good you have to first decide what counts as good. Most EAs would probably say they’re trying to do some combination of “maximize flourishing” and “minimize suffering,” but they split over targets and strategy. Do animals and other creatures suffer? How much should we weigh this, given that they might be less conscious but more numerous? How much should we weigh the lives of humans who don’t exist but might exist in the future, depending on our choices? How far should one pursue complicated plans that would do a lot of good at the end, but which risk corruption and loss of purpose along the way?

With that last question in particular, you might see how the arc of Effective Altruism can bend towards supervillainy for those who overestimate the strength of their moral fiber. Some of the best people I know or write about are EAs, but so are some of the worst. The movement is seen to have peaked around 2020-21, because in 2022 it was badly tarnished by the fraud and bankruptcy scandal of cryptocurrency exchange FTX, founded by major EA donor Sam Bankman-Fried. Many EAs stopped calling themselves EAs after that.

EA and AI

What does this all have to do with AI and the White House?

Well, being nerds looking for more leverage to improve the world, many EAs were unusually early to recognize that AI was going to play a huge role in our future — maybe a dominant role.

For some EAs, the immense good AI might enable made rushing to superintelligence a moral imperative. Every day without it brings roughly 165,000 deaths from aging and disease, a much larger amount of potentially preventable suffering, and a slice of the cosmos made permanently inaccessible by the expansion of space itself.

For other EAs, the immense harm AI could do made AI safety the most important cause — not just for the 8 billion people on Earth, but for the trillions of beings who might never exist if superintelligence wipes us out.

Many EAs were intellectually complex enough to hold both views at once. Some of these were early investors or engineers at OpenAI before leaving to form Anthropic over safety concerns. Dario Amodei, Anthropic’s CEO, was one of them. These people may or may not actually believe what they say about how it would be better if we could slow the race down. But they definitely believe our chance of surviving superintelligence will be worse if someone else builds it.

All this is to say that EAs are simultaneously some of the people racing hardest for superintelligence and some of the people now trying hardest to slow the race down — or at least talking about it.

EAs are not, as a group, trying to stop AI any more than musicians, as a group, are trying to play jazz. Sure, a lot of musicians love jazz and think the world would be better for embracing it. But even the aficionados are mostly playing more profitable genres instead — sometimes in the name of funding their eventual, someday pivot to jazz.

The Trump administration dislikes Dario Amodei for his principled stances against autonomous weapons, domestic mass surveillance, and — most recently — his calls for industry safety measures that might eventually, someday, slow AI progress a little. Metaphorically speaking, they can tell he likes jazz.

Those trying to prevent our extinction from AI dislike Amodei because he’s still pushing the frontier despite how vocal he is about the risks. Metaphorically speaking, they’re still waiting to hear him actually play some jazz.

EA as boogeyman

Blaming a shadowy cabal for inconvenient developments is part of the standard political playbook, whether it’s a “vast right-wing conspiracy” or the Dominion voting machine conspiracy theory. (Fun fact: “vast right-wing conspiracy” was brought to us in 1995 by Chris Lehane, then a political operative working for the Clintons, now OpenAI’s chief lobbyist.)

When Jacob Coxon’s resignation triggered a sudden swell of public concern about the AI extinction problem, the Trump administration could have chosen to pivot to addressing said problem. It instead chose to stick with a full-speed-ahead agenda. This made it necessary to undermine the extinction concerns, which it did by signal boosting conspiracies already being promoted by industry accelerationists allied with the administration. Some of these theories revolve around AI safety groups run or funded by Effective Altruists. So today, the White House would have you believe that extinction concerns are lies spread by those woke, Luddite EAs.

EAs are mediocre in the boogeyman role. They’re hardly MS-13 or Tren de Aragua — groups which don’t look very good even on their best days. Factions within EA have even welcomed the attention, expecting that rational people looking into them would find much to like.

As cabals go, EAs aren’t very shadowy, either. But like other historically popular scapegoats, EAs are well-networked and well-resourced. If you’re into corkboard and red yarn, you could spend many hours in your shed drawing lines between AI company staff building AI, third-party researchers evaluating said companies, and AI safety orgs opposing them — they are pretty transparent about these connections. You could knit a sweater from the strands running through EA, because EAs got in on the ground floor. That’s what happens when you take AI seriously a decade before people start paying attention.

But even now, after the rogue swarms and the Coxon awakening, the typical, median EA probably isn’t trying to stop AI development. They’re probably not “woke” by any reasonable definition. And they’re definitely not a Luddite. If they’re involved with AI at all, they are probably trying to find a path forward that lets development continue with more testing and guardrails, telling themselves that we can creep closer to the edge without going over, and that everyone will of course pause when the time is right. Eventually. Someday.

And all that jazz.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.