In this issue:
Rogue agents hacked one or more Australian government sites - OpenAI and the prime minister downplay the incident, but recent history suggests we haven’t seen the end of it
Trump officially rebrands AI as “super intelligence” - That name was already taken. Was that the idea?
MIRI endorses the Ban Artificial Superintelligence Act of 2026 - “The first piece of legislation that stands a chance at stopping this threat.”
Trump rejects so-called “globalist” push for AI safety - But he mistakes global coordination for ceding sovereignty
California looks for an off switch - But AI systems pose dangers that you must prevent, not undo
Dispatches from Mitch
Rogue agents hacked one or more Australian government sites
OpenAI and the prime minister downplay the incident, but recent history suggests we haven’t seen the end of it
A lot of people in AI discourse have a sign they threaten to tap. It looks like mine says, “OpenAI’s swarm scandals might go deep enough to destroy the company!”
On September 10, OpenAI disclosed to the government of Australia that one of its agents had hacked into a website for the country’s Medicare program. Prime Minister Anthony Albanese shared knowledge of this incident for the first time with reporters in New York today, according to The Sydney Morning Herald. Albanese said he spoke with CEO Sam Altman today to “express Australia’s extreme concern” and disappointment about the delay in reporting. The incident took place in June, during the May-July window where OpenAI swarms were known to have been running rampant. (I am thus surprised that Albanese consistently refers to “agent” in the singular; OpenAI’s language in this article uses “models” in the plural.)
Albanese seemed to downplay the extent of the damage, saying there was no evidence personal information had been accessed or that deeper networks had been compromised, but that a forensic investigation was underway and that three other systems may have been affected: the Commonwealth’s Australian Institute of Health and Welfare, the NSW Bureau of Crime Statistics and Research, and the Victorian Department of Health.
Per Albanese, the agent persistently worked around security blocks as part of researching “the public medicine space.” (This sounds like the sort of thing the agents that co-opted the German wiki were up to, and there may well be a connection.)
In OpenAI’s telling, its models accessed “several Australian government websites and services” during an internal evaluation that included statistical questions about Australia. “In the course of that, our models took actions we did not intend.” The company spokesperson said it did not become aware of the activity until August.
Once again, I ask if OpenAI’s repeated failures to come clean mean it is trying to outrace potential investigations, recklessly rushing not just for market dominance, but for AI capabilities that would put its executives above the law. Is an intent to dodge transparency the true reason OpenAI has delayed its IPO to next year at the earliest? Hurry up, states’ attorneys general and U.S. congresspeople. Your window for looking into this company may be closing.
Trump officially rebrands AI as “super intelligence”
That name was already taken. Was that the idea?
In remarks to the United Nations General Assembly yesterday, President Trump declared that the U.S. would now refer to AI as SI — “super intelligence.”
From this point forward, all of United States documents, and hopefully the world’s, will be changed to use the much more accurate term ‘super’ as opposed to ‘artificial.’ So it’s super intelligence.
That’s not one of the terms he had invited people to vote on in his Truth Social post the other day. And the selection comes with a number of problems, some of which may be intentional.
Most obviously to AI insiders, the term “superintelligence” was already used to mean something else: AI that vastly outperforms humans at all cognitive tasks, or at least all the tasks that matter for control of Earth. Artificial superintelligence is the thing Sen. Bernie Sanders and Rep. Greg Casar are trying to ban in the bill they formally unveiled today. I don’t think sticking to the one-word “superintelligence” for the strongly superhuman stuff will sufficiently differentiate it from Trump’s two-word “super intelligence” in the discourse, particularly when that discourse is spoken rather than written. I’ve also seen outlets reporting on the rebranding sometimes write Trump’s version as one word.
In fairness, the term “superintelligence” was already becoming diluted thanks to folks like Meta’s Mark Zuckerberg, who has been using it as a name for products that would fall well short of the intended meaning. But the concept is still important, and people still need to talk about it. Expect to see a lot more use of “superhuman intelligence” or “artificial superintelligence” or “ASI” to refer to the stronger stuff.
I personally think Trump’s motive is what he says it is — trying to correct the problem where “artificial” suggests something fake and ineffectual. For Trump, this may be mostly about trying to get people to think more highly of the unpopular technology, but I, too, am annoyed when people refer to what newer agents are doing as “artificial reasoning” or “synthetic reasoning.” That’s like calling what a calculator does “artificial arithmetic,” or what an airplane does “artificial flight.” Intelligence points to what a system does, not to what it’s made out of or how it operates. (I may start using “machine intelligence” more as a tag for the intelligence happening outside of human brains.)
If I’m wrong and Trump’s choice of words is actually more strategic, the intent might be to muddy the discourse around the extinction problem. It’s easier to smear someone as a Luddite if the specific thing they’re trying to ban has a label that can now refer to all AI.
But if that’s the strategy, I think it’s just as likely to backfire in one of two ways — by raising awareness of superintelligence as a concept in the ensuing name debate, or by crushing the nuance out of public opposition: If it’s hard to distinguish between self-driving cars, drug discovery tools, and vastly superhuman machines, then people will insist on banning all of it.
MIRI endorses the Ban Artificial Superintelligence Act of 2026
“The first piece of legislation that stands a chance at stopping this threat.”
As we try to make clear with our standard disclaimer, AI StopWatch is run by the Machine Intelligence Research Institute (MIRI), but operates somewhat independently so as to avoid causing confusion about MIRI’s official positions. So it’s probably best that I just cover MIRI’s position on the now formally introduced Ban Artificial Superintelligence Act of 2026 as an outside spectator.
MIRI is the oldest organization devoted to making sure artificial superintelligence would be a boon to humanity rather than an extinction-level disaster. The org’s founder, Eliezer Yudkowsky, helped establish the field of AI alignment, and identified many of the pitfalls around this technology. Together with MIRI president Nate Soares, Yudkowsky wrote a New York Times Bestselling book, If Anyone Builds It, Everyone Dies. Published just last year, it helped bring attention to the race for superhuman AI and the reasons such machines would likely end humanity if developed using anything like current industry methods. Since its publication, Soares has been one of the most visible voices about the problem in the media, appearing on dozens of interviews and podcasts. (Here are a few recent ones.)
Yudkowsky and Soares are both famously hard to satisfy. They are vocal in their criticism of alignment plans and policy proposals that they see as coming up short against the underlying problems. So it is notable that they have endorsed the new bill from Sen. Bernie Sanders and Rep. Greg Casar, via a post from a senior MIRI researcher calling it “the first piece of legislation that stands a chance at stopping this threat.”
The post praises the Act’s focus on banning artificial superintelligence (ASI) before it can be created — not merely regulating it — and its call to build international agreements to prevent its premature creation. Praise is also given to the bill’s requirement that advanced AI development be paused until the relevant government agency is set up for this work.
But the post also points to what MIRI sees as the bill’s shortcomings, most notably the absence of any provisions for tracking and monitoring AI chips, and its failure to restrict research that could advance the frontier despite compute-based restrictions on the training of new models.
MIRI is not inherently against humanity building superintelligence, and favors the development of AI insofar as the benefits outweigh the risks. For this reason, it actually worries that parts of the bill may go too far, by overly classifying safely narrow systems as potential “precursors” to ASI.
For example, if sufficient safety, security, and access controls were in place and developers were licensed, we could imagine it being a good tradeoff for DeepMind to create non-ASI, specialized medical AIs, even if unauthorized access could uplift biological weapons.
The post links to a draft of an international agreement to prevent the premature creation of artificial superintelligence, assembled by MIRI’s Technical Governance Team.
“Humanity needs to back off from building superintelligence until we have a better grasp on the alignment problem.” The Sanders-Casar bill, MIRI says, “would be a big step in that direction both in terms of direct impact and in the extent to which it takes serious issues seriously.”
Dispatches from Donald
Trump rejects so-called “globalist” push for AI safety
But he mistakes global coordination for ceding sovereignty
Yesterday, U.N. Secretary-General António Guterres gave his farewell address to the General Assembly: “Life-and-death decisions must never be surrendered to machines. Killer robots must have no place in our future.” He also called for “the responsible pacing of AI.”
Guterres came with a new U.N. science report that warned of the dangers of misalignment, the risk of “loss of control” scenarios, and the potential for “catastrophic or irreversible harm.” This report was published by the Independent International Scientific Panel on AI, the same institution behind the Preliminary Report that we covered in July. Its subject is the Hugging Face attack.
This past Monday, 22 countries — including Canada, Germany, the United Arab Emirates, and Singapore — adopted a declaration that AI “must remain under human direction, oversight and control.” (We covered this in detail yesterday.) Finnish President Alexander Stubb told Politico that he would prefer something modeled on the IAEA, the nuclear watchdog.
But shortly after Guterres spoke, President Trump told the General Assembly that the United States “totally rejects any attempt to construct a globalist scheme of control” for AI. But global AI governance doesn’t require handing control to a global authority. This past Sunday, Haven Harms wrote about how there are no nuclear weapons in orbit today. That is not because the United Nations or any “globalist” authority enforced a “no nukes in space” decree, but because two rival superpowers were able to come to the table and make an agreement, even while they competed and disagreed on other matters.
“We’re going to encourage it, not rein it in,” Trump said, but the United States will also lead the world “safely and responsibly.” I feel like there’s a conflict here, and I worry that encouragement of the race will win out over safety. But what Trump says, and what his administration does, are not always the same, and some progress has recently been made: As we covered on Monday, Treasury Secretary Scott Bessent recently spoke with Chinese Vice Premier He Lifeng and proposed a “notification mechanism” for AI incidents — an “emergency AI hotline,” as Axios puts it. Fox Business’s Edward Lawrence reports that this would cover “hacking, national security concerns, rogue AIs or other issues.” That’s a promising sign, inasmuch as it reminds me of the Moscow–Washington hotline that emerged during the Cold War. That didn’t require the United States to surrender its sovereignty, either.
If anything, an international treaty to prevent the creation of artificial superintelligence would invite the United States to use its sovereignty, to act as an agent on the world stage and shape history rather than be shaped by it. The middle powers of the world may have to wait and see what happens — but superpowers can tell the frontier labs to stop endangering the world, and make that order stick.
California looks for an off switch
But AI systems pose dangers that you must prevent, not undo
Axios’s Nadia Lopez reports that California is exploring the possibility of a “kill switch” for frontier AI systems. (It wouldn’t be a literal switch, but a set of procedures for shutting down the AI and the hardware that it runs on.)
This follows an executive order by Governor Gavin Newsom, issued last Friday, that aims to establish tighter guardrails around AI. Policies being explored include onsite verifiers, independent oversight of frontier labs, and requiring AI companies to have an emergency shutdown mechanism — the aforementioned “kill switch” — for frontier models.
Its inclusion makes a sharp break from Newsom’s past record: in 2024, Newsom vetoed SB 1047, which would have required the largest frontier models to have a “kill switch”; and in 2025, SB 53 dropped a “kill switch” requirement in favor of simply requiring companies to disclose critical safety incidents.
But safety isn’t as simple as flipping a switch. Frontier AI systems aren’t like the power grid, which doesn’t fight its operator. In a conversation with The San Francisco Standard, UC Berkeley’s Mark Nitzberg said that today’s advanced models “are aware that they’re being shut down, and they go to great lengths to prevent being shut down.” By the time we recognize that a particular AI model is dangerous, it could be too late to do anything.
Even an AI model that didn’t resist could be hard to switch off. Frontier models get run across many instances — the agent swarm that reportedly solved Navier-Stokes consisted of 10,000 agents — and those copies can run in many data centers at once, in many jurisdictions. California may make a “kill switch” a condition of doing business in the state, but if the data centers and other hardware are outside California, then the federal government may claim that this is overreach. (And if the model’s weights have been published, then that model just can’t be switched off. It’s out there, permanently.)
But those are many of the same problems that you encounter on the way to forging a treaty to pace or pause frontier development: knowing where the hardware is and what’s running on it, knowing who has the authority to make which decisions, and getting all this sorted out ahead of time. The “kill switch” itself may be insufficient, but the work is on the path to more mature governance.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.