In this issue:
Sanders and Casar propose superintelligence ban - A major policy breakthrough may set the world on course for survival
Congress is right to worry about rogue AI - New bill introduced for technical AI security standards, while OpenAI still has not released full incident records
Drawing a line, 111 times - New survey asks 111 national security experts how much catastrophic risk they’ll accept from AI
A year without - NYC enacts a one-year ban on classroom AI use through 8th grade
Dispatches from Joe
Sanders and Casar propose superintelligence ban
A major policy breakthrough may set the world on course for survival
I am not generally an emotional man, but the news I read this morning moved me almost to tears. Seeing in the Washington Post that U.S. Senator Bernie Sanders (I-VT) and Representative Greg Casar (D-TX) had announced the Ban Artificial Superintelligence Act, I felt a mixture of hope and trepidation. What sort of policies go with the provocative name? Would they be sensible, or slop?
After I read the details, it was hope that won out.
The legislation is forthcoming, so we don’t yet know the full text, but the summary does not mince words. The bill would outlaw attempts to develop or deploy superintelligence, defined as:
An artificial intelligence system that exhibits or can easily be modified to exhibit capabilities that match or exceed human cognitive performance and capabilities across a broad range of domains or tasks, [or] AI systems that have sufficient capabilities to plan and execute the disempowerment of humanity, including by overthrowing or undermining the U.S. government.
The bill would also:
Halt “advanced AI development” until a federal regulatory body develops clear standards, enforced by corporate shutdown and jail time for violators;
Establish “a new cabinet-level federal agency to safeguard the public from the dangers of artificial intelligence” and assign an advisory board of AI experts; and
Set U.S. international policy on a path towards international agreements that make this ban global.
I am impressed. I have some concerns — we’ll get there — but this is the first piece of proposed AI legislation I’ve seen that might drastically improve humanity’s chance of survival.
The proposals are mostly common sense: We should not build something smarter than us that we lack the means to understand or control, and we should stop private companies who attempt this. AI technology is far too important and dangerous not to have a dedicated federal agency, and a panel of knowledgeable experts would help non-technical authorities set better policy. Of course, a unilateral U.S. halt isn’t enough when other countries continue, so international coordination is urgently needed.
Some open concerns temper my enthusiasm. First of all, I’m aware this is likely something of a messaging bill, unlikely to pass in the current session of Congress. But I’m going to take its message seriously, and look at what we know from the summary so far.
The definition of superintelligence looks reasonable as written, but “advanced AI development” is much less so, and may be overbroad. I still respect the decision to err on the side of caution while no serious regulation exists.
Speaking of regulation, I’m unsure whether the regulatory body and the “cabinet-level agency” are intended to be separate entities. My guess is that they are, which means that a fully fleshed-out bill ought to include explicit language and a timeline for creating both.
It makes sense that the proposed federal agency would “Monitor frontier AI systems at all stages of the lifecycle” and “Supervise the removal of dangerous capabilities,” but I’m not sure what is meant by the third responsibility, “Supervise the destruction of artificial superintelligence.” The main way humanity defeats a superintelligence is by not building it in the first place. But I can imagine some scenarios where a nascent, dangerous AI, not yet fully superhuman, exists alongside a government-ready kill switch, and it does make sense that this agency would be charged with the shutdown.
I would also make the agency — or perhaps the regulatory body — responsible for research into technology that can verify international agreements. A key part of such deals is making sure everyone keeps them, like the International Atomic Energy Agency helps do for nuclear nonproliferation agreements.
The proposed international policy is thin on details. Understandable for a one-page summary, but a real policy will need more flesh. Starting off with a unilateral U.S. halt, without negotiating a deal with China first, may not be the best method; but it still seems better than the current hands-off approach, and it might inspire China to treat superintelligence with similar gravity.
In 2024, my colleagues in the MIRI technical governance team drafted a more gradual plan for securing international coordination, published in an appendix to a proposed international agreement. (Disclosure: I also had a hand in editing both.) It’s only an example, though, and there are many different paths the world might take to a global halt.
Ultimately, even if the proposed bill dies in committee, it still represents a massive breakthrough in sane AI policy, and a huge leap relative to what was being considered a couple years ago. We are finally beginning to see what it looks like for the AI problem to be treated with respect, at least from the policy side.
I look forward to seeing the full text.
Congress is right to worry about rogue AI
New bill introduced for technical AI security standards, while OpenAI still has not released full incident records
Axios covered another bipartisan AI bill that was introduced today by U.S. Representatives Josh Gottheimer (D-NJ) and Mike Lawler (R-NY). Within a year of its passage, the Stop Rogue AI Act would direct the National Institute of Standards and Technology (NIST) “to develop and publish standards, guidelines and best practices for how organizations can securely deploy AI agents.”
Per Axios, the required standards would concern evaluating AI agents, monitoring and logging their actions, protecting those logs from tampering, and tracking agent populations.
I see this as a positive development. As I’ve previously written, the AI industry has long lacked the robust technical standards that characterize mature industries.
This bill is one of several introduced after news of the Hugging Face attack smacked into Washington’s AI complacency like a wrecking ball. Others include the AI Kill Switch Act and the Secure AI Development Act, which my colleague Donald covered in July, and the FRONTIER Act.
Together, these bills indicate to me that U.S. leaders are rapidly waking up to the threats posed by AI. Many of their provisions are worthwhile in their own right, although (with one notable exception) they fall short of a full-blown halt.
On a related note, in early August U.S. Representative Greg Casar (D-TX) sent a public letter to OpenAI demanding answers on the Hugging Face incident, including a detailed log of the attacks and the events that led to it. OpenAI responded with a letter mostly rehashing its blog post about the incident and promising to monitor its own AI systems, to record their actions, and to develop unspecified “additional automated shutdown capabilities.”
Casar, evidently unimpressed, replied yesterday citing many of the same concerns I mentioned last week:
Providing hand-picked investigators six days of supervised access is not public release, and those investigators themselves flagged that they could not rule out errors in their AI-assisted analysis. I am deeply concerned about the limited scope of that investigation.
Casar goes on to cite a long list of questions from the original letter that remain unanswered, like “How many times OpenAI models have obtained unauthorized access to the internet” or “What became of the credentials and data the models took.”
I am deeply glad to see more of our leaders taking the risks seriously. Unfortunately, the window they have to act is narrowing, partly because dangerous capabilities proliferate. For example, last week the Chinese AI company Z.ai released the weights of GLM-5.3, an open model with advanced cyber capabilities (though not quite, as the company would have us believe, a rival to Anthropic’s Fable). Z.ai delayed the release for two weeks (an unusual occurrence), to complete “safety evaluation and hardening.”
Within days of the release, a different company scrubbed the safeguards out of that same model and released a guardrail-free edition, boasting on social media about “offensive cyber” capabilities and their server’s lack of logging.
As I put the finishing touches on this dispatch, OpenAI announced the staggered release of its latest model Astra. Open models tend to lag behind American frontier AI by around four months. Whatever frightening capabilities we see from Astra, they may be present in a guardrails-free open model by spring.
If the “Stop Rogue AI Act” passed Congress and was signed into law this very day, we’d still have to wait an entire year for any standards to kick in. At the rate AI companies are currently racing towards an unseen brink, that’s practically a lifetime. For this and other reasons, the most urgently needed AI policy is to slow the heck down.
Dispatch from Donald
Drawing a line, 111 times
New survey asks 111 national security experts how much catastrophic risk they’ll accept from AI
Yesterday, the Institute for Security and Technology released a survey of what national security professionals expect from advanced AI: the hazards, the opportunities, and how much risk they think is worth running. The 111 people surveyed were former Under Secretaries and Assistant Secretaries, deputy assistant secretaries, NSC senior directors, senior diplomats, intelligence leaders, retired lieutenant generals and major generals, plus the academics and technologists who work alongside them. Eighty-seven percent currently serve or have served in the U.S. government.
The questions were unusually precise: Most polls will ask, “How worried are you?” and then I’ll have to write that such-and-such a number were quote-unquote “worried or highly worried.” Questions in this poll read, “What is the probability that an AI-caused global catastrophe (defined here as an event causing the death of more than 10 million people) occurs before 2050?” and “What is your estimate of the likelihood that AI-enabled capabilities could contribute to a global catastrophic event leading to human extinction (or irreversible civilizational collapse) within the next 50 years?” 40% of respondents put the odds of ten-million-deaths at 5% or higher, and 26% put the odds at 10% or higher. On extinction or irreversible collapse, a majority came in under 1%, but 28% said 1% or higher, and 16% said 5% or higher.
A third question asked what probability of such a catastrophe “would be unacceptable in exchange for the economic and strategic benefits of AI leadership.” Respondents’ answers were set against their estimates of the actual risk: Among the 56 who gave concrete answers to both that question and the ten-million-deaths question, 61% put the risk at a level they had marked as unacceptable. The same precision shows up on loss of control. Asked for the probability that AI operates outside human control within ten years, the median answer was that there is a 33% chance that humans lose control. 87% of respondents put it at 10% or higher.
The sample may, in the report’s own phrasing, “overrepresent people who view AI as an important policy issue.” Respondents came through professional networks and referrals, and didn’t have to answer everything (hence my qualifier about “56 respondents”).
Another thing: The responses were collected between April 30 and July 15, so the Hugging Face attack didn’t influence anyone’s response. Polls are fuzzy and track moving targets, but they can still give useful information. My takeaway is that, even before the events of July and August, a large share of the people who have spent careers pricing risk for the U.S. government had already run the numbers, counted the upside, and deemed the threat unacceptable.
Dispatch from Alana
A year without
NYC enacts a one-year ban on classroom AI use through 8th grade
New York City has banned classroom AI use for one year, for students in grades 2-K (2-year-olds) through 8th grade. This means the AI components of over 38 tools previously used in the classroom will be “disabled or discontinued,” Mayor Mamdani told Politico. According to the New York Times, teachers also will not be allowed to ask AI to grade assignments, to decide whether a student should graduate, or to advise on crisis counseling.
Politico calls the policy “one of the most aggressive a major school district has taken to date,” though some parents wish it had gone further, such as covering all grades or lasting longer than a year.
Mamdani says the one-year ban will be evaluated after it ends and possibly extended or amended. I was interested to learn, elsewhere in the article, that NYC “initially blocked student and teacher access to ChatGPT in 2023 on Department of Education devices or internet networks, citing learning and safety concerns” but “reversed its position months later.”
I’m curious to see if NYC can demonstrate concrete benefits or disadvantages after this one-year test — a task that seems pretty difficult. Evaluating changes to education practices is always tricky, since effects don’t always show up right away, and there aren’t always clear benchmarks to compare to. If NYC is lucky or skilled enough to get clear results, these could inform decisions in other cities or states, making a chaotic mess of policies slightly more data-based.
As with other disruption-prone areas — including elections, hiring, health care, policing, and deepfakes — the absence of federal legislation means states and municipalities are largely on their own to navigate AI disruption as best they can. It would be nice if they could start sharing some data, even if limited.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.