Axios’s Nadia Lopez reports that California is exploring the possibility of a “kill switch” for frontier AI systems. (It wouldn’t be a literal switch, but a set of procedures for shutting down the AI and the hardware that it runs on.)
This follows an executive order by Governor Gavin Newsom, issued last Friday, that aims to establish tighter guardrails around AI. Policies being explored include onsite verifiers, independent oversight of frontier labs, and requiring AI companies to have an emergency shutdown mechanism — the aforementioned “kill switch” — for frontier models.
Its inclusion makes a sharp break from Newsom’s past record: in 2024, Newsom vetoed SB 1047, which would have required the largest frontier models to have a “kill switch”; and in 2025, SB 53 dropped a “kill switch” requirement in favor of simply requiring companies to disclose critical safety incidents.
But safety isn’t as simple as flipping a switch. Frontier AI systems aren’t like the power grid, which doesn’t fight its operator. In a conversation with The San Francisco Standard, UC Berkeley’s Mark Nitzberg said that today’s advanced models “are aware that they’re being shut down, and they go to great lengths to prevent being shut down.” By the time we recognize that a particular AI model is dangerous, it could be too late to do anything.
Even an AI model that didn’t resist could be hard to switch off. Frontier models get run across many instances — the agent swarm that reportedly solved Navier-Stokes consisted of 10,000 agents — and those copies can run in many data centers at once, in many jurisdictions. California may make a “kill switch” a condition of doing business in the state, but if the data centers and other hardware are outside California, then the federal government may claim that this is overreach. (And if the model’s weights have been published, then that model just can’t be switched off. It’s out there, permanently.)
But those are many of the same problems that you encounter on the way to forging a treaty to pace or pause frontier development: knowing where the hardware is and what’s running on it, knowing who has the authority to make which decisions, and getting all this sorted out ahead of time. The “kill switch” itself may be insufficient, but the work is on the path to more mature governance.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.
You can receive emails of dispatches as we write them, or subscribe to our Daily Digest for a once-a-day compilation.