Of dawns and miracles
A proposal from Demis Hassabis, a warning from Daniel Kokotajlo, and a data center moratorium in New York
In this issue:
The risks of making sand think - Google DeepMind’s Demis Hassabis pitches an AI regulatory regime
A chilling conversation - Daniel Kokotajlo takes the AI risk discussion mainstream
New York imposes one-year moratorium on data centers - More Americans oppose local data centers than nuclear power plants, Gallup finds
Dispatches from Robert
The risks of making sand think
Google DeepMind’s Demis Hassabis pitches an AI regulatory regime
Exactly one month ago, Dean Ball, former White House AI adviser and current Head of Strategic Futures at OpenAI, wrote that the U.S. has an informal licensing regime for AI, albeit one without consistent rules or firm boundaries. This morning, Demis Hassabis published his proposal for a formal version.
The CEO of Google DeepMind proposes that a U.S. standards body for AI be created, similar to the Financial Industry Regulatory Authority (FINRA), which oversees Wall Street. In an interview with Axios, Hassabis said he would like to see this board be majority-independent and staffed with Turing Award winners and other experts.
This new body — and the benchmarks it develops — would then determine which models qualify as “frontier-class.” Stricter guidelines would apply to the “Frontier Labs” behind these models, for example, regarding security screening of personnel and safety budgets. The Frontier Labs would also be required to have all new models tested by the new body 30 days prior to release, with a focus on cyber, bio, and deception risks. Passing these tests would eventually become a prerequisite for releasing the model on the U.S. market. The system is designed to regulate frontier models from around the world without stifling startups or academic research.
Credit where credit is due: This is probably the boldest and most concrete proposal for a regulatory regime a frontier lab CEO has made to date. Hassabis has given thought to the fact that such a licensing regime must be as universal as possible and that the benchmarks must be designed in such a way that the frontier labs cannot specifically train their AI for them in advance. This is important so AIs don’t adopt behavior that maximizes training scores rather than what the engineers wanted those scores to represent, as we recently discussed.
Above all, however, he mentions quite casually and hidden in the middle of a paragraph that one task of the institution he proposes could be to coordinate a slowdown of the race, “if the seriousness of the situation demands.” This seems surprisingly vague given that it’s one of the most important parts of his proposal. Who would make this call? How would that happen? And is the situation not already serious enough?
Let’s take a look at how he describes the situation we currently find ourselves in:
At the moment, we are locked in an extremely intense, multilayered commercial and geopolitical race. While these competitive dynamics fuel rapid progress and accelerate the incredible upsides, advances on the frontier are outpacing our understanding of the technology. Nobody in the world knows for sure what is going to happen from here, and even the experts disagree.
He goes even further. Hassabis writes that while he believes the technical risks are solvable, this is only possible if industry and society give themselves the necessary time — and that we are currently failing to do so. It is the strongest and most compelling passage in his post, and it refutes his own proposal. After issuing this clear warning, predicting ten industrial revolutions at ten times the speed for the next decade, discussing recursively self-improving AIs, admitting that no one knows exactly what lies ahead… his proposal is to proceed with “cautious optimism,” a voluntary 30-day preview testing window and a new benchmark?
Furthermore, a race dynamic cannot be resolved unilaterally. Sure, on paper, this proposed U.S. body is supposed to have universal jurisdiction over all models. But would China’s Z.ai submit GLM 5.2 for 30-day preview tests just to gain access to the U.S. market? Would Beijing let them, even if they wanted to?
Hassabis describes the possibilities of AI in poetic terms. He speaks of “dawns” and “miracles,” the “foothills of singularity,” and about “making sand think.” He speaks of the risks in far more prosaic terms. But the brighter the light, the sharper the shadow. If we take seriously what he says about what’s at stake, then it seems necessary that we take the time to get this right. Hassabis has written a strong case for stopping and stapled it on a plan to race on. There is no such thing as racing cautiously. They taught sand to think and now it’s running through our fingers.
A chilling conversation
Daniel Kokotajlo takes the AI risk discussion mainstream
This week, Daniel Kokotajlo, the AI researcher leading the AI Futures Project, appeared on the highly popular YouTube channel The Diary of a CEO. For two hours, he spoke with host Steven Bartlett about AI 2040, his team’s new scenario that we’ve already covered several times.
“The core problem is that people aren’t taking it seriously yet,” says the former OpenAI governance researcher, using his two hours in a friendly space to present his warnings to a wide audience. Bartlett makes a noticeable effort to give Kokotajlo the moral weight he deserves. He explicitly mentions the $2 million in equity that Kokotajlo was prepared to walk away from because he refused to sign OpenAI’s anti-disparagement clause, which would have prohibited him from criticizing the company after he left.
Kokotajlo sees a 70% chance of a catastrophic outcome from current AI developments, and he seems able to convey the gravity of the situation. In a particularly striking moment, Bartlett speaks with Kokotajlo about the personal consequences he draws from his predictions. When Kokotajlo describes how carefully he and his wife have weighed whether to have children because of his shortened timelines, Bartlett calls it “chilling” — an assessment he repeats several times.
In his closing message, Kokotajlo warns against dismissing the dangers of artificial intelligence simply because they sound like science fiction.
You’re going to hear a lot of things and you already have been hearing a lot of things about AI and it’s going to sound like science fiction. But sometimes things which sound like science fiction happen in reality. And in fact, many times, historically, things that used to be science fiction have then become reality.
Of course he is right about that. Not too long ago, AI itself was still science fiction. And just because H. G. Wells first conceived of “atom bombs” in a 1914 novel doesn’t mean the danger posed by nuclear weapons was any less real a few decades later.
All in all, Kokotajlo managed to convey both how dire the situation is and that there’s still hope.
BARTLETT: Do you think it’s too late?
KOKOTAJLO: No, I don’t think it’s too late.
KOKOTAJLO: If I thought it was too late, I wouldn’t be here.
BARTLETT: Where would you be?
KOKOTAJLO: With my family.
My colleague Mitch wrote last month, that some disasters are built one excuse and one rationalization at a time. Hopefully we can also achieve prevention one chilling conversation at a time.
Dispatch from Donald
New York imposes one-year moratorium on data centers
More Americans oppose local data centers than nuclear power plants, Gallup finds
New York has become the first U.S. state to impose a moratorium on data center construction. Governor Kathy Hochul signed an executive order pausing environmental permits for facilities that consume at least 50 megawatts. The executive order lasts for one year, and is meant to give regulators time to write standards on energy demand, water use, and other issues. New York’s state legislature recently passed its own one-year moratorium, targeting facilities that consume 20+ megawatts; at this time, Hochul has neither signed nor vetoed that bill.
Governments are not all reacting to data centers in the same way, as noted by Reuters. Earlier this year, Governor Janet Mills of Maine vetoed legislation that would have imposed an eighteen-month moratorium on facilities that consume 20+ megawatts. In the Netherlands, Amsterdam banned new data centers and expansions to data centers until 2030; a similar freeze around Dublin, Ireland, in place since 2021, was recently lifted.
Whether their concern is water usage, noise pollution, or something else, ordinary people strongly dislike data centers. A March Gallup poll found that 71% of Americans oppose data centers in their community, more than oppose living near nuclear power plants. This is bipartisan opposition: Democrats, Republicans, and Independents were all more likely to oppose than support data centers. It is also a strong opposition: For comparison, “since Gallup first asked the nuclear power plant question in 2001, the high point in opposition has been 63%.”
While congressional support for a data center moratorium is concentrated among Democrats (and Bernie Sanders), this doesn’t mean that voters will organize themselves along those lines. Opposition to data centers, to quote Charlie Berens, is “the most bipartisan issue since beer.”
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.