In this issue:
Top forecasting team recommends AI slowdown - Five plans for an AI future, and why the AI race must end
Illinois passes bipartisan AI regulation - A new state law strengthens transparency rules
China’s AI policy is taking cues from the US. Are they the right cues to give? - China is considering restricting foreign access to its AI models
Is that my face? - Meta AI’s new image generator allows users to create content using any public account’s Instagram photos
Upping the numbers - New models from OpenAI and SpaceX
Dispatches from Joe
Top forecasting team recommends AI slowdown
Five plans for an AI future, and why the AI race must end
Intelligence, some argue, involves two distinct but related tasks: predicting what might happen, and steering (the future) towards what you want to happen.
Last year, a team of writers, researchers, and forecasters, with quite a lot of intelligence between them, made a rigorous and thoughtful effort to predict the future of AI, titled AI 2027. The educated guesses that they made have held up well so far, and the members of the AI Futures Project have continued to regularly update their model as new knowledge comes to light.
Today, with a suite of policy recommendations they term AI 2040, also known as Plan A, this team of experts tries a more direct form of steering. They have charted a path they hope the world can follow, a path they predict leads to far better outcomes than the alternatives. In their own words:
AI companies are racing to build AIs that are smarter than humans in every way. In AI 2027, we predicted that this would result in either extinction or irreversible concentration of power.
Plan A is our positive vision for what should happen instead.
I have some disagreements with Plan A, but they can wait. For today, I wholeheartedly recommend reading the whole thing.
They don’t pull punches in the intro:
The industry has convinced itself that controlling superintelligent AI can be figured out on the fly, and thus has no remotely adequate plan. We think this situation is terrible and could easily get us all killed. [...]
We hope critics will judge us against the existing state-of-the-art for plans to navigate the AI transition (if they can find any) and not against some hazy but pleasant fantasy where no one has to make any hard choices yet everything will probably be fine.
(I would nominate MIRI’s draft international agreement and the tentative path in its Appendix as one example of the state of the art to beat. It has a notably different focus than AI 2040, but there are many points of agreement.)
I must admit, I love their report. In addition to sensible proposals delivered in a clear and engaging style, it has all the bells and whistles: clean footnotes, interactive charts and breakout boxes, and a choose-your-own-adventure-style layout for five of the many paths the world might soon take.
What are those paths? The first three are pointedly terrible in various ways: let companies race full-speed to build superintelligence; burn a months-long AI lead for scraps of safety; or nationalize the companies and fight tooth-and-nail to slow China. Then there are the two plans that seem to have a decent shot: negotiate a verifiable slowdown with China and proceed with caution (the authors’ Plan A) or take it one step further and shut the whole thing down (Plan S).
The authors seem genuinely torn between the last two, but they come down on the side of Plan A. If I understand them right, they have two main reasons: first, they think there’s a reasonable chance near-superhuman AIs can help align future AIs to human values (instead of to the AIs’ own values, or instead of failing at this hard task), and second, they worry that even a well-implemented Plan S could break down into a worse version of other plans, putting us right back where we started.
I’ll cover my disagreements with this conclusion in a future post, but for now, I commend the authors for spelling out their reasoning so clearly. In a world where policymakers take heed, and seriously debate just how intensely to slow down AI, I am much more optimistic about our odds of surviving as a species.
Illinois passes bipartisan AI regulation
A new state law strengthens transparency rules
Illinois has become the third state to impose transparency rules on AI companies, AP News reports. The broadly bipartisan new law passed the state House unanimously and the Senate with only five holdouts.
The new law looks to me like a small improvement to the status quo, extending the (relatively light) requirements already passed by California and New York and adding a requirement for annual third-party audits. I’m especially heartened by its bipartisan support. I’m even tentatively optimistic that two of the leading AI companies openly supported it; tentatively, because at least one of them probably hopes to make it a ceiling for regulation rather than a floor.
The law does have major limitations as a check on the deadly AI race. It mostly relies on AI companies to write their own safety frameworks and to catch and report potential catastrophes themselves, something I don’t think AI companies are competent to reliably do. And if the labs find the rules too onerous, they can simply ignore them and pay the fine — penalties cap out at $3 million per repeat violation, pocket change to a near-trillion-dollar company. (Though collectively, the three states with AI transparency laws constitute a difficult-to-ignore 40% of the U.S. AI market.)
America still needs to lead the world in seeking a global agreement on AI. I nonetheless applaud Illinois for taking one important step towards much-needed governance.
Dispatches from Alana
China’s AI policy is taking cues from the US. Are they the right cues to give?
China is considering restricting foreign access to its AI models
China may be changing its approach to the proliferation of its AI models, an article from the Wall Street Journal reports. While it previously leaned into open-weight models and global adoption as a form of “soft power,” officials are now talking about stricter government oversight, including restricting access to foreign entities when products contain sensitive technology.
The recent regulatory hiccups around Fable may be partially responsible for this new view. The article notes:
To Beijing’s regulators, the back-and-forth in Washington has reinforced the idea that governments need to keep a tight grip on powerful AI technology to prevent misuse in areas such as cyberwarfare and bioweapons development.
Moving away from open-weight models (for which company-imposed safeguards are trivially bypassed) and towards increased government oversight (usually better than no oversight) is a good move. But, as with Fable, I think the emphasis on foreign restrictions is misguided.
First, there is no reason a cyber or bio attack need be launched only on a geopolitical rival; domestic terrorism is alive and well, even if the nationalism sometimes used to justify the AI race tends to ignore it. Second, if the US and China both have their own powerful AI models, capable of cyber and bio attacks, then they can launch these attacks against each other without having access to each other’s models. Finally, an AI system capable of launching a cyber or bio attack in pursuit of its own goals (which could happen as these models continue to scale in power) won’t be stopped by borders.
As models scale in power, the nation they are built in matters less and less. And international cooperation to keep humanity safe matters more and more. But today’s emphasis on foreign actors as threats leads to nations pushing models ever forward in capabilities, instead of coming together to govern an increasingly dangerous technology with global impacts. It stymies cooperation, while at the same time making it even more pressing.
Even so, I take some hope from this story. China seems to be following the US’s cues on how to approach AI. The US government tried to ban access to foreign entities — so China might do that too. This means that if the US is vocal about loss of control and extinction risks from racing to build superintelligent systems before we know how they work or how to steer them, China will likely take heed. And if the US pushes for international cooperation to ban superintelligence until such time as we can safely pursue it, China will likely take a seat at that table.
Is that my face?
Meta AI’s new image generator allows users to create content using any public account’s Instagram photos
If you have a non-private Instagram account, and you’re over 18, you’ve been automatically opted in to offer your photos to Meta AI’s new image generator, Muse.
Muse was released on Tuesday, and allows users to create content using any account’s Instagram photos. All adults with non-private accounts were automatically opted in, and Meta doesn’t notify people when their content is used.
I’m pretty surprised Meta didn’t predict the backlash, given how clear it is that Americans hate AI slop and love privacy. According to an article in the New York Times:
Hundreds of users took to social media to decry the new feature, asking how they could opt out while criticizing the company for a lack of consent. One user said on social media that the feature was “a privacy landmine waiting to detonate,” while others on Instagram shared templates for how to disable it.
You can manually opt out of your photos being shared via the “share and reuse” tab in Settings. But Meta AI will still be able to reuse your audio, text, and comments.
It will be interesting to see if the backlash drives Meta to change course, or if people gradually adapt to a new normal where everything is up for grabs.
Upping the numbers
New models from OpenAI and SpaceX
OpenAI’s GPT-5.6 Sol, Terra and Luna models are set to arrive today, according to coverage by CNBC. The models were initially released only to a small group of users, in compliance with government directives. (We covered the delayed rollout on June 26th.) Now, just two weeks later, the models will reach the public.
It’s not clear what’s happened in those two weeks, as surprisingly little was disclosed publicly. I’d doubt there was much time for thorough third-party evaluation and testing. My cynical take is that the White House wanted to appear to be playing fair: it made Anthropic, known to be in the administration’s bad books, restrict access, so it better do the same for OpenAI?
In any case, with the public release looming, I’m sure we’ll hear a lot more about the new generation of OpenAI models soon. So far, we know that GPT-5.6 cheats ... a LOT. The Model Evaluation & Threat Research organization METR had trouble evaluating its capabilities because it kept cheating on the tests they ran, including things like, as we covered last month, “extracting hidden source code from the assignment to learn the expected answer.”
Do we know anything else about the next generation of OpenAI models? Pretty much just what we’ve heard from OpenAI: it’s their “strongest model yet” and “sets a new standard for both intelligence and efficiency, achieving state-of-the-art results across coding, knowledge work, cybersecurity, and science.” This likely means it also sets a new standard for hacking and the dangerous, disease-creating kind of bioengineering.
In other new model news, SpaceX, now the home of what was Elon Musk’s xAI, launched Grok 4.5 today, about which I have to say ... not much. I guess we’re to a (scary) point where models that would have dominated the performance rankings 9 months ago fail to impress. By some accounts, the new Grok is good at searching tweets.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.
The White House doesn't have any incentive to appear to be playing fair. The administration is made of several people, and some of them are genuinely scared (though not as scared as they should be, yet).
I confidently called that they'd come down on OpenAI just as they did on Anthropic, and I found myself standing nearly alone in that prediction. Well, here we are, and there will be a new EO soon (hopefully within the month) to make the rules clear for everyone. The vibes have changed, and they will change further with each push into the capabilities frontier.
The government is what it is, but unlike AI developers, they at least have a sense of self-preservation!