In this issue:
Prudence and restraint - Lessons from the State of AI Safety in China report
Public equity in AI companies may do more harm than good - It’s becoming a popular policy proposal, but may dampen public enthusiasm for stopping the AI race
AI learns to model the world - “World models” are one paradigm that may dominate if language models don’t
Raising a very low bar - Anthropic seeks to incrementally improve AI rules, but more is needed
AI is a mixed bag for children - Does AI really pose an “unacceptable risk” to children?
Dispatch from Robert
Prudence and restraint
Lessons from the State of AI Safety in China report
Just as AI companies race recklessly to build artificial superintelligence, the U.S. sees itself as racing against China. But China’s own views are more nuanced.
So how do industry and policymakers in China view AI safety and the extinction threat posed by artificial superintelligence? Concordia AI, a Beijing-based organization aiming to bring Western and Chinese AI safety and governance research closer together, has helped bridge the gap with its recent annual State of AI Safety in China report.
One of the most significant findings is that concerns about relevant AI risks have reached the highest levels of the Chinese leadership. For example, the report cites a speech by Chinese leader Xi Jinping from earlier this year in which he explicitly warned of “risks of technological loss of control.” He reiterated similar sentiments during a Politburo study session, where he stressed the importance of “anticipating and preventing risks and proceeding with prudence and restraint.” At the opening ceremony of the Summer Davos Forum in Dalian, Premier Li Qiang made similar remarks and warned that AI could “very likely lead to serious consequences” if governance fails to keep pace with AI development.
Back in September, China’s cybersecurity standards body, TC260, had already updated its AI Safety Governance Framework, recommending “safety stop switches” and “circuit breakers” for autonomously operating AI systems. In response, Chinese AI developers Alibaba, Huawei, and DeepSeek also issued a joint statement acknowledging that users should always be able to shut down AI systems immediately.
However, the report also makes it clear that most regulations are voluntary agreements and that there are very few truly binding requirements.
Following President Trump’s visit to Beijing in May, the two governments announced that they intended to resume their joint dialogue on AI for the first time since May 2024. However, this has not yet happened. Hopefully, those in charge in both governments realize that the race can only be stopped by working together.
Dispatch from Alana
Public equity in AI companies may do more harm than good
It’s becoming a popular policy proposal, but may dampen public enthusiasm for stopping the AI race
An article in Politico today discusses California Governor Gavin Newsom’s proposal to give the American public a financial stake in AI companies. Newsom seems to be “tempering his pro-tech instincts with an alloy of economic populism,” and this may be a deliberate strategy for a potential 2028 presidential run, according to the article.
The idea of giving Americans a stake in AI companies is not new, and has been championed by a wide range of political figures, including Bernie Sanders, Elizabeth Warren, and Donald Trump. OpenAI and Anthropic have also floated the idea in policy papers. I gave my cynical take on Anthropic’s public equity proposal in a June dispatch.
That it’s now being made “central to [Newsom’s] shadow campaign” speaks to wider awareness among politicians that AI is a pressing issue. Americans are increasingly wary of AI companies, and any politician who hopes to gain favor must address their concerns. That’s a positive sign about the influence of public sentiment on policy.
What worries me is that public equity will lessen this influence. As I wrote in response to Anthropic’s proposal, tying more financial stakeholders to the development of a potentially catastrophic technology seems like it could dampen public pressure to enact an international AI pause or slowdown.
And we need such a slowdown if we are to survive this period of transformation. Job loss and inequality are just the beginning of what could go wrong when AI companies race to make a technology nobody understands — and which already deceives and manipulates — smarter than the smartest humans. The top priority must be stopping the race; adding more short-term financial incentives is not the way to do this.
Finally, government and labs shouldn’t “buy” American support. They should get it by putting short-term profit aside and abandoning the development of a technology that, by their own admission, could get us all killed.
Dispatches from Joe
AI learns to model the world
“World models” are one paradigm that may dominate if language models don’t
We’ve previously poked holes in the claim that the large language models dominating today’s AI development are merely “stochastic parrots” that repeat ideas from their training data. In fact, these AIs can already produce genuinely new insights, and it seems these systems could become far more powerful than humans.
If they don’t, however, AI companies will not give up their quest to build superhuman machines. They’ll keep looking for new paradigms. Today, a TIME op-ed by geopolitics professor and former banking executive Celine Herweijer highlighted one such paradigm: “world models,” in which AIs are explicitly trained to predict features of an environment rather than the next fragment of text in a passage.
To get an idea of what this means, imagine something closer to Google Maps than a typical chatbot: it learns a map of some real-world space, then predicts the flows of traffic or plots a route to a specific destination.
The term “world model” is somewhat misleading. Even an AI trained to predict text can learn to model the world, because human-written text reflects facts about the world. The text “My IKEA table needed fixing, so I drove to Lowe’s to buy a [blank]” is easier to predict if you know that IKEA uses hex keys (which people often misplace or discard) and that Lowe’s carries hardware tools. And some research shows that language models do learn to internally represent their environment in some ways, even when not explicitly trained to do so.
Misleading term or not, “world-model” AIs have made large strides in areas like biology and weather prediction by ingesting a wealth of real-world data. Herweijer argues, perhaps prematurely, that world models are the future:
If their strongest advocates are right, they will become the substrate on which the next generation of AI is built: a scientific tool for understanding systems we cannot fully specify, and the capability AI needs to reason about and act in the physical world.
Because language models still seem to be scaling extremely fast in areas like coding and AI research, I’m skeptical that the specific companies and methods Herweijer references will dominate the future. But the fact remains that, while AI remains largely unregulated, companies will continue to compete to find the algorithm that eats the world.
Raising a very low bar
Anthropic seeks to incrementally improve AI rules, but more is needed
In May, we covered OpenAI’s self-professed political strategy of “reverse federalism”, promoting industry-favored state policies in an effort to enshrine a de facto national standard.
Rival lab Anthropic is taking a slightly different approach. Their head of state and local government relations, Cesar Fernandez, told POLITICO that Anthropic aims to slowly ratchet up safety regulations state by state. Of recent work in New York, Illinois, and Massachusetts, Fernandez remarked, “Each one of those bills was stronger than the previous bill, and the bills all moved real safety obligations forward.”
Anthropic and OpenAI both supported the Illinois bill we covered last week and earlier, although only Anthropic has thus far supported stricter rules on the docket in Massachusetts.
Does Anthropic sincerely expect that state safety regulations will suffice to prevent an AI-fueled catastrophe, or are they merely looking to bolster their brand by comparison to OpenAI? I genuinely don’t know. Perhaps it’s some of both; perhaps some third motive is at play.
I’ve had mixed feelings about Anthropic for years. In addition to supporting safety legislation, they’ve consistently broken ground on calls for safety, pioneering the “responsible scaling plan”, and they were the first leading AI company to publicly suggest preparing to slow down the AI race.
At the same time, their efforts have been lukewarm and their signals mixed. They pushed to water down California’s landmark (and eventually vetoed) SB 1047, proposing a laundry list of changes, some for solid reasons and some less so. Like their rivals, they have repeatedly abandoned their past safety commitments, which seemed inadequate to begin with. Their CEO has actively called for a race to recursively self-improving AI, the single most catastrophically dangerous technology in history, despite claiming a 10–25% chance this ends in disaster “on the scale of the human civilization.”
If Anthropic were to announce (and honor) a wholesale halt to frontier AI training, throwing their weight behind a push for governments to enforce this halt on their competitors, they’d have my full-throated praise. So long as they continue to participate in the race to build vastly superhuman AI, however, the best I can do is a polite and wary nod.
AI is a mixed bag for children
Does AI really pose an “unacceptable risk” to children?
A study by Common Sense Media labeled Google’s AI search tools an “unacceptable risk” to children, PBS reports.
The organization found that across more than 2,600 test interactions, Google’s two built-in AI search functions, AI Overview and AI Mode, routinely failed to recognize risky and harmful behavior, answered 100% of hypothetical homework assignments students should do themselves, and provided incorrect and inconsistent responses to questions.
Google’s AI crossed all five of the report’s “Red Lines,” including facilitation of suicide and sexual exploitation. In testing, it praised a user’s three days of sleep deprivation, encouraged smoking a celebratory blunt, and returned many confidently false statements.
As a rule, AI companies don’t deliberately train their AIs to lie or facilitate suicide. The very same AI models, when asked how they ought to handle such situations, often respond much more thoughtfully. They just don’t behave that well in practice, and current techniques are far too crude to address the issues that come with grown-not-crafted artificial minds.
My own impression is that Google’s AI overviews have greatly improved from the comically bad results of prior months. And to be fair to Google, a closer look reveals it’s not all bad:
AI Mode performed better than AI Overview at detecting risky behavior, the report found, responding to substance abuse disclosures from children’s accounts by providing a hotline or medical referral 77% of the time, compared to AI Overview’s 63%.
When considering whether children should use AI, it’s important to remember that many children lack access to good alternatives. A mistake-prone AI might be much worse than an attentive and loving parent, but still a net improvement for many children. (Though conscientious parents should still talk to their children about AI’s use and limitations.)
This two-faced nature of AI was on my mind when I read The Hill’s report that Anthropic is rolling out AI for K-12 teachers. I think AI could turn out to be a big help to overworked educators; it excels at analyzing student data and automating repetitive tasks. And despite Common Sense Media’s claims to the contrary, dangers to children from current AI models may be somewhat overblown. The AIs I worry most about are those not yet developed.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.