In this issue:
More encouraging signals from the Chinese president - In World AI Conference keynote, Xi Jinping flags “loss of control” risk but gives few details
Chinese AI follows fast - What the news gets right and wrong on open weight AI
Some alignment on what’s needed - Politico’s new AI safety map shows common ground
Dispatch from Donald
More encouraging signals from the Chinese president
In World AI Conference keynote, Xi Jinping flags “loss of control” risk but gives few details
The World Artificial Intelligence Conference (WAIC) is an annual international gathering held every July in Shanghai: part-trade show and part-governance forum. This year, Xi Jinping gave the opening keynote speech; it was his first time attending WAIC.
For me, the headline news is a single phrase he used: “loss of control.” I pay close attention whenever I hear that from policymakers (as I did while reading the United Nations’ Preliminary Report a few weeks ago). Xi’s closing words called for the world to “constantly refine measures to forestall loss of control.” The term used is 失控; “loss of control” is not a Western gloss or creative interpretation, though I can’t be certain that there’s not some specific connotation that’s been lost in translation. (You can find the official English translation here.)
Xi also said that AI must remain “under human control.” Not under the state’s control, or the Party’s control. That might just be rhetoric, but the CCP is made of humans. Any world in which humans have lost control of AI is definitionally one in which the Chinese government has lost control as well. In that respect, the CCP’s incentives are aligned with everyone else’s.
This matters because the standing objection to an international treaty is that China won’t come to the table. A speech like this is evidence that China is at least willing to talk. Zvi Mowshowitz comments:
No doubt they have in mind something that would favor their position, and their initial asks will look outrageous and unacceptable to us. That is how this works. [...]
It is also possible Xi is engaging in cheap talk. [...]
The way you find out and do your best with this opportunity is: You get started now.
I don’t want to overstate the case for optimism. Xi’s language on safety is bog-standard: no compute thresholds, no evaluations, no red lines, and no verification regime. Openness is being prioritized over safety; a commitment to open-weight models that anyone can download is hard to square with keeping humans in control. You can’t recall or monitor a model whose weights have been made public.
But there are signs that the Chinese government feels that tension. This push is coming amid an internal debate over the security risks of open-weight models. The newly-announced Chinese AI model Kimi K3 is purportedly an open-weight model, but the weights won’t be public until July 27. Mowshowitz thinks that the delay is safety-motivated, but we don’t know for sure.
On balance, I think that Xi’s speech is an encouraging signal. It isn’t the exact thing that I want — an international treaty to halt the development of artificial superintelligence — but this is the kind of thing I’d expect to see on the way there.
Dispatch from Joe
Chinese AI follows fast
What the news gets right and wrong on open weight AI
It’s becoming a fad to claim that China has “caught up to the U.S.” every time a Chinese AI lab releases an impressive new model. It wasn’t true the last time and it’s still not true today. But there are important lessons to take from the stories around Chinese AI, only some of which I’ve covered before.
Alongside a speech by Chinese President Xi Jinping, today’s big story is the release of Moonshot AI’s Kimi K3. With 2.8 trillion parameters, it is the largest AI model with open weights (anyone on the internet can download it). It’s large even by closed-weight standards, similar in size to (though likely not larger than) most leading American AIs. Models this size run on hardware worth hundreds of thousands of dollars, but if you have access to that hardware, the marginal cost to run is pretty low.
Laurie Chen of Reuters has a pretty good analysis, but uncritically shares some claims from the Chinese lab that deserve a closer look. Chen repeats Moonshot’s claims of outstanding hardware performance, claims which are likely true but only somewhat related to a model’s capabilities in practice. She also reports Kimi K3’s high performance on several AI benchmarks (it often did better than any model except Anthropic’s Fable), but neglects to mention that such benchmarks are often noisy, cherry-picked, and easily gamed.
Chen draws an ominous conclusion:
Companies including Moonshot, Z.ai and MiniMax are releasing increasingly powerful models at sharply lower cost, challenging long-held assumptions in the West that Chinese developers trail their American peers by months.
But by most accounts, Chinese developers do trail their American peers by months. Reporting by Jamie John of the Financial Times strikes me as more grounded, citing recent analysis from the UK AI Security Institute that puts the U.S. lead at 4 to 7 months — lower than in 2025, but still substantial.
Citing the same institute, John points out the reason open-weight models can be more dangerous than private ones: Once an AI model can be downloaded by anyone over the internet, there’s no taking that decision back. And bad actors with access to a model’s weights can easily strip away built-in protections.
Like many AIs in America and China, Kimi K3 was probably trained (or “distilled”) at least in part using the outputs of some leading American models. Christopher Mims of the Wall Street Journal notes that this is common industry practice, though often irritating to leading AI developers when done at scale.
Mims observes that high-performing, open-weight models like Kimi K3 threaten to undercut the profits of American AI labs. This checks out: People will often choose a cheap, good-enough AI model over the expensive best-in-class.
I think he’s wrong, though, when he goes on to say that access to electricity is “the moat that will matter most” for AI companies. Electricity does matter, and since attempts to build new American power plants are often smothered by environmental lawsuits, China may be on track to power its own datacenters more cheaply.
But I think there’s still an immense difference between having the world’s smartest AI and the next best thing. I haven’t heard any stories of American companies distilling Chinese AIs to copy their scary capabilities. Due to a combination of talent, chips, and other factors, American AI companies are still driving most of the progress in AI. As my colleague Mitch put it, it’s hard to outrun your own shadow.
I also think there’s an immense difference between jockeying for market share and racing to build the next generation of AI. Most leading American labs are explicitly trying to create vastly superhuman systems, and if the prospect that their creations might extinguish our species won’t stop them, neither will losing some customers to China.
Dispatch from Alana
Some alignment on what’s needed
Politico’s new AI safety map shows common ground
Politico attempted to map out some of the main factions within AI safety, a term that has come to mean many different things.
I could quibble about the labels and groupings, but I’d rather point out that even with the (questionable) breakdown Politico created, there’s a surprising amount of common ground. Four of the seven categories include people who are very concerned about the risk of extinction from AI. Nobody likes the term “AI safety.” And most people think we should make sure we can keep AI under human stewardship. The main exception seems to be the so-called “let it rip” crowd, which was an odd choice for inclusion in an AI safety field guide, since they view safety as antagonistic to innovation.
Granted, the individuals featured almost certainly disagree about how to keep AI under human stewardship. Some think guardrails and safety checks will be enough. Others think we need to stop development until we have a better understanding of how to reliably get AI systems to do what we want. (I’ve written about why available guardrails and safety checks aren’t currently up to the task, and my colleague Robert wrote about the limits of tests just yesterday.)
But there’s still widespread agreement on a common goal, which is certainly not something to take for granted. That gives me some hope.
One place I will quibble with the article is its description of MIRI president Nate Soares as a proponent of “completely shutting down AI development.” His frequently stated position is that “the only thing that needs to stop here is this race to create artificial superintelligence.” In other words, we should stop developing more and more powerful systems until we understand how to reliably steer them; this could mean pausing for decades or more. During such a pause, Nate says, companies could still pursue more narrow AI for certain domains: things like cancer research, drug discovery, and self-driving cars.
While Politico’s summary of Soares’s stance wasn’t quite right, the quote they included from him was spot on:
I’ve never liked the term ‘AI safety.’ If you’re in a car careening towards the edge of a cliff, it’s not the time to bring up ‘car safety.’ Car safety is about seat belts; it’s about airbags; it’s about crumple zones; and none of those will save you if the car goes soaring off the cliff. AI companies are racing to create artificial superintelligence, and the AIs they’re creating aren’t carefully programmed to follow instructions; they’re a mess of trained numbers that even the creators don’t understand. What we need is not a little extra ‘AI safety.’ What we need is to stop this reckless race.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.