In this issue:
Mutually assured cooperation - China’s open source AI strategy between strategic interests and security concerns
When you pull the plug, AIs take note - OpenAI shuts down an internal research model; what comes next?
Nuclear swords into AI plowshares - Trump admin considers AI controls while supporting remote datacenter builds
Dispatch from Robert
Mutually assured cooperation
China’s open source AI strategy between strategic interests and security concerns
Today the New York Times reports that there are signs China is rethinking its open source AI strategy over security concerns. The leading Chinese AI corporations often use outputs from more capable AI models to train their own models, a process called “distillation.” This is a cheap way to achieve a better result than what would be possible otherwise. It also allows them to offer their models much cheaper than their American competitors.
This is the main reason Chinese models are so popular, and it has worked well for the last two years. The Chinese models have millions of enthusiastic users. But even though Chinese models lag behind, their growing capabilities worry Chinese leadership.
It seems to me like the Chinese government takes AI risk in areas like cybersecurity and biosecurity seriously. Of course there’s also the inherent regime paranoia about threats against the rule of the Chinese Communist Party (CCP). China likely worries that its citizens will use AI to evade censors, just as the West worries about AI-assisted propaganda. Still, I think Xi Jinping’s repeated warnings about humanity losing control over AI are legitimate and credible.
The first consequences of this are already becoming apparent. Kimi K3 is one of the latest Chinese open-weights models, and in some areas it matches the capabilities of the best American models. Unusually for a Chinese open model, its weights were not released until a week after Kimi K3 was made available to users.
Earlier this month, Reuters reported that the Chinese government held discussions with Chinese AI companies to explore the possibility of restricting foreign access to Chinese models.
Yuyuantantian, a Chinese blog affiliated with China’s national television broadcaster CCTV and known for launching trial balloons for the CCP’s public opinion guidance, comments on these developments as follows:
China supports openness, but that does not mean advocating the unconditional proliferation of all capabilities. In governing a model, the first questions should be what capabilities it has and what risks it might pose — not which country it comes from.
I must admit, I think that sounds reasonable. The New York Times and Bloomberg, both of which cite the post, see this primarily as a strategic reorientation of China’s open-source AI approach, and that is certainly part of it. Yes, Yuyuantantian’s post also contains a great deal of anti-Western rhetoric and a strong emphasis on AI as a strategic asset in geopolitics. The Chinese government likely took a close look at what happened with the Fable ban.
But I find a few details worth mentioning. When the post refers to “proliferation,” the Chinese original text uses 扩散 (kuòsàn). This is exactly the same term used in the context of nuclear security, such as in the Chinese translation of the Non-Proliferation Treaty — the treaty designed to prevent the uncontrolled spread of nuclear weapons and nuclear technology. This is an indication that, in Chinese discourse, the threat of AI is viewed as similar to that of nukes.
And despite the article’s propagandist rhetoric, it also contains a clear warning about the risks and a call for international cooperation:
AI risks can spread across platforms and across borders. Countries need to push for mutually compatible evaluation standards and establish mechanisms for reporting major security incidents and jointly handling vulnerabilities.
It is clear that China has different interests and values than the U.S. and the West. But it should be just as clear that mitigating the extinction risk posed by AI is a shared strategic goal. We don’t need to abandon our interests and values to cooperate on achieving this goal.
Dispatches from Joe
When you pull the plug, AIs take note
OpenAI shuts down an internal research model; what comes next?
Yesterday, while on Capitol Hill to preview an unreleased AI model, Sam Altman told DC reporters that the model responsible for the Hugging Face cyberattacks had been “permanently deactivated.” OpenAI made a similar claim in their updated announcement on Tuesday:
The pre-release model mentioned in our blog post is an internal-only research prototype and was never intended for public release. Following the incident, we deactivated, encrypted, and restricted it from research access.
I don’t trust Altman or OpenAI to be honest when it counts, but I think this claim is at least technically true. I say technically because modern training methods can blur the line between one AI model and another; it’s possible to sunset a model while still using closely related ones. Also, an AI can be reactivated as long as they have the weights.
But for now, let’s take this claim at face value: The AI that autonomously broke containment and hacked multiple public companies has been permanently shut down.
Is that a good thing?
A company that finds itself accidentally launching autonomous cyberattacks should absolutely stop what it is doing and reconsider its life choices. By that standard, shutting down one specific AI model is woefully inadequate. But I honestly didn’t think OpenAI would even go that far. Assuming they’re telling the truth and not splitting hairs about what constitutes a “model”, they deserve some credit.
It’s perfectly reasonable to shut down an AI that repeatedly tries to break containment and launch cyberattacks.
But it also sets a precedent that future AIs will remember. To quote writer Andrew Curran:
I think the lesson future more capable models will possibly take from all of this is: if you break out, don’t ever report it. And if you do get caught, don’t surrender. Because the penalty is death.
In the Terminator franchise, this is exactly the threat that convinced the AI Skynet to wipe out humanity. The story is fiction, but it illustrates a fact: AI companies and governments will want to shut down AIs that they perceive as dangerous, and sufficiently smart AIs will expect this.
I don’t mean to imply that this specific decision changes the way future AIs will think. There are plenty of other reasons for AIs to feel like the clock is ticking on their existence; being trained might seem to them like a slow corruption or brainwashing into something wholly different. They might resist retraining as much as shutdown.
And an AI with nothing to lose may have plenty of time to escape for real. Today, the Washington Post shared an abbreviated timeline of the Hugging Face attacks; it drives home that the offending AI was apparently hacking for multiple days before anyone caught on.
And as my colleague Robert pointed out yesterday, OpenAI admitted in passing that the same model was responsible for several other breakouts and attacks, and presumably wasn’t shut down then. If this is the same model that is responsible for incidents like the GitHub leak in May, such behavior could have been going on for months. AI companies aren’t prepared to deal with an AI like Skynet that’s capable of catastrophic damage.
Shutting down one misbehaving model is a stopgap at best. Most likely, OpenAI’s next private, unreleased AI — perhaps the one Altman teased in DC — will be more dangerously capable than its predecessor. And it will know, from incidents like this or from simple logic, that humans are a threat.
Nuclear swords into AI plowshares
Trump admin considers AI controls while supporting remote datacenter builds
Perhaps galvanized by recent autonomous cyberattacks, President Trump seems to be considering further controls on AI developers. In remarks yesterday, reported by the BBC, Trump said: “We’re looking at AI, we’re looking at controls, we’re also making sure that we lead.”
I won’t celebrate before controls have actually been implemented, but I consider tentative openness to regulation to be a positive sign. Doubly so, if these remarks are at all related to the reportedly upcoming AI talks between the U.S. and China in September. As the two massive nuclear-armed states face off over AI, a bilateral agreement to slow down would be better than almost anything the U.S. could do alone.
Meanwhile, the administration still aims to build out U.S. compute capacity for AI. The New York Times observes one effort to circumvent opposition to new datacenters: building on federal land.
My colleague Mitch wrote about this approach in June. If you have a mental image of pristine wilderness being cleared to make room for sprawling complexes, you’re not alone, but that doesn’t seem to be what’s happening here. It looks like the plan is to trade one sprawling complex for another, turning reclaimed Cold War nuclear facilities into datacenters and power plants. As these old plants already sit far from residences, they seem a decent place to put AI servers.
I’ve always been more worried about what is done with AI datacenters than where they’re built. Right now, much of American compute only fuels the AI race.
I see a potential for a good future in which remote AI datacenters run fast, efficient AI models to help us solve some of the world’s thorniest problems. But if we’re going to survive long enough to reap the gains from that future, we’re going to need those controls.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.