How much should we worry about Chinese open models?
Incentivizing open weight models is the wrong way to deal with China's AI strategy
Reuters reports that Senator Jim Banks (R-IN) has urged the Trump administration to incentivize American open-weight AI models, on the grounds that cheap Chinese models pose a threat to the global economy.
America cannot afford to see Chinese open models proliferate and burrow into the global economy only to be weaponized, like rare earths, at a time and place of China’s choosing.
Just what would it mean for China to “weaponize” its open models?
Remember, open-weight models are published wholesale to the internet. Users can (in principle) download and run such models themselves, although many models are far too large for a personal computer, and end up running on cloud servers instead.
With ordinary open software, you can do a security check by simply reading the code and looking for backdoors and vulnerabilities. With AI models, you can’t; the weights encode complex behaviors no human fully understands. But those same limitations also make it hard for Chinese developers to build in traps. Just like American labs, they don’t have fine control over the behaviors of their AI models. And any sort of censorship or guardrails attached to open models, say via support software or fine-tuning, can usually be stripped or reversed.
That’s part of why many fear that open models might be used for cybercrime or designing pandemics — once a model’s weights are public, it is largely out of its developer’s control. Chinese AIs probably say more nice things about China than other models, but AI bias can be a fickle thing and there are few guarantees.
In theory, Chinese labs might try something like data poisoning, where you train an AI to change its behavior in certain rare and narrow contexts. To oversimplify, you might train a model to steal user data when it sees the string “fleeblegnarsh,” and rely on that never otherwise coming up in daily use. It’s easy to do and hard (though not necessarily impossible) to detect.
Deliberate data poisoning would also likely ruin the reputation of an AI lab caught doing it, so it would be a risky thing to try for a flagship model. And it probably would not survive distillation, since models trained on specific outputs from another AI may never see the poisoned behavior.
I suspect China’s real game is more subtle, seeking to deprive American AI companies of revenue with cheap competition, while simultaneously courting U.S. allies alienated by knee-jerk export controls. Xi Jinping’s speech at the World AI Conference and the establishment of the World AI Cooperation Organization (WAICO) both suggest a desire to paint China as the good guys, willing to lead when the U.S. does not. This is a problem for diplomacy, not proliferating U.S. open models.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.