Open-weights debate hits fever pitch with open letter
Profit motives explain hypocritical positions and dismissals of risks
Even as the Hugging Face hack still had everyone talking yesterday — and perhaps partly because of it — the leaders of dozens of U.S. companies rallied behind an open letter in support of Chinese models.
Er, I mean open-weights models. But since the best open-weights models are all Chinese right now, this amounts to the same thing. So I find this letter ironic, given the way many of these same CEOs use China as a boogeyman whenever people talk about regulation.
It’s just business. The letter’s driving sponsors are those with the most to gain from continued access to cheap, guardrail-free AI. These include startups with AI-first business models, the venture capitalists funding those startups, and the also-rans of the AI race — companies like Mistral and Microsoft that would like to compete at the frontier and benefit from undercutting the business models of Anthropic and OpenAI. (Kevin Roose explained the dynamics well in his latest, and especially excellent, Hard Fork podcast.)
The letter was prompted by recent White House signals about potentially banning or restricting the use of open-weights models from China. This would be over frustrations at the way Chinese models are consistently, flagrantly distilled from (trained on the outputs of) leading American closed-weights models like Fable. National security factions within the administration are also concerned about the potential for Chinese backdoors in these models, which could allow them to steal sensitive secrets from corporate users even if hosted on U.S. servers.
The Hugging Face hack threw a curveball into the open-weights conversation, initially because Hugging Face had painted Chinese open-weights models as the heroes that thwarted the attacks after U.S. models refused to engage in sketchy-looking cyber tasks. But the subsequent reveal that OpenAI’s models had independently carried out the attacks woke people up to the risk of open-weights models behaving similarly. If your nightmare scenario begins with rogue AI escaping its creators’ servers, then letting anyone download rogue AI for free looks like a good way to speedrun to the part where you wake up screaming.
Open-weights advocates have been spinning this fear as further reason to not restrict open-weights models. (If you outlaw them, then only outlaws will have them!)
Some are hoping the U.S. will incentivize American firms to release open-weights agents that can compete at the frontier, or at least with the best of the Chinese models, so companies don’t have to choose between American, capable, and open.
My own sense is that any White House moves on this in the coming weeks are likely to be short-lived. I think the administration will soon be spooked enough by model capabilities to seriously dislike the thought of anyone running frontier models that aren’t on a tight leash and easy to shut down, regardless of where they were born.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.