Amodei declines to flinch
Anthropic's CEO shared reasons he won't sign the open-weights letter despite intense peer pressure
Among tech leaders and experts over the past several days, the open letter against bans on open-weights models from China or elsewhere fell into the sort of cascading peer pressure I associate with episodes of social injustice: Anyone who dislikes the injustice but has reservations about proposed remedies can find themselves cornered by claims of “You’re either with us or you support the injustice.”
In part, the letter was a transparent move by underdogs to form a disapproving circle around the leading AI companies, painting them as greedy totalitarians for trying to keep their frontier models locked down. It seemed to say, “You’re either with us or you’re against open weights, open source, and freedom itself.” And it worked: Even Google and OpenAI ended up signing the letter.
That left all eyes on Anthropic, the only big dog still in the center of the circle. It would have been easy for CEO Dario Amodei to cynically sign the letter with no intention of doing anything differently, as I fully suspect OpenAI’s Sam Altman did. Instead, Amodei put out a blog post yesterday explaining why he wasn’t going to sign.
He clarifies that “Anthropic has never advocated for a ban on open-weights models,” and that open-weights models without dangerous capabilities are a “public good.” But he’s worried that powerful AI models “may be misused to carry out cyberattacks or biological attacks, and may have serious alignment problems,” linking to the TIME article about the Hugging Face attack. Open-weights models, he reminds readers, are hard to monitor, impossible to take back, and trivially freed from guardrails against misuse.
Amodei says we should instead:
Stop selling powerful AI chips to China.
Crack down on the distillation operations used by Chinese companies (and others) to transplant frontier capabilities into models that may lack frontier safeguards.
Subject all models — open and closed — to mandatory safety testing.
He disputes the open letter’s thesis about defensive AI being the solution to offensive AI, writing:
I worry that biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials, whereas defense against these agents is a multi-year operational task in the best case.
As I’ve just shared a bunch of Amodei’s words at face value, let me be clear: Amodei is leading a race to superintelligence that would kill us all. The race must be stopped.
I also think Amodei is wrong to imply that it’s safe to continue developing frontier AI, as long as it’s thoroughly tested; the most dangerous models are already criminally dangerous and prone to escaping during testing — as seen in the Hugging Face attack.
But I respect Amodei for standing up to the open-weights open letter posse, and I think his stated reasons for doing so are sound. As bad as things are, they could always be worse. Could we please not give everyone with a GPU and a grudge the means to design a pandemic?
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.