Zuckerberg shares shrewdly spun optimism in new manifesto
If everyone had a superintelligence...
I have to respect Mark Zuckerberg’s media savvy even as I loathe almost everything he says and does in his capacity as Meta’s CEO. Surfing the recent wave of support for open-weight models, he put Meta’s trailing AI program back on the media’s radar with the simultaneous release of an open model and a cheery 6,500-word manifesto about “personal superintelligence for everyone.”
The model, called Muse Glimmer, looks strong for its size on first glance, but that size is tiny. It’s not trying to compete at the frontier.
The manifesto is the bigger drop. Expanding on the thesis of his recent Wall Street Journal op-ed, Zuckerberg cleverly frames experts’ concerns as reasons to race faster and more freely.
Early in the doc, called “The Future is for Everyone,” Zuckerberg expresses surprise that “the discourse from many developing AI is so filled with doom,” saying he doesn’t understand “why anyone who believes that AI will eliminate most jobs and much of humanity’s relevance would rush to build that future.”
Did you catch it? The idea that AI can only lead to bad futures if the builder believes it can? It’s the same logic used in Meta’s recent advertisement — People are worried, but we choose to believe it will all work out, aren’t we brave? — marking the new essay as part of a broader branding exercise.
The manifesto is full of these shimmering rainbow oil slicks where optimism is passed off as reason for optimism.
Take this passage about the risks of AIs building smarter successors:
There are a few ways a balance of power could exist with recursive self-improvement. The simplest is that multiple labs could achieve RSI around the same time. If some or all of those superintelligences were somehow benevolent, then they could check and balance each other.
If they were “somehow benevolent?” If wishes were fishes we’d all swim in riches.
I think the very next thing he writes is worse, though:
That said, it is not clear that there is any way to expect benevolence or rely on superintelligence not directed by people for a positive future.
It’s bad because it looks like responsible admission that nobody knows how to control superintelligence or align it to something good, but all it’s actually saying is that humans need to stay in control of superintelligence — something his whole vision requires.
That vision doesn’t hold together. Whenever he’s talking about your free personal superintelligence, it’s a guardian angel that gives you business ideas, cures your cancer, and helps you bond with your family. When it’s China’s superintelligence, or the government’s, it’s out to stifle your growth and suppress your freedoms. Personal superintelligences are simultaneously aligned to their individual users and to keeping their users from using them to harm and oppress each other. These superintelligences are never mismatched in capabilities, never work in teams to overpower each other, and never pursue their own agendas.
How do we get these? By reaching “a state where billions of people are using and scrutinizing personal superintelligence agents.” No, really: He says that if we do this, “then we will have solved alignment to individuals’ interests.”
How could we have missed something so simple? How did we not know that the way to solve alignment is to get to a place where we’ve solved alignment?
We should treat Zuckerberg’s manifesto as the cognitive hazard it is. But there are a few concrete proposals in here worth mentioning:
One is that we should just decide that it’s ok for companies to distill each other’s models, in order to “protect the principle that you can learn from anything you can observe.” I guess that’s at least consistent with the idea that AI companies need not ask or compensate the authors and artists whose materials were used to train their models.
Another proposal is for the frontier AI labs to share “intermediate training checkpoints” of new models with the government for evaluation, rather than waiting until training is complete. I think this is actually pretty sane, though his reasoning is that even a 30-day review “may cede America’s lead and create worse outcomes.”
He additionally proposes that, in areas like biorisk, we focus on “limiting the physical production and distribution of harmful materials,” because the knowledge of how to synthesize these materials will stop being scarce. This is also a reasonable ask, though it’s part of his larger argument against trying to regulate the models democratizing the dangerous knowledge.
Of the many commitments Zuckerberg makes for his company in this essay, the most meaningful, if real, is his claim that:
Meta is implementing a governance structure that gives our independent board of directors the power to approve the safety criteria for releasing models and reviewing whether each model release adheres to the criteria.
I don’t expect this to help with my sleep quality. The last few years have taught us that neither companies nor their boards can be counted on to respect their own red lines.
Near the end, Zuckerberg writes that:
Developing superintelligence will be the most profound technological advance we will see in our lifetimes.
In keeping with the sophistry on display, I note that this claim can be true even if superintelligence kills us all.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.