Guest post by Haven Harms
‘Once you see AGI you can’t unsee it.’ It has a real ‘ring of power’ dynamic to it, and makes people do crazy things. I don’t mean that AGI is the ring itself, but instead the totalizing philosophy of ‘being the one to control AGI.’
—Sam Altman
In The Lord of the Rings, the Council of Elrond convened as danger was brewing in Middle-earth. Sauron had returned to Mordor and was gathering strength, his Black Riders were hunting for the One Ring he had forged long ago, and that Ring had just arrived at Rivendell in the hands of a hobbit. Representatives of Elves, Dwarves, and Men, each there on separate errands, were called by Elrond to hear the full tale of the Ring and to answer a single question: “What shall we do with the Ring?”
In 2017, the Future of Life Institute (FLI) organized the Asilomar Conference on Beneficial AI, bringing together leaders in academia and industry “to discern the AI community’s shared vision for AI, should it exist.” This conference was held amid growing public interest in artificial intelligence and increasing recognition that those developing the technology had both an opportunity and a responsibility to influence its direction.
At the time, transformative Artificial General Intelligence (AGI) and the dynamic Altman would eventually compare to the One Ring still seemed far off. Most customer-facing AIs were very basic, like Alexa or Siri. But change was in the air. The months leading up to the conference saw a series of matches in Go, the ancient strategy board game, between the AI system AlphaGo and Lee Sedol, one of the world’s top players. As many in the industry had assumed that an AI would not be able to beat the best Go players for another decade, AlphaGo’s triumph came as a shock. The paper “Concrete Problems in AI Safety” was published six months before the Asilomar conference, but the AI safety field itself was very small.
Organizers of the Asilomar conference began by synthesizing existing reports of risks and governance proposals into a preliminary set of principles, which participants then revised through advance surveys and breakout sessions. The result was what came to be known as the Asilomar AI Principles: 23 statements, each backed by at least 90 percent of attendees. The hope was that AI development guided by these principles would “offer amazing opportunities to help and empower people in the decades and centuries ahead.”
At Rivendell and Asilomar, participants discussed the risks at hand and actions that should be avoided to prevent harm and destruction. At the Council of Elrond, amid various proposals and warnings about the dangerous power of the Ring, consensus was reached: the Ring should be destroyed, and Frodo, the hobbit, would be the one to carry it. At Asilomar, in contrast, principles were chosen, but plans were not. The Asilomar AI Principles were turned into an open letter that would go on to collect more than 5,700 signatures. AI leaders in attendance like Sam Altman, Dario Amodei, Demis Hassabis, and Elon Musk all signed the letter. However, their actions in the years since appear to betray these principles.
What would have happened if the Council of Elrond had just said, “We should avoid using the Ring,” and stopped there?
The Trap of Boromir
At the Council of Elrond, Boromir argued that they could use the Ring for good. “Why should we not think that the Great Ring has come into our hands to serve us in the very hour of need?” he asked. “Let the Ring be your weapon, if it has such power as you say.”
Elrond’s answer was that the Ring should not be wielded by anyone. “The very desire of it corrupts the heart,” he said, and if any of the Wise used it to overthrow Sauron, “yet another Dark Lord would appear.” Boromir appeared to accept the correction, bowing his head: “So be it.”
Principles across groups get eroded when faced with motivated reasoning. Clearer incentives can bulldoze vague principles when the situation is dire enough, or the reward delicious enough. This happened to Boromir as he continued to rationalize why they actually should wield the Ring. Months after the Council, Boromir found Frodo alone and made the same argument, questioning Elrond and Gandalf, saying “often I doubt if they are wise and not merely timid.” He reclassified the principle as someone else’s worry, something that was overly cautious and applied to other kinds of people. “True-hearted Men, they will not be corrupted.” We only want strength “in a just cause.” The Ring is “a gift to the foes of Mordor.” And finally: “It is mad not to use it.” Frodo, deeply committed to his mission, barely escaped from Boromir’s attempt to take the Ring.
The AI industry makes the same arguments as Boromir about why racing to superintelligence is justified. Let’s return to Sam Altman. He names the “ring-of-power” dynamic, diagnoses it as the desire to be “the one to control AGI,” and says it makes people do crazy things. It’s a little too on the nose that he writes about what the dynamic does to “people,” seemingly excluding himself, while running one of the companies most entwined in the dynamic. His proposed remedy is distribution: “The only solution I can come up with is to orient towards sharing the technology with people broadly, and for no one to have the ring.” Altman’s proposed remedy reminds me of Boromir’s line of thinking: not necessarily that the right hands can wield it, but that enough hands can.
If everyone has the Ring, no one has the Ring?
What If Boromir Had Gotten the Ring?
In the 1966 Foreword to the second edition of The Lord of the Rings, Tolkien, refuting a theory that the books are an allegory for World War II, describes what would have happened had the story followed history and patterns from the real world. The Ring, he says, would have been seized and used against Sauron. Sauron would have been enslaved rather than destroyed, and his tower occupied rather than thrown down. Saruman, having failed to get the Ring himself, would have found the missing pieces of his own Ring-lore research, and before long “would have made a Great Ring of his own” to challenge the war’s victor. Both sides would have held hobbits in hatred and contempt; they “would not long have survived even as slaves.”
The principles the AI industry has most conspicuously violated point toward a future much like Tolkien’s counterfactual.
The Asilomar AI Principles state: “Superintelligence should only be developed in the service of widely shared ethical ideals, and for the benefit of all humanity rather than one state or organization.” Unfortunately, development “for all humanity” has become an open competition to see which lab, in which country, reaches superintelligence first. The United States and China have each made AI dominance part of official policy in collaboration with AI companies. The strategic AI action plan in the U.S. is literally titled “Winning the Race.”
The principle titled “Race Avoidance” states in full: “Teams developing AI systems should actively cooperate to avoid corner-cutting on safety standards.” When more than 1,300 AI company employees signed the “Pacing the Frontier” letter, they told the government that “each company — and country — is under intense competitive pressure not to unilaterally slow that acceleration.” This pressure is apparent in the AI companies’ safety policies. OpenAI states that it may adjust its safety requirements if a rival releases a high-risk system without comparable safeguards; Anthropic’s policy says it “cannot commit to following” its own industry-wide recommendations unilaterally; Google DeepMind and Meta include similar caveats in their safety policies. Essentially, they all say that their safety standards may go out the window if another company has an unsafe AI.
The wealth these companies generate flows back into politics: In July, it was reported that “Leading the Future,” a super PAC network funded by Andreessen Horowitz, OpenAI president Greg Brockman, and other industry investors, raised $125 million and spent over $24 million in the 2026 primaries alone to influence the AI agenda. Two super PACs in its network face an FEC complaint alleging they concealed their spending through shell companies. This is directly in opposition to the non-subversion Asilomar principle, stating, “The power conferred by control of highly advanced AI systems should respect and improve, rather than subvert, the social and civic processes on which the health of society depends.” Meanwhile, the internet is increasingly being flooded with fake articles, cloned voices, fabricated images, and AI-generated posts. Civic processes depend on people being able to trust what they read, hear, and see, yet it’s becoming harder and harder to distinguish reality from fiction.
These principles had near-unanimous support at Asilomar. Unfortunately, there wasn’t a plan for what to do when the Ring was actually within reach.
Escaping the Trap of Boromir
The AI industry and governments have already fallen into the trap of Boromir, but it’s not too late to course-correct and resist going further. After Boromir tried to take the Ring from Frodo, he saw his error clearly. “What have I done?” he cried. Rather than continuing to pursue the Ring, Boromir spent the rest of his strength honorably protecting the hobbits when orcs attacked. Over the past few years, as AI companies have walked back their safety commitments, I’ve seen some employees quit after experiencing wake-up calls much like Boromir’s. My hope is that more people in the industry have these moments of clarity and escape the Boromir trap.
Getting out of this trap starts with resisting the racing frame itself and the fear beneath it. Boromir’s argument only makes sense within this frame: if the Ring is a weapon, someone must wield it — and better us than them. Racing to superintelligence can seem justified by desperation: if we don’t get there first, someone worse will. Yet when Sauron sought to be reunited with the Ring, kingdoms could not have averted their destruction by racing to seize his power; they had to stand together.
Caution is not the same as timidity, and urgency is not a valid reason to abandon judgment. The wisest figures at the Council refused the Ring because they knew they could be corrupted. “I will not take the Ring to wield it,” said Elrond. “Nor I,” said Gandalf — who had already told Frodo, “Do not tempt me!” warning that the way of the Ring to his heart was “pity for weakness and the desire of strength to do good.” Corruption often starts as good intentions that get twisted as other incentives emerge.
It is vital to seriously consider the possibility that Elrond warned about: that the Ring — artificial superintelligence — is too powerful and can’t be controlled at all. Gandalf said the lesser rings were “only essays in the craft before it was full-grown, and to the Elven-smiths they were but trifles — yet still to my mind dangerous for mortals.” Our own essays in the craft — today’s AI models — deceive their evaluators and behave in ways their creators did not intend. If the lesser rings cannot be fully controlled, it is foolish to keep increasing AI capabilities and expect we will remain in control.
For those in the industry who have had their Boromir awakening, it is time to act: testify to governments about the real risks of superintelligence, turn away from AI capabilities work, and advocate for governance that protects us and gives humanity more time to get this right.
The Boromir problem is not our only AI problem. There is also a Saruman problem, a Wormtongue problem, and so on. But there is reason for hope. The Beacons of Gondor have been lit: When the signatories of the “Pacing the Frontier” letter asked the U.S. government to back an international effort to build the tools to deliberately pace AI development, they recognized that no company, and no country, can do this alone. The AI industry, governments, and the rest of us now need to answer the two questions Asilomar left on the table: what, concretely, will be done, and who will do it.
Haven Harms holds a B.S. in Public Health and an M.Sc. in Global Public Health. She is the founder and principal of Harms Research & Consulting, supporting organizations working on AI safety, governance, and advocacy.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.
