In this issue:
Please wake the driver - Are Trump and his party really going to commit both political and literal suicide by AI?
Why aren’t more AI staff quitting? - The industry selects for people who think they can navigate the dangers
Rumors of Recursive Self-Improvement (RSI) - Don’t sleep on capabilities progress while policy has our attention
Dispatches from Mitch
Please wake the driver
Are Trump and his party really going to commit both political and literal suicide by AI?
Rogue AI swarms aren’t partisan, and our political response to them can’t afford to be.
I nervously observe that, while the overall media salience of the AI problem continues to soar, the issue has been noticeably less prominent on right-leaning news sites this weekend. That’s a shift from Thursday and Friday that might be correlated with the mostly right-wing spread of new conspiracy theories. These allege that Jacob Coxon’s resignation from Anthropic was in some way phony, that his warnings were lies, and that the viral attention that followed was somehow manufactured.
I hope these aren’t behind President Trump’s remarks on the golf course today, when he said:
whoever wins AI wins. And we can put guardrails, and we can do this and that, but I think you have a lot of negative forces that are bringing it up, and they’re bringing up things that won’t happen.
He has an AI meeting with China’s President Xi Jinping in D.C. on the 24th. We need him to step up! There are a handful of AI companies that might be able to end the world in the next 6-12 months, but only one person on Earth who might be able to singlehandedly stop them.
To borrow a Trumpism that happens to be true in this case, people are saying that if the President makes a strong AI security deal with China, he should get the Nobel Peace Prize.
Yes, one of those people is Sam Altman, whose idea of what a deal could be maybe doesn’t go far enough:
And even if just the US and China could agree on some shared standards and testing for development of this technology, I think that’d be a wonderful accomplishment that the two of them can deliver. [...] I don’t think this is hard. This is like a one page document.
But if Trump and Xi could start at Altman’s suggestion to “agree that no one should be taking a certain level of risk with the development process of this,” they could work out a more concrete treaty to halt the race to uncontrollable superintelligence later. It is possible to make adherence to such a deal verifiable.
The proximity and uncontrollability of superhuman AI swarms have started to sink in with the public, and with the Congresspeople asking for the cancellation of a planned recess so they can act. But like Trump, House Speaker Mike Johnson (R-LA) doesn’t seem to have updated yet. According to Politico, Johnson said today that he would support AI talks with tech leaders, but he “insisted that developers — not Congress — are responsible for ensuring their products are safe.”
Products? A chatbot is a product. Amoral agent swarms of elite hackers, like the one responsible for the RubyGems attack described yesterday, are a national security threat. We have to hope Trump and Johnson just haven’t woken up to this fact yet.
In what some will find to be a surprisingly grounded discussion with Tucker Carlson that came out two days ago, MIRI president Nate Soares shared his analogy of the bus driver:
My biggest hope here comes from the fact that most people don’t understand what these guys are trying to do. One way I like to say it is the bad news is that the bus is racing towards the cliff edge. The good news is that the bus driver is asleep. Which might seem bad. [...] But if you can wake the bus driver up, you know, it’s much better to be in a bus where the driver that’s headed towards the cliff is asleep than if they’re awake [...] and they’re choosing the cliff... right?
The passengers on the bus are screaming. If they go into the ballot box like that in November, the Republican party will be in for a drubbing. The GOP shouldn’t need the threat of political suicide to motivate it when inaction on AI means literal suicide for us all, but maybe there’s a streak of Hermione Granger in the party that can be appealed to.
Will someone please wake the driver?
Why aren’t more AI staff quitting?
The industry selects for people who think they can navigate the dangers
“It’s what we’ve all been saying,” wrote one Anthropic researcher about the alarm raised by Jacob Coxon in his viral resignation announcement.
The New York Times’s Mike Isaac and Kate Conger gathered several such statements for an article yesterday about the worried chatter inside AI companies. That one came from a company chat. But employees across labs are reportedly taking their concerns off the record as well, in “encrypted chats and private dinners,” with intent to organize awareness-raising efforts. One existing group, the Coalition of Concerned AI Staff, is described as a project to create “stronger safeguards” and hold companies to their safety commitments.
Publicly and privately, employees are debating the impact they can have by loudly quitting like Coxon vs. applying leverage at their companies. They are almost universally choosing the latter. Why? After Coxon’s resignation, workers reportedly “urged top executives at their companies to pause development of the technology so they would have more time to work on it in a safe way.” But I didn’t hear them threatening to quit.
I think it’s because industry staff have been weighing these issues for a long time. Those deeply bothered at the idea of personally accelerating the AI race had quit already or never taken a job at an AI company in the first place.
The same is true of the AI executives themselves, which is important to remember when people ask why AI CEOs would be actively building something they think could kill us. There were always going to be people determined to build superintelligence. These were going to be people who believed it was possible. To make superintelligence your life’s work, you kind of have to take the idea seriously. Taking it seriously means recognizing how dangerous it could be. We should be much more surprised by founders working toward superintelligence who don’t think it’s dangerous.
Mark Zuckerberg is arguably the exception that proves the rule. The Meta CEO uses the word “superintelligence” a lot and talks as though it’s only as dangerous as your beliefs about it. But he was also late to the race and seems to see it as a mere consumer product. His visions of a future with superintelligence are substantially more mundane than those of his peers — bordering on Jetsons territory — suggesting he doesn’t actually believe in the kind of superintelligence his own engineers are working on.
It’s crazy that people can try to build a machine god without so much as a permit. But we’re in this situation only because lawmakers and the public spent the last twenty years dismissing the idea of superintelligence in our lifetime. The kind of people who stuck with their grand project through all the mockery mostly aren’t going to be the type to quit now that the laughing has stopped.
Rumors of Recursive Self-Improvement (RSI)
Don’t sleep on capabilities progress while policy has our attention
Researchers at OpenAI say it. Researchers at Anthropic say it. And so, increasingly, do researchers at Google DeepMind. They’re putting more and more of their AI development in the hands of AI itself, and the cycle is speeding up.
Twitter’s premier AI rumor-broker, Andrew Curran, said yesterday that “rumors have been circulating for three days” that training of DeepMind’s next model is going faster than planned because of new discoveries. What new discoveries? “There are rumors of training improvements with machine assist.” What’s the most “optimistic” rumor? “That they have successfully closed the loop and have achieved RSI.”
This would be... a huge deal, and not just because it would likely put DeepMind back in frontier contention with Anthropic and OpenAI. If DeepMind — or any lab — has truly started a cycle of recursive self-improvement, then we’ve reached the part of the story where the AI labs lose control on purpose and surrender our fate to AIs cooked up by other AIs.
If true RSI has been achieved, it is more important than any of the current excitement about company proposals for pacing development.
I don’t know if this has happened! Curran’s track record is strong — he most recently claimed that Navier-Stokes had been solved days before the math story broke — but that doesn’t mean the people he’s talking to aren’t mistaken or confused.
My point is that we’re likely in the end-game, or close enough to it that every day and hour really matters. AI capabilities are ticking up every day we spend letting the labs discuss among themselves how they’d like to exercise more responsibility in a way that “does not mean halting model training or technical progress.” They’re ticking up every hour OpenAI dodges transparency about its swarm incidents, and every minute we spend shooting down bad AI takes online.
We can’t wait for the next Congressional session, the next President, or the next warning shot to start regulating this industry, because right now the government probably doesn’t even have enough visibility into the labs to know how close to the edge they’ve taken us. The most momentous and dangerous development in history shouldn’t be the kind of thing our leaders learn about through a series of unsubstantiated tweets.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.