"Friend of precious things"
Startup hubris, AI awareness in Congress, cross-Pacific cooperation potential, and more
Dispatches from Mitch
RSI as a service
There are times when the AI startup scene becomes a dark parody of its own worst impulses, and I can’t help but laugh.
I caught a full dose of gallows giggles today from this Wall Street Journal profile of Mirendil, a company founded by two ex-Anthropic researchers that just raised $200 million to sell AIs that self-improve at building AIs tailored to client needs.
Recursive self-improvement (RSI) — AI making smarter AI that makes smarter AI — is the most dangerous milestone any company could aim for, given the AI training methods in use. It could quickly lead to superintelligent systems clever enough to outmaneuver the people who think they own them. Because companies don’t know how to keep their existing AIs consistently safe and controllable, aiming for RSI right now is telling the universe that you’re getting your butt kicked on level 3 but would like to skip straight to level 20 — no, wait, 40 — oh, fine, make that 400.
The fact that most of the top AI companies have recently pointed to RSI as their goal and haven’t already been shut down by their governments is a grave indictment of our era — one I hope future generations will be around to look back on in horror.
But Mirendil’s pitch is a whole new tier of crazy. The big AI companies, remember, claim to be racing to superintelligence because they’re worried about less responsible actors getting there first; to that view, the only thing scarier than RSI is someone else doing RSI.
Mirendil wants to just give RSI to everyone.
I don’t want to oversell it. Their pitch is basically “instead of trying to train your own AI specialized in your area, have our AI train an AI to train your AI, etc.” I’d be willing to bet that this product never materializes — not because I think it’s impossible, but because I would expect one of the big companies to beat them to it or buy them out first.
I’m sure the latter outcome would be fine to the sort of amoral investors willing to fund such a socially irresponsible project in the first place. These include the venture capital firm Andreessen Horowitz, proud supporter of a bot farm and an anti-AI-regulation super PAC.
If you’re wondering, yes, Mirendil’s name is yet another bit of Lord of the Rings lore co-opted by a startup in a dark industry. (See mass surveillance analysis provider Palantir and autonomous weapons maker Anduril.) The Wall Street Journal explains that Mirendil is roughly Elvish for “friend of precious things.”
RAISE US
In what strikes me as evidence of a strong comms effort, many news outlets this morning simultaneously reported on the launch of RAISE US, a non-profit intended to help America’s workers “transition to an AI economy.”
When I saw who was involved, the media splash made perfect sense. This is a big, bipartisan, public-private partnership spearheaded by former Biden administration Secretary of Commerce (and former governor of Rhode Island) Gina Raimondo. She’s supported by political figures on the right, like former Republican governor of Indiana, Eric Holcomb; by top-tier philanthropists like Melinda French Gates and Laurene Powell Jobs; and by corporate giants including Amazon, Bank of America, Anthropic, Microsoft, and OpenAI. The group has raised $500 million.
The plan is to experiment, working closely with governors of states already trying out different programs to improve worker resilience, from retraining programs to “wage insurance.”
It would be easy for me to be cynical about RAISE US. On some level, it is almost certainly a form of charity-washing and misdirection from some of the groups involved — a thing they can point to as evidence that they’re doing something about some of the problems they’re causing. Based on descriptions of some of the programs, I can also imagine a lot of this money going right back to the AI companies in the form of programs that train workers to use their products.
And of course there is the issue of whether we will reach the sort of “AI economy” that one can “transition to” at all, if the AI race isn’t halted in time. When I first read the organization’s pitch, I pictured something like the Titanic transitioning to the bottom of the North Atlantic.
But I can’t help but like RAISE US’s experimental approach, and the scale of its ambitions. If or when governments and corporations reach a point where workers need a massive shot in the arm in order to keep our economy or our democracy from falling over, it will be good for us to already know what does and doesn’t work, and have funding channels ready to receive the infusion.
Growing awareness in Congress of AI’s catastrophic potential
In the brief-but-encouraging news bucket today, we have two items pointing to growing awareness of AI’s catastrophic potential within the halls of Congress.
POLITICO’s Kelsey Brugger documented the evolution of Ted Cruz, the Senate Commerce Chair with outsized influence over AI policy. In 2024, he was against applying regulation to AI at all. He was warning that “Big Tech and the Radical Left” would use AI regulation to (in the article’s terms) “empower the administrative state, kill innovation and cause the U.S. to lose the AI race with China.”
But now, Cruz’s staffers are saying the senator thinks government should take “targeted” action in cases that fall outside of existing law. These cases, according to Brugger, include “catastrophic risk, deepfakes and chatbots.”
Cruz also seems reluctant to continue pushing for federal preemption of state AI laws — an agenda item dear to the White House but very unpopular with voters and the senators who represent them.
Separately, Texas Rep. Nathaniel Moran plans to introduce a new AI Incident Reporting Act. As the name implies, the proposed legislation would require AI developers to report “dangerous capabilities, security breaches, and safety incidents” to the Commerce Department within seven days of their discovery, Reuters reports.
Moran is pitching it as a “catch-it-early and sound-the-alarm” bill.
The most severe threats from AI really need to be caught before they are built, not after there is an incident, but Moran is at least looking for the right things. Reuters says the bill’s “reportable activities” include:
a model attempting to evade human oversight, circumvent safeguards, and otherwise undermine the ability of human operators to control the model. It also includes unauthorized access to model weights, which help determine a machine’s decision-making, and chemical, biological, nuclear, and other threats to public safety.
Dispatches from Joe
Experts on both sides of the Pacific urge cooperation
WIRED’s Will Knight came back from a conference in Beijing with a message for all of us: When it comes to AI, China’s researchers are deeply worried too.
Frontier AI’s cybersecurity and systemic risks are too serious to ignore, and increasingly capable agentic models could soon cause chaos unless the world’s AI superpowers can work together.
To make this case, Knight cites experts on both sides of the Pacific. MIT computer scientist Stephen Casper observes that “AI is a global technology with global benefits, global harms, and a consistent tendency for new capabilities to eventually proliferate.” Like Billy Perrigo yesterday, Casper likens the current race to the Cold War and urges the U.S. and China to collaborate before it’s too late. “AI doesn’t need a Chernobyl moment,” he adds, echoing a recent warning by AI researcher Stuart Russell.
Casper cites research that proposes international cooperation on verification mechanisms, standards and best practices, AI infrastructure, and evaluations. This call is a familiar one; the MIRI technical governance team has made similar proposals, as in a 2024 paper on verification and evaluation.
Knight also quotes AI and security expert Lin Yun of Shanghai Jiao Tong University. “If different countries understand the risks in similar ways, it becomes easier to develop shared safety principles and technical standards.” There are signs that China does indeed see the same risks we do; I previously compiled a list of signals from China to that effect. Last year, China’s Vice Premier warned against “allow[ing] this reckless competition among countries to continue.”
Experts on both sides have long been calling for the U.S. and China to put aside their differences and coordinate a halt to the AI race. The Superintelligence Statement, which calls for “a prohibition on the development of superintelligence,” is signed by some of the most prominent researchers in the U.S. and in China.
Knight’s article focuses on the pressing dangers of cybersecurity and misuse, rather than extinction by rogue AI. The fact that many of the same experts worry about extinction only deepens the need for action. Collaboration between the U.S. and China lays essential groundwork for any international action on AI, and I hope policymakers on both sides heed their own scientists in this.
China piggybacks on American AI progress
Reuters reports on a letter that Anthropic sent to Congress two weeks ago which has just now been made public. The letter accuses Chinese tech giant Alibaba of attempting to “distill” Anthropic’s Claude AI.
Distillation itself is a common technique in machine learning research. In brief, it takes the outputs of one AI model and uses them as training data for another (typically cheaper) model. Anthropic claims to have caught Alibaba harvesting Claude’s outputs, in a recent campaign involving nearly 25,000 fraudulent accounts and nearly 30 million interactions.
I notice they conspicuously don’t say which model or models were accessed. Was Mythos, the potent cyber-hacker, among them? Would they admit it if it were?
Anthropic does say (correctly) that exposing their models to foreign distillation makes it more likely that bad actors will get access to Mythos-tier cyberattack capabilities. We’re already seeing a rapid rise in the capabilities of Chinese models available on the open internet, though it is debatable how strong a role distillation plays in their training.
Anthropic also doesn’t explain how they caught the distillation, nor why they failed to prevent it. But it shouldn’t come as a surprise; we’ve known for months that China has a thriving black market in cheap access to American AIs like Claude, and that security at the AI labs is not great.
It’s also interesting to note that this letter was sent just two days before the U.S. government issued a directive blocking access to Anthropic’s Mythos on June 12, worded in a way that forced Anthropic to cut off access entirely in order to comply. This was probably a coincidence; the directive doesn’t make much sense as a response to the letter.
The policies Anthropic actually proposed in the letter seem mostly sensible to me. They ask the U.S. government to help AI labs beef up their security (a long overdue step, mirrored in a White House memo we covered in April) and clarify antitrust laws to allow labs to collaborate on safety and security. They renew the call for export controls on AI chips, an important nonproliferation step. Both of these seem like no-brainers, and not just because of distillation.
Anthropic also proposes making it more costly for Chinese labs to distill American models, though the letter is somewhat vague as to how. I am less confident in the U.S. government’s ability to do this well, especially given some ambiguity about what exactly qualifies as distillation.
These steps are reasonable, and compatible with international cooperation on safety; negotiating with the Soviet Union during the Cold War did not mean giving up nuclear secrets.
But the elephant being ignored is Anthropic itself. It, and other frontier labs, are racing towards superhuman AI, dragging Chinese copycats along for the ride. The U.S. should take steps to make it harder for China to access dangerous capabilities, but this is not enough; our national security is at risk as long as those capabilities exist at all. And our survival as a species is threatened as long as AI capabilities continue to escalate beyond our ability to contain or control.
One of the main drivers of Chinese AI progress is American AI progress. Perpetually denying dangerous AI capabilities to rivals and bad actors while developing them ourselves isn’t really feasible. To quote my colleague Mitch, we may as well try to outrun our own shadow.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.