AI companies plan to hand the future to machines, writes Kelsey Piper in The Argument, and too many people fail to appreciate this fact.
These companies explicitly aim for recursive self-improvement (RSI), a process in which AI models design increasingly powerful successors. AI companies say this, loudly and repeatedly, in public and in private. They have already automated most of their coding work; they are eager to automate more.
Earlier this summer, OpenAI announced:
Over the past six months, the share of research compute devoted to internal coding inference grew 100-fold, while internal agentic token usage increased approximately 22-fold.
and a few days ago, it said:
We are making strong progress toward creating an automated AI researcher by March of 2028.
Anthropic likewise says it is “delegating a growing share of AI development to AI systems themselves.”
As Piper puts it, “AI companies are not, primarily, trying to automate your job — they are trying to automate their own jobs.”
What does this look like in practice? An ever-growing mass of poorly understood alien minds, far too many for any human institution to track, constantly self-enhancing in increasingly incomprehensible ways.
Just a couple weeks ago, OpenAI tasked more than ten thousand AI agents with a hard math problem. That is a fraction of a fraction of the total compute dedicated to AI research, and yet that single group contained more AIs than OpenAI has employees. They cannot possibly be looking hard enough at what those AIs are doing.
The AI companies seeking RSI are forced to rely on yet more AIs for monitoring — either the very same AIs that are being watched for misbehavior, or older and dumber models. All are unreliable, and any can lie or mislead.
The absolute deluge of AI activity is already a problem for investigators, too. When a mere 700 agents attacked Hugging Face, the third-party investigation had to lean on unreliable AI models to even begin to understand what happened.
Piper explains:
There simply isn’t enough human attention available to monitor the number of AIs the companies hope to put to work on independent AI research. As a result, we’ll be reliant on the AIs to understand what is happening inside fully automated data centers where more and more powerful AIs are developed.
This isn’t some pessimistic projection of what might go wrong — it’s actually the plan. Not the plan for the distant future; it’s the plan for next spring.
When policymakers and think tanks propose AI governance principles, they almost universally advocate for practices that keep humans in control of AI. It’s in the Asilomar AI Principles, the EU AI Act, and a UN framework on autonomous weapons. Even Chinese President Xi Jinping has urged “measures to forestall loss of control.”
Many of these proposals do not seem to grapple with the true problem, which is that AI companies do not have a workable plan which permits them to retain control. Their explicit and publicly declared approach, which they will evidently not abandon unless forced by law, is to ask some AIs to do their jobs for them, ask untrustworthy AIs to read everything and write reports, and hope their employees can keep up.
This plan is exactly as insane as it sounds, and it cannot be allowed to proceed.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.