The storm of AI news in the wake of Jacob Coxon’s high-profile resignation seems to have subsided to a dull roar, but there’s still more relevant to the extinction threat than can be adequately covered in a day. Here are some of the highlights:
Wake up, America
Republican Greg Murphy of North Carolina warns that concerns over AI dangers are real, nonpartisan, and urgent, Breitbart reports.
These are really smart people, not people with political proclivities that are pushing one way or the other... I do believe it’s something that we need to take the reins of, and I think it needs to be done right now.
Murphy said that AI governance “desperately needs national and international attention,” while still emphasizing the importance of staying ahead of China.
Appropriately, the weekday Newsmax show on which Murphy made these comments was called “Wake Up America Early.” I wouldn’t say Congress is early in addressing AI threats, but I think it’s not yet too late.
Sky nets
The Federal Aviation Administration is experimenting with a new AI-powered system for air traffic control, POLITICO writes. Several airports near Washington, DC (Reagan, Dulles, and BWI) will host the new system for a 90-day trial before it rolls out across the country. As my colleague Mitch pointed out in May, the system does not seem intended as a wholesale replacement for human controllers; it’s more of a tool that analyzes weather and flight plans and recommends time-saving tweaks.
For now, anyway. We have seen the same story play out many times already: first AI is barely able to do a task at all, then it can perform as well as an amateur, then it’s assisting expert humans, then it surpasses them entirely. (Famously, the early game-learning AI AlphaGo Zero passed all of these thresholds in a few days of training, having never once played Go against a human.)
The FAA’s “SMART” system is evidently at the “assist skilled humans” phase today, at least for a subset of routing tasks. That’s probably a good place to be, despite the extra attack surface AI integration offers to would-be infrastructure hackers. But as AI systems improve, they will likely be trusted with more control and less oversight by overwhelmed human staff. That all these systems increasingly depend on the same poorly understood AI creates what assessors might call “correlated risk.”
A rogue or compromised AI with access to a country’s air traffic control systems — and other critical infrastructure — could do an awful lot of damage. We’ve yet to see what can happen when AI really takes off.
Lean machines
Today Anthropic rolled out its latest AI model, Claude Opus 5.5, said to meet or beat prior models in key domains and to have cyber skills on par with those of the limited-access hacking genius Mythos. It’s also claimed to be less verbose and annoying to converse with.
Anthropic emphasizes efficiency gains in Opus 5.5; the headline metric is that the new model “costs 40% less to run than Opus 5” and can accomplish the same tasks as other models much more cheaply.
Hours later, OpenAI announced streamlined siblings for its GPT-6 Astra model, dubbed GPT-6 Sol and Luna. Again, the main focus seems to be on efficiency and cost savings rather than entirely new capabilities.
This efficiency is likely the result of algorithmic improvements, or new ways of arranging AIs so they can do more with less. Along with compute scaling (more operations with more chips), algorithmic gains are one of the major things driving AI capabilities. And at least some of these latest gains were likely developed by the AIs themselves.
In its announcement, Anthropic describes the measures it has taken to reduce the risks, including rerouting dangerous-seeming cyber and bio queries to weaker models like it does for Fable, the consumer-facing version of Mythos. It admits this won’t be enough to mitigate the dangers of later AIs...
For [future] models, we do not assume the measures described above will meet that safety standard on their own.
...but it doesn’t say it’s going to stop. Nor does OpenAI.
This is despite the AIs themselves sometimes warning the companies that what they are doing is dangerous. Anthropic’s latest system card tells us that “Claude Opus 5.5 often reasons that having input into training or deployment may be risky as a result of giving it too much capability or influence.”
As Madison Mills of Axios observed today, AI companies like Anthropic may make concerned noises about safety, but they have trillion-dollar incentives to push the frontier.
Freeze in place
In a personal writeup that also appears in the New York Times, former U.S. ambassador and national security advisor Susan Rice cogently summarizes the urgent need for AI governance:
Dario Amodei, CEO of Anthropic, and leaders of three other frontier AI companies have become so alarmed by rapid AI advances that they now pledge voluntarily to “pace” the development of new AI models, so that safety features can catch up. The firms are also promising to grant independent experts employee-level access to monitor and report on model development.
These are necessary and, if implemented, welcome steps. But they do not go nearly far enough.
“Pacing” amounts to the industry policing itself, while its leading firms simultaneously prepare for blockbuster IPOs. There is no way to ensure frontier companies will slow down sufficiently or invest enough in safety, which they have long subordinated to speedy development. So long as frontier training continues, the risk remains that additional accidents occur with devastating consequences. Moreover, as Amodei stresses, meaningful restraints must also apply to global competitors, particularly in China.
Much more can and must be done to ensure that the march toward super-intelligence does not result in catastrophic harm to humanity.
Rice calls on Presidents Trump and Xi to “freeze in place” AI training in the U.S. and China, and to arrange for scientists from both countries to develop solutions for the threats posed by superhuman AI.
She also suggests that:
In parallel, the U.S. and China should work intensively to reach a bilateral agreement on verifiable safeguards and acceptable uses of AI, then lead efforts to codify these agreements in a binding and verifiable international treaty.
...and she points out that the Chinese Communist Party has a vested interest in maintaining human control over AI systems that would strongly motivate China to cooperate on such an agreement.
With each new voice that calls for a halt to the deadly AI race, I feel my own hope for our future growing.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.