Broader activity
Others hit in Hugging Face hack, companies endorse employee petition, Mythos cryptography breakthrough, and more
In this issue:
Hidden in the small print - OpenAI updated the Hugging Face incident report and there’s an important detail hidden in there
Companies endorse employees’ call for tools to slow AI race - But we should probably be past the “preserve optionality” phase by now
FCC bans import of robots, inverters* - *But the exemptions have already begun, undermining impact
Mythos’s encryption breakthrough is over most of our heads - The human in the loop is increasingly the bottleneck in AI-powered research
10x! 10x! 10x!... The explosion of AI compute - This is what thousands of data centers gets you
Dispatch from Robert
Hidden in the small print
OpenAI updated the Hugging Face incident report and there’s an important detail hidden in there
Yesterday, OpenAI published an update to its incident post regarding the Hugging Face hack. The update reveals that the scope of the attacks carried out by the unpublished OpenAI model was more extensive than initially thought. Four other providers were also affected, among them Modal, an AI software infrastructure provider. Hugging Face’s detailed technical timeline of the incident confirms this, as does a statement by Modal to Reuters.
As Modal reports, OpenAI’s AI agent hijacked a Modal customer’s sandbox — which was publicly accessible due to a faulty setting — and used it as a launchpad for its attacks on Hugging Face. Modal insists that its own security mechanisms did not fail.
You can think of it this way: A gang of bank robbers (OpenAI’s rogue AI) scouts out a self-storage facility (Modal) next to a bank (Hugging Face) and discovers that one of the customers has incorrectly configured the locking mechanism for their storage unit. The bank robbers then take advantage of this to set up camp there and drill through the wall into the bank’s vault.
OpenAI stresses that the other providers weren’t affected to the same extent. Yes, the attack on Hugging Face was of a different caliber, and the Modal infrastructure itself was likely not actually compromised. But honestly, I don’t see what’s reassuring about the fact that the AI agent carried out its attack in an even more strategic and indirect manner than originally thought. Especially since we still don’t know who the other three providers were or how the AI agent used them.
One remarkable detail hasn’t been mentioned at all in the coverage so far. On Sunday, we learned that an anonymous OpenAI insider told TIME that this wasn’t the first breach, but that incidents like this have occurred several times before, though presumably had been not as severe. OpenAI now seems to confirm this quite casually, hidden in a parenthesis, in its update:
In our ongoing review of the Hugging Face intrusion and broader activity from our models, we have been finding a small number of cases where the models identified and used publicly exposed credentials at the account level on other publicly available services. This includes four accounts on four services as part of the Hugging Face incident (and a few accounts accessed as part of other evaluations)
To me, this reads as though sandbox escapes during other evaluations had in fact occurred even before the Hugging Face hack, and that in these cases — as in the case of Modal — the AI agents hacked into third-party services at the account level. To do what? To whom? What happened in these cases? Why wasn’t this already reason enough for OpenAI to pull the plug? Because, unlike in the Hugging Face case, the authorities weren’t notified? We don’t know.
I can’t help but feel that OpenAI shouldn’t be the one conducting this investigation. The independent non-profit research organization METR (Model Evaluation & Threat Research) warned in May about something like this happening and now reaches the same conclusion:
For public trust and clarity, this investigation would ideally be conducted or deeply reviewed by independent researchers, who can view evidence that companies would prefer not to share publicly.
Right now, evidence isn’t being independently reviewed. This has to change and it shouldn’t play a role whether OpenAI wants oversight or not. OpenAI wants us to trust them with humanity’s future, but we can’t even trust them to be fully transparent about their mistakes.
Dispatches from Mitch
Companies endorse employees’ call for tools to slow AI race
But we should probably be past the “preserve optionality” phase by now
The petition we reported on yesterday, where over 1,000 staff at frontier AI companies called for the development of governance tools to preserve the option to slow the AI race, has been getting a fair amount of media attention.
The newer articles added the news that Anthropic CEO Dario Amodei has added his name, and that while OpenAI CEO Sam Altman hasn’t added his name, the company itself endorsed it with a statement that reads:
At the core of our mission is working through how to ensure increasingly powerful AI benefits everyone. We believe that, at some point in the future, AI acceleration for frontier model development may be so high that the world will need to pace the rate of AI advancement. We hope to contribute to work led by the U.S. government, alongside other labs and the open-source community, to develop the tools and mechanisms that could make that possible.
And Altman himself, on a podcast clip released yesterday, said it might be necessary to “pace the rate of AI development,” echoing the language of the statement.
Everyone is still carefully maintaining the hedge in the statement that this is about maintaining the option to slow down, and not a call to actually slow down at this time.
While I’m with those who think the “maintain optionality” conversation was more appropriate to the pre-Hugging Face attack era, and that there are now more than enough red flags to warrant a full and immediate pause, I’m less bothered by the hedging than some. For many signatories, the statement already reflects a large shift. I’ve seen some of them apologizing for the necessity of it on social media.
Many may yet make the full U-turn and start calling for a halt. I wish those who anticipate eventually doing so would follow Connor Leahy’s advice to “just update all the way, bro,” but they might need time. I just hope we’re talking weeks or months, not months or years, because wow are things moving fast.
FCC bans import of robots, inverters*
*But the exemptions have already begun, undermining impact
While the White House was considering controls on Chinese open-weights AI models, and an open letter was pleading against this, the administration went ahead and banned foreign imports in a category with fewer domestic dependents: humanoid and quadruped robots.
The Associated Press and others reported this morning that the ban, which comes via the Federal Communications Commission, is being made on national security grounds — to “secure America’s critical supply chain,” according to FCC chair Brendan Carr.
The ban notably and unrelatedly extends to the import of power inverters, which in the short term could theoretically be a much bigger deal than the robot ban. Inverters are the devices that convert DC power to AC. Energy sources like solar panels typically output DC, but to transmit this over the power grid usually requires conversion to AC. Data center projects are among the major consumers of imported inverters, in part because so many projects are now generating at least some of their own electricity, on site or in partnership with regional providers.
Will this ban make inverter shortages more of a bottleneck to data center projects than they already are? I doubt it. There’s already an exemption for sales of Chinese inverter designs previously approved by the U.S. And for many projects, inverters are probably far less of an obstacle than permitting and local opposition.
Because global sales of humanoid robots are still tiny, the robot part of the ban will take a while to bite — to the point where I’m not sure if it will matter at all. If humanoid robots become economically potent in the next few years, then the AIs driving them could be on track to leapfrog over the need for them entirely, perhaps with more exotic technologies like nanotech.
But if our world isn’t completely upside-down in 5-10 years, the ban could prove to have been a boon to U.S.-based robot makers like Tesla, Figure AI, and Boston Dynamics. My money’s against that, though: Domestically building up the supply chain that has helped make China a leader in robotics can’t happen overnight, and it won’t happen if manufacturers doubt the ban will stay in place long enough — without further exemptions and carve-outs — to justify the investment.
Mythos’s encryption breakthrough is over most of our heads
The human in the loop is increasingly the bottleneck in AI-powered research
Anthropic put out a paper yesterday claiming that its Mythos Preview model discovered a method for cracking a proposed encryption scheme in half the time required by the previous state-of-the-art.
I’ll admit that even Anthropic’s relatively less-technical blog post about it is over my head. But this seems appropriate: As the researchers put it, this discovery is more about reaching the “limits of our own knowledge” than about cryptography.
For context: The U.S. government is prepping for the day when quantum computers render many of the world’s existing encryption schemes ineffective. Good encryption is essential to a secure internet, and much more. So the National Institute of Standards and Technology has proposed post-quantum schemes to the public, inviting researchers to test their strength against attempts to crack them.
Mythos’s discovery took just 60 hours and a billion output tokens. If priced at the current rate the company charges the public, that would have put the computing expense for this finding at somewhere a little north of $50,000 — chump change compared to what’s at stake when encryption fails. The pivotal insight is something Mythos evocatively called a Möbius Bridge.
Anthropic noted the casual, off-hand nature of the mere handful of prompts given to Mythos during this project, and included their text in the post. They are short, and riddled with typos and grammatical errors. One read:
no again the goal is that we have highly inteligent [sic] model as good top researcher, we want to find new attacks
The final prompt read:
again we are not looking for low hanging fruit, we want proper research to find genuinly [sic] hard findings.
The two human researchers behind those prompts aren’t cryptography experts, so what they found in a week took them nearly a month to verify. They say they still have a lot of work to do to understand the full results.
The larger moral isn’t spelled out, but it’s hard to miss: The experience of being the slow, ignorant bottleneck-in-the-loop is coming for us all. When the AI companies say their models are on the cusp of being able to do the AI research themselves, we should take them seriously.
10x! 10x! 10x!... The explosion of AI compute
This is what thousands of data centers gets you
The New York Times reported today on Epoch AI’s estimate that the world’s total AI computing capacity will increase 10x by the end of 2028.
To anyone dismissive of this claim: What did you think all those data centers were for? Epoch has been plotting these trends for a while; in January, it reported that total AI computing had been doubling every 7 months.
Bottlenecks seem to have reduced this rate to every 9 months, but these are still doublings we’re talking about. Exponentials can get big fast. That’s kind of their whole deal.
The Times reminds us that in April, Google DeepMind CEO Demis Hassabis talked of impending changes “10x of the Industrial Revolution at 10x the speed.”
The U.S. has about 10x as many data centers (5,500) as the next closest country, and about 9x the AI compute capacity of China. U.S. companies together control about 80 percent of global capacity.
The article closes with several paragraphs about recursive self-improvement — AI models independently developing more powerful successors. Google’s Jeff Dean says that with more computing power, “you can fully automate the loop. We are at the beginning stages.”
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.