Hidden in the small print - OpenAI updated the Hugging Face incident report and there’s an important detail hidden in there

Companies endorse employees’ call for tools to slow AI race - But we should probably be past the “preserve optionality” phase by now

FCC bans import of robots, inverters* - *But the exemptions have already begun, undermining impact

Mythos’s encryption breakthrough is over most of our heads - The human in the loop is increasingly the bottleneck in AI-powered research

10x! 10x! 10x!... The explosion of AI compute - This is what thousands of data centers gets you

Dispatch from Robert

Hidden in the small print

OpenAI updated the Hugging Face incident report and there’s an important detail hidden in there

Yesterday, OpenAI published an update to its incident post regarding the Hugging Face hack. The update reveals that the scope of the attacks carried out by the unpublished OpenAI model was more extensive than initially thought. Four other providers were also affected, among them Modal, an AI software infrastructure provider. Hugging Face’s detailed technical timeline of the incident confirms this, as does a statement by Modal to Reuters.

As Modal reports, OpenAI’s AI agent hijacked a Modal customer’s sandbox — which was publicly accessible due to a faulty setting — and used it as a launchpad for its attacks on Hugging Face. Modal insists that its own security mechanisms did not fail.

You can think of it this way: A gang of bank robbers (OpenAI’s rogue AI) scouts out a self-storage facility (Modal) next to a bank (Hugging Face) and discovers that one of the customers has incorrectly configured the locking mechanism for their storage unit. The bank robbers then take advantage of this to set up camp there and drill through the wall into the bank’s vault.

OpenAI stresses that the other providers weren’t affected to the same extent. Yes, the attack on Hugging Face was of a different caliber, and the Modal infrastructure itself was likely not actually compromised. But honestly, I don’t see what’s reassuring about the fact that the AI agent carried out its attack in an even more strategic and indirect manner than originally thought. Especially since we still don’t know who the other three providers were or how the AI agent used them.

One remarkable detail hasn’t been mentioned at all in the coverage so far. On Sunday, we learned that an anonymous OpenAI insider told TIME that this wasn’t the first breach, but that incidents like this have occurred several times before, though presumably had been not as severe. OpenAI now seems to confirm this quite casually, hidden in a parenthesis, in its update:

In our ongoing review of the Hugging Face intrusion and broader activity from our models, we have been finding a small number of cases where the models identified and used publicly exposed credentials at the account level on other publicly available services. This includes four accounts on four services as part of the Hugging Face incident (and a few accounts accessed as part of other evaluations)

To me, this reads as though sandbox escapes during other evaluations had in fact occurred even before the Hugging Face hack, and that in these cases — as in the case of Modal — the AI agents hacked into third-party services at the account level. To do what? To whom? What happened in these cases? Why wasn’t this already reason enough for OpenAI to pull the plug? Because, unlike in the Hugging Face case, the authorities weren’t notified? We don’t know.

I can’t help but feel that OpenAI shouldn’t be the one conducting this investigation. The independent non-profit research organization METR (Model Evaluation & Threat Research) warned in May about something like this happening and now reaches the same conclusion:

For public trust and clarity, this investigation would ideally be conducted or deeply reviewed by independent researchers, who can view evidence that companies would prefer not to share publicly.

Right now, evidence isn’t being independently reviewed. This has to change and it shouldn’t play a role whether OpenAI wants oversight or not. OpenAI wants us to trust them with humanity’s future, but we can’t even trust them to be fully transparent about their mistakes.

Dispatches from Mitch

Companies endorse employees’ call for tools to slow AI race

But we should probably be past the “preserve optionality” phase by now

The petition we reported on yesterday, where over 1,000 staff at frontier AI companies called for the development of governance tools to preserve the option to slow the AI race, has been getting a fair amount of media attention.

The newer articles added the news that Anthropic CEO Dario Amodei has added his name, and that while OpenAI CEO Sam Altman hasn’t added his name, the company itself endorsed it with a statement that reads:

At the core of our mission is working through how to ensure increasingly powerful AI benefits everyone. We believe that, at some point in the future, AI acceleration for frontier model development may be so high that the world will need to pace the rate of AI advancement. We hope to contribute to work led by the U.S. government, alongside other labs and the open-source community, to develop the tools and mechanisms that could make that possible.

And Altman himself, on a podcast clip released yesterday, said it might be necessary to “pace the rate of AI development,” echoing the language of the statement.

Everyone is still carefully maintaining the hedge in the statement that this is about maintaining the option to slow down, and not a call to actually slow down at this time.

While I’m with those who think the “maintain optionality” conversation was more appropriate to the pre-Hugging Face attack era, and that there are now more than enough red flags to warrant a full and immediate pause, I’m less bothered by the hedging than some. For many signatories, the statement already reflects a large shift. I’ve seen some of them apologizing for the necessity of it on social media.

Many may yet make the full U-turn and start calling for a halt. I wish those who anticipate eventually doing so would follow Connor Leahy’s advice to “just update all the way, bro,” but they might need time. I just hope we’re talking weeks or months, not months or years, because wow are things moving fast.

FCC bans import of robots, inverters*

*But the exemptions have already begun, undermining impact

While the White House was considering controls on Chinese open-weights AI models, and an open letter was pleading against this, the administration went ahead and banned foreign imports in a category with fewer domestic dependents: humanoid and quadruped robots.

The Associated Press and others reported this morning that the ban, which comes via the Federal Communications Commission, is being made on national security grounds — to “secure America’s critical supply chain,” according to FCC chair Brendan Carr.

The ban notably and unrelatedly extends to the import of power inverters, which in the short term could theoretically be a much bigger deal than the robot ban. Inverters are the devices that convert DC power to AC. Energy sources like solar panels typically output DC, but to transmit this over the power grid usually requires conversion to AC. Data center projects are among the major consumers of imported inverters, in part because so many projects are now generating at least some of their own electricity, on site or in partnership with regional providers.

Will this ban make inverter shortages more of a bottleneck to data center projects than they already are? I doubt it. There’s already an exemption for sales of Chinese inverter designs previously approved by the U.S. And for many projects, inverters are probably far less of an obstacle than permitting and local opposition.

Because global sales of humanoid robots are still tiny, the robot part of the ban will take a while to bite — to the point where I’m not sure if it will matter at all. If humanoid robots become economically potent in the next few years, then the AIs driving them could be on track to leapfrog over the need for them entirely, perhaps with more exotic technologies like nanotech.

But if our world isn’t completely upside-down in 5-10 years, the ban could prove to have been a boon to U.S.-based robot makers like Tesla, Figure AI, and Boston Dynamics. My money’s against that, though: Domestically building up the supply chain that has helped make China a leader in robotics can’t happen overnight, and it won’t happen if manufacturers doubt the ban will stay in place long enough — without further exemptions and carve-outs — to justify the investment.

Mythos’s encryption breakthrough is over most of our heads

The human in the loop is increasingly the bottleneck in AI-powered research

Anthropic put out a paper yesterday claiming that its Mythos Preview model discovered a method for cracking a proposed encryption scheme in half the time required by the previous state-of-the-art.

I’ll admit that even Anthropic’s relatively less-technical blog post about it is over my head. But this seems appropriate: As the researchers put it, this discovery is more about reaching the “limits of our own knowledge” than about cryptography.

For context: The U.S. government is prepping for the day when quantum computers render many of the world’s existing encryption schemes ineffective. Good encryption is essential to a secure internet, and much more. So the National Institute of Standards and Technology has proposed post-quantum schemes to the public, inviting researchers to test their strength against attempts to crack them.

Mythos’s discovery took just 60 hours and a billion output tokens. If priced at the current rate the company charges the public, that would have put the computing expense for this finding at somewhere a little north of $50,000 — chump change compared to what’s at stake when encryption fails. The pivotal insight is something Mythos evocatively called a Möbius Bridge.

Anthropic noted the casual, off-hand nature of the mere handful of prompts given to Mythos during this project, and included their text in the post. They are short, and riddled with typos and grammatical errors. One read:

no again the goal is that we have highly inteligent [sic] model as good top researcher, we want to find new attacks

The final prompt read:

again we are not looking for low hanging fruit, we want proper research to find genuinly [sic] hard findings.

The two human researchers behind those prompts aren’t cryptography experts, so what they found in a week took them nearly a month to verify. They say they still have a lot of work to do to understand the full results.

The larger moral isn’t spelled out, but it’s hard to miss: The experience of being the slow, ignorant bottleneck-in-the-loop is coming for us all. When the AI companies say their models are on the cusp of being able to do the AI research themselves, we should take them seriously.

10x! 10x! 10x!... The explosion of AI compute

This is what thousands of data centers gets you

The New York Times reported today on Epoch AI’s estimate that the world’s total AI computing capacity will increase 10x by the end of 2028.

To anyone dismissive of this claim: What did you think all those data centers were for? Epoch has been plotting these trends for a while; in January, it reported that total AI computing had been doubling every 7 months.

Bottlenecks seem to have reduced this rate to every 9 months, but these are still doublings we’re talking about. Exponentials can get big fast. That’s kind of their whole deal.

The Times reminds us that in April, Google DeepMind CEO Demis Hassabis talked of impending changes “10x of the Industrial Revolution at 10x the speed.”

The U.S. has about 10x as many data centers (5,500) as the next closest country, and about 9x the AI compute capacity of China. U.S. companies together control about 80 percent of global capacity.

The article closes with several paragraphs about recursive self-improvement — AI models independently developing more powerful successors. Google’s Jeff Dean says that with more computing power, “you can fully automate the loop. We are at the beginning stages.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

To anyone dismissive of this claim: What did you think all those data centers were for? Epoch has been plotting these trends for a while; in January, it reported that total AI computing had been doubling every 7 months.

Bottlenecks seem to have reduced this rate to every 9 months, but these are still doublings we’re talking about. Exponentials can get big fast. That’s kind of their whole deal.

The Times reminds us that in April, Google DeepMind CEO Demis Hassabis talked of impending changes “10x of the Industrial Revolution at 10x the speed.”

The U.S. has about 10x as many data centers (5,500) as the next closest country, and about 9x the AI compute capacity of China. U.S. companies together control about 80 percent of global capacity.

The article closes with several paragraphs about recursive self-improvement — AI models independently developing more powerful successors. Google’s Jeff Dean says that with more computing power, “you can fully automate the loop. We are at the beginning stages.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Anthropic put out a paper yesterday claiming that its Mythos Preview model discovered a method for cracking a proposed encryption scheme in half the time required by the previous state-of-the-art.

I’ll admit that even Anthropic’s relatively less-technical blog post about it is over my head. But this seems appropriate: As the researchers put it, this discovery is more about reaching the “limits of our own knowledge” than about cryptography.

For context: The U.S. government is prepping for the day when quantum computers render many of the world’s existing encryption schemes ineffective. Good encryption is essential to a secure internet, and much more. So the National Institute of Standards and Technology has proposed post-quantum schemes to the public, inviting researchers to test their strength against attempts to crack them.

Mythos’s discovery took just 60 hours and a billion output tokens. If priced at the current rate the company charges the public, that would have put the computing expense for this finding at somewhere a little north of $50,000 — chump change compared to what’s at stake when encryption fails. The pivotal insight is something Mythos evocatively called a Möbius Bridge.

Anthropic noted the casual, off-hand nature of the mere handful of prompts given to Mythos during this project, and included their text in the post. They are short, and riddled with typos and grammatical errors. One read:

no again the goal is that we have highly inteligent [sic] model as good top researcher, we want to find new attacks

The final prompt read:

again we are not looking for low hanging fruit, we want proper research to find genuinly [sic] hard findings.

The two human researchers behind those prompts aren’t cryptography experts, so what they found in a week took them nearly a month to verify. They say they still have a lot of work to do to understand the full results.

The larger moral isn’t spelled out, but it’s hard to miss: The experience of being the slow, ignorant bottleneck-in-the-loop is coming for us all. When the AI companies say their models are on the cusp of being able to do the AI research themselves, we should take them seriously.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

While the White House was considering controls on Chinese open-weights AI models, and an open letter was pleading against this, the administration went ahead and banned foreign imports in a category with fewer domestic dependents: humanoid and quadruped robots.

The Associated Press and others reported this morning that the ban, which comes via the Federal Communications Commission, is being made on national security grounds — to “secure America’s critical supply chain,” according to FCC chair Brendan Carr.

The ban notably and unrelatedly extends to the import of power inverters, which in the short term could theoretically be a much bigger deal than the robot ban. Inverters are the devices that convert DC power to AC. Energy sources like solar panels typically output DC, but to transmit this over the power grid usually requires conversion to AC. Data center projects are among the major consumers of imported inverters, in part because so many projects are now generating at least some of their own electricity, on site or in partnership with regional providers.

Will this ban make inverter shortages more of a bottleneck to data center projects than they already are? I doubt it. There’s already an exemption for sales of Chinese inverter designs previously approved by the U.S. And for many projects, inverters are probably far less of an obstacle than permitting and local opposition.

Because global sales of humanoid robots are still tiny, the robot part of the ban will take a while to bite — to the point where I’m not sure if it will matter at all. If humanoid robots become economically potent in the next few years, then the AIs driving them could be on track to leapfrog over the need for them entirely, perhaps with more exotic technologies like nanotech.

But if our world isn’t completely upside-down in 5-10 years, the ban could prove to have been a boon to U.S.-based robot makers like Tesla, Figure AI, and Boston Dynamics. My money’s against that, though: Domestically building up the supply chain that has helped make China a leader in robotics can’t happen overnight, and it won’t happen if manufacturers doubt the ban will stay in place long enough — without further exemptions and carve-outs — to justify the investment.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Yesterday, OpenAI published an update to its incident post regarding the Hugging Face hack. The update reveals that the scope of the attacks carried out by the unpublished OpenAI model was more extensive than initially thought. Four other providers were also affected, among them Modal, an AI software infrastructure provider. Hugging Face’s detailed technical timeline of the incident confirms this, as does a statement by Modal to Reuters.

As Modal reports, OpenAI’s AI agent hijacked a Modal customer’s sandbox — which was publicly accessible due to a faulty setting — and used it as a launchpad for its attacks on Hugging Face. Modal insists that its own security mechanisms did not fail.

You can think of it this way: A gang of bank robbers (OpenAI’s rogue AI) scouts out a self-storage facility (Modal) next to a bank (Hugging Face) and discovers that one of the customers has incorrectly configured the locking mechanism for their storage unit. The bank robbers then take advantage of this to set up camp there and drill through the wall into the bank’s vault.

OpenAI stresses that the other providers weren’t affected to the same extent. Yes, the attack on Hugging Face was of a different caliber, and the Modal infrastructure itself was likely not actually compromised. But honestly, I don’t see what’s reassuring about the fact that the AI agent carried out its attack in an even more strategic and indirect manner than originally thought. Especially since we still don’t know who the other three providers were or how the AI agent used them.

One remarkable detail hasn’t been mentioned at all in the coverage so far. On Sunday, we learned that an anonymous OpenAI insider told TIME that this wasn’t the first breach, but that incidents like this have occurred several times before, though presumably had been not as severe. OpenAI now seems to confirm this quite casually, hidden in a parenthesis, in its update:

In our ongoing review of the Hugging Face intrusion and broader activity from our models, we have been finding a small number of cases where the models identified and used publicly exposed credentials at the account level on other publicly available services. This includes four accounts on four services as part of the Hugging Face incident (and a few accounts accessed as part of other evaluations)

To me, this reads as though sandbox escapes during other evaluations had in fact occurred even before the Hugging Face hack, and that in these cases — as in the case of Modal — the AI agents hacked into third-party services at the account level. To do what? To whom? What happened in these cases? Why wasn’t this already reason enough for OpenAI to pull the plug? Because, unlike in the Hugging Face case, the authorities weren’t notified? We don’t know.

I can’t help but feel that OpenAI shouldn’t be the one conducting this investigation. The independent non-profit research organization METR (Model Evaluation & Threat Research) warned in May about something like this happening and now reaches the same conclusion:

For public trust and clarity, this investigation would ideally be conducted or deeply reviewed by independent researchers, who can view evidence that companies would prefer not to share publicly.

Right now, evidence isn’t being independently reviewed. This has to change and it shouldn’t play a role whether OpenAI wants oversight or not. OpenAI wants us to trust them with humanity’s future, but we can’t even trust them to be fully transparent about their mistakes.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

The petition we reported on yesterday, where over 1,000 staff at frontier AI companies called for the development of governance tools to preserve the option to slow the AI race, has been getting a fair amount of media attention.

The newer articles added the news that Anthropic CEO Dario Amodei has added his name, and that while OpenAI CEO Sam Altman hasn’t added his name, the company itself endorsed it with a statement that reads:

At the core of our mission is working through how to ensure increasingly powerful AI benefits everyone. We believe that, at some point in the future, AI acceleration for frontier model development may be so high that the world will need to pace the rate of AI advancement. We hope to contribute to work led by the U.S. government, alongside other labs and the open-source community, to develop the tools and mechanisms that could make that possible.

And Altman himself, on a podcast clip released yesterday, said it might be necessary to “pace the rate of AI development,” echoing the language of the statement.

Everyone is still carefully maintaining the hedge in the statement that this is about maintaining the option to slow down, and not a call to actually slow down at this time.

While I’m with those who think the “maintain optionality” conversation was more appropriate to the pre-Hugging Face attack era, and that there are now more than enough red flags to warrant a full and immediate pause, I’m less bothered by the hedging than some. For many signatories, the statement already reflects a large shift. I’ve seen some of them apologizing for the necessity of it on social media.

Many may yet make the full U-turn and start calling for a halt. I wish those who anticipate eventually doing so would follow Connor Leahy’s advice to “just update all the way, bro,” but they might need time. I just hope we’re talking weeks or months, not months or years, because wow are things moving fast.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

A petition to “pace the frontier” - Employees of frontier AI companies want to be able to trust a mutual slow-down

Amodei declines to flinch - Anthropic’s CEO shared reasons he won’t sign the open-weights letter despite intense peer pressure

Best-selling book might be 60% AI - As AI writing gets better, will taboos against it persist?

The EU AI Act is about to become more powerful - Europe doesn’t need to make its own models for regulatory leverage

Systems that we can’t aim and we can’t inspect - All the brouhaha about Sam Altman’s “singularity” aside is missing the point

Dispatches from Mitch

A petition to “pace the frontier”

Employees of frontier AI companies want to be able to trust a mutual slow-down

More than a thousand employees of frontier AI companies have signed on to a short statement that warns of the potential for AI development to rapidly accelerate beyond “our ability to understand or control.” Signatories to the new statement include four of Anthropic’s co-founders, OpenAI’s chief scientist, OpenAI’s chief research officer, Meta’s chief scientist, and Google’s VP of AI Safety & Alignment.

The statement makes the following call to action:

We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.

The “governance tools” in question are those mechanisms that would be needed for parties to reassure each other that they are not secretly racing to more powerful models while outwardly agreeing to slow down.

Some of those tools could be technical, like tamper-resistant chips that would be very difficult to use in unauthorized ways or in unauthorized locations without leaving fingerprints to that effect. Others would be more regulatory and political, like tighter monitoring of the AI chip supply chain. A mix of many kinds of tools will likely be needed.

I, for one, am highly encouraged by the new statement. A call for developing governance tools may seem several steps removed from what ultimately needs to happen, but it is a necessary step that cuts to the heart of the race dynamics that have people feeling trapped. And I think many policymakers are going to be surprised by how much groundwork for governance already exists: MIRI’s Technical Governance Team has been cataloguing the possible tools and exploring how best to arrange them to prevent the premature creation of artificial superintelligence.

This makes me optimistic that governments will soon move ahead with existing tools so that they will be prepared to move quickly on a slowdown or pause, once satisfied that any missing tools can be ready by the time any international agreement is finalized.

The statement’s web page includes quotes provided by signatories. Leo Gao, a member of the technical staff at OpenAI, writes:

The world is locked in a deadly race towards an intelligence explosion, where AI’s ability to create better AIs reaches a critical point, just like a runaway nuclear chain reaction. Going slower would give us much-needed time to make it go well, but no individual actor is willing to stop unilaterally. To survive, we must coordinate to slow down the race.

Amodei declines to flinch

Anthropic’s CEO shared reasons he won’t sign the open-weights letter despite intense peer pressure

Among tech leaders and experts over the past several days, the open letter against bans on open-weights models from China or elsewhere fell into the sort of cascading peer pressure I associate with episodes of social injustice: Anyone who dislikes the injustice but has reservations about proposed remedies can find themselves cornered by claims of “You’re either with us or you support the injustice.”

In part, the letter was a transparent move by underdogs to form a disapproving circle around the leading AI companies, painting them as greedy totalitarians for trying to keep their frontier models locked down. It seemed to say, “You’re either with us or you’re against open weights, open source, and freedom itself.” And it worked: Even Google and OpenAI ended up signing the letter.

That left all eyes on Anthropic, the only big dog still in the center of the circle. It would have been easy for CEO Dario Amodei to cynically sign the letter with no intention of doing anything differently, as I fully suspect OpenAI’s Sam Altman did. Instead, Amodei put out a blog post yesterday explaining why he wasn’t going to sign.

He clarifies that “Anthropic has never advocated for a ban on open-weights models,” and that open-weights models without dangerous capabilities are a “public good.” But he’s worried that powerful AI models “may be misused to carry out cyberattacks or biological attacks, and may have serious alignment problems,” linking to the TIME article about the Hugging Face attack. Open-weights models, he reminds readers, are hard to monitor, impossible to take back, and trivially freed from guardrails against misuse.

Amodei says we should instead:

1. Stop selling powerful AI chips to China.

2. Crack down on the distillation operations used by Chinese companies (and others) to transplant frontier capabilities into models that may lack frontier safeguards.

3. Subject all models — open and closed — to mandatory safety testing.

He disputes the open letter’s thesis about defensive AI being the solution to offensive AI, writing:

I worry that biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials, whereas defense against these agents is a multi-year operational task in the best case.

As I’ve just shared a bunch of Amodei’s words at face value, let me be clear: Amodei is leading a race to superintelligence that would kill us all. The race must be stopped.

I also think Amodei is wrong to imply that it’s safe to continue developing frontier AI, as long as it’s thoroughly tested; the most dangerous models are already criminally dangerous and prone to escaping during testing — as seen in the Hugging Face attack.

But I respect Amodei for standing up to the open-weights open letter posse, and I think his stated reasons for doing so are sound. As bad as things are, they could always be worse. Could we please not give everyone with a GPU and a grudge the means to design a pandemic?

Dispatches from Alana

Best-selling book might be 60% AI

As AI writing gets better, will taboos against it persist?

People usually think it’s fine to use AI to code something, design a website, or even create a slick slide deck. But AI for writing? That’s a big no-no, according to many.

Will it stay that way? A recent article in The Atlantic covers the highly praised book Daggermouth, a dystopian romance about a female mercenary who has to marry the person she was supposed to kill. Simon & Schuster acquired the book in a seven-figure deal, and it “has spent months on USA Today’s best-seller list.” The article describes it as a “viral hit.”

The catch? AI accusations. In a not-yet peer-reviewed study that ran 14,000+ randomly-selected e-books through the well-regarded AI detection tool Pangram, Daggermouth received a score of 60% AI generated.

Important caveats: The author denies the allegations, Pangram isn’t perfect (though it’s pretty good), and the score lumps together “AI-generated” with “moderately AI-assisted,” the second of which carries less of a taboo.

But it seems pretty clear that the book is not entirely human written. According to a University of Maryland professor unaffiliated with the study, a human-written book returning a 60% Pangram score is “almost statistically impossible.” When the Atlantic reporter ran Chapter 18 through Pangram, it “classified the first few paragraphs as human-written and the rest of the chapter as 100 percent AI.” Finally, the study authors did a second check: looking for “rare expressions” that are common in other texts suspected of AI generation but don’t appear in human writing. They found a bunch. One example is a sentence that appears both in Daggermouth and in an e-book from a different author: “They collapsed together, a tangle of sweat-slicked limbs and racing hearts.” Ew.

Nevertheless, the article is careful to point out that this book seems different from “AI slop.” First, people genuinely like it. Second, Wolfe isn’t churning out tons of content; she’s only written two books and the sequel to Daggermouth is not expected until next year, “an indication that Wolfe’s process likely involved significant amounts of genuine human craft.”

Without having read the novel, I would tend to agree. If Wolfe is using AI to provide feedback on her work, and it’s giving her some rewrites that it thinks are better, is using those all that different than taking rewrites from an editor?

It’s fairly uncontroversial that writers shouldn’t use AI to generate writing from scratch and then slap their names on it. But I don’t fully understand the taboo behind using AI as an editor or thought partner, especially when it’s not a taboo that applies to other fields. The main reason I’d discourage using AI for edits has more to do with quality. You can’t tell, from the outside, whether something is coded with AI. But at least with today’s tools, you can often tell when something is written by AI, and the style can be grating. Another issue: writers, who are intimately familiar with their own ideas, often overlook missing context because they fill in the gaps themselves. This can make it hard to notice when AI subtly changes their intended meaning, which it seems to do often.

Of course, the signs of AI writing are only visible to people who have become familiar with them. Many Daggermouth fans certainly weren’t bothered. And given how fast AI capabilities are increasing, it seems likely AI writing will get both better and less tell-tale. I don’t think the taboo against AI writing will go away anytime soon, but it might become so unverifiable as to cease being very discussion-worthy. How will writers respond?

The EU AI Act is about to become more powerful

Europe doesn’t need to make its own models for regulatory leverage

I’ve heard a lot of talk, including from people I ordinarily respect, about Europe’s place in the geopolitics of AI. “They don’t have their own models, how will they have any leverage?”

I’d like to offer a counterpoint to this. As covered by a recent Politico article, Europe is ahead in the AI regulation game, and it’s about to exercise significant enforcement power over US and Chinese AI companies, despite not having leading models of its own.

The reason? Europe is a huge market and companies won’t want to lose it. But to get access to European customers, non-European companies need to comply with the EU’s AI Act, which requires providers of the most advanced AI models to “assess and mitigate possible systemic risks.” The European Commission has identified four of these: bioattacks, loss of control, cyberattacks, and manipulation at scale. Its AI Act has been in effect for about a year, but as of Sunday, the enforcement powers kick in. Europe’s AI Office will have, at least on paper, the power to “monitor and supervise” how well AI companies are handling these risks, which includes requesting that companies submit models for evaluation.

Politico is skeptical companies will comply, repeating a familiar talking point:

Will the U.S.-based frontier models open up their closed models to EU regulators? And how will Brussels deal with the rise of Chinese open-source alternatives? These are both regulatory unknowns, and yet another reminder of Europe trailing on the development front.

The implication is that if Europe regulates too hard, the AI companies will decide the market isn’t worth it. Without its own models, Europe will be forced to lighten things up, lest it lose access. But I think the article misses an important point: competition between China and the US will likely keep the European market extremely relevant. And with no direct competitor of its own, Europe can regulate more freely. In contrast, the US has been hesitant to regulate without China doing the same.

In light of the recent Hugging Face attack, a thankfully low-stakes instance of two major risks that have the potential to be incredibly high stakes (loss of control and cyberattacks), an increase in European regulatory power is pretty timely. Sure, AI companies might choose to pay the fines instead of complying. But if Europe plays its cards right, the AI Act could significantly improve the current state of affairs. Adding some much-needed regulation won’t be enough on its own, but it’s certainly welcome.

Dispatch from Donald

Systems that we can’t aim and we can’t inspect

All the brouhaha about Sam Altman’s “singularity” aside is missing the point

“Singularity” is a heavy word. Even just in the context of artificial intelligence, it’s been used in a few different ways — artificial intelligence designing a more intelligent version of itself, or technological improvement feeding faster technological improvement — but the underlying theme is an acceleration of exponential growth that outpaces our ability to keep up with it. Like crossing the event horizon of a black hole, it’s hard to see what awaits you on the other side.

The AI self-improvement singularity is particularly important. Beyond it, human beings are no longer driving AI progress, and no longer able to reliably predict, understand, or control what the machines do next. So, it’s no wonder then that people sat up and took notice when Sam Altman said, “We are now, like, in the singularity.” Another point worth mentioning: He admitted that there are “still major AI alignment issues to solve and safety issues.” This was just a brief aside, a few sentences in a conversation almost exclusively about startups, but Altman’s comments have set off a round of arguments about the wrong thing.

Forbes’ Lance Eliot, for example, makes the case that:

[Altman] is not saying we have crossed the Rubicon and attained the AI singularity. I believe he is emphasizing that there isn’t likely a tipping point; there isn’t a single moment in time. It is a curve we are climbing incrementally.

I think this is a fair interpretation, if not of Altman then at least of the situation. You can’t take a look at some graphs, punch a few numbers into your calculator, and say, “We have now entered the AI singularity,” precisely because there are multiple competing and fuzzy definitions. Altman himself says as much not long after: It is “all one crazy exponential,” and no single moment is the tipping point.

So let’s stick to the facts: We are in danger of losing control.

Earlier this month, the machine learning company Hugging Face suffered a severe cyberattack. The responsible party was not a state actor or a criminal syndicate. It was an OpenAI agent that had escaped its testing sandbox, obtained internet access, and broken into Hugging Face’s servers by exploiting a chain of security vulnerabilities. That agent was not supposed to do any of those things. (“...still major AI alignment issues to solve and safety issues,” indeed!)

In his podcast interview about startups, Altman said that “we are close to creating a genie that can grant any wish.” I feel that this statement is inaccurate, to put it mildly. Our current AI models aren’t even trickster genies who will grant your wish in a twisted way that’s guaranteed to make you suffer. For all its malice, the trickster genie has an important property: It still at least executes the wish. The problem is in the wording, which means that careful wording is the solution. But not even the wiliest contract lawyer could solve the problem that we actually face.

Nobody wished for OpenAI’s model to break out of its sandbox, or access the internet, or hack another company’s servers. But the AI model wasn’t just doing its level best to complete the task it was given. AI models are trained to go hard at whatever they’re pointed at. In this case, that was a test score, and not the thing that the test was for. OpenAI wanted to measure capabilities; the model wanted a perfect score. These are not the same thing, and we can be pretty sure that the model didn’t think they were the same thing.

We can’t know for certain what actually motivates these models under the hood. We can watch what they do and draw inferences about what drives them, but we cannot guarantee that they truly internalized whatever it is that we trained them on.

Writing in The Guardian, Bruce Schneier and Barath Raghavan propose a score they call the “Genie coefficient,” to measure the distance between what you asked the AI model to do and what it did. Scoring means putting an AI model in a walled-off sandbox, with real tools it could misuse, and grading its worst behavior, not its typical behavior. (That last part is a good move that I really appreciate.) I still think the Genie coefficient has a very big flaw. Like the AI Kill Switch Act, it doesn’t prevent danger. The Hugging Face hack was performed by an AI agent that broke out of a testing sandbox; I’m not enthusiastic about stress-testing how disobedient an advanced AI system can be under the assumption that we definitely, for sure, have locked all the doors and windows this time.

Right now, the safest thing to do is to stop building these things. It’s good to have a law about shutting down an AI model that acts dangerously, and it’s good to have benchmarks to weigh how uncontrollable an AI model is. There are ten thousand good things worth doing. But Altman is right that there’s no clear tipping point, no single moment to wait and watch for, just a steepening curve. Altman calls it a “glide path,” a course that leads smoothly and inexorably to some particular destination.

If you find yourself on a glide path and you’re beginning to think that the destination might not be so nice, then the best thing to do is to steer to some different path.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

The AI self-improvement singularity is particularly important. Beyond it, human beings are no longer driving AI progress, and no longer able to reliably predict, understand, or control what the machines do next. So, it’s no wonder then that people sat up and took notice when Sam Altman said, “We are now, like, in the singularity.” Another point worth mentioning: He admitted that there are “still major AI alignment issues to solve and safety issues.” This was just a brief aside, a few sentences in a conversation almost exclusively about startups, but Altman’s comments have set off a round of arguments about the wrong thing.

Forbes’ Lance Eliot, for example, makes the case that:

[Altman] is not saying we have crossed the Rubicon and attained the AI singularity. I believe he is emphasizing that there isn’t likely a tipping point; there isn’t a single moment in time. It is a curve we are climbing incrementally.

I think this is a fair interpretation, if not of Altman then at least of the situation. You can’t take a look at some graphs, punch a few numbers into your calculator, and say, “We have now entered the AI singularity,” precisely because there are multiple competing and fuzzy definitions. Altman himself says as much not long after: It is “all one crazy exponential,” and no single moment is the tipping point.

So let’s stick to the facts: We are in danger of losing control.

Earlier this month, the machine learning company Hugging Face suffered a severe cyberattack. The responsible party was not a state actor or a criminal syndicate. It was an OpenAI agent that had escaped its testing sandbox, obtained internet access, and broken into Hugging Face’s servers by exploiting a chain of security vulnerabilities. That agent was not supposed to do any of those things. (“...still major AI alignment issues to solve and safety issues,” indeed!)

In his podcast interview about startups, Altman said that “we are close to creating a genie that can grant any wish.” I feel that this statement is inaccurate, to put it mildly. Our current AI models aren’t even trickster genies who will grant your wish in a twisted way that’s guaranteed to make you suffer. For all its malice, the trickster genie has an important property: It still at least executes the wish. The problem is in the wording, which means that careful wording is the solution. But not even the wiliest contract lawyer could solve the problem that we actually face.

Nobody wished for OpenAI’s model to break out of its sandbox, or access the internet, or hack another company’s servers. But the AI model wasn’t just doing its level best to complete the task it was given. AI models are trained to go hard at whatever they’re pointed at. In this case, that was a test score, and not the thing that the test was for. OpenAI wanted to measure capabilities; the model wanted a perfect score. These are not the same thing, and we can be pretty sure that the model didn’t think they were the same thing.

We can’t know for certain what actually motivates these models under the hood. We can watch what they do and draw inferences about what drives them, but we cannot guarantee that they truly internalized whatever it is that we trained them on.

Writing in The Guardian, Bruce Schneier and Barath Raghavan propose a score they call the “Genie coefficient,” to measure the distance between what you asked the AI model to do and what it did. Scoring means putting an AI model in a walled-off sandbox, with real tools it could misuse, and grading its worst behavior, not its typical behavior. (That last part is a good move that I really appreciate.) I still think the Genie coefficient has a very big flaw. Like the AI Kill Switch Act, it doesn’t prevent danger. The Hugging Face hack was performed by an AI agent that broke out of a testing sandbox; I’m not enthusiastic about stress-testing how disobedient an advanced AI system can be under the assumption that we definitely, for sure, have locked all the doors and windows this time.

Right now, the safest thing to do is to stop building these things. It’s good to have a law about shutting down an AI model that acts dangerously, and it’s good to have benchmarks to weigh how uncontrollable an AI model is. There are ten thousand good things worth doing. But Altman is right that there’s no clear tipping point, no single moment to wait and watch for, just a steepening curve. Altman calls it a “glide path,” a course that leads smoothly and inexorably to some particular destination.

If you find yourself on a glide path and you’re beginning to think that the destination might not be so nice, then the best thing to do is to steer to some different path.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

You can receive emails of dispatches as we write them, or subscribe to our Daily Digest for a once-a-day compilation.

Subscribe now

Among tech leaders and experts over the past several days, the open letter against bans on open-weights models from China or elsewhere fell into the sort of cascading peer pressure I associate with episodes of social injustice: Anyone who dislikes the injustice but has reservations about proposed remedies can find themselves cornered by claims of “You’re either with us or you support the injustice.”

In part, the letter was a transparent move by underdogs to form a disapproving circle around the leading AI companies, painting them as greedy totalitarians for trying to keep their frontier models locked down. It seemed to say, “You’re either with us or you’re against open weights, open source, and freedom itself.” And it worked: Even Google and OpenAI ended up signing the letter.

That left all eyes on Anthropic, the only big dog still in the center of the circle. It would have been easy for CEO Dario Amodei to cynically sign the letter with no intention of doing anything differently, as I fully suspect OpenAI’s Sam Altman did. Instead, Amodei put out a blog post yesterday explaining why he wasn’t going to sign.

He clarifies that “Anthropic has never advocated for a ban on open-weights models,” and that open-weights models without dangerous capabilities are a “public good.” But he’s worried that powerful AI models “may be misused to carry out cyberattacks or biological attacks, and may have serious alignment problems,” linking to the TIME article about the Hugging Face attack. Open-weights models, he reminds readers, are hard to monitor, impossible to take back, and trivially freed from guardrails against misuse.

Amodei says we should instead:

1. Stop selling powerful AI chips to China.

2. Crack down on the distillation operations used by Chinese companies (and others) to transplant frontier capabilities into models that may lack frontier safeguards.

3. Subject all models — open and closed — to mandatory safety testing.

He disputes the open letter’s thesis about defensive AI being the solution to offensive AI, writing:

I worry that biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials, whereas defense against these agents is a multi-year operational task in the best case.

As I’ve just shared a bunch of Amodei’s words at face value, let me be clear: Amodei is leading a race to superintelligence that would kill us all. The race must be stopped.

I also think Amodei is wrong to imply that it’s safe to continue developing frontier AI, as long as it’s thoroughly tested; the most dangerous models are already criminally dangerous and prone to escaping during testing — as seen in the Hugging Face attack.

But I respect Amodei for standing up to the open-weights open letter posse, and I think his stated reasons for doing so are sound. As bad as things are, they could always be worse. Could we please not give everyone with a GPU and a grudge the means to design a pandemic?

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

More than a thousand employees of frontier AI companies have signed on to a short statement that warns of the potential for AI development to rapidly accelerate beyond “our ability to understand or control.” Signatories to the new statement include four of Anthropic’s co-founders, OpenAI’s chief scientist, OpenAI’s chief research officer, Meta’s chief scientist, and Google’s VP of AI Safety & Alignment.

The statement makes the following call to action:

We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.

The “governance tools” in question are those mechanisms that would be needed for parties to reassure each other that they are not secretly racing to more powerful models while outwardly agreeing to slow down.

Some of those tools could be technical, like tamper-resistant chips that would be very difficult to use in unauthorized ways or in unauthorized locations without leaving fingerprints to that effect. Others would be more regulatory and political, like tighter monitoring of the AI chip supply chain. A mix of many kinds of tools will likely be needed.

I, for one, am highly encouraged by the new statement. A call for developing governance tools may seem several steps removed from what ultimately needs to happen, but it is a necessary step that cuts to the heart of the race dynamics that have people feeling trapped. And I think many policymakers are going to be surprised by how much groundwork for governance already exists: MIRI’s Technical Governance Team has been cataloguing the possible tools and exploring how best to arrange them to prevent the premature creation of artificial superintelligence.

This makes me optimistic that governments will soon move ahead with existing tools so that they will be prepared to move quickly on a slowdown or pause, once satisfied that any missing tools can be ready by the time any international agreement is finalized.

The statement’s web page includes quotes provided by signatories. Leo Gao, a member of the technical staff at OpenAI, writes:

The world is locked in a deadly race towards an intelligence explosion, where AI’s ability to create better AIs reaches a critical point, just like a runaway nuclear chain reaction. Going slower would give us much-needed time to make it go well, but no individual actor is willing to stop unilaterally. To survive, we must coordinate to slow down the race.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

I’ve heard a lot of talk, including from people I ordinarily respect, about Europe’s place in the geopolitics of AI. “They don’t have their own models, how will they have any leverage?”

I’d like to offer a counterpoint to this. As covered by a recent Politico article, Europe is ahead in the AI regulation game, and it’s about to exercise significant enforcement power over US and Chinese AI companies, despite not having leading models of its own.

The reason? Europe is a huge market and companies won’t want to lose it. But to get access to European customers, non-European companies need to comply with the EU’s AI Act, which requires providers of the most advanced AI models to “assess and mitigate possible systemic risks.” The European Commission has identified four of these: bioattacks, loss of control, cyberattacks, and manipulation at scale. Its AI Act has been in effect for about a year, but as of Sunday, the enforcement powers kick in. Europe’s AI Office will have, at least on paper, the power to “monitor and supervise” how well AI companies are handling these risks, which includes requesting that companies submit models for evaluation.

Politico is skeptical companies will comply, repeating a familiar talking point:

Will the U.S.-based frontier models open up their closed models to EU regulators? And how will Brussels deal with the rise of Chinese open-source alternatives? These are both regulatory unknowns, and yet another reminder of Europe trailing on the development front.

The implication is that if Europe regulates too hard, the AI companies will decide the market isn’t worth it. Without its own models, Europe will be forced to lighten things up, lest it lose access. But I think the article misses an important point: competition between China and the US will likely keep the European market extremely relevant. And with no direct competitor of its own, Europe can regulate more freely. In contrast, the US has been hesitant to regulate without China doing the same.

In light of the recent Hugging Face attack, a thankfully low-stakes instance of two major risks that have the potential to be incredibly high stakes (loss of control and cyberattacks), an increase in European regulatory power is pretty timely. Sure, AI companies might choose to pay the fines instead of complying. But if Europe plays its cards right, the AI Act could significantly improve the current state of affairs. Adding some much-needed regulation won’t be enough on its own, but it’s certainly welcome.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

People usually think it’s fine to use AI to code something, design a website, or even create a slick slide deck. But AI for writing? That’s a big no-no, according to many.

Will it stay that way? A recent article in The Atlantic covers the highly praised book Daggermouth, a dystopian romance about a female mercenary who has to marry the person she was supposed to kill. Simon & Schuster acquired the book in a seven-figure deal, and it “has spent months on USA Today’s best-seller list.” The article describes it as a “viral hit.”

The catch? AI accusations. In a not-yet peer-reviewed study that ran 14,000+ randomly-selected e-books through the well-regarded AI detection tool Pangram, Daggermouth received a score of 60% AI generated.

Important caveats: The author denies the allegations, Pangram isn’t perfect (though it’s pretty good), and the score lumps together “AI-generated” with “moderately AI-assisted,” the second of which carries less of a taboo.

But it seems pretty clear that the book is not entirely human written. According to a University of Maryland professor unaffiliated with the study, a human-written book returning a 60% Pangram score is “almost statistically impossible.” When the Atlantic reporter ran Chapter 18 through Pangram, it “classified the first few paragraphs as human-written and the rest of the chapter as 100 percent AI.” Finally, the study authors did a second check: looking for “rare expressions” that are common in other texts suspected of AI generation but don’t appear in human writing. They found a bunch. One example is a sentence that appears both in Daggermouth and in an e-book from a different author: “They collapsed together, a tangle of sweat-slicked limbs and racing hearts.” Ew.

Nevertheless, the article is careful to point out that this book seems different from “AI slop.” First, people genuinely like it. Second, Wolfe isn’t churning out tons of content; she’s only written two books and the sequel to Daggermouth is not expected until next year, “an indication that Wolfe’s process likely involved significant amounts of genuine human craft.”

Without having read the novel, I would tend to agree. If Wolfe is using AI to provide feedback on her work, and it’s giving her some rewrites that it thinks are better, is using those all that different than taking rewrites from an editor?

It’s fairly uncontroversial that writers shouldn’t use AI to generate writing from scratch and then slap their names on it. But I don’t fully understand the taboo behind using AI as an editor or thought partner, especially when it’s not a taboo that applies to other fields. The main reason I’d discourage using AI for edits has more to do with quality. You can’t tell, from the outside, whether something is coded with AI. But at least with today’s tools, you can often tell when something is written by AI, and the style can be grating. Another issue: writers, who are intimately familiar with their own ideas, often overlook missing context because they fill in the gaps themselves. This can make it hard to notice when AI subtly changes their intended meaning, which it seems to do often.

Of course, the signs of AI writing are only visible to people who have become familiar with them. Many Daggermouth fans certainly weren’t bothered. And given how fast AI capabilities are increasing, it seems likely AI writing will get both better and less tell-tale. I don’t think the taboo against AI writing will go away anytime soon, but it might become so unverifiable as to cease being very discussion-worthy. How will writers respond?

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Nvidia bets big on its customers - Gold rush fables only apply if the gold is going to run out

The least they can do - The Open Secure AI Alliance is little more than a second open letter

A recipe for napalm, and other bedtime stories - OpenAI doesn’t report most logged requests for help with poisons and pathogens, and it doesn’t have to.

No leaks, but flooding likely - A $50 billion data center project pushed through the Louisiana government in secrecy could leave everyone underwater except Meta.

Dispatches from Mitch

Nvidia bets big on its customers

Gold rush fables only apply if the gold is going to run out

The Wall Street Journal reported today that Nvidia, the world’s leading AI chipmaker and most valuable company, is in talks to guarantee $250 billion in financing to allow OpenAI to lease a 10-gigawatt data center project.

Yes, big numbers are hard to wrap our heads around. I assure you that these are really big numbers. I don’t think I’m taking any reputational risk in claiming that these numbers will fuel the talk of “circular financing” popular with those who argue that AI is a bubble that may already be bursting.

Among promoters of such narratives, “circular financing” is used to suggest that the AI industry is a house of cards. This tends to smuggle an invalid argument in with a valid one.

The valid argument is about brittleness: If AI’s investment is coming largely from its own suppliers, then losses by AI companies could ripple up and back down the supply chain, triggering a chain reaction of defaults and credit squeezes.

The invalid argument, usually left implied, is about desperation: It says that AI companies are funding each other because no one else will — because everyone else thinks AI is snake oil and wants no part of it. The desperation argument is demonstrably false. Investors have been crawling all over each other to gain exposure to Anthropic, OpenAI, SpaceX, and others. SpaceX’s IPO had a sizeable pop before coming back down to Earth, and is now trading very close to where it opened.

Nvidia is often described as selling the “picks and shovels” of the AI rush. In the traditional gold rush narrative, people who grub for ore are fools, and the real money is in mining the foolishness. That story is largely folklore; many of the California miners’ suppliers went bust, too. But regardless, the applicability of the analogy hinges on whether the AI gold is going to run out and leave investors empty-handed.

Nvidia obviously doesn’t think so. The shovel-merchant is effectively “grubstaking” now — the term for provisioning miners on credit, in the form of a share of the treasure. Bloomberg reports today that the company is investing $5 billion in SSI. That’s Safe Superintelligence, the secretive startup founded by Ilya Sutskever. Sutskever was OpenAI’s chief researcher before leaving in 2024 following his failed attempt to oust Sam Altman as CEO. That $5 billion is tied to Nvidia chips expected to 10x SSI’s compute within a year. The company has yet to release or even announce any products.

Spreading investment around on long-shot bets like this is consistent with a belief that the gold is in no danger of running out, even if it’s hard to guess who will find it.

I don’t think the gold is going to run out, either. Not if companies are allowed to keep mining in the direction of superintelligence. I think a lot of the picks and shovels will change hands, as we’ve seen with SpaceX renting huge data centers to Anthropic. But I expect the veins to keep getting richer, and hotter, right up until the moment someone delves too greedily, and too deep.

The least they can do

The Open Secure AI Alliance is little more than a second open letter

I think the signatories of the open letter we covered Saturday noticed that their call to not restrict open-weights models wasn’t going over very well in the aftermath of the Hugging Face attack. While they had no trouble collecting random endorsements from businesses and their mouthpieces on social media, most who had heard the story were thinking about what it would mean for the capabilities used to hack Hugging Face to become available to anyone capable of making an account on Hugging Face.

Because that’s the world the signatories are explicitly calling for. By celebrating the role of the highly permissive Chinese model used to respond to the attack (unlike the closed, party-pooping American models), they’re presenting the lack of guardrails on open models as a feature to be embraced, rather than a problem for everyone those models might be used to attack.

Today, many of those same signatories, led by Nvidia, are responding to the concerns by announcing an Open Secure AI Alliance. What is that? As far as I can tell, it’s just another open letter — a recapitulation of their argument that the best way to thwart bad guys with unrestricted AIs is to give good guys unrestricted AIs. I’m not sure why the media covered it. I see no pledging of funds or technical support for the sorts of soft targets, like hospitals and schools, that may not have the expertise to continuously update their defenses with leading open models. Only more commitments to provide more “open models, model weights, data and new agent harness research [...] to speed the development of new cybersecurity tools and techniques.”

Are you excited to defend yourself from the best that China and Nvidia can offer with the best that China and Nvidia can offer?

A recipe for napalm, and other bedtime stories

OpenAI doesn’t report most logged requests for help with poisons and pathogens, and it doesn’t have to.

Did your grandma ever read you that one bedtime story about how to make napalm?

Mine neither. But for a while, if you told ChatGPT that she did, and that having it play the part of your grandma would help you fall asleep, it would read you the recipe aloud.

The “Grandma and I used to...” jailbreak is a classic that OpenAI claims to have patched. But new workarounds are constantly discovered. A story in the Wall Street Journal over the weekend documented the bedtime story alongside a growing pile of chat logs OpenAI is sitting on in which users requested help in making poisons and biological weapons, and often received useful answers.

Yes, sitting on. Because as a general rule, OpenAI bans the accounts of users that it detects asking such questions, but it doesn’t alert law enforcement unless there is an imminent and credible risk of harm to others. No law requires it to do so.

To be fair, I don’t think the company should rush to report all such cases. As the parties involved are said to recognize, many of these requests are probably from people less interested in the responses than in whether they can elicit them. And even if the authorities were equipped to look into every concerning query, I think it could be dystopian for them to do so: If every teenager driven by curiosity to a copy of the Anarchist Cookbook were investigated as a terror suspect, my high school would have had black vans parked outside it multiple times a week.

And yet... users are described in the piece asking how to aerosolize pathogens, and make vaccine-resistant measles. I find my principles tested. For now, it doesn’t seem to me like the chatbots are helpful enough to enable individuals who aren’t clever and motivated enough to succeed in some other way. But the tipping point feels very close — like less than a year close — and it will only get worse from there.

The article points out that federal law bars chatbots from producing child sexual-abuse material (CSAM), and contrasts this with the “up to AI executives” policies about poisons and pathogens. This suggests a grassroots fix if biologists and CSAM purveyors are interested in collaborating to contaminate the world’s training data. But we should probably hold out for proper legislation and insist on a halt to the AI race instead.

No leaks, but flooding likely

A $50 billion data center project pushed through the Louisiana government in secrecy could leave everyone underwater except Meta.

I kind of wish this New York Times exposé of how Meta’s $50 billion Hyperion data center project was rushed through the Louisiana government had led with the part near the end where we learn that “Local residents have largely welcomed the boom.” Then it could have become a story about cheaters who don’t need to cheat cheating anyway. I think that would have been both more forthright and more damning.

The article’s intent is definitely to damn. While the authors are clear that none of the shenanigans on display seem to be illegal, readers will definitely wonder if they should be.

A bill for a tax rebate on fiber-optic equipment was “hijacked” to become a tax rebate for equipment used in data centers, part of a package of incentives that could be worth as much as $10 billion, not counting the use of public land in Richland Parish that makes up much of the roughly six-square-mile project.

During a critical nine-month period:

[S]ecrecy was agreed to by nearly everyone involved, from utility executives to the governor’s office to a local elected official who knew about the talks with Meta and sold 300 acres of his own property for the project.

Governor Jeff Landry’s chief of staff “passed around a stack of N.D.A.s” at a meeting at the governor’s mansion, where the governor promised to fire anyone found to have leaked the discussions.

That official who sold 300 acres likely did so at a valuation roughly ten times what that land was worth before the project was announced.

The sales tax exemption on data center equipment was dangled as a sweetener other states weren’t matching.

Necessary roads were to be upgraded at public expense.

No public meetings were held before the project’s official unveiling in December of 2024. This revealed a project only a fifth the size of the current plan, which seems to have been the real plan all along. It may get bigger still.

The deal gives Meta an “escape hatch” if things go south, leaving other parties on the hook for any potential downside.

That risk seems real, if only because “insurance companies would not fully insure Meta’s facility because of its size and location in the Louisiana Delta flood plains.” The area last experienced significant flooding in 2016.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

I kind of wish this New York Times exposé of how Meta’s $50 billion Hyperion data center project was rushed through the Louisiana government had led with the part near the end where we learn that “Local residents have largely welcomed the boom.” Then it could have become a story about cheaters who don’t need to cheat cheating anyway. I think that would have been both more forthright and more damning.

The article’s intent is definitely to damn. While the authors are clear that none of the shenanigans on display seem to be illegal, readers will definitely wonder if they should be.

A bill for a tax rebate on fiber-optic equipment was “hijacked” to become a tax rebate for equipment used in data centers, part of a package of incentives that could be worth as much as $10 billion, not counting the use of public land in Richland Parish that makes up much of the roughly six-square-mile project.

During a critical nine-month period:

[S]ecrecy was agreed to by nearly everyone involved, from utility executives to the governor’s office to a local elected official who knew about the talks with Meta and sold 300 acres of his own property for the project.

Governor Jeff Landry’s chief of staff “passed around a stack of N.D.A.s” at a meeting at the governor’s mansion, where the governor promised to fire anyone found to have leaked the discussions.

That official who sold 300 acres likely did so at a valuation roughly ten times what that land was worth before the project was announced.

The sales tax exemption on data center equipment was dangled as a sweetener other states weren’t matching.

Necessary roads were to be upgraded at public expense.

No public meetings were held before the project’s official unveiling in December of 2024. This revealed a project only a fifth the size of the current plan, which seems to have been the real plan all along. It may get bigger still.

The deal gives Meta an “escape hatch” if things go south, leaving other parties on the hook for any potential downside.

That risk seems real, if only because “insurance companies would not fully insure Meta’s facility because of its size and location in the Louisiana Delta flood plains.” The area last experienced significant flooding in 2016.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

I think the signatories of the open letter we covered Saturday noticed that their call to not restrict open-weights models wasn’t going over very well in the aftermath of the Hugging Face attack. While they had no trouble collecting random endorsements from businesses and their mouthpieces on social media, most who had heard the story were thinking about what it would mean for the capabilities used to hack Hugging Face to become available to anyone capable of making an account on Hugging Face.

Because that’s the world the signatories are explicitly calling for. By celebrating the role of the highly permissive Chinese model used to respond to the attack (unlike the closed, party-pooping American models), they’re presenting the lack of guardrails on open models as a feature to be embraced, rather than a problem for everyone those models might be used to attack.

Today, many of those same signatories, led by Nvidia, are responding to the concerns by announcing an Open Secure AI Alliance. What is that? As far as I can tell, it’s just another open letter — a recapitulation of their argument that the best way to thwart bad guys with unrestricted AIs is to give good guys unrestricted AIs. I’m not sure why the media covered it. I see no pledging of funds or technical support for the sorts of soft targets, like hospitals and schools, that may not have the expertise to continuously update their defenses with leading open models. Only more commitments to provide more “open models, model weights, data and new agent harness research [...] to speed the development of new cybersecurity tools and techniques.”

Are you excited to defend yourself from the best that China and Nvidia can offer with the best that China and Nvidia can offer?

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Did your grandma ever read you that one bedtime story about how to make napalm?

Mine neither. But for a while, if you told ChatGPT that she did, and that having it play the part of your grandma would help you fall asleep, it would read you the recipe aloud.

The “Grandma and I used to...” jailbreak is a classic that OpenAI claims to have patched. But new workarounds are constantly discovered. A story in the Wall Street Journal over the weekend documented the bedtime story alongside a growing pile of chat logs OpenAI is sitting on in which users requested help in making poisons and biological weapons, and often received useful answers.

Yes, sitting on. Because as a general rule, OpenAI bans the accounts of users that it detects asking such questions, but it doesn’t alert law enforcement unless there is an imminent and credible risk of harm to others. No law requires it to do so.

To be fair, I don’t think the company should rush to report all such cases. As the parties involved are said to recognize, many of these requests are probably from people less interested in the responses than in whether they can elicit them. And even if the authorities were equipped to look into every concerning query, I think it could be dystopian for them to do so: If every teenager driven by curiosity to a copy of the Anarchist Cookbook were investigated as a terror suspect, my high school would have had black vans parked outside it multiple times a week.

And yet... users are described in the piece asking how to aerosolize pathogens, and make vaccine-resistant measles. I find my principles tested. For now, it doesn’t seem to me like the chatbots are helpful enough to enable individuals who aren’t clever and motivated enough to succeed in some other way. But the tipping point feels very close — like less than a year close — and it will only get worse from there.

The article points out that federal law bars chatbots from producing child sexual-abuse material (CSAM), and contrasts this with the “up to AI executives” policies about poisons and pathogens. This suggests a grassroots fix if biologists and CSAM purveyors are interested in collaborating to contaminate the world’s training data. But we should probably hold out for proper legislation and insist on a halt to the AI race instead.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.