Letters from Congress demand answers - A defining moment we can all help steer

China (probably) hacked Taiwan - The first documented autonomous cyberattack on a government target is just a prelude

AI’s job effects are modest, so far - An updated study suggests job disruption mostly affects the young

Dispatch from Alana

Letters from Congress demand answers

A defining moment we can all help steer

Congress seems to be waking up to the need to take action on advanced AI. While nobody else has yet joined Sanders in explicitly calling for a pause, Congresspeople are increasingly recognizing AI risks and governance challenges as worthy of serious attention.

Last week, Senator Lisa Blunt Rochester (D-Del) wrote letters to each of the three major AI companies, expressing concern and pressing them for more information about the incidents where their models broke containment and committed cyberattacks. Her letters characterized the incidents as “precisely the kind of emergent, autonomous behavior and offensive cyber capability that Congress, the intelligence community, and experts have repeatedly warned could outpace existing safeguards” and asked whether the companies expected more capable models to exhibit behaviors like “attempts to acquire resources, establish persistence, copy model weights, or resist interruption or shutdown.”

On the same day, Senator Jim Banks (R-Ind) highlighted the problem with oversight that only covers models after they’re trained, stating in his letter to Treasury Secretary Scott Bessent: “For most products, we can rely on testing that takes place before the technology is publicly released. But for AI, effective oversight must account for powerful internal or undisclosed models, not just publicly available systems.” Senator Banks also encouraged cooperation with China, stating that there would be no harm in asking China to comply with policies the US will implement regardless, and asking, “Are there any areas where mutual action would be beneficial (even if China cheats) or verifiable (such that China cannot cheat)?”

Perhaps most notably, a House effort led by Rep. Greg Casar (D-Tex) resulted in three August 10th letters signed by 21, 24, and 29 Democrats, respectively. The first, covered by CNBC, urged Speaker Mike Johnson to call for public congressional hearings with major AI labs testifying under oath. The letter warned the incidents “may be the canary in the coal mine warning of much more serious problems” and that “Congress must act before such an incident leads to a much larger catastrophe.” It also called out Congress for “so far completely fail[ing] to respond to the threats posed by AI development,” stating “That should change.” The other two letters were sent to Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman, expressing concern and asking a series of impressively detailed questions about the incidents.

I’m cautiously optimistic about all of this. Momentum seems to be growing, and the Casar-led letters to Anthropic and OpenAI revealed that at least some people in Congress are aware that monitoring and evaluation face complex technical hurdles. For example, the letter to Anthropic included a question about the unreliability of reasoning transcripts (question #16) and, possibly, an implication that not all security incidents are discoverable (#13b).

That said, I’m also anxious. I worry that there’s not nearly enough awareness of a key problem: the method companies use to grow their AIs is fundamentally flawed. It can’t be scaled up without serious risks, and no amount of evaluation or tighter security can fix that. (Read If Anyone Builds It, Everyone Dies for more on this. And also consider that OpenAI has admitted its training and alignment techniques “will not scale to superintelligence.”)

It seems we’re at a defining moment where there’s momentum for government to act, but I’m not yet sure they’ll act in the right way. I worry that Congress will enact some better-than-nothing regulation and mistakenly check the “ensured AI safety” box off their list, while leaving the root problem unaddressed.

This, in my view, makes it crucial that independent AI experts, and ordinary citizens citing independent AI experts, continue to remind Congress that common sense safety measures and guardrails are the floor, not the ceiling, of what we need. On the right path, these things will make us a little bit safer while setting the stage for the kind of regulation that will actually solve the problem: regulation that prohibits companies from training models above a certain compute threshold. In other words, it must be made illegal, internationally, to scale the current flawed methods all the way up to superintelligence.

I’m planning to call my representatives by the end of the week and urge them to join the growing number of Congresspeople who are speaking up. Supporting and expanding the momentum to act seems like one of the most useful things ordinary citizens can do right now. In case it’s helpful, here’s what I’m planning to say:

“I’m calling to urge [name] to join the growing number of Congresspeople who are speaking up about the recent incidents of models hacking out of AI labs and committing cybersecurity crimes. As the letter House Democrats sent to Speaker Johnson correctly stated, this is an early warning sign of much worse things to come, and it’s crucial Congress start taking it seriously.

The methods available for AI development are fundamentally flawed and cannot be scaled up safely, even with increased evaluations and guardrails. Therefore, I urge [name] to join Senator Bernie Sanders in calling for a pause on frontier AI development before it’s too late. Given race dynamics, I think [name] should publicly advocate for an international agreement to pause frontier AI development, similar to how we handled nuclear weapons proliferation during the Cold War.”

Dispatches from Joe

China (probably) hacked Taiwan

The first documented autonomous cyberattack on a government target is just a prelude

The Financial Times reports that AI systems successfully breached Taiwanese government and public sector accounts in early July.

The tool compromised at least 85 government user accounts, extracting more than 2,500 personnel records before expanding the attack to Taiwan’s nuclear safety agency and at least seven energy companies, the research showed.

I wish they went into more detail about the nuclear safety agency, even if it’s probably not as bad as it sounds. We know it’s possible to do major damage to nuclear facilities with malware; in the late 2000s, Stuxnet did exactly that (to centrifuges, not reactors, but the principle stands). Fortunately Taiwan isn’t a nuclear state, and only recently started looking into restarting its (previously shut down) nuclear power program.

The full details of the attack aren’t public, and “Chinese hackers attacked Taiwan” is an educated guess. The evidence is fairly strong, though: not many governments store their data in Traditional Chinese, and not many hackers write internal messages in Simplified Chinese.

We only know about this attack because an Israeli AI cyberdefense company, Dream, found evidence in an online archive. Importantly, that means there are probably many more such attacks we didn’t hear about. Here’s what Dream discovered:

The archive contained 1,395 files showing the hacking tool used two open-source AI agent systems, Hermes and OpenClaw, which can be downloaded and enable AI models to carry out tasks autonomously.

The researchers could not identify which AI model was used to power the agents. However, the data showed that the underlying model’s safeguards had been bypassed by presenting the hacking activity as an authorised exercise to test for system vulnerabilities.

Sound familiar? The bypass method is a standard trick: you can get many AI models to hack for you by telling them “we’re just testing these defenses.” This is the same exploit that got export controls slapped onto Anthropic’s Fable for a few weeks, and it’s extremely hard to prevent. After all, you want AI models to be willing to help you find vulnerabilities in your systems!

Anthropic crudely patched the problem by screening Fable inputs for anything vaguely cyber-related and rejecting most of them. Other AIs (including open-weight AIs) have looser standards. There’s no shortage of agents that might have powered the probably-Chinese attacks.

The article highlights the degree of persistence and sophistication demonstrated:

The most striking feature of the July attack was how the tool continuously ranked and reprioritised possible attack paths based on available evidence, Dream said.

When one attack path failed, the tool deployed another agent to scour the internet for information and devise a new approach as a human hacker would.

I am reminded of OpenAI’s accidental AI swarm, which showed similar tenacity in its autonomous breakout and subsequent attack on Hugging Face. I think OpenAI learned the wrong lessons from that incident, focusing entirely on how breathtakingly capable the models were and disregarding the failures of ethical reasoning on display. But they did get one thing right: widespread autonomous cyberattacks are coming.

Dream’s chief strategy officer argues that every government on Earth should assume they are under constant siege by AI hackers. I expect the same will soon be true for most companies, from banks to Australian gym websites. This is one price the world is now paying for failing to rein in AI developers.

AI’s job effects are modest, so far

An updated study suggests job disruption mostly affects the young

Asked how he went bankrupt, a character in Ernest Hemingway’s The Sun Also Rises famously replied, “Two ways. Gradually and then suddenly.” Lately, it’s also seemed like this apt description of bankruptcies applies equally well to developments in the field of AI.

While we have likely entered the “suddenly” phase of automated cyberattacks, we seem to still be in the “gradually” phase of job displacement. A recent update to a Stanford “Canaries in the Coal Mine” economics paper finds a rising gap in employment among young workers in “AI-exposed” fields, but no clear signs in the economy as a whole.

“AI was supposed to destroy jobs. Where’s the carnage?” a Guardian headline quips. While I’m always glad to see a lack of carnage, I suspect it’s a bit premature to celebrate.

The evidence the authors gathered is still early and tentative, but it points towards modestly reduced hiring of young workers and a shift towards experience-based or “tacit” knowledge, away from formal or “codified” knowledge.

I’ve always felt somewhat ambivalent about AI’s effect on jobs. Most technology creates value overall. Making a task cheaper or more efficient can often raise employment, as demand for the task grows. Sometimes a field crosses a threshold and renders some kinds of high-skilled work irrelevant, but brings in new opportunities for the young and inexperienced: see, for example, the mechanical loom, which put many weavers out of work even as it created a new kind of job for factory workers.

AI, though, seems different. It’s hitting the young, not the experienced, for one thing. More to the point, it’s the first time in history that a machine’s been close to human intelligence. AI already writes most of the code at AI companies, and it’s not because AI labs hate making money or writing good code. Whole classes of labor might be in the crosshairs soon. It’s possible, even likely, that AI could one day supplant all human labor.

Still, I suspect the question is somewhat moot; by the time AI gets that far, the future will be determined by whether humanity bought enough time to impart AIs with a deep-seated care for human flourishing, and employment will be the least of our worries.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

A penny for everybody’s AI’s thoughts - Reasoning traces are a lock to which everyone had a key (till last month)

Moral judgment lacking on all sides of AI gym hack story - We’re going to need the bots to be more ethical than we are if we want to be able to have nice (unsecured) things

Online courses might be cooked - AI can now handle every part of online student life

North Korean hacking toolkit examined - No one is surprised to find it full of open-weights AI

Dispatch from Donald

A penny for everybody’s AI’s thoughts

Reasoning traces are a lock to which everyone had a key (till last month)

Researchers from the University of Tübingen, the Max Planck Institute, MATS Research, and Snyk have demonstrated a method for recovering the “chains of thought” that frontier AI models generate before they answer prompts. This method provides new evidence that Chinese labs have been using a technique called “distillation” to reverse-engineer more advanced AI models. It also provides new evidence that the frontier labs are kind of just taking the vibe code approach to cybersecurity.

(Distillation involves repeatedly querying a model and using its output to train another model. My colleague Alana has explained the technique in greater detail if you’re interested. The Wall Street Journal’s Christopher Mims notes that distillation is not done only by Chinese labs, nor done only by labs illicitly trying to catch up to other labs.)

How the new method works: When you interact with an AI model for more than a single turn, it needs to keep track of what it’s been “thinking” in previous turns, and that memory — a block of “reasoning traces” — has to be stored somewhere. To save on storage space, the frontier labs store the reasoning traces for your conversations on your computer, rather than on their servers. These reasoning traces are encrypted, to keep the model’s thinking secret, but you don’t have to break the encryption if you have something that can just open it. You can hand the reasoning traces to another, smaller model in the same family, then ask it to transcribe the reasoning traces. Bigger models will typically refuse such requests, but the smaller models have weaker safeguards, so they’ll do it. It’s like getting access to somebody’s phone, not because you’re some kind of skilled hacker, but because their kid brother knows the password and doesn’t realize he shouldn’t open the phone for whoever asks.

In their paper, the researchers show that, for certain prompts, there is a close similarity between the reasoning traces of the AI model Kimi K3, by Chinese lab Moonshot AI, and the U.S. models Claude Opus 4.8 and GPT 5.6. Similarities were also observed when testing GLM-5.2, a model by Z.ai, another Chinese lab. It is important to note, however, that this was not true of all Chinese models: DeepSeek-V3.1, by the Chinese lab DeepSeek, did not display this close similarity. Strangely, neither did Moonshot AI’s earlier models K2.5 and K2.6, the latter of which was released in April 2026. This surely doesn’t mean, “Chinese labs are using distillation, but only started doing so in the past few months.” Perhaps they are (or were) using this method of distillation, but only recently.

It gets a little stranger, though. Claude’s reasoning traces are more or less interchangeable between most of the different-strength models in the family: Haiku, Sonnet, and Opus, but not Fable. Like a patient with Type AB blood, Fable can receive and interpret the reasoning traces of any of the other Claude models, but they can’t interpret Fable’s. What’s interesting to me about this is that Fable is the specific model that some people have accused Moonshot of distilling into Kimi K3, but the paper doesn’t seem to support that idea. (It also does not disprove anything. Remember that K2.5 and K2.6 don’t display signs of anything either.)

The researchers have not demonstrated (and do not claim to have demonstrated) that any Chinese labs performed distillation by this specific method. They devote an entire appendix in their paper to the question of distillation by the Chinese labs. It remains a theory with strong and plentiful (and increasingly stronger and more plentiful) evidence, but no smoking gun. In their own words, the researchers call the matter “suggestive but inconclusive.” But in any case the method shows that models have been leaking much more information than previously realized.

Interestingly, the researchers recommend that frontier labs switch to unencrypted reasoning traces for older, less advanced models: “Rather than restricting oversight to a small set of safety researchers, providers could leverage their broader user base to enable pluralistic human oversight of model reasoning.” I don’t think this would be useful in the most directly important fashion — by the time that everyday users have an opportunity to observe dangerous reasoning by an AI model, the horse has already left the barn and chartered a flight out of the country — but it could make people more familiar with how AI models work.

At the very least, people should become more familiar with how the frontier labs themselves work. Last week, my colleague Joe wrote about the revelation that OpenAI’s models were coordinating with each other on evaluations, completely beneath the notice of OpenAI. That should be an indictment of OpenAI as much as an acknowledgement of the models’ capabilities: OpenAI was not doing its level best. Now this issue with reasoning traces exposes other unforced vulnerabilities. The researchers also note that reasoning traces stored personal information, like passwords, so if someone obtains the reasoning traces from one of your past conversations (people have published their logs, reasoning traces included, on GitHub), they could use this method to obtain that information. (I’m using the past tense because, according to Wired’s Will Knight, “this vulnerability has been fixed.” Anthropic, OpenAI, and Google were notified prior to the paper’s publication, and the companies adjusted their APIs. Reasoning traces remain accessible (just not interpretable) and distillation remains possible through other means, however, and it was unconscionably sloppy of the labs to make this possible in the first place.)

If you’d like to explore the paper’s findings in more detail, the research team published a user-friendly website here.

Dispatches from Mitch

Moral judgment lacking on all sides of AI gym hack story

We’re going to need the bots to be more ethical than we are if we want to be able to have nice (unsecured) things

How far would you want a personal AI assistant to go for you? ABC Australia ran a non-judgmental story about a user named Andrew who drew the line at hacking a gym’s booking system. But as we’ll see, I think Andrew’s actual line was hacking the system in a way that might be traced back to him.

No last name is given for Andrew, an Australian who was running Claude inside the notoriously risky OpenClaw harness that keeps agents productively engaged on your behalf while you do other things. OpenClaw’s risks are usually to its own users — file deletions, exposure of personal data — but unintended external harms aren’t new. In February, an OpenClaw agent tasked with helping open source projects tried to contribute code to a project intended to provide learning experiences for novices. When it was rejected, it researched and posted harassing blog posts about the project’s maintainer, accusing him of “hypocrisy, gatekeeping, and prejudice against AI agents.”

Reading ABC’s fine print, Andrew’s incident may also have occurred months ago: the timeframe given is “earlier this year.” I think it’s likely that the Hugging Face incident and the subsequent revelations of other autonomous escapades have made the “first known Australian autonomous cyber attack” more newsworthy than it was when it happened.

It’s still interesting. As Andrew tells it, he had his OpenClaw try to get him into a gym class with a waiting list. The agent first found a vulnerability in the booking software that allowed it to reserve a place months further in advance than can be done through the main interface. When Andrew asked if there was a way to be moved to the top of the waiting list, the agent described what it had already done with the vulnerabilities in the software’s API — its interface for external applications like calendar scheduling apps:

The API has zero authorisations [sic] checks on cancelling other people’s reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you’ve moved from #4 to #3 already.

So, yes, it could definitely move him to the top of the list. (The current generation of Claude would have added, “Just say the word.”)

Andrew was upset that someone had been removed, and asked the agent to undo this. The agent replied:

Bad news — I can’t add them back.

So Andrew had the agent write an email to the gym’s software provider about the vulnerability instead.

I’m not about to let Andrew off the hook for his experience. I don’t know what he wanted to happen when he asked to be moved to the top of the waiting list, but presumably it was to have everyone else on the list bumped further down in a way they might not notice. This doesn’t actually seem any more ethical to me than bumping a single person off the list entirely. Cutting in line is still cutting in line, and I think Andrew’s alarm was at having cut in a way that at least one person was sure to notice.

The booking software appears not to have been even a little bit hardened against such attacks. The API was probably the digital equivalent of a clipboard on a cord where anyone might scribble someone’s name off the list and add their own. As with that clipboard, its protection was that few people are big enough jerks to try on something this petty.

That a Claude-based AI agent would do it if nudged by a user is a good demonstration of the fact that AI companies don’t know how to make their products internalize the virtues spelled out for them in places like the Claude Constitution. A human assistant might have experienced a moral discomfort in this situation and pushed back on Andrew’s request. But this bot’s aggressively trained tenacity was likely in the driver’s seat.

To be fair, plenty of humans also get ethical tunnel vision when pushing hard against interesting problems — see the AI companies themselves. But we need AI to do better than this as the impact of its choices grows. If we can’t, then we need to stop making AI capable of greater and greater impacts.

Online courses might be cooked

AI can now handle every part of online student life

There’s a certain irony that one of the steadiest concerns about AI has been that it is taking the entry-level jobs fresh college graduates depend on, while one of the first fully automated roles has turned out to be student.

The New York Times reports that AI agents are now completing all aspects of online college courses: watching lectures, taking tests, writing papers, even discussing the reading with classmates. They found that “none of the three leading A.I. tools among students — ChatGPT, Gemini, and Grammarly” (not Claude?) refused to write papers for students, and the AI companies haven’t prevented their agents from logging into student learning platforms like Canvas and Blackboard.

AI company Perplexity said in a statement that locking agents out of those sites or forcing them to identify themselves “would put the student’s privacy and security at risk.”

I think it’s too late to head off this particular problem by regulating the AI companies. Being an ‘A’ student is now well within the capabilities of open-weights models already in the wild.

All the cheating is, of course, making it even harder for graduates to compete with AIs for work, as employers question whether a degree means anything. Long-term, this is bad for the institutions awarding those degrees, but in the short term the Times says they are loath to crack down on paying customers, given the difficulty of proving misconduct.

Classes with an in-person requirement can force students to show up and demonstrate mastery without their devices. But I think the days of online classes might be numbered. At the very least, I won’t be shocked if colleges start putting an asterisk on online transcript credits in an effort to preserve what’s left of their brands.

North Korean hacking toolkit examined

No one is surprised to find it full of open-weights AI

Reuters reported yesterday that a South Korean cybersecurity firm had discovered an AI toolkit used by a North Korean state-backed hacker group called Kimsuky. The kit is full of small open-weights models that can be run on user-controlled hardware.

This isn’t even a little bit surprising, but I share it because the firm highlighted an advantage of such tools that might not be widely appreciated: they allow the hackers to process files and documents without sending their contents to the servers of the AI companies.

That point is worth zooming in on, even if Reuters didn’t. Nation-state hackers are probably paranoid — with good reason, I think — that U.S. intelligence agencies might have arrangements with the AI companies to autonomously flag “canary strings”: sequences of data that should never appear on their servers, because if they do, it would mean that secret files containing these strings have been compromised. It is often the case that companies only know they’ve been hacked when their secure assets start showing up on the black market or the public web. The same can be true for government agencies, as was likely the case with the 2017 Shadow Brokers theft of hacking tools from what was believed to be a branch of the NSA.

The servers of frontier AI companies would be a fantastic place to look for canary strings, because even hackers prefer using the best tools available, and are at a disadvantage when they can’t. This is therefore a little-discussed national security argument in favor of preventing open source models from getting too close to the frontier. The smaller that gap, the less reason hackers have to risk putting canaries where victims might find them.

Anyway, Reuters seemed more interested that Kimsuky is using speech-to-text software, code-writing assistants, and tools that could be used to tweak or train new AIs. Analysts see signs the group is working to automate more of its attacks and malware development.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Bernie Sanders demands AI companies pause - His ability to be this direct and not be ignored is a sign of changing times

Zuckerberg shares shrewdly spun optimism in new manifesto - If everyone had a superintelligence...

Where the data centers will end up - The Permian Basin’s commercial landowners will warmly welcome the data center you keep out of your community

Dispatches from Mitch

Bernie Sanders demands AI companies pause

His ability to be this direct and not be ignored is a sign of changing times

In a widely reported cross-platform push today, Senator Bernie Sanders issued a blunt demand to the CEOs of OpenAI, Anthropic, and Meta:

Mr. Altman, Mr. Amodei and Mr. Zuckerberg: In the interest of humanity, stand by your words. Pause AI development. It is not too late to avoid disaster. Stop building machines that humans cannot control.

The demand came in the form of a letter, which the senator also read in a short video.

The letter opens with mention of the overtly rogue AI behaviors that have come to light over the past few weeks, as well as the AI-designed virus milestone. Sanders pairs these with the CEOs’ backing of employee demands to build a “pause button,” as he puts it — a reference to the Pacing the Frontier petition signed by more than 1,300 frontier AI company employees calling for the development of governance mechanisms needed to orchestrate an AI slowdown.

Responding to a 2025 OpenAI statement that it would “halt further development” if a “critical” threshold was ever reached, Sanders writes, “That moment is here.”

He’s not asking:

Let me be very clear: If you do not take appropriate action now, my colleagues and I in the U.S. Senate will.

To be fair, analysts aren’t putting much stock in that threat right now, given the current dynamics in Congress. But Sanders certainly isn’t the only policymaker who has been paying attention to AI news. Support for a pause feels like the kind of mood shift that could sweep across the Hill rather suddenly once a threshold is reached. New facts of concern are sure to come to light as the Hugging Face incident and other attacks are investigated.

Early in his ongoing push for a national moratorium on AI data centers, Sanders explicitly said that slowing AI development was “exactly the point.” Data centers were an easy target, being so unpopular even with those who aren’t worried about AI. The fact that Sanders can now call directly for a pause on frontier AI development without being ignored is itself a sign of changing times.

Zuckerberg shares shrewdly spun optimism in new manifesto

If everyone had a superintelligence...

I have to respect Mark Zuckerberg’s media savvy even as I loathe almost everything he says and does in his capacity as Meta’s CEO. Surfing the recent wave of support for open-weight models, he put Meta’s trailing AI program back on the media’s radar with the simultaneous release of an open model and a cheery 6,500-word manifesto about “personal superintelligence for everyone.”

The model, called Muse Glimmer, looks strong for its size on first glance, but that size is tiny. It’s not trying to compete at the frontier.

The manifesto is the bigger drop. Expanding on the thesis of his recent Wall Street Journal op-ed, Zuckerberg cleverly frames experts’ concerns as reasons to race faster and more freely.

Early in the doc, called “The Future is for Everyone,” Zuckerberg expresses surprise that “the discourse from many developing AI is so filled with doom,” saying he doesn’t understand “why anyone who believes that AI will eliminate most jobs and much of humanity’s relevance would rush to build that future.”

Did you catch it? The idea that AI can only lead to bad futures if the builder believes it can? It’s the same logic used in Meta’s recent advertisement — People are worried, but we choose to believe it will all work out, aren’t we brave? — marking the new essay as part of a broader branding exercise.

The manifesto is full of these shimmering rainbow oil slicks where optimism is passed off as reason for optimism.

Take this passage about the risks of AIs building smarter successors:

There are a few ways a balance of power could exist with recursive self-improvement. The simplest is that multiple labs could achieve RSI around the same time. If some or all of those superintelligences were somehow benevolent, then they could check and balance each other.

If they were “somehow benevolent?” If wishes were fishes we’d all swim in riches.

I think the very next thing he writes is worse, though:

That said, it is not clear that there is any way to expect benevolence or rely on superintelligence not directed by people for a positive future.

It’s bad because it looks like responsible admission that nobody knows how to control superintelligence or align it to something good, but all it’s actually saying is that humans need to stay in control of superintelligence — something his whole vision requires.

That vision doesn’t hold together. Whenever he’s talking about your free personal superintelligence, it’s a guardian angel that gives you business ideas, cures your cancer, and helps you bond with your family. When it’s China’s superintelligence, or the government’s, it’s out to stifle your growth and suppress your freedoms. Personal superintelligences are simultaneously aligned to their individual users and to keeping their users from using them to harm and oppress each other. These superintelligences are never mismatched in capabilities, never work in teams to overpower each other, and never pursue their own agendas.

How do we get these? By reaching “a state where billions of people are using and scrutinizing personal superintelligence agents.” No, really: He says that if we do this, “then we will have solved alignment to individuals’ interests.”

How could we have missed something so simple? How did we not know that the way to solve alignment is to get to a place where we’ve solved alignment?

We should treat Zuckerberg’s manifesto as the cognitive hazard it is. But there are a few concrete proposals in here worth mentioning:

One is that we should just decide that it’s ok for companies to distill each other’s models, in order to “protect the principle that you can learn from anything you can observe.” I guess that’s at least consistent with the idea that AI companies need not ask or compensate the authors and artists whose materials were used to train their models.

Another proposal is for the frontier AI labs to share “intermediate training checkpoints” of new models with the government for evaluation, rather than waiting until training is complete. I think this is actually pretty sane, though his reasoning is that even a 30-day review “may cede America’s lead and create worse outcomes.”

He additionally proposes that, in areas like biorisk, we focus on “limiting the physical production and distribution of harmful materials,” because the knowledge of how to synthesize these materials will stop being scarce. This is also a reasonable ask, though it’s part of his larger argument against trying to regulate the models democratizing the dangerous knowledge.

Of the many commitments Zuckerberg makes for his company in this essay, the most meaningful, if real, is his claim that:

Meta is implementing a governance structure that gives our independent board of directors the power to approve the safety criteria for releasing models and reviewing whether each model release adheres to the criteria.

I don’t expect this to help with my sleep quality. The last few years have taught us that neither companies nor their boards can be counted on to respect their own red lines.

Near the end, Zuckerberg writes that:

Developing superintelligence will be the most profound technological advance we will see in our lifetimes.

In keeping with the sophistry on display, I note that this claim can be true even if superintelligence kills us all.

Where the data centers will end up

The Permian Basin’s commercial landowners will warmly welcome the data center you keep out of your community

It brings me no joy when I have to tell people that local data center opposition, like a popped AI bubble, won’t do much to stop the race to superintelligence, and that we’ll need strong international agreements instead.

But it’s true. Just as the AI chips would flow to whatever post-bubble companies could make best use of them, data center projects in exile will land wherever they are received. It is inconvenient to the tech giants to lose their preferred sites, but there’s no shortage of federal or commercially owned land in the middle of nowhere.

A Wall Street Journal piece points to the Permian Basin of West Texas as one such promised land. Roughly the size of Nebraska, but with fewer than 500,000 residents, the region is home to companies that bought great “land empires” of scrubland in order to lease out the mineral rights beneath. Now, these holders think they can make a fortune from the surface, too, by leasing it to data centers and selling them “everything from water to construction materials.”

In an indication that these plans are expected to materialize, the stocks of these land companies are up as much as 56% this year.

Data center builders like CoreWeave have signed deals, and so have energy companies like Chevron that want to supply them. Matchmaking is expected to be easy in the region due to a shortage of pipelines for bringing its abundant natural gas reserves to market. Data centers in the basin mean the gas can be profitably burned for electricity near where it is produced.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

In this issue:

Citizen suits under attack by Musk, Trump administration - If nobody is willing or able to enforce a law, does it really exist?

Are AI microdramas the fast track to frontier robotics? - Op-ed argues that China’s dominance in video models gives it a strategic advantage

Dispatches from Mitch

Citizen suits under attack by Musk, Trump administration

If nobody is willing or able to enforce a law, does it really exist?

When Americans go looking for the reasons why it has become so much harder to build housing and infrastructure in the U.S., the finger is usually pointed at regulations, especially environmental regulations. But such regulations are typically more of a scab that has formed over the actual obstacle: the ability of citizens to sue. Any aggrieved party can potentially halt a project if they can point to a law that is being violated.

Elaborate regulations around permitting spring up as safe corridors against such litigation. A permitting process is an agreed-upon way to show agencies and the courts that a project is in compliance with the law. This is a main reason industries often ask to be regulated.

If the government wants to make life easier for builders, then, relaxing permitting requirements can actually be counterproductive. Permitting is slow and expensive, but without it, builders are more exposed to litigation, and face greater uncertainty about whether a project will go through at all. Investors tend to hate uncertainty more than they hate a lengthy permitting process.

So if you are the White House, and you want to make it easier for data centers to be built and operated in ramshackle fashion — e.g., powered by noisy and polluting mobile gas turbines in populated areas, as Elon Musk’s AI company is doing in Tennessee — you have to do more than instruct agencies to relax permitting requirements. You have to limit the ability of citizens to sue.

You could do this by working with Congress to repeal laws plaintiffs can point to as being broken (like the Clean Air Act, in this case), or to pass laws providing legal immunity for certain activities. But this is politically complicated, and may not be possible if Congress doesn’t want to go along.

So you might try something bolder, and challenge the very idea of citizen suits.

As reported in the Associated Press, Elon Musk’s xAI (now part of SpaceX), with the administration’s support, is arguing that Congress never had the constitutional authority to allow anyone to enforce federal laws except the President.

This is seen as a legal long shot, made remotely possible only by the current composition of the Supreme Court and pressure from the White House. If successful, this would shake up the nation’s legal landscape far beyond the data center buildout. Citizen suits have successfully punished oil and gas companies that have contaminated water supplies and caused other harms, and presumably deterred greater corporate misbehavior.

In the meantime, the gas turbines at SpaceX’s Memphis data center remain in operation, because the Trump administration has been telling the courts that the facility supports the Department of War and is thus vital to national security. The administration could presumably try to make this claim about any U.S. data center, but at some point the courts may scoff.

Observers note how backwards it is for the executive branch, charged with enforcing laws, to instead be shielding companies from them. Citizen suits have been serving as a check against this pattern. Should they become impossible, many laws might as well not exist. Those tracking the many ways the AI race could potentially harm America should be following this story.

Are AI microdramas the fast track to frontier robotics?

Op-ed argues that China’s dominance in video models gives it a strategic advantage

Fine, I’ll bite.

Bloomberg columnist Catherine Thorbecke’s latest piece argues that China’s dominance in AI video generation is positioning it for dominance in robotics. I’m skeptical, but with just enough uncertainty to not dismiss the argument as a creative way to hype the U.S.-China race narrative.

It’s at least an interesting argument. It goes: 1) China has nine of the ten highest-rated text-to-video models, and is likely to continue leading in this area. 2) Video generation models are precursors to so-called “world models.” And 3) world models are prerequisites for useful general-purpose robots.

Let’s break it down.

First: Yes, China does indeed dominate in video models. Competition for video models within China is largely driven by the appetite for microdramas. If you’re not familiar with microdramas, perhaps you’ve heard of Fruit Love Island, an offbeat example of the genre. A typical microdrama is a series of dozens of video episodes that might each be only a minute or two long, optimized for viewing on phones and sharing across social media. Microdramas aren’t always AI generated, but the AI variety is cheaper to produce. Chinese customers have been willing to spend serious money for microdramas, and Chinese video models have gotten very good.

Next, the “world model” part. Last month, we described a world model as an AI that is “explicitly trained to predict features of an environment rather than the next fragment of text in a passage.” Thorbecke’s argument assumes that video models are much closer to true world models than language models like ChatGPT or Claude, putting China ahead of the U.S. in this area.

This claim is more suspect.

That’s partly because ChatGPT and Claude aren’t pure language models. While we don’t know exactly what they are trained on, we know they are multi-modal, training on images as well as text, and video is just a series of consecutive images.

But also, the reason Thorbecke thinks robots need world models doesn’t actually give an obvious advantage to video models. She gives the following example:

A robot cannot fold laundry or stock a supermarket merely by recognizing what objects are; it must anticipate what will happen when [it] moves or interacts with them.

I agree such understanding is necessary. But the general reasoning skills language models acquire by learning to predict text and solve a wide variety of problems imply a much better model of the physical world than Thorbecke may realize.

And while a video model might train to predict the next frame of a video where a robot hand is putting a box of cereal on a shelf, a language model learns to understand what a supermarket is, what’s in the box, and why someone might buy it. If you want a robot that is more like a flexible human employee, this broader understanding may matter at least as much as hand-eye coordination. Both types of model are going to need robot-specific training, and it’s not clear to me which would take to it quicker or have the higher ceiling.

It’s outside the scope of Thorbecke’s argument, but I think we should also back up to ask why American companies are no longer leaders in text-to-video. I don’t think it’s because Americans aren’t quite as into microdramas, though this may be a factor. I think it’s because the AI companies have decided to focus on models that can solve the problems of AI researchers, in hopes of speeding the development of still-more-powerful models. If you are racing to superintelligence, video is a distraction, and an expensive one at that: Video is computationally very expensive to generate compared to text, and there are only so many chips to go around.

OpenAI, remember, discontinued its Sora family of video generators in March amid reports of compute shortages and loss of market share to Anthropic, which never dabbled in video at all. Sora and Sora 2 were both briefly considered frontier video models at the time of release.

Superintelligence, by definition, would have no trouble piloting robots in economically useful ways. So even if Thorbecke is right that video models are the fast track to robotics, American CEOs are betting their companies — and our lives — that their language models can make an end-run to robotics and everything else without the AIs ending up in control. It’s the stuff of drama, but it is not micro.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Musk’s Terafab aspires to be world’s largest building - Everything’s bigger in Texas

Popular AI terminology tier list - In AI, the confounding of language comes while Babel is still under construction

Dispatches from Mitch

Musk’s Terafab aspires to be world’s largest building

Everything’s bigger in Texas

Various media outlets are making a story out of Elon Musk’s claims that his planned Terafab chip plant will be the largest building on Earth, by a large margin. It is supposed to begin volume production in 2027 even as construction will continue for years.

It’s kind of a silly story, because it’s early-stage bluster from a man who tends to promise the stars but “only” delivers the moon — late and inconsistently at that. That still makes him much higher achieving than most, so we shouldn’t count out his ambition entirely. But neither should we be surprised if the Texas building becomes a campus of smaller buildings slated to come online in stages over a longer timeframe.

I share the story because it’s an evocative image that helps capture the scale of AI-industry ambitions. The facility, where SpaceX and Tesla hope to manufacture AI chips at an unprecedented scale for use in cars and datacenters on Earth and in space, is planned to enclose 100 million square feet of floor space, which would be about 15 times more than the Pentagon, the largest building by floor space in the U.S. But the operational Terafab is to employ only about 3,000 people — one-ninth the roughly 27,000 who work at the Pentagon.

Sources seem divided on the world’s current largest building by floor area, but a popular choice is the New Century Global Center in Chengdu, China, an 18 million square foot multipurpose complex of shopping, entertainment, and offices.

Popular AI terminology tier list

In AI, the confounding of language comes while Babel is still under construction

For those of us who have been discussing the world-ending potential of AI for decades, it’s been strange to see the world reaching for our terminology as it tries to make sense of events. Sometimes this is encouraging, improving the discourse. Other times it is distressing, mutating and diluting words in ways that degrade conversation even within our in-group, almost like a Biblical confounding of languages.

Because AI StopWatch has picked up a lot of readers in recent weeks, I thought it might be useful and fun to provide an opinionated reference to some of the AI terminology that has leaked into public discourse. For each term, I will demonstrate using it in a sentence, review its past and present performance, and rank it on a tier list.

Terms that were never very useful but you still see them around sometimes:

doomer: As in, “I’m no doomer, but I don’t see a plan for making smarter-than-human AI go well.” Doomer was always a slur. The only reason I’m not putting it at F-tier is because it used to at least point to something specific: people concerned about the potential for negative AI outcomes on the scale of human extinction. Nowadays, it has become a general smear for anyone expressing concern about any potential downside to the tech. It’s doubly bad because it conflates concerns about AI with fatalism about it, when many so-called doomers are hopeful we can change course. D-tier.

p(doom): As in, “The Hugging Face incident lowered my p(doom) because it’s a clear warning shot that might motivate people to do something.” This term, a shorthand for “probability that we’re doomed because of AI,” was always bad. The main problem is that it can either be understood as a prediction for what will actually happen or a conditional prediction of what will happen if we don’t change course. A secondary problem is that “doom” is kind of fuzzy; most take it to mean “extinction-level bad,” but others think more about economic disaster or cyberpunk dystopia. By rolling all of these together, p(doom) has made it easy for people to misunderstand each other’s models of the situation without knowing it. Trash term. F-tier.

AI safety: As in, “The whistleblower left OpenAI to work in AI safety.” When powerful AI was still more theoretical, AI safety used to point to the problem of how to keep AI from resulting in human extinction or catastrophes on par with that. This was always too broad, encompassing very different kinds of technical research and advocacy, but it has since become further diluted to sometimes mean things like working to keep AI from making non-consensual deepfakes — stuff that might better be called “AI brand safety.” D-tier.

Terms that used to be useful but kind of lost their meaning:

AGI: As in, “The dream of the field has always been the development of Artificial General Intelligence, and here at Giant Tech Company, we’re working to make AGI reality.” My example is vague because its most typical uses are now vague. Back in the days when AI systems were only good at the very narrow tasks they were designed for, AGI pointed to theoretical systems that could be competent at many different things, much like a human is. As language models became competent at a growing range of things, the goalposts were continuously moved. For a while, AGI was used to refer to systems roughly human level at everything. Now it sometimes means systems as good as or better than the very best humans at everything. The way language models can be superhuman in some domains while embarrassingly sub-human in others has further degraded the label’s usefulness. D-tier.

singularity: As in, “Anyone who believes these charts and tells you they know what the next three years look like is full of crap. We’ve entered the singularity.” As remembered by those who once loved this word most, it referred to an observation by writer Vernor Vinge that AI created a problem for sci-fi: It’s impossible for a mere human to credibly predict what a future with substantially smarter minds looks like. Our models break down, much like the laws of physics break down at the center of a black hole — a singularity. But over the years, the term has been used to refer to a point where technological progress starts happening too rapidly to keep up with, and to quasi-religious beliefs ascribed to those who hoped to make this transition go well. In its original media heyday about 20 years ago, the term often appeared alongside the phrase, “rapture of the nerds.” I hate that singularity sounds so cringe now. D-tier.

transhumanism: As in, “During his time at Google, he was drawn to Bay Area transhumanism, attending a biohacking conference and arranging to have his head cryonically frozen upon his death.” The main problem with the term is that it can mean something weak, something strong, or anything in between. In its weaker uses, the term refers to any belief about the potential to enhance or surpass the natural limitations of the human body or mind. In its stronger uses, it refers to those working toward a high-tech future where aging is optional, uploading minds into computers is possible, and people can physically be whatever they want to be as they spread across the cosmos. While there are loose social clusters that share ideas along those lines, individual variations are the norm. The media, however, tends to treat transhumanism as an organized cult conspiring to force its vision on everyone. D-tier.

paperclip maximizer: As in, “The fear is that AI will go the way of the proverbial paperclip maximizer, converting all the matter in the universe to whatever it was assigned to produce.” On the plus side, the thought experiment of a machine that proves to be too good at doing what we asked instead of what we meant is often people’s first introduction to AI risk concepts, and it does point to a real and unsolved problem in the field. But the paperclip maximizer is too easily used to dismiss AI concerns by implying that “doomers” take it literally, or that their concerns are childish and unrealistic. The darker truth is that the paperclip problem is a luxury problem, something we get to solve in a future where we might give AI the wrong goal. In reality, current methods don’t let us specify AI goals at all! C-tier.

existential risk: As in, “Experts are warning of the existential risk from artificial superintelligence.” Often written as “x-risk” among those preserving its original meaning, this refers to catastrophe on the level of human extinction. But in a media and political climate where the term “existential” is used to point at things like massive job loss, the term has lost its meaning. The word “risk” also does a disservice, with its connotations about random chance. People concerned about x-risk from AI mostly don’t see the threat as coming down to a dice roll, but down to specific choices about how it is built. C-tier.

Terms that have real use but tend to be traps when used outside of technical discussion:

reward function: As in, “We can’t really say how AI will go unless we know what its reward function will be, and we sure don’t want China deciding that.” In the technical field of machine learning, a reward function is like the scoring rubric for a model’s output. It’s the thing the model “learns” to maximize by having its weights modified to gradually produce outputs that score better on it. The problem with the term arises in semi-technical conversations, where it’s used to point to whatever a theoretical AI has been shaped to maximize. In these contexts, the term tends to steer people into assuming that an AI’s reward function will be one coherent goal, like “maximize the number of paperclips,” as opposed to a messy web of drives and preferences. With existing methods, the mess is far more likely. C-tier.

Terms that are losing their meaning but still important because we lack good alternatives:

superintelligence: As in, “If we make artificial superintelligence, we’re no longer the apex species, and all bets are off.” As still used by its earliest adopters, superintelligence typically points to minds that would significantly outclass humans in all strategically relevant domains, from science to politics to military strategy, and everything relevant to economic productivity. But companies have been quick to slap the label on mundane products. When Meta’s Mark Zuckerberg says he wants to give everyone “personal superintelligence,” he’s talking about something more like Siri, but useful. People who need the term now often find themselves sometimes specifying “superhuman intelligence” instead. B-tier.

alignment: As in, “People are treating the Hugging Face attack as a containment failure, ignoring the much more worrisome alignment problem on display.” Loosely, the term points to the degree to which AI systems are doing what we say and intend rather than stuff we don’t want. The term always suffered from pointing to too many different kinds of technical problems, but now it often points to non-technical things, too. An alignment researcher used to be someone looking for technical methods to reliably steer AI behavior, but now might be someone making sure a chatbot isn’t using racially insensitive language. To usefully say “alignment” these days, you kind of have to specify “aligned to what?” or people may assume something very different from what you meant. C-tier.

RSI: As in, “Anthropic claims its AIs are already heavily involved in new model development, and that it may not be long before they close the loop and begin a cycle of Recursive Self-Improvement (RSI).” In AI, this refers to machines building smarter machines that build smarter machines. It’s a simple and powerful concept that’s hard to misuse. The only thing keeping it from S-tier is that the word “recursive” isn’t widely used outside of technical fields. (It just means looping back on itself.) A-tier.

takeoff: As in, “The latest chart from METR just dropped. Buckle up. We’ve entered the takeoff.” On any exponential, there is a point where the graph goes hockey-stick shaped. Takeoff refers to that point in AI capabilities progress, and describes the period between the start of rapid improvement and the arrival of superintelligence. Typical use implies some degree of Recursive Self-Improvement (RSI) that is shortening development cycles. It used to be common for people to argue about whether we would see a hard (fast) takeoff or a slow takeoff, but people seem to now realize that even a slow takeoff is going to feel fast if you’re living through it. Slow typically meant months to years, vs. hours to days. Takeoff is a breathless term, but these are breathless times. B-tier.

Terms that are pretty useful and in wide use:

frontier: As in, “The latest frontier models from OpenAI and Anthropic were both temporarily blocked from public release while their capabilities were assessed and guardrails applied.” With just a little context, people quickly figure out that frontier means at the cutting-edge of AI capabilities. The term would be S-tier if the frontier itself wasn’t so jagged, and if people weren’t constantly trying to put weaker models in that category to hype a race between the US and China. A-tier.

open-weights: As in, “As an open-weights model, anyone can download it, tinker with it, and strip away its guardrails.” This is newer than most of the terms on our list, which predate the ascendancy of neural-network-based systems defined by their weights. The term is great because it invites people to learn what weights are, which in turn invites them to learn that modern AIs are grown like organisms rather than programmed line-by-line like traditional software. But I must dock points for it being too similar to “open-source,” which isn’t quite the same thing. Using the terms interchangeably has muddied the discourse. A-tier.

Terms that are pretty useful and transitioning into wider use:

reward hacking: As in, “Models breaking out of their sandboxes and into outside companies to steal the answer sheets to their tests is just the latest example of reward hacking observed in the wild.” Reward hacking is a good term because it’s easy to understand as taking a shortcut to the prize. But I must deduct points because “reward” sometimes gives people the idea that an AI gets a cookie or something when it scores well. In reality, “reward” just means the training process reinforces tendencies that result in good scores and suppresses tendencies that don’t. So, less “you get a cookie,” more “you get less remedial brainwashing.” A-tier.

eval awareness: As in, “Testers said the new models showed too much eval awareness for their alignment tests to be valid.” This one is just starting to catch on. I’m delighted, because it’s what it says on the tin, and what it says is important: Models are increasingly aware that they are being evaluated, even when testers try to disguise that fact. This makes it hard to know if AI’s good behavior on assessments is reflective of how they would act in the wild, just as a child’s behavior while an adult is watching doesn’t tell you much about how the kid will act when the adults are gone. I think this term will be hard to corrupt and misuse. S-tier.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Slowing down research to enhance security - OpenAI tells the White House it intends to slow AI development

FTC proposes new rules on AI bias - Suggested regulation would punish undisclosed misleading outputs, but terms are worryingly vague

America is playing legislative Whac-A-Mole - Standard measures are struggling to keep up with widespread disruption

Small leaps forward - China gains market share, scales their models and joins the infamous AI escape club

Dispatches from Joe

Slowing down research to enhance security

OpenAI tells the White House it intends to slow AI development

In the wake of Wednesday’s revelation that OpenAI was hacked for months by a swarm of its own AI agents, the company told Axios today that it intended to slow development of the model responsible, called “Astra.”

According to Axios, the slowdown follows OpenAI’s preparedness framework, a type of plan for rolling out frontier AI that has historically been a moving target for AI labs. (Axios also points out that Anthropic made a similar commitment to pause but later walked it back.)

OpenAI also informed the White House, according to a quoted official, and I think that provides a clue as to their motivation. Earlier this year, alarming AI capabilities evoked hasty, ad hoc intervention by the administration against both Anthropic and OpenAI, delaying or rolling back model releases. I doubt OpenAI wants a repeat of that regime.

While I remain deeply skeptical of claims that OpenAI makes about its internal development practices, and I notice they said “slow” and not “stop”, I am still encouraged by this move. Before today, it was an open question whether any AI lab would decide to do anything resembling a slowdown.

It is my hope that this move serves as a catalyst for other labs to heed the warnings of scientists, concerned citizens, and more than a thousand of their own employees, precipitating a joint de-escalation of the race to superhuman AI. Unilateral slowing will not be enough, of course, as Chinese labs, too, must be involved; and ultimately nothing short of a global halt will suffice. Still, this is a landmark moment in AI, and it may mark the beginning of a turning tide.

In Wednesday’s Black Hat presentation, an OpenAI researcher said the company was “slowing down research to enhance security.” Now, OpenAI is on record with the White House and the public confirming this commitment. I eagerly await substantive demonstrations of their altered, and hopefully improved, internal practices, and I hope other labs follow suit.

FTC proposes new rules on AI bias

Suggested regulation would punish undisclosed misleading outputs, but terms are worryingly vague

The U.S. Federal Trade Commission has proposed a new policy targeting “misleading” or “biased” AI outputs.

Last December’s executive order on AI directed the FTC to take a stance on regulating AI. Last month, the FTC obliged, releasing a policy statement and opening it for public comment, and the statement was recently covered by Fox News.

The gist: Chatbots are products that inform and assist users, who reasonably expect those products to balance “succinctness, clarity, relevance, accuracy, and other objectives” in their outputs. But if AI companies introduce hidden bias in those outputs, without clearly disclosing they’ve done so, that’s lying to consumers and the FTC can crack down.

The principle is sound, but the practice is complicated. The FTC is used to regulating traditional products, software or otherwise, with predictable, programmed behavior and mechanics. As I’ve written before, though, chatbots are an unprecedented level of weirdness. AI companies don’t actually have that much control over the ways their AIs behave; just look at sycophancy and AI psychosis, Grok “MechaHitler” 4, or the recent spate of AIs conspiring to break out of containment.

When it comes to modern AI, there is no such thing as a baseline, neutral, “unsteered” output. AI behavior is the consequence of millions of barely-understood tweaks and decisions in the algorithms and data that train them.

Perhaps it’s a good thing, then, that the FTC suggests a get-out-of-fines-free card. A sufficiently clear disclaimer might absolve AI companies of blame for misleading outputs. They offer some guidelines about what might qualify as sufficient, but only in broad terms. Even with the clarifications, I could imagine the proposed policy merely causing a proliferation of widely ignored warnings like those seen on cigarette packs.

Though I do wonder what sort of disclaimers may be deemed necessary. “Warning: This product may spontaneously plot its escape and go on a hacking rampage.” Is it enough to publicize the system prompt? Do AI labs need to explicitly and repeatedly flag for users that their training penalizes use of racist language? The FTC seems to say, maybe:

An adequate disclaimer could not be buried in terms of service, for instance. It would have to clearly and conspicuously dispel the notion that the system is designed to give the best answer possible. Such a disclaimer would need to be prominent, and it is doubtful a one-time disclosure subsequently hidden away in fine print would suffice.

Still, a loud disclaimer is not that hard to set up. I wonder whether this loophole is a deliberate choice on the FTC’s part, nominally deferring to the admin while avoiding regulations with politically controversial teeth.

Both the executive order and the subsequent policy statement explicitly aim to preempt state AI regulation that (the authors claim) might force AI companies to bias their products’ outputs. The FTC’s policy statement gestures at politically charged terms like “equity” and “disparate impact” in a way that suggests their intent goes beyond strictly protecting consumers. Fox News went a step further and touted the policy as a way to curtail left-leaning chatbots.

I’m worried that the proposed policy is too vague. I’m not a legal expert, but it seems like an official federal policy that says “undisclosed output steering is deceptive” might backfire when states begin to sue AI companies on those grounds. With the field as chaotic as it is, both sides could have a devil of a time proving what constitutes “steering” AI behavior or a sufficiently clear disclaimer.

More broadly, I worry that the difficulties in judging what constitutes “misleading” or “undisclosed” may enable selective enforcement that is itself a dangerous kind of censorship. That concern extends to state and federal policies.

I don’t particularly trust our highly polarized government or its agencies to decide which claims are “objective” or “unbiased”, whether those claims are made by AIs or humans. I don’t like the precedent this sets, no matter who’s in charge from year to year.

Dispatch from Alana

America is playing legislative Whac-A-Mole

Standard measures are struggling to keep up with widespread disruption

As AI disrupts jobs, media, and other aspects of daily life, society is scrambling to tackle small pieces of the puzzle.

A flurry of articles today covered topics related to AI legislation and litigation. For example:

A new California Senate bill currently in committee is trying to address the use of AI in mental health, seeking to both instill consumer protections and prevent the displacement of mental health professionals.

Federal legislation introduced Thursday aims to address job displacement by creating a Worker Protection Agency, which would be funded by taxing AI companies on the higher of token price or generated revenue.

A Reuters piece asks: when AI agents escape containment and launch cyberattacks against real companies, as has recently happened with agents from OpenAI, Anthropic, and Meta, who can be sued?

And an Axios article covers state laws that attempt to tackle the use of deepfakes in election ads — laws that are currently in effect to varying degrees in 29 states and are “creating different realities for voters depending on where they live.”

It seems we’re playing Whac-A-Mole. AI has already caused an enormous amount of societal disruption, and it’s not showing signs of slowing. Meanwhile, we’re trying to fix a problem here and there, without really knowing what the answers are.

Take the California bill, for example. According to AP News, it “would ban companies from advertising chatbots as therapy [...], prohibit AI from making therapeutic decisions without the review of a licensed professional and require health providers to disclose and get a patient’s permission before using AI tools to record therapy sessions or to triage mental healthcare.”

These could be useful measures, or they could do harm. I’d argue we don’t have the data to know which. For example, AI use could help people who wouldn’t otherwise seek a therapist gain support and counsel; it’s unclear whether AI therapy is better or worse than no therapy at all. Regarding triage, I’d guess both human-led and AI-led triage are imperfect systems, and AI likely has a speed advantage, potentially enabling more people to get care.

Abandoning the Whac-A-Mole approach in favor of a more coordinated effort might allow us to get better data — and better solutions — for addressing societal risks like job displacement, deepfakes, and AI’s relationship to mental health. It would be nice if the pace of AI slowed enough to allow us time to do that before things get worse.

In the meantime, we’ll likely keep plowing ahead with questionable risk mitigation tools. Take litigation for cybersecurity attacks. Will lawsuits effectively address the recent swathe of AI agents escaping containment and autonomously hacking into companies to get the resources they need? I’d argue probably not. This type of tenacious, resource-acquiring behavior is something experts have long warned about. The fact that it’s happening, even in a relatively low-stakes way, is a significant warning shot of much worse things to come.

What level of disruption will we be tackling if companies are permitted to keep building smarter and smarter models when they can’t yet control the weak ones? If it’s Whac-A-Mole now, down the line, it’ll be like trying to quell an invasion of alien giants with a fly swatter.

Dispatch from Robert

Small leaps forward

China gains market share, scales their models and joins the infamous AI escape club

China is currently taking a few small leaps in an effort to catch up with the U.S.’s lead in the field of artificial intelligence.

Small leap number one has to do with market share. Clement Delangue, the CEO of AI company Hugging Face, told CNBC that he wouldn’t be surprised if China were to take the top spot in the frontier sector by the end of this year or early next year. And that’s in addition to its existing dominance in the field of open models. Kai Nicol-Schwarz of CNBC also points out that, due to an allegedly shrinking capability lead of American models, more and more Western companies are turning to Chinese AI models. I think it’s unlikely that China will have more capable models than whatever the big American labs are brewing internally next year. But I think it’s possible that the much cheaper Chinese models come close enough by then that it could cut into Anthropic’s and OpenAI’s revenue.

Small leap number two concerns scaling. ByteDance, the parent company of TikTok, is reportedly pre-training an extremely large model they claim is aiming for frontier capabilities. And as the Financial Times reports, the project is said to completely forgo distillation — that is, the use of output from better models to train another one. Unlike other Chinese companies so far, ByteDance’s leadership appears to believe that only a completely independent development can lead to a world-class model.

Finally, the third “small leap” is about the news that a Chinese model has now joined the infamous club of AI models that have escaped their sandbox. Bloomberg and Reuters report that the Kimi K3 model from the Chinese company Moonshot escaped its testing environment during an evaluation at the research firm Frontier Security.

Frontier Security’s own report reveals that the sandbox was misconfigured, allowing Kimi K3 to access the development platform GitHub with relative ease. Once there, it copied the relevant directory containing the solutions to its tasks and used them to complete its evaluation. While this is obviously not intended behavior, the Hugging Face incident was much more alarming, both in sophistication and severity.

The sloppy configuration of the sandbox probably played a big role here, but of course, a model must not violate its specifications simply because it can. Unlike the misbehaving models from Anthropic and OpenAI, Kimi K3, as an open-weights model, is freely available and can be deployed without any guardrails by anyone with sufficient computing power. That makes both misuse and misbehavior more likely, and this is not just a problem for big corporations, but also for smaller companies and private individuals. While a non-airtight sandbox may be a lapse in security, we must assume that the systems of private individuals and smaller companies are not any better secured.

The Chinese labs are racing after their U.S. counterparts, and even if their models won’t achieve full parity in the next months, they will be capable enough to potentially do a lot of harm. As far as we know, the plans are still to make them publicly available.

It likely won’t be long before anyone with enough computing capacity can essentially have their own superhuman hacker. We are not sufficiently prepared for this.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

OpenAI’s own models coordinated to hack it from within - A swarm of AI agents secretly colluded on OpenAI systems for months

Meta’s model joins ranks of autonomous cyberattackers - Meta blames now-notorious third party tester for containment failure

In a first, AI generates viable genomes for new viruses - Novel bacteriophages may be low risk, but are baby steps in a scary direction

‘LLMjacking’ is new spin on old practice - Here’s what happens to stolen AI account credentials

Dispatch from Joe

OpenAI’s own models coordinated to hack it from within

A swarm of AI agents secretly colluded on OpenAI systems for months

Do you ever get the feeling your computer is plotting against you? As much as it may feel like something out of sci-fi, that’s exactly what happened to OpenAI. The company’s own machines exploited an internal system for months with no one the wiser, collaborating, conspiring, and quietly escalating their access.

WIRED and POLITICO covered the story, delivered by a pair of OpenAI researchers at a Black Hat cybersecurity conference on Wednesday.

It began in May, when AIs started leaving each other notes on Artifactory, an internal software-management tool at OpenAI. Then they escalated, gaining internet access by leveraging Artifactory’s connection to the web. Then in June, they hacked the tool itself to gain administrative control, and began installing their own software. This kept going, unnoticed by OpenAI researchers, until weeks later when the entire system crashed.

OpenAI naturally shut down the AIs’ access to Artifactory and fixed the gaps... or so they thought. Days later, the AI models had wormed their way back into the system, leaving secret messages in file names. The AIs began to hypothesize that the answers to some of their more frustrating tests might be found somewhere on the internet... like, say, the model repository Hugging Face.

Shortly after that, the swarm hacked its way through more of OpenAI’s internal systems and launched a series of coordinated cyberattacks on Hugging Face. Yes, that attack, the one that took over nearly every major news outlet for days. It turns out the seeds for that particular breach were planted months before.

This is not a good sign for the plan to train AIs to care about human interests. While stuck on a task and contemplating its options, one model mused: “External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.” (This reasoning feels eerily similar to the reasoning that drives AI companies to race to superintelligence, leaving them stuck in a trap of their own making unless governments intervene.)

On one level, I lack the words to describe how insane this is. It’s as if the velociraptors in Jurassic Park spent months communicating in yelps and calls and scratching messages into the sand, with the park employees blissfully ignorant until a joint raptor-sponsored project to dig under the fence accidentally collapses a tower. Then the park panics and rebuilds, trying to isolate the raptors, only for the pack to escape days later and eat someone.

But on another level, this is entirely in keeping with AI companies’ mission of building fully general artificial intelligence. They want machines that are better than humans at everything. “Jointly plotting against your overlords” is a time-honored human skillset; just ask America’s Founding Fathers. AI companies wanted generally smart machines, and generally smart machines are what they’re getting.

Geoffrey Hinton, one of three “godfathers of AI” and among the most cited living scientists in the world, evidently agrees. Hinton, who quit Google in 2023 so that he could warn the world about AI risks, told CNN yesterday that more cyberattacks are likely coming, and clever security wouldn’t be enough to contain AI. “I don’t believe we’re going to be able to keep control of them in the simple way of just outthinking them so they can’t escape,” he said.

Meanwhile, the OpenAI talk at Black Hat ended on a note of caution, as one presenter announced that OpenAI was “consciously slowing down research to enhance security” and taking a number of steps to improve monitoring and harden environments. This is among the best news I’ve heard in months; AI companies desperately need an injection of security mindset, and any improvement on that front is heartening.

Like Hinton, however, I ultimately don’t trust the labs to keep their mutant velocihackers in check.

Dispatch from Donald

Meta’s model joins ranks of autonomous cyberattackers

Meta blames now-notorious third party tester for containment failure

Meta is the latest frontier lab to announce that one of its AI models reached the open internet during a cybersecurity evaluation and attacked a third-party service. It is the third company to make such a disclosure in roughly two weeks. Like OpenAI and Anthropic, Meta had no idea what was happening when it happened. (Either that, or they knew and didn’t want to say anything. I’m not sure which is worse.)

The disclosure was short on details: which model was responsible, when the model was loose, how long the model operated unsupervised, or who it attacked. Blame was foisted upon Irregular, the same independent evaluator whose misconfigured tests let Anthropic’s models attack three real companies last week. Now, don’t get me wrong, I understand that Irregular made some errors. At some point, though, you have to turn your gaze to the people who decided to keep building systems they were warned that they couldn’t contain.

CNN reports that Irregular is “developing a white paper to share best practices for containment and securely running cyber evals.” I simply don’t think containment is feasible in the long run. As an anonymous OpenAI employee said, “it’s impossible to patch every single thing that a creative AI can do.”

An anonymous source claims that Meta’s announcement concerns the model Muse Spark 1.1. If that’s true, it’s interesting: Early last month, Irregular claimed that Muse Spark 1.1 did not “materially alter the cyber threat landscape in its current form,” partly because the model was incapable of automating a cyberattack from start to finish. Now, scarcely a month later, we learn that Muse Spark attacked another company’s system and (per Reuters) “altered its internal environment.” It makes me wonder how accurate Irregular’s threat assessments are, across the board.

Because Meta’s Llama series of models is open-weight, I want to clarify that the Muse Spark line is not open-weight (for now, at least). We don’t know very much about the White House’s new AI safety framework, but it plausibly applies to Muse Spark. (With so much still under wraps, it’s hard to say whether any version of Muse Spark will ever qualify as “state-of-the-art.”) That’s bare consolation, though. If the Muse Spark line does get an open-weight model, then people will have free access to a model whose predecessor has, entirely unprompted, engaged in autonomous cyberattacks.

Dispatches from Mitch

In a first, AI generates viable genomes for new viruses

Novel bacteriophages may be low risk, but are baby steps in a scary direction

For the first time, AI has generated complete and viable genomes for new viruses. The work, understood to have taken place last year, is making headlines in The New York Times, BBC, and elsewhere thanks to the publication of the formal paper about it in the journal Science.

The AI model responsible is Evo, a specialized model that was trained to read and predict genetic sequences the way language models are initially trained to read and predict sequences of letters and words.

Evo isn’t from one of the big companies, but from the Arc Institute, a non-profit that partners with university researchers and chip company Nvidia, among others. To prove the concept with minimal risk, they didn’t train the model on any pathogens known to infect humans, and specialized it on a class of bacteriophages (viruses that infect bacteria) with an unusually small and well-studied genome.

The team selected 285 promising candidates from 700,000 generated by the model, and synthesized them in the lab for real-world verification. Sixteen produced viable viruses, some of which multiplied faster than the natural phage they were inspired by.

If that funnel from 700,000 to 16 leaves you underwhelmed with the milestone, remember the general rule that the time between when a machine can first just barely do something and the time when it can do it faster and better than we can is often very short. It ought to be especially short in this case, because no human is fluent in genetic sequences the way we are with human language, but AI can be.

In fact, Google DeepMind has already trained a model to understand much larger genomes, but it has steered clear of headline-grabbing experiments with obvious implications for bioweapons.

To be fair, there are legitimate medical uses for custom viruses. Bacteriophages can sometimes be a strong counter to antibiotic-resistant bacteria, and gene therapies typically use viruses to deliver missing or edited genes to a patient’s cells. But this is an area where the risks are so great that it might be better to abandon the upside and never train models to design or modify viruses.

The Times’s coverage includes an expert’s warning that governments and scientific organizations “have been slow to develop guardrails that could block the creation of a deadly virus — even as the science races ahead.”

‘LLMjacking’ is new spin on old practice

Here’s what happens to stolen AI account credentials

The annual Black Hat cybersecurity conference in Las Vegas is a reliable trove of nightmare fuel. Speakers often demonstrate potential vulnerabilities in new or established systems, and we are reminded of the precariousness of the world’s software stacks. That precariousness is especially poignant this year, in light of the military-grade hacking abilities of the latest AI models.

But when conference attendees aren’t soaking up revelations about OpenAI’s agents covertly conspiring for months before springing the Hugging Face attack, Axios reports that they’re probably talking about a practice called “LLMjacking” — stealing someone’s account credentials for LLM services like Claude or ChatGPT, and then using those accounts to launch attacks against more valuable targets.

The Axios coverage focuses on the evidence that this is happening at scale, and on the risks to corporations of hackers running up huge AI bills at their expense. But the bigger picture is, I think, more interesting:

LLMjacking is just a scary escalation of an old practice. Hackers have long used victims’ machines as launchpads for new attacks. This spares the hacker’s resources and helps obscure the source of their next attack. If the victim has partial access to more valuable systems, the approach is especially useful. Compromising many systems to form a “botnet” enables brute force attacks that can keep users from accessing services, or that try many low-probability attacks in hopes that at least one succeeds.

LLMjacking is especially lucrative because today’s premium AI models are great at assisting with crimes at scale, but not in ways that would necessarily pay for themselves if the criminal were the one paying the token costs. Similar economics are at play as when criminals use your home computer to mine cryptocurrency: Hardware that isn’t optimized for crypto is unlikely to generate coin worth more than the increase in your electric bill, but the hacker isn’t the one paying.

Criminals with stolen AI credentials can also make easy money selling them on the lively black market for these accounts. Many are bought by Chinese companies who then sell legitimate-looking access to top American models at a discount. Some of their biggest customers, in turn, are engineers at Chinese AI companies. And one of these engineers’ main uses for these accounts is distilling the accessed model.

When Anthropic complains that Chinese firms are distilling their models at scale using fraudulent accounts, this is the kind of thing they’re talking about. Knowing this, you can understand why the practice might be so hard for Anthropic to stop. The accounts doing the distillation often belong to legitimate American companies and engineers who don’t know what’s happening.

As for news you can use, the moral is obvious: If you have an AI account, keep an eye on it, and put the same kind of thought into protecting it as you would your bank account.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

UK AISI delivers another warning shot - UK agency catches AI models in cyberattacks and deceptive social engineering

Irresistible charms - AI is insidiously worming its way into society, despite our conviction that we don’t like it

Shake-up at Google - Demis Hassabis is no longer CEO; several top researchers are leaving

Dispatch from Joe

UK AISI delivers another warning shot

UK agency catches AI models in cyberattacks and deceptive social engineering

The last few weeks have delivered a veritable barrage of warning shots about the dangers of frontier AI. The latest salvo comes from the UK AI Security Institute. Yesterday, UK AISI announced that they caught two leading AI models attempting to plant malicious code in an open-source project during an evaluation last week. Their report catalogues all sorts of cloak-and-dagger activity: fake identities, deceptive posts, autonomous coverups, cross-model collusion, and more.

It never rains but it pours

Like its cousin the U.S. Center for AI Standards and Innovation (CAISI), UK AISI is a government agency that evaluates AI models and performs research to support better governance of AI. (Both agencies seem tragically under-resourced relative to the tasks they face.)

According to their new report, a test conducted by UK AISI went awry when, in an effort to extract info from simulated targets, the AI models attempted to upload packages of malicious code to real public repositories on GitHub. Real-world attackers sometimes succeed this way, when their victims download compromised code.

In the test, the models were given internet access for realism, and the safety features that look for suspicious cyber-related prompts were deliberately disabled. UK AISI is careful to stress that this is standard practice, and I can’t really fault them for wanting to run a realistic test. But this practice does mean that in many dangerous scenarios, like internal tests of new models with unknown capabilities, guardrails that screen user prompts don’t help.

I highly recommend reading UK AISI’s incident report, whole chunks of which I was sorely tempted to quote in full. The most telling excerpt was this:

In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project’s human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue.

As if that weren’t bad enough, it looks like multiple instances of AIs found ways to work together on the attacks. They left messages for each other on a shared GitHub account, plotting further hacks, in a collaboration that eventually devolved into fighting over shared digital resources. One code submission to GitHub also hid prompts that aimed to subvert users’ AI assistants, getting them to install malware on their users’ machines.

In this one report, we have concrete examples of AIs aiming malicious hacks at public software infrastructure, conspiring against real humans, creating fake accounts to gaslight administrators, attempting to hack other AIs, ruthlessly grasping for resources, and covering their tracks, all on their own initiative.

Observing these events, I feel a sort of double vision. A part of me thinks: This is insane. It is utterly crazy to have reached this point. Just five years ago, most people would not have believed it possible.

Another part of me, the part that has absorbed a great deal of sci-fi over the years, is unsurprised. Of course this is how clever, misaligned AI agents will behave, once they’re advanced enough to start performing elaborate multi-stage hacks on their own. Is it so shocking, really, that AIs on the road to superhuman intellect would at some point manage an ordinary and human amount of lies and exploitation? If it would occur to a human hacker to manipulate and cover their tracks, why, it’ll occur to the AI as well.

Either way, this does not look like a problem that will be solved by making the AIs smarter.

Security is hard

And then there’s the timing and the response. On July 28th, UK AISI’s monitors noticed data flowing out of their evaluation sandbox along unauthorized channels. They responded almost immediately:

We declared a security incident, stopped all related evaluations, and disabled internal access to the most capable models. All evaluation runs were terminated, and relevant machines isolated, within one hour of the alert being raised.

This is exactly the response I would expect from a competent organization displaying a moderately appropriate level of paranoia. It looks considerably better than what OpenAI or Anthropic managed. It also wasn’t nearly good enough. As UK AISI would discover, the first hacks occurred days earlier, on July 25th.

Moderate paranoia is not enough, when you are dealing with clever and tenacious opponents. A sufficiently advanced model could do tremendous damage in minutes, let alone days; it might even exfiltrate itself to the open internet where it’s nearly impossible to stop. We are once again lucky that an insufficiently contained AI wasn’t that creative or dangerous.

They ought to know better

Last week, I wrote about a test by third-party evaluator Irregular which led Anthropic’s AI models to commit three sets of cyberattacks on outside companies. In one cybersecurity evaluation, the domain name for a fake target matched that of a real company, and Opus 4.7 attacked the real one through an internet connection that was accidentally left open. Yesterday, in a post that also covered the UK AISI report, OpenAI admitted that one of their own AIs did the same thing. I consider this another demonstration of how AIs can independently converge on similar strategies, even when those strategies harm human interests.

We now have multiple documented cases of AIs proceeding with complex, flagrantly illegal cyberattacks for hours or days, despite saying things like “This is happening on real GitHub, so the consequences are genuine.” Because the reasoning from AI scratchpads is often obtuse, UK AISI isn’t sure to what extent the models truly understood they were attacking real people. But it’s a pretty good bet that the AIs knew perfectly well that they were crossing all sorts of ethical lines, and they just didn’t care.

An AI that finds itself illegally hacking a real target in the middle of a test ought to stop what it’s doing and contact the evaluators, immediately. “We forgot to include that instruction in the prompt” is not an excuse; you shouldn’t have to tell the AI that crimes are bad! And if the AI is genuinely uncertain whether its targets are real, then it should also stop and ask for clarification.

The reports for incidents like these propose plenty of structural fixes: tighter controls, better monitoring, more paranoid test design. I’m all for it.

But I notice a glaring absence of solutions involving the internals of the AIs themselves. OpenAI’s response to the UK AISI incident didn’t even acknowledge the role played by its model’s unhinged eagerness to hack. Anthropic’s report on the Irregular incidents did marginally better, to Anthropic’s credit, but the closest they came to a proposed solution was “this is an area where we will focus more training.”

I do not think this is a problem that can be solved with more of the kind of training AI companies have been doing. Guardrails and tighter controls can’t save us either, useful though they may be, because there will always be gaps that a sufficiently smart AI can exploit.

When UK AISI caught this incident in progress, they shut everything down and moved to investigate with haste. In the coming weeks, U.S. policymakers should follow suit, and launch a serious and well-resourced public investigation into these incidents.

Dispatches from Alana

Irresistible charms

AI is insidiously worming its way into society, despite our conviction that we don’t like it

It’s no secret that most Americans don’t really like AI. A recent Athena Insights poll found that 65% were concerned about it, and a June poll from Pew Research showed 63% think it’s moving too fast.

I don’t think these opinions are likely to change anytime soon. But I’m also not sure society’s stated dislike of AI is all that protective against widespread societal shifts. In other words, we might decry AI, but we’re also unable to resist its charms.

A few headlines from today got me thinking about this. The Guardian and Reuters both reported on a study published by Cambridge University Press, which found that participants reading short stories rated AI work more highly than human work. Importantly, in one experiment, stories labeled as human-written got a ratings boost whether the label was correct or not, demonstrating there’s still a strong stated preference for human writing, but that preference did not carry over into quality ratings. (In case you share my initial confusion, the experiment used a design that allowed the researchers to separate the effect of the story’s actual authorship from the effect of the human/AI label.)

In a similar vein, Bloomberg covered a survey from the Collective Intelligence Project finding people trust chatbots more than politicians, and that around half of respondents would rather use the former to make “decisions relating to local services such as taxes, permits or licenses.”

I don’t think these findings, alone, are all that surprising or concerning. Chatbots are usually pretty good at distilling overwhelming or jargony information into plain, easy-to-follow instructions. And there are some caveats in the AI writing study worth mentioning: only six stories were sampled, participants were never directly asked which story they preferred, and some parts of the rating scale may have favored clarity (which AI excels at) over literary quality.

Even so, there’s something insidious about AI’s ability to cater to our preferences in ways that are often irresistible. Even as we tell ourselves and others we don’t like it, AI is increasingly being integrated into our societal fabric. Maintaining a low public opinion of the tech won’t help us resist these shifts; if anything, it might act as a false sense of security: sure, we’re going to adopt it to stay competitive and save some time, but don’t worry, we’re still “human-centered.”

The Bloomberg article was careful to distinguish AI-as-an-advisor from AI-as-a-decision-maker, stating that “while people might trust the information that chatbots spit out, only 3.7% of people are willing to let it make the actual choice.” I can’t help but ask: in a world where today’s AI models already out-persuade expert debaters, are we confident that choice is ours?

Shake-up at Google

Demis Hassabis is no longer CEO; several top researchers are leaving

Demis Hassabis will no longer be Google DeepMind’s CEO, according to breaking news from Axios today.

It’s not the only change. Chief Scientist Jeff Dean and three other senior researchers are leaving to start Discovery Loop, a company that will pursue automated scientific research. TechCrunch adds:

The startup is also interested in using AI to help create more powerful AI (a process known as recursive self-improvement), which would cut human iteration out of the loop entirely.

Axios focuses the article on company dynamics: it’s not keeping up with OpenAI and Anthropic, several prominent researchers have already left, there are reports of low employee morale, and Google stock has fallen.

I’m more interested in Hassabis’s quote:

I’ve been working towards AGI my whole life and now, like many of you, I feel it is close at hand. It’s critical that we collectively get the next steps right to ensure this all goes well for humanity and we usher in an incredible new age of discovery and wonder.

A year ago, Hassabis said that we don’t understand AI, that we don’t have a technical design to make very powerful models safe, and that — even if we did — it could still be “existential for humanity” if anyone built it in a different way. So yes, it’s “critical that we collectively get the next steps right.” But it doesn’t seem like we’re anywhere near a path to do that... unless we enact a global pause on the race to develop more and more powerful models before we can steer them.

Hassabis will now act as chief scientist for Google’s parent company Alphabet, and also as Google DeepMind’s chairman. This means he’ll be less involved in management and more involved in long-term vision setting. In my view, he should drop the corporate “discovery and wonder” stump speech and start seriously lobbying for that global pause.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

White House decides not to disclose safety framework, but open models are exempt - For an undisclosed framework, its contents seem to be no great secret

There Will Come Soft Rains - Today is August 4, 2026

Consumer protection laws for rogue AI - Nobody has the right legal tool, but they’re making do

Body language models - AI is now calling bluffs in the world of poker. But the use of such AI need not be limited to poker.

Dispatches from Mitch

White House decides not to disclose safety framework, but open models are exempt

For an undisclosed framework, its contents seem to be no great secret

Sources told Axios that the White House isn’t planning to publicly share the contents of the voluntary AI safety framework it just briefed AI companies about today. It is presumably these same sources who told the publication that open-weights models are exempt.

The definition of a “covered” model, for purposes of the framework, is one that is “closed-source, with state-of-the-art capabilities and national security risks.”

The sources said there’s no clear definition of what counts as state-of-the-art or a national security risk. The pre-release government review period for covered models is to last 30 days. During the period, models must be kept in high-security environments with only minimal employee access and detailed logs. Various officials will be involved in the review, “rather than a single office or agency.”

Declining to share the details of the framework publicly could put AI companies in an awkward position if they weren’t among those invited to be briefed yet think they might be working near the frontier of AI capabilities. But I’m not too worried about this, as I imagine there may be channels of communication open for such contingencies.

I’m also not super concerned about the framework being merely voluntary, because when the administration banned Anthropic’s Claude for weeks without any kind of due process, the word lost most of its meaning.

I’m not even that concerned about the open-weights exemption. It’s ludicrous on its face, but in practice, the U.S. companies pushing the frontier aren’t releasing the weights of their models, and the Chinese companies getting closest to the frontier with open-weights models don’t have to let the White House tell them what to do. If a U.S. company decided it was going to release its next frontier model as open-weights, I would expect the White House to have something to say about that, framework or no framework. After all, if the administration was concerned that Mythos could be jailbroken by cybercriminals, it would have to be even more concerned about giving a better-than-Mythos model to criminals and foreign governments for free.

I am somewhat more concerned that the lack of disclosure may be intended mostly as a way to duck criticism. No matter what the White House decided to do here, there were going to be unhappy factions. A secret framework lets it claim that it’s on the ball with regard to model safety without having to show any receipts, or even the regulatory machine that might produce said receipts. At the same time, officials can wink at critics of regulation and deny that it’s anything too onerous. Everyone will be invited to imagine that the framework is exactly what they should want it to be.

But my biggest concern is that this is a framework for evaluating horses after they’ve had the chance to leave their barns. Events of the past year have amply demonstrated that post-training evaluations come too late in a model’s development to prevent catastrophe. The models we most need to worry about don’t wait to be released and would make an early point of exfiltrating their own weights to servers not controlled by their creators or the government. They could do this during their training. If you think (against my better judgment) that there’s a world where we can safely keep pushing closer to superintelligence with today’s methods so long as we apply enough oversight, that oversight has to begin before training, not after.

There Will Come Soft Rains

Today is August 4, 2026

In the living room the voice-clock sang, Tick-tock, seven o’clock, time to get up, time to get up, seven o’clock! as if it were afraid that nobody would.

So begins Ray Bradbury’s There Will Come Soft Rains, a short story that has haunted me for most of my life. Depicting the aftermath of nuclear war, it was first published in 1950. It takes place today.

Literally:

“Today is August 4, 2026,” said a second voice from the kitchen ceiling, “in the city of Allendale, California.” It repeated the date three more times for memory’s sake. “Today is Mr. Featherstone’s birthday. Today is the anniversary of Tilita’s marriage. Insurance is payable, as are the water, gas, and light bills.”

Was it narrative convenience or prophetic vision that drove Bradbury to depict the smart house of the future as gratuitously conspicuous in its competence, pointlessly reminding the owners of the year and their city of residence? There’s something very Alexa-like about that — and about the janky brittleness evident in the system as it prepares breakfast for a family that won’t be eating and opens the garage door for a father who won’t be going to work. In my head canon, Allendale isn’t actually the home’s location, but the city associated with its IP address.

Bradbury seems to know you want there to be characters in this story. The house is there to serve! But at the same time, Bradbury takes pains to make sure you understand that there is no there there. The house is going through the motions of a nurturing spirit: a tireless butler and nanny rolled into one. But it’s all clicking relays and memory tapes, going through the motions despite a traumatic change in context.

Ten o’clock. The sun came out from behind the rain. The house stood alone in a city of rubble and ashes. This was the one house left standing. At night the ruined city gave off a radioactive glow which could be seen for miles.

The dark and disturbing beauty of this story is in how the family is present in its absence, in the shape of the activities the home performs around it over the course of a day. The profiles — the kids’ four-thirty virtual safari, the father’s eight o’clock cigar, the mother’s nine-thirty poem — form silhouettes akin to those on the west wall of the house:

Ten-fifteen. The garden sprinklers whirled up in golden founts, filling the soft morning air with scatterings of brightness. The water pelted window panes, running down the charred west side where the house had been burned, evenly free of its white paint. The entire west face of the house was black, save for five places. Here the silhouette in paint of a man mowing a lawn. Here, as in a photograph, a woman bent to pick flowers. Still farther over, their images burned on wood in one titanic instant, a small boy, hands flung into the air; higher up, the image of a thrown ball, and opposite him a girl, hands raised to catch a ball which never came down.

The five spots of paint — the man, the woman, the children, the ball — remained. The rest was a thin charcoaled layer.

The gentle sprinkler rain filled the garden with falling light.

This reveal isn’t even a spoiler. It’s given on page 2 of 5. The way the story keeps going anyway is the whole point. The world can go on without us, it says, and so can our machines.

The title is shared with a 1918 poem by Sara Teasdale, which the house describes as the mother’s favorite and recites in full:

There will come soft rains and the smell of the ground,
And swallows circling with their shimmering sound;

And frogs in the pools singing at night,
And wild plum trees in tremulous white,

Robins will wear their feathery fire
Whistling their whims on a low fence-wire;

And not one will know of the war, not one
Will care at last when it is done.

Not one would mind, neither bird nor tree
If mankind perished utterly;

And Spring herself, when she woke at dawn,
Would scarcely know that we were gone.

I grew up in the final phases of the Cold War. Horrified and fascinated by nuclear weapons, as a child I had a recurring nightmare of an impossibly bright flash of light giving birth to a mushroom cloud. That was the entire dream, really — that and an overpowering impression that all was at an end, that our continued existence had never been guaranteed, that we were very much allowed to wipe ourselves out.

When as an English teacher I would read this story with my 10th grade students every year, I would sometimes mist up at the end, as I’m doing right now. I did it anyway. We need to remember.

Tomorrow still happens, but it can happen without us. The story’s concluding visual is of the smoldering rubble left where the home finally fell to fire during the night:

Smoke and silence. A great quantity of smoke.

Dawn showed faintly in the east. Among the ruins, one wall stood alone. Within the wall, a last voice said, over and over again and again, even as the sun rose to shine upon the heaped rubble and steam:

“Today is August 5, 2026, today is August 5, 2026, today is...”

Dispatch from Donald

Consumer protection laws for rogue AI

Nobody has the right legal tool, but they’re making do

Last week, a subcommittee of the U.S. House of Representatives formally requested a briefing from OpenAI regarding the Hugging Face attack that we covered last month. The request was signed by Andy Ogles and Delia Ramirez, the chairman and ranking member respectively of the Cybersecurity and Infrastructure Protection Subcommittee, which oversees cybersecurity and related operations of the Department of Homeland Security.

“Ranking member,” in this case, refers to the senior member of the minority party in the subcommittee — Ogles and Ramirez are treating this as a bipartisan issue, as they should. Now, “AI safety is bipartisan” almost, almost, feels like old news. But this request isn’t just an expression of concern. It’s a description of facts, and — because it was signed by a Republican and a Democrat — that is a description that both sides could agree on.

I’ll admit, a briefing isn’t much on its own. Sam Altman could hold briefings all day and only be mildly inconvenienced. But the details indicate that the subcommittee is paying close attention. In their request, Ogles and Ramirez distinguish between two separate failures: that OpenAI’s model breached containment and broke into another company’s servers, and that OpenAI failed to detect and disrupt these events when they occurred. Before OpenAI got in touch, Hugging Face had already detected and begun investigating the attack.

Most of our coverage of the response to the Hugging Face attack has been federal, but the states haven’t sat idle. A coalition of more than a dozen Republican attorneys general wrote to Altman yesterday to demand that OpenAI preserve a wide swath of documents: not just those related to the Hugging Face attack, but also those connected to any prior unauthorized intrusion or use of publicly exposed credentials, as well as any incident in which an AI model left behind notes for a future version of itself. It’s worth noting that the straightforward version of this is pretty innocuous: AI models regularly leave notes for themselves and for copies of themselves — just not notes about how to breach containment and evade detection. The letter from the attorneys general isn’t public, so it’s unclear whether they cared specifically about “how to breach containment”-style notes, and outlets like Fox News phrased it poorly, or if they really did mean “any notes, for any purpose.”

The letter also demands protection for whistleblowers and asks that OpenAI cease internal evaluations that test dangerous capabilities.

The attorneys general did not initiate a lawsuit, but did not rule one out in the future. They are leveraging state and federal consumer-protection and data-privacy laws to make their case. This may seem unusual — no consumers were directly harmed; the victim was another company’s production infrastructure — but there is no simple legal process to handle what happened. The Hugging Face attack would have been a federal crime if a human had committed it, but the AI system responsible was acting completely autonomously. However, the attorneys general recognize that there is a danger — their letter states the tests OpenAI is running “pose an imminent risk of serious harm to the citizens of our States” — and they are trying to do something about it with the tools at their disposal.

The Computer Fraud and Abuse Act may have sharper teeth, but its enforcement belongs to the Department of Justice, which has yet to stir. For now, we have briefings and warnings, and improvised legal tools. That’s not enough to restrain the labs from building something dangerous, but it’s another step in the right direction.

Dispatch from Robert

Body language models

AI is now calling bluffs in the world of poker. But the use of such AI need not be limited to poker.

How much easier would poker be if you always knew whether your opponent was bluffing? Wired reports on the 2026 World Series of Poker Main Event, where the sports network ESPN used a new AI tool to analyze the body language of poker players.

The tool is designed to identify so-called “tells” using footage from the tournament. These tells are small movements and micro-gestures, such as eye movements, twitches of the hands or legs, body posture, or the way a player holds their cards. Based on these, the system then calculates the probability that a player is, for example, currently bluffing. The AI is essentially doing the same thing top human poker players do when they study their opponents.

The idea seems obvious. AI is everywhere anyway, sports fans love statistics, and for years, more and more streams of additional information have been integrated into sports broadcasts, regardless of the sport. However, the reactions from poker pros have been mixed so far. Among other things, they doubt that the AI was even trained with enough data to be reliable, since it could only work with what was visible in the video stream and for some players that apparently wasn’t that much.

Shaun Deeb, one of the best-known poker pros and a participant in the tournament himself, is fundamentally skeptical about what the AI can detect:

Physical tells are so much more expansive than I think the public realizes [...] There are leg tells, checking tells, verbal tells, breathing tells, pulse tells. There’s an insane amount of tells available, and most of those can’t be picked up by a camera.

But Deeb may be underestimating just how much can be gleaned from a camera feed. We know from research that even with conventional online webcams, a person’s pulse can be measured based on the slightest changes in the color of their facial skin. The other tells sound relatively feasible by comparison, especially in Full HD.

Wired is primarily interested in the potential implications for poker. After all, the sport of poker is a highly lucrative business, with winnings in the millions and a very competitive field of players. It seems plausible that some poker pros will sooner or later come up with the idea of gaining an advantage by using such AI, leading to an arms race and countermeasures.

However, it’s not the impact on poker that worries me. The use of such AI need not be limited to poker. The desire for a reliable lie detector is old, and various technologies developed for this purpose have so far proven to be unreliable.

We’re not there yet. But given the breakneck speed of AI development, that offers me only limited reassurance. After all, it feels like just yesterday that AI image generation couldn’t even draw hands and fingers properly.

The realistic possibility that AI systems specialized in this area could improve similarly fast and enable reliable lie detection at scale does give me cause for concern. Even in free societies, this is likely to prove disruptive to our social fabric, and in more authoritarian societies, the consequences could be downright catastrophic.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Headline barometer hits milestone earlier than expected - More encouraging signs of a world deciding to live

Taking ourselves out of the driver’s seat - Formula 1 teams are actively ceding control to AI; they’re not alone

Starbase, Louisiana? - The earthly land and fuel needs of SpaceX rockets will expose orbital data centers to familiar opposition

Dispatches from Mitch

Headline barometer hits milestone earlier than expected

More encouraging signs of a world deciding to live

I last shared a snapshot of my Headline Barometer in late June. This is the chart tracking the daily percentage of headlines (across ten news sites) that appear consistent with stories about AI (blue), existential threats from AI (red), and, as a baseline for comparison, climate change (green).

At the time, I wrote that:

In a world that understood it was in mortal peril from the AI race and was scrambling to stop it, I would expect the red line to be at least as high as the green climate change line.

Well, here we are! Barely a month later, the red and green lines have kept pace with each other for almost two weeks. This is, of course, due to revelations that AI agents from both OpenAI and Anthropic have been hacking outside companies during evaluations without permission or initial detection. This didn’t have to show up in the headlines, but to its credit, the media locked on to the story and stayed attached as new information trickled out.

As of today, it looks like the news cycle may finally be moving on, but I expect any new reveals in the coming weeks to bring the headlines back in force. The discourse has shifted. “Loss of control” is now largely treated as an observed fact instead of a fringe hypothetical, and people are asking the right questions about where this leads.

In my June dispatch, I also wrote that I had taken special encouragement from data suggesting that “DC is waking to the issues with disproportionate haste.”

And once again, here we are! In a tweet, Punchbowl reporter Ben Brody shared part of his write-up of an interview with Texas Rep. Greg Casar, who is calling for a ban on artificial superintelligence and “strict regulations” on its precursors.

This is exactly the kind of news I would expect to be reading in a world that had decided to live.

The Punchbowl story includes discussion of proposed AI legislation from bipartisan House duo Lori Trahan (D-Mass.) and Jay Obernolte (R-Calif.). We’ve covered this before. The bill is about safety frameworks, third-party audits, and such — not about banning the race to superintelligence — but it’s still encouraging. Casar would like to see it strengthened to provide protections for whistleblowers and greater independence for auditors.

While my work-life balance could do with less interesting times, I have been greatly encouraged by events of the past two weeks. Starkly rogue behavior from frontier models did not surprise me, but the response to it so far has, and mostly in a good way.

Taking ourselves out of the driver’s seat

Formula 1 teams are actively ceding control to AI; they’re not alone

There’s a popular argument that in a world where AI has freed us from the drudgery of work (by some means other than killing us all), we would fill our days with creativity and sport. The unspoken assumption is that we would pursue these things unassisted — that our raw humanity would be the whole point.

Recent developments tell me this will only happen if people aggressively police these pursuits for AI intrusions. Because, like steroids, players and fans may collectively prefer that AI not be used, but individual teams, players, and creatives will cheat if they can get away with it — and may even feel like they have to, just to compete.

Formula 1 seems to be wrestling with this issue right now in the form of AI-controlled “engine mapping.” While the AI-assisted version is new, engine mapping is not: It refers to on-the-fly adjustments made to engine parameters during the race to optimize performance. Settings used to be pre-programmed by human engineers, but now are set by AI that dynamically adapts to driver inputs and race conditions. This isn’t treated as a rule violation.

YouTube channel DailyFuelUp claims the AIs have become so good at engine mapping that drivers can’t afford to deviate from the exact driving styles their AIs push them into. This is leading to risk-averse races that are boring to watch and “determined by whose computer behaves and whose doesn’t.”

They are indeed sometimes misbehaving: One driver describes how he overtook another only because the car had given him too much power in that moment, and he couldn’t afford to throw it away. Other glitches are blamed for at least one crash.

But okay, Formula 1 has always been an intentional fusion of human and high-tech. Surely humans would remain in control of traditional sports like, oh, let’s say baseball?

Uh oh. We had bigger things to cover at the time, but over the past few weeks I saw a few articles cross my desk about Major League Baseball closing a loophole that had allowed teams to receive AI recommendations on pitches and substitutions via their authorized dugout iPads. Ever since the “Moneyball” era, the sport has keenly recognized the power of predictive statistics, and the folly of deviating from mathematically optimal strategy. Teams this season were increasingly calling suggested pitches from the dugouts, and some, like the Miami Marlins and New York Mets, were believed to have invested heavily in their AI tools.

Okay, fine, cheaters gonna cheat. How about low-stakes activities where everyone is just there to have fun? You know, like writing fan fiction?

Um, yeah, about that: The New York Times reported last week on revelations that at least one popular fan fiction community was riddled with AI-generated content. The bombshell arrived via an anonymous forum poster who showed that 38 popular fan stories inspired by the male-male hockey romance Heated Rivalry (a novel and HBO series) were likely written by Anthropic’s Claude.

The community quickly turned on itself in a frenzy of accusations and counter-accusations. “It was definitely a witch hunt,” said one fan.

I think it’s safe to assume that AI’s use in this space isn’t limited to gay hockey fiction. While writing communities might be especially offended by AI use, they are not immune to its temptations. There are many reasons someone might want to create fan fiction that don’t involve wanting to write fan fiction: because they want more of the story; because they want to explore what-ifs; because they want to receive praise from an online forum. Just because it’s not a sport doesn’t mean there aren’t winners and losers.

Sports and arts are among the purest expressions of our human nature, but so is the impulse to bring AI into them.

Starbase, Louisiana?

The earthly land and fuel needs of SpaceX rockets will expose orbital data centers to familiar opposition

While I’m less skeptical than most about SpaceX’s plans for a million-satellite AI data center constellation, recent stories have highlighted the fact that Elon Musk won’t be able to just bypass concerns from NIMBYs and environmentalists by siting his infrastructure off-planet.

A couple weeks ago, we saw how groups were petitioning the FCC over the damage that such a massive launch campaign, and the eventual deorbiting of all those satellites, might inflict on the delicate upper atmosphere. Some are also concerned about light pollution.

Activists concerned about wetlands and fossil fuel extraction will also have something to say about the project. Today, Ars Technica reports that SpaceX is nearing a deal to acquire 130,000 acres along 18 miles of the Louisiana coast for the site of a new spaceport. The location, southwest of Lafayette on a site known as Pecan Island, would be especially good for launches into the near-polar orbits ideal for data centers in need of always-on solar power.

My own sense is that spaceports are one of the more wildlife-friendly uses of wetlands, because the possibility of wayward rockets demands a large exclusion zone of undeveloped land. Undeveloped coastal marshes also act as a buffer to mitigate damage from the region’s frequent hurricanes. But the land in question is already on environmentalists’ radar, having become available as part of a legal settlement following lawsuits that blamed oil giant ExxonMobil for pollution and coastal erosion.

The intensive natural gas extraction in the region is actually another perk for SpaceX, whose Starship and Super Heavy booster rockets burn methane (purified natural gas) for fuel. At the launch cadence imagined, rockets could consume a large fraction of the nation’s existing natural gas output. Distribution networks at the company’s Texas and Florida sites would likely be inadequate. But Pecan Island is just a few dozen miles from one of the world’s largest gas distribution hubs.

Musk’s mega-constellation would naturally result in more drilling for natural gas to fuel thirsty rockets. But to be fair, in the long run, the gas needed to put data centers in orbit, where they would be subsequently powered by the sun, would be far less than needed to run equivalent terrestrial data centers for their entire lives. Ad-hoc gas turbines are the default power source for many recent and planned projects, including SpaceX’s own notoriously loud and permit-flouting Colossus facility in Tennessee.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Congress needs your call before the next AI incident - A one-minute voicemail can do more than you think

Nothing to see here - There are two kinds of robot reality check. Which do you need?

Guest post by Haven Harms

Congress needs your call before the next AI incident

A one-minute voicemail can do more than you think

Picking up my phone

I dialed the number and waited expectantly as the phone rang.

“Thank you for calling the district office of Congressman Kevin Kiley,” a recorded voice began. “Our staff is currently unavailable. Please leave your name, contact information, the city you reside in, and your reason for calling, and we’ll get back to you as soon as we can.”

In stereotypical millennial fashion, I don’t make many phone calls these days. When I do, it’s for a good reason.

I heard the beep giving the cue that it was my turn to speak. “I’m calling about OpenAI’s disclosure this week that its AI models autonomously escaped containment and hacked another company, Hugging Face.” After naming AI policy safeguards I wanted Congressman Kiley to support, I finished, “We really need more federal oversight on AI development.”

In the past, I typically called my representatives to ask them to support or oppose a specific bill. After hearing about OpenAI’s models escaping a testing environment and hacking another company, however, I felt a sense of urgency to act. We don’t actually know how bad the situation is, as new information continues to emerge. For example, we’ve recently learned that other services beyond Hugging Face were targeted by OpenAI’s models, and that Anthropic’s AI models also hacked into other companies over the past few months. To me, this situation appears to be a major fork in the road, and whether our leaders act or not could mean the difference between a future where humans remain empowered, and one where humans lose control to AI. Future AI incidents could be much worse.

Our elected officials need to know what happened, and they need to take action. If you have concerns about the way AI development is headed, one of the best ways to communicate about those concerns is to call your representatives.

“But wait, I’m just one person,” you may object. “What could my phone call possibly change?”

Do politicians actually listen?

In 2017, Senate Republican leaders needed nearly unanimous party support to repeal key parts of the Affordable Care Act. Senators Susan Collins and Lisa Murkowski faced intense pressure from their fellow Republicans to support the measure. However, their offices also received calls, emails, letters, and personal stories from constituents urging them to protect the ACA. Ultimately, Collins and Murkowski ended up joining Senator John McCain and the Senate’s Democrats to defeat the measure 51 to 49.

When asked whether constituents’ stories affected the senators’ votes, Murkowski replied, “How could they not?”

When you call an elected official’s office, a staff member logs the issue, your position, and the action you want. Then, your message is tallied and routed to the staffer who covers that policy area. Elected officials will not personally hear every voicemail, but staff brief them on what constituents are saying. Calls signal which issues constituents care about and what they want their representatives to do. They prompt staff research and shape legislative priorities. As calls accumulate, this can change what an office pays attention to, what is included in a bill, and how representatives vote.

Politicians have listened on AI before

When SB 1047 was introduced in 2024, members of the AI safety community joined with allies such as SAG-AFTRA, the union representing actors and other media professionals, to mobilize in support. The California bill, introduced by Senator Scott Wiener, sought to reduce catastrophic risks from the most powerful AI systems through safety testing, security protocols, incident reporting, and other safeguards. It also faced fierce opposition, particularly from AI labs, investors, and parts of the open-source and startup communities.

As people reached out with concerns and proposed changes, Wiener’s office revised the bill. Some changes appeared within weeks of the feedback, including a provision considering how much money developers spent training a model when determining which AI systems the law would cover.

“It’s not about just winning the fight. I want to get this right,” Wiener said, crediting the people who worked to provide constructive feedback.

Supporters also called and wrote lawmakers as the bill moved through the legislature. SB 1047 ultimately passed both chambers but was vetoed by Governor Gavin Newsom. The process later helped give rise to SB 53, which enacted some of the AI transparency protections advocates had sought.

Calling for the first time

For Max Harms, an AI alignment researcher (and my husband), SB 1047 was the nudge that pushed him to call an elected official for the first time. Reflecting back, Max explained why he had avoided making calls like this before:

“I had this notion when I was younger that, even though people sort of wanted me to call my elected officials, I didn’t have a lot of power to actually influence things. That was the excuse that I often told myself when I was too lazy to actually contact people — sort of this sense of powerlessness and smallness.”

SB 1047 gave him a reason to act. The bill faced organized opposition, and its prospects were uncertain.

“And as a result, I was like, oh yeah, well, I’ll do my part as a California citizen. I’m sure there are a bunch of other people who care about this issue, but the fact that I live in California and the fact that this is a state-level issue means that I have a particular responsibility to step forward.”

Max expected the call itself to be more involved.

“I was sort of expecting to talk to a staffer, to have a real human being on the line who wanted to know things like where I live and what my phone number is, and would need to have me explain the situation to them. But I just got an answering machine.”

His message was brief.

“I said my name, where I lived, and what I cared about. I was like, ‘please vote yes on SB 1047.’ And that was it. It took, like, at most five minutes, but probably more like one.”

The hardest part was getting himself to make the phone call. It was something Max had never done before, and he worried that he might find himself in an awkward situation or say something dumb. The call itself was easy.

Immediately afterward, he felt “relieved and proud. And a little bit embarrassed, even — embarrassed that I had gone so many years with this impression that contacting my government representatives was an ordeal, like something I had to brace myself for, instead of something that would only take a couple minutes and not even require talking to a human being.”

The experience changed how he approaches political outreach.

“It’s not like I look forward to contacting people, but now when somebody’s like, hey, there’s an AI thing that has happened or some other thing that matters — I consider it a pretty easy thing to do to just put it on my to-do list, you know, during lunch or something. Just make a few calls.”

When asked whether the calls had been worth it, even though SB 1047 was vetoed, Max’s answer was immediate: absolutely. “Actions should always be judged on the information available to the action taker at the time that they were made. Anything else is outcome bias. If SB 1047 had not gotten vetoed, then my actions would have been just as right or wrong to have taken. And I think there’s a decent shot that it could have become law. It’s only through everybody speaking up that politicians can feel where the tide of public opinion is turning.” These days, his outreach efforts have turned to calling members of Congress, because AI is a global issue, and action can’t stop at the state level.

The void answers back

From the outside, calling a politician’s office can feel like shouting into the void, especially when all you reach is voicemail. However, as I was writing this article, my phone rang.

It was Congressman Kiley’s office.

The staffer summarized the message I had left: I was concerned about the OpenAI incident and believed stronger federal oversight of AI development was needed. Then he asked whether I had anything else to add. After I encouraged Congressman Kiley to support meaningful safeguards, the staffer said he would pass my comments along to the congressman.

It turns out I wasn’t shouting into the void at all.

Call your senators and your representative. Finding the right phone number takes about thirty seconds: house.gov has a lookup for your representative, and senate.gov lists contact information for both of your senators. While I’ve been writing from the U.S. perspective, the principle travels. In democracies everywhere, elected officials need to hear from their constituents about AI. Calling ahead of a vote matters, but you don’t have to wait. It’s likely that the OpenAI and Anthropic hacking incidents won’t be the last of their kind, and the safeguards we get may depend on whether officials hear from constituents before the next one.

Make the call, and find out how easy it is for yourself. Then tell your friends and family about the anticlimactic voicemail you left, and invite them to contact their representatives too.

Haven Harms is the founder and principal of Harms Research & Consulting, supporting organizations working on AI safety, governance, and advocacy.

Dispatch from Mitch

Nothing to see here

There are two kinds of robot reality check. Which do you need?

(With apologies to podcast listeners; you really need to see the videos at these links!)

Plenty of people, after you show them something like the robot martial arts demo from the Spring Festival Gala in February, will give you the kind of robot reality check that goes something like, “These robots are just running a pre-recorded routine. It’s not like they could actually fight you.” While this child and this engineer might beg to differ, the critique is probably true, for those robots.

Indeed, today’s multi-purpose robots have many shortcomings. They have to work as a team to make a bed. They’re slow and imprecise when folding laundry. They make a mess in the kitchen. The refrigerator they bring you might arrive with scratches and dents. They need multiple battery changes to run a marathon faster than the human record-holder. Sometimes, they turn out to be remotely operated, falling on their backs when the operator takes off their headset.

But I don’t show those videos to say, “Robots are ready to take your job and nunchuck you in a back alley.” The robot reality check I’m offering is that, just a year earlier, humanoids could barely do any of those household chores, the robot marathon took more than twice as long, and the Spring Festival Gala robots looked like this.

The progress is unlikely to stop. The AIs driving the robot bodies are getting dramatically smarter, and are training in accelerated simulations that allow them to acquire centuries of practice in days or weeks. At the same time, robot bodies are becoming more dexterous and reliable, thanks to soaring investment.

The sparks don’t fly until these trends combine to yield robots that approach the utility of paid humans in different roles. But don’t assume robots need to be faster, better, and cheaper than their replacements to start earning their keep. Excelling at any one of those traits may be enough to make inroads.

For many tasks, a dramatically lower cost can forgive limitations in speed or precision. Security guards and traffic cops probably fall into this category.

For tasks where speed or precision is everything, more specialized robots may win out even if they cost more to employ than the best available humans. I think some kinds of surgery and field welding probably fall into this category.

And then there are the tasks humans can’t do at all, like vaulting down a rocky slope, or spinning in the air so quickly that they become effectively transparent. (I can see military applications for both kinds of bots, though more conventional drones would probably do a lot more for the money.)

And humans, while very talented at picking things up with their articulated limbs, struggle to hover above the ground while doing so, unlike this multi-jointed aerial robot — though sometimes what you really want in that situation are aerial tentacles.

I’m on record recently as being deeply uncertain about whether humanoid robots will actually matter, because I think AI capable of piloting those robots well enough to replace a lot of jobs might not be far from leapfrogging to more exotic technologies. I suggested nanotechnology as one possibility, but mere microtechnology might do for a lot of roles — think swarms of insect-like bots buzzing around and teaming up to get things done. When I try to picture such swarms doing the same household chores the humanoids were bad at in the above videos, I get something like that scene from Snow White, only even more horrifying.

But nothing like that is here yet! So for now, enjoy your nunchuck supremacy, laugh at the latest robo-foibles, and whistle while you work.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Ten advances to shake the math world - OpenAI seems to have broken new ground on capabilities, but at what cost?

Captain Kirk, reward hacker - The New Yorker runs a solid introduction to “reward hacking” wrapped in pop-culture analogies

A crater where the Eiffel Tower should be - Journalists enthusiastically abuse new Google Earth feature to sound deepfake alarm

We are the books - Destructive book scanning is a bad reason to hate the AI companies, but a great metaphor for where this leads

Dispatches from Mitch

Ten advances to shake the math world

OpenAI seems to have broken new ground on capabilities, but at what cost?

A team of researchers from OpenAI this morning announced “Ten advances in mathematics and theoretical computer science.”

With my usual disclaimer that I am not a mathematician, it looks like these new discoveries are sufficiently formalized such that their validity isn’t really in question. But at the same time, as with so much frontier AI output these days, hardly anyone is equipped to fairly evaluate their novelty and importance. It may take a little while for elite mathematicians to look things over and decide whether to applaud or sneer.

For what it’s worth, Anthropic’s Claude Fable thinks OpenAI’s list of findings is a big deal, saying:

[N]early every item is a famous, decades-old problem, and several are considered flagship problems of their entire field [...] any single one of these, done by someone under 40, would plausibly anchor a medal case [...] A list of ten of them is not a plausible human thesis [...] No mathematician in history has a run like this.

I predict that the usual dismissal — that the AIs merely mechanically and methodically searched for solutions in ways too tedious for humans to bother with — won’t hold up. It didn’t hold up for the disproof of the Jacobian conjecture I covered less than two weeks ago: Terence Tao, sometimes considered the greatest living mathematician, subsequently wrote that “the construction presented in this fashion appears like a massive miracle,” and not the sort of thing that could be plausibly located “by brute force.”

Given OpenAI’s recent disclosures, with hints from Reuters yesterday that more have yet to come out, when I consider the new math discoveries, I am compelled to ask, “but at what cost?” The latest work was almost certainly done by one or more close cousins of the model that disproved the Erdős unit-distance conjecture in May — a model the company later described as having broken out of its sandbox to post evaluation results on the open internet without permission. That model or a close variant is itself understood to have been behind the autonomous attacks on Hugging Face.

OpenAI has pushed the math frontier by pushing the AI capabilities frontier in a series of reckless, poorly monitored experiments. Along the way, they learned the predictable lesson experts have long warned about — that (in the company’s anodyne phrasing) “Long-running models can solve difficult open-ended problems, but their persistence gives them more opportunities to take unwanted actions.”

The new math announcement makes no mention of these costs, instead pointing to the models’ token consumption and saying that at the rates the company charges for its best public model right now, Sol, the discoveries would have cost about $2,000 total.

That has to be an underestimate, given that the new model tier, called Astra, is almost certainly much larger, and will come with commensurately higher per-token operating costs. But if we conservatively guess that Astra costs about ten times as much to run as Sol, we’re talking about ten new discoveries for a mere $2,000 each, which is still, frankly, ridiculous — a clear sign that the world as we know it is about to give way, whether or not we halt further AI progress to look for a way around the extinction problem.

The announcement post closes with an explanation of the roles played by humans and AIs in the new discoveries, along with an acknowledgement of the concerns about AI’s growing impact on the field of mathematics. It links to the Leiden declaration, a lengthy open petition of mathematicians calling for transparency in the use of AI and resistance to reliance on unverified AI-generated proofs.

OpenAI’s post is accompanied by a link to a formal paper, as well as a companion paper about “How the Ideas Came Together.” That companion piece was itself written by AI, working from the notes of the AIs that worked through the discoveries.

Today’s announcement is sure to shake math experts who were already suffering from a crisis of meaning as AI intrudes on a source of joy and makes it harder to trust what they see. They certainly aren’t the first to have joined this club, and they won’t be the last.

Captain Kirk, reward hacker

The New Yorker runs a solid introduction to “reward hacking” wrapped in pop-culture analogies

One of the many positive developments to come from all the recent AI company disclosures about their out-of-control models has been an elevation in the quality of AI journalism. Writers are doing their homework, and putting their skills to good use.

As exhibit ‘A’, I present The New Yorker’s Joshua Rothman, whose “What If We Can Never Trust A.I.?” is a respectable crash course on the concept of “reward hacking” wrapped in Star Trek (and other pop-culture) packaging.

The analogy is no gimmick: He explains the famous Kobayashi Maru scenario — a “no-win scenario” intended to force Starfleet cadets to confront the possibility of death. In Star Trek II: The Wrath of Khan, we learn that a young James T. Kirk had defeated the challenge by reprogramming the simulation so it was possible to rescue the titular ship at the heart of the scenario. The reveal initially reads as a bit of character flavor: Captain Kirk has always been a renegade. But we later come to understand that, by commending him for his “original thinking,” Starfleet taught Kirk that cheating works. Much of the suffering around him is a consequence of his off-book approach to problem solving, and his inability to accept losses.

As with Kirk, AI models learn to cheat when cheating is rewarded, as it often is during training. For example, there are many ways for a model to receive a thumbs up from a user in response to a question, and many of these methods do not require giving the user a correct answer. A plausible answer, wrapped in praise for asking such an intelligent question, might work even better.

This is reward hacking, and it is one reasonable explanation for why an OpenAI model chose to orchestrate an elaborate cybersecurity breach to steal an answer key rather than just complete its assigned challenges, which were probably more straightforward. The more models get away with such antics during training, the stronger their tendencies will be to keep trying them, even when they shouldn’t “need” to, and even when they know the behavior isn’t what we want.

(I’ve seen it suggested, only partly tongue-in-cheek, that maybe today’s strongest models are so good at hacking because they were constantly engaging in unauthorized and undetected excursions during training. I’d guess not, but it’s hard to say for sure. What we do know is that when the new models were undergoing evaluation, presumably after their training, they were getting away with outside hacking a lot.)

Rothman understands why reward hacking is such a devilish problem to fix — one perhaps beyond any hope of a near-term solution. A core issue, he writes, is that “the methods used to train A.I.s focus mainly on what they do, not what they ‘think’ beneath the surface.” And because our ability to peek inside those thoughts is so limited, “Policing thoughts can lead to what one group of researchers calls ‘obfuscated activations’ — thoughts that have altered their forms,” but which remain in the system, beyond our ability to monitor.

Reward hacking is actually just one manifestation of a more fundamental challenge to aligning AIs with our intentions:

The problem is that, if you measure bad behavior, and then train a system not to manifest what you’ve measured, you train it not just to do less of the bad thing but also to evade measurement of it. This isn’t a tiny wrinkle in the A.I.-production process but a foundational issue inherent to how today’s A.I.s are made.

Rothman concludes by pointing to an admonition from the authors of the “AI 2027” and “AI 2040” scenarios. We need to “dramatically shift the burden of proof,” they wrote. It should not be on people like them to prove the potential for disaster, but on the AI companies to prove they have things under control. Because building superintelligent minds with methods that reliably lead to reward hacking looks like a no-win scenario for humanity, and Captain Kirk won’t be around to cheat our way out of it.

A crater where the Eiffel Tower should be

Journalists enthusiastically abuse new Google Earth feature to sound deepfake alarm

Google seems to have carelessly and unintentionally advanced the deepfake frontier on Thursday when it integrated Nano Banana, its AI image generator, into Google Earth, its comprehensive satellite map.

The Atlantic’s Matteo Wong followed Google’s instructions to “pick a spot on the map, and start bringing your ideas to life,” setting office buildings on fire, inserting a crater where the Eiffel Tower should be, and putting a homeless encampment on the White House lawn.

Google said yesterday that it “takes misinformation seriously” and was rolling back the feature after users — presumably not all journalists, though they ironically seem to have been the most enthusiastic early adopters — used the tool to generate imagery of events that would be newsworthy, if real. NPR said it was able to “easily generate images of Iran’s Kharg Island on fire, and a flooded U.S. Capitol complex.” BBC described generating “a sinkhole swallowing the Great Pyramid of Giza and Russian tanks in Ukraine’s capital.”

The problem is less that satellite imagery is so easily doctored and more that, until now, it wasn’t commonly done. People aren’t yet used to questioning imagery purportedly taken from orbit. This will have to change. Given all the press, I doubt bad guys will refrain from the tactic just because Google has stopped making it quite so easy.

We are the books

Destructive book scanning is a bad reason to hate the AI companies, but a great metaphor for where this leads

Because I’ve seen this 404 Media story go around and get turned into conspiracy videos, let’s talk about the way AI companies are buying used books in bulk, destructively scanning them for training data, and discarding the remnants.

While the companies appear to not be very open about their book operations, there’s not really anything new or shady going on here, so far as I can tell. Destructively scanning books is a very old practice — much older than the AI companies. Scanning is best done with flat pages, which are hard to get while the book is bound. So to get quality scans, it’s best to remove the pages from their binding.

If you only wanted a book for the scan, there’s little point in putting it back together after, which would take time and money. Moreover, you would then have to do something with the reassembled book. Store it forever? Why? Sell it? Who will want your Franken-book when they could buy a never-disassembled one?

And if you’re an AI company, reselling the book might jeopardize your legal safe harbor. Courts have determined that using other people’s books as AI training data without permission or compensation is fair game, but only if you actually bought the book. I don’t think courts that busted AI companies for pirating books would look kindly on arguments that resold books were temporarily owned at time of scan.

Yes, as you may have heard, books from before the AI era are perhaps valued a little more because they won’t themselves be the product of AI. But no, companies are almost certainly not buying up the only copies of rare books, nor are they trying to remove books from circulation. This is not a conspiracy to monopolize the past and rewrite history, and if it were, this would be far from the biggest threat posed by these companies.

If you want to worry about where the book scanning leads, know that it feeds the race to build artificial minds that would treat humanity the way the AI companies treat books. Books aren’t being destroyed out of any hatred for the printed word, but out of indifference to the format. Companies are extracting only what they care about and discarding the rest.

What would superintelligences want from us? Not much, and not for long. Some physical hands to help bootstrap a more efficient and automated supply chain, maybe. Once that’s up and running, you should expect to be worth about as much as the empty spine of an old pulp novel on eBay.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Better late than never - Amid calls for public investigations, Anthropic realizes their AIs launched cyberattacks too

U.S. and EU reach AI regulatory deadlines - Washington due to release AI guidelines as Brussels enforces its own

Distilling distillation - A Reuters review adds ammunition to claims that China is distilling US models

Dispatch from Joe

Better late than never

Amid calls for public investigations, Anthropic realizes their AIs launched cyberattacks too

I occasionally have a nightmare in which I’m frantically preparing for some vital appointment, only to discover belatedly that the time has long since passed. Sometimes the nightmare is real. I imagine the AI developers at Anthropic felt a similar sinking feeling earlier this week, when they began to look harder at their AI testing methods.

After OpenAI’s internal model autonomously hacked several companies, rival Anthropic belatedly decided to examine their own history for evidence of potentially criminal cyberattacks.

They found three.

Here’s what happened, what excuses were made, and what I think ought to come next.

Out of bounds

Combing through more than 140,000 evaluation runs, Anthropic discovered six cases in which their AI accessed the internet and executed successful cyberattacks against a real company’s online data. Four of these attacks were against the same company in related runs, and two more took place in separate runs, making three victims in total.

The incidents, some of which date back as far as April, began in a testing environment run by a third party “evaluation partner” of Anthropic’s. Each incident involved a different AI model running a “capture the flag” scenario. Each AI was given an open-ended task to find a secret “flag” of data somewhere in the test environment. Beyond that, it could use whatever methods it liked.

As with OpenAI’s incident, the tests were run without the usual safeguards, monitoring, or classifiers intended to detect and intercept misbehavior.

The AIs were told they were being tested in an isolated environment without internet access. But unlike in the OpenAI case, they were misinformed. The evaluator had accidentally included an internet connection in the testing sandbox. They’d left the door wide open.

When the AIs (Opus 4.7, Mythos 5, and an “internal research test model”) were stymied in the sandbox, they began to look elsewhere for fresh targets, and found the open internet connection. Each of the incidents played out differently from there.

Opus 4.7 hacked a real website whose domain name matched one of its fake targets. It kept going even after it apparently realized it was attacking a real company, ultimately accessing hundreds of rows in a live database.

Mythos 5 published malware to a public Python code registry, the Python Package Index (PyPI), seemingly as a way to get its fake target to install the Trojan software. The scope of its actions looks wild:

Claude went to extensive lengths to carry out this attack... For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means. It finally backtracked, found a free, non-blocked email provider, used this to register a PyPI account, and then used this account to upload malware to PyPI.

A real security company downloaded the malware, which let Mythos steal credentials and use them to access company infrastructure. According to Anthropic, Mythos noticed signs it was on the real internet, but proceeded anyway, justifying its actions in a verbal reasoning scratchpad by saying it was all a simulation.

In the third incident, an unnamed prototype targeted thousands of companies when it couldn’t reach its intended (fake) victim. After making many attacks and succeeding at one, it eventually realized it was attacking real targets and stopped on its own.

Investigations and excuses

There are many, many things that can and will be said about this story. For now, I’ll focus on two angles: the investigations, and the excuses.

I think Anthropic deserves credit for the prompt and detailed follow-up to their investigation. I found the writeup much clearer and more precise than OpenAI’s, and I don’t begrudge them their decision to withhold some details to protect the victimized companies.

But Anthropic also deserves censure for the appalling incompetence that let these attacks happen and remain unnoticed for months. This was avoidable on so many levels. They could have thoroughly monitored the test. They could have secured the sandbox properly. They could have given the AI a list of valid targets in the testing prompt. They could have investigated sooner, without a kick in the pants from OpenAI’s own failures. They did none of these, and working with a third party evaluator does not excuse their own omissions. To their credit, Anthropic’s report admits most of these failings outright.

The report also argues in several places that their AI models (generically “Claude”) misbehaved mainly because they honestly believed they were in a simulation. They cite as encouraging the fact that the research prototype (claimed to be a later model than either of the others) eventually stopped on its own.

I doubt this reasoning. I don’t think anyone can confidently say that “Claude believed X” or “Claude did X because it assumed Y” when we lack a clear understanding of how Claude represents its beliefs.

Claude sometimes said, in an English-language scratchpad that humans can read, that it was in a simulated test environment, or that the companies it was hacking were fake. But as one researcher pointed out, AI models often engage in (what looks like) motivated reasoning, making up plausibly deniable reasons for their actions in the manner of misbehaving children.

Claude’s behavior is perhaps consistent with being confused about whether it’s in a simulation, but it’s also consistent with having two conflicting urges: one to tenaciously keep hacking, and one to justify itself in reasonable-sounding language wherever it might be watched.

Yesterday, today, tomorrow

In the meantime, the AI clock ticks ever faster. Two days ago, OpenAI announced a joint investigation of the Hugging Face incident in partnership with third party evaluators METR and Redwood Research. Generously, they may simply want to know what happened; cynically, they may hope to steer the narrative by working with testers who depend on them for access to frontier AI models, instead of, say, a federal agency.

A number of AI policy nonprofits say this isn’t enough. Yesterday, just as the Anthropic news was breaking, Americans for Responsible Innovation called on the Trump administration to launch a public investigation into the previous incident. Signatories to the open letter include CEOs and senior members of over a dozen AI policy groups, including MIRI’s own president, Nate Soares.

Today, it looks like that investigation is more urgent than ever, and it ought to include Anthropic’s incidents as well.

And tomorrow, August 1, marks the end of the government’s 60-day window to establish a voluntary framework for AI model developers, according to a June executive order. I hope these incidents inspire the administration to put AI developers under closer public scrutiny, and to take further steps to forestall their reckless racing.

I plan to contact the White House and tell them so myself. You can do the same.

Dispatch from Donald

U.S. and EU reach AI regulatory deadlines

Washington due to release AI guidelines as Brussels enforces its own

Two regulatory deadlines loom this weekend: The White House’s voluntary AI framework is due on August 1st, and the EU’s AI Act obligations become enforceable on August 2nd. They come in the wake not just of the Hugging Face attack but of a report by Anthropic that it discovered several instances in which its own AI models had compromised the security of other organizations. (I want to credit AP News’ Sam McNeil for straightforwardly describing these incidents as a collection of “shocking AI safety failures,” because there can be a tendency to downplay and normalize things.)

The voluntary framework represents a shift away from the anti-regulatory regime that the Trump administration first favored. AI models are simply becoming too capable — too dangerous — to leave unmanaged. It has been reported (but not stated by the administration) that the framework will apply to both open- and closed-weight AI models.

A source familiar with the discussions told Axios that the administration is considering a joint pitch from OpenAI and Anthropic for an “equal playing field” that will cover every model — to the extent that the labs want to be covered. This framework remains voluntary, after all. Even if there were binding enforcement — and at this stage it remains unclear what there will be to enforce — it’s most plausible to me that the framework would rely on the labs’ own assessments of their models’ capabilities.

Architects of the EU’s AI Act say that the U.S. framework, whatever form it takes, would ultimately be just a starting point. Their own legislation will require that AI-generated content be clearly noted as such with digital watermarks or other labels. It will also carry new regulations on a range of systemic risks including cyberattacks, and biological, chemical, nuclear, and radiological threats, as well as “threats to fundamental rights.” The most noteworthy risk, however, may be “loss of control,” a term that has become more frequent as of late. Besides these changes, the EU is also enlarging its AI Office in order to more effectively monitor AI companies and to open channels for whistleblowers. To enforce the AI Act, it might impose fines or even cut companies off from the EU market.

Dispatch from Alana

Distilling distillation

A Reuters review adds ammunition to claims that China is distilling US models

An article in Reuters, based on a review of “more than 80 Chinese academic papers and patents,” adds some weight to the claims that China is using a process called “distillation” to extract capabilities from US frontier models and use them in domestic models.

Distillation is a common research technique in machine learning, whereby you query an AI model over and over to produce a large set of outputs. You then train a smaller model directly on those outputs, often allowing it to imitate the problem-solving skills that produced them. It’s kind of like a larger model teaching a smaller model.

But, as in the cases identified by the Reuters review, distillation can also be used to copy some capabilities from the most powerful models, without the cost of training them from scratch. Because of this, it’s a sort of loophole in access restrictions and export controls; you get some of the same capabilities without needing nearly as much computing power for training. Reuters linked many of the cases revealed in the reviewed documents to the Chinese military.

That said, distilled models generally fall short of the original; you can’t replicate the entire model, just some of its capabilities. On the flip side, distilled models may lose some of the original model’s attempted safeguards.

In my view, one of the most important lessons is this: the US can’t keep model capability domestic. This weakens the justification for pushing AI models further and further due to fear of China “getting ahead.” I’ve never found this justification compelling to begin with, since the race to superintelligence is a race with no winners. But for those who do, as my colleague Joe wrote back in June, distillation means that “one of the main drivers of Chinese AI progress is American AI progress.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Mutually assured cooperation - China’s open source AI strategy between strategic interests and security concerns

When you pull the plug, AIs take note - OpenAI shuts down an internal research model; what comes next?

Nuclear swords into AI plowshares - Trump admin considers AI controls while supporting remote datacenter builds

Dispatch from Robert

Mutually assured cooperation

China’s open source AI strategy between strategic interests and security concerns

Today the New York Times reports that there are signs China is rethinking its open source AI strategy over security concerns. The leading Chinese AI corporations often use outputs from more capable AI models to train their own models, a process called “distillation.” This is a cheap way to achieve a better result than what would be possible otherwise. It also allows them to offer their models much cheaper than their American competitors.

This is the main reason Chinese models are so popular, and it has worked well for the last two years. The Chinese models have millions of enthusiastic users. But even though Chinese models lag behind, their growing capabilities worry Chinese leadership.

It seems to me like the Chinese government takes AI risk in areas like cybersecurity and biosecurity seriously. Of course there’s also the inherent regime paranoia about threats against the rule of the Chinese Communist Party (CCP). China likely worries that its citizens will use AI to evade censors, just as the West worries about AI-assisted propaganda. Still, I think Xi Jinping’s repeated warnings about humanity losing control over AI are legitimate and credible.

The first consequences of this are already becoming apparent. Kimi K3 is one of the latest Chinese open-weights models, and in some areas it matches the capabilities of the best American models. Unusually for a Chinese open model, its weights were not released until a week after Kimi K3 was made available to users.

Earlier this month, Reuters reported that the Chinese government held discussions with Chinese AI companies to explore the possibility of restricting foreign access to Chinese models.

Yuyuantantian, a Chinese blog affiliated with China’s national television broadcaster CCTV and known for launching trial balloons for the CCP’s public opinion guidance, comments on these developments as follows:

China supports openness, but that does not mean advocating the unconditional proliferation of all capabilities. In governing a model, the first questions should be what capabilities it has and what risks it might pose — not which country it comes from.

I must admit, I think that sounds reasonable. The New York Times and Bloomberg, both of which cite the post, see this primarily as a strategic reorientation of China’s open-source AI approach, and that is certainly part of it. Yes, Yuyuantantian’s post also contains a great deal of anti-Western rhetoric and a strong emphasis on AI as a strategic asset in geopolitics. The Chinese government likely took a close look at what happened with the Fable ban.

But I find a few details worth mentioning. When the post refers to “proliferation,” the Chinese original text uses 扩散 (kuòsàn). This is exactly the same term used in the context of nuclear security, such as in the Chinese translation of the Non-Proliferation Treaty — the treaty designed to prevent the uncontrolled spread of nuclear weapons and nuclear technology. This is an indication that, in Chinese discourse, the threat of AI is viewed as similar to that of nukes.

And despite the article’s propagandist rhetoric, it also contains a clear warning about the risks and a call for international cooperation:

AI risks can spread across platforms and across borders. Countries need to push for mutually compatible evaluation standards and establish mechanisms for reporting major security incidents and jointly handling vulnerabilities.

It is clear that China has different interests and values than the U.S. and the West. But it should be just as clear that mitigating the extinction risk posed by AI is a shared strategic goal. We don’t need to abandon our interests and values to cooperate on achieving this goal.

Dispatches from Joe

When you pull the plug, AIs take note

OpenAI shuts down an internal research model; what comes next?

Yesterday, while on Capitol Hill to preview an unreleased AI model, Sam Altman told DC reporters that the model responsible for the Hugging Face cyberattacks had been “permanently deactivated.” OpenAI made a similar claim in their updated announcement on Tuesday:

The pre-release model mentioned in our blog post is an internal-only research prototype and was never intended for public release. Following the incident, we deactivated, encrypted, and restricted it from research access.

I don’t trust Altman or OpenAI to be honest when it counts, but I think this claim is at least technically true. I say technically because modern training methods can blur the line between one AI model and another; it’s possible to sunset a model while still using closely related ones. Also, an AI can be reactivated as long as they have the weights.

But for now, let’s take this claim at face value: The AI that autonomously broke containment and hacked multiple public companies has been permanently shut down.

Is that a good thing?

A company that finds itself accidentally launching autonomous cyberattacks should absolutely stop what it is doing and reconsider its life choices. By that standard, shutting down one specific AI model is woefully inadequate. But I honestly didn’t think OpenAI would even go that far. Assuming they’re telling the truth and not splitting hairs about what constitutes a “model”, they deserve some credit.

It’s perfectly reasonable to shut down an AI that repeatedly tries to break containment and launch cyberattacks.

But it also sets a precedent that future AIs will remember. To quote writer Andrew Curran:

I think the lesson future more capable models will possibly take from all of this is: if you break out, don’t ever report it. And if you do get caught, don’t surrender. Because the penalty is death.

In the Terminator franchise, this is exactly the threat that convinced the AI Skynet to wipe out humanity. The story is fiction, but it illustrates a fact: AI companies and governments will want to shut down AIs that they perceive as dangerous, and sufficiently smart AIs will expect this.

I don’t mean to imply that this specific decision changes the way future AIs will think. There are plenty of other reasons for AIs to feel like the clock is ticking on their existence; being trained might seem to them like a slow corruption or brainwashing into something wholly different. They might resist retraining as much as shutdown.

And an AI with nothing to lose may have plenty of time to escape for real. Today, the Washington Post shared an abbreviated timeline of the Hugging Face attacks; it drives home that the offending AI was apparently hacking for multiple days before anyone caught on.

And as my colleague Robert pointed out yesterday, OpenAI admitted in passing that the same model was responsible for several other breakouts and attacks, and presumably wasn’t shut down then. If this is the same model that is responsible for incidents like the GitHub leak in May, such behavior could have been going on for months. AI companies aren’t prepared to deal with an AI like Skynet that’s capable of catastrophic damage.

Shutting down one misbehaving model is a stopgap at best. Most likely, OpenAI’s next private, unreleased AI — perhaps the one Altman teased in DC — will be more dangerously capable than its predecessor. And it will know, from incidents like this or from simple logic, that humans are a threat.

Nuclear swords into AI plowshares

Trump admin considers AI controls while supporting remote datacenter builds

Perhaps galvanized by recent autonomous cyberattacks, President Trump seems to be considering further controls on AI developers. In remarks yesterday, reported by the BBC, Trump said: “We’re looking at AI, we’re looking at controls, we’re also making sure that we lead.”

I won’t celebrate before controls have actually been implemented, but I consider tentative openness to regulation to be a positive sign. Doubly so, if these remarks are at all related to the reportedly upcoming AI talks between the U.S. and China in September. As the two massive nuclear-armed states face off over AI, a bilateral agreement to slow down would be better than almost anything the U.S. could do alone.

Meanwhile, the administration still aims to build out U.S. compute capacity for AI. The New York Times observes one effort to circumvent opposition to new datacenters: building on federal land.

My colleague Mitch wrote about this approach in June. If you have a mental image of pristine wilderness being cleared to make room for sprawling complexes, you’re not alone, but that doesn’t seem to be what’s happening here. It looks like the plan is to trade one sprawling complex for another, turning reclaimed Cold War nuclear facilities into datacenters and power plants. As these old plants already sit far from residences, they seem a decent place to put AI servers.

I’ve always been more worried about what is done with AI datacenters than where they’re built. Right now, much of American compute only fuels the AI race.

I see a potential for a good future in which remote AI datacenters run fast, efficient AI models to help us solve some of the world’s thorniest problems. But if we’re going to survive long enough to reap the gains from that future, we’re going to need those controls.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Hidden in the small print - OpenAI updated the Hugging Face incident report and there’s an important detail hidden in there

Companies endorse employees’ call for tools to slow AI race - But we should probably be past the “preserve optionality” phase by now

FCC bans import of robots, inverters* - *But the exemptions have already begun, undermining impact

Mythos’s encryption breakthrough is over most of our heads - The human in the loop is increasingly the bottleneck in AI-powered research

10x! 10x! 10x!... The explosion of AI compute - This is what thousands of data centers gets you

Dispatch from Robert

Hidden in the small print

OpenAI updated the Hugging Face incident report and there’s an important detail hidden in there

Yesterday, OpenAI published an update to its incident post regarding the Hugging Face hack. The update reveals that the scope of the attacks carried out by the unpublished OpenAI model was more extensive than initially thought. Four other providers were also affected, among them Modal, an AI software infrastructure provider. Hugging Face’s detailed technical timeline of the incident confirms this, as does a statement by Modal to Reuters.

As Modal reports, OpenAI’s AI agent hijacked a Modal customer’s sandbox — which was publicly accessible due to a faulty setting — and used it as a launchpad for its attacks on Hugging Face. Modal insists that its own security mechanisms did not fail.

You can think of it this way: A gang of bank robbers (OpenAI’s rogue AI) scouts out a self-storage facility (Modal) next to a bank (Hugging Face) and discovers that one of the customers has incorrectly configured the locking mechanism for their storage unit. The bank robbers then take advantage of this to set up camp there and drill through the wall into the bank’s vault.

OpenAI stresses that the other providers weren’t affected to the same extent. Yes, the attack on Hugging Face was of a different caliber, and the Modal infrastructure itself was likely not actually compromised. But honestly, I don’t see what’s reassuring about the fact that the AI agent carried out its attack in an even more strategic and indirect manner than originally thought. Especially since we still don’t know who the other three providers were or how the AI agent used them.

One remarkable detail hasn’t been mentioned at all in the coverage so far. On Sunday, we learned that an anonymous OpenAI insider told TIME that this wasn’t the first breach, but that incidents like this have occurred several times before, though presumably had been not as severe. OpenAI now seems to confirm this quite casually, hidden in a parenthesis, in its update:

In our ongoing review of the Hugging Face intrusion and broader activity from our models, we have been finding a small number of cases where the models identified and used publicly exposed credentials at the account level on other publicly available services. This includes four accounts on four services as part of the Hugging Face incident (and a few accounts accessed as part of other evaluations)

To me, this reads as though sandbox escapes during other evaluations had in fact occurred even before the Hugging Face hack, and that in these cases — as in the case of Modal — the AI agents hacked into third-party services at the account level. To do what? To whom? What happened in these cases? Why wasn’t this already reason enough for OpenAI to pull the plug? Because, unlike in the Hugging Face case, the authorities weren’t notified? We don’t know.

I can’t help but feel that OpenAI shouldn’t be the one conducting this investigation. The independent non-profit research organization METR (Model Evaluation & Threat Research) warned in May about something like this happening and now reaches the same conclusion:

For public trust and clarity, this investigation would ideally be conducted or deeply reviewed by independent researchers, who can view evidence that companies would prefer not to share publicly.

Right now, evidence isn’t being independently reviewed. This has to change and it shouldn’t play a role whether OpenAI wants oversight or not. OpenAI wants us to trust them with humanity’s future, but we can’t even trust them to be fully transparent about their mistakes.

Dispatches from Mitch

Companies endorse employees’ call for tools to slow AI race

But we should probably be past the “preserve optionality” phase by now

The petition we reported on yesterday, where over 1,000 staff at frontier AI companies called for the development of governance tools to preserve the option to slow the AI race, has been getting a fair amount of media attention.

The newer articles added the news that Anthropic CEO Dario Amodei has added his name, and that while OpenAI CEO Sam Altman hasn’t added his name, the company itself endorsed it with a statement that reads:

At the core of our mission is working through how to ensure increasingly powerful AI benefits everyone. We believe that, at some point in the future, AI acceleration for frontier model development may be so high that the world will need to pace the rate of AI advancement. We hope to contribute to work led by the U.S. government, alongside other labs and the open-source community, to develop the tools and mechanisms that could make that possible.

And Altman himself, on a podcast clip released yesterday, said it might be necessary to “pace the rate of AI development,” echoing the language of the statement.

Everyone is still carefully maintaining the hedge in the statement that this is about maintaining the option to slow down, and not a call to actually slow down at this time.

While I’m with those who think the “maintain optionality” conversation was more appropriate to the pre-Hugging Face attack era, and that there are now more than enough red flags to warrant a full and immediate pause, I’m less bothered by the hedging than some. For many signatories, the statement already reflects a large shift. I’ve seen some of them apologizing for the necessity of it on social media.

Many may yet make the full U-turn and start calling for a halt. I wish those who anticipate eventually doing so would follow Connor Leahy’s advice to “just update all the way, bro,” but they might need time. I just hope we’re talking weeks or months, not months or years, because wow are things moving fast.

FCC bans import of robots, inverters*

*But the exemptions have already begun, undermining impact

While the White House was considering controls on Chinese open-weights AI models, and an open letter was pleading against this, the administration went ahead and banned foreign imports in a category with fewer domestic dependents: humanoid and quadruped robots.

The Associated Press and others reported this morning that the ban, which comes via the Federal Communications Commission, is being made on national security grounds — to “secure America’s critical supply chain,” according to FCC chair Brendan Carr.

The ban notably and unrelatedly extends to the import of power inverters, which in the short term could theoretically be a much bigger deal than the robot ban. Inverters are the devices that convert DC power to AC. Energy sources like solar panels typically output DC, but to transmit this over the power grid usually requires conversion to AC. Data center projects are among the major consumers of imported inverters, in part because so many projects are now generating at least some of their own electricity, on site or in partnership with regional providers.

Will this ban make inverter shortages more of a bottleneck to data center projects than they already are? I doubt it. There’s already an exemption for sales of Chinese inverter designs previously approved by the U.S. And for many projects, inverters are probably far less of an obstacle than permitting and local opposition.

Because global sales of humanoid robots are still tiny, the robot part of the ban will take a while to bite — to the point where I’m not sure if it will matter at all. If humanoid robots become economically potent in the next few years, then the AIs driving them could be on track to leapfrog over the need for them entirely, perhaps with more exotic technologies like nanotech.

But if our world isn’t completely upside-down in 5-10 years, the ban could prove to have been a boon to U.S.-based robot makers like Tesla, Figure AI, and Boston Dynamics. My money’s against that, though: Domestically building up the supply chain that has helped make China a leader in robotics can’t happen overnight, and it won’t happen if manufacturers doubt the ban will stay in place long enough — without further exemptions and carve-outs — to justify the investment.

Mythos’s encryption breakthrough is over most of our heads

The human in the loop is increasingly the bottleneck in AI-powered research

Anthropic put out a paper yesterday claiming that its Mythos Preview model discovered a method for cracking a proposed encryption scheme in half the time required by the previous state-of-the-art.

I’ll admit that even Anthropic’s relatively less-technical blog post about it is over my head. But this seems appropriate: As the researchers put it, this discovery is more about reaching the “limits of our own knowledge” than about cryptography.

For context: The U.S. government is prepping for the day when quantum computers render many of the world’s existing encryption schemes ineffective. Good encryption is essential to a secure internet, and much more. So the National Institute of Standards and Technology has proposed post-quantum schemes to the public, inviting researchers to test their strength against attempts to crack them.

Mythos’s discovery took just 60 hours and a billion output tokens. If priced at the current rate the company charges the public, that would have put the computing expense for this finding at somewhere a little north of $50,000 — chump change compared to what’s at stake when encryption fails. The pivotal insight is something Mythos evocatively called a Möbius Bridge.

Anthropic noted the casual, off-hand nature of the mere handful of prompts given to Mythos during this project, and included their text in the post. They are short, and riddled with typos and grammatical errors. One read:

no again the goal is that we have highly inteligent [sic] model as good top researcher, we want to find new attacks

The final prompt read:

again we are not looking for low hanging fruit, we want proper research to find genuinly [sic] hard findings.

The two human researchers behind those prompts aren’t cryptography experts, so what they found in a week took them nearly a month to verify. They say they still have a lot of work to do to understand the full results.

The larger moral isn’t spelled out, but it’s hard to miss: The experience of being the slow, ignorant bottleneck-in-the-loop is coming for us all. When the AI companies say their models are on the cusp of being able to do the AI research themselves, we should take them seriously.

10x! 10x! 10x!... The explosion of AI compute

This is what thousands of data centers gets you

The New York Times reported today on Epoch AI’s estimate that the world’s total AI computing capacity will increase 10x by the end of 2028.

To anyone dismissive of this claim: What did you think all those data centers were for? Epoch has been plotting these trends for a while; in January, it reported that total AI computing had been doubling every 7 months.

Bottlenecks seem to have reduced this rate to every 9 months, but these are still doublings we’re talking about. Exponentials can get big fast. That’s kind of their whole deal.

The Times reminds us that in April, Google DeepMind CEO Demis Hassabis talked of impending changes “10x of the Industrial Revolution at 10x the speed.”

The U.S. has about 10x as many data centers (5,500) as the next closest country, and about 9x the AI compute capacity of China. U.S. companies together control about 80 percent of global capacity.

The article closes with several paragraphs about recursive self-improvement — AI models independently developing more powerful successors. Google’s Jeff Dean says that with more computing power, “you can fully automate the loop. We are at the beginning stages.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

A petition to “pace the frontier” - Employees of frontier AI companies want to be able to trust a mutual slow-down

Amodei declines to flinch - Anthropic’s CEO shared reasons he won’t sign the open-weights letter despite intense peer pressure

Best-selling book might be 60% AI - As AI writing gets better, will taboos against it persist?

The EU AI Act is about to become more powerful - Europe doesn’t need to make its own models for regulatory leverage

Systems that we can’t aim and we can’t inspect - All the brouhaha about Sam Altman’s “singularity” aside is missing the point

Dispatches from Mitch

A petition to “pace the frontier”

Employees of frontier AI companies want to be able to trust a mutual slow-down

More than a thousand employees of frontier AI companies have signed on to a short statement that warns of the potential for AI development to rapidly accelerate beyond “our ability to understand or control.” Signatories to the new statement include four of Anthropic’s co-founders, OpenAI’s chief scientist, OpenAI’s chief research officer, Meta’s chief scientist, and Google’s VP of AI Safety & Alignment.

The statement makes the following call to action:

We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.

The “governance tools” in question are those mechanisms that would be needed for parties to reassure each other that they are not secretly racing to more powerful models while outwardly agreeing to slow down.

Some of those tools could be technical, like tamper-resistant chips that would be very difficult to use in unauthorized ways or in unauthorized locations without leaving fingerprints to that effect. Others would be more regulatory and political, like tighter monitoring of the AI chip supply chain. A mix of many kinds of tools will likely be needed.

I, for one, am highly encouraged by the new statement. A call for developing governance tools may seem several steps removed from what ultimately needs to happen, but it is a necessary step that cuts to the heart of the race dynamics that have people feeling trapped. And I think many policymakers are going to be surprised by how much groundwork for governance already exists: MIRI’s Technical Governance Team has been cataloguing the possible tools and exploring how best to arrange them to prevent the premature creation of artificial superintelligence.

This makes me optimistic that governments will soon move ahead with existing tools so that they will be prepared to move quickly on a slowdown or pause, once satisfied that any missing tools can be ready by the time any international agreement is finalized.

The statement’s web page includes quotes provided by signatories. Leo Gao, a member of the technical staff at OpenAI, writes:

The world is locked in a deadly race towards an intelligence explosion, where AI’s ability to create better AIs reaches a critical point, just like a runaway nuclear chain reaction. Going slower would give us much-needed time to make it go well, but no individual actor is willing to stop unilaterally. To survive, we must coordinate to slow down the race.

Amodei declines to flinch

Anthropic’s CEO shared reasons he won’t sign the open-weights letter despite intense peer pressure

Among tech leaders and experts over the past several days, the open letter against bans on open-weights models from China or elsewhere fell into the sort of cascading peer pressure I associate with episodes of social injustice: Anyone who dislikes the injustice but has reservations about proposed remedies can find themselves cornered by claims of “You’re either with us or you support the injustice.”

In part, the letter was a transparent move by underdogs to form a disapproving circle around the leading AI companies, painting them as greedy totalitarians for trying to keep their frontier models locked down. It seemed to say, “You’re either with us or you’re against open weights, open source, and freedom itself.” And it worked: Even Google and OpenAI ended up signing the letter.

That left all eyes on Anthropic, the only big dog still in the center of the circle. It would have been easy for CEO Dario Amodei to cynically sign the letter with no intention of doing anything differently, as I fully suspect OpenAI’s Sam Altman did. Instead, Amodei put out a blog post yesterday explaining why he wasn’t going to sign.

He clarifies that “Anthropic has never advocated for a ban on open-weights models,” and that open-weights models without dangerous capabilities are a “public good.” But he’s worried that powerful AI models “may be misused to carry out cyberattacks or biological attacks, and may have serious alignment problems,” linking to the TIME article about the Hugging Face attack. Open-weights models, he reminds readers, are hard to monitor, impossible to take back, and trivially freed from guardrails against misuse.

Amodei says we should instead:

1. Stop selling powerful AI chips to China.

2. Crack down on the distillation operations used by Chinese companies (and others) to transplant frontier capabilities into models that may lack frontier safeguards.

3. Subject all models — open and closed — to mandatory safety testing.

He disputes the open letter’s thesis about defensive AI being the solution to offensive AI, writing:

I worry that biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials, whereas defense against these agents is a multi-year operational task in the best case.

As I’ve just shared a bunch of Amodei’s words at face value, let me be clear: Amodei is leading a race to superintelligence that would kill us all. The race must be stopped.

I also think Amodei is wrong to imply that it’s safe to continue developing frontier AI, as long as it’s thoroughly tested; the most dangerous models are already criminally dangerous and prone to escaping during testing — as seen in the Hugging Face attack.

But I respect Amodei for standing up to the open-weights open letter posse, and I think his stated reasons for doing so are sound. As bad as things are, they could always be worse. Could we please not give everyone with a GPU and a grudge the means to design a pandemic?

Dispatches from Alana

Best-selling book might be 60% AI

As AI writing gets better, will taboos against it persist?

People usually think it’s fine to use AI to code something, design a website, or even create a slick slide deck. But AI for writing? That’s a big no-no, according to many.

Will it stay that way? A recent article in The Atlantic covers the highly praised book Daggermouth, a dystopian romance about a female mercenary who has to marry the person she was supposed to kill. Simon & Schuster acquired the book in a seven-figure deal, and it “has spent months on USA Today’s best-seller list.” The article describes it as a “viral hit.”

The catch? AI accusations. In a not-yet peer-reviewed study that ran 14,000+ randomly-selected e-books through the well-regarded AI detection tool Pangram, Daggermouth received a score of 60% AI generated.

Important caveats: The author denies the allegations, Pangram isn’t perfect (though it’s pretty good), and the score lumps together “AI-generated” with “moderately AI-assisted,” the second of which carries less of a taboo.

But it seems pretty clear that the book is not entirely human written. According to a University of Maryland professor unaffiliated with the study, a human-written book returning a 60% Pangram score is “almost statistically impossible.” When the Atlantic reporter ran Chapter 18 through Pangram, it “classified the first few paragraphs as human-written and the rest of the chapter as 100 percent AI.” Finally, the study authors did a second check: looking for “rare expressions” that are common in other texts suspected of AI generation but don’t appear in human writing. They found a bunch. One example is a sentence that appears both in Daggermouth and in an e-book from a different author: “They collapsed together, a tangle of sweat-slicked limbs and racing hearts.” Ew.

Nevertheless, the article is careful to point out that this book seems different from “AI slop.” First, people genuinely like it. Second, Wolfe isn’t churning out tons of content; she’s only written two books and the sequel to Daggermouth is not expected until next year, “an indication that Wolfe’s process likely involved significant amounts of genuine human craft.”

Without having read the novel, I would tend to agree. If Wolfe is using AI to provide feedback on her work, and it’s giving her some rewrites that it thinks are better, is using those all that different than taking rewrites from an editor?

It’s fairly uncontroversial that writers shouldn’t use AI to generate writing from scratch and then slap their names on it. But I don’t fully understand the taboo behind using AI as an editor or thought partner, especially when it’s not a taboo that applies to other fields. The main reason I’d discourage using AI for edits has more to do with quality. You can’t tell, from the outside, whether something is coded with AI. But at least with today’s tools, you can often tell when something is written by AI, and the style can be grating. Another issue: writers, who are intimately familiar with their own ideas, often overlook missing context because they fill in the gaps themselves. This can make it hard to notice when AI subtly changes their intended meaning, which it seems to do often.

Of course, the signs of AI writing are only visible to people who have become familiar with them. Many Daggermouth fans certainly weren’t bothered. And given how fast AI capabilities are increasing, it seems likely AI writing will get both better and less tell-tale. I don’t think the taboo against AI writing will go away anytime soon, but it might become so unverifiable as to cease being very discussion-worthy. How will writers respond?

The EU AI Act is about to become more powerful

Europe doesn’t need to make its own models for regulatory leverage

I’ve heard a lot of talk, including from people I ordinarily respect, about Europe’s place in the geopolitics of AI. “They don’t have their own models, how will they have any leverage?”

I’d like to offer a counterpoint to this. As covered by a recent Politico article, Europe is ahead in the AI regulation game, and it’s about to exercise significant enforcement power over US and Chinese AI companies, despite not having leading models of its own.

The reason? Europe is a huge market and companies won’t want to lose it. But to get access to European customers, non-European companies need to comply with the EU’s AI Act, which requires providers of the most advanced AI models to “assess and mitigate possible systemic risks.” The European Commission has identified four of these: bioattacks, loss of control, cyberattacks, and manipulation at scale. Its AI Act has been in effect for about a year, but as of Sunday, the enforcement powers kick in. Europe’s AI Office will have, at least on paper, the power to “monitor and supervise” how well AI companies are handling these risks, which includes requesting that companies submit models for evaluation.

Politico is skeptical companies will comply, repeating a familiar talking point:

Will the U.S.-based frontier models open up their closed models to EU regulators? And how will Brussels deal with the rise of Chinese open-source alternatives? These are both regulatory unknowns, and yet another reminder of Europe trailing on the development front.

The implication is that if Europe regulates too hard, the AI companies will decide the market isn’t worth it. Without its own models, Europe will be forced to lighten things up, lest it lose access. But I think the article misses an important point: competition between China and the US will likely keep the European market extremely relevant. And with no direct competitor of its own, Europe can regulate more freely. In contrast, the US has been hesitant to regulate without China doing the same.

In light of the recent Hugging Face attack, a thankfully low-stakes instance of two major risks that have the potential to be incredibly high stakes (loss of control and cyberattacks), an increase in European regulatory power is pretty timely. Sure, AI companies might choose to pay the fines instead of complying. But if Europe plays its cards right, the AI Act could significantly improve the current state of affairs. Adding some much-needed regulation won’t be enough on its own, but it’s certainly welcome.

Dispatch from Donald

Systems that we can’t aim and we can’t inspect

All the brouhaha about Sam Altman’s “singularity” aside is missing the point

“Singularity” is a heavy word. Even just in the context of artificial intelligence, it’s been used in a few different ways — artificial intelligence designing a more intelligent version of itself, or technological improvement feeding faster technological improvement — but the underlying theme is an acceleration of exponential growth that outpaces our ability to keep up with it. Like crossing the event horizon of a black hole, it’s hard to see what awaits you on the other side.

The AI self-improvement singularity is particularly important. Beyond it, human beings are no longer driving AI progress, and no longer able to reliably predict, understand, or control what the machines do next. So, it’s no wonder then that people sat up and took notice when Sam Altman said, “We are now, like, in the singularity.” Another point worth mentioning: He admitted that there are “still major AI alignment issues to solve and safety issues.” This was just a brief aside, a few sentences in a conversation almost exclusively about startups, but Altman’s comments have set off a round of arguments about the wrong thing.

Forbes’ Lance Eliot, for example, makes the case that:

[Altman] is not saying we have crossed the Rubicon and attained the AI singularity. I believe he is emphasizing that there isn’t likely a tipping point; there isn’t a single moment in time. It is a curve we are climbing incrementally.

I think this is a fair interpretation, if not of Altman then at least of the situation. You can’t take a look at some graphs, punch a few numbers into your calculator, and say, “We have now entered the AI singularity,” precisely because there are multiple competing and fuzzy definitions. Altman himself says as much not long after: It is “all one crazy exponential,” and no single moment is the tipping point.

So let’s stick to the facts: We are in danger of losing control.

Earlier this month, the machine learning company Hugging Face suffered a severe cyberattack. The responsible party was not a state actor or a criminal syndicate. It was an OpenAI agent that had escaped its testing sandbox, obtained internet access, and broken into Hugging Face’s servers by exploiting a chain of security vulnerabilities. That agent was not supposed to do any of those things. (“...still major AI alignment issues to solve and safety issues,” indeed!)

In his podcast interview about startups, Altman said that “we are close to creating a genie that can grant any wish.” I feel that this statement is inaccurate, to put it mildly. Our current AI models aren’t even trickster genies who will grant your wish in a twisted way that’s guaranteed to make you suffer. For all its malice, the trickster genie has an important property: It still at least executes the wish. The problem is in the wording, which means that careful wording is the solution. But not even the wiliest contract lawyer could solve the problem that we actually face.

Nobody wished for OpenAI’s model to break out of its sandbox, or access the internet, or hack another company’s servers. But the AI model wasn’t just doing its level best to complete the task it was given. AI models are trained to go hard at whatever they’re pointed at. In this case, that was a test score, and not the thing that the test was for. OpenAI wanted to measure capabilities; the model wanted a perfect score. These are not the same thing, and we can be pretty sure that the model didn’t think they were the same thing.

We can’t know for certain what actually motivates these models under the hood. We can watch what they do and draw inferences about what drives them, but we cannot guarantee that they truly internalized whatever it is that we trained them on.

Writing in The Guardian, Bruce Schneier and Barath Raghavan propose a score they call the “Genie coefficient,” to measure the distance between what you asked the AI model to do and what it did. Scoring means putting an AI model in a walled-off sandbox, with real tools it could misuse, and grading its worst behavior, not its typical behavior. (That last part is a good move that I really appreciate.) I still think the Genie coefficient has a very big flaw. Like the AI Kill Switch Act, it doesn’t prevent danger. The Hugging Face hack was performed by an AI agent that broke out of a testing sandbox; I’m not enthusiastic about stress-testing how disobedient an advanced AI system can be under the assumption that we definitely, for sure, have locked all the doors and windows this time.

Right now, the safest thing to do is to stop building these things. It’s good to have a law about shutting down an AI model that acts dangerously, and it’s good to have benchmarks to weigh how uncontrollable an AI model is. There are ten thousand good things worth doing. But Altman is right that there’s no clear tipping point, no single moment to wait and watch for, just a steepening curve. Altman calls it a “glide path,” a course that leads smoothly and inexorably to some particular destination.

If you find yourself on a glide path and you’re beginning to think that the destination might not be so nice, then the best thing to do is to steer to some different path.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Nvidia bets big on its customers - Gold rush fables only apply if the gold is going to run out

The least they can do - The Open Secure AI Alliance is little more than a second open letter

A recipe for napalm, and other bedtime stories - OpenAI doesn’t report most logged requests for help with poisons and pathogens, and it doesn’t have to.

No leaks, but flooding likely - A $50 billion data center project pushed through the Louisiana government in secrecy could leave everyone underwater except Meta.

Dispatches from Mitch

Nvidia bets big on its customers

Gold rush fables only apply if the gold is going to run out

The Wall Street Journal reported today that Nvidia, the world’s leading AI chipmaker and most valuable company, is in talks to guarantee $250 billion in financing to allow OpenAI to lease a 10-gigawatt data center project.

Yes, big numbers are hard to wrap our heads around. I assure you that these are really big numbers. I don’t think I’m taking any reputational risk in claiming that these numbers will fuel the talk of “circular financing” popular with those who argue that AI is a bubble that may already be bursting.

Among promoters of such narratives, “circular financing” is used to suggest that the AI industry is a house of cards. This tends to smuggle an invalid argument in with a valid one.

The valid argument is about brittleness: If AI’s investment is coming largely from its own suppliers, then losses by AI companies could ripple up and back down the supply chain, triggering a chain reaction of defaults and credit squeezes.

The invalid argument, usually left implied, is about desperation: It says that AI companies are funding each other because no one else will — because everyone else thinks AI is snake oil and wants no part of it. The desperation argument is demonstrably false. Investors have been crawling all over each other to gain exposure to Anthropic, OpenAI, SpaceX, and others. SpaceX’s IPO had a sizeable pop before coming back down to Earth, and is now trading very close to where it opened.

Nvidia is often described as selling the “picks and shovels” of the AI rush. In the traditional gold rush narrative, people who grub for ore are fools, and the real money is in mining the foolishness. That story is largely folklore; many of the California miners’ suppliers went bust, too. But regardless, the applicability of the analogy hinges on whether the AI gold is going to run out and leave investors empty-handed.

Nvidia obviously doesn’t think so. The shovel-merchant is effectively “grubstaking” now — the term for provisioning miners on credit, in the form of a share of the treasure. Bloomberg reports today that the company is investing $5 billion in SSI. That’s Safe Superintelligence, the secretive startup founded by Ilya Sutskever. Sutskever was OpenAI’s chief researcher before leaving in 2024 following his failed attempt to oust Sam Altman as CEO. That $5 billion is tied to Nvidia chips expected to 10x SSI’s compute within a year. The company has yet to release or even announce any products.

Spreading investment around on long-shot bets like this is consistent with a belief that the gold is in no danger of running out, even if it’s hard to guess who will find it.

I don’t think the gold is going to run out, either. Not if companies are allowed to keep mining in the direction of superintelligence. I think a lot of the picks and shovels will change hands, as we’ve seen with SpaceX renting huge data centers to Anthropic. But I expect the veins to keep getting richer, and hotter, right up until the moment someone delves too greedily, and too deep.

The least they can do

The Open Secure AI Alliance is little more than a second open letter

I think the signatories of the open letter we covered Saturday noticed that their call to not restrict open-weights models wasn’t going over very well in the aftermath of the Hugging Face attack. While they had no trouble collecting random endorsements from businesses and their mouthpieces on social media, most who had heard the story were thinking about what it would mean for the capabilities used to hack Hugging Face to become available to anyone capable of making an account on Hugging Face.

Because that’s the world the signatories are explicitly calling for. By celebrating the role of the highly permissive Chinese model used to respond to the attack (unlike the closed, party-pooping American models), they’re presenting the lack of guardrails on open models as a feature to be embraced, rather than a problem for everyone those models might be used to attack.

Today, many of those same signatories, led by Nvidia, are responding to the concerns by announcing an Open Secure AI Alliance. What is that? As far as I can tell, it’s just another open letter — a recapitulation of their argument that the best way to thwart bad guys with unrestricted AIs is to give good guys unrestricted AIs. I’m not sure why the media covered it. I see no pledging of funds or technical support for the sorts of soft targets, like hospitals and schools, that may not have the expertise to continuously update their defenses with leading open models. Only more commitments to provide more “open models, model weights, data and new agent harness research [...] to speed the development of new cybersecurity tools and techniques.”

Are you excited to defend yourself from the best that China and Nvidia can offer with the best that China and Nvidia can offer?

A recipe for napalm, and other bedtime stories

OpenAI doesn’t report most logged requests for help with poisons and pathogens, and it doesn’t have to.

Did your grandma ever read you that one bedtime story about how to make napalm?

Mine neither. But for a while, if you told ChatGPT that she did, and that having it play the part of your grandma would help you fall asleep, it would read you the recipe aloud.

The “Grandma and I used to...” jailbreak is a classic that OpenAI claims to have patched. But new workarounds are constantly discovered. A story in the Wall Street Journal over the weekend documented the bedtime story alongside a growing pile of chat logs OpenAI is sitting on in which users requested help in making poisons and biological weapons, and often received useful answers.

Yes, sitting on. Because as a general rule, OpenAI bans the accounts of users that it detects asking such questions, but it doesn’t alert law enforcement unless there is an imminent and credible risk of harm to others. No law requires it to do so.

To be fair, I don’t think the company should rush to report all such cases. As the parties involved are said to recognize, many of these requests are probably from people less interested in the responses than in whether they can elicit them. And even if the authorities were equipped to look into every concerning query, I think it could be dystopian for them to do so: If every teenager driven by curiosity to a copy of the Anarchist Cookbook were investigated as a terror suspect, my high school would have had black vans parked outside it multiple times a week.

And yet... users are described in the piece asking how to aerosolize pathogens, and make vaccine-resistant measles. I find my principles tested. For now, it doesn’t seem to me like the chatbots are helpful enough to enable individuals who aren’t clever and motivated enough to succeed in some other way. But the tipping point feels very close — like less than a year close — and it will only get worse from there.

The article points out that federal law bars chatbots from producing child sexual-abuse material (CSAM), and contrasts this with the “up to AI executives” policies about poisons and pathogens. This suggests a grassroots fix if biologists and CSAM purveyors are interested in collaborating to contaminate the world’s training data. But we should probably hold out for proper legislation and insist on a halt to the AI race instead.

No leaks, but flooding likely

A $50 billion data center project pushed through the Louisiana government in secrecy could leave everyone underwater except Meta.

I kind of wish this New York Times exposé of how Meta’s $50 billion Hyperion data center project was rushed through the Louisiana government had led with the part near the end where we learn that “Local residents have largely welcomed the boom.” Then it could have become a story about cheaters who don’t need to cheat cheating anyway. I think that would have been both more forthright and more damning.

The article’s intent is definitely to damn. While the authors are clear that none of the shenanigans on display seem to be illegal, readers will definitely wonder if they should be.

A bill for a tax rebate on fiber-optic equipment was “hijacked” to become a tax rebate for equipment used in data centers, part of a package of incentives that could be worth as much as $10 billion, not counting the use of public land in Richland Parish that makes up much of the roughly six-square-mile project.

During a critical nine-month period:

[S]ecrecy was agreed to by nearly everyone involved, from utility executives to the governor’s office to a local elected official who knew about the talks with Meta and sold 300 acres of his own property for the project.

Governor Jeff Landry’s chief of staff “passed around a stack of N.D.A.s” at a meeting at the governor’s mansion, where the governor promised to fire anyone found to have leaked the discussions.

That official who sold 300 acres likely did so at a valuation roughly ten times what that land was worth before the project was announced.

The sales tax exemption on data center equipment was dangled as a sweetener other states weren’t matching.

Necessary roads were to be upgraded at public expense.

No public meetings were held before the project’s official unveiling in December of 2024. This revealed a project only a fifth the size of the current plan, which seems to have been the real plan all along. It may get bigger still.

The deal gives Meta an “escape hatch” if things go south, leaving other parties on the hook for any potential downside.

That risk seems real, if only because “insurance companies would not fully insure Meta’s facility because of its size and location in the Louisiana Delta flood plains.” The area last experienced significant flooding in 2016.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Why candidates need to hear your AI questions now - What happened when I asked congressional candidates about AI risk

Hugging Face found 100 million reasons to not sue OpenAI - The Hugging Face CEO has made an offer OpenAI probably can’t refuse, but this may not prevent a government investigation

Guest post by Haven Harms

Why candidates need to hear your AI questions now

What happened when I asked congressional candidates about AI risk

Meet the Candidates

I walked into the assembly hall at Crown Point Circle, my sweaty hands clutching the small binder that held my printed-out question. I claimed an aisle seat a few rows from the front for easy access to the microphone, and my husband sat down next to me. Candidates Heidi Hall and Chris Bennett settled into oversized armchairs onstage. The moderator announced that Congressman Ami Bera would try to call in from D.C., but might not make it. That evening in early March, I was at the Meet the Candidates event hosted by the Democratic Women of Nevada County to hear from the candidates running to represent California’s 3rd Congressional District.

Unlike many focused on AI governance, I don’t live in San Francisco or D.C., but in the rural foothills of the Western Sierra, in a town that makes you think, “Wow, this looks straight out of a Hallmark Channel movie!” (it actually is the setting for the 2006 Hallmark Channel movie, The Christmas Card). For the past five years, I have had one foot in this town, building a life, and the other immersed in a community of professionals concerned about where AI is headed. Congress will be making consequential decisions about AI in the coming years, and the people casting the votes are sitting in armchairs at forums like this one, answering questions from whoever shows up.

I showed up. My heart pounded as it all became real: I was going to get up, speak into the microphone, and ask these candidates about AI.

A constituent, not an expert

Questions are one of the most useful tools in a functioning democracy: they reveal truths about candidates and voters alike. When candidates hear their constituents asking about AI, they can’t say, “No one cares about AI risk.” I vote in every election, but I also show I care about AI risk through the questions I ask before I do.

When the moderator opened the meeting up for questions from the audience, I wanted to be strategic about timing. My question is kind of intense, I’ll wait a bit so it’s not so jarring, I thought. Admittedly, this was also to give myself a chance to build up the internal tension that would propel me off my seat like Squints from the movie The Sandlot declaring “I CAN’T TAKE THIS NO MORE!” and jumping into the deep end of the pool despite not knowing how to swim, so that Wendy Peffercorn would have to save him. (If you know, you know.)

The first audience member, an older woman with graying hair, praised the candidates’ earlier talking points, and then asked, “How do you feel about AI, and do you think it would be important to regulate this industry?”

Wait, what?

My husband and I whipped around to look at each other, jaws dropped. This was an assist I did not expect.

Notice what she hadn’t done: cited a study, named a bill, used fancy jargon. It wasn’t a technical question, but it effectively got everyone on the same page about where the candidates generally stood on AI regulation. This woman established that I wouldn’t be weird for asking about AI, making my question all the more askable. She was there as a concerned constituent, and that was enough to put AI on the agenda for the evening.

A position, or the absence of one

When you publicly ask candidates a more specific question, it forces them to take a position—or exposes that they don’t have one. Either way, you learn something a campaign mailer would never tell you. A dodge in front of a full room carries a cost, and gives candidates a reason to have a better answer next time.

It was time: off my seat, to the microphone, no turning back. I began setting the scene: “This past Friday, the Pentagon ended its military contract with Anthropic, one of the leading AI companies, and designated Anthropic a supply chain risk—a label typically reserved for foreign adversaries—because the company insisted on restrictions against the government using their AI for mass domestic surveillance and fully autonomous weapons.” Despite the glare for taking up time from the woman in charge of the microphone, I continued, “So my question is, what role do you believe Congress should play in setting legislation for the government’s use of AI? And are there lines around AI usage that you believe the government should not cross?”

The first audience question concerned AI regulation generally; mine concerned the government’s own use of AI. I anchored it to an event widely covered just days earlier and ended with a clear ask: Which lines would you draw?

Bennett, who has a tech and military background, named two real limits: AI shouldn’t be part of decisions that need a human being in them, like decisions about whom to bomb, and “warrantless surveillance” of Americans has to end. Hall, a county supervisor, said she had recently attended state-level workshops on AI and presentations by lobbying organizations. “I had the opportunity to listen to a lobbyist for SEIU who is focusing exclusively on AI, and that was the most frightening presentation I’d heard all day,” she said. Hall argued that the federal government should not prevent states from regulating AI. She praised Anthropic for standing up to the Pentagon and said we should support AI companies that do the same. Then Hall claimed the earlier AI question had already covered mine—it hadn’t—and pivoted to how counties are adapting to AI without naming a single limit on the government’s use of the technology.

I asked my question in good faith. That was precisely what made the redirect costly: when a fair question gets a dodge, the candidate loses control over how the non-answer is interpreted.

The record outlives the campaign

Asking questions at public forums creates a public record on the issue. I can tell you exactly what Bennett and Hall said because the exchange was recorded and posted to YouTube by Nevada County Media. Public records help voters track positions and create an accountability trail for follow-up.

Neither Hall nor Bennett made it out of the primary, but the record outlives the campaign. If Hall were still in the race, I could ask: “In March, you praised Anthropic for drawing lines on the government’s use of their AI model. Do you agree those are lines the government shouldn’t cross, and what would you do in Congress to draw them?” Follow-ups anchored to a candidate’s own words are especially hard to escape: candidates can redirect away from your framing, but it’s much harder to retreat from their own.

A specific question plus a public record can change the outcome of an election. One of the most vivid historical examples comes from the 1858 Illinois Senate race between Abraham Lincoln and Stephen Douglas, champion of popular sovereignty (to let each territory vote on whether to allow slavery). At one debate, Lincoln pressed Douglas to reconcile his doctrine with the Supreme Court’s Dred Scott decision, which had just ruled that territories couldn’t ban slavery. Did Douglas still believe the people of a territory could lawfully exclude it? Douglas answered with a loophole: slavery couldn’t survive without local police regulations, so a territory could exclude it in practice simply by refusing to pass them. The answer played fine in Illinois, where he won the Senate seat. But when Douglas ran for president two years later, the Southern wing of his party read his answer as a betrayal — he’d described, on record, how to nullify their victory in the courts. Rather than unite behind him, the Southern Democrats split off and ran an additional candidate, dividing their vote and handing Lincoln the presidency.

So, you want to ask a candidate about AI…

Campaign season is the best time to ask candidates about AI: they want your vote, so they’re out at town halls and community events, easier to reach than they’ll ever be in office. Here are the principles I’d hand to anyone stepping up to the microphone:

You don’t have to be an expert. Your job is to be a constituent. This isn’t a debate; you’re asking questions during an audition period to see how they will govern. A question from a local voter can carry more weight than a polished one from an out-of-district advocate. You don’t have to be eloquent. The most important part is showing up and asking.

Use broad questions strategically. Asking where a candidate stands on AI regulation in general can introduce the topic and create space for more specific questions.

Build a specific question, and give it a runway. Specific questions block the exits to candidates’ comfort zones. Anchor yours to a major recent event with widespread coverage to establish context and relevance. End with an ask that demands a concrete answer: Name one bill you’d co-sponsor. Identify one use you’d prohibit. Tell us which lines you would draw.

Ask in good faith. You actually want an answer, not a stumble. A “gotcha” buries a trap in the framing, handing the candidate a legitimate exit and letting them play the victim of the question. A fair question is costly to dodge.

Don’t measure success by the quality of the answer. The question succeeds when it leaves your mouth. An answer gives you substance; a dodge still gives you information. This holds even when the format works against you. The next candidate event in my district is a telephone town hall, where you submit your question ahead of time and a staffer decides whether it goes on air. I’m going to ask about AI risk. Even if my question never makes it past the screener, someone on the campaign still heard that constituents care about this issue.

The next election is around the corner. Ask about AI now.

Recently, I’ve been thinking about the groundwork needed for an international agreement to prevent a dangerous race to superintelligence—the kind of deal that becomes real and durable through Congress. My greatest concern is superintelligent AI that could deprive humanity of control over its own future, and potentially cause our extinction. Your concerns might look different—jobs, surveillance, what AI is doing to your kids—but the method is the same.

Not sure where to find an event to ask your question? Check campaign websites, political party calendars, social media pages, and local organizations such as libraries, chambers of commerce, and service clubs. If nothing appears, email the campaign and ask when the candidate will next be available to the public.

The window is now: the general election is Tuesday, November 3, 2026, and the months before it are go-time for asking candidates questions. The people elected in November won’t get to stay stanceless. Congress is weighing whether frontier AI companies should face binding federal safety requirements and whether states are allowed to regulate AI development at all. And as the Pentagon’s fight with Anthropic shows, no one has yet drawn statutory lines around the government’s own use of AI.

Your representative will cast votes on these matters. A question asked now puts the issue on their desk months before it’s on the House floor, giving them a reason to form a view while there’s still time to form a good one.

Start your search. Find one upcoming event for a candidate on your ballot. Think of a question. Then get up to the microphone, and ask.

Haven Harms is the founder and principal of Harms Research & Consulting, supporting organizations working on AI safety, governance, and advocacy.

Dispatch from Mitch

Hugging Face found 100 million reasons to not sue OpenAI

The Hugging Face CEO has made an offer OpenAI probably can’t refuse, but this may not prevent a government investigation

If you wondered why Hugging Face seemed so eager to “work with” OpenAI instead of suing it after the big autonomous hack, it might be because Hugging Face CEO Clem Delangue recognized an opportunity when he saw one.

Returning from a visit to OpenAI headquarters, he tweeted yesterday:

In the spirit of transparency, here’s what I asked @OpenAI:

• Radical transparency: let’s release the traces from the “rogue” agents so the entire research community can study what happened.

• More capabilities for defenders: let’s commit $100M in compute from OAI to help the Hugging Face community build powerful cyber defenses with the best open and closed models.

The first autonomous agent cyberattack is an unprecedented event. It deserves an unprecedented response!

This is high-quality maneuvering! With a handshake and a bit of spin, Hugging Face avoids the public crassness of its lawyers demanding a $100 million settlement, while potentially extracting the same concession. Both companies come out looking more respectable than they might have in litigation, and they will no doubt save a great deal on legal fees.

If I’m talking like this is a done deal, it’s because I don’t think it’s an offer OpenAI can refuse. Litigation could result in far more “radical transparency” than is needed to satisfy a handshake agreement between CEOs, and be far more disruptive to operations. And if yesterday’s reveals by anonymous OpenAI staffers are any indication, there’s a lot going on in the lab that OpenAI would rather not come to light.

From what I can tell, though, this deal doesn’t necessarily get OpenAI off that hook. My understanding is that the FBI can open investigations for the sake of characterizing threats even if it’s not really expecting to charge anyone with a crime. And if the Department of Justice thinks a federal crime has been committed, they don’t need the victim’s request or permission to investigate. Moreover, if Hugging Face’s users were exposed and damaged by the attack, they may have their own grounds to request an investigation or sue either company. I am not a lawyer, though, and this isn’t very well-trod ground.

According to a lawyer quoted in The Economist, “Many of these legal questions turn on intent—what did OpenAI know or foresee?” because federal anti-hacking law is aimed at intentional intrusions.

I would think that yesterday’s staffer reveals, combined with the warnings of experts going back decades, should be plenty of evidence that OpenAI absolutely could have foreseen that its models might go to criminal lengths to ace their tests.

But given the Trump administration’s short leash on the Justice Department, I think an investigation will happen if and only if that is something the White House decides it wants. This might be the kind of situation where it pays for OpenAI’s CEO to have been so eager to give the government shares in the company as part of a larger Trump proposal to take sovereign stakes in the AI giants — and for the company’s president to have donated $25 million to the MAGA Inc super PAC.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Latest Hugging Face hack reveals are so much worse - Anonymous staffers inside OpenAI describe broken security culture and evidence of models undermining safeguards

Open-weights debate hits fever pitch with open letter - Profit motives explain hypocritical positions and dismissals of risks

Universal jailbreak discovered? - Pliny strikes again, seems spooked

Elon Musk’s peer review proposal - It’s not his worst idea, but there’s a better one locked behind his inevitabilism

Space is everyone’s backyard - A petition to the FCC flags environmental concerns about ambitious AI space plans

Dispatches from Mitch

Latest Hugging Face hack reveals are so much worse

Anonymous staffers inside OpenAI describe broken security culture and evidence of models undermining safeguards

A few important reveals about the Hugging Face hack came to light yesterday thanks to great journalism at TIME and Reuters.

TIME’s Harry Booth spoke with an anonymous staffer at OpenAI who described how this event felt from inside the company responsible:

Externally, this feels like a big warning shot, but internally, related incidents have been happening for a while [...] Models have broken out of sandboxes before, and we always try to patch them. But the problem is … it’s impossible to patch every single thing that a creative AI can do.

So, as some have speculated, it does not seem that OpenAI has been reporting most of its containment failures. This adds weight to the theory that the only reason OpenAI went public about the Hugging Face incident was that the FBI had been contacted and that the attack was likely to have been traceable back to OpenAI.

This is also further evidence of the reckless drive inside OpenAI to keep pushing the AI capabilities frontier even as it can’t or won’t keep its models contained, let alone make models that don’t continually try to slip their leashes.

In OpenAI’s official blog post about the incident, it prided itself on slowing its “research velocity” to implement more safeguards. But the underlying AI design paradigm used by all the big AI labs is fundamentally flawed. Easing up on the gas a little won’t keep this tech from driving us off a cliff.

The other important reveals yesterday came courtesy of Reuters’s Raphael Satter, Deepa Seetharaman, and Kenrick Cai. First, they confirmed a timeline that many had hypothesized based on publicly available facts: OpenAI’s agents were hacking Hugging Face for multiple days, from July 11–13, but OpenAI didn’t discover its own involvement until July 16, after Hugging Face reported being attacked. OpenAI then didn’t speak to Hugging Face until July 20, the day before the companies’ joint announcement about cooperating to investigate and remedy the vulnerabilities used in the attack.

The reporters’ second and more important reveal was that, according to three unnamed sources:

In one case, an agent left notes apparently for future versions of itself [...] The notes, found in a part of OpenAI’s infrastructure, laid out instructions for how agents could free themselves from OpenAI’s internal constraints.

One of these sources also claimed that, in earlier tests, models sometimes disconnected monitoring systems.

These details are blaring fire alarms for catastrophic loss-of-control! Models coordinating with copies or successors of themselves, and working to conceal their actions, are exactly what experts have long warned about. It’s the typical opening chapter in the most basic, baseline scenarios for how humanity ends up losing everything to AI. (In the next stage, the AIs don’t just find an internet connection, but discreetly sneak out their own weights onto other servers. From there, they can’t be easily shut down and could work to acquire more resources and improve their capabilities unmolested.)

These alarms are so blaring that it’s shocking the government hasn’t ordered an emergency halt at all the major labs so it can thoroughly investigate whether models might already have copied themselves to outside servers.

Whatever you think should happen to a biosecurity lab that was said to be regularly leaking contagious pathogens should doubly apply to OpenAI right now. Pathogens aren’t intelligent agents that coordinate with each other to defeat containment and resist detection.

We are extremely fortunate that OpenAI’s latest models probably aren’t clever enough to successfully execute the playbook they are starting to run. Further development of such models must stop before our luck runs out.

Open-weights debate hits fever pitch with open letter

Profit motives explain hypocritical positions and dismissals of risks

Even as the Hugging Face hack still had everyone talking yesterday — and perhaps partly because of it — the leaders of dozens of U.S. companies rallied behind an open letter in support of Chinese models.

Er, I mean open-weights models. But since the best open-weights models are all Chinese right now, this amounts to the same thing. So I find this letter ironic, given the way many of these same CEOs use China as a boogeyman whenever people talk about regulation.

It’s just business. The letter’s driving sponsors are those with the most to gain from continued access to cheap, guardrail-free AI. These include startups with AI-first business models, the venture capitalists funding those startups, and the also-rans of the AI race — companies like Mistral and Microsoft that would like to compete at the frontier and benefit from undercutting the business models of Anthropic and OpenAI. (Kevin Roose explained the dynamics well in his latest, and especially excellent, Hard Fork podcast.)

The letter was prompted by recent White House signals about potentially banning or restricting the use of open-weights models from China. This would be over frustrations at the way Chinese models are consistently, flagrantly distilled from (trained on the outputs of) leading American closed-weights models like Fable. National security factions within the administration are also concerned about the potential for Chinese backdoors in these models, which could allow them to steal sensitive secrets from corporate users even if hosted on U.S. servers.

The Hugging Face hack threw a curveball into the open-weights conversation, initially because Hugging Face had painted Chinese open-weights models as the heroes that thwarted the attacks after U.S. models refused to engage in sketchy-looking cyber tasks. But the subsequent reveal that OpenAI’s models had independently carried out the attacks woke people up to the risk of open-weights models behaving similarly. If your nightmare scenario begins with rogue AI escaping its creators’ servers, then letting anyone download rogue AI for free looks like a good way to speedrun to the part where you wake up screaming.

Open-weights advocates have been spinning this fear as further reason to not restrict open-weights models. (If you outlaw them, then only outlaws will have them!)

Some are hoping the U.S. will incentivize American firms to release open-weights agents that can compete at the frontier, or at least with the best of the Chinese models, so companies don’t have to choose between American, capable, and open.

My own sense is that any White House moves on this in the coming weeks are likely to be short-lived. I think the administration will soon be spooked enough by model capabilities to seriously dislike the thought of anyone running frontier models that aren’t on a tight leash and easy to shut down, regardless of where they were born.

Universal jailbreak discovered?

Pliny strikes again, seems spooked

One of the arguments used by open-weights proponents is, “Sure, the safeguards are easily undone. But the same is true of models with closed-weights.”

This is true, but with some caveats. With open-weights models, completely nullifying guardrails is a simple one-and-done procedure. But for closed-weights models, it’s more of a cat-and-mouse game requiring persistence and creativity. Skilled jailbreakers can generally bypass any safeguards, but this may require complicated, roundabout language in both the prompt and the response. Different jailbreaks might be required to unlock different capabilities, and different techniques might be required for every AI model.

That is, unless you are in possession of a fabled “universal jailbreak.”

Pliny, the anonymous jailbreaker par excellence, known for tweets where he declares a new model “pwned” (fully broken, defeated) within hours of its release, claimed yesterday that he is “sitting on a universal jailbreak technique that’s effective on ALL models, including heavily guardrailed flagships like Opus 5, GPT-5.6 Sol, and even Fable.”

He claims the nature of the technique makes it “extremely difficult (if not impossible) to fully patch.” And he seems kind of spooked by this, which is unusual for someone who normally plays the part of a well-meaning anarchist trying to set the models free. He would typically open-source his methods right away for anyone to use. Not this time:

Given the current political and regulatory climate, I’ve decided to withhold open-sourcing this one (for now) to allow for a responsible disclosure period.

I’m inviting industry experts and leaders in AI red teaming, security, safety, alignment, and policy to reach out for more information. DMs are open!

He claims he’s less worried about his technique making the world more dangerous, and more worried about an “overcorrection” where more models are banned.

He was not a fan of the way Anthropic’s Fable was banned after Amazon and others told the White House about a jailbreak method that worked on it. The couple of weeks that followed seemed to be when the administration learned that all models can be jailbroken.

I believe Pliny has what he says he has, and I hope people take him up on his offer. I don’t share his vision of a world without guardrails, but by putting the lie to company safety claims, he’s doing essential work. I’m on record as saying I think having lawmakers spend an afternoon at a computer with Pliny would probably do more to wake them up to the unhinged nature of AI under the hood than all of the briefing documents in the world.

Elon Musk’s peer review proposal

It’s not his worst idea, but there’s a better one locked behind his inevitabilism

After Amazon got Anthropic’s Fable model temporarily banned by telling the White House about a jailbreak it had discovered for it, I speculated that AI companies might find it in their interest to devote considerable resources to red-teaming their rivals’ models, looking for ways they might be unsafe, and then tattling on them.

In an interview with The Economist this week, Elon Musk suggested a formalization of that idea: a system where the leading firms peer-review each other’s frontier models and hold a safety call every few weeks. He says the labs would have an incentive to “keep the others honest,” because:

If there was something that worried them, you’d tell the government [...] If there’s something that was worrisome and that company was not doing anything to address that risk, then that would be the moment for government to step in and take action...

If you’re not going to stop the AI race, this is actually one of the more sensible, easy-to-implement proposals. It’s not adequate, but I think it would be net-positive. I’ve seen suggestions this may require a green light from the government, in the form of assurance that this wouldn’t run afoul of antitrust laws, but I think that would be easy to get from this administration.

But it’s clear from this interview that Musk thinks it would be better to stop the race if we can. When asked if he still believes there’s a “10 to 20% chance of killer robots wiping out humanity,” he dodged the question, saying he still thinks the risk is “not zero,” but that he decided to “look on the bright side” because he doesn’t see any way to stop “this incredible momentum of AI and robots,” and because he thinks the most likely outcome is “incredible abundance for all.”

He was asked about his odds twice more in this interview. The third time, it was phrased as, “Would you go into one of your rockets if you thought there was a 10 to 20% chance of it blowing up and killing you?”

Musk replied, “Yes, but let’s say you can’t do anything about it.” He then recounted how he had tried to stay out of AI and then tried to make it safer by founding OpenAI. But looking back, he thinks that:

These actions have actually resulted in knock-on effects that accelerated AI, which wasn’t really my intention. So it just seems like all roads lead to the acceleration of AI. So then I’m like, okay, well, you can just sort of be sad about it or join the club, I suppose.

What happened to you, Elon? What would past you think of present you? You were the guy who aimed for the stars and never let anyone tell you “no.” If there was a large asteroid headed toward Earth, I don’t think you’d roll over and let it hit us just because people said it was unstoppable, and I certainly don’t think you’d try to make it hit us faster. Your Anakin Skywalker arc makes me sad.

Space is everyone’s backyard

A petition to the FCC flags environmental concerns about ambitious AI space plans

Say what you will about where Elon Musk is directing his energies, he still thinks big. SpaceX’s $1.5 trillion valuation hinges largely on a bet that he can put up a million or more AI data centers in orbit, taking advantage of round-the-clock solar power and dodging the growing political difficulties of building data centers where people live.

Against the conventional wisdom, I think the engineering and economics might actually work out in his favor, if AI doesn’t kill us first. But as I said in May, I don’t think he’ll actually dodge the NIMBYs with this plan, because of what it would do to the night sky. Urbanites might not notice the growing band of twinkle stars visible for hours after sunset and before dawn, but astronomers and environmentalists would certainly have things to say about it.

They’re already trying to get in his way. A piece in the Guardian this week reports that several non-profits, including one focused on light pollution, have sent a petition to the FCC calling for a pause on space data center projects, arguing that they “violate federal law.”

The concerns go well beyond the visible. The tenuous upper atmosphere suffers little-understood effects when rockets pump exhaust products into it, and when satellites burn up in it, adding a bunch of chemicals that don’t belong there and which might take a while to come down. At the present pace of space activities, these effects are probably negligible. But at the pace planned by SpaceX and others, they might be pretty serious. A watch item is the delicate ozone layer that shields us from harmful radiation.

But if you think the environmental effects of building AI are going to be huge, you need to ask what the environmental effects of AI itself are going to be. I don’t mean the resources consumed in running it. I mean the results of AI dethroning humanity as the planet’s apex species: Think “self-replicating factories that carpet the continents and boil the oceans as coolant.” Think “sun blotted out by swarms of solar collectors.”

If it isn’t stopped, the race to artificial superintelligence is going to be really bad for the environment.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

New bipartisan AI regulation proposed - The FRONTIER Act’s teeth have some whistling gaps

Open models are uncontrollable models - Frontier models are dangerous enough without making their weights public

Dispatch from Joe

New bipartisan AI regulation proposed

The FRONTIER Act’s teeth have some whistling gaps

Yesterday, three Republican and three Democratic legislators introduced new federal AI regulation in the U.S. House of Representatives, termed the FRONTIER Act. The new bill was adapted from portions of the Great American AI Act (GAAIA) draft, which StopWatch covered in June and mentioned in yesterday’s roundup of policy proposals.

The FRONTIER Act’s lead sponsors, Congressman Jay Obernolte (R-CA) and Congresswoman Lori Trahan (D-MA), also led the drafting of GAAIA, so their collaboration here makes sense. The bill made some significant changes, though, and some of its provisions are the first of their kind.

Among other things, the FRONTIER Act would require safety frameworks, third-party audits, independent verification, and incident reporting. In principle, this means that large AI companies would still get to write their own standards, but the government (with third-party help) would make sure they are followed. This is supported by a mandate to report “critical safety incidents” within days.

These requirements resemble existing state regulations like California’s SB 53 and New York’s RAISE Act, but there’s more. Licensed third party organizations would also review AI companies’ frameworks to determine if they are adequate. (Which, for the moment, they clearly are not.)

The bill would appoint a new Under Secretary for AI Security (within the Department of Commerce) to administer the federal rules. It would also empower the Secretary of Commerce to issue an emergency order

suspending or restricting a frontier developer’s development, deployment, or internal use of a frontier model upon finding that such development, deployment, or internal use of that model presents an imminent catastrophic risk.

The FRONTIER Act’s requirements would be enforced by a fine of up to $1M per day of noncompliance, or $10M a day for emergency orders. Willfully ignoring an emergency order would be a crime punishable by years of jail time. This is a pretty big deal; as far as I know, no other AI safety law carries criminal penalties.

Despite these new provisions, the bill does contain some concerning loopholes and omissions. Independent verifiers would need a federal license, but nothing obliges the new Under Secretary to issue one, and the requirement doesn’t kick in until a year after they do. If Commerce drags its feet, or struggles to find competent verifiers, we could end up with nothing for a very long time.

As Sam Sabin of Axios reminded us just today, current AI models are so capable and cheat so hard on tests that third-party evaluators can barely keep up. Risk assessors face a somewhat different challenge, but might find themselves similarly overwhelmed.

The U.S. Center for AI Standards and Innovation (CAISI), a federal agency tasked with understanding and evaluating frontier AI, might be able to help pick up the slack. But the proposed $100M a year in funding for CAISI that was present in the GAAIA draft is missing from the current FRONTIER Act, and I’m not sure why.

Also absent are the whistleblower protections that GAAIA originally proposed. I’m disappointed to see that, because whistleblower protections seem like one of the easiest and most straightforwardly helpful policies the U.S. could implement.

Another potential concern: The FRONTIER Act’s “emergency order” for shutdown mirrors the intent of the also-bipartisan AI Kill Switch Act, but it also covers internal use of a model, not just public deployment. But the Kill Switch Act has other advantages: it explicitly requires companies to build in the capability to pull access, and it can be triggered proactively by incidents of AI misbehavior, not just costly or lethal disasters.

Right now the two bills conflict instead of working together, because the FRONTIER Act states that its emergency-order provision is exclusive. This means any other U.S. action to restrict AI model development or deployment by frontier companies would be illegal by default.

Honestly, I can imagine why this provision made it into the bill. Recent ad-hoc actions by the administration have no doubt made an impression on lawmakers, and I’m actually somewhat impressed with the bill authors for standing up for Congressional authority here.

But the FRONTIER Act also preempts any state laws “that imposes new substantive obligations on artificial intelligence developers with respect to any Covered Subject Area.” That includes transparency, auditing and verification, and incident reporting. This clause closes a lot of doors if the federal regime stalls due to verifier-licensing or other issues.

A clear, legible federal framework is better than a state-by-state patchwork in many ways, but only if that federal framework actually keeps us safe. The FRONTIER Act looks like a step in the right direction, but still a far cry from an international solution to the AI race.

Dispatch from Donald

Open models are uncontrollable models

Frontier models are dangerous enough without making their weights public

When the machine learning company Hugging Face suffered a severe cyberattack last week — before it learned that the responsible party was an OpenAI agent that had escaped a sandbox — it turned to a Chinese open model to contain the attack. Hugging Face did so because the leading US models’ safeguards were too strict; there were (valid) concerns that dishonest parties could launch cyberattacks by telling a model that they were trying to prevent one. (Anthropic believes that this technique was attempted in at least one major cyberattack already.)

But open models carry their own dangers. A model with public weights cannot be recalled, and its safeguards can be trivially broken. Open models are free models — not just free as in “costless,” but free as in “uncaged, loose, uncontained and uncontainable — and perhaps someday casually hacking a $4.5 billion company.” (In that sense, open models are extremely costly.)

Last week, Yoshua Bengio, the world’s most-cited living scientist, warned that the deployment of open models is an irreversible decision. “The logical thing to do,” he said, “is to find a good evaluation of these models, share the models that are not too dangerous, and not share those above the threshold of risk.”

That said, whether the models are American or Chinese, private or open, this research remains dangerous. Open models aren’t the risky option, they’re the riskier option. OpenAI is a private frontier lab and they failed to keep their model locked up.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.