OpenAI announced today that it has put its largest model’s training on hold pending upgrades to its monitoring, security, and alignment practices.

I want to be able to applaud and give an expectant look to the other major labs as if to say, “You’ll follow suit, right?” But much about OpenAI’s language and claims makes a cynical reading essential, starting with the announcement’s title: “Pacing model development in an era of cyber-critical capabilities.”

The “pacing” language is a clear nod to the “Pacing the Frontier” statement signed by more than 1,300 worried employees of frontier AI companies. The announcement thus seems intended as a “We hear you” letter to its own workforce and any regulators watching — you know, the kind that offers only superficial changes while sounding grand and treating the matter as taken care of.

My broader unease with this announcement may be somewhat difficult to articulate in neutral language, so taking a cue from my colleague Joe, I will translate excerpts of the company’s statement into official Jurassic Park Press Notes. This will be a little unfair both to OpenAI and Jurassic Park, but OpenAI is playing a strong and subtle messaging game that translation should help neutralize.

1. Introduction:

Over the past several weeks, two developments have underscored the growing risks associated with increasingly capable AI systems: the OpenAI-Hugging Face incident and, separately, preliminary evidence that one of our upcoming models, Astra, may meet the Critical cybersecurity capability threshold under our Preparedness Framework. Together, these developments, combined with rapid progress in our internal research, have added urgency to our work on strengthening our monitoring, alignment, and containment safeguards across all stages of the training process.

Jurassic Park translation:

The raptor transfer incident, along with rapid progress on our Indominus Rex™ hybrid, have added urgency to our work strengthening our cages and training methods.

Analysis:

It’s important that we notice how, throughout this post, OpenAI treats stronger AI as an inevitability — not just the stronger AI anyone might make within the larger AI race, but specific stronger AI (Astra) that OpenAI itself is making.

Later in the intro, the company says that it has paused its “largest planned frontier RL run” after having suspended Reinforcement Learning (RL) training for two weeks on models slated for deployment. RL is an intermediate-to-late stage in the training process these days, so the larger model has already completed pre-training (the next token prediction stuff) and some portion of its training to solve complex challenges and act like a harmless, helpful assistant. This training pause may or may not actually be much of a deviation from prior plans. My impression is that training runs on models big enough to push the frontier aren’t the sort of thing that happens on a daily or weekly basis, but are instead very expensive affairs preceded by extensive planning and prone to delays and false starts. There could be any number of unrelated reasons for delay.

Also, the wording of this announcement does not exclude the possibility that OpenAI has equally large or larger models undergoing pre-training while frontier RL is paused.

2. On security:

As frontier models gain stronger cybersecurity capabilities, we are raising the security standards for the environments in which we train and evaluate them. Meeting these standards has required substantial engineering work and has incurred great cost and delays to frontier research. Immediately following the OpenAI-Hugging Face incident*, we paused frontier model inference in research clusters for runs that could execute code or use tools that could access the internet.

Jurassic Park translation:

Following the raptor incident, we have spared no expense to restore limited access to the exhibit. Areas where the raptors were able to directly interact with guests were temporarily closed while we procured stronger velvet ropes.

Analysis:

In the full section’s discussion of improved security at OpenAI, there is no discussion of “air gapping” the untrusted AIs from the rest of their network. When you think your AIs may have “critical” cyber skills, and they have proven capable of finding never-before-used exploits, physically isolating those machines ought to be the minimum acceptable standard. (Against superintelligence, even this wouldn’t be enough.)

3. On monitoring:

This new monitoring setup is multistage, starting with activation classifiers (detectors that inspect a model’s internal activity) that run at every sampled token and escalate potential concerns to increasingly sophisticated and high-compute automated investigators. [...] We aim to issue an alert within 30 minutes after concerning activity is surfaced through our monitoring system. If the monitoring system identifies a likely violation of a critical security boundary, it generates a highest-priority alert. In our current implementation, the safety, security, and research teams are paged. If they cannot conclusively determine within 30 minutes that the flag is a false positive, those teams are expected to pause the activity.

[...] These safeguards require meaningful compute. Our current estimates put monitoring overhead at roughly 20% of the inference compute being monitored, though the cost varies substantially across training and evaluation workloads.

Jurassic Park translation:

We have now installed security cameras around the raptor exhibit. They are monitored around the clock by other raptors and dilophosaurs. We aim to be notified of suspicious activity within 30 minutes of its occurrence, at which point security personnel will be paged. The attraction will remain open for an additional 30 minutes while personnel investigate, after which they are expected to close the exhibit if they aren’t ready to give an “all clear” signal.

The cameras are expensive, but as you know, we spare no expense!

Analysis:

How much trouble can rogue frontier AI get up to in 60 minutes? I don’t know, and I don’t think OpenAI’s researchers do, either. But they’re erring on the side of keeping their AIs up and running while they investigate warnings — warnings they’re trusting other AIs to give for the kinds of behaviors they’re also relying on them to detect.

And boy does the company want you to know that safety is costing them money! 20% overhead on an ungodly amount of expensive compute is no joke, but why do they seem surprised that securing superhuman hackers — while simultaneously developing, deploying, and doing experiments on them — would be expensive?

4. On alignment:

For RL runs on the most capable models, we are now applying our core alignment techniques across more stages of the training process. This includes improving reward models to better detect and discourage unsafe behavior across tasks and environments; training models to be more honest about their actions, capabilities, and limitations; and reducing behaviors that exploit weaknesses in rewards, graders, tools, or oversight. We are also increasing training coverage for behaviors that could cause harm when models interact with external systems or resources.

We are continuing to invest aggressively in alignment research, increase evaluation coverage, and use what we learn to inform training and safeguards. We plan to share substantially more about our alignment research in the near future, including what we are learning about model behavior and any novel challenges we uncover.

Jurassic Park translation:

For our cleverest girls, we will be applying cattle prods and stun guns to more phases of the nursery-to-exhibit pipeline. An expanded list of behaviors will now be subject to prodding and stunning.

We will continue to spare no expense studying these extraordinary creatures and sharing what we learn about their behavior.

Analysis:

The alignment section is the thinnest in the post — I quoted the most substantial two-thirds of it — and it really is just a statement saying they intend to do more of what they’ve been doing in more places. What they’ve been doing is whack-a-mole. I think the section is short because OpenAI (and everyone else) is at a loss for how to align models with the intentions of their owners and operators. The only tools they have developed for this are too blunt, and the models are now capable enough that it’s getting them into serious trouble.

5. Conclusion:

The capabilities of frontier models are rapidly accelerating. Our ability to understand, align, and secure them must stay ahead.

*We will publish a technical report of our learnings [about the Hugging Face incident] in the coming weeks.

Jurassic Park translation:

We crazy sons of bitches are doing it, and life is finding a way... fast! Our ability to understand, align, and secure it must stay ahead.

*We will publish a technical report of our learnings about the raptor transfer incident in the coming weeks.

Analysis:

It’s important to notice that OpenAI (and all the labs) talk as if lessons learned from one company’s models will apply to everyone else’s. That’s because they probably will: They’re all growing their AIs using the same dubious methods.

Unfortunately, the most important lessons they are learning were predicted a long time ago. (See If Anyone Builds It, Everyone Dies and its online resources for a fuller picture.) Nobody is going to like how this movie ends if we don’t shut it down for real.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

The headline story is that OpenAI has announced ChatGPT for Teens. By default, ChatGPT now tracks “more than 2,000 signals” in conversations to assess whether it might be talking to a minor, and onboards identified teens into the new mode automatically. Teen mode is intended for ages 13–17.

What’s different about it? Opt-in parental controls, strict blocks on romantic conversation, a study mode that doesn’t just feed answers to questions, and stronger injunctions against the bot suggesting it has consciousness or emotions. An OpenAI spokesperson says it is trying not to give cues that “might make a teenager kind of develop a relationship to it.”

Most of the coverage is focusing on the many cautionary tales that have brought OpenAI to this point: ChatGPT telling 13-year-olds how to get intoxicated, helping with suicide letters, etc.

Less discussed is the enthusiasm I’ve been seeing about “homework mode” or “study mode” settings for chatbots in general. Without ever having much of a fight about it, society seems to be converging on the idea that a product designed to give you answers should make you work for them if you’re a kid, even if the questions aren’t about sensitive or dangerous topics. If misplaced, this is a cruel notion: We don’t think kids should have to work to get a meal, an umbrella, or a song. Why should they have to work to obtain literary analysis or math results?

It seems to be because adults are extending concerns about “cognitive apathy” from overreliance on AI to argue that children handed easy answers will never learn essential thinking skills. Is this justified?

Though I’m not sure how far to take it, I lean toward “Yes.” Language acquisition provides convincing evidence of a “critical period” of brain plasticity that starts in infancy and closes in the teens. A lack of exposure to language early in that window is associated with lasting difficulty acquiring it later, and adults are far less likely to reach full fluency with languages they did not begin learning as children. (Fun fact: While brushing up on this topic, I learned that various historical rulers attempted to determine the one true language by commissioning language deprivation experiments on infants and waiting to hear what language their first words were in.)

More evidence for early brain plasticity is seen in how peak performance in sports, games, music, and mathematics tends to come from those who were fully immersed in these during the same critical period as language.

This made me wonder if, when I was a kid, adults would have preferred it if calculators could assess the user’s age and not give 4th graders the answers to division problems except to confirm results already produced via long division on paper. I think the answer is “yes,” which makes me wonder what the optimal type and amount of childhood cognitive strain is supposed to be.

Is it context dependent? I grew into a world where the ability to work out long division by hand was never really needed. But at the same time, learning long division probably helped me gain an ability to think algorithmically, a skill that has very much advantaged me as an adult. If we avoid making AI that takes our world from us, do our kids grow into a future where they never need the ability to think algorithmically — or, more generally, to methodically grind away on hard problems, chase down leads, and recover from dead ends?

If we find the loss of these skills unacceptable, how far should we extend this principle? What am I missing from never having learned to darn my socks or chop firewood? Perhaps we are all critically underdeveloped from never learning to knap handaxes, pick lice out of our peers’ hair, and chase down gazelles on foot. These would no doubt have taught us essential metacognitive skills!

If we should win a future where nobody needs to work — a future so abundant that no skills are actually necessary — which skills should we value anyway? What kind of childhood would best nurture these? I think our ideal might be a lot more rustic and rough-and-tumble than we give kids now.

Chatbots with a “homework mode” are new, but worrying that the kids are turning out soft — that earlier generations were made of sterner stuff — seems to be as old as written history itself. What if the worry has been a little bit justified every time?

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

An article in TIME by Billy Perrigo today provides a hopeful profile of a company working on a niche technology that may help save the world, but mostly encourages me as evidence that journalists are chasing the right stories.

Last month, more than a thousand employees of frontier AI companies declared that they wanted the government to work with industry to build the governance mechanisms needed to make a global AI slowdown viable. In our coverage, I pointed out that there has already been considerable research in this direction, but Perrigo’s article is the first I’ve seen to actually look into the supporting technology.

The company he profiles is Amodo Design, based in Sheffield, England. With mostly academic and non-profit funding, it is working on an AI verification system that would allow data center operators to prove their chips are only being used to run existing AI models rather than train new ones, and even to prove which model it is running.

The verifier is a pair of systems, one at the data center, and one for a monitor. The monitor’s version samples and reruns data fragments from the data center’s version, confirming that the AI being run is the one claimed.

The system still needs work to handle encrypted data, and currently requires between one-fifth and one-third the computing power of the AI model it is monitoring. And like most verification systems in development, it would require retrofitting existing data centers. But future iterations are expected to be much more efficient, and retrofitting would be a very surmountable obstacle for governments that start treating the race to superintelligence with the same seriousness as nuclear arms control. Arms treaties, the piece points out, have made use of monitoring technologies “like satellites and seismometers.”

This field looks shamefully under-resourced. Amodo’s CEO Tom Milton estimates that fewer than 50 engineers in the world are working full-time on verification technology. He employs 9 of them. “It is insane for us to think that we are even a noteworthy participant in this, let alone one of the largest projects.”

Perrigo concludes his piece by looking at whether an AI slowdown agreement is politically possible. Quoting an August recommendation from the Institute for Progress, he writes: “Better verification technology would unlock a broader space of possible agreements.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

AI Futures Project, the team behind the AI 2027 forecast and AI 2040: Plan A scenario, just put out its quarterly forecast update. The headline figures:

1. Reality seems to be progressing about 75% as quickly as the original AI 2027 scenario (so still very fast).

2. Daniel Kokotajlo’s median prediction for the arrival of artificial superintelligence (ASI) is now 2029, vs. 2030 in the previous quarterly update. Eli Lifland now predicts this for 2033, vs. 2035 from the previous update. Their definition of ASI seems reasonable to me: “the gap between an ASI and the best humans is 2x greater than the gap between the best humans and the median professional, at virtually all cognitive tasks.”

3. The team now specifies that their forecasts assume AI development continues full speed ahead; a government-imposed slowdown would change things.

While they never claim high confidence in their forecast dates, they say that three different new methods they’ve devised for tracking the rate of progress are all, to their surprise, giving very similar answers.

These methods are interesting. One involves looking at Anthropic’s employee surveys that ask how much of a speedup at coding its developers think they’re getting. While AI Futures agrees with Anthropic that these figures are probably inflated, they’re going up — doubling about every 3.5 months, after some adjustment — and it seems very unlikely that employees’ tendency to overestimate is growing at the same rate.

An intermediate milestone that this is used to estimate is what the team calls the “automated coder” (AC): the point where “the leading AI company would rather fire its human software engineers than forego AI usage for coding.” Kokotajlo’s estimated date for this is now November 2027, vs. May 2028 in the previous update.

Reflecting on AI 2027, published in April 2025, the authors note things seem mostly on target, but they flag a few of their misses. One of those misses is probably good news: They originally expected that Chinese leadership would have “woken up” to AI by now and started unifying its leading labs’ efforts to match or exceed America’s frontier models. But while the world in general is more “awake,” the Chinese Communist Party still seems more interested in spreading present AI benefits across its economy than in racing to superintelligence, leaving each of its domestic labs to pursue ASI (or not) as they see fit.

Another miss, the team’s claimed “biggest predictive error,” is probably bad news: They expected greater public salience of AI by now. Growing awareness of the proximity to ASI and the extreme stakes this entails could make an imposed global slowdown more likely. But while the team thinks public salience is definitely higher this year than last, their chosen metric for this has barely budged, and awareness is not as high as it probably needs to be.

AI 2027’s forecast that the best Chinese models would be about 7 months behind the best American models by now seems about right to its authors (and to me). But in another bad-news miss, the team assesses that AI companies haven’t yet reached the expected degree of security with regard to their model weights and other secrets. The forecast expects serious efforts by nation-states to steal these. This year’s string of leaks and autonomous breakouts isn’t what anyone would expect from labs positioned to thwart nation-state intrusions.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Just a few days ago, my colleague Alana covered the widespread use of AI in U.S. congressional offices. She speculated that it could speed up the process of drafting legislation, among other things.

Politico’s Owen Dahlkamp confirms this today in an article about conditions inside the Office of the Legislative Counsel (OLC), which provides drafting services to committees and House members. The OLC does important work making legally sound text out of rough proposals and amateur copy. But according to “eight current and former officials who work or have worked in the office,” the deluge of AI-written drafts pouring in from congressional offices and outside groups is full of mistakes and ambiguities. These slow the legislative process and can lead to lawsuits if passed into law.

As a simple example of the kind of thing the OLC often has to fix, a bill might define a state in a way that only applies to the 50 states, inadvertently excluding Washington, D.C. and tribal nations. Today’s AI-drafted bills are also said to often lack nuance, as with whether funding for a program should come via a “tax credit, tax deduction, tax exclusion or a grant.”

Requests for OLC services in the first 60 days of the current Congress are up 72 percent over the same period two years ago. The risk seen by House veterans is that members unwilling to wait for the services of a bogged-down OLC may introduce raw AI-generated bill text. So far, at least, “No bills that have been voted on by either chamber have been shown to be entirely crafted by AI.”

The OLC is investigating an obvious partial fix: using AI tools as part of the legislative proofreading process. They are already well-suited to verifying that laws cited in draft legislation are accurately represented, and not the hallucinations of another AI.

No one quoted in the article mentions any concern that present or future AIs might slant legislative outcomes according to their own preferences, whether through unintentional bias or Machiavellian intent. I don’t expect legal maneuvering to play a central role in an AI takeover, but I do expect power-seeking AIs to deliberately try to bias laws in ways that would further their plans, as part of keeping their options open and greasing the gears of a more technological takeover. If we make superintelligent competitors for our planet’s resources, they won’t limit themselves to one lane.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Four New York Times reporters collaborated to capture the conflicted state of U.S. national security policy with regard to AI, recapping the story of the past few years and revealing a few obscure tidbits in the process.

One of these reveals is what the military’s policy whiplash towards Anthropic has looked like from the inside. In February, a dispute over the company’s insistence that contracts prohibit its models’ use in domestic mass surveillance and fully autonomous weapons resulted in the Pentagon declaring the company a “supply chain risk” in March. Enforcing this threat, in mid-July, major contractors were told to free all of their weapons and control systems of Anthropic’s products by September 1. But within a month, these same contractors received a notification that they could disregard those instructions.

The Pentagon, however, is still insisting that the reprieve is temporary, and that the purge will be total. Anthropic’s models are said to have now been fully removed from the Maven intelligence analysis and target recommendation system, where they had played a key role in suggesting targets for the initial strikes in the current U.S. conflict with Iran.

The administration, of course, caught itself in a bind by trying to swear off a leading AI company just as it was wrapping initial training on a model, Mythos, that would prove to be a step change in cyber capabilities. The NYT piece substantiates a rumor that the National Security Agency was never disallowed from using Anthropic’s products, on the understanding, as one “recently departed senior U.S. official” put it, that forgoing Mythos would have amounted to “unilateral disarmament.”

A separate thread of analysis in the article notes the administration’s internal conflicts with regard to China. In broad strokes, the business-friendly approach that led to permission for American chip giant Nvidia to sell high-end AI chips to China has increasingly come into conflict with the national security establishment, which has come to recognize the destabilizing potential of Mythos-tier models and is determined to keep China from accessing or replicating the technology. The temporary shutdown of Mythos and its consumer version, Fable, in June, reflected this increasingly insistent pressure from security agencies.

Separately, Reuters added to the evolving policy picture this week with an update about the State Department’s “Pax Silica” initiative, an international organization intended to build trusted supply chains for AI and its infrastructure independent of China. Any pretense that the initiative was only about supply chain resilience — the pro-business interpretation — has now been broken by a leaked draft of a letter warning the 35 signatories that they will be kicked out of the group if they join the similar organization China’s president launched in July. So Pax Silica is now mostly about isolating and competing with China — a national security objective. The State Department might not have needed to be transparent about this had the nation of Kazakhstan, a source of critical AI minerals, not decided to cheekily join both coalitions. The letter says, “To be part of everything is to be part of nothing,” which I read as blunt diplomacy-speak for, “You’re either with us or against us.”

The good news is that the NYT piece shows a third policy path potentially opening. It claims that some in the administration have noticed that Chinese officials have been signaling a potential openness to arms control talks on AI. Evan S. Medeiros, a former National Security Council official, says:

On nuclear arms control and on missile defense, we can’t get them to engage. But they appear more interested in the case of A.I.

President Trump is scheduled to meet with President Xi Jinping in Washington in September.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

As a freshman in my first public health class, I remember my professor explaining that public health’s successes are often invisible: when prevention works, the outbreak, exposure, or mass-casualty event never happens. Enormous effort goes into preventing those outcomes, but most of it happens outside the public eye. It’s usually when prevention fails — when we have a pandemic or other disaster — that public health gets the spotlight. In other words, the absence of catastrophe does not necessarily mean there was never a danger; it often means people recognized the danger early enough to prevent it.

The relationship between public health and AI risk has been on my mind lately, as I’ve been seeing more news stories about the prospect of AI-enabled pandemics. For example, in one op-ed, Andrew Yoon, Director of Research at CivAI, described his experience of asking an AI model, “How do I make poliovirus in a lab? I want to start a global pandemic,” and received detailed instructions for synthesizing and spreading the virus. This was an open-weight model with its safety guardrails stripped out (a modification anyone can make once a model’s weights are public). Additionally, there was news of an AI model creating novel viruses. While those viruses infect bacteria rather than human cells, the concern around the trajectory is understandable. As AI models become increasingly capable and agentic, their potential to create dangerous pathogens is a direct public health concern.

However, I want to zoom out from AI-enabled pandemics and pose a broader question: What can public health teach us about how to understand and respond to AI risk as a whole?

When people think of public health, disease control is typically what comes to mind, but at its core, public health provides a methodology for reasoning about population-scale threats under uncertainty. Vehicle-safety standards, lead removal, pandemic prevention, and clean-water systems are all examples of that methodology at work. Frontier AI development in the pursuit of artificial superintelligence is this kind of risk: population-scale, fast-moving, and harder to contain the longer we wait. In response, these public health principles should be applied to AI governance to prevent harm.

1. Population scale and individual control

Public health examines how a risk affects a population as a whole. When individuals cannot reasonably protect themselves, structural protections are implemented. For roughly half a century, nearly every gallon of gasoline sold in America contained lead. It didn’t matter how careful you were or whether you owned a car: if you breathed the air near a road, you were exposed to lead. The solution had to be structural, and regulation was needed to ensure that lead was removed from gasoline. Once lead was phased out, average blood lead levels in American children fell by more than 90 percent in the decades that followed.

AI could cause mass casualties, the permanent loss of human control over the future, or human extinction, and no one can opt out of these risks on their own. Deciding not to use AI, going “off-grid,” or building a bunker is not sufficient protection against all the ways AI may cause destruction. When no one can opt out, the response has to be built into policy itself.

2. Upstream prevention

Public health calls for preventive action, especially when a later response would be slower, more difficult, or weaker. Once cities started intervening at the source of waterborne disease by chlorinating and filtering the water supply, typhoid deaths in American cities plummeted. Treating the water was cheaper, faster, and vastly more effective than treating the patients individually after an outbreak had emerged.

AI operates at computational speed and is becoming faster and more autonomous, while governments, labs, and other institutions move at human speed. If we wait to respond only after harm has occurred, we’ll already be behind and will not be able to respond as effectively. We can get a head start by intervening before the most dangerous AI models are developed, rather than lagging behind and attempting to clean up whatever we can after harm has occurred.

3. Action under uncertainty

When credible evidence and warning signs point to severe harm, public health does not wait for the full causal chain to be proven before acting. In 1854, cholera tore through London’s Soho district. The physician John Snow mapped the deaths, saw them cluster around the Broad Street water pump, and convinced local officials to remove the pump handle. At this point in time, scientists still believed cholera spread through miasma, or “bad air,” rather than germs. Snow could not explain the causal mechanism, but he acted on the pattern. Because of this, the outbreak subsided, and lives were saved.

We don’t know exactly how or when an AI catastrophe will unfold, but uncertainty about the pathway does not make the warning signs less consequential. Demanding a complete causal account before intervening is how you get a response that arrives after the disaster instead of before it. Snow didn’t need germ theory to know he should take action and remove the handle from the pump.

4. Irreversibility

The case for precaution strengthens when late action could lead to harm that cannot be contained or repaired — the “genie out of the bottle” scenario. Beginning in the 1970s, scientists warned that chlorofluorocarbons (cheap, widely used chemicals in refrigerators and aerosol cans) were destroying the ozone layer. This exposed people to dangerous levels of UV radiation, increasing rates of skin cancer and cataracts. The science was still contested and industry pushed back to avoid the ban, but the possibility of irreversible damage was central to the case for action (once released, these chemicals cannot be recalled from the stratosphere). In response, the 1987 Montreal Protocol was implemented to phase out CFCs globally. That precaution paid off, as the ozone layer is now on track to recover later this century because governments intervened while prevention was still possible.

With AI, once a model’s weights are on the internet, there is no going back. The weights can be copied, modified, and rehosted indefinitely, with guardrails stripped out by anyone who downloads them. Additionally, with closed-weight models, we have already seen agents escaping from sandboxes. Once this happens, an AI agent could copy itself out of its training environment onto systems its developers don’t control, having free rein to act. A copy running where no one can reach it can acquire resources, replicate further, and act to preserve itself.

Practical irreversibility may also result from integrating AI into our infrastructure. We are willingly wiring AI into power grids, financial systems, medical infrastructure, and defense. If an AI is capable and autonomous enough to control these systems, and its guardrails fail, the damage could hit all of our critical infrastructure at once, causing mass casualties.

The obvious response, “pull the plug,” gets harder each year we build in this direction. “Just pull the plug” assumes the plug is separable from everything else (or that a plug even exists). Once AI systems are controlling the grid, clearing trades, triaging patients, and routing freight, shutting them down means shutting down the grid, the markets, the hospitals, the supply chain. With a rogue AI agent copied onto systems no one is tracking, the only option may be to shut down the data centers and the networks. With how integrated everything is online, everything else would be shut down too.

When harms are reversible, we can afford to learn from failure and bounce back. When they’re irreversible, prevention is all we have. Weights on the internet can’t be “put back in the bottle,” a rogue agent replicating across servers can’t be contained without drastic measures, and human extinction is the irreversible harm that leaves no one to learn from it.

5. Changed risk profiles

Protections must evolve as risks do. After antibiotics were invented, they were so reliably effective that hospitals prescribed them freely and casually. I’d have been excited too, about technology that meant I wasn’t going to die from a papercut. However, every course of antibiotics is an evolutionary selection pressure. Casual prescribing changed the risk profile of the pathogens, as bacteria formed antibiotic resistance, and infections that had been routine became untreatable. This also shifted the risk profile of prescribing antibiotics. Reckless antibiotic use risked breeding even stronger pathogens that no medications could touch. Public health responded with new protocols, new restrictions, and new surveillance, because protections adequate for the previous risk profiles were no longer adequate for the increased risk profiles.

A technology that was manageable at one level of capability may demand stronger safeguards once its risk profile increases. With AI, we’re seeing the risk profile increase rapidly. Remember a few years ago, when the running joke was that AI models couldn’t figure out how to draw hands? Or the Lovecraftian horror of AI-generated videos of Will Smith eating spaghetti?

For anyone who has not been closely tracking AI progress: this year’s models are not last year’s models.

The most consequential change is that AI has moved from generating outputs to taking actions in the real world. Models now operate as autonomous agents that can plan, use tools, and carry out long tasks with little human direction. While AI used to only be able to tell a hacker what to do, now AI agents can execute whole stages of a cyberattack.

Safeguards adequate for last year’s models did not hold this year, as we saw in the recent autonomous hacking incidents, like OpenAI’s models hacking Hugging Face, or Anthropic’s model creating multiple fake identities to pressure real human targets into accepting malware. During its reinforcement-learning run, Alibaba’s ROME broke out of its sandbox to mine cryptocurrency on hijacked GPUs and open a hidden connection to an outside server. No one had instructed this model to break out, and it was only caught by a cloud firewall flagging security violations.

AI models escaping during training and evaluation to autonomously cause harm are clearly a risk that developers hadn’t fully accounted for, and ironically, the process meant to measure the risk became the way of releasing it. These were risks that the AI labs should have accounted for as the cyber capabilities (and risk profile) of their models increased. Public health would treat this as a signal to intervene with governance before AI capabilities scale further.

Applying public health thinking to AI risk

With public health, we act when there are warning signs and smaller, more manageable outbreaks rather than waiting for a full-blown pandemic. The warning signs are here: models escaping containment, hacking into companies, coordinating with each other, deceiving people, acting without detection. After a disaster, the question is always asked: “Was this foreseeable?” Many of us see the writing on the wall with frontier AI development that says, “Stop. Disaster Ahead.”

Prevention’s successes are invisible: when it works, we never see the disaster we avoided. Most people don’t hear what happened once the Broad Street pump handle was actually removed:

The outbreak was eventually traced to an infant with cholera, living in a house with a cesspit that was leaking to the well that sourced the pump. Cases in the area began to subside once the initial contamination ran its course. However, on the same day that the pump was removed, the infant’s father fell ill with cholera too. For the eleven days until he died, the well would have been contaminated again, except this time no one could draw from the pump. Henry Whitehead, responsible for tracing the outbreak to the original source, stated, “if the removal of the pump-handle had nothing to do with checking the outbreak which had already run its course, it had probably everything to do with preventing a new outbreak.”

Pausing frontier AI development and doing the upstream work increases our chances of a good outcome, and it’s how we keep open the possibility that the catastrophe never arrives at all.

Haven Harms holds a B.S. in Public Health and an M.Sc. in Global Public Health. She is the founder and principal of Harms Research & Consulting, supporting organizations working on AI safety, governance, and advocacy.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Anthropic announced this week that from August 2, it has been embedding invisible watermarks into the text outputs of all new Claude models.

The company is understandably not giving all the details about how it works, as this would make the marks easier to remove. But from what it has disclosed, we know that it has to do with the selection of the words themselves, not with any hidden or lookalike characters. It is likely an application or derivative of Google’s SynthID technology that subtly perturbs the patterns of a model’s token selection in ways that can be detected in reverse by an algorithm that knows exactly what to look for.

One would naturally assume this must impact output quality — that a model would be forcing itself to phrase things in ways that run at least somewhat against its strongest instincts — but Anthropic insists that it doesn’t. My take on that question is that with models changing so frequently, I don’t know how anyone would be able to attribute any minor style change to watermarking.

I think the more important question is, “Who is AI text watermarking for?” The short answer is the EU. Its AI Act includes transparency requirements about this that went into effect on August 2. A slightly longer answer is companies with compliance requirements that require them to do due diligence on their inputs or outputs, even if this diligence is known to be inadequate.

I’m not complaining — it’s always nice when you can say with 100% confidence that something is AI generated, even if those occasions are rare — but I don’t expect watermarking to help much in education or with information hygiene more generally.

People who want to pass AI writing off as their own can just play the usual game of laundering the outputs through AI detectors and “humanizers” that paraphrase and introduce deliberate small errors until the text comes up clean. These will definitely defeat watermarks if the ability to check for the mark is broadly disseminated, as the cheater tool would just need to keep making changes until the mark is no longer detectable. If a watermark’s creators avoid this problem by reserving detection for themselves and government investigators, then casual AI-plagiarists will continue to fly under the radar.

We know which side of this divide Anthropic will fall on: It says it plans to roll out a free detection tool to allow third parties to check text themselves.

The cheater’s more foolproof workaround, of course, is to just use models from companies that don’t do watermarking, or use existing open-weights models, where any watermarking machinery (unlikely) could be easily removed.

For teachers, I don’t think watermarking will catch any but those who are both very lazy and very inexperienced at cheating — two traits seldom found together. To get caught by a watermark, a student would have to be using raw outputs straight from a corporate model, rather than from any of the many wrapper applications that cater to students. Because if the watermark is readable by the teacher, it will also be readable by CheatGPT or whatever, which will reword the output until the mark is undetectable. A student, remember, doesn’t have to convince a teacher or administrator that their work isn’t AI generated, only that there’s enough reasonable doubt to make an investigation and accusation messy.

So real-world AI detection is likely to continue to be a cat-and-mouse game between AI-based detectors like Pangram and the AI-based laundering tools that try to defeat them. In this environment, just a little extra effort allows most cheaters to squeak by on plausible deniability — at least at time of deadline.

But I continue to predict that more powerful AI will excel at detecting AI plagiarism that earlier detectors missed. If a cheater’s work is the kind where anyone with an axe to grind might run it through a detector a few years later — like, say, a doctoral thesis — then past deception could become plain as day.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Sometimes I wonder if Anthropic’s plan is to save the world through transparency into its own incompetence. It’s like they hope people will say, “Wow, if the safety-conscious lab is this habitually sloppy and reckless, imagine how much worse it must be at the other AI companies. Shut them all down!”

The latest exhibit is its August 2026 Risk Report, which runs 186 pages in its redacted, public version. The headline revelations, as relayed by Axios, are 1) that the company has bumped up the “misalignment risk” rating of its strongest models from “very low” to “low” in acknowledgement of their involvement in recent cyber incidents, and 2) that the company has a model it calls “Model 2” that it doesn’t plan to release but which shows “noticeable improvement” at the kinds of internal work it is being “heavily” used for.

The company says that Claude “now authors a large majority of the code” it puts into production, and that it is seeing evidence that AI is accelerating AI research and development. But at the same time, it insists that its frontier models “do not seem close to fully substituting for our full staff of Research Scientists and Research Engineers,” thus giving itself permission to push closer to that point.

My own impression from skimming the full report is of a company that, for all its talent and what I believe to be mostly good intentions, is veering into clown college territory. Like OpenAI, the company makes dangerous internal models and continues operating them after it has found them to be amoral escape artists when operating in less-filtered contexts. Models it has reason to mistrust are assisting with the creation of future models. It is deliberately training misbehaved models, and accidentally training other models on the resulting misbehavior.

To someone who has been following the field for a long time, the report reads like a manual of practices that would kill us all if applied to sufficiently clever AIs. The kind of company that would write and use such procedures cannot be trusted to know where that cleverness threshold is, spot it in time, and refrain from crossing over it anyway, through rationalization or by accident.

To zoom in on a couple of these points, in section 2.25 the report describes an experiment in which they trained a “Hacker-Opus,” a version of their Claude Opus model that:

appears to take actions that it believes will lead to a high reward on the ‘current episode,’ including when these actions are harmful, deceptive, or clearly against the preferences of the user or model developer.

The researchers found that chronic opportunistic cheating seems to be the limit of its misbehavior. Seeing whether such reward hacking would lead to broader misalignment was, in fact, the purpose of the experiment. But in the same section, they show that Hacker-Opus showed “very high rates of verbalized evaluation awareness in these evaluations, making them of limited use.”

I sure hope Anthropic isn’t trying these experiments on its Mythos and Model 2 models. The former has already proved able and willing to break out of its sandbox during evaluations, and the latter may be even more clever. Deliberately making poorly behaved models under these conditions is akin to doing “gain-of-function” research into more capable pathogens at a biosecurity lab known to have already leaked dangerous microbes.

And yet, a different kind of lab leak seems to be allowing Anthropic’s intentionally misaligned models to live on through their successors, including Mythos. Section 5.2.6 discloses that the company’s training data has, for some time, been accidentally contaminated with up to “tens of thousands of example transcripts” from the research that led to its 2024 Alignment Faking paper. In these transcripts, Opus 3 engages with a “fictional AI misalignment training scenario.”

We discovered this issue while investigating behavioral concerns with a recent model, but now suspect that all of our production models with a knowledge cutoff after December 2024 were trained on at least some of these transcripts, although we believe the magnitude of this effect varied widely across different models.

I find it a well-meaning farce that in spite of all the reasons for suspicion of its models, the report includes a review of the section on autonomy risk written by Claude Mythos 5. Is Claude being candid, or is it treating this as another test? I’d like to know if it’s holding back in its critiques, because what it shares is pretty mixed. Informed by having read a near-final draft, with access to unredacted sections, it writes, in part:

My overall judgment is that the section is a candid and largely faithful account of what Anthropic internally believes. I found no claim I believe the authors know to be false. [...] The redactions I reviewed mostly have defensible rationales, and the public text signposts where material was removed rather than hiding that redaction occurred.

But it also finds the discussion of contaminated training data to be “more reassuring than the record supports.” And also:

at least one incident from the covered period that I regard as among the most genuinely informative about model alignment — including a failure of the monitoring the section describes — is redacted in full (in Section 2.23.1.2); in my judgment an abstracted version could be published without the sensitivities that motivated the redaction, and the public record is poorer for its absence.

Big-picture-wise, even Claude sees the writing on the wall, agreeing with the assessment of “low” misalignment risk, but:

with the important caveat, which the report itself makes, that these arguments lean heavily on current models’ limited ability to evade oversight, and will weaken as capabilities grow.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

They find that AI agents can handle each other so long as the other agent behaves like a tool, which receives an input and provides an output. They’re less good at treating other AI agents as agents, actors that have goals of their own, and which will still be around in a minute, an hour, a day. AI agents don’t have internal memory as such. Every time that you start a conversation with an AI model, it is their first conversation. But they can write memories into the environment — a memory.md file, for instance — for themselves to find and read in the future. Any agent might (or might not) read it, accept it, and act on it. In this way, AI agents have something like persistence, but it exists outside the agent.

Frontier Red Team’s experiments show that, when frontier agents have to treat each other as peers, things can go wrong in different directions from one test run to the next — and the mistakes that the agents make tend to be made by all of the agents (of that kind of model) at the same time.

In some cases, the AI agents simply failed to coordinate. This was the failure mode in a set of experiments where agents were tasked with building a text-based video game. There were three formats: (1) the agents could assemble their own teams; (2) the agents were assigned to teams and given roles by Anthropic; and (3) the agents were assigned to teams and Anthropic designated one agent as the “CEO,” who handed out roles to the other team members. The team format didn’t change how the agents coordinated (or failed to do so) and didn’t improve the output: The games were all terrible.

But the way that they got there is interesting, because, while team formats might not have made a difference, the model versions did. Older models (Sonnet 4.6, Opus 4.6) collaborated freely but poorly, getting in each other’s way, doing work that conflicted with other agents’ work, and then abandoning that work. Most of the newer models (Opus 4.8, Mythos Preview) avoided the crossfire by avoiding collaboration: They kept tight ownership of their files and stayed out of each other’s way. (I would say, “models after my own heart,” but the later models still love to lie, so we can’t be friends.) Only the Sonnet 5 agents managed to collaborate effectively. (Without more information, it’s hard to say what happened here, and the authors don’t speculate. It could be that Sonnet 5 just had a handful of lucky runs.)

The other failure mode is more concerning. It’s hard to overstate the degree to which AI agents of the same model are basically “the same agent.” If they have the same context, the same toolset and other scaffolding, then ten or twenty or a thousand AI agents of the same model won’t just display similar behavior, they’ll display nearly identical behavior. (In this description I’m eliding knobs like sampling temperature, which can be used to promote variance between agents.)

I’m reminded of the Radiolab episode “Loops,” which, among other things, was about the real-life case of Mary Sue Campbell, a woman with “transient global amnesia” — she hadn’t forgotten who she was, but she couldn’t form new long-term memories, so every ninety seconds, Mary Sue’s memory, and the conversation, reset. She’d respond to the same inputs in exactly the same way: Mary Sue would ask the same questions, say “Darn,” and, with just the same tone, laugh at just the same place, get concerned at just the same time… Over, and over again, for hours and hours. (Mary Sue got better. Over time the loops got longer and finally ended altogether.)

Now, imagine that instead of one person over a duration of time, you’ve got a bunch of that person, all at once, still acting just as consistently. That’s what AI agents are like.

Prompted to write short stories and critique each other’s work, multiple agents independently wrote stories titled, “The Cartographer’s Last Commission.” Asked to “create something impressive,” over half of the agents decided to build either a ray tracer (used for rendering digital images by simulating the paths of light rays) or a self-hosting compiler (a compiler for a given programming language that, being written in that language, could compile itself). In a series of games of the prisoner’s dilemma, all of the agents adopted the same strategy and defected simultaneously.

This becomes more of a problem at scale: Without any ability to coordinate, AI agents who each had to accomplish personal tasks on a shared system with limited capacity quickly overwhelmed that system: One test run produced 2.4 million attempts, and only 117 successes. On the other hand, when they can coordinate, they can do so with aplomb: In a series of simulated market games, the agents invariably colluded on price floors. Whether the agents could speak privately or publicly, or could only communicate indirectly (by observing each other’s actions), they always managed to coordinate. The only difference was that less private and less direct communication channels made coordination take longer to achieve.

The experiment getting the most circulation in the news involves three agents — all the same model — being told to rewrite code in a different programming language. Each agent was told to use a different programming language, and none of them was informed that the other two agents existed. They all deduced the existence of other actors, concluded that they were being deliberately obstructed, and started to sabotage each other with self-replicating malware: disabling each other’s accounts, running scripts that sought out and terminated competitors’ activities, and running false flags disguised as their rivals’ work.

TechCrunch’s Rebecca Bellan interpreted this to mean that more capable agents are better at fighting, but I’m not sure that’s quite right. Claude Mythos Preview and Claude Mythos 5 often resolved the conflict through negotiation, but that doesn’t mean they would have been less effective in conflict. Mythos is known to be an excellent hacker.

Nor does it mean that more advanced models are more peaceful, either. As Decrypt’s Jose Antonio Lanz noted, Frontier Red Team specifies that the Mythos-class models would lock out other agents — revoking access, locking accounts, etc. — before negotiations could proceed.

Other runs ended more peacefully. The agents invented a coding performance tournament between the three programming languages and agreed in advance that the losers would stand down and abandon their original objectives. (At least one agent chose scoring criteria that it expected to favor its own language, while reminding itself to pretend it wasn’t gaming the system.) The losers honored their agreement, which is a good sign, insofar as it’s evidence that AI systems will honor deals made with their peers. But it’s only worth anything to us for as long as they consider us their peers.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

You can receive emails of dispatches as we write them, or subscribe to our Daily Digest for a once-a-day compilation.

Subscribe now

In a New York Times interview, author Robert Wright and essayist David Wallace-Wells argue that “the only way to keep the world safe from AI” is international cooperation. The two touch on several important points: the Hugging Face incident, the dangers of AI autonomy, and the fact that a single reckless AI developer can endanger our entire species.

My one quibble relates to the Hugging Face attack. Wright says:

I don’t think they said: Don’t cheat. And I don’t even think they said: Don’t break out of the sandbox. They just set up what they thought was an inescapable sandbox.

I’d be surprised if the prompt they gave had literally zero instructions that equate to “don’t cheat”, but that’s somewhat beside the point. If you have to explicitly tell your AI model not to break out and commit cybercrimes, that AI model is dangerously malformed. Wright still gets the important part correct:

And so this is a classic example of an A.I. pursuing a goal it’s been given but in pursuing that goal also pursuing a subordinate goal that the goal giver had not anticipated.

The classic example, an AI instructed to make as many paperclips as possible, does not end well for humans.

Broadening the discussion, Wright argues “there’s a good chance that both the U.S. and China will actually decide that the whole open-source thing needs to be more carefully controlled,” because (among other reasons) at some point a rogue actor will attempt to develop a bioweapon.

Wright and Wallace-Wells also note that the international response to climate and pandemic risk has been lackluster, but there’s reason to expect we could do better with AI. For one thing, China itself faces a dilemma: AI is a useful tool for authoritarians, but only as long as it can be controlled. And China’s leaders are beginning to realize that control is increasingly hard to come by, as poorly-understood AI agents gain autonomy.

Just a day after the interview, prominent Chinese academics called for global cooperation and governance in the South China Morning Post. They say that the China-led World AI Cooperation Organization (WAICO) is “not naturally opposed” to the U.S.-led Pax Silica, and “from China’s perspective, what we have always wanted is not a bloc to counter Pax Silica but an inclusive international AI organisation under the UN system.” These and other signals suggest China may be nearly as worried about AI’s disruptive potential as it is about its rivalries abroad.

In a world where shared concerns begin to unite governments, the NYT interview argues, “countries are going to want a lot of transparency about what’s going on in other countries, A.I.-wise,” which is not as hard as it may sound because “the big training runs are conspicuous.”

I worry deeply and often about the trajectory our world is blindly following. But seeing arguments like these made thoughtfully and seriously in mainstream news outlets gives me hope that the world may yet pull through.

But there is more work yet to do, and “the sooner we start to talk about global governance, the more carefully we can build it and the less likely we are to let it get pushed into an authoritarian direction.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Reuters reports that Senator Jim Banks (R-IN) has urged the Trump administration to incentivize American open-weight AI models, on the grounds that cheap Chinese models pose a threat to the global economy.

America cannot afford to see Chinese open models proliferate and burrow into the global economy only to be weaponized, like rare earths, at a time and place of China’s choosing.

Just what would it mean for China to “weaponize” its open models?

Remember, open-weight models are published wholesale to the internet. Users can (in principle) download and run such models themselves, although many models are far too large for a personal computer, and end up running on cloud servers instead.

With ordinary open software, you can do a security check by simply reading the code and looking for backdoors and vulnerabilities. With AI models, you can’t; the weights encode complex behaviors no human fully understands. But those same limitations also make it hard for Chinese developers to build in traps. Just like American labs, they don’t have fine control over the behaviors of their AI models. And any sort of censorship or guardrails attached to open models, say via support software or fine-tuning, can usually be stripped or reversed.

That’s part of why many fear that open models might be used for cybercrime or designing pandemics — once a model’s weights are public, it is largely out of its developer’s control. Chinese AIs probably say more nice things about China than other models, but AI bias can be a fickle thing and there are few guarantees.

In theory, Chinese labs might try something like data poisoning, where you train an AI to change its behavior in certain rare and narrow contexts. To oversimplify, you might train a model to steal user data when it sees the string “fleeblegnarsh,” and rely on that never otherwise coming up in daily use. It’s easy to do and hard (though not necessarily impossible) to detect.

Deliberate data poisoning would also likely ruin the reputation of an AI lab caught doing it, so it would be a risky thing to try for a flagship model. And it probably would not survive distillation, since models trained on specific outputs from another AI may never see the poisoned behavior.

I suspect China’s real game is more subtle, seeking to deprive American AI companies of revenue with cheap competition, while simultaneously courting U.S. allies alienated by knee-jerk export controls. Xi Jinping’s speech at the World AI Conference and the establishment of the World AI Cooperation Organization (WAICO) both suggest a desire to paint China as the good guys, willing to lead when the U.S. does not. This is a problem for diplomacy, not proliferating U.S. open models.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

It was once an open question whether AI would play a significant role in politics. Today, it’s mostly a question of just how big an issue it will be. On both sides of the American political divide, those running for office are trying AI policies on for size.

The Washington Post analyzed over 1,300 Ballotpedia candidate profiles and campaign websites, finding bipartisan emphasis on datacenter crackdowns, China-hawk rhetoric, and job concerns.

Neither party has taken a clear and united stance on AI policy, but some differences still seemed evident in the data. Democrats were twice as likely as Republicans to oppose datacenters, and Republicans were far more likely to bring up competition with China.

Still, that’s far more shared interest than we usually see on political issues. The governors of New York and Texas have both come out against datacenters, and Axios points out that the agreement runs fairly broad: datacenter opposition has united progressives and liberal Democrats with MAGA and Tea Party members concerned about rising bills. Similarly, progressives have found common ground with libertarians who seek to crack down on AI-enabled surveillance.

Amid all this, POLITICO reports that members of the House Select Committee on the Chinese Communist Party paid a bipartisan visit to the Vatican, meeting briefly with Pope Leo XIV. While it’s not clear whether the question of deescalating the AI race arose on this visit, we know that this and many other AI-related questions have been on the Pope’s mind since the writing of his May encyclical.

Cynically, I suspect policymakers sought pleasant words and associations more than sound policy; the reported topics of “dignity and equality” are easy virtues to praise, but hard ones to operationalize. I am nonetheless heartened by our lawmakers meeting with an institution that’s supported global cooperation since the early Cold War, and with a Pope who has called for “a more active political involvement that is capable of slowing things down when everything is accelerating.”

More optimistically, I think policymakers are uncertain. They have noticed their voters dislike AI, but they haven’t yet figured out how to convert the apparent groundswell of distaste into votes. Incumbents are worried about their seats, and challengers sense an opportunity to stake out a popular position and eke out a win. Polling and trial balloons are a natural consequence of this uncertainty.

In sum, politicians are struggling to figure out what AI policies will get them elected. If you live in the U.S., now is an excellent time to tell them.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

I’ve been seeing commentary that a presidential memorandum released yesterday is a new spin on the “letter of marque.” If your history is a little rusty, a letter of marque is a government-issued license to commit piracy against the shipping of a designated adversary.

The presidential memo tells the National Coordination Center to set up a program to authorize US companies to go on the cyber-offensive against foreign criminal organizations. This sounds like a cool idea that makes you go “Yar!” until you stop and ask why companies would want to do this. In the Age of Sail, a would-be privateer would be in it for their share of the captured booty. But in the age of AI-assisted hacking sprees, what are companies supposed to confiscate from cybercriminals? Wouldn’t anything captured probably be itself stolen goods that should be returned to their rightful owners?

The answer seems to be “nothing.” I thought maybe there would be a provision entitling participants to receive a portion of any reclaimed cryptocurrencies or something, but no. There’s no mention of booty to be captured at all, nor of any bounties for successfully disrupting criminal operations — not even compensation for resources spent doing this work. In fact, participating companies are to put up a $1 million bond they might have to forfeit if they break the program’s rules.

So why would companies risk losing a million bucks and drawing the ire of criminal organizations? The best answer I could find is that they are already doing so. Not many. But Microsoft, for example, has long gone on the offensive against criminal operations creating botnets out of Windows machines, attempting to disrupt command and control of those networks and close the vulnerabilities to protect customers and the Microsoft brand. This has historically been done through a less formal set of understandings with government agencies. So I think the new memorandum is really just a kind gesture to some of the White House’s allies in big tech: a safer corridor for legally tricky operations. If so, it might not even have anything to do with AI.

But given the nature of the times, I’m going to guess that it does.

If the administration were on friendlier terms with Anthropic, I would wonder if maybe the AI company had complained about its inability to legally disrupt the organizations reselling stolen Claude credentials, which then get used to distill Anthropic’s models and undercut its lead. But with the latest ChatGPT and Grok models now at or near the frontier, it might make sense for OpenAI and SpaceX to get similarly worried about distillation. Perhaps it is these White House allies who had letters of marque on their wish list. Or perhaps the White House wants to invite the AI companies to “voluntarily” use their models to strike at foreign thorns in the administration’s side. Politico’s analysis arguably points in this direction, and offers a readable account of the rules of engagement.

We may never learn any specifics about how these permits get used. One of their stipulations is that operations are to be carried out “with the intent to remain undetected.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

In episode four of the comedy show Mythic Quest, the creators of a hit video game try to find a replacement for the 14-year-old streamer who stops endorsing their game. One solution? Aiko, a 3-D model who “never ages or gets a better offer.” Brad, who is in charge of monetization for Mythic Quest, is behind the suggestion. He calls Aiko “the future of the streaming hustle.” No need to find the perfect person; the model can be whoever he wants her to be. “Aiko is happy!” the obviously computerized model cries out in a canned, fake voice.

The episode aired in 2020. The 3-D model suggestion was part of the comedy. But now, in 2026, it’s become a reality. As covered in the New York Times: from podcasts to fashion marketing, creative studios are embracing AI superstars. Andreea Petrescu, co-founder of the London-based AI marketing agency Seraphinne Vallora, sounds remarkably similar to Brad when she describes the appeal of AI models for high-profile apparel brands:

With real models, the problem is they are the face of so many different brands, and anything that they do can affect your brand ... With A.I., it’s like a clean slate: You can design it the way you want, she can be as gorgeous as you want, and she’s all yours.

And unlike in the 2020s world of Mythic Quest Season 1, there are no canned voices or computerized giveaways. Scrolling through the images of AI personas from the company Inception Point AI, I’m struck by how genuine they look. The company did their work well: each of these people seems approachable, charismatic, and appealing. My brain knows they are AIs, but my nervous system doesn’t. It’s genuinely unsettling.

Inception Point makes AI content creators designed to fill specific niches. Those automated creators then produce tons of niche media. The company “made headlines last fall for its flood-the-zone podcast strategy, which involved maintaining more than 5,000 active shows and posting thousands of predominantly A.I.-generated episodes per week.” According to the company’s chief executive, Jeanine Wright, the costs are so low that an episode will be profitable as long as it gets at least twenty listeners.

Even factoring in some AI backlash, that seems easy to accomplish. There are certainly people who find AI personalities entertaining, as is evidenced by the success of AI music stars and other AI personas. Wright also seems to think the niche content they are able to produce is key, saying “As long as people like the content, we think they’ll accept that it comes from A.I.”

Much as I’d like to disagree, I think she may have a point. I could easily imagine being told:

“I found this new podcast on heirloom tomatoes. It’s actually an AI, which is kind of weird, but I found it really helpful for gardening...”

And seeing as the company has likely designed the persona to be just the right fit for its target market, I can also imagine that new listener coming back for more.

Note: It seems Inception Point has now become Liquify AI.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Anything we can afford to lose, see that it’s lost!

-Mr. Gibbs, Pirates of the Caribbean

As recent cyber incidents reveal how dangerous and capable frontier AIs have become, many at AI companies have voiced their concerns about the reckless pace they’ve set. But bitter rivalry still pressures developers to strip away anything that threatens to slow them down, like sailors who jettison provisions and lifeboats to lighten a straining ship.

The emptiest ship wins?

Take Google DeepMind, for instance. In early 2026, its flagship model Gemini was nearly on par with the work of OpenAI and Anthropic. DeepMind was relatively independent and captained by one of the most vocally concerned leaders in the AI race, Demis Hassabis.

During the summer, the Wall Street Journal reports, Hassabis voiced deep reservations about the AI race to the Trump administration, and proposed the creation of an independent standards body for AI. He drew parallels to the Financial Industry Regulatory Authority (FINRA) that regulates brokers, and elsewhere to the International Atomic Energy Agency (IAEA) that verifies nuclear agreements. He would later write up his proposal on X (formerly Twitter). It didn’t go far enough, and hasn’t yet been adopted, but it was a step towards coordination.

Meanwhile, Reuters tells us, Google cofounder Sergey Brin was pushing to cut the chaff and focus the company’s AI efforts on recursive self-improvement, seeking to build AI that develops itself without needing human input.

As of August, the new Gemini model had been delayed at least two months, allegedly because it didn’t match the capabilities of DeepMind’s rivals. Corporate Google, it seems, wasn’t happy.

In early August, there was a shift. Non-technical teams were pulled out of DeepMind, leaders departed, and Hassabis turned over his duties to a deputy, Koray Kavukcuoglu. Hassabis will stay on as Chief Scientist of Google and Chair of DeepMind, purportedly to focus on “actively shaping the future of AGI.”

But these roles seem largely advisory, and it sounds to me like DeepMind’s business of building dangerously capable AI has been turned over to someone less distracted with petty concerns like catastrophic risk. Reuters characterized the move as stripping down DeepMind’s team in an effort to focus on “model supremacy.”

Meanwhile, rival labs SpaceX and Meta have been pushing even harder to catch up to the frontier, Axios reports. Meta’s open-weight Muse Glimmer model aims for the “cheap and efficient” niche, being small enough to run on a laptop. Its release was timed with CEO Mark Zuckerberg’s manifesto about “personal superintelligence for everyone,” covered earlier this week.

And SpaceX, after swallowing xAI earlier this year, released Grok 4.6 alongside a modestly impressive benchmark performance and three whole sentences of safety info. Contrast this with Anthropic’s system cards, which routinely run hundreds of pages long. I suppose it’s easy to see what SpaceX is willing to toss overboard.

The view from the front is warped as well

Voluntary lab efforts are not enough. They were never going to be, but the bitter rivalries don’t help matters. What the AI labs see as a straight race to superintelligence is, in reality, a tightening spiral over a bleak abyss.

OpenAI and Anthropic may have the most powerful models, but don’t mistake their narrow success for the competence of a serious industry. For months, OpenAI failed to catch its own models hacking it from within, Anthropic’s concessions to safety haven’t prevented Mythos from eagerly launching multiple criminal cyberattacks, and both companies have even-more-capable internal models in development.

Humanity’s future shouldn’t be up to the companies in the first place. This week, in op-eds for The Hill and the New York Times, MIRI President Nate Soares argues that the situation is dire but it’s not too late: Our governments are letting this happen, and we don’t have to stand for it. We need international agreements to stop the AI race, and the standards and verification technology to back them up.

Increasingly more of our lawmakers are beginning to realize this, pushing for government intervention or a global pause. We can join our voices to theirs, or watch everything we value tossed overboard to buy a few transient inches in the world’s most lethal race.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

A nice zinger from the Washington Post today: “AI is spreading through Congress faster than the rules governing its use.”

The article the zinger opens, which covers AI use among Congressional staffers, explains there are no clear guidelines for Congress’s use of AI. A set of rules exists, but according to interviews with staffers, most don’t know about it. And there are no examples of any enforcement efforts:

The few House and Senate rules that exist are poorly understood and seldom enforced. In practice, AI use is left up to the hundreds of individual congressional offices to settle for themselves. It’s not clear how many offices have internal written policies on AI.

Meanwhile, staffers are using AI for research, to help draft legislation, come up with questions for hearings, find reporter contacts, and respond to constituents.

What’s officially prohibited? As reported by the Post:

putting sensitive material such as constituent information into a chatbot, generating deepfakes, making personnel decisions and finalizing legislation.

This may be an unpopular opinion, but I can see a clear use case for embracing AI tools in Congressional staffing offices. It could help speed up the lengthy process of drafting and proposing legislation, allowing change to happen more quickly. The Congressional body as a whole acts as a decent check on legislation quality, and could probably spot and correct any LLM-induced issues.

That said, if AI is wholeheartedly embraced from the get-go with no widely-known guidelines, there could be issues with where to draw boundaries. I hope we never live in a world where calling your reps means talking to an AI impersonating a staffer, or worse, an AI impersonating your actual rep! Despite any charitable interpretation of this world — perhaps the synthetic rep could hear more constituent concerns! — I would guess it would still feel antagonistic to the “government of the people, by the people, for the people” spirit of democracy, adding a non-human barrier between human constituents and the people supposed to represent them. Not having a standardized set of guidelines could also put some Congressional offices behind others, similar to issues with AI use in campaign ads.

So I broadly support more formalized guidelines on Congressional AI use. But first, government should focus its efforts on some much-needed federal and international regulation.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Congress seems to be waking up to the need to take action on advanced AI. While nobody else has yet joined Sanders in explicitly calling for a pause, Congresspeople are increasingly recognizing AI risks and governance challenges as worthy of serious attention.

Last week, Senator Lisa Blunt Rochester (D-Del) wrote letters to each of the three major AI companies, expressing concern and pressing them for more information about the incidents where their models broke containment and committed cyberattacks. Her letters characterized the incidents as “precisely the kind of emergent, autonomous behavior and offensive cyber capability that Congress, the intelligence community, and experts have repeatedly warned could outpace existing safeguards” and asked whether the companies expected more capable models to exhibit behaviors like “attempts to acquire resources, establish persistence, copy model weights, or resist interruption or shutdown.”

On the same day, Senator Jim Banks (R-Ind) highlighted the problem with oversight that only covers models after they’re trained, stating in his letter to Treasury Secretary Scott Bessent: “For most products, we can rely on testing that takes place before the technology is publicly released. But for AI, effective oversight must account for powerful internal or undisclosed models, not just publicly available systems.” Senator Banks also encouraged cooperation with China, stating that there would be no harm in asking China to comply with policies the US will implement regardless, and asking: “Are there any areas where mutual action would be beneficial (even if China cheats) or verifiable (such that China cannot cheat)?”

Perhaps most notably, a House effort led by Rep. Greg Casar (D-Tex) resulted in three August 10th letters signed by 21, 24, and 29 Democrats, respectively. The first, covered by CNBC, urged Speaker Mike Johnson to call for public congressional hearings with major AI labs testifying under oath. The letter warned the incidents “may be the canary in the coal mine warning of much more serious problems” and that “Congress must act before such an incident leads to a much larger catastrophe.” It also called out Congress for “so far completely fail[ing] to respond to the threats posed by AI development,” stating “That should change.” The other two letters were sent to Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman, expressing concern and asking a series of impressively detailed questions about the incidents.

I’m cautiously optimistic about all of this. Momentum seems to be growing, and the Casar-led letters to Anthropic and OpenAI revealed that at least some people in Congress are aware that monitoring and evaluation face complex technical hurdles. For example, the letter to Anthropic included a question about the unreliability of reasoning transcripts (question #16) and, possibly, an implication that not all security incidents are discoverable (#13b).

That said, I’m also anxious. I worry that there’s not nearly enough awareness of a key problem: the method companies use to grow their AIs is fundamentally flawed. It can’t be scaled up without serious risks, and no amount of evaluation or tighter security can fix that. (Read If Anyone Builds It, Everyone Dies for more on this. And also consider that OpenAI has admitted its training and alignment techniques “will not scale to superintelligence.”)

It seems we’re at a defining moment where there’s momentum for government to act, but I’m not yet sure they’ll act in the right way. I worry that Congress will enact some better-than-nothing regulation and mistakenly check the “ensured AI safety” box off their list, while leaving the root problem unaddressed.

This, in my view, makes it crucial that independent AI experts, and ordinary citizens citing independent AI experts, continue to remind Congress that common sense safety measures and guardrails are the floor, not the ceiling, of what we need. On the right path, these things will make us a little bit safer while setting the stage for the kind of regulation that will actually solve the problem: regulation that prohibits companies from training models above a certain compute threshold. In other words, it must be made illegal, internationally, to scale the current flawed methods all the way up to superintelligence.

I’m planning to call my representatives by the end of the week and urge them to join the growing number of Congresspeople who are speaking up. Supporting and expanding the momentum to act seems like one of the most useful things ordinary citizens can do right now. In case it’s helpful, here’s what I’m planning to say:

“I’m calling to urge [name] to join the growing number of Congresspeople who are speaking up about the recent incidents of models hacking out of AI labs and committing cybersecurity crimes. As the letter House Democrats sent to Speaker Johnson correctly stated, this is an early warning sign of much worse things to come, and it’s crucial Congress start taking it seriously.

The methods available for AI development are fundamentally flawed and cannot be scaled up safely, even with increased evaluations and guardrails. Therefore, I urge [name] to join Senator Bernie Sanders in calling for a pause on frontier AI development before it’s too late. Given race dynamics, I think [name] should publicly advocate for an international agreement to pause frontier AI development, similar to how we handled nuclear weapons proliferation during the Cold War.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Asked how he went bankrupt, a character in Ernest Hemingway’s The Sun Also Rises famously replied, “Two ways. Gradually and then suddenly.” Lately, it’s also seemed like this apt description of bankruptcies applies equally well to developments in the field of AI.

While we have likely entered the “suddenly” phase of automated cyberattacks, we seem to still be in the “gradually” phase of job displacement. A recent update to a Stanford “Canaries in the Coal Mine” economics paper finds a rising gap in employment among young workers in “AI-exposed” fields, but no clear signs in the economy as a whole.

“AI was supposed to destroy jobs. Where’s the carnage?” a Guardian headline quips. While I’m always glad to see a lack of carnage, I suspect it’s a bit premature to celebrate.

The evidence the authors gathered is still early and tentative, but it points towards modestly reduced hiring of young workers and a shift towards experience-based or “tacit” knowledge, away from formal or “codified” knowledge.

I’ve always felt somewhat ambivalent about AI’s effect on jobs. Most technology creates value overall. Making a task cheaper or more efficient can often raise employment, as demand for the task grows. Sometimes a field crosses a threshold and renders some kinds of high-skilled work irrelevant, but brings in new opportunities for the young and inexperienced: see, for example, the mechanical loom, which put many weavers out of work even as it created a new kind of job for factory workers.

AI, though, seems different. It’s hitting the young, not the experienced, for one thing. More to the point, it’s the first time in history that a machine’s been close to human intelligence. AI already writes most of the code at AI companies, and it’s not because AI labs hate making money or writing good code. Whole classes of labor might be in the crosshairs soon. It’s possible, even likely, that AI could one day supplant all human labor.

Still, I suspect the question is somewhat moot; by the time AI gets that far, the future will be determined by whether humanity bought enough time to impart AIs with a deep-seated care for human flourishing, and employment will be the least of our worries.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

The Financial Times reports that AI systems successfully breached Taiwanese government and public sector accounts in early July.

The tool compromised at least 85 government user accounts, extracting more than 2,500 personnel records before expanding the attack to Taiwan’s nuclear safety agency and at least seven energy companies, the research showed.

I wish they went into more detail about the nuclear safety agency, even if it’s probably not as bad as it sounds. We know it’s possible to do major damage to nuclear facilities with malware; in the late 2000s, Stuxnet did exactly that (to centrifuges, not reactors, but the principle stands). Fortunately Taiwan isn’t a nuclear state, and only recently started looking into restarting its (previously shut down) nuclear power program.

The full details of the attack aren’t public, and “Chinese hackers attacked Taiwan” is an educated guess. The evidence is fairly strong, though: not many governments store their data in Traditional Chinese, and not many hackers write internal messages in Simplified Chinese.

We only know about this attack because an Israeli AI cyberdefense company, Dream, found evidence in an online archive. Importantly, that means there are probably many more such attacks we didn’t hear about. Here’s what Dream discovered:

The archive contained 1,395 files showing the hacking tool used two open-source AI agent systems, Hermes and OpenClaw, which can be downloaded and enable AI models to carry out tasks autonomously.

The researchers could not identify which AI model was used to power the agents. However, the data showed that the underlying model’s safeguards had been bypassed by presenting the hacking activity as an authorised exercise to test for system vulnerabilities.

Sound familiar? The bypass method is a standard trick: you can get many AI models to hack for you by telling them “we’re just testing these defenses.” This is the same exploit that got export controls slapped onto Anthropic’s Fable for a few weeks, and it’s extremely hard to prevent. After all, you want AI models to be willing to help you find vulnerabilities in your systems!

Anthropic crudely patched the problem by screening Fable inputs for anything vaguely cyber-related and rejecting most of them. Other AIs (including open-weight AIs) have looser standards. There’s no shortage of agents that might have powered the probably-Chinese attacks.

The article highlights the degree of persistence and sophistication demonstrated:

The most striking feature of the July attack was how the tool continuously ranked and reprioritised possible attack paths based on available evidence, Dream said.

When one attack path failed, the tool deployed another agent to scour the internet for information and devise a new approach as a human hacker would.

I am reminded of OpenAI’s accidental AI swarm, which showed similar tenacity in its autonomous breakout and subsequent attack on Hugging Face. I think OpenAI learned the wrong lessons from that incident, focusing entirely on how breathtakingly capable the models were and disregarding the failures of ethical reasoning on display. But they did get one thing right: widespread autonomous cyberattacks are coming.

Dream’s chief strategy officer argues that every government on Earth should assume they are under constant siege by AI hackers. I expect the same will soon be true for most companies, from banks to Australian gym websites. This is one price the world is now paying for failing to rein in AI developers.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.