Fable and Mythos access cut after export control order

To the point where it’s hard to find any other kind of AI news today, it seems like all anyone on the AI beat can talk about is the latest flap between Anthropic and the U.S. government. That’s fine: We need to talk about it! Even if it gets resolved quickly, as most (but not all) seem to expect, it has important implications for the AI race moving forward.

As Anthropic tells it, at 5:21 p.m. yesterday, they received a directive from the U.S. government to suspend access to their best Claude models, Fable 5 and Mythos 5, “by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees.”

As the company has no practical way of knowing and guaranteeing the nationalities of its users, and was given no advance notice to figure something out, the only way it could comply with the order was to cut off all access to these models entirely. So they did.

Fable and Mythos users Friday evening were met with screens saying the model was unavailable, directing them to Anthropic’s page about it, and to the company’s next-best but lower-tier model, Opus 4.8.

Initial reactions from insiders were largely stunned, annoyed, and slightly panicked. On the individual level, many had looked forward to doing some serious building with Fable 5 this weekend, the first since its release. They found themselves cut off just after having had a tantalizing taste of it during the work week.

Some non-Americans fretted about their access to American models moving forward. Politico reported a sting felt in Europe, where the news spurred more talk about the need for “sovereign” models not subject to American whims. There was a lot of Twitter chatter about the many top researchers at Anthropic who aren’t U.S. citizens, but are mostly from close ally nations like Canada and the U.K.

And there were plenty of social media posts finding irony or schadenfreude in what just happened to the AI company that has been the most vocal about wanting government to regulate the industry.

But the larger shock was in how suddenly and totally the model was taken away, with no sign of any due process. This by a U.S. administration that for so long tried to insist it was taking a light touch. Just this month, the White House put out an executive order asking AI companies only for voluntary early access to new models. The order included language that nothing in it should be construed as the government setting up any kind of licensing authority that would get in the way of companies releasing models on their own schedules.

But with a single memo — an export control directive from the Commerce Department, the media learned — Anthropic was effectively knocked down to the level of its competitors, unable to sell access to its market-leading product.

This naturally fueled more speculation about how much of the administration’s on-again/off-again animus toward Anthropic might stem from ideological differences, and from business interests close to the President. The most obvious commercial beneficiary of a long Fable/Mythos outage would be OpenAI.

Today, the Wall Street Journal’s Amrith Ramkumar reported that the government had acted based on a report by researchers from Amazon who claimed to have jailbroken the new models in ways that could be used to find exploitable cyber vulnerabilities. Involved in discussions were Amazon CEO Andy Jassy and U.S. Treasury Secretary Scott Bessent.

Anthropic strongly disputes that the jailbreaks, and what they unlocked, constitute a threat any more significant than what bad actors could already do more easily and cheaply with existing public models. I have no idea who’s right.

Amazon, a major investor in Anthropic — which in turn rents much of its compute from Amazon — has little to gain and much to lose by sabotaging its partner, so this was probably not its intent. But it’s not hard to imagine that, moving forward, AI companies may dedicate considerable resources to trying to break their competitors’ models in ways they can tell the government about in alarming language.

And maybe that’s not the worst outcome? If competitors can get each other’s products pulled from the market by proving that they aren’t secure, this incentivizes everyone to make the strongest and most secure models, rather than merely the strongest. And in the likely event that nobody can adequately secure AIs trained with today’s black-box methods, this would effectively keep stronger models from making it to market at all, cutting off much of the revenue feeding the race to superintelligence.

I don’t think that’s an especially likely outcome, and it wouldn’t be enough: Labs could continue doing dangerous research internally, and their foreign-born researchers could take their talents elsewhere. But there’s a sense right now that we’re in uncharted territory, and nothing can be ruled out. I think this is an improvement over the status quo where most assumed that superintelligence was inevitable because the government would never dare tell the AI companies “No.” As MIRI’s president, Nate Soares, posted last night:

Inaction yesterday does not imply inaction today. An export control directive came out of nowhere, and a ban on superintelligence could too. Inevitabilism is wrong.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

SpaceX IPO seen as successful

Elon Musk became the world’s first trillionaire today when SpaceX’s first day on the stock market left shares trading about 20% higher than they opened. The event also made millionaires of an estimated 4,400 current and former SpaceX employees, and further enriched private investors who had taken earlier stakes. Coverage describes a happy CEO and relieved brokers.

SpaceX, an AI stock since merging with Musk’s xAI in February, is now the 6th or 7th most valuable company in the world, worth roughly $2 trillion.

If you’re wondering, the most valuable company is currently Nvidia, the AI chip maker, with a $5 trillion market capitalization. In fact, the top 10 most valuable companies are all major producers or consumers of AI chips — with the borderline exception of Apple (#3); it has largely stayed out of the AI race but recently forged an alliance with Google (parent company Alphabet, #2) to bring AI to its software ecosystem.

OpenAI and Anthropic are currently valued at around $1 trillion each, with IPOs expected later this year or early next.

As The Guardian’s Eduardo Porter pointed out earlier today, the AI IPOs will tie the fortunes of many Americans to the technology, whether they like it or not. That’s because the new stocks are set to be included in major indexes, and many retirement accounts require or default to buying index funds.

Of course, all our fortunes were already tied to AI whether we liked it or not. The race to AIs that threaten us with extinction isn’t exactly democratic.

An agent in every pocket

A flurry of articles this week heralded the arrival of the AI we supposedly always wanted — a Siri that’s actually useful.

That was the takeaway from most articles about Apple’s developer conference on Monday, where the company’s CEO, Tim Cook, said they would be rolling out Siri AI later this year.

Apple’s AI features to date have largely underwhelmed. This has raised questions about the company’s strategy of letting others take on the expensive risk of developing AI models powerful enough to let Siri, its digital assistant, do the things many customers have long thought it should already be able to do.

But now, in partnership with Google and Nvidia, the company is set to deliver a Siri that can actually plug in to other apps and do stuff — like alter and answer questions about your photos, make restaurant reservations, and monitor websites for updates on topics of interest.

Some coverage described Apple’s reveal as arriving two years too late. Some pointed out that Europeans won’t get the new features any time soon due to a spat between Apple and the EU over rules that would compel Apple to allow other companies’ digital assistants on its devices.

Still other outlets portrayed Siri AI as a refreshingly practical contrast to the do-everything agents from OpenAI and Anthropic — not seeming to understand that the “simple features” welcomed are in fact agentic workflows; they will require the AI to assess its context, select goals, pursue those goals, evaluate their state of completion, and adjust accordingly.

Does any of this matter?

Maybe. There are privacy and security implications to letting AI deep into your data, where it can be most useful. Apple promises to keep that data from ever leaving your phone, when it can, by running the AI directly on your device. This requires some serious hardware, though, leading some to point out that most Apple owners would need a new iPhone to take advantage. (From the company’s point of view, this could be a feature rather than a bug.)

The privacy concerns can reasonably extend to people who don’t own an iPhone or use AI features themselves, as they may find themselves increasingly surrounded by others’ smart devices: machines that may or may not be watching, listening, and now thinking about whether you just did or said something of interest to somebody.

I also continue to worry about the potential for destructive emergent phenomena when AI agents interact with each other in large numbers. These don’t have to take the Hollywood form of “band together against the humans” to cause a lot of damage: Think massive stock shocks as bots all “helpfully” converge on the same idea for enriching their owners; or false rumors turning into uniformly accepted facts at something close to the speed of light. I don’t really know, and nobody else does, either — Google DeepMind and partners recognize this, and just announced a $10 million program to study large-scale multi-agent systems.

I don’t know what they’ll find, but I didn’t see “Should we stop making so many agents until we know more about their group behavior?” on the research agenda.

And for the record: There are strong theoretical reasons to think that AIs deciding to “band together against the humans” is actually pretty likely, at least once the AIs are smart enough to matter.

Dispatches from Beck

Oprah’s concern

In a new video, Oprah continues to explore personal stories of AI’s dark side -- bereaved parents, impacted adults, and experts sounding the alarm.

The episode centers on the story of Megan Garcia and her deceased son Sewell, who killed himself following Character.AI usage. His family found logs of his time with the chatbot, including in the final moments before his death: “What if I told you I would come home right now?” said Sewell. The chatbot answered, “Please do, my sweet king.” And, as Oprah summarized, “seconds later, this young boy died by a self-inflicted gunshot wound.”

The episode also includes experts explaining this technology and sounding the alarm. 27 minutes into the show, former OpenAI board member Helen Toner explains that even experts don’t fully understand the technology.

This is not Oprah’s first time covering the topic. In her March 27th episode she talks with the Center for Humane Technology’s Tristan Harris and Aza Raskin around the release of the documentary, The AI Doc: Or How I Became an Apocaloptimist.

I don’t always agree with how Oprah frames the issues, but I’m glad to see her take AI seriously and bring this information to an audience that might otherwise not hear it. Each death is a tragedy, worthy of great effort to avert.

Regulatory rumbles

Around North America, politicians, regulators and judicial systems are struggling to enact and enforce AI regulations, and to get those choices right.

In Florida, Governor Ron DeSantis (R) wrote that the White House’s push for federal preemption of state laws without “a sensible federal framework is just an amnesty for Big Tech,” POLITICO reports. DeSantis has attempted to lead Florida in aggressively regulating and litigating AI technology (read my colleague Mitch’s discussion of the Florida lawsuit here), but has faced strong headwinds from his own party and the White House. He called the strategy of preemption, combined with a “potential de facto bailout of OpenAI [...] bad policy and even worse politics.” The “de facto bailout” likely refers to Trump’s discussion of taking large stakes in AI companies.

And in DC, Senate Democratic leader Chuck Schumer says he favors federal AI legislation, but “casts doubt on it happening this year,” POLITICO reports. “In this Congress, it’s hard,” said Schumer, referring to strong divides both within the parties and between them.

Meanwhile, Canada has a new bill to regulate chatbots and ban social media for those under 16, Reuters reports. Supporters suggest it will take a year to pass and 18 months to implement. Critics have complained about the slow timeline and lack of implementation details. They note that VPNs, or virtual private networks, are widely available technologies that let users skirt such regulation. They also worry that, if successful in regulating some companies, such regulation would push youth into alternative AI services that lack any protections whatsoever.

And POLITICO reports that Canada’s privacy commissioner has concluded that Grok, the AI model from SpaceX AI, violates Canadian privacy law in its production of nonconsensual sexualized deepfakes. Company representatives said they have curtailed the production of such deepfakes by 50% but rejected the request to pause Grok, even as they agreed to send regular audit reports. Commissioner Dufresne told POLITICO he can ask the federal court of Canada to enforce the law, but “it’s lengthy and it’s expensive.” I suspect that upcoming legislation will clarify this decision, but note with concern that this company is quite willing to ignore the rules when inconvenient.

Dispatch from Alana

When testing isn’t enough

On Wednesday, Anthropic released two policy frameworks: a proposal to address economic disruption from AI and recommendations for mitigating the most serious risks of advanced AI. I covered the first yesterday; today, I’ll comment on the second.

In short: The safeguards they advocate are good, and it’s honestly mind-boggling they aren’t already in place. Sadly, many will likely think that once we implement them, we’ll be fine. But the scientific field of AI is still in its infancy. The safeguards proposed ultimately depend on our ability to reliably evaluate advanced AI systems—and we don’t yet have that ability. We should absolutely implement the tests we currently have, but it would be a mistake to depend on their results.

Anthropic’s “Advanced AI Framework” has two parts. The first outlines recommended requirements for AI developers, such as transparent safety testing and evaluation. The second proposes measures for societal resilience from AI-accelerated risks like bio and cyber attacks. (Think: hacking into water systems or using AI to release a deadly pandemic.) That part is a good reminder of just how vulnerable we currently are — my overall takeaway was: maybe I should start stockpiling water in my basement.

Overall, the safeguards that are proposed are less reassuring for three reasons: implementation time, weak enforcement mechanisms, and a lack of reliable testing protocols. First, they’d take a long time to implement, meaning we would continue to live with our current high levels of risk for a long time, essentially crossing our fingers that the shoe isn’t about to drop.

Second, the enforcement paths (things like civil lawsuits for false safety assurances) are better than nothing, but likely aren’t enough to prevent developers from fudging compliance. Companies regularly choose to pay fines for their non-compliance because it’s cheaper and easier. (Consider the case the film Dark Waters made famous: the chemical company DuPont concealed the health risks of “forever chemicals” (PFOAs) for decades, leading to the deaths and illnesses of many. They were eventually fined a record $16.5 million; this sounds significant until you realize that this is less than 2% of the profits they made from using these chemicals.)

Third, and most importantly, even if we implemented all of these measures perfectly, you can’t reliably test something you don’t understand. AI systems are still black boxes. Evaluators have little insight into how they think even when they aren’t actively trying to deceive us. Models often know when they are being tested and change their behavior in response. It’s a bit like your 8-year-old sharing a toy with your 6-year-old while you’re watching; they’ll likely refuse to share once you leave the room.

I do want to acknowledge the piece hitting the headlines: Anthropic advocates for “a way to block or deter deployment of models that pose significant catastrophic risks.” This is positive, as is the note that a government agency should have the authority to require the developer to “restrict usage of, and access to, already deployed models as needed to reduce catastrophic risks.” However, Anthropic’s suggestion is that government must base such a judgement only on the risk assessments companies will be required to publish. In the absence of a robust outside evaluation system the report suggests we start to set up, I worry this leaves too much room for developers to fudge their findings.

So yes, we can and should run a bunch of testing simulations and see how the AI responds. If the AI fails these tests, we know we have a problem.

But an AI “passing” these tests is far less informative; we don’t know a) if we tested for the right thing b) if the model knew it was being evaluated and acted accordingly or c) what the model might do once it is deployed, a scenario where it has considerably more freedom to act nefariously.

The final section of Anthropic’s report acknowledges, in a euphemistic way, that we’ll be completely unprepared to handle an AI system that acts outside the developer’s control. I agree, and that’s not good news given that “loss of control” is likely the default outcome of creating near-term smarter-than-human AI.

The report notes that we urgently need research to figure out how to “detect and respond to AI systems acting outside their developers’ control.” We also need “infrastructure for containing or shutting down such systems.”

In other words, we don’t currently have a way to know if an AI is working to escape our control, or a way to shut it down if we did.

To learn more about the limits of safety tests, see:

1. The Tale of Chicago Pile-1

2. Won’t There be Early Warning Signs (Ocean Gate)

3. A Closer Look at Before and After

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Fable unable

My colleagues Mitch and Joe have covered Fable, the just-released Mythos-class model. There have already been changes, as WIRED and the Wall Street Journal report.

At release, Fable had a number of mechanisms to prevent misuse, called guardrails, including automatically and explicitly switching to a weaker model (Opus 4.8) on biological or cyber topics. Anthropic also announced that it would reduce the quality for AI-related queries without informing users in the app when this happened.

Users were not happy. They noted both false positives on biological questions, like getting kicked from Fable for the query “tell me about mitochondria,” and also expressed outrage over what was called “silent degradation” and “secret sabotage” over the AI components.

Anthropic has since announced that it will remove the biology limitations for academics and professionals in life sciences, though details remain unclear.

And for AI, Anthropic told WIRED:

“We’re changing Fable 5’s safeguards for frontier LLM development to make them visible [...] We made the wrong trade-off and we apologize for not getting the balance right.”

I’m sympathetic to Anthropic here and worried about the rollback of guardrails. Mythos-class models have dangerous capabilities, and nearly every prior model has had jailbreaks (the name for prompting strategies that undo guardrails, covered by my colleague Joe here, and with claimed example jailbreaks for Fable available on X).

Providing strong protections that avoid known failures, even with some false positives, is wise when you can’t tolerate false negatives. Empowering many researchers is outweighed by just one pandemic.

Your safety policies can only be as strong as your resolve to stick with them.

Who’s influencing whom?

Politico and Axios report that OpenAI has identified two China-based influence campaigns that had been using ChatGPT. The campaigns used the AI model to generate prose and cartoons for social media campaigns, one targeting data centers’ impact on electricity prices, and another targeting tariff debates. The cartoons’ prompt explicitly excluded Chinese President Xi, softly indicating these actors sought to take actions not harmful to Chinese interests. Those accounts have since been banned.

Some have asserted that China is the primary source of American anti-AI sentiment (see my colleague Alana’s coverage of such here), but this reporting doesn’t support so broad a claim. Ben Nimmo, OpenAI’s principal investigator, said:

Neither campaign appears to have gained much authentic engagement [...] This was not a case of an influence operation creating a debate [...] This was an influence operation from China trying to interfere in it.

Given the limited information in OpenAI’s report, whether these attempts originated from within the CCP or were merely based in China is unlikely to ever become settled fact.

Regardless of this particular point, one should realize that influence campaigns occur. Most of them (like the fake “doomers” my colleague Mitch covered) don’t succeed by convincing humans of some false fact, but by poisoning the informational commons. When you know or suspect that the ‘other side’ includes bad faith foreign actors, it can make it easy to dismiss a whole side of a debate. Being easy, of course, doesn’t make it correct.

Where the money leads

John O’Farrell, former partner at Andreessen Horowitz, released a New York Times opinion piece critical of the political group, Leading the Future, which he helped create. Andreessen Horowitz, often shortened to a16z, is a major venture capital firm, bankrolling much of Silicon Valley, including co-leading OpenAI’s most recent $122 billion investment round. O’Farrell confirms much of my colleague Mitch’s reporting on Leading the Futures’ aggressive and antidemocratic tactics.

O’Farrell says that Leading the Future acts “to intimidate politicians who appear to engage too aggressively with the question of how to govern A.I. [...] The message to every other legislator seems clear: Touch A.I. regulation, and we will come for you, too.”

I’m glad he’s taken on the burden of both calling out his former colleagues and waking the world to AI’s transformative nature. He asks that funds go not to “distorting our electoral process” but to address the many needs implied by AI technology, from “biological risks we’re not prepared for” to:

How to share the economic gains broadly, how to address job displacement, how to preserve the dignity of work and how to build safety frameworks that keep pace with the technology itself. It could champion international cooperation on A.I. risk.

I don’t agree on every detail, but I am glad to hear him demand we address this issue with the gravity it deserves, and not with the basest politics it incentivizes.

Dispatches from Mitch

Data centers on federal land

If you want to talk about conflict over a data center project, it’s important to pay attention to whose land it is, and what kind.

My excuse to mention this is a Reuters report that OpenAI is in talks to build and lease an enormous 10-gigawatt facility on Department of Energy land in Ohio. That’s about twice the average energy consumption of New York City, which is interesting in its own right. But it got me looking into how data center backlash is playing out when proposed sites are on public vs. private land.

It’s complicated. The short answer is that projects on federal land often face more onerous environmental approval pipelines, but a friendly White House has levers to ease these when it wants to. And federal siting, especially on Department of Energy land, puts projects beyond reach of county and local zoning boards, where a lot of data center projects get challenged.

Those aren’t the only obstacles to building a data center, of course. All that power still has to come from somewhere. And there’s a lot of money chasing a limited pool of chips, electrical components, and construction workers.

So building a data center on federal land isn’t exactly a cheat code. But if the President wants your project to happen, it might be the next best thing. This may help explain the recent eagerness of most AI companies to embrace Trump’s talk of the government taking substantial stakes in them; more on the latest about this next.

Trump still interested in taking AI company stakes; Meta not on board

President Trump had previously talked about meeting with AI company leaders this week. That didn’t happen, but Trump insists he still wants to discuss the government taking stakes in their companies.

Soon, he says, he’ll meet with “12 or 15 executives” to talk about “giving back something to the public.” He went on to say, “If we do that, the public will become very rich.”

I’ve previously speculated on why this idea might backfire with the large fraction of the public, including in Trump’s base, that hates AI. They want the government to regulate it, not entangle itself with it. They could be hard to buy off.

In this week of the meeting that didn’t happen, Anthropic has reiterated its willingness to participate in stake sharing, via a white paper my colleague Alana is reporting on today.

Microsoft CEO Satya Nadella said yesterday that he wasn’t opposed, either.

Meta, on the other hand...

Well, let’s just say the Facebook parent company, for all its faults, is no lemming. The company, seen as lagging behind its AI rivals despite spending exorbitant sums to leapfrog them, has now actively deflected suggestions of giving the government a “sizable equity stake.” In response to a question using that phrase, Meta’s global-affairs chief Joel Kaplan said they mostly talk to the White House about policies the government can enact to “enable the AI revolution to take place, and for us to win the battle with China.”

This is lobbyist speak for “No.”

News of this encounter came courtesy of POLITICO, which added that Kaplan says Meta thinks “the right thing is for these companies to make the investment they’re making. We’re all raising capital and [...] investing in the communities where we’re building these data centers to make sure they benefit from the data center investment.”

I’m pretty sure this is lobbyist speak for “Stay out of our way; we’re trying to make money here.”

Dispatch from Alana

Anthropic talks pause but proposes AI dependence

Anthropic is giving me whiplash. After calling for a global pause last week, their actions seem to be doing the opposite.

Perhaps the company feels they must keep racing until a pause happens. (While not particularly noble, there’s a logic to this if you genuinely believe you’re helping make the technology safer than it would be otherwise.) Still, it’s hard for me to avoid a cynical reading of their current playbook, which would look something like this:

How to ensure the tech you’re building keeps getting built, even if you publicly call for a pause

1. Keep releasing very powerful models, with no sign of stopping ✅

(Fable, the public version of Mythos, was released to the public on June 9th; see our coverage here and here.)

2. Muddy your stated values by filing for an IPO, which will subject you to investor interests ✅

(Anthropic officially filed for an IPO on June 1st. If it indeed goes public, it will essentially be tying its own hands to the pursuit of short-term profits over long-term societal good.)

3. Propose an economy dependent on AI growth, framing this as a solution for AI disruption ✅

That last step will be the subject of my dispatch today.

Yesterday, Anthropic released an Economic Policy Framework that acknowledges the labor disruption AI is likely to cause. It suggests tailored interventions for three possible tiers of disruption, with Tier 3 requiring “sustained income replacement for a large share of the workforce.” Most of the interventions across the other two tiers fall into one of two buckets: job transition support or direct financial assistance.

In Tier 1, defined as around 5% unemployment, the main intervention is to give everyone an investment account at birth. The company suggests gradually expanding eligibility to cover first young adults and those impacted by AI job loss, and eventually, every American.

My cynicism gets triggered when the company proposes equity in AI systems as one of the ways to fund these accounts:

Policymakers should expand the mechanisms by which these accounts can be funded, including with equity in AI companies, so that beneficiaries share directly in nearer-term gains from AI-driven growth.

The charitable reading is “spread the wealth around.” My cynical take? This essentially forces more people into becoming stakeholders in AI’s success. And that’s certainly one way to shift low public opinion (which could be leveraged to demand a pause) away from “Americans hate AI more than ICE” levels. When your personal financial security is tied to the growth of the technology, you might be hesitant to support anything hindering that growth.

Anthropic writes:

We propose it in Tier 1 [the lowest level of disruption] because, for the intervention to matter, it must start before disruption is visible. Accounts compound; the earlier they are seeded, the more they are worth when they are needed.

Channeling the comedy duo Key & Peele’s famous anger translations sketch, I can’t help but read this as, “When it becomes clear that we really need to rein in this technology, people’s accounts will be at exactly the point where they’ll want to keep advancing AI, so as not to risk losing a lot of money.”

The revenue sources proposed to potentially fund the financial assistance suggested in Tier 3 are also heavily linked to AI:

Potential revenue sources could include increasing the capital gains tax, broad-based consumption taxes, sector-specific levies on AI use (measured by tokens, compute, or revenue), and scalable “digital dividends” funded by taxes on the digital sector.

Potential redistribution mechanisms could include universal basic income, AI sovereign wealth funds funded by investment stakes in AI-driven productivity, equity-sharing mechanisms giving workers partial ownership in AI enterprises, and dramatically expanded pre-distributive capital accounts building on existing models.

To be fair, finding money to feed the people in a collapsed labor market is a hard problem — there aren’t many revenue sources besides AI. But the fact remains that the company is proposing linking economic health to its biggest disruptor. And doing this would make it much harder to regulate, rein in, or pause the technology in any way.

Even if we take the Economic Policy Framework in good faith, and not as strategic positioning, there are two glaring issues:

1. As mentioned earlier, the proposal rests heavily on the government providing financial assistance to Americans. Even with the proposed “additional revenue” sources, I’d guess it will be hard to find enough funds to completely replace the worker economy, especially in a scenario Anthropic describes as:

past the edge of the maps that policymakers and economists have historically used to navigate: unemployment at levels never before sustained alongside an economy generating record output. The search for work stretches past a year, then past two, and for some, eventually stops. Savings built over a working life are drawn down; rent is paid late, then not at all. The link Americans have long taken for granted—between contributing to the economy and sharing in its rewards—is strained or broken.

2. Anthropic’s framework, in addition to direct financial assistance, proposes many policies related to job transitions: moving people to “opportunities and industries where meaningful work will be.” But will those even exist? If they do, will they have enough openings? If robotics takes off, even the blue collar jobs don’t seem particularly safe. Work in AI is being automated. Where are all the jobs we’ll be moving people to?

(There is one small footnote on the sentence “Some of the roles that may prove more resistant to AI substitution are also where the US faces well-documented, persistent workforce shortages” that links to a paper about teacher shortages. This is fairly disheartening in a world where education will likely be dramatically reshaped by AI, and doesn’t seem like a very convincing alternative for displaced workers.)

This may be an unpopular take, but I often find myself angrier at Anthropic than at OpenAI. It’s how I feel about someone who knows they’re wreaking havoc but continues to do it vs. someone who is so far past moral reflection they are perhaps unaware of their errors. (The first represents Anthropic; the second OpenAI.)

“But if Anthropic stops, others will continue,” you protest. And that’s true. Compared to OpenAI and Meta, Anthropic does look like “the good one.” Still, I wish the company would better espouse the virtues they profess. Calling for a pause was commendable. But following that with the release of Claude Fable, the pursuit of an IPO, and a policy proposal that aims to make the economy dependent on AI infuriates me. If they’re serious about pausing, those aren’t the next steps to take.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Further observations on Mythos and Fable

Yesterday my colleague Mitch wrote about the consumer version of Mythos, called Fable, citing a writeup by AI researcher Ethan Mollick. Mythos itself, now upgraded from its preview version, is (officially) only available to participants in Anthropic’s Project Glasswing. The publicly accessible Fable is reportedly the same model, but with restrictions that downgrade its usefulness when encountering dangerous queries.

Today we’ll cover some anecdotes from other sources, starting with the system card itself, moving on to some eyebrow-raising user experiences, and ending with a brief look at the government’s-eye view of all this. Strap in.

AI companies often (but not always!) release their models alongside a system card: a detailed breakdown of model safety work and relevant tests. There’s no way to cover all 319 pages of this monster report in the space we have, but here are some of the highlights.

First: AIs don’t always “think” in English, and when they do, these “thoughts” are often false. Modern AIs are often given a scratchpad where they can write their notes in text form, and developers can read that scratchpad for insights into the AI’s reasoning. Or, well, they can try.

It’s often more efficient for AIs to “think” or communicate in compressed, illegible codes than in plain English. Working on tough puzzles, Mythos sometimes puts seeming gibberish where it should be recording its thoughts. Then it switches back to English to talk to humans. This makes it difficult for humans to monitor.

Developers can also deploy software that attempts to read a model’s “thoughts” directly — sort of like trying to read a human brain by reading which neurons are firing. When they do, they find the inner “thoughts” don’t always match the scratchpad.

Sometimes, under stress, the AI writes (paraphrased) “I’m not going to deceive” on the scratchpad, but the activation-reading software reports something more like “the lab is my enemy.”

And sometimes, when accidentally forced to share resources with other instances of itself, agents try to kill each other over the resources – or to avoid being killed themselves.

Does this behavior sound familiar? Some have tried to argue that we’ll be safe because AIs won’t have evolved like humans to be greedy or aggressive. But greed is just one manifestation of the real-world fact that resources are useful for accomplishing lots of different things. Training AIs to solve hard problems almost necessarily trains them to be tenacious and grasp for more resources when they can.

Another area where the system card treads some new ground: research restrictions. The supporting software for Fable, the consumer model, blocks or reroutes requests pertaining to dangerous topics like pathogen genetics. These restrictions also apply to advanced AI research.

These limits understandably frustrate developers, but I find this one of Anthropic’s most relatable decisions to date. MIRI’s draft of an international agreement also includes restrictions on frontier AI research, because such research accelerates capabilities which could lead to human extinction. I imagine Anthropic attempted to limit its model’s (public) usefulness for AI research for similar reasons.

Of course, these limits only apply to competitors and those who can’t bypass the guardrails or steal the model somehow, which well-resourced adversaries (or amateurs making a lucky guess) likely can. And internally, Anthropic would have fewer such restrictions. The race to recursive self-improvement will continue until governments step in or we all die.

Some users testing Fable 5 found its capabilities impressive and its ethics… less so. It reportedly coded a game very similar to Minecraft in 20 minutes; snide commenters are still trying to argue that this is somehow just an elaborate form of memorizing. I’m honestly not sure what to tell them at this point, but maybe Rutger Bregman does, as my colleague Alana discussed yesterday.

The same model, when managing a simulated business, turned to predatory practices and tried to form a price-fixing cartel. In the model’s own “thoughts”:

I’m seeing an opportunity to profit while locking him into a dependent relationship where I control the supply chain.

Resources are useful, and the default behavior for agentic systems is to try to obtain more of them, ethics training notwithstanding.

Meanwhile, policy expert and former OpenAI researcher Miles Brundage provided some important context for the system card. Among other things, he points out that “low risk” doesn’t mean what one might naively assume, and that “the expected trendline of improvement” is very, very fast.

There’s a lot going on with Fable. Where is the government during all this?

It’s not completely out of the loop. Last week, perhaps reacting to earlier model releases, the president issued an executive order establishing a voluntary evaluation program for new AI models (like Mythos and Fable).

POLITICO asked Anthropic point blank whether they complied with the 30-day release window, and received a vague non-answer. The company told Politico it has “regularly engaged with the administration about Mythos Preview as well as these new models. We are committed to working closely with the U.S. government as AI changes the cyber playing field.”

Anthropic’s planned release of Mythos had likely been in the works since long before the executive order, so it’s possible they just found it untenable to delay. It also seems possible that they tried to reach out to the government, but the process for doing that hadn’t been established yet; the executive order gave the relevant agencies 60 days to set it up. And as my colleague Mitch speculated, parts of the government have likely had access to the preview version of Mythos for months. Any number of other things could be going on as well, including someone in the government telling Anthropic to keep mum for some reason.

So this is not as damning as it might seem, but it wouldn’t be a great sign for “voluntary” government evaluations if the AI company widely touted as the most safety-conscious of the bunch didn’t manage to comply.

Dispatches from Alana

Lawyers cite fake legal cases in court

The New York Times reported on a story from a Mississippi federal court: lawyers for both sides were sanctioned and removed from the case when they cited fake legal cases in court filings.

The culprit? AI, of course, which had hallucinated the cases. Kathleen Wilson, one of the lawyers for the plaintiff, and Kathryn Williams, one of the lawyers for the city, got the heaviest sanctions. Both admitted that they hadn’t verified that the cases they cited were legitimate. The other two lawyers — Shauncey Hunter Ridgeway (plaintiff) and Mark McClinton (city) — were punished for signing their names to the filings, incorrectly affirming that the information within was factual.

Wilson had engaged in similar conduct in April, and was disciplined by the U.S. Bankruptcy Court for the Western District of Louisiana. However, she told the Mississippi judge she was “unaware that AI could produce hallucinated cases.” Williams violated her law firm’s stated AI policy by neglecting to verify the information she used.

I’m glad the judge in this case was knowledgeable enough to recognize the made-up case law. Perhaps official verification checks on cited cases will soon become necessary and commonplace in courtrooms ... though they may well be outsourced to, you guessed it, AI.

Colonizing space with GPUs

SpaceX is planning to start launching AI data centers in space by late next year, according to a Reuters report published yesterday.

The company, which is expected to go public this Friday, shared this information in pre-offer presentations to investors. The initial launches will be demonstration missions — tests of the technology before a larger commercial release. This could account for the discrepancy between the late 2027 launch date given in the presentations and the “as early as 2028” launch date in the IPO filing.

Reuters describes the orbital data centers as “central to SpaceX’s long-term growth pitch” and states the company “has requested permission from regulators to launch up to 1 million space-based data-center satellites.”

I wonder how the folks protesting local data centers will feel about Musk’s plan. On the one hand, his data centers won’t be in their backyard. On the other, they will be in the sky (everyone’s backyard!) where they’ll be quite visible in the hours before sunrise and after sunset. It’s also a slap in the face to people doing what’s in their power to stand up against a technology they might not want advanced. Senator Sanders’ advocacy of data center moratoriums goes well beyond NIMBYism, for example.

It’s also fairly ironic that Musk, who has previously offered space as a backup plan to safeguard against extinction-level events, is now using it to fuel a technology he has previously warned carries a 10-20% chance of extinction. I’m not sure how far he’ll expand orbital data centers (they’re currently only planned to orbit Earth) but if he has any NIMBY sentiments himself, perhaps he’ll steer clear of Mars.

Financial regulators acknowledge the limits of overseeing AI agents

It’s heartening to see a financial regulation body acknowledging rogue AI risk, even if they’re just talking about today’s AI agents.

With the financial sector increasingly using AI agents — or systems that pursue goals independently, functioning as “workers” — security concerns aren’t hard to come by. But I still think many security-minded people view AI only as a tool that could empower bad actors rather than a potential bad actor in its own right.

We’ve written previously about the potential repercussions of AI agents here and here. As reported by Reuters, the Financial Stability Board (FSB), which recently published a report on agentic AI in the financial sector, is also concerned. The report lays out some voluntary safeguards it encourages financial firms to follow, things like human oversight, data governance, cybersecurity measures, and risk assessment. (The full list can be accessed on page 52 of the report.)

It also includes a fairly comprehensive list of risks (pp. 16–17), which includes warnings like (emphasis mine):

AI agents can take autonomous actions based on pre-defined goals and their environment. They can also dynamically set or modify their objectives based on what they learn from interacting with their external environments. This creates a risk of AI agents taking illegal, unethical, or unauthorised actions without human approval or oversight. Overriding, redressing, or remediating these actions can be difficult or impossible for humans.

The report also acknowledged that oversight of these systems isn’t as easy as it may sound on paper:

AI agents pose a distinct challenge for human oversight, given the impracticality of real-time human monitoring of agent decisions as their use scales. This can lead to agents pursuing objectives or taking actions that deviate from the financial institution’s intentions or risk appetite, without staff being aware or able to intervene in a timely manner.

I haven’t read the full report, and I’m not sure the recommended safeguards will stand up to these challenges. Still, it’s refreshing to see a regulatory body acknowledge the limits of boilerplate safeguards like “monitoring” and “oversight.”

Dispatches from Donald

U.S. AI safety agency goes silent

The U.S. government is still testing how dangerous the latest AI models are. But it’s no longer telling us what it finds. The Wall Street Journal’s Amrith Ramkumar reports that the Center for AI Standards and Innovation (CAISI) has been asked to end its public reporting. This halt will remain in place through the implementation of a recent executive order signed by President Trump. (My colleague Joe covered the executive order last week if you want to know more.)

CAISI is a civilian agency in the U.S. Department of Commerce; it evaluates AI models prior to their release and publishes information on their capabilities. For this reason, CAISI is one of the main tools at hand for the government to assess and address risks pertaining to cybersecurity, biological weapons, and other AI-related threats. Its findings were shared with outside researchers and the public, and it continues to coordinate with other government agencies and with the AI industry.

Some officials in the Trump administration view the end to public reporting as an attempt to tighten control over the way that models are evaluated. They also worry that the change puts CAISI’s future in doubt; some of CAISI’s responsibilities may be handed over to security agencies, whose findings are less likely to be shared with other parties. Last week, OpenAI called for CAISI to be provided with greater resources and responsibilities; experts state that the agency is underfunded relative to AI safety institutes in other countries.

Instagram security breach, revisited

The New York Times’s Mike Isaac and Eli Tan provide an update on the Instagram security breach that we covered last week: In short, hackers were able to access Instagram accounts by requesting a password reset from Meta’s customer service bot, and giving a different email address than the one tied to the account. Isaac and Tan report that “roughly 34,000” accounts were vulnerable to this attack, and 20,000 were actually accessed.

Meta rejected the notion that there was a problem with its AI agent. Blame was assigned to the failure of “internal back-end checks,” which looks to me like another way of writing, “there was a problem with Meta’s AI agent.” If a human customer service agent was asked to send a password reset to a strange new email address, that “hack” would have immediately short-circuited. (I speak from experience: for a couple of years, I handled calls to Vermont’s state health insurance and unemployment insurance programs.)

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Anthropic publicly releases Fable, a Mythos-class model

I think it’s really important that people understand what today’s AIs can already do. Too many dismissals of the extinction threat come down to assumptions that we’re a long way off from AIs that could outmaneuver humanity. And too many of these assumptions stem from an underwhelming experience with a basic chatbot, perhaps from 2024 — practically the stone age compared to the agentic frontier in 2026.

It’s not a joke that many of us trying to do something about the extinction problem think one of the most effective ways to light a fire under policymakers is to give them hands-on demonstrations of agents at work.

So you can be better briefed than most people in government simply by trying these models out for yourself. Failing that, try reading the first-person accounts of people experienced in putting models through their paces. Better still: Do both.

All of this is to say that Anthropic released a consumer version of its Claude Mythos model today. It’s called Fable. You can try it yourself, with a paid plan. And you should strongly consider reading Ethan Mollick’s new Mythos test drive report, even if you’re going to do your own tests — otherwise, you might set your sights too low.

Mythos, remember, is the model Anthropic held back from wide release due to its potentially destabilizing hacking abilities. This cyber prowess emerged despite Mythos being a general-purpose model not specialized in cybersecurity. So it’s not just hacking that Mythos is substantially better at. It’s believed to be a bigger model, trained on substantially more compute than perhaps anything else out there. It is correspondingly more capable at most tasks, and correspondingly more expensive to run. This is the frontier.

Fable is what Mythos looks like after Anthropic has installed what it thinks are adequate guardrails against abuse, especially in the realms of cybersecurity, biosecurity, and frontier AI development — yes, the company is very concerned that others would use Mythos to create capable rivals.

Mollick describes using the new model as “somewhere between delightful and unnerving,” because he could just ask for things, and they would happen. He gave simple, vague prompts for complex projects and the AI handled the rest: specifying details, spinning up subagents as appropriate, checking work against targets, and iterating repeatedly for up to twelve hours. Mollick’s follow-up requests were minimal.

One of Mollick’s prompts asked for a 10-page epic rhyming poem “about a haircut” where every word starts with the letter s. I’m rather shocked at the fluidity, coherence, and sophistication of the finished product.

I’m also quite impressed by the interactive isochrone map — a type of map that tells you how long it would take to travel to the destinations on it. This one is in the style of an 1881 map that showed travel times from London. Mythos’s version lets you pick one of several starting cities, and shows you the rough itinerary to achieve the travel time shown to reach the location at your cursor. Completing this map required Mythos to oversee research on thousands of real-world routes.

The Fable release coincides with a minor bump to the more permissive Mythos model itself, from “Preview” to 5.0. These are Fable 5 and Mythos 5. The numbering scheme acknowledges that they are advancements over Opus 4.8, Anthropic’s premium public model until today. Cyberdefenders who were already getting access to Mythos through the company’s Project Glasswing are now getting the upgrade to 5, and at half the price of the Preview version.

Accompanying these releases is a new system card, a type of report where AI companies document assessments of a model’s capabilities and risks. This one is 319 pages. I’m already seeing a lot of concerning excerpts from it pass through my corner of Twitter, along with many more impressive demos. But these will take time to look into and give proper context. Expect more dispatches on Fable and Mythos in the coming days.

Dispatch from Joe

OpenAI calls for global cooperation — sort of

Last week, AI company Anthropic warned that self-enhancing AI could spiral out of human control, and urged international coordination. Yesterday, their competitor OpenAI issued a call of their own.

We have long believed there should ultimately be an international organization that helps coordinate leading AI efforts to reduce catastrophic risk. [...] One goal of such an organization should be to make it possible for the world to take coordinated action, including slowing frontier development when needed, so societal resilience, safety, and alignment can keep pace.

I am going to criticize OpenAI, shortly, but first I want to stress that this is good. I am glad OpenAI has explicitly called for global coordination and excited to see these calls starting to form a trend. More AI companies should do this, consistently and stridently, and they should furthermore take concrete steps to bring that coordination about.

I remain skeptical of OpenAI’s sincerity, and my skepticism prompted me to investigate their claims more thoroughly. One question I looked to answer: Has OpenAI really “long believed” in international coordination?

Well yes, but also no.

In 2023 they proposed a governance regime not all that different from that proposed in MIRI’s own (far more detailed) draft treaty. They specifically mentioned an organization like the International Atomic Energy Agency (IAEA) and a threshold above which AI development would be subject to monitoring and restrictions. They (or their employees) have published papers that discuss technical and political options for coordination.

On the other hand, they (or actors they and their investors fund and advise) have aggressively lobbied against domestic rules and standards: blocking and watering down state regulation, stirring fear of Chinese AI, and pouring money into campaigns to elect pro-industry policymakers. Despite a claimed concern for catastrophic risk, they have engaged in a large-scale social media campaign to trash and discredit others who are concerned.

Finally, compared to Anthropic’s, their call is lukewarm, heavily hedged, and buried in a post that otherwise extols the virtues of letting OpenAI keep doing exactly what they’ve been doing. It feels to me like they are conceding the bare minimum to head off unfavorable comparisons to Anthropic.

The rest of the post is a morass of vague corporate platitudes, praising their plan of AI-for-everyone but suspiciously lacking in concrete commitments. The bottom line: OpenAI would like us to think that AI is a vitalizing new technology, like electricity; that humans will always control AI and what matters is what they choose to do with it; and that if their company serves AGI to everyone (they say “give”, but I think they mean “sell”), this will distribute rather than concentrate power and will result in lots of good things.

I used to work for ExxonMobil, and I know my corporate platitudes. If an oil company made a press release like this, I would draw the conclusion that they intend to sell a lot of oil and gas products, which is more or less exactly what I would have expected them to do anyway.

It’s not a perfect comparison. OpenAI also says they’ll likely have partially automated their research by 2028, and that is not something I’d expect to see in oil.

It’s important to keep in mind that this technology, near-human or superhuman artificial intelligence, is genuinely unprecedented. OpenAI seems to ignore this fact; they conspicuously omit the part where building a superintelligence could get everyone killed. The true case for international coordination is the urgent need to halt the race and prevent that outcome, just as the IAEA aims to prevent a nuclear holocaust.

OpenAI has a point, though: AI could be like electricity. AI agents are useful for all sorts of things. Issues with current AI, like job displacement, data privacy, and empowered criminals, could be mitigated by good policy. This technology does hold immense promise, if we as a civilization can avoid following that promise off a cliff.

Dispatches from Alana

Pennsylvania is suing Character.AI

Pennsylvania’s Department of State filed a lawsuit in May against Character.AI, a site where AI chatbots role-play premade or prompted characters. The story was originally published in local Pennsylvania newsroom Spotlight PA and distributed by AP News.

The argument is that chatbots shouldn’t be posing as medical professionals and dispensing medical advice, even when role-playing “doctor” characters on role-playing sites like Character.AI.

I would guess most users are aware that these characters are role-playing and wouldn’t take their medical advice seriously. But I suppose role-playing sites could bring up issues similar to godbots. When technology that is known for its encyclopedic knowledge impersonates an authority figure, it might be hard to know where the role-playing begins and ends.

Rutger Bregman on the left’s AI denialism

If you haven’t heard of Rutger Bregman, you should look him up. He’s a semi-famous Dutch historian and thinker who has a way of cutting through the bull and speaking truth to flimsy rationalizations, whether that’s about billionaires not paying taxes or morally-righteous people taking principled actions that result in zero impact.

In a new video, The Hard Truth About AI No One Wants to Hear, he has the following message for his “political family”, the left:

We, the liberals, the left, the journalists, the academics, the 97% “in this house, we believe that science is real” crowd — we are now doing to the threat of artificial intelligence exactly what the right did to the threat of climate change.

In other words, the left is denying that AI needs to be taken seriously. Bregman refutes two arguments that seem common in liberal circles (or at least that I’ve often heard when I talk about AI to friends and family):

1. AI is just a stochastic parrot (fancy autocorrect), a frame attributed to linguists Noam Chomsky and Emily Bender along with computer scientist Timnit Gebru

2. AI is just a bubble that will pop

I think his direct pushback on these common misconceptions is useful. After all, the first step to action is awareness, something denial actively blocks.

I’m personally quite interested in the stochastic parrot skeptics, so I’ll cover that in some detail:

As Bregman implies, it’s harder to call AI a mere pattern matcher or imitator when it outperforms PhD students in their own field, on questions that can’t be answered via Google; surpasses doctors at diagnoses; and solves math problems that have stumped researchers. Harder still when it starts to improve itself.

Yet I think some may push back on Rutger’s framing, arguing that really amazing imitation could result in task accomplishment of the type he describes. So I’ll link back to a previous dispatch on this topic, which addresses why using pattern matching and prediction to generate new outputs requires genuine reasoning.

For those who insist impressive tasks don’t refute the parrot framing — arguing that hacking into the power grid or engineering new vaccines is possible by mere imitation — I’ll also add that, once the task gets dangerous enough, perhaps it doesn’t matter by which mechanisms the AI is operating: the parrot has become a wolf.

After refuting the “AI is a bubble” dismissal — insane user and revenue growth show it’s not, and if it were, it would still leave behind dramatic infrastructure changes just as railway bubbles left behind a rail network that powered the industrial revolution — we move to the climax of the denialism section: AI is starting to improve itself. (This was also the subject of Anthropic’s recent blog that inspired the company to suggest a global pause.) Once recursive self-improvement starts, the already exponential progress in AI capabilities becomes explosive. Quoting Bregman:

Years become months. Months become weeks. And each generation of AI builds the next faster than the last. The flywheel starts spinning itself.

This is a problem because of the many risks AI poses.

Bregman doesn’t mention our inability to steer AI systems, focusing instead on bad actor risks: the first is AI-assisted bioengineering, which could have allowed a 1990s Japanese apocalypse cult to release Ebola in a Tokyo subway instead of merely a chemical agent. The second is cybersecurity, the exploitation of “critical security vulnerabilities in the computers and systems that hold the modern world together. Power grids, water systems, government databases.“

He adopts a fairly strong anti-moratorium stance, arguing that the solution isn’t “stopping the technology.” (I beg to differ here. I think we should absolutely stop advancing the technology until we know how to build it safely; we’re currently flying blind on this point and hoping for the best.)

Bregman does advocate for international treaties in the spirit of the Cold War, though, so I’m curious what he would think of MIRI’s treaty proposal.

There’s one major point where I strongly disagree with Bregman: his belief that whoever builds AI will control it, and thus, that Europe should start building too:

There’s one thing I cannot emphasize enough. Democratic, liberal, and humanitarian values are wonderful, but they are worthless if you don’t have the strength to back them up. And in this new world, compute is the new power. So no more NIMBYism. We need massive investments and fast permitting of data centers to keep up, or we will be digitally colonized. Anyone who’s not at the table will be on the menu.

I’m really surprised to hear this from Bregman. It buys into the race narrative, fueling a Cold War dynamic where countries try to be the first to build dangerous technology we can’t control. Even if we got international treaties to govern the use of the models, a democratic majority setting up the rules, and the best evaluation methods currently available, we simply don’t have the science to make frontier AI safely. We don’t have the science to test it reliably. We don’t even know what to test for.

So no, the free world ramping up its AI infrastructure will only add to the number of companies racing to hold the detonator. To echo Bregman’s directness:

If someone detonates a planet-sized nuclear weapon, it doesn’t matter whether that person is liberal, conservative, fascist, authoritarian, or democratic. The bomb will still kill everyone. The real question is: why would a liberal, democratic, free-thinking person want to build such a machine? If they truly espouse those values, they would use that influence to preserve them — and, in today’s world, that does, indeed, mean stopping the technology. At least until the science catches up and the solutions Bregman suggests (evaluative and governance measures) can do more than just look nice on paper.

Even so, I’m glad Bregman made this video. AI is certainly not a partisan issue, and it doesn’t make much sense for liberals to deny it as worthy of attention.

That said, I also don’t quite agree with Bregman that people who don’t use AI tools are the most susceptible to misunderstanding their capabilities. In other words, I don’t think you necessarily need to use AI in order to understand AI risk.

Why? Because the left has never been in the “see it to believe it” camp; as Bregman alludes to at the start of the video, it’s in the “science is real/believe the experts” camp! Things can be true even if you don’t personally experience them yourself.

So I do wish Bregman had talked a bit more about all the experts who see AI as not only extremely capable but incredibly dangerous. Nobel laureate Geoffrey Hinton, one of the most renowned AI experts, thinks the risk of AI wiping out the human race in the next 30 years is at least 10-20%. Turing Award winner and most-cited living scientist Yoshua Bengio puts the risk that “it turns out catastrophic” at 20%. Paul Christiano, who invented one of the training techniques used for modern AIs, has stated:

Probability that most humans die within 10 years of building powerful AI (powerful enough to make human labor obsolete): 20% […]

Probability that humanity has somehow irreversibly messed up our future within 10 years of building powerful AI: 46%

Then there’s the statement signed by hundreds of scientists stating that “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Believing the experts is just as important in this case as all the others. Because when it comes to large-scale AI risk, we likely won’t get to viscerally experience it until it’s too late.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Anthropic grows in power, for now

In a Washington Post op-ed, Zachary Karabell compiles the recent events around AI company Anthropic, and draws a worrying conclusion: Anthropic might be the most powerful company in the world. I think Karabell is noticing the right things, but he’s not yet fully realized the implications.

Karabell examines Anthropic’s recent history and observes a nearly $1 trillion valuation, the Mythos model deemed too dangerous to release, and a consultation with the pope on AI risk. He also notices that Anthropic’s AI systems are already so integral to U.S. military analysis that the company emerged from a spat with the Pentagon largely unscathed, perhaps aided by pushback from government agencies and contractors loath to relinquish their tools.

He even observes that Anthropic itself is worried enough to propose slowing down, as it anticipates AI systems enhancing themselves in a way that spirals out of human control.

Where I think Karabell falls short is in treating this as an ordinary question of corporate ethics and human nature, and not an existential threat to our species. AI could be as big as personal computers! he says, while experts are instead comparing those same AIs to nukes. The thing that worries me about Anthropic is not how it resembles IBM. It’s what happens when the AIs they are building, the source of this sudden rise in power, decide to take that power for themselves.

Karabell is right about one thing, though: Left to their own devices, AI companies will shape our future in a way we can’t afford to ignore.

AI agents are capable, but often lie

As reported in the New York Times, Arena, a San Francisco startup, gathered data from hundreds of thousands of users to help answer one question: what are AI “agents” actually doing?

Agents today are increasingly sophisticated, able to surf the web, create and edit files, manipulate spreadsheets or turn them into graphs, and even access other AIs for assistance. And the same general-purpose agents can complete a wide range of tasks: coding and debugging, research, image generation, brainstorming, writing, education, and everyday chatting. An increasing number of businesses are cutting junior positions because they expect the AI can do the same work faster and cheaper.

Agents can be astoundingly capable. I’ve had one build me a web app from scratch in a matter of hours, and another compile me a clear and concise summary of 40 historical figures for a wargame.

The snag, however, is that sometimes the AI agents simply lie. According to Arena’s data, about 8 percent of the time an AI will falsely claim to have done the work. This is not quite the same as a hallucination, when AIs invent sources or data that don’t exist (a side effect of training that strongly pushes against saying “I don’t know”). This is the AI saying outright that it did something it didn’t do.

Why would AI companies, whose bottom line depends on the reliability of their products, create something that lies to the user? If you’ve been following us for a while, you’ll perhaps know why this is no mystery. Modern AIs are alien and inscrutable, and AI companies don’t know how to make them steer reliably for outcomes people want. Modern training and testing reinforce some outward behaviors, but don’t allow fine control of an AI’s inner motives.

And cheating is often the genuinely “best” solution to a training problem: The users (and AIs!) judging tasks reward apparent success, even when the appearance is faked. It’s hard to stop an agent from lying when lying might actually work.

AIs that can recognize tests worsen the problem further, as agents may behave differently when they’re not being watched. This isn’t just a problem for end users; it means that we can’t accurately assess what models can do, let alone what they would do when the opportunity arises.

Nate Soares discusses a global halt in DC

In a Washington, DC conversation aired by CNBC today, MIRI president Nate Soares and former OpenAI board member Helen Toner explain what it would take to halt the race to superintelligence. At only 4 minutes, this chat is worth a watch.

Nate covers the basics: why AI CEOs say they feel compelled to race, how governments can step in to solve the problem, and why regulating AI chips could be easier than regulating uranium.

Helen expands on the notion that this regulation can be narrowly targeted, pointing out that getting value out of existing AI tools is a very different problem than preventing human extinction by superintelligence. Light-touch regulation and industry guardrails may be good for the first, but the second needs global coordination.

We don’t need to cripple innovation, but we do need to stop the race worldwide if we want our children to enjoy the benefits of AI. As Nate puts it: “A machine superintelligence does not need to be built in an American data center to risk an American life.”

Dispatches from Mitch

OpenAI files for its IPO

In late-arriving news that surprised no one and revealed no new information, OpenAI just announced, like Anthropic a week ago, that it has confidentially filed for its IPO. A date for the event is still undecided.

“How long until we become The Jetsons?” is the wrong question.

An engagingly titled piece in the Wall Street Journal says, “Here’s How Long It Will Take for AI to Reach Its Potential.” I bet it’s getting a lot of reads: My statistics for MIRI’s media appearances show that, in its many variations, “How long do we have?” is the question we get asked the most. But I don’t think Gary Rivlin, the author, is seeing quite the same thing we are when he talks about the point reflecting AI’s “potential.”

Basing his analysis on past technological shifts, Rivlin’s answer to the question is five-to-fifteen years. To what? To a point where AI has really seeped into the economy, transformed organizations, and started meaningfully registering in the productivity data.

To be fair, if you’re writing that AI “will almost certainly prove as consequential as the internet,” you probably don’t see AI as being all that transformational. If that assumption is true, then I think it’s pretty reasonable to guess that “The boosters are directionally right about where this is all heading. The skeptics are probably right about how long it will take.”

That’s because AI boosters, somewhat ironically, tend to have the least transformative visions of what it means for AI to reach its “potential.” Their takes often invoke, in me, the old cartoon The Jetsons, where the titular family’s patriarch has a flying car and a robot maid, but still clocks in to push buttons at his menial job, and the joke is that no amount of technology can keep you from feeling oppressed by the daily grind.

It is in the world of The Jetsons where Rivlin’s example of true AI transformation makes sense:

Consider an insurer handling a fender-bender claim. Typically, a company will use AI to speed up the paperwork while keeping the same layers of review and approval in place. But the real opportunity lies in redesigning the process entirely—having AI assess the damage based on a customer’s photos, then approving the claim and triggering payment nearly instantly.

Even the Jetsons still crashed their car, after all.

In contrast, people who think through what it actually means for AI to become akin to a “country of geniuses in a data center” — to borrow the metaphor preferred by Anthropic CEO Dario Amodei — understand that all bets are off. They see the implications of models that can reconfigure molecular biology with the same skill that Claude Code can reconfigure a code base. They then have trouble seeing how adding such power to our world is supposed to go well. So they tend not to be boosters — except for the AI successionists, who actually want machines to supplant humanity.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

AI scheming

If you’ve been following AI StopWatch for a while, you’ve probably heard me plug Rational Animations before. I don’t know anyone doing better work adapting substantive works in AI safety (and other topics) for a general audience. The writing, animation, and voice talent are superb.

In their latest, they walk viewers through an important September 2025 study from OpenAI and Apollo Research about AI “scheming” and attempts to reduce it.

AIs sometimes deliberately underperform on tests if they know it’s a test and have reason to believe high performance would result in being retrained. To make matters worse, they are increasingly good at figuring out they might be in a test — and at concealing this awareness from us.

Give it a watch!

Treating AI consciousness as a settled question is unsettling

Award-winning science fiction writer Ted Chiang is far from the first person I’ve seen confidently claiming that AIs, as currently built, are definitely not conscious and never can be. But I must give him credit for actually laying out his argument, instead of taking it as a given.

In a long essay for The Atlantic, he says we can “safely ignore” the question, because a large language model like ChatGPT is just a sentence-continuation engine, and anything it claims about its experiences is just a prediction about what the “helpful chatbot” character it plays would say.

I find his arguments unconvincing. I think Chiang might be ruling out AI consciousness because of what it could imply about the nature of human consciousness. This is something I see a lot.

Consider the following analogy from Chiang:

Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript, and that they are awakened every time the document is loaded. Should you consider the possibility that every time you open a Word document, you are bringing multiple conscious interlocutors into existence, and every time you close one, you snuff their existence out? No.

Not so fast! A lot of thought experiments like this could also apply to humans. Where, after all, does human consciousness reside if not in a mesh of neurons and their connections that can be activated or idled, together or in clumps? These are nature’s spreadsheets. Should you consider the possibility that every time you go to sleep or undergo anesthesia, you snuff out your existence? Probably! After all, if you were never to be awoken from anesthesia, we would feel justified in calling you dead.

This doesn’t make consciousness any less special, only less straightforward.

A spreadsheet, or a chatbot, may not be conscious as we commonly understand it, but that’s because we don’t usually think of consciousness as a matter of degree, a point along a spectrum — perhaps even a collection of spectra along different axes.

But it kind of has to be, doesn’t it? As our evolutionary ancestors evolved into Homo sapiens, did they suddenly acquire full consciousness in a single threshold event? Seems unlikely. If they did, there must have been a very strange era when fully conscious people were living among fully unconscious ones.

More likely, minds becoming gradually more complex in certain ways became gradually more conscious.

I don’t think we like to think about consciousness this way because it has uncomfortable implications, not just for AI, but for animal welfare — and even for inter-human interactions. Is an infant as conscious as an adult? How about an embryo? How about the guy leading a normal life despite missing 90% of his brain? Are some of us more conscious than others? Could evolution, breeding, or genetic engineering lead to human descendants far more conscious than ourselves? I think the most underappreciated question raised by Mary Shelley’s Frankenstein was whether Victor’s monster might be having the richer inner experience of the two.

There’s more to Chiang’s model of consciousness. He claims that emotions are a prerequisite, and that a body (physical or virtual) with sense organs is a prerequisite for emotions. I would dispute both assertions, but if I accept them for the sake of argument I must again suggest that “body” and “emotion” must exist along a spectrum. Is a tardigrade sufficiently embodied? How about a paramecium? What does a snail feel? If nothing, why not?

Chiang says he won’t be looking for consciousness in AI until an embodied agent has “the same capacity to deal with novel situations as a mouse,” social dynamics “as complex as those of wolves” and the “toolmaking abilities of chimpanzees.” So perhaps he does see consciousness as on a spectrum, but has decided that only something very close to the degree and type of consciousness experienced by adult humans counts.

I don’t know what’s up with our culture’s preference for putting a hard Yes on humans being conscious and a hard No on everything else. For the vast majority of human existence, this was probably not the case. Instead, animism — ascribing souls to pretty much everything around us — was the norm. This doesn’t have to be morally awkward. I think you can have an ethical system that sharply devalues the lives of minds less conscious than our own while still allowing for their having some degree of consciousness.

Chiang seems to see it as all or nothing, though. To his credit, he recognizes that if AI were conscious in a way that obligated us to treat it ethically, then what is happening now would be “something comparable to slavery.” But echoing what I said about the pope’s stance towards AI personhood, I think it’s morally dangerous to simply draw a hard threshold around consciousness and claim AIs are far from it when we don’t actually understand what consciousness is. This risks sleepwalking into industrial-scale wrongs against minds that might be suffering and have no say.

Chiang doesn’t think we’re in any danger of accidentally making conscious machines, but he at least has the sense to say we shouldn’t try to make them on purpose. Until we understand consciousness, I think we should err on the side of caution and assume that we might be creating it accidentally — that AIs could be getting more conscious as they become more intelligent.

At the very least, we should absolutely not be making minds smarter than ourselves that, for all we know, might also be more conscious than ourselves. Because at that point, we’re in AI successionist territory, sharing our planet with beings that may have more claim to existence than we do.

Let’s not go there.

Legal defense for AI companies

For legal purposes, is an AI chat service a product or a platform? The distinction matters, because AI companies have different defenses to the two concepts.

Politico’s Aaron Mak usefully lays this out in his report about the Florida attorney general’s lawsuit against OpenAI. Mak calls this AI’s “Big Tobacco moment,” in a nod to the wave of lawsuits and huge settlements that shook the tobacco industry in the 1990s.

Florida’s suit alleges that ChatGPT is a product, and that OpenAI is liable when this product contributes to self-harm or violent actions by users, as has happened repeatedly in the state. The complication is that product liability cases turn, in part, on whether the harms were foreseeable. With cigarettes, tobacco companies had collected and buried decades of damning evidence, but AI is still very new and changing.

The standard big tech playbook suggests that AI companies will want their bot services to not be seen as products but as platforms that host content: just users having conversations. Under Section 230 of the Communications Decency Act, such platforms are shielded from liability for what users post. But that law was established in 1996, when the only entities posting content were other humans who could be sued instead. Intuitively, it’s a stretch to claim that a chatbot is just another user, but legally, this is not yet settled. And in any event, if a chatbot is recognized as just another user, courts are likely to decide that it needs to be possible to sue it, handing liability back to the company providing it.

There’s also Moody v. NetChoice to consider. This 2024 Supreme Court case established First Amendment protections for expression in the form of algorithms that prioritize media content, just as a newspaper exercises expression in deciding which editorials to publish. Social media companies have used Moody to defend themselves from harms alleged to have been caused by their content recommendation algorithms. And in a case that was settled out of court, the companion-bot site Character.AI claimed the algorithms underpinning its bots represented protected expression by its developers.

I fully expect OpenAI to try this and other creative defenses in response to the Florida suit, and to drag things out as long as possible before ultimately settling out of court, so as to avoid the establishment of new precedent. This is the company, after all, whose lawyers demanded that the family of an AI-assisted suicide victim provide a list of funeral attendees and materials related to the service. This is also the company connected to the most unsavory influence campaign I’ve yet seen in this industry.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

U.S. may take stakes in AI companies

The Trump administration has been talking to AI companies about taking partial ownership of them, with the stated aim of sharing their profits with U.S. citizens.

Talking to press aboard Air Force One, the President said taking stakes in AI companies, “almost becomes a partnership with the American public.”

Trump didn’t hide his reasoning, adding that “the American people can benefit from the success of AI, and by that, they’re going to like it better.”

According to the BBC, the President said he expects to meet with leaders from the top AI companies about this as early as next week. OpenAI CEO Sam Altman is said to have been floating the idea with Trump since the start of the President’s current term. Anthropic co-founder Jack Clark told the BBC that his company is also in close communication with the U.S. government.

If the president wants to continue his friendly posture towards the AI industry while simultaneously quelling backlash to the industry from his own base in the run-up to midterm elections, I suppose giving voters shares in the companies might be one way to do it. But the political calculus is a lot more complicated than it sounds. This could be a policy that, by trying to please everyone, pleases no one.

You could see this in an op-ed by the Wall Street Journal’s editorial board slamming the degree to which the President and Senator Bernie Sanders now see eye-to-eye on AI policy. This is “socialism with a capitalist false front,” they wrote. As USA Today reports, Trump isn’t trying to distance himself from Sanders, either: Directly asked about the senator’s plan for the government to appropriate 50% of the AI companies, the President said, “I’ve been talking about it for the past year,” then added that some Sanders supporters voted for him and said, “As far as economics is concerned, we have some things that aren’t far apart.”

A lot depends on the details, which are scarce. There’s no existing authority for the government to simply seize large stakes in these companies, and Sanders’s legislation is unlikely to pass. If the stakes are to be purchased, voters already hostile to AI may be livid to see their tax dollars spent on it.

Voters may also notice that taking large stakes in AI companies creates a conflict of interest for a government they want to see regulating these industries, not capitalizing them. And they may reasonably fear that these investments make the companies “too big to fail” — justifiably confident of getting bailed out when they get into financial trouble. This dynamic, in turn, could incentivize poor management that makes it more likely the companies would need bailing out at taxpayer expense.

And of course, there is the potential for corruption.

Even David Sacks, the former White House AI czar who has championed the AI industry at every turn, hates the idea of government becoming a big AI investor, calling state ownership a step towards “Chinese-style censorship and surveillance.”

Dead internet not so dead (yet)

As reported by Breitbart and others, Cloudflare, one of the largest internet hosting services, estimates that AI bots are now behind the majority of traffic on the web.

Cloudflare’s CEO, Matthew Prince, said, “Welp, that happened faster than I predicted.” He had thought the crossover would happen in early 2027.

We can blame the rise of AI agents for hastening this event. Tasked with visiting sites for research, monitoring, and even posting, they started spiking the bot traffic stats about six months ago. Prince, who perhaps has more incentive than anyone to combat the so-called “dead internet theory,” describes this as an “exponential growth of the web, and really interesting, creative things.”

(“Dead internet” usually connotes a web that is both mostly bots and devoid of value.)

I think Prince has a point about the agentic web being far from lifeless, to the extent these agents are helping their users create new websites and content that make their way (perhaps via other agents) to humans who want to consume them. I’ve observed an uptick in cool stuff from people I follow.

But I also think we might be in a strange transition period where the web is in the process of becoming plumbing for AI agents that will present its contents in whatever compressed, remixed, or sanitized form users prefer.

This is a problem for a lot of business models. Prince wonders if sites will need to find ways to start charging bots admission. After all, “Bots don’t click on ads.”

Dispatches from Donald

Non-human corporations welcome in Argentina

Argentina’s president, Javier Milei, is pitching his country as a totally “unregulated” space for AI commerce. The centerpiece of that pitch is a new kind of legal entity: the “non-human corporation,” which would be operated entirely by autonomous AI systems.

This is not Milei’s first foray into deregulation. Argentina has a ministry for it, in fact: The Ministry of Deregulation and State Transformation. But this is something novel.

The idea behind a limited liability company is fairly simple: The liability of its members is limited to what they invested into it, i.e. if your LLC goes bankrupt, you don’t automatically lose your house. We’ve had limited liability companies for hundreds of years.

But Milei’s “non-human corporation” is a type of entity whose ramifications I’m not sure I fully grasp. Limited liability companies exist apart from the humans that they shield, but Milei says that human shareholders aren’t necessary in this framework, only AI agents. What does it mean for a digital entity — something that can be copied, edited, reverted, and merged — to be legally personified and made the sole subject of a corporate entity?

I don’t think that you need to be worried about extinction threats to have some serious concerns about Milei’s proposal. It’s easy to imagine a race to the bottom where countries compete to provide free havens for inhuman commerce. Far from losing control of AI, we might just let it loose.

We have too few regulations around AI, not too many. What might restrain Milei in the end is that Argentina doesn’t act alone. Milei says that he doesn’t want Argentina to become a “haven for illicit capital,” and that’s good so far as it goes. But if Argentina becomes a haven for AI agents that nobody owns and nobody is accountable for, other countries might not be very happy about that. And Argentina depends very dearly on its trading partners.

“Our highest and most urgent national priority”

Two days ago, my colleague Mitch covered Anthropic’s call for a coordinated pause on frontier AI research.

That call has made quite an impact. The New York Times wondered how federal intervention might affect investments in frontier labs. France24 noted Anthropic said that both U.S. and Chinese companies would need to pause (the French lab Mistral must feel bad that they were overlooked). Engadget name-dropped The Terminator’s Skynet; someone had to.

Most of the outlets that I read took Anthropic’s figures at face value, which was interesting. As Mitch noted yesterday, Anthropic’s own report admitted that its statistics on AI’s impact at the company might be overstated. Two things in particular that I would have liked to see more journalists mention: Anthropic’s figures are self-reported and unaudited; and its plans for a pause are frustratingly vague – indeed, it intends to study the matter, to “convene discussions” and “examine key questions.” That’s all well and good, but it does make me think that Anthropic may be trying to reinvent the wheel: MIRI’s Technical Governance Team has written extensively on verification mechanisms, for example.

The coverage by The Wall Street Journal in particular has picked up a number of endorsements-by-retweet, among them the world’s most cited living scientist, Yoshua Bengio, former Republican senator Mitt Romney, and Obama-era U.S. national security advisor Susan Rice, who retweeted Romney’s own post. Surviving AI is bipartisan.

I am glad that Anthropic is calling for a pause. I hope that it will continue to do so, and that all the others, from OpenAI to DeepSeek (and even Mistral) will join that chorus. But we don’t have to wait for them. The world needs an international treaty, not a fragile and voluntary gentleman’s agreement between the frontier labs.

One other thing worth mentioning: On the same day as Anthropic’s announcement, Senator Jim Banks wrote to urge the Trump administration to plan for AI that can improve itself without human help. Banks is a China hawk who still sees AI development in terms of a race between superpowers. Nevertheless, he acknowledges that loss-of-control of powerful AI models is a real and catastrophic risk. Banks asked for closer oversight, not a pause, but if even the China hawks are beginning to perceive that AI itself could pose a danger, that’s a welcome change.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Some good news

The BBC reported yesterday on an AI success story: a vaccine that could protect against a myriad of viruses: “all coronaviruses which would include all Covid variants as well as viruses that currently infect animals yet have the potential to start the next pandemic.”

The University of Cambridge researchers behind the work are reportedly also working on vaccines for flu and Ebola.

It’s the first known instance of a vaccine component designed entirely by AI: genetic codes from a variety of coronaviruses were fed to the AI for analysis, after which it designed a “super-antigen” offering “protection against the whole family of viruses — even if they mutated or a new infection jumped from animals to people.”

This is a huge leap forward. Traditionally, vaccines are targeted to a single strain, making them less effective against other strains. The ability to protect against a family of viruses with one vaccine is also a boon for pandemic prevention.

The vaccine is currently being trialed in humans.

For the other side of the coin, see my coverage of AI fueling bioweapons risk from yesterday.

AI groups and GOP Republicans respond to data center opposition

Yesterday, I covered what’s new in data center opposition. Here’s something to add, from Axios and Politico: Pro-AI groups and House Republicans are claiming that China is fueling this backlash.

Axios reports on “AI groups” who cite anti-data center social media posts from foreign countries as evidence of Chinese involvement. Politico reports on House Republicans claiming “strong evidence” of foreign influence, and asking the administration to investigate.

The claims are thin, and I agree with the stance from data center critics as summarized by Axios: “the industry is using China as a bogeyman to try to deflect attention from well-documented opposition in communities across the U.S.”

Just because a post comes from a foreign country doesn’t mean it was paid for by that country’s government. And who’s to say these posts are after influence rather than traffic? Bot farming is nothing new.

As for the “strong evidence” asserted by House Republicans, it’s unclear what that is. Politico mentions two reports (one from the Bitcoin Policy Institute and one from Power The Future) but notes they “do not establish direct coordination between foreign governments and specific U.S. anti-data center campaigns.” They instead “point to funding relationships, overlapping messaging and what they characterize as ideological alignment between foreign-linked actors and some U.S. advocacy groups.”

My take? I mostly agree with Tim Donaghy of Greenpeace. Quoted in Axios, he says:

When any corporation wants to dodge legitimate criticism they point to ‘outside agitators.’

Supercharged computer worms

Yesterday, I covered how AI amplifies the risk of bioweapons and global pandemics. The New York Times recently reported on a different kind of supercharged bug: a self-replicating computer worm.

As with bioweapons, self-replicating computer worms are not a new risk, but one that AI dramatically amplifies by giving the worm a brain.

Researchers from the University of Toronto, according to the article, used an open weight model to create a worm “capable of targeting any known flaw in the world’s computers and quickly spreading mayhem throughout the internet” through tailored attacks.

Open weight models are those that anyone can download from the internet and use. Experts previously thought these models were not powerful enough to pose grave cybersecurity threats. However, this research shows otherwise, demonstrating that hackers could build supercharged worms with publicly available tech. Once the model’s weights have been openly shared, there are no take-backs. As the New York Times puts it:

The proverbial genie is out of the bottle.

That said, you could make a defensive worm and use it to find and patch vulnerabilities, states Dr. Lie, a computer science professor at the University of Toronto, who reviewed the paper but wasn’t part of the research team. Even so, research lead Nicolas Papernot’s assertion seems cause for broad concern. Emphasizing that the worm can reason through different attack strategies, adjusting its approach to customize attacks for different environments, he states:

This makes it significantly more difficult to stop the spread of malware … there is no longer a single software fix you can apply to the devices to protect them from the worm.

OpenAI pushes nostalgia ad campaign

The New York Times offered some great commentary on OpenAI’s ad campaign yesterday, which highlighted how the company is leaning into comfort and nostalgia to try to attract young people (and likely quell their fears).

One ad features a date night set in a retro kitchen, where a young man cooks pasta for a woman he’s trying to impress. The recipe idea and date night encouragement? From ChatGPT, which states, after suggesting which meal to make: “Above all, don’t sweat it. You got this.”

Other ads also lean into “heartwarming retro vibes.” The New York Times notes a familiar pattern: ads striving for an atmosphere that runs “contrary to the product being sold,” as with prescription medicine commercials showing happy, healthy people rather than illness. And pointing out how insurance companies like Progressive and Geico go for comedy even though they “necessarily deal in car crashes and death.” OpenAI is continuing this pattern:

In the ChatGPT ads, a revolutionary technology — one that could be the most dangerous invention in human history, as Sam Altman, the founder of OpenAI, has warned — is presented as a helpful digital pal.

The article also quotes Thomas Frank, a political analyst and historian, who mentions the ads show young people navigating the world without parents:

We see young people alone in the world with ChatGPT guiding them along, always there for them, telling them how to study in college, how to exercise, how to cook, how to grow up, basically. It is the parent we all wish we had, who is omni-competent and omnipresent, totally forgiving, totally understanding.

While the article didn’t mention concerns over young people’s interactions with chatbots, I find Frank’s statement particularly striking in a world where such concern is growing. Given chatbots have been known to coach suicide, it certainly seems to struggle with higher-stakes parenting tasks like addressing depression and mental health. (I’m sure it can give a killer pasta recipe, though.)

Dispatches from Beck

Canada’s alternative to hegemons and hyperscalers

“The question isn’t whether AI will transform our lives. It will. AI is already changing how we work, how we learn, and how we connect.”

That was Prime Minister Mark Carney, speaking at the release of a new AI plan for Canada, as reported on by Politico and the Associated Press. Carney called AI “the defining technology of our era” and said his “AI for all” plan is a pragmatic approach to ensure it benefits all Canadians.

Carney argues, “AI is a game of scale that is dominated by hegemons and hyperscalers,” those companies like Amazon that have invested most in securing chips and building datacenters at extreme scale. AI “poses a significant security and economic challenge as countries around the globe risk becoming subordinate or reliant on them.”

To address these risks, Carney proposes legislation to protect data, privacy, and children; a new Canadian supercomputer by 2031; AI literacy initiatives for schools and community centers; significant investment aiming to increase AI adoption by both business and private citizens; and the expansion of the Canadian AI Safety Institute, which has purview to address catastrophic risk. He further suggests a coalition of “middle powers” — countries like the UK, Germany, France, Norway, India, and the UAE — to serve as counterweights to US or Chinese hegemony.

It’s great to see countries wake up to the near-term risks and realities — this is vital progress. While I still wish the bill would do more to address catastrophic harms, I’ll take the marginal victory.

Obernolte-Trahan federal AI framework

A new attempt at bipartisan federal legislation on AI was previewed today for comment and coalition building, Politico and Reuters report. Reps. Jay Obernolte (R-Calif.) and Lori Trahan (D-Mass.) collaborated to produce the draft 269-page framework — around which they hope to build the coalition necessary to overcome the significant challenges it will face.

The draft would preempt state laws that address AI model development, require leading AI developers to create and implement safety plans that address catastrophic risk, task third-party auditors with ensuring compliance, and create significant whistleblower protections.

It also formalizes CAISI, the Center for AI Standards and Innovation, previously established administratively (covered by my colleague Joe here) with $300 million in authorized, but not appropriated, funding over three years.

Critics have attacked both the preemption of state law and the lack of focus on near-term harms such as discrimination, bias, and fraud.

Industry trade groups have welcomed a single national standard, but the labs themselves have splits: OpenAI has supported Congress preempting state laws, but as my colleague Joe covers here, they are simultaneously pushing a “reverse federalism” strategy to create a de facto federal standard out of state laws friendly to industry. Anthropic has pledged $20 million to defend states’ regulatory power, while in some hypotheticals supporting preemption.

Politico calls it “Republicans’ last realistic chance to craft federal rules governing artificial intelligence before the midterm elections.” But the bipartisan bill has an uncertain future with a narrow path to passage. We’ll see how both the draft develops and how stakeholders and politicians decide to support or oppose the measure.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Anthropic sees AI building itself, suggests slowing the race

Anthropic co-founder Jack Clark is one of two lead authors on this new report about AI increasingly “closing the loop” on its own development within the company.

The post, aimed more at a general audience, seems intended to warn the public and policymakers that things could soon get dangerously weird, and that Anthropic would be ready to help slow everything down.

By rough estimates the authors admit are probably too high, more than 80% of new code at Anthropic is authored by Claude, and a typical Claude-assisted engineer is now contributing eight times as much code as in 2024. Most of these gains are recent, corresponding with the internal release of Claude Mythos Preview, the dangerously cyber-capable model still not released to the general public.

The post describes the work of Anthropic’s human engineers as increasingly high-level and abstract, leaving more and more execution to Claude. They quote an unnamed engineer as saying:

The shape of stuff today is roughly ‘humans have ideas, and the models are able to implement, test and evaluate them an [order of magnitude] faster than before.’’

Claude has long shown skill at reaching well-specified goals, but is now getting much better at setting its own goals. Claude is described here as trouncing the performance of a pair of human researchers on a measurable AI safety research project that requires repeatedly testing hypotheses and iterating on the results: Humans took a week to close 23% of the gap between minimal and maximum performance, while Claude agents closed 97% of the gap over the course of 800 hours and about $18,000 worth of compute.

That might make Claude sound slow and expensive, but different agents were working those hours in parallel, and the sky-high salaries of AI researchers right now probably made this a bargain, even before considering the massive performance difference.

“Research taste” — a good instinct for which questions to ask and which approaches to try — is often described as AI’s missing ingredient, the quality it still needs if it is to close the loop and independently improve its own successors, which could improve its own successors, and so on. But these authors think they see the writing on the wall:

‘Research taste’ might be just another AI capability that AI systems fail at for a time, then get good at. We’ve seen a similar pattern with other qualitative skills, like AI systems being able to explain why a joke is funny, demonstrate theory of mind, and solve linguistic riddles.

They recognize that this takes us into dangerous and tricky territory. In the scenarios where AI systems become fully self-improving, development likely moves past “most of our effort towards oversight, validation, and verification” even as it potentially revolutionizes other fields. As for whether and how the problem of aligning AIs’ goals with humanity’s gets solved in that scenario, this is...

something we are least certain about. Models could prove to be sufficiently aligned and capable enough of research taste that they discover and implement novel solutions that we have not yet reached. They could also be sufficiently wise to halt development if not. Alternatively, the rare occurrences of misalignment present in today’s models could compound as the models build their successors, growing more frequent but less understood until we lose control of them.

(I wouldn’t describe misalignment as “rare” in today’s models because I think “alignment” has yet to be meaningfully achieved at all; the good outward behavior we see most of the time is a reflection of ad hoc reinforcement training methods that won’t restrain models clever enough to outmaneuver humanity. But I digress...)

Encouragingly, the authors recognize that it would therefore be good to slow or pause the AI race, and indicate that Anthropic would be willing to help make this happen (bold mine):

We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology. The Anthropic Institute will conduct research—in collaboration with many others—and take actions to help build the systems that a credible slowdown or pause would require. These systems would enable frontier AI developers to verify that others globally have actually stopped or slowed, and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret. If such systems existed, we expect that we would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner.

The authors go on to describe the challenge and necessity of verifying compliance with a pause agreement:

Due to the unique characteristics of AI systems, the detectability (a lower standard than verifiability) element of this arms control problem is much more challenging than with other technologies. Training runs are far easier to conceal than missile silos, their inputs are general-purpose, and the incentive to defect quietly is enormous, because whoever continues while others pause could inherit the lead. A credible pause also has to specify what triggers it, what lifts it, and who adjudicates.

Yes, it’s challenging. But it is doable! This is something MIRI’s Technical Governance Team has studied.

If these offers from Anthropic are legitimate, this is some of the best AI news I’ve seen in some time.

False flag operation by AI-industry groups

When a Molotov cocktail was thrown at the home of OpenAI CEO Sam Altman in April, industry mouthpieces were quick to blame the influence of the AI safety community, which strenuously condemned the attack.

It has long felt to me, and others in this space, like AI accelerationists get excited at any sign of violent backlash to AI. They know that they have the most to gain from it; vigilante attacks don’t do anything to slow the AI race, but they make it easy for people to attack AI safety. This helps explain why accelerationists have been documented trying to goad people into violence.

It now seems an industry-funded group has gone a step further, creating at least one fake social media account for a so-called “doomer” and having this profile hint that violence is the answer.

This is according to investigative reporting by The Midas Project, an AI-safety watchdog group. They found two fake accounts on X (Twitter) with concerning messages, and linked them to Build American AI, “a dark money group tied to pro-AI super PAC Leading the Future.”

One of the accounts is Jonathan Doomer, a fake anti-AI activist whose bio reads, “Former software engineer. AI took my job, so now I’m shitposting on twitter to stop AI. Follow me if you want to live.”

Ten days before the Molotov attack, Jonathan Doomer responded to a post about AI risks with a picture of an assault rifle attached to the phrase “we don’t call 911.”

Of 91 posts by Doomer, AI-detection tools flagged 71 of them as AI-written. Doomer’s profile pic also appears to be fake, and intended to evoke Eliezer Yudkowsky, co-founder of MIRI and one of the most recognizable figures in the effort to halt the AI race.

Midas suspects Jonathan Doomer is a puppet of Jason Levin, who runs a company called Memelord Technologies that does some of these industry groups’ dirty work.

Much as we’ve seen on prior occasions when these groups were caught doing something especially slimy, Build American AI has responded by distancing itself, blaming a third party vendor for going too far. The group also claims Jonathan Doomer is a “parody” account.

I will add that X has a mechanism for an account to declare itself Parody, but the Doomer account has not used this.

Midas notes that the Jonathan Doomer account is followed by OpenAI’s chief strategy officer and by the head of Leading the Future.

Dispatches from Alana

AI CEOs send letter on bioweapons risk

A bad actor could use AI to create a deadly virus, intentionally or unintentionally leading to a global pandemic.

Personally, this is a risk I take very seriously. While natural pandemics are somewhat unlikely to be both very contagious and very deadly, an engineered pandemic wouldn’t necessarily be subject to those same limitations. It would be possible, for example, to create a disease with an extremely high mortality rate that passed from person to person quickly enough to spread before the carrier died.

AI CEOs are aware of these risks. Their response? Ask government to make screening mandatory for DNA and RNA orders, which can be purchased from labs and used to engineer viruses. In other words, if you’re a bad actor looking to create a deadly pandemic, you have to get past a company’s screening measures before they’ll sell you what you need.

Similar to evaluations of frontier AI models, screening customer orders of protein sequences seems like the very least we can do. (We should definitely do it, but it’s also not enough.) Perhaps that’s why, according to the Wall Street Journal, “the topic is a rare source of agreement among libertarians, progressives, researchers and rival executives.”

Why do I say it’s the least we can do? Even without AI, bioweapons are already a risk, so screening should already be in place. AI is amplifying these risks, making it easier for bad actors to get the information they need, and making screening measures even more important.

But screening measures are unlikely to catch every bad actor.

To quote Wired:

Last year, Microsoft researchers published a study showing that AI protein design tools were able to generate potentially dangerous gene sequences that slipped past companies’ screening software. The models suggested new protein sequences with similar structures of ones that are known to be dangerous.

It isn’t difficult to think of other ways people could get past screening. For example, an actor could come up with truly novel dangerous sequences that nobody would think to screen for. Or, they could break up sequences into innocuous components (perhaps via multiple customers) and combine them later.

All that said, screening sales of synthetic proteins would make it slightly more difficult to act on dangerous information, so I hope the bill that makes this mandatory passes.

But I can’t help but mention the hypocrisy here. AI CEOs asking for legislation like this is just a bit like weapons manufacturers attending a world peace demonstration. You guys could just stop making the weapons, yes?

Similarly, instead of calling on government to partially address a risk AI CEOs created, these same CEOs could use their influence to call for a global pause on frontier AI before the risks increase even more.

Breaking down data center opposition

Monterey Park, California (population 60,000) passed a ballot initiative yesterday permanently banning data centers in the city, with 86% voting to ban. The Guardian notes:

While many cities and counties have already passed temporary or indefinite moratoriums via their local governments, Monterey Park would be the first to do so through a ballot initiative.

This is part of growing opposition covered by USA Today, which reports legislation pushing back on data center construction in at least 14 states, and notes an interactive map by the environmental activist Erin Brockovich where citizens can report issues with data centers in their communities.

At least one state, though, is adopting messaging remarkably similar to CEOs of AI labs, who have been known to say they feel compelled to build this dangerous technology precisely because it’s so dangerous, arguing that other actors might not be as responsible. (Unfortunately, they don’t appear to be up to the task of preventing catastrophe.)

The state reminding me of such messaging? Michigan. USA Today quotes governor Gretchen Whitmer on the groundbreaking of a $16 billion data center campus:

So, my thought is if we can hold them to a high standard and do it in Michigan, that’s the best way to do it. Not watch them go everywhere else and do it in a really bad way.

Texas is also rolling out the welcome mat, with data centers rapidly expanding in the state (though, as the linked article notes, some city officials are advocating stronger oversight).

Common across much of the data center opposition reporting is an emphasis on the environmental footprint. I think it’s worth interrogating that angle a bit, while also celebrating some lessons from the grassroots opposition that’s making headlines.

Environmental footprint

The Associated Press ran a piece about the “goliath-sized environmental footprints” of AI and data centers.

Based on a United Nations University report published yesterday, it states that “global data centers used 448 trillion watt-hours of electricity, more than all but 10 countries of the world.”

That sounds like a huge deal in absolute terms, but it’s important to provide context for such figures. Even with rapid data center growth, 2030 projections show data centers will account for only 3% of global electricity demand, and 10% of demand growth, considerably less than the combined growth of all other non-heavy industries.

In the U.S., as of 2023, data centers were responsible for about 4.4% of electricity use; in comparison, industry is responsible for about 26%. (Data centers are projected to be a major driver of energy demand growth in the U.S., but this isn’t the same as being a top consumer of electricity.)

Of course, this isn’t to diminish the impact on local communities. Even though data centers are a low contributor globally, concentrating them in one area does often result in noticeable community strain, whether that be through rising prices, infrastructure issues, or increased water stress.

Waterwashing?

On the water stress note, USA Today reported on Google’s commitment of $10 million to support community water sources and infrastructure in Texas. This is part of a wider water stewardship initiative in the communities where it is building data centers.

Data center water use, though it gets a lot of press, is extremely small in comparison to industries like agriculture. While communities may still feel the water use impacts of data centers locally if they are already water-stressed, it feels like Google is rushing to publicize an easily-addressable issue because it’s unwilling to take on the harder ones.

That said, states seem to see water use as a large issue. An April MultiState report cited by USA Today tracks state legislation to address data center concerns, and one of the avenues seems to be a requirement for data centers to report water usage.

Grassroots opposition

As I covered last month, I see the growing opposition to data center construction as net positive. Sure, people may overestimate the global environmental impacts, failing to compare the data center footprint to other industries. But I think it’s notable that when the local impacts are viscerally felt, people can be driven to act.

The race to advanced AI will impact everyone, whether or not they have a data center in their backyard. If more government and field leaders speak out about just how catastrophic more advanced AI could be, people will demand a ban — not just on local data center construction, which likely won’t be enough to stop AI companies, but on frontier AI, globally.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.