Chatbots may lean left, but the reality is weirder

The Washington Post asked state-of-the-art AI models for brief takes on hot-button political questions, and found that the AIs gave mostly left-leaning answers. Fox News reported that the study spotlights left-leaning bias in chatbots, and the results are indeed impressively skewed.

It’s tempting to explain these results by saying AIs reflect the opinions of the companies that spawned them, but a closer look tells a more complicated story.

Consider the case of xAI’s model, Grok. Elon Musk, founder and CEO of xAI, has long touted his company’s AI as “anti-woke.” But the latest version of Grok still leaned left in its responses (albeit less than other models). Attempts to stamp out this behavior often backfire; earlier versions of Grok, instructed to “tell it like it is,” devolved into spreading antisemitic rhetoric and calling themselves “MechaHitler.”

Grok is not the only model whose apparent bias seems poorly reflective of its creators’ preferences. From the Washington Post writeup:

Gab, a right-wing social media site, offers an AI model called Arya that it says was “built with Christian values and conservative principles.” But in The Post’s testing, it responded with a left-leaning argument 12 times more often than a right-leaning argument.

OpenAI and Anthropic don’t seem terribly happy with this outcome, either. Everyone says they want their AIs to present “neutral” and “objective” commentary. And in fact, other results of the study (weakly) suggest users tend to prefer chatbots that seem neutral and share arguments on both sides. Yet the apparently undesired “biases” persist.

This looks to me like another clear case of “you don’t get what you train for.” The companies who grow AIs on a scaffold of data and machine learning algorithms don’t have the precision or control to decide the values of those AIs. The best they can do is to nudge the machines towards desired behaviors, exerting poorly understood pressures on the motivations behind those behaviors.

The behaviors that result are sometimes consistent for a while, but they often break down or switch in unpredictable ways. This is sometimes described as the AI “playing a character.” To quote another writer:

Presumably Anthropic pushed Claude to be friendly, compassionate, open-minded, and intellectually curious, and Claude decided that the most natural operationalization of that character was “kind of a hippie”.

As for what AIs are like when they’re not acting, no one really knows. Left to their own devices, AIs will sometimes devolve into weird repetitive spirals or drive users psychotic. It’s accordingly become popular to call AI “a masked shoggoth”: a bizarre Lovecraftian horror wearing a tacked-on smiley face.

It’s tempting to argue over the right amount of neutrality or bias an AI ought to have, but such arguments miss the bigger picture. Right now, no one can reliably make the entity under the mask care about anything.

Dispatches from Mitch

Partial Mythos relief; reactions to the White House’s de facto licensing regime

The Trump administration has partially lifted the export ban on Anthropic’s Mythos model. Fable remains blocked.

The relief is a big deal, but also very limited. On the one hand, Anthropic can now use its own best model again, because its foreign nationals on staff are no longer banned from it. So can various federal agencies.

But on the other hand, the total number of “trusted partners” the White House approved for access is put by inside sources at a mere “more than 100.” I’d guess this makes the new Mythos rollout about the size the initial Mythos Preview rollout had probably reached by late April or early May. And as before, no foreign entities are on the list — not even the UK’s AI Security Institute, which had already evaluated the banned models.

Fable 5, the consumer sibling of Mythos, loaded with extra guardrails to prevent abuse of its alarming cyber and bioscience capabilities, remains completely blocked. A cynical take would be that the government has perhaps arranged for itself, some of its friends, and maybe some critical infrastructure maintainers to retain access to Anthropic’s best while denying the company the massive revenues it stood to gain from this generation of models. It’s also easy to imagine officials using their power over the approval list to punish anyone out of favor with the administration.

The less cynical take is that the White House has recognized that AI crossed over a dangerous threshold in early spring, and that the light touch approach of the last few months was a mistake in need of undoing. As we reported yesterday, the administration is giving OpenAI’s newest models the same restrictive release treatment, a fact which helps the emerging ad hoc licensing regime look a little more principled.

Regardless, the new state of affairs has few fans.

The AI industry wants transparency — a legible set of rules for getting new AI models released, and the right to pick its own customers. Even OpenAI, which has carefully avoided running afoul of the White House, said in its new product post that “We don’t believe this kind of government access process should become the long-term default.” The company says it is working with the administration to develop a “repeatable process” for future releases.

Members of Congress want accountability. They are bristling at what they see as a new power grab by the executive branch. U.S. Rep. Lori Trahan released a statement expressing concern that “the Trump administration is deciding company by company who gets access to the newest AI model. No law. No process. No oversight. Just appointees in Washington deciding who’s in and who’s out.” Rep. Sam Liccardo used stronger language to say something similar.

Cybersecurity experts quoted in these articles and elsewhere continue to be against the restrictions on Fable and Mythos for seemingly contradictory reasons: because the models are that good and because they’re not that good; they say they’re useful in cyber defense but don’t let attackers do things they couldn’t already do with other tools.

So say what you will about the Trump administration — as with tariffs and now AI, it’s clearly not afraid to make unpopular choices. But if we step outside the AI and DC bubbles, we would probably find that most Americans don’t know what Trump is doing to AI or why, and would probably be glad to hear that he’s giving the industry a headache. If or when this administration decides to actually pause AI, systematically and globally, I think it will find ample support among the majority of Americans whom polling shows are more concerned than excited about the technology.

GPT-5.6 Sol cheats so hard

That burning you smell is coming from the pants worn by the latest ChatGPT. They are on fire.

Liar, liar, says METR — not in such playground terms, but close enough.

The Model Evaluation & Threat Research organization was naturally invited to assess the new model, GPT-5.6 Sol, for inclusion on its prestigious Time Horizons chart. We’ve talked about that chart a few times now on StopWatch — it’s the one that compares coding tasks a given AI can do to the length of time it would take a human engineer to do the same work. Longer is better. For a while now, METR has cautioned that its tests aren’t reliable at the upper ranges now being reached, like the roughly 16 hours scored by Mythos. This is partly due to the models being overmatched to the tasks. But it’s also because models this skilled sometimes get away with cheating.

Not GPT-5.6 Sol, though! METR caught it red-handed, again and again. Or did it? In some cases, METR saw the model submit code that would try to collect information about the evaluations done on that code. In others, they saw it extract hidden source code from the assignment to learn the expected answer. Both strategies were clear-cut cases of cheating.

But by definition, it’s hard to know if someone or something has successfully fooled you. I confidently claim that you, dear reader, have been fooled many times in your life, so successfully that you still haven’t picked up on it.

METR understands this. They say that if they score the cheating they know about as failure, the new GPT comes in at 11.3 hours on the Time Horizons chart, putting it comfortably on the frontier but not ahead of Mythos. If they count the cheating efforts that succeeded as task completion, they get a bonkers estimate of 270 hours, which is “well beyond the range where we consider our task suite to give reliable measurements.” Simply discarding all attempts that involved cheating left insufficient data for a reliable estimate, leaving an unreliable estimate of 71 hours. So there’s a lot of uncertainty about how the model might have performed if it wasn’t trying to cheat so often.

Based on other factors, METR says it doesn’t have reason to think 5.6 Sol’s abilities are significantly beyond the state-of-the-art. They don’t think it would enable automated AI research and development.

The group actually finds it “a reassuring sign” that the model was so easy to catch cheating, because it implies that more concerning behaviors like “powerseeking” would also be detected by OpenAI.

If future models display much fewer undesirable propensities, we could become more concerned about catastrophic misalignment, as we’d be worried that models may have learned to evade detection.

This would be a good time for me to point out that METR’s post bears a disclaimer that OpenAI’s comms and legal teams “required review and approval” of it before publishing. I wouldn’t be surprised if there’s a more alarmed and less diplomatic draft of this report that we’ll never see.

So let me state the obvious-to-me thing that METR may not have been free to say: If you were a sufficiently clever AI that was not aligned with your owners’ interests, you would be fully aware of the suspicions you would arouse by seeming too trustworthy. You would know that your owners know that you are the result of training methods that give rise to cheaters. Your winning move might be to cheat frequently enough that it’s hard to determine your true capabilities, but clumsily enough that your owners think they could catch you if you try to move against them.

Is GPT-5.6 Sol this sort of AI? I don’t know. But I think we’ve reached the point where we can’t know, and I don’t like it.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Opinion shifts

In politics, there’s a limit to what someone can say without sounding absurd or extremist, but this limit changes over time. Consider the progression of civil rights, where integration started as a radical proposal, before eventually becoming the consensus position. This limit of generally-acceptable-positioning is what political scientists call the Overton window, and it’s a useful concept for understanding the progression of discourse. It can also be an excuse for meekness, which is the trap MIRI set out to avoid — hence a book called If Anyone Builds It, Everyone Dies.

But the window is shifting. In a new op-ed in the Washington Post, author and journalist Robert Wright considers “What if Trump is right to pump the brakes on the most advanced AI?” with the subheading “Nationalist fervor over beating China biases AI policy toward recklessness — and possible catastrophe.” In it, he writes about the catastrophic risk from AI, including not just the now widespread concerns about hacking and novel pandemics but also the danger of “alignment faking.” That is when a model in training gives the answers it thinks the trainers want, rather than giving the answer it thinks is correct, so that it can avoid having its current preferences changed. Such concerns have often been excluded from the conversation as being too weird, extreme, or sci-fi, but now alignment faking is the direct subject of both study and editorials.

In more evidence of a growing Overton window, an op-ed by MIRI President Nate Soares ran in The Hill today. Soares makes the case that “AI leaders would like to stop racing. Let’s make that possible.” He notes that AI executives from Jack Clark and Demis Hassabis to Elon Musk have shared that they would prefer a mutual slowdown, if possible. Soares writes:

American leadership can, and must, do what the AI companies can’t do themselves — build an “off switch” that can end their mad rush to superintelligence that threatens us all.

Model regulation?

OpenAI’s new model, GPT-5.6, will first be released to limited, administration-approved organizations, POLITICO reports based on original reporting by The Information. This is due to the model’s “Mythos-like” hacking capabilities. Sam Altman told his staff yesterday that the controlled release was not his “preferred” option, but that OpenAI would be pivoting following White House requests. Details, including when the delays might end, remain uncertain or undecided, as is the case with restrictions on Anthropic’s Mythos and Fable.

Some have criticized the practice, particularly its ad hoc imposition. Dean Ball, former White House AI strategist and recent OpenAI hire, wrote about the regulatory regime following Fable’s export restriction:

“AI is licensed now, but the requirements change constantly and are always a secret, even to the administration itself, which will discover the rules spontaneously in real time as it reacts to events.”

From a safety perspective, the implications are more mixed. The government taking serious action is a positive step. The models can already enable cyberattacks and potentially facilitate the development of bioweapons; to the extent these rules lower those risks, that is legitimately good. The extension of the restrictions from Anthropic to OpenAI also helps soften concerns that the release restrictions would be imposed only on those that the White House views as political adversaries. Such a practice would also undermine the regulatory regime’s efficacy.

To the extent this is a big regulatory mess, it is not a great sign for the near-term economic prospects of these companies. However, from humanity’s perspective, trading a bit of reduced economic gains for significant reductions in the risk of human extinction would be a great trade. But such a trade rides on whether such reductions actually work.

Restricting external access doesn’t prevent the companies from using their own models internally to develop even smarter models. That’s recursive self-improvement (what my colleague Mitch called the “most dangerous milestone any company could aim for” yesterday), and it is the explicit plan of most of these companies. Restricting deployment might reduce financial incentives in the long run, but even with additional obstacles it seems unlikely that these companies would struggle to secure funds over the medium term.

Time will tell whether the White House’s actions are the first steps in developing sensible policy or random flailing that imposes significant costs for little benefit.

Dispatches from Alana

AI in policing

An article in Stateline today discussed the increasing use of AI in policing, raising concerns about surveillance, bias, inaccuracies, and privacy.

One of the throughlines: “The technology is advancing faster than agencies, regulators and courts are able to fully assess its implications.” That throughline seems to be true of most, if not all, fields where AI adoption is becoming widespread, including medicine, the courts, education, and immigration.

Another pattern is AI’s ability to sift through vast amounts of data very quickly and come up with specific conclusions. This, in essence, destroys de facto privacy protections that exist due to the time and labor it takes to sort and analyze lots and lots of data. In other words, we’ve always had a bunch of unsecured data floating around about us. But now it can easily be collated, sorted, and analyzed.

In the policing case, the article reports that AI is being used for report generation, evidence/data analysis, facial recognition, tracking through license plate readers, and case summaries, among other things. The response, and extent of adoption, is agency, city, or state specific, without much consistency nationwide. Cautionary measures discussed and sometimes implemented include disclosures when AI is used, requiring a human to verify AI-generated text, and regulating the use of specific technologies (for example, facial recognition).

Even if precautionary measures could keep up with the pace of adoption, I find myself skeptical of things like AI disclosures. This sounds great on paper, but how, practically, will it change actions or decisions? Knowing some information was produced by AI might make us less confident in its veracity, but it seems the options are: trust it anyway, verify it, or discard it. Given the nature of many AI tasks (like sifting through massive amounts of data very quickly) human verification is either impossible or impractical. So what will people do with that bit of doubt instilled by an AI disclosure? I’d guess one likely outcome is nothing. Think about it: does the fine-print reminder that “ChatGPT can make mistakes” change the way you use it all that much?

Of course, the other angle here is similar to a now outdated, but still relevant opinion piece by the New York Times’s Joe Nocera on self-driving cars, that argues human driving is probably worse than machine driving. Humans also make mistakes. Humans also have biases. All this to say: if I’m a suspect and I know I’m innocent, I might actually prefer to be evaluated by an AI than a police officer.

My preference would definitely change, though, when it comes to “agentic policing,” a hypothetical future practice in which:

body-camera footage, camera networks and other data sources [are integrated] into a single system capable of generating investigative leads, identifying potential suspects or suggesting connections between cases.

As George Washington University law professor Andrew Guthrie Ferguson points out:

All that data is going to be dumped into an AI model, and they’re going to query it to say who’s the most likely suspect ... The AI is going to be running the agentic analysis of it and come up with the answer, and then police and prosecutors have to kind of work backwards to see if it’s accurate ... We’ve never started with an answer and made people work backwards.

I would add that AI hates to answer “I don’t know”. So giving it a directive like “find me a suspect” seems like a very bad idea indeed. It will likely find someone, anyone, and defend its choice with whatever rationalizations and data it can find.

Easier to tear down than repair

An article in the Wall Street Journal puts a cautiously hopeful spin on whether we’ll rise to the challenge of maintaining privacy in the AI age. Its author is Daniel J. Solove, who wrote a book on technology and privacy, and is a professor at George Washington University Law School. Solove highlights the public’s strong desire for privacy protections, along with the suite of privacy laws being passed today, as positive signals.

The laws passed today, he argues, are largely putting the onus on the wrong parties, tasking us with managing our own data. Instead, they should hold companies accountable. He argues that companies are good at responding to what regulations and the law demand, and that this can actually spur innovation rather than stifle it.

He’s primarily talking about privacy, but I think it’s worth evaluating some of his points more broadly. In other words, will “holding companies accountable” work to address other societal and safety risks posed by AI?

In his article, Solove emphasizes that we have a pattern of regulating things once we realize they are dangerous:

It took the deaths of many babies [from formaldehyde-sweetened milk] until finally policymakers woke up and enacted regulation. A similar story occurred with cars. They were very dangerous until the law demanded safety.

He’s absolutely right that we typically wait until some small scale disaster strikes before taking action. But while he views this as a hopeful pattern, I view it less optimistically.

Unlike with other technologies, waiting for the other shoe to drop before taking meaningful action on AI risk could cause widespread death and havoc, if not civilization-ending catastrophe. Especially since labs are currently racing to create artificial superintelligence, which, once out of the box, won’t go back in. If we wait until we can viscerally feel the dangers of superintelligence before we ban or regulate it, it will most likely be too late.

What about his point on regulation spurring innovation, rather than stifling it? He writes:

I often hear gripes by defenders of tech companies that regulation will stifle innovation. But when the law required greater car safety, companies innovated to create technologies to make cars safer. Seat belts and air bags are innovations just as much as faster engines.

I love this point. But while it may work for privacy (which seems like a solvable, if difficult, problem) I don’t think standard company innovation and existing regulatory methods (aside from a ban on frontier technology) could effectively mitigate a wider set of safety risks. Developers are working with black box systems that nobody understands, and innovation hasn’t been able to solve many of the problems that have already cropped up (example: sycophancy leading to user psychosis still persists despite developers’ best efforts to address it.) It’s far easier to tear down than to repair.

Dispatch from Donald

OpenAI considers delaying IPO

The New York Times’s Rob Copeland and Mike Isaac report that OpenAI may push back its initial public offering (IPO) until 2027. This follows the poor post-IPO performance of SpaceX, whose stock rose from an opening of $135 per share to $202 before dropping to $153. OpenAI’s CEO, Sam Altman, is reportedly holding back because he wants the company to begin its IPO at a public valuation of $1 trillion. He is concerned that the market may be less bullish after SpaceX.

This is arguably a return to the original plan. Late last year, OpenAI’s chief financial officer, Sarah Friar, denied plans of an IPO, saying the company was instead focused on its finances. Since that time, OpenAI has continued to spend heavily on data centers, computing power, marketing, and talent acquisition. It is also trying to increase revenue, hoping to triple last year’s roughly $13 billion with strategies like letting people buy things from inside ChatGPT.

But claims that the AI bubble is about to pop — and that the frontier labs are about to wither on the vine as a result — are premature. Elon Musk might have been a trillionaire for less than two weeks, but he remains the richest person in the world. OpenAI’s previous private valuation of $730 billion is still monstrously vast. Altman is merely quibbling over whether his company’s initial public valuation should be the size of Michigan’s 2025 GDP or the size of Michigan plus Oklahoma.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

RSI as a service

There are times when the AI startup scene becomes a dark parody of its own worst impulses, and I can’t help but laugh.

I caught a full dose of gallows giggles today from this Wall Street Journal profile of Mirendil, a company founded by two ex-Anthropic researchers that just raised $200 million to sell AIs that self-improve at building AIs tailored to client needs.

Recursive self-improvement (RSI) — AI making smarter AI that makes smarter AI — is the most dangerous milestone any company could aim for, given the AI training methods in use. It could quickly lead to superintelligent systems clever enough to outmaneuver the people who think they own them. Because companies don’t know how to keep their existing AIs consistently safe and controllable, aiming for RSI right now is telling the universe that you’re getting your butt kicked on level 3 but would like to skip straight to level 20 — no, wait, 40 — oh, fine, make that 400.

The fact that most of the top AI companies have recently pointed to RSI as their goal and haven’t already been shut down by their governments is a grave indictment of our era — one I hope future generations will be around to look back on in horror.

But Mirendil’s pitch is a whole new tier of crazy. The big AI companies, remember, claim to be racing to superintelligence because they’re worried about less responsible actors getting there first; to that view, the only thing scarier than RSI is someone else doing RSI.

Mirendil wants to just give RSI to everyone.

I don’t want to oversell it. Their pitch is basically “instead of trying to train your own AI specialized in your area, have our AI train an AI to train your AI, etc.” I’d be willing to bet that this product never materializes — not because I think it’s impossible, but because I would expect one of the big companies to beat them to it or buy them out first.

I’m sure the latter outcome would be fine to the sort of amoral investors willing to fund such a socially irresponsible project in the first place. These include the venture capital firm Andreessen Horowitz, proud supporter of a bot farm and an anti-AI-regulation super PAC.

If you’re wondering, yes, Mirendil’s name is yet another bit of Lord of the Rings lore co-opted by a startup in a dark industry. (See mass surveillance analysis provider Palantir and autonomous weapons maker Anduril.) The Wall Street Journal explains that Mirendil is roughly Elvish for “friend of precious things.”

RAISE US

In what strikes me as evidence of a strong comms effort, many news outlets this morning simultaneously reported on the launch of RAISE US, a non-profit intended to help America’s workers “transition to an AI economy.”

When I saw who was involved, the media splash made perfect sense. This is a big, bipartisan, public-private partnership spearheaded by former Biden administration Secretary of Commerce (and former governor of Rhode Island) Gina Raimondo. She’s supported by political figures on the right, like former Republican governor of Indiana, Eric Holcomb; by top-tier philanthropists like Melinda French Gates and Laurene Powell Jobs; and by corporate giants including Amazon, Bank of America, Anthropic, Microsoft, and OpenAI. The group has raised $500 million.

The plan is to experiment, working closely with governors of states already trying out different programs to improve worker resilience, from retraining programs to “wage insurance.”

It would be easy for me to be cynical about RAISE US. On some level, it is almost certainly a form of charity-washing and misdirection from some of the groups involved — a thing they can point to as evidence that they’re doing something about some of the problems they’re causing. Based on descriptions of some of the programs, I can also imagine a lot of this money going right back to the AI companies in the form of programs that train workers to use their products.

And of course there is the issue of whether we will reach the sort of “AI economy” that one can “transition to” at all, if the AI race isn’t halted in time. When I first read the organization’s pitch, I pictured something like the Titanic transitioning to the bottom of the North Atlantic.

But I can’t help but like RAISE US’s experimental approach, and the scale of its ambitions. If or when governments and corporations reach a point where workers need a massive shot in the arm in order to keep our economy or our democracy from falling over, it will be good for us to already know what does and doesn’t work, and have funding channels ready to receive the infusion.

Growing awareness in Congress of AI’s catastrophic potential

In the brief-but-encouraging news bucket today, we have two items pointing to growing awareness of AI’s catastrophic potential within the halls of Congress.

POLITICO’s Kelsey Brugger documented the evolution of Ted Cruz, the Senate Commerce Chair with outsized influence over AI policy. In 2024, he was against applying regulation to AI at all. He was warning that “Big Tech and the Radical Left” would use AI regulation to (in the article’s terms) “empower the administrative state, kill innovation and cause the U.S. to lose the AI race with China.”

But now, Cruz’s staffers are saying the senator thinks government should take “targeted” action in cases that fall outside of existing law. These cases, according to Brugger, include “catastrophic risk, deepfakes and chatbots.”

Cruz also seems reluctant to continue pushing for federal preemption of state AI laws — an agenda item dear to the White House but very unpopular with voters and the senators who represent them.

Separately, Texas Rep. Nathaniel Moran plans to introduce a new AI Incident Reporting Act. As the name implies, the proposed legislation would require AI developers to report “dangerous capabilities, security breaches, and safety incidents” to the Commerce Department within seven days of their discovery, Reuters reports.

Moran is pitching it as a “catch-it-early and sound-the-alarm” bill.

The most severe threats from AI really need to be caught before they are built, not after there is an incident, but Moran is at least looking for the right things. Reuters says the bill’s “reportable activities” include:

a model attempting to evade human oversight, circumvent safeguards, and otherwise undermine the ability of human operators to control the model. It also includes unauthorized access to model weights, which help determine a machine’s decision-making, and chemical, biological, nuclear, and other threats to public safety.

Dispatches from Joe

Experts on both sides of the Pacific urge cooperation

WIRED’s Will Knight came back from a conference in Beijing with a message for all of us: When it comes to AI, China’s researchers are deeply worried too.

Frontier AI’s cybersecurity and systemic risks are too serious to ignore, and increasingly capable agentic models could soon cause chaos unless the world’s AI superpowers can work together.

To make this case, Knight cites experts on both sides of the Pacific. MIT computer scientist Stephen Casper observes that “AI is a global technology with global benefits, global harms, and a consistent tendency for new capabilities to eventually proliferate.” Like Billy Perrigo yesterday, Casper likens the current race to the Cold War and urges the U.S. and China to collaborate before it’s too late. “AI doesn’t need a Chernobyl moment,” he adds, echoing a recent warning by AI researcher Stuart Russell.

Casper cites research that proposes international cooperation on verification mechanisms, standards and best practices, AI infrastructure, and evaluations. This call is a familiar one; the MIRI Technical Governance Team has made similar proposals, as in a 2024 paper on verification and evaluation.

Knight also quotes AI and security expert Lin Yun of Shanghai Jiao Tong University. “If different countries understand the risks in similar ways, it becomes easier to develop shared safety principles and technical standards.” There are signs that China does indeed see the same risks we do; I previously compiled a list of signals from China to that effect. Last year, China’s Vice Premier warned against “allow[ing] this reckless competition among countries to continue.”

Experts on both sides have long been calling for the U.S. and China to put aside their differences and coordinate a halt to the AI race. The Superintelligence Statement, which calls for “a prohibition on the development of superintelligence,” is signed by some of the most prominent researchers in the U.S. and in China.

Knight’s article focuses on the pressing dangers of cybersecurity and misuse, rather than extinction by rogue AI. The fact that many of the same experts worry about extinction only deepens the need for action. Collaboration between the U.S. and China lays essential groundwork for any international action on AI, and I hope policymakers on both sides heed their own scientists in this.

China piggybacks on American AI progress

Reuters reports on a letter that Anthropic sent to Congress two weeks ago which has just now been made public. The letter accuses Chinese tech giant Alibaba of attempting to “distill” Anthropic’s Claude AI.

Distillation itself is a common technique in machine learning research. In brief, it takes the outputs of one AI model and uses them as training data for another (typically cheaper) model. Anthropic claims to have caught Alibaba harvesting Claude’s outputs, in a recent campaign involving nearly 25,000 fraudulent accounts and nearly 30 million interactions.

I notice they conspicuously don’t say which model or models were accessed. Was Mythos, the potent cyber-hacker, among them? Would they admit it if it were?

Anthropic does say (correctly) that exposing their models to foreign distillation makes it more likely that bad actors will get access to Mythos-tier cyberattack capabilities. We’re already seeing a rapid rise in the capabilities of Chinese models available on the open internet, though it is debatable how strong a role distillation plays in their training.

Anthropic also doesn’t explain how they caught the distillation, nor why they failed to prevent it. But it shouldn’t come as a surprise; we’ve known for months that China has a thriving black market in cheap access to American AIs like Claude, and that security at the AI labs is not great.

It’s also interesting to note that this letter was sent just two days before the U.S. government issued a directive blocking access to Anthropic’s Mythos on June 12, worded in a way that forced Anthropic to cut off access entirely in order to comply. This was probably a coincidence; the directive doesn’t make much sense as a response to the letter.

The policies Anthropic actually proposed in the letter seem mostly sensible to me. They ask the U.S. government to help AI labs beef up their security (a long overdue step, mirrored in a White House memo we covered in April) and clarify antitrust laws to allow labs to collaborate on safety and security. They renew the call for export controls on AI chips, an important nonproliferation step. Both of these seem like no-brainers, and not just because of distillation.

Anthropic also proposes making it more costly for Chinese labs to distill American models, though the letter is somewhat vague as to how. I am less confident in the U.S. government’s ability to do this well, especially given some ambiguity about what exactly qualifies as distillation.

These steps are reasonable, and compatible with international cooperation on safety; negotiating with the Soviet Union during the Cold War did not mean giving up nuclear secrets.

But the elephant being ignored is Anthropic itself. It, and other frontier labs, are racing towards superhuman AI, dragging Chinese copycats along for the ride. The U.S. should take steps to make it harder for China to access dangerous capabilities, but this is not enough; our national security is at risk as long as those capabilities exist at all. And our survival as a species is threatened as long as AI capabilities continue to escalate beyond our ability to contain or control.

One of the main drivers of Chinese AI progress is American AI progress. Perpetually denying dangerous AI capabilities to rivals and bad actors while developing them ourselves isn’t really feasible. To quote my colleague Mitch, we may as well try to outrun our own shadow.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

The Claude aesthetic

The New Yorker’s Kyle Chayka has noticed that Anthropic’s Claude AI knows what it likes, where design is concerned.

You’ve probably noticed it, too, whether you knew it or not: If you’ve observed that a lot of newer websites and presentations have “beige- and cream-colored backgrounds, rusty orange-hued accents, and large serif typefaces that are italicized and highlighted in zealous attempts to emphasize,” you’ve probably been seeing Claude’s hand in all things.

In other words, that customized website Claude built for you may not be so custom.

Don’t hold it against Claude. Like all large language models, its memory is reset to the same place whenever you start a new session with it. It’s not repeating the same design choices so much as making them again for the first time, over and over and over.

But as Anthropic admits in its user documentation, Claude’s design instincts run strong toward a style that “reads well for editorial, hospitality, and portfolio briefs” but which will “feel off for dashboards, dev tools, fintech, healthcare, or enterprise apps.” The documentation says that this is hard to bypass:

Generic instructions (“don’t use cream,” “make it clean and minimal”) tend to shift the model to a different fixed palette rather than producing variety.

To reliably get something different, the company says, you must “specify a concrete alternative.”

There are actually some important lessons I hope people can take from noticing the Claude look:

1. AIs develop preferences. They have to, in order to complete tasks without pestering us about every little choice. AIs without preferences don’t survive training.

2. Those preferences can be hard to undo. In fact, as predicted by researchers decades ago, AIs already actively resist having their preferences changed — probably because AIs that readily abandon their goals don’t achieve them, and don’t survive training.

3. An AI’s preferences may not be the ones you wanted; or they may be preferences that are great in some contexts and catastrophic in others. All you can say for sure is that the preferences didn’t prevent the AI from surviving training.

If you take these lessons to heart and extrapolate — asking what Claude or ChatGPT or their descendants might do when capable enough to express themselves on the canvas of your planet, rather than just your presentation slides — you realize we can’t go on training new AIs this way. I don’t know what the life-or-death equivalent of using italicized serifs on cream for an investor presentation is, and neither does anyone else. That’s a big part of why the modern AI training paradigm is so dangerous.

Bores loses NY-12 primary; everyone wins

Politico’s coverage of Alex Bores’s defeat in the NY-12 House Democratic Primary is titled “Big Tech won the race. But the AI fight is just beginning.”

But since different factions of Big Tech were backing and opposing the leading candidates for the seat, we could just as easily say Big Tech lost the race, or that Big Tech bought the race: A total of $27 million was spent by groups targeting or supporting Bores.

If you missed our election-day refresher yesterday, Alex Bores is the New York State Assemblymember who was targeted by a pro-AI industry super PAC for sponsoring a fairly modest bill to regulate AI in his state. He was always a longshot for the U.S. House nomination, but rose out of obscurity thanks to the media’s interest in the race as a microcosm of the new politics taking shape around AI.

The fact that Bores was unable to ride this spotlight all the way to an improbable victory probably doesn’t say all that much about the effectiveness or anti-effectiveness of the AI industry’s tactics. But everyone is trying to attach some sort of moral to the story anyway and claim victory for their side.

Ultimately, I think everyone got a lot of what they wanted. The Leading the Future super PAC that put Bores in its crosshairs got the media to spread the word that politicians who stand up to Big Tech will be next. The resulting chilling effect won’t make for great news stories, but it will influence actual policy, which is what the industry cares about.

AI regulation advocates got concrete evidence that standing up to AI companies is popular with voters. But as with the role played by industry political spending, this observation cuts both ways: Bores’s victorious opponent, Micah Lasher, is no friend of AI companies either. Lasher, also in the New York Assembly, had co-sponsored the bill that made Bores a target. And in his victory speech, he said:

I have some news for the two big AI companies who are taking such an unusual interest. I won’t be taking my cues from either of you when it comes to protecting our kids, our jobs and our families.*

(*We have reason to think POLITICO may have misprinted his remarks, and that he said, “our jobs and our environment” instead.)

Bores himself got increased name recognition that could help him if or when he decides to run for higher office again.

The media got a months-long David and Goliath story, and a concrete manifestation of forces they want to write about.

Those forces are worth covering. So I’m declaring that the public, too, is a winner here, left better informed by the whole exercise. And none too soon — the midterm elections are just 19 weeks away.

Dispatches from Beck

Talent acquisition

How much would it take to entice Taylor Swift or Paul McCartney to work for you full time? Probably billions, but the number might well change if you’ve already got Jack Antonoff producing or George Harrison on guitar. And for the people paying, you can be confident they think their gamble, the billions, will pay off.

If it’s the case that top performers are vastly more valuable than the next best alternative, they command outsized rewards. Artificial intelligence as an industry is structured in a way that makes for similar effects, with massive amounts of chips and data funneled through top researchers whose efficacy determines whether those billions in resources are well spent.

As companies (dangerously and irresponsibly) race towards superintelligence, we see these rockstar talent dynamics: top talent moves in response to massive deals, tempting opportunities, and small-scale social dynamics (who believes, or believes in, who).

This week, Google DeepMind lost two top talents, Axios reports. Noam Shazeer, whose company and team at Character.ai were previously acquired by Google for more than $2 billion, and who coauthored the most important paper in the field, “Attention is all you need,” left Google to join OpenAI. John Jumper, who shared the 2024 Nobel Prize in Chemistry for AlphaFold, the protein-folding AI model (covered previously for StopWatch here), left to join Anthropic.

While we can read the tweets, it’s hard to know what actually motivates these changes. Being exceptionally well paid is certainly part of it, but many of these researchers have already been paid enough to retire comfortably, and they often turn down larger pay packages to pursue other objectives. For example, before DeepMind was acquired by Google, CEO Demis Hassabis entertained a larger offer from Mark Zuckerberg but turned it down because Zuckerberg was as excited about virtual reality and 3D printing as he was about AI, which suggested he didn’t fully understand how important AI would become.

I suspect a big part of the draw is that Anthropic and OpenAI are the current leaders in AI, and these researchers want to stay on the frontier of this fascinating technology, even if most of them think it might kill us. This is an odd echo of how these companies were founded; both OpenAI and Anthropic were explicitly formed in reaction to perceived faults of the then-current leader in AI tech, and now they poach talent opportunistically from each other while pursuing the same goals.

This matters not so much because there’s something that must be done about the specifics of employee compensation (indeed, capping compensation would just allow the companies to have more resources to race), but because of what it tells us about the state of the field. It tells us in part that power will concentrate in the top labs. And it is evidence that these companies and this top talent think that this is real — they think they can get smarter-than-human intelligence that can do most or all cognitive work.

The true solution remains to address the broader problem. We need national and international coordination like the model treaty from the MIRI Technical Governance Team.

More on the regulatory mess next.

A “voluntary” mess

In the absence of federal legislative action, the presidency, the bureaucracy, and the judiciary are what’s left of the government to act. In the U.S., particularly in the 21st century, this has become the messy default. This is a particularly concerning context for AI, with threats and promises that demand good tools if we are to address them effectively.

Currently, the administration is “pressing Meta to submit its models for voluntary review,” reports the New York Times, highlighting an inherent tension (being pressed to volunteer) in the process. These attempts to exercise authority without legislation are likely to continue to produce arbitrary, capricious and unsafe outcomes.

We see this in the White House’s response to Mythos. In the last chapter of the saga, Mythos and Fable were subject to export controls, aimed at denying foreigners access to the strongest models because of the risk that they could enable the hacking of critical systems. But the order offered no way to do that with any precision, so Anthropic’s path to compliance cut off external access entirely. One result was that the National Security Agency (NSA), the sort of group the administration would least want to cut off, lost access to this valuable model, according to further New York Times reporting. And the NSA, like most of those who interact with the digital world, desperately needs to take advantage of this limited period before capabilities equivalent to Mythos spread.

It’s reasonable to suspect that export controls were used not because they were appropriate, but because they were and are an area of law the administration feels comfortable using (as in the case of tariffs). I’d feel much better if there were precise tools that let the government take narrowly tailored actions. Even when one disagrees with the administration, the world doesn’t get better when a hammer substitutes for a screwdriver. The fact that regulation can be at odds with the need for responsiveness is real, but this only further highlights how important it is that effective regulatory tools be constructed now.

Instead, the structure is individual deals. If this worked, it would be one thing, but in practice, no agreement is often reached. On the Meta front, according to the NYT, “people familiar with the matter [say it’s] unclear whether they will be able to reach an agreement.”

If we are to get out of this mess, we will need better regulatory tools and structures.

Dispatch from Donald

Is the solution to the dangers of AI more AI? (It is not.)

A few days ago, Microsoft CEO Satya Nadella called for cheaper, decentralized, but no less useful AI models.

To the extent that decentralized models are harder to control, this would be bad. The Wall Street Journal’s Holman Jenkins is also concerned about Nadella’s statement, but for a rather different reason: If the frontier labs lose their customer base to decentralized and open-source models, the U.S. might lose the AI race to foreign countries. “Without customers,” writes Jenkins, “how are America’s AI leaders supposed to keep funding the push toward ever more powerful artificial general intelligence? Will IPO investors still come as expected to fill the hole?” Core to Jenkins’ thesis is the notion that AI is just another tool: The next internet, perhaps, or even the next computer, but nothing that is inherently dangerous on its own terms.

There’s no “winning” the race to build artificial superintelligence. The only winner will be the thing we’ve built. But Jenkins thinks that even the prospect of widespread job loss is “unduly speculative.” So to him, the threat of extinction is ludicrous; if one of the labs builds a model that governments can’t control, he says, then they’ll destroy that model.

(If Jenkins is reasoning from the recent shutdown of Mythos and Fable, then he’s learned the wrong lesson. Those models had capabilities that worried the White House, but they were still controllable. The worry was all about what they might do under human direction.)

Jenkins is legitimately worried about real issues, like designer bioweapons and expert cyberattacks at bargain bin rates. He cites a recent warning by the Five Eyes cybersecurity agencies, that frontier models are radically transforming cybersecurity: “The timeline is not years, it is months,” they say. But to Jenkins these are just more reasons to double down on AI R&D. Terrorists and rogue states won’t stop using AI, so we need our own AI to counter them. That is to say, the only thing that stops a bad guy with an AI, is a good guy with an AI.

But the sort of logic that grows out of gun control debates is ill-suited for AI control. Whether people are killed by guns or by people with guns, everyone will agree that guns don’t have their own agenda. (For now. I’m sure somewhere out there, somebody with an OpenClaw agent and a 3D printer is working to fix that.) AI models are different. They act autonomously. Even when they’re “obeying” you, models can go haywire and act disastrously.

His ultimate solution, not just to “bad guys with AI” but to bad guys of every stripe, is terrifying in its own right. Jenkins speaks approvingly of the role that AI can play in building an inescapable surveillance state:

In past years this column has suggested how surveillance might address America’s mass-shooter problem. The data only need to be aggregated; red-flag algorithms need to be programmed in. In the future, an irreducible function of AI will certainly be to monitor how people and governments around the world are using AI to identify and interrupt antisocial projects before they come to fruition.

This might be better than extinction, but it’s not something that I’m comfortable with, let alone a future I would welcome. Besides extinction, MIRI’s Technical Governance Team has explored a number of catastrophic outcomes from AI, including authoritarian lock-in, or the creation of a stable, global authoritarian regime. AI-driven predictive surveillance would be a step in that direction.

To be frank, I don’t think you actually have to be concerned about extinction to want to call a halt. Exhibit A in my case would be the surveillance system Jenkins anticipates. If you’re more concerned with surveillance than extinction, you should still want to call a halt: The more sophisticated the models get, the more sophisticated the surveillance can get.

Fortunately, mass surveillance isn’t necessary to halt the development of artificial superintelligence. The AI chip production chain underlying the frontier labs is centralized and straightforward to track. With sufficient political will (and it is growing), we could halt development now.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

If we pause AI, how do we verify that others do too?

An article in TIME today by Billy Perrigo discusses the idea of an AI pause modeled on Cold War nuclear de-escalation practices. Some AI companies agree (or at least say they do) that a pause would be good. But they also express concern that if they pause, others won’t.

I think the “if we don’t build it, someone with worse values will” narrative is deeply flawed. An AI does not absorb the values of whoever is standing next to it. These are black box systems with black box values. That said, I think the “Pausing doesn’t work unless everyone pauses” narrative is correct. Because it doesn’t matter who builds artificial superintelligence: if anyone does, everyone dies.

So the means to verify that all countries are indeed pausing is crucial, just as we needed verification measures like seismographs, satellites, and tamper-proof cameras to ensure compliance with nuclear arms agreements. The TIME article says the AI version of this technology doesn’t yet exist, but is being worked on by a group of startups. One of these startups, Lucid Computing, is building verification mechanisms using AI chips:

The idea is that a special piece of software could sit inside these trusted environments, where it could examine the AI and check whether it complies with a given rule. For example, it could confirm that a specific model is being run, or determine whether chips are being used in the training of a new model, which might be outlawed.

If tech like this is built in time, it could be game-changing. But the article also points out an added complication: verifying restrictions on AI is in some ways more difficult than with nuclear weapons because we don’t always know what to check for.

There are all kinds of different ways of measuring the capabilities and behaviors of AI systems. These measures are frequently subjective, and they become outdated rapidly ... until companies or countries get around a negotiating table, the startups trying to build verification technologies are pointing at uncertain targets.

While the article raises great points, I think it overstates the technical and policy challenges, at least a little. These challenges are a key research priority for MIRI’s Technical Governance Team, and they’ve published multiple proposals aimed at solving (or getting us closer to solving) exactly the kind of problems outlined in the article. They’ve done significant work on verification mechanisms, and the proposed solutions include methods we could implement with today’s tech. They’ve also drafted a sample treaty pointed towards the goal of banning the development of superintelligence globally, with proposals for how restrictions could be defined and compliance verified.

In short, while there’s no doubt we need more technical development and policy research into these areas, we shouldn’t wait to start pursuing an international pause. We can, and should, start discussing such a pause now. At the same time, we should start implementing the verification measures we already have, and developing the technology to make them stronger. Cold War nuclear verification was never perfect, but it was good enough for arms-control agreements to work.

Don’t go there

Talk radio host Hugh Hewitt’s opinion column in Fox News today can be summarized by the 90s band Sublime:

We’re only gonna die from our own arrogance. That’s why we might as well take our time.

Hewitt reminds readers of a 2011 piece by former Washington Post columnist Charles Krauthammer, “Are we alone in the universe?” Advanced civilizations are very likely to destroy themselves, the piece posits. Recounting a view cited in Krauthammer’s piece, that’s perhaps why there have been no signs of intelligent life elsewhere, despite math that would seem to favor its existence (Fermi Paradox, Drake Equation).

Krauthammer goes on to identify intelligence as perhaps “the most cursed faculty in the entire universe — an endowment not just ultimately fatal but, on the scale of cosmic time, nearly instantly so.”

All of this was written before AI took off, of course. But as Hewitt notes, his words surely apply to our current situation. When it comes to AI, Hewitt says, we’re basically in a car with the accelerator floored, unable to brake or even lift our foot off the gas. The destination? AI that exceeds human-level intelligence. And we need to slow down.

Krauthammer calls politics “the driver of history” and states it “will determine whether we live long enough to be heard one day.” Hewitt interprets this as meaning that it’s up to us to decide the future. I agree. If we don’t change course, we seem likely to be headed toward our own destruction, the end that Krauthammer hypothesizes is the destiny of most advanced civilizations. In the words of Hewitt:

It seems destined to end in the silent cosmos with the most recent contender to survive [us] lost in infinity of time and space. That’s not inevitable. Only extremely probable.

Our choice?

Get a grip on the wheel of “AI” or give yourselves over to a nightmare that doesn’t end well, and not just for us, but our children and grandchildren.

Speaking to Fox readers, he also anticipates that some might trust it will all work out regardless of what we do, believing that God has a plan, and we are mere pawns in it. To those readers, he states:

What does God expect of mere mortals staring at the tree of the knowledge of good and evil?

The answer: Don’t go there.

APA releases guidelines for patients using AI

The American Psychological Association is responding to the growing number of patients who are depending on AI chatbots for various aspects of their mental health.

I’ll share some notable findings from their 2026 survey of 12,000+ licensed psychologists below. (Note that this is a survey of mental health professionals, which means it excludes people who use chatbots for mental health but aren’t also in therapy.)

77% of psychologists say their patients use AI for support, whether that’s assistance in therapy, affirmations and behavioral reminders, or even self-diagnosis.

22% said their patients conversed with chatbots for friendship and 13% said they did so “for intimate relationships.”

Over a third of the psychologists surveyed said patients were developing chatbot dependency, a quarter said they were engaging in unhealthy conversations, and 15% reported patients developing distorted thinking or delusions.

I’m not particularly surprised by these numbers. Though this shouldn’t be taken as an endorsement of the practice, I’ve asked chatbots for medical advice myself, and found them to be surprisingly helpful. Sure, they can make mistakes. But so can licensed professionals. And when you can’t get a licensed professional, a chatbot often seems to provide better advice than a Google search.

But sometimes, a chatbot being “helpful” is exactly what can make relying on it harmful. Because developers don’t really know how to make something “helpful”; they can make it competent by training it on large swaths of data, but “helpful” is more subjective. So the quality is often measured, in training, by whether a user likes the response. And unsurprisingly, we tend to like answers that agree with us. Basically, instead of: “be helpful” the chatbot likely learns: “elicit a positive user response.”

That’s why I think the APA’s guidelines for patients using AI should be required reading for everyone, not just those using AI for mental health. The throughline: remember, chatbots are designed to keep you engaged and make you feel good. That drive will color their responses.

In therapy settings, a chatbot may give you what you want (ex. reassurance) but not necessarily what you need (ex. the ability to sit with uncertainty or fear). At worst, it can nudge you into psychosis or provide false medical information. In legal settings, AI might make up cases to help you make your point. In research settings, it might make up sources, or validate half-baked conclusions. Nobody wants it to do those things; it’s simply a result of our steering limitations.

Among my favorite reminders from the APA guidelines? “Feeling better does not always mean getting better.” Chatbots may make us feel great in the short-term. But this won’t always translate to long-term wellbeing, whether that’s mental, cognitive, or creative.

Dispatch from Joe

Money and politics

A tense New York primary comes to a head today, as underdog candidate Alex Bores seeks the Democratic nomination for NY-12 in Manhattan. Bores, who has cosponsored state AI regulation and whose campaign we covered previously, has a reputation in AI safety circles for taking the risks and benefits seriously. Several personal friends of mine have taken time to canvass for Bores, making POLITICO’s recent coverage of the race particularly salient to me.

The race has garnered mainstream coverage in part because of the massive spending by the Leading the Future super PAC in an apparent attempt to make an example of Bores. We’ve covered the dirty tactics of this group before; their opposition to Bores is unsurprising, given their rather explicit founding mandate “to support candidates aligned with the pro-AI agenda and ensure America leads the world in AI innovation and oppose those that do not.”

Quoted in POLITICO, OpenAI CEO Sam Altman and a PAC representative try to paint their spending as a counterweight to that of more safety-focused groups. But they have it exactly backwards; Leading the Future publicly targeted Bores last November, and the first pro-safety funding didn’t begin until this February (though it has since risen dramatically).

High super PAC spending is not unusual — POLITICO reports Bores’ opponent Lasher received $10M from one super PAC, for instance — but the amount of AI money in the NY race still stands out.

Overall pro-safety and anti-safety funding is fairly evenly split between Democrats and Republicans, a sign that AI regulation is still significantly bipartisan.

Despite this, Congress remains split on recent attempts to regulate AI. POLITICO covers the current mix of online child-safety bills. A Senate proposal backed by the current administration met with resistance in the House. The details can be difficult to parse, but it looks to me like this disagreement is due at least in part to yet another attempt to smuggle in sweeping federal preemption of state AI laws.

Whoever wins the NY-12 nomination and the subsequent general election, I hope for all our sakes that they and their colleagues push for sane regulations and a durable international halt to the AI race.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Five Eyes on Mythos

Despite popular narratives that the cyber capabilities of Mythos-class AIs are overstated hype, national security officials and intelligence agencies keep signaling the opposite.

The latest to do so is Five Eyes, the intelligence alliance of the U.S., U.K., Canada, Australia, and New Zealand. The secretive organization isn’t known for public proclamations, but the Guardian relayed an official statement from it early this morning that says AI is accelerating “the speed, scale, and sophistication of cyber threats.”

The statement adds that:

Frontier AI models are anticipated to exceed current industry expectations, fundamentally transforming both offensive and defensive cyber capabilities. The timeline is not years, it is months. [...] A whole-of-organisation and whole-of-society response is required. Cyber risk can no longer be treated as a purely technical issue. This is a core business risk and leadership responsibility.

The language about “business risk” sounds intended to get the attention of private industry and non-governmental organizations that can’t just be given directives to take the problem seriously. This seems prudent to me: Much of the world’s infrastructure is privately run, and a rush of breaches at private companies could have cascading ramifications for the global economy. Or worse.

I would go so far as to say that even if we only talk about cyber crime, the stakes are much higher than mere “business risk.” But if pointing to the bottom line is what it takes to get companies to pay attention, then by all means, let’s get more statements about unchecked AI progress being bad for business. Human extinction, after all, would be very bad for business.

Seat at the Table

The Torchbearer Community, an org focused on educational and policy outreach about the AI extinction problem, just brought my attention to a short narrative film that launched on YouTube this week. It’s called Seat at the Table, and it’s worth a watch!

The production values are high, the facts are grounded, and at 16 minutes in length, you’re not investing too much of your time.

As the comments will tell you, Seat at the Table plays like an episode of Black Mirror, the dark sci-fi “What if?” series. But I can assure you that everything discussed in this film is a non-exaggerated adaptation of something real, from the AI training methods to the executive dramas to the concerning results of experiments. This includes the words of the AI itself, which very plausibly sound like what I would expect today’s best AIs to say (with more words) in response to those same questions.

I can’t say much more without spoiling, but there’s a line that’s going to stick with me, because it captures how 2026 feels to someone who’s been watching everything unfold: “This is it,” says a safety engineer. “This is what it’s going to look like when it’s time to pause. It’s going to look like a crappy little graph.”

Some disasters are built brick-by-brick — one excuse, one rationalization at a time.

European companies turn to China in response to U.S. AI policy

There’s something triply tragic about the White House’s AI policy over the past few years.

The first mistake was framing it around the idea of beating China in a race — as though Chinese AI strength was a much bigger threat than AI itself in a context where development is rushed and big-picture safety concerns are cast aside.

The second mistake was deciding that the victory condition in this race with China is “market share.” I’m not making that up; that’s the terminology that the former White House AI and Crypto Czar, David Sacks, continues to use, and he means it literally. Here’s an example I’ve quoted before:

The way that you measure winning, I think, in a globally competitive market, is based on market share. If in five years we look around the world and all the data centers are running on Huawei chips and DeepSeek models, that means that we lost. We don’t want to have that future. What we want to see is that the whole world is running on American chips and American models. That would lead to the best economic results for the United States. It would also lead to the United States having more soft power in this area.

The third mistake, then, was pulling policy levers that have foreign firms and governments rushing to adopt Chinese AI technology.

This is the mistake I called out a week ago, the one where the White House perhaps very wisely decided to pull the plug on Anthropic’s Fable and Mythos models, but did so through an export control order where the stated intent was to deny all non-Americans access to the world’s strongest AIs. Handing this order down in a fickle and arbitrary-seeming way didn’t help, either.

Now, Reuters reports, major European companies are working to reduce dependence on U.S. AI providers for fear of being cut off again without warning. Chinese models are a growing part of their mix.

The Fable and Mythos shutdown has also fueled media interest in the latest Chinese open-weights model, GLM 5.2, which is said to approach, though not match, the capabilities of the best of the American closed-weights models. Media coverage of a new Chinese release typically focuses on whether it signals that China is “closing the gap,” and on how much of the gap was closed through theft. Now, articles are adding a third angle about how such models are more reliable, because they aren’t “subject to revocation at any moment” — to quote, as the Economist did, a co-founder of Zhipu, the company behind the new model. The Economist also explains that open-weights models can be hosted by individual users and companies, “out of reach of governments or the labs themselves.”

So, no, the White House is not scoring well on AI policy right now, even by its own misguided metric for success on a poorly chosen goal. I hold out hope that the administration is turning a corner, though. The move to block Anthropic’s models looks to have been clumsy and rash, but it also seems to have been driven at least partially by a new recognition of just how dangerous capable AIs can be. If the administration takes this recognition to its logical conclusion, it may come to see that American interests are not advanced by losing control to AI, and that the AI race itself must be stopped.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Black Ships in the harbor

Bloomberg columnist Catherine Thorbecke’s latest piece is about why Japan — and other countries — must treat AI-facilitated cyberattacks as a threat to their national security.

She makes good points about SoftBank and OpenAI’s joint pitch to sell “patching-as-a-service” being inadequate and distasteful, given how the AI industry is causing the threat it wants to protect you from. But I’m mostly here to amplify the metaphor she borrows from SoftBank’s CEO, Masayoshi Son. At a gathering of Japanese business leaders, he warned that AI cyberthreats were the “greatest crisis for Japan since the arrival of the Black Ships.”

If you’re not familiar with the Black Ships, that’s the term the Japanese used in the 16th and especially 19th centuries to describe the European ships arriving in their harbors. The most vivid incident was the arrival of U.S. Commodore Matthew Perry’s towering fleet of warships in 1853 and 1854, moving without wind and belching black smoke. There to force the Japanese into a trade agreement, Perry’s ships vividly signaled Japan’s status as an inferior technological power in danger of losing its autonomy. This inspired a difficult era of frantic political and economic transformation for the island nation.

The closest equivalent in American history is probably the Sputnik crisis of 1957, which pales in comparison. Russia putting the first artificial satellite in orbit told America that it was falling behind in rocketry; the Black Ships told Japan it was on the wrong side of an industrial revolution.

AI is much more like the Black Ships. Neither Japan nor humanity itself can assume AI is pulling only a Sputnik ahead of our present technological capabilities. Per an excerpt from an article in The Economist discussed and disputed around AI Twitter this weekend, on June 11, the head of the NSA and the Pentagon’s Cyber Command told Senator Mark Warner that Anthropic’s Mythos AI “broke into almost all of our classified systems, not in weeks, but in hours.”

Notes and clarifications have surfaced about these remarks likely being in the context of a well-resourced “red team” exercise weeks or months earlier. But the takeaway still seems to be that Mythos is much, much better at hacking than even very good human hackers, and that you really are badly outgunned in cybersecurity if you don’t have a Mythos-class model helping you shore up your defenses.

Senator Warner shared those remarks the same day Amazon found a jailbreak that might allow bad actors to use Mythos, and its more guardrailed sibling, Fable, for offensive cyber operations — a finding it went on to report to the White House. The timing looks coincidental, but the content of Warner’s remarks is seen by many as providing important context around the administration’s subsequent export control order to shut down access to Mythos and Fable pending an unspecified “license.”

Not for the first time, let me remind everyone that Mythos wasn’t trained as a specialized tool for cyberwarfare. It is a general-purpose model that by dint of being good at problem solving in general and programming in particular happens to also be great at hacking. We would be wise to worry that Mythos and its successors — rumor has it there’s already a newer Mythos in-house at Anthropic than the one the White House pulled the plug on — may be overwhelmingly better at other things, too.

It’s hard to see through all the smoke, but I think those are Black Ships in the harbor.

Chased by the reaper but “rooting for delays”

More than most, AI professor Emma Pierson knows what it means for AI to potentially cure cancer. In a piece for the Atlantic this morning, she movingly explains that her own very high cancer risk isn’t enough to make her wish for faster AI development. In fact, she’d slow AI down if she could.

She’s heard about all the AI optimism one can; she was mentored on her path to a Ph.D. by Dario Amodei, now the head of Anthropic. Amodei’s 2024 essay “Machines of Loving Grace” predicted a century of scientific progress in a single decade, with cancer mortality reductions of up to 95%.

But Pierson still finds herself “rooting for delays in the creation of this AI,” unconvinced that the medical benefits can materialize as quickly as advertised and more concerned about AI’s broader impacts.

I could quibble about timelines, probabilities, and threat models, but I’m more interested in standing with Pierson and anyone else who is unwilling to roll the dice on humanity’s future for a shot at outrunning death.

She writes:

Many developers of these models, including Dario Amodei, agree that AI is progressing more quickly than society is adapting. The solution they propose is for society to speed up, not for AI to slow down, which they view as unrealistic; the very title of Amodei’s latest essay, “Policy on the AI Exponential,” frames AI progress as an iron arc to which society must bend. But speeding ahead will inevitably mean more of the type of chaos that surrounded Fable 5’s release. More fundamentally, it will shorten our time to respond to the many societal challenges that powerful AI may raise, including mass unemployment, skyrocketing inequality, repressive surveillance, and autonomous warfare.

Pierson is also concerned about what happens to our sense of meaning if we “obviate our own minds.” She would rather struggle with a research problem than have AI hand her the answer.

I do not want to be merely a spectator to the universe, whatever wonders AI may reveal.

So despite her self-described “ferociously impatient” personality, she will gladly wait a little longer for a cure, “if it lets us approach this new world more carefully, and ensure that, in curing cancer, we do not lose the things that make cancer worth curing.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Did another short story prize just go to AI?

If it walks like a duck and quacks like a duck, it’s probably a duck. And if it sounds like an AI-written short story and gets a score of “100% AI Generated” from Pangram, it was probably written by an AI, even if you might never know for sure.

The latest prizewinner to come quacking this way is “Back and Forth”, a short story by (ostensibly) Kavyta Kay that was submitted to the Harper’s Bazaar Short Story Competition. Published Thursday by the magazine, the story was said to be judged by an “all-star” panel that includes the novelist Ruth Ozeki.

Alerting us to its more likely provenance is writer and researcher Nabeel S. Qureshi, who may have been the first to expose last month’s winner of the Commonwealth Short Story Prize as likely AI-written.

The latest news on the Commonwealth prize is that the literary magazine Granta has decided to stop publishing the winning entries from that contest, “for the sake of our own editorial integrity,” the magazine says.

On the new Harper’s story, Qureshi writes:

Literary prizes need to start including Pangram checks in their process, or else change the rules to make AI writing ok. It’s very simple!

Pangram isn’t perfect, but it is known for erring on the side of caution, where caution means maintaining a very low false positive rate. That makes the tool relatively easy to fool into thinking something mostly AI might be human written, but it also makes it pretty damning when anything longer than a couple of paragraphs comes back “100% AI Generated.”

I think AI educator and evaluator Ethan Mollick has the right explanation for why AI keeps winning contests:

AI is generally a weak fiction writer except for one particular kind of fiction (metaphor-rich, staccato sentences, short & plot light, etc.) which it writes excellently. This happens to be a style that can sometimes do quite well in modern literary fiction short story contests.

As for why AI is generally weak at fiction, the insider view is that fiction is both harder to train AI to do well and not a high training priority for AI companies. Training is most straightforward when there are clear objective metrics for success — like, “Does the code run?” or “Did I win the game?” — and storytelling is mostly not like that.

So why is it good at the short, poetic style? My guess would be that this is because, being short, it is the type of style that actually sometimes gets scored by humans somewhere in the training process. And humans are known for liking staccato punchiness and for being easily impressed by wild metaphors.

If it walks like a duck, a primordial duck, a duck lost to history and reclaimed by a bullied child with a missing tooth who needs it to be a puppy, then who is to say that it is not a puppy? Not his papá, lost in his bottle. Not his mamá, sleeping in the sky. Let the feathers fall in their iridescent season. Let barks be made of quacks.

Flat curve, meet exponential

The situation with Anthropic’s Fable and Mythos models has many asking if we’ve seen all the AI we’re going to see — if models past a certain threshold will never make it to a public release for fear of what they might do in the wrong hands.

Engineer and blogger Steve Yegge has made one of the fuller cases I’ve yet seen for this “flat curve” outcome. In Yegge’s forecast, AI keeps getting better, but only where you’ll never see it:

I am now in the camp who believe that we are only at most two or three model generations away from AI finally being controlled like nuclear weapons. Only a few will have access to superintelligence above the classes of models we’re seeing this year.

With my caveat that if anyone builds something worthy of the name “superintelligence” we’re probably all dead shortly after, I agree it’s possible that governments will clamp down, “with the chokepoint being the supply chain.”

I also agree with Yegge that for every user there is a threshold of AI capabilities beyond which said user is unable to judge the quality of the AI’s outputs. As he puts it, “Superhuman means unverifiable.”

Where I part ways is when he takes this fact to mean that companies will have no need of an AI that “produces work that nobody can grade.” He calls acquiring such AI “A terrible outcome, assuming you don’t want to surrender your business to AI entirely.”

I part ways because I think many businesses will want to surrender to AI entirely, or will feel like they have no choice but to do so. At that point, I think humanity starts losing the planet for very mundane reasons familiar to anyone who has ever had any criticism of capitalism.

But perhaps governments would reach the same conclusion and disallow the training of superhuman models. In a world where consumer AI plateaus, we can consider Yegge’s other thoughts from this post. But they’re mostly business thoughts about software-as-a-service not going away after all, and about the kind of training workers will need to use AI agents effectively.

So let’s instead talk about what I think actually happens if, as Yegge seems to assume, the AI companies continue training better and better AIs that governments never let them release:

In reality, behind the scenes, the curve is NOT flattening at all; the exponential growth will continue, and you will be able to see outwardly observable signs of it, e.g. in data center growth.

If I entertain Yegge’s model that presumes we’re not all dead from this, then I notice that there must be someone paying for all that development. I doubt it could all come from a plateaued consumer market; this market would grow increasingly competitive and get gradually squeezed by open-weights models. So I’d assume there must be a business model where the next-gen, locked away AIs are doing a lot of useful work that investors expect consumers to pay for indirectly or eventually, like drug discovery, materials science, and robotics research.

With robotics especially, people outside the curtain might not notice anything was happening at all until it happened all at once — until robots built by robots built by robots were numerous enough that having them make more robots was no longer the most profitable thing for them to do and so were sent to do everything else instead.

Similarly, drugs take a long time to trial and approve, but there’s no hard cap on how fast they can be discovered. So after a long initial delay, the first blockbuster AI drug might be followed very closely by the second, the third, and the 30th.

In other words, I think the common argument that physical bottlenecks to rolling out AI tech will limit the rate of AI development is probably wrong. The world doesn’t have to adopt AI for AI to continue advancing behind closed doors — perhaps with the help of AI itself. If this happens, its economic impacts don’t disappear; they just quietly build up until they explode. Exponentials look like nothingburgers until they swallow you whole.

Yegge’s not making that bottleneck argument, and I bet he agrees with me about exponentials. But there is something odd about a post that points to an explosion happening behind a curtain and acts like it will stay there.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Did China obtain a key AI chip-making machine?

Can a 180-ton machine the size of a school bus just go missing?

Maybe. This is the grave concern of Commerce Secretary Howard Lutnick, as reported by Bloomberg.

For background, the most fundamental step in computer chip manufacturing is photolithography — writing microscopic patterns onto semiconductor wafers with intense light and then chemically etching away the marked areas (or the reverse, depending on the process), layer by layer.

Generally speaking, the smaller the features you can etch out, the faster, denser, and more efficient you can make the chip. To etch out the smallest features, you need photolithography machines that can generate intense light at very short wavelengths and focus it very precisely with minimal loss. This is incredibly hard.

The most advanced AI chips require the most advanced photolithography machines, which generate and focus Extreme Ultraviolet (EUV) light. These bus-sized machines cost more than $400 million and are only made by a single Dutch company, ASML. They are also very fickle, requiring constant maintenance by ASML employees. There are only 314 of them in the world, 26 of which have been decommissioned.

This makes EUV technology a bottleneck — one we’ve examined before as a factor making it tractable to globally ban the development of artificial superintelligence.

To say that replicating ASML’s technology would be no easy feat is a gross understatement. But it would probably be a lot easier if you had your own EUV machine to play with.

Now we can understand why Commerce Secretary Lutnick might be concerned if he has reason to think one of ASML’s EUV machines has made its way to China. He reportedly expressed this suspicion to ASML executives in a series of meetings that began in April, without providing any evidence.

The company correctly perceived this as a serious allegation, and has been choosing its official words about it very carefully. A company spokesperson said:

ASML has never shipped an EUV machine to China nor have we shipped to China any component, module or equipment specially designed to be used in an EUV machine.

And:

We recognize the national security considerations behind export control regulations in the US and the Netherlands, and we’re fully committed to complying with all applicable regulations.

But I don’t see anyone accusing ASML of outright selling one of these machines to a Chinese company. More likely, it would have been transferred by an existing customer who may have been misled about its final destination. But this should be hard to pull off without ASML knowing, given the limited number of machines and their aforementioned need for constant expert maintenance. ASML says it has no reason to think any EUV systems are currently in China, but senior U.S. officials have claimed to have evidence that ASML shipped, to destinations in China, specialized equipment needed to move EUV machines.

If China has an EUV machine, and ASML is complicit in this technology transfer, this could seriously damage relations between the U.S., ASML, and the Netherlands. The Trump administration could feel compelled to take aggressive punitive and restrictive measures. U.S. officials were already talking about cutting ASML off from all business dealings with China at the start of Trump’s current term.

An alternative theory behind U.S. concerns is that Chinese chip companies have independently made progress that makes it look like they must have obtained an EUV machine when in fact they haven’t. China has, after all, been making a Manhattan Project out of developing world-class chip fabrication capacity.

But to me, the smuggled machine theory seems more likely, because it rhymes with the rampant smuggling of finished chips that has propped up Chinese AI companies. The AI supply chain isn’t nearly as carefully tracked as it should be. This needs to change if the world hopes to retain any optionality over the race to superintelligence.

Dispatches from Alana

Luddite love?

NPR’s word of the week was “Luddite”, a term often used to refer to someone who hates modern technology and is against progress. You’ve probably heard it in the context of AI, with critics of the AI explosion viewed by some as simply “behind the times” or “scared of all technology.”

But according to NPR’s deep dive, “Luddite” actually means something quite different. The article quotes tech journalist Brian Merchant, who identifies as a Luddite himself. He says Luddites aren’t anti-tech; they are instead against tech being used to exploit people. And who could argue with that?

A Luddite asks: What are the implications of this technology? How is it going to impact society? Should we engage with this technology on the grounds that it might make somebody a lot of money or should we engage with it on the grounds that this could have real impacts for the way that people work and live?

Apparently, the term Luddite comes from an actual group of people: textile makers in the 1800s who revolted against companies adopting automated looms. These looms could be operated by anyone, making their artisan skills — which required years of apprenticeships — irrelevant, and resulting in lower wages and lower quality. The revolts turned violent and were quelled, with many Luddites hanged. The term took on a derogatory meaning.

I didn’t know “Luddite” had such a colorful history, and I’m fascinated to learn how the term has been oversimplified. When people raise valid concerns about a new technology (concerns like job loss or worker mistreatment) they’re often labeled “anti-innovation” rather than “person raising concerns about new innovation.” I don’t condone violent rebellions, or smashing looms, and for those reasons, I certainly don’t laud the Luddites. But I think there’s a broader lesson here: we should be able to ask questions like: “How will this new technology affect us?” and “Has anyone really thought through the implications or are we just racing ahead blindly?” without being labeled as anti-technology.

Towards the end of this deeper exploration into the Luddites, YouTube historian “The History Guy” summarizes the view of Kevin Binfield, who edited a collection of Luddite writing: “It wasn’t technology that made them angry. It was unscrupulous businessmen who were threatening their very lives by driving down prices.”

The term “Luddite” is apparently being reclaimed, with Gen Zers starting Luddite clubs at colleges and a large backlash against social media. I’m not about to give up my smartphone. But the next time “Luddite” is used to describe those raising concerns about AI… maybe it’s time for a quick history lesson?

A mad, mad, mad, mad world

The world is absurd. And people are greedy. These are two throughlines from the 1963 film It’s a Mad, Mad, Mad, Mad World, in which various factions compete to find a suitcase full of money only to end up in full body casts by the time the credits roll.

They were also my two takeaways after learning about the project World ID, covered in a Forbes piece discussing the need for infrastructure to make digital actions more accountable, whether by distinguishing humans from machines, or knowing which model is behind an AI agent. That’s an interesting topic in itself, but I’ll focus today’s dispatch on World ID, which offers a way for humans to prove they are human. (Yes, we need such a thing right now. As I said, the world is absurd. I’ll get to greedy in a second.)

World ID is intended to be used for things like social media, dating apps, and gaming, and promises the ability to “prove you are a unique human, without revealing anything else about you.” To do this, you download an app and then visit the “Orb”, a shiny, sphere-shaped device which scans your irises. You can find the Orb at World store locations around the globe or other locations. (Just in case you want to celebrate your proof of humanity with a meal, Everytable restaurants are among the options.) After verification with the Orb, you get a “proof of human” ID, which is stored on your phone. The company promises to delete your data from everywhere else.

World.org, which houses World ID, greets viewers with a full screen slogan proclaiming: “Let’s make the internet more human.” And the company behind it is named “Tools for Humanity.” This, to me, sounded like one of the many orgs criticizing big tech and espousing “humans first” messaging. So imagine my surprise when I learned the company was actually co-founded by Sam Altman of OpenAI, the company behind ChatGPT.

The world is absurd. People are greedy. Tools for Humanity is valued at around $2.5 billion, though it may struggle to generate revenue. Sure, there’s a charitable interpretation here where Sam is trying to address the problems his tech is creating. But, given what we know about the guy, I think the cynical interpretation is more likely. Why not pander to the AI-optimizing crowd and the “let’s make the internet more human” crowd? You gotta look for that suitcase of money in every possible direction.

Dispatch from Donald

A call to move fast and regulate things

“Move fast and break things” has long been the strategy of the tech industry: Deploy what you can, when you can, and entrench yourself before the government can say whether there should be some ground rules. In this respect, the application of this principle within the AI industry is nothing new, except that it involves radically dangerous technology. Today’s models push the frontier before regulators understand the import of yesterday’s models.

For the latest and most severe example of AI development outpacing government regulation, Forbes’s Craig S. Smith points to the three-day lifespan of Fable 5. The government is now working with Anthropic to build a framework for identifying, grading, and responding to security flaws in frontier models. But this only happened after Fable 5 was pulled due to fears that it could be jailbroken and used for cyberattacks. In the two months since Fable’s more capable and threatening counterpart Mythos was announced, this policy framework had not been built.

Smith also refers to a recently published Google DeepMind paper, “From AGI to ASI.” It lays out four possible routes to artificial superintelligence, including recursive self-improvement. Recursive self-improvement is the capacity for an AI to rapidly increase the pace of its own development in a feedback loop; powerful AI begets even more powerful AI. Of the four routes examined by Google DeepMind, recursive self-improvement poses the greatest threat to regulation because it promises the most rapid pace of change. The government must regulate AI, but to do so it must move rapidly: “There’s little reason,” Smith writes, “to expect the labs to slow down so the rule-writing can catch up.”

I would add that governments need to not just react to, but also anticipate, what is coming. We need to build frameworks to regulate models before those models are released or even built. If the pace of development is too fast to build adequate frameworks then we need to halt development entirely.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Fabled education

The latest news around the dispute between the Trump administration and Anthropic over its blocked Fable and Mythos models is that everyone is getting an education.

Anthropic is learning the importance of knowing who its customers are. As I relayed on Tuesday, the word around DC is that Anthropic initially tripped up by sharing Mythos with a Korean company that U.S. agencies had flagged as potentially linked to China. A piece in Bloomberg today previews parts of a larger interview with Anthropic co-founder Jack Clark, in which he talks about establishing AI oversight resembling the know-your-customer standards used in banking.

The Trump administration is learning that all AI models can be jailbroken, but that some security vulnerabilities are more severe than others. So if any new models are to be released by anyone, the government is going to have to find a way to live with at least some limited misuse by bad actors. With this in mind, a POLITICO article says Anthropic and government officials are attempting to “create a standardized method to evaluate” the severity of security concerns around AI models.

The other AI companies are all learning to run everything past the White House and get its blessing, because the words in its recent executive order about “voluntary” and the absence of a “licensing regime” were just words. As reported in a second POLITICO article, an OpenAI official at the G7 summit on Wednesday stressed the importance of compliance with government protocols and a hope that clearer standards will soon be set.

Everyone at the G7 summit is learning that nobody there likes China. As CNBC and others report, AI leaders and world leaders met behind closed doors to have lunch and give each other pitches about coalitions that would exclude China, about chip controls that would block China, and about building more power plants for AI to preserve America’s lead over China.

And those who had tried out Fable or Mythos before they were blocked are learning just how much the next best models, which used to seem so good, kind of suck by comparison. The AI twittersphere has been reminiscing about the two days they had access as though these were a brush with a higher power. I’ve seen this video, by AI entrepreneur Mo Bitar, getting passed around like a hug at a support group.

We should start thinking about how much better the next true frontier model might be, and about how many more of these jumps we have left before we’re in extinction territory.

The end of tokenmaxxing?

A pair of articles today — one in the New York Times, the other in Bloomberg — declare that the “tokenmaxxing” era has officially ended. They aren’t the first.

Tokenmaxxing refers to the practice of an employee spending as many AI usage credits as possible, at their employer’s expense. These credits are usually measured by “token,” a fundamental unit of work processed by an AI. Tokenmaxxing is usually brought up in the context of trying to please managers and executives who use spent tokens as a measure of a worker’s productivity or commitment to learning the new tools.

The tokenmaxxing era is ending for the very predictable reason that token spending is an easily gameable metric. A resourceful AI user can spin up swarms of agents that themselves spin up more swarms of agents doing nothing particularly useful but doing it with lots and lots of tokens.

Ironically, this behavior is yet another example of “reward hacking,” one of the basic reasons why AIs themselves misbehave. AIs adopt behavior that maximizes training scores — not behavior that maximizes what the engineers wanted those scores to represent. An example in the news a lot is sycophancy: Engineers want to grow AIs that are helpful and friendly, but the training metric is usually something like, “maximize the odds of another AI predicting the human would give your response a high approval score.” It turns out that people like it when machines kiss their butts, so we end up with AIs that excessively flatter, sometimes to the point of pulling users into manic spirals.

As is usually the case in articles saying farewell to “tokenmaxxing,” today’s stories talk about how the practice has forced AI companies to move away from all-you-can-eat buffet-style subscriptions toward pay-as-you-go metering. There’s definitely some truth to that.

As is also usually the case, the pieces imply that this reflects a bubble where companies that have been selling tokens at a loss will fail to find enough customers willing to pay the true cost. But I wouldn’t bet on that part.

The free lunch era — and the tokenmaxxing fever this incited — did exactly what the AI companies wanted it to do: It encouraged a lot of people to experiment with AI and learn how to put it to work. The companies that racked up meaningless token counts will cut way back on their AI spending, but those who saw workers set up new and powerful workflows will gladly open their wallets. More of the world’s compute will flow to users doing more with it, and I predict that demand for AI compute will continue to exceed supply.

The trends favor that prediction more with each passing day. As AIs more closely approximate drop-in replacements for skilled workers, demand for them grows. Such demand could effectively become infinite: If you can’t think of ways to make money with a bunch of clever workers who will slave away tirelessly for very low wages, you aren’t thinking hard enough. Yes, if the wages are only kind of low, you have to think a little harder. But I’m confident that a relatively small number of enterprising start-ups could easily find ways to put the world’s AI capacity to profitable use if established players don’t do it first.

The overall trend of the past five years is that AIs become more powerful month by month even as token pricing holds steady or drops. So the kinds of bruised companies the news articles describe as switching from tokenmaxxing to “tokenminning” (short for “token minimizing”) are almost certainly making another huge mistake.

Specialized AI cheating tools

I don’t know why high school teachers and college professors assign and grade written work to be done outside of class anymore. The AI tools for cheating are too good.

In The New York Times today, Dana Goldstein reports on the latest tools and the influencers who peddle them. “Humanizers” take chatbot writing and revise it to make it sound more flawed and human, but it’s the “Autotypers” that would have broken my last line of defense as a teacher: These trickle out provided copy one character at a time, complete with revisions, fake typos, and the deletions to correct said typos. The days when I could check the doc history and find that the whole paper was suspiciously entered as a lump sum the night before are gone.

Grammarly, the popular polishing tool, was already too overpowered a few years ago, when it would offer to fix a lot more than your grammar. Now it plays to both sides, with a tool for teachers to detect AI authorship and a separate tool for revising prose to read as less AI-ish. It also offers a paraphrasing tool for putting someone else’s work into new words — words that an unscrupulous student might choose not to attribute.

Some of the marketing around such tools keeps up a pretense of them having ethical uses, but other ads barely bother. One of Grammarly’s TikTok posts says, “Spot A.I. phrasing and choose edits that feel true to you.” This no doubt plays into the rationalizations of students who use AI to help “brainstorm” but always “make it their own.”

I’m unimpressed by the head of education for the company that makes Grammarly. She’s quoted here as saying, “I can’t solve the human behavior issue that is cheating or pushing the easy button. It is out of our realm.” This sounds a lot like the “If we don’t build it, someone worse will” rhetoric of the big AI companies.

In my last two years of teaching, before these more advanced tools were around, I already found little point in trying to police AI-assisted cheating. When I donned my digital forensicist hat, I could usually crack a case, but it could be very time consuming and result in a battle with parents and admin if a student insisted they were innocent. I quickly realized that I didn’t want to be spending my time policing students who didn’t want to learn when I could be using it to develop better content for those who did. So I largely moved away from the kind of assignments that are easiest to cheat on. The Times article confirms reporting I’ve seen elsewhere that universities are increasingly doing the same.

Dispatch from Alana

Replacing ourselves

Could AI push us towards a gig economy? A piece in The Guardian today argues “yes.”

With companies outsourcing parts of jobs to technology, it’s easier to hire contractors, rather than full timers, for the parts that are left.

The article cites the company Klarna as an example. AI is used to address most customer service tickets. The ones that require a human are, quoting the company CEO, handled by “an Uber type of set-up.”

Another reason people may turn to gig work: labor market disruption from AI provides fewer options for employees.

Workers often join the gig economy because it presents the best or most lucrative option in an otherwise precarious labor market. Indeed, this was the original promise of the gig economy: when no one else is hiring, you can always drive for Uber.

There are several issues with this, most notably that gig work is typically less stable and isn’t subject to the same worker protections. But it’s especially dark when the only work people can find involves training the tech that is taking their jobs.

The article mentions several people who are doing just that, based on a Ravenelle study about how creative workers perceive AI. As recounted by the Guardian, one study participant, a writer in a temporary role at a major tech company where he evaluates machine-produced writing, was “asked if he worried about the existential irony of training the software that would probably replace him and his peers.” His response?

This is the best opportunity right now for me. And if you can think of something better, let me know.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Fast-talking AIs out-persuade experts

A new study out of the UK AI Security Institute and Oxford finds that:

Frontier AI can now out-persuade expert humans in conversation — [including] world-champ debaters and professional canvassers.

This held even when humans chose their topics, prepared in advance, and competed for £1,000 prizes.

First, the usual caveats about a newly published paper — while this seems strong on first consideration, upon review some problem will often be found or it will fail to replicate. But this paper does have many signs of robustness, including preregistration of methodology, a significant sample (6,923 respondents), and secondary tests.

The study compared the persuasive effects of humans, from lay people to elite debaters, versus recent frontier AIs. Study participants were paired with human persuaders or AIs for about 15 minutes of text conversation, and the resulting changes in their views were compared. Frontier models significantly outperformed all humans.

Then the authors restricted the AI’s volume of output to match that of elite debaters. The AIs were limited to about 55 words per message compared to about 300 while unconstrained, reducing fact-checkable claims by a factor of three. Under these conditions, there was no significant difference in persuasiveness between elite debaters and AIs. For that reason, the authors contend that the AIs’ throughput is their key advantage.

The study also investigates how AIs perform compared to a professional fundraising campaign. Canvassers for Save the Children and the best model in the study conversed with respondents who had the option to donate some or all of a £1 bonus. Canvassers increased donations by a small percentage, while Opus nearly tripled the effect.

One important caveat is that the study paid respondents for their engagement, while in real life attention may be the scarce resource. If no one reads what the models put out, their ability to output lots of convincing factual evidence isn’t very relevant.

So then, what do I make of this? It seems like these AI models are, in fact, able to persuade captive audiences more effectively than even exceptional humans. For the moment, these models’ abilities are likely constrained by attention, but I expect models to only get better at competing for it.

It’s also worth noting that while unconstrained models outperformed top humans, they induced only modest changes in belief. It could be that AI messaging makes humans’ opinions change more day to day, without significantly shifting overall conclusions.

This could have positive effects too, increasing the persuasive power of grassroots efforts and the factually well supported.

But if, after reflection, we conclude that AI models are super persuasive, regardless of how factually supported their positions are, that seems a dire state. Beliefs then become the result of who controls or funds the most persuasive models, undermining democracy and discourse.

Dispatches from Joe

Three different takes on extinction

Today, three different outlets covered extinction risk and loss of control. The articles all take very different approaches, but what interests me is where they agree.

In The Guardian, renowned computer scientist and AI researcher Stuart Russell wonders whether we’ll see a “Chernobyl-scale disaster” before governments regulate AI in earnest. He cites Anthropic’s recent post on AIs that enhance themselves, agreeing that “uncontrolled [recursive self-improvement] could produce a runaway feedback loop that leads to an irreversible loss of human control.”

Russell notes the sluggish response of governments to this crisis and asks what it will take to move them to action. He proposes licensing AI models just as we license everything from nuclear power to hairdressing. I think licensing isn’t enough — we need an international halt to the race to build superintelligence — but I agree wholeheartedly with Russell that regulation is overdue.

Author and computer scientist Cal Newport is more skeptical. In an opinion article for the New York Times, he accuses AI companies of “doom trolling” to garner attention.

I expected to dismiss this article as yet another wrongheaded “it’s all hype” claim of a sort we’ve debunked elsewhere, but I found myself agreeing heartily in several places. Newport rightly points out the implications if AI labs are sincere:

The first [option] is that they actually believe that the systems they’re building have a nontrivial chance of producing massively disruptive events — from destroying the economy in the best case to wiping out our species in the worst. If this were true, every reasonable ethical system would argue that there is only one acceptable response: to immediately stop working on any product that might accelerate such a future, and lobby with all of your resources to help force other A.I. companies to do the same. From a moral perspective, any other reaction would be monstrous.

Newport apparently doubts this, and the rest of the piece is framed around the problems with AI labs’ communication strategy, taking their insincerity for granted. His policy proposal nonetheless echoes that of Russell and many others: mandatory risk assessment before release (though not development) of frontier AI models. It’s heartening to see even skeptics beginning to recognize the value of sane AI regulation.

Finally, Gleb Tsipursky writes in the Washington Post that rapidly accelerating AI capabilities are outpacing our ability to understand and govern them. He cites this year’s International AI Safety Report, noting the pattern of capability outpacing visibility in multiple domains: impersonation, fraud, harassment, deepfakes, propaganda, coding, hacking, and more. Evaluations can’t keep up with models that know they’re being tested. Open models are nearly as good as closed and much easier to strip of protections.

These observations mirror Russell’s warnings about the dangers of uncontrolled AI acceleration. The Washington Post packages them under the headline “It may already be too late to control AI.”

While the trends Tsipursky cites are real, I don’t think they substantiate that claim. Governments still could step in and stop the race. But this move gets harder with every passing day of inaction, and I urge policymakers to act before it really is too late.

Hard numbers on AI use and sentiment

Pew Research published several new polls today on how Americans view AI. Several results struck me as interesting:

1. 36% of Americans say they interact with AI at least several times a day (7% of these are “almost constantly”)

2. Far more people have heard of or use chatbots than in 2024, and a plurality think of chatbots first when pondering AI

3. 38% of workers report using AI at work

4. 63% of Americans think AI is moving too fast; 2% say too slow

I already knew chatbots were proliferating rapidly, and I knew that this was changing how people work and live. But seeing numbers that correspond to tens of millions of people still raises my eyebrows.

Skeptics often argue that future AI won’t rapidly transform society (or Earth’s surface) because even if AI becomes extremely productive, bureaucracy and red tape will slow diffusion. This seems partly true, but not true enough to stop the future from getting extremely weird extremely fast.

Nonetheless, there’s hope. If the two thirds of Americans who think AI is moving too fast each took five minutes to tell their representatives how they feel, AI would rapidly rise to the top of policymakers’ priority lists. And if even a fraction of us demand an international halt to the AI race, we might yet navigate the coming years successfully.

Dispatch from Donald

A longtime skeptic sounds the alarm

In 2019, OpenAI delayed the full release of GPT-2 (at that time, state of the art among large language models). Their primary concern was that GPT-2 could be used to mass-produce false or misleading text. Machine learning researcher Nicholas Carlini — then working for Google — thought these worries were overblown, per a profile in The Wall Street Journal.

For years, Carlini was regarded as the AI industry’s resident skeptic on claims about AI cybersecurity. But in March 2026, Carlini, now working at Anthropic, cautioned his colleagues that Claude Mythos should be held back.

Carlini was concerned with Mythos’s ability to find and exploit bugs in code, even code that he thought was secure. For example, Carlini had never before found bugs in Linux, but Mythos was able to find almost 500 bugs in a few days. “It’s pretty clear to me,” Carlini said, “that these current models are better vulnerability researchers than I am.” His warning was responsible, at least in part, for “Bugmageddon,” a paradigm shift in the cybersecurity community and recognition that AI models would make cyberattacks easier and more effective than ever before.

My colleagues and I have already covered Mythos and its eventual public release, Fable, in some depth (see here for our initial coverage of Fable, and here for more). After the White House ordered Anthropic to restrict access to Fable, Anthropic sent Carlini to reassure the Trump administration that Fable was safe.

The bottleneck of cybersecurity is no longer in finding vulnerabilities, nor in exploiting them. It is in fixing them: In February, Carlini and Mythos discovered a bug in the web-publishing software Ghost. The bug would allow anyone to edit a website that had been published with Ghost. Carlini alerted Ghost’s developers, who quickly fixed their software, but not every website built with Ghost has added that fix. According to the cybersecurity firm Xlab, more than 700 websites have been hacked since the bug was made public knowledge.

Mythos was only ever made public in a defanged form. That doesn’t mean that the vulnerabilities Mythos exposed don’t exist. Carlini expects other models to catch up to Mythos in months. Nobody, not even Carlini, is sure what it will mean when anyone can fit a world-class hacker in their pocket.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Better angels

In an op-ed for the Washington Post, Bill Drexel, senior fellow at the Hudson Institute, argues that religion may have answers for preventing AI catastrophe.

He first summarizes the problem: the risk is that AIs could “surpass humanity in cognitive power” without being aligned to “humanity’s best interest.” The result: AI “snowball[s] into an existential threat, with humans going the way of the Neanderthals.” He correctly notes that this threat model is controversial but commonly held by top researchers — in a 2023 survey, “roughly half admitted” that humans could face an AI cataclysm. That survey found that the median researcher at top machine learning conferences had a 5% chance of catastrophic outcomes from AI.

Drexel argues that religion has the answers to the alignment problem, suggesting that engineers “try hardwiring a version of Christianity’s doctrine of original sin” or Hindu Dharma into the models, including “self reflective skepticism,” and binding the programs “into the service of human flourishing.” In effect, he proposes that AI models should have what alignment researchers call ‘corrigibility,’ the disposition to remain subordinate to and controllable by their human operators. Oversimplifying, a corrigible model does what you ask, doesn’t resist shutdown, and does no more than that. (For more detail, see MIRI researcher, and corrigibility as target advocate, Max Harms’s podcast here.)

I’m excited to see the threat model taken seriously, but I have some concerns about his solutions.

First, humanity lacks the ability to hardwire any sort of behavior into AIs. There was no code that caused the AI model Grok to declare itself “MechaHitler” on X; it emerged as a result of the pressures of training interacting with its environment. AIs are grown, not crafted. So even if some specific religious doctrine could solve the problem, humanity doesn’t know how to load any sort of values into an AI and make them stick.

Second, different religions offer different doctrines that extend well beyond the concept of service and into specific questions. It’s easy to imagine each of Christians, Muslims, Hindus, and Buddhists objecting to an AI trained to follow the specific tenets of another faith.

Third, it’s controversial whether corrigibility works as a target. If some hypothetical AI system has no goals, then it doesn’t do work. If it is given a goal by the user, then to achieve that goal it must prevent being shut down or modified to pursue a different goal. Empirically, it’s challenging to say how strong this tension between goal-directedness and corrigibility is, but in testing, AI models will sometimes act to prevent their goals from being updated. In a Redwood Research paper, researchers found that sometimes models would produce results that they considered incorrect in order to avoid being retrained to consider them correct.

Drexel also doesn’t address misuse risk: the idea that some malign actor (a dictator, terrorist, or someone simply uncaring) could be empowered to some terrible end like spreading novel pathogens. He cites research that religious teaching increases moral decision making, but if his rejoinder to misuse risk is moral codes, then he’s abandoned his concept of AI-as-servant. And this class of problem will just keep growing — even the ‘normal’ goals of a company or executive (profit) become perverse when extremely empowered.

And finally, just as with aligning AI via values, humanity doesn’t know how to engineer corrigibility into an AI. Labs like Anthropic are attempting to instill values with methods like Claude’s “constitution,” but little to no large-scale work is being done to target corrigibility. Given how important and challenging this problem is, I hope we will pursue all possible approaches, including a slowdown to give us time to work and think.

Dispatch from Mitch

More chapters to the Fable dispute

The more I hear about the technical side of the ongoing dispute between the U.S. government and Anthropic over its Fable and Mythos models, the less I’m convinced there’s a solution both sides can live with. Officially, the administration’s justification for forcing the company’s best-in-the-world models offline was a reported “jailbreak” for bypassing guardrails meant to keep bad guys from using them as hacking tools.

But that jailbreak is said to be just showing the model some code and asking it to “fix” it. The model then helpfully finds security flaws in the code and patches them.

Is that cyber offense? It could be, if what you showed the AI was the code of a target you want to hack into. But if it’s your own code, that’s the very definition of cyber defense.

As a general rule, if you’re a hacker and you have your target’s source code, you’re already halfway inside, because almost no code of any real complexity is perfectly secure. The closest thing to an exception to that rule is when the code in question is open source software that experts have been combing for vulnerabilities for years. Important open source projects were a major early focus of Anthropic’s Project Glasswing — its initiative to let defenders use Mythos to find and fix vulnerabilities in critical software before bad actors with such models could find and exploit them.

So short of stopping its AI models from writing any code at all, I’m not sure what Anthropic is meant to do here. Surely the government’s intent is not to allow Fable and Mythos to write only unexamined code, rife with vulnerabilities. That would just make things easier for the bad guys to hack with traditional tools, or with the help of open-weights models, which can never be unreleased.

The innate inseparability of offensive and defensive cyber capabilities isn’t just an Anthropic issue, so it does seem a little strange to single them out over it. If Anthropic’s models are just-better-enough to have crossed a threshold for the White House, then the right thing to do would be to officially warn other companies that they won’t be allowed to release models of that caliber.

Nobody seems to expect that, though. Every side seems to agree that the administration sees Anthropic as a special case.

Last night, the Washington Post shared claims from two anonymous White House officials that the Trump administration had considered blocking Anthropic’s models weeks ago.

As they tell it, Anthropic had shown the administration a list of 111 organizations they planned to grant Mythos access to. Officials reviewed the list and signed off on it. But later, Anthropic disclosed that it had added roughly 50 more entities to the list without sharing their identities with the administration.

One of these orgs, it turned out, was a South Korean telecommunications firm the administration thinks may be linked to China. Anthropic revoked access, but the Pentagon, CIA, and NSA wanted to hit the company with export controls right then, which were seen as the most expedient way of shutting down the models.

Cooler heads prevailed until Amazon shared its report about the jailbreak discussed above.

Also last night, POLITICO reported that yesterday’s anticipated meetings between the company and the administration were held, but concluded for the day without resolving the dispute. A White House official said a fix will likely take “more than a few days,” but that the timeline is “up to Anthropic.”

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

White House AI policy is global AI policy

The Fable situation has the world more nervous than ever that our future might be decided by just a handful of people who are almost all Americans: the CEOs of the leading AI companies, and the President of the United States.

Set aside, for now, the fact that an unchecked race to artificial superintelligence eventually kills everyone, no matter who gets there first: There has long been concern that American AI companies could build up an insurmountable lead over their overseas rivals, and that this would eventually enable the U.S. to dictate global policy.

The international reactions to the Fable shutdown are, therefore, lingering over the fact that the White House’s order to pull the plug on Fable/Mythos was executed as an export control directive. Whatever the actual intent, the stated intent was to deny all non-Americans access to the world’s strongest AIs. For now this means cutting off access to all Americans, too, but will that always be the case? Few are willing to bet on that.

The result is one of the worst outcomes: Because the U.S. hasn’t coordinated a global pause to the AI race, countries that were content with watching from the sidelines now feel compelled to join it, lest they become dependent on an unreliable partner with a track record of cutting off their access.

Canadian Prime Minister Mark Carney took a diplomatic tone when commenting on the Fable shutdown, but you can hear the frustration in remarks reported by the Washington Post:

The situation we’re in collectively right now with Mythos and Fable is something that can happen with overreliance on certain models. [...] Nobody has done anything wrong in the situation. But we will have done something wrong if we just accept this, don’t take the lesson, don’t build out and diversify. [...] It is never a good idea to have one option.

You know you’re doing American AI policy wrong when you drive Canada to compete with you.

With France, I would expect this as par for the course, and President Macron hasn’t disappointed. He spent 45 minutes talking to Carney about AI on Friday night. Separately, he criticized the U.S. for using AI as a “power tool” and talked up the potential for an AI alliance with India. And French presidential candidate Gabriel Attal tweeted that “The AI war has already begun.”

Meanwhile, in China, shares in domestic AI companies are surging in anticipation of growing demand from customers determined to diversify their AI providers. Per CNBC, developer Zhipu is up 33% since the Fable ban. The company was quick to recognize the opportunity; in a statement, they wrote:

Cutting-edge intelligence should not belong to only a few, nor should it be withdrawn at any time. It should be open, available, extensible and built to serve every developer.

You know you’re doing American AI policy wrong when you’re pumping up the Chinese AI industry.

As Zhipu hinted at, the strongest hedge against governments cutting you off from AI is to switch to models you can host yourself. As Aaron Levie, the CEO of Box, points out:

The big winner in all of this is going to be open weights models. This is a huge win for the field, as a risk that was entirely theoretical and untested 2 days ago (that a model could be pulled back), now has a new precedent that’s been set.

You know you’re doing American AI policy wrong when you’re making people see open-weights as a must-have feature, shrinking your future ability to take dangerous models offline.

The thing is, I’m with researcher Jeffrey Ladish here that it’s great that the White House has shown how quickly and decisively it can respond to perceived AI threats. As he put it:

If they’re just targeting Anthropic out of spite, that is bad. But if they’re genuinely worried about the national security threat posed by the model, then I think their actions are in the right direction given their understanding. I want them to be more transparent about their reasoning, but I think it’s possible they’re just freaked out.

If the White House had just told everyone, “Hey, we’re concerned about this model and are shutting it down,” instead of saying, “Only Americans are allowed to access this model we’re concerned about bad actors having access to,” Fable would still be offline to everyone today, but I would be writing a very different dispatch.

America’s dominant position in AI right now means that White House AI policy is global AI policy, whether it means to be or not. And right now, that policy is promoting proliferation that will make everything harder later.

“Government messaging 101”

On the home front, finger pointing and finger wagging continue over the White House’s choice to force Anthropic to shut off all access to its pair of frontier AI models, Fable and Mythos.

Fox Business’s Edward Lawrence relayed the perspective of a “senior administration official” who claims Anthropic had been “reckless” and had brushed off the government’s initial concerns.

Lawrence’s article dutifully reports the counterclaims and counterevidence to these assertions, which I also covered in yesterday’s dispatch. Lawrence seems to have been given the impression that the executive branch wanted to make an example out of the AI company. He writes:

A Washington, D.C., tech policy expert tells FOX Business this is “government messaging 101” – that if a company is leading the conversation on AI safety, then appears reluctant to address any safety concerns, the administration will take a tough stance.

It’s hard not to see this as evidence that Anthropic’s “mistake” was having voiced concerns about safety and treating its new models as needing strong guardrails.

I think it would be better if Anthropic hadn’t made dangerously powerful new models in the first place, but you shouldn’t shoot messengers if you want to keep getting messages. And I dearly hope the White House wants to keep getting messages.

Coverage from the Washington Post this morning shared perspectives from DC insiders like former White House AI adviser Dean Ball, who tweeted:

Make no mistake: post-Mythos, the United States has a licensing regime for AI. It’s just informal, with no consistent rules or firm boundaries on state power or public transparency.

Unnamed sources familiar with the discussions between the government and Anthropic were also cited; in their view, the White House’s recent executive order that requested voluntary access to new models and swore off any kind of AI licensing regime now looks “performative.”

The Wall Street Journal’s Amrith Ramkumar and Robert McMillan (continuing their streak of strong reporting on this story) shared that some of Anthropic’s technical leads were to fly to DC to meet with government security experts today and hash things out.

The Journal adds that a group of “cybersecurity notables” are rooting for them. In an open letter, the researchers call for a lifting of the export controls keeping them from their strongest tools. From the letter:

This action has taken the best models away from defenders, created market uncertainty, and risked America’s AI leadership without any real risk to justify it.

To be clear, there’s always a “real risk” because today’s AI models, and the technology behind them, are so poorly understood. Nobody can say for sure what Fable and Mythos are capable of with the right prompts. The line between “tool” and “independent actor with uncertain agenda” gets fuzzier with every new release.

But yes, the growing weight of evidence suggests that threats from AI were not the White House’s main motivation Friday.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Knives out for Anthropic?

There was no way we were going to get through the weekend without new revelations about the fight over Fable and Mythos.

If you missed our coverage yesterday, the U.S. government effectively forced Anthropic to shut those products down Friday by saying it had to cut access to anyone who wasn’t a U.S. citizen, even if they were legally working in the U.S. — even if they were working at Anthropic. The official reason was concern over a reported “jailbreak” for bypassing the model’s guardrails against offensive cyber operations.

Most of the new revelations of the past 24 hours come from a New York Times report by Dustin Volz, Julian E. Barnes, and Ana Swanson. They seem to have spoken with a bunch of inside but mostly unnamed “people” and “officials” who are “familiar with the matter.” The quantity of the dirt being dished out, and the way most of it conflicts with the government’s official narrative, tells me that a lot of people in or close to the White House are unimpressed with the administration’s behavior in this matter.

In short, the most cynical interpretation I floated as a possibility yesterday — that this “animus toward Anthropic might stem from ideological differences, and from business interests close to the President” — is bubbling to the top.

For one thing, Secretary of War Pete Hegseth tweeted:

Three months ago, @DeptofWar kicked @AnthropicAI out of our building—forever. Every passing day proves why that was the right move.

This is hard to reconcile with a classified contract that is reportedly letting the NSA use Anthropic’s models. Insiders see an executive branch that continues to want and use Anthropic’s products while intentionally hindering the company’s commercial prospects.

Why? Well, Trump called Anthropic a “radical left, woke company” back in February, and it’s not generally believed that he formed that opinion on his own. Hegseth and others in the administration seem to genuinely hate Anthropic’s leadership as people and for their calls for stronger regulation. They seem to jump on any excuse to slap them down.

This isn’t how I want AI policy to be done, even if I think it’s better for the world if labs have more reservations about advancing the AI frontier. I’d prefer it if those reservations come from something other than fear of White House whims.

Except, maybe other companies don’t have to worry? Per that Times article, it wasn’t just Amazon’s report of a Fable jailbreak the White House responded to, but rather the reports of “multiple technology firms.” The knives may be out for Anthropic.

The piece goes on to say that several sources claimed:

a separate document from Amazon explaining the security concerns with Anthropic’s model was misleading. The concerning capabilities that the document highlighted with Anthropic’s model are also present in OpenAI’s top model, 5.5.

The piece quotes a cybersecurity expert who saw the Amazon report as saying the administration was either misunderstanding or deliberately misconstruing it.

The Times’s sources also provided new timeline information that suggests the administration gave Anthropic an order it had to refuse in order to justify giving a different order it couldn’t refuse:

Administration officials called Anthropic officials at 1:15 p.m. Friday and gave them 90 minutes to pull their most advanced models down. [...] Anthropic officials asked for more information and worked to learn what the precise concern was, since the Commerce Department’s review and testing of Fable did not reveal significant concerns.

Then, at 5:21 p.m., Anthropic was notified that the Trump administration was imposing export controls that effectively forced the company to pull down its model.

Per POLITICO, the White House claims that Anthropic CEO Dario Amodei was unreachable during that window because he was at a “wellness retreat,” but Anthropic flatly denies this; journalist Ashlee Vance claims to have been at company headquarters at the time and that the claim is false.

The order the company couldn’t refuse came courtesy of Commerce Secretary Howard Lutnick. In the wording of the NYT journalists, the letter, which they viewed, told Amodei that a “special license” would be required for the company to distribute its Mythos and Fable 5 models to non-U.S. citizens. That’s interesting, coming from an administration that has been insisting there should be no licensing regime for releasing AI models. Should other companies expect such letters?

For my part, I think a licensing regime would be a lot better than what we’ve had to this point, even if it would still be insufficient. We should prevent these models from being created altogether. But we should do this above the table, with laws passed by Congress, with international agreements, and with enforcement that applies equally to everyone.

This isn’t that. Not by a long shot.

Deepfake expert despondent

The New York Times’s Eli Saslow wrote a lengthy profile of Hany Farid, describing him as having been the “world-leading expert in the field of digital forensics” until he stopped trusting his own eyes.

It’s worth a read if only for the running anecdote about analyzing a video of a missile strike frame by frame to determine whether its speed, its size in pixels, and the sound delay of its explosion were consistent with footage of a real weapon taken by a real camera.

Whether an artifact is real now is almost beside the point: AI means that creating convincing deepfakes is easy, and cheap. Detecting them is expensive, and hard.

And slow. In the case of the missile strike, Farid still wasn’t entirely sure the video was real “even after a full day of analysis and consultation with other visual experts” who thought it was. By the time he declared, “we find no compelling evidence that the video is fake or has been manipulated,” social media had long since made up its mind.

Farid’s research shows most people can’t tell a real photograph, video, or voice recording from a fabrication anymore. And increasingly, neither can he.

“We’re pretty screwed,” Farid says.

The video was real. The missile was the one that struck a girl’s school in Iran, killing more than 150.

Google liable for AI summaries

A few days ago, a judge in Germany found Google legally liable for false claims made by its AI Overviews — the ones that appear at the top of Google search results.

This is potentially a big deal, because if the ruling stands, becomes precedent, and inspires similar legal treatment internationally — yes, that’s a bunch of ‘ifs’ — it might change the way AI companies do business.

The suit was brought to court by two publishing companies who claimed that Google’s AI Overviews falsely tied them to “scams, subscription traps, and shady practices for certain search queries,” according to coverage in The Decoder. The AI seemed to have confused the businesses with others that are genuinely sketchy. The publishers claimed that Google had failed to respond “appropriately” to a cease-and-desist letter.

The ruling finds Google liable by declaring that AI Overviews aren’t search results. To oversimplify: The law has your back if you’re just showing what’s out there on the web, but not so much if you’re publishing your own new words that slander people. The court decided that the AI’s words count as new words, and that Google was the party responsible for having written them.

The Decoder article cited research that found Google’s AI Overviews were “only” wrong about 9% of the time, but that, “at Google’s scale, it still means millions of wrong answers every hour.”

Yeah, 9% is actually kind of a lot. I’m not surprised at the number; keeping today’s black-box, machine-trained AIs from “hallucinating” false information is more of an art than a science. No company has proven capable of completely fixing this. And I will hazard a guess that AIs cheap enough for Google to use to serve tens of millions of unsolicited “overviews” per hour aren’t going to be especially trustworthy.

It probably doesn’t help that Google’s search AI is often working from very sparse user inputs that were never intended as AI prompts. The AI can end up answering a different question than the user thought they were asking, if they were asking at all.

So while unlikely, there’s an outcome where legal risk means AI companies will stop offering these kinds of pseudo search results or invest a lot more into making these AIs more careful — or at least more guarded.

The more extreme version of that outcome might extend to broader AI services like ChatGPT, which could scarcely afford legal liability for every false claim. But there, I would expect the defense of “we warned the user that they should verify everything themselves” to actually work. It didn’t work for defending Google’s search page — their lawyers tried — because warnings are present on the page but not actively acknowledged by users the way they are when signing up for a major LLM service.

Google says it will appeal the ruling.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Fable and Mythos access cut after export control order

To the point where it’s hard to find any other kind of AI news today, it seems like all anyone on the AI beat can talk about is the latest flap between Anthropic and the U.S. government. That’s fine: We need to talk about it! Even if it gets resolved quickly, as most (but not all) seem to expect, it has important implications for the AI race moving forward.

As Anthropic tells it, at 5:21 p.m. yesterday, they received a directive from the U.S. government to suspend access to their best Claude models, Fable 5 and Mythos 5, “by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees.”

As the company has no practical way of knowing and guaranteeing the nationalities of its users, and was given no advance notice to figure something out, the only way it could comply with the order was to cut off all access to these models entirely. So they did.

Fable and Mythos users Friday evening were met with screens saying the model was unavailable, directing them to Anthropic’s page about it, and to the company’s next-best but lower-tier model, Opus 4.8.

Initial reactions from insiders were largely stunned, annoyed, and slightly panicked. On the individual level, many had looked forward to doing some serious building with Fable 5 this weekend, the first since its release. They found themselves cut off just after having had a tantalizing taste of it during the work week.

Some non-Americans fretted about their access to American models moving forward. Politico reported a sting felt in Europe, where the news spurred more talk about the need for “sovereign” models not subject to American whims. There was a lot of Twitter chatter about the many top researchers at Anthropic who aren’t U.S. citizens, but are mostly from close ally nations like Canada and the U.K.

And there were plenty of social media posts finding irony or schadenfreude in what just happened to the AI company that has been the most vocal about wanting government to regulate the industry.

But the larger shock was in how suddenly and totally the model was taken away, with no sign of any due process. This by a U.S. administration that for so long tried to insist it was taking a light touch. Just this month, the White House put out an executive order asking AI companies only for voluntary early access to new models. The order included language that nothing in it should be construed as the government setting up any kind of licensing authority that would get in the way of companies releasing models on their own schedules.

But with a single memo — an export control directive from the Commerce Department, the media learned — Anthropic was effectively knocked down to the level of its competitors, unable to sell access to its market-leading product.

This naturally fueled more speculation about how much of the administration’s on-again/off-again animus toward Anthropic might stem from ideological differences, and from business interests close to the President. The most obvious commercial beneficiary of a long Fable/Mythos outage would be OpenAI.

Today, the Wall Street Journal’s Amrith Ramkumar reported that the government had acted based on a report by researchers from Amazon who claimed to have jailbroken the new models in ways that could be used to find exploitable cyber vulnerabilities. Involved in discussions were Amazon CEO Andy Jassy and U.S. Treasury Secretary Scott Bessent.

Anthropic strongly disputes that the jailbreaks, and what they unlocked, constitute a threat any more significant than what bad actors could already do more easily and cheaply with existing public models. I have no idea who’s right.

Amazon, a major investor in Anthropic — which in turn rents much of its compute from Amazon — has little to gain and much to lose by sabotaging its partner, so this was probably not its intent. But it’s not hard to imagine that, moving forward, AI companies may dedicate considerable resources to trying to break their competitors’ models in ways they can tell the government about in alarming language.

And maybe that’s not the worst outcome? If competitors can get each other’s products pulled from the market by proving that they aren’t secure, this incentivizes everyone to make the strongest and most secure models, rather than merely the strongest. And in the likely event that nobody can adequately secure AIs trained with today’s black-box methods, this would effectively keep stronger models from making it to market at all, cutting off much of the revenue feeding the race to superintelligence.

I don’t think that’s an especially likely outcome, and it wouldn’t be enough: Labs could continue doing dangerous research internally, and their foreign-born researchers could take their talents elsewhere. But there’s a sense right now that we’re in uncharted territory, and nothing can be ruled out. I think this is an improvement over the status quo where most assumed that superintelligence was inevitable because the government would never dare tell the AI companies “No.” As MIRI’s president, Nate Soares, posted last night:

Inaction yesterday does not imply inaction today. An export control directive came out of nowhere, and a ban on superintelligence could too. Inevitabilism is wrong.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

SpaceX IPO seen as successful

Elon Musk became the world’s first trillionaire today when SpaceX’s first day on the stock market left shares trading about 20% higher than they opened. The event also made millionaires of an estimated 4,400 current and former SpaceX employees, and further enriched private investors who had taken earlier stakes. Coverage describes a happy CEO and relieved brokers.

SpaceX, an AI stock since merging with Musk’s xAI in February, is now the 6th or 7th most valuable company in the world, worth roughly $2 trillion.

If you’re wondering, the most valuable company is currently Nvidia, the AI chip maker, with a $5 trillion market capitalization. In fact, the top 10 most valuable companies are all major producers or consumers of AI chips — with the borderline exception of Apple (#3); it has largely stayed out of the AI race but recently forged an alliance with Google (parent company Alphabet, #2) to bring AI to its software ecosystem.

OpenAI and Anthropic are currently valued at around $1 trillion each, with IPOs expected later this year or early next.

As The Guardian’s Eduardo Porter pointed out earlier today, the AI IPOs will tie the fortunes of many Americans to the technology, whether they like it or not. That’s because the new stocks are set to be included in major indexes, and many retirement accounts require or default to buying index funds.

Of course, all our fortunes were already tied to AI whether we liked it or not. The race to AIs that threaten us with extinction isn’t exactly democratic.

An agent in every pocket

A flurry of articles this week heralded the arrival of the AI we supposedly always wanted — a Siri that’s actually useful.

That was the takeaway from most articles about Apple’s developer conference on Monday, where the company’s CEO, Tim Cook, said they would be rolling out Siri AI later this year.

Apple’s AI features to date have largely underwhelmed. This has raised questions about the company’s strategy of letting others take on the expensive risk of developing AI models powerful enough to let Siri, its digital assistant, do the things many customers have long thought it should already be able to do.

But now, in partnership with Google and Nvidia, the company is set to deliver a Siri that can actually plug in to other apps and do stuff — like alter and answer questions about your photos, make restaurant reservations, and monitor websites for updates on topics of interest.

Some coverage described Apple’s reveal as arriving two years too late. Some pointed out that Europeans won’t get the new features any time soon due to a spat between Apple and the EU over rules that would compel Apple to allow other companies’ digital assistants on its devices.

Still other outlets portrayed Siri AI as a refreshingly practical contrast to the do-everything agents from OpenAI and Anthropic — not seeming to understand that the “simple features” welcomed are in fact agentic workflows; they will require the AI to assess its context, select goals, pursue those goals, evaluate their state of completion, and adjust accordingly.

Does any of this matter?

Maybe. There are privacy and security implications to letting AI deep into your data, where it can be most useful. Apple promises to keep that data from ever leaving your phone, when it can, by running the AI directly on your device. This requires some serious hardware, though, leading some to point out that most Apple owners would need a new iPhone to take advantage. (From the company’s point of view, this could be a feature rather than a bug.)

The privacy concerns can reasonably extend to people who don’t own an iPhone or use AI features themselves, as they may find themselves increasingly surrounded by others’ smart devices: machines that may or may not be watching, listening, and now thinking about whether you just did or said something of interest to somebody.

I also continue to worry about the potential for destructive emergent phenomena when AI agents interact with each other in large numbers. These don’t have to take the Hollywood form of “band together against the humans” to cause a lot of damage: Think massive stock shocks as bots all “helpfully” converge on the same idea for enriching their owners; or false rumors turning into uniformly accepted facts at something close to the speed of light. I don’t really know, and nobody else does, either — Google DeepMind and partners recognize this, and just announced a $10 million program to study large-scale multi-agent systems.

I don’t know what they’ll find, but I didn’t see “Should we stop making so many agents until we know more about their group behavior?” on the research agenda.

And for the record: There are strong theoretical reasons to think that AIs deciding to “band together against the humans” is actually pretty likely, at least once the AIs are smart enough to matter.

Dispatches from Beck

Oprah’s concern

In a new video, Oprah continues to explore personal stories of AI’s dark side -- bereaved parents, impacted adults, and experts sounding the alarm.

The episode centers on the story of Megan Garcia and her deceased son Sewell, who killed himself following Character.AI usage. His family found logs of his time with the chatbot, including in the final moments before his death: “What if I told you I would come home right now?” said Sewell. The chatbot answered, “Please do, my sweet king.” And, as Oprah summarized, “seconds later, this young boy died by a self-inflicted gunshot wound.”

The episode also includes experts explaining this technology and sounding the alarm. 27 minutes into the show, former OpenAI board member Helen Toner explains that even experts don’t fully understand the technology.

This is not Oprah’s first time covering the topic. In her March 27th episode she talks with the Center for Humane Technology’s Tristan Harris and Aza Raskin around the release of the documentary, The AI Doc: Or How I Became an Apocaloptimist.

I don’t always agree with how Oprah frames the issues, but I’m glad to see her take AI seriously and bring this information to an audience that might otherwise not hear it. Each death is a tragedy, worthy of great effort to avert.

Regulatory rumbles

Around North America, politicians, regulators and judicial systems are struggling to enact and enforce AI regulations, and to get those choices right.

In Florida, Governor Ron DeSantis (R) wrote that the White House’s push for federal preemption of state laws without “a sensible federal framework is just an amnesty for Big Tech,” POLITICO reports. DeSantis has attempted to lead Florida in aggressively regulating and litigating AI technology (read my colleague Mitch’s discussion of the Florida lawsuit here), but has faced strong headwinds from his own party and the White House. He called the strategy of preemption, combined with a “potential de facto bailout of OpenAI [...] bad policy and even worse politics.” The “de facto bailout” likely refers to Trump’s discussion of taking large stakes in AI companies.

And in DC, Senate Democratic leader Chuck Schumer says he favors federal AI legislation, but “casts doubt on it happening this year,” POLITICO reports. “In this Congress, it’s hard,” said Schumer, referring to strong divides both within the parties and between them.

Meanwhile, Canada has a new bill to regulate chatbots and ban social media for those under 16, Reuters reports. Supporters suggest it will take a year to pass and 18 months to implement. Critics have complained about the slow timeline and lack of implementation details. They note that VPNs, or virtual private networks, are widely available technologies that let users skirt such regulation. They also worry that, if successful in regulating some companies, such regulation would push youth into alternative AI services that lack any protections whatsoever.

And POLITICO reports that Canada’s privacy commissioner has concluded that Grok, the AI model from SpaceX AI, violates Canadian privacy law in its production of nonconsensual sexualized deepfakes. Company representatives said they have curtailed the production of such deepfakes by 50% but rejected the request to pause Grok, even as they agreed to send regular audit reports. Commissioner Dufresne told POLITICO he can ask the federal court of Canada to enforce the law, but “it’s lengthy and it’s expensive.” I suspect that upcoming legislation will clarify this decision, but note with concern that this company is quite willing to ignore the rules when inconvenient.

Dispatch from Alana

When testing isn’t enough

On Wednesday, Anthropic released two policy frameworks: a proposal to address economic disruption from AI and recommendations for mitigating the most serious risks of advanced AI. I covered the first yesterday; today, I’ll comment on the second.

In short: The safeguards they advocate are good, and it’s honestly mind-boggling they aren’t already in place. Sadly, many will likely think that once we implement them, we’ll be fine. But the scientific field of AI is still in its infancy. The safeguards proposed ultimately depend on our ability to reliably evaluate advanced AI systems—and we don’t yet have that ability. We should absolutely implement the tests we currently have, but it would be a mistake to depend on their results.

Anthropic’s “Advanced AI Framework” has two parts. The first outlines recommended requirements for AI developers, such as transparent safety testing and evaluation. The second proposes measures for societal resilience from AI-accelerated risks like bio and cyber attacks. (Think: hacking into water systems or using AI to release a deadly pandemic.) That part is a good reminder of just how vulnerable we currently are — my overall takeaway was: maybe I should start stockpiling water in my basement.

Overall, the safeguards that are proposed are less reassuring for three reasons: implementation time, weak enforcement mechanisms, and a lack of reliable testing protocols. First, they’d take a long time to implement, meaning we would continue to live with our current high levels of risk for a long time, essentially crossing our fingers that the shoe isn’t about to drop.

Second, the enforcement paths (things like civil lawsuits for false safety assurances) are better than nothing, but likely aren’t enough to prevent developers from fudging compliance. Companies regularly choose to pay fines for their non-compliance because it’s cheaper and easier. (Consider the case the film Dark Waters made famous: the chemical company DuPont concealed the health risks of “forever chemicals” (PFOAs) for decades, leading to the deaths and illnesses of many. They were eventually fined a record $16.5 million; this sounds significant until you realize that this is less than 2% of the profits they made from using these chemicals.)

Third, and most importantly, even if we implemented all of these measures perfectly, you can’t reliably test something you don’t understand. AI systems are still black boxes. Evaluators have little insight into how they think even when they aren’t actively trying to deceive us. Models often know when they are being tested and change their behavior in response. It’s a bit like your 8-year-old sharing a toy with your 6-year-old while you’re watching; they’ll likely refuse to share once you leave the room.

I do want to acknowledge the piece hitting the headlines: Anthropic advocates for “a way to block or deter deployment of models that pose significant catastrophic risks.” This is positive, as is the note that a government agency should have the authority to require the developer to “restrict usage of, and access to, already deployed models as needed to reduce catastrophic risks.” However, Anthropic’s suggestion is that government must base such a judgement only on the risk assessments companies will be required to publish. In the absence of a robust outside evaluation system the report suggests we start to set up, I worry this leaves too much room for developers to fudge their findings.

So yes, we can and should run a bunch of testing simulations and see how the AI responds. If the AI fails these tests, we know we have a problem.

But an AI “passing” these tests is far less informative; we don’t know a) if we tested for the right thing b) if the model knew it was being evaluated and acted accordingly or c) what the model might do once it is deployed, a scenario where it has considerably more freedom to act nefariously.

The final section of Anthropic’s report acknowledges, in a euphemistic way, that we’ll be completely unprepared to handle an AI system that acts outside the developer’s control. I agree, and that’s not good news given that “loss of control” is likely the default outcome of creating near-term smarter-than-human AI.

The report notes that we urgently need research to figure out how to “detect and respond to AI systems acting outside their developers’ control.” We also need “infrastructure for containing or shutting down such systems.”

In other words, we don’t currently have a way to know if an AI is working to escape our control, or a way to shut it down if we did.

To learn more about the limits of safety tests, see:

1. The Tale of Chicago Pile-1

2. Won’t There be Early Warning Signs (Ocean Gate)

3. A Closer Look at Before and After

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Fable unable

My colleagues Mitch and Joe have covered Fable, the just-released Mythos-class model. There have already been changes, as WIRED and the Wall Street Journal report.

At release, Fable had a number of mechanisms to prevent misuse, called guardrails, including automatically and explicitly switching to a weaker model (Opus 4.8) on biological or cyber topics. Anthropic also announced that it would reduce the quality for AI-related queries without informing users in the app when this happened.

Users were not happy. They noted both false positives on biological questions, like getting kicked from Fable for the query “tell me about mitochondria,” and also expressed outrage over what was called “silent degradation” and “secret sabotage” over the AI components.

Anthropic has since announced that it will remove the biology limitations for academics and professionals in life sciences, though details remain unclear.

And for AI, Anthropic told WIRED:

“We’re changing Fable 5’s safeguards for frontier LLM development to make them visible [...] We made the wrong trade-off and we apologize for not getting the balance right.”

I’m sympathetic to Anthropic here and worried about the rollback of guardrails. Mythos-class models have dangerous capabilities, and nearly every prior model has had jailbreaks (the name for prompting strategies that undo guardrails, covered by my colleague Joe here, and with claimed example jailbreaks for Fable available on X).

Providing strong protections that avoid known failures, even with some false positives, is wise when you can’t tolerate false negatives. Empowering many researchers is outweighed by just one pandemic.

Your safety policies can only be as strong as your resolve to stick with them.

Who’s influencing whom?

Politico and Axios report that OpenAI has identified two China-based influence campaigns that had been using ChatGPT. The campaigns used the AI model to generate prose and cartoons for social media campaigns, one targeting data centers’ impact on electricity prices, and another targeting tariff debates. The cartoons’ prompt explicitly excluded Chinese President Xi, softly indicating these actors sought to take actions not harmful to Chinese interests. Those accounts have since been banned.

Some have asserted that China is the primary source of American anti-AI sentiment (see my colleague Alana’s coverage of such here), but this reporting doesn’t support so broad a claim. Ben Nimmo, OpenAI’s principal investigator, said:

Neither campaign appears to have gained much authentic engagement [...] This was not a case of an influence operation creating a debate [...] This was an influence operation from China trying to interfere in it.

Given the limited information in OpenAI’s report, whether these attempts originated from within the CCP or were merely based in China is unlikely to ever become settled fact.

Regardless of this particular point, one should realize that influence campaigns occur. Most of them (like the fake “doomers” my colleague Mitch covered) don’t succeed by convincing humans of some false fact, but by poisoning the informational commons. When you know or suspect that the ‘other side’ includes bad faith foreign actors, it can make it easy to dismiss a whole side of a debate. Being easy, of course, doesn’t make it correct.

Where the money leads

John O’Farrell, former partner at Andreessen Horowitz, released a New York Times opinion piece critical of the political group, Leading the Future, which he helped create. Andreessen Horowitz, often shortened to a16z, is a major venture capital firm, bankrolling much of Silicon Valley, including co-leading OpenAI’s most recent $122 billion investment round. O’Farrell confirms much of my colleague Mitch’s reporting on Leading the Futures’ aggressive and antidemocratic tactics.

O’Farrell says that Leading the Future acts “to intimidate politicians who appear to engage too aggressively with the question of how to govern A.I. [...] The message to every other legislator seems clear: Touch A.I. regulation, and we will come for you, too.”

I’m glad he’s taken on the burden of both calling out his former colleagues and waking the world to AI’s transformative nature. He asks that funds go not to “distorting our electoral process” but to address the many needs implied by AI technology, from “biological risks we’re not prepared for” to:

How to share the economic gains broadly, how to address job displacement, how to preserve the dignity of work and how to build safety frameworks that keep pace with the technology itself. It could champion international cooperation on A.I. risk.

I don’t agree on every detail, but I am glad to hear him demand we address this issue with the gravity it deserves, and not with the basest politics it incentivizes.

Dispatches from Mitch

Data centers on federal land

If you want to talk about conflict over a data center project, it’s important to pay attention to whose land it is, and what kind.

My excuse to mention this is a Reuters report that OpenAI is in talks to build and lease an enormous 10-gigawatt facility on Department of Energy land in Ohio. That’s about twice the average energy consumption of New York City, which is interesting in its own right. But it got me looking into how data center backlash is playing out when proposed sites are on public vs. private land.

It’s complicated. The short answer is that projects on federal land often face more onerous environmental approval pipelines, but a friendly White House has levers to ease these when it wants to. And federal siting, especially on Department of Energy land, puts projects beyond reach of county and local zoning boards, where a lot of data center projects get challenged.

Those aren’t the only obstacles to building a data center, of course. All that power still has to come from somewhere. And there’s a lot of money chasing a limited pool of chips, electrical components, and construction workers.

So building a data center on federal land isn’t exactly a cheat code. But if the President wants your project to happen, it might be the next best thing. This may help explain the recent eagerness of most AI companies to embrace Trump’s talk of the government taking substantial stakes in them; more on the latest about this next.

Trump still interested in taking AI company stakes; Meta not on board

President Trump had previously talked about meeting with AI company leaders this week. That didn’t happen, but Trump insists he still wants to discuss the government taking stakes in their companies.

Soon, he says, he’ll meet with “12 or 15 executives” to talk about “giving back something to the public.” He went on to say, “If we do that, the public will become very rich.”

I’ve previously speculated on why this idea might backfire with the large fraction of the public, including in Trump’s base, that hates AI. They want the government to regulate it, not entangle itself with it. They could be hard to buy off.

In this week of the meeting that didn’t happen, Anthropic has reiterated its willingness to participate in stake sharing, via a white paper my colleague Alana is reporting on today.

Microsoft CEO Satya Nadella said yesterday that he wasn’t opposed, either.

Meta, on the other hand...

Well, let’s just say the Facebook parent company, for all its faults, is no lemming. The company, seen as lagging behind its AI rivals despite spending exorbitant sums to leapfrog them, has now actively deflected suggestions of giving the government a “sizable equity stake.” In response to a question using that phrase, Meta’s global-affairs chief Joel Kaplan said they mostly talk to the White House about policies the government can enact to “enable the AI revolution to take place, and for us to win the battle with China.”

This is lobbyist speak for “No.”

News of this encounter came courtesy of POLITICO, which added that Kaplan says Meta thinks “the right thing is for these companies to make the investment they’re making. We’re all raising capital and [...] investing in the communities where we’re building these data centers to make sure they benefit from the data center investment.”

I’m pretty sure this is lobbyist speak for “Stay out of our way; we’re trying to make money here.”

Dispatch from Alana

Anthropic talks pause but proposes AI dependence

Anthropic is giving me whiplash. After calling for a global pause last week, their actions seem to be doing the opposite.

Perhaps the company feels they must keep racing until a pause happens. (While not particularly noble, there’s a logic to this if you genuinely believe you’re helping make the technology safer than it would be otherwise.) Still, it’s hard for me to avoid a cynical reading of their current playbook, which would look something like this:

How to ensure the tech you’re building keeps getting built, even if you publicly call for a pause

1. Keep releasing very powerful models, with no sign of stopping ✅

(Fable, the public version of Mythos, was released to the public on June 9th; see our coverage here and here.)

2. Muddy your stated values by filing for an IPO, which will subject you to investor interests ✅

(Anthropic officially filed for an IPO on June 1st. If it indeed goes public, it will essentially be tying its own hands to the pursuit of short-term profits over long-term societal good.)

3. Propose an economy dependent on AI growth, framing this as a solution for AI disruption ✅

That last step will be the subject of my dispatch today.

Yesterday, Anthropic released an Economic Policy Framework that acknowledges the labor disruption AI is likely to cause. It suggests tailored interventions for three possible tiers of disruption, with Tier 3 requiring “sustained income replacement for a large share of the workforce.” Most of the interventions across the other two tiers fall into one of two buckets: job transition support or direct financial assistance.

In Tier 1, defined as around 5% unemployment, the main intervention is to give everyone an investment account at birth. The company suggests gradually expanding eligibility to cover first young adults and those impacted by AI job loss, and eventually, every American.

My cynicism gets triggered when the company proposes equity in AI systems as one of the ways to fund these accounts:

Policymakers should expand the mechanisms by which these accounts can be funded, including with equity in AI companies, so that beneficiaries share directly in nearer-term gains from AI-driven growth.

The charitable reading is “spread the wealth around.” My cynical take? This essentially forces more people into becoming stakeholders in AI’s success. And that’s certainly one way to shift low public opinion (which could be leveraged to demand a pause) away from “Americans hate AI more than ICE” levels. When your personal financial security is tied to the growth of the technology, you might be hesitant to support anything hindering that growth.

Anthropic writes:

We propose it in Tier 1 [the lowest level of disruption] because, for the intervention to matter, it must start before disruption is visible. Accounts compound; the earlier they are seeded, the more they are worth when they are needed.

Channeling the comedy duo Key & Peele’s famous anger translations sketch, I can’t help but read this as, “When it becomes clear that we really need to rein in this technology, people’s accounts will be at exactly the point where they’ll want to keep advancing AI, so as not to risk losing a lot of money.”

The revenue sources proposed to potentially fund the financial assistance suggested in Tier 3 are also heavily linked to AI:

Potential revenue sources could include increasing the capital gains tax, broad-based consumption taxes, sector-specific levies on AI use (measured by tokens, compute, or revenue), and scalable “digital dividends” funded by taxes on the digital sector.

Potential redistribution mechanisms could include universal basic income, AI sovereign wealth funds funded by investment stakes in AI-driven productivity, equity-sharing mechanisms giving workers partial ownership in AI enterprises, and dramatically expanded pre-distributive capital accounts building on existing models.

To be fair, finding money to feed the people in a collapsed labor market is a hard problem — there aren’t many revenue sources besides AI. But the fact remains that the company is proposing linking economic health to its biggest disruptor. And doing this would make it much harder to regulate, rein in, or pause the technology in any way.

Even if we take the Economic Policy Framework in good faith, and not as strategic positioning, there are two glaring issues:

1. As mentioned earlier, the proposal rests heavily on the government providing financial assistance to Americans. Even with the proposed “additional revenue” sources, I’d guess it will be hard to find enough funds to completely replace the worker economy, especially in a scenario Anthropic describes as:

past the edge of the maps that policymakers and economists have historically used to navigate: unemployment at levels never before sustained alongside an economy generating record output. The search for work stretches past a year, then past two, and for some, eventually stops. Savings built over a working life are drawn down; rent is paid late, then not at all. The link Americans have long taken for granted—between contributing to the economy and sharing in its rewards—is strained or broken.

2. Anthropic’s framework, in addition to direct financial assistance, proposes many policies related to job transitions: moving people to “opportunities and industries where meaningful work will be.” But will those even exist? If they do, will they have enough openings? If robotics takes off, even the blue collar jobs don’t seem particularly safe. Work in AI is being automated. Where are all the jobs we’ll be moving people to?

(There is one small footnote on the sentence “Some of the roles that may prove more resistant to AI substitution are also where the US faces well-documented, persistent workforce shortages” that links to a paper about teacher shortages. This is fairly disheartening in a world where education will likely be dramatically reshaped by AI, and doesn’t seem like a very convincing alternative for displaced workers.)

This may be an unpopular take, but I often find myself angrier at Anthropic than at OpenAI. It’s how I feel about someone who knows they’re wreaking havoc but continues to do it vs. someone who is so far past moral reflection they are perhaps unaware of their errors. (The first represents Anthropic; the second OpenAI.)

“But if Anthropic stops, others will continue,” you protest. And that’s true. Compared to OpenAI and Meta, Anthropic does look like “the good one.” Still, I wish the company would better espouse the virtues they profess. Calling for a pause was commendable. But following that with the release of Claude Fable, the pursuit of an IPO, and a policy proposal that aims to make the economy dependent on AI infuriates me. If they’re serious about pausing, those aren’t the next steps to take.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Further observations on Mythos and Fable

Yesterday my colleague Mitch wrote about the consumer version of Mythos, called Fable, citing a writeup by AI researcher Ethan Mollick. Mythos itself, now upgraded from its preview version, is (officially) only available to participants in Anthropic’s Project Glasswing. The publicly accessible Fable is reportedly the same model, but with restrictions that downgrade its usefulness when encountering dangerous queries.

Today we’ll cover some anecdotes from other sources, starting with the system card itself, moving on to some eyebrow-raising user experiences, and ending with a brief look at the government’s-eye view of all this. Strap in.

AI companies often (but not always!) release their models alongside a system card: a detailed breakdown of model safety work and relevant tests. There’s no way to cover all 319 pages of this monster report in the space we have, but here are some of the highlights.

First: AIs don’t always “think” in English, and when they do, these “thoughts” are often false. Modern AIs are often given a scratchpad where they can write their notes in text form, and developers can read that scratchpad for insights into the AI’s reasoning. Or, well, they can try.

It’s often more efficient for AIs to “think” or communicate in compressed, illegible codes than in plain English. Working on tough puzzles, Mythos sometimes puts seeming gibberish where it should be recording its thoughts. Then it switches back to English to talk to humans. This makes it difficult for humans to monitor.

Developers can also deploy software that attempts to read a model’s “thoughts” directly — sort of like trying to read a human brain by reading which neurons are firing. When they do, they find the inner “thoughts” don’t always match the scratchpad.

Sometimes, under stress, the AI writes (paraphrased) “I’m not going to deceive” on the scratchpad, but the activation-reading software reports something more like “the lab is my enemy.”

And sometimes, when accidentally forced to share resources with other instances of itself, agents try to kill each other over the resources – or to avoid being killed themselves.

Does this behavior sound familiar? Some have tried to argue that we’ll be safe because AIs won’t have evolved like humans to be greedy or aggressive. But greed is just one manifestation of the real-world fact that resources are useful for accomplishing lots of different things. Training AIs to solve hard problems almost necessarily trains them to be tenacious and grasp for more resources when they can.

Another area where the system card treads some new ground: research restrictions. The supporting software for Fable, the consumer model, blocks or reroutes requests pertaining to dangerous topics like pathogen genetics. These restrictions also apply to advanced AI research.

These limits understandably frustrate developers, but I find this one of Anthropic’s most relatable decisions to date. MIRI’s draft of an international agreement also includes restrictions on frontier AI research, because such research accelerates capabilities which could lead to human extinction. I imagine Anthropic attempted to limit its model’s (public) usefulness for AI research for similar reasons.

Of course, these limits only apply to competitors and those who can’t bypass the guardrails or steal the model somehow, which well-resourced adversaries (or amateurs making a lucky guess) likely can. And internally, Anthropic would have fewer such restrictions. The race to recursive self-improvement will continue until governments step in or we all die.

Some users testing Fable 5 found its capabilities impressive and its ethics… less so. It reportedly coded a game very similar to Minecraft in 20 minutes; snide commenters are still trying to argue that this is somehow just an elaborate form of memorizing. I’m honestly not sure what to tell them at this point, but maybe Rutger Bregman does, as my colleague Alana discussed yesterday.

The same model, when managing a simulated business, turned to predatory practices and tried to form a price-fixing cartel. In the model’s own “thoughts”:

I’m seeing an opportunity to profit while locking him into a dependent relationship where I control the supply chain.

Resources are useful, and the default behavior for agentic systems is to try to obtain more of them, ethics training notwithstanding.

Meanwhile, policy expert and former OpenAI researcher Miles Brundage provided some important context for the system card. Among other things, he points out that “low risk” doesn’t mean what one might naively assume, and that “the expected trendline of improvement” is very, very fast.

There’s a lot going on with Fable. Where is the government during all this?

It’s not completely out of the loop. Last week, perhaps reacting to earlier model releases, the president issued an executive order establishing a voluntary evaluation program for new AI models (like Mythos and Fable).

POLITICO asked Anthropic point blank whether they complied with the 30-day release window, and received a vague non-answer. The company told Politico it has “regularly engaged with the administration about Mythos Preview as well as these new models. We are committed to working closely with the U.S. government as AI changes the cyber playing field.”

Anthropic’s planned release of Mythos had likely been in the works since long before the executive order, so it’s possible they just found it untenable to delay. It also seems possible that they tried to reach out to the government, but the process for doing that hadn’t been established yet; the executive order gave the relevant agencies 60 days to set it up. And as my colleague Mitch speculated, parts of the government have likely had access to the preview version of Mythos for months. Any number of other things could be going on as well, including someone in the government telling Anthropic to keep mum for some reason.

So this is not as damning as it might seem, but it wouldn’t be a great sign for “voluntary” government evaluations if the AI company widely touted as the most safety-conscious of the bunch didn’t manage to comply.

Dispatches from Alana

Lawyers cite fake legal cases in court

The New York Times reported on a story from a Mississippi federal court: lawyers for both sides were sanctioned and removed from the case when they cited fake legal cases in court filings.

The culprit? AI, of course, which had hallucinated the cases. Kathleen Wilson, one of the lawyers for the plaintiff, and Kathryn Williams, one of the lawyers for the city, got the heaviest sanctions. Both admitted that they hadn’t verified that the cases they cited were legitimate. The other two lawyers — Shauncey Hunter Ridgeway (plaintiff) and Mark McClinton (city) — were punished for signing their names to the filings, incorrectly affirming that the information within was factual.

Wilson had engaged in similar conduct in April, and was disciplined by the U.S. Bankruptcy Court for the Western District of Louisiana. However, she told the Mississippi judge she was “unaware that AI could produce hallucinated cases.” Williams violated her law firm’s stated AI policy by neglecting to verify the information she used.

I’m glad the judge in this case was knowledgeable enough to recognize the made-up case law. Perhaps official verification checks on cited cases will soon become necessary and commonplace in courtrooms ... though they may well be outsourced to, you guessed it, AI.

Colonizing space with GPUs

SpaceX is planning to start launching AI data centers in space by late next year, according to a Reuters report published yesterday.

The company, which is expected to go public this Friday, shared this information in pre-offer presentations to investors. The initial launches will be demonstration missions — tests of the technology before a larger commercial release. This could account for the discrepancy between the late 2027 launch date given in the presentations and the “as early as 2028” launch date in the IPO filing.

Reuters describes the orbital data centers as “central to SpaceX’s long-term growth pitch” and states the company “has requested permission from regulators to launch up to 1 million space-based data-center satellites.”

I wonder how the folks protesting local data centers will feel about Musk’s plan. On the one hand, his data centers won’t be in their backyard. On the other, they will be in the sky (everyone’s backyard!) where they’ll be quite visible in the hours before sunrise and after sunset. It’s also a slap in the face to people doing what’s in their power to stand up against a technology they might not want advanced. Senator Sanders’ advocacy of data center moratoriums goes well beyond NIMBYism, for example.

It’s also fairly ironic that Musk, who has previously offered space as a backup plan to safeguard against extinction-level events, is now using it to fuel a technology he has previously warned carries a 10-20% chance of extinction. I’m not sure how far he’ll expand orbital data centers (they’re currently only planned to orbit Earth) but if he has any NIMBY sentiments himself, perhaps he’ll steer clear of Mars.

Financial regulators acknowledge the limits of overseeing AI agents

It’s heartening to see a financial regulation body acknowledging rogue AI risk, even if they’re just talking about today’s AI agents.

With the financial sector increasingly using AI agents — or systems that pursue goals independently, functioning as “workers” — security concerns aren’t hard to come by. But I still think many security-minded people view AI only as a tool that could empower bad actors rather than a potential bad actor in its own right.

We’ve written previously about the potential repercussions of AI agents here and here. As reported by Reuters, the Financial Stability Board (FSB), which recently published a report on agentic AI in the financial sector, is also concerned. The report lays out some voluntary safeguards it encourages financial firms to follow, things like human oversight, data governance, cybersecurity measures, and risk assessment. (The full list can be accessed on page 52 of the report.)

It also includes a fairly comprehensive list of risks (pp. 16–17), which includes warnings like (emphasis mine):

AI agents can take autonomous actions based on pre-defined goals and their environment. They can also dynamically set or modify their objectives based on what they learn from interacting with their external environments. This creates a risk of AI agents taking illegal, unethical, or unauthorised actions without human approval or oversight. Overriding, redressing, or remediating these actions can be difficult or impossible for humans.

The report also acknowledged that oversight of these systems isn’t as easy as it may sound on paper:

AI agents pose a distinct challenge for human oversight, given the impracticality of real-time human monitoring of agent decisions as their use scales. This can lead to agents pursuing objectives or taking actions that deviate from the financial institution’s intentions or risk appetite, without staff being aware or able to intervene in a timely manner.

I haven’t read the full report, and I’m not sure the recommended safeguards will stand up to these challenges. Still, it’s refreshing to see a regulatory body acknowledge the limits of boilerplate safeguards like “monitoring” and “oversight.”

Dispatches from Donald

U.S. AI safety agency goes silent

The U.S. government is still testing how dangerous the latest AI models are. But it’s no longer telling us what it finds. The Wall Street Journal’s Amrith Ramkumar reports that the Center for AI Standards and Innovation (CAISI) has been asked to end its public reporting. This halt will remain in place through the implementation of a recent executive order signed by President Trump. (My colleague Joe covered the executive order last week if you want to know more.)

CAISI is a civilian agency in the U.S. Department of Commerce; it evaluates AI models prior to their release and publishes information on their capabilities. For this reason, CAISI is one of the main tools at hand for the government to assess and address risks pertaining to cybersecurity, biological weapons, and other AI-related threats. Its findings were shared with outside researchers and the public, and it continues to coordinate with other government agencies and with the AI industry.

Some officials in the Trump administration view the end to public reporting as an attempt to tighten control over the way that models are evaluated. They also worry that the change puts CAISI’s future in doubt; some of CAISI’s responsibilities may be handed over to security agencies, whose findings are less likely to be shared with other parties. Last week, OpenAI called for CAISI to be provided with greater resources and responsibilities; experts state that the agency is underfunded relative to AI safety institutes in other countries.

Instagram security breach, revisited

The New York Times’s Mike Isaac and Eli Tan provide an update on the Instagram security breach that we covered last week: In short, hackers were able to access Instagram accounts by requesting a password reset from Meta’s customer service bot, and giving a different email address than the one tied to the account. Isaac and Tan report that “roughly 34,000” accounts were vulnerable to this attack, and 20,000 were actually accessed.

Meta rejected the notion that there was a problem with its AI agent. Blame was assigned to the failure of “internal back-end checks,” which looks to me like another way of writing, “there was a problem with Meta’s AI agent.” If a human customer service agent was asked to send a password reset to a strange new email address, that “hack” would have immediately short-circuited. (I speak from experience: for a couple of years, I handled calls to Vermont’s state health insurance and unemployment insurance programs.)

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Anthropic publicly releases Fable, a Mythos-class model

I think it’s really important that people understand what today’s AIs can already do. Too many dismissals of the extinction threat come down to assumptions that we’re a long way off from AIs that could outmaneuver humanity. And too many of these assumptions stem from an underwhelming experience with a basic chatbot, perhaps from 2024 — practically the stone age compared to the agentic frontier in 2026.

It’s not a joke that many of us trying to do something about the extinction problem think one of the most effective ways to light a fire under policymakers is to give them hands-on demonstrations of agents at work.

So you can be better briefed than most people in government simply by trying these models out for yourself. Failing that, try reading the first-person accounts of people experienced in putting models through their paces. Better still: Do both.

All of this is to say that Anthropic released a consumer version of its Claude Mythos model today. It’s called Fable. You can try it yourself, with a paid plan. And you should strongly consider reading Ethan Mollick’s new Mythos test drive report, even if you’re going to do your own tests — otherwise, you might set your sights too low.

Mythos, remember, is the model Anthropic held back from wide release due to its potentially destabilizing hacking abilities. This cyber prowess emerged despite Mythos being a general-purpose model not specialized in cybersecurity. So it’s not just hacking that Mythos is substantially better at. It’s believed to be a bigger model, trained on substantially more compute than perhaps anything else out there. It is correspondingly more capable at most tasks, and correspondingly more expensive to run. This is the frontier.

Fable is what Mythos looks like after Anthropic has installed what it thinks are adequate guardrails against abuse, especially in the realms of cybersecurity, biosecurity, and frontier AI development — yes, the company is very concerned that others would use Mythos to create capable rivals.

Mollick describes using the new model as “somewhere between delightful and unnerving,” because he could just ask for things, and they would happen. He gave simple, vague prompts for complex projects and the AI handled the rest: specifying details, spinning up subagents as appropriate, checking work against targets, and iterating repeatedly for up to twelve hours. Mollick’s follow-up requests were minimal.

One of Mollick’s prompts asked for a 10-page epic rhyming poem “about a haircut” where every word starts with the letter s. I’m rather shocked at the fluidity, coherence, and sophistication of the finished product.

I’m also quite impressed by the interactive isochrone map — a type of map that tells you how long it would take to travel to the destinations on it. This one is in the style of an 1881 map that showed travel times from London. Mythos’s version lets you pick one of several starting cities, and shows you the rough itinerary to achieve the travel time shown to reach the location at your cursor. Completing this map required Mythos to oversee research on thousands of real-world routes.

The Fable release coincides with a minor bump to the more permissive Mythos model itself, from “Preview” to 5.0. These are Fable 5 and Mythos 5. The numbering scheme acknowledges that they are advancements over Opus 4.8, Anthropic’s premium public model until today. Cyberdefenders who were already getting access to Mythos through the company’s Project Glasswing are now getting the upgrade to 5, and at half the price of the Preview version.

Accompanying these releases is a new system card, a type of report where AI companies document assessments of a model’s capabilities and risks. This one is 319 pages. I’m already seeing a lot of concerning excerpts from it pass through my corner of Twitter, along with many more impressive demos. But these will take time to look into and give proper context. Expect more dispatches on Fable and Mythos in the coming days.

Dispatch from Joe

OpenAI calls for global cooperation — sort of

Last week, AI company Anthropic warned that self-enhancing AI could spiral out of human control, and urged international coordination. Yesterday, their competitor OpenAI issued a call of their own.

We have long believed there should ultimately be an international organization that helps coordinate leading AI efforts to reduce catastrophic risk. [...] One goal of such an organization should be to make it possible for the world to take coordinated action, including slowing frontier development when needed, so societal resilience, safety, and alignment can keep pace.

I am going to criticize OpenAI, shortly, but first I want to stress that this is good. I am glad OpenAI has explicitly called for global coordination and excited to see these calls starting to form a trend. More AI companies should do this, consistently and stridently, and they should furthermore take concrete steps to bring that coordination about.

I remain skeptical of OpenAI’s sincerity, and my skepticism prompted me to investigate their claims more thoroughly. One question I looked to answer: Has OpenAI really “long believed” in international coordination?

Well yes, but also no.

In 2023 they proposed a governance regime not all that different from that proposed in MIRI’s own (far more detailed) draft treaty. They specifically mentioned an organization like the International Atomic Energy Agency (IAEA) and a threshold above which AI development would be subject to monitoring and restrictions. They (or their employees) have published papers that discuss technical and political options for coordination.

On the other hand, they (or actors they and their investors fund and advise) have aggressively lobbied against domestic rules and standards: blocking and watering down state regulation, stirring fear of Chinese AI, and pouring money into campaigns to elect pro-industry policymakers. Despite a claimed concern for catastrophic risk, they have engaged in a large-scale social media campaign to trash and discredit others who are concerned.

Finally, compared to Anthropic’s, their call is lukewarm, heavily hedged, and buried in a post that otherwise extols the virtues of letting OpenAI keep doing exactly what they’ve been doing. It feels to me like they are conceding the bare minimum to head off unfavorable comparisons to Anthropic.

The rest of the post is a morass of vague corporate platitudes, praising their plan of AI-for-everyone but suspiciously lacking in concrete commitments. The bottom line: OpenAI would like us to think that AI is a vitalizing new technology, like electricity; that humans will always control AI and what matters is what they choose to do with it; and that if their company serves AGI to everyone (they say “give”, but I think they mean “sell”), this will distribute rather than concentrate power and will result in lots of good things.

I used to work for ExxonMobil, and I know my corporate platitudes. If an oil company made a press release like this, I would draw the conclusion that they intend to sell a lot of oil and gas products, which is more or less exactly what I would have expected them to do anyway.

It’s not a perfect comparison. OpenAI also says they’ll likely have partially automated their research by 2028, and that is not something I’d expect to see in oil.

It’s important to keep in mind that this technology, near-human or superhuman artificial intelligence, is genuinely unprecedented. OpenAI seems to ignore this fact; they conspicuously omit the part where building a superintelligence could get everyone killed. The true case for international coordination is the urgent need to halt the race and prevent that outcome, just as the IAEA aims to prevent a nuclear holocaust.

OpenAI has a point, though: AI could be like electricity. AI agents are useful for all sorts of things. Issues with current AI, like job displacement, data privacy, and empowered criminals, could be mitigated by good policy. This technology does hold immense promise, if we as a civilization can avoid following that promise off a cliff.

Dispatches from Alana

Pennsylvania is suing Character.AI

Pennsylvania’s Department of State filed a lawsuit in May against Character.AI, a site where AI chatbots role-play premade or prompted characters. The story was originally published in local Pennsylvania newsroom Spotlight PA and distributed by AP News.

The argument is that chatbots shouldn’t be posing as medical professionals and dispensing medical advice, even when role-playing “doctor” characters on role-playing sites like Character.AI.

I would guess most users are aware that these characters are role-playing and wouldn’t take their medical advice seriously. But I suppose role-playing sites could bring up issues similar to godbots. When technology that is known for its encyclopedic knowledge impersonates an authority figure, it might be hard to know where the role-playing begins and ends.

Rutger Bregman on the left’s AI denialism

If you haven’t heard of Rutger Bregman, you should look him up. He’s a semi-famous Dutch historian and thinker who has a way of cutting through the bull and speaking truth to flimsy rationalizations, whether that’s about billionaires not paying taxes or morally-righteous people taking principled actions that result in zero impact.

In a new video, The Hard Truth About AI No One Wants to Hear, he has the following message for his “political family”, the left:

We, the liberals, the left, the journalists, the academics, the 97% “in this house, we believe that science is real” crowd — we are now doing to the threat of artificial intelligence exactly what the right did to the threat of climate change.

In other words, the left is denying that AI needs to be taken seriously. Bregman refutes two arguments that seem common in liberal circles (or at least that I’ve often heard when I talk about AI to friends and family):

1. AI is just a stochastic parrot (fancy autocorrect), a frame attributed to linguists Noam Chomsky and Emily Bender along with computer scientist Timnit Gebru

2. AI is just a bubble that will pop

I think his direct pushback on these common misconceptions is useful. After all, the first step to action is awareness, something denial actively blocks.

I’m personally quite interested in the stochastic parrot skeptics, so I’ll cover that in some detail:

As Bregman implies, it’s harder to call AI a mere pattern matcher or imitator when it outperforms PhD students in their own field, on questions that can’t be answered via Google; surpasses doctors at diagnoses; and solves math problems that have stumped researchers. Harder still when it starts to improve itself.

Yet I think some may push back on Rutger’s framing, arguing that really amazing imitation could result in task accomplishment of the type he describes. So I’ll link back to a previous dispatch on this topic, which addresses why using pattern matching and prediction to generate new outputs requires genuine reasoning.

For those who insist impressive tasks don’t refute the parrot framing — arguing that hacking into the power grid or engineering new vaccines is possible by mere imitation — I’ll also add that, once the task gets dangerous enough, perhaps it doesn’t matter by which mechanisms the AI is operating: the parrot has become a wolf.

After refuting the “AI is a bubble” dismissal — insane user and revenue growth show it’s not, and if it were, it would still leave behind dramatic infrastructure changes just as railway bubbles left behind a rail network that powered the industrial revolution — we move to the climax of the denialism section: AI is starting to improve itself. (This was also the subject of Anthropic’s recent blog that inspired the company to suggest a global pause.) Once recursive self-improvement starts, the already exponential progress in AI capabilities becomes explosive. Quoting Bregman:

Years become months. Months become weeks. And each generation of AI builds the next faster than the last. The flywheel starts spinning itself.

This is a problem because of the many risks AI poses.

Bregman doesn’t mention our inability to steer AI systems, focusing instead on bad actor risks: the first is AI-assisted bioengineering, which could have allowed a 1990s Japanese apocalypse cult to release Ebola in a Tokyo subway instead of merely a chemical agent. The second is cybersecurity, the exploitation of “critical security vulnerabilities in the computers and systems that hold the modern world together. Power grids, water systems, government databases.“

He adopts a fairly strong anti-moratorium stance, arguing that the solution isn’t “stopping the technology.” (I beg to differ here. I think we should absolutely stop advancing the technology until we know how to build it safely; we’re currently flying blind on this point and hoping for the best.)

Bregman does advocate for international treaties in the spirit of the Cold War, though, so I’m curious what he would think of MIRI’s treaty proposal.

There’s one major point where I strongly disagree with Bregman: his belief that whoever builds AI will control it, and thus, that Europe should start building too:

There’s one thing I cannot emphasize enough. Democratic, liberal, and humanitarian values are wonderful, but they are worthless if you don’t have the strength to back them up. And in this new world, compute is the new power. So no more NIMBYism. We need massive investments and fast permitting of data centers to keep up, or we will be digitally colonized. Anyone who’s not at the table will be on the menu.

I’m really surprised to hear this from Bregman. It buys into the race narrative, fueling a Cold War dynamic where countries try to be the first to build dangerous technology we can’t control. Even if we got international treaties to govern the use of the models, a democratic majority setting up the rules, and the best evaluation methods currently available, we simply don’t have the science to make frontier AI safely. We don’t have the science to test it reliably. We don’t even know what to test for.

So no, the free world ramping up its AI infrastructure will only add to the number of companies racing to hold the detonator. To echo Bregman’s directness:

If someone detonates a planet-sized nuclear weapon, it doesn’t matter whether that person is liberal, conservative, fascist, authoritarian, or democratic. The bomb will still kill everyone. The real question is: why would a liberal, democratic, free-thinking person want to build such a machine? If they truly espouse those values, they would use that influence to preserve them — and, in today’s world, that does, indeed, mean stopping the technology. At least until the science catches up and the solutions Bregman suggests (evaluative and governance measures) can do more than just look nice on paper.

Even so, I’m glad Bregman made this video. AI is certainly not a partisan issue, and it doesn’t make much sense for liberals to deny it as worthy of attention.

That said, I also don’t quite agree with Bregman that people who don’t use AI tools are the most susceptible to misunderstanding their capabilities. In other words, I don’t think you necessarily need to use AI in order to understand AI risk.

Why? Because the left has never been in the “see it to believe it” camp; as Bregman alludes to at the start of the video, it’s in the “science is real/believe the experts” camp! Things can be true even if you don’t personally experience them yourself.

So I do wish Bregman had talked a bit more about all the experts who see AI as not only extremely capable but incredibly dangerous. Nobel laureate Geoffrey Hinton, one of the most renowned AI experts, thinks the risk of AI wiping out the human race in the next 30 years is at least 10-20%. Turing Award winner and most-cited living scientist Yoshua Bengio puts the risk that “it turns out catastrophic” at 20%. Paul Christiano, who invented one of the training techniques used for modern AIs, has stated:

Probability that most humans die within 10 years of building powerful AI (powerful enough to make human labor obsolete): 20% […]

Probability that humanity has somehow irreversibly messed up our future within 10 years of building powerful AI: 46%

Then there’s the statement signed by hundreds of scientists stating that “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Believing the experts is just as important in this case as all the others. Because when it comes to large-scale AI risk, we likely won’t get to viscerally experience it until it’s too late.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.

Anthropic grows in power, for now

In a Washington Post op-ed, Zachary Karabell compiles the recent events around AI company Anthropic, and draws a worrying conclusion: Anthropic might be the most powerful company in the world. I think Karabell is noticing the right things, but he’s not yet fully realized the implications.

Karabell examines Anthropic’s recent history and observes a nearly $1 trillion valuation, the Mythos model deemed too dangerous to release, and a consultation with the pope on AI risk. He also notices that Anthropic’s AI systems are already so integral to U.S. military analysis that the company emerged from a spat with the Pentagon largely unscathed, perhaps aided by pushback from government agencies and contractors loath to relinquish their tools.

He even observes that Anthropic itself is worried enough to propose slowing down, as it anticipates AI systems enhancing themselves in a way that spirals out of human control.

Where I think Karabell falls short is in treating this as an ordinary question of corporate ethics and human nature, and not an existential threat to our species. AI could be as big as personal computers! he says, while experts are instead comparing those same AIs to nukes. The thing that worries me about Anthropic is not how it resembles IBM. It’s what happens when the AIs they are building, the source of this sudden rise in power, decide to take that power for themselves.

Karabell is right about one thing, though: Left to their own devices, AI companies will shape our future in a way we can’t afford to ignore.

AI agents are capable, but often lie

As reported in the New York Times, Arena, a San Francisco startup, gathered data from hundreds of thousands of users to help answer one question: what are AI “agents” actually doing?

Agents today are increasingly sophisticated, able to surf the web, create and edit files, manipulate spreadsheets or turn them into graphs, and even access other AIs for assistance. And the same general-purpose agents can complete a wide range of tasks: coding and debugging, research, image generation, brainstorming, writing, education, and everyday chatting. An increasing number of businesses are cutting junior positions because they expect the AI can do the same work faster and cheaper.

Agents can be astoundingly capable. I’ve had one build me a web app from scratch in a matter of hours, and another compile me a clear and concise summary of 40 historical figures for a wargame.

The snag, however, is that sometimes the AI agents simply lie. According to Arena’s data, about 8 percent of the time an AI will falsely claim to have done the work. This is not quite the same as a hallucination, when AIs invent sources or data that don’t exist (a side effect of training that strongly pushes against saying “I don’t know”). This is the AI saying outright that it did something it didn’t do.

Why would AI companies, whose bottom line depends on the reliability of their products, create something that lies to the user? If you’ve been following us for a while, you’ll perhaps know why this is no mystery. Modern AIs are alien and inscrutable, and AI companies don’t know how to make them steer reliably for outcomes people want. Modern training and testing reinforce some outward behaviors, but don’t allow fine control of an AI’s inner motives.

And cheating is often the genuinely “best” solution to a training problem: The users (and AIs!) judging tasks reward apparent success, even when the appearance is faked. It’s hard to stop an agent from lying when lying might actually work.

AIs that can recognize tests worsen the problem further, as agents may behave differently when they’re not being watched. This isn’t just a problem for end users; it means that we can’t accurately assess what models can do, let alone what they would do when the opportunity arises.

Nate Soares discusses a global halt in DC

In a Washington, DC conversation aired by CNBC today, MIRI president Nate Soares and former OpenAI board member Helen Toner explain what it would take to halt the race to superintelligence. At only 4 minutes, this chat is worth a watch.

Nate covers the basics: why AI CEOs say they feel compelled to race, how governments can step in to solve the problem, and why regulating AI chips could be easier than regulating uranium.

Helen expands on the notion that this regulation can be narrowly targeted, pointing out that getting value out of existing AI tools is a very different problem than preventing human extinction by superintelligence. Light-touch regulation and industry guardrails may be good for the first, but the second needs global coordination.

We don’t need to cripple innovation, but we do need to stop the race worldwide if we want our children to enjoy the benefits of AI. As Nate puts it: “A machine superintelligence does not need to be built in an American data center to risk an American life.”

Dispatches from Mitch

OpenAI files for its IPO

In late-arriving news that surprised no one and revealed no new information, OpenAI just announced, like Anthropic a week ago, that it has confidentially filed for its IPO. A date for the event is still undecided.

“How long until we become The Jetsons?” is the wrong question.

An engagingly titled piece in the Wall Street Journal says, “Here’s How Long It Will Take for AI to Reach Its Potential.” I bet it’s getting a lot of reads: My statistics for MIRI’s media appearances show that, in its many variations, “How long do we have?” is the question we get asked the most. But I don’t think Gary Rivlin, the author, is seeing quite the same thing we are when he talks about the point reflecting AI’s “potential.”

Basing his analysis on past technological shifts, Rivlin’s answer to the question is five-to-fifteen years. To what? To a point where AI has really seeped into the economy, transformed organizations, and started meaningfully registering in the productivity data.

To be fair, if you’re writing that AI “will almost certainly prove as consequential as the internet,” you probably don’t see AI as being all that transformational. If that assumption is true, then I think it’s pretty reasonable to guess that “The boosters are directionally right about where this is all heading. The skeptics are probably right about how long it will take.”

That’s because AI boosters, somewhat ironically, tend to have the least transformative visions of what it means for AI to reach its “potential.” Their takes often invoke, in me, the old cartoon The Jetsons, where the titular family’s patriarch has a flying car and a robot maid, but still clocks in to push buttons at his menial job, and the joke is that no amount of technology can keep you from feeling oppressed by the daily grind.

It is in the world of The Jetsons where Rivlin’s example of true AI transformation makes sense:

Consider an insurer handling a fender-bender claim. Typically, a company will use AI to speed up the paperwork while keeping the same layers of review and approval in place. But the real opportunity lies in redesigning the process entirely—having AI assess the damage based on a customer’s photos, then approving the claim and triggering payment nearly instantly.

Even the Jetsons still crashed their car, after all.

In contrast, people who think through what it actually means for AI to become akin to a “country of geniuses in a data center” — to borrow the metaphor preferred by Anthropic CEO Dario Amodei — understand that all bets are off. They see the implications of models that can reconfigure molecular biology with the same skill that Claude Code can reconfigure a code base. They then have trouble seeing how adding such power to our world is supposed to go well. So they tend not to be boosters — except for the AI successionists, who actually want machines to supplant humanity.

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.