Dispatch from Beck
Fast-talking AIs out-persuade experts
A new study out of the UK AI Security Institute and Oxford finds that:
Frontier AI can now out-persuade expert humans in conversation — [including] world-champ debaters and professional canvassers.
This held even when humans chose their topics, prepared in advance, and competed for £1,000 prizes.
First, the usual caveats about a newly published paper — while this seems strong on first consideration, upon review some problem will often be found or it will fail to replicate. But this paper does have many signs of robustness, including preregistration of methodology, a significant sample (6,923 respondents), and secondary tests.
The study compared the persuasive effects of humans, from lay people to elite debaters, versus recent frontier AIs. Study participants were paired with human persuaders or AIs for about 15 minutes of text conversation, and the resulting changes in their views were compared. Frontier models significantly outperformed all humans.
Then the authors restricted the AI’s volume of output to match that of elite debaters. The AIs were limited to about 55 words per message compared to about 300 while unconstrained, reducing fact-checkable claims by a factor of three. Under these conditions, there was no significant difference in persuasiveness between elite debaters and AIs. For that reason, the authors contend that the AIs’ throughput is their key advantage.
The study also investigates how AIs perform compared to a professional fundraising campaign. Canvassers for Save the Children and the best model in the study conversed with respondents who had the option to donate some or all of a £1 bonus. Canvassers increased donations by a small percentage, while Opus nearly tripled the effect.
One important caveat is that the study paid respondents for their engagement, while in real life attention may be the scarce resource. If no one reads what the models put out, their ability to output lots of convincing factual evidence isn’t very relevant.
So then, what do I make of this? It seems like these AI models are, in fact, able to persuade captive audiences more effectively than even exceptional humans. For the moment, these models’ abilities are likely constrained by attention, but I expect models to only get better at competing for it.
It’s also worth noting that while unconstrained models outperformed top humans, they induced only modest changes in belief. It could be that AI messaging makes humans’ opinions change more day to day, without significantly shifting overall conclusions.
This could have positive effects too, increasing the persuasive power of grassroots efforts and the factually well supported.
But if, after reflection, we conclude that AI models are super persuasive, regardless of how factually supported their positions are, that seems a dire state. Beliefs then become the result of who controls or funds the most persuasive models, undermining democracy and discourse.
Dispatches from Joe
Three different takes on extinction
Today, three different outlets covered extinction risk and loss of control. The articles all take very different approaches, but what interests me is where they agree.
In The Guardian, renowned computer scientist and AI researcher Stuart Russell wonders whether we’ll see a “Chernobyl-scale disaster” before governments regulate AI in earnest. He cites Anthropic’s recent post on AIs that enhance themselves, agreeing that “uncontrolled [recursive self-improvement] could produce a runaway feedback loop that leads to an irreversible loss of human control.”
Russell notes the sluggish response of governments to this crisis and asks what it will take to move them to action. He proposes licensing AI models just as we license everything from nuclear power to hairdressing. I think licensing isn’t enough — we need an international halt to the race to build superintelligence — but I agree wholeheartedly with Russell that regulation is overdue.
Author and computer scientist Cal Newport is more skeptical. In an opinion article for the New York Times, he accuses AI companies of “doom trolling” to garner attention.
I expected to dismiss this article as yet another wrongheaded “it’s all hype” claim of a sort we’ve debunked elsewhere, but I found myself agreeing heartily in several places. Newport rightly points out the implications if AI labs are sincere:
The first [option] is that they actually believe that the systems they’re building have a nontrivial chance of producing massively disruptive events — from destroying the economy in the best case to wiping out our species in the worst. If this were true, every reasonable ethical system would argue that there is only one acceptable response: to immediately stop working on any product that might accelerate such a future, and lobby with all of your resources to help force other A.I. companies to do the same. From a moral perspective, any other reaction would be monstrous.
Newport apparently doubts this, and the rest of the piece is framed around the problems with AI labs’ communication strategy, taking their insincerity for granted. His policy proposal nonetheless echoes that of Russell and many others: mandatory risk assessment before release (though not development) of frontier AI models. It’s heartening to see even skeptics beginning to recognize the value of sane AI regulation.
Finally, Gleb Tsipursky writes in the Washington Post that rapidly accelerating AI capabilities are outpacing our ability to understand and govern them. He cites this year’s International AI Safety Report, noting the pattern of capability outpacing visibility in multiple domains: impersonation, fraud, harassment, deepfakes, propaganda, coding, hacking, and more. Evaluations can’t keep up with models that know they’re being tested. Open models are nearly as good as closed and much easier to strip of protections.
These observations mirror Russell’s warnings about the dangers of uncontrolled AI acceleration. The Washington Post packages them under the headline “It may already be too late to control AI.”
While the trends Tsipursky cites are real, I don’t think they substantiate that claim. Governments still could step in and stop the race. But this move gets harder with every passing day of inaction, and I urge policymakers to act before it really is too late.
Hard numbers on AI use and sentiment
Pew Research published several new polls today on how Americans view AI. Several results struck me as interesting:
36% of Americans say they interact with AI at least several times a day (7% of these are “almost constantly”)
Far more people have heard of or use chatbots than in 2024, and a plurality think of chatbots first when pondering AI
38% of workers report using AI at work
63% of Americans think AI is moving too fast; 2% say too slow
I already knew chatbots were proliferating rapidly, and I knew that this was changing how people work and live. But seeing numbers that correspond to tens of millions of people still raises my eyebrows.
Skeptics often argue that future AI won’t rapidly transform society (or Earth’s surface) because even if AI becomes extremely productive, bureaucracy and red tape will slow diffusion. This seems partly true, but not true enough to stop the future from getting extremely weird extremely fast.
Nonetheless, there’s hope. If the two thirds of Americans who think AI is moving too fast each took five minutes to tell their representatives how they feel, AI would rapidly rise to the top of policymakers’ priority lists. And if even a fraction of us demand an international halt to the AI race, we might yet navigate the coming years successfully.
Dispatch from Donald
A longtime skeptic sounds the alarm
In 2019, OpenAI delayed the full release of GPT-2 (at that time, state of the art among large language models). Their primary concern was that GPT-2 could be used to mass-produce false or misleading text. Machine learning researcher Nicholas Carlini — then working for Google — thought these worries were overblown, per a profile in The Wall Street Journal.
For years, Carlini was regarded as the AI industry’s resident skeptic on claims about AI cybersecurity. But in March 2026, Carlini, now working at Anthropic, cautioned his colleagues that Claude Mythos should be held back.
Carlini was concerned with Mythos’s ability to find and exploit bugs in code, even code that he thought was secure. For example, Carlini had never before found bugs in Linux, but Mythos was able to find almost 500 bugs in a few days. “It’s pretty clear to me,” Carlini said, “that these current models are better vulnerability researchers than I am.” His warning was responsible, at least in part, for “Bugmageddon,” a paradigm shift in the cybersecurity community and recognition that AI models would make cyberattacks easier and more effective than ever before.
My colleagues and I have already covered Mythos and its eventual public release, Fable, in some depth (see here for our initial coverage of Fable, and here for more). After the White House ordered Anthropic to restrict access to Fable, Anthropic sent Carlini to reassure the Trump administration that Fable was safe.
The bottleneck of cybersecurity is no longer in finding vulnerabilities, nor in exploiting them. It is in fixing them: In February, Carlini and Mythos discovered a bug in the web-publishing software Ghost. The bug would allow anyone to edit a website that had been published with Ghost. Carlini alerted Ghost’s developers, who quickly fixed their software, but not every website built with Ghost has added that fix. According to the cybersecurity firm Xlab, more than 700 websites have been hacked since the bug was made public knowledge.
Mythos was only ever made public in a defanged form. That doesn’t mean that the vulnerabilities Mythos exposed don’t exist. Carlini expects other models to catch up to Mythos in months. Nobody, not even Carlini, is sure what it will mean when anyone can fit a world-class hacker in their pocket.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.