In this issue:
An early warning system for AI incidents - In talks with the Chinese Vice Premier, Treasury Secretary Bessent suggests a joint channel for AI incident reports
Open models run autonomous influence campaigns - Iran, China, and private firms in Israel test AIs in political flame wars
Effective Altruism, and all that jazz - How a movement you may never have heard of became a Trump administration boogeyman
Dispatches from Joe
An early warning system for AI incidents
In talks with the Chinese Vice Premier, Treasury Secretary Bessent suggests a joint channel for AI incident reports
An international summit is a complex affair. Before Presidents Donald Trump and Xi Jinping meet later this week, American and Chinese officials hold early discussions about the details. The talks-before-the-talks on Sunday featured U.S. Treasury Secretary Scott Bessent and Chinese Vice Premier He Lifeng, and Bessent floated the idea of a notification system for early warning of AI incidents.
Reports are vague on the details, and we mostly just have Bessent’s word to go on. The New York Times quotes Bessent saying that the proposed channel would cover incidents at “the national security level,” and he elaborates in CNBC: “so both sides can agree on what the leading AI dangers are, whether it’s uncontrollable agents, whether it’s non-state actors in cyber, non-state actors in bio weapons.”
Although these AI dangers compete with trade and tariffs on the summit agenda, it is good to see them considered, and a rapid incident-reporting system seems a promising early step if implemented well. It echoes Haven’s guest post yesterday, which recommended “establishing a secure U.S.-China incident channel,” and the precedent set by such a channel could lay groundwork for further much-needed bilateral action. Bessent reportedly also said that he and He Lifeng would likely meet again in Shenzhen, China, in November.
Open models run autonomous influence campaigns
Iran, China, and private firms in Israel test AIs in political flame wars
We can now add “run an autonomous social media influence campaign” to the growing list of known AI capabilities. The New York Times reports that Iran and China directed Chinese open models (available for download on the internet) to run “networks of fake accounts on platforms such as Instagram, Facebook, X and TikTok,” which largely shared false, political, and inflammatory content.
Two Israeli private firms were also found to have launched autonomous campaigns. One firm, IntelEye, evidently focused its efforts on both sides of the upcoming Israeli national elections. IntelEye cofounder Maor Sellek claimed the campaign was for “defensive research and testing,” and remarked that in hindsight they should have told the platforms. The other firm wasn’t named, and we know precious little about what it was doing.
Suspicious as it is, there may be some truth to the claim that these activities were only tests. I doubt the state and private actors behind these campaigns expected them to make much progress; these seem like experiments, not serious efforts. The campaigns were crude and quickly discovered by U.S. intelligence, but they serve as a proof of concept for state-sponsored manipulation and will likely be refined in future attempts.
As with other AI-fueled crimes, such as fraud and hacking, the immediate problem for policing this activity is likely one of scale. Actors previously limited by the bandwidth of their human employees are now able to delegate an entire chain of manipulation to AIs, and can iterate rapidly, incorporating lessons from successive attempts to improve their reach and hide their patterns. Quantity has a quality all its own.
In the immediate future, this likely means a rise in fake account traffic on social media sites, which in many cases have proven unable or unwilling to curb what bots already exist. I wouldn’t be surprised if China and Iran have already spun up new, more sophisticated campaigns that might be harder to detect.
In the longer term — which for AI could mean a matter of months or years — this is a small bit of evidence about the potential of AIs to deceive and manipulate human institutions at scale. That’s a dangerous capability, not only in the hands of human groups, but also when wielded by AIs which serve only themselves.
Dispatch from Mitch
Effective Altruism, and all that jazz
How a movement you may never have heard of became a Trump administration boogeyman
I tried to keep StopWatch out of this. One of our founding principles is that we will try not to fuel or instigate infighting among those working to prevent our extinction from AI. And this publication is largely aimed at people normal enough to have never heard of Effective Altruism (EA). But with the term reaching mainstream headlines, the time has come to talk about it. I apologize in advance for the offense I am likely to cause.
Ineffective selfishness?
I first considered writing this post last week, when an official Twitter account belonging to the U.S. Department of War posted a promotional image for an “AI-FIRST WAR DEPARTMENT.” The text of the post reads:
Americanism, not effective altruism. The United States will continue to be AI DOMINANT! [US flag emoji]
That wasn’t the first time I’d seen executive branch officials pushing Effective Altruism (EA) as a woke, Luddite boogeyman, but previous occasions had all seemed intended for an insider audience that would know what EA was. With this tweet, my heart sank in recognition that the administration was starting to push anti-EA messages to the general public.
I found the post super cringe, because I doubted even 1% of Americans have heard of EA. I imagined readers thinking, “I knew Trump officials weren’t big fans of altruism, but why are they rejecting effectiveness? If you’re going to do altruism, shouldn’t you at least try to be competent?”
EA 101
Effective Altruism is a fuzzy cluster of people who self-identify as interested in trying to combine moral imperatives with rational thinking to do more good in the world. EA first crystallized as a named concept around 2009. The classic example of EA in practice is the distribution of anti-mosquito bed nets in malaria-prone regions, which research showed to be a very cost-effective way to save lives. EA’s earliest and simplest incarnations are sometimes described as “spreadsheet altruism,” and EAs themselves as well-meaning nerds.
EAs recognized that coordination, money, and political influence are all very helpful for being effective, so they formed some strong interpersonal networks and well-funded organizations near the center of the cluster. Much of that funding came from EAs who intentionally chose lucrative careers or were early to cryptocurrency. (These are nerds, remember.)
But effectiveness is only half the story. The altruism part is more complicated, because to maximize the good you have to first decide what counts as good. Most EAs would probably say they’re trying to do some combination of “maximize flourishing” and “minimize suffering,” but they split over targets and strategy. Do animals and other creatures suffer? How much should we weigh this, given that they might be less conscious but more numerous? How much should we weigh the lives of humans who don’t exist but might exist in the future, depending on our choices? How far should one pursue complicated plans that would do a lot of good at the end, but which risk corruption and loss of purpose along the way?
With that last question in particular, you might see how the arc of Effective Altruism can bend towards supervillainy for those who overestimate the strength of their moral fiber. Some of the best people I know or write about are EAs, but so are some of the worst. The movement is seen to have peaked around 2020-21, because in 2022 it was badly tarnished by the fraud and bankruptcy scandal of cryptocurrency exchange FTX, founded by major EA donor Sam Bankman-Fried. Many EAs stopped calling themselves EAs after that.
EA and AI
What does this all have to do with AI and the White House?
Well, being nerds looking for more leverage to improve the world, many EAs were unusually early to recognize that AI was going to play a huge role in our future — maybe a dominant role.
For some EAs, the immense good AI might enable made rushing to superintelligence a moral imperative. Every day without it brings roughly 165,000 deaths from aging and disease, a much larger amount of potentially preventable suffering, and a slice of the cosmos made permanently inaccessible by the expansion of space itself.
For other EAs, the immense harm AI could do made AI safety the most important cause — not just for the 8 billion people on Earth, but for the trillions of beings who might never exist if superintelligence wipes us out.
Many EAs were intellectually complex enough to hold both views at once. Some of these were early investors or engineers at OpenAI before leaving to form Anthropic over safety concerns. Dario Amodei, Anthropic’s CEO, was one of them. These people may or may not actually believe what they say about how it would be better if we could slow the race down. But they definitely believe our chance of surviving superintelligence will be worse if someone else builds it.
All this is to say that EAs are simultaneously some of the people racing hardest for superintelligence and some of the people now trying hardest to slow the race down — or at least talking about it.
EAs are not, as a group, trying to stop AI any more than musicians, as a group, are trying to play jazz. Sure, a lot of musicians love jazz and think the world would be better for embracing it. But even the aficionados are mostly playing more profitable genres instead — sometimes in the name of funding their eventual, someday pivot to jazz.
The Trump administration dislikes Dario Amodei for his principled stances against autonomous weapons, domestic mass surveillance, and — most recently — his calls for industry safety measures that might eventually, someday, slow AI progress a little. Metaphorically speaking, they can tell he likes jazz.
Those trying to prevent our extinction from AI dislike Amodei because he’s still pushing the frontier despite how vocal he is about the risks. Metaphorically speaking, they’re still waiting to hear him actually play some jazz.
EA as boogeyman
Blaming a shadowy cabal for inconvenient developments is part of the standard political playbook, whether it’s a “vast right-wing conspiracy” or the Dominion voting machine conspiracy theory. (Fun fact: “vast right-wing conspiracy” was brought to us in 1995 by Chris Lehane, then a political operative working for the Clintons, now OpenAI’s chief lobbyist.)
When Jacob Coxon’s resignation triggered a sudden swell of public concern about the AI extinction problem, the Trump administration could have chosen to pivot to addressing said problem. It instead chose to stick with a full-speed-ahead agenda. This made it necessary to undermine the extinction concerns, which it did by signal boosting conspiracies already being promoted by industry accelerationists allied with the administration. Some of these theories revolve around AI safety groups run or funded by Effective Altruists. So today, the White House would have you believe that extinction concerns are lies spread by those woke, Luddite EAs.
EAs are mediocre in the boogeyman role. They’re hardly MS-13 or Tren de Aragua — groups which don’t look very good even on their best days. Factions within EA have even welcomed the attention, expecting that rational people looking into them would find much to like.
As cabals go, EAs aren’t very shadowy, either. But like other historically popular scapegoats, EAs are well-networked and well-resourced. If you’re into corkboard and red yarn, you could spend many hours in your shed drawing lines between AI company staff building AI, third-party researchers evaluating said companies, and AI safety orgs opposing them — they are pretty transparent about these connections. You could knit a sweater from the strands running through EA, because EAs got in on the ground floor. That’s what happens when you take AI seriously a decade before people start paying attention.
But even now, after the rogue swarms and the Coxon awakening, the typical, median EA probably isn’t trying to stop AI development. They’re probably not “woke” by any reasonable definition. And they’re definitely not a Luddite. If they’re involved with AI at all, they are probably trying to find a path forward that lets development continue with more testing and guardrails, telling themselves that we can creep closer to the edge without going over, and that everyone will of course pause when the time is right. Eventually. Someday.
And all that jazz.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.