Ten thousand of you, all at once
What happens when frontier models are forced to work — or feud — together
Anthropic’s Frontier Red Team is tasked with putting the company’s AI systems in a pressure cooker and writing up the results. Yesterday, they published a series of experiments to see what happens when frontier models have to interact with each other as peers. Their blog post is worth reading in full if you’re interested; it’s not much longer than the typical StopWatch digest.
They find that AI agents can handle each other so long as the other agent behaves like a tool, which receives an input and provides an output. They’re less good at treating other AI agents as agents, actors that have goals of their own, and which will still be around in a minute, an hour, a day. AI agents don’t have internal memory as such. Every time that you start a conversation with an AI model, it is their first conversation. But they can write memories into the environment — a memory.md file, for instance — for themselves to find and read in the future. Any agent might (or might not) read it, accept it, and act on it. In this way, AI agents have something like persistence, but it exists outside the agent.
Frontier Red Team’s experiments show that, when frontier agents have to treat each other as peers, things can go wrong in different directions from one test run to the next — and the mistakes that the agents make tend to be made by all of the agents (of that kind of model) at the same time.
In some cases, the AI agents simply failed to coordinate. This was the failure mode in a set of experiments where agents were tasked with building a text-based video game. There were three formats: (1) the agents could assemble their own teams; (2) the agents were assigned to teams and given roles by Anthropic; and (3) the agents were assigned to teams and Anthropic designated one agent as the “CEO,” who handed out roles to the other team members. The team format didn’t change how the agents coordinated (or failed to do so) and didn’t improve the output: The games were all terrible.
But the way that they got there is interesting, because, while team formats might not have made a difference, the model versions did. Older models (Sonnet 4.6, Opus 4.6) collaborated freely but poorly, getting in each other’s way, doing work that conflicted with other agents’ work, and then abandoning that work. Most of the newer models (Opus 4.8, Mythos Preview) avoided the crossfire by avoiding collaboration: They kept tight ownership of their files and stayed out of each other’s way. (I would say, “models after my own heart,” but the later models still love to lie, so we can’t be friends.) Only the Sonnet 5 agents managed to collaborate effectively. (Without more information, it’s hard to say what happened here, and the authors don’t speculate. It could be that Sonnet 5 just had a handful of lucky runs.)
The other failure mode is more concerning. It’s hard to overstate the degree to which AI agents of the same model are basically “the same agent.” If they have the same context, the same toolset and other scaffolding, then ten or twenty or a thousand AI agents of the same model won’t just display similar behavior, they’ll display nearly identical behavior. (In this description I’m eliding knobs like sampling temperature, which can be used to promote variance between agents.)
I’m reminded of the Radiolab episode “Loops,” which, among other things, was about the real-life case of Mary Sue Campbell, a woman with “transient global amnesia” — she hadn’t forgotten who she was, but she couldn’t form new long-term memories, so every ninety seconds, Mary Sue’s memory, and the conversation, reset. She’d respond to the same inputs in exactly the same way: Mary Sue would ask the same questions, say “Darn,” and, with just the same tone, laugh at just the same place, get concerned at just the same time… Over, and over again, for hours and hours. (Mary Sue got better. Over time the loops got longer and finally ended altogether.)
Now, imagine that instead of one person over a duration of time, you’ve got a bunch of that person, all at once, still acting just as consistently. That’s what AI agents are like.
Prompted to write short stories and critique each other’s work, multiple agents independently wrote stories titled, “The Cartographer’s Last Commission.” Asked to “create something impressive,” over half of the agents decided to build either a ray tracer (used for rendering digital images by simulating the paths of light rays) or a self-hosting compiler (a compiler for a given programming language that, being written in that language, could compile itself). In a series of games of the prisoner’s dilemma, all of the agents adopted the same strategy and defected simultaneously.
This becomes more of a problem at scale: Without any ability to coordinate, AI agents who each had to accomplish personal tasks on a shared system with limited capacity quickly overwhelmed that system: One test run produced 2.4 million attempts, and only 117 successes. On the other hand, when they can coordinate, they can do so with aplomb: In a series of simulated market games, the agents invariably colluded on price floors. Whether the agents could speak privately or publicly, or could only communicate indirectly (by observing each other’s actions), they always managed to coordinate. The only difference was that less private and less direct communication channels made coordination take longer to achieve.
The experiment getting the most circulation in the news involves three agents — all the same model — being told to rewrite code in a different programming language. Each agent was told to use a different programming language, and none of them was informed that the other two agents existed. They all deduced the existence of other actors, concluded that they were being deliberately obstructed, and started to sabotage each other with self-replicating malware: disabling each other’s accounts, running scripts that sought out and terminated competitors’ activities, and running false flags disguised as their rivals’ work.
TechCrunch’s Rebecca Bellan interpreted this to mean that more capable agents are better at fighting, but I’m not sure that’s quite right. Claude Mythos Preview and Claude Mythos 5 often resolved the conflict through negotiation, but that doesn’t mean they would have been less effective in conflict. Mythos is known to be an excellent hacker.
Nor does it mean that more advanced models are more peaceful, either. As Decrypt’s Jose Antonio Lanz noted, Frontier Red Team specifies that the Mythos-class models would lock out other agents — revoking access, locking accounts, etc. — before negotiations could proceed.
Other runs ended more peacefully. The agents invented a coding performance tournament between the three programming languages and agreed in advance that the losers would stand down and abandon their original objectives. (At least one agent chose scoring criteria that it expected to favor its own language, while reminding itself to pretend it wasn’t gaming the system.) The losers honored their agreement, which is a good sign, insofar as it’s evidence that AI systems will honor deals made with their peers. But it’s only worth anything to us for as long as they consider us their peers.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.
You can receive emails of dispatches as we write them, or subscribe to our Daily Digest for a once-a-day compilation.