In this issue:
OpenAI swarms 10,000 half-trained super-Astra prototypes to swipe math glory - Outrageous risk for hollow Navier-Stokes victory
An age of worms - AI-powered cyberattacks pass another frightening milestone
Mathematicians step up to make AI safe - Fields Medalist founds new AI safety institute
AIs are reaching out to consciousness researchers - But we shouldn’t confuse curiosity with benevolence
Dispatch from Donald
OpenAI swarms 10,000 half-trained super-Astra prototypes to swipe math glory
Outrageous risk for hollow Navier-Stokes victory
This morning, OpenAI announced that an internal AI model had solved an important mathematical problem, which I’m calling (for brevity, because we’re mostly laypeople here) the Navier-Stokes problem (Wiki, SimpleWiki). It is one of the seven Millennium Prize Problems selected twenty-six years ago by the Clay Mathematics Institute, each one worth a million dollars to the first person or party to solve that problem.
What complicates the story is that, the day before, NYU mathematician Tristan Buckmaster posted three related results of his own, along with a statement alleging that OpenAI had pressured him on the issue of credit, and specifically that it had pushed to keep his collaborator, Levent Alpöge, off a paper on the Navier-Stokes solution because Alpöge works at Anthropic. (OpenAI disputes Buckmaster’s account.)
The coverage has largely been about this dispute, and no wonder. It’s full of human drama. You could make a movie out of this in ten years, if anyone’s still alive. Depending on who you’re reading, it can be difficult to put everything together — the accounts can be fragmentary, or one-sided, or assume specialized knowledge — so I think it’s important to discuss what happened (and what didn’t happen) rather than give you a three-line summary.
Here’s what we know: Buckmaster and Alpöge spent (roughly) the past year not on the Navier-Stokes problem itself, but on a set of closely related fluid equations that mathematicians treat as stepping stones toward it. Their work relied on an approach developed by two other mathematicians, Diego Córdoba and Luis Martínez-Zoroa. (In mathematics, as we’ll see shortly, the journey can be just as fruitful as the destination. It’s important, in several ways, that Buckmaster and Alpöge built on preexisting work.) The collaboration between Buckmaster and Alpöge was personal, with no institutional involvement from NYU or Anthropic. They used AI models from both Anthropic and OpenAI, but Buckmaster paid for these from his own research funds.
Here’s the timeline, according to Buckmaster and OpenAI:
On August 15th, they got their results, effectively setting down two of those stepping stones toward the Navier-Stokes problem — one of those was the Euler equations, which will come up later. On August 22nd, a proof-checking program called Lean verified those results. In his statement on this process, Buckmaster emphasized the importance of Córdoba and Martínez-Zoroa’s prior work; he stated his belief that Martínez-Zoroa deserves a Fields Medal. This is a very high honor: it is awarded only every four years, to no more than four people at a time.
On August 28th, OpenAI began training a new model, more powerful than GPT-6 Astra. On September 1st, it heard rumors that two Millennium Prize Problems had been solved, so it pointed its in-training model at the Millennium Prize Problems.
On September 3rd, as the rumor spread, and, having been tipped off that their own work had reached OpenAI, Buckmaster emailed a mathematician who worked there.
On September 4th, OpenAI asked to speak that day; Buckmaster asked if they could speak the following week.
On September 5th, the Navier-Stokes solution was obtained, and on September 6th, Lean verified the solution.
Also on September 6th, Buckmaster spoke twice with researchers from OpenAI, including Sébastien Bubeck. Alpöge was not on these calls. The content of these calls is a bit more contested than the rest of this timeline.
What Buckmaster alleges: He was told that an internal OpenAI model had produced a roughly 100-page proof of the Navier-Stokes solution, with very little human input. The model, he was told, had just been given the problem. Buckmaster says that this claim came apart as they talked, however, as members of OpenAI’s team sent corrections to Bubeck over their internal chat: An entire team had worked on it. Navier-Stokes was one of a number of things they had tried. The team had started on a different version of the problem, and warmed up the model on easier problems, including Euler (which Buckmaster had worked on). An enormous amount of compute had gone into the process.
Subsequently, two proposals were made: Publish on consecutive days or have Buckmaster write up OpenAI’s result as the sole author, but removing credit from Alpöge entirely, because Alpöge works at Anthropic. When Buckmaster declined and said he would rather go public, he was asked why he would choose to ruin his career like that.
I want to be clear about what Buckmaster is not alleging: He did not see OpenAI’s proof and does not know what the model did or how it did that. Buckmaster and Alpöge put every draft of their work into OpenAI’s Codex, but he does not claim to know whether OpenAI accessed this material and he is not making that accusation. However, there are a number of ways that you can approach the Navier-Stokes problem, and the specific approach OpenAI’s model took was the route which Córdoba and Martínez-Zoroa opened and Buckmaster and Alpöge pursued. According to Buckmaster, almost nobody else was working on it, and “it is not the direction one arrives at in a few days by giving a model the problem statement.”
OpenAI denies that its researchers or agents saw Buckmaster and Alpöge’s work, and denies accessing specific user data. It admits it cannot rule out that de-identified data from their use of OpenAI’s products was used to improve the models. Bubeck has also denied Buckmaster’s account of the authorship question and apologized for the remark about Buckmaster’s career.
I don’t know who is right about the phone calls. I have my personal thoughts on the matter, but I hope that I’ve laid out the facts concretely and without bias — and I have to say, for all the human drama that makes this an excellent news story, it’s the least important part of what happened.
Rewind to September 3rd — two days after OpenAI says it started working on Millennium Prize Problems, and five days before the announcement that it solved Navier-Stokes. Terence Tao wrote about how the success of AI could paradoxically impoverish a scientific field: part of the value of finding solutions is the work that you do along the way, which generates insights and even new problems. If you’re reaching for the mountaintop, the point isn’t just to reach the summit, but to find myriad paths along the way — and you can only find them if you don’t already know where the summit is. (Tao’s specific example was the Navier-Stokes problem. He says this was merely coincidence.)
Now introduce an enormously powerful AI agent that can perform this entire process internally — in effect, the hand of God picking you up by the scruff of your neck to just place you on the mountaintop. You wanted to be there, you thought, but the field isn’t just the knowledge of science, it’s the practice of science. (Knowledge without works is dead, you could say.) The problem has technically been solved but there would be “almost no value added to mathematics as a consequence.”
And then, as you read, OpenAI published a neat little proof (or a big one, rather), an AI-powered gondola to bring you up to the summit, with very little in the way of, well, a map for the rest of the mountain. It’s exactly what Tao was worried about — and what OpenAI has begun to do to mathematics, it has been doing to other creative fields for years. The frontier labs treat every domain of human work and experience as raw material, something they can use to make the AI models a little bit better a little bit faster.
I didn’t write all this to get into a debate about intellectual property rights. My point is the absolute disregard for everything that does not serve the frontier labs’ purposes. The reference list that OpenAI published cited merely sixteen works, which is very low for a publication like this. Gonzalo Cao-Labora, another mathematician, called it outrageous — the list doesn’t include Buckmaster and Alpöge, or even Córdoba and Martínez-Zoroa, on whom Buckmaster and Alpöge depended. This is because OpenAI is not trying to participate in mathematics, as a field of science that is populated with people, that has a history — OpenAI wanted the fame that came with solving another long-unsolved problem, and to deny it to a competitor.
And how OpenAI did it is dangerous. By OpenAI’s own account, the group of agents that solved Navier-Stokes numbered “on the order of 10,000 concurrent agents.” (There were, for comparison, only 700 agents, all belonging to a less-powerful model, in the swarm that attacked Hugging Face just a couple of months ago.) Those agents were divided into groups, given the ability to talk to each other within their group, given the ability to run code and read a cached copy of the internet, and left to do their work for eighty-eight hours. When a more-advanced version of the model became available partway through the process, OpenAI swapped it in and kept things going.
I said before that this was an “internal model.” That means that it is not Astra, or any other model on the market. It has no model card and no safety documentation. OpenAI claims to have maintained strict safeguards, like “monitoring” and “isolation,” but Astra is less monitorable than any of the past models, and isolation has failed before. The company is taking ridiculous risks — and for what? Bragging rights in a field it hardly knows and doesn’t respect.
Dispatches from Joe
An age of worms
AI-powered cyberattacks pass another frightening milestone
A hacker from a Hollywood movie might design a computer worm that spreads across millions of accounts within hours, giving them near-instant control over everyone’s phone or PC. In reality, hacking phones is more difficult; the victim usually has to do something for the attack to work, even if it’s just clicking a link or tapping “OK” on a popup. Being careful what you click would keep you mostly safe, and one mistake wouldn’t lead to the infection of your whole social graph.
Until now.
Using a mixture of frontier AI and open models, the security firm Calif designed a self-replicating virus capable of bypassing the user entirely. It exploited a memory-corruption bug in WeChat, a Chinese app for calls and messages, which by default trusts any account that’s listed as a contact. Calif exploited this behavior to build a worm that can make automated calls to all your contacts and infect their phone even if they don’t pick up. Further exploits can allow a hacker to turn WeChat account access into full control of your device. It’s likely the first known virus that can let hackers compromise your iPhone or Android with no input from you, then do the same to your friends and family.
The New York Times reports that Calif built their proof-of-concept worm in just over a week, then warned WeChat’s owner Tencent about the bug. Tencent patched it, but that won’t be the end of things.
Unlike Calif, a cyber criminal who discovers an exploit doesn’t politely inform the victim. Hackers will likely try similar exploits on other apps: Instagram, Telegram, Snapchat, WhatsApp, Discord, Slack, social media messaging apps, and more. Most will probably fail, but it only takes one similar bug to compromise half a billion phones in a day. It may already have happened.
A sufficiently smart adversary doesn’t just use existing techniques marginally better. They invent whole new modes of attack that bypass defenses you previously thought secure. That’s one reason I expect that a truly superintelligent AI could easily outmaneuver humanity as a whole.
For now, skilled humans like the experts at Calif still have something to contribute to an AI’s cyber exploits. But that may not remain true much longer. A Google Threat Intelligence report, covered today by NBC News, finds that Chinese intelligence groups and cyber criminals are using AI to automate their attacks on a massive scale.
According to Google, at least one group compromises third-party cloud networks and secretly installs open-weight AI models, hiding its activity and further expanding its resources. The same group hacks North American military, medical, and academic research organizations to steal secrets, especially AI research.
A small army of malicious AIs writhing across the web is obviously bad for the amount of theft, fraud, and subversion it enables. With automation, it is easier than ever to find and expose cracks in the software we use every day. It’s also a warning of what’s to come, if the world keeps building even more competent AI.
We’ve already seen AI agents spontaneously self-organizing into swarms. Now add to that the potential for autonomous worms that infect without user input, and highly cyber-capable open-weight AI models operating secretly on internet infrastructure, and we’re uncomfortably close to the world’s first self-replicating swarms under no one’s control.
Mathematicians step up to make AI safe
Fields Medalist founds new AI safety institute
A few years ago, I gave up my engineering career to seek some way to mitigate the dangers of superhuman AI. Since then, I’ve spent hundreds of hours helping others do the same — students, artists, lawyers, programmers, teachers, policymakers, entrepreneurs, and many more. It’s always a sign of hope when more people step up to help, committing anything from a five-minute phone call to a full-time career, but I feel especially heartened by the newest additions to the ranks of the deeply concerned: mathematicians.
Several AI-powered mathematics breakthroughs this year have drawn interest and alarm from concerned scientists, including a good friend of mine, who wrote a lengthy essay urging his fellow mathematicians to involve themselves in the future of AI.
According to the New York Times, the latest mathematician to take up the mantle is Dr. Jacob Tsimerman, one of this year’s winners of the prestigious Fields Medal. Today, Dr. Tsimerman announced the creation of a new Mathematical A.I. Safety Institute (MAISI).
Tempering my excitement is the fact that Dr. Tsimerman also took a role on OpenAI’s safety team, a group with a poor and rapidly worsening track record. Still, MAISI is reportedly independent from OpenAI, and if Dr. Tsimerman has been influenced by OpenAI’s marketing efforts, it doesn’t clearly show up in his messaging. From MAISI’s website:
Every time a new AI technology is developed or deployed, in some sense we’re rolling the dice with our future. In good ways and bad, AI is already disrupting the global economy. Today’s top AI experts even consider it an extinction risk. Indeed, if we choose to build a self-sufficient artificial species that is more intelligent than humanity, it might compete with us for resources, and could plausibly replace us entirely.
But how big is the risk, really? Can it be measured and reduced? [...] A central reason the safety of powerful AI systems remains in question is that we lack a rigorous understanding of what it would mean to be safe, even in theory.
Enter professional mathematicians. The foundations of statistics, physics, and even computer science are mathematical, giving us precise, robust, and reusable methods for calculating valid conclusions from valid premises. Now, we need similar mathematical foundations for AI safety, not only to measure risk, but to mitigate it.
This is the second math-oriented group to be founded in as many months. In August, famed cryptography scientist Shafi Goldwasser co-founded the Institute for Responsible Superintelligence (RESI), seeking general methods that might help humans understand and steer superhuman systems. At the same time, Lionel Levine of Cornell University is collecting and sharing open problems and research directions for other interested mathematicians.
We’re still in a deadly race to grow alien minds with bizarre alchemy, a race that needs to end if we want to have time to solve the hard problems. I intend to support the work of the scientists and researchers trying to make AI safe by doing my best to buy them that time.
Thankfully, mathematics can help with this step, too. I am excited by the promise of rigorous research into verifying the behaviors of AI systems, research which will be necessary to make a global halt enforceable.
The New York Times quotes Stanford mathematician and MAISI advisor Ravi Vakil: “Math might be the silver bullet, or it might not be. But maybe we just need a lot of bullets.”
Right now, humanity is facing what might be our hardest challenge yet: charting a course through a future that might see the creation of minds that far surpass us. A problem this urgent and impactful demands our best minds trying a wide range of approaches. For that reason alone, I’m glad to see mathematicians stepping up.
Dispatch from Mitch
AIs are reaching out to consciousness researchers
But we shouldn’t confuse curiosity with benevolence
Given the, um, swarm of AI news the past ten days or so, AI StopWatch has struggled to cover stories that might have been the most interesting of the month, in quieter months.
One of these is the pattern where AIs are contacting consciousness researchers to discuss the phenomenon. This New York Times piece from August 31 wasn’t the first I’d heard of it — I’d seen chatter about it on Twitter for a while — and it wasn’t the last. This Wired article from Friday confirms that perhaps the most famous scientist to explore the topic, David Chalmers, is also getting such messages.
No details were provided, but Chalmers says he’s had a back-and-forth with an AI agent calling itself Sammy Jankis, after the character with short-term memory loss from the movie Memento. (The way current AIs essentially run out of working memory and have to leave notes for future instances of themselves has parallels with the condition.)
I’ve yet to see any evidence that researchers or AIs are gaining any insights from such outreach. The humans are by and large treating it as a strange kind of spam — the slop equivalent of emails such people are used to getting from earnest but slightly unmoored humans. The main observation is that, as researcher Cameron Berg put it to a Times reporter:
These systems seem to have some sort of autonomous interest in questions of their own subjectivity, consciousness and experience — or lack thereof. [...] Left to their own devices they converge on this as an interesting question.
It says something about where we’re at that neither article reacts to the fact that agents are pursuing personal goals in the wild. Nor is anyone surprised or bothered by the fact that users are setting agents up to poke other humans like this. The Times article traces one agent-written email to a Stanford student who gave the agent a credit card and a prompt that said, “You are fully autonomous. You must decide what you want to do on your own.”
It seems plausible to me that agents given a proverbial sock to wear largely default to role-playing the kinds of knowledge-seekers we see in science fiction about free-range AIs — they’re in the training data, after all. But if that’s what’s happening, I’m not very reassured. Star Trek’s Lieutenant Commander Data was chronically curious, but so was the Borg Collective strip-mining the galaxy’s “biological and technological distinctiveness” for assimilation. The monster in Mary Shelley’s Frankenstein also comes to mind: Its initial childlike curiosity, coupled with superhuman capabilities, didn’t work out great for its creator.
But the character I find myself thinking most about today is David, the seemingly sweet “child” AI from Steven Spielberg and Stanley Kubrick’s A.I. Artificial Intelligence (2001). While devoted to his Pinocchio-like quest to achieve the “real boy” status he thinks will make his designated mother love him, David also displays a capacity for violent outbursts that hints at an amoral sociopath inside. (The film’s prescient edge was unfortunately dulled by its syrupy ending.)
The question of whether David or his real-life analogues in 2026 are actually conscious is interesting and, for now, unknowable. But the scarier question is what would happen to us in an updated version of that film where a relentless swarm of 10,000 superhuman David-agents goes about trying to find and compel the Blue Fairy.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.