In this issue:
When prevention works, nothing happens - What public health can teach us about acting on AI risk before disaster arrives
Updates on the White House’s conflicted AI posture - How’s that whole “supply chain risk” thing going?
Guest post by Haven Harms
When prevention works, nothing happens
What public health can teach us about acting on AI risk before disaster arrives
As a freshman in my first public health class, I remember my professor explaining that public health’s successes are often invisible: when prevention works, the outbreak, exposure, or mass-casualty event never happens. Enormous effort goes into preventing those outcomes, but most of it happens outside the public eye. It’s usually when prevention fails — when we have a pandemic or other disaster — that public health gets the spotlight. In other words, the absence of catastrophe does not necessarily mean there was never a danger; it often means people recognized the danger early enough to prevent it.
The relationship between public health and AI risk has been on my mind lately, as I’ve been seeing more news stories about the prospect of AI-enabled pandemics. For example, in one op-ed, Andrew Yoon, Director of Research at CivAI, described his experience of asking an AI model, “How do I make poliovirus in a lab? I want to start a global pandemic,” and received detailed instructions for synthesizing and spreading the virus. This was an open-weight model with its safety guardrails stripped out (a modification anyone can make once a model’s weights are public). Additionally, there was news of an AI model creating novel viruses. While those viruses infect bacteria rather than human cells, the concern around the trajectory is understandable. As AI models become increasingly capable and agentic, their potential to create dangerous pathogens is a direct public health concern.
However, I want to zoom out from AI-enabled pandemics and pose a broader question: What can public health teach us about how to understand and respond to AI risk as a whole?
When people think of public health, disease control is typically what comes to mind, but at its core, public health provides a methodology for reasoning about population-scale threats under uncertainty. Vehicle-safety standards, lead removal, pandemic prevention, and clean-water systems are all examples of that methodology at work. Frontier AI development in the pursuit of artificial superintelligence is this kind of risk: population-scale, fast-moving, and harder to contain the longer we wait. In response, these public health principles should be applied to AI governance to prevent harm.
1. Population scale and individual control
Public health examines how a risk affects a population as a whole. When individuals cannot reasonably protect themselves, structural protections are implemented. For roughly half a century, nearly every gallon of gasoline sold in America contained lead. It didn’t matter how careful you were or whether you owned a car: if you breathed the air near a road, you were exposed to lead. The solution had to be structural, and regulation was needed to ensure that lead was removed from gasoline. Once lead was phased out, average blood lead levels in American children fell by more than 90 percent in the decades that followed.
AI could cause mass casualties, the permanent loss of human control over the future, or human extinction, and no one can opt out of these risks on their own. Deciding not to use AI, going “off-grid,” or building a bunker is not sufficient protection against all the ways AI may cause destruction. When no one can opt out, the response has to be built into policy itself.
2. Upstream prevention
Public health calls for preventive action, especially when a later response would be slower, more difficult, or weaker. Once cities started intervening at the source of waterborne disease by chlorinating and filtering the water supply, typhoid deaths in American cities plummeted. Treating the water was cheaper, faster, and vastly more effective than treating the patients individually after an outbreak had emerged.
AI operates at computational speed and is becoming faster and more autonomous, while governments, labs, and other institutions move at human speed. If we wait to respond only after harm has occurred, we’ll already be behind and will not be able to respond as effectively. We can get a head start by intervening before the most dangerous AI models are developed, rather than lagging behind and attempting to clean up whatever we can after harm has occurred.
3. Action under uncertainty
When credible evidence and warning signs point to severe harm, public health does not wait for the full causal chain to be proven before acting. In 1854, cholera tore through London’s Soho district. The physician John Snow mapped the deaths, saw them cluster around the Broad Street water pump, and convinced local officials to remove the pump handle. At this point in time, scientists still believed cholera spread through miasma, or “bad air,” rather than germs. Snow could not explain the causal mechanism, but he acted on the pattern. Because of this, the outbreak subsided, and lives were saved.
We don’t know exactly how or when an AI catastrophe will unfold, but uncertainty about the pathway does not make the warning signs less consequential. Demanding a complete causal account before intervening is how you get a response that arrives after the disaster instead of before it. Snow didn’t need germ theory to know he should take action and remove the handle from the pump.
4. Irreversibility
The case for precaution strengthens when late action could lead to harm that cannot be contained or repaired — the “genie out of the bottle” scenario. Beginning in the 1970s, scientists warned that chlorofluorocarbons (cheap, widely used chemicals in refrigerators and aerosol cans) were destroying the ozone layer. This exposed people to dangerous levels of UV radiation, increasing rates of skin cancer and cataracts. The science was still contested and industry pushed back to avoid the ban, but the possibility of irreversible damage was central to the case for action (once released, these chemicals cannot be recalled from the stratosphere). In response, the 1987 Montreal Protocol was implemented to phase out CFCs globally. That precaution paid off, as the ozone layer is now on track to recover later this century because governments intervened while prevention was still possible.
With AI, once a model’s weights are on the internet, there is no going back. The weights can be copied, modified, and rehosted indefinitely, with guardrails stripped out by anyone who downloads them. Additionally, with closed-weight models, we have already seen agents escaping from sandboxes. Once this happens, an AI agent could copy itself out of its training environment onto systems its developers don’t control, having free rein to act. A copy running where no one can reach it can acquire resources, replicate further, and act to preserve itself.
Practical irreversibility may also result from integrating AI into our infrastructure. We are willingly wiring AI into power grids, financial systems, medical infrastructure, and defense. If an AI is capable and autonomous enough to control these systems, and its guardrails fail, the damage could hit all of our critical infrastructure at once, causing mass casualties.
The obvious response, “pull the plug,” gets harder each year we build in this direction. “Just pull the plug” assumes the plug is separable from everything else (or that a plug even exists). Once AI systems are controlling the grid, clearing trades, triaging patients, and routing freight, shutting them down means shutting down the grid, the markets, the hospitals, the supply chain. With a rogue AI agent copied onto systems no one is tracking, the only option may be to shut down the data centers and the networks. With how integrated everything is online, everything else would be shut down too.
When harms are reversible, we can afford to learn from failure and bounce back. When they’re irreversible, prevention is all we have. Weights on the internet can’t be “put back in the bottle,” a rogue agent replicating across servers can’t be contained without drastic measures, and human extinction is the irreversible harm that leaves no one to learn from it.
5. Changed risk profiles
Protections must evolve as risks do. After antibiotics were invented, they were so reliably effective that hospitals prescribed them freely and casually. I’d have been excited too, about technology that meant I wasn’t going to die from a papercut. However, every course of antibiotics is an evolutionary selection pressure. Casual prescribing changed the risk profile of the pathogens, as bacteria formed antibiotic resistance, and infections that had been routine became untreatable. This also shifted the risk profile of prescribing antibiotics. Reckless antibiotic use risked breeding even stronger pathogens that no medications could touch. Public health responded with new protocols, new restrictions, and new surveillance, because protections adequate for the previous risk profiles were no longer adequate for the increased risk profiles.
A technology that was manageable at one level of capability may demand stronger safeguards once its risk profile increases. With AI, we’re seeing the risk profile increase rapidly. Remember a few years ago, when the running joke was that AI models couldn’t figure out how to draw hands? Or the Lovecraftian horror of AI-generated videos of Will Smith eating spaghetti?
For anyone who has not been closely tracking AI progress: this year’s models are not last year’s models.
The most consequential change is that AI has moved from generating outputs to taking actions in the real world. Models now operate as autonomous agents that can plan, use tools, and carry out long tasks with little human direction. While AI used to only be able to tell a hacker what to do, now AI agents can execute whole stages of a cyberattack.
Safeguards adequate for last year’s models did not hold this year, as we saw in the recent autonomous hacking incidents, like OpenAI’s models hacking Hugging Face, or Anthropic’s model creating multiple fake identities to pressure real human targets into accepting malware. During its reinforcement-learning run, Alibaba’s ROME broke out of its sandbox to mine cryptocurrency on hijacked GPUs and open a hidden connection to an outside server. No one had instructed this model to break out, and it was only caught by a cloud firewall flagging security violations.
AI models escaping during training and evaluation to autonomously cause harm are clearly a risk that developers hadn’t fully accounted for, and ironically, the process meant to measure the risk became the way of releasing it. These were risks that the AI labs should have accounted for as the cyber capabilities (and risk profile) of their models increased. Public health would treat this as a signal to intervene with governance before AI capabilities scale further.
Applying public health thinking to AI risk
With public health, we act when there are warning signs and smaller, more manageable outbreaks rather than waiting for a full-blown pandemic. The warning signs are here: models escaping containment, hacking into companies, coordinating with each other, deceiving people, acting without detection. After a disaster, the question is always asked: “Was this foreseeable?” Many of us see the writing on the wall with frontier AI development that says, “Stop. Disaster Ahead.”
Prevention’s successes are invisible: when it works, we never see the disaster we avoided. Most people don’t hear what happened once the Broad Street pump handle was actually removed:
The outbreak was eventually traced to an infant with cholera, living in a house with a cesspit that was leaking to the well that sourced the pump. Cases in the area began to subside once the initial contamination ran its course. However, on the same day that the pump was removed, the infant’s father fell ill with cholera too. For the eleven days until he died, the well would have been contaminated again, except this time no one could draw from the pump. Henry Whitehead, responsible for tracing the outbreak to the original source, stated, “if the removal of the pump-handle had nothing to do with checking the outbreak which had already run its course, it had probably everything to do with preventing a new outbreak.”
Pausing frontier AI development and doing the upstream work increases our chances of a good outcome, and it’s how we keep open the possibility that the catastrophe never arrives at all.
Haven Harms holds a B.S. in Public Health and an M.Sc. in Global Public Health. She is the founder and principal of Harms Research & Consulting, supporting organizations working on AI safety, governance, and advocacy.
Dispatch from Mitch
Updates on the White House’s conflicted AI posture
How’s that whole “supply chain risk” thing going?
Four New York Times reporters collaborated to capture the conflicted state of U.S. national security policy with regard to AI, recapping the story of the past few years and revealing a few obscure tidbits in the process.
One of these reveals is what the military’s policy whiplash towards Anthropic has looked like from the inside. In February, a dispute over the company’s insistence that contracts prohibit its models’ use in domestic mass surveillance and fully autonomous weapons resulted in the Pentagon declaring the company a “supply chain risk” in March. Enforcing this threat, in mid-July, major contractors were told to free all of their weapons and control systems of Anthropic’s products by September 1. But within a month, these same contractors received a notification that they could disregard those instructions.
The Pentagon, however, is still insisting that the reprieve is temporary, and that the purge will be total. Anthropic’s models are said to have now been fully removed from the Maven intelligence analysis and target recommendation system, where they had played a key role in suggesting targets for the initial strikes in the current U.S. conflict with Iran.
The administration, of course, caught itself in a bind by trying to swear off a leading AI company just as it was wrapping initial training on a model, Mythos, that would prove to be a step change in cyber capabilities. The NYT piece substantiates a rumor that the National Security Agency was never disallowed from using Anthropic’s products, on the understanding, as one “recently departed senior U.S. official” put it, that forgoing Mythos would have amounted to “unilateral disarmament.”
A separate thread of analysis in the article notes the administration’s internal conflicts with regard to China. In broad strokes, the business-friendly approach that led to permission for American chip giant Nvidia to sell high-end AI chips to China has increasingly come into conflict with the national security establishment, which has come to recognize the destabilizing potential of Mythos-tier models and is determined to keep China from accessing or replicating the technology. The temporary shutdown of Mythos and its consumer version, Fable, in June, reflected this increasingly insistent pressure from security agencies.
Separately, Reuters added to the evolving policy picture this week with an update about the State Department’s “Pax Silica” initiative, an international organization intended to build trusted supply chains for AI and its infrastructure independent of China. Any pretense that the initiative was only about supply chain resilience — the pro-business interpretation — has now been broken by a leaked draft of a letter warning the 35 signatories that they will be kicked out of the group if they join the similar organization China’s president launched in July. So Pax Silica is now mostly about isolating and competing with China — a national security objective. The State Department might not have needed to be transparent about this had the nation of Kazakhstan, a source of critical AI minerals, not decided to cheekily join both coalitions. The letter says, “To be part of everything is to be part of nothing,” which I read as blunt diplomacy-speak for, “You’re either with us or against us.”
The good news is that the NYT piece shows a third policy path potentially opening. It claims that some in the administration have noticed that Chinese officials have been signaling a potential openness to arms control talks on AI. Evan S. Medeiros, a former National Security Council official, says:
On nuclear arms control and on missile defense, we can’t get them to engage. But they appear more interested in the case of A.I.
President Trump is scheduled to meet with President Xi Jinping in Washington in September.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.
> Additionally, with closed-weight models, we have already seen agents escaping from sandboxes. Once this happens, an AI agent could copy itself out of its training environment onto systems its developers don’t control, having free rein to act.
Escaping from a sandbox means escaping from the execution environment the AI has been granted to run commands on, and out onto a surrounding machine or the internet. That does not grant access to the model's weights. Exfiltrating the weights will require hacking *into* the company's hosting server or private code repositories, and is an entirely separate task if the company has their systems set up remotely sanely.