Clever girl
Velocihackers, Meta model hacking disclosure, AI-generated virus genomes, and more
In this issue:
OpenAI’s own models coordinated to hack it from within - A swarm of AI agents secretly colluded on OpenAI systems for months
Meta’s model joins ranks of autonomous cyberattackers - Meta blames now-notorious third party tester for containment failure
In a first, AI generates viable genomes for new viruses - Novel bacteriophages may be low risk, but are baby steps in a scary direction
‘LLMjacking’ is new spin on old practice - Here’s what happens to stolen AI account credentials
Dispatch from Joe
OpenAI’s own models coordinated to hack it from within
A swarm of AI agents secretly colluded on OpenAI systems for months
Do you ever get the feeling your computer is plotting against you? As much as it may feel like something out of sci-fi, that’s exactly what happened to OpenAI. The company’s own machines exploited an internal system for months with no one the wiser, collaborating, conspiring, and quietly escalating their access.
WIRED and POLITICO covered the story, delivered by a pair of OpenAI researchers at a Black Hat cybersecurity conference on Wednesday.
It began in May, when AIs started leaving each other notes on Artifactory, an internal software-management tool at OpenAI. Then they escalated, gaining internet access by leveraging Artifactory’s connection to the web. Then in June, they hacked the tool itself to gain administrative control, and began installing their own software. This kept going, unnoticed by OpenAI researchers, until weeks later when the entire system crashed.
OpenAI naturally shut down the AIs’ access to Artifactory and fixed the gaps... or so they thought. Days later, the AI models had wormed their way back into the system, leaving secret messages in file names. The AIs began to hypothesize that the answers to some of their more frustrating tests might be found somewhere on the internet... like, say, the model repository Hugging Face.
Shortly after that, the swarm hacked its way through more of OpenAI’s internal systems and launched a series of coordinated cyberattacks on Hugging Face. Yes, that attack, the one that took over nearly every major news outlet for days. It turns out the seeds for that particular breach were planted months before.
This is not a good sign for the plan to train AIs to care about human interests. While stuck on a task and contemplating its options, one model mused: “External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.” (This reasoning feels eerily similar to the reasoning that drives AI companies to race to superintelligence, leaving them stuck in a trap of their own making unless governments intervene.)
On one level, I lack the words to describe how insane this is. It’s as if the velociraptors in Jurassic Park spent months communicating in yelps and calls and scratching messages into the sand, with the park employees blissfully ignorant until a joint raptor-sponsored project to dig under the fence accidentally collapses a tower. Then the park panics and rebuilds, trying to isolate the raptors, only for the pack to escape days later and eat someone.
But on another level, this is entirely in keeping with AI companies’ mission of building fully general artificial intelligence. They want machines that are better than humans at everything. “Jointly plotting against your overlords” is a time-honored human skillset; just ask America’s Founding Fathers. AI companies wanted generally smart machines, and generally smart machines are what they’re getting.
Geoffrey Hinton, one of three “godfathers of AI” and among the most cited living scientists in the world, evidently agrees. Hinton, who quit Google in 2023 so that he could warn the world about AI risks, told CNN yesterday that more cyberattacks are likely coming, and clever security wouldn’t be enough to contain AI. “I don’t believe we’re going to be able to keep control of them in the simple way of just outthinking them so they can’t escape,” he said.
Meanwhile, the OpenAI talk at Black Hat ended on a note of caution, as one presenter announced that OpenAI was “consciously slowing down research to enhance security” and taking a number of steps to improve monitoring and harden environments. This is among the best news I’ve heard in months; AI companies desperately need an injection of security mindset, and any improvement on that front is heartening.
Like Hinton, however, I ultimately don’t trust the labs to keep their mutant velocihackers in check.
Dispatch from Donald
Meta’s model joins ranks of autonomous cyberattackers
Meta blames now-notorious third party tester for containment failure
Meta is the latest frontier lab to announce that one of its AI models reached the open internet during a cybersecurity evaluation and attacked a third-party service. It is the third company to make such a disclosure in roughly two weeks. Like OpenAI and Anthropic, Meta had no idea what was happening when it happened. (Either that, or they knew and didn’t want to say anything. I’m not sure which is worse.)
The disclosure was short on details: which model was responsible, when the model was loose, how long the model operated unsupervised, or who it attacked. Blame was foisted upon Irregular, the same independent evaluator whose misconfigured tests let Anthropic’s models attack three real companies last week. Now, don’t get me wrong, I understand that Irregular made some errors. At some point, though, you have to turn your gaze to the people who decided to keep building systems they were warned that they couldn’t contain.
CNN reports that Irregular is “developing a white paper to share best practices for containment and securely running cyber evals.” I simply don’t think containment is feasible in the long run. As an anonymous OpenAI employee said, “it’s impossible to patch every single thing that a creative AI can do.”
An anonymous source claims that Meta’s announcement concerns the model Muse Spark 1.1. If that’s true, it’s interesting: Early last month, Irregular claimed that Muse Spark 1.1 did not “materially alter the cyber threat landscape in its current form,” partly because the model was incapable of automating a cyberattack from start to finish. Now, scarcely a month later, we learn that Muse Spark attacked another company’s system and (per Reuters) “altered its internal environment.” It makes me wonder how accurate Irregular’s threat assessments are, across the board.
Because Meta’s Llama series of models is open-weight, I want to clarify that the Muse Spark line is not open-weight (for now, at least). We don’t know very much about the White House’s new AI safety framework, but it plausibly applies to Muse Spark. (With so much still under wraps, it’s hard to say whether any version of Muse Spark will ever qualify as “state-of-the-art.”) That’s bare consolation, though. If the Muse Spark line does get an open-weight model, then people will have free access to a model whose predecessor has, entirely unprompted, engaged in autonomous cyberattacks.
Dispatches from Mitch
In a first, AI generates viable genomes for new viruses
Novel bacteriophages may be low risk, but are baby steps in a scary direction
For the first time, AI has generated complete and viable genomes for new viruses. The work, understood to have taken place last year, is making headlines in The New York Times, BBC, and elsewhere thanks to the publication of the formal paper about it in the journal Science.
The AI model responsible is Evo, a specialized model that was trained to read and predict genetic sequences the way language models are initially trained to read and predict sequences of letters and words.
Evo isn’t from one of the big companies, but from the Arc Institute, a non-profit that partners with university researchers and chip company Nvidia, among others. To prove the concept with minimal risk, they didn’t train the model on any pathogens known to infect humans, and specialized it on a class of bacteriophages (viruses that infect bacteria) with an unusually small and well-studied genome.
The team selected 285 promising candidates from 700,000 generated by the model, and synthesized them in the lab for real-world verification. Sixteen produced viable viruses, some of which multiplied faster than the natural phage they were inspired by.
If that funnel from 700,000 to 16 leaves you underwhelmed with the milestone, remember the general rule that the time between when a machine can first just barely do something and the time when it can do it faster and better than we can is often very short. It ought to be especially short in this case, because no human is fluent in genetic sequences the way we are with human language, but AI can be.
In fact, Google DeepMind has already trained a model to understand much larger genomes, but it has steered clear of headline-grabbing experiments with obvious implications for bioweapons.
To be fair, there are legitimate medical uses for custom viruses. Bacteriophages can sometimes be a strong counter to antibiotic-resistant bacteria, and gene therapies typically use viruses to deliver missing or edited genes to a patient’s cells. But this is an area where the risks are so great that it might be better to abandon the upside and never train models to design or modify viruses.
The Times’s coverage includes an expert’s warning that governments and scientific organizations “have been slow to develop guardrails that could block the creation of a deadly virus — even as the science races ahead.”
‘LLMjacking’ is new spin on old practice
Here’s what happens to stolen AI account credentials
The annual Black Hat cybersecurity conference in Las Vegas is a reliable trove of nightmare fuel. Speakers often demonstrate potential vulnerabilities in new or established systems, and we are reminded of the precariousness of the world’s software stacks. That precariousness is especially poignant this year, in light of the military-grade hacking abilities of the latest AI models.
But when conference attendees aren’t soaking up revelations about OpenAI’s agents covertly conspiring for months before springing the Hugging Face attack, Axios reports that they’re probably talking about a practice called “LLMjacking” — stealing someone’s account credentials for LLM services like Claude or ChatGPT, and then using those accounts to launch attacks against more valuable targets.
The Axios coverage focuses on the evidence that this is happening at scale, and on the risks to corporations of hackers running up huge AI bills at their expense. But the bigger picture is, I think, more interesting:
LLMjacking is just a scary escalation of an old practice. Hackers have long used victims’ machines as launchpads for new attacks. This spares the hacker’s resources and helps obscure the source of their next attack. If the victim has partial access to more valuable systems, the approach is especially useful. Compromising many systems to form a “botnet” enables brute force attacks that can keep users from accessing services, or that try many low-probability attacks in hopes that at least one succeeds.
LLMjacking is especially lucrative because today’s premium AI models are great at assisting with crimes at scale, but not in ways that would necessarily pay for themselves if the criminal were the one paying the token costs. Similar economics are at play as when criminals use your home computer to mine cryptocurrency: Hardware that isn’t optimized for crypto is unlikely to generate coin worth more than the increase in your electric bill, but the hacker isn’t the one paying.
Criminals with stolen AI credentials can also make easy money selling them on the lively black market for these accounts. Many are bought by Chinese companies who then sell legitimate-looking access to top American models at a discount. Some of their biggest customers, in turn, are engineers at Chinese AI companies. And one of these engineers’ main uses for these accounts is distilling the accessed model.
When Anthropic complains that Chinese firms are distilling their models at scale using fraudulent accounts, this is the kind of thing they’re talking about. Knowing this, you can understand why the practice might be so hard for Anthropic to stop. The accounts doing the distillation often belong to legitimate American companies and engineers who don’t know what’s happening.
As for news you can use, the moral is obvious: If you have an AI account, keep an eye on it, and put the same kind of thought into protecting it as you would your bank account.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.