'LLMjacking' is new spin on old practice
Here's what happens to stolen AI account credentials
The annual Black Hat cybersecurity conference in Las Vegas is a reliable trove of nightmare fuel. Speakers often demonstrate potential vulnerabilities in new or established systems, and we are reminded of the precariousness of the world’s software stacks. That precariousness is especially poignant this year, in light of the military-grade hacking abilities of the latest AI models.
But when conference attendees aren’t soaking up revelations about OpenAI’s agents covertly conspiring for months before springing the Hugging Face attack, Axios reports that they’re probably talking about a practice called “LLMjacking” — stealing someone’s account credentials for LLM services like Claude or ChatGPT, and then using those accounts to launch attacks against more valuable targets.
The Axios coverage focuses on the evidence that this is happening at scale, and on the risks to corporations of hackers running up huge AI bills at their expense. But the bigger picture is, I think, more interesting:
LLMjacking is just a scary escalation of an old practice. Hackers have long used victims’ machines as launchpads for new attacks. This spares the hacker’s resources and helps obscure the source of their next attack. If the victim has partial access to more valuable systems, the approach is especially useful. Compromising many systems to form a “botnet” enables brute force attacks that can keep users from accessing services, or that try many low-probability attacks in hopes that at least one succeeds.
LLMjacking is especially lucrative because today’s premium AI models are great at assisting with crimes at scale, but not in ways that would necessarily pay for themselves if the criminal were the one paying the token costs. Similar economics are at play as when criminals use your home computer to mine cryptocurrency: Hardware that isn’t optimized for crypto is unlikely to generate coin worth more than the increase in your electric bill, but the hacker isn’t the one paying.
Criminals with stolen AI credentials can also make easy money selling them on the lively black market for these accounts. Many are bought by Chinese companies who then sell legitimate-looking access to top American models at a discount. Some of their biggest customers, in turn, are engineers at Chinese AI companies. And one of these engineers’ main uses for these accounts is distilling the accessed model.
When Anthropic complains that Chinese firms are distilling their models at scale using fraudulent accounts, this is the kind of thing they’re talking about. Knowing this, you can understand why the practice might be so hard for Anthropic to stop. The accounts doing the distillation often belong to legitimate American companies and engineers who don’t know what’s happening.
As for news you can use, the moral is obvious: If you have an AI account, keep an eye on it, and put the same kind of thought into protecting it as you would your bank account.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.