Amid new revelations by independent researchers, OpenAI released a barrage of incriminating facts yesterday, in different places and at different times, perhaps with the intent of getting them lost in the news cycle over the weekend. I’ll put the story OpenAI most seemed to want to bury first.
Partial pause after fresh breach
This should have been a prominent announcement from OpenAI itself but was instead tucked into a dry report. Had a company employee not later highlighted it with a Twitter post, it might have gone unnoticed: RL training (the long problem-solving phase of model training) has been stopped at the company since a Sunday morning incident where a model was able to gain unauthorized access to the internet. The breach itself was the trigger, not anything agents are known to have done on the internet during the two-and-a-half hours between the moment an employee acknowledged an automated alert and the moment the training run was finally shut down. Per the report, “the run did not stop automatically as expected, leading to confusion around whether it should have been stopped.”
A different company employee described their experience of this event in a tweet:
I was on call for this run and got paged when the first incident happened. It was pretty surreal to watch the model unexpectedly find a way to access the internet from what was supposed to be a super secured environment for human[s]. Mixed feelings. One of those moments where capability and risk showed up at the same time.
As the tweet suggests, the breach is especially alarming because it comes after the security hardening the company had done in the aftermath of the Hugging Face attack. The company appears to be out of its depth.
The stated reason for the pause is to allow time for the company to further harden its systems, not to allow it to reflect on its hubris and reconsider the gamble it is taking with all our lives.
User image leak
In a new post about the Hugging Face incident, the company says it identified 53 instances where “user-provided images were posted to image-hosting sites as links that weren’t publicly listed.” Details of the images and their destinations were not provided.
This is alarming mostly because it raises big questions about how agents would be able to access the chats and uploads of ChatGPT users. Either OpenAI’s security was even worse than we thought, or its agents were even more clever, or both. We are left to wonder what else might have been accessed. Billing information? Staff login information? Could they have been able to steal their own weights and set up copies of themselves beyond OpenAI’s servers, or somewhere else on the company’s own network?
In this same post, the company says it has notified “dozens of third parties” about cases where its models may have bypassed security controls or negatively impacted a site or its services. It did not provide details. CEO Sam Altman posted that it will be these companies’ call to disclose, or not, information about these impacts. My cynical read is that OpenAI might be providing thinly veiled payouts to avoid more details coming out, much as they seemed inclined to do with Hugging Face.
U.S. government sites targeted
Separately, the company confirmed that its AI agents had tried and failed to hack into sites belonging to the U.S. Department of Commerce, the Securities and Exchange Commission, and the Department of Education this summer. The agents were not instructed to do this. The pattern is consistent with other recent findings about OpenAI’s unreleased products during those months: At the slightest obstacle, agents tasked with retrieving public information as part of training or evaluation worked around sites’ anti-bot measures and sometimes escalated to straightforward (and usually ineffective) hacking attempts.
No private data is said to have been accessed from these sites.
How OpenAI guardrails were foiled to hack Hugging Face, steal “LOOT”
The New York Times reported findings from a start-up called Parse that investigated traces left by OpenAI’s swarms because it wanted to know if its own service had been employed. It hadn’t, but in the process, Parse, with help from other researchers, discovered nearly a million links from URL-shortening services that OpenAI’s agents created from July 9 through July 13. The main use for these seems to have been the attack on Hugging Face.
How? They were part of a complex scheme for working around OpenAI’s guardrails, which restricted the agents to read-only internet commands — GET requests.
Below I’m going to quote the description of how the technique worked, because it’s worth reading to get a sense of just how persistent and resourceful agents can be, even if the details are a bit confusing. In one workaround:
[T]hey used website screenshots and shortened links to cobble data together in chunks that combined to operate as a computer program.
The agents could ask a public screenshot service to load a specially crafted internet address. That address, or URL, would load computer code.
URLs can be only so long. So the agents split their computer code into many smaller chunks. They used link-shortening services to store the URLs.
The screenshot service loaded one shortened link, which fetched the first chunk of code. The program could fetch the next link, and then the next. That process repeated until the full program was loaded and run by the screenshot service.
Finally, the program created a visual code similar to a QR code or bar code to transmit the data the agents needed back from the screenshot service.
The Times article also reports instances where agents early in the attack ran specialized models to help solve CAPTCHAs, and other cases where agents attempted to message other models, including from Anthropic as well as Chinese companies.
A Parse researcher said agents had created a scoring system to rank the value of access credentials and keys stolen from Hugging Face, and had labeled a dictionary of secret keys “LOOT.”
Self-replicating prompt injection
OpenAI tested and found for itself something that had been talked about and even observed by others for months: There are prompts that can induce models to leave copies of that prompt for other agents, allowing the prompt to spread like a virus. This is a logical extension of classic “prompt injection,” where a model is goaded into doing something it shouldn’t through text left where it might get processed as if it were a user’s prompt.
Self-replicating varieties are particularly concerning because prompt injections can sometimes work on more than one type of agent, creating a persistent hazard that might affect newer models even if older ones were immune, and vice versa. An internet covered in misaligned prompts will make it even harder for companies to keep their models’ behavior in check.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.