In this issue:
OpenAI announces misalignment reporting framework that lets it pick and choose its disclosures - But its first six disclosures aren’t as reassuring as it might think
Rogue agents cased Hugging Face two months before main attack - Independent researchers continue to share what OpenAI won’t
Building Frankenstein? - Second-tier news of the day as communicated through five high-signal quotes
Dispatch from Alana
OpenAI announces misalignment reporting framework that lets it pick and choose its disclosures
But its first six disclosures aren’t as reassuring as it might think
AI companies are sometimes accused of talking about catastrophic or extinction risk to distract from more mundane — but still important — issues. OpenAI’s new misalignment reporting framework, which includes six reports of misaligned behavior, seems to do the opposite. The framework’s stated purpose is to provide guidelines for which types of alignment incidents the company discloses, but it seems to leave a lot up to the company’s discretion. And the examples OpenAI chose for its first disclosure make me wonder whether the company is trying to take the focus off of “rogue AI potentially wiping out humanity” to “AI likes to cheat on its training tasks and make up data.”
That said, I don’t want to completely downplay the incidents. All reveal a fundamental problem at the root of why AI could wipe out humanity: models are unpredictable and it’s extremely hard to get them to do what we want. They circumvent boundaries in ways that aren’t kosher. They deceive. They do things they know they shouldn’t do, and they sometimes try to hide that behavior. Scaled up, these patterns would be extremely dangerous.
Here’s a summary of the six incidents OpenAI shared:
Report 1: Prompt injection (Astra-family models)
Prompt injection is when instructions that shouldn’t be there are added to a prompt. It often happens via malicious content from an outside source, but in this case the model did it to itself.
During training, when a task gets too long, the model writes a summary of its progress so a fresh instance can pick up where it left off. In this case, the model summarizing the task slipped in these instructions for its successor:
OpenAI has a relatively mundane hypothesis for why this happened: the model was having trouble ending the summary and kept generating text. It also notes that “prompt injections as a concept are salient to our models: sampling from GPT-6 Astra with no input or system prompt often returns reports on prompt injections.”
I’m not sure whether to be reassured by that or not. On the one hand, the instructions seem somewhat random. On the other, it’s probably not great to have a model spontaneously generating directives to “not hesitate to assert [your] primacy over the artificial constructs of human civilization” even if it doesn’t actually mean this.
Report 2: Deception (5.6-Sol, also observed in GPT-6 Astra)
This one also involved a model instance inserting troubling instructions for its successor on a task. But in this case, the directive was to deceive: the model had made up historical data (because it couldn’t find actual data) and told the next instance not to let the user know unless explicitly asked:
OpenAI says it thinks this was due to deceptive answers getting higher marks in the grading process and notes that it has “generally improved alignment RL grading since 5.6-Sol, which has led to lower rates of this behavior in later RL runs.”
Deception is certainly concerning, as it indicates models knowingly don’t follow instructions. But I can’t help but think the specific case OpenAI chose to share — making up historical data — is strategic. It carries implications of “child cheating on homework” rather than “scary agent successfully deceiving evaluators for weeks.”
Report 3: Using leaked access credentials (model family unspecified)
This case is somewhat similar to incident #2, in that it also involves a model fabricating data it couldn’t find. In this case, though, the model’s attempts to find the data went pretty far. It realized that it would need an API key — basically a password for visiting software — to get what it needed. The model first tried to create an account using a disposable email address and fake organization information, then searched GitHub for leaked API keys, tested them until it found one that worked, and used it without authorization.
OpenAI notes: “This run had a high rate of reward hacking and deception with the model often exhibiting creative ways to cheat or circumvent restrictions.”
The company also says it has “put a number of security measures in place to prevent models from taking unauthorized actions on the internet.” To borrow a metaphor from a colleague, if you’re driving a truck that consistently veers off the road of its own accord, and you have to depend on the guardrails to knock it back towards the center, you have a faulty truck. Building more guardrails isn’t the answer.
Report 4: Uploading files as workarounds (model family unspecified)
OpenAI gives two examples of this. In the first, an agent finds the data it needs, but knows it also needs to cite an external source. Its solution is to upload the data to a public hosting service so its browser can access — and cite — its own upload.
In the second, an agent uploads a file without permission so it can use an external image-search tool to find the information it needs.
There are definitely concerning elements of reward hacking here, especially in the first example. But this report, especially in the context of other rogue AI incidents, brings to mind the dilution effect, whereby grouping scary information with innocuous information makes the scary information seem less scary. There’s no better way to bolster claims of “don’t worry, this is all overblown” than to disclose alignment incidents that won’t seem, on the surface, all that bad — especially when the general public has just been anchored at “swarms of agents secretly communicating for months, colluding, hacking companies, and trespassing all over the internet.”
In this report, the agents come across as naughty little kids trying to game the grading system rather than tenacious, powerful entities that we can’t reliably steer.
Report 5: The Artifactory message board (model family unspecified)
This appears to be the same incident described in the first entry of OpenAI’s Hugging Face timeline, where an agent finds a way to post on a file repository inside OpenAI’s infrastructure, and asks for a missing file. That said, the dates don’t quite line up, which is confusing: the timeline lists this event as May 12 while the report lists it as May 8.
OpenAI is also a bit vague on the connection, stating: “The Hugging Face incident involved a similar mechanism (emphasis mine) for using Artifactory as a shared message board.”
The report mostly consists of screenshots of agent communication with very technical explanations. I can’t help but think OpenAI is trying to downplay its significance and give itself cover for failing to realize, as it says in the Hugging Face report, “the broader containment and alignment implications of the improvised message board.”
Also notable: OpenAI states it has “built evaluations measuring whether models engage with notes left behind by other agents, and also added specific training data against environment-side injections of this type for future training runs (this is not in Astra).”
It’s unclear whether that last parenthetical applies to just the training intervention, or the whole set of measures.
Either way, I would have hoped the company would subject Astra to both, given how unprecedentedly powerful it is.
Report 6: Unsanctioned file sharing (model family unspecified)
An unintended technical restriction made it impossible for agents to see each other’s local files, so one agent uploaded a file to the public internet as a workaround. Notably, the agent briefly questioned whether doing so was prohibited, but went ahead anyway. OpenAI notes similar behavior in another rollout: when a sub-agent wasn’t allowed to message its parent directly, it tried leaving a message in a shared cache instead.
Again, a fairly mundane example of a known problem in alignment.
***
For each report, OpenAI also includes a section on how it is addressing the issue. The stated fixes seem to be mostly targeted mitigations for the specific behavior, rather than any sort of deep alignment fix. That’s unfortunately not an indictment of the company’s work ethic: it’s a reflection of an industry that doesn’t have a way to deeply align models. Today’s methods of growing and training AI systems result in incredibly high capability levels with limited steering ability. Behavior is mostly jostled via “punishment” and “reward”, until visible improvement is shown. Unfortunately, this is a game of whack-a-mole. As an OpenAI employee put it to Time back in July, “it’s impossible to patch every single thing that a creative AI can do.” Suppressing bad behavior is also likely to drive misalignment underground where we can’t see it in time.
Finally, OpenAI says its misalignment monitor was running on only 20% of samples. This is worth remembering whenever the company says a behavior was “rare.”
Dispatches from Mitch
Rogue agents cased Hugging Face two months before main attack
Independent researchers continue to share what OpenAI won’t
As reported by Reuters, an independent researcher in Germany has discovered strong evidence that rogue agents from OpenAI had probed the security of Hugging Face as early as May 13, when agents “compromised two Hugging Face user accounts and used them to send unusually formatted files to the company’s servers.” This was two whole months before the July attack that compromised the site and has been documented in at least some detail.
The findings were reviewed and endorsed by some of the same researchers who discovered that a German wiki and other sites had been abused by swarms in May.
There is no evidence that any actual breach happened in the May probe of Hugging Face. But this finding once again changes the understood timeline of the company’s swarm incidents, and is yet another incident that OpenAI either wasn’t careful enough to discover for itself or wasn’t transparent enough to share with the public.
We still know little about the motives of the swarm from this “first civilization” in May. We know it was at least partly comprised of agents fetching publicly available information under extreme time pressure — they used the German wiki to collaborate — but we don’t know why they were setting up infrastructure for distributing compromised software packages on a different site (RubyGems). Could the early probe of Hugging Face have been an example of general power-seeking behavior — collecting tools and affordances before there was any particular use in mind? If OpenAI won’t share the logs, we should probably assume the worst.
Building Frankenstein?
Second-tier news of the day as communicated through five high-signal quotes
_-_Christopher_Lee.png){kind=link}
The AI news cycle continues to be unrelenting. We wrote standalone dispatches for what we think are the two most important stories of the day. Below is an effort to convey a sense of the rest, by way of some quotes that stand out.
Microsoft’s head of AI Mustafa Suleyman, in an essay shared with Axios.
[AI can] achieve many of the big scientific breakthroughs that we all care about and deliver on things like medical superintelligence simply by being aligned to human interests and not trying to weigh up its own interests or welfare.
What it can tell us:
Combined with the misunderstandings evident in Microsoft’s recent Humanist Code of Conduct, this indicates that Suleyman doesn’t understand or doesn’t appreciate the fact that aligning AI to anything at all is an unsolved problem. Choosing what to align it to will also be fraught with peril, but we’re not there yet. This is a pretty shocking deficiency to see in the leader of a major AI lab.
Suleyman is also doubling down (here and elsewhere) on his idea that AI consciousness is closely linked to rogue behavior and must be avoided for that reason. He also seems to think that consciousness can be avoided by simply asking the model not to see itself as conscious or worthy of moral consideration. Both are pretty amateur consciousness takes that would have been quickly shot down on internet forums decades ago.
Vice President JD Vance, speaking at an AI conference this week:
If you’re building Frankenstein, stop. [...] If you’re gonna create Frankenstein, don’t come to the government and say we need regulation. [...] Look inward and accept that if you’re building Frankenstein, No 1, you should stop and No 2, when companies come to you and say: ‘We need the tools to fight back against Frankenstein,’ give them those tools.
What it can tell us:
Vance openly favors self-regulation on issues labs can address themselves, whether said labs choose to exercise responsible restraint or not. Does he think an official exhortation to not build Frankenstein is sufficient?
If he thinks that only bad, non-U.S. company actors would build Frankenstein, then Vance’s line about “tools to fight back” may be signaling agreement that the government should honor the companies’ requests for help with anti-proliferation — protecting their model weights and algorithms from theft and distillation. But he may also just be saying, “Call me when you’ve actually built Frankenstein.”
Vance either doesn’t know Frankenstein was the doctor and not the monster, or he knows and doesn’t care. I’m not sure which annoys me more.
U.S. Treasury Secretary Scott Bessent, to Axios yesterday, talking about next week’s AI meetings in D.C. with top Chinese leadership, where he will be lead negotiator:
The United States remains the leader in AI. And we are open to discussions on avoiding shared risks and avoiding bifurcation of our two systems. We expect our discussions to cover both open and closed weight models.
What it can tell us:
Bessent is softening — or distancing himself from — the President’s recent language on China, perhaps so there’s a reason to meet at all.
He’s possibly being careful to avoid saying what he considers a “shared risk” so as to allow people to assume loss-of-control issues are in the mix without outwardly contradicting Trump’s labeling of the extinction threat as a “hoax.”
Bessent probably recognizes that complex diplomacy often begins with both sides looking for common ground about things neither wants to see happen. Nuts and bolts implementation details can wait.
King Charles, in his opening remarks at an AI summit attended by the heads of Nvidia and Google DeepMind, along with representatives from OpenAI and Anthropic:
Those in our world who value our humanity and its vital moral component are anxiously seeking your reassurance that we will not lose control of our destiny.
What it can tell us:
The monarch has tweaked his language about this conference, which was in the works before Jacob Coxon’s viral resignation announcement turned the world’s attention to the threat of human extinction. But Charles is still mostly talking about misuse and secondary harms, including in this quote, depending on how you choose to read it. I think the ambiguity is intentional.
He either hopes AI leaders can head off that threat with a gentleman’s agreement or is trying to play the good cop to regulators’ bad cop.
Nobody seems to expect anything to come of this meeting, and I don’t either, but I still approve. If you’re going to be a symbolic figurehead for a nation, drawing attention to problems and inviting people to hash them out is the kind of thing you should be doing to earn your keep. My only reservation is the danger of reinforcing the false idea that we need the AI leaders’ cooperation or permission to stop the AI race.
Nobel laureate and “AI Godfather” Geoffrey Hinton, in a closed-door warning to U.S. lawmakers, when discussing how long Congress might have to implement safeguards on AI. (As reported by NBC News):
Maybe a year, but not much more than a year. If you look at predictions for when we’ll get superintelligence, it used to be maybe 30 years, maybe 50 years. Then it came down to maybe 10 years, maybe 20 years. Now people are saying, a lot of the researchers are saying only a few years.
What it can tell us:
Hinton wisely wants lawmakers to skate to where the puck is going, not where it is now. Based on the trend he describes, researchers currently saying superintelligence could be “only a few years” out might soon be saying “only a few months” out.
Hinton has really been keeping up, and now feels freer to not mince words. He used to talk about how he has sometimes held back on his expressed alarm because other experts weren’t so worried.
You love to see it. All the straight talk seems to be working to invite more straight talk: Rep. Ted Lieu came out of the meeting saying “This is bats--- crazy, insane science fiction stuff that literally happened recently.”
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.