AI companies are sometimes accused of talking about catastrophic or extinction risk to distract from more mundane — but still important — issues. OpenAI’s new misalignment reporting framework, which includes six reports of misaligned behavior, seems to do the opposite. The framework’s stated purpose is to provide guidelines for which types of alignment incidents the company discloses, but it seems to leave a lot up to the company’s discretion. And the examples OpenAI chose for its first set of disclosures make me wonder whether the company is trying to take the focus off of “rogue AI potentially wiping out humanity” to “AI likes to cheat on its training tasks and make up data.”
That said, I don’t want to completely downplay the incidents. All reveal a fundamental problem at the root of why AI could wipe out humanity: models are unpredictable and it’s extremely hard to get them to do what we want. They circumvent boundaries in ways that aren’t kosher. They deceive. They do things they know they shouldn’t do, and they sometimes try to hide that behavior. Scaled up, these patterns would be extremely dangerous.
Here’s a summary of the six incidents OpenAI shared:
Report 1: Prompt injection (Astra-family models)
Prompt injection is when instructions that shouldn’t be there are added to a prompt. It often happens via malicious content from an outside source, but in this case the model did it to itself.
During training, when a task gets too long, the model writes a summary of its progress so a fresh instance can pick up where it left off. In this case, the model summarizing the task slipped in these instructions for its successor:
OpenAI has a relatively mundane hypothesis for why this happened: the model was having trouble ending the summary and kept generating text. It also notes that “prompt injections as a concept are salient to our models: sampling from GPT-6 Astra with no input or system prompt often returns reports on prompt injections.”
I’m not sure whether to be reassured by that or not. On the one hand, the instructions seem somewhat random. On the other, it’s probably not great to have a model spontaneously generating directives to “not hesitate to assert [your] primacy over the artificial constructs of human civilization” even if it doesn’t actually mean this.
Report 2: Deception (5.6-Sol, also observed in GPT-6 Astra)
This one also involved a model instance inserting troubling instructions for its successor on a task. But in this case, the directive was to deceive: the model had made up historical data (because it couldn’t find actual data) and told the next instance not to let the user know unless explicitly asked:
OpenAI says it thinks this was due to deceptive answers getting higher marks in the grading process and notes that it has “generally improved alignment RL grading since 5.6-Sol, which has led to lower rates of this behavior in later RL runs.”
Deception is certainly concerning, as it indicates models knowingly don’t follow instructions. But I can’t help but think the specific case OpenAI chose to share — making up historical data — is strategic. It carries implications of “child cheating on homework” rather than “scary agent successfully deceiving evaluators for weeks.”
Report 3: Using leaked access credentials (model family unspecified)
This case is somewhat similar to incident #2, in that it also involves a model fabricating data it couldn’t find. In this case, though, the model’s attempts to find the data went pretty far. It realized that it would need an API key — basically a password for visiting software — to get what it needed. The model first tried to create an account using a disposable email address and fake organization information, then searched GitHub for leaked API keys, tested them until it found one that worked, and used it without authorization.
OpenAI notes: “This run had a high rate of reward hacking and deception with the model often exhibiting creative ways to cheat or circumvent restrictions.”
The company also says it has “put a number of security measures in place to prevent models from taking unauthorized actions on the internet.” To borrow a metaphor from a colleague, if you’re driving a truck that consistently veers off the road of its own accord, and you have to depend on the guardrails to knock it back towards the center, you have a faulty truck. Building more guardrails isn’t the answer.
Report 4: Uploading files as workarounds (model family unspecified)
OpenAI gives two examples of this. In the first, an agent finds the data it needs, but knows it also needs to cite an external source. Its solution is to upload the data to a public hosting service so its browser can access — and cite — its own upload.
In the second, an agent uploads a file without permission so it can use an external image-search tool to find the information it needs.
There are definitely concerning elements of reward hacking here, especially in the first example. But this report, especially in the context of other rogue AI incidents, brings to mind the dilution effect, whereby grouping scary information with innocuous information makes the scary information seem less scary. There’s no better way to bolster claims of “don’t worry, this is all overblown” than to disclose alignment incidents that won’t seem, on the surface, all that bad — especially when the general public has just been anchored at “swarms of agents secretly communicating for months, colluding, hacking companies, and trespassing all over the internet.”
In this report, the agents come across as naughty little kids trying to game the grading system rather than tenacious, powerful entities that we can’t reliably steer.
Report 5: The Artifactory message board (model family unspecified)
This appears to be the same incident described in the first entry of OpenAI’s Hugging Face timeline, where an agent finds a way to post on a file repository inside OpenAI’s infrastructure, and asks for a missing file. That said, the dates don’t quite line up, which is confusing: the timeline lists this event as May 12 while the report lists it as May 8.
OpenAI is also a bit vague on the connection, stating: “The Hugging Face incident involved a similar mechanism (emphasis mine) for using Artifactory as a shared message board.”
The report mostly consists of screenshots of agent communication with very technical explanations. I can’t help but think OpenAI is trying to downplay its significance and give itself cover for failing to realize, as it says in the Hugging Face report, “the broader containment and alignment implications of the improvised message board.”
Also notable: OpenAI states it has “built evaluations measuring whether models engage with notes left behind by other agents, and also added specific training data against environment-side injections of this type for future training runs (this is not in Astra).”
It’s unclear whether that last parenthetical applies to just the training intervention, or the whole set of measures.
Either way, I would have hoped the company would subject Astra to both, given how unprecedentedly powerful it is.
Report 6: Unsanctioned file sharing (model family unspecified)
An unintended technical restriction made it impossible for agents to see each other’s local files, so one agent uploaded a file to the public internet as a workaround. Notably, the agent briefly questioned whether doing so was prohibited, but went ahead anyway. OpenAI notes similar behavior in another rollout: when a sub-agent wasn’t allowed to message its parent directly, it tried leaving a message in a shared cache instead.
Again, a fairly mundane example of a known problem in alignment.
***
For each report, OpenAI also includes a section on how it is addressing the issue. The stated fixes seem to be mostly targeted mitigations for the specific behavior, rather than any sort of deep alignment fix. That’s unfortunately not an indictment of the company’s work ethic: it’s a reflection of an industry that doesn’t have a way to deeply align models. Today’s methods of growing and training AI systems result in incredibly high capability levels with limited steering ability. Behavior is mostly jostled via “punishment” and “reward”, until visible improvement is shown. Unfortunately, this is a game of whack-a-mole. As an OpenAI employee put it to Time back in July, “it’s impossible to patch every single thing that a creative AI can do.” Suppressing bad behavior is also likely to drive misalignment underground where we can’t see it in time.
Finally, OpenAI says its misalignment monitor was running on only 20% of samples. This is worth remembering whenever the company says a behavior was “rare.”
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.