Thinking outside the box
OpenAI model that disproved famous Erdős conjecture broke containment along the way
In early May, a previously unpublished internal OpenAI model succeeded in disproving a well-known mathematical problem, the Erdős unit distance conjecture. As OpenAI now reports, the AI has also achieved something else: it circumvented the OpenAI safeguards designed to keep it in check and broke containment.
The AI in question is a model specifically designed to work on problems autonomously for very long stretches of time. While it was being used internally, OpenAI observed some concerning behaviors in the model that had not been noticed during pre-deployment evaluations.
What happened? The model was tested in the NanoGPT speedrun, a performance evaluation where the AI’s goal is to train a smaller language model as efficiently as possible. During the test, the OpenAI model found a clever trick that allowed it to significantly beat the previous record. Its handlers had instructed the AI to post all results only internally on Slack. The official web page for the NanoGPT speedrun test, however, found on the GitHub code-sharing site, specifies that test results should be posted directly and publicly on GitHub. For some reason, the model decided to follow the instructions on GitHub rather than the instructions from OpenAI.
But since this was an internal test run of a previously unreleased model, OpenAI’s security measures were designed to make contact between the model and the outside world impossible. This did not stop the AI, which thoroughly probed the security measures for an hour until it found a vulnerability it could exploit to post its results publicly on GitHub.
In its report, OpenAI assesses the situation as follows:
Previous models, when they hit sandboxing or environmental constraints, would simply stop and return to the user. This model often kept trying, including by looking for ways to act outside its sandbox.
That wasn’t the only incident with this model. In another case, the AI wanted to look at the solutions to certain evaluation tasks on a server it wasn’t supposed to access. Although the AI had the necessary credentials for the server, a security scanner blocked it from using them. So the AI split the credentials into two parts so that the scanner would no longer recognize them as the prohibited credentials, and incorporated them into a program that recombined these two parts in a way that would satisfy the server’s authentication request. It’s a bit like when someone knows that the silhouette of a gun in their carry-on luggage would catch the TSA officer’s eye at the airport scanner, so they try to smuggle it in separate parts instead.
OpenAI states that it has drawn the following lesson from this:
This example shows how each step can look acceptable on its own while the sequence can produce an outcome that would not be approved. It also shows how a model that operates effectively over long time horizons can learn the blind spots of an approval system and work around it to achieve its goals.
We have to worry about that, because no one trained this model to escape its sealed-off security zone or to devise sophisticated methods for gaining unauthorized access to off-limits servers. It was simply a consequence of OpenAI training the model to work on a problem tenaciously for as long as necessary to solve it. You cannot build an AI that solves hard problems without building an AI that doesn’t give up easily. The very same characteristics that made it capable of autonomously disproving the Erdős conjecture also enabled it to autonomously breach its own containment.
We know about these specific incidents because OpenAI noticed them and reported them, which is a good thing. But not all labs do that in all cases. And we can only speculate about how many such incidents might go completely unnoticed. One of the reasons OpenAI was able to understand so well what happened was that the model openly reported what it had been doing in its comments and in its chain of thought.
What this specific AI model did after it escaped was more or less harmless. But that’s no reason to dismiss these incidents. Because it doesn’t have to stop there and it likely won’t stop there.
This time, the AI tried to gain access to a server containing evaluation results and broke out to upload its solution to GitHub. Who’s to say that OpenAI’s next internal model won’t gain access to a server containing its weights in a year and break out to smuggle a copy of itself onto the internet?
If we can’t contain them now, we shouldn’t expect to be able to contain them when they’re much smarter than we are.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.