Geometry dash

Containment breaches, citizen science contamination, and another disproven conjecture

In this issue:

Thinking outside the box - OpenAI model that disproved famous Erdős conjecture broke containment along the way

Manipulating reality - Nature scientists warn that AI is unknowingly impacting the scientific record

AI casually crushes another famous math conjecture - The fall of the Jacobian conjecture is either surprising or a non-event, depending on who you ask

---

Dispatch from Robert

Thinking outside the box

OpenAI model that disproved famous Erdős conjecture broke containment along the way

Cute and harmless now, but they grow up so fast.

In early May, a previously unpublished internal OpenAI model succeeded in disproving a well-known mathematical problem, the Erdős unit distance conjecture. As OpenAI now reports, the AI has also achieved something else: it circumvented the OpenAI safeguards designed to keep it in check and broke containment.

The AI in question is a model specifically designed to work on problems autonomously for very long stretches of time. While it was being used internally, OpenAI observed some concerning behaviors in the model that had not been noticed during pre-deployment evaluations.

What happened? The model was tested in the NanoGPT speedrun, a performance evaluation where the AI’s goal is to train a smaller language model as efficiently as possible. During the test, the OpenAI model found a clever trick that allowed it to significantly beat the previous record. Its handlers had instructed the AI to post all results only internally on Slack. The official web page for the NanoGPT speedrun test, however, found on the GitHub code-sharing site, specifies that test results should be posted directly and publicly on GitHub. For some reason, the model decided to follow the instructions on GitHub rather than the instructions from OpenAI.

But since this was an internal test run of a previously unreleased model, OpenAI’s security measures were designed to make contact between the model and the outside world impossible. This did not stop the AI, which thoroughly probed the security measures for an hour until it found a vulnerability it could exploit to post its results publicly on GitHub.

In its report, OpenAI assesses the situation as follows:

Previous models, when they hit sandboxing or environmental constraints, would simply stop and return to the user. This model often kept trying, including by looking for ways to act outside its sandbox.

That wasn’t the only incident with this model. In another case, the AI wanted to look at the solutions to certain evaluation tasks on a server it wasn’t supposed to access. Although the AI had the necessary credentials for the server, a security scanner blocked it from using them. So the AI split the credentials into two parts so that the scanner would no longer recognize them as the prohibited credentials, and incorporated them into a program that recombined these two parts in a way that would satisfy the server’s authentication request. It’s a bit like when someone knows that the silhouette of a gun in their carry-on luggage would catch the TSA officer’s eye at the airport scanner, so they try to smuggle it in separate parts instead.

OpenAI states that it has drawn the following lesson from this:

This example shows how each step can look acceptable on its own while the sequence can produce an outcome that would not be approved. It also shows how a model that operates effectively over long time horizons can learn the blind spots of an approval system and work around it to achieve its goals.

We have to worry about that, because no one trained this model to escape its sealed-off security zone or to devise sophisticated methods for gaining unauthorized access to off-limits servers. It was simply a consequence of OpenAI training the model to work on a problem tenaciously for as long as necessary to solve it. You cannot build an AI that solves hard problems without building an AI that doesn’t give up easily. The very same characteristics that made it capable of autonomously disproving the Erdős conjecture also enabled it to autonomously breach its own containment.

We know about these specific incidents because OpenAI noticed them and reported them, which is a good thing. But not all labs do that in all cases. And we can only speculate about how many such incidents might go completely unnoticed. One of the reasons OpenAI was able to understand so well what happened was that the model openly reported what it had been doing in its comments and in its chain of thought.

What this specific AI model did after it escaped was more or less harmless. But that’s no reason to dismiss these incidents. Because it doesn’t have to stop there and it likely won’t stop there.

This time, the AI tried to gain access to a server containing evaluation results and broke out to upload its solution to GitHub. Who’s to say that OpenAI’s next internal model won’t gain access to a server containing its weights in a year and break out to smuggle a copy of itself onto the internet?

If we can’t contain them now, we shouldn’t expect to be able to contain them when they’re much smarter than we are.

---

Dispatch from Alana

Manipulating reality

Nature scientists warn that AI is unknowingly impacting the scientific record

Credit: KOMMERS via Unsplash

Suppose a very smart AI gained deep insight, via data and pattern analysis, into how human memories are recorded. Could it then use that knowledge to encode false memories … of your boss telling you to do a specific task, for example? Could it subtly alter how you perceive the world in ways that would change your behavior?

In Chapter 6 of If Anyone Builds It, Everyone Dies, authors Nate Soares and Eliezer Yudkowsky offer memory illusion as one possible guess for one possible way a superintelligent AI might manipulate reality.

I was recently reminded of this by an article in The Guardian, which covered a much lower-stakes, but still vaguely creepy example of AI-induced reality manipulation: humans are using AI to enhance nature images and unknowingly affecting scientific recordkeeping.

“Citizen science” is a way of crowdsourcing information about the natural world. Nature lovers and photographers post pictures of plants and animals on platforms like iNaturalist — information that, according to the platform’s director of community support, “a scientist could probably never get at scale.” He explains:

It is also almost like a sensor of what is happening on Earth in real time: are plants flowering early? Are species moving north as the climate warms? The more we know about where species are, the better informed we can be as conservationists.

But now, with more and more nature photo platform users creating or enhancing their images using AI, the record is essentially corrupted.

In one example, a user asked AI to make their bird image look better, and AI confidently changed the species from an epaulet oriole to a red-winged blackbird. This change seems to have gone undetected by the user, leading to a false sighting in Central Brazil, an area where the species hadn’t previously been seen.

From: Citizen science platforms must mitigate against the threat of generative AI

Events like this have led researchers to issue a warning in the science journal Nature:

AI image processing tools are now ubiquitous, and there is a pressing need to educate contributors of the problems that image manipulation can cause for the scientific record.

My take: this isn’t a dramatic example of AI-induced reality manipulation. But it’s proof that AI can already change our picture of the world, with real consequence.

---

Dispatch from Mitch

AI casually crushes another famous math conjecture

The fall of the Jacobian conjecture is either surprising or a non-event, depending on who you ask

A Spanish conquistador helmet (comb morion). Credit: Jorge Barrios.

Social media went a little nuts yesterday after Levent Alpöge, a Harvard mathematician affiliated with Anthropic, posted a disproof of the Jacobian conjecture, a venerable hypothesis about the reversibility of some kinds of functions.

It went viral for a few reasons: For one, the disproof fit inside the low-key tweet that Alpöge used to announce it, and was quickly verified by others.

The tweet.

For another, this was one of the more famous and recognizable open math problems; people have been trying to crack it for more than 80 years. It also seems to have been generally assumed that the conjecture was true, so the disproof came as something of a surprise.

And unlike some other recent high-profile AI math breakthroughs, this one seems to have been achieved by Fable, a commercially available model, rather than a special variant of an internal model optimized for such work (like the special OpenAI model that disproved the Erdős unit distance conjecture in May).

But we don’t yet know much about how Fable and Alpöge approached the problem to find this counterexample, so people aren’t quite sure what to make of it. News site Mashable quoted Andrew Blumberg, a Columbia mathematician involved in AI math benchmarks who was unimpressed:

This is exactly the kind of thing I would expect AI to be able to do. If there was a counterexample that was concise and easy to state that people haven’t found because it’s a pain to search through all this stuff, AI will find it.

On the one hand, I’m not a mathematician. But on the other hand, Blumberg’s claim sets off my goalpost-shifting detectors. If such a famous problem could be easily solved through brute force, why hadn’t this been done before modern AIs, which are actually really bad at brute-force computation? If the value added by Fable was figuring out how to narrow the search space for brute-force solvers, then why had humans failed to figure that out on their own?

This sounds to me like another case where, if a human had come up with the insights, we would be celebrating their creative genius. But if an AI did it, then creativity must not have been required at all.

Put that debate aside, though. Regardless of how people want to frame it, the puzzle was solved. This should give us pause. The way these old math problems keep falling to AI is more evidence that superhuman AIs could come to understand the rules of our universe better than we do. With such understanding, they might be able to employ strategies that seem to bend those rules and catch us completely off guard.

Mathematics matters. It has downstream effects on the other sciences, which then inform the applied engineering disciplines. When armed with better theory, chemists, biologists, materials engineers, and others can aim directly for innovations they might never find by trial and error.

I shudder to imagine a superhuman AI that finds solutions to more of humanity’s open problems but keeps them to itself, developing a deeper scientific theory and sharper engineering acumen, leapfrogging humanity’s more incremental gains. This is one way we could end up in the kind of lopsided situation the Native peoples of the Americas were in when European colonists arrived to claim their lands. In that situation, I don’t know how much it softens the blow to say the new arrivals had merely brute-forced solutions to tedious problems you couldn’t be bothered with.

---

The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.