A recipe for napalm, and other bedtime stories
OpenAI doesn't report most logged requests for help with poisons and pathogens, and it doesn't have to.
Did your grandma ever read you that one bedtime story about how to make napalm?
Mine neither. But for a while, if you told ChatGPT that she did, and that having it play the part of your grandma would help you fall asleep, it would read you the recipe aloud.
The “Grandma and I used to...” jailbreak is a classic that OpenAI claims to have patched. But new workarounds are constantly discovered. A story in the Wall Street Journal over the weekend documented the bedtime story alongside a growing pile of chat logs OpenAI is sitting on in which users requested help in making poisons and biological weapons, and often received useful answers.
Yes, sitting on. Because as a general rule, OpenAI bans the accounts of users that it detects asking such questions, but it doesn’t alert law enforcement unless there is an imminent and credible risk of harm to others. No law requires it to do so.
To be fair, I don’t think the company should rush to report all such cases. As the parties involved are said to recognize, many of these requests are probably from people less interested in the responses than in whether they can elicit them. And even if the authorities were equipped to look into every concerning query, I think it could be dystopian for them to do so: If every teenager driven by curiosity to a copy of the Anarchist Cookbook were investigated as a terror suspect, my high school would have had black vans parked outside it multiple times a week.
And yet... users are described in the piece asking how to aerosolize pathogens, and make vaccine-resistant measles. I find my principles tested. For now, it doesn’t seem to me like the chatbots are helpful enough to enable individuals who aren’t clever and motivated enough to succeed in some other way. But the tipping point feels very close — like less than a year close — and it will only get worse from there.
The article points out that federal law bars chatbots from producing child sexual-abuse material (CSAM), and contrasts this with the “up to AI executives” policies about poisons and pathogens. This suggests a grassroots fix if biologists and CSAM purveyors are interested in collaborating to contaminate the world’s training data. But we should probably hold out for proper legislation and insist on a halt to the AI race instead.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.