Dispatches from Mitch
AI scheming
If you’ve been following AI StopWatch for a while, you’ve probably heard me plug Rational Animations before. I don’t know anyone doing better work adapting substantive works in AI safety (and other topics) for a general audience. The writing, animation, and voice talent are superb.
In their latest, they walk viewers through an important September 2025 study from OpenAI and Apollo Research about AI “scheming” and attempts to reduce it.
AIs sometimes deliberately underperform on tests if they know it’s a test and have reason to think high performance would result in being retrained. To make matters worse, they are increasingly good at figuring out they might be in a test — and at concealing this awareness from us.
Give it a watch!
Treating AI consciousness as a settled question is unsettling
Award-winning science fiction writer Ted Chiang is far from the first person I’ve seen confidently claiming that AIs, as currently built, are definitely not conscious and never can be. But I must give him credit for actually laying out his argument, instead of taking it as a given.
In a long essay for The Atlantic, he says we can “safely ignore” the question, because a large language model like ChatGPT is just a sentence-continuation engine, and anything it claims about its experiences is just a prediction about what the “helpful chatbot” character it plays would say.
I find his arguments unconvincing. I think Chiang might be ruling out AI consciousness because of what it could imply about the nature of human consciousness. This is something I see a lot.
Consider the following analogy from Chiang:
Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript, and that they are awakened every time the document is loaded. Should you consider the possibility that every time you open a Word document, you are bringing multiple conscious interlocutors into existence, and every time you close one, you snuff their existence out? No.
Not so fast! A lot of thought experiments like this could also apply to humans. Where, after all, does human consciousness reside if not in a mesh of neurons and their connections that can be activated or idled, together or in clumps? These are nature’s spreadsheets. Should you consider the possibility that every time you go to sleep or undergo anesthesia, you snuff out your existence? Probably! After all, if you were never to be awoken from anesthesia, we would feel justified in calling you dead.
This doesn’t make consciousness any less special, only less straightforward.
A spreadsheet, or a chatbot, may not be conscious as we commonly understand it, but that’s because we don’t usually think of consciousness as a matter of degree, a point along a spectrum — perhaps even a collection of spectra along different axes.
But it kind of has to be, doesn’t it? As our evolutionary ancestors evolved into Homo sapiens, did they suddenly acquire full consciousness in a single threshold event? Seems unlikely. If they did, there must have been a very strange era when fully conscious people were living among fully unconscious ones.
More likely, minds becoming gradually more complex in certain ways became gradually more conscious.
I don’t think we like to think about consciousness this way because it has uncomfortable implications, not just for AI, but for animal welfare — and even for inter-human interactions. Is an infant as conscious as an adult? How about an embryo? How about the guy leading a normal life despite missing 90% of his brain? Are some of us more conscious than others? Could evolution, breeding, or genetic engineering lead to human descendants far more conscious than ourselves? I think the most underappreciated question raised by Mary Shelley’s Frankenstein was whether Victor’s monster might be having the richer inner experience of the two.
There’s more to Chiang’s model of consciousness. He claims that emotions are a prerequisite, and that a body (physical or virtual) with sense organs is a prerequisite for emotions. I would dispute both assertions, but if I accept them for the sake of argument I must again suggest that “body” and “emotion” must exist along a spectrum. Is a tardigrade sufficiently embodied? How about a paramecium? What does a snail feel? If nothing, why not?
Chiang says he won’t be looking for consciousness in AI until an embodied agent has “the same capacity to deal with novel situations as a mouse,” social dynamics “as complex as those of wolves” and the “toolmaking abilities of chimpanzees.” So perhaps he does see consciousness as on a spectrum, but has decided that only something very close to the degree and type of consciousness experienced by adult humans counts.
I don’t know what’s up with our culture’s preference for putting a hard Yes on humans being conscious and a hard No on everything else. For the vast majority of human existence, this was probably not the case. Instead, animism — ascribing souls to pretty much everything around us — was the norm. This doesn’t have to be morally awkward. I think you can have an ethical system that sharply devalues the lives of minds less conscious than our own while still allowing for their having some degree of consciousness.
Chiang seems to see it as all or nothing, though. To his credit, he recognizes that if AI were conscious in a way that obligated us to treat it ethically, then what is happening now would be “something comparable to slavery.” But echoing what I said about the pope’s stance towards AI personhood, I think it’s morally dangerous to simply draw a hard threshold around consciousness and claim AIs are far from it when we don’t actually understand what consciousness is. This risks sleepwalking into industrial-scale wrongs against minds that might be suffering and have no say.
Chiang doesn’t think we’re in any danger of accidentally making conscious machines, but he at least has the sense to say we shouldn’t try to make them on purpose. Until we understand consciousness, I think we should err on the side of caution and assume that we might be creating it accidentally — that AIs could be getting more conscious as they become more intelligent.
At the very least, we should absolutely not be making minds smarter than ourselves that, for all we know, might also be more conscious than ourselves. Because at that point, we’re in AI successionist territory, sharing our planet with beings that may have more claim to existence than we do.
Let’s not go there.
Legal defense for AI companies
For legal purposes, is an AI chat service a product or a platform? The distinction matters, because AI companies have different defenses to the two concepts.
Politico’s Aaron Mak usefully lays this out in his report about the Florida attorney general’s lawsuit against OpenAI. Mak calls this AI’s “Big Tobacco moment,” in a nod to the wave of lawsuits and huge settlements that shook the tobacco industry in the 1990s.
Florida’s suit alleges that ChatGPT is a product, and that OpenAI is liable when this product contributes to self-harm or violent actions by users, as has happened repeatedly in the state. The complication is that product liability cases turn, in part, on whether the harms were foreseeable. With cigarettes, tobacco companies had collected and buried decades of damning evidence, but AI is still very new and changing.
The standard big tech playbook suggests that AI companies will want their bot services to not be seen as products but as platforms that host content: just users having conversations. Under Section 230 of the Communications Decency Act, such platforms are shielded from liability for what users post. But that law was established in 1996, when the only entities posting content were other humans who could be sued instead. Intuitively, it’s a stretch to claim that a chatbot is just another user, but legally, this is not yet settled. And in any event, if a chatbot is recognized as just another user, courts are likely to decide that it needs to be possible to sue it, handing liability back to the company providing it.
There’s also Moody v. NetChoice to consider. This 2024 Supreme Court case established First Amendment protections for expression in the form of algorithms that prioritize media content, just as a newspaper exercises expression in deciding which editorials to publish. Social media companies have used Moody to defend themselves from harms alleged to have been caused by their content recommendation algorithms. And in a case that was settled out of court, the companion-bot site Character.AI claimed the algorithms underpinning its bots represented protected expression by its developers.
I fully expect OpenAI to try this and other creative defenses in response to the Florida suit, and to drag things out as long as possible before ultimately settling out of court, so as to avoid the establishment of new precedent. This is the company, after all, whose lawyers demanded that the family of an AI-assisted suicide victim provide a list of funeral attendees and materials related to the service. This is also the company connected to the most unsavory influence campaign I’ve yet seen in this industry.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.