If you haven’t actually read George Orwell’s 1984 (you should!), there are a few things I as your former English teacher want to correct about your culturally absorbed understanding of it:
The mustachioed Big Brother always watching you from omnipresent posters probably isn’t any one actual person, if he ever was.
Most people aren’t actually under constant surveillance, because most people don’t matter. The watchful eye is reserved for the people who would be in a position to upset the system, which in practical terms means people who are part of the Party machinery.
The “thought police” can’t actually read minds.
Greed has little to do with it. Even the Inner Party elites are trapped by the system, and are materially much worse off than the upper classes of old.
Most people don’t recognize that they are being oppressed and deprived, thanks to the Party’s control over information. Those who buy into the Party line are actually pretty ecstatic, much of the time, because the Party line is that they’re always winning.
The terrifying truth about 1984’s INGSOC, one of the three oppressive parties that rule most of the planet and performatively squabble over the rest, is that it is like a living machine. The people who comprise it are as helpless to change its form or function as individual cells in your body are to change yours. No one person is powerful enough to cause chaos or reform through their death, sabotage, or incompetence. As a result, the oppressor is effectively immortal, a zombified vice-grip that only squeezes tighter over time. Any surplus production not required for its own persistence is deliberately destroyed in a forever war there is no intention of winning, preventing economic growth and any technological advancement that doesn’t help with surveillance and control. Progress is halted. The world is frozen.
Orwell’s book was and still is a horror story because it seemed plausible even without its fairly minimal surveillance technology. It makes stable totalitarianism look like the logical endpoint of the standard dictator’s playbook if spliced with the kind of banal, self-perpetuating bureaucracy readers encounter all the time.
I don’t think AI is needed for humanity to slide into that trap. It just makes it much easier to move in that direction. The latest video from Rational Animations is an accessible introduction to the idea, and I think the art style in this one is especially compelling.
AI makes Orwell’s vision much easier to implement and much harder to undo because AI can be effectively everywhere at once, putting the “total” in totalitarianism in a way that INGSOC never could.
In 1984’s Oceania, the non-Party “proles” are mostly left to their own devices, unmonitored, in an animal-like ignorance about how much better life used to be. But AI could be tasked with monitoring everything.
In Oceania, the Thought Police have to infer thoughtcrime from external behavior. But we already have proof-of-concept demonstration that AI can learn to read human thoughts. The hardware is clunky and impractical today, but we must always remember the rule that once the machines can do something at all, it’s often not long before they can do it far better than we could ever hope to. And if AI becomes adept at reading thoughts, it might not be such a leap for it to start implanting them.
In Oceania, the Party machinery keeps itself in line by pitting its members against each other in an atmosphere of perpetual suspicion where deviations are always ratted out in the end. But AI could be the perfectly reliable “rat” that precludes even the kind of doomed tryst between a jointly misaligned pair of Party members that makes up the middle act of 1984.
In Oceania, there probably isn’t an actual Big Brother, because he would be a mortal point of failure. With AI technology, lifespans of dictators might be greatly extended, but the most stable Big Brother would be AI itself, and I would expect AI to eventually become the tyrant in any AI-enabled tyranny.
Good news: If you’re worried that AI-enabled totalitarianism might be even worse than extinction, I think the latter is still much more likely if we don’t halt the AI race. For AI totalitarianism to remain stable, AI must never be able and willing to completely replace the unreliable populace with machines. That would probably require it to be designed to carry out a stagnant totalitarian vision while never improving on its own capabilities. But nobody knows how to design AIs that actually internalize intended goals like that, especially with regard to not acquiring more power. So any dictators trying to lock the world into stable, stagnant tyranny through superintelligence would probably just get us all killed the same way corporations and democracies racing for superintelligence would.
But again, I don’t think superintelligence — or AI at all — is needed for humanity to slide into stable totalitarianism. As the Rational Animations video points out:
[P]erhaps the most broadly useful way to defend against totalitarianism is to strengthen and improve today’s democratic societies [...] like promoting civics education or building more trustworthy institutions.
If that sounds vague and wishful to you, I’m with you. But there is clearly some functional resilience in the world’s existing institutions that could be built upon, because these institutions have so far avoided nuclear war and other traps that once seemed inevitable. I’m confident they can be steered to head off global stable totalitarianism and human extinction from AI, too.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.