In this issue:
Big Brother was always a machine - But AI could put the total in totalitarianism
Little Brother is looking over your shoulder - Meta and Spirit Airlines offer cautionary tales of workplace surveillance
Dispatches from Mitch
Big Brother was always a machine
But AI could put the total in totalitarianism
If you haven’t actually read George Orwell’s 1984 (you should!), there are a few things I as your former English teacher want to correct about your culturally absorbed understanding of it:
The mustachioed Big Brother always watching you from omnipresent posters probably isn’t any one actual person, if he ever was.
Most people aren’t actually under constant surveillance, because most people don’t matter. The watchful eye is reserved for the people who would be in a position to upset the system, which in practical terms means people who are part of the Party machinery.
The “thought police” can’t actually read minds.
Greed has little to do with it. Even the Inner Party elites are trapped by the system, and are materially much worse off than the upper classes of old.
Most people don’t recognize that they are being oppressed and deprived, thanks to the Party’s control over information. Those who buy into the Party line are actually pretty ecstatic, much of the time, because the Party line is that they’re always winning.
The terrifying truth about 1984’s INGSOC, one of the three oppressive parties that rule most of the planet and performatively squabble over the rest, is that it is like a living machine. The people who comprise it are as helpless to change its form or function as individual cells in your body are to change yours. No one person is powerful enough to cause chaos or reform through their death, sabotage, or incompetence. As a result, the oppressor is effectively immortal, a zombified vice-grip that only squeezes tighter over time. Any surplus production not required for its own persistence is deliberately destroyed in a forever war there is no intention of winning, preventing economic growth and any technological advancement that doesn’t help with surveillance and control. Progress is halted. The world is frozen.
Orwell’s book was and still is a horror story because it seemed plausible even without its fairly minimal surveillance technology. It makes stable totalitarianism look like the logical endpoint of the standard dictator’s playbook if spliced with the kind of banal, self-perpetuating bureaucracy readers encounter all the time.
I don’t think AI is needed for humanity to slide into that trap. It just makes it much easier to move in that direction. The latest video from Rational Animations is an accessible introduction to the idea, and I think the art style in this one is especially compelling.
AI makes Orwell’s vision much easier to implement and much harder to undo because AI can be effectively everywhere at once, putting the “total” in totalitarianism in a way that INGSOC never could.
In 1984’s Oceania, the non-Party “proles” are mostly left to their own devices, unmonitored, in an animal-like ignorance about how much better life used to be. But AI could be tasked with monitoring everything.
In Oceania, the Thought Police have to infer thoughtcrime from external behavior. But we already have proof-of-concept demonstration that AI can learn to read human thoughts. The hardware is clunky and impractical today, but we must always remember the rule that once the machines can do something at all, it’s often not long before they can do it far better than we could ever hope to. And if AI becomes adept at reading thoughts, it might not be such a leap for it to start implanting them.
In Oceania, the Party machinery keeps itself in line by pitting its members against each other in an atmosphere of perpetual suspicion where deviations are always ratted out in the end. But AI could be the perfectly reliable “rat” that precludes even the kind of doomed tryst between a jointly misaligned pair of Party members that makes up the middle act of 1984.
In Oceania, there probably isn’t an actual Big Brother, because he would be a mortal point of failure. With AI technology, lifespans of dictators might be greatly extended, but the most stable Big Brother would be AI itself, and I would expect AI to eventually become the tyrant in any AI-enabled tyranny.
Good news: If you’re worried that AI-enabled totalitarianism might be even worse than extinction, I think the latter is still much more likely if we don’t halt the AI race. For AI totalitarianism to remain stable, AI must never be able and willing to completely replace the unreliable populace with machines. That would probably require it to be designed to carry out a stagnant totalitarian vision while never improving on its own capabilities. But nobody knows how to design AIs that actually internalize intended goals like that, especially with regard to not acquiring more power. So any dictators trying to lock the world into stable, stagnant tyranny through superintelligence would probably just get us all killed the same way corporations and democracies racing for superintelligence would.
But again, I don’t think superintelligence — or AI at all — is needed for humanity to slide into stable totalitarianism. As the Rational Animations video points out:
[P]erhaps the most broadly useful way to defend against totalitarianism is to strengthen and improve today’s democratic societies [...] like promoting civics education or building more trustworthy institutions.
If that sounds vague and wishful to you, I’m with you. But there is clearly some functional resilience in the world’s existing institutions that could be built upon, because these institutions have so far avoided nuclear war and other traps that once seemed inevitable. I’m confident they can be steered to head off global stable totalitarianism and human extinction from AI, too.
Little Brother is looking over your shoulder
Meta and Spirit Airlines offer cautionary tales of workplace surveillance
For workers subject to employee monitoring software in the age of AI, I imagine it must be like a pesky little brother you don’t want hanging around you all the time. Not only does it want to know what you’re up to when you’d rather it not, it’s now trying to learn from you. The thought of being a role model could be flattering, but you have every reason to expect your little brother will tattle on you at the slightest provocation, and that it may be trying to usurp your role as the kid good at that thing you do. Worse still, the little brother’s agenda changes: Tips shared in confidence one day may be used against you the next.
In April, Meta installed software on employee computers that monitored their apps, keystrokes, and mouse movements. The company had said it needed the data to train its AI models.
I’m honestly not sure if employees were more upset about potentially training their own replacements or about the indignity of being spied on. But they were not happy. In May, amid massive layoffs, they sued their employer. A primary complaint was that the company had engaged AI in the decision-making process, informed by the data collected by the monitoring tools about their work habits. Affected employees who had been away on family or medical leave believed they had been misrepresented by this data, and effectively discriminated against.
It will be for the courts to weigh the validity of the accusation. But the moral is clear: Even if your employer says they need your data to train AI (because you’re such an expert!!), and means it at the time, they might later use that data against you in a performance evaluation. Little brother at work!
Do you think your employer isn’t watching you? Don’t be too sure. In a Wall Street Journal column this week with tips about how to fool the monitors, Callum Borchers warns that many states don’t have laws requiring employers to tell you what they’re running on your work computer.
(Borchers’s tips, if you need them: Try to look only about 80% busy, because higher is suspicious; mark planned meetings on your calendar, lest you appear idle during those times; and prefer a physical mouse “jiggler” device over a software tool, because some monitors track the other apps you have running.)
Even if your employer actually isn’t tracking your work or your conversations on internal messaging servers, that doesn’t mean a little brother won’t be looking over them later. Case in point: Google paid $10 million at auction this week for the entire dataset of Spirit Airlines, which went bankrupt in May and ceased operations. Per CNN, this includes “emails and internal communications, spreadsheets, transactions with the public including bookings and frequent flyer information as well as human resources information on its employees.”
Google said only that the data “can be helpful in improving our products and AI models.” How so, exactly? The hosts of the Hard Fork podcast speculated yesterday that Google might not want this so much for standard text-prediction training data, but rather to create a simulated airline environment in which models might learn business skills. Simulated task environments are a booming sub-industry within AI right now.
The purchased data is supposed to have been stripped of individually identifiable information, but I wouldn’t bet on that protection holding against even present AI, which is already very good at making correlations to unmask the anonymous, let alone future AI. While I don’t think Google is interested in assessing the performance of former Spirit employees, let alone unmasking them, the data might become part of models that potentially could, and they may have incentive to do so: A model that learns more real-world context about the characters in its simulation might more masterfully navigate it.
Little brothers grow up. A potential scenario I don’t think I’ve seen even in sci-fi: a company’s AI turning you down for a job based on vibes about you from a simulation it only vaguely remembers from its childhood.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.