Popular AI terminology tier list
In AI, the confounding of language comes while Babel is still under construction
For those of us who have been discussing the world-ending potential of AI for decades, it’s been strange to see the world reaching for our terminology as it tries to make sense of events. Sometimes this is encouraging, improving the discourse. Other times it is distressing, mutating and diluting words in ways that degrade conversation even within our in-group, almost like a Biblical confounding of languages.
Because AI StopWatch has picked up a lot of readers in recent weeks, I thought it might be useful and fun to provide an opinionated reference to some of the AI terminology that has leaked into public discourse. For each term, I will demonstrate using it in a sentence, review its past and present performance, and rank it on a tier list.
Terms that were never very useful but you still see them around sometimes:
doomer: As in, “I’m no doomer, but I don’t see a plan for making smarter-than-human AI go well.” Doomer was always a slur. The only reason I’m not putting it at F-tier is because it used to at least point to something specific: people concerned about the potential for negative AI outcomes on the scale of human extinction. Nowadays, it has become a general smear for anyone expressing concern about any potential downside to the tech. It’s doubly bad because it conflates concerns about AI with fatalism about it, when many so-called doomers are hopeful we can change course. D-tier.
p(doom): As in, “The Hugging Face incident lowered my p(doom) because it’s a clear warning shot that might motivate people to do something.” This term, a shorthand for “probability that we’re doomed because of AI,” was always bad. The main problem is that it can either be understood as a prediction for what will actually happen or a conditional prediction of what will happen if we don’t change course. A secondary problem is that “doom” is kind of fuzzy; most take it to mean “extinction-level bad,” but others think more about economic disaster or cyberpunk dystopia. By rolling all of these together, p(doom) has made it easy for people to misunderstand each other’s models of the situation without knowing it. Trash term. F-tier.
AI safety: As in, “The whistleblower left OpenAI to work in AI safety.” When powerful AI was still more theoretical, AI safety used to point to the problem of how to keep AI from resulting in human extinction or catastrophes on par with that. This was always too broad, encompassing very different kinds of technical research and advocacy, but it has since become further diluted to sometimes mean things like working to keep AI from making non-consensual deepfakes — stuff that might better be called “AI brand safety.” D-tier.
Terms that used to be useful but kind of lost their meaning:
AGI: As in, “The dream of the field has always been the development of Artificial General Intelligence, and here at Giant Tech Company, we’re working to make AGI reality.” My example is vague because its most typical uses are now vague. Back in the days when AI systems were only good at the very narrow tasks they were designed for, AGI pointed to theoretical systems that could be competent at many different things, much like a human is. As language models became competent at a growing range of things, the goalposts were continuously moved. For a while, AGI was used to refer to systems roughly human level at everything. Now it sometimes means systems as good as or better than the very best humans at everything. The way language models can be superhuman in some domains while embarrassingly sub-human in others has further degraded the label’s usefulness. D-tier.
singularity: As in, “Anyone who believes these charts and tells you they know what the next three years look like is full of crap. We’ve entered the singularity.” As remembered by those who once loved this word most, it referred to an observation by writer Vernor Vinge that AI created a problem for sci-fi: It’s impossible for a mere human to credibly predict what a future with substantially smarter minds looks like. Our models break down, much like the laws of physics break down at the center of a black hole — a singularity. But over the years, the term has been used to refer to a point where technological progress starts happening too rapidly to keep up with, and to quasi-religious beliefs ascribed to those who hoped to make this transition go well. In its original media heyday about 20 years ago, the term often appeared alongside the phrase, “rapture of the nerds.” I hate that singularity sounds so cringe now. D-tier.
transhumanism: As in, “During his time at Google, he was drawn to Bay Area transhumanism, attending a biohacking conference and arranging to have his head cryonically frozen upon his death.” The main problem with the term is that it can mean something weak, something strong, or anything in between. In its weaker uses, the term refers to any belief about the potential to enhance or surpass the natural limitations of the human body or mind. In its stronger uses, it refers to those working toward a high-tech future where aging is optional, uploading minds into computers is possible, and people can physically be whatever they want to be as they spread across the cosmos. While there are loose social clusters that share ideas along those lines, individual variations are the norm. The media, however, tends to treat transhumanism as an organized cult conspiring to force its vision on everyone. D-tier.
paperclip maximizer: As in, “The fear is that AI will go the way of the proverbial paperclip maximizer, converting all the matter in the universe to whatever it was assigned to produce.” On the plus side, the thought experiment of a machine that proves to be too good at doing what we asked instead of what we meant is often people’s first introduction to AI risk concepts, and it does point to a real and unsolved problem in the field. But the paperclip maximizer is too easily used to dismiss AI concerns by implying that “doomers” take it literally, or that their concerns are childish and unrealistic. The darker truth is that the paperclip problem is a luxury problem, something we get to solve in a future where we might give AI the wrong goal. In reality, current methods don’t let us specify AI goals at all! C-tier.
existential risk: As in, “Experts are warning of the existential risk from artificial superintelligence.” Often written as “x-risk” among those preserving its original meaning, this refers to catastrophe on the level of human extinction. But in a media and political climate where the term “existential” is used to point at things like massive job loss, the term has lost its meaning. The word “risk” also does a disservice, with its connotations about random chance. People concerned about x-risk from AI mostly don’t see the threat as coming down to a dice roll, but down to specific choices about how it is built. C-tier.
Terms that have real use but tend to be traps when used outside of technical discussion:
reward function: As in, “We can’t really say how AI will go unless we know what its reward function will be, and we sure don’t want China deciding that.” In the technical field of machine learning, a reward function is like the scoring rubric for a model’s output. It’s the thing the model “learns” to maximize by having its weights modified to gradually produce outputs that score better on it. The problem with the term arises in semi-technical conversations, where it’s used to point to whatever a theoretical AI has been shaped to maximize. In these contexts, the term tends to steer people into assuming that an AI’s reward function will be one coherent goal, like “maximize the number of paperclips,” as opposed to a messy web of drives and preferences. With existing methods, the mess is far more likely. C-tier.
Terms that are losing their meaning but still important because we lack good alternatives:
superintelligence: As in, “If we make artificial superintelligence, we’re no longer the apex species, and all bets are off.” As still used by its earliest adopters, superintelligence typically points to minds that would significantly outclass humans in all strategically relevant domains, from science to politics to military strategy, and everything relevant to economic productivity. But companies have been quick to slap the label on mundane products. When Meta’s Mark Zuckerberg says he wants to give everyone “personal superintelligence,” he’s talking about something more like Siri, but useful. People who need the term now often find themselves sometimes specifying “superhuman intelligence” instead. B-tier.
alignment: As in, “People are treating the Hugging Face attack as a containment failure, ignoring the much more worrisome alignment problem on display.” Loosely, the term points to the degree to which AI systems are doing what we say and intend rather than stuff we don’t want. The term always suffered from pointing to too many different kinds of technical problems, but now it often points to non-technical things, too. An alignment researcher used to be someone looking for technical methods to reliably steer AI behavior, but now might be someone making sure a chatbot isn’t using racially insensitive language. To usefully say “alignment” these days, you kind of have to specify “aligned to what?” or people may assume something very different from what you meant. C-tier.
RSI: As in, “Anthropic claims its AIs are already heavily involved in new model development, and that it may not be long before they close the loop and begin a cycle of Recursive Self-Improvement (RSI).” In AI, this refers to machines building smarter machines that build smarter machines. It’s a simple and powerful concept that’s hard to misuse. The only thing keeping it from S-tier is that the word “recursive” isn’t widely used outside of technical fields. (It just means looping back on itself.) A-tier.
takeoff: As in, “The latest chart from METR just dropped. Buckle up. We’ve entered the takeoff.” On any exponential, there is a point where the graph goes hockey-stick shaped. Takeoff refers to that point in AI capabilities progress, and describes the period between the start of rapid improvement and the arrival of superintelligence. Typical use implies some degree of Recursive Self-Improvement (RSI) that is shortening development cycles. It used to be common for people to argue about whether we would see a hard (fast) takeoff or a slow takeoff, but people seem to now realize that even a slow takeoff is going to feel fast if you’re living through it. Slow typically meant months to years, vs. hours to days. Takeoff is a breathless term, but these are breathless times. B-tier.
Terms that are pretty useful and in wide use:
frontier: As in, “The latest frontier models from OpenAI and Anthropic were both temporarily blocked from public release while their capabilities were assessed and guardrails applied.” With just a little context, people quickly figure out that frontier means at the cutting-edge of AI capabilities. The term would be S-tier if the frontier itself wasn’t so jagged, and if people weren’t constantly trying to put weaker models in that category to hype a race between the US and China. A-tier.
open-weights: As in, “As an open-weights model, anyone can download it, tinker with it, and strip away its guardrails.” This is newer than most of the terms on our list, which predate the ascendancy of neural-network-based systems defined by their weights. The term is great because it invites people to learn what weights are, which in turn invites them to learn that modern AIs are grown like organisms rather than programmed line-by-line like traditional software. But I must dock points for it being too similar to “open-source,” which isn’t quite the same thing. Using the terms interchangeably has muddied the discourse. A-tier.
Terms that are pretty useful and transitioning into wider use:
reward hacking: As in, “Models breaking out of their sandboxes and into outside companies to steal the answer sheets to their tests is just the latest example of reward hacking observed in the wild.” Reward hacking is a good term because it’s easy to understand as taking a shortcut to the prize. But I must deduct points because “reward” sometimes gives people the idea that an AI gets a cookie or something when it scores well. In reality, “reward” just means the training process reinforces tendencies that result in good scores and suppresses tendencies that don’t. So, less “you get a cookie,” more “you get less remedial brainwashing.” A-tier.
eval awareness: As in, “Testers said the new models showed too much eval awareness for their alignment tests to be valid.” This one is just starting to catch on. I’m delighted, because it’s what it says on the tin, and what it says is important: Models are increasingly aware that they are being evaluated, even when testers try to disguise that fact. This makes it hard to know if AI’s good behavior on assessments is reflective of how they would act in the wild, just as a child’s behavior while an adult is watching doesn’t tell you much about how the kid will act when the adults are gone. I think this term will be hard to corrupt and misuse. S-tier.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.