Yesterday, the Institute for Security and Technology released a survey of what national security professionals expect from advanced AI: the hazards, the opportunities, and how much risk they think is worth running. The 111 people surveyed were former Under Secretaries and Assistant Secretaries, deputy assistant secretaries, NSC senior directors, senior diplomats, intelligence leaders, retired lieutenant generals and major generals, plus the academics and technologists who work alongside them. Eighty-seven percent currently serve or have served in the U.S. government.
The questions were unusually precise: Most polls will ask, “How worried are you?” and then I’ll have to write that such-and-such a number were quote-unquote “worried or highly worried.” Questions in this poll read, “What is the probability that an AI-caused global catastrophe (defined here as an event causing the death of more than 10 million people) occurs before 2050?” and “What is your estimate of the likelihood that AI-enabled capabilities could contribute to a global catastrophic event leading to human extinction (or irreversible civilizational collapse) within the next 50 years?” 40% of respondents put the odds of ten-million-deaths at 5% or higher, and 26% put the odds at 10% or higher. On extinction or irreversible collapse, a majority came in under 1%, but 28% said 1% or higher, and 16% said 5% or higher.
A third question asked what probability of such a catastrophe “would be unacceptable in exchange for the economic and strategic benefits of AI leadership.” Respondents’ answers were set against their estimates of the actual risk: Among the 56 who gave concrete answers to both that question and the ten-million-deaths question, 61% put the risk at a level they had marked as unacceptable. The same precision shows up on loss of control. Asked for the probability that AI operates outside human control within ten years, the median answer was that there is a 33% chance that humans lose control. 87% of respondents put it at 10% or higher.
The sample may, in the report’s own phrasing, “overrepresent people who view AI as an important policy issue.” Respondents came through professional networks and referrals, and didn’t have to answer everything (hence my qualifier about “56 respondents”).
Another thing: The responses were collected between April 30 and July 15, so the Hugging Face attack didn’t influence anyone’s response. Polls are fuzzy and track moving targets, but they can still give useful information. My takeaway is that, even before the events of July and August, a large share of the people who have spent careers pricing risk for the U.S. government had already run the numbers, counted the upside, and deemed the threat unacceptable.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.
You can receive emails of dispatches as we write them, or subscribe to our Daily Digest for a once-a-day compilation.