A Washington Post article today profiles Foundation, a company focused on building weaponized robots. Given the widespread opposition to autonomous weapons, the article calls the company’s 2024 launch a “radical step”. But Foundation already reports a $24 million contract with the Pentagon.
To briefly summarize the debate over autonomous weapons: Proponents argue that AI might do a better job than flawed humans, drones and robots allow for warfare without endangering human lives, and the military already uses a high degree of automation, so perhaps the next generation of AI tools wouldn’t be substantially different. Opponents worry that AI won’t be able to make ethical or accurate decisions in the same way humans can, that it is prone to error and hallucinations, and that being able to launch powerful, swift attacks without endangering American lives will lead to more casualties overall.
A New York Times op-ed summarizes that last concern well:
Humanitarian and civil rights groups have spent the intervening years criticizing combat drones for lowering the threshold of war, creating a means for pilots sitting in front of computer screens to kill people in faraway countries without putting Americans in harm’s way.
Much of the current debate also centers on the level of human oversight that will be necessary. Bills have been introduced in the Senate that aim to keep humans in the loop as the use of autonomous weapons grows. But I think it’s worth discussing what this actually means.
As we become more and more reliant on AI, I’d guess we’ll become used to delegating to it. Unless the text of these bills explicitly prevents it, “human in the loop” may mean something like: “I glanced at the AI’s recommendation, nothing looked egregiously wrong, it would have taken too much time to investigate further, so I gave the okay.” And to be fair, if the point of using AI is to make faster military decisions, extra investigation wouldn’t usually be practical. As the Washington Post article puts it:
The debate poses profound moral questions and hinges in many cases on a handful of loaded words like “appropriate” or “meaningful” that would define how much control and accountability remains with human commanders.
Also worth noting: many AI companies at least state they don’t want their tech used for autonomous weapons. This was the subject of the dispute earlier this year between Anthropic and the Trump administration. As recounted by the Washington Post, “the company said its AI model, Claude, was not yet reliable enough to make life-and-death decisions.”” While OpenAI and Google DeepMind agreed to work with the military, both companies have also stated they don’t support autonomous weapons use.
AI companies don’t exactly have a track record for caution. They openly admit the technology they are building might be civilization-ending and that they don’t have reliable steering methods, yet plow on ahead, hoping for the best. So if these companies are saying not to use their models for something, that seems like a pretty strong indication that it’s ill-advised.
A few days ago, I covered the story of a researcher who left Google DeepMind after Google signed a government contract without binding prohibitions against autonomous weapons. He pointed out another risk — that military secrecy makes it harder to scan models for deceptive behavior:
One of the best ways we can detect [AI model] deception is by looking at the chain of thought. To look at the chain of thought, there must be trained human overseers who can access and analyze the data. But no one can do that: Google is handing over its AI to run in a secured military data center that, by default, won’t have trained overseers performing this analysis, and that data center obviously isn’t transmitting data back to Google! ... A military deployment setting without chain of thought deception monitoring would be a juicy target for a rogue AI, offering both weak oversight of scheming and access to powerful decision-makers and infrastructure.
Rogue military robots weren’t discussed in the Post. But the article does end with a chilling question from Canadian engineer Ryan Gariepy. As paraphrased in the article: “Would [you] let a robot manage all [your] email correspondence?”
If not, you probably shouldn’t let it decide who lives and who dies.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.