War machines can run amok with AI in control

AIs not only make life-and-death targeting decisions on the battlefield, but also involve humans in only some stages of the decision-making process. To illustrate how that happens and prompt greater scrutiny of model-enabled killing, Airwars, a not-for-profit transparency watchdog, has published a detailed report, taking apart the ways machine learning powers modern warfare. The investigation, titled Anatomy of an AI Kill Chain, offers a visual guide through a fictitious kill chain – the steps real militaries go through to identify targets and eliminate them, with an emphasis on decisions delegated to AI. Authors Sophia Goodfriend, Heidy Khlaaf, Namir Shabibi, Joe Dyke, and Nathan Walker cover six stages: the decision support systems used to assist data gathering, surveillance technology, intelligence and identification, target selection, strikes on targets, and post-strike assessments. The report notes, "A recent book about US military AI revealed that in some operations only two of the six stages of the US military kill chain now involved humans in the loop, with a third involving some human oversight. The rest are now fully automated." Given that targeting mistakes have been widely documented, it's worth wondering whether AI is making such errors more common. And since military officials have, in the past, insisted AI's role cannot be known, there's reason to look for ways to better understand when AI is involved in life-and-death decisions. "In journalism and in policy, there's a tendency to focus on maybe one autonomous weapon system, like an Anduril drone or a Palantir anomaly detection system, and to look at the specific companies that are producing a few single machine learning algorithms that are undergirding those systems," said Sophia Goodfriend, research fellow at the University of Cambridge’s Pembroke College and co-author of the report, in a phone interview with The Register. "What we wanted to do … is to really underscore the stack of AI systems that are upending what it means to wage war, specifically what it means to surveil, target, and kill on the battlefield, and also emphasize how various technologies work together or don't work together throughout the kill chain." The project's goal, she said, is to look beyond specific technical systems at the way warfare is mediated by machine learning algorithms and to highlight the specific limitations of those systems. For each stage of the kill chain, the report touches on sources of potential errors: the reliability and accuracy of decision support systems, automated translation errors during surveillance of a target's text messages, target risk scoring systems derived from algorithmic assessment of social media data, computer vision systems that may misidentify objects and people, the shortcomings of recurrent neural networks used during drone strikes when there's GPS and/or electronic jamming, and the potential problems with AI-assisted battle damage assessment systems. The specific kill chain described is fictitious, but Goodfriend said that this was necessary to avoid making overreaching claims about how specific militaries operate. "Obviously it's hard to reconstruct proprietary systems used by militaries in real time today because they are often censored and secret and reporters cannot pry open exactly how a military is operating," she said, adding that she hopes the report prompts people to question claims about having a human in the loop. "What comes across in this piece is just how difficult it is for a human to intervene in a machine learning decision, the pernicious problem of automation bias, especially under the time constraint of war, and just how thoroughly everything is mediated by AI and automation already on the battlefield," she said. "So even if you have somebody who's reviewing a risk profiling system by looking at a social media post, that social media post will be translated by a machine learning algorithm and they won't be able to verify if that translation is correct because they're dealing with this time constraint and it will be really tempting and much easier to defer to an AI-generated output." Goodfriend acknowledges that it's difficult for civilians to make decisions about technologies they use that might amplify physical safety risks from military AI systems. That's why, she said, it's important to draw attention to the way these systems work. "The stack of AI systems integrated into warfare are the subject of a lot of hype and a lot of mystique and it's rare for those making or deploying these systems to really break down how they actually operate it and what it entails for civilians living under warfare," she said. "So, in illustrating how these technologies work, the massive amounts of data, of people's private information, of passive surveillance that is the foundation and lifeblood of automated warfare, I think that that's one way of moving the needle in terms of advocacy." The other key takeaway of the report, she said, is to underscore that machine learning systems make errors. "The point is not that these are teething problems that will be mitigated or ironed out once these technologies are deployed more and refined more, but really to emphasize the inherent flaws of machine learning and how they can't perform the tasks that militaries expect them to perform when they're deployed on the battlefield," she said. ®

Indexed and credited by AIPROPX. Originating outlet: The Register. Open at source →