Recent reports on the NSA’s use of metadata and machine learning to generate intelligence for drone strikes in Pakistan spotlights the somewhat less-discussed legal concerns over the use of metadata in targeted killing operations.
Patrick Ball, a highly respected data scientist, was highly critical of the NSA’s SKYNET program in a recent report by Ars Technica. In particular, Ball attacked the NSA program’s method of “machine learning” for the purposes of classifying individuals as terrorists based on metadata of Pakistani mobile phones. Martin Robbins of The Guardian, however, pushes back on both The Intercept and Ars Technica accounts of the NSA program. Most helpfully, Robbins questions whether and how SKYNET information was actually used as a basis for targeting decisions. (Others including Chris Jenks and Geoffrey Corn have previously raised similar objections to the media’s characterization of SIGINT-based targeting.) Too often ignored by defenders of the drone program, as Gen. Hayden does in his recent op-ed, this question of uncertainty in data-driven intelligence and doubt over status deserves much more attention, and is essential to the interpretation and application of international humanitarian law (IHL) to drone operations.
From an IHL perspective, there are three questions worth separating out. First, is using probabilistic data analysis and conclusions as a basis for targeting decisions and assessments of status permissible under IHL? Second, and likely more relevant, can such probabilistic analysis be utilized in targeting decision-making processes, by humans, in a manner that is consistent with IHL? And, third, what are the long-term implications of the increasing reliance on probabilistic data analysis in intelligence-driven targeting decisions for accountability and promoting compliance with international legal norms?
Additional Protocol I states: “In case of doubt whether a person is a civilian, that person shall be considered to be a civilian.”
But the US Defense Department’s Law of War Manual explicitly rejects the idea that this presumption in cases of doubt exists as a legal obligation: “Under customary international law, no legal presumption of civilian status exists for persons or objects, nor is there any rule inhibiting commanders or other military personnel from acting based on the information available to him or her in doubtful cases. … In assessing whether a person or object that normally does not have any military purpose or use is a military objective, commanders and other decision-makers must make the decision in good faith based on the information available to them in light of the circumstances ruling at the time. A legal presumption of civilian status in cases of doubt may demand a degree of certainty that would not account for the realities of war.”
The ICRC notes that because many States have objected to strict interpretation of this rule “it is fair to conclude that when there is a situation of doubt, a careful assessment has to be made under the conditions and restraints governing a particular situation as to whether there are sufficient indications to warrant an attack. … In the case of non-international armed conflicts, the issue of doubt has hardly been addressed in State practice, even though a clear rule on this subject would be desirable as it would enhance the protection of the civilian population against attack.”
Intelligence-driven drone strikes, and revelations like SKYNET, push this question of doubt to the fore.
If US government officials were using programs like SKYNET, alone, to select targets, this would raise nearly identical concerns that Lethal Autonomous Weapons Systems raise under IHL. Humans would be “in-the-loop” but insofar as the actual target selection is left, uncritically, to the computer program, similar legal concerns would likely be present, including compliance with the principle of distinction in the absence of genuine, qualitative human judgment as well as the accountability gap.
However, it seems unlikely that SKYNET or other similar programs operate within the targeted killing program in this way. Instead, the issue of doubt creeps into the picture in a less obvious — though no less significant — manner. It seems much more likely that, as the US has done in Afghanistan, tools like metadata analysis are used as one of several sources of intelligence and information feeding human drone operators, intelligence analysts, and commanders. Uncertainty in intelligence tools and analysis, and inevitable doubt over individuals’ combatant status, therefore, are embedded more deeply in the process and exists at multiple levels and decision points.
Under IHL, there does not seem to be any particular reason to treat a program like SKYNET’s mode of information collection and analysis as inherently legally different from, say, human intelligence from an informant on the ground or observations from a joint tactical air controller calling in an airstrike. These sources of information are also vulnerable to some probability of error. The pertinent question remains how is that information ultimately used by those responsible? How much weight are its results given in human-led decisions to target individuals? Under IHL, what level of “doubt” or uncertainty as to civilian status, if any, is permissible? And what precautions are feasible and legally required?
It is impossible without the long-demanded transparency over the drone program to know the answers to those questions. However, there is reason to worry that, in practice, the reliance on metadata could undermine several core IHL protections and principles.
First, is the determination of combatant status and direct participation in hostilities (DPH). Being a “terrorist” or a member of a suspect network, the kinds of results programs like SKYNET produce, is not itself a sufficient basis to determine whether an individual is targetable. Over time, the reliance on SIGINT and metadata analysis, especially in the absence of reliable HUMINT and other on-the-ground sources of intelligence, could erode the presumption of civilian status, and as a practical matter, determinations made by metadata analysis could substitute or over-determine DPH assessments. In addition, direct participation in hostilities is based largely on an individual’s actions. Metadata analysis’ focus on associations between individuals and correlations of individual behavioral variables as predictors for target identification may, in effect, substitute DPH with a membership- or social association-based standard.
Second, is the undermining of the principle of proportionality. The use of SIGINT and metadata analysis, and the deployment of such significant resources on the side of identifying “targets” dwarfs the resources and analytical tools used to identify and verify civilian status. This could, in practice, weaken the presumption of civilian status of individuals that may be incidentally harmed in an attack. Records showing that US officials classified the vast majority of individuals killed in drone strikes as “unknown” provide a strong reason to worry that this has indeed happened. The application of a “good faith” standard, as set out in the DOD Law of War Manual, may exacerbate this risk of systematic bias in status determinations.
Third, is the feasibility of precautions. As the ICRC notes, feasibility has been interpreted to mean those precautions which are practically possible taking into account all of the circumstances. Among the possible, relevant circumstances, the DOD Law of War Manual lists: “the likelihood and degree of humanitarian benefit from taking the precaution” and “the cost of taking the precaution, in terms of time, resources, or money.”
The use and possible over-reliance on SIGINT and metadata analysis for targeting purposes, and the institutional investment and confidence in such tools, may render possible precautions less feasible or practical in the eyes of military commanders. For example, the plethora of data and analytical models developed to identify targets, and the comparably fewer resources dedicated to identifying and verifying civilian status with such sophistication, may lead commanders to underestimate the likelihood of a given precaution yielding humanitarian benefit. In addition, decisions at strategic and planning levels to invest money and resources in programs like SKYNET may, down the line, make precautions more expensive and less feasible at the operational or tactical level.
Fourth, is accountability. As The Intercept’s revelations and recent films like “National Bird” have powerfully highlighted, modern drone warfare is not simply remote-controlled killing. It is a complex system consisting of human pilots, analysts, and operators, interwoven with highly technical and data-driven technologies like SKYNET. Defenders of the drone program often highlight this web of analysts and decision-makers behind drone operations to assuage concerns. However, this may also complicate accountability. What does accountability mean when no one person or commander has his proverbial “finger on the trigger” but instead many individuals and several government agencies feed into a highly bureaucratized decision-making process? When complex data-driven intelligence is utilized, this diffusion of responsibility means determining accountability is even more difficult, and identifying where in that process mistakes amount to legal violations is a qualitatively different challenge than in more conventional uses of force.
Finally, on an intuitive level, one reason why revelations like SKYNET trigger alarm in some is the use of seemingly innocuous facts about people’s lives, such as sharing your mobile phone or travel to nearby cities, are combined and analyzed to make predictions about not so innocent behavior — potentially for the purposes of killing. That may strike some experts as techno-phobia, but that’s too dismissive.
From an IHL perspective, it is helpful to translate this anxiety into the perspective of the civilian on the ground. One purpose of IHL is to clarify, for non-combatants, what kinds of actions and associations could put them in harm’s way during armed conflict, and in turn, how they can better protect themselves. Drones generally upset this sense of predictability and ability to self-protect — reliance on programs like SKYNET to drive targeting decisions only amplify this concern.
As a legal matter, the revelations about SKYNET highlight the paucity of analysis on the concept of “doubt” in IHL and the need for more rigorous thinking about how to guard against the erosion of legal norms and properly interpret and apply IHL in an era of big data and intelligence-driven targeted killing.