Few phrases dominate debates about autonomous weapons more than “meaningful human control.” It has become a central topic in diplomatic forums, academic discussions, and civil society campaigns. States negotiating at the United Nations Group of Governmental Experts on lethal autonomous weapons (GGE LAWS) have invoked the phrase regularly, and advocates see it as the minimum requirement for regulating AI-enabled autonomous weapons systems (AWS). Despite its prominence and controversy, the phrase is often considered vague, impractical, and misunderstood. What exactly does meaningful control mean? What is the threshold for meaningful? What does control look like and entail? And, importantly, who exercises it, when, and under what limits?
In a recent three-part series in International Law Studies, I provide some answers (and more questions) about human control. The analysis views human control not as a single act at a moment in time by one person—such as pressing a trigger or turning a system left or right—but as a series of distributed, embedded human judgments throughout a system’s lifecycle. Each article examines the three types of actors with distinct roles in the human control framework—software designers/developers, commanders, and operators—all of whom make decisions that influence whether an AWS can be used legally and responsibly. Each group faces unique technical, cognitive, legal, and operational challenges across three stages:
- Design and development decisions: where engineers, data scientists, and software designers set the parameters through, inter alia, data training, software architecture, and interface design.
- Command decisions: where commanders authorize deployment, establish mission parameters, and impose operational constraints.
- Operator decisions: where individuals in the field guide, observe, and terminate systems to avoid unintended outcomes.
Each stage involves distinct forms of human control. Treating them as part of a continuum reveals both the opportunities for embedding judgment and the risks of assuming that human control exists when, in practice, it may not.
Control Starts with the Code
The design and development phase has received more attention in the AI context than in the development of traditional weapon systems, for obvious reasons. Decisions made during this phase have a significant impact on the ultimate performance of an AI system. In the case of AI-enabled weapons, international humanitarian law (IHL), or the law of armed conflict, begins to apply well before a weapon is deployed. Article 36 of Additional Protocol I (AP I) to the Geneva Conventions requires States to determine if the employment of a new means or method of warfare would, in some or all circumstances, be prohibited by AP I. In this context, fulfilling that obligation is impossible without scrutinizing design choices.
Software developers and engineers are responsible for key decisions, including the selection of machine learning techniques, the compilation and cleaning of training data, and the design and presentation of data at the operator interface. Each of these choices carries legal implications. For example, a system trained on unrepresentative or biased data may systematically misidentify targets, undermining the principle of distinction. In another example, a poorly designed interface may result in cognitive overload for operators or inadvertently bias their decisions toward a particular outcome. Clearly, at this stage, recognizing how developer decisions are opportunities to embed human judgment is a critical piece of a larger puzzle for harnessing human control, identifying foreseeable risks or shortcomings, and implementing responsible AI.
Human control, therefore, starts with the code. If other stakeholders—notably legal advisors—are not involved from the beginning, important decisions about compliance with IHL are essentially delegated to technical design choices. Bringing in multidisciplinary expertise early on guarantees that data curation, algorithm development, and testing procedures align with both technical instruments and States’ legal obligations under IHL.
Commanders and the Weight of Deployment
Commanders serve as a crucial link between design and deployment. Commanders determine whether to deploy an autonomous system, the conditions under which it will be deployed, and other associated constraints. Despite holding such an important decision-making role, the opportunities commanders have to incorporate human judgment are often overlooked within the broader human control debate.
Commanders can exercise human control through procedures such as testing and training, setting mission parameters, and developing rules of engagement. For example, an AWS might be authorized to operate only within a geographically-defined area, against specific categories of military objectives, or for a limited period of time. These constraints are intended to mitigate the risk of civilian harm and define the conditions for optimal system performance.
However, commanders rarely have complete knowledge of how systems will perform or how an environment may change over time. No testing regime can predict every possible outcome, especially against adaptable or near-peer adversaries. Therefore, commanders need access to essential information about the risks of system failures and malfunctions under specific conditions to understand these limitations during planning. Uncertainties will always be present, and commanders must use their judgment to balance these uncertainties with operational demands.
Additionally, at a tactical level, commanders will be responsible for implementing maintenance and monitoring regimes. This includes unit training with new systems, overseeing system updates and testing and evaluation of deployed systems, and, if necessary, legal reviews of substantially altered systems.
Commanders thus carry the burden of embedding human judgment through procedures such as testing systems at the edge, training units responsible for new weapon systems, operational and mission planning, and maintenance and monitoring. By defining mission constraints and ensuring that system capabilities align with legal objectives, they exercise a form of control that can be more impactful than last-minute interventions.
Operators at the Edge
Finally, operators are often portrayed as the ultimate safeguard against failures in AWS. Public debate tends to assume that their ability to authorize or abort engagements is the essence of human control. But in practice, their position may be far less impactful.
Cognitive considerations and limitations are key issues in harnessing human control with operators. Operators face challenges such as automation bias, vigilance fatigue, and cognitive overload. Furthermore, systems may operate at machine speeds difficult for humans to keep pace, or they may monitor for long periods, leading to cognitive drift. Interfaces that hide uncertainty or limit override functions exacerbate the issue, making intervention more symbolic than substantive.
Despite these challenges, operators remain essential. And importantly, operators have unique functions separate from developers and commanders—operators guide, observe, and terminate.
Operators guide AWS by authorizing updates or adjustments to ensure alignment with command intent and legal constraints. They observe system behavior in real-time, providing oversight that machines alone cannot achieve. Additionally, they terminate engagements when the system’s actions risk violating IHL or diverging from mission parameters. These functions are unique to operators, making them the final—but not the only—safeguard of human judgment.
Across design, command, and operation, human control is distributed rather than centralized. No single actor bears all responsibility. Instead, a chain of embedded judgments is incorporated into AWS.
Policy Implications
The lifecycle approach demonstrates that States and militaries cannot rely on vague assurances of human control in AWS; they must specify where, when, and by whom judgment is exercised. Four policy recommendations result from this:
1. Harness embedded judgment throughout the lifecycle of AWS.
A key idea connecting these stages is embedded human judgment. Instead of viewing human control as a final override, this framework highlights that human decisions are woven throughout an AWS’s lifecycle, and a comprehensive human control policy should harness that timeline.
During design, judgment is embedded through decisions like data selection, machine learning techniques, and the construction of testing regimes, as well as interface design. These decisions determine parameters that will guide a system’s performance in a combat environment. At the command level, judgment is embedded through processes like mission planning and rules of engagement, which shape the system’s operating environment and permissible targets. In operations, judgment is embedded through operator functions of guiding, observing, and terminating, which are critical to mitigating risk and unintended outcomes.
2. Integrate legal and technical design choices.
Recognizing embedded judgment highlights the significance of technical considerations in shaping legal decisions. Data selection, system architecture, and verification methods directly affect whether IHL principles can be upheld. States must actively harness that technical-legal relationship through greater discourse, cross recruitment, and training.
3. Enhance command accessibility and responsibility.
Commanders at all levels must understand their role within the larger framework of this emerging capability. However, policymakers also need to be realistic about their expectations of commanders. There are many calls for commanders to be more tech-savvy in order to understand or accept responsibility for their use of AI. Commanders do not need to code to understand the risks of a specific weapon system, just as we expect for non-AI weapon systems. For example, commanders are not expected to have an in-depth understanding of the physics behind bullets or bombs. Subject matter experts will be able to provide that necessary information. Nonetheless, commanders must have a thorough understanding of the risks associated with particular systems to establish and maintain acceptable environments for their use, monitoring, and maintenance.
4. Maintain perspective on the limited role of operators.
Operators are part of a far larger network of human control. And in some ways, they have the least amount of ‘control’ relative to other actors within the lifecycle. Future policies surrounding human control must acknowledge their limited functions and utilize earlier stages to mitigate biases or other limitations that affect operators in their guiding, observing, and terminating functions.
Looking Ahead
The framework outlined above provides a foundation for further inquiry, but several critical questions must still be addressed. What levels of autonomy are appropriate for specific use cases? How can we truly operationalize “meaningful” control? Can States converge on common standards for testing, constraint-setting, and oversight? Are there legal or operational implications of States with different or contrasting approaches to human control?
This lifecycle perspective shows that human control must be more than a policy slogan; it should drive practical actions. What matters is who makes judgments, when, and how. Maintaining and utilizing embedded human judgment throughout the lifecycle is vital for ensuring that autonomous weapons are developed responsibly and used lawfully. Identifying where that judgment exists is crucial for the lawful and responsible use of emerging military technologies.
