March 16, 1968, is one of the darkest days in U.S. military history. On that day, the soldiers of C Company, 1st Battalion, 20th Infantry Regiment, who had suffered dozens of casualties in the campaign against the Viet Cong, assaulted the village of My Lai. Under the command of Lieutenant William Calley, the soldiers attacked the village based on faulty intelligence about the location of a Viet Cong unit. Instead of the expected enemy, they found local civilians, mostly old men, women, and children. In the end, a U.S. Army investigation found that C Company soldiers “massacred a large number of noncombatants” and committed torture, rape, and infanticide. The precise number of Vietnamese killed was between 175 and 500 people.
While it is difficult to believe, the My Lai massacre would have been worse had it not been interrupted by a U.S. Army helicopter crew led by then-Warrant Officer Hugh Thompson, Jr. Thompson witnessed the actions of C Company soldiers while circling above the village. At several points during the massacre, Thompson landed his helicopter to help the locals in an attempt to stop the killing, challenging Calley’s orders directly.
Now imagine a future battlefield, with soldiers as emotionally charged or misguided as those under Calley’s command. But on this future battlefield, Hugh Thompson’s counterpart might not be there. Instead, a drone will likely be flying overhead.
Could that drone play the same role as Hugh Thompson did in the My Lai massacre? This is a complex question military leaders must begin to confront.
While such a drone — let’s call it a Thompson drone — is not possible today, it is increasingly plausible. Infantry units are already training with drones to support ground assaults. Computer vision algorithms on drones are being advertised as able to distinguish unarmed civilians from combatants. And those same drones could use generative artificial intelligence (AI) to convey information about civilians or combatants to troops on the ground, either via text message or with increasingly realistic voices. It is therefore plausible that a future Thompson drone could be deployed to support military operations and intervene in a ground assault by communicating information in a way that could prevent or stop violations of the Law of Armed Conflict, the Geneva Conventions, and local rules of engagement.
Putting aside current technical capabilities and limitations of drones for the moment, our hypothetical Thompson drone should prompt us to consider the relationship between increasingly capable AI and humans engaged in armed conflict. What if AI could coach soldiers through difficult situations like My Lai? Does it mean we should consider AI to be not just a mere tool, but a coach instead? Or is “coach” too soft a concept? Could there be a potential role for AI as an “enforcer” of the principles regulating the professional conduct of armed conflict and the protection of civilians?
The discretion granted to drones and the relative agency retained by humans will determine whether an AI-enabled military system has the role of tool, coach, or enforcer. Future military commanders will increasingly face difficult decisions about employing AI as a tool, coach, or enforcer, and should thus think carefully about the ethical implications of each of the three roles.
Thinking About AI as a Tool or as a Coach
A dominant interpretation of an AI system is that it functions merely as a tool. AI can be directed to follow a specific order and serve a specific purpose set by humans for their own benefit. In this sense, it is like using a hammer to drive a nail into a wall, rather than using your fist. The remit is narrow and the goal well-specified. Examples of tools could include target recognition algorithms or AI-enhanced missile defense. If a drone was just a way to conduct aerial surveillance or to provide a communications relay, it too could be considered a tool.
Our hypothetical Thompson drone, however, would have a larger remit and be allowed to arbitrate among “local” or “global” goals. Local goals in this case are short-term objectives tied to missions, sub-tasks, or decision points within a larger operation. These are often instrumental steps toward achieving global goals — high-level, overarching objectives that guide an entire operation. The discretion to arbitrate among local or global goals involves shaping these objectives or determining their relative priority when they come into conflict.
For instance, in sending a message to Lieutenant Calley that directly conflicts with his actions or stated intent, the Thompson drone engages in the arbitration of both global and local goals. At the global level, it weighs the overarching objective of destroying the enemy and protecting friendly forces against protecting non-combatants, and makes a recommendation on their relative priority for the unit. At the local level, the drone must navigate more immediate tasks, like temporarily halting fire while reassessing information.
Regardless of local or global goals, an AI with the discretion to recommend a wide range of options is much more than a simple tool, especially when it involves shaping and arbitrating among goals. For example, an alarm clock can effectively interfere with a person’s desire to sleep more, at least when they’re sound asleep in the morning. Yet, we wouldn’t think that an alarm clock is more than a useful tool for simply waking up on time for work. By contrast, what if an AI-enabled alarm clock went beyond simple ringing and instead recommended you get a new job that starts later in the day? An alarm with such discretion would be more like a coach than a mere tool. Accordingly, we can think of AI with limited or low discretion over arbitrating goals as a tool and AI with greater discretion over tasks as a coach.
Human Agency and the Distinction Between Coach and Enforcer
While the discretion of AI may delineate its role as a tool from that of a coach, it is crucial to consider the role of a human in relation to the AI. Here, we must bring in the concept of human agency — specifically, the agency a human has to ignore or contradict an AI system.
First, to explain what we mean by human agency, recall the example of an alarm clock that can interfere with a human’s intent to sleep but can also be snoozed, turned off, or thrown at a wall. Similarly, our hypothetical Thompson drone could easily interfere with human decision-making by, for example, presenting overhead imagery of civilians or sending persistent alerts. The Thompson drone could even assess the mental state of the soldiers on the ground and tune its communications accordingly. If these messages can be ignored or countermanded, much like a personal trainer’s instructions to exercise can be ignored, then the human has a high degree of agency relative to the AI coach.
But what if the unit incurred some consequence for ignoring the Thompson drone’s guidance? For example, what if the Thompson drone records the local commander’s decision to disregard the information and reports the violation to a higher headquarters? In this scenario, the human still has agency, though less than if the drone were merely relaying information to the soldiers on the ground. In these situations where humans have less agency, it is not like ignoring a personal trainer’s advice. Instead, it is more like disregarding a coach who could pull you off the field or kick you off the team.
Consider for a moment, however, what happened in My Lai:
[Hugh Thompson] tried to explain that these people appeared to be civilians, that we hadn’t taken any fire and there was no evidence of combatants in that area. The lieutenant [Calley] told him to mind his own business and get out of the way. They were face to face, screaming at each other. Hugh came back to the aircraft … He said: ‘They are coming this way. I’m going to go over to the bunker myself and get these people out. If they fire on these people, or fire on me while I’m doing that, shoot ’em!’
What if the Thompson drone had threatened the same? This would, rightly, give many commanders and soldiers pause. Humans on the ground would not be able to ignore or override the command of a drone that is ready to shoot. In this case, such a drone with lethal ability to enforce rules goes beyond the role of coach.
From this analysis, we find that an AI system that can enforce certain actions or decisions is not a coach but more closely resembles an “enforcer.” The dividing line between the status of an AI as a coach or as an enforcer hinges on the question of human agency. Where the human retains sufficient agency to disregard the AI, the AI functions as a coach. Where the human does not have agency to disregard the AI, the AI functions as an enforcer.
Of course, there is no clear dividing line between agency and no agency. Rather, human agency exists on a spectrum. At one end of the spectrum is an AI that only responds when prompted and can be ignored or disabled at the will of humans. At the other end of the spectrum is an AI that threatens to shoot if you do not follow its order. In between these two extremes there is a spectrum of different types of human-machine interactions with varying levels of human agency. There are also questions about which external factors might influence the exercise of human agency. For example, will humans be more hesitant to exercise their agency to ignore an AI recommendation if it is delivered through a human-like synthetic voice and anthropomorphized design, rather than a simple message displayed on a screen? Will humans be more inclined to heed the warning coming from a system labeled as an “expert advisor” than one called a “support tool,” even if it is an identical system? Would a persuasive chatbot reduce human agency? These are important questions, but beyond the scope of our analysis here.
Implications of Treating AI as a Tool, Coach, or Enforcer
U.S. military culture is accustomed to human coaches and technological tools, not technological coaches for human tools, much less AI systems as enforcers. In establishing an AI system as a tool, a coach, or an enforcer, militaries will be making choices that either conform with these cultural norms or begin to shift the norms entirely. Not all of these choices will be easy or straightforward.
Using AI as a tool to repel an incoming missile strike is a straightforward decision with historical precedent. The Department of Defense has established a policy on using autonomous and semi-autonomous functions in weapons systems. The narrow discretion allotted to the AI in tool-like employments can afford the human greater control over actions taken in specific conditions.
Granting wider discretion to AI, especially in shaping local and global goals for military operations, is more novel than using AI as a tool, though also not without precedent. For example, many U.S. servicemembers use apps that coach them toward fitness goals with motivational prompts. While less widespread, there are examples of emerging AI coaches. The Intelligence Advanced Research Projects Agency’s REASON project, for example, prompts intelligence analysts to seek out specific evidence to substantiate their conclusions or to consider alternative explanations. Moreover, a new class of AI-enabled decision support systems are emerging to coach military commanders through the decision-making process, especially at the operational level of war.
Further, as the sci-fi My Lai scenario suggests, there may be cases where AI could be used as an enforcer. It is unclear whether such a decision would be effective in reducing civilian harm — a topic worthy of investigation before our sci-fi scenario becomes reality. That said, no commander wants to be responsible for the willful killing of civilians. The My Lai massacre brought deep shame to the Army — which tried to cover it up — and became a global scandal still studied at military academies as a cautionary tale. The prospect of an AI backstop that a military commander could use to prevent or interrupt such a disaster has obvious appeal from both humanitarian and operational perspectives.
We have already shown how increasing AI discretion and reducing human agency shifts AI roles from tool to coach to enforcer. U.S. military leaders should therefore consider how AI system design and employment choices reflect their desire to use each role and their comfort level with the implications of that choice. Instructors in leadership and ethics should prompt conversations in their classrooms about what military leaders at all levels ought to consider in employing AI, including the potential implications for human agency. Theorists and ethicists, informed by AI researchers and developers, should offer their thoughts on the practical and ethical tradeoffs that commanders must consider between AI enforcement and servicemember agency. Program managers should reflect on ways to enable command discretion with respect to these technologies through training and human factors design choices. Technologists should consider their roles in both supporting these discussions and designing AI solutions and interfaces that will ethically serve operational goals. There is no clear line between “human agency” and “no human agency.” Even when AI coaches are easily ignored, the messages they send will influence human decision-making. That influence could be minor — a text message displayed on a screen — or more forceful, as in a loud voice on a radio or an emotionally manipulative message. These will be design choices made by technologists and commanders, and both should keep in mind the effects of such design decisions on the tool, coach, or enforcer framework.
Conclusion
Some may find a fictional Thompson drone which can override military orders under certain circumstances as infeasible because of long-established U.S. military principles of command, delegation, and human autonomy. However, there has been a steady diminution of individual unit independence and autonomy, going back to the installation of radios on navy warships and continuing through to the live streaming of combat operations.
Beyond the United States, our scenario may be even more realistic. Consider the Russian military’s embrace of algorithmic warfare and the struggles of China’s People’s Liberation Army in establishing competent and independent mid-level leaders. In both cases, there are signs that senior leaders might try to rely on technical systems to avoid relying on lower-level soldiers, who may lack training or good judgement.
Moreover, AI playing the role of a coach does not necessarily imply less agency for troops on the ground. The unit can still fully retain the ability to ignore or override recommendations made by an AI coach. In fact, the inclusion of an AI coach that can more effectively shape and arbitrate local and global goals might offer a way to support, rather than undermine, the continued exercise of autonomy and agency by the unit. What constitutes coaching and what might be considered AI manipulation is difficult to determine. Similarly, what might constitute human agency? Or is consent to deploy with an AI overseer sufficient to be considered an exercise of agency? These questions require further research and exploration.
That said, AI functioning as merely a coach may not be sufficient to prevent catastrophes like My Lai. What if a local commander ignored the Thompson drone’s order to stop attacking civilians, or fired on it to be rid of its pestering? To more effectively prevent a massacre, should humans be prevented from ignoring the Thompson drone?
The answers to these questions depend on the effectiveness of the drone in shaping outcomes. At a point, however, the ability to shape outcomes could come into tension with the agency of humans. In this sense, the military could face a critical tradeoff between enhancing operational effectiveness on the one hand, and preserving human agency and judgments on the ground, on the other. We aren’t advocating that the military grant AI greater discretion or more authority simply to enhance operational effectiveness. Rather, our aim is to encourage military leaders to carefully consider how their decisions to employ AI might affect the independent decision making of their service members.
Emelia Probasco is a former naval officer and senior fellow at Georgetown University’s Center for Security and Emerging Technology, where she studies the military applications of AI.
Minji Jang is a postdoctoral fellow at Georgetown University’s Kennedy Institute of Ethics (Ethics Lab) and Tech & Society Initiative. She holds a Ph.D. in Philosophy from the University of North Carolina at Chapel Hill.
Image: Senior Airman Paige Weldon
Jesus Christ is King
