Introduction

The 21^st-century military landscape is transforming rapidly, with armed forces in many parts of the world beginning work on modernisation strategies that seek to leverage technological developments such as artificial intelligence (AI), autonomous systems, and robotics. While the motivations behind the various national strategies may differ, most modern militaries are looking to adopt Lethal Autonomous Weapons Systems (LAWS)—which are capable of detecting, selecting and engaging targets without human intervention.

By nature, LAWS are morally complex and paradigmatically unfamiliar. As such, their emergence has elicited calls for nurturing international norms and regulations that will guide their development and use. Consequently, the United Nations Group of Governmental Experts (UN GGE) on LAWS was established in 2016, after three years of informal expert meetings under the auspices of the Convention on Certain Conventional Weapons (CCW). The GGE’s most recent meeting took place in August 2018, focusing on the human element in the development and deployment of LAWS.^[i]

The GGE’s Report outlines the following three basic policy guidelines: ^[ii]

1. Human accountability cannot be transferred to machines.
   Machines and human beings cannot be treated the same way under the law. Humans are accountable at all stages of the development, deployment and use of LAWS.
2. States are responsible for physical and non-physical safeguards for LAWS.
   States must take measures to secure their weapons systems against theft, damage and cyber attacks by other state and non-state actors.
3. International humanitarian law (IHL) is applicable to the development, deployment and use of all emerging weapons systems.

These guidelines are a balancing act between competing interests inextricably tied with power dynamics, whether these be geopolitical considerations, institutional interests, or humanitarian concerns. Twenty-six countries have called for a ban on LAWS, yet at the same time, four of the P-5 have explicitly rejected such a ban.^[iii]China called for a ban on the use but not the development of such systems. Some analysts see China’s move as a diplomatic stalling tactic given that it is actively pursuing lethal autonomous technologies of its own.^[iv] The GGE’s guiding principles therefore reflect the ground reality that the most powerful states will continue to pursue LAWS long before other powers reach any consensus on stringent international regulations.

This paper examines the possible use cases of LAWS for middle powers, and how the UN GGE’s human accountability tenet specifically would shape such systems. It concludes with some recommendations for the Indian context. The focus on human accountability is due to its being the most significant of the GGE’s recommendations, with the others serving as corollaries. That is because IHL does not have provisions for accountability within autonomous entities; only states, and to some extent individuals, can be held accountable under its provisions.^[v] In turn, the human accountability tenet will shape how and in what contexts LAWS can be deployed. This analysis focuses on middle powers because, although global LAWS discourse tends to centre on the ‘AI race’ between great powers, other countries are following developments in this field, guided by their own context. Thus, while the US may want to develop LAWS to maintain their overwhelming military edge, and China would want to leverage AI to challenge US dominance, the motivations for a middle power like South Korea or Singapore are to gradually close the gaps left by their dwindling military manpower and enhance their capabilities in a strategically challenging neighbourhood. Similarly, the Indian military requires capabilities suited to its diverse geographies and theatres of conflict.

The security benefits of LAWS

Middle powers operating in complex, multiple theatres of conflict within their own borders see the following utilities in developing and deploying LAWS.

Force multiplier and contested borders

LAWS can augment the response times and effectiveness of border forces. Several middle powers, particularly in Asia, share contested borders and volatile security environments. Of the total number of militarised territorial disputes globally, 34 percent are in Asia.^[vi] For such states, LAWS present an opportunity to effectively police borders and respond to skirmishes.

For countries that have a heavily militarised border, and are undergoing a demographic crunch, AI-enabled systems are a pathway to maintaining border security in the absence of a sufficiently large standing military or border police force. For instance, South Korea’s birth rate hit a record low this year, and its population is projected to fall by 13 million or 25 percent by 2060.^[vii] At the same time, its border with North Korea is one of the most heavily militarised in the world and is littered with an estimated 1.1 million mines planted by both sides, making it dangerous for soldiers to patrol.^[viii]

Meanwhile, for countries with no shortage of manpower, autonomous systems can supplement human capabilities, and improve work conditions for soldiers on the ground, thereby improving the force’s overall effectiveness.  These features would be particularly useful for border patrol forces like India’s Border Security Force (BSF), the largest force of its kind in the world with over 2.5 lakh active personnel.^[ix] BSF personnel endure inhospitable conditions for extended periods under considerable duress and risk to their lives: the Home Ministry reported that there were 529 suicides and 491 killed in action between 2012-2018.^[x]

These forces can benefit, for example, from static and mobile autonomous surveillance units that should provide high marginal utility for policed borders like the LOC or DMZ. ^[xi]   Such surveillance units have been deployed by the US Customs and Border Protection at the southern border and by Israel in area bordering the Gaza Strip. They can feed into networked systems which would “sift through visual data and other intelligence, and direct it to humans to take the appropriate actions.”[xii] AI technology firms agree, and say that at the most basic level, “you could make use of autonomous robotic vehicles capable of difficult terrain to patrol, flag anomalies, and report those anomalies.”^[xiii]

Surveillance and detection are therefore the low-hanging fruits of autonomous capabilities for border security for middle powers. In this vein is the BSF’s pilot electronic surveillance programme—called the Comprehensive Integrated Border Management System (CIBMS)—which seeks to integrate thermal, radar and aerostat capabilities into a unified system.^[xiv] The system has immense potential in improving the ability of the BSF to detect and monitor threats at the border by broadening the range of actionable information available to decision-makers. However, CIBMS is far from being fully realised, held up as it is by delays, insufficient border infrastructure, and lack of technically skilled manpower.

In cases of border conflict, a network of air, ground and maritime autonomous surveillance systems and lethal autonomous units can give states an early edge, especially against asymmetric tactics. For instance, Pakistan’s New Concept War Fighting strategy – developed as a counter to India’s Cold Start Doctrine – relies on rapid mobilisation and tactical nuclear weapons.^[xv] An autonomous conventional defense system consisting of armed units can use information gathered by a networked surveillance system and potentially eliminate Pakistan’s escalatory tactical nuclear arsenal before it can be used.

Integrating principles of human accountability into a border security system can take two forms: human-in-the-loop and human-on-the-loop.  An example of a human-in-the-loop is South Korea’s SGR-A1 sentry gun, deployed along the DMZ. The robot has two modes: human-supervised, where it alerts a human operator to seek authorisation to engage a target, and fully autonomous, where no human authorisation is required. In either scenario, the SGR-A1 can issue verbal warnings and recognise physical cues of surrender, including the raising of one’s arms and dropping one’s weapon.^[xvi] After criticism grew around the fully autonomous mode, South Korea  operationalised the permanent inclusion of a human-in-the-loop, that is, a human operator decides whether to fire.^[xvii]  Effectively, the SGR-A1s “gives a single human operator control over several guns”, meaning they act as a force multiplier.^[xviii]

A human-on-the-loop system would be closer to the definition of fully autonomous: it can detect, select and attack without human intervention, unless a human operator intervenes. Such a system, while fraught with ethical issues, would provide substantial advantages in terms of improving the speed of response. During a border incursion, such a system could serve as a first line of defence, and even as a deterrent by raising the immediate costs for the attacking actor.

While the exact dynamics and viability of autonomous systems in border security will need to be explored further, they are a plausible tool for preventing and de-escalating border conflicts.

Reducing human costs in urban theatres of conflict

Fewer boots on the ground, paired with continued improvements in image recognition and pinpoint-accuracy in targeting can greatly reduce both civilian casualties and harm to friendly forces. All these effects can help resolve a complex issue: (democratic) governments around the world seek to avoid the humanitarian outcry and resultant damage to their international reputation that arise from civilian casualties, and because such deaths adversely affect the state’s credibility with local authorities and citizens.

Urban theatres of conflict present numerous challenges to conventional militaries: they are crowded with structures that hinder situational awareness, and civilians are frequently caught in the crossfire between unfriendly and friendly forces. For most middle powers, these urban theatres are within their own territories.

For instance, India faces a protracted conflict in Kashmir, where civilian casualties jumped from 15 in 2016 to 40 in 2017, making it the deadliest year since 2010. The same year, 75 police officers and soldiers died in the line of duty.^[xix] In Kashmir, the stakes are high in the ability to discriminate between civilian and combatant, because the affected civilians are often Indian citizens and the cooperation of the local population is crucial for the success of counter-insurgency operations.

The Philippines recently undertook a long and bloody battle with ISIL-affiliated foreign fighters in Marawi, which displaced 98 percent of the total population of the city, many of whom have been unable to return.^[xx] The town, according to a 2018 news report, remains in ruins and is contaminated by 500 lbs of explosive remnants of war dropped by the Philippine air force.^[xxi]

Autonomous systems have promise in improving situational awareness and protecting friendly forces as well as civilians. Some of the utilities outlined by Paul Scharre of CNAS are: one, a distributed swarm that creates a protective cloud around friendly forces on the ground; two, providing logistical support to ground forces; three, evacuating wounded soldiers; and four, shooting down enemy drones.^[xxii]

Thus, an urban conflict can be visualised in the following way: a swarm of surveillance drones is sent in to map the area and detect concentrations of people. Civilian structures such as hospitals and schools are identified and tagged for protection by the swarm. Once a human commander has the necessary information, parsed by the autonomous surveillance system, they can make the necessary adjustments to their operational plans. Friendly forces enter the city, shielded from aerial or enemy drone attacks by a swarm of armed drones. The forces are kept well-supplied with munitions and medical provisions by Unmanned Aerial Vehicles (UAVs) equipped with non-lethal means of defence to prevent supplies from falling into enemy hands. Injured soldiers and civilians caught in the crossfire can be escorted by armoured Unmanned Ground Vehicles (UGVs).

The component technologies are already available or have been tested in laboratory conditions. For instance, in 2015, ETH Zurich successfully tested an AscTec Firefly drone that can autonomously navigate a structure and create a 3-D map of the unfamiliar environment.^[xxiii] The same lab has also built micro-helicopters which, at a diameter of a mere 10 cm, are far more unobtrusive than the Firefly.^[xxiv] Image and object recognition have also made significant strides: in 2018, researchers from Sony trained an AI algorithm to 75 percent accuracy using ImageNet/ResNet-50 in around four minutes.^[xxv] As SparkCognition’s Amir Husain says, “Essentially, despite the seeming complexity of the scenarios described [in this paper], the defense technical base, particularly in the US, and leading AI companies already possess a wealth of technology that could fundamentally change how these situations are approached and alleviate long-standing challenges.”^[xxvi] However, even with all these developments, a fully autonomous system in an urban theatre is a stretch of imagination as the state of machine learning is still not robust or reliable enough. LAWS can only feasibly be deployed in conjunction with human forces, serving to supplement and enhance their capabilities and situational awareness.

Human accountability is both, more essential and more challenging in an urban theatre in comparison to a militarised zone. Being able to hold human operators, commanders and leaders accountable for civilian casualties is a key safeguard to ensure that an autonomous system does not allow policing forces to sidestep issues of responsibility. Provisions for accountability would also need to be integrated into national laws to prevent misuse, such as in cases of non-violent dissent. However, the sheer volume of decisions human operators would need to make to remain in-the-loop in a crowded, hostile urban battlefield could reduce the efficacy of such a system in protecting friendly forces and civilians, where mere seconds can make a massive difference. Skilled human operators can play a supervisory rather than deciding role to tread this balance. Thus, while there is promise in deploying LAWS to reduce civilian and military casualties in urban theatres of conflict, rushing ahead with such a system without the necessary policy and operational infrastructure could end up doing more harm than good.

Defence-in-depth through persistent presence in the maritime domain

LAWS can help maintain persistent presence in areas that are difficult to monitor due to risks arising from climate, vast or difficult terrain, or unexploded ordnance, enabling defence-in-depth even with complex geographies.

The Indian military is engaged in a diverse range of theatres: mountains to the north, deserts to the west, and India’s island territories and oceanic borders. The 2017 Joint Doctrine of the Indian Armed Forces highlights the centrality of all-terrain capabilities in securing India’s territory:

“[U]ndertaking ‘Integrated Theatre Battle’ with an operationally adaptable force, to ensure decisive victory in a network centric environment across the entire spectrum of conflict in varied geographical domains, will be the guiding philosophy for evolution of force application and war fighting strategies.”^[xxvii]

An example of this complex geography is the Indian Ocean, which has both connected India to global maritime routes, and also made the country vulnerable to sea-borne threats. In the immediate maritime territory surrounding its coast, India faces the threat of sea-borne terrorism as epitomised in the 26/11 Mumbai attacks. In its wider Exclusive Economic Zone, the Indian Navy must contend with piracy and naval incursions by hostile states.

For its part, Indonesia’s geography challenge is policing its 16,000 or so islands.^[xxviii]Its coastline is the third-longest in the world.^[xxix] Cognisant of the centrality of the maritime domain in Indonesia’s security, President Joko Widodo announced the ‘Global Maritime Fulcrum’ (GMF) policy during his speech at the 2014 East Asia Summit. In 2016, Jokowi elaborated:

“Our strategic position between two oceans and as the fulcrum of the global shipping route, is an important factor in maintaining security and safety of international shipping navigation, especially in the sea intersection that links West-East trade routes and North- South trade routes, where more than 60,000 ships trespass our seas every year,”^[xxx]

The challenge presented by vast maritime territories—as well as other difficult-to-police terrains such as mountains and forests—is maintaining a continuous presence. A defending military force can only cover a portion of these territories at a time, giving an intruding actor multiple points of entry before they can be detected and countered. In the maritime sphere, autonomous systems can enhance maritime domain awareness (MDA) and serve as an effective deterrent against incursions. For instance, the Indian Navy inaugurated the Integrated Underwater Harbour Defence and Surveillance System (IUHDSS) in 2016.^[xxxi] Currently operational in the port cities of Vishakhapatnam and Mumbai, this Israel-made multi-sensor system (radar, electro-optic cameras and sonar) can identify, track and generate warnings for underwater and surface threats near the coasts. However, the system cannot act against these threats. An autonomous system along the lines of the IUHDSS can deploy armed Unmanned Underwater Vehicles (UUVs) and UAVs to conduct additional reconnaissance of detected objects, issue verbal warnings where necessary, fire warning shots or otherwise engage the target without lethal force. If enough actionable information is available, the autonomous system could potentially use lethal force to destroy a vessel, after which manned vessels could be sent in to detain the passengers. In doing so, an autonomous IUHDSS-like system can one, deter intruding civilian vessels, as in Indonesia’s case with Chinese fishing vessels; or, two, provide the first layer of defence in case of an imminent threat while information is relayed via the chain of command.

Another useful application of autonomous systems would be in detecting and eliminating mines. The Indonesian Navy, for instance, deployed a STERNA UAV in July 2018 to detect and assess underwater mines.^[xxxii] In India, minesweepers are essential in maintaining the safety of its maritime routes. Currently, India only has two functional minesweepers for the entirety of its maritime territory. The Indian Navy could take a future-oriented approach and upgrade to an autonomous minesweeping system that can remove mines without endangering any human crew.

LAWS in naval and coastal defence is perhaps the lowest-hanging fruit for maritime states and is a pressing need due to the uncertain security environment in the Indo-Pacific. Human accountability can be locked into maritime LAWS in the lethal force stage, in that a human with the necessary authority must decide at their own discretion after being presented with actionable information, since consideration of diplomatic issues cannot be feasibly integrated into an autonomous system.

As this section has elaborated, there is immense potential in deploying autonomous systems within each middle power’s unique strategic and operational context.  Although the UN GGE’s human accountability clause could in some cases restrain the effectiveness of LAWS, it can work toward the overall goal of reducing casualties and de-escalating conflicts. It is also a necessary element to ensure countries do not find themselves on a slippery slope toward war without consequences. At the same time, the defence and policy ecosystems in most middle powers lack crucial elements that they need to form the base upon which such capabilities can be built. 

Recommended policy measures

This paper makes the following policy recommendations. While they build on the peculiarities of the Indian context, these can be adapted and applied to other national contexts as well. 

1.  Invest in talent needed to build and use Autonomous Systems suited to the needs of the military.

India suffers from a shortage of talent needed to develop and secure AI systems. A report by Chinese technology giant Tencent estimates that while millions of roles are available in the field of AI, there are only 300,000 AI practitioners and researchers worldwide. In India, the shortage is compounded by its inability to develop and retain top talent: the 2017 Global Talent Competitiveness Index ranked India 92^nd out of 118 countries in its ability to attract talent.^[xxxiii]

Experts highlight the following skills as necessary for building an AI workforce in a developing country: digital literacy to use platforms such as Amazon Web Services or Mechanical Turk, deep learning and robotics.^[xxxiv] Other core skills needed to build AI algorithms include computer science fundamentals, probability and statistics, as well as data modeling and evaluation.

Private-sector actors can play a lead role here by instituting programmes to impart AI skills to India’s STEM workforce. Tata Consultancy Services, for instance, has announced that it would sponsor a Centre on Intelligent Systems in IIT Hyderabad, which will soon begin offering an AI Executive Programme to bridge the talent gap.^[xxxv] The three branches of the Indian military should forge partnerships through such programmes as well by having arrangements with educational institutions to have a military pathway, whereby aspiring AI talent can learn the basics of military operations in order to gain an understanding on how AI technologies can be used in a military context. SparkCognition’s Amir Husain identifies the following as a pressing need in military AI skilling: “At the moment what every country needs is a cross-pollination between military experience and technology – specifically software technology expertise. These have traditionally been two fundamentally separate silos of talent, but innovation with AI in the military requires collaboration between these skill sets and a rethinking of strategy and tactics in light of what technology makes possible.”^[xxxvi] By establishing this collaboration through educational programmes, the Indian military can leverage AI talent to meet its operational and strategic needs and adapt accordingly to the changing technological context.

2.  Create Military Futures Commands to identify long-term strategic acquisitions and forge lasting partnerships with domestic private defence manufacturers.

India’s defence acquisitions have historically been dominated by imports from Russia and Israel, as well as state-owned enterprises (SOEs). However, in the field of AI, the growth in speed and capability has risen exponentially and will continue to do so. As an example, researchers from Sony have trained an image recognition AI algorithm to 75-percent accuracy using ImageNet/ResNet-50 in around four minutes; the same task, in 2015, took 29 hours.^[xxxvii] Additionally, it will become necessary to be able to quickly integrate developments in the technology by the private sector into the military, something that the often slow and cumbersome SOEs and military procurement process are ill-equipped to undertake. Furthermore, India must foster a competitive domestic defence industrial base to eventually transition from being a buyer of autonomous systems to a producer. This transition is crucial because as the country begins to employ LAWS and AI systems in diverse operational contexts, systems must be shaped by particular needs, and not the other way around. The Defence Procurement Procedure (DPP) 2016 was an attempt to move in this direction by integrating elements of the ‘Make in India’ policy into defence procurement.

Commodore (Retd.) Sujeet Samaddar, Senior Consultant at NITI Aayog, who served as commodore with the Indian Navy for 31 years, identified the core issue in the defence procurement process: lack of balancing between force modernisation and building national capability.^[xxxviii] The issue manifests in the Indian military in the form of “an input-specification based approach to acquisition requests as opposed to a focus on outcome requirements”—i.e., the military branches often narrow their available options by focusing too heavily on what goes into making a weapons system as opposed to the goals they hope to achieve through them.

The immediacy of demand for weapons systems also frequently results in the breaking up of RFPs for procurement, leading to piecemeal acquisitions as opposed to those that are future-oriented and strategic. “This,” he says, “prevents the kind of demand aggregation needed to generate the scale and size for creating a pipeline production model so that force modernisation also builds the national defence industrial base.” The piecemeal approach is in part due to military commanders wanting to acquire and deploy based on immediate ground requirements, and in part to account for the long delays due to the procurement procedure itself that frequently clog up the process. For instance, the Indian Navy has attempted to acquire minesweepers thrice over the last decade, including through domestic production using the government-owned Goa Shipyard, and a contract with South Korea’s Kangnam Corporation. In the meantime, its existing minesweepers fell apart, and the Navy currently has none left in its fleets.^[xxxix]

The three military branches should set up dedicated Futures Commands that will take a long-run approach to defence acquisitions and should focus on emerging technologies in the field of autonomous systems.^[xl] It is crucial that such a Command account for the diversity of perspectives reflected in academia, the private sector and government, as any exercise in military modernisation would need to consider the social and ethical impact of the technologies it seeks to adopt.

The Future Commands, in addition to continuing relationships with government defence manufacturers and DRDO, must foster long-term ones with private defence manufacturers as well. This could take the form of periodic ‘challenges’ where the Command provides a specific problem statement and invites manufacturers to develop and demonstrate prototypes. Military personnel could then test these prototypes and provide feedback to the manufacturers on the spot. A second mechanism to develop long-term ties with private industry would be through consultation mechanisms whereby the Futures Commands bring in technical experts to brief the Command on the current state of AI, machine learning, robotics and allied technologies and brainstorm on possibilities for use in a military context.

3.  Institute Committees on Human Accountability within the proposed Military Futures Commands

As cited in the UN GGE’s report, Article 36, Additional Protocol of the Geneva Conventions lays out the fundamentals of state accountability in the development of weapons systems. India, however, has not signed any of the additional protocols, ostensibly due to reservations about the classification of self-determination movements, insurgencies and resistance to alien occupation as international armed conflicts.^[xli] However, even in the absence of these protocols, India can institute policies and practices that address accountability issues.

The Military Futures Commands should formulate guidelines and principles on accountability in autonomous systems that underpin their strategies for acquisition of such systems. Basic guidelines could include permissible cases for use of lethal force in non-conflict scenarios (such as naval patrol), as well as a requirement that all AI systems that are acquired by the Indian military undergo some form of Failure Modes and Effects Analysis (FMEA), which focuses on the ways in which a process or product can fail and seek to limit such failures. In the case of AI systems, this would include protocols for scenarios where these systems undergo (internal) critical failures or are under external attack (cyberattacks, adversarial imaging, among others).

Conclusion

• Media and Internet
• Internet Governance
• Africa
• India
• East Asia
• West Asia
• AI TECHNOLOGY
• Artificial intelligence
• GGE
• LAWSLETHAL AUTONOMOUS WEAPONS SYSTEMS (LAWS)

The views expressed above belong to the author(s). ORF research and analyses now available on Telegram! Click here to access our curated content — blogs, longforms and interviews.