Introduction

Artificial Intelligence (AI) in medicine relies on an ecosystem of health data to train machines that learn responses to diagnose, predict, or perform more complex medical tasks. Patient data is leveraged for supporting clinicians in decision-making, bringing to the fore patterns in the data that were not discernible to a clinician’s eyes, and in some cases even charting out medical prognosis. Its uses have been well documented: electronic health record (EHR) systems have used machine learning algorithms to detect data from text^[1] as well as undertaking predictive analysis to warn clinicians about high-risk conditions and co-morbidities.^[2] This is in addition to guiding drug discovery^[3] and more topically, allowing population-analysis for pandemic preparedness and response measures.^[4]

The Indian government has been trying to nudge the health system towards greater overall digitisation for the last two decades. Frontline health workers are being trained to adopt digital health: moving from paper-and-pen-based entries that are transferred to a centralised digital portal, to now using mobile-phone applications that allow real-time information upload.^[5] The shift to digitisation has been codified in the National Health Policy (2017) and is represented in the National Health Stack vision, detailing the need to leverage technologies such as Big Data analytics for data stored in universal registries.^[6] The National Digital Health Blueprint (NDHB 2019) further builds on this vision to identify building blocks that leverage foundational technology towards expansive application development for varied uses and rely in a most rudimentary manner on high data integrity of the health system.^[7]

While digitisation is a promising first step to creating interoperable digital systems, there are plenty of challenges to its adoption. EHR adoption, for example, has been laggard in public health institutions due to its high cost of implementation and high burden on clinicians owing to cumbersome input and maintenance procedures.^[8] Even with well-integrated EHR systems in the West, clinicians are known to spend more time with the technology than the patient.^[9] Such situation is likely to be exacerbated in India, where the public health system is under-staffed and technologically averse.

For emerging technology that relies on robust data systems for innovation, this is an existential challenge. The same user reluctance that plagues EHR is likely to curtail the uptake of more advanced technological tools. Quality benchmarks like EHR standards can address this reluctance to an extent by ensuring the standardisation of the tool’s design and function to allow the data collected at different sources to be accessible and functional to different users in the same way. Once trained in the setting up and use of one system, the seamless integration and accessibility of a patient’s health records for rapid diagnosis and treatment is likely to help users hurdle their technological reluctance. However, this may come at the cost of imposing heavy burdens on EHR developers.

In the nascent industry of emerging technology like AI and ML-based healthcare solutions, the technology is far more advanced than the standards, which are yet to be established. In the absence of a clear approval and market-access pathway, innovators have a higher price to pay to enter the healthcare innovations market. A regulator in this context must not only function to create boundary conditions to preserve patient safety, but also allow reasonable room for innovation and efficacy for promising solutions (See Figure 1).

Potential of AI for Public Health Outcomes in India

India’s public health ecosystem provides service delivery through vertical programs for immunisation, and disease surveillance and management that focuses on population health maintenance. It also boasts of a formidable network of health services that encompasses 18 percent of the country’s total outpatient care, and 44 percent of total inpatient care,^[10]  all of which are highly subsidised or free for citizens. Although the reliance on public versus private healthcare varies across state, public healthcare centres often serve as the only point of care for the country’s 66 percent rural population. Yet it suffers from staff shortages, low staff motivation, inadequate or outdated medical equipment, and slow-responding medical institutions.^[11] Notwithstanding the progress made by Ayushman Bharat (AB-PMJAY) and its pursuit of ‘health for all’ through health and wellness centres and health insurance, public health service in India is overburdened.^[12]

With a doctor-patient ratio of 1:10,189^[13] (10 times short of the World Health Organization’s [WHO] recommended ratio^[14]) and severe resource shortages, the clarion call has never been louder for technology at scale to support healthcare delivery in the country. The response has been hopeful—more recently from emerging technological solutions. For example, an AI-based breast cancer screening device that uses a non-invasive, low-cost solution based on heat-mapping for early detection of breast cancer has been able to detect breast cancer up to five years earlier than a mammography with reduced reliance on trained technicians.^[15] A smartphone-based anthropometry technology enables frontline health workers to accurately report baby weight,^[16] solving for incongruencies in field reported data which is popularly tied to insufficient focus on and incorrect interventions for malnutrition on the field. In countries in the West, a rapid detection and response device directly alerts radiologists when it spots pneumothorax.^[17] Various states have taken the initiative to embrace this mission. Telangana for example, has declared 2020 as the year of AI, with the intention of making AI-based innovation successful across e-governance, agriculture, healthcare and education.^[18]

Healthcare is surely and steadily embracing digital health innovation to respond to critical health challenges. In response, regulations have been established for standardising the design and function of these technologies (as is the case with EHR, or medical devices) that recognise the risks associated with their use and protect the patient and user’s safety and rights. AI-based solutions have not only variable conditions of risk associated with their use,^[a] but the risks associated with their use are also still being understood. In preparing for regulation of AI-based health technology, it is important to recognise the context and risks associated with each of these categories.

The problem with regulating AI

In many parts of the world, the use of AI in public healthcare delivery has increased in recent years. In the United Kingdom (UK), for example, the National Health Service or NHS adopted an AI chatbot-based triage system in 2019.^[19] However, the known and unknown risks of making AI the norm for health service delivery have threatened to upend the values of equitable access that are synonymous with public health. While there are AI solutions that exist in speciality or tertiary care hospitals (especially diagnostic assistive tools), few solutions effectively reach out to the primary care setups, perhaps due to the high cost of development and operationalisation that deters affordable pricing for scale.^[20]

Moreover, the more widespread use of AI is hampered by its complexity, rendering certain principles inexplicable to users and untrustworthy (‘AI black box’).^[b],[21] Due to its aggregation of several thousand data points, machine learning algorithms’ decision trajectories are often too complex to be traced back and made explainable to its users without human intervention.^[22] Given the potential of AI to learn pre-existing patterns in data, AI has also been critiqued for replicating biases against disadvantaged social groups that clinicians would otherwise consciously rule out.^[23] Concerns around the discrimination that might be inherent to using AI in medical contexts (that is further challenging to identify and isolate) also have severe implications in a medico-legal context where liability is difficult to ascertain and is instead shared.^[24]

Systems lens for AI regulation

National AI strategies have committed ambitious targets for capital investments towards research and application of artificial intelligence. Encapsulating a proactive stance, these strategies have highlighted how research, innovation and permissive markets can catapult economies into the 4th Industrial revolution and also occupy a significant position in the welfare discourse.^[25]

India’s National Strategy for AI sets precedent for AI capacity development through the institution of Centres of Research Excellence (COREs) focused on fundamental research, as well as International Centres on Transformational AI (ICTAIs) for applied research. In parallel, it acknowledges critical challenges around issues of privacy and safety, data integrity, and technical resource capacity. In a context of emerging technology such as AI finding a way to address public health challenges, regulation for standards of safety and efficacy cannot afford to simply react to known risks of technology^[26] but must be proactive in collaborating for better safety standards.

The US Food and Drug Administration (USFDA), borrowing from the work of International Medical Device Regulators Forum or IMDRF, provides a useful lens for regulating AI/ML models in healthcare, categorising them as AI/ML SaMD or Software as a Medical Device.^[27] Following a risk categorisation that ascertains an AI’s potential risk to the patient and its intended use, USFDA’s proposal involves treating regulation for AI as a series of iterative checkpoints rather than a one-time certification model. Given the potential threat considered against the intended use of the AI, specific clinical evidence is required to be submitted both before and after deployment of the SaMD. In weaning off a static regulation model, the USFDA upholds Good Machine Learning Practice (GMLP) on expectations of quality systems responsible for generating SaMD, including ensuring quality and relevance of data, and transparency of the output aimed at users.^[28] In establishing checkpoints that include manufacturers reporting on specific performance and safety indicators post deployment, the SaMD regulation process allows for modifications to approved devices for greater efficacy of use. Yet in the absence of domestic regulatory expertise in AI regulation, adopting this gold-standard for regulation might be more expensive for domestic innovators.

The ‘pacing problem’^[c] witnessed in the case of AI regulations for healthcare is stark. Historically utilised to safeguard social welfare, regulations have been risk-averse and prioritised consumer safety. This is an outcome that follows systematic review of the costs and benefits of innovation, in addition to striking its balance with relevant stakeholder interests. However, the rise of emerging technology such as AI has raised an important critique of slow-moving, non-adaptive regulation regimes that have not only challenged innovation but also curtailed economic growth.^[29] It is predicted that the application of AI in healthcare in India will be worth INR 431.97 billion by 2021;^[30] this is juxtaposed against a regulations system that has only just acknowledged software as a medical device^[31] and an innovation ecosystem that is still burdened with high costs of experimentation and evaluation. A systematic review of the role of regulation in incentivising the uptake of AI for addressing public health’s woes, while prioritising patient safety, is essential to guiding a regulations framework for AI-based SaMD in developing countries like India.

Pre-regulatory Conditions for AI in Healthcare

India has had the experience of building supportive regulations for the pharmaceutical sector that helped it develop from almost non-existent to one of the world’s leading suppliers of generic drugs. This was achieved through a mix of price controls, experimenting with process patents, and industrial promotion policies.^[32] However, this agile and responsive policy development has yet to translate to medical devices or technologies, and India’s health system continues to be 75-percent dependent on imported medical technology.^[33] The imperative for India is to develop its own medical innovations ecosystem.^[34] This section outlines the existing context within which the regulatory system for AI in healthcare will have to function (See Figure 2).

Inadequate Data Infrastructure

An AI model is built on the foundation of robust and accurate data. Some innovators are able to invest in cumbersome primary data collection and create their own proprietary datasets while buy commercially available datasets to train their models. Both pathways require intensive capital investments that are not available with early-stage start-ups that create tools for the public health system at large.

In India, the government owns large swathes of data, both from public health facilities and national programmes. However, this data lacks accuracy and completeness, which usually results in incorrect conclusions. On aggregation, small errors like misspelled names or inaccurate counts at the facility level can cumulate into glaring misinterpretations.^[35] This can also detract from the representativeness of the datasets used for training and potentially amplify data biases in the AI models, which can have severe social fallouts.

Therefore, a critical challenge for the government is to enable digitisation of most clinical transactions where citizens partake. Thereafter, it is necessary to develop a data culture and quality systems to enable that digital health data accurately depicts the realities of health outcomes at the population, sub-national and even individual facility or patient level. To achieve this goal, the government of India has already commenced an ecosystem building effort for digital health, at the core of which is the concept of EHR of all citizens along with a health information exchange platform to enable sharing of data across the continuum of care.

These building blocks are envisioned as free-flowing data exchange, but presently face immense challenges of portability, especially when it comes to including the private sector in this ecosystem. Physician compliance and adoption of standard terminologies like Systematized Nomenclature of Medicine- Clinical Terms (SNOMED CT) is not particularly incentivised in India as it was in the United States (US), where one could secure substantially higher rates of EHR development through a system of financial incentives and sanctions. Beyond this, private institutions with digital systems face roadblocks due to the absence of mechanisms for sharing data with the government or each other due to technical interoperability challenges. For instance, the government’s Revised National TB Control Program cannot follow patients or monitor their care once they choose to seek treatment in the private sector, due to absence of sharing pathways.^[36]

Unless interoperability across software systems and terminologies is uniformly secured across the healthcare system in India, the digital health ecosystem will remain fragmented and incomplete. While this normative ecosystem can aspire to digitise data that traditionally exists in paper registers, it does not necessarily assist AI innovators in their work unless easy, cost-effective and convenient modalities for sharing this data are instituted.

As health data is considered as Sensitive Personal Information under the Information Technology (Reasonable security practices and procedures and sensitive personal data or information) Rules 2011,^[37] it is also necessary to have stronger privacy and security measures for digital health data, especially when it comes to sharing it. This is where privacy preserving processes like anonymisation and de-identification fit in: they will remove all personally identifiable marks from the data and prepare it for sharing for training AI models. However, it is now widely accepted that anonymisation is not absolute.^[38] At the same time, annotation of health data, including pathological reports and radiological scans, is necessary for data to be usable for the machine to learn and draw patterns. Both privacy-preserving and annotation processes are cumbersome and investment-heavy activities^[39] that can ultimately make the development process expensive and create entry barriers for new enterprises. 

Extensive Capital Requirements

The role of technological innovation in addressing large-scale access challenges that are typical of a developing nation’s public healthcare system is also widely recognised in India. Investment patterns reflect this: the medical devices sector has seen an inflow of FDI worth US$1.8 billion between April 2000 and June 2019.^[40] The pivotal drivers for this sector-specific growth have included increased healthcare consumption and insurance penetration, growing investment from private equity models, and diversified healthcare delivery mechanisms.^[41]

In parallel, the government’s flagship universal health coverage scheme AB-PMJAY is set to be established as the world’s largest health assurance scheme- providing INR 0.5 million per family to nearly 40 percent of the country’s population. Aiming to mainstream transformative technology and boost innovation for healthcare delivery through a dedicated Innovation Unit, AB-PMJAY has established the call for public health innovation in a version that is accessible and affordable to the most economically vulnerable. However, managing a precarious balance between long gestation periods of investments in medical technology and accelerating access for its 107.4 million target users is a direct challenge to the success of the scheme, and to revolutionising public health through innovation in general. The hope is for regulation to favour the market for innovation.

Extensive evaluation and testing processes of deep-tech solutions result in prolonged time spent at this stage, delaying the time for deployment and requiring a relatively longer period of lock-in for investors. For example, the average time taken by US medical technology companies for pre-market clearance is 5.6 years,^[42] with 61 percent of them taking more than four years to get initial market approvals. Further, even medical technology with inconsequential risks to patients—such as external aids like hearing aids—could be treated as high-risk investments due to the high uncertainty that comes with long-term health outcomes. It is not surprising that the average time taken to exit a medical device startup is 8.8 years, with the company burning an average of US$6.25 million every year.^[43]

There is increased ambiguity around perceived risks of AI-based technology and a need for stricter vigilance that follow post-market modifications. It is therefore fair to assume that without significant market incentives, promising emerging technological solutions cannot perform without supportive regulation to bolster its entry into public health.

The role of regulation in making high-risk industries attractive for private investments can be illustrated through the pharmaceutical industry. The value chain in the industry is characterised by two specific kinds of activities: those involved in drug discovery, and those in the manufacturing and selling of the drug. The latter being relatively low-risk, drug discovery involves high and inherently unpredictable risks with returns being 10 percent of the cost of capital for the process^[44] and can only be afforded by pharmaceutical sales giants. The high cost of innovation here is offset by patent protection measures and value-based pricing of drugs manufactured, irrespective of their capital costs. Regulation in the pharmaceutical industry has in this way offset high R&D costs undertaken by manufacturers and made the investments in innovation possible, an approach that might not be feasible in the field of AI.

An exciting opportunity for infusing capital in AI for healthcare lies in mobilising investments towards core infrastructure for digital health innovation. Supporting the development of core capacities like generating standardised and annotated health records and health data exchanges will allow higher penetration for emerging technology that leverages robust data systems and atop building blocks to optimise its application in healthcare. In turn, more accessible markets attract private capital to relatively high-risk solutions (like clinical decision support systems, for example) in the AI-based development value chain.

Difficulty in Assessing the Human-AI Interaction

As regulation pursues the alignment of clinical performance with patient safety, an important consideration is how AI solutions interact with its users and in turn, how that affects clinical efficacy. Superior clinical evidence of an AI-based solution might not necessarily translate to superior adoption, or necessitate that the solution addresses the clinical condition it was meant to because of variables between the clinical environment and algorithm’s practice environment.

Unlike drugs, software and Information Technologies (IT) tools are known to be highly affected by organisational factors such as resources, staffing, skills, training, culture, workflow and processes^[45] as delivery of healthcare interventions using these tools requires the healthcare staff to take on a more active role. A tale of caution comes from using CAD (computer-aided detection) for mammography to improve breast cancer detection wherein the CAD procedure performed no better (and in some ways worse) than the procedure without involving CAD.^[46] Despite no real benefit to women for breast cancer screening, CAD-based mammographies increased nearly 70 percent after insurance reimbursement increased for this procedure in 2002.^[47] Regulators thus need to account not only for the proven clinical efficacy of the solution, but the result of its presence in the market that might serve as a nudge for altering clinician behaviour around the target condition. Another element to consider is creating trust in AI models when it comes to patients, especially in cases where there is no human in the middle, like in the case of chatbots.^[d]

In healthcare, human factors validation testing serves as a meaningful way to address adoption challenges that signal ‘human interaction’ issues for the AI-based SaMD. This demonstrates that the final finished combination product-user interface can be used by intended users without serious issues, for its intended uses and under the expected use conditions.^[48] In the public health context that is still struggling with technology adoption of the more fundamental applications (like EHR patient recording systems),  AI explainability is an important consideration, in order to increase trust in these new systems, while studying and testing for possible risks of human-AI interaction.

Challenges in Dynamic Evaluation of AI

Consensus from a study panel organised as part of Stanford University’s One Hundred Year Study of Artificial Intelligence reflected that “…attempts to regulate ‘AI’ in general would be misguided, since there is no clear definition of AI (it isn’t any one thing), and the risks and considerations are very different in different domains.”^[49] Limited understanding of AI helps articulate the reluctance of regulatory bodies to deconstruct it for purposes of regulation.

At present, there is no domestic regulatory oversight in India for SaMD interventions, leaving AI-driven SaMD further out of its purview. Even if SaMD were recognised, across the world its regulatory approval is based on repeatability and certainty. However, when a software learns on its own and its outputs vary, the regulations need overhaul to adapt to it.^[50]

Due to the evolving nature of algorithms and tedious standard regulatory processes, it is not hard to imagine that after an approval is granted and the product is marketed, an improved version of the algorithm can be released periodically as it collects and analyses new data. To eliminate the need to seek new approvals every single time a version of the algorithm has to be released, the USFDA has implemented a total product life cycle (TPLC) regulatory approach. This approach facilitates the rapid cycle of product improvement and requires pre-market submission for changes that affect safety or effectiveness, such as new indications for use, new clinical effects, or significant technology modifications that affect performance characteristics.^[51] Incorporating a change management protocol is the welcome necessary step in dynamic evaluation of AI-driven products.

Acceptability of results of AI products is another impediment to its adoption. On the field, startups are advised to conduct clinical trials that are time consuming and expensive.^[52] While rulebooks exist for drug-related clinical trials, regulations are scant in the context of medical devices, let alone AI-enabled SaMD. In the absence of a unified Medical Devices Policy, different agencies including the Central Drugs Standard Control Organization (CDSCO) and Bureau of Indian Standards (BIS) have enlisted their own set of requirements, but there is a lack of coordination amongst these agencies.^[53] The absence of an overall guide has led to interpretation issues and prolonged approval times in complying with these interim measures.

Inadequate Regulatory Capacity

Regulatory agility and responsiveness have a direct impact on the adoption of innovation. This regulatory framework needs to be continually fine-tuned to enable optimal innovations while controlling healthcare expenditure.^[54] Regulatory certainty offers benefits to companies by increasing predictability and transparency. Moreover, regulations and standards can also increase compatibility of products^[55] (interoperability for software products) that can lead to cost savings,^[56] which are particularly beneficial for public health units. India’s medical device market has leaped and will continue to grow (pegged to be valued at US$50 billion by 2025),^[57] but its regulatory infrastructure is likely to be a hurdle in many ways because of inherent deficits.

At their core, regulatory frameworks seek to fulfill the dual objective of ascertaining that a product’s probable benefits for its intended use trump its probable risks, and ensuring that these products are easily available to patients in need. This also involves undertaking an enabling function to kickstart industries and innovations.

The first challenge in this pursuit concerns the purview of Indian medical device regulations. Since 1989, when the first medical device was regulated in India, the regulators have only regulated hardware devices, treating them as identical to drugs.^[58] A clear distinction between medical devices and pharmaceuticals for the purpose of regulation was made only in 2017 with the new Medical Device Rules.^[59] These Rules expanded the scope of ‘medical devices’ to all medical devices and in-vitro diagnostic devices that are notified by the government on the basis of their risk. However, these Rules did not recognise software as a medical device, something that was mentioned in its earlier 2016 draft.

It is only through two notifications issued on 11 February 2020 that India moved ahead of regulating just 37 categories of medical devices to bringing all devices, including a software or an accessory, intended to be used for a medical purpose under the purview of regulation.^[60] Through these notifications, the government has also sought to ensure that all importers and manufacturers of medical devices have to be certified as compliant with ISO-13485 (Medical Devices – Quality Management Systems – Requirements for Regulatory Purposes). While the need for compliance with international quality norms can bring a certain assurance of product quality and safety, the standard is still not fit for quality assessments of dynamic and emerging technologies that are increasingly being integrated into health systems, including AI.

Overall, Indian regulators fall behind their international counterparts to truly promote innovations. At present, there are only nascent attempts at creating an ecosystem and infrastructure to conduct quality testing for devices similar to CE or USFDA.^[61] The Gujarat government has already approved the setting up of India’s first medical device testing lab,^[62] but there is still much to be done for putting the right framework in place that can give impetus to local quality testing.

Industry players have been pushing for a separate and comprehensive regulatory regime for medical devices separate from the Drugs and Cosmetics Act. Such a legislation was also being proposed by the NITI Aayog with the Draft Medical Devices (Safety, Effectiveness and Innovation) Bill with its own proposed authority along the lines of the FSSAI.^[63]  This Bill with changes incorporating the consensus achieved with the Ministry of Health and Family Welfare will be introduced in the parliament in the near future.^[64] However, there is little indication that this proposed regulatory framework will have specific provisions to deal with the dynamic demands of emerging technological solutions.

While India is moving slowly towards regulating a wider ambit of medical products being used de facto within the health system, its regulators need to play multiple roles, including that of protecting the patients through rigorous pre- and post-market evaluations as well as that of ensuring access to these products through affordability-inducing measures. For software solutions, India can swiftly adapt existing reference regulations (International Medical Device Regulators Forum or IMDRF) combined with institution of oversight procedures from local regulatory bodies. This might also require India to reassess its policymaking process and make it more participatory, with greater involvement of industry and academic stakeholders to create a synergetic ecosystem for AI in healthcare products.

Recommendations

The dynamic nature of artificial intelligence, coupled with variables introduced from its interaction with users make it apparent that a regulation for balancing patient safety with product efficiency will need to be monitored and reviewed, well into the deployment of the solution. This begs for the role of the regulator to be multifaceted and progressive, which in turn might necessitate structural changes in how regulations, evaluations and certifications, and monitoring is traditionally conducted in India for health-related products.

An overview of regulatory capacity-building for highly specialised markets such as health technology provides a useful insight: semi-governmental regulation (involving specialised functionaries to inform standards and their implementation) allows regulatory agencies to borrow technical standards from international bodies, while exercising care in adopting the same to their social and economic context.^[65]

However, these approaches cannot be adopted as is into India, given the country’s unique ecosystem, industry and regulatory constraints. Adopting the USFDA-based quality and efficacy standards and mechanisms might also limit the AI innovations in healthcare to innovators that have the financial and technological resources to pursue the international gold-standard, and in turn make AI that much less accessible to public health at large.

In regulating AI-based medical devices to mitigate its potential risks to patient safety, the IMDRF risk-assessment framework of SaMD allows identifying categories of risk that require a higher degree of evaluation and monitoring. Focusing on the clinical acuity of the location of care (e.g., intensive care unit versus general preventive care setting), type of decision being suggested (immediately life-threatening versus clinical reminder), and type of decision support being provided (e.g., interruptive alert versus invisible “nudge”), the framework justifiably requires high-risk medical devices to be substantiated with evidence for its validity, reliability and clinical association, and also for the way in which it mitigates known risks to patients. Basing regulations on a risk-based evaluation can help prioritise deployment of lower-risk medical devices in the short-term^[e] ), and resolve for more stringent regulatory concerns around high-risk medical devices in the long term.

Therefore, what is needed is an ecosystem building role where the regulator catalyses the industry through ensuring availability of foundational building blocks like data, promulgating regulatory processes that secures patient interest without overburdening the fledgling industry, and works through an experimental and consultative approach with all relevant stakeholders to institutionalize these frameworks. Key recommendations of how the regulators can fulfill these expectations are presented in the sections below.

Enabling data democratisation with citizens’ interest at the core

For AI to truly permeate healthcare, data access cannot be centralised and cordoned off from those who need to use it. Privacy preservation and protection measures are largely in conflict with the access to large datasets needed for the development, certification and supervision of AI in health solutions. While there are innovative technological options like differential privacy, and comprehensive and dynamic consent management that can resolve the conflict, they are not widely available for an ecosystem that is already propelling at speed. Meanwhile, it is the regulator’s role to ensure data is democratised in a way that keeps the interest of its citizens at the forefront—both in terms of protecting their privacy and ensuring their safety as patients.

Distinguishing between personal and non-personal data as well as setting up access pathways for both separately can be the first step towards data democratisation. To protect the citizen’s interest in the former, it might be reasonable to insist on in-depth documentation of data operating procedures along with regular audits. ISO-13485 and the General Data Protection Regulation (GDPR) requirements (in the absence of the Indian Personal Data Protection Bill) can provide broad guidance on data privacy and security practices that must be instituted.

For non-personal data, the government has a facilitator’s role to play, especially with respect to the data gathered through its own efforts and programmes. It also has the greater possibility of being representative and equally accessible to all.^[66] Exploring pathways to publicly release government data in anonymised and digitised form should be the priority for enabling the industry. This effort needs to go beyond existing efforts like data.gov, which face their own challenges,^[67] into a concentrated effort for investing in infrastructure and capacity building that enables quality data collection. This also requires a conscious effort to develop large, quality, consensual datasets fit for clinical AI innovations.

The regulatory role should also extend to standard-setting for data collection and consent, quality management, and consolidation, which the Health Ministry has been trying to fulfill with the EHR Standards (2016) and the NDHB (2019). This will propel the ecosystem and give it the technological interoperability to share data and aggregate it fit for AI. However, the Government should go beyond defining standards and future strategies, by creating data marketplaces and collaborative schemes to enable this data sharing.

Data quality issues are critical when it comes to building AI for clinical settings. It is, therefore, incumbent on the regulators of AI models to also ensure that the data used adheres to the FAIR (findability, accessibility, interoperability, and reusability) principles and is collected in an ethical manner before certifying the model as fit for the market. This could be further supplemented by organisational quality assessment in pre-market checkpoints. These conditions can signal to the industry that data integrity and ethical collection is of paramount importance to be eligible for the market, and lead to positive structural changes in how enterprises function.

Building a lean approval process 

The regulatory requirements of AI in healthcare continues to evolve as the industry is still in nascent stages. This is also the opportunity to have flexible regulation and learn from experience in striking the right balance between over-regulation (which may delay large-scale public health deployment for meaningful impact) and under-regulation (which may pose challenges to safety, effectiveness, adoption and user-trust). This results in two areas of consideration for the regulator: clinical evaluation of AI models, and post-market monitoring and surveillance of AI models in use.

This is what the USFDA’s Pre-Certification Program intended to do, i.e. institute a least-burdensome regulatory oversight mechanism by ensuring that developers are trustworthy and have adequate quality management systems (organisational excellence and culture) that can support and maintain a safe and highly effective SaMD across its life-cycle. This is followed by a pre-market review process of the safety and efficacy of the model itself in the least intrusive way possible, and finally the USFDA uses post-market monitoring mechanisms to ensure continued safety, effectiveness, and quality performance of SaMD in the real world using real world data.

When it comes to clinical evaluations, the purpose of regulatory oversight is to prevent false results, errors and misinterpretations in the outputs of the AI models that could be detrimental to the clinical outcome it targets. Therefore, the checkpoint for the regulator might be satisfied in the leanest way possible by ensuring accuracy and relevancy of data inputs and its outputs generated through the operation of the algorithm.

A framework used in ethics of genome-wide association studies for multifactorial diseases to identify which genes are useful can be applied to the question of data for AI models as well. The framework identified three criteria necessary for a gene to be useful, which are: (i) data in the studies and work products derived from these genes must be reproducible and applicable to the target population; (ii) data and derived work products should have significant benefits for the patient population to whom they are applied; and (iii) resulting knowledge should lead to quantifiable utility for the patient in excess of the potential harm.^[68]

Therefore, at the clinical evaluation stage, the regulator might be satisfied by evidence proving the benefits through a suite of options, viz. pilot data, observational and risk-adjusted assessment results, and even clinical trials. It is the risk classification of the device that should define the stringency of evidentiary requirements. At the same time, evidence that can point to the efficiency with which the AI prediction interacts with the human element in the loop can also be mandated for clinical high-risk devices. Even highly accurate predictions might be not fit to improve clinical outcomes unless they are followed up with effective interventions (actions) that are integrated into the clinical workflow.^[69] Thus, evidence that not only points to the high predictive value of the model but how the prediction-action pair operates in the clinical setting might be better suited, but may be  cumbersome to obtain and assess.

For a traditional regulatory framework like India’s, it might be challenging to leapfrog into complex institutional changes that AI evaluations and monitoring might necessitate. Effective use of regulatory sandboxes with relaxed regulations and anonymised data availability can help experiment with regulatory models to strike the effective balance needed while allowing for innovations to prosper. Sandboxing^[f] can help decipher new models of collaborations between industry and the government while also helping understand the boundary conditions of effective regulation and ethics to drive innovations. Sandboxes are common and effective across the world. Most recently, UK’s NHSx has called for a joint regulatory sandbox for AI in healthcare bringing together all the sandbox initiatives by different regulators and giving innovators a single, end-to-end safe space to develop and test their AI systems.^[70]

Ensuring an adaptive and consultative monitoring mechanism

The mechanisms to monitor the performance of the model following its deployment is a complex task involving collection and interpretation of real-world information. Further, self-learning models keep refining themselves from on-going datastreams making them complicated to monitor for safety periodically using a static and limited dataset. There are two possibilities for a nascent ecosystem here: limit itself to locked models that can be easily monitored, or develop novel ways to evaluate self-learning (unlocked or reinforced) models. The latter approach will require a consultative approach to work alongside the industry in instituting a balanced and cost-effective system, as experience shows that enhanced post-market surveillance has faced hardships in terms of compliance from developers and enforcement powers of regulators.^[71] Some feasible pathways could be periodic evaluation of performance on stratified patient subgroups to assess if the model performs equally effectively across sub-categories of patients. Flagging of certain outputs as anomalous^[g] and manual auditing of these can help improve reliability of the model.

Another element where the regulator might need a consultative approach with the developers is to set up processes for risk reporting to ensure absolute patient safety. Medical devices usually rely on hazard and operability studies, which can be used for clinical AI devices as well. However, since continuous learning and adaptive aspects of AI bring with it newer risks, it might also be necessary to adapt these risk assessment processes accordingly. Iterative system testing for risk on a continuous basis or period risk audits could be explored, but in ways that do not substantially add to developers’ operational costs. Developers can also list out the dependencies on which their node’s operations are based at the outset (e.g. continued access to user’s data on which the model is based) to be able to control and manage each of them.

Designing for user and patient conditions 

The dynamism of AI-based technology in the clinical context means that its users are pushed to adapt to new workflows that integrate its functions to positively influence health outcomes or, conversely, having no positive influence but instead distorting the treatment pathway. Thus, even if a technology has no proven risk to the patient under given conditions, it needs to be tested for how it adapts with user workflows.^[72]

During clinical evaluation, if a given medical device responds to the clinical outcome it intends to, there is merit in undertaking human factors validation testing considering the environment in which it will be used. The USFDA recommends that manufacturers determine whether the population using the device comprises professionals or nonprofessionals, what the users’ education levels are, what age the users are, what functional limitations they may have, and their mental and sensory conditions. Clinical efficacy for a specific device can be radically influenced by how the device’s testing environment (a controlled laboratory ecosystem) is different from its application environment (a primary health clinic with limited internet connectivity). For frontline health workers with minimum digital literacy, complex interface functions on digital health applications could compromise the volume of beneficiaries they can respond to in a limited period of time, thus compromising health outcomes for the community. Regulation for medical devices therefore needs to articulate similar conditions that need to be tested for, and articulated for its specific usability in a public health context.

Given how trustworthy AI is likely to be adopted better and in many cases is a condition for operating in healthcare, regulators are expected to articulate how much evidence proves this trust. Deep learning methods have been lauded for higher accuracy but are also sufficiently opaque for users to distrust them and hold them accountable for high-risk clinical output. Given the high impact-high risk devices that AI promises to deliver for healthcare, simply prohibiting AI solutions employing opaque decision pathways is counter-productive. Instead, regulations could play a pivotal role in guiding manufacturers to a need-based framework for explainability including (1) articulating the operational and legal needs for regulation, (2) examination of technical tools available to address the same and (3) value the required level of explanation needed against the costs involved, emphasising that explanations are socially useful only when total social benefits exceed costs.^[73]

Regulatory imperatives for ensuring innovations respond to immediate public health needs

In many ways, the true litmus test for an innovation is its responsiveness to the actual needs of the ecosystem in which it integrates. As regulators deliberate over conditioning the innovation ecosystem for AI in healthcare, favouring its responsiveness to public health goals allows manufacturers to innovate directly in response to a need. For this, the regulator needs to build a larger foundational ecosystem and take the role of an enabler, while simultaneously focusing on low-hanging fruits to start introducing emerging technologies in a substantial way into the market.

Under the Medical Devices Rules (2017), USFDA and CE certified medical devices can be marketed in India without having to undergo lengthy clinical trials. While it may be prudent to extend the regulation to include SaMDs, the step may not necessarily spur homegrown innovation. Certification programs under the USFDA and CE are prohibitively expensive for most startups owing to the high costs of clinical trials and regulatory filing in the US and Europe, respectively. Therefore, regulatory authorities in India should explore mechanisms to subsidise these certification expenses by providing direct financial incentives to the startups and MSMEs working on solutions for public health in the absence of Indian quality certification mechanisms. Further, subsidising costs at source could be explored via international agreements and partnerships with external certifying agencies.

There may already be solutions being developed internationally that can be easily contextualised to India. Incentivising these international companies to test solutions on Indian patients for their global trials and working with international agencies to accept and assess these tests for certification may be an important first step in preparing the Indian ecosystem.  From a commercial point of view, the international solutions may have the first mover advantage but can prepare the market for indigenous solutions and lower the barriers to entry in the longer run. For this, regulators in India will need to quickly adapt its vigilance mechanisms as a first goal (as compared to comprehensive clinical evaluations) and ensure safe deployment in India. Learning from this experience, regulators can move further to define holistic certification and benchmarking guidelines for India.

International patent pooling for life-saving technologies can be negotiated by international consortiums on similar lines of Medical Patent Pools for life-saving drugs. While the technology will remain proprietary to the parent firm, the on-ground implantation of these technologies will have to be taken up by local firms that understand the diverse contexts of Indian health systems. Investment to ensure uptake of these solutions may lead to the creation of a smaller auxiliary industry that can quickly test and operationalize health technologies on the ground.

In an enabling role, the regulator also must take a forward-looking approach in building the foundational layers of the ecosystem through collaborations with other governmental, private sector and civil society players. The NDHB is a prime example of how an enterprise architectural approach with focus on base principles, standard-setting and open source technology layers can achieve the goal of kickstarting sustainable and scalable innovations on the top-most application layers. For the AI in the health ecosystem, the government can play a facilitator’s role in creating open technology layers like anonymisers and annotation tools, which can bring down the cost and effort required for innovators in developing and deploying solutions.

Finally, domestic regulatory clarity is pivotal for certainty amongst innovators innovating for an Indian market. India should freely borrow and co-opt norms for clinical assessment being set in AI by international organizations such as WHO-ITU.^[74] Following an iterative approach to the discovery of India-specific norms by working with medical research institutes and AI solution providers in controlled environments such as AI sandboxes can prove to be hugely beneficial to all stakeholders involved – the medical community, the regulator, the innovator and the citizen seeking health services.

Conclusion

While AI shows immense potential in meeting the needs of an under-resourced and overburdened health system, there is much to be done to create and institutionalise structures that can propel its development and optimise its benefits to all. A systems lens to regulate AI can help achieve this goal by drawing domestic regulators’ attention to conditions of the ecosystem that can allow emerging technology to thrive.

Investments are required to build the digital health ecosystem in the country and unlock the large amounts of data that exists with stakeholders, which form the base for AI-driven systems. Further, capacities need to be built not only by the regulators and the government but by private firms and solutioners in the space to assess and ensure the long-term viability of AI. As for any emerging technology, adapting the current static regulatory approach to a more dynamic, iterative one will be key in allowing AI-based technology to thrive rather than struggle against laggard regulation. Perhaps most importantly, as was highlighted in a letter signed by eminent scholars such as Stephen Hawking and industry leaders like Elon Musk, this is an opportunity for the ecosystem to be developed in a way which maximises the social benefit of AI.^[75] Ethical concerns, including, but not limited to biases may surreptitiously slip in, and can go unnoticed if sufficient checks are not put in place. It is the regulators’ responsibility to build a vibrant ecosystem which can fruitfully deliver systems that minimise bias and maximise social benefit as much as possible.

Gartner, a leading IT research and advisory company, said that AI in healthcare is on the rise or in some cases may have hit the peak of the ‘Technology Hype Cycle’.^[76] These are still early days for AI in healthcare and much is to be ascertained in terms of the scalability into real-world use cases. Just as was the case in Artificial General Intelligence, there is a danger of overestimating the use of and the ability to build these complex systems to augment and replace existing healthcare systems. What is clear is that even narrow AI solutions (those operating in predetermined range and scope) have been demonstrated to be of help to medical professionals- influencing health outcomes in a promising way.^[77] It is now for governments (as regulators), clinicians (as users) and patients (as beneficiaries) to collaboratively shape the terms that allow emerging technology to urgently respond to the country’s developmental goals.

Endnotes

• Healthcare
• Media and Internet
• CYBER AND TECHNOLOGY
• healthcare
• India
• MEDIA AND INTERNET
• occasional papers
• public health
• TECH AND MEDIA

The views expressed above belong to the author(s). ORF research and analyses now available on Telegram! Click here to access our curated content — blogs, longforms and interviews.