Quantifying COVID: All pandemic models are wrong; some are useful and some aren’t

In an attempt to model COVID-19-related mortality, researchers of the Institute for Health Metrics and Evaluation (IHME) have presented an estimate of excess mortality from the pandemic in 191 countries, including India, for 2020 and 2021. The results were recently published in The Lancet. The Indian part of the global study is largely based on empirical excess mortality—difference between estimated deaths for the pandemic years and ‘expected deaths’ based on reported all-cause deaths in recent pre-pandemic years—observed in 12 Indian states, mostly during the surges in COVID-19 cases in 2020 and 2021. The authors have claimed that although the official global COVID-19 death toll was 59 lakh between 1 January 2020, and 31 December 2021, the actual ‘excess’ deaths that occurred over the same period amounted to 1.82 crore. According to the study, the highest number of estimated COVID-19 plus non-COVID-19 excess deaths occurred in India (41 lakh), and in an indirect way, equating such deaths to COVID-19 deaths, it has reiterated that this number is eight times more than the number of officially recorded COVID-19 deaths.

Questionable conclusions

Although there have been several other similar efforts, a global race of sorts was on to bring out the first estimate of global COVID-19-related excess deaths in a peer-reviewed journal. The IHME team has won that race. However, this estimate is problematic for several reasons. First, any estimate is only as good as the quality of data available, how uncertainty is incorporated in the model, and the nature of assumptions made. Secondly, during the pandemic, attributing death to COVID-19 per se, and disaggregating the data between non –COVID-19 and COVID-19 deaths, have been a fraught with difficultly. Thirdly, assuming death reporting of pre-pandemic times as a proxy for death reporting during the pandemic is not correct. In this context, it is wrong to assume that 41 lakh are “additional” deaths, or that these were COVID-19 deaths, as media articles have been claiming.

The paper accounts for underreporting of CRS in pre-pandemic years and the estimated deaths for the pandemic years are calculated by comparing reported registered deaths for a limited number of months,

The current IHME study is based on unvalidated Civil Registration System (CRS) data from 12 states during the high transmission months of 2020 and 2021, collected largely from journalists and secondary sources, and uses it to extrapolate mortality for two years. It is unclear from the paper if any validation or quality check of the data were done. The paper accounts for underreporting of CRS in pre-pandemic years and the estimated deaths for the pandemic years are calculated by comparing reported registered deaths for a limited number of months, mostly between July 2020 and June 2021, with average deaths for the same period in 2018 and 2019. However, given how sharply the whole health system was repurposed to respond to the pandemic, the level of death underreporting during normal times would no longer be a good approximation of COVID-19 death underreporting. Also, families of COVID-19 victims have been made eligible to financial compensation, and this has created incentives in the system for better reporting, as demonstrated by more than 80,000 backlog deaths reported by 18 states from mid-2021. Again, while reported backlogs are used in The Lancet analysis, we do not know if or how these details were incorporated in the calculations by the authors. Interestingly, for richer countries for which routine CRS data are available, the IHME authors excluded data from periods affected by delays in death registration in their analysis. In India, death registration delays are a systemic feature in CRS, and it is unclear how data reported by media platforms were used without validation. Furthermore, the Government of India (GoI) routinely publishes death estimates to achieve accuracy. In fact, the GoI also points to the gap between registered versus estimated deaths. For example, in 2018 and 2019, 19.2 lakh was the difference between the 1.46 crore deaths that were registered in CRS across India and 1.65 crore deaths that were estimated to have taken place in the country, according to the GoI. The second set of estimates of deaths are arrived at by using sample registration system (SRS) data. The data and approach are in the public domain. As death registrations have improved (92 percent in 2019), the gap between registered and estimated have reduced consistently. All these re-estimates are available in the public domain for free. And rarely have these gaps in projection exercises led to dramatic and sensationalist coverage in the Indian media. In fact, experts from the GoI have published scientific papers—some in The Lancet itself—revising its own earlier estimates upwards.

In India, death registration delays are a systemic feature in CRS, and it is unclear how data reported by media platforms were used without validation.

It is important to understand India’s health and data systems and the complexities in estimating deaths, before getting carried away by the narrative of 41 lakh excess deaths in two years. Given how tertiary healthcare facilities are distributed, deaths missing in one state can be found recorded in other states: People from different states often seek care for serious ailments in states like Kerala with better healthcare delivery systems. As a result, death registration has often been more than 100 percent in Kerala in normal years, and Kerala’s 58,500 reported COVID-19 deaths are counterintuitively much more than what the “excess deaths” CRS system reported for the same time period. The fact that state-level underreporting is often overestimated, is not likely accounted for by the IHME study. According to some estimates, if 2020 and 2021 were “normal years”, two crore people or more than 27,000 people per day would’ve died due to non-COVID-19 reasons,. However, during the pandemic years, regardless of underlying conditions or other life threatening ailments, those who were COVID-19 positive during or close to their death have been counted as COVID-19 deaths in India. COVID-19 positive status of many patients with severe non-COVID-19 conditions would have come to notice only at the point of hospital entry due to mandatory testing. This implies that rather than being an ’excess death’ category, COVID-19 is also a major subset of the 27,000 who were expected to die in India every day. In addition, particularly in states with weak health systems—many of the excess deaths that may have happened in 2020 and 2021 would have been due to delayed or unavailable care for non-COVID-19 conditions. WHO’s COVID-19 global death toll project, has not yet put out a South Asia estimate, presumably due to the difficulty in arriving at estimates in a context of weak data systems. According to WHO, only 16 of the 106 member states in the African, Eastern Mediterranean, Southeast Asian, and Western Pacific regions have sufficient data to make such empirical calculations. However, the IHME team seems to have sidestepped this binding constraint, and used whatever data—of variable quality—that could be found, often from media articles, so that disaggregated global estimates could be calculated. Alarmist and misleading media coverage of the study fails to mention this detail.

Conclusion

COVID-19 modelling based on insufficient and poor quality data has already taken a toll causing the heavy-handed responses, including a lockdown, a quick and single-minded repurposing of the health system at the cost of non-COVID-19 healthcare. We cannot afford to make the same mistakes again in the impending waves by inflating COVID-19 mortality. In fact, although based on a thin layer of evidence as of now, this study is more about non-COVID-19 deaths during the pandemic than it is about COVID-19 deaths. It is hoped that as and when more systematic CRS data is made available by Indian states, the authors will revise their current estimates and deploy more context-specific assumptions. Given the continent-like diversity within Indian states, it is an exercise to be done when state-level data is available; and not solely riding on strong and unreal assumptions.

• Healthcare
• India
• Covid-19
• CRS
• health
• IHME
• Model
• pandemic
• SRS

The views expressed above belong to the author(s). ORF research and analyses now available on Telegram! Click here to access our curated content — blogs, longforms and interviews.