TY - JOUR

T1 - Implementation of the COVID-19 Vulnerability Index Across an International Network of Health Care Data Sets: Collaborative External Validation Study

AU - Reps, Jenna M

AU - Kim, Chungsoo

AU - Williams, Ross D

AU - Markus, Aniek F

AU - Yang, Cynthia

AU - Salles, Talita Duarte

AU - Falconer, Thomas

AU - Jonnagaddala, Jitendra

AU - Williams, Andrew

AU - Fernández-Bertolín, Sergio

AU - DuVall, Scott L

AU - Kostka, Kristin

AU - Rao, Gowtham

AU - Shoaibi, Azza

AU - Ostropolets, Anna

AU - Spotnitz, Matthew E

AU - Zhang, Lin

AU - Casajust, Paula

AU - Steyerberg, Ewout W

AU - Nyberg, Fredrik

AU - Kaas-Hansen, Benjamin Skov

AU - Choi, Young Hwa

AU - Morales, Daniel

AU - Liaw, Siaw-Teng

AU - Abrahão, Maria Tereza Fernandes

AU - Areia, Carlos

AU - Matheny, Michael E

AU - Aragón, María

AU - Park, Rae Woong

AU - Hripcsak, George

AU - Reich, Christian G

AU - Suchard, Marc A

AU - You, Seng Chan

AU - Ryan, Patrick B

AU - Prieto-Alhambra, Daniel

AU - Rijnbeek, Peter R

PY - 2021

Y1 - 2021

N2 - BACKGROUND: SARS-CoV-2 is straining healthcare systems globally. The burden on hospitals during the pandemic could be reduced by implementing prediction models that can discriminate between patients requiring hospitalization and those who do not. The COVID-19 vulnerability (C-19) index, a model that predicts which patients will be admitted to hospital for treatment of pneumonia or pneumonia proxies, has been developed and proposed as a valuable tool for decision making during the pandemic. However, the model is at high risk of bias according to the "prediction model risk of bias assessment" criteria and has not been externally validated.OBJECTIVE: Externally validate the C-19 index across a range of healthcare settings to determine how well it broadly predicts hospitalization due to pneumonia in COVID-19 cases.METHODS: We followed the OHDSI framework for external validation to assess the reliability of the C-19 model. We evaluated the model on two different target populations: i) 41,381 patients that have SARS-CoV-2 at an outpatient or emergency room visit and ii) 9,429,285 patients that have influenza or related symptoms during an outpatient or emergency room visit, to predict their risk of hospitalization with pneumonia during the following 0 to 30 days. In total we validated the model across a network of 14 databases spanning the US, Europe, Australia and Asia.RESULTS: The internal validation performance of the C-19 index was a c-statistic of 0.73 and calibration was not reported by the authors. When we externally validated it by transporting it to SARS-CoV-2 data the model obtained c-statistics of 0.36, 0.53 (0.473-0.584) and 0.56 (0.488-0.636) on Spanish, US and South Korean datasets respectively. The calibration was poor with the model under-estimating risk. When validated on 12 datasets containing influenza patients across the OHDSI network the c-statistics ranged between 0.40-0.68.CONCLUSIONS: The results show that the discriminative performance of the C-19 model is low for influenza cohorts, and even worse amongst COVID-19 patients in the US, Spain and South Korea. These results suggest that C-19 should not be used to aid decision making during the COVID-19 pandemic. Our findings highlight the importance of performing external validation across a range of settings, especially when a prediction model is being extrapolated to a different population. In the field of prediction, extensive validation is required to create appropriate trust in a model.CLINICALTRIAL:

AB - BACKGROUND: SARS-CoV-2 is straining healthcare systems globally. The burden on hospitals during the pandemic could be reduced by implementing prediction models that can discriminate between patients requiring hospitalization and those who do not. The COVID-19 vulnerability (C-19) index, a model that predicts which patients will be admitted to hospital for treatment of pneumonia or pneumonia proxies, has been developed and proposed as a valuable tool for decision making during the pandemic. However, the model is at high risk of bias according to the "prediction model risk of bias assessment" criteria and has not been externally validated.OBJECTIVE: Externally validate the C-19 index across a range of healthcare settings to determine how well it broadly predicts hospitalization due to pneumonia in COVID-19 cases.METHODS: We followed the OHDSI framework for external validation to assess the reliability of the C-19 model. We evaluated the model on two different target populations: i) 41,381 patients that have SARS-CoV-2 at an outpatient or emergency room visit and ii) 9,429,285 patients that have influenza or related symptoms during an outpatient or emergency room visit, to predict their risk of hospitalization with pneumonia during the following 0 to 30 days. In total we validated the model across a network of 14 databases spanning the US, Europe, Australia and Asia.RESULTS: The internal validation performance of the C-19 index was a c-statistic of 0.73 and calibration was not reported by the authors. When we externally validated it by transporting it to SARS-CoV-2 data the model obtained c-statistics of 0.36, 0.53 (0.473-0.584) and 0.56 (0.488-0.636) on Spanish, US and South Korean datasets respectively. The calibration was poor with the model under-estimating risk. When validated on 12 datasets containing influenza patients across the OHDSI network the c-statistics ranged between 0.40-0.68.CONCLUSIONS: The results show that the discriminative performance of the C-19 model is low for influenza cohorts, and even worse amongst COVID-19 patients in the US, Spain and South Korea. These results suggest that C-19 should not be used to aid decision making during the COVID-19 pandemic. Our findings highlight the importance of performing external validation across a range of settings, especially when a prediction model is being extrapolated to a different population. In the field of prediction, extensive validation is required to create appropriate trust in a model.CLINICALTRIAL:

U2 - 10.2196/21547

DO - 10.2196/21547

M3 - Journal article

C2 - 33661754

VL - 9

JO - JMIR Medical Informatics

JF - JMIR Medical Informatics

SN - 2291-9694

IS - 4

ER -