Key Takeaways

• The current COVID-19 pandemic has challenged healthcare systems with increased emergency department (ED) visits and hospitalizations, highlighting a need for intuitive approaches to rapidly and accurately triage patients presenting with respiratory infections as positive for SARS-CoV 2 or influenza infection.
• Our machine learning framework highlights the importance of developing dynamic disease models that control for confounding comorbidities, such as influenza, and the rapidly evolving nature of COVID-19.
• Our approach of using a machine learning model to differentiate between COVID-19 and influenza can be applied to other diseases that are either symptomatically or phenotypically similar to aid in making prompt and accurate diagnoses and facilitating appropriate treatment paradigms.

Motivation

It is estimated that the influenza virus contributes to 31.4 million outpatient visits and over 200,000 hospitalizations within the United States every year ¹. Coupled with the recent astronomical rise in COVID-19 related hospitalizations, the consistent annual influx of influenza patients has the potential to overburden the already strained healthcare system, especially during the flu season. Therefore, to aid frontline healthcare workers in rapidly diagnosing and treating patients, we developed a machine learning framework that could potentially differentiate between the two viral infections when they may coexist within the community.

The Challenge in Differentiating COVID-19 from Influenza

            Clinical presentations of COVID-19 exist along a spectrum, with some patients exhibiting milder symptoms similar to those of the common cold, such as a cough, fever, fatigue, sore throat, or muscle aches ². And while influenza and COVID-19 tend to present with similar symptoms, it is becoming evident that patient vital signs, measurements of basic bodily functions, are unique in COVID-19 infections³. Our goal was to use these routinely acquired patient vital signs to develop a practical application for health care workers to differentiate between COVID-19 and influenza through machine learning algorithms. The various clinical variables investigated in the study included: vital signs (e.g. body temperature, heart rate, respiratory rate, oxygen saturation, and blood pressure), encounter details (e.g. month and reason for visit), and basic demographic characteristics (e.g. age, gender, race, and ethnicity).

The Machine Learning Model

            An ensemble-based (i.e., XGBoost) supervised machine learning models were developed using a cohort of 3883 patients from ED visits and hospitalizations at West Virginia University hospitals, from which 747 (19%) had tested positive for COVID-19, 1913 (49%) tested negative for COVID-19, and 1223 (31%) had influenza. This patient cohort was randomly split into two sub cohorts: a training (80%) set to build the model, and a testing set (20%) to validate the model. Moreover, to further evaluate the generalizability of the predictive framework, we validated our models using a patient cohort from the TriNetX research network, a cloud-based database that provides researchers access to external de-identified patient data. This dataset included 6,613 encounters of 1,057 COVID-19 patients and 9,084 encounters related to 2,068 influenza patients.

How Accurate are the Predictions?

            We developed four different context specific XGBoost (i.e., ensemble-based) predictive models that could be used in a clinical setting. Receiver operating characteristic (ROC) curves were used to illustrate the diagnostic ability to distinguish between the different scenarios in each model. The first model categorized patients as either influenza or COVID-19 positive and achieved an ROC area under the curve (AUC) of 98.8%. The second model distinguished influenza patients from all other patients, irrespective of a patient’s COVID-19 test, achieving an ROC AUC of 98.7%. The third model discriminated between COVID-19 positive and COVID-19 negative patients with a ROC AUC of 97.2%. Finally, the last model used a multi-class framework trained to discriminate between all three different types of patients and had an ROC AUC of 98%.

The superior discrimination of the classifier was validated in the external dataset as well, with the influenza versus COVID-19 model demonstrating a ROC AUC of 92.3%, and the model to separate patients with influenza, irrespective of their COVID-19 status, also showing a similar AUC of 92.3%. These predictions suggested that the developed models were generalizable to varying patient populations in EDs and/or admitted to hospitals with influenza or COVID-19 infections.

What Drives the Model’s Performance?

            We used the Shapley Additive Explanations (SHAP) method, an interpretability method that helps analyze feature importance based on their impact on the model’s output (Figure 1). The analysis showed that vital signs such as body temperature, heart rate, and blood pressure played a more significant role in distinguishing between influenza and COVID-19 positive encounters. Additionally, stepwise removal and addition of vital signs in the machine learning model based on feature importance suggested that of all the vital signs, body temperature, followed by heart rate and SPO₂, could impact the predictive models’ performance in discriminating between influenza, COVID-19 positive, and negative encounters.

Figure 1: SHapley Additive exPlanations (SHAP) beeswarm summary plot of SHAP values distribution of each feature in the dataset for the (a) COVID-19 versus Influenza, (b) Influenza versus Others and (c) COVID-19 positive vs negative models respectively. They depict the relative importance, impact and contribution of features in making predictions. From the plots, it is evident that vital signs such as heart rate, body temperature, diastolic blood pressure and SPO₂ are consistently among the top-ranked features for the models. Additionally, encounter- and patient-specific features such as month, Age, reason for visit, and BMI are highly important.

Summary & Outlook

Our study highlights how machine learning can effectively classify influenza and COVID-19-positive cases through vital signs on clinical presentation. This work presents a low-cost, robust classification system using ensemble-based supervised learning approaches for the appropriate triage of patients displaying symptoms of a viral respiratory infection. With these algorithms, the identification of proper treatment modalities for both COVID-19 and influenza can be made more rapidly, increasing the effectiveness of patient care. We certainly hope that our current work can aid healthcare workers and clinicians to rapidly identify, triage, and guide treatment decisions when the two viral infections start cocirculating in the communities.

To allow for reproducibility and share our work with the research community, the associated models were made available at https://github.com/ynaveena/COVID-19-vs-Influenza for non-commercial use.

Acknowledgements

The authors would like to acknowledge Muhammad Raafay Uqaily for assisting in writing this article.

References

[1] Carias, Cristina et al. “Net Costs Due to Seasonal Influenza Vaccination--United States, 2005-2009.” PloS one vol. 10,7 e0132922. 31 Jul. 2015, doi:10.1371/journal.pone.0132922

[2] “Similarities and Differences between Flu and COVID-19​.” Centers for Disease Control and Prevention, Centers for Disease Control and Prevention, 6 May 2021, www.cdc.gov/flu/symptoms/flu-vs-covid19.htm#:~:text=Both%20COVID%2D19%20and,tiredness

[3] Soltan,  A. A. S. et al. Artificial intelligence driven assessment of routinely collected healthcare data is an effective screening test for COVID-19 in patients presenting to hospital. doi:https://doi.org/10.1101/2020.07.07.20148361; (2020).