Developing and validating a prediction model for in-hospital mortality in patients with ventilator-associated pneumonia in the ICU

Introduction

Ventilator-associated pneumonia (VAP) is a very common nosocomial infection in the intensive care unit (ICU) (1-3). VAP is defined as pneumonia occurring in patients who were subject to invasive mechanical ventilation at least 48 hours before the onset of infection (4,5), and its incidence is reported to be 5–40% of patients receiving invasive mechanical ventilation (2,6). VAP is associated with some adverse events, such as atelectasis, aspiration, pulmonary edema, venous thromboembolism, delirium, and acute respiratory distress syndrome (ARDS), leading to continuing high morbidity and mortality in the ICU (7,8).

Previous clinical studies have investigated a wide range of risk factors associated with the mortality of patients with VAP, including age, inappropriate initial treatment, duration of mechanical ventilation, length of hospital stay, comorbidities, and invasive operations (9-15). However, the judgment criteria of these risk factors vary between studies and cannot be applied directly to clinical practice, which requires a scoring system convenient for clinical use. At present, only a few scoring systems for predicting the mortality risk of VAP are recognized as effective, such as the Acute Physiology and Chronic Health Evaluation (APACHE II), the Simplified Acute Physiology Score (SAPS II), and the Sequential Organ Failure Assessment (SOFA). These models have limitations when used for predicting the risk of VAP mortality. APACHE II and SAPS II are both time-consuming to use and require large amounts of data for accurate analysis (16). In addition, APACHE II does not include the effects of mechanical ventilation and the use of vasopressor drugs and so may not be appropriate for identifying organ dysfunction and mortality associated with VAP (17). Moreover, Gaudet et al. reported that SOFA has low sensitivity, poor accuracy, and cannot be used to distinguish ventilator-associated tracheobronchitis (VAT) from VAP (18). Furthermore, the number of measurement time points (19) and the age proportion of the sample (20) affects the discriminative power of these models.

Given the multiple limitations of these prediction models mentioned above, we aimed to develop a comprehensive prediction model with demographic and clinical data in a large sample cohort to assess the in-hospital mortality risk of VAP patients from the Medical Information Mart for Intensive Care (MIMIC-III) database between 2001 and 2012. We present the following article in accordance with the TRIPOD reporting checklist (available at https://apm.amegroups.com/article/view/10.21037/apm-22-502/rc).

Methods

Study design

This was a retrospective cohort study. Using the International Classification of Diseases (ICD-9) diagnosis code (99731: ventilator-associated pneumonia) and keywords (VAP, ventilator-associated pneumonia, or venting-associated pneumonia), we selected adult individuals diagnosed with VAP from the MIMIC-III database. The MIMIC-III database was developed by an interdisciplinary team of data scientists and practicing physicians from the Laboratory for Computational Physiology at Massachusetts Institute of Technology (21). It contains detailed information of 38,597 distinct adult patients and 49,785 hospital admissions downloaded from archives from critical care information systems, hospital electronic health record databases, and the United States Social Security Administration Death Master File. The data are publicly available at https://physionet.org/content/mimiciii/1.4/. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013).

Potential variables

The demographic and clinical data were extracted including gender; ethnicity; last care unit; oral care; duration of invasive ventilation; comorbidities including chronic obstructive pulmonary disease (COPD), lung cancer, heart failure, diabetes mellitus, hypertension, and septicemia; hospital length of stay (LOS), and ICU LOS. The SOFA and SAPS II scores were also collected to assess the severity of the disease. In addition, we collected the laboratory data from 6 hours before to 24 hours after admission to the ICU, including white blood cell (WBC) count, red blood cell (RBC) count, blood urea nitrogen (BUN), mean arterial pressure (MAP), fraction of inspired oxygen (FIO₂), oxygen saturation (SPO₂), and bacteria that are common pathogens carried by patients with VAP (Klebsiella pneumonia, Pseudomonas aeruginosa, Acinetobacter baumannii, yeast, Aspergillus fumigatus, Staphylococcus, and microorganism quantity) (22).

Among the variables mentioned above, the categorical variables included gender, ethnicity, last care unit, oral care, comorbidities, lung cancer, heart failure, diabetes mellitus, hypertension, septicemia, Klebsiella pneumoniae, pseudomonas aeruginosa, Acinetobacter baumannii, yeast, Aspergillus fumigatus, Staphylococcus, and microorganism quantity. Continuous variables included invasive ventilation, hospital LOS, ICU LOS, MAP, FIO₂, SPO₂, WBC, RBC, BUN, SAPS II score, and SOFA score. The primary outcome was in-hospital mortality of hospitalized patients with VAP.

Development and validation of the prediction model

Random sampling was used to divide the data into training sets and test sets, with 70% of the data forming the training set and the remaining 30% forming the test set. An equilibrium test was performed between the training set and the test set. After ensuring that the 2 groups of data were balanced, univariate logistic regression analysis was used to screen for possible predictors in the training set. According to the results of the univariate logistic regression analysis, factors with P<0.05 were included in the multivariate logistic analysis. The backward selection method was adopted to establish the model in the training set, and internal validation was carried out in the test set. The receiver operating characteristic (ROC) curve and the calibration curve were plotted to evaluate the performance of the model, and the DeLong test was used to compare the area under the curve (AUC) between the single-index models (SAPS II, and SOFA) and the combined model. The value of AUC was over 0.8, indicating the performance of the model was good.

Statistical analysis

Enumeration data were expressed as the number of cases and the constituent ratio [N (%)]. A chi-square test was used for comparison between groups. Measurement data in normal distribution were expressed as mean and standard deviation (mean ± SD), and a t-test was used for comparison between groups. Non-normal distributed measurement data were expressed as median and interquartile range [M (Q1, Q3)], and the Wilcoxon rank-sum test was used for comparison.

Data were randomly grouped by using the “scikit-learn” module in Python 3 (Python Software Foundation, Wilmington, DE, USA), and univariate and multivariate logistic regression analyses were performed using R version 4.0.2 (The R Foundation for Statistical Computing, Vienna, Austria). P<0.05 was considered statistically significant.

Results

Patient characteristics

A total of 8,182 adult individuals diagnosed with VAP were selected from the MIMIC database. Individuals who were missing any information relating to laboratory data (n=1,278), clinical data (n=285), or disease severity data (n=6) were excluded. Finally, 6,613 eligible subjects were enrolled in the dataset.

The whole dataset was divided into a training set (n=4,629) and a test set (n=1,984). In the training set, there were 2,788 (60.23%) males and 1,841 (39.77%) females. Among them, 3,297 (71.22%) subjects were white people, 330 (7.13%) were black people, and 1,002 (21.65%) were of other ethnicities. The median duration of invasive ventilation was 24.908 (9.779, 94.738) hours. The median hospital LOS was 8.70 (5.25, 14.65) days, and the median ICU LOS was 3.24 (1.76, 6.98) days. More details are shown in Table 1. The results suggested that there were no significant differences in the distribution between the training set and the test set (all P>0.05).

Developing the prediction model

According to the univariate logistic regression analysis, female gender (P=0.003); other races (P<0.001); oral care of toothbrush (P=0.004); comorbidities of COPD (P=0.002), lung cancer (P<0.001), hypertension (P=0.001), and septicemia (P<0.001); hospital LOS (P=0.021); ICU LOS (P<0.001); FIO₂ (P=0.012); SPO₂ (P<0.001); WBC (P<0.001); BUN (P<0.001); bacteria including Klebsiella pneumonia (P<0.001), Pseudomonas aeruginosa (P<0.001), yeast (P<0.001), Aspergillus fumigatus (P<0.001), and Staphylococcus (P<0.001); microorganism quantity (P<0.001); SAPS II score (P<0.001); SOFA score (P<0.001); and duration of invasive ventilation (P<0.001) were found to be potential predictors of the mortality of VAP (Table 2).

All variables with statistical significance were enrolled in the multivariate logistic regression for further analysis. The results showed that other races [odds ratio (OR) =1.637, 95% CI: 1.330–2.011, P<0.001], lung cancer (OR =2.202, 95% CI: 1.226–3.884, P=0.007), septicemia (OR =2.000, 95% CI: 1.620–2.466, P<0.001), hospital LOS (OR =0.928, 95% CI: 0.915–0.941, P<0.001), FIO₂ (OR =0.992, 95% CI: 0.988–0.995, P<0.001), SPO₂ (OR =0.969, 95% CI: 0.952–0.985, P<0.001), SAPS II score (OR =1.048, 95% CI: 1.041–1.055, P<0.001), SOFA score (OR =1.109, 95% CI: 1.077–1.143, P<0.001), and duration of invasive ventilation (OR =1.004, 95% CI: 1.004–1.005, P<0.001) were all independently associated with the mortality of VAP (Table 3).

The algorithm for the mortality risk was as follows: lnP/(1-P) = −0.700 + 0.493 Other Ethnicity + 0.789 Lung Cancer (Yes) + 0.693 Septicemia (Yes) – 0.074 Hospital LOS – 0.008 FIO₂ – 0.032 SPO₂ + 0.104 SOFA Score + 0.047 SAPS II + 0.004 Invasive Ventilation. The nomogram was also plotted, as shown in Figure 1.

Figure 1 The nomogram for predicting the mortality risk of patients with VAP in the ICU. VAP, ventilator-associated pneumonia; ICU, intensive care unit; SPO₂, oxygen saturation; FIO₂, fraction of inspired oxygen; SOFA, Sequential Organ Failure Assessment; LOS, length of stay; SAPS II, Simplified Acute Physiology Score.

Example

As shown in Figure 2, we randomly took 1 patient from the training set as an example. The patient was Caucasian without lung cancer or septicemia. The duration of invasive ventilation was 38.5 hours, and the hospital LOS was 24.35 days, with an FIO₂ of 100% and an SPO₂ of 83%. The SOFA score was 9 points, and the SAPS II score was 47 points. The total score of this patient was calculated to be 409 points, and the corresponding predicted probability of death was 0.1416, which was consistent with the fact that the patient reported no in-hospital death.

Figure 2 The example for practical use of the nomogram. SPO₂, oxygen saturation; FIO₂, fraction of inspired oxygen; SOFA, Sequential Organ Failure Assessment; LOS, length of stay; SAPS II, Simplified Acute Physiology Score.

Validating the prediction model

According to the ROC analysis, the AUC in the training set was 0.837 (95% CI: 0.821, 0.853), with a sensitivity of 0.734 (95% CI: 0.704, 0.764) and a specificity of 0.796 (95% CI: 0.783, 0.809), which suggested that the model performed well. The cutoff value was 0.185. In the test set, the AUC was 0.817 (95% CI: 0.791, 0.843), with a sensitivity of 0.657 (95% CI: 0.608, 0.707) and a specificity of 0.797 (95% CI: 0.778, 0.817; Table 4 and Figure 3). The calibration curve also confirmed the good calibration of the model (Figure 4).

Figure 3 The ROC curve for the training set and the test set. ROC, receiver operating characteristic; AUC, area under the curve.

Figure 4 The calibration curve for the training set and the test set.

Table 5 presents the result of the comparison between the combined model and the single-index models. The AUCs of SAPS II and SOFA were 0.760 (95% CI: 0.733–0.787) and 0.685 (95% CI: 0.653–0.716), respectively, which were both lower than the AUC of the combined prediction model, indicating that the combined model had a better ability to predict the individual death risk of patients with VAP in the ICU than SAPS II or SOFA.

Discussion

Currently, VAP is a common cause of nosocomial infections and even death of ICU patients during hospitalization. Therefore, rapid and accurate identification of patients at a higher risk of death from VAP is critical for better prevention and management of VAP. In this study, lung cancer, septicemia, other races, hospital LOS, FIO₂, SPO₂, SAPS II score, SOFA score, and duration of invasive ventilation were identified as independent predictors of VAP. We developed a prediction model for mortality in patients with VAP using the following algorithm: lnP/(1-P) = −0.700 + 0.493 Other Ethnicity + 0.789 Lung Cancer (Yes) + 0.693 Septicemia (Yes) – 0.074 Hospital LOS – 0.008 FIO₂ – 0.032 SPO₂ + 0.104 SOFA Score + 0.047 SAPS II + 0.004 Invasive Ventilation. Internal verification confirmed that the model had good predictive value, and that its curve was close to the ideal curves.

Previous studies have reported that nosocomial infection is associated with invasive mechanical ventilation in the ICU, including reintubation, tracheostomy, and fiberoptic bronchoscopy (13,23-25). The present study suggested that the duration of invasive ventilation was a predictor of VAP mortality, which was consistent with previous studies. ICU patients receiving invasive ventilation are easily exposed to stress, which leads to a decrease in patients’ resistance. A longer duration of invasive ventilation may lead to multiple stress responses, which can further reduce the function of the body barrier and increase the risk of respiratory infections (12,26). Furthermore, bacteria colonized in the stomach can be colonized in the pharynx and then enter the lower respiratory tract to cause infection (12). Prolonged mechanical ventilation can also lead to a variety of complications. In our study, lung cancer and septicemia significantly increased the risk of VAP mortality. Studies have reported that comorbidities, such as diabetes, respiratory diseases, and renal failure, etc., might be a risk factor for VAP (15,27,28). These diseases can lead to immune suppression, which can impair vital organs such as the heart, liver, kidney, and lungs and make the patient more vulnerable to infection.

To date, only a few prediction models, such as APACHE II, SAPS II, and SOFA, can effectively score the mortality risk of patients with VAP in the ICU. Čiginskienė et al. observed that the AUCs of APACHE II, SAPS II, and SOFA were 0.62 (95% CI: 0.54–0.84), 0.68 (95% CI: 0.54–0.84), and 0.73 (95% CI: 0.59–0.86), respectively, when predicting in-hospital mortality in drug-resistant patients with VAP caused by Acinetobacter baumannii (29). Studies assessing the discriminative power of the APACHE II score for VAP have reported AUCs of 0.53 (95% CI: 0.47–0.58) (30) and 0.743 (95% CI: 0.628–0.857) (31), while a study by Gursel et al. evaluating the discriminative power of the SOFA score for VAP reported an AUC of 0.71 (95% CI: 0.58–0.84) (32). Compared to these single-index models, our multivariate prediction model had a higher AUC [0.817 (95% CI: 0.791, 0.843)] based on a relatively large sample size, and its accuracy and reliability were confirmed by internal validation. This suggests that our model may perform better than these single-index models when predicting the risk of VAP mortality. In addition, our model contains relatively comprehensive variables (e.g., demographic, clinical, and laboratory data) and predictors that are easy to obtain in clinical practice with high clinical practical value. Therefore, our model might assist in clinical decision-making by predicting patient outcomes, recommending timely intervention measures, and improving the survival and prognosis of patients with VAP.

This study has some limitations. First, our study was a single-center study with a study population from the United States only, which may have affected the general applicability of our results. Secondly, the data of the medical cost and administration time were not available in the database. Thirdly, our model lacks external validation. A multicenter, prospective study including different populations is necessary for further validation.

Conclusions

In this study, we developed and validated a practical VAP prediction model with good performance. This model could provide ancillary data to help clinicians predict the individual death risk of patients with VAP in the ICU and make informed decisions regarding VAP diagnosis.

Acknowledgments

Funding: None.

Footnote

Reporting Checklist: The authors have completed the TRIPOD reporting checklist. Available at https://apm.amegroups.com/article/view/10.21037/apm-22-502/rc

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://apm.amegroups.com/article/view/10.21037/apm-22-502/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013).

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Li Z, Ma X, Gao S, et al. Association between hospital and ICU structural factors and patient outcomes in China: a secondary analysis of the National Clinical Improvement System Data in 2019. Crit Care 2022;26:24. [Crossref] [PubMed]
2. Papazian L, Klompas M, Luyt CE. Ventilator-associated pneumonia in adults: a narrative review. Intensive Care Med 2020;46:888-906. [Crossref] [PubMed]
3. Kock KS, Maurici R. Respiratory mechanics, ventilator-associated pneumonia and outcomes in intensive care unit. World J Crit Care Med 2018;7:24-30. [Crossref] [PubMed]
4. Martin-Loeches I, Rodriguez AH, Torres A. New guidelines for hospital-acquired pneumonia/ventilator-associated pneumonia: USA vs. Europe. Curr Opin Crit Care 2018;24:347-52. [Crossref] [PubMed]
5. Kalil AC, Metersky ML, Klompas M, et al. Management of Adults With Hospital-acquired and Ventilator-associated Pneumonia: 2016 Clinical Practice Guidelines by the Infectious Diseases Society of America and the American Thoracic Society. Clin Infect Dis 2016;63:e61-e111. [Crossref] [PubMed]
6. Razazi K, Arrestier R, Haudebourg AF, et al. Risks of ventilator-associated pneumonia and invasive pulmonary aspergillosis in patients with viral acute respiratory distress syndrome related or not to Coronavirus 19 disease. Crit Care 2020;24:699. [Crossref] [PubMed]
7. Ścisło L, Walewska E, Bodys-Cupak I, et al. Nutritional Status Disorders and Selected Risk Factors of Ventilator-Associated Pneumonia (VAP) in Patients Treated in the Intensive Care Ward-A Retrospective Study. Int J Environ Res Public Health 2022;19:602. [Crossref] [PubMed]
8. Watson K, Reoch J, Heales LJ, et al. The incidence and characteristics of ventilator-associated pneumonia in a regional nontertiary Australian intensive care unit: A retrospective clinical audit study. Aust Crit Care 2021; [Epub ahead of print]. [PubMed]
9. Ribeiro ILA, Bellissimo-Rodrigues WT, Mussolin MG, et al. Impact of a dental care intervention on the hospital mortality of critically ill patients admitted to intensive care units: A quasi-experimental study. Am J Infect Control 2022; [Epub ahead of print]. [Crossref] [PubMed]
10. Zhu S, Wang W, Kang Y, et al. Clinical outcomes and risk factors for mortality from ventilator-associated events: A registry-based cohort study among 30,830 intensive care unit patients. Infect Control Hosp Epidemiol 2022;43:48-55. [Crossref] [PubMed]
11. Nseir S, Le Gouge A, Pouly O, et al. Relationship Between Obesity and Ventilator-Associated Pneumonia: A Post Hoc Analysis of the NUTRIREA2 Trial. Chest 2021;159:2309-17. [Crossref] [PubMed]
12. Wu D, Wu C, Zhang S, et al. Risk Factors of Ventilator-Associated Pneumonia in Critically III Patients. Front Pharmacol 2019;10:482. [Crossref] [PubMed]
13. Othman HA, Gamil NM, Elgazzar AEM, et al. Ventilator associated pneumonia, incidence and risk factors in emergency intensive care unit Zagazig university hospitals. Egyptian Journal of Chest Diseases and Tuberculosis 2017;66:703-8. [Crossref]
14. Liu Y, Di Y, Fu S. Risk factors for ventilator-associated pneumonia among patients undergoing major oncological surgery for head and neck cancer. Front Med 2017;11:239-46. [Crossref] [PubMed]
15. Jiménez-Trujillo I, Jiménez-García R, de Miguel-Díez J, et al. Incidence, characteristic and outcomes of ventilator-associated pneumonia among type 2 diabetes patients: An observational population-based study in Spain. Eur J Intern Med 2017;40:72-8. [Crossref] [PubMed]
16. Larsson J, Itenov TS, Bestle MH. Risk prediction models for mortality in patients with ventilator-associated pneumonia: A systematic review and meta-analysis. J Crit Care 2017;37:112-8. [Crossref] [PubMed]
17. Karakuzu Z, Iscimen R, Akalin H, et al. Prognostic Risk Factors in Ventilator-Associated Pneumonia. Med Sci Monit 2018;24:1321-8. [Crossref] [PubMed]
18. Gaudet A, Devos M, Keignart S, et al. Usefulness of Sepsis-3 in diagnosing and predicting mortality of ventilator-associated lower respiratory tract infections. PLoS One 2021;16:e0245552. [Crossref] [PubMed]
19. Boeck L, Eggimann P, Smyrnios N, et al. The Sequential Organ Failure Assessment score and copeptin for predicting survival in ventilator-associated pneumonia. J Crit Care 2012;27:523.e1-9. [Crossref] [PubMed]
20. Srinivasan M, Shetty N, Gadekari S, et al. Comparison of the Nosocomial Pneumonia Mortality Prediction (NPMP) model with standard mortality prediction tools. J Hosp Infect 2017;96:250-5. [Crossref] [PubMed]
21. Johnson AE, Pollard TJ, Shen L, et al. MIMIC-III, a freely accessible critical care database. Sci Data 2016;3:160035. [Crossref] [PubMed]
22. Liu Q, Yang J, Zhang J, et al. Description of Clinical Characteristics of VAP Patients in MIMIC Database. Front Pharmacol 2019;10:62. [Crossref] [PubMed]
23. Horan TC, Andrus M, Dudeck MA. CDC/NHSN surveillance definition of health care-associated infection and criteria for specific types of infections in the acute care setting. Am J Infect Control 2008;36:309-32. [Crossref] [PubMed]
24. Wałaszek M, Kosiarska A, Gniadek A, et al. The risk factors for hospital-acquired pneumonia in the Intensive Care Unit. Przegl Epidemiol 2016;70:15-20, 107-10. [PubMed]
25. Joseph NM, Sistla S, Dutta TK, et al. Ventilator-associated pneumonia in a tertiary care hospital in India: incidence and risk factors. J Infect Dev Ctries 2009;3:771-7. [Crossref] [PubMed]
26. Apostolopoulou E, Bakakos P, Katostaras T, et al. Incidence and risk factors for ventilator-associated pneumonia in 4 multidisciplinary intensive care units in Athens, Greece. Respir Care 2003;48:681-8. [PubMed]
27. But A, Yetkin MA, Kanyilmaz D, et al. Analysis of epidemiology and risk factors for mortality in ventilator-associated pneumonia attacks in intensive care unit patients. Turk J Med Sci 2017;47:812-6. [Crossref] [PubMed]
28. Chang L, Dong Y, Zhou P. Investigation on Risk Factors of Ventilator-Associated Pneumonia in Acute Cerebral Hemorrhage Patients in Intensive Care Unit. Can Respir J 2017;2017:7272080. [Crossref] [PubMed]
29. Čiginskienė A, Dambrauskienė A, Rello J, et al. Ventilator-Associated Pneumonia due to Drug-Resistant Acinetobacter baumannii: Risk Factors and Mortality Relation with Resistance Profiles, and Independent Predictors of In-Hospital Mortality. Medicina (Kaunas) 2019;55:49. [Crossref] [PubMed]
30. Lisboa T, Diaz E, Sa-Borges M, et al. The ventilator-associated pneumonia PIRO score: a tool for predicting ICU mortality and health-care resources use in ventilator-associated pneumonia. Chest 2008;134:1208-16. [Crossref] [PubMed]
31. Mirsaeidi M, Peyrani P, Ramirez JA, et al. Predicting mortality in patients with ventilator-associated pneumonia: The APACHE II score versus the new IBMP-10 score. Clin Infect Dis 2009;49:72-7. [Crossref] [PubMed]
32. Gursel G, Demirtas S. Value of APACHE II, SOFA and CPIS scores in predicting prognosis in patients with ventilator-associated pneumonia. Respiration 2006;73:503-8. [Crossref] [PubMed]