Risk predictive models for delirium in the intensive care unit: a systematic review and meta-analysis

Introduction

Delirium is an acute, reversible, and widespread cognitive disorder characterized by fluctuating consciousness, inattention, and disorganized thinking, with a prevalence of 30% to 80% in ICU (1-3). ICU delirium is associated with a lot of severe consequences, such as elevated ICU mortality, prolonged duration of mechanical ventilation and length of hospital stays, increased medical costs, and reduced patients’ long-term cognitive function and quality of life (4-6). Besides, ICU staff caring for delirium patients experienced an increased workload and severe psychological stress (7,8). Furthermore, family members will suffer distress when they witness their loved one in this altered state (9).

Evidence shows that delirium can be prevented. The most commonly recommended measure is implementing delirium prevention clustering measures, such as the ABCDEF bundle (10). Along with universal prophylaxis, tailor preventive measures depending on risk factors and risk levels may be a more effective strategy (11). The risk prediction model was used to predict the independent influence of various risk factors on the disease's occurrence and evaluate the possibility of conditions. Prediction of the ICU delirium risk can help medical personnel effectively identify high-risk patients and develop appropriate clinical decisions. For instance, it can be utilized to better inform family members about the patient's risk of developing delirium and stratify patients in delivering future delirium prevention studies (11-14).

Many scholars have previously established models for predicting delirium of ICU patients based on single-center or multi-center research designs. Additionally, prediction models of delirium for postoperative patients and elderly hospitalized patients were systematically reviewed (15,16), but they were not explicitly targeted at ICU patients. Therefore, the purpose of this study was to comprehensively search studies on prediction models of delirium risk in intensive care unit patients and systematically review them from the aspects such as the basic features, statistical methods, the methodological quality, predictors, and performance of various prediction model, to provide a theoretical basis for clinical practice and scientific research. We present the following article in accordance with the PRISMA reporting checklist (available at http://dx.doi.org/10.21037/apm-20-1183).

Methods

Search strategy

The following databases were searched: Cochrane Library, PubMed, Embase, Web of Science, CINAHL Complete, ProQuest, Chinese Biomedical Literature Database (CBM), China National Knowledge Infrastructure (CNKI), Wanfang and VIP database, from inception to 1^st June 2019. Medical Subject Headings (MeSH) terms and free words were combined: (predict* OR prognos* OR risk*) AND (critical care OR critical illness OR intensive car* OR icu* OR critically ill) AND (delirium OR icu psychosis OR icu syndrome OR acute confusional state OR acute brain dysfunction). The computer search was supplemented by hand search of citations. Literature was searched by 2 researchers (Y Zhang and L Qiao) independently, and limited to papers published in English and Chinese. The protocol was registered in PROSPERO (CRD42019130802). Available from: https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42019130802. The search strategy for PubMed was outlined below (((((((predict*[Title/Abstract]) OR prognos*[Title/Abstract]) OR risk*[Title/Abstract])) OR "Prognosis"[Mesh])) AND ((((("Critical Care"[Mesh] OR "Critical Care Nursing"[Mesh]) OR "Intensive Care Units"[Mesh]) OR "Critical Illness"[Mesh])) OR ((((((critical care[Title/Abstract]) OR intensive care unit*[Title/Abstract]) OR critical illness[Title/Abstract]) OR icu*[Title/Abstract]) OR intensive care[Title/Abstract]) OR critically ill[Title/Abstract]))) AND (("Delirium"[Mesh]) OR (((((delirium[Title/Abstract]) OR icu psychosis[Title/Abstract]) OR icu syndrome[Title/Abstract]) OR acute confusional state[Title/Abstract]) OR acute brain dysfunction[Title/Abstract])) .

Inclusion and exclusion criteria

Articles were included according to the following criteria: (I) Study participants: ICU patients (age 18 years or older); (II) Study content: a study using multivariable design to describe the development or validation of patient delirium risk prediction model (at least two predictors); (III) Study type: cohort or case-control studies. Meanwhile, the exclusion criteria were as follows: (I) A risk prediction model was not established; (II) The model building process or method was not described; (III) The outcome was evaluated in the absence of the use of tools for reliability and validation; (IV) The original text was not available or the information was incomplete; (V) To avoid extracting the same data. We only included the latest or most complete study.

Data extraction and study quality

The design of the data extraction table included the year of publication, country, name of the risk prediction model, participants, research design, sample size, measurement of results, the process of testing results, risk factors, development and verification methods (statistical methods used and outcomes, i.e., % delirium), and the performance of the prediction model. For evaluating the model's prediction performance, model discrimination and calibration are two important dimensions (17). The discrimination reflects the ability of the prediction model to distinguish whether the final event will occur or not, and the most generally used evaluation index is AUC value (18). The calibration reflects the consistency between the predicted results and the model’s observed results (17). Two reviewers (X Chen and Y Lao) collected data independently. The risk of bias (ROB) and clinical applicability was assessed by the PROBAST tool (19,20) from 20 questions in four key domains: participants, predictors, outcome, and analysis. Signaling questions are rated as: “yes/probably yes”, “probably no/no”, and “no information”. According to the signal problem and the author’s judgment, each field was divided into “high”, “low” and “unclear”. All the methodological quality evaluations included in the literature were independently completed by two reviewers (X Chen and Y Zhang). If there were differences, the decision was made by another researcher (Y Lao).

Statistical analysis

All statistical analyses were performed using the forest plot package in R3.6.1. We plotted AUC to summary the predictive performance by forest plot. The Q test was used to determine whether there was heterogeneity between studies. If I²≤50%, it was considered that the studies did not show statistical homogeneity, and a fixed-effects model was used for analysis. If I²>50%, it was considered that the studies were statistically heterogeneous, and the data were analyzed by using a random-effects model; the sensitivity analysis was used to examine the robustness of the findings. Descriptive analysis was used for studies that data could not be merged.

Results

Eligible articles and study characteristics

According to the search strategy, 11,165 articles were initially obtained, and 21 articles were finally registered (Figure 1) (21-41). Table 1 summarized the primary characteristics of the included study. This review included 14 prediction models. Totally, 22,113 patients were included in our study; most patients were from mixed ICU. Nineteen studies (90.5%) used the Confusion Assessment Method for Intensive Care Unit (CAM-ICU) to evaluate the results, and delirium assessment frequency in 15 studies was two or more times a day. Three models (19,36,41) were dynamic prediction models (daily prediction), and the rest were static prediction models, of which five prediction models used at ICU admission and six models used within 24 hours after admission to ICU. Only two studies followed the TRIPOD statement.

Figure 1 PRISMA Search flow diagram.

Table 1 Primary characteristics of the prediction model included in the review
Full table

Study quality

The ROB and clinical applicability were assessed with the PROBAST tool (19) (Table 2). Although all 21 studies demonstrated a low applicability risk, only one of the articles (25) had low bias risk, and the remaining studies were associated with a high ROB. High ROB was mostly in the domain of analysis. The reasons for the high risk of bias in the analysis are as follows: (I) Insufficient sample size. Five studies (23-25,30,32) met the sample size standard (events per variable, EPV >10) in model development studies. Whereas validation studies, only six studies (25,30,31,35-38) met the standard of including at least 100 participants with the outcome. (II) Continuous predictors handled unreasonably. There were seven models (21-24,27,29,41) in which collected continuous predictors were converted into ≥ two categories without using a prespecified method. (III) Selection of predictors based on univariable analysis. (IV) Lack of standardized evaluation of model calibration. Model calibration was not assessed or only assessed using the Hosmer-Lemeshow test. Only six studies (23-25,31,36,37) used a calibration plot or belt to assess the calibration. (V) Lack of internal validation techniques. Four studies (24,27,30,40) lacked internal validation in model development. (VI) Predictors and their weights were inconsistent with the report or not reported. The final model calculation formula was not reported after model development in three studies (21,22,36), and the regression coefficients in the final model were adjusted in another three studies (27,39,41). In terms of predictors and outcomes, the risk of bias was unclear in six articles (26,27,30,33-35) because they did not report blind methods in the literature. Details were shown in Appendix 1.

Table 2 ROB and clinical applicability of included studies
Full table

Statistical methods of models

In this review, 14 studies reported model development (21-25,27-30,32,36,39-41), and 19 studies reported model validation (21-23,25-29,31-41). The method of model development was logistic regression. Six studies (22,27,29,39-41) selected variables for inclusion in the model based on univariable analysis. For 14 included prediction models, ten models were verified internally, of which Bootstrap was used in five studies (21-23,36,39). Random split validation was used in four models (25,28,29,32) internal validation, and the bootstrap validation and random split validation were used in one model (41). In terms of external validation, ten articles reported calibration, four articles (22 27,29,39) carried out the Hosmer-Lemeshow test, and six articles (23-25,31,36,37) reported the calibration plot or belt. Specific statistical methods for the included models were shown in Table 3.

Table 3 Statistical methods and performance of prediction models
Full table

Predictors of models

Among the 14 prediction models included, the number of candidate predictors was 11–116, and the number of predictors in the final model was 3–14, with a total of 40 predictors. Furthermore, the predictors included in the prediction models were divided into two categories: predisposing factors and precipitating factors. In this systematic review, the most common risk factor for delirium in the intensive care unit was cognitive dysfunction (including the history of Alzheimer’s disease, coma, and cognitive impairment). In terms of precipitating factors, sedation and analgesics were the most common predictors, followed by infection and mechanical ventilation. The predictors in the final model were presented in Table 4.

Table 4 Predictors of selected models
Full table

Predictive performance

Of the 14 articles associated with model development, 13 reported discrimination, and the C-statistics ranged from 0.73 to 0.93. For model validation, 14 articles reported AUC, ranging from 0.62 to 0.94, indicating that the included models were quite different. Nine studies were calibrated using the Hosmer-Lemeshow test or a calibration plot, and all of them showed moderate goodness of fit (P>0.05).

In this review, three delirium prediction models were externally validated in at least two studies, including Prediction of Delirium in ICU Patients (PRE-DELIRIC model), Early Prediction of Delirium in ICU Patients (E-PRE-DELIRIC model), and recalibrated PRE-DELIRIC model. Pooled AUC was conducted, and the result was shown in Figure 2. For PRE-DELIRIC model and E-PRE-DELIRIC model, heterogeneity among studies were high (I²=94.66%, P<0.001 and I²=95.83%, P<0.001). The pooled AUC was calculated using a random-effects model. The results showed the good accuracy of PRE-DELIRIC model and E-PRE-DELIRIC model (pooled AUC =0.84; 95% CI: 79.3–89.6% and pooled AUC =0.76; 95% CI: 68.0–84.6%). The sensitivity analysis demonstrated that a single study could not significantly impact the pooled AUC with 95% CIs. For calibration of the PRE-DELIRIC model, heterogeneity among articles was low (I²=0%, P=0.32). The pooled AUC was calculated using a fixed-effects model, and the result also indicated good accuracy (pooled AUC =0.78; 95% CI: 73.8–81.3%).

Figure 2 Meta-analysis of delirium prediction models.

Discussion

In this systematic review, we found 21 studies involving 14 models for predicting ICU delirium. Except for the post-cardiac surgery model (22) and the E-PRE-DELIRIC model (37), whose reported AUC <0.7 in external validation, most models had excellent discrimination (AUC >0.7). However, for the E-PRE-DELIRIC model, the summary AUC score was >0.7. The main reason was that patients received preventive measures according to two models in ref 37, which reduced the incidence of delirium and hence lowered the AUC value of the E-PRE-DELIRIC model. In this systematic review, nine studies reported the calibration and showed that the calibration accuracy was satisfactory overall (P>0.05). However, due to insufficient data, quantitative synthesis cannot be conducted.

At present, the PRE-DELIRIC and E-PRE-DELIRIC model are the most common prediction model of ICU delirium. The meta-analysis indicates that they have a good ability to distinguish delirium. PRE-DELIRIC model has been externally validated by scholars in China (26), Lithuania (34), Argentina (35), Australian (38), and pooled AUC are 0.844, which indicates superior performance. However, the results are partially different from the present one meta-analysis of PRE-DELIRIC by Ho (42), which shows a cumulative AUC is 0.78 (95% CI: 0.74–0.81). The main reason is that we analyze the performance of PRE-DELIRIC without including the recalibrated model, whose accuracy is not as good as the former one. The calibrated model is considered a newly developed model and should not be included. Plus, we have included two Chinese studies whose AUC is nearly 0.93, which attributes to better performance. The PRE-DELIRIC model contains ten predictive factors: age, APACHE II score, admission category, coma, infection, metabolic acidosis, emergency admission, blood urea nitrogen and sedative use, morphine dose within 24 hours. It was designed to predict delirium in surgery, medical, trauma, or neurology adult patients. According to the prediction model, the risks are divided into 4 levels: low-risk group (0–20%), intermediate-risk group (20–40%), high-risk group (40–60%) and extremely high-risk group (>60%).

However, ICU delirium tends to occur in the first day of ICU stay (39), PRE-DELIRIC model only predicts delirium 24 h later after ICU admission. E-PRE-DELIRIC model could predict delirium risk as soon as the patient entered the ICU, which had advantages in a clinical application (43). It includes nine predictors: age, history of cognitive dysfunction, history of alcoholism, blood urea nitrogen, admission disease group, emergency admission, mean arterial pressure (MAP) at admission, glucocorticoid use, and respiratory failure at admission. The risks are divided into four levels: extremely low-risk group (0–10%), low-risk group (10–20%), intermediate-risk group (20–35%), and high-risk group (>35%). Both models are suitable for mixed ICU patients. However, the E-PRE-DELIRIC model (the pooled AUC was 0.78) was not as good as that of the PRE-DELIRIC model. Wassenaar (37) conducted a prospective study of 2,178 ICU patients in 11 hospitals in seven countries. It was concluded that the E-PRE-DELIRIC model, in conjunction with the PRE-DELIRIC model, is more sensitive in the prediction of delirium. Hence, it is recommended to use E-PREDELIRIC at ICU admission. If delirium did not occur, PREDELIRIC should be completed within 24 h to improve the detection of low-risk cases.

However, there is substantial heterogeneity between different studies. On the one hand, the difference in study characteristics (such as where studies were conducted and what participants were included) (Table 1) might lead to significant heterogeneity. On the other hand, the diversity of data collection among studies might affect the model’s performance. The resulting variable, delirium, is a fluctuating, sudden acute condition that requires objective and accurate assessment. However, the outcome assessed by clinical staff and the frequency of assessment was quite non-systematic in some studies (26,35), decreasing consistency. Data loss is also a significant source of heterogeneity. For example, van den Boogaard (23) reported that some values (such as urea, metabolic acidosis, and sodium) were missing, and the mean normal value was imputed. To some extent, the variance and standard deviation of the data will be reduced, and the degree of variation will be underestimated. Therefore, the results of the meta-analysis should be carefully considered.

In this systematic review, the number of predictors in the final model ranged from 3 to 14. Prior cognitive impairment was the most common predisposing factor in 14 included models. Moreover, sedative and analgesics were the most precipitating factors, followed by infection and mechanical ventilation. These risk factors were included in the PRE-DELIRIC and E-PRE-DELIRIC model, the most extensively used models. However, the definition of prior cognitive impairment was inconsistent, and the drugs used for sedative and analgesics were not the same among studies, which would make predictors’ clinical application challenging.

In this systematic evaluation, the quality of evidence was low because of the overall high ROB. The main reason for high ROB was in the analysis field, such as insufficient sample size, lack of internal or external verification, and selection of prediction factors based on univariate analysis. It is suggested that scholars should follow the rigorous research scheme designed by the PROBAST tool when developing or verifying the delirium risk prediction model in ICU in later research. Besides, the TRIPOD list should be submitted as an attachment when submitting articles on predictive models, so that academic editors, peer-reviewed experts, and researchers can evaluate the methodological quality of the articles. However, in this review, only two articles followed the TRIPOD reporting standard, which might prevent readers from assessing the quality of a prognostic model (44).

In this review, the probability for delirium development was divided into groups, like very low, low, moderate, and high risk of delirium. Sensitivity and specificity were calculated for these different groups. We cannot derive the model’s sensitivity and specificity based on the available data, which is regarded as suitable measures to express discriminative abilities. In future research, the overall specificity and sensitivity of the model can be reported. Moreover, it is of considerable significance to promote the external verification and clinical application of the model. Most models have been developed in the past five years, and the number of studies is limited. Specifically, there is a lack of prospective cohort validation studies on population models of intensive care units in different hospitals and regions and comparative studies on the prediction efficiency of different models in the same population. Many models have not been verified externally, so it is difficult to compare their accuracy. At present, few scholars have discussed the application effect of the model in clinical practice (45), which may be related to the increase of staff workload and the tedious calculation of the model. Some scholars believe that intelligent risk prediction is conducive to promoting the model’s clinical application of (46). It is necessary to rely on information systems to automatically collect prediction factor information and calculate prediction results in future experiments. At the same time, RCTs should include delirium risk score as inclusion/stratification criteria if a risk cut-off must be chosen to deliver delirium preventive intervention. The stratified research design will make the groups more comparable. In the clinic, we can inform the family caregivers whether the patient is at high risk and what we can do together to prevent it according to the ICU delirium prediction model.

Conclusions

This meta-analysis indicates that PRE-DELIRIC and E-PRE-DELIRIC model yield a good prediction accuracy, and we suggest they could be the consideration to predict the risk of ICU delirium. However, heterogeneity does exist among studies, so the suggestion should be carefully considered. As for the future research direction, it is suggested that researchers should combine advanced statistical techniques and strictly follow the statement of the TRIPOD statistical performance report.

Acknowledgments

We would like to thank Prof. Qi-Sheng Gao for statistical assistance.

Funding: This work was supported by the Chinese Nursing Association Annual Research Project (Grant number: ZHKY201820) and Zhejiang Medicine Health Science and Technology Project (Grant number: 2018KY472).

Footnote

Reporting Checklist: The authors have completed the PRISMA reporting checklist. Available at http://dx.doi.org/10.21037/apm-20-1183

Peer Review File: Available at http://dx.doi.org/10.21037/apm-20-1183

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/apm-20-1183). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Mehta S, Cook D, Devlin JW, et al. Prevalence, risk factors, and outcomes of delirium in mechanically ventilated adults. Crit Care Med 2015;43:557-66. [Crossref] [PubMed]
2. Krewulak KD, Stelfox HT, Leigh JP, et al. Incidence and Prevalence of Delirium Subtypes in an Adult ICU: A Systematic Review and Meta-Analysis. Crit Care Med 2018;46:2029-35. [Crossref] [PubMed]
3. Ely EW, Shintani A, Truman B, et al. Delirium as a predictor of mortality in mechanically ventilated patients in the intensive care unit. JAMA 2004;291:1753-62. [Crossref] [PubMed]
4. Salluh JI, Wang H, Schneider EB, et al. Outcome of delirium in critically ill patients: systematic review and meta-analysis. BMJ 2015;350:h2538. [Crossref] [PubMed]
5. Abelha FJ, Luis C, Veiga D, et al. Outcome and quality of life in patients with postoperative delirium during an ICU stay following major surgery. Crit Care 2013;17:R257. [Crossref] [PubMed]
6. Vasilevskis EE, Chandrasekhar R, Holtze CH, et al. The Cost of ICU Delirium and Coma in the Intensive Care Unit Patient. Med Care 2018;56:890-7. [Crossref] [PubMed]
7. Öztürk Birge A, Beduk T. The relationship of delirium and risk factors for cardiology intensive care unit patients with the nursing workload. J Clin Nurs 2018;27:2109-19. [Crossref] [PubMed]
8. LeBlanc A, Bourbonnais FF, Harrison D, et al. The experience of intensive care nurses caring for patients with delirium: A phenomenological study. Intensive Crit Care Nurs 2018;44:92-8. [Crossref] [PubMed]
9. Day J, Higgins I. Adult family member experiences during an older loved one's delirium:a narrative literature review. J Clin Nurs 2015;24:1447-56. [Crossref] [PubMed]
10. Devlin JW, Skrobik Y, Gelinas C, et al. Clinical Practice Guidelines for the Prevention and Management of Pain, Agitation/Sedation, Delirium, Immobility, and Sleep Disruption in Adult Patients in the ICU. Crit Care Med 2018;46:e825-73. [Crossref] [PubMed]
11. Paton L, Elliott S, Chohan S. Utility of the PRE-DELIRIC delirium prediction model in a Scottish ICU cohort. J Intensive Care Soc 2016;17:202-6. [Crossref] [PubMed]
12. Wassenaar A, Rood P, Schoonhoven L, et al. The impact of nUrsiNg DEliRium Preventive INnterventions in the Intensive Care Unit (UNDERPIN-ICU): A study protocol for a multi-centre, stepped wedge randomized controlled trial. Int J Nurs Stud 2017;68:1-8. [Crossref] [PubMed]
13. Smithburger PL, Korenoski AS, Alexander SA, et al. Perceptions of Families of Intensive Care Unit Patients Regarding Involvement in Delirium-Prevention Activities: A Qualitative Study. Crit Care Nurse 2017;37:e1-9. [Crossref] [PubMed]
14. Fiest KM, Krewulak KD, Sept BG, et al. A study protocol for a randomized controlled trial of family-partnered delirium prevention, detection, and management in critically ill adults: the ACTIVATE study. BMC Health Serv Res 2020;20:453. [Crossref] [PubMed]
15. van Meenen LCC, van Meenen DMP, de Rooij SE, et al. Risk Prediction Models for Postoperative Delirium: A Systematic Review and Meta-Analysis. J Am Geriatr Soc 2014;62:2383-90. [Crossref] [PubMed]
16. Lindroth H, Bratzke L, Purvis S, et al. Systematic review of prediction models for delirium in the older adult inpatient. BMJ Open 2018;8:e019223. [Crossref] [PubMed]
17. Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. Eur J Clin Invest 2015;45:204-14. [Crossref] [PubMed]
18. Moons KG, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015;162:W1-73. [Crossref] [PubMed]
19. Moons KGM, Wolff RF, Riley RD, et al. PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration. Ann Intern Med 2019;170:W1-W33. [Crossref] [PubMed]
20. Wolff RF, Moons K, Riley RD, et al. PROBAST:A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Ann Intern Med 2019;170:51-8. [Crossref] [PubMed]
21. Pisani MA, Murphy TE, Van Ness PH, et al. Characteristics Associated With Delirium in Older Patients in a Medical Intensive Care Unit. Arch Intern Med 2007;167:1629-34. [Crossref] [PubMed]
22. Katznelson R, Djaiani GN, Borger MA, et al. Preoperative Use of Statins Is Associated with Reduced Early Delirium Rates after Cardiac Surgery. Anesthesiology 2009;110:67-73. [Crossref] [PubMed]
23. van den Boogaard M, Pickkers P, Slooter AJC, et al. Development and validation of PRE-DELIRIC (PREdiction of DELIRium in ICu patients) delirium prediction model for intensive care patients: observational multicentre study. BMJ 2012;344:e420. [Crossref] [PubMed]
24. van den Boogaard M, Schoonhoven L, Maseda E, et al. Recalibration of the delirium prediction model for ICU patients (PRE-DELIRIC): a multinational observational study. Intensive Care Med 2014;40:361-9. [Crossref] [PubMed]
25. Wassenaar A, van den Boogaard M, van Achterberg T, et al. Multinational development and validation of an early prediction model for delirium in ICU patients. Intensive Care Med 2015;41:1048-56. [Crossref] [PubMed]
26. Li Y, He T, Wu W. Study for the reliability and validity of Chinese version PRE‐DELIRIC delirium prediction model. J Nurs Training 2015;30:1356-9.
27. Yuan J. Development of mathematical model and metabonomic study of blood plasma for ICU patients with delirium[dissertation]. Anhui: Wan Nan Medical College; 2017.
28. Chen Y, Du H, Wei B, et al. Development and validation of risk-stratification delirium prediction model for critically ill patients. Medicine 2017;96:e7543. [Crossref] [PubMed]
29. Zhu X. Analysis and development of a delirium prediction model for intensive care patients. [dissertation]. Chongqing: Third Military Medical University; 2017.
30. Xiaobo L, Jinsong L, Mingchun L, et al. To Establish and Evaluate a Prediction Model for Delirium. Neural Injury and Functional Reconstruction 2017;12:532-5.
31. Lee A, Mu JL, Joynt GM, et al. Risk prediction models for delirium in the intensive care unit after cardiac surgery: a systematic review and independent external validation. Br J Anaesth 2017;118:391-9. [Crossref] [PubMed]
32. Moon KJ, Jin Y, Jin T, et al. Development and validation of an automated delirium risk assessment system (Auto-DelRAS) implemented in the electronic health record system. Int J Nurs Stud 2018;77:46-53. [Crossref] [PubMed]
33. Luxi D, Lan C, Yan H, et al. Comparison of PRE-DELIRIC and E-PRE-DELIRIC delirium prediction models in ICU patients. Chinese Journal of Practical Nursing 2018;34:1172-6.
34. Linkaitė G, Riauka M, Buneviciute I, et al. Evaluation of PRE-DELIRIC (PREdiction of DELIRium in ICu patients) delirium prediction model for the patients in the intensive care unit. Acta Med Litu 2018;25:14-22. [Crossref] [PubMed]
35. Sosa FA, Roberti J, Franco MT, et al. Assessment of delirium using the PRE-DELIRIC model in an intensive care unit in Argentina. Revista Brasileira de Terapia Intensiva 2018;30:50-6. [Crossref] [PubMed]
36. Marra A, Pandharipande PP, Shotwell MS, et al. Development and Validation of a Daily Prediction Model. Chest 2018;154:293-301. [Crossref] [PubMed]
37. Wassenaar A, Schoonhoven L, Devlin JW, et al. Delirium prediction in the intensive care unit: comparison of two delirium prediction models. Crit Care 2018;22:114. [Crossref] [PubMed]
38. Green C, Bonavia W, Toh C, et al. Prediction of ICU Delirium. Crit Care Med 2019;47:428-35. [Crossref] [PubMed]
39. Chaiwat O, Chanidnuan M, Pancharoen W, et al. Postoperative delirium in critically ill surgical patients: incidence, risk factors, and predictive scores. BMC Anesthesiol 2019;19:39. [Crossref] [PubMed]
40. Huanmin X, Dongmei LU, Xiaohui W, et al. The development and application of a risk prediction model for postoperative delirium in ICU patients. Chinese Journal of Nursing 2019;54:8-13.
41. Fan H, Ji M, Huang J, et al. Development and validation of a dynamic delirium prediction rule in patients admitted to the Intensive Care Units (DYNAMIC-ICU):A prospective cohort study. Int J Nurs Stud 2019;93:64-73. [Crossref] [PubMed]
42. Ho MH, Chen K, Montayre J, et al. Diagnostic test accuracy meta-analysis of PRE-DELIRIC (PREdiction of DELIRium in ICu patients): A delirium prediction model in intensive care practice. Intensive Crit Care Nurs 2020;57:102784. [Crossref] [PubMed]
43. Han QY, Hu Z. Clinical prediction model construction and evaluation: a challenge for clinical researchers. Ann Transl Med 2019;8:74. [Crossref] [PubMed]
44. Logullo P, MacCarthy A, Kirtley S, et al. Reporting guideline checklists are not quality evaluation forms: they are guidance for writing. Health Sci Rep 2020;3:e165. [Crossref] [PubMed]
45. van den Boogaard M, Slooter AJC, Brüggemann RJM, et al. Effect of Haloperidol on Survival Among Critically Ill Adults With a High Risk of Delirium. JAMA 2018;319:680-90. [Crossref] [PubMed]
46. Duceppe MA, Williamson DR, Elliott A, et al. Modifiable Risk Factors for Delirium in Critically Ill Trauma Patients: A Multicenter Prospective Study. J Intensive Care Med 2019;34:330-6. [Crossref] [PubMed]