Are we better a decade later in the accuracy of survival prediction by palliative radiation oncologists?

Introduction

Predicted survival as determined by clinicians, serves as a decisive factor in end-of-life planning in palliative care (1). Despite the emotional strain of receiving terminal diagnoses, many patients strongly value their prognoses (2). Furthermore, survival predictions provide clinicians with critical information used to adapt care, allowing for the determination of treatment setting and intent best suited to the patient (1). For radiation oncologists, predictions of survival serve as indicators of suitable dose fractionation schedules, which are adapted based on the palliative needs of the patient (3).

A general trend of inaccurate clinician predicted survival (CPS) has been reported in the literature, with most clinicians having a tendency to overestimate survival (4). Inaccuracies in prognoses may lead to inappropriate treatment-related decisions, such as the administration of aggressive therapies that result in unnecessary toxicities (3). Ultimately, the goal of researching CPS is to improve the quality of life (QOL) of patients by determining how to provide them with more accurate prognoses, allowing them to make critical decisions and have open communication with those closest to them about their wishes nearing the end-of-life.

The primary objective of the present study was to determine the accuracy of survival estimates for patients receiving palliative radiotherapy, as predicted by four radiation oncologists in the Rapid Response Radiotherapy Program (RRRP). Secondary objectives of this study include (I) an analysis of factors predictive of accurate CPS, (II) comparisons of the accuracy of survival predictions over subsequent clinic visits, and (III) comparisons of the accuracy of predictions in this study to those of the previous study in the RRRP in 2005 (5).

Methods

Patient population

The RRRP clinic at Sunnybrook Health Sciences Centre in Toronto offers prompt palliative radiotherapy to patients with advanced cancer. The RRRP primarily alleviates symptoms related to advanced stages of cancer to improve QOL. Most patients are referred to the RRRP for symptomatic bone metastases. Patients referred to RRRP from August 2014 to March 2017 with a reported CPS and date of death (DOD) were included in the study. The study was approved by the institutional ethics board of Sunnybrook Health Sciences Centre (No. 371-2014).

Survival prediction and data collection

CPS was provided by one of four radiation oncologists as a specific length of time in months or years. Patients did not have direct involvement in the study and were not provided their CPS unless they requested it, as such informed consent was not obtained. Additional patient characteristics including Karnofsky Performance Status (KPS), primary cancer site, and sites of metastases were obtained from medical records by research assistants and inputted into a secure database. The previously listed data points were collected at each of the patients’ subsequent clinic visits within the study period. DOD was collected for each patient with CPS on February 1^st, 2018 from the Patient Care System (PCS) and Excelicare.

Statistical analysis

Median, inter-quartiles (IQR), and range were reported for continuous variables related to patient demographics and clinical information, whereas number of patients and proportions (percentages) were reported for categorical variables. The total number of visits per patient within the study period were calculated and reported as frequencies.

Descriptive analysis was conducted on actual survival (AS), CPS, and the difference between AS and CPS (i.e., AS minus CPS). The mean, 95% confidence interval (CI), and standard deviation (SD) were also calculated for the difference between AS and CPS. The accuracy of CPS was defined as the mean difference between AS and CPS in weeks. A negative difference in AS and CPS indicated an overestimation of survival, whereas a positive difference indicated an underestimation of survival. Weighted Kappa was used for testing the agreement between AS and CPS, where Kappa >0.75 was excellent, 0.40–0.75 was fair to good, and <0.40 was poor (6). A Wilcoxon rank-sum test was used for comparing the accuracy of CPS in patients with shorter (≤26 weeks) and longer (>52 weeks) survival. A t-test was conducted to determine significant difference between our study and the 2005 study. P<0.05 were considered to be statistically significant.

For comparisons of the accuracy of CPS between the first visit and all other visits, and also between subsequent visits, a general linear mixed model was applied. The outcome was the mean difference between AS and CPS in weeks and the independent variable was the categorical variable of visit number.

To search for significant covariates related to the accuracy of CPS at the first visit, univariate general linear mixed model was performed. The difference between AS and CPS in weeks was the outcome and the independent variables included gender, sites of metastases, KPS (binary or categorical), and primary cancer site. The coefficient and its standard error (SE), P, and Alaike information criterion (AIC, where lower AIC indicates better fit of model) were calculated for each factor. KPS of 0–40 and a primary cancer site of prostate were used as reference groups for comparisons in the model. Using a backward selection procedure, a multivariable linear mixed model was conducted. Variables from univariate analysis with P<0.10 were included in the backward selection. The final multivariate model only included significant predictive factors with P<0.05.

Significant changes in the accuracy of CPS over successive years from first to last clinic date were determined using a general linear mixed model. The difference between AS and CPS in weeks was the outcome and the independent variable was the continuous variable of time in months. The coefficient and its SE, and P were calculated. All analyses were conducted using Statistical Analysis Software (SAS version 9.4 for Windows), where P<0.05 was considered statistically significant.

Results

A summary of patient characteristics is provided in Table 1. One-hundred seventy-two patients with reported DOD and CPS were included in the final analysis. The median patient age was 75.2 years (range: 38.4–93.9 years), with most being male (n=106, 61.6%). Lung was found to be the most common primary tumour site (n=46, 26.74%), followed by prostate (n=41, 23.84%), and gastrointestinal (GI) (n=30, 17.44%). The most prevalent sites of metastases were bone (n=149, 86.63%), liver (n=43, 25.0%), and lung (n=40, 23.26%). Median KPS was found to be 60 (range: 20–100), with the majority of patients having a KPS level between 50 and 70 (n=83, 48.26%). The number of patients with subsequent visits gradually decreased, with 88 (49.44%) patients having only one visit, 44 (24.72%) patients having two, and 19 (10.67%) patients having three. The four radiation oncologists provided a total of 373 predictions for the 172 patients.

Table 1 Patient demographics
Full table

A summary of AS, CPS, and accuracy of survival predictions is provided in Table 2. The mean AS from the first visit was 26.76 weeks (SD ±30.49), whereas median AS was 12.9 weeks (IQR: 5.9 and 35.5). Eighty-two (47.7%) patients passed away within 12 weeks of their first consultation. The mean and median CPS, as determined by clinicians on the patients’ first visit, was found to be 45.71 weeks (SD ±38.77) and 28.5 weeks (IQR: 13.0 and 52.0), respectively. The mean and median difference between AS and CPS for the entire patient population from the first visit was −18.95 (SD ±36.14) and −14.0 (IQR: −36.4 and −1.3) weeks, respectively. For patients with an AS ≤12 (n=82), 13–26 (n=31), 27–52 (n=28), and >52 weeks (n=31), the mean difference in AS and CPS was −24.9, −27.0, −13.7, and −0.1 weeks, respectively. Figure 1 shows a bar graph of the agreement between AS and CPS category at the first visit and Figure 2 shows a scatter plot of the same information for each patient. For patients with an AS ≤12, 13–26, 27–52, and >52 weeks, CPS was within the same time frame in 24.4%, 35.5%, 50.0%, and 54.8% of patients, respectively. The Kappa Cohen value between CPS and AS was 0.2 (95% CI between 0.1 and 0.3), which indicates poor agreement (6). The accuracy of CPS was significantly greater in patients with an AS >52 weeks when compared to patients with an AS ≤26 weeks (P=0.02). The accuracy of CPS in the present study was significantly lower than that of the 2005 study (P=0.04), where mean difference in AS & CPS was −12.3 weeks (SD ±38.8, n=739) (5).

Table 2 Actual survival (AS), clinician predicted survival (CPS), and accuracy of CPS
Full table

Figure 1 Agreement between AS and CPS categories. AS, actual survival; CPS, clinician predicted survival.

Figure 2 Scatter plot of agreement between AS and CPS. Dotted line represents points where CPS and AS are equivalent. AS, actual survival; CPS, clinician predicted survival.

Table 3 shows the P comparing the accuracy of CPS between subsequent visits, and between every visit and the first visit. There was no significant difference in the accuracy of CPS between all visits (P=0.5), which includes comparisons between subsequent visits and between every visit and the first visit. Additionally, the difference between AS and CPS did not significantly change over the course of this study from the first to last clinic date (P=0.1).

Table 3 General linear mixed model comparison of mean AS-CPS between visits
Full table

KPS overall was not found to be significantly related to mean difference in AS and CPS (P=0.2) at the first visit upon univariate general linear mixed model (Table 4). Additionally, no sites of metastases were identified as significantly correlated to the difference in AS and CPS at the first visit. Primaries in the breast, GI tract, and lung were found to be significantly related to mean difference in AS and CPS relative to a primary of prostate (P=0.01, 0.01, 0.04, respectively), with prostate being significantly less predictive of accurate CPS. Three variables with P<0.1 were included in the backward selection (gender, P=0.04; KPS <40, P=0.06; and primary cancer site, P=0.08).

Table 4 Univariate analysis of predictive factors
Full table

Upon multivariable analysis (Table 5) only gender was found to be significantly related to mean difference in AS and CPS (P=0.04) at the first visit, with females having a significantly smaller absolute mean difference between AS and CPS than males (−11.7 vs. −23.5 weeks).

Table 5 Multivariate analysis of predictive factors of the accuracy of CPS at the first visit (n=172)
Full table

In order to eliminate potential bias, a second multivariable linear mixed model was conducted in which prostate and breast cancer patients were excluded (n=107). It was found that gender was no longer significantly related to mean difference in AS and CPS (P=0.2), and KPS >40 almost reached statistical significance (P=0.06) (Table 6).

Table 6 Multivariate analysis of predictive factors of the accuracy of CPS at the first visit with the exclusion of breast and prostate primaries (n=107)
Full table

Discussion

In the present study, the CPS of radiation oncologists was generally inaccurate, with most survival estimates being overly optimistic. KPS, primary cancer site, and sites of metastases were not indicative of accurate CPS and CPS did not show any significant changes over subsequent visit dates. Only female gender was correlated with accurate CPS.

The trend of overly optimistic survival predictions is consistent with previously reported results in the literature, as shown in the review by Cheon et al. (4). Such trends can potentially be attributed to the ‘Illusion of Control’, a form of cognitive bias which describes an expectancy of success and greater control over a situation, which may be reflected in physicians’ wishes for their patients to have the best possible outcomes and reluctance to accept their inability to increase longevity (3,7).

The current study found that patients with longer survival (>52 weeks) had significantly more accurate CPS than patients with shorter survival (≤26 weeks) (P=0.02). A phenomenon known as the horizon effect suggests that CPS accuracy improves the closer it is made to the patient’s death (8). However, the horizon effect could not be appropriately evaluated in the present study as patients who are perceived to die within a few weeks are often not referred to RRRP.

Three survival prediction analysis studies by Chow et al. (5,9) and Gripp et al. (3) found that patients with a survival of ≤6 months were more likely to have overestimates in CPS and patients with survival ≥9 months were more likely to have underestimates in CPS. In two sequential prospective studies by Morita et al. (n=150 and 108), it was found that serious overestimates in survival (difference between AS and CPS was ≥28 days and CPS was two times greater than AS) were related to the rapid onset of complications in advanced-stage disease, while also stating that serious underestimates (difference between AS and CPS was ≥28 days and AS was two times greater than CPS) were related to sudden remission of tumour-related pathologies (e.g., bowel obstruction, ascites, etc.) or benign complications (10). As such, a variety of factors may contribute to inaccurate CPS that clinicians may not be able to account for.

KPS at all levels was not found to be significantly predictive of accurate CPS (P=0.2). A prospective cohort study by Fairchild et al. (n=126) found KPS at all levels to be a significantly predictive factor of accurate CPS made by physicians (P=0.0003) and nurses (P=0.04) (8). However, some studies have found KPS to be at risk for greater subjectivity at lower levels of functional status, as it was initially created with a focus on curative rather than palliative treatment (11). Given this and the present results, the use of KPS in the formation of CPS should be limited until greater consistency in the literature is available on its efficacy or an alternative measure of performance status for patients receiving palliative care should be used to inform CPS.

Neither primary cancer site nor sites of metastases were found to be significantly related to CPS accuracy in the present study. Breast and colorectal primaries have been found to be correlated to greater overall survival when compared to lung, lymphoma, and head and neck primaries (P=0.002 and 0.01, respectively), while a primary of lung (P<0.0001) was correlated to poorer overall survival when compared to the previously mentioned primaries in the same patient population (3,12). Bone metastases have been correlated with greater overall survival when compared to brain, liver, lung or multiple sites of metastases (P<0.001), while brain metastases (P=0.01) were significantly correlated to reduced survival (3,13). Despite this knowledge, in the present study CPS was generally inaccurate and neither primary cancer site nor sites of metastases had any bearing on the accuracy of CPS; thus other factors or tools should be used to improve CPS accuracy.

Upon multivariable analysis, only female gender (P=0.04) was found to be predictive of accurate CPS, potentially reflecting gender differences in the reporting of symptoms. A study by Ladwig et al. on a sample of German patients (n=7,460, not all were cancer patients) found that females reported significantly more symptoms when compared to males (P≤0.001) (14). The increased reporting of symptoms in females may provide clinicians with a better understanding of the patients’ disease, potentially resulting in more accurate CPS in females compared to males. However, upon excluding breast and prostate cancer patients, gender was no longer significantly related to CPS accuracy (P=0.2), reducing the validity of this finding as it may be confounded by this potential bias.

CPS accuracy remained stable over subsequent visit dates (P=0.5), contrary to older studies by Parkes (15) and Oxenham (16) which reported mild improvements in CPS accuracy over subsequent visits, attributed to increased familiarity of the physician with the patient’s case. This difference may be attributable to the fact that the RRRP operates as an outpatient clinic, whereas the previous studies by Parkes and Oxenham were conducted in inpatient settings which allow for more direct contact with patients and progressive familiarity with their cases. In the RRRP, regular contact with radiation oncologists is limited and there is a greater focus on the need for palliation of symptoms rather than on the overall progression of disease, thereby limiting the radiation oncologists’ understanding of factors affecting patients’ survival. Future studies should continue to compare the accuracy of CPS between subsequent patient visits and varying specialities, as information on this is somewhat limited and outdated (15,16).

The accuracy of CPS in this study (−19.0 weeks, SD ±36.1) was significantly inferior when compared to the previous study conducted in the RRRP (P=0.04), which reported the mean difference between AS and CPS to be −12.3 weeks (SD ±38.8) (5). The degree of overestimation in CPS has increased since the 2005 study, which may be due to differences in the characteristics of the patient population. The 2005 study had a larger proportion of patients with lung cancer when compared to the present study (51% vs. 27%), which has been associated with decreased survival, potentially aiding physicians in making more accurate CPS (3,8). However, it should be noted that the present study reported a significant decrease in the accuracy of CPS for patients with shorter overall survival, despite contradictions found in the literature.

Within the present study, the accuracy of survival estimates did not significantly increase over successive years between 2014 and 2017 (P=0.6). However, this study was only conducted over the course of three and a half years. The stagnation in the accuracy of survival predictions has been reported for decades, with a commentary by Parkes stating that the prognostic abilities of physicians had not changed since their previous study in the early 1970s (15,17).

Considering the generally poor prognostic ability of CPS as reported in the present study and others in the literature, future research should seek to validate prognostic models to aid clinicians in forming accurate predictions. It would be worthwhile to compare the accuracy of these previously mentioned models against CPS in order to quantify the degree of discordance in the future. It is improbable that education measures will improve the predictions of clinicians; therefore, the use of such models developed for an array of patients receiving palliative radiotherapy, including patients with brain and bone metastases, may be valuable as they take into account factors such as KPS, WHO performance status, and age (18,19). Additionally, further research into the use of artificial intelligence and data mining of patient databases for the development of electronic survival estimation applications for end-of-life conditions may be a potential consideration.

This study is limited by a small sample size of 172 patients. Additionally, patients who are within the last few weeks of survival may not be referred to the RRRP as it would be burdensome to the patient and radiation may or may not improve their QOL at this point. As such, this study may not have captured representative information on the accuracy of CPS in patients who are rapidly approaching death.

Conclusions

Survival estimates for patients seen in RRRP were found to be inaccurate and overestimated. A significant decrease in accuracy was observed since the previous RRRP study in 2005. KPS, primary cancer site, and sites of metastases were not predictive of accurate CPS. Female gender was found to be significantly predictive of accurate CPS, which may reflect gender differences in symptom reporting. No improvements were seen in the accuracy of survival predictions over subsequent clinic visits, potentially due to the RRRP’s status as an outpatient radiotherapy clinic. Due to the limited accuracy of CPS alone, the validation and use of prognostic models should be a future consideration.

Acknowledgements

We thank the generous support of Bratty Family Fund, Michael and Karyn Goldstein Cancer Research Fund, Joey and Mary Furfari Cancer Research Fund, Pulenzas Cancer Research Fund, Joseph and Silvana Melara Cancer Research Fund, and Ofelia Cancer Research Fund.

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.

Ethical Statement: The study was approved by the institutional ethics board of Sunnybrook Health Sciences Centre (No. 371-2014).

References

1. Thai V, Wolch G, Tarumi Y. Survival Prediction of End Stage Cancer Patients: A Quick Review. J Palliat Care Med 2013;3:164.
2. Gwilliam B, Keeley V, Todd C, et al. Prognosticating in patients with advanced cancer--observational study comparing the accuracy of clinicians’ and patients’ estimates of survival. Ann Oncol 2013;24:482-8. [Crossref] [PubMed]
3. Gripp S, Moeller S, Bölke E, et al. Survival prediction in terminally ill cancer patients by clinical estimates, laboratory tests, and self-rated anxiety and depression. J Clin Oncol 2007;25:3313-20. [Crossref] [PubMed]
4. Cheon S, Agarwal A, Popovic M, et al. The accuracy of clinicians’ predictions of survival in advanced cancer: a review. Ann Palliat Med 2016;5:22-9. [PubMed]
5. Chow E, Davis L, Panzarella T, et al. Accuracy of survival prediction by palliative radiation oncologists. Int J Radiat Oncol Biol Phys 2005;61:870-3. [Crossref] [PubMed]
6. Fleiss JL. Statistical Methods for Rates and Proportions. 2nd ed. New York: John Wiley & Sons, Inc; 1981.
7. Langer EJ. The illusion of control. J Pers Soc Psychol 1975;32:311-28. [Crossref]
8. Fairchild A, Debenham B, Danielson B, et al. Comparative multidisciplinary prediction of survival in patients with advanced cancer. Support Care Cancer 2014;22:611-7. [Crossref] [PubMed]
9. Chow E, Hruby G, Harris K, et al. Determining the accuracy of health care professionals in predicting the survival of patients with advanced metastatic cancer. J Pain Manage 2010;3:73-9.
10. Morita T, Tsunoda J, Inoue S, et al. Improved accuracy of physicians’ survival prediction for terminally ill cancer patients using the Palliative Prognostic Index. Palliat Med 2001;15:419-24. [Crossref] [PubMed]
11. Abernethy AP, Shelby-James T, Fazekas BS, et al. The Australia-modified Karnofsky Performance Status (AKPS) scale: a revised scale for contemporary palliative care clinical practice BMC Palliat Care 2005;4:7. [ISRCTN81117481]. [Crossref] [PubMed]
12. Chow E, Fung K, Panzarella T, et al. A predictive model for survival in metastatic cancer patients attending an outpatient palliative radiotherapy clinic. Int J Radiat Oncol Biol Phys 2002;53:1291-302. [Crossref] [PubMed]
13. Chen MT, Sun HF, Zhao Y, et al. Comparison of patterns and prognosis among distant metastatic breast cancer patients by age groups: a SEER population-based analysis. Sci Rep 2017;7:9254. [Crossref] [PubMed]
14. Ladwig KH, Marten-Mittag B, Formanek B, et al. Gender differences of symptom reporting and medical health care utilization in the German population. Eur J Epidemiol 2000;16:511-8. [Crossref] [PubMed]
15. Parkes CM. Accuracy of predictions of survival in later stages of cancer. Br Med J 1972;2:29-31. [Crossref] [PubMed]
16. Oxenham D. Accuracy of prediction of survival by different professional groups in a hospice. Palliat Med 1998;12:117-8. [Crossref] [PubMed]
17. Parkes CM. Commentary: prognoses should be based on proved indices not intuition. BMJ 2000;320:473. [PubMed]
18. Zindler JD, Jochems A, Lagerwaard FJ, et al. Individualized early death and long-term survival prediction after stereotactic radiosurgery for brain metastases of non-small cell lung cancer: Two externally validated nomograms. Radiother Oncol 2017;123:189-94. [Crossref] [PubMed]
19. Willeumier JJ, van der Linden YM, van der Wal CWPG, et al. An Easy-to-Use Prognostic Model for Survival Estimation for Patients with Symptomatic Long Bone Metastases. J Bone Joint Surg Am 2018;100:196-204. [Crossref] [PubMed]