Outcome measures in palliative care training interventions: a systematic review of trial-based studies
Review Article

Outcome measures in palliative care training interventions: a systematic review of trial-based studies

Hsien Seow1^, Valerie C. Bishop1^, Jeff Myers2, Kelli I. Stajduhar3^, Denise I. Marshall4^, Nadia K. Incardona2, Oren H. Levine1, Leah J. Steinberg2, Daryl B. Bainbridge1^

1Department of Oncology, McMaster University, Hamilton, ON, Canada; 2Department of Family and Community Medicine, University of Toronto, Toronto, ON, Canada; 3School of Nursing, University of Victoria, Victoria, BC, Canada; 4Department of Family Medicine, McMaster University, Hamilton, ON, Canada

Contributions: (I) Conception and design: All authors; (II) Administrative support: H Seow; (III) Provision of study materials or patients: None; (IV) Collection and assembly of data: VC Bishop, DB Bainbridge; (V) Data analysis and interpretation: All authors; (VI) Manuscript writing: All authors; (VII) Final approval of manuscript: All authors.

^ORCID: Hsien Seow, 0000-0001-6701-1714; Valerie C. Bishop, 0000-0001-6769-2783; Kelli I. Stajduhar, 0000-0003-2381-4712; Denise I. Marshall, 0000-0002-2366-7065; Daryl B. Bainbridge, 0000-0001-9249-8229.

Correspondence to: Daryl B. Bainbridge, PhD. 699 Concession St. Room G-201, Hamilton, ON L8V5C2, Canada. Email: bainbridgd@hhsc.ca.

Background: The ability of health care clinicians to offer a palliative approach to care to their patients with progressive, life-limiting illness has become critical as demand for these services increases. Numerous training initiatives exist to assist clinicians who are not palliative care specialists in the development of palliative care skills, however there is little consensus on how to best measure the effectiveness of these education programs. We conducted a systematic review of palliative care training intervention trials to examine the outcomes measures used.

Methods: We searched MEDLINE, CINAHL, PsycINFO, Embase, HealthSTAR, and five trial registries for studies and protocols published since 2000. Eligible studies were trials assessing palliative care training for clinicians. Interventions had to address at least two of six palliative care-related domains, based on the National Consensus Project: identification or assessment; illness understanding; symptom management; decision making (e.g., advance care planning); coping (patient and caregivers); and referral (coordination/care planning). Each article was reviewed independently by a minimum of two reviewers for inclusion and extraction of relevant data.

Results: Of 1,383 articles reviewed, 36 studies met the inclusion criteria, 16 (44%) of which focused on palliative care communication skills. Among all the trials, 190 different measures were reported. Only 11 validated measures were used in at least 2 studies, including the End-of-Life Professional Caregiver Survey (EPCS) for clinicians and the Quality of Dying and Death Questionnaire (QODD) for caregivers. Clinician and patient/caregiver reported outcomes were measured in 75% and 42% of studies, respectively. Half of the trials employed a study-created questionnaire. Data from administrative (n=14) and/or qualitative (n=7) sources were also used. Nine studies, almost exclusively those with a communication skills focus, assessed clinician interactions as an outcome.

Conclusions: We found considerable diversity in outcomes among the trials reviewed. Further examination of the outcomes used in the broader literature and development of these measures is needed. This will assist towards establishing meaningful and consistent metrics for assessing the impact of palliative care education, to inform evidence-based scaling of effective programs.

Keywords: Palliative care; outcome and process assessment; education; health personnel; training programs

Submitted Aug 11, 2022. Accepted for publication Jan 04, 2023. Published online Feb 23, 2023.

doi: 10.21037/apm-22-947

Highlight box

Key findings

• Our review identified 36 trials of palliative care training programs.

• Among these studies 190 different measures were reported. Eleven validated measures were used in two or more trials.

What is known and what is new?

• Many palliative care training initiatives exist, however there remains little consensus on how to best measure the effectiveness of these education programs.

• We examined the outcome measures used across trials to identify common measures that may inform future high-level studies of palliative care training interventions.

What is the implication, and what should change now?

• We found little commonality among the measures employed and many of the trials did not assess actual behavior change or its impact.

• We provide recommendations for evaluating palliative care training interventions and for further research in its measurement.

Introduction

There is growing consensus among health care clinicians and researchers that palliative care should be integrated earlier in the serious illness trajectory (1-3) and that this can be facilitated by generalist clinicians providing primary palliative care to their patients (4,5). Many local and national training programs have been initiated to help develop these skills among primary care physicians, nurses, and non-palliative care specialists (6-10). A systematic review by Brighton et al. in 2017 which evaluated studies of end-of-life care communication skills training, found 153 unique training interventions (11). We recently completed a systematic review that reported on the effectiveness of 22 palliative care education programs that had been evaluated in controlled trials (10). We identified five main outcome themes: palliative care awareness, palliative care delivery, communication, well-being, and care processes, with most studies reporting a significant positive impact in at least one outcome. However, no reviews have gone in depth about the outcome measures used in the assessment of palliative care education interventions (8,10-15).

Several reviews have examined measures used to assess the impact of specialized palliative care, such as in advanced cancer care, on patient outcomes (16-23). A compilation by Teno et al. in 2005 of measures deemed pertinent to end-of-life care identified nearly 300 instruments (24). This list was expanded upon in a 2007 systematic review which found an additional 64 measures of specialized end-of-life care (18). The measures cited largely consist of patient or family reported outcomes, including satisfaction with care, quality of life, extent of pain and symptoms, functionality, and health care use. Place of care/death and health care costs were also commonly used system-level outcomes for evaluating palliative care program effectiveness (25). Nonetheless, outcomes for assessing the effectiveness of palliative care programs tend to be different and further downstream from outcomes relevant to evaluating the effectiveness of education interventions on health care clinicians. Medical training is intended to promote the uptake of knowledge and a change in practice by clinicians. This purpose needs to be reflected in the assessment measures. To our knowledge, no current reviews specifically examine outcome measures used in palliative care education interventions delivered to interprofessional clinicians.

We completed a systematic review of controlled trials and protocols of palliative care education interventions for interprofessional health care clinicians to address this gap in knowledge. Our objective was to examine the outcome measures evaluated across these studies to identify common measures or themes that may inform future high-level studies of palliative care training interventions. We present the following article in accordance with the PRISMA reporting checklist for systematic reviews (26) (available at https://apm.amegroups.com/article/view/10.21037/apm-22-947/rc).

Methods

Review aims

We completed a systematic literature review to find and describe outcome measurements used in trial studies and protocols of palliative care training interventions. We completed our review in spring 2022. Our goal was to find palliative care training initiatives for health care professionals and compare the measures used. The protocol for this systematic review was registered with PROSPERO (reg CRD42022271741), as part of a larger review.

Search strategy

We searched seven bibliographic databases: MEDLINE, CINAHL (Cumulative Index to Nursing & Allied Health Literature), Embase, PsycINFO, HealthSTAR, the Cochrane Database of Systematic Reviews and Clinical Trials and, the Web of Science (Arts & Humanities Citation Index; Science Citation Index Expanded, Social Sciences Citation Index, Conference Proceedings Citation Indices). We also searched the following five trial registries: ClinicalTrials.gov, EU Clinical Trials Register, Australian New Zealand Clinical Trials Registry (ANZCTR), International Standard Registered Clinical/Social Study Number Registry (ISRCTN), and International Clinical Trials Registry Platform (ICTRP). We searched for English language studies published from 2000 to February 2022 (though training could occur in languages other than English).

We used the following subject headings and related terms in our search: Palliative Care (including end of life, terminal care, etc.); Health Personnel (including clinician, medical staff, etc.); Health Knowledge, Attitudes, Practice (including skill, experience, etc.); and Education/Training combined with the “AND” Boolean operator. We included search strategies (hedges) for choosing Trials or Reviews in the search algorithm to identify these studies (for the full search strategy, refer to Appendix 1) (27). We also searched through the reference lists of all original study articles and reviews found. The trial registration numbers of all registered or published study protocols identified were searched using Google Scholar to identify any related publications, i.e., published protocol or completed study.

Inclusion and exclusion criteria

Eligible studies were either a randomised controlled trial or a quasi-experimental study (i.e., had a control group) that had been registered with a trial registry. We included completed published studies, published protocols, and trial registries of research analyzing the effect of palliative care training for generalist clinicians. The intervention population could include members of any health care profession who can provide primary palliative care (excluding palliative care specialists), including health care students and postgraduate medical students, working in any setting. We defined palliative care as care of any serious illness requiring palliation or hospice care that cause a decline in health, resulting in death (10,28). We defined training as an in-person or online intervention, consisting of interactive instruction (10,29). The focus of this education could range from teaching early palliative care to that at end of life. Additionally, to meet the criteria of teaching a palliative approach to care, the intervention needed to address at least two of the following palliative care components: assessment/identification (early identification of those who may benefit from a palliative approach to care and/or comprehensive assessment of their needs); illness understanding; decision making (such as advance care planning); symptom management; coping (assessing the ability of the caregiver and/or family to cope); and referral (such as care planning and/or coordinating with specialists). These components were derived from Jacobsen et al. (30). guidelines for outpatient palliative care, originally described by the National Consensus Project for Quality Palliative Care (31). We excluded studies of pediatric palliative care interventions, those without an interactive piece, and those based solely on educating patients or family caregivers.

Selection process

We used the Rayyan collaborative review platform to manage title, abstract, and full text screening (32). A minimum of two reviewers from the research team completed screening independently and in duplicate. All articles potentially meeting the inclusion criteria at the title/abstract stages advanced to the next screening stage. At the full text screening stage, we resolved disagreements through discussion in the research team.

Data extraction and analysis

We extracted data using a standardised template with specified fields, including study design, data sources, measurement intervals, and specific measures. Two reviewers (VCB and DBB) independently extracted data from included study publications and then consolidated findings; differences were resolved through joint review and discussion. We reported all study measures listed in all publications of included studies. Measures identified were grouped by theme to describe and compare these outcomes across the studies. We also identified “commonly” used measures, defined as being used by two or more studies (18). In our previous systematic review, we assessed the risk of bias of the published, completed trials using the Cochrane Risk of Bias 2 tool (RoB 2) or the ROBINS-I tool (Risk of Bias In Non-randomized Studies) (10).

Kirkpatrick Training Evaluation Model

The Kirkpatrick Training Evaluation Model describes four levels of evidence for measuring the effectiveness of an intervention based on four levels of impact: reaction (i.e., satisfaction with the training), learning, behavior, and results (i.e., effect of behavior change) (33). This model is an internationally recognized framework for assessing the outcomes of educational programs (34). We classified the outcomes in each study relative to the Kirkpatrick model to determine the levels of evidence represented in the study. We also assessed all commonly used measures specifically, in terms of the Kirkpatrick level(s) of evidence addressed.

Results

Study selection

Our search strategy found 1,383 articles or trial registries, including original papers and review studies. In total, 51 review studies were identified, of which 37 were found to be relevant, and hand searched for original papers that the search did not capture. After the initial screening of all original papers, 55 full text articles and 27 published protocols were examined; 36 studies were eligible for inclusion, 28 with published results and 8 with published protocols only (see PRISMA flow chart Figure 1). We did not find nor exclude any papers or reviews that had a strictly pediatric focus, that also passed the other exclusion criteria. Our search using the trial registry numbers identified the published protocols of five included studies (included in the 36 reviewed studies) that were not found from the search of the bibliographic databases. Overall, 16 (44%) of the study interventions focused on palliative care communication, while the others covered palliative care in general. Thus, we separated General Palliative Care and Communication studies into Table 1 and Table 2, respectively, to allow for comparison of measures used across like studies.

Figure 1 PRISMA study search strategy flow diagram.

Table 1

General palliative care focus studies

Lead author, year, intervention name Objective Target professions Data sources (TYPE) Outcome measures Kirkpatrick model levels£
Agar, 2015†; 2017†, IDEAL Project (Implementing Dementia End of life care At Local aged care facilities) (35,36) To compare facilitated family case conferencing to usual care for end-of-life care and outcomes for residents with advanced dementia living in nursing homes Nurses, community nurses, care assistants, general practitioners, allied health professionals Self-completed survey: Clinicians, and Family Members, Patient clinical data (nurse assessed), Administrative data Clinicians—nurses: - Comfort Assessment in Dying with Dementia Scale - Symptom Management at the End of life in Dementia Scale - Quality of Life in Late-stage Dementia Scale - EuroQol-5 Dimension-5 Level (EQ-5D-5 L) Scale - Palliative Care for Advanced Dementia Questionnaire (qPAD) Families/caregivers: - End of Life in Dementia Scale - Comfort Assessment in Dying with Dementia Scale - Symptom Management at the End of life in Dementia Scale - Satisfaction with Care at the End of life in Dementia Scale Patient clinical data: - Functional Assessment Staging of Alzheimer’s Disease Scale - Australia–modified Karnofsky Performance Status Scale - Bedford Alzheimer Nursing Severity Scale - Aged Care Funding Instrument Scale - Palliative Care Outcomes Collaborative Scale Administrative data: - Place of death - Involvement of GP/other services - Emergency department visits - Hospital admissions (number and days) - Formal symptom assessments - Symptom management last day - Health care cost II, III, IV
Ahlström, 2018** (37); Bökberg, 2019** (38) To evaluate the effectiveness of an educational intervention on palliative care in nursing homes Staff (commonly assistant nurses, nurses, occupational therapists and physiotherapists) Self-completed survey: Clinicians, and Family Members, Interviews with Clinicians, Patients, and Caregivers, Administrative data (registry based on clinician submitted data), Qualitative focus groups, Participant observations Clinicians—mixed: - Person-Centered Care Assessment Tool for staff - Person-Centered Climate Questionnaire for staff - Your Experience of Palliative Care Questionnaire - Preparedness for implementation [Focus Groups] Patient interviews: - WHO Quality of Life Questionnaire (modified) - Person-Centered Care Assessment Tool for patients (modified) - Person-Centered Climate Questionnaire for patients (modified) Families/caregivers: - WHO Quality of Life Questionnaire - Participation of next of kin in care in nursing homes questionnaire (study created) - Quality of life (study created) [Interviews] - WHO Quality of Life Questionnaire (modified) [Interviews] - Participation of next of kin in care in nursing homes questionnaire (study created) [Interviews] - Experience with intervention [Interviews] Administrative data: - Symptom management provided - Oral health provided - Bereavement follow-up offered, etc. - Participant observations (ethnography) I, II, III, IV
Bishop, 2019, End-of-Life Nursing Education Consortium project (12) To increase competence in primary palliative care Registered nurses (acute and ambulatory care units) Clinician self-completed survey Clinicians—nurses: - Palliative Care Practices of Registered Nurses Survey (adapted) - Practice change 3 months post education (Y/N) II
Broese, 2020†, COMPASSION (COPD Palliative and Supportive care Implementation) study—protocol published [completed trial not published] (39) To study the effect of implementing an integrated palliative care approach Pulmonologist, respiratory nurses, hospital palliative care consultants, general practitioners specialized in asthma and COPD or palliative care, regional palliative care consultants Self-completed survey: Clinicians, Patients, and Caregivers, Administrative data, Interviews with Clinicians, Patients, and Caregivers Clinicians—mixed: - End-of-Life Professional Caregiver Survey - Measurement Instrument for Determinants of Innovation Questionnaire (role identity) (modified) - Satisfaction with care provided to patients question (study created) - Intervention dosage delivered report (study created) Patients: - Functional Assessment of Chronic Illness Therapy Palliative Care Scale - Functional Assessment of Chronic Illness Therapy - Spiritual Well-Being Scale (FACIT-Sp-12) - Hospital Anxiety and Depression Scale - Clinical COPD Questionnaire - Intervention dosage received (modified) - Satisfaction with provided care (study created) Families/caregivers: - Caregiver Reaction Assessment Scale - Satisfaction with care provided to patients question (study created) Administrative data (1 year pre to 1 year post): - Emergency department visits - Hospital admissions (number and days) - ICU admissions (number and days) - Place of death - Place of care final week Interviews—all: - Experience with intervention and acceptability I, II, III, IV
Catania, 2017**; 2021**, INFO-QoL (40,41) To determine the feasibility of a nursing intervention focused on quality of life assessment in palliative care Hospice/palliative care team members: physicians, nurses, nurse assistants Patient self-completed survey, Patient clinical data (nurse assessed), Administrative data, Clinician interviews Patients: - Acceptability of intervention survey (study created) Patient clinical data: - INFO-QoL questionnaire (study created) - Palliative Outcome Scale (Italian version) Administrative data: - Use of INFO-QoL questionnaire - Composite Patient Management Score (adapted) Clinicians—nurse interviews: - Experience with intervention I, III, IV
Chang, 2021, End of Life Care for All, Palliative Care Education and Practice, PUMCH-Johns Hopkins Geriatrics conference, and Doctor-Patient Communication Teachers’ training Program (42) To teach the concept and core principles of palliative care and develop these skills Residents Assessed observations in practice/simulation (simulated patients), Clinician self-completed survey Assessed observations in practice/simulation: - Humanities and medical skills test score (study created) Clinicians—medical residents: - Self-assessment questionnaire on interest in humanistic medicine (study created) II, III
Curtis, 2011 (43) To improve ICU end-of-life care ICU staff (nurses) Self-completed survey: Clinicians and Family Members, Administrative data Clinicians—nurses: - Quality of Dying and Death Questionnaire Families/caregivers: - Quality of Dying and Death Questionnaire - Family Satisfaction in the ICU Administrative data: - ICU days before death - Time from admission to withdrawal of mechanical ventilation II, III, IV
Göriş, 2017 (44) To analyze the effect of terminal patient care training on nurses’ attitudes toward death in an oncology hospital Nurses Clinician self-completed survey Clinicians—nurses: - Death Attitude Profile-Revised Scale - Nurse description form questionnaire (nurse characteristics including training) (study created) II
Kinley, 2014†, GSFCH programme (45) To facilitate the provision of quality end-of-life care Nurse managers Administrative data, Qualitative focus groups Administrative data: - Resident place of death - Use of integrated care pathway for last days of life - Use of other end-of-life care tools - Completion of GSFCH accreditation Clinicians—nurse focus groups: - Themes of challenging issues IV
Kruse, 2008*, Teaching Toolkit for Nursing Excellence at End-of-Life Transition and End-of-Life Nursing Education Consortium curriculum (46) To change perceptions of end-of-life care Nurses Clinician self-completed survey Clinicians—nurses: - Perceptions on End-of-Life Care Questionnaire II
Lamppu, 2019; 2021† (47,48) To determine whether staff training in palliative and end-of-life care would benefit residents’ health-related quality of life or reduce their hospital days Registered nurses, practical nurses, physicians Clinician self-completed survey, Patient clinical data (nurse assessed) Clinicians—mixed: - Intervention evaluation and meeting of objectives rating (study created) Patient clinical data: - 15-Dimensional Health-Related Quality-of-Life Instrument (15D) - Mini-Mental State Examination - Mini Nutritional Assessment - Clinical Dementia Rating Scale - Edmonton symptom assessment system - Pain Assessment in Advanced Dementia scale - Psychosocial Well-Being scale - Satisfaction with Care at the End-of-Life in Dementia scale Administrative data: - Hospital days over 24 months - Emergency department visits over 24 months - Cost of all hospital services over 24 months I, III, IV
Llobera, 2017†, Protocol—published [completed trial not published] (49) To assess the effect of appointing a palliative care leader/expert to each health care team Physicians, Nurses, Psychologists Clinician self-completed survey, Administrative data Clinicians—mixed: - Palliative Needs Questionnaire (NECPAL CCOMS-ICO) - Identification and Classification of Patients Needing Palliative Care Questionnaire Administrative data: - Early identification (90 days before death) - Home death - Hospital admissions in final month - Emergency department visits in final month - Health care cost in final month II, III, IV
Okumura-Hiroshige, 2020**, End-of-Life Nursing Education Consortium-Geriatric (50) To improve attitudes and knowledge in palliative care provision Nurses Clinician self-completed survey Clinicians—nurses: - End-of-Life care nursing attitude scale for Japanese geriatric nurses - End-of-Life nursing knowledge scale for Japanese geriatric nurses II
Omidi, 2020 (51) To improve perceptions of and clinical competency in providing end-of-life care to patients with cancer Oncology nurses Clinician self-completed survey Clinicians—nurses: - Oncology Nurses’ Perception Of End-of-Life Care Questionnaire - Nurses’ Competency in the Provision of Palliative Care Survey II
Rosemann, 2007** (52); Hermann, 2012** (53); Engeser, 2014** (54), PAMINO (Palliative Medical Initiative North Baden) To compare an interdisciplinary palliative care training program vs. usual palliative care for malignant tumour patients in a primary care setting Primary care physicians Self-completed survey: Clinicians, Patients, and Families Clinicians—physicians: - Palliative Outcome Scale (clinician rating) - Health service resource use questionnaire (study created) - Process indicators (Advance directive, Alternative arrangement for care, Cooperation with nursing services) Patients: - Palliative Outcome Scale (self rating) - Quality of Life Questionnaire Core-15 Palliative Care - Visual analogue scale for assessing pain Families/caregivers: - Burden Scale for Family Caregivers II, IV
Seow, 2021†, CAPACITI (Community Access to PAlliative Care via Interprofessional Teams Improvement) Protocol—unpublished (registered NCT05120154) To help primary care teams to operationalize an early palliative care approach Primary care providers (physicians, nurses, social workers, managers, etc.) Clinician self-completed survey qualitative focus groups Clinicians—mixed: - Identification of patients requiring palliative care (study created) - End-of-Life Professional Caregiver Survey - Assignment completion & change survey (study created) - CAPACITI Competencies Survey (study created) - Assessment of Interprofessional Team Collaboration Scale II - Organizational Readiness to Change Assessment survey - Session evaluation survey (study created) - Experience with intervention [Focus Groups] I, II
Thoonsen, 2015, 2019, RADboud university medical centre indicators for Palliative Care needs tool and training programme (55,56) To increase early identification of patients requiring palliative care and to facilitate anticipatory, interdisciplinary palliative care planning General practitioners Clinician self-completed survey (including reporting from medical records) Clinicians—physicians: - Patient identification using RADboud indicators for PAlliative Care tool - Survey reporting indicators from patient medical record (study created) including cause/place of death, GP contacts and home visits in final month, GP out-of-hours service and hospital admissions in final 3 months, explored dimensions [somatic, social and financial, activities of daily living, and spiritual and psychological] II, III
Uslu-Sahan, 2020, Interprofessional Gynecologic Oncology Palliative Care Training (57) To increase gynecologic oncology palliative care knowledge Interprofessional students (nursing, medical, nutrition-dietician, and social work) Clinician self-completed survey Clinicians—mixed: - Palliative Care Knowledge Test - Interdisciplinary Education Perception Scale - Teamwork Attitudes Questionnaire II
Van den Block, 2020†, Palliative Care for Older People Steps to Success Program (58) To improve quality of palliative care provision Nurses, care assistants Self-completed survey: Clinicians and Patients Clinicians—mixed: - Palliative Care Survey (PCS-Knowledge of palliative care) - Self-Efficacy in End-of-Life Care Survey - End-of-Life Professional Caregiver Survey - Opinions on palliative care (RotterdamMove2PC) survey Patients: - Comfort Assessment in Dying with Dementia Scale - Quality of Dying in Long-Term Care Families/caregivers: - Comfort Assessment in Dying with Dementia Scale - Satisfaction with Care at the End-of-Life in Dementia scale - Family Perception of Physician-Family Communication II, III, IV
Yeun, 2015*, Palliative Care Professional Education program (59) To increase palliative care knowledge and recognition of a good death Nurses Clinician self-completed survey Clinicians—nurses: - Good Death Recognition Tool - Modified Palliative Care Tool (recognition of palliative care) - Meaning of Life Tool II

All randomized controlled trials except where noted. We include the published protocol reference in cases where the published study featured only one aspect of a multi-part study. †, clustered randomized controlled trials; *, randomized quasi-experimental study; **, non randomized controlled trial; £, Level I = reaction, II = learning, III = behaviour, IV = results. GP, general practitioner; Y/N, yes/no; COPD, chronic obstructive pulmonary disease; ICU, intensive care unit; INFO-QoL, intervention focused on quality of life measurement; GSFCH, Gold Standards Framework for Care Homes.

Table 2

Communication focus studies

Lead author, year, intervention name Objective Target professions Data sources (TYPE) Outcome measures Kirkpatrick model levels£
Brown, 2018, Codetalk (60) To improve competence in palliative care communication Internal medicine residents, medicine subspecialty fellows, nurse practitioner students, or community-based advanced practice nurses Clinician self-completed survey Clinicians—mixed: - Palliative care communication competence survey (study created) II
Curtis, 2013 (61) To improve provider communication with seriously ill patients Internal medicine trainees and nurse practitioner students Self-completed survey: Patients, Families, and Clinician assessors Patients: - Quality of Communication Questionnaire - Quality of End-Of-Life Care Questionnaire - Personal Health Questionnaire (PHQ-8) - Short-Form Health Survey (SF-12) Families/caregivers: - Quality of Communication Questionnaire - Personal Health Questionnaire (PHQ-8) Clinician assessors: - Quality of Communication Questionnaire II, III, IV
Epstein, 2017, Values and Options in Cancer Care program (62) To promote patient-centered communication about disease course, prognosis, treatment decisions, and end-of-life care Medical oncologists Assessed observations in practice/simulation (real patients), Self-completed survey: Clinicians and Patients, Administrative data Assessed observations in practice/simulation: - Active Patient Participation Coding (engaging patients in consultation) - Verona VR-CoDES (responding to patients’ emotions) Clinicians—physicians: - The Human Connection scale Health Care Climate Questionnaire - Perceived Efficacy in Patient-Physician Interactions Scale - Estimated 2-year survival and curability of the patient’s cancer (agreement with patient’s estimate) Patients: - McGill Quality of Life Single-Item Scale - McGill Psychological Well-Being subscale - McGill Existential Well-Being subscale - Functional Assessment of Cancer Therapy—General Questionnaire Physical Functioning subscale - Functional Assessment of Cancer Therapy—General Questionnaire Social Functioning subscale Administrative data: - Aggressive treatment last 30 days of life (chemotherapy, potentially burdensome interventions, emergency department /hospital admission) - Hospice utilization II, III, IV
Goelz, 2011, COM-ON-p (SPIKES/Oncotalk based) (63) To improve competence in early palliative care communication Physicians Assessed observations in practice/simulation (simulated patients) Assessed observations in practice/simulation: - COM-ON Checklist III
Hallford 2011, protocol—published [completed trial not published] (64) To evaluate the effect of an intervention on palliative care staff to improve the recognition of depression and provide support Non-physician palliative care staff Clinicians self-completed survey, Interviews: Staff and Family, Administrative data Clinicians—mixed: - Knowledge of depression survey (study created) - Depression Attitude Questionnaire (adapted) - Self-efficacy in Detecting and Managing Depression Questionnaire (adapted) - Barriers to detecting and managing depression survey (study created) Administrative data: - Referrals made for depressive symptoms Interviews (Staff and Family): - Changes in staff practices II, III
Harnischfeger, 2020, PALLI-KOM, protocol—published [completed trial not published] (65) To evaluate a newly developed communication skills training program Physicians Assessed observations in practice/simulation (simulated patients), Self-completed survey: Clinicians and Patients (simulated) Assessed observations in practice/simulation: - COM-ON Checklist (German version) - Consideration of palliative care principles (based on EAPC-competencies) (study created) Clinicians—physicians: - Self-Efficacy in Palliative Care Scale (German version) - Thanatophobia Scale - Knowledge about palliative care services (study created) - Acceptance of and satisfaction with the training (study created) - Confidence in dealing with palliative care related topics (study created) Patients (simulated): - Quality of Physician-Patient-Interaction Questionnaire (German version) II, III, IV
Koropchak, 2006, SCOPE (Studying Communication in Oncologist Patient Encounters), protocol—published [completed trial not published] (66) To evaluate the effect of a communication intervention on oncologists Oncologists Assessed observations in practice/simulation (real patients), Self-completed survey: Clinicians and Patients Assessed observations in practice/simulation: - Communication behaviors scale (study created) Clinicians—oncologists: - Study created survey based on items from prior surveys (outcome expectancies; confidence; comfort level for addressing patients’ emotional concerns) Patients: - Study created survey based on items from prior surveys (anxiety and depression; self-rating of communication with their oncologists; satisfaction with care, quality of life, social support) II, III, IV
Lin, 2021 (67) To evaluate the effectiveness of an end-of-life education program on shared decision making attitudes among nurses Nursing professionals (who have received formal nursing education) Assessed observations in practice/simulation (simulated patients), Clinician self-completed survey Assessed observations in practice/simulation: - Standardized patient survey scale (study created) - Standardized patient survey score (study created) - Standardized patient survey global score (study created) Clinicians—nurses: - Substitute decision maker attitude scale (study created) - Substitute decision maker attitude score (study created) - Substitute decision maker attitude global score (study created) II, III
Liu, 2020, Palliative Care Needs Rounds (68) To determine if the intervention improves staff perceptions of residents’ quality of death/dying, staff self-reported capability to care for people in the last months of life, and completion of anticipatory care documents Registered nurses, enrolled nurses, nursing aides, activities coordinators, managers Clinician self-completed survey, Administrative data Clinicians—mixed: - Quality of Dying and Death Questionnaire Short Form - Capacity to Adopt a Palliative Approach Tool Administrative data: - Completion of advance care plan - Appointment of medical power of attorney II, III
Martinsson, 2016, Information about the transition to end-of-life (69) To determine if an educational intervention for physicians and nurses in nursing homes and hospitals increases the number of patients who receive information about the transition to end-of-life Physicians, nurses Administrative data (registry based on clinician submitted data) Administrative data: - Patients who received information about the transition to end-of-life III
Paladino, 2019† (70); Bernacki, 2019† (71), Serious Illness Conversation Guide training To improve the occurrence, timing, quality, and accessibility of documented serious illness conversations Physicians, advanced-practice clinicians Self-completed survey: Patient and Family Caregiver, Administrative data Patients: - Life Priorities survey for patients survey (study created) - Family perceptions survey (study created) - Peace, Equanimity, and Acceptance in the Cancer Experience Questionnaire - The Human Connection scale - Generalized Anxiety Disorder (GAD-7) Scale - Patient Health Questionnaire (PHQ-9) Families/caregivers: - Concordance with patient’s end-of-life goals (study created) Administrative data: - Median 2-year survival - Serious illness conversation completed - Timing of first conversation - Discussion about values or goals - Discussions about end-of-life care planning - Conversations and domains discussed - Discussion documented in the accessible structured EMR module III, IV
Slort, 2013, ACA (Availability, Current Issues and Anticipation) training programme (72) To improve competence in palliative care communication General practitioners Assessed observations in practice/simulation (simulated patients) Assessed observations in practice/simulation: - Roter Interaction Analysis System availability items III
Szmuilowicz, 2010 (73) To enhanced performance of and confidence with end-of-life conversations Internal medical residents Assessed observations in practice/simulation (simulated patients), Clinician self-completed survey Assessed observations in practice/simulation: - Communication competence (study created tool) Clinicians—medical residents: - Communication competence (study created tool) II, III
Totten, 2019†, SICP, protocol—published [completed trial not published] (74) To compare a team based vs. clinician focused approach to implementing an existing advanced care planning program Administrators, nurses and/or medical assistants, clinicians Self-completed survey: Clinicians, Patients, and Caregivers Clinicians—mixed: - Experience with SICP questions (study created) - Reported use of SICP Patients: - Goal-concordant care survey (study created) - Patient-Reported Outcomes Measurement Information System (PROMIS-29) (anxiety, depression, and health) - Control Preferences Scale (decision making) - Use of health care services survey (study created) - Experience with SICP questions (study created) Families/caregivers: - Zarit Burden Interview Scale - Patient-Reported Outcomes Measurement Information System (PROMIS-29) - Hospice use, location of death, and family bereavement questions (study created) - Experience with SICP questions (study created) I, II, IV
Walczak, 2014, CSP (Communication Support Programme), protocol—published [completed trial not published] (75) To facilitate patients’ and caregivers’ ability to communicate about life expectancy and end-of-life care with their healthcare team Currently practicing medical oncologists Self-completed survey: Patients and Caregivers, Interviews: Caregivers, Administrative data Patients: - Perceived Efficacy in Patient Physician Interactions Scale - Functional Assessment of Cancer Therapy—General Questionnaire - McGill Quality of Life Questionnaire - Control Preferences Scale (decision making) (adapted) - Information Styles Questionnaire (adapted - 2 items) - Peace, Equanimity, and Acceptance in the Cancer Experience Questionnaire - Understanding of life expectancy from physician scale (study created) - Rating of doctor’s communication skills and manner during consultations scale (study created) - Satisfaction with care (study created) Families/caregivers: - Perceived Efficacy in Patient-Physician Interactions Scale (adapted) - Short Form Health Survey (SF-36 V.2) - Control Preferences Scale (decision making) (adapted) - Information Styles Questionnaire (adapted-2 items) - Caregivers’ understanding of the patients’ prognosis (study created) - Understanding of life expectancy from physician scale (study created) - Quality of Dying and Death Questionnaire [Interviews] Administrative data: - Health care in the final month including cancer-focused treatment, emergency intervention, and palliative care III, IV
Wilkinson, 2008, Wilkinson training course from the National Advanced Communication Skills Programme for Senior Health Care Professionals in Cancer Care (76) To increase awareness of end-of-life care communication skills and to explore strategies to implement these skills in practice Nurses Assessed observations in practice/simulation (real patients), Patient self-completed survey Assessed observations in practice/simulation: - Communication Skills Rating Scale - Communication Skills Confidence Questionnaire Patients: - State Anxiety Scale - General Health Questionnaire (GHQ-12) - Patient Satisfaction Questionnaire III, IV

All randomized controlled trials except where noted. We include the published protocol reference in cases where the published study featured only one aspect of a multi-part study. †, clustered randomized controlled trials; £, Level I = reaction, II = learning, III = behaviour, IV = results. VR-CoDES, Verona coding definitions of emotional sequences; COM-ON, Communication in Oncology; EAPC, European Association for Palliative Care; EMR, electronic medical record; SICP, Serious Illness Care Program.

Description of studies and interventions

Thirty of the studies included were randomized controlled trials (RCTs) (9 clustered RCTs, 21 RCTs), along with four non-RCTs and two randomized quasi experimental studies (Tables 1,2). The studies were conducted in Europe (n=14), North America (n=11), Asia (n=7), and Australia (n=4). The target audience for almost half (n=16) of the educational interventions was interprofessional, the rest were aimed specifically at physicians/residents (n=10) or nurses (n=10). Six studies included oncology care providers (including 3 studies with nurses), however two of these studies were research protocols with no results. Participant numbers among the completed studies ranged from about 20 to over 1,000 individuals in each arm. The education program duration ranged from a half day to 20 months, with 16 (44%) of the training interventions being completed in days (less than a week), 9 (25%) completed in 1 to 4 weeks, and 11 (31%) completed over a month or more. Details on the education interventions and groups of comparisons for the completed studies are provided in our prior review (10).

Description of measures used and common measures

Over half of the trials (n=23) used data from multiple data sources including participant questionnaires, administrative records, and external observer ratings. Overall, 28 studies (78%) measured clinician-reported outcomes and 15 (42%) measured a combination of patient or family reported outcomes and clinician-reported outcomes, using questionnaires (Tables 1,2). Half of all trials employed a study-created questionnaire(s). Data from administrative (n=14) sources were also used to examine changes in clinicians’ practice and/or patient outcomes, following the training intervention. Seven trials used a mixed methods approach that included qualitative data (semi-structured interview or focus group) with clinicians (n=6) and/or patients/families (n=4), largely examining perceptions of the education intervention or resulting changes to practice. Specific to communication focused studies, half of these analysed clinician interactions with real or simulated patients as an outcome. Many of the trials (58%) collected final data within three months of completion of the training program, with 6 studies (17%) measuring final outcomes immediately after the intervention.

A total of 11 different validated questionnaire measures were used in at least 2 trials each, including the End-of-Life Professional Caregiver Survey (EPCS) (clinicians) and the Quality of Dying and Death Questionnaire (QODD) (caregivers) (see Table 3). Based on administrative data, 12 studies examined health care use, including costs (n=3), hospitalizations (n=5) and/or patient place of death (n=3) (see Table S1).

Table 3

Commonly used questionnaire measures and Kirkpatrick level of impact*

Measure Kirkpatrick framework level
Reaction: I Learning: II Behavior: III Results: IV
Communication in Oncology Checklist ×
Control Preferences Scale (decision making) ×
End-of-Life Professional Caregiver Survey ×
McGill Quality of Life Questionnaire ×
Palliative Outcome Scale ×
Peace, Equanimity, and Acceptance in the Cancer Experience Questionnaire ×
Perceived Efficacy in Patient Physician Interactions Scale adapted × ×
Quality of Dying and Death Questionnaire ×
Satisfaction with Care at the End-of-Life in Dementia scale ×
Short Form Health Survey (SF-36/12) ×
The Human Connection scale × ×

*, used by at least two unrelated studies.

Study outcome categories

Among all the trials, 190 different measures were reported. We grouped all measures used/proposed into 14 outcome categories (see Table S1 for all measures by category). These categories were determined based on our review and thematic ordering of the phenomena being examined among the measures. Table 4 presents a summary of outcome categories and the number of trials with measures in each category, for all studies and those specific to communication. Approximately one third (n=13) of the trials measured clinician’s palliative care knowledge or confidence in the skills they had acquired through the intervention. Most of the communication skill trials (12 of 16) measured “quality of communication”, this being through an external source (rater or patient). The remaining four studies measured clinician-reported use (74), clinician-rated competency in initiating appropriate dialogue with their patients (64,68), or evidence of these conversations in the administrative record (68,69). The other outcome categories were each represented in a third or less of the trials identified.

Table 4

Number of studies with measures in each outcome category

Outcome categories Number of studies (n=36) % of all studies Number of communication studies (n=16) % of communication studies
Quality of communication 13 36.1 12 75.0
Palliative care knowledge/confidence 13 36.1 4 25.0
Patient or family satisfaction/experience 12 33.3 6 37.5
Health care costs and/or use 12 33.3 3 18.8
Patient general health/functionality/symptoms 11 30.6 5 31.3
Provider use of intervention tools 11 30.6 5 31.3
Mental health patient reported outcomes 9 25.0 6 37.5
Quality of life 9 25.0 3 18.8
Provider satisfaction/experience with intervention 8 22.2 2 12.5
Palliative care attitudes 7 19.4 2 12.5
Provider/patient relationship 4 11.1 3 18.8
Caregiver burden/involvement 4 11.1 1 6.3
Teamwork/collaboration 3 8.3 0 0.0
Other measures, e.g., survival, goal concordance, symptom management, advance care planning 13 36.1 6 37.5

Kirkpatrick’s four levels of training evaluation

We mapped the trial outcomes onto the Kirkpatrick model evidence levels (33) (Tables 1,2). Most studies (78%) used clinician reported outcomes, such as self-reported competency, which relate to Level I: satisfaction with the intervention or Level II: perception of learning. An example of a Level I measure is “Experience with SICP (Serious Illness Conversation Program)” survey and Level II, the EPCS. Assessed observations in practice/simulation (25% of all studies, 50% of communication-focused studies), which involves assessment of the clinician’s interaction with a patient by an external reviewer, offers more objective evidence of clinician behavior change, denoting Level III. Level III outcomes also include clinician use of palliative care processes or tools, reported by the patient/family or extracted from administrative data. Overall, 24 (67%) trials measured Level III and 18 (50%) trials measured Level IV evidence, the latter signifying patient and system outcomes.

Table 2 presents the Kirkpatrick model level of impact assessed among the commonly used questionnaire measures identified. The majority (73%) of these measures are patient-reported outcomes (PROs) and therefore address Level IV outcomes on the Kirkpatrick model.

Discussion

Our review identified a relatively large number of trials (n=36), including protocols, of education interventions to increase generalist clinician capacity for providing a palliative approach to care. Nearly half of the studies were explicitly communication skills programs. As with prior reviews of patient palliative care interventions (16,18,24), our current study found a multitude of different measures, with extensive diversity in the types of outcomes assessed. There was little consistency in measures used between the studies; only 11 validated questionnaires were used in at least 2 of the trials we identified. Half of the studies employed a study-created questionnaire, often to measure clinician competency and/or knowledge. A vast majority (n=28, 78%) of the education intervention trials evaluated clinician reported outcomes. In contrast, fewer (n=15, 42%) of the studies assessed patient or family reported outcomes, a shortcoming inherent to the body of palliative care training research noted in prior reviews (8,17,77,78). To our knowledge, our systematic review is the first to closely examine the outcome measures of palliative care training programs for clinicians.

In reviewing studies of education interventions in health care, a broad range of outcomes are apparent, from clinician attendance at the training and their satisfaction, to system-level impact, such as acute care costs (10,18,79). Multiple levels of evidence are required to critically appraise educational interventions aimed at clinicians (33,80,81). While participant satisfaction with the program and the perception of skill acquisition are formative to knowledge uptake and application in practice, objective measures of behavior change—such as assessment of observations with simulated or real patient encounters—are necessary to empirically demonstrate the effectiveness of an education intervention (82-84).

Insights and recommendations emerged from our review regarding the incorporation of higher level outcomes as described in the Kirkpatrick model (33), namely Level III: behavior and Level IV: results (PROs). The use of validated patient/family reported outcomes relevant to palliative care, particularly those that capture care satisfaction, as opposed to less modifiable qualities such as health status, can provide objective metrics of clinician behavior. Patient or family satisfaction is more likely to be sensitive to clinician education interventions in palliative care than clinical outcomes, especially considering the majority of these studies have a short duration (≤3 months) between the intervention and post-assessment (10). In fact, it may be unrealistic to expect patient-reported outcomes such as quality of life, mental health, or physical health to be immediately impacted by clinician training. Lastly, health care system outcomes, such as acute care use, are a high level result of behavior change, though these outcomes may be too far downstream from a newly implemented education program for clinicians to show an effect (16,25,85,86).

Using assessed observations in practice or simulation shows promise for realizing more objective measurement in studies of palliative care education. Indeed, many of the trials of communication skills training used assessed observations of clinician interactions with patients as the main study outcome. However, our review identified only one non-communication specific trial that used observation of practice (42). This finding is consistent with that of a 2018 review of simulation training in palliative care which reported that the vast majority of these studies centre on communication skills, rather than including other domains of palliative care (87). The development and use of observed assessment to measure clinician behavior change has many challenges, including the cost, time, technology, and the logistics involved in implementation (88), in addition to capturing the multiple dimensions that a palliative approach to care entails. There is a need for further validated approaches that can be used to assess clinician/patient interactions in providing comprehensive palliative care.

Our study reveals several recommendations that align with many of those put forth by the European Association for Palliative Care (EAPC) for outcome measurement in palliative care (89). Many trials used data from multiple data sources, including administrative and/or qualitative sources. Using a mixed methods approach with multiple sources, and multidimensional instruments can more holistically capture the impact of a palliative care training intervention. Moving towards some level of international consistency, standard or universality of use of measures is key; the ability to compare across studies that are also evaluating other palliative care education interventions would then be possible. The few common questionnaire examples from this review are the EPCS to assess general palliative care competency and the QODD completed by caregivers, as well as the Communication in Oncology (COM-ON) Checklist for communication focused interventions. Most important is the use of validated measures that reflect the content of the intervention. Assessed observations in practice/simulation, especially for communication focused studies where the expected outcomes are more clearly defined, should be considered along with patient reported outcomes for objectively assessing behavior change in clinicians. Finally, a longer duration between the intervention and final data collection will help to evaluate long term effects of education training and will allow more time for behavior changes to potentially transpire.

Our review has several limitations. We did not assess the suitability of the given measures for each individual intervention and therefore cannot comment on the extent to which the measures represented in each study are reasonable and valid indicators of program impact. We did not assess the psychometric properties of the measures used. Our review includes studies across countries, so factors such as language and practice norms may undermine universal tools. Finally, we limited our review to trials (including registered protocols) to capture what measures were used by studies with the highest purported level of evidence. We recognize that our review excludes a large number of non-trial, single cohort studies; synthesis of the outcomes used could identify additional measures for evaluating palliative care education interventions. Future reviews should examine relevant approaches described in the broader education science literature, particularly methods for objectively capturing clinician behavior change.

Conclusions

We found considerable diversity and little commonality among the measures used in our systematic review of 36 trials examining palliative care education programs for clinicians. Nearly 200 different measures were reported among the studies. Only 11 validated measures were used in at least 2 trials. Further work is required to establish consistent measures of assessing the impact of palliative care education, to inform evidence-based scaling of effective programs.

Acknowledgments

Funding: This work was funded by the Canadian Institutes of Health Research (CIHR) (Grant #408099). See https://cihr-irsc.gc.ca/e/193.html. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Footnote

Reporting Checklist: The authors have completed the PRISMA reporting checklist. Available at https://apm.amegroups.com/article/view/10.21037/apm-22-947/rc

Peer Review File: Available at https://apm.amegroups.com/article/view/10.21037/apm-22-947/prf

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://apm.amegroups.com/article/view/10.21037/apm-22-947/coif). All authors report that this work was funded by the Canadian Institutes of Health Research (CIHR) (Grant #408099). The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

  1. Evans JM, Mackinnon M, Pereira J, et al. Integrating early palliative care into routine practice for patients with cancer: A mixed methods evaluation of the INTEGRATE Project. Psychooncology 2019;28:1261-8. [Crossref] [PubMed]
  2. Tassinari D, Drudi F, Monterubbianesi MC, et al. Early Palliative Care in Advanced Oncologic and Non-Oncologic Chronic Diseases: A Systematic Review of Literature. Rev Recent Clin Trials 2016;11:63-71. [Crossref] [PubMed]
  3. Vanbutsele G, Van Belle S, Surmont V, et al. The effect of early and systematic integration of palliative care in oncology on quality of life and health care use near the end of life: A haracteri controlled trial. Eur J Cancer 2020;124:186-93. [Crossref] [PubMed]
  4. Sawatzky R, Porterfield P, Lee J, et al. Conceptual foundations of a palliative approach: a knowledge synthesis. BMC Palliat Care 2016;15:5. [Crossref] [PubMed]
  5. Quill TE, Abernethy AP. Generalist plus specialist palliative care–creating a more sustainable model. N Engl J Med 2013;368:1173-5. [Crossref] [PubMed]
  6. Li WW, Chhabra J, Singh S. Palliative care education and its effectiveness: a systematic review. Public Health 2021;194:96-108. [Crossref] [PubMed]
  7. Pereira J, Chary S, Faulkner J, et al. Primary-level palliative care national capacity: Pallium Canada. BMJ Support Palliat Care 2021;bmjspcare-2021-003036.
  8. Selman LE, Brighton LJ, Hawkins A, et al. The Effect of Communication Skills Training for Generalist Palliative Care Providers on Patient-Reported Outcomes and Clinician Behaviors: A Systematic Review and Meta-analysis. J Pain Symptom Manage 2017;54:404-416.e5. [Crossref] [PubMed]
  9. DeNofrio JC, Verma M, Kosinski AS, et al. Palliative Care Always: Hepatology-Virtual Primary Palliative Care Training for Hepatologists. Hepatol Commun 2022;6:920-30. [Crossref] [PubMed]
  10. Bainbridge D, Bishop V, Myers J, et al. Effectiveness of Training Programs About a Palliative Care Approach: A Systematic Review of Intervention Trials for Health Care Professionals. J Palliat Med 2022; Epub ahead of print. [Crossref] [PubMed]
  11. Brighton LJ, Koffman J, Hawkins A, et al. A Systematic Review of End-of-Life Care Communication Skills Training for Generalist Palliative Care Providers: Research Quality and Reporting Guidance. J Pain Symptom Manage 2017;54:417-25. [Crossref] [PubMed]
  12. Bishop CT, Mazanec P, Bullington J, et al. Online End-of-Life Nursing Education Consortium Core Curriculum for Staff Nurses: An Education Strategy to Improve Clinical Practice. J Hosp Palliat Nurs 2019;21:531-9. [Crossref] [PubMed]
  13. Chung HO, Oczkowski SJ, Hanvey L, et al. Educational interventions to train healthcare professionals in end-of-life communication: a systematic review and meta-analysis. BMC Med Educ 2016;16:131. [Crossref] [PubMed]
  14. Anstey S, Powell T, Coles B, et al. Education and training to enhance end-of-life care for nursing home staff: a systematic literature review. BMJ Support Palliat Care 2016;6:353-61. [Crossref] [PubMed]
  15. Bassah N, Seymour J, Cox K. A modified systematic review of research evidence about education for pre-registration nurses in palliative care. BMC Palliat Care 2014;13:56. [Crossref] [PubMed]
  16. Kamal AH, Gradison M, Maguire JM, et al. Quality measures for palliative care in patients with cancer: a systematic review. J Oncol Pract 2014;10:281-7. [Crossref] [PubMed]
  17. Kearns T, Cornally N, Molloy W. Patient reported outcome measures of quality of end-of-life care: A systematic review. Maturitas 2017;96:16-25. [Crossref] [PubMed]
  18. Mularski RA, Dy SM, Shugarman LR, et al. A systematic review of measures of end-of-life care and its outcomes. Health Serv Res 2007;42:1848-70. [Crossref] [PubMed]
  19. Parker D, Hodgkinson B. A comparison of palliative care outcome measures used to assess the quality of palliative care provided in long-term care facilities: a systematic review. Palliat Med 2011;25:5-20. [Crossref] [PubMed]
  20. Stevens E, Martin CR, White CA. The outcomes of palliative care day services: a systematic review. Palliat Med 2011;25:153-69. [Crossref] [PubMed]
  21. Hearn J, Higginson IJ. Outcome measures in palliative care for advanced cancer patients: a review. J Public Health Med 1997;19:193-9. [Crossref] [PubMed]
  22. Potts M, Cartmell KB, Nemeth L, et al. A Systematic Review of Palliative Care Intervention Outcomes and Outcome Measures in Low-Resource Countries. J Pain Symptom Manage 2018;55:1382-97.e7. [Crossref] [PubMed]
  23. Smith TJ, Cassel JB. Cost and non-clinical outcomes of palliative care. J Pain Symptom Manage 2009;38:32-44. [Crossref] [PubMed]
  24. Teno JM, Okun SN, Casey V, et al. Toolkit of Instruments to Measure End of Life Care (TIME). Resource Guide: Achieving Quality of Care at Life’s End. 2005.
  25. Bainbridge D, Seow H. Measuring the quality of palliative care at end of life: an overview of data sources. Healthy aging & Clinical care in the elderly 2016;8:9-15. [Crossref]
  26. Page MJ, Moher D, Bossuyt PM, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ 2021;372: [Crossref] [PubMed]
  27. McMaster University. Search Filters for MEDLINE in Ovid Syntax and the PubMed translation 2022. Available online: https://hiru.mcmaster.ca/hiru/hiru_hedges_medline_strategies.aspx
  28. Van Mechelen W, Aertgeerts B, De Ceulaer K, et al. Defining the palliative care patient: a systematic review. Palliat Med 2013;27:197-208. [Crossref] [PubMed]
  29. Rowe AK, Rowe SY, Peters DH, et al. Effectiveness of strategies to improve health-care provider practices in low-income and middle-income countries: a systematic review. Lancet Glob Health 2018;6:e1163-75. [Crossref] [PubMed]
  30. Jacobsen J, Jackson V, Dahlin C, et al. Components of early outpatient palliative care consultation in patients with metastatic nonsmall cell lung cancer. J Palliat Med 2011;14:459-64. [Crossref] [PubMed]
  31. Ferrell BR, Twaddle ML, Melnick A, et al. National Consensus Project Clinical Practice Guidelines for Quality Palliative Care Guidelines, 4th Edition. J Palliat Med 2018;21:1684-9.
  32. Ouzzani M, Hammady H, Fedorowicz Z, et al. Rayyan-a web and mobile app for systematic reviews. Syst Rev 2016;5:210. [Crossref] [PubMed]
  33. Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick’s four levels of training evaluation: Association for Talent Development. 2016.
  34. Allen LM, Hay M, Palermo C. Evaluation in health professions education-Is measuring outcomes enough? Med Educ 2022;56:127-36. [Crossref] [PubMed]
  35. Agar M, Beattie E, Luckett T, et al. Pragmatic cluster haracteri controlled trial of facilitated family case conferencing compared with usual care for improving end of life care and outcomes in nursing home residents with advanced dementia and their families: the IDEAL study protocol. BMC Palliat Care 2015;14:63. [Crossref] [PubMed]
  36. Agar M, Luckett T, Luscombe G, et al. Effects of facilitated family case conferencing for advanced dementia: A cluster haracteri clinical trial. PLoS One 2017;12:e0181020. [Crossref] [PubMed]
  37. Ahlström G, Nilsen P, Benzein E, et al. Implementation of knowledge-based palliative care in nursing homes and pre-post post evaluation by cross-over design: a study protocol. BMC Palliat Care 2018;17:52. [Crossref] [PubMed]
  38. Bökberg C, Behm L, Wallerstedt B, et al. Evaluation of person-centeredness in nursing homes after a palliative care intervention: pre- and post-test experimental design. BMC Palliat Care 2019;18:44. [Crossref] [PubMed]
  39. Broese JMC, van der Kleij RMJJ, Kerstjens HAM, et al. A cluster randomized controlled trial on a multifaceted implementation strategy to promote integrated palliative care in COPD: study protocol of the COMPASSION study. BMC Palliat Care 2020;19:155. [Crossref] [PubMed]
  40. Catania G, Bagnasco A, Signori A, et al. A phase 2 quasi-experimental trial evaluating the feasibility, acceptability, and potential effectiveness of complex nursing intervention focused on QoL assessment on advanced cancer patients with palliative care needs: study protocol. Pilot Feasibility Stud 2017;3:54. [Crossref] [PubMed]
  41. Catania G, Zanini M, Signori A, et al. Providing a nurse-led complex nursing Intervention Focused on quality of life assessment on advanced cancer patients: The INFO-QoL pilot trial. Eur J Oncol Nurs 2021;52:101961. [Crossref] [PubMed]
  42. Chang J, Qi Z, Jiang S, et al. The impact of palliative care education and training program on the resident physicians. Ann Palliat Med 2021;10:2758-65. [Crossref] [PubMed]
  43. Curtis JR, Nielsen EL, Treece PD, et al. Effect of a quality-improvement intervention on end-of-life care in the intensive care unit: a randomized trial. Am J Respir Crit Care Med 2011;183:348-55. [Crossref] [PubMed]
  44. Göriş S, Taşcı S, Özkan B, et al. Effect of Terminal Patient Care Training on the Nurses’ Attitudes Toward Death in an Oncology Hospital in Turkey. J Cancer Educ 2017;32:65-71. [Crossref] [PubMed]
  45. Kinley J, Stone L, Dewey M, et al. The effect of using high facilitation when implementing the Gold Standards Framework in Care Homes programme: a cluster haracteri controlled trial. Palliat Med 2014;28:1099-109. [Crossref] [PubMed]
  46. Kruse BG, Melhado LW, Convertine L, et al. Evaluating strategies for changing acute care nurses’ perceptions on end-of-life care. Am J Hosp Palliat Care 2008;25:389-97. [Crossref] [PubMed]
  47. Lamppu PJ, Finne-Soveri H, Kautiainen H, et al. Effects of Staff Training on Nursing Home Residents’ End-Of-Life Care: A Randomized Controlled Trial. J Am Med Dir Assoc 2021;22:1699-705.e1. [Crossref] [PubMed]
  48. Lamppu PJ, Laurila J, Finne-Soveri H, et al. Training nursing home staff to improve residents’ end-of-life care: design and baseline findings from a randomized controlled trial. Eur Geriatr Med 2019;10:649-57. [Crossref] [PubMed]
  49. Llobera J, Sansó N, Ruiz A, et al. Strengthening primary health care teams with palliative care leaders: protocol for a cluster randomized clinical trial. BMC Palliat Care 2017;17:4. [Crossref] [PubMed]
  50. Okumura-Hiroshige A, Fukahori H, Yoshioka S, et al. Effect of an end-of-life gerontological nursing education programme on the attitudes and knowledge of clinical nurses: A non-randomised controlled trial. Int J Older People Nurs 2020;15:e12309. [Crossref] [PubMed]
  51. Omidi K, Dehghan M, Shahrbabaki PM. Effectiveness of A Traditional Training Method in Increasing Long-Term End-of-Life Care Perception and Clinical Competency among Oncology Nurses: A Pilot Clinical Trial. Asia Pac J Oncol Nurs 2020;7:287-94. [Crossref] [PubMed]
  52. Rosemann T, Hermann K, Miksch A, et al. The PAMINO-project: evaluating a primary care-based educational program to improve the quality of life of palliative patients. BMC Palliat Care 2007;6:5. [Crossref] [PubMed]
  53. Hermann K, Engeser P, Szecsenyi J, et al. Palliative patients cared for at home by PAMINO-trained and other GPs – health-related quality of life as measured by QLQ-C15-PAL and POS. BMC Palliat Care 2012;11:13. [Crossref] [PubMed]
  54. Engeser P, Hermann K, Szecsenyi J, et al. Symptom control and place of death in palliative cancer patients in primary care. Results of the controlled PAMINO evaluation study. Schmerz 2014;28:607-13. [Crossref] [PubMed]
  55. Thoonsen B, Gerritzen SHM, Vissers KCP, et al. Training general practitioners contributes to the identification of palliative patients and to multidimensional care provision: secondary outcomes of an RCT. BMJ Support Palliat Care 2019;9:e18. [Crossref] [PubMed]
  56. Thoonsen B, Vissers K, Verhagen S, et al. Training general practitioners in early identification and anticipatory palliative care planning: a randomized controlled trial. BMC Fam Pract 2015;16:126. [Crossref] [PubMed]
  57. Uslu-Sahan F, Terzioglu F. Interprofessional simulation-based training in gynecologic oncology palliative care for students in the healthcare profession: A comparative randomized controlled trial. Nurse Educ Today 2020;95:104588. [Crossref] [PubMed]
  58. Van den Block L, Honinx E, Pivodic L, et al. Evaluation of a Palliative Care Program for Nursing Homes in 7 Countries: The PACE Cluster-Randomized Clinical Trial. JAMA Intern Med 2020;180:233-42. [Crossref] [PubMed]
  59. Yeun YR, Kwon M, Lee KS. Development and evaluation of an education program for professional palliative care nursing. J Korean Acad Nurs 2015;45:139-46. [Crossref] [PubMed]
  60. Brown CE, Back AL, Ford DW, et al. Self-Assessment Scores Improve After Simulation-Based Palliative Care Communication Skill Workshops. Am J Hosp Palliat Care 2018;35:45-51. [Crossref] [PubMed]
  61. Curtis JR, Back AL, Ford DW, et al. Effect of communication skills training for residents and nurse practitioners on quality of communication with patients with serious illness: a randomized trial. JAMA 2013;310:2271-81. [Crossref] [PubMed]
  62. Epstein RM, Duberstein PR, Fenton JJ, et al. Effect of a Patient-Centered Communication Intervention on Oncologist-Patient Communication, Quality of Life, and Health Care Utilization in Advanced Cancer: The VOICE Randomized Clinical Trial. JAMA Oncol 2017;3:92-100. [PubMed]
  63. Goelz T, Wuensch A, Stubenrauch S, et al. Specific training program improves oncologists’ palliative care communication skills in a randomized controlled trial. J Clin Oncol 2011;29:3402-7. [Crossref] [PubMed]
  64. Hallford DJ, McCabe MP, Mellor D, et al. Intervention for depression among palliative care patients and their families: A study protocol for evaluation of a training program for professional care staff. BMC Palliat Care 2011;10:11. [Crossref] [PubMed]
  65. Harnischfeger N, Rath HM, Ullrich A, et al. Evaluation of a communication skills training to facilitate addressing palliative care related topics in advanced cancer patients: study protocol of a multicenter randomized controlled trial (PALLI-KOM). BMC Palliat Care 2020;19:67. [Crossref] [PubMed]
  66. Koropchak CM, Pollak KI, Arnold RM, et al. Studying communication in oncologist-patient encounters: the SCOPE Trial. Palliat Med 2006;20:813-9. [Crossref] [PubMed]
  67. Lin MH, Lin SC, Lee YH, et al. The effectiveness of simulation education program on shared decision-making attitudes among nurses in Taiwan. PLoS One 2021;16:e0257902. [Crossref] [PubMed]
  68. Liu WM, Koerner J, Lam L, et al. Improved Quality of Death and Dying in Care Homes: A Palliative Care Stepped Wedge Randomized Control Trial in Australia. J Am Geriatr Soc 2020;68:305-12. [Crossref] [PubMed]
  69. Martinsson L, Heedman PA, Eriksson M, et al. Increasing the number of patients receiving information about transition to end-of-life care: the effect of a half-day physician and nurse training. BMJ Support Palliat Care 2016;6:452-8. [Crossref] [PubMed]
  70. Paladino J, Bernacki R, Neville BA, et al. Evaluating an Intervention to Improve Communication Between Oncology Clinicians and Patients With Life-Limiting Cancer: A Cluster Randomized Clinical Trial of the Serious Illness Care Program. JAMA Oncol 2019;5:801-9. [Crossref] [PubMed]
  71. Bernacki R, Paladino J, Neville BA, et al. Effect of the Serious Illness Care Program in Outpatient Oncology: A Cluster Randomized Clinical Trial. JAMA Intern Med 2019;179:751-9. [Crossref] [PubMed]
  72. Slort W, Blankenstein AH, Schweitzer BP, et al. Effectiveness of the ACA (Availability, Current issues and Anticipation) training programme on GP-patient communication in palliative care; a controlled trial. BMC Fam Pract 2013;14:93. [Crossref] [PubMed]
  73. Szmuilowicz E, el-Jawahri A, Chiappetta L, et al. Improving residents’ end-of-life communication skills with a short retreat: a randomized controlled trial. J Palliat Med 2010;13:439-52. [Crossref] [PubMed]
  74. Totten AM, Fagnan LJ, Dorr D, et al. Protocol for a Cluster Randomized Trial Comparing Team-Based to Clinician-Focused Implementation of Advance Care Planning in Primary Care. J Palliat Med 2019;22:82-9. [Crossref] [PubMed]
  75. Walczak A, Butow PN, Clayton JM, et al. Discussing prognosis and end-of-life care in the final year of life: a haracteri controlled trial of a nurse-led communication support programme for patients and caregivers. BMJ Open 2014;4:e005745. [Crossref] [PubMed]
  76. Wilkinson S, Perry R, Blanchard K, et al. Effectiveness of a three-day communication skills course in changing nurses’ communication skills with cancer/palliative care patients: a haracteri controlled trial. Palliat Med 2008;22:365-75. [Crossref] [PubMed]
  77. Lendon JP, Ahluwalia SC, Walling AM, et al. Measuring Experience With End-of-Life Care: A Systematic Literature Review. J Pain Symptom Manage 2015;49:904-15.e1-3.
  78. Kavalieratos D, Corbelli J, Zhang D, et al. Association Between Palliative Care and Patient and Caregiver Outcomes: A Systematic Review and Meta-analysis. JAMA 2016;316:2104-14. [Crossref] [PubMed]
  79. Chauhan BF, Jeyaraman MM, Mann AS, et al. Behavior change interventions and policies influencing primary healthcare professionals’ practice-an overview of reviews. Implement Sci 2017;12:3. Erratum in: Implement Sci 2017;12:38. [Crossref] [PubMed]
  80. Reio TG, Rocco TS, Smith DH, et al. A critique of Kirkpatrick’s evaluation model. New Horizons in Adult Education and Human Resource Development 2017;29:35-53. [Crossref]
  81. Smidt A, Balandin S, Sigafoos J, et al. The Kirkpatrick model: A useful tool for evaluating training outcomes. J Intellect Dev Disabil 2009;34:266-74. [Crossref] [PubMed]
  82. Atkins L, Francis J, Islam R, et al. A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems. Implement Sci 2017;12:77. [Crossref] [PubMed]
  83. Johnson MJ, May CR. Promoting professional behaviour change in healthcare: what interventions work, and why? A theory-led overview of systematic reviews. BMJ Open 2015;5:e008592. [Crossref] [PubMed]
  84. Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for haracterizing and designing behaviour change interventions. Implement Sci 2011;6:42. [Crossref] [PubMed]
  85. Barbera L, Paszat L, Chartier C. Indicators of poor quality end-of-life cancer care in Ontario. J Palliat Care 2006;22:12-7. [Crossref] [PubMed]
  86. Pasman HR, Brandt HE, Deliens L, et al. Quality indicators for palliative care: a systematic review. J Pain Symptom Manage 2009;38:145-56. [Crossref] [PubMed]
  87. Kozhevnikov D, Morrison LJ, Ellman MS. Simulation training in palliative care: state of the art and future directions. Adv Med Educ Pract 2018;9:915-24. [Crossref] [PubMed]
  88. So HY, Chen PP, Wong GKC, et al. Simulation in medical education. J R Coll Physicians Edinb 2019;49:52-7. [Crossref] [PubMed]
  89. Bausewein C, Daveson BA, Currow DC, et al. EAPC White Paper on outcome measurement in palliative care: Improving practice, attaining outcomes and delivering quality services – Recommendations from the European Association for Palliative Care (EAPC) Task Force on Outcome Measurement. Palliat Med 2016;30:6-22. [Crossref] [PubMed]
Cite this article as: Seow H, Bishop VC, Myers J, Stajduhar KI, Marshall DI, Incardona NK, Levine OH, Steinberg LJ, Bainbridge DB. Outcome measures in palliative care training interventions: a systematic review of trial-based studies. Ann Palliat Med 2023;12(2):399-417. doi: 10.21037/apm-22-947

Article Options

Download Citation

Share