Electronic Patient Reported Outcome Measures after palliative radiotherapy: evaluation of implementation

Introduction

Background

Radiotherapy is a frequently employed palliative treatment for patients with advanced cancer (1,2). In our hospital, approximately 30% of radiation treatments are given with palliative intent, equating to around 750 palliative radiotherapy (PRT) courses annually. Despite preventive measures, treatment-related symptoms or other issues can persist post-treatment, leading to a decline in patients’ quality of life (QoL). Due to the short treatment courses used for PRT, these symptoms often manifest after treatment completion. Consequently, accurately capturing treatment-induced toxicities from patients is challenging, as monitoring typically only occurs during scheduled visits. Additionally, there is a lack of guidelines detailing the best practices for systematic symptom monitoring after PRT (3,4).

Rationale and knowledge gap

Remote follow-up through web-based care is a potential strategy to enhance care for patients. Electronic Patient-Reported Outcome Measures (ePROMs) could enable patients to report symptoms and concerns from home, facilitating timely and appropriate care (5). In research settings, ePROMs have been shown to be acceptable to both patients and healthcare providers (HCPs) and are associated with improved patient-provider communication, enhanced health-related QoL, reduced emergency department visits, and even improved cancer-related survival (6-11). However, ePROM implementation in routine clinical practice has shown to be challenging, with ongoing concerns about their associated burden and workload for both patients and clinicians, limited integration of ePROMs into clinical workflows and the electronic medical record (EMR), and insufficient technical support (12-14). Limited data are available on the application of ePROM-based symptom follow-up after PRT (15).

Objective

Our team has developed an ePROM intervention for symptom follow-up after PRT, named the ePRomT diary (12,15,16). The current study aimed to evaluate the implementation of this intervention using the RE-AIM framework. In Phase I, we investigated: (I) which patients were most likely to use the ePROM intervention; (II) how patients would perceive and use the intervention; (III) which HCPs within our staff were motivated to participate in implementation and how they viewed their role; (IV) which aspects of our intervention needed modification to increase implementation and enhance patients’ satisfaction; and (V) what resources HCPs require for long-term success. Based on the user feedback received in Phase I, modifications were made to the intervention and its implementation strategy, and its target audience was redefined. The impact of these changes on relevant RE-AIM aspects was assessed in Phase II of the study (Table 1). We presented this article in accordance with the TREND reporting checklist (available at https://apm.amegroups.com/article/view/10.21037/apm-24-74/rc).

Methods

Study design

A mixed-method two-phase implementation evaluation of an ePROM based follow-up intervention after PRT was performed in a single, tertiary hospital, University Hospitals Leuven in Belgium.

ePROM intervention

Drawing on clinical experience, literature on ePROMs in advanced cancer and palliative care, previous ePROM diaries developed in our hospital, and our own exploratory research on ePROMs in PRT, we developed an ePROM intervention for symptom follow-up after PRT: the ePRomT diary (12,17,18).

In this diary, symptoms were assessed using the Dutch translation of the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE), with the exception of pain, which was measured using a numeric rating scale ranging from 0–10 (19-22). The symptoms evaluated were selected based on clinical experience, patient feedback, and existing literature, as specific PRO-CTCAE items have not yet been established for PRT (4,23,24). In Phase I of the study, nine general symptoms were evaluated. Additionally, patients could report any other relevant symptom(s) they experienced in a ‘free text’ box. In Phase II, treatment region-specific items were added to the general symptoms on patient request (Table 2).

The ePRomT diary was integrated into the online patient portal of the hospital’s EMR, MyNexUZHealth (25). This platform allows patients to view appointments, consultation reports, scans, and other health-related information. Importantly, patients already receive ePROMs, Patient Reported Experience Measures, diaries, and other questionnaires through this platform.

Patients were automatically prompted daily to complete the ePRomT diary, for a period of 21 days following PRT. Upon completion of the diary, immediate automatic feedback was provided based on the input: reporting of limited symptom severity resulted in online self-management advice (e.g., references to specific hospital information folders and leaflets, general self-care instructions), while reporting severe symptoms led to advice to contact an HCP. There was no active follow-up of the diary by HCPs. However, when patients received advice to contact an HCP, an automatic email was also sent to researcher E.O. for research purposes.

In Phase II, one hour per week was allocated for telephone consultations with the clinical support managers (CSMs), based on patients’ feedback. CSMs are nurses and radiotherapy technologists (RTTs) with several years of experience who see patients after their consultation with the physician for additional support and education. Information about the role of the CSM and their availability was added to the study materials. Patients were explicitly encouraged to contact a CSM or HCP of choice in case of questions or concerns during the study.

Implementation plan

The RE-AIM framework was used to evaluate the implementation and to identify factors for sustainable adoption of this intervention (Table 1). RE-AIM is an acronym for five domains: (I) reach of the target population; (II) effectiveness on key outcomes; (III) adoption by individuals responsible for delivery; (IV) barriers, facilitators, and resources necessary for implementation; and (V) potential for maintenance of the intervention (26-28).

Given that residents in radiation oncology (N=9) and CSMs (N=4) would be the primary recruiters for the study, they received additional training on the eligibility criteria, the ePRomT diary, and MyNexUZHealth.

All staff in the Department of Radiation Oncology were educated about (e)PROMs, their potential benefits and current use in oncology, the ePRomT diary and the study objectives via presentations, flyers, meetings, and emails. HCP from outside the department were informed about the study through a trial notification in the patients’ EMR, which included information about the ePRomT diary as well as PRT, its side effects, and potential supportive measures.

The implementation strategy was discussed in advance with the medical staff, a resident representative, the head nurse, and one of the CSMs to ensure that the intervention was seamlessly incorporated into routine clinical practice as much as possible.

Both implementation phases had a recruitment period of 10 weeks, with Phase I conducted in late spring 2021 and Phase II in late spring 2022. The research group met regularly during these periods to discuss the implementation findings.

Participants

Patients were recruited during appointments and bedside consultations, with about 50% of patients referred for PRT seen in each setting. Patients were eligible for the ePRomT diary if they (I) could give informed consent; (II) could speak/read and write Dutch; (III) started PRT; (IV) were ambulatory or assumed to be ambulatory after treatment; (V) were willing and able (computer and internet access) to complete ePROMs (themselves or with assistance); (VI) were not enrolled in another study that required the completion of (e)PROMs. Patients treated in oligometastatic setting could not participate.

In Phase II, we excluded patients with a Karnofsky Index (KI) of ≤50/World Health Organization (WHO) score ≤3.

Data collection

To assess the proportion and representativeness of participating patients and their characteristics compared to the total patient group referred for PRT, patient data such as age, primary tumor, treatment localization, performance status, and all ePROM data (number of completed ePROMs, reported symptoms, and reported number of contacts triggered by the diary) were extracted from the EMR (reach, implementation) (Tables 3,4).

All patients who agreed to take part in the study, completed a survey (Appendix 1) and participated in a semi-structured interview 3–6 weeks after completing their treatment (i.e., 0–3 weeks after completing the diary), regardless of ePROM completion. Themes explored were reasons for (non-)use of the diary, satisfaction with the intervention, and suggested changes (effectiveness, implementation). Three to four weeks after each inclusion period, CSMs and residents participated in semi-structured interviews to review the recruitment and implementation process as well as other feedback (implementation, adoption and maintenance).

In addition to the surveys and interviews, field notes were taken about implementation success, HCPs’ comments, and other relevant information during the recruitment period.

Analyses

Quantitative data from the patient surveys were analyzed using (descriptive) statistics. Qualitative data from field notes and interviews were thematically analyzed, focusing on themes associated with (non-)use of intervention using NVivo.

All interviews were conducted by researcher E.O., a radiation oncologist working in the department, during a scheduled appointment. All interviews were audio recorded, transcribed verbatim, coded and analyzed by E.O. Two additional researchers (A.C. and S.I.) independently read several transcripts, to help generate initial codes with discrepancies resolved through discussion and consensus. Coding and analysis was done using NVivo.

Ethical considerations

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). Permission for this study was obtained from the Ethics Committee Research UZ/KU Leuven (No. B3222021000603). All participants in this study signed an informed consent form (ICF). The Standards for Reporting Implementation Studies of Complex Interventions (StaRI) were followed (29).

Results

Reach

In Phase I, 128 patients were referred to our department for PRT (Table 3). Of these patients, 87 (68%) met the eligibility criteria for the ePROM intervention, and 42 patients (33%) agreed to participate. Forty-five patients (35%) declined participation for various reasons. The primary reason for refusal by 25/45 (or 56%) patients was the electronic completion of the PROMs. Three patients were offered ePROMs during systemic treatment shortly after completing PRT, which was an exclusion criterion. Consequently, their ePRomT diary was discontinued after a telephone discussion, resulting in 39 participants (30% of the total patient population or 45% of eligible patients).

Phase I revealed that four patients with a KI of ≤50 were either hospitalized or received palliative care at home with daily visits from a home care nurse (HCN) after their PRT. These patients either did not complete the ePROMs and/or found them relatively burdensome; with two patients dying during the intervention period. This resulted in 37 patients that could be analyzed. Based on these findings, it was decided to only include ambulatory patients with a KI of >50 in Phase II (Tables 3,4).

In Phase II, 109 patients were referred for PRT (Table 3). Sixty (55%) were unable to participate, of which 25 were not offered the ICF. In 19/25 (or 76%) of these patients, this was due to workflow changes during coronavirus disease 2019 (COVID-19). Additionally, 3 patients were not offered the ICF due to uncertainty regarding their treatment setting (palliative or curative), and at the request of a supervisor: in one case because of the patient’s age, in the other two for unknown reasons. Of the remaining 49 patients, 40 (37% of the entire patient population or 82% of eligible patients) participated in the ePRomT diary (Table 3), and 9 (18%) declined participation, all citing a lack of interest in increased symptom follow-up (Table 4). All of the 40 patients who participated in the trial agreed to complete the study survey and were interviewed.

In both phases, reasons for participation included the opportunity for additional symptom follow-up beyond standard-of-care, self-management support associated with increased independence and patient-centered care, and the fact that the ePRomT diary was introduced as a potentially important addition to their care (Table 5). In the interviews, even patients who did not complete any ePROMs reported being pleased to have been invited to participate.

Patients generally expressed high satisfaction with the information provided about the study and its objectives as well as the intervention itself in the survey (Tables S1,S2). A limited number of patients gave a neutral answer to the survey question regarding their satisfaction with written information provided. In the interviews, it became clear that these patients referring to the ICF, which they found hard to understand. The information provided to them about how to use the app, the goal of the ePROmT diary etc. was regarded clear and complete by all patients.

Both CSMs and residents mentioned that the ICF acted as a barrier for recruitment. The ICF was considered disruptive in the consultation and patients were put off by the size of the ICF. Additionally, the terms ‘palliative’ and ‘palliative radiotherapy’ deterred several patients from participating, as they did not identify with these labels.

Effectiveness

In both phases, patients responded neutrally when asked if the diary met their expectations in the survey (Tables S1,S2). When asked about this in the interviews, all of these patients indicated that they had never used an ePROM diary before and therefore did not really have any expectations about the ePROM diary or how it would work.

Patients generally found the questions in the diary to be relevant and concise (Tables S3,S4). In the interviews, patients were highly positive about the free-text box, with many suggesting the inclusion of more such boxes to individualize the diary. In Phase I interviews, some patients mentioned missing questions related to their specific situation and recommended adding additional questions specific to the treatment area, as they found the symptom list too generic (Table 5). This feedback was incorporated into Phase II, after which the questionnaire being ‘too generic’ was no longer mentioned (Table 2). However, providing more than one free-text box was requested by several patients in Phase II as well.

In Phase I, about two-thirds of patients indicated that the diary made them more aware of their symptoms; in Phase II, this was about half of the patients (Tables S3,S4). In the interviews, several patients mentioned that they would want to use the diary questions as a sort of checklist to know which symptoms to look out for. In Phase I, this was one of the reasons patients requested questions related to their specific symptoms, as mentioned above.

The automatic feedback was generally viewed as important, relevant, and useful. However, in the survey, a portion of patients did not associate the diary and the feedback it provided with a feeling of independence (Tables S3,S4). In the interviews, however, patient indicated that the self-management strategy enabled them to be active participants in their care, which they appreciated (Table 5). The feedback was considered “helpful”, making patients feel “supported” and more “secure”.

Some patients expressed a desire for self-management information more tailored to their specific situation. They found the advice provided on the feedback pages to be useful but rather generic. Some even indicated a preference for receiving a personalized email or message with personalized information instead of automatically generated feedback (Table 5).

All patients who received advice to contact an HCP considered this necessary and appropriate (Tables S5,S6). However, this advice did not seem to motivate patients to contact an HCP or make them more secure to do so. In Phase II, the motivation to contact an HCP was higher than in Phase I (Tables S5,S6). In both phases, patients perceived a barrier to reaching out. Several patients expressed a preference for being contacted by an HCP, especially when experiencing more severe symptoms. The primary reason for this preference was the fear of inconveniencing HCPs with their concerns and questions, particularly because they lacked a specific contact person (Table 5). Consequently, many patients mainly used the diary and its advice to guide discussions on their most severe symptoms during face-to-face consultations, such as scheduled appointments or day ward visits. In Phase I, two patients suggested specifying a recommended time or period for contacting the hospital to further facilitate communication. In Phase II one hour per week was allocated for telephone consultations with the CSMs. This was very positively perceived and reviewed in the interviews, as most patients preferred discussing their symptoms directly with a hospital HCP and were reluctant to schedule a (telephone) appointment with their general practitioner (GP) as recommended in the diary.

In Phase II, three patients in very poor condition found three weeks of (daily) ePROMs too burdensome. One answered in their survey that 3 weeks was too long and suggested that an alternative follow-up method might be more suitable for patients in their situation during the interview. The two other patients indicated that a duration of 3 weeks was no problem, but maybe a lower frequency of ePROMs would decrease completion burden (Table 5). However, no negative effects related to the ePRomT diary or participation in the study were reported by these or other patients.

Overall, only one patient from Phase I indicated being disappointed with the intervention (Tables S1,S3,S5). The diary did not meet his expectations, and he was not happy with the questions asked nor the feedback provided. During the interview, he mentioned that his wife was a registered nurse in the department of oncology, and that he would discuss his symptoms and questions with her, so the diary did not provide any additional information or benefit.

In Phase I, 93% of patients using the ePRomT diary found it to be of added value in their care, 75% were willing to use it again for retreatment, and 70% would recommend the diary to others. In Phase II, these figures increased to 98% finding it valuable, 84% willing to reuse it, and 80% recommending it to others (Tables S1,S2). Using the diary did not result in a subjective impact on the duration, intensity, or overall impact of their symptoms. Nevertheless, patients in both phases mentioned feeling “safer” with the diary, despite the fact that their entries were not actively monitored.

Adoption

CSMs generally expressed satisfaction with recruitment for the study. They believed they had ample time to introduce the study, explain the diary, and assist patients with installing the software on their phones/demonstrating the diary if needed. They also felt that they were able to enroll a proper number of patients in the study. In contrast, residents, who introduced the ePRomT diary during bedside consultations in hospital and day wards, found it more challenging to discuss the study or offer assistance with the diary. They felt that considering the study and eligibility criteria was disruptive amidst their hectic schedules. Additionally, carrying ICFs during their clinical rounds was seen as burdensome. Residents also noted the difficulty of recruiting patients in shared rooms, which compromised privacy. Furthermore, they frequently encountered patients who had not anticipated being referred for PRT, making it challenging to discuss topics such as research and enroll patients in the trial.

The same CSMs were involved in the two phases of the study, so limited time in education of ePROMS or the trial was necessary. However, several residents had changed between Phase I and II and several requested information and education on an individual basis.

Despite these (perceived) challenges, the recruitment rates for the study were comparable between residents and CSMs.

Implementation

In Phase I, 17 of the 37 included patients (46%) completed at least one assessment, with an average of 10 assessments per patient (range, 1–20). Of these 17 patients, 10 received self-care advice at least once. The primary reason for limited ePROM completion was a low symptom burden, leading to a perceived lack of need for self-management or HCP advice. Twenty-two patients (56%) did not complete any assessments for various reasons (Table 4). Ten of these patients indicated a loss of interest in the intervention after completion of their radiotherapy.

Of the 10 patients that received self-management advice, reported reading this and implementing (some of) it. Eight patients were advised to contact an HCP. Six of these patients indicated that this advice motivated them to contact an HCP (Table S3); however, only four followed this advice to some degree: three discussed their symptoms during a scheduled consultation, and one discussed his symptoms when the hospital contacted him to reschedule an appointment. This could be because only 3 patients replied ‘Yes’ to the survey question if they felt more confident to contact an HCP after being advised to do so by the diary (Table S2).

Key aspects of the ePRomT diary, according to both patients and HCPs, included the use of the hospital’s platform for the diary. This familiar platform enabled patients and HCPs to focus on the ePRomT diary without the need to familiarize themselves with a new medium. The (daily) completion reminders were also considered essential by many patients to ensure consistent ePROM completion. However, especially patients in poorer condition or those who did not perceive side-effect burden, mentioned that modification of the completion reminder frequency would motivate them to use the ePRomT diary in the future. However, the daily reminders were not perceived as bothersome by patients who did not want daily ePROM completion.

The focus of the diary on self-care was viewed positively by all patients, with several patients feeling empowered by this approach (Table 5).

One modification made after Phase I was a change in the title of the intervention. Several patients encountered difficulties finding the ePRomT diary under its original title, “ePROM PROGRAM”, with one patient failing to complete any ePROMs as a result (Tables S3,S5). Based on patient feedback, the title was changed to “ePRomT diary—radiotherapy diary” for Phase II. Another adjustment was the inclusion of treatment site-specific radiotherapy-related symptoms (Table 2). While the general questions were deemed highly relevant and were retained, the request to add more free-text boxes was not implemented. Adaptations to the frequency of eRPOMs (once-daily) could not be made due to software restrictions.

During Phase II, 26/40 patients (65%) completed at least one assessment, with an average of 10 assessments per patient (range, 1–21). The main reason for discontinuing ePROM completion was once again a limited symptom burden. Of these 26 patients, 16 received self-care advice at least once. Fifteen patients reported reading the advice, and nine implemented some of the recommendations. Five patients discussed their symptoms and self-care management during a day ward consultation. Five patients were advised to contact an HCP, of which two followed this advice due to experiencing pain. Among the remaining patients, one felt a barrier to contacting an HCP between scheduled visits, while the other two did not consider their reported symptoms severe enough to warrant additional HCP contact. There were no patients who contacted the CSMs during the allocated telephone consultation hour. One patient mentioned that it felt odd to contact an HCP who was no longer responsible for his care, as the radiotherapy was finished (Table 5).

Maintenance

The initiation of the ePRomT diary was reported to be easier by CSMs compared to residents. Furthermore, CSMs expressed interest in an active role in further ePROM implementation, as they believed the ePRomT diary enhanced clinical service. However, they emphasized the need for dedicated time to follow-up on ePROM data. Residents recognized the potential benefits of ePROMs and hoped that discussing them outside of a study context would be less disruptive to their workflow. They suggested that having a flyer about the ePRomT diary, similar to other departmental flyers, would aid maintenance. Residents also believed that integrating ePROMs into the clinical workflow would eliminate the need for an ICF and, more importantly, remove the ‘palliative’ label that some patients perceived so negatively (Table 5).

However, CSMs mentioned in their interviews that the lack of feedback on patients’ well-being after treatment could potentially demotivate them from recruiting patients for ePROMs, as they would not observe the effects of the ePROM interventions. This was confirmed by the residents interviewed, who considered receiving feedback from or about patients after treatment as a very valuable learning opportunity.

Discussion

We conducted a two-phase implementation evaluation of ePROMs for symptom follow-up after PRT. Despite the recognized benefits of ePROMs, integrating them into routine clinical care remains challenging, with a lack of guidance on successful implementation in standardized, evidence-based multidisciplinary care pathways (30). However, having ePROMs become more established in routine clinical practice may increase operational efficiencies, reduce traveling time for patients, and increase the willingness of patients to enroll in clinical trials, especially in local intervention trials with the intent of symptom control (31).

The intervention in this study was designed based on previous ePROM research and our own data, in alignment with the European Society for Medical Oncology (ESMO) recommendation on ePROMs (32). The objective of this paper was to evaluate the implementation of this intervention using the RE-AIM framework, to guide other researchers and clinicians in their implementation efforts.

An important factor affecting reach was the research set-up requiring an ICF. This ICF, which consisted of several pages of complex information, acted as a barrier to participation, which is a known problem (33). Eliminating the research aspect and incorporating ePROMs into routine care for advanced cancer patients would likely improve reach. Additionally, the term ‘palliative’ used in the ICF posed a challenge for several patients, who did not consider themselves palliative or did not wish to be labeled as such, a finding consistent with previous studies (34,35). Even patients who did agree to participate in the study mentioned having difficulties with this label.

Currently, our hospital offers ePROMs in many cancer care trajectories, either for follow-up or benchmarking. Developing a comprehensive ePROM-based follow-up system across the cancer trajectory could familiarize both patients and HCPs with ePROMs, and potentially enhance their use and benefits. By offering ePROMs throughout the cancer trajectory, terms like ‘curative’ or ‘palliative’ can be avoided, alleviating the emotional burden associated with these terms and possibly further improve reach.

Patients’ satisfaction with intervention (effectiveness) was high. Patients did not feel that using the diary affected their perception of symptom intensity, duration, or impact on their lives neither positively nor negatively. It should be noted that the majority of patients did not report significant side-effects. On the other hand, patients did report satisfaction with the self-management tools provided, which made them feel empowered, in control, and “safe”. Additionally, no patients reported negative experiences with either the diary or the self-management advice. We found no differences in the opinions on self-management between patients of different age groups or performance statuses. However, we did not assess the extent to which the self-management advice was actually used. Self-management in patients with advanced cancer is highly personal and multifaceted. Self-management support can benefit from an individualized approach embedded in strong partnerships with relatives and HCPs (36). Some patients expressed a desire to tailor the ePRomT diary to their specific capabilities and needs and several patients did not wish to initiate or discontinued the intervention due to its intensity, with daily ePROMs for three weeks. Patients also would have preferred individualized feedback. However, individualized feedback would likely entail active follow-up of the ePROMs by an HCP, resulting in a significant increase in workload. A potential alternative could be the use of health chatbots for patients who prefer more individualized information. While data are still limited, patients’ willingness to interact with these chatbots appears to be high (37,38). Nevertheless, even with the use of chatbots, adequate human resources to address ePROM data should be allocated to this responsibility (32).

A recent study found no difference in health-related QoL between ePROM symptom monitoring via a reactive approach, where patients receive alerts to contact an HCP, or an active approach, where the HCP receives an alert to contact the patients (39). However, we found that patients who were prompted to contact an HCP still faced barriers to doing so. Patients who received an alert tended to delay discussing their symptoms, waiting for a planned hospital appointment, typically at the medical oncology department. This could partly be due to the difference in rapport patients have with their HCPs in medical oncology compared to radiation oncology. It has been shown that sufficient time with HCPs in radiotherapy is critical for developing trust and reducing anxiety and stress. However, the shorter treatment schedules of palliative patients may hinder the development of the patient-professional relationship, while the symptoms, added disease burden, and existential distress associated with a palliative outlook may actually demand enhanced support, interpersonal engagement, and communication (40). Improving rapport with HCPs in radiotherapy and motivating patients to contact them will certainly aid in the implementation and effectiveness of ePROM-based PRT symptom follow-up. This is especially relevant as our CSMs are very willing to play an active role in supporting these patients, during and after their treatment.

Despite challenges, both CSMs and residents were willing to offer patients ePROMs, which is a crucial finding. Staff in organizations with higher readiness are more likely to initiate change, be collaborative and cooperative, and willing to implement new evidence-based practices, essential for the adoption and maintenance (30,41,42). Several studies have highlighted the benefit of having project managers or coordinators skilled in both knowledge translation and facilitating practice changes (14,42). Furthermore, the presence of local staff champions to advocate for ePROMs usage has been shown to be an important facilitator in implementation, adoption, and maintenance (42). As CSMs are responsible for optimizing clinical care and implementing care initiatives, their positive evaluation of our intervention and willingness to take up an active role in further ePROM endeavors is encouraging. Their willingness to play an active role in supporting these patients can also enhance the effectiveness and reach of the ePRomT diary, thereby promoting patient-centered care and improving the overall quality of care provided. Where CSMs see patients during initial consultation and sometimes during their treatment, they are not very often informed about the patients’ wellbeing after treatment is completed as it is the resident that is contacted for deliberation or re-referral to the department. This organizational factor could be a potential maintenance issue.

It should be noted however, that about half of patients were recruited by residents. Whereas CSMs have extensive experience in oncology care pathways and can act as local champions, residents have varying levels of experience with radiation oncology in general, PRT and patient care. Additionally, they rotate between different care paths every few months and do not complete their entire residency in our hospital. Continued education on PRT and its symptoms as well as their use, benefits and the ePRomT diary intervention will be necessary for maintenance.

For this study, we had access to an established hospital-based online platform, that made it possible to integrate ePROM data in the EMR. Limited integration has been shown to be a significant barrier for implementation (42). Many hospitals may not have our resources, making the interpretation of our results for other applications challenging.

A potential limitation of this study is the representativeness of the patients who participated. While those who took part were highly satisfied with the intervention, it’s crucial to acknowledge that only one-third of all patients referred for PRT met the eligibility criteria for our implementation evaluation study, for various reasons. Another limitation could be the role of the main researcher, E.O., who serves as a radiation oncologist and palliative care physician at the hospital where the study was conducted. Although none of the patients included in the study were treated by E.O., her position in the hospital may have influenced the responses of both patients and HCPs who participated in the interviews, potentially introducing bias into the study. Additionally, since we did not audio-record, transcribe, and analyze the research team meetings, there might be some nuances in these discussions that were missed. Nonetheless, we believe this would not have significantly impacted the decisions made during the study.

Conclusions

This study indicates that the ePRomT diary is feasible and well-received. However, further research is required to address the challenges related to patient recruitment and commitment to the intervention, optimize the frequency and duration of ePROMs, and integrate ePROMs into existing workflows to maximize the potential benefits of ePROM-based symptom follow-up after PRT.

Acknowledgments

We would like to thank the clinical support managers and residents of the Department of Radiation Oncology of the University Hospitals Leuven for their assistance in this study.

Funding: None.

Footnote

Reporting Checklist: The authors have completed the TREND reporting checklist. Available at https://apm.amegroups.com/article/view/10.21037/apm-24-74/rc

Data Sharing Statement: Available at https://apm.amegroups.com/article/view/10.21037/apm-24-74/dss

Peer Review File: Available at https://apm.amegroups.com/article/view/10.21037/apm-24-74/prf

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://apm.amegroups.com/article/view/10.21037/apm-24-74/coif). E.O. serves as an unpaid editorial board member of Annals of Palliative Medicine from December 2022 to November 2024. The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by The Ethics Committee Research UZ/KU Leuven (EC Research, No. B3222021000603) and informed consent was obtained from all individual participants.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Wu SY, Singer L, Boreta L, et al. Palliative radiotherapy near the end of life. BMC Palliat Care 2019;18:29. [Crossref] [PubMed]
2. Spencer K, Morris E, Dugdale E, et al. 30 day mortality in adult palliative radiotherapy--A retrospective population based study of 14,972 treatment episodes. Radiother Oncol 2015;115:264-71. [Crossref] [PubMed]
3. Lin VQ, Riviere P, Murphy JD, et al. Retrospective Review of Follow-up Strategies for Patients Receiving Palliative Radiotherapy. J Pain Symptom Manage 2023;66:238-41. [Crossref] [PubMed]
4. Underwood J, McCloskey S, Raldow A, et al. Developing a Mobile Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events Administration System to Capture Postradiation Toxicity in Oncology: Usability and Feasibility Study. JMIR Form Res 2022;6:e27775. [Crossref] [PubMed]
5. Aiyegbusi OL, Nair D, Peipert JD, et al. A narrative review of current evidence supporting the implementation of electronic patient-reported outcome measures in the management of chronic diseases. Ther Adv Chronic Dis 2021;12:20406223211015958. [Crossref] [PubMed]
6. Girgis A, Durcinoska I, Levesque JV, et al. eHealth System for Collecting and Utilizing Patient Reported Outcome Measures for Personalized Treatment and Care (PROMPT-Care) Among Cancer Patients: Mixed Methods Approach to Evaluate Feasibility and Acceptability. J Med Internet Res 2017;19:e330. [Crossref] [PubMed]
7. Girgis A, Bamgboje-Ayodele A, Rincones O, et al. Stepping into the real world: a mixed-methods evaluation of the implementation of electronic patient reported outcomes in routine lung cancer care. J Patient Rep Outcomes 2022;6:70. [Crossref] [PubMed]
8. Basch E, Iasonos A, McDonough T, et al. Patient versus clinician symptom reporting using the National Cancer Institute Common Terminology Criteria for Adverse Events: results of a questionnaire-based study. Lancet Oncol 2006;7:903-9. [Crossref] [PubMed]
9. Basch E, Deal AM, Kris MG, et al. Symptom Monitoring With Patient-Reported Outcomes During Routine Cancer Treatment: A Randomized Controlled Trial. J Clin Oncol 2016;34:557-65. [Crossref] [PubMed]
10. Denis F, Lethrosne C, Pourel N, et al. Randomized Trial Comparing a Web-Mediated Follow-up With Routine Surveillance in Lung Cancer Patients. J Natl Cancer Inst 2017; [Crossref] [PubMed]
11. Rotenstein LS, Huckman RS, Wagle NW. Making Patients and Doctors Happier - The Potential of Patient-Reported Outcomes. N Engl J Med 2017;377:1309-12. [Crossref] [PubMed]
12. Oldenburger E, Neyens I, Coolbrandt A, et al. Using ePROMs for follow-up after palliative radiotherapy: An exploratory study with patients and health care providers. Patient Educ Couns 2022;105:2355-61. [Crossref] [PubMed]
13. Stover AM, Haverman L, van Oers HA, et al. Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings. Qual Life Res 2021;30:3015-33. [Crossref] [PubMed]
14. Foster A, Croot L, Brazier J, et al. The facilitators and barriers to implementing patient reported outcome measures in organisations delivering health related services: a systematic review of reviews. J Patient Rep Outcomes 2018;2:46. [Crossref] [PubMed]
15. Oldenburger E, Oldenburger F, Coolbrandt A, et al. The use of patient reported outcome measures (PROMs) in palliative radiotherapy: A topical review. Radiother Oncol 2020;149:94-103. [Crossref] [PubMed]
16. Oldenburger E, Isebaert S, Coolbrandt A, et al. The use of electronic Patient Reported Outcomes in follow-up after palliative radiotherapy: A survey study in Belgium. PEC Innov 2023;3:100243. [Crossref] [PubMed]
17. Coolbrandt A, Steffens E, Wildiers H, et al. Use of a symptom diary during chemotherapy: A mixed-methods evaluation of the patient perspective. Eur J Oncol Nurs 2017;31:37-45. [Crossref] [PubMed]
18. Coolbrandt A, Bruyninckx E, Verslype C, et al. Implementation and Use of a Patient Symptom Diary During Chemotherapy: A Mixed-Methods Evaluation of the Nurse Perspective. Oncol Nurs Forum 2017;44:E213-22. [Crossref] [PubMed]
19. Smith AW, Mitchell SA, K, De Aguiar C, et al. News from the NIH: Person-centered outcomes measurement: NIH-supported measurement systems to evaluate self-assessed health, functional performance, and symptomatic toxicity. Transl Behav Med 2016;6:470-4. [Crossref] [PubMed]
20. Kluetz PG, Chingos DT, Basch EM, et al. Patient-Reported Outcomes in Cancer Clinical Trials: Measuring Symptomatic Adverse Events With the National Cancer Institute's Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). Am Soc Clin Oncol Educ Book 2016;35:67-73. [Crossref] [PubMed]
21. Basch E, Reeve BB, Mitchell SA, et al. Development of the National Cancer Institute's patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE). J Natl Cancer Inst 2014;106:dju244. [Crossref] [PubMed]
22. Dueck AC, Mendoza TR, Mitchell SA, et al. Validity and Reliability of the US National Cancer Institute's Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). JAMA Oncol 2015;1:1051-9. Erratum in: JAMA Oncol 2016;2:146. [Crossref] [PubMed]
23. Körner P, Ehrmann K, Hartmannsgruber J, et al. Patient-reported symptoms during radiotherapy : Clinically relevant symptom burden in patients treated with palliative and curative intent. Strahlenther Onkol 2017;193:570-7. [Crossref] [PubMed]
24. Spencer K, Parrish R, Barton R, et al. Palliative radiotherapy. BMJ 2018;360:k821. [Crossref] [PubMed]
25. Uw medisch dossier. [cited 2020 Aug 26]. Available online: https://www.nexuzhealth.be/nl/mynexuzhealth
26. Glasgow RE, Harden SM, Gaglio B, et al. RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review. Front Public Health 2019;7:64. [Crossref] [PubMed]
27. King DK, Glasgow RE, Leeman-Castillo B. Reaiming RE-AIM: using the model to plan, implement, and evaluate the effects of environmental change approaches to enhancing population health. Am J Public Health 2010;100:2076-84. [Crossref] [PubMed]
28. Glasgow RE, Estabrooks PE. Pragmatic Applications of RE-AIM for Health Care Initiatives in Community and Clinical Settings. Prev Chronic Dis 2018;15:E02. [Crossref] [PubMed]
29. Pinnock H, Barwick M, Carpenter CR, et al. Standards for Reporting Implementation Studies (StaRI): explanation and elaboration document. BMJ Open 2017;7:e013318. [Crossref] [PubMed]
30. Bamgboje-Ayodele A, Avery S, Pearson J, et al. Adapting an integrated care pathway for implementing electronic patient reported outcomes assessment in routine oncology care: Lessons learned from a case study. J Eval Clin Pract 2022;28:1072-83. [Crossref] [PubMed]
31. Gaertner J, Zimmermann F, Simone CB 2nd. The value of patient reported outcomes in palliative radiotherapy: A discussion in light of current findings. Ann Palliat Med 2023;12:611-5. [Crossref] [PubMed]
32. Di Maio M, Basch E, Denis F, et al. The role of patient-reported outcome measures in the continuum of cancer clinical care: ESMO Clinical Practice Guideline. Ann Oncol 2022;33:878-92. [Crossref] [PubMed]
33. King-Kallimanis BL, Ferris A, Dropkin L, et al. Initial Steps in Creating a Patient-Centric Addendum to Clinical Trial Informed Consent Forms. JTO Clin Res Rep 2023;4:100575. [Crossref] [PubMed]
34. Shen MJ, Wellman JD. Evidence of palliative care stigma: The role of negative stereotypes in preventing willingness to use palliative care. Palliat Support Care 2019;17:374-80. [Crossref] [PubMed]
35. Bandieri E, Borelli E, Gilioli F, et al. Stigma of Palliative Care among Patients with Advanced Cancer and Their Caregivers on Early Palliative Care. Cancers (Basel) 2023;15:3656. [Crossref] [PubMed]
36. van Dongen SI, de Nooijer K, Cramm JM, et al. Self-management of patients with advanced cancer: A systematic review of experiences and attitudes. Palliat Med 2020;34:160-78. [Crossref] [PubMed]
37. Bibault JE, Chaix B, Nectoux P, et al. Healthcare ex Machina: Are conversational agents ready for prime time in oncology? Clin Transl Radiat Oncol 2019;16:55-9. [Crossref] [PubMed]
38. Ma DC, Potters L, Lee L, et al. Patient-Reported Outcomes Using Automated Chatbot for Breast Cancer Patients Receiving Radiation Therapy. Int J Radiat Oncol Biol Phys 2021;111:e96. [Crossref]
39. Billingy NE, Tromp VNMF, Aaronson NK, et al. Quality of life after patient-initiated vs physician-initiated response to symptom monitoring: the SYMPRO-Lung trial. J Natl Cancer Inst 2023;115:1515-25. [Crossref] [PubMed]
40. Dennis K, Harris G, Kamel R, et al. Rapid Access Palliative Radiotherapy Programmes. Clin Oncol (R Coll Radiol) 2020;32:704-12. [Crossref] [PubMed]
41. Stover AM, Tompkins Stricker C, Hammelef K, et al. Using Stakeholder Engagement to Overcome Barriers to Implementing Patient-reported Outcomes (PROs) in Cancer Care Delivery: Approaches From 3 Prospective Studies. Med Care 2019;57 Suppl 5 Suppl 1:S92-9.
42. Glenwright BG, Simmich J, Cottrell M, et al. Facilitators and barriers to implementing electronic patient-reported outcome and experience measures in a health care setting: a systematic review. J Patient Rep Outcomes 2023;7:13. [Crossref] [PubMed]