Improving the methods for patient-reported experience measures in palliative care: findings from a cognitive interview study

Introduction

The global shift toward ‘value-based’ healthcare has been accompanied by an increasing emphasis on consumer appraisal of the quality of care (1). Patient-reported experience measures (PREMs) are questionnaires designed for asking consumers about their lived experience of healthcare to identify strengths and weaknesses and so inform service improvements (2,3). Patient experience has been defined as “the sum of all interactions, shaped by our [i.e., healthcare’s] culture, that influence patient perceptions across the continuum of care” [New South Wales Ministry of Health, 2020 (4) p.8].

People with palliative care needs due to life-limiting illness constitute a patient population for whom healthcare experience has a special significance because care is primarily focused on improving patients’ subjective appraisal of their quality of life rather than on disease-related outcomes (5). Measuring patient experience of palliative care also requires special considerations compared to other kinds of care. First, palliative care patients’ capacity to self-report on PREMs is likely to be compromised by negative impacts of disease and treatment on cognitive processing, including impairments in verbal working/delayed episodic memory from opioids (6); such effects can also fluctuate over time (7). Second, research suggests that palliative care patients prioritise certain aspects of care that may be less focal in other kinds of care. These include: clinician-patient communication, shared decision-making and advance care planning (ACP), including the option of declining treatment based on information regarding prognosis and net benefit vis-à-vis adverse effects; and key qualities of care such as how ‘kind’, ‘caring’ and ‘respectful’ health professionals are, and whether patients feel treated with ‘dignity’ (8,9). To date, however, it is unclear what preferences palliative care patients might have for being asked about these aspects of care in PREMs, as well as more generic qualities of care that typically feature in PREMs across patient populations such as ‘safety’ and ‘privacy’.

According to theory, completing a questionnaire requires respondents to engage in four cognitive operations, namely: understanding what each item is asking (‘comprehension’), retrieving relevant information or knowledge from memory (‘recall’), making an evaluation of each item based on this recall (‘judgment’), and selecting from the available response options accordingly (‘response’) (10). An interest in how respondents engage in these operations has given rise to a qualitative method called ‘cognitive interviewing’, which involves asking people to ‘think aloud’ as they consider each questionnaire item, usually with ‘verbal probes’ to gain further insights as needed (11). While a small number of studies have undertaken cognitive interviewing with people with palliative care needs (12-16), they have focused on refining a specific questionnaire rather than on drawing more general inferences regarding cognitive operations in this population. One exception was a cognitive interviewing study with nursing home residents, which revealed some unexpected patterns when participants completed a quality of life questionnaire (17). First, participants struggled to comprehend whether questionnaire items were asking about actual or desired care. Second, participants had difficulty recalling and making judgments based on their personal preferences. Finally, when it came to responding, participants struggled to distinguish between “very important” versus “important” on a verbal response scale (VRS), and to select from values between 0 and 5 on a numerical response scale (NRS).

The current study aimed to explore cognitive operations among people with palliative care needs when completing a PREM focused on the care domains known to be important to this patient population in order to inform future administration of questionnaires for this purpose. We present the following article in accordance with the CIRF (18) and COREQ (19) reporting checklists (available at https://apm.amegroups.com/article/view/10.21037/apm-22-9/rc).

Methods

This study was commissioned by the New South Wales Bureau of Health Information (BHI) and took a qualitative approach using cognitive interviews (11). Data were collected between May and December 2021. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by Institutional Ethics Committee of South Eastern Sydney Local Health District (No. 2020/ETH03036) and informed consent was taken from all the participants. Participant confidentiality was protected by replacing names with study numbers in all reports and storing data separately from consent forms and other identifying information.

Participants

Participants were adults receiving inpatient, outpatient or community specialist palliative care from one sub-acute hospital in Sydney, Australia. While palliative care can be delivered by generalist as well as specialist services, limiting to specialist palliative care reduced requirements for additional eligibility screening and ensured that participants were aware their illness was serious.

Participants needed sufficient spoken English skills and cognitive capacity to give informed consent and participate in cognitive interviewing. To avoid placing undue burden on patients who were very unwell, we excluded those with an Australian Karnofsky Performance Status (AKPS) score of 40 or below (i.e., “in bed 50% of the time” or worse) (20), and those with a Palliative Care Outcome Collaborative (PCOC) Palliative Care Phase of “deteriorating” or “terminal” (21).

While this paper focuses on patients, interviews were also conducted with family carers.

Recruitment

We used a consecutive approach to sampling as follows. Eligible patients were identified by a female clinical trials nurse (AR) at weekly multi-disciplinary team meetings and nurse practitioner clinics, as well as via twice-weekly liaisons with clinicians running the hospital rehabilitation gym for palliative care patients. The nurse also checked with the clinical team whether there were additional considerations that might warn against approaching a patient beyond the inclusion criteria (e.g., distress). Potential participants across care settings were then approached by the clinical trials nurse either in person (inpatients) or by telephone (community). Participants were offered an AU$50 gift voucher to acknowledge the time commitment required. In the absence of published guidance on sample size requirements for cognitive interviewing studies, we aimed to recruit 15 participants as a mid-range estimate from similar previous studies (12-17).

Data collection

Patient-reported experience measure

This paper focuses not on developing a specific PREM but rather on distilling patterns in cognitive operations that have broader implications for questionnaire administration in this patient population.

The study used a PREM developed under the auspices of the BHI specifically to capture experiences of healthcare in the domains identified to be most important to palliative care patients, as described in the Introduction (8,9). The PREM comprised items mapped against these domains from existing PREMs for people with palliative care needs identified by a systematic review (22). Initial testing of the PREM was carried out in five focus groups and a modified Delphi survey with 21 patients and carers aimed at verifying the sufficiency and necessity of domains and reviewing draft items. PREM development was also informed by extensive consultation with a broad range of stakeholders including clinicians and other providers of palliative care services to identify their priorities for ‘actionable’ information.

The PREM was intended to measure experience in both patients and carers, and included 33 items. Items were nominally grouped according to content as follows, although had not yet been tested for structural validity: care environment (5 items), information and communication (10 items), compassion, respect and kindness (5 items), support for daily functioning and care in the preferred place (4 items), care of symptoms (2 items) and organizing care (5 items), as well as 2 open-ended items asking about the “best part of care” and “what most needs improving”. The list included items on aspects of care usually considered central to palliative care as outlined in the introduction, including advance care planning and shared-decision-making, communication about prognosis and declining treatment, and consideration of cultural and spiritual beliefs, as well as reference to the concepts of ‘kindness’, ‘care’, ‘respect’ and ‘dignity’, and ‘safety’ and ‘privacy’. Response options for each item variously used one of the two following VRS: “yes, always”, “yes, sometimes”, “no”; “yes, definitely”, “yes, to some extent”, “no”. Some items also included an option for “not applicable” and/or “don’t know/can’t remember”. The questionnaire underwent modification part-way through the cognitive interviews in an attempt to make items more readily interpretable, especially with regard to different settings.

Cognitive interviews

Interviews in the inpatient setting were conducted face-to-face, whereas those conducted with community participants were conducted via telephone. No-one else was present besides the participant and interviewers. Interviews were conducted by two researchers—the female clinical trials nurse (AR) and a male academic social scientist with a PhD (TL)—the first of whom led the interview and the second of whom took notes over the telephone, adding supplementary probes as necessary. The use of two interviewers with differing perspectives was intended to enrich interpretation of participant responses and guard against potential bias from prior assumptions. Both researchers were experienced in qualitative research, and neither had any previous contact with the participants. TL had prior experience of cognitive interviewing and trained AR by demonstration.

Only one interview was conducted per participant. At the beginning of interviews, it was explained that the purpose was to test the survey items rather than the person completing them and that, therefore, there were no right or wrong answers. Instructions made it clear that participants could take rests as needed and skip items that they felt were distressing or were otherwise uncomfortable answering. Additional self-reported data were collected on participants’ gender, life-limiting illness, and settings of care experienced (inpatient, community). Participants who had experience of more than one setting of care were asked to focus on only one when completing the draft PREM.

For cognitive interviews, we used both ‘think aloud’ and ‘verbal probing’ methods, following evidence that each can provide complementary information (23). Instructions for thinking aloud invited participants to say whatever came into their minds when thinking about each questionnaire item. Rather than wait until the end of the questionnaire, we used verbal probes at the time participants answered each item to clarify cognitive operations (24). We took this approach because we anticipated that cognitive impacts from illness and treatment might impede participants’ recall of earlier items by the time they reached the end of the survey (13). After completing the survey, retrospective probes were used to explore participants’ perspectives on the survey as a whole, including whether they felt the survey was easy to complete (e.g., was difficult to understand or tiring), was too long/short, and contained any repetition.

Notes taken by TL during the interview used a template pre-populated for each item and captured pertinent verbatim responses as well as hesitancies and other evidence of confusion. The note-taker went back to audio-recordings/transcripts as required following the interviews to verify quotes, and notes were compared with transcripts of 20% of interviews by another member of the team (CV) to appraise their veracity.

Analysis

Analysis was conducted first within each interview (‘within-case’) to identify patterns in cognitive operations across different items and then as a comparison between responses from different participants on the same item (‘cross-analysis’) (11). Analysis was conducted by a single researcher (TL), with regular review and discussion with all other members of the team, which included nurses (AR, JP, CV) and BHI staff experienced in questionnaire design (LD, NH, AW). No efforts to triangulate with other data (qualitative or quantitative) were made, no software was used, and neither transcripts nor findings were returned to participants for verification.

Results

Participant characteristics

In total, 29 patients were screened, 20 were eligible and 15 were interviewed. Reasons for non-participation included being too unwell and/or preoccupied with prognosis and healthcare. Six participants were interviewed about the inpatient setting, and nine about the community. All except one participant had experienced care across settings, with the exception being a man who had experienced only community care. Eight participants were women, and seven were men. All participants had cancer except for one inpatient who had motor neuron disease. Interviews varied from 18 minutes to 90 minutes in length.

Cognitive operations

Comprehension

Several participants struggled to understand the conceptual underpinnings of PREMs, including why value was being placed on their subjective experience (rather than more objective measures of care), and whether they were being asked about actual or desired care. The inpatient with MND (P7) expressed doubt about the relevance of PREMs based on the following reasoning: “if you’ve put yourself in that position [i.e., of receiving poor care] it’s your fault … If you’re not getting the support, you should leave”.

Most participants lacked experience of declining treatment and ACP, some of whom did not understand what these referred to (e.g., on declining treatment: inpatient P7, “[do you mean when] something causing pain or that’s ineffective?”; community patient P11, “I usually agree… if it’s something to do with improving my health…help me get better… I have faith in my doctors”; on ACP: inpatient P7, “[I think it relates to] my will, where they’re going to bury me”). On the other hand, all participants seemed familiar with the idea that their healthcare teams might not have fully disclosed prognosis, with nearly all indicating a preference for open disclosure. Two participants became visibly distressed at completing items of this kind, including one man who chose to discontinue the interview.

The concepts of ‘safety’ and ‘privacy’ appeared to have a range of interpretations, substantially related to setting. The concept of ‘safety’ was variously interpreted in both physical and psychological senses. Physical meanings attached to ‘safety’ included security (community P9, “no-one can get in and I have good neighbours”), safeguards against injury (community patients P1 and P20, bathroom hand rail; inpatient P5, nurse call button; inpatient P7, “something sticking out that you could trip over”), and being monitored for symptoms (inpatient P17, “if I was awake I’d be asked if I was alright”) and access to symptom management (inpatient P14, “if you’re in pain… they attend to it”). Psychological meanings attached to ‘safety’ included ease of contact after-hours (community P3, “I always can contact [them]”) and confidentiality (inpatient P4, “I can tell them anything”; community patient P20, “I feel safe from a confidentiality point of view”). One community patient (P20) suggested ‘vulnerability’ as a term that he felt better encompassed both physical and psychological senses of safety. The concept of 'privacy’ was interpreted by some participants from both settings to refer to dignity (inpatient P7, “you have to shower and you might have a different nurse and I don’t think it’s very nice… but at the same time, I understand it may be necessary to prevent falls… could have a male on male”; community P19, “not intruding on your private life; asking questions that are out in the open and not embarrassing”). Inpatients also interpreted privacy to refer to interruptions or disturbances (inpatient P6, “lots of coming and going”), while community participants referred to concerns regarding confidentiality (community P16, “…one time when they weren’t very good with my records…I got a random call, like cold calling, only after I gave my details to the hospital, which made me quite angry”).

The concepts of ‘respect’ and ‘dignity’ were interpreted by most participants to be similar and include the degree to which health professionals treated them like an individual person (community patient P16, “listening to my concerns … not making me feel like a bother”) rather than ‘box ticked’ (community patient P20). The amount of time given to them was a key indicator for many. One participant viewed ‘respect’ as a reciprocal process for which the patient shared responsibility (community P3, “it depends on how you treat them”). Participants appeared to differ in whether they distinguished ‘kind’ as a separate concept from ‘respect’. While some considered the concepts to be closely related, two community patients who seemed especially able to reflect on their cognitive operations (P16 and P20) considered ‘kindness’ to require a level of empathic engagement not necessary for respectful interactions, which they considered to be more formal and professional (“you can be respectful [to someone] but not give a shit about them”). P16 further distinguished ‘kind’ from ‘caring’, which others had not (“kind is emotion... similar to compassion... whereas caring is a physical act”). One inpatient participant acknowledged that terms like ‘respect’, dignity’, ‘kind’ and ‘caring’ are inherently subjective (P17, “everyone has their own meaning—it depends how people interpret the questions”). Unsurprisingly, participants varied widely in whether they felt related items were duplicative and could be reduced, and their preferred term if they had to choose only one.

All participants appeared able to understand the survey’s two open-ended items on ‘best part of care’ and ‘what most needs improving’ respectively, including those who struggled with closed items.

Recall

Many participants appeared to struggle with recall, including one inpatient (P6) who disclosed a severe memory impairment part-way through the survey (“I will forget who you are when you walk out the door”). Nearly all participants drew on experience from various services and settings, even though they had been asked to focus on only one. This was especially frequent for community participants who often considered home help (e.g., community P9, “I have someone once a fortnight do the cleaning… she makes the bed because I can’t lift the mattress to tuck in… and she puts the washing out”) and even informal care (e.g., community P8, “I’ve got the wife at home, and she supports me”) as well as formal healthcare.

Participants who understood what declining treatment and ACP referred to sometimes became mildly distressed when recalling relevant experience, which seemed to distract them from making a relevant judgment (e.g., inpatient P7 “I’ve got nothing to look forward to. It’s no good hiding your head”). This appeared to be compounded by items on these aspects of care having been ordered one after the other, prolonging the duration over which they were asked to reflect. One community patient initially eluded answering this question before indicating that ACP was “something I don’t want to discuss” (P8).

Two participants were clearly preoccupied with a particular issue, which they returned to wherever it could be made relevant to an item, including community patient P11 who was worried about stoma care, inpatient P14 who was upset about lack of sleep due to noise on the ward at night, and community patient P16 who was concerned about the lack of access to doctors.

Judgement

A small number of participants showed signs that they were responding based on a global assessment of the quality of care rather than with specific reference to each item. In all cases, these participants had a favourable impression of care and therefore selected the most positive response option for every item. Where participants had limited experience of an aspect of care being asked about, they often made a hypothetical judgement based on their experience of other aspects of care (“community patient P18, “well I haven’t come across anything like that… ‘yes always’”).

Participants sometimes answered more positively than might be expected after saying their experience of care did not meet their needs because they made allowances for health professionals’ effort (community patient P18, “they always try”) or under-resourcing (e.g., inpatient P4 “within the means possible… given the type of care that they’re giving, and the type of patients they have, I think it’s ‘[yes] always’ the case”). Participants also made their judgement based on comparison with experiences of care from other providers (inpatient P17 “you just don’t get that [level of family involvement] everywhere”). Finally, responses by one community participant appeared to be positively skewed by social desirability and courtesy biases or a fear of recrimination [“I don’t want to complain … I don’t want to offend any of the health professionals at (Service name)].

When making a judgment about the questionnaire’s two open items, none of the participants raised a new issue that had not already been considered in responding to earlier items. Instead, participants either summarised their responses over previous items or distilled the most salient issues from their previous responses, either in terms of importance (evidenced by their repeated reference to the issue earlier in the survey) or recency (i.e., their responses to items immediately preceding). However, even though they were not providing new information, most participants welcomed the opportunity to provide feedback in their own words.

Response

Several participants made no use of the middle response option on proportion of time (‘yes, sometimes’). These participants responded with a simple ‘yes’ initially, and then defaulted to ‘yes always’ when asked to choose from the graded options, even when their understanding of the question was doubtful (e.g., inpatient P5 when asked about involvement in decision-making, “‘yes always’…whatever they say, I do”). Three participants who were asked about this confirmed that they would have preferred dichotomous ‘yes’/’no’ response options throughout. While they were unable to explain why, these participants were all patients who also struggled with comprehension and recall, suggesting cognitive burden might have been a factor.

In contrast, participants who were not so unwell tended to make more use of all three options in the scale. However, ‘yes, sometimes’ was often used in a way that deviated from proportion of time to reflect proportions of other entities, including symptoms, health professionals and other less identifiable entities. ‘Yes, always’ was also used by one participant (inpatient P17) where a deviation from the norm ‘was a one off’. The response scale ‘yes, definitely’, ‘yes, to some extent’ appeared to have broader applicability across items, given ‘extent’ could be applied to a wider range of entities than time.

All participants were able to briefly summarise a response to the questionnaire’s two open-ended items, even when they had struggled to select from 3-point scales for the previous items.

Discussion

The current cognitive interviewing study found that participants with palliative care needs faced significant challenges in fulfilling each of the four cognitive operations required to complete a draft PREM, including comprehending each item, recalling necessary information, making a judgement based on this information, and responding accordingly. Given our inclusion criteria excluded people who were bedbound, had unstable disease or were unable to give informed consent, the observed challenges likely under-estimate those that would be encountered in a more representative sample of the palliative care population. Moreover, participants in our study were supported to complete the PREM via interview administration; self-administration in written format may pose further challenges that our study did not investigate.

The challenges we identified are consistent with a previous cognitive interviewing study carried out with nursing home residents completing a measure of quality of life (17), and suggest that design and administration of questionnaires for older and/or unwell populations should support cognitive operations to enhance the quality and interpretability of data. This may require clarification of whether people should answer items: hypothetically or with ‘not applicable’ where they lack relevant experience; with reference only to healthcare or other forms of support (especially in the community); and based on their experience of care vis-à-vis their needs or what they feel they can reasonably expect. Respondents may need reminding of these requirements at various points during the questionnaire, and are likely to benefit from monitoring and support when they deviate from the desired focus, suggesting that interviewer administration may be advantageous. Problems that participants faced in our study with staying focused on one setting could be addressed by offering alternative questions for inpatient versus community experience. Consolidating the focal setting with respondents at the beginning of the questionnaire will help to embed it in respondents’ working memory only if this is relatively unaffected (25).

Participants in the current study varied in the degree to which they wanted and used the mid-point of the 3-point verbal response scale. Ideally, respondents might be given the choice of answering ‘yes’/‘no’ instead of using an ordinal scale if they prefer. A dichotomous scale might also be more immediately interpretable for services seeking to identify which aspects of care they are performing poorly on and so should focus on improving. Where an ordinal scale is used, our findings support ‘yes, definitely/to some extent’ instead of ‘yes, always/sometimes’ because this scale has broader applicability than just proportion of time. Our study also suggests that, in verbal format, open-ended items are easy for people with palliative care needs to respond to, and can produce a useful summation of the most salient experiences of care. While written responses to open-ended items might also be feasible and acceptable for some people with palliative care needs, we speculate that those from people who are very unwell are likely to be highly abbreviated and lack richness, or even be illegible or uninterpretable. Future research is needed to develop ways that responses to open-ended items can be audio-recorded and analysed to inform service improvement with minimal burden to healthcare teams, perhaps using artificial intelligence (26).

While all the participants in the current study reported themselves satisfied with the length of the questionnaire, it seems likely that fatigue may have been a contributing factor for some people. Shortening questionnaires can be achieved by one of two approaches; either by adopting a modular approach whereby services focus on improving one aspect of care at a time and therefore administer only the most relevant section of the questionnaire during a given period, or by using a short-form questionnaire consisting of only one high-level item per domain to screen for problems that could then be followed up by more in-depth assessment. The ‘consideRATE Questions’ recently developed in the US (27) provide an example of a short-form questionnaire suited to initial screening of this kind, which is based on palliative patient’s expressed domains of importance related to inpatient care (8,9).

Future research is needed to test whether the above supports might improve the quality and interpretability of PREM data from people with palliative care needs. Research is also needed to explore other factors that may be influential. For example, a previous cognitive interviewing study that focused on a symptom assessment scale in 11 patients with cancer recruited through a palliative care service found participant responses to be influenced by contextual factors, such as mood and time of day (16). Ideally, future research should formally assess participants’ cognitive functioning at the time of PREM completion to better understand the various contributions of problems in attention, memory, executive functions and language to cognitive operations required. However, such research is ethically complex given that people with palliative care needs may find batteries of cognitive tests burdensome and will require clinical debriefing and support for any impairments that are identified.

Limitations

Findings from the current study are limited in a number of ways. In the absence of established guidance on sample size for cognitive interviews, we aimed for and achieved recruitment of 15 participants as a mid-range estimate from similar previous studies (12-17). However, a larger sample might have revealed further considerations for PREM administration, especially if we had been able to purposively sample from patients with a broader range of perspectives. Perhaps most importantly, the sample only included people receiving specialist palliative care, limiting generalisability to people receiving palliative care from generalist providers. Even in our specialist palliative care sample, some participants were unfamiliar with concepts pertaining to end-of-life care, such as ACP and declining treatment, and this is likely to over-estimate understanding in the palliative care population more generally. Also, our study did not sample for or explore diversity among different cultural perspectives on such aspects of palliative care (28).

It is also important to highlight that we used a PREM without established psychometric properties, limiting potential to interpret our results beyond the item level within the context of evidence regarding scale structure and reliability. Also, while PREM items were included to offer the best available coverage of aspects of care found by previous research to be important to people with palliative care needs, it was not possible to explore all aspects without over-burdening participants—for example, we did not consider maintaining roles and identity, or minimising the risk of infection in the context of compromised immunity (8,9). Further research will be needed to investigate cognitive operations in relation to these concepts.

Another limitation concerns that fact that problems with cognitive operations when answering questionnaire items inevitably also affected the quality of data that could be collected via think aloud and verbal probe methods. Many participants struggled to understand the think aloud technique, requiring regular verbal probing to gather insights into cognition. Even then, several participants were unable to reflect much on their deliberations or even to understand why they were being asked (e.g., Interviewer: “Can I ask why you chose that response?” P8: “Because it’s the truth!”). Indeed, verbal probing is likely to be more cognitively challenging than responding to questionnaire items itself. This is because they require meta-cognition (i.e., thinking about thinking) as well as an ability to communicate these reflections. In the current study, we found data to be especially limited on recall versus judgement, which are hypothesised to be separate cognitive operations but which were difficult to distinguish during cognitive interviews and analysis. Participants tended to report their judgement rather than recall, and struggled to reflect meaningfully on what factors they had taken into account when deciding, even when probed immediately after responding.

Finally, analysis focused on notes taken during the interviews rather than of transcripts line-by-line. The note-taker had previous experience of cognitive interviewing, went back to the transcripts or recordings when needed to clarify, and a second researcher verified 20% of notes against transcripts. However, it is possible that some participant quotes were slightly mis-represented, or some data with important implications were missed.

Conclusions

People with palliative care needs may face challenges with the cognitive operations required to complete PREMs, and cannot be assumed to be familiar with end-of-life care concepts like ACP and declining treatment. Supports are needed to improve the quality and interpretability of data. Future research should aim to further our understanding of influences from common impairments in cognitive functioning from disease and treatment and how these can be addressed through questionnaire design and administration. In the meantime, this paper has made tentative suggestions for supporting comprehension, recall, judgement and response that require further testing.

Acknowledgments

Funding: This work was supported by the Bureau of Health Information, New South Wales, Australia—a state government organization.

Footnote

Reporting Checklist: The authors have completed the CIRF and COREQ reporting checklists. Available at https://apm.amegroups.com/article/view/10.21037/apm-22-9/rc

Data Sharing Statement: Available at https://apm.amegroups.com/article/view/10.21037/apm-22-9/dss

Peer Review File: Available at https://apm.amegroups.com/article/view/10.21037/apm-22-9/prf

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://apm.amegroups.com/article/view/10.21037/apm-22-9/coif). LD, NH and AW are employees of the Bureau of Health Information (BHI), which is a New South Wales state government organization that funded the conduct of this work, and they were involved in deciding which research team to award the tender to; CV was part of the team, JLP led the team, that was awarded the BHI tender to conduct this work. The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Institutional Ethics Committee of South Eastern Sydney Local Health District (No. 2020/ETH03036) and informed consent was taken from all the participants.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Teisberg E, Wallace S, O'Hara S. Defining and Implementing Value-Based Health Care: A Strategic Framework. Acad Med 2020;95:682-5. [Crossref] [PubMed]
2. Jayakumar P, Bozic KJ, Lee TH. Information asymmetry: The untapped value of the patient. NEJM Catalyst 2019: October 15th. Available online: https://catalyst.nejm.org/doi/full/10.1056/CAT.19.0700
3. Bull C, Byrnes J, Hettiarachchi R, et al. A systematic review of the validity and reliability of patient-reported experience measures. Health Serv Res 2019;54:1023-35. [Crossref] [PubMed]
4. New South Wales Ministry of Health. Elevating the human experience: Our guide to action for patient, family, carer and caregiver experiences. Sydney: NSW Health; 2020.
5. World Health Organization. Palliative care 2020. Available online: https://www.who.int/health-topics/palliative-care.
6. Pask S, Dell'Olio M, Murtagh FEM, et al. The Effects of Opioids on Cognition in Older Adults With Cancer and Chronic Noncancer Pain: A Systematic Review. J Pain Symptom Manage 2020;59:871-893.e1. [Crossref] [PubMed]
7. Kurita GP, Benthien KS, Sjøgren P, et al. Identification of the predictors of cognitive impairment in patients with cancer in palliative care: a prospective longitudinal analysis. Support Care Cancer 2017;25:941-9. [Crossref] [PubMed]
8. Virdun C, Luckett T, Davidson PM, et al. Dying in the hospital setting: A systematic review of quantitative studies identifying the elements of end-of-life care that patients and their families rank as being most important. Palliat Med 2015;29:774-96. [Crossref] [PubMed]
9. Virdun C, Luckett T, Lorenz K, et al. Dying in the hospital setting: A meta-synthesis identifying the elements of end-of-life care that patients and their families describe as being important. Palliat Med 2017;31:587-601. [Crossref] [PubMed]
10. Tourangeau R. Cognitive science and survey methods: A cognitive perspective. In: National Research Council, editor. Cognitive aspects of survey design: Building a bridge between disciplines Washington, DC: National Academy Press; 1984. p. 73-100.
11. Peterson CH, Peterson NA, Powell KG. Cognitive interviewing for item development: Validity evidence based on content and response processes. Meas Eval Couns Dev 2017;50:217-23. [Crossref]
12. Veronese S, Rabitti E, Costantini M, et al. Translation and cognitive testing of the Italian Integrated Palliative Outcome Scale (IPOS) among patients and healthcare professionals. PLoS One 2019;14:e0208536. [Crossref] [PubMed]
13. Murtagh FE, Addington-Hall JM, Higginson IJ. The value of cognitive interviewing techniques in palliative care research. Palliat Med 2007;21:87-93. [Crossref] [PubMed]
14. Antunes B, Ferreira PL. Validation and cultural adaptation of the Integrated Palliative care Outcome Scale (IPOS) for the Portuguese population. BMC Palliat Care 2020;19:178. [Crossref] [PubMed]
15. Beck I, Olsson Möller U, Malmström M, et al. Translation and cultural adaptation of the Integrated Palliative care Outcome Scale including cognitive interviewing with patients and staff. BMC Palliat Care 2017;16:49. [Crossref] [PubMed]
16. Bergh I, Kvalem IL, Aass N, et al. What does the answer mean? A qualitative study of how palliative cancer patients interpret and respond to the Edmonton Symptom Assessment System. Palliat Med 2011;25:716-24. [Crossref] [PubMed]
17. Meyer R, Drewniak D, Hovorka T, et al. Questioning the Questionnaire: Methodological Challenges in Measuring Subjective Quality of Life in Nursing Homes Using Cognitive Interviewing Techniques. Qual Health Res 2019;29:972-86. [Crossref] [PubMed]
18. Boeije H, Willis G. The cognitive interviewing reporting framework (CIRF). Methodology (Gott) 2013;9:87-95. [Crossref]
19. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007;19:349-57. [Crossref] [PubMed]
20. Abernethy AP, Shelby-James T, Fazekas B, et al. The Australia-modified Karnofsky Performance Status (AKPS) scale: a revised scale for contemporary palliative care clinical practice. BMC Palliat Care 2005;4:7. [Crossref] [PubMed]
21. Eagar K, Watters P, Currow DC, et al. The Australian Palliative Care Outcomes Collaboration (PCOC)--measuring the quality and outcomes of palliative care on a routine basis. Aust Health Rev 2010;34:186-92. [Crossref] [PubMed]
22. Virdun CLT, Garcia M, Disalvo D, Phillips J. Palliative Care – experiences of patients and carers project report 2: Focus groups undertaken in partnership with people (patients and carers) who have experience of palliative and end of life care. Sydney, Australia: IMPACCT (Improving Palliative, Aged and Chronic Care through Clinical Research and Translation); 2021.
23. Priede C, Farrall S. Comparing results from different styles of cognitive interviewing: ‘verbal probing’ vs. ‘thinking aloud’. Int J Soc Res Methodol 2011;14:271-87. [Crossref]
24. Buers C, Triemstra M, Bloemendal E, et al. The value of cognitive interviewing for optimizing a patient experience survey. Int J Soc Res Methodol 2014;17:325-40. [Crossref]
25. Burkard C, Rochat L, Juillerat Van der Linden AC, et al. Is working memory necessary for implementation intentions to enhance prospective memory in older adults with cognitive problems? J Appl Res Mem Cogn 2014;3:37-43. [Crossref]
26. Maadi M, Akbarzadeh Khorshidi H, Aickelin U. A Review on Human-AI Interaction in Machine Learning and Insights for Medical Applications. Int J Environ Res Public Health 2021;18:2121. [Crossref] [PubMed]
27. Saunders CH, Durand MA, Scalia P, et al. User-Centered Design of the consideRATE Questions, a Measure of People's Experiences When They Are Seriously Ill. J Pain Symptom Manage 2021;61:555-565.e5. [Crossref] [PubMed]
28. McGrath P, Ogilvie KF, Rayner RD, et al. The "right story" to the "right person": communication issues in end-of-life care for Indigenous people. Aust Health Rev 2005;29:306-16. [Crossref] [PubMed]