Outcome measurement—a scoping review of the literature and future developments in palliative care clinical practice

Introduction

“Measurement is an abstraction. It involves developing a set of rules to assign numbers to represent a concept.” (p. 8) (1). An individual concept can be deconstructed in ordered levels, usually represented by numerical values, which constitutes a system. This system is the commonality which allows for communicating and understanding about that concept. It is then possible to represent quantities of a trait, an attribute or a state and classify subjects against a standardised scale which will aid in the interpretation and meaning of that quantity. To measure a concept, one must define it and deconstruct it in its several components to build an outcome measure (2).

Depending on the field of inquiry, outcome measurement is defined in different ways. A broad definition would be: “to determine and evaluate results of an action, or program and their comparison with the intended or projected results” (3). However, to understand an outcome and give it meaning, one needs to measure it and consider its context. Choosing which outcomes to measure and why will first depend on the field of use. In health, an outcome is “the change in a patient’s current and future health status that can be attributed to preceding healthcare” (4).

In an era of evidence-based medicine, measurement is a fundamental aspect of medical research and clinical practice: it makes diagnosis, prognosis and evaluation of interventions possible (5). Hence, the rigor of the methodological steps regarding measurement properties, the correct use of the measure, and the interpretation of results are paramount because these will influence the quality of care provided to patients (5). Choosing the correct measure requires taking the purpose of measuring into consideration, the population under study, the context and defining what will be measured. Moreover, the measure should be culturally adapted and validated for the targeted population.

Since Dame Cicely Saunders introduced, on the palliative care arena, the concept of holistic care for the dying patient and the concept of “total pain”, it became clear that patient-centred care, which takes into account all that it is considered to be important by the individual, is the appropriate care to be delivered to this population (6). This includes understanding and respecting individual and cultural preferences and belief systems about treatments and death for each individual patient. Palliative care strives toward providing holistic support by assessing all multidimensional aspects of the patient whilst addressing diagnosis, prognosis and the complexity of problems that arise during the disease trajectory (7-10). Cultural values, beliefs, needs and individual preferences of each patient should, ideally, be considered in a joint clinical decision-making process (11).

Methods

Scoping reviews are a relatively new approach for synthesizing research evidence and lack a definitive methodology internationally accepted (12,13). Based on recommendations by Levac et al. (14), whose work is built on the framework developed by Arksey and O’Malley (12), we present the following five key phases: (I) identifying the research question; (II) identifying relevant studies; (III) study selection; (IV) charting the data; and (V) collating, summarizing, and reporting the results. The optional final step, “consultation exercise”, was not performed. For steps 4 and 5 we present the data according to the following headings: “Outcome measurement in health care: historical development” and headings based on the User’s Guide for Implementing Patient-Reported Outcomes Assessment in Clinical Practice (15) proposed by The International Society for Quality of Life Research (ISOQOL): (I) identifying the goals for collecting patient reported outcome measures (PROMs) in clinical practice; (II) selecting the patients, setting, and timing of assessments; (III) determining which questionnaire(s) to use; (IV) choosing a mode for administering the questionnaire; (V) designing processes for reporting results and identifying aids to facilitate score interpretation; (VI) developing strategies for responding to issues identified by the questionnaires; and (VII) evaluating the impact of the PRO intervention on the practice.

Identifying the research question

This review was guided by the question “What are the issues to be considered and the decisions to be made when using outcome measurement in palliative care clinical practice?”.

Identifying relevant studies and study selection

This scoping review is based on a search and search results of a published systematic review on implementing PROMs in palliative care clinical practice (16). Medline, PsycInfo, Cumulative Index to Nursing and Allied Health Literature, Embase and British Nursing Index were systematically searched from 1985. Hand searching of reference lists for all included articles and relevant review articles was performed. A total of 3,863 articles were screened. Sixty were included in this scoping review.

Charting the data and collating, summarizing, and reporting the results

Outcome measurement in health care: historical development

Outcome measurement has a long history in health care, especially using mortality to assess outcomes. One of the early examples of routine collection of mortality rates goes back to 1532 when Henry VII began gathering weekly “Bills of Mortality” due to epidemic plague-related deaths (17). In 1754 the first trial was conducted for the treatment of scurvy in British sailors, by Lind (18,19). Lind studied 12 scurvy sailors by observing putrefaction of gums, spots, weakness of knees and overall health condition. He gave six different treatments to 6 pairs of sailors and observed that within 6 days the oranges and lemon group had returned to being fit and healthy. The other groups did not have any major changes (20). By recording and learning from outcomes it was then possible to make recommendations for dietary modifications, although the British Navy took almost 50 years to implement the recommended intervention (dietary modification) (19).

In the 19^th century Florence Nightingale developed a routine clinical outcomes system during the Crimean war in 1854 to study, and try to reduce, the number of deaths. She collected data on cause of death and showed the association between sanitary conditions and mortality rates: wounded soldiers were not dying solely due to actual injuries they had sustained in the battle field but rather due to insanitary conditions in the hospital they were being cared for (21). By 1856, after several improvements to the hospitals and the care provided, mortality fell (17). However, Nightingale was not keen on hospital mortality numbers alone as an outcome measure and maintained the belief that collection of non-mortal information was more appropriate and useful to understand interventions (22).

During the same century John Snow became the precursor of epidemiology after elucidating the cholera outbreak source in the Broad Street water pump in London, by associating contaminated water and gastrointestinal symptoms (19). The outcomes collected were incidence of cholera and mortality throughout the weeks of the occurrence (23).

In the early 20^th century, closer to the current concept of outcome as a result of an intervention, Ernest Amory Codman, an American orthopaedic surgeon developed the notion of following the patient’s recovery long enough to observe whether or not the treatment (intervention) had been successful, and if it had not, to ask why (24). This was the “end result idea”, a major step in collecting outcomes beyond just mortality rates, and Codman operationalised it further by developing a card system which was filled with details of each case before and after surgery. One year later he went back to the card, examine the patient and the surgery outcome would be evaluated based on the condition of the patient. Some of Codman’s colleagues did the same in his own hospital and other hospitals, making evaluation of the outcome of the surgical treatment and comparison of individual surgeons and hospitals available to the public (24). This was not welcomed by most of his peers, especially the more senior ones, “whose status was measured by seniority and not by the results of their practice” (24). Nevertheless, it was a major advance since the early 1800s and the solely collection of hospital mortality rates. About a century ago, Codman advocated that in order to clearly establish a relation between care and its results, it was paramount to record data on a large number of observations throughout time and that the different uses of those that included monitoring quality, advancing clinical science, establishing accountability, allocating resources and managing them efficiently, setting personnel policies, promoting functional differentiation, allowing informed choice by physicians and prospective patients, pricing services and remunerating providers, and stimulating fair competition (25).

By 1966 Donabedian takes this work further and publishes “Evaluating the quality of medical care” describing the structure-process-outcome model and concluding that it is only by evaluating the outcome of the intervention that one can understand the effectiveness and quality of the care provided (26). In addition, he noted that choosing the correct outcome(s) is paramount and that measuring easily-collected outcomes, which are irrelevant, is of little use. Also, having different perspectives (patient, clinician, family member) and considering the context of those outcomes is the best way to avoid several pitfalls of health outcome measurement (27).

Nowadays, an era of evidence-based practice, routine outcome measurement is not only seen as a necessity but also as a requirement to improve the quality of patient care. But the questions “Who”, “When”, “How”, “What” to measure and how to store, analyse and interpret the data remain as barriers to its use in clinical practice (28). Moreover, defining the quality of care to enable measurement has its own pitfalls: on the one hand the definition of quality of care is influenced by societal values such as compassion and value for money, by who sets the criteria and how those criteria are set and by the technicality of medical procedures: “Too much concern with the technical management of illness will result in a diminution in attention to prevention, rehabilitation, and coordination and continuity of care, and the consequent effects on the clinician-patient relationship” (19); on the other, it depends on the individual’s subjective experience with health and disease, and their expectations based on their life experiences (29).

Identifying the goals for collecting PROMs in clinical practice

PROMs are a category of outcomes that one can distinguish from other types of outcomes, including laboratory measures, clinician ratings, and caregiver reports because PROMs require that the information captured comes directly from patients, informing about symptoms and constructs on how patients function or feel in relation to a health condition and its therapy (30). Due to its subjective nature, each construct is inherent to each patient and only they can inform how they feel at a particular time. Hence, PROMs are more and more considered to be the gold-standard of outcome measurement of subjective experiences (31,32).

Having relevant questions on a particular disease, may help acceptance of the questionnaires by patients. In addition, by being sensitive to the condition these measures might detect small clinical changes relevant to patient care (33). Traditionally, PROMs have been developed for clinical research. However, in recent years there has been a shift and the use of PROMS in clinical practice has increased, as well as for health policy decisions. Data collected in a systematic way, may benefit clinical practice both at the individual patient level by aiding in clinical decision making, and at the population level to support efficient health service delivery processes, by performing audits and benchmarking (34).

Care of individual patient

Measuring health outcomes is essential in making a diagnosis, because based on scores obtained decisions are made on possible diagnosis and/or application of subsequent diagnostic tests; for the decision-making process on managing symptoms and other phenomenon which occur in patients. In some cases, outcome measurement helps to predict which patients could benefit from a particular intervention and allows to document whether there was an improvement or not after an intervention. PROMs can be an aid to identify and screen physical, psychological, spiritual and social unmet needs. Additionally, these measures can act as a communication aid between the patient and their family and the healthcare professional, as well as, between different clinical teams. This is crucial since evidence suggests that continuity of care and multidisciplinary collaboration improve the experience and patients and families when transitioning from curative to palliative care (35).

Service/population level

As a means to evaluate the quality of care provided to patients, health services and organisations can benchmark their own processes with others in a continuous way by measuring and comparing outcomes of the same processes. By doing this, it is possible to reveal leaders in particular areas, which can inform others on their processes, so they can improve, and, to identify issues which need to be refined. Hence, benchmarking is a team effort because the outcome will involve changes of current practices, with effects felt throughout the service/organisation (36). Aggregating data by patient population, healthcare professional, service or organisation will aid in answering questions of improvement of care by conducting periodical internal audits on specific issues, to continually measure current practice against a defined (desired) standard (36).

In a reality of limited resources, evidence on costs and cost-effectiveness of healthcare interventions is paramount to aid in deciding how those resources are allocated (19). It is estimated that 20% to 25% of hospital beds are allocated to end-of-life care (37). There is no doubt that there are difficulties in capturing relevant outcomes to be able to conduct economic studies in different palliative care populations and settings (37,38). In general, tools used to collect cost data are not standardised, but rather developed for each individual study, which presents a major problem to make any sort of comparison (39). Another problem is deciding which perspective(s) should be taken for the analysis, and indeed, there is a lack of studies which use the patient and family perspective in term of cost, even though there is evidence that direct and indirect costs of care have an impact regarding health and well-being (39).

Selecting the patients, setting, and timing of assessments

Regardless of a specific diagnosis, patients with advanced disease have unique needs and, as the disease progresses and their physical condition deteriorates, changes in cognitive abilities are also expected to occur closer to the time of death, making it increasingly challenging to capture the desired outcomes directly from the patient (40). It would be important to plan for assisting the patient when collecting those data and to think about proxy collection when the patient is no longer able to answer the questionnaires alone. One can refer to patient reported and proxy reported measures as patient-centred outcome measures (PCOM) (41).

In 1995, Wilson and Cleary published a conceptual model of patient outcomes arguing that, Health Related Quality of Life (HRQoL) measures are responsive to crucial clinical changes and therefore important supplements to physiological and biological measures of health status (42). These authors incite thinking of different measures in a continuum of complexity of five levels: (I) biological and physiological factors; (II) symptoms; (III) functioning; (IV) general health perceptions and (V) overall quality of life, and describe the relationships among them. The model takes into consideration the characteristics of the individual and of the environment they are a part of. The higher the level, the more difficult it is to define and measure constructs because the number of variables or inputs increases. They advocated that if the main goal of care is to improve patient outcomes, then there is a need to identify causal pathways that link different types of outcome to each other to facilitate the association between diagnosis and therapy. A flow in the reverse direction might also be possible, which might have important implications for clinical practice. This was explored by Ferrans and colleagues (43) when developing a modified Wilson and Cleary model: if, for instance, anxiety is the cause for low overall QoL, then anxiety should be treated, but, if anxiety is the result of low overall QoL, then the cause of low overall QoL should be diagnosed and treated which should make anxiety levels decrease. Building on the modified Wilson and Cleary model, in 2007 Osoba published a model for HRQoL assessment in clinical practice, describing how to incorporate HRQoL measures in clinical practice, throughout the disease trajectory (44). Four key moments are described, in which patient reported outcomes should be collected in clinical practice, to inform and assist in clinical decision making: (I) during initial history and physical examination to provide baseline data and understand multidimensional needs of the patient at that moment; (II) during laboratory and imaging testing so that all information on health outcomes relating to each particular patient is available to the clinician; (III) during treatment(s) to understand, and, if needed act upon, the impact of the treatment on the patient; and (IV) during monitoring and follow up to determine if the treatment had the expected outcomes and if not, to inform and aid on the best way to proceed. Hence, this is an iterative process, especially if the aim of the treatment(s) is to keep the disease under control or if the disease is progressing. The timing of assessments and the setting will inevitably vary and must be taken into account when planning.

Determining which questionnaire(s) to use

Whether collecting health outcomes is for patient assessment, for gathering population level data or for an audit cycle, the development of new measures continues to thrive and choosing an outcome measure is increasingly complex. In the past two decades many PROMs have been developed, and increasingly their role in clinical practice has been stressed. Harding et al. (45) published results of the first pan-European survey describing the views of professionals on the use and preferred features of outcome measures in palliative care. The main findings suggest that new tools are not required but that refinement of existing ones with appropriate scientific properties should be advocated. There is now international consensus on what psychometric properties outcome measures should be assessed for, whether during the development of new measures or assessing existing ones (46,47). Those properties are: reproducibility, internal consistency, content validity, construct validity, criterion validity, floor and ceiling effects, responsiveness and interpretability (28,31,48,49). See Table 1 for an example of a measure and Mokkink et al. (47) for definitions of these concepts.

Table 1 Measurement properties of the Palliative care Outcome Scale (POS)
Full table

For advance disease populations, PROMs should be simple, short, multidimensional so that patients are not burdened with data collection. When considering using them in clinical practice, these measures should provide relevant clinical data which can be used in real time, assisting healthcare professionals caring for their patients. The tendency seems to be of a modular approach: a core, condition specific instrument and several modules specific to a disease (50-52).

Choosing a mode for administering the questionnaire

The introduction of these measures in clinical practice has been slow and difficult (16,53). This leads to a lack of standardised measurement, making it difficult to compare health outcomes in populations, settings or even countries (54-58). The frequency of administration can vary greatly depending on setting, but also, as the disease progresses and if the patient is in active treatment. Whether it is at each visit, between visits, weekly or monthly, it is important to be vigilant of changes in patient’s outcomes (59,60). Using paper questionnaires or electronic versions has advantages and disadvantages. Although not all settings/organizations may have both options, it is important to understand if patients are comfortable and are able to complete PROMs with the selected mode. Other methods published in the literature include phone, email, mail, web-based, app for smartphone (15). Nurses are the healthcare providers at a privileged position to contribute to using PROMs in clinical practice. The role of nurses in the provision of palliative care is paramount regardless of the patient diagnosis and the context of care. Whether by direct contact with patients and families by assessing and responding to physical, psychosocial, emotional and spiritual needs or by indirect contact by making use of management and multidisciplinary skills, nurses could potentially contribute to the implementation and development of outcome measurement in palliative care clinical practice (61).

Designing processes for reporting results and identifying aids to facilitate score interpretation

A systematic review has looked at means of capturing the desired patient reported outcomes in palliative care settings, including pen and paper, laptop, and tablet, and feeding them back to clinicians. Feedback consisting of a one-page report with numerical and graphical display of main needs at that point in time does improve clinician awareness of patient unmet needs, hence benefiting patient outcomes. However, most evidence comes from cancer populations, thus, more evidence on non-cancer populations is needed (41). This information can be available prior to the consultation, during or after the consultation, depending on when the questionnaire is filled by the patient and whether it is in paper or electronic format. Electronic questionnaires’ software makes it possible to immediately have the results available, and, if there are multiple assessments in time, to have those longitudinal data displayed in a way that helps clinicians understand how the patients’ condition is evolving throughout time. It is important to explore the clinical utility of each item of a measure as well as defining possible cut-offs to aid in clinical decision making to make a PROM more suitable for routine use (62).

Developing strategies for responding to issues identified by the questionnaires

Collecting data with validated measures and not use the information it provides in clinical practice to the immediate benefit of the patient and family, seems like a big waste of resources, and raises ethical issues because if an individual is providing clinical information, then it should be used for their clinical benefit.

Predicting the course of the illness to make the best treatment decision for each individual patient has relied mostly on using multivariate models (frequentist approach). Typically, predictors are selected by adjusting the variables and their effect on the chosen outcome is checked. If there is statistical significance the variable is called an independent predictor. One of the main criticisms to this approach is that often correlation is confused with causation (63). Just because there is a correlation does not mean that the independent predictors are causal factors, it could be that there is correlation due to the available data and the selection of variables made to conduct the analysis (63). Another issue is heterogeneity: relying on the population “mean” treatment effect may not be the correct choice for an individual patient. Other approaches are being advocated to support clinical decisions, since these allow for a more personalised prediction of the disease trajectory. Bayesian networks are probabilistic graphical models suitable for inference and can be used to model causality (64). They allow to incorporate prior knowledge, multiple variables and independence assumptions, while learning the structure and parameters of the network. Each time new data are added to the model there is the possibility to calculate new probabilities which in turn updates the evidence and allows to manage uncertainty (65,66). By collecting data in clinical practice, we are gathering data which are much closer to the “truth” or reality of patients and families. However, one of the main criticisms to this approach is its subjectivity (66) because you select the parameters to be measured.

Evaluating the impact of the PRO intervention on the practice

In the past years, the gap between research findings and their application in clinical practice has been well documented (67,68). Different interventions to bridge that gap, such as development of clinical guidelines, continuing professional education and financial incentives have been developed and continue to be used today (69,70). However, one of the main challenges in the implementation field is how to conceptualise, measure and evaluate whether those interventions are successful (71). In 2013 the World Health Organization (WHO) published a Practical Guide for implementation research in health, field which has been expanding and becoming more and more important as the global health community continues to grow (72). The document describes basic concepts and language, definitions and who should be involved, as well as, methodology, study designs and implementation outcomes. It has been recognised that implementation research is a fundamental research topic in health, since it contributes to maximise the beneficial impact of interventions.

Modern statistical models allied with computer technology, help making this a reality. Palliative care is multidisciplinary and often deals with the last phase of someone’s life. So, having those data available for clinicians would be a tremendous help in gathering and understanding the patient clinical history. Ultimately, feeding back data to clinicians in real time will allow better individual care and feeding back aggregated data at nation level will allow international comparisons and inform decision makers to improve/change health policies nationally (73). This will only be feasible and effective when PROM scores are integrated in the health organization’s information system (74). One successful example is The Palliative Care Outcomes Collaboration (PCOC) in Australia, a national program which uses standardised clinical assessment tools to measure and benchmark patient outcomes in palliative care by coordinating patient outcomes reporting, education program, and quality activities (74-76). See Table 2 which summarizes the PCOC model as an example of a PROM system.

Table 2 A summary of the PCOC model as an example of a PROM system
Full table

Conclusions

Outcome measurement has a long history in health care. The routine use of PCOMs will change and improve how healthcare is delivered and ultimately organised. It allows to monitor and document patient progress in real time, as well as, over time. Additionally, there is possibility to provide audits and comparisons, cost-effectiveness studies and, at an international level, for cross-country comparisons. Ultimately it will inform policy decision makers to improve the quality of services and the care provided and by allowing and facilitating decisions on allocating resources.

Acknowledgements

Funding: Bárbara Antunes is funded by Foundation for Science and Technology (FCT) PhD Fellow (grant number: PD/BD/113664/2015). Doctoral Program in Clinical and Health Services Research (PDICSS) is funded by FCT (grant number: PD/0003/2013).

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.

References

1. Roach KE. Measurement of health outcomes: Reliability, validity and responsiveness. Proceedings of the American Academy of Orthotists and Prosthetists. 2006:8-12.
2. Fawcett AJL. Principles of assessment and outcome measurement for occupational therapists and physiotherapists: theory, skills and application. Chichester: John Wiley & Sons, 2007, p.xxi, 467.
3. Business dictionary. 2012. Available online: http://www.businessdictionary.com/definition/outcome-measure.html
4. Donabedian A. Explorations in quality assessment and monitoring. Ann Arbor, Mich.: Health Administration Press, 1980.
5. De Vet HCW, Terwee CB, Mokkink LB, et al. Measurement in Medicine. 1st edition. Cambridge: Cambridge University Press, 2011.
6. Clark D. Between hope and acceptance: the medicalisation of dying. BMJ 2002;324:905-7. [Crossref] [PubMed]
7. Addington-Hall JM. Research methods in palliative care. Oxford: Oxford University Press, 2007, p.xii, 315 p.
8. Bajwah S, Higginson IJ, Ross JR, et al. Specialist Palliative Care is More Than Drugs: A Retrospective Study of ILD Patients. Lung 2012;190:215-20. [Crossref] [PubMed]
9. Jocham HR, Dassen T, Widdershoven G, et al. Quality of life in palliative care cancer patients: a literature review. J Clin Nurs 2006;15:1188-95. [Crossref] [PubMed]
10. Stewart AL, Teno J, Patrick DL, et al. The concept of quality of life of dying persons in the context of health care. J Pain Symptom Manage 1999;17:93-108. [Crossref] [PubMed]
11. Kupfer JM, Bond EU. Patient satisfaction and patient-centered care: necessary but not equal. JAMA 2012;308:139-40. [Crossref] [PubMed]
12. Arksey H, O’Malley L. Scoping studies: Towards a Methodological Framework. Int J Soc Res Methodol 2005;8:19-32. [Crossref]
13. Pham MT, Rajić A, Greig JD, et al. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synth Methods 2014;5:371-85. [Crossref] [PubMed]
14. Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci 2010;5:69. [Crossref] [PubMed]
15. User’s Guide for Implementing Patient-Reported Outcomes Assessment in Clinical Practice. visited on the 27th July 2014. Available online: http://www.isoqol.org/UserFiles/file/UsersGuide.pdf
16. Antunes B, Harding R, Higginson IJ, et al. Implementing patient-reported outcome measures in palliative care clinical practice: a systematic review of facilitators and barriers. Palliat Med 2014;28:158-75. [Crossref] [PubMed]
17. Iezzoni LI. 100 apples divided by 15 red herrings: a cautionary tale from the mid-19th century on comparing hospital mortality rates. Ann Intern Med 1996;124:1079-85. [Crossref] [PubMed]
18. Bloom BS. Does it work? The outcomes of medical interventions. Int J Technol Assess Health Care 1990;6:326-32. [Crossref] [PubMed]
19. O'Connor RJ, Neumann VC. Payment by results or payment by outcome? The history of measuring medicine. J R Soc Med 2006;99:226-31. [Crossref] [PubMed]
20. Lind J. A teatrise of the scurvy. Edinburgh: Sands, Murray and Cochran, 1753.
21. Cohen IB. Florence Nightingale. Sci Am 1984;250:128-37. [Crossref] [PubMed]
22. Nightingale F. Notes on Matters affecting the Health, Efficiency and Hospital Administration of the British Army. London: Harrison and Sons, 1858.
23. Snow J. On the Mode of Communication of Cholera. Second ed. London: John Churchill, 1855.
24. Kaska SC, Weinstein JN. Historical perspective. Ernest Amory Codman, 1869-1940. A pioneer of evidence-based medicine: the end result idea. Spine 1998;23:629-33. [Crossref] [PubMed]
25. Donabedian A. The end results of health care: Ernest Codman's contribution to quality assessment and beyond. Milbank Q 1989;67:233-56; discussion 257-67. [Crossref] [PubMed]
26. Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q 1966;44 Suppl:166-206. [Crossref] [PubMed]
27. Donabedian A. The quality of care. How can it be assessed? JAMA 1988;260:1743-8. [Crossref] [PubMed]
28. Turner-Stokes L, Williams H, Sephton K, et al. Engaging the hearts and minds of clinicians in outcome measurement - the UK Rehabilitation Outcomes Collaborative approach. Disabil Rehabil 2012;34:1871-9. [Crossref] [PubMed]
29. Carr AJ, Gibson B, Robinson PG. Measuring quality of life - Is quality of life determined by expectations or experience? Brit Med J 2001;322:1240-3. [Crossref] [PubMed]
30. Patrick DL, Burke LB, Powers JH, et al. Patient-reported outcomes to support medical product labeling claims: FDA perspective. Value Health 2007;10 Suppl 2:S125-37. [Crossref] [PubMed]
31. Higginson IJ, Carr AJ. Using quality of life measures in the clinical setting. BMJ 2001;322:1297-300. [Crossref] [PubMed]
32. Vignaroli E and Bruera E. Multidimensional assessment in palliative care. In: Bruera E, Higginson I, von Gunten CF, et al. editors. Textbook of Palliative Medicine and Supportive Care. London: Hodder Arnold, 2006:319-32.
33. Malý M, Vondra V. Generic versus disease-specific instruments in quality-of-life assessment of chronic obstructive pulmonary disease. Methods Inf Med 2006;45:211-5. [Crossref] [PubMed]
34. Currow DC, Eagar K, Aoun S, et al. Is It Feasible and Desirable to Collect Voluntarily Quality and Outcome Data Nationally in Palliative Oncology Care? J Clin Oncol 2008;26:3853-9. [Crossref] [PubMed]
35. Gardiner C, Ingleton C, Gott M, et al. Exploring the transition from curative care to palliative care: a systematic review of the literature. BMJ Support Palliat Care 2015;5:335-42. [Crossref] [PubMed]
36. Benson HR. An introduction to benchmarking in healthcare. Radiol Manage 1994;16:35-9. [PubMed]
37. Smith S, Brick A, O’Hara S, et al. Evidence on the cost and costeffectiveness of palliative care: A literature review. Palliat Med 2014;28:130-50. [Crossref] [PubMed]
38. Coast J. Strategies for the economic evaluation of end of life care: Making a case for the capability approach. Expert Rev Pharmacoecon Outcomes Res 2014;14:473-82. [Crossref] [PubMed]
39. Gardiner C, Ingleton C, Ryan T, et al. What cost components are relevant for economic evaluations of palliative care, and what approaches are used to measure these costs? A systematic review. Palliat Med 2017;31:323-37. [Crossref] [PubMed]
40. Bausewein C, Daveson B, Benalia H, et al. Outcome Measurement in Palliative Care. PRISMA, London, 2011. Available online: http://pos-pal.org/Resources.php
41. Etkind SN, Daveson BA, Kwok W, et al. Capture, transfer, and feedback of patient centred outcomes data in palliative care populations: does it make a difference? A systematic review. J Pain Symptom Manage 2015;49:611-24. [Crossref] [PubMed]
42. Wilson IB, Cleary PD. Linking clinical variables with health-related quality of life. A conceptual model of patient outcomes. JAMA 1995;273:59-65. [Crossref] [PubMed]
43. Ferrans CE, Zerwic JJ, Wilbur JE, et al. Conceptual Model of Health-Related Quality of Life. J Nurs Scholarsh 2005;37:336-42. [Crossref] [PubMed]
44. Osoba D. Translating the Science of Patient-Reported Outcomes Assessment Into Clinical Practice. J Natl Cancer Inst Monogr 2007;37:5-11. [Crossref] [PubMed]
45. Harding R, Simon ST, Benalia H, et al. The PRISMA Symposium 1: outcome tool use. Disharmony in European outcomes research for palliative and advanced disease care: too many tools in practice. J Pain Symptom Manage 2011;42:493-500. [Crossref] [PubMed]
46. Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res 2010;19:539-49. [Crossref] [PubMed]
47. Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol 2010;63:737-45. [Crossref] [PubMed]
48. Streiner DL. Clinimetrics vs. psychometrics: an unnecessary distinction. J Clin Epidemiol 2003;56:1142-5; discussion 1146-9. [Crossref] [PubMed]
49. Terwee CB, Bot SD, de Boer MR, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007;60:34-42. [Crossref] [PubMed]
50. Sprangers MA, Cull A, Bjordal K, et al. The European Organization for Research and Treatment of Cancer. Approach to quality of life assessment: guidelines for developing questionnaire modules. EORTC Study Group on Quality of Life. Qual Life Res 1993;2:287-95. [Crossref] [PubMed]
51. Aaronson NK, Ahmedzai S, Bergman B, et al. The European Organization for Research and Treatment of Cancer QLQ-C30: a quality-of-life instrument for use in international clinical trials in oncology. J Natl Cancer Inst 1993;85:365-76. [Crossref] [PubMed]
52. Cella DF, Tulsky DS, Gray G, et al. The Functional Assessment of Cancer Therapy scale: development and validation of the general measure. J Clin Oncol 1993;11:570-9. [Crossref] [PubMed]
53. Dawson J, Doll H, Fitzpatrick R, et al. The routine use of patient reported outcome measures in healthcare settings. BMJ 2010;340:c186. [Crossref] [PubMed]
54. Cormier JN, Askew RL. Assessment of patient-reported outcomes in patients with melanoma. Surg Oncol Clin N Am 2011;20:201-13. [Crossref] [PubMed]
55. Department of Primary Care Health Sciences HSaPRG. The use of Patient Reported Outcome Measures (PROMs) for clinical decision making and health policy 2011. Available online: http://www.primarycare.ox.ac.uk/hsprg/research/proms
56. Rose M, Bezjak A. Logistics of collecting patient-reported outcomes (PROs) in clinical practice: an overview and practical examples. Qual Life Res 2009;18:125-36. [Crossref] [PubMed]
57. Sloan JA, Halyard MY, Frost MH, et al. The Mayo Clinic manuscript series relative to the discussion, dissemination, and operationalization of the Food and Drug Administration guidance on patient-reported outcomes. Value Health 2007;10 Suppl 2:S59-63. [Crossref] [PubMed]
58. Bossola M, Murri R, Onder G, et al. Physicians' knowledge of health-related quality of life and perception of its importance in daily clinical practice. Health Qual Life Outcomes 2010;8:43. [Crossref] [PubMed]
59. Valderas JM, Alonso J. Patient reported outcome measures: a model-based classification system for research and clinical practice. Qual Life Res 2008;17:1125-35. [Crossref] [PubMed]
60. Snyder CF, Aaronson NK, Choucair AK, et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res 2012;21:1305-14. [Crossref] [PubMed]
61. Husaini A. The role of nurses in providing palliative care for dying cancer patients: a meta-ethnographic synthesis. BMJ Support Palliat Care 2014;4:A1-2. [Crossref]
62. Antunes B, Murtagh F, Bausewein C, et al. Screening for Depression in Advanced Disease: Psychometric Properties, Sensitivity and Specificity of Two Items of the Palliative care Outcome Scale (POS). J Pain Symptom Manage 2015;49:277-88. [Crossref] [PubMed]
63. Pallant JF, Tennant A. An introduction to the Rasch measurement model: an example using the Hospital Anxiety and Depression Scale (HADS). Br J Clin Psychol 2007;46:1-18. [Crossref] [PubMed]
64. Yet B, Perkins ZB, Marsh W, et al. Towards a Method of Building Causal Bayesian Networks for Prognostic Decision Support. ProBioMed 11, Bled, Slovenia, July 2011.
65. Billheimer D, Gerner EW, McLaren CE, et al. Combined Benefit of Prediction and Treatment: A Criterion for Evaluating Clinical Prediction Models 2014. Cancer Inform 2014;13:93-103. [PubMed]
66. Sesen MB. Bayesian Networks for Clinical Decision Support in Lung Cancer Care 2013. Plos One 2013;8. [Crossref] [PubMed]
67. Green LA, Seifert CM. Translation of research into practice: why we can’t “just do it”. J Am Board Fam Pract 2005;18:541-5. [Crossref] [PubMed]
68. Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet 2003;362:1225-30. [Crossref] [PubMed]
69. Boaz A, Baeza J, Fraser A. the European Implementation Score Collaborative Group (EIS). BMC Res Notes 2011;4:212. [Crossref] [PubMed]
70. Lewis CC, Fischer S, Weiner BJ, et al. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci 2015;10:155. [Crossref] [PubMed]
71. Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011;38:65-76. [Crossref] [PubMed]
72. Peters DH, Tran NT, Adam T. Implementation research in health: a practical guide. Alliance for Health Policy and Systems Research. World Health Organization, 2013.
73. Greenhalgh J, Dalkin S, Gooding K, et al. Functionality and feedback: a realist synthesis of the collation, interpretation and utilisation of patient-reported outcome measures data to improve patient care. Southampton (UK): NIHR Journals Library; 2017 Jan.
74. Australian Health Services Research Institute, University of Wollongong. 2017. Accessed 27.02.2017. Available online: http://ahsri.uow.edu.au/pcoc/about/index.html
75. Eagar K, Watters P, Currow DC, et al. The Australian Palliative Care Outcomes Collaboration (PCOC) – measuring the quality and outcomes of palliative care on a routine basis. Aust Health Rev 2010;34:186-92. [Crossref] [PubMed]
76. Currow DC, Allingham S, Yates P, et al. Improving national hospice/palliative care service symptom outcomes systematically through point-of-care data collection, structured feedback and benchmarking. Support Care Cancer 2015;23:307-15. [Crossref] [PubMed]