Perspective

Stakeholder engagement in comparative effectiveness research: how will we measure success?

Danielle C Lavallee

Surgical Outcomes Research Center, University of Washington, 1107 NE 45th Street, Suite 502 Seattle, WA 98105, USA

Search for more papers by this author

Carla J Williams

Johns Hopkins School of Medicine, 733 N Broadway, Baltimore, MD 21205, USA

Search for more papers by this author

Ellen S Tambor

Center for Medical Technology Policy, 401 E Pratt Street, Suite 631, Baltimore, MD 21201, USA

Search for more papers by this author

Patricia A Deverka

* Author for correspondence

Center for Medical Technology Policy, 401 E Pratt Street, Suite 631, Baltimore, MD 21201, USA.

Search for more papers by this author

Email the corresponding author at pat.deverka@cmtpnet.org

Published Online:11 Sep 2012https://doi.org/10.2217/cer.12.44

View article

Abstract

Stakeholder engagement in comparative effectiveness research continues to gain national attention. While various methods are used to gather stakeholder expertise and form recommendations, evaluation of the stakeholder experience is often missing. The lack of evaluation prohibits assessing how effective and meaningful engagement practices are for enhancing research efforts and limits the ability to identify areas for future improvement. We propose that an evaluation plan of engagement processes be developed before stakeholder involvement begins and be required as part of a request for proposal or research grant where stakeholder input is being sought. Furthermore, we recommend the inclusion of six meta-criteria that represent normative goals of multiple studies: respect, trust, legitimacy, fairness, competence and accountability. To aid in the development of future evaluations, we have developed definitions for and matched specific examples of measuring each meta-criterion to serve a guide for others in the field.

Keywords:

Papers of special note have been highlighted as: ▪▪ of considerable interest

References

1 Washington AE, Lipstein SH. The Patient-Centered Outcomes Research Institute – promoting better information, decisions, and health. N. Engl. J. Med.365(15),e31 (2011).Crossref, Medline, Google Scholar
2 Manchikanti L, Falco F, Benyamin R, Helm S, Parr A, Hirsch J. The impact of comparative effectiveness research on interventional pain management: evolution from Medicare Modernization Act to Patient Protection and Affordable Care Act and the Patient-Centered Outcomes Research Institute. Pain Physician14,249–282 (2011).Crossref, Medline, Google Scholar
3 Helfand M, Tunis S, Whitlock EP et al. A CTSA agenda to advance methods for comparative effectiveness research. Clin. Transl. Sci.4(3),188–198 (2011).Crossref, Medline, Google Scholar
4 Deverka P, Lavallee D, Desai PJ et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J. Compar. Effect. Res.1(2),181–194 (2012).▪▪ Development of definitions for stakeholder engagement in comparative effectiveness research and a conceptual model for engaging stakeholders.Link, Google Scholar
5 Sox HC, Greenfield S. Comparative effectiveness research: a report from the Institute of Medicine. Ann. Intern. Med.151(3),203–205 (2009).Crossref, Medline, Google Scholar
6 Hoffman A, Montgomery R, Aubry W, Tunis SR. How best to engage patients, doctors, and other stakeholders in designing comparative effectiveness studies. Health Aff.29(10),1834–1841 (2010).Crossref, Medline, Google Scholar
7 Abelson J, Forest P-G, Eyles J, Smith P, Martin E, Gauvin F-P. Deliberations about deliberative methods: issues in the design and evaluation of public participation processes. Soc. Sci. Med.57(2),239–251 (2003).Crossref, Medline, Google Scholar
8 Chess C. Evaluating environmental public participation: methodological questions. J. Environ. Plan. Manage.43(6),769–784 (2000).Crossref, Google Scholar
9 Oels A. Evaluating stakeholder dialogues. In: Stakeholder Dialogues in Natural Resources Management. Stollkleemann S, Welp M (Eds.). Springer Berlin Heidelberg, Germany, 117–151 (2006).Google Scholar
10 Butterfoss FD. Process evaluation for community participation. Annu. Rev. Public Health27(1),323–340 (2006).Crossref, Medline, Google Scholar
11 Rowe G, Frewer LJ. Public participation methods: a framework for evaluation. Sci. Technol. Human Values25(1),3–29 (2000).Crossref, Google Scholar
12 Webler T, Tuler S, Krueger ROB. What is a good public participation process? Five perspectives from the public. Environ. Manage.27(3),435–450 (2001).Crossref, Medline, CAS, Google Scholar
13 Gardner J, Dowd A, Mason C, Ashworth P. A framework for stakeholder engagement on climate adaptation. CSIRO Climate Adaptation Flagship Working Paper (2009).Google Scholar
14 Rowe G, Marsh R, Frewer LJ. Evaluation of a deliberative conference. Sci. Technol. Human Values29(1),88–121 (2004).▪▪ Use of a questionnaire, evaluation checklist and telephone interviews to evaluate the success of a deliberative conference according to acceptance criteria (representativeness, early involvement, influence and transparency) and process criteria (resource accessibility, task definition, structured decision making and cost–effectiveness).Crossref, Google Scholar
15 Rowe G, Frewer LJ. Evaluating public-participation exercises: a research agenda. Sci. Technol. Human Values29(4),512–556 (2004).Crossref, Google Scholar
16 Abelson J, Giacomini M, Lehoux P, Gauvin F-P. Bringing “the public” into health technology assessment and coverage policy decisions: from principles to practice. Health Policy82(1),37–50 (2007).Crossref, Medline, Google Scholar
17 Hailey D, Nordwall M. Survey on the involvement of consumers in health technology assessment programs. Int. J. Technol. Assess. Health Care22(4),497–499 (2006).Crossref, Medline, Google Scholar
18 Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: assessing partnership approaches to improve Public Health. Annu. Rev. Public Health19,173–202 (1998).Crossref, Medline, CAS, Google Scholar
19 Sandoval JA, Lucero J, Oetzel J et al. Process and outcome constructs for evaluating community-based participatory research projects: a matrix of existing measures. Health. Educ. Res.27(4),680–690 (2011).Crossref, Medline, Google Scholar
20 Braun KL, Nguyen TT, Tanjasiri SP. Operationalization of community-based participatory research principles: assessment of the National Cancer Institute’s Community Network Programs. Am. J. Public Health102(6),1195–1203 (2011).Crossref, Medline, Google Scholar
21 Burgess J, Stirling A, Clark J et al. Deliberative mapping: a novel analytic-deliberative methodology to support contested science-policy decisions. Public Understand. Sci.16(3),299–322 (2007).Crossref, Google Scholar
22 Beierle TC, Konisky DM. Values, conflict, and trust in participatory environmental planning. J. Policy Anal. Manage.19(4),587–602 (2000).Crossref, Google Scholar
23 Santos SL, Chess C. Evaluating citizen advisory boards: the importance of theory and participant-based criteria and practical implications. Risk Analysis23(2),269–279 (2003).Crossref, Medline, Google Scholar
24 Renn O. Risk communication: towards a rational discourse with the public. J. Hazard. Mater.29,465–519 (1992).Crossref, Google Scholar
25 Webler T. ‘Right’ discourse in citizen participation: an evaluative yardstick. In: Fairness and Competence in Citizen Participation: Evaluating Models for Environmental Discourse. Renn O, Webler T, Wiedelmann P (Eds). Kluwar Academic Press, Boston, MA, USA, 35–86 (1995).Google Scholar
26 Carnes SA, Schweitzer M, Peelle EB, Wolfe AK, Munro JF. Measuring the success of public participation on environmental restoration and waste management activities in the US Department of Energy. Technol. Soc.20(4),385–406 (1998).Crossref, Google Scholar
27 Halvorsen KE. Assessing public participation techniques for comfort, convenience, satisfaction, and deliberation. Environ. Manage.28(2),179–186 (2001).Crossref, Medline, CAS, Google Scholar
28 Lauber TB, Knuth BA. Measuring fairness in citizen participation: a case study of moose management. Soc. Nat. Resources12,19–37 (1999).Crossref, Google Scholar
29 Franche R-L, Baril R, Shaw W, Nicholas M, Loisel P. Workplace-based return-to-work interventions: optimizing the role of stakeholders in implementation and research. J. Occup. Rehabil.15(4),525–542 (2005).Crossref, Medline, Google Scholar
30 Reed MS. Stakeholder participation for environmental management: a literature review. Biolog. Conserv.141(10),2417–2431 (2008).Crossref, Google Scholar
31 Webler T, Tuler S. Fairness and competence in citizen participation. Admin. Soc.32(5),566–595 (2000).Crossref, Google Scholar
32 Caron-Flinterman J, Broerse J, Teerling J et al. Stakeholder participation in health research agenda setting: the case of asthma and COPD research in the Netherlands. Sci. Public Policy33,291–304 (2006).Crossref, Google Scholar
33 Enright K, Bourns C. The case for stakeholder engagement. Stanford Soc. Innov. Rev.40–45 (2010).Google Scholar
34 Gable C, Shireman B. Stakeholder engagement: a three-phase methodology. Environ. Qual. Manage.14(3),9–24 (2005).Crossref, Google Scholar
35 Ellerbusch F, Gute D, Desmarais A, Woodin M. Community engagement as a component of revitalization: lessons learned from the Technical Outreach Services to Communities programme. Local Environ.11,515–535 (2006).Crossref, Google Scholar
36 Tindana PO, Singh JA, Tracy CS et al. Grand challenges in global health: community engagement in research in developing countries. PLoS Med.4(9),e273 (2007).Crossref, Medline, Google Scholar
37 Oxman A, Lewin S, Lavis J, Fretheim A. SUPPORT tools for evidence-informed health Policymaking (STP) 15: engaging the public in evidence-informed policymaking. Health Res. Policy Syst.7,1478–4505 (2009).Google Scholar
38 Mitton C, Smith N, Peacock S, Evoy B, Abelson J. Public participation in health care priority setting: a scoping review. Health Policy91(3),219–228 (2009).▪▪ Recognizes that deliberative approaches to engaging the public are becoming more common but formal evaluation of public engagement efforts are rare.Crossref, Medline, Google Scholar
39 Bruni RA, Laupacis A, Martin DK, Group for the U of TPS in HCR. Public engagement in setting priorities in health care. Can. Med. Assoc. J.179(1),15–18 (2008).Crossref, Google Scholar
40 Tompkins EL, Few R, Brown K. Scenario-based stakeholder engagement: incorporating stakeholders preferences into coastal planning for climate change. J. Environ. Manage.88(4),1580–1592 (2008).Crossref, Medline, Google Scholar
41 Laurian L, Shaw MM. Evaluation of public participation. J. Plan. Edu. Res.28(3),293–309 (2009).Crossref, Google Scholar
42 Padgett SM, Bekemeier B, Berkowitz B. Collaborative partnerships at the state level: promoting systems changes in public health infrastructure. J. Public Health Manage. Prac.10(3),251–257 (2004).Crossref, Medline, Google Scholar
43 Amaeshi KM, Crane A. Stakeholder engagement: a mechanism for sustainable aviation. Corp. Soc. Responsibil. Environ. Manage.13(5),245–260 (2006).Crossref, Google Scholar
44 Burton H, Adams M, Bunton R, Schröder-Bäck P. Developing stakeholder involvement for introducing public health genomics into public policy. Public Health Genomics12(1),11–19 (2009).Crossref, Medline, CAS, Google Scholar
45 Rowe G, Horlick-Jones T, Walls J, Pidgeon N. Difficulties in evaluating public engagement initiatives: reflections on an evaluation of the UK GM Nation? Public debate about transgenic crops. Public Understand. Sci.14(4),331–352 (2005).Crossref, Google Scholar
46 Shindler B, Neburka J. Public participation in forest planning: 8 attributes of success. J. Forestry95,17–19 (1997).Google Scholar
47 Pedersen ER. Making corporate social responsibility (CSR) operable: how companies translate stakeholder dialogue into practice. Business Soc. Rev.111(2),137–163 (2006).Crossref, Google Scholar
48 Hanson CE, Palutikof JP, Dlugolecki A, Giannakopoulos C. Bridging the gap between science and the stakeholder: the case of climate change research. Climate Res.31(1),121–133 (2006).Crossref, Google Scholar
49 Sibbald S, Singer P, Upshur R, Martin D. Priority setting: what constitutes success? A conceptual framework for successful priority setting. BMC Health Serv. Res.9(1),43 (2009).▪▪ Development of a framework for successful stakeholder priority setting in healthcare that relates to both process and outcome components by asking relevant stakeholders what they understand successful priority setting to mean.Crossref, Medline, Google Scholar
50 Apostolakis GE, Pickett SE. Deliberation: integrating analytical results into environmental decisions involving multiple stakeholders. Risk Analysis18(5),621–634 (1998).Crossref, Google Scholar
51 Gibson J, Martin D, Singer P. Setting priorities in health care organizations: criteria, processes, and parameters of success. BMC Health Serv. Res.4(1),25 (2004).Crossref, Medline, Google Scholar
52 Goldenhar LM, LaMontagne AD, Katz T, Heaney C, Landsbergis P. The intervention research process in occupational safety and health: an overview from the National Occupational Research Agenda Intervention Effectiveness Research Team. J. Occup. Environ. Med.43(7),616–622 (2001).Crossref, Medline, CAS, Google Scholar
53 Tuler S, Webler T. Voices from the forest: what participants expect of a public participation process. Soc. Nat. Resources12,437–453 (1999).Crossref, Google Scholar
54 Renn O. A model for an analytic-deliberative process in risk management. Environ. Sci. Technol.33(18),3049–3055 (1999).Crossref, CAS, Google Scholar
55 Daniels N, Sabin JE. Accountability for reasonableness: an update. BMJ337 (2008).Medline, Google Scholar
56 Menon D, Stafinski T, Martin D. Priority-setting for healthcare: who, how, and is it fair? Health Policy84(2–3),220–233 (2007).Crossref, Medline, Google Scholar
57 Jones J, Hunter D. Consensus methods for medical and health services research. BMJ311,376–380 (1995).Crossref, Medline, CAS, Google Scholar
58 Delbecq AL, Van de Ven AH. A group process model for problem identification and program planning. J. Appl. Behav. Sci.7(4),466–492 (1971).Crossref, Google Scholar
59 Chappel D, Bailey J, Stacy R, Rodgers H, Thomson R. Implementation and evaluation of local-level priority setting for stroke. Public Health115(1),21–29 (2001).▪▪ Semi-structured interviews used to evaluate stakeholder engagement in a priority setting process with a discussion of areas of success and areas for improvement.Crossref, Medline, CAS, Google Scholar
101 O’Haire C, McPheeters M, Nakamoto E et al. Engaging stakeholders to identify and prioritize future research needs. Methods Future Research Needs Report No 4 (2011). www.effectivehealthcare.ahrq.gov/ehc/products/200/698/MFRNGuide04--Engaging_Stakeholders--6–10–2011.pdfGoogle Scholar
102 Preskill H, Jones N. A practical guide for engaging stakeholders in developing evaluation questions (2009). www.rwjf.org/pr/product.jsp?id=49951Google Scholar

Vol. 1, No. 5

Metrics

Downloaded 335 times

History

Published online 11 September 2012

Published in print September 2012

Information

Keywords

Financial & competing interests disclosure

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

No writing assistance was utilized in the production of this manuscript.

PDF download