Real World Evaluation Strategies

by

Valéry Ridde, Émilie Robert

• LAST REVIEWED: 25 November 2014
• LAST MODIFIED: 25 November 2014
• DOI: 10.1093/obo/9780199756797-0140

Introduction

Public health interventions are by nature complex and are implemented in contexts and with social actors who have a mutual influence on each other. It thus becomes nearly impossible, and more or less irrelevant, to control the interventions or to find direct causal relationships. It follows that most public health interventions are natural experiments for which appropriate methodological fit should be the gold standard. Therefore, researchers should adopt a pragmatic, non dogmatic position; be responsive; and adapt to the natural environment of the real world (as there is no such thing as a laboratory setting for public health interventions). Because of their complexity, we need to go beyond just knowing the interventions’ effects; we also want to know how those effects are produced, for whom, and under what circumstances. Methods should be selected that allow us to understand the inherent complexity of public health interventions. The social sciences have contributed significant methodological suggestions to the field of evaluation of complex interventions. In this context, the war of paradigms is unproductive. However, taking into account real-life situations does not mean abandoning the use of theories, conceptual frameworks, or the intervention theories identified in the preevaluation phase (evaluability assessment). Moreover, the issues related to an evaluation often extend beyond just methods. Budget constraints need to be considered, as well as the usefulness of the responses to the questions and concerns in terms of effecting change in practices and interventions (see the Oxford Bibliographies article “Knowledge Translation and Exchange”). Beyond studying effects, it also becomes essential to examine interventions’ implementation (fidelity and adaptation) and operationalization, as well as their relevance, appropriateness, sustainability, and acceptability. Thus the evaluation questions and concerns should guide the methods, and not the reverse. To the extent possible, we will present all these consubstantial aspects of a pragmatic and real-world evaluation approach from a conceptual perspective and then illustrate their application. (The authors wish to thank Donna Riley for translation and editing support.)

Journals

Generally speaking, interdisciplinary journals in the public health field publish evaluations, including those that meet the criteria described in this article. For this reason, those journals are not listed here. Notable exceptions are Global Health Promotion, which focuses specifically on the themes of population health and health promotion, and the Milbank Quarterly, which does not hesitate to offer evaluations using innovative methods, such as realistic evaluation. The American Journal of Evaluation and Evaluation are devoted to advancing practices and knowledge about evaluation and represent streams of thought from both sides of the Atlantic. The Journal of Mixed Methods Research and Qualitative Health Research also publish real-world evaluations, but from a methodological perspective, which is their focus. Finally, given that the aim of real-world evaluation is also to respond to questions arising from the field, Implementation Science is useful for understanding issues concerning the use of knowledge coming out of evaluation and research in the health field.

• American Journal of Evaluation. 1981–.

  Formerly Evaluation Practice (1981–), this journal is known for having hosted a debate between Carol H. Weiss and Michael Quinn Patton on the use of evaluation. The interdisciplinary journal aims to advance both theoretical and practical knowledge about evaluation.

• Evaluation. 1995–.

  This is a European journal devoted to evaluation in several fields, including health.

• Global Health Promotion. 1994–.

  Although it does not specialize in evaluation, this journal regularly publishes evaluations of programs in health promotion and population health and also provides more theoretical articles in this field. Articles may be in English, Spanish or French, and all supplements are open access.

• Implementation Science. 2006–.

  This open-access journal focuses on the use of research and evaluation in the health field. The journal contains not only empirical articles, but also articles that are conceptual or theoretical.

• Journal of Mixed Methods Research. 2007–.

  The primary aim of this journal is to advance knowledge and practice with respect to mixed-methods research in the social sciences, humanities, and health. For this reason, particular attention is given to empirical studies, particularly evaluations.

• Milbank Quarterly. 1923–.

  Formerly New York Health Demonstrations (1923–1930), the Milbank Memorial Fund Quarterly Bulletin (1931–1933), the Milbank Memorial Fund Quarterly (1934–1972), and the Milbank Memorial Fund Quarterly, Health and Society (1973–1985). This is a high-caliber journal that concentrates on different types of evaluations, as well as on theoretical and conceptual reflections on evaluation, in the fields of health and health policy.

• Qualitative Health Research.

  This is one of the few journals that have allowed qualitative research specialists to demonstrate the relevance of this approach in the field of health sciences, especially through program evaluations.