Social Work Meta-analysis
by
Paul Montgomery, Matthew Morton
  • LAST REVIEWED: 28 April 2017
  • LAST MODIFIED: 14 December 2009
  • DOI: 10.1093/obo/9780195389678-0054

Introduction

This article organizes a wide range of selected resources for understanding and executing meta-analyses. Meta-analysis refers to “statistical techniques used to analyse and synthesise results of multiple studies on the same topic” (see Introductory Works, Littell, et al. 2008, pp. 176). Although meta-analysis encompasses the statistical aspects of research synthesis, the process is situated within other processes involving the selection and inclusion of studies as well as the reporting and interpretation of results. In particular meta-analysis should generally be conducted within the context of a systematic review in order to ensure that the process of gathering and appraising data for meta-analysis is as credible and transparent as possible. Therefore this bibliography also provides resources related to bias, assessing study quality, unpublished data, and other nonquantitative aspects of research synthesis that have direct implications for meta-analysis.

Introductory Works

This section highlights a few of the many articles and texts that have been written in recent decades in an effort to introduce the concept, principles, and procedures of meta-analysis to a wider audience. Together the British Medical Journal’s “Meta-Analysis: Education and Debate” series on meta-analysis (Egger, et al. 1998) may provide the most concise and Internet accessible overview introduction on the topic. The Littell, et al. 2008 pocket guide text should also be prioritized; it is the most up-to-date introductory text available on meta-analysis, is nontechnical, and helps readers understand how meta-analysis and systematic reviews are unique but complementary endeavors. The Cooper, et al. 2000 and Schmidt and Hunter 2003 chapters offer nonmathematical overviews of how meta-analysis has developed over time, controversies involved, and key principles and language regarding statistical procedures. These chapters are written for specific research areas (education and psychology, respectively) but would be useful for a more general audience. Greenhalgh 1997 helps guide nonexperts in making sense of meta-analytic studies and appraising the quality of the paper. Though targeted toward a medical audience, Chaturvedi, et al. 2007 and Zwahlen, et al. 2008 effectively illustrate key limitations argued of meta-analyses when not conducted or presented appropriately.

  • Chaturvedi, Nishi, Rudy Bilous, Rebecca Hardy, Giuseppe Remuzzi, Piero Ruggenenti, and Giancarlo C. Viberti. 2007. Misleading meta-analysis: A need to look beyond the headlines. Diabetic Medicine 24 (6): 587–591.

    DOI: 10.1111/j.1464-5491.2007.02139.xSave Citation »Export Citation »E-mail Citation »

    Using a meta-analytic example from medicine, the authors illustrate the limitations of meta-analysis when executed or presented inadequately and provide guidance for overcoming, or at least being transparent with, these issues.

    Find this resource:

    • Cooper, H. M., J. C. Valentine, and K. Charlton. 2000. The methodology of meta-analysis. In Contemporary Special Education Research: Syntheses of the Knowledge Base on Critical Instructional Issues. Edited by Russell Gersten, Ellen P. Schiller, and Sharon Vaughn, 263–281. Mahwah, NJ: Lawrence Erlbaum.

      Save Citation »Export Citation »E-mail Citation »

      This chapter provides a brief overview of the history of meta-analysis, how methods and approaches have developed over time, and key concepts for understanding meta-analysis.

      Find this resource:

      • Cwikel, Julie, Lynn Behar, and June Rabson-Hare. 2000. A comparison of a vote count and a meta-analysis review of intervention research with adult cancer patients. Research on Social Work Practice 10 1 (1): 139–158.

        Save Citation »Export Citation »E-mail Citation »

        Within a social work context, the authors explore the merits and limitations of meta-analysis as well as an alternative research synthesis procedure (vote counting) and compare the two approaches for social workers and social work researchers. A research example of a care intervention for adult cancer patients is used for demonstration.

        Find this resource:

        • Egger, Matthias, et al. 1998. Meta-Analysis: Education and Debate.

          Save Citation »Export Citation »E-mail Citation »

          During 1997–1998 the British Medical Journal printed a series of six articles, all written by Matthias Egger along with other colleagues, examining the procedures in conducting reliable meta-analysis in medical research. The articles are available free online and together provide a useful introductory guide through the major issues involved with meta-analysis. One complementary article also provides a review of meta-analysis software packages, but given the rapid advances of computer technology, this review is now largely outdated.

          Find this resource:

          • Greenhalgh, Trisha. 1997. How to read a paper: Papers that summarise other papers (systematic reviews and meta-analyses). British Medical Journal 315: 672–675.

            Save Citation »Export Citation »E-mail Citation »

            The How to Read a Paper series of ten articles is geared toward helping nonexperts appraise the quality of studies—in this case, systematic reviews and meta-analyses.

            Find this resource:

            • Littell, Julia H., Jacqueline Corcoran, and Vijayan Pillai. 2008. Systematic reviews and meta-analysis. Oxford: Oxford Univ. Press.

              DOI: 10.1093/acprof:oso/9780195326543.001.0001Save Citation »Export Citation »E-mail Citation »

              This pocket guide text provides a step-by-step introduction to the process of research synthesis using systematic review and meta-analysis that employs specific social work examples and is geared toward students and researchers alike.

              Find this resource:

              • Lundahl, Brad, and Joanne Yaffe. 2007. Use of meta-analysis in social work and allied disciplines. Journal of Social Service Research 33:1–11.

                DOI: 10.1300/J079v33n03_01Save Citation »Export Citation »E-mail Citation »

                This study analyzes publication patterns of meta-analyses across five disciplines—family studies, nursing, psychology, psychiatry, and social work—and finds that in all disciplines meta-analyses are rising, though social work journals have done much less publishing and commenting on meta-analyses. Implications and recommendations are discussed for improving effective use of meta-analysis in the social work context.

                Find this resource:

                • Schmidt, Frank L., and John E. Hunter. 2003. Meta-analysis. In Handbook of Psychology. Vol. 2: Research Methods in Psychology. Editor in chief Irving B. Weiner, volume editors John A. Schinka and Wayne F. Velicer, 533–554. New York: Wiley.

                  Save Citation »Export Citation »E-mail Citation »

                  Provides a succinct overview of the purpose and methods behind meta-analysis and discusses the application of meta-analysis to the field of psychology.

                  Find this resource:

                  • Zwahlen, Marcel, Andrew Renehan, and Matthias Egger. 2008. Meta-analysis in medical research: Potentials and limitations. Urologic Oncology: Seminars and Original Investigations 26 (3): 320–329.

                    DOI: 10.1016/j.urolonc.2006.12.001Save Citation »Export Citation »E-mail Citation »

                    Though geared toward a medical audience, this article provides an updated overview of common forms of bias and quality issues that can compromise the value and validity of meta-analyses. The authors argue that meta-analyses should generally be conducted in the context of a systematic review to try to avoid these limitations as much as possible.

                    Find this resource:

                    Textbooks

                    The number of comprehensive, textbook-style books on meta-analysis that would be useful in teaching undergraduate or graduate courses is still relatively meager. Lipsey and Wilson 2001 reads more like a practical guide than a textbook but would provide a useful overview for students and serves as a common reference for researchers conducting meta-analyses. Sutton, et al. 2000 is a useful text for helping students delve further into the statistical procedures and considerations involved with meta-analysis. Egger, et al. 2001 and Hunter and Schmidt 2004 help acquaint readers with the historical background of systematic reviews.

                    • Egger, Matthias, George Davey Smith, and Douglas Altman, eds. 2001. Systematic reviews in health care: Meta-analysis in context. London: BMJ.

                      DOI: 10.1002/9780470693926Save Citation »Export Citation »E-mail Citation »

                      This text helps acquaint readers with the terminology and history involved with systematic reviews and approaches for reducing or eliminating bias. The book offers little technical support for the statistical methods of meta-analysis.

                      Find this resource:

                      • Hunter, John E., and Frank L. Schmidt. 2004. Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.

                        Save Citation »Export Citation »E-mail Citation »

                        Offers statistical procedures that would be the preferred approach for researchers in organizational or industrial psychology. For researchers in other fields, it provides an overview of meta-analysis history and concepts as well as an alternative perspective for meta-analytic statistical methods.

                        Find this resource:

                        • Lipsey, Mark W., and David B. Wilson. 2001. Practical meta-analysis. Thousand Oaks, CA: Sage.

                          Save Citation »Export Citation »E-mail Citation »

                          Provides a practical overview of the stages of meta-analyses and an introduction to meta-analytic statistical procedures that is accessible to readers with a low technical knowledge of statistics.

                          Find this resource:

                          • Sutton, Alex J., Keith R. Abrams, David R. Jones, Trevor Sheldon, and Fujian Song. 2000. Methods for meta-analysis in medical research. Chichester, UK: Wiley.

                            Save Citation »Export Citation »E-mail Citation »

                            Using a nontechnical approach, this book is helpful for understanding the statistical issues involved with meta-analysis. Examples are rooted in medical research, but the concepts would be useful and accessible to anyone interested in conducting or reading meta-analyses.

                            Find this resource:

                            Manuals and Guides

                            Manuals and guides will provide step-by-step instructions supporting researchers through the process (or part of the process) of conducting meta-analysis. It is advisable for anyone interested in conducting a Cochrane or Campbell systematic review (or a high-quality meta-analysis or systematic review) to become familiar with the Cochrane Handbook for Systematic Reviews of Interventions (Higgins and Green 2008), which is frequently updated, is available for free online, and provides a comprehensive reference for conducting a review and meta-analysis. The Littell, et al. 2008 text is consistent with the Cochrane Handbook for Systematic Reviews of Interventions (the authors have themselves produced several Cochrane and Campbell reviews) but provides a more condensed and readable reference to keep on hand. Each chapter of Cooper and Hedges 1994 is on a different aspect of the meta-analysis process. Lipsey and Wilson 2001 does the same but is also accessible to readers with a low technical knowledge of statistics. Streiner 2003 offers a brief and simplified step-by-step introductory guide to conducting meta-analysis.

                            • Cooper, Harris, and Larry V. Hedges, eds. 1994. The handbook of research synthesis. New York: Russell Sage Foundation.

                              Save Citation »Export Citation »E-mail Citation »

                              This book is a widely used resource on research synthesis. Chapters are written by experts in different aspects of the meta-analytic process, but the same data sets are used across chapters for consistency.

                              Find this resource:

                              • Higgins, Julian, and Sally Green, eds. 2008. Cochrane handbook for systematic reviews of interventions. Oxford: Cochrane Collaboration.

                                DOI: 10.1002/9780470712184Save Citation »Export Citation »E-mail Citation »

                                The Cochrane Handbook for Systematic Reviews of Interventions provides a detailed, step-by-step description of the process of preparing and maintaining a Cochrane systematic review on the effects of health-care interventions. While aimed at Cochrane reviewers, the handbook is an advisable reference for anyone interested in conducting a high-quality systematic review or meta-analysis.

                                Find this resource:

                                • Lipsey, Mark W., and David B. Wilson. 2001. Practical meta-analysis. Thousand Oaks, CA: Sage.

                                  Save Citation »Export Citation »E-mail Citation »

                                  This book provides a practical overview of the stages of meta-analyses and an introduction to meta-analytic statistical procedures that is accessible to readers with a low technical knowledge of statistics.

                                  Find this resource:

                                  • Littell, Julia H., Jacqueline Corcoran, and Vijayan Pillai. 2008. Systematic reviews and meta-analysis. Oxford: Oxford Univ. Press.

                                    DOI: 10.1093/acprof:oso/9780195326543.001.0001Save Citation »Export Citation »E-mail Citation »

                                    This pocket guide text provides a step-by-step introduction to the process of research synthesis using systematic review and meta-analysis that uses specific social work examples and is geared toward students and researchers alike.

                                    Find this resource:

                                    • Streiner, David L. 2003. Meta-analysis: A 12-step program. Electronic Journal of Gambling Issues 9:1–23

                                      Save Citation »Export Citation »E-mail Citation »

                                      For readers interested in a quick and simplified, step-by-step summary guide through the process of conducting a meta-analysis, this article provides an easy introduction and highlights key aspects of meta-analysis beyond aggregating numbers (that is, how to properly identify studies and minimize opportunities for bias). The article is available free online.

                                      Find this resource:

                                      Reference Resources

                                      The following online resources provide assistance, tools, and examples for students, researchers, and decision makers interested in meta-analysis. The Campbell Collaboration and Cochrane Collaboration websites constitute the foremost databases for high-quality systematic reviews and meta-analyses. Campbell prepares, maintains, and disseminates systematic reviews in education, crime and justice, and social welfare. Cochrane, having been established earlier than Campbell, has a larger database of both systematic reviews and primary trials, which pertain broadly to research on health and medical interventions. The Cochrane Handbook for Systematic Reviews of Interventions also provides detailed, step-by-step guidance in conducting a systematic review and meta-analysis. The Equator Network, like Comprehensive Meta-Analysis—a commercial site—has various useful resources on the different areas of meta-analysis. Psychwiki.com offers a more simplified online resource that introduces a novice audience to basic concepts and resources for meta-analysis. The Lyons 1997–2003 website deals mostly with a statistical introduction to meta-analysis using one common statistical approach.

                                      Bibliographies

                                      There are apparently no reliable online bibliographies that address a wide range of meta-analytic issues and remain regularly updated; however, Lipsey and Wilson 2001 and Moher, et al. 1995 provide a number of useful resources for interested readers.

                                      • Lipsey, Mark W., and David B. Wilson. 2001. Practical meta-analysis. Thousand Oaks, CA: Sage.

                                        Save Citation »Export Citation »E-mail Citation »

                                        Page 233 provides a recommended reading list that consists of most of the standard texts on meta-analyses.

                                        Find this resource:

                                        • Moher, David., Alejandro R. Jadad, Graham Nichol, Marie Penman, Peter Tugwell, and Sharon Walsh. 1995. Assessing the quality of randomized controlled trials: An annotated bibliography of scales and checklists. Controlled Clinical Trials 16 (1): 62–73.

                                          DOI: 10.1016/0197-2456(94)00031-WSave Citation »Export Citation »E-mail Citation »

                                          This article cites and reviews scales and checklists for assessing the quality of randomized controlled trials developed by 1995.

                                          Find this resource:

                                          Software

                                          Software programs have been developed for conducting statistical functions and meta-analyses electronically. Such programs allow users to combine and analyze data and often enable researchers to produce graphs, charts, and models with their data. Many commonly used statistical software programs in social science, such as SAS, Statistical Package for the Social Sciences (SPSS), and Stata, support basic statistical functions for the purposes of meta-analysis. Some macros programs have been developed to cater to more advanced statistical functions and graphing needs that can work with the aforementioned general statistical software packages. Comprehensive Meta-Analysis and Rosenberg, et al. both present commercial software programs that generally require purchasing but can offer attractive features and benefits for researchers interested in conducting high-quality meta-analysis. Review Manager 5, a program designed for the Cochrane Collaboration, may be appropriate for meta-analysis and other systematic review uses, but notably the software can only be accessed for accepted research and educational purposes. Wang and Bushman 1999 and Wilson both provide free resources online to aid meta-analysis using several common general statistical and data software programs. Finally, Littell, et al. 2008 offers a recent summary of software programs for conducting systematic reviews and meta-analysis currently in print.

                                          • Comprehensive Meta-Analysis. Biostat.

                                            Save Citation »Export Citation »E-mail Citation »

                                            A commercial software program compatible with Windows, Comprehensive Meta-Analysis provides a comprehensive, well-developed software package for meta-analysis. The software provides sophisticated features and options for entering data, computing effect sizes, assessing heterogeneity, and producing funnel and forest plots, among other things.

                                            Find this resource:

                                            • Littell, Julia H., Jacqueline Corcoran, and Vijayan Pillai. 2008. Systematic reviews and meta-analysis. Oxford: Oxford Univ. Press.

                                              DOI: 10.1093/acprof:oso/9780195326543.001.0001Save Citation »Export Citation »E-mail Citation »

                                              Appendix B (page 143) provides references and summaries for software programs for conducting systematic reviews and meta-analysis.

                                              Find this resource:

                                              • Lunn, D. J., A. Thomas, N. Best, and D. Spiegelhalter. 2000. WinBUGS: A Bayesian modelling framework; Concepts, structure, and extensibility. Statistics and Computing 10:325–337.

                                                DOI: 10.1023/A:1008929526011Save Citation »Export Citation »E-mail Citation »

                                                The WinBUGS/BUGS (Bayesian Inference Using Gibbs Sampling) website provides programs for researchers interested in performing Bayesian analyses with complex statistical models for meta-analysis. The software is free online.

                                                Find this resource:

                                                • Review Manager 5 (RevMan 5). Copenhagen, Denmark: Nordic Cochrane Centre, Cochrane Collaboration.

                                                  Save Citation »Export Citation »E-mail Citation »

                                                  Review Manager 5 (RevMan 5) is the software used for conducting multiple stages of Cochrane systematic reviews, including meta-analysis. It works on Windows, Mac OS X, and Linux. RevMan 5 is free for preparing Cochrane and Campbell reviews and other accepted purely academic purposes. Others using Review Manager 5 for commercial purposes will need to purchase a license. The software calculates odds ratios, risk ratios, weighted mean differences, and standardized mean differences. It can fit both random- and fixed-effect models, provides Q and I2 heterogeneity tests, and provides forest and funnel plots.

                                                  Find this resource:

                                                  • Rosenberg, Michael S., Dean C. Adams, and Jessica Gurevitch. MetaWin. Sinauer.

                                                    Save Citation »Export Citation »E-mail Citation »

                                                    A commercial software program compatible with Windows, MetaWin allows for computing various effect sizes, performing heterogeneity analyses, displaying graphs and plots, and conducting subgroup analyses.

                                                    Find this resource:

                                                    • Wang, Morgan C., and Brad J. Bushman. 1999. Integrating results through meta-analytic review using SAS software. Cary, NC: SAS Institute.

                                                      Save Citation »Export Citation »E-mail Citation »

                                                      This text guides investigators in using SAS software to perform statistical functions for meta-analysis.

                                                      Find this resource:

                                                      • Wilson, David B. Meta-Analysis Stuff.

                                                        Save Citation »Export Citation »E-mail Citation »

                                                        Wilson’s website provides macros programs that work with Excel, SAS, Statistical Package for the Social Sciences (SPSS), and Stata that can calculate effect sizes, perform various analyses, perform a Q test for heterogeneity, and perform moderator analyses. The files are free online.

                                                        Find this resource:

                                                        Origins of Meta-Analyses

                                                        Meta-analysis constituted a major development in research methods during the last three decades of the 20th century and the early 21st century. The discourse and application of meta-analysis generated a number of approaches, discoveries, and controversies during that time. Many texts have made major contributions to this discourse, but of the early literature only Glass 1976 and Glass, et al. 1981 are referenced here, as Glass was the first to coin the term “meta-analysis” and in many ways ignited the modern developments in research synthesis (though the first meta-analysis is believed to have been conducted by Karl Pearson in 1904). Nevertheless, Glassian approaches to research synthesis have always been contentious, and alternative approaches have taken root. Hunt 1997 provides the most comprehensive historical review to date on research synthesis, and Chalmers, et al. 2002 offers a briefer overview of meta-analytic developments across multiple disciplines.

                                                        • Chalmers, Iain, Larry V. Hedges, and Harris Cooper. 2002. A brief history of research synthesis. Evaluation and the Health Professions 25 (1): 12–37.

                                                          DOI: 10.1177/0163278702025001003Save Citation »Export Citation »E-mail Citation »

                                                          Provides a relatively concise account of the development of meta-analysis across multiple disciplines and more recent meta-analytic developments aimed at reducing bias.

                                                          Find this resource:

                                                          • Glass, Gene V. 1976. Primary, secondary, and meta-analysis of research. Educational Researcher 5 (10): 3–8.

                                                            Save Citation »Export Citation »E-mail Citation »

                                                            Geared toward education research. In this article Glass coined the term “meta-analysis” and advanced early procedures for conducting meta-analysis.

                                                            Find this resource:

                                                            • Glass, Gene V., Barry McGaw, and Mary Lee Smith. 1981. Meta-analysis in social research. Beverly Hills, CA: Sage.

                                                              Save Citation »Export Citation »E-mail Citation »

                                                              This book was the earliest overview text on the subject of meta-analysis. It does not reflect the advances and some of the common practices in meta-analysis today, but it constitutes an important contribution to the early development of the meta-analytic field.

                                                              Find this resource:

                                                              • Hunt, Morton. 1997. How science takes stock: The story of meta-analysis. New York: Russell Sage Foundation.

                                                                Save Citation »Export Citation »E-mail Citation »

                                                                Discusses how meta-analysis emerged as a field, major contributors, and controversies around meta-analysis.

                                                                Find this resource:

                                                                Information Retrieval Strategies

                                                                The result of a meta-analysis can only be as good as the studies it includes. For this reason the line “garbage in, garbage out” has become a popular expression encapsulating the importance of strong information retrieval strategies for meta-analyses. The Cochrane Handbook for Systematic Reviews of Interventions (Higgins and Green 2008) and Littell, et al. 2008 both provide guidance aimed at helping researchers and practitioners conduct high-quality systematic reviews from which meta-analyses may be best produced. These texts include explicit information for preparing and conducting far-reaching search strategies. Johnson and Eagly 2000 is a good summary text that introduces readers to key issues to consider when retrieving studies. The University of Illinois at Chicago reference is one of several websites that take researchers and practitioners through the critical steps of forming and executing a professional search and the screening process for studies to be included in reviews and meta-analyses.

                                                                • Higgins, Julian, and Sally Green, eds. 2008. Cochrane handbook for systematic reviews of interventions. Oxford: Cochrane Collaboration.

                                                                  DOI: 10.1002/9780470712184Save Citation »Export Citation »E-mail Citation »

                                                                  The Cochrane Handbook for Systematic Reviews of Interventions provides explicit guidance for how to search and retrieve studies for high-quality systematic reviews, which would generally provide the best basis of research evidence for synthesis in a meta-analysis.

                                                                  Find this resource:

                                                                  • Johnson, Blair T., and Alice H. Eagly. 2000. Quantitative synthesis of social psychological research. In Handbook of Research Methods in Social and Personality Psychology. Edited by Harry T. Reis and Charles M. Judd, 496–528. London: Cambridge Univ. Press.

                                                                    Save Citation »Export Citation »E-mail Citation »

                                                                    This chapter introduces readers to common procedures for locating and screening relevant studies and provides illustrative examples in various sections. The authors walk readers through the concepts of searching online databases, why and when to include unpublished studies, contacting other researchers, examining the reference lists in other reviews, and otherwise executing a comprehensive search strategy.

                                                                    Find this resource:

                                                                    • Littell, Julia H., Jacqueline Corcoran, and Vijayan Pillai. 2008. Systematic reviews and meta-analysis. Oxford: Oxford Univ. Press.

                                                                      DOI: 10.1093/acprof:oso/9780195326543.001.0001Save Citation »Export Citation »E-mail Citation »

                                                                      This user-friendly pocket guide dedicates a chapter specifically to locating and screening studies for systematic reviews and meta-analysis.

                                                                      Find this resource:

                                                                      • University of Illinois at Chicago. Evidence Based Medicine: Finding the Best Clinical Literature.

                                                                        Save Citation »Export Citation »E-mail Citation »

                                                                        This online library resource walks researchers through the basic steps of developing evidence-based research, including formulating a proper research question, conducting an online search, navigating databases, and appraising studies. Though specific to a medical audience, the basic tenants hold true for researchers and practitioners in social work looking to conduct a professional search for studies to include in meta-analysis.

                                                                        Find this resource:

                                                                        Appraising Meta-Analyses

                                                                        Jørgensen, et al. 2006 is included because it represents a growing body of evidence suggesting substantial differences in findings produced by meta-analyses conducted under different standards. The QUOROM Statement developed in Moher, et al. 1999 has become a key guide in the field for sufficiently reporting meta-analyses; its recommendations are helpful guides for understanding what to look for in a meta-analytic study and critical information by which meta-analyses should be appraised. The Centre for Evidence Based Medicine (CEBM) and Public Health Resource Unit (PHRU) websites provide professionally developed, user-friendly tools for critically appraising several kinds of studies, including systematic reviews and experimental and quasi-experimental trials.

                                                                        • Centre for Evidence Based Medicine (CEBM).

                                                                          Save Citation »Export Citation »E-mail Citation »

                                                                          The Centre for Evidence Based Medicine provides short, user-friendly forms to aid critical appraisal of systematic reviews and randomized controlled trials, among other things.

                                                                          Find this resource:

                                                                          • Jørgensen, Anders W., Jørgen Hilden, and Peter C. Gøtzsche. 2006. Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: Systematic review. British Medical Journal 333 (7572): 782–785.

                                                                            DOI: 10.1136/bmj.38973.444699.0BSave Citation »Export Citation »E-mail Citation »

                                                                            Examines twenty-four meta-analyses conducted in the context of a Cochrane Collaboration systematic review and matching meta-analyses that were conducted outside of Cochrane. Investigators found significantly stronger quality among meta-analyses that were conducted in Cochrane reviews. Findings support the advances to situate meta-analyses in the context of well-conducted, highly sensitive, and transparent systematic reviews of relevant studies.

                                                                            Find this resource:

                                                                            • Moher, David, A. R. Jadad, and T. P. Klassen. 1998. Guides for reading and interpreting systematic reviews III: How did the authors synthesize the data and make their conclusions? Archives of Pediatrics and Adolescent Medicine 152 (9): 915–920.

                                                                              Save Citation »Export Citation »E-mail Citation »

                                                                              Offers a brief guide for nonexperts to key concepts for understanding and appraising the meta-analytic aspect of a systematic review.

                                                                              Find this resource:

                                                                              • Moher, David, Deborah J. Cook, Susan Eastwood, Ingram Olkin, Drummond Rennie, and Donna F. Stroup. 1999. Improving the Quality of Reports of Meta-Analyses of Randomised Controlled Trials: The QUOROM Statement. Lancet 354 (9193): 1896–1900.

                                                                                DOI: 10.1016/S0140-6736(99)04149-5Save Citation »Export Citation »E-mail Citation »

                                                                                Responding to a widespread problem of poor presentation of meta-analyses, a meeting of clinicians, epidemiologists, researchers, editors, and statisticians resulted in this QUOROM Statement, which provides detailed and explicit guidance to those reporting meta-analyses of randomized controlled trials. This statement can also be used as a guide for consumers of meta-analyses in appraising quality and reporting of methods and findings.

                                                                                Find this resource:

                                                                                • Public Health Resource Unit (PHRU), National Health Service. Appraisal Tools.

                                                                                  Save Citation »Export Citation »E-mail Citation »

                                                                                  This National Health Service website provides several downloadable user-friendly forms to aid critical appraisal of systematic reviews, randomized controlled trials, and quasi-experimental trials, among other things.

                                                                                  Find this resource:

                                                                                  Unpublished Data

                                                                                  One important thematic issue in the development of meta-analysis involves the question of what kinds of studies should be included and how that choice influences the findings of meta-analyses. Among the most prevalent themes in this discourse is whether to include unpublished data. The tendency for studies sharing certain trends (for example, null or negative effects) to go unpublished has become popularly known as the “file drawer problem” for meta-analyses. Rothstein, et al. 2005 is the most comprehensive text on this issue, though Cook, et al. 1993 and McAuley, et al. 2000 are two among the many studies empirically demonstrating how commonly unpublished studies are excluded from meta-analyses and further how that exclusion tends to generate exaggerated findings regarding the effectiveness of interventions.

                                                                                  • Cook, Deborah J., Gordon H. Guyatt, Gerard Ryan, Joanne Clifton, Lisa Buckingham, Andrew Willan, William McIlroy, and Andrew D. Oxman. 1993. Should unpublished data be included in meta-analyses? Current convictions and controversies. Journal of the American Medical Association 269 (21): 2749–2753.

                                                                                    DOI: 10.1001/jama.269.21.2749Save Citation »Export Citation »E-mail Citation »

                                                                                    This paper discusses the debates about using gray literature in meta-analysis and surveys relevant investigators and journal editors to garner views on the use of unpublished data from the field.

                                                                                    Find this resource:

                                                                                    • McAuley, Laura, Ba Pham, P. Tugwell, and David Moher. 2000. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 356 (9237): 1228–1231.

                                                                                      DOI: 10.1016/S0140-6736(00)02786-0Save Citation »Export Citation »E-mail Citation »

                                                                                      Findings from this empirical study of meta-analyses indicate that exclusion of unpublished studies can lead to exaggerated statements of intervention effectiveness.

                                                                                      Find this resource:

                                                                                      • Rothstein, Hannah R., Alexander J. Sutton, and Michael Bronstein, eds. 2005. Publication bias in meta-analysis: prevention, assessment, and adjustments. Chichester, UK: Wiley.

                                                                                        DOI: 10.1002/0470870168Save Citation »Export Citation »E-mail Citation »

                                                                                        The chapters of this book provide the most comprehensive, if dry, overview of the history and issues involved with publication bias and meta-analysis as well as the latest approaches for identifying and avoiding the influences of publication bias on meta-analytic findings.

                                                                                        Find this resource:

                                                                                        Quality of Data

                                                                                        The quality of the findings from a meta-analysis is only so good as the quality of the studies it includes. The Moja, et al. 2005 study provides updated evidence on the state of accounting for study quality in meta-analyses and systematic reviews. The Delphi List (Verhagen, et al. 1998) offers a brief list of critical criteria by which study quality for randomized controlled trials should be considered in research synthesis. The CONSORT and TREND statements (Altman, et al. 2001; des Jarlais, et al. 2004) are carefully developed consensus statements among research experts for reporting the methods and results of trials that provide guides by which one can assess study quality. Mayo-Wilson 2007 adds to these statements by emphasizing the reporting of trial implementation. Herbison, et al. 2006 and Jüni, et al. 2001 caution against summary scores of study quality given the lack of validity found in existing tools, but they nonetheless urge for an assessment of study quality in meta-analyses and recommend the exploration of improved ways for doing this quantitatively.

                                                                                        • Altman, Douglas G., Kenneth F. Schulz, David Moher, Matthias Egger, Frank Davidoff, Diana Elbourne, Peter C. Gøtzsche, and Thomas Lang. 2001. The revised CONSORT Statement for reporting randomized trials: Explanation and elaboration. Annals of Internal Medicine 134 (8): 663–694.

                                                                                          Save Citation »Export Citation »E-mail Citation »

                                                                                          The CONSORT Statement synthesizes a substantial evidence base in order to produce a minimum set of recommendations for reporting randomized controlled trials. The twenty-two-item checklist can serve as a useful tool for conductors of meta-analyses in appraising the studies considered for inclusion.

                                                                                          Find this resource:

                                                                                          • Des Jarlais, Don C., Cynthia Lyles, Nicole Crepaz, and TrendGroup. 2004. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND Statement. American Journal of Public Health 94 (3): 361–366.

                                                                                            DOI: 10.2105/AJPH.94.3.361Save Citation »Export Citation »E-mail Citation »

                                                                                            The TREND Statement is a similar resource to the CONSORT Statement, but TREND provides a well-established guide for properly reporting and appraising nonrandomized trials rather than randomized trials.

                                                                                            Find this resource:

                                                                                            • Herbison, Peter, Jean Hay-Smith, and William J. Gillespie. 2006. Adjustment of meta-analyses on the basis of quality scores should be abandoned. Journal of Clinical Epidemiology 59 (12): 1249–1256.

                                                                                              Save Citation »Export Citation »E-mail Citation »

                                                                                              Based on an empirical investigation of systematic reviews and meta-analyses, the investigators found existing quality scores to be unhelpful. The authors recommend improved quantitative methods for judging the quality of studies included in meta-analyses.

                                                                                              Find this resource:

                                                                                              • Jüni, Peter, Douglas G. Altman, and Matthias Egger. 2001. Systematic reviews in health care: Assessing the quality of controlled clinical trials. British Medical Journal 323 (7303): 42–46.

                                                                                                DOI: 10.1136/bmj.323.7303.42Save Citation »Export Citation »E-mail Citation »

                                                                                                This paper helps prioritize critical aspects of judging study quality, though it cautions against producing summary scores from quality scales to rank studies.

                                                                                                Find this resource:

                                                                                                • Mayo-Wilson, Evan. 2007. Reporting implementation in randomized trials: Proposed Additions to the consolidated standards of reporting trials statement. American Journal of Public Health 97 (4): 630–633.

                                                                                                  DOI: 10.2105/AJPH.2006.094169Save Citation »Export Citation »E-mail Citation »

                                                                                                  This article adds to previous literature on the discussion of study quality and reporting by emphasizing trial implementation considerations.

                                                                                                  Find this resource:

                                                                                                  • Moja, Lorenzo P., Elena Telaro, Roberto D’Amico, Ivan Moschetti, Laura Coe, and Alessandro Liberati. 2005. Assessment of methodological quality of primary studies by systematic reviews: Results of the metaquality cross sectional study. British Medical Journal 330 (7499): 1053–1058.

                                                                                                    DOI: 10.1136/bmj.38414.515938.8FSave Citation »Export Citation »E-mail Citation »

                                                                                                    This study reviewed 956 systematic reviews and found that Cochrane reviews fared better than others in assessing the quality of trials, but Cochrane and non-Cochrane reviews alike largely failed in accounting for study quality in the interpretation of results. The article provides updated references and further guidance for research.

                                                                                                    Find this resource:

                                                                                                    • Verhagen, Arianne P., Henrica C. W. de Vet, Robert A. de Bie, Alphons G. H. Kessels, Maarten Boers, Lex M. Bouter, and Paul G. Knipschild. 1998. The Delphi List: A criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. Journal of Clinical Epidemiology 51 (12): 1235–1241.

                                                                                                      DOI: 10.1016/S0895-4356(98)00131-0Save Citation »Export Citation »E-mail Citation »

                                                                                                      The Delphi List constitutes a consensus statement among multiple experts that recommends brief criteria for assessing the quality of randomized controlled trials for the purpose of conducting systematic reviews and meta-analyses.

                                                                                                      Find this resource:

                                                                                                      Heterogeneity and Subgroup Analysis

                                                                                                      One common criticism of and challenge for meta-analysis is combining the results of trials with study designs or interventions that have considerable differences from each other. Much of this discourse revolves around differences between study quality, for which references are provided. These references can be helpful in delving further into the issues of heterogeneity and subgroup analysis, but note that a better introduction to the subject in the context of meta-analyses is available in the major overview texts presented in the Textbooks and Manuals and Guides sections. Counsell, et al. 1994 and Smith, et al. 1997 warn against certain types of subgroup analysis, while Higgins, et al. 2003 proposes a way to improve measures of consistency between trials. Patsopoulos, et al. 2008 seeks to account for heterogeneity between studies.

                                                                                                      • Counsell, C. E., M. J. Clarke, J. Slattery, and P. A. G. Sandercock. 1994. The miracle of DICE therapy for acute stroke: Fact or fictional product of subgroup analysis? British Medical Journal 309 (6970): 1677–1681.

                                                                                                        Save Citation »Export Citation »E-mail Citation »

                                                                                                        Provides a facetious illustration of how excessive and inappropriate subgroup analysis can lead one to draw conclusions from data by overmanipulating data and playing on the effects of chance. Could be a good illustrative teaching paper.

                                                                                                        Find this resource:

                                                                                                        • Higgins, Julian P. T., Simon G. Thompson, Jonathan J. Deeks, and Douglas G. Altman. 2003. Measuring inconsistency in meta-analyses. British Medical Journal 327 (7414): 557–560.

                                                                                                          DOI: 10.1136/bmj.327.7414.557Save Citation »Export Citation »E-mail Citation »

                                                                                                          Higgins and colleagues explore the issue of heterogeneity of effects between studies with meta-analyses as well as subgrouping, and the authors propose a new quantity, I2, as an improved measure of consistency between trials.

                                                                                                          Find this resource:

                                                                                                          • Patsopoulos, Nikolaos A., Evangelos Evangelou, and John P. A. Ioannidis. 2008. Sensitivity of between-study heterogeneity in meta-analysis: proposed metrics and empirical evaluation. International Journal of Epidemiology 37 (5): 1148–1157.

                                                                                                            DOI: 10.1093/ije/dyn065Save Citation »Export Citation »E-mail Citation »

                                                                                                            This paper aims to work toward a standardized approach to sensitivity analysis that can be used to provide different models in order to account for heterogeneity between studies.

                                                                                                            Find this resource:

                                                                                                            • Smith, George Davey, Matthias Egger, and Andrew N. Phillips. 1997. Meta-analysis: Beyond the grand mean? British Medical Journal 315 (7122): 1610–1614.

                                                                                                              Save Citation »Export Citation »E-mail Citation »

                                                                                                              Addresses the use and misuse of subgroup analysis in meta-analysis and briefly discusses alternative methods preferred by the authors for assessing differences in treatment effects.

                                                                                                              Find this resource:

                                                                                                              Bias

                                                                                                              Many forms of bias can influence the results of a study and skew the conclusions regarding an intervention’s effectiveness. There is a significant amount of literature on the subject of bias; this section only highlights a small amount of this literature, particularly as it pertains to meta-analysis. For a concise overview of the most common kinds of bias, readers should consider Delgado-Rodriguez and Llorca 2004a good starting place. Egger, et al. 1997 presents evidence of bias in meta-analyses, thus justifying further exploration of the issue, and Egger and Smith 1998 briefly introduces readers to major forms of bias they may confront in the process of locating and selecting studies for meta-analysis. Jüni, et al. 2002 and Williamson, et al. 2005 focus on specific kinds of bias (publication bias and outcome selection bias, respectively) that may have special relevance to meta-analysis. Littell 2008 helps readers identify a broader range of bias specifically relevant to research synthesis.

                                                                                                              • Delgado-Rodriguez, Miguel and Javier Llorca. 2004. Bias. Journal of Epidemiology and Community Health 58 (8): 635–641.

                                                                                                                DOI: 10.1136/jech.2003.008466Save Citation »Export Citation »E-mail Citation »

                                                                                                                Describes the most common forms of bias across a range of stages in the research process.

                                                                                                                Find this resource:

                                                                                                                • Egger, Matthias, and George Davey Smith. 1998. Meta-analysis: Bias in location and selection of studies. British Medical Journal 316 (7124): 61–66.

                                                                                                                  Save Citation »Export Citation »E-mail Citation »

                                                                                                                  Calls special attention to major forms of publication bias, location bias in finding studies (for example, biases that arise from language limitations, databases used, and citations), and bias from inclusion criteria for study searches, all of which can significantly alter the results and credibility of a meta-analysis.

                                                                                                                  Find this resource:

                                                                                                                  • Egger, Matthias, George Davey Smith, Martin Schneider, and Christoph Minder. 1997. Bias in meta-analysis detected by a simple, graphical test. British Medical Journal 315 (7109): 629–634.

                                                                                                                    Save Citation »Export Citation »E-mail Citation »

                                                                                                                    This paper investigates and illustrates empirical evidence of how publication bias and other related bias can skew the results of meta-analyses.

                                                                                                                    Find this resource:

                                                                                                                    • Jüni, Peter, Franziska Holenstein, Jonathan Sterne, C. Bartlett, and Matthias Egger. 2002. Direction and impact of language bias in meta-analyses of controlled trials: Empirical study. International Journal of Epidemiology 31 (1): 115–123.

                                                                                                                      DOI: 10.1093/ije/31.1.115Save Citation »Export Citation »E-mail Citation »

                                                                                                                      The paper deals with the issue of language bias and investigates whether including only English-language studies in meta-analysis appears to influence the overall results.

                                                                                                                      Find this resource:

                                                                                                                      • Littell, Julia. 2008. Evidence-based or biased? The quality of published reviews of evidence-based practices. Children and Youth Services Review 30 (11): 1299–1317.

                                                                                                                        DOI: 10.1016/j.childyouth.2008.04.001Save Citation »Export Citation »E-mail Citation »

                                                                                                                        This paper identifies substantial opportunities for various forms of bias in unscientifically developed reviews. The criteria by which Littell assesses bias may also be helpful for guiding readers and conductors of meta-analyses in knowing what to look for.

                                                                                                                        Find this resource:

                                                                                                                        • Williamson, Paula R., Carol Gamble, Douglas G. Altman, and Jane L. Hutton. 2005. Outcome selection bias in meta-analysis. Statistical Methods in Medical Research 14 (5): 515–524.

                                                                                                                          DOI: 10.1191/0962280205sm415oaSave Citation »Export Citation »E-mail Citation »

                                                                                                                          This article goes a step further from the discourse on publication bias and focuses on the prevalence of and dealing with the issue of selective reporting bias in which some outcomes from studies are preferentially reported while others are not.

                                                                                                                          Find this resource:

                                                                                                                          Statistical Issues and Procedures

                                                                                                                          As the core function of meta-analysis involves quantitatively combining data, there has been a wide range of statistical developments and texts presented on the subject. This section provides a starting place for key papers and books on statistical issues concerning meta-analysis. Overview literature referenced in the Textbooks and Manuals and Guides sections will provide an introduction to statistical issues and procedures often sufficient to many readers’ needs. The literature here, however, focuses almost exclusively on the statistical aspects of meta-analysis and allows for further grappling with the quantitative elements of research synthesis. The main contemporary texts, depending on the numerate level of the reader, include Gaver, et al. 1992; Hedges and Olkin 1985; Stangl and Berry 2000; and Whitehead 2002. Hedges and Olkin 1985 was an early but authoritative contribution to the meta-analytic field, and while many updates and developments have occurred since its publication, it is still a common reference. Morris and DeShon 2002 provides support for statistically dealing with multiple types of experiment designs included in one meta-analysis. Becker, et al. 2004; Gøtzsche, et al. 2007; Hedges and Pigott 2001; and Hunter and Schmidt 2000 all address specific issues that will be of interest to many students, researchers, and readers of meta-analyses.

                                                                                                                          • Becker, Betsy Jane, Larry V. Hedges, and Therese D. Pigott. 2004. Campbell Collaboration Statistical Analysis Policy Brief. Olso, Norway: Campbell Collaboration.

                                                                                                                            Save Citation »Export Citation »E-mail Citation »

                                                                                                                            Identifies key statistical issues that arise in meta-analyses for Campbell Collaboration systematic reviews and provides policy guides for reviewers (not intended as a teaching text). Available free on the Campbell Collaboration website.

                                                                                                                            Find this resource:

                                                                                                                            • Gaver, Donald. P., David Draper, Prem K. Goel, Joel B. Greenhouse, Larry V. Hedges, Carl N. Morris, and Christine Waternaux. 1992. Combining information: Statistical issues and opportunities for research. Washington, D.C.: National Academy Press.

                                                                                                                              Save Citation »Export Citation »E-mail Citation »

                                                                                                                              Provides a cross-disciplinary overview of quantitative issues and proposed statistical approaches for conducting meta-analysis.

                                                                                                                              Find this resource:

                                                                                                                              • Gøtzsche, Peter C., Asbjørn Hróbjartsson, Katja Marić, and Britta Tendal. 2007. Data extraction errors in meta-analyses that use standardized mean differences. Journal of the American Medical Association 298 (4): 430–437.

                                                                                                                                DOI: 10.1001/jama.298.4.430Save Citation »Export Citation »E-mail Citation »

                                                                                                                                Examines the challenges of combining outcomes from studies that use similar but not the same measures and thus commonly use standardized mean differences (SMDs) as a proxy measure for combining data.

                                                                                                                                Find this resource:

                                                                                                                                • Hedges, Larry V., and Ingram Olkin. 1985. Statistical methods for meta-analysis. Orlando, FL: Academic.

                                                                                                                                  Save Citation »Export Citation »E-mail Citation »

                                                                                                                                  Like The Handbook of Research Synthesis (Cooper and Hedges 1994; see Manuals and Guides), this book became a key text on the topic of meta-analysis. It focuses more exclusively on the meta-analysis process within research synthesis and teaches relevant statistical methods. Hedges and Olkin differed from previous literature on meta-analysis by, among other things, proposing that chi-square statistical tests be used to determine the extent to which sampling error explains variability between study outcomes in a meta-analysis.

                                                                                                                                  Find this resource:

                                                                                                                                  • Hedges, Larry V., and Therese D. Pigott. 2001. The power of statistical tests in meta-analysis. Psychological Methods 6 (3): 203–217.

                                                                                                                                    DOI: 10.1037/1082-989X.6.3.203Save Citation »Export Citation »E-mail Citation »

                                                                                                                                    As stated in the abstract, “the authors describe procedures to compute statistical power of fixed- and random-effects tests of the mean effect size, tests for heterogeneity (or variation) of effect size parameters across studies, and tests for contrasts among effect sizes of different studies” (pp. 203).

                                                                                                                                    Find this resource:

                                                                                                                                    • Hunter, John E., and Frank L. Schmidt. 2000. Fixed effects vs. random effects meta-analysis models: Implications for cumulative research knowledge. International Journal of Selection and Assessment 8 (4): 275–292.

                                                                                                                                      DOI: 10.1111/1468-2389.00156Save Citation »Export Citation »E-mail Citation »

                                                                                                                                      While fixed-effect methods are commonly used in meta-analyses, this article argues that random-effects methods are less likely to produce type 1 errors and too narrow confidence intervals.

                                                                                                                                      Find this resource:

                                                                                                                                      • Morris, Scott B., and Richard P. DeShon. 2002. Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods 7 (1): 105–125.

                                                                                                                                        DOI: 10.1037/1082-989X.7.1.105Save Citation »Export Citation »E-mail Citation »

                                                                                                                                        This paper is unique in that it provides guidance specifically with respect to using proper statistical procedures when including various types of study designs from included experiments in a meta-analysis—a rather common issue.

                                                                                                                                        Find this resource:

                                                                                                                                        • Stangl, Dalene K., and Donald A. Berry, eds. 2000. Meta-analysis in medicine and health policy. New York: Marcel Dekker.

                                                                                                                                          Save Citation »Export Citation »E-mail Citation »

                                                                                                                                          This text assumes a relatively high level of statistical understanding for several chapters and thus would be particularly helpful for researchers using more advanced meta-analytic statistical methods.

                                                                                                                                          Find this resource:

                                                                                                                                          • Whitehead, Anne. 2002. Meta-analysis of controlled clinical trials. Chichester, UK: Wiley.

                                                                                                                                            Save Citation »Export Citation »E-mail Citation »

                                                                                                                                            Addresses the mathematical aspects of meta-analyses, covering a range of issues for readers interested in either simple or advanced technical issues.

                                                                                                                                            Find this resource:

                                                                                                                                            Teaching

                                                                                                                                            As Dodd 2000 points out, because contemporary meta-analysis is a relatively recent research development engendered during the 1980s and 1990s, many faculty have had little formal training themselves in meta-analysis, let alone feel comfortable with teaching it to others. The following references may be helpful guides or resources for faculty or presenters looking to teach meta-analysis. Teachers unfamiliar with meta-analysis will want to refer to some of the selections included in Introductory Works, Textbook, and Manuals and Guides. Barnes and Levin 2006 reviews the necessary content for teaching meta-analysis to nursing students. Read Cumming 2006 if you want to incorporate software programs into teaching. The lesson plans in Dodd 2000 have been evaluated by students. The Evidence Based Social Intervention website of the Department of Social Policy and Social Work at the University of Oxford has materials for graduate-level training.

                                                                                                                                            • Barnes, Michael J., and Roña F. Levin. 2006. Teaching meta-analysis: Summarizing quantitative research. In Teaching Evidence-based Practice in Nursing. Edited by Roña F. Levin and Harriet R. Feldman, 99–120 New York: Springer.

                                                                                                                                              Save Citation »Export Citation »E-mail Citation »

                                                                                                                                              Reviews the necessary content for teaching the concepts and procedures of meta-analysis in an introductory fashion to nursing students or other largely nonquantitative course audiences. The chapter also provides guidance specifically for teaching the most statistically necessary aspects of meta-analysis to a nontechnical student audience and provides model narratives as a supportive tool.

                                                                                                                                              Find this resource:

                                                                                                                                              • Cumming, Geoff. Meta-analysis: Pictures that explain how experimental findings can be integrated. Paper presented at the Seventh International Conference on Teaching Statistics, Salvador, Brazil, 2006.

                                                                                                                                                Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                This presentation argues for including meta-analysis in introductory statistical textbooks and suggests ways to use software programs to help teach meta-analysis.

                                                                                                                                                Find this resource:

                                                                                                                                                • Dodd, David K. 2000. Teaching meta-analysis in the undergraduate experimental psychology course. Teaching of Psychology 27 (1): 54–57.

                                                                                                                                                  Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                  Provides teachers with a student-evaluated ninety-minute lesson plan for teaching meta-analysis at an undergraduate level along with references and examples to use in class and recommended introductory texts for faculty who would like to develop a stronger familiarity with meta-analysis themselves before teaching the subject.

                                                                                                                                                  Find this resource:

                                                                                                                                                  • Evidence Based Social Intervention (EBSI). Department of Social Policy and Social Work, Univ. of Oxford.

                                                                                                                                                    Save Citation »Export Citation »E-mail Citation »

                                                                                                                                                    The Evidence Based Social Intervention website provides information for those interested in graduate-level training with heavy emphasis on research synthesis and other methods supporting evidence-based intervention and research as well as references to various publications and resources on research synthesis and evidence-based intervention.

                                                                                                                                                    Find this resource:

                                                                                                                                                    back to top

                                                                                                                                                    Article

                                                                                                                                                    Up

                                                                                                                                                    Down