Meta-analysis in Criminology

by

Travis Pratt

• LAST REVIEWED: 25 May 2011
• LAST MODIFIED: 25 May 2011
• DOI: 10.1093/obo/9780195396607-0101

Introduction

The empirical status of the research on any particular topic in criminology is usually established through the traditional narrative review of the research literature. In this method, studies are described discursively and afforded relative importance determined by the reviewer. These reviews can be especially useful since, if focused on the details of the studies, they may yield special insights and illuminate new pathways for future research. Such narrative reviews do, however, depend on the judgment of their authors to tell the reader what the research shows. Furthermore, when the research findings are mixed—with some studies finding significant results while others do not—there can be a legitimate dispute over what the body of research “really says.” In such cases, a range of biases may creep into the discussion and influence the lens through which the body of research is perceived. Indeed, conflicting interpretations of the existing empirical evidence on a given research question are not uncommon, and even consistent interpretations could conceivably be attributable to similar biases and misreadings of the literature. Finally, the variability in study outcomes makes it difficult to falsify a given thesis or theory. Thus, there is the potential for ideas to linger on longer than they should, not because they have scientific merit, but rather because the imprecise way in which knowledge is organized in a narrative review leaves the door open for bad ideas still to claim empirical legitimacy if studies supporting those ideas can still be located. As an alternative to the traditional narrative review, meta-analysis entails the quantitative synthesis of empirical literature. Meta-analysis attempts to integrate the findings of multiple independent tests of a similar hypothesis in a more objective manner by treating the empirical study as the unit of analysis. Researchers may then draw inferences based on the effect size (or predictive capacity) of relationships between variables. What follows here is an overview of the major sources for meta-analysis within criminology.

General Overviews

Much of the published work related to meta-analysis is focused on the more technical aspects of the various statistical methods that have been developed. While these works are quite useful, it is equally important to understand the broader—and perhaps more critical—debates concerning various issues associated with the method. In short, there is a risk that if we get too hyperfocused on the technical nuances of this method, then larger considerations may end up getting lost, particularly with regard to those issues most relevant to criminologists faced with the task of systematically organizing the empirical research in a given area. Accordingly, the works listed below are intended to provide more general assessments of meta-analysis for criminologists. These works include broad-based “how-to” guides such as Borenstein, et al. 2009 and Cooper 1998; historical/contextual overviews, as in Hunt 1999 and Pratt 2002; discussions of the larger implications of the technique in Hunter and Schmidt 1996; and an analysis of the “state of the field” of meta-analysis in criminology, in Wells 2009.

• Borenstein, Michael, Larry V. Hedges, Julian P. T. Higgins, and Hannah R. Rothstein. 2009. Introduction to meta-analysis. New York: Wiley.

  DOI: 10.1002/9780470743386

  Provides the most current broad overview of the various uses and forms of meta-analysis. Although this work is not necessarily targeted at criminologists—or even at social scientists—the book itself serves as a solid set of guidelines for conducting a meta-analysis.

• Cooper, Harrison. 1998. Synthesizing research: A guide for literature reviews. 3d ed. Thousand Oaks, CA: SAGE.

  Technically a bit deeper than Borenstein, et al. 2009, Cooper’s book provides more detail on the appropriateness of various meta-analysis techniques under varying methodological conditions. Like most SAGE volumes, it is readable and user-friendly.

• Hunt, Morton M. 1999. How science takes stock: The story of meta-analysis. New York: Russell Sage Foundation.

  This work is largely a historical overview of where the technique of meta-analysis came from and how it has developed into the various forms of it that we see today. It reads more like a narrative than almost any other work on the subject and provides a level of insider knowledge as to how the technique developed that any reader will likely find engaging.

• Hunter, John E., and Frank L. Schmidt. 1996. Cumulative research knowledge and social policy formulation: The critical role of meta-analysis. Psychology, Public Policy, and Law 2:324–347.

  DOI: 10.1037/1076-8971.2.2.324

  This piece highlights one of the more important developments in the application of meta-analytic research: its ability to inform policy decisions. Hunter and Schmidt do a fine job of detailing how researchers can shape their meta-analyses in ways that will maximize their impact in the policy arena.

• Pratt, Travis C. 2002. Meta-analysis and its discontents: Treatment destruction techniques revisited. Journal of Offender Rehabilitation 35:23–40.

  Taking a page from Gottfredson’s 1979 “Treatment destruction techniques” article in the Journal of Research in Crime and Delinquency, Pratt’s article is a tongue-in-cheek account of how critics of meta-analysis have often relied on faulty arguments and sketchy logic.

• Wells, Edward. 2009. Uses of meta-analysis in criminal justice research: A quantitative review. Justice Quarterly 26:268–294.

  DOI: 10.1080/07418820802119984

  Wells provides the most up-to-date assessment of the state of the field of meta-analysis in criminology and criminal justice research. This work is essentially a survey of the various kinds of meta-analyses that have appeared in the criminological literature to date, with additional analyses of current trends in the discipline.