Research Integrity in Public Health

by

Dennis M. Gorman

• LAST REVIEWED: 28 August 2018
• LAST MODIFIED: 28 August 2018
• DOI: 10.1093/obo/9780199756797-0189

Introduction

It is now recognized that a crisis of credibility exists in many academic disciplines, which manifests itself in the form of too many positive results and too many results that cannot be replicated or reproduced. In some academic disciplines such nonreproducible false discoveries are estimated to be a majority of what is published in the research literature. One of the major ways in which false discoveries enter the research literature is through the use of flexible data analysis. This can take the form of p-hacking a marginally significant result (i.e., one just above the 0.05 alpha level) into a statistically significant one (i.e., one just below the 0.05). The other main way that false discoveries are produced is through investigators conducting a myriad of analyses and reporting only those that produced statistically significant results in such a way that it appears these were the outcome of a priori hypothesis testing. In either case, investigators are free to engage in such practices and submit the results from these flexible analyses for review and publication in peer-reviewed journals, as neither the reviewer nor the reader have any idea how many and what types of analyses were originally planned, nor how these differed from those submitted for publication. This problem is exacerbated by the fact that academic journals are very reluctant to publish null or negative findings. The need to produce positive results for publication is at the heart of a dysfunction incentive system that exists within academia, which has created a hypercompetitive environment in which quantity of output, rather than quality and scientific contribution, is rewarded. Analytic flexibility and selective reporting are difficult to identify in the published literature, since investigators obviously do not admit to having used such practices to manufacture the statistically significant results they report in their published studies, nor do they admit to the severe limitations of null hypothesis statistical testing (NHST) using p-values that is standard practice in most disciplines. However, statistical techniques have been developed within the emerging field of meta-research to identify such practices, and alternatives to NHST have been proposed. In addition, statistical tests have been developed to try to estimate the prevalence of false discoveries within specific areas of research. Investigators from the meta-research community have also proposed various solutions to the crisis in academic credibility, focusing on the ways that data are collected, analyzed, reported, and shared with colleagues.

General Overviews

Alberts, et al. 2014 is a systems analysis of the crisis in biomedical science; the authors propose a set of solutions that require upstream planning and policy initiatives on the part of the federal government to correct the deep-seated structural flaws in the way scientific research and institutions are funded. This is a much more radical set of solutions to the problem of the deterioration in research quality and integrity than those generally proposed, which focus on downstream factors such as disciplinary norms and values or the policies and practices of peer-reviewed journals. Ioannidis, et al. 2015 is an overview of the emerging field of meta-research, which the authors define as the scientific study of research. They see this as a cross-cutting discipline that involves both theoretical and empirical investigations, and that takes a broad view of science and research. Institute of Medicine and National Research Council 2002 and National Academies of Sciences, Engineering, and Medicine 2017 are comprehensive overviews of what research integrity entails and what the main threats to it are in the current environment in which scientific research is funded, conducted, and reported. Both use a systems perspective to identify the leverage points at which threats to research integrity arise and interventions can be introduced. Both also make specific recommendations for improvements to the present system. Marusic, et al. 2016 is a Cochrane review of interventions to promote research integrity and reduce research misconduct. Thirty-one studies met the inclusion criteria. Of these, fifteen were randomized controlled trials. Many of the studies were judged to be at high risk of bias. The interventions were often poorly described, but they were mostly educational and training efforts intended to change the knowledge and attitudes of individuals. None of the interventions targeted organizational change. The edited volume Lilienfeld and Waldman 2017 contains seventeen chapters by some of the leading academics conducting research on issues related to research integrity. While the focus is on research conducted in the field of psychology, most of the issues discussed in these chapters (e.g., reproducibility of results, aversion to null findings, allegiance bias, and transparency) are relevant to public health research.

• Alberts, B., M. W. Kirschner, S. Tilghman, and H. Varmus. 2014. Rescuing US biomedical research from its systemic flaws. Proceedings of the National Academy of Science 111.16: 5773–5777.

  DOI: 10.1073/pnas.1404402111

  Presents a macrolevel analysis of the crisis in biomedical research, identifying the sources of the deterioration in quality in the hypercompetition among researchers that results from the imbalance between the number of individuals trained and seeking funding and the dramatic cuts in federal funding that have occurred since the 1990s. Biomedical research is presented as a system in constant disequilibrium and not subject to correction through traditional market forces.

• Institute of Medicine and National Research Council. 2002. Integrity in scientific research: Creating an environment that promotes responsible conduct. Washington, DC: National Academies Press.

  Chapter 3 of this report presents a dynamic open-systems model of research integrity that conceptualizes the behavior of individual researchers as influenced by the structures and norms of the institutions in which they work, which are, in turn, influenced by the inputs they receive (such as funding) and demand for their outputs (such as publications). Reviews institutional approaches to maintaining research integrity, and the role of self-assessment and education. Available for purchase or as a free PDF.

• Ioannidis, J. P. A., D. Fanelli, D. D. Dunne, and S. N. Goodman. 2015. Meta-research: Evaluation and improvement of research methods and practices. PLoS Biology 13.10: e1002264.

  DOI: 10.1371/journal.pbio.1002264

  Presents an overview of the discipline of meta-research that focuses on the methods, reporting, reproducibility, evaluation, and incentives of the scientific enterprise. Contains a list of thirty-four ongoing initiatives (with web addresses) within these five domains. Discusses the challenges to the growth of the discipline, including its fragmentary nature, lack of methodological cohesion, and limited funding opportunities.

• Lilienfeld, S. O., and I. Waldman. 2017. Psychological science under scrutiny: Recent challenges and proposed solutions. Hoboken, NJ: Wiley-Blackwell.

  DOI: 10.1002/9781119095910

  The most comprehensive text available on the reproducibility crisis within psychology and its causes. Includes contributions from leading scholars that discuss reproducibility, statistical power, problems with null hypothesis significance testing, Bayesian hypothesis testing, and the adverse influence of allegiance effects on clinical research. Problems that pervade specific research domains are reviewed, including social psychology, brain imaging research, and gene association research. Also includes a detailed discussion of blind data analysis.

• Marusic, A., E. Wager, A. Utrobicic, H. R. Rothstein, and D. Sambunjak. 2016. Interventions to prevent misconduct and promote integrity in research and publication. Cochrane Database of Systematic Reviews 4:MR000038.

  DOI: 10.1002/14651858.MR000038.pub2

  Using standard Cochrane methodological procedures, the authors identified thirty-one studies that evaluated interventions to reduce research misconduct (twenty-one focused on plagiarism and ten on ethics and integrity). Most interventions involved some form of training. The methodological quality of the studies was judged to be poor, and the data extracted could not be used in a formal meta-analysis. The authors conclude that the effects of training on research integrity are uncertain.

• National Academies of Sciences, Engineering, and Medicine. 2017. Fostering integrity in research. Washington, DC: National Academies Press.

  DOI: 10.17226/21896

  Adopts a perspective that views the research enterprise as a complex adaptive system of interacting individuals and organizations. Discusses recent trends that present threats to research integrity, including increased competition, conflicts of interest, and new forms of scholarly communication. Reviews the core values of research (e.g., objectivity) and examines the costs and consequences of both research misconduct and use of detrimental research practices. Concludes with specific recommendations for improvement. Available for purchase or as a free PDF.