Replication is one key component toward a robust cumulative knowledge base. It plays a critical function in assessing the stability of the scientific literature. Replication involves closely repeating the procedure of a study and determining if the results are similar to the original. For decades, behavioral scientists were reluctant to publish replications. Reasons were epistemic and pragmatic. First of all, original studies were viewed as conclusive in most cases, and failures to replicate were often attributed to mistakes by the replicating researcher. In addition, failures to replicate may be caused by numerous factors. This inherent ambiguity made replications less desirable to journals. On the other hand, replication successes were expected and considered to contribute little beyond what was already known. Finally, editorial policies did not encourage the publication of replications, leaving the robustness of scientific findings largely unreported. A series of events ultimately led the research community to reconsider replication and research practices at large: the discovery of several cases of large-scale scientific misconduct (i.e., fraud); the invention and application of new statistical tools to assess strength of evidence; high-profile publications suggesting that some common practices may be less robust than previously assumed; failure to replicate some major findings of the field; and the creation of new, online tools aimed to promote transparency in the field. To deal with what is often regarded as the crisis of confidence, initiatives have been developed to increase the transparency of research practices, including (but not limited to) preregistration of studies; effect size predictions and sample size/power estimation; and, of course, replications. Replication projects themselves evolved in quality: from replications that were originally as small in sample as problematically small original studies to large-scale “Many Labs” collaborative projects. Ultimately, the development of higher-quality replication projects and open science tools has led (and will continue to lead) to a clearer understanding of human behavior and cognition and have contributed to a clearer distinction between exploratory and confirmatory behavioral science. The current article gives an overview of the history of replications, of the development of tools and guidelines, and of review papers discussing theoretical implications of replications.
A Historical Perspective on Replication in Psychological Science
Smith 1970 alerted the community to the lack of replication practices in psychology, and reminded the community of the importance of replication for cumulative science. More than forty years later, Makel, et al. 2012 reiterates this concern, showing that from 1900 to 2012, only 1.07 percent of the published research were replications. Scholars may have been reluctant to conduct and publish replications because of the difficulty in interpreting replication failures as well as the community’s preference for clear and easy patterns/conclusions from data. Giner-Sorolla 2012 for example describes how the preference for positive (i.e., significant) results and well-written narratives may have introduced new publication bias and decreased reproducibility. Greenwald 1975 explains how a bias against publishing null results may discourage researchers from pursuing research involving null hypotheses, resulting in an unrepresentative literature (e.g., a greater proportion of papers that erroneously reject the null). A publication in 2016 of a “replication” of Stapel and Semin 2007 made clear, however, that replications were often completed, yet went unpublished (IJzerman, et al. 2015). Nosek, et al. 2012 argues that the struggle between innovation and accumulation in science led scholars to neglect close replication practices. Preferences for clear patterns from data and difficulty interpreting replication data manifested in editorial policies, which, according to Neuliep and Crandall 1990, encourage the publication of new findings over replications.
Giner-Sorolla, R. 2012. Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science 7.6: 562–571.
This article criticizes scientific journals for favoring aesthetics of writing and for “clean” results. The authors show that emphasizing “clean” results over and beyond theoretical reasoning and methodological rigor leads scientists to favor the submission—and thus the publication—of flawed results.
Greenwald, A. G. 1975. Consequences of prejudice against the null hypothesis. Psychological Bulletin 82.1: 1–20.
The author surveyed seventy-five social psychologists about their research and publication practices relating to null and/or positive results. Using these responses as input parameters, the author presents a mathematical model simulating the outcome of the research process and argues that publication of null results is necessary for a healthy scientific literature. The author argues that scholars should evaluate adequacy of procedure and importance of findings, rather than emphasizing only significant results. Available online.
IJzerman, H., N. F. E. Regenberg, J. Saddlemyer, and S. L. Koole. 2015. Perceptual effects of linguistic category priming: The Stapel and Semin (2007) paradigm revisited in twelve experiments. Acta Psychologica 157:23–29.
This paper reports twelve nonsignificant studies on “linguistic category priming” that were originally thought to be replications of studies in Stapel and Semin 2007, originally published in the Journal of Social and Personality Psychology. Even up until 2013 (after Stapel was caught for mass data fabrication and the original was retracted), Journal of Personality and Social Psychology did not accept these studies, as they were seen as replications.
Makel, M. C., J. A. Plucker, and B. Hegarty. 2012. Replications in psychology research: How often do they really occur? Perspectives on Psychological Science 7.6: 537–542.
This paper investigates how many replications were conducted in psychology from 1900 to 2012. The authors searched for papers containing the word stem “replicat,” finding that only 1.6 percent of papers used terms like “replicate” or “replication.” Further examination revealed only ~1.07 percent were actual replications. Most of these published replications were successful.
Neuliep, J. W., and R. Crandall. 1990. Editorial bias against replication research. Journal of Social Behavior and Personality 5.4: 85–90.
This article surveyed the opinions of journal editors in social and behavioral science. Globally, results indicate that editorial boards’ reluctance to publish replications may account for the lack of published replications before 1990.
Nosek, B. A., J. R. Spies, and M. Motyl. 2012. Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science 7.6: 615–631.
This paper illustrates how publication practices favored novelty and positive results over replication and stability. The authors suggest a disconnect exists between scientific ideals and scientists’ interests, and this contributes to the current replication crisis. The authors propose strategies for better aligning research incentives with discovering truth.
Smith, N. 1970. Replication studies: A neglected aspect of psychological research. American Psychologist 25.10: 970–975.
This paper discusses the research community’s lack of interest in replication and discusses reasons why researchers do not publish replications. Available online.
Stapel, D. A., and G. R. Semin. 2007. The magic spell of language: Linguistic categories and their perceptual consequences. Journal of Personality and Social Psychology 93.1: 23–33.
This article contains four studies showing that the level of language abstraction influences perceptual focus. The Levelt committee proved that these findings were based on fraudulent data created by the first author. Eventually dozens of papers authored by Stapel would be retracted. Available online. Retraction published in 2013, Journal of Personality and Social Psychology 104.1: 197.
Users without a subscription are not able to see the full content on this page. Please subscribe or login.
How to Subscribe
Oxford Bibliographies Online is available by subscription and perpetual access to institutions. For more information or to contact an Oxford Sales Representative click here.
- Abnormal Psychology
- Action Research
- Addictive Behavior
- Affective Forecasting
- Allport, Gordon
- Alzheimer’s Disease
- Animal Behavior
- Animal Learning
- Anxiety Disorders
- Art and Aesthetics, Psychology of
- Attachment in Social and Emotional Development across the ...
- Attention-Deficit/Hyperactivity Disorder (ADHD) In Adults
- Attention-Deficit/Hyperactivity Disorder (ADHD) in Childre...
- Attraction in Close Relationships
- Attribution Theory
- Authoritarian Personality
- Behavior Therapy, Rational Emotive
- Behavioral Economics
- Behavioral Genetics
- Bereavement and Grief
- Biological Psychology
- Birth Order
- Body Image in Men and Women
- Bystander Effect
- Clinical Neuropsychology
- Clinical Psychology
- Cognitive Consistency Theories
- Cognitive Dissonance Theory
- Cognitive Neuroscience
- Communication, Nonverbal Cues and
- Comparative Psychology
- Conformity, Compliance, and Obedience
- Coping Processes
- Counseling Psychology
- Critical Thinking
- Cross-Cultural Psychology
- Cultural Psychology
- Death and Dying
- Deceiving and Detecting Deceit
- Defensive Processes
- Depressive Disorders
- Development, Prenatal
- Developmental Psychology (Cognitive)
- Developmental Psychology (Social)
- Diagnostic and Statistical Manual of Mental Disorders (DSM...
- Dissociative Disorders
- Drugs and Behavior
- Eating Disorders
- Ecological Psychology
- Educational Settings, Assessment of Thinking in
- Embodiment and Embodied Cognition
- Emerging Adulthood
- Emotional Intelligence
- Environmental Neuroscience and Environmental Psychology
- Ethics in Psychological Practice
- Evolutionary Psychology
- Exploratory Data Analysis
- Eyewitness Testimony
- Factor Analysis
- Festinger, Leon
- Five-Factor Model of Personality
- Flynn Effect, The
- Friendships, Children's
- Fundamental Attribution Error/Correspondence Bias
- Gambler's Fallacy
- Game Theory and Psychology
- Geropsychology, Clinical
- Health Psychology
- Heuristics and Biases
- History of Psychology
- Human Factors
- Implicit Association Test (IAT)
- Industrial and Organizational Psychology
- Intelligence, Crystallized and Fluid
- Intercultural Psychology
- Intergroup Conflict
- International Classification of Diseases and Related Healt...
- International Psychology
- Interviewing in Forensic Settings
- Intimate Partner Violence, Psychological Perspectives on
- Item Response Theory
- Law, Psychology and
- Learned Helplessness
- Learning versus Performance
- LGBTQ+ Romantic Relationships
- Lie Detection in a Forensic Context
- Life-Span Development
- Locus of Control
- Meaning in Life
- Mechanisms and Processes of Peer Contagion
- Media Violence, Psychological Perspectives on
- Memories, Autobiographical
- Memories, Flashbulb
- Memories, Repressed and Recovered
- Memory, Human
- Memory, Implicit versus Explicit
- Minnesota Multiphasic Personality Inventory (MMPI)
- Moral Development
- Moral Psychology
- Moral Reasoning
- Neuroscience of Associative Learning
- Obsessive-Complusive Disorder (OCD)
- Occupational Health Psychology
- Operant Conditioning
- Optimism and Pessimism
- Organizational Justice
- Parenting Stress
- Path Models
- Peace Psychology
- Perception, Person
- Performance Appraisal
- Personality Disorders
- Personality Psychology
- Phenomenological Psychology
- Placebo Effects in Psychology
- Positive Psychology
- Post-Traumatic Stress Disorder (PTSD)
- Prejudice and Stereotyping
- Prisoner's Dilemma
- Prosocial Behavior
- Prosocial Spending and Well-Being
- Psychology, Political
- Psychophysics, Visual
- Psychotic Disorders
- Reasoning, Counterfactual
- Rehabilitation Psychology
- Religion, Psychology and
- Replication Initiatives in Psychology
- Research Methods
- Risk Taking
- Schizophrenic Disorders
- School Psychology
- Self, Gender and
- Self, Psychology of the
- Self-Regulation in Educational Settings
- Sensation Seeking
- Sex and Gender
- Sexual Minority Parenting
- Sexual Orientation
- Signal Detection Theory and its Applications
- Single People
- Skinner, B.F.
- Sleep and Dreaming
- Small Groups
- Social Class and Social Status
- Social Cognition
- Social Neuroscience
- Social Support
- Social Touch and Massage Therapy Research
- Somatoform Disorders
- Spatial Attention
- Sports Psychology
- Stereotype Threat
- Stress and Coping, Psychology of
- Subjective Wellbeing Homeostasis
- Teaching of Psychology
- Terror Management Theory
- Testing and Assessment
- Theory of Mind
- Therapies, Person-Centered
- Therapy, Cognitive-Behavioral
- Thinking Skills in Educational Settings
- Time Perception
- Trait Perspective
- Twin Studies
- Type A Behavior Pattern (Coronary Prone Personality)
- Women and Science, Technology, Engineering, and Math (STEM...
- Women, Psychology of