Observational (Non-Randomized) Studies

by

Paul Rosenbaum

• LAST MODIFIED: 24 April 2023
• DOI: 10.1093/obo/9780199828340-0312

Introduction

In 1965, the statistician William G. Cochran defined an observational study as an attempt to draw inferences about the effects caused by a treatment, a policy, a program, an exposure, or an intervention in a context in which randomized experimentation is not possible for practical or ethical reasons. A “treatment” or “control” condition is something that could, in principle, be imposed or withheld; however, in most observational studies, there are overwhelming ethical or practical barriers to imposing the treatment on human subjects for experimental purposes. For instance, the treatment may be harmful, such as the sudden death of a spouse and its possible effects on depression, or a potentially traumatic event and its possible effects as a cause of post-traumatic stress syndrome. Alternatively, the treatment might be a prison sentence that is imposed by a court, in a typical context in which it is neither reasonable nor realistic to alter the prison sentence for research purposes. Observational studies became distinct from experiments when Sir Ronald Fisher invented randomized experimentation. In a randomized experiment, individuals are assigned to treatment or control by a truly random mechanism, perhaps by flips of a fair coin, or more commonly today by random numbers generated by a computer. Randomized treatment assignment ensures that there is no systematic reason that certain types of people receive treatment and other types receive control; it is simply luck. The benefits of randomized treatment assignment are absent in observational studies. Treated individuals and controls may differ systematically prior to treatment, so a difference in outcomes observed after treatment may not be an effect caused by the treatment. Some pretreatment differences are visible in data; others are not. Treated individuals might typically be somewhat older than controls, so treated individuals might be matched to controls of the same age to remove or adjust for that difference in age. There are many methods of adjusting for measured covariates, and matching is the simplest of these. Treated and control individuals may also differ prior to treatment in terms of covariates that were not measured, and it is not possible to match or adjust for an unmeasured covariate. Most of the controversy that commonly attends an observational study consists of debate about possible biases from the failure to adjust for some unmeasured covariate. Most of the creative effort in designing a successful observational study focuses on addressing possible biases from unmeasured covariates.

Overviews

In the year following the publication of the US Surgeon General’s Report Smoking and Health, a member of the panel that wrote the report, the statistician William G. Cochran, wrote a survey of concepts and methods for observational studies. The survey Cochran 1965 was published with extensive discussion, and it remains an excellent place to begin reading about observational studies. Since the 1960s, causal inference has developed as a shared subfield of several disciplines that study the effects of treatments on human beings, including economics as discussed by Angrist and Krueger 1999, epidemiology and public health as discussed by Hernán and Robins 2020, medicine as discussed by Vandenbroucke 2004, and the behavioral sciences as discussed by Shadish, et al. 2002. Imbens and Rubin 2015, Rosenbaum 2017, and Rutter 2007 provided interdisciplinary surveys of causal inference in observational studies.

• Angrist, J. D., and A. B. Krueger. 1999. Empirical strategies in labor economics. In Handbook of labor economics. Vol. 3. By J. D. Angrist and A. B. Krueger, 1277–1366. Amsterdam: Elsevier.

  DOI: 10.1016/S1573-4463(99)03004-7

  This is a survey of methods for observational studies in labor economics; however, the same methods are applicable in many fields. Owing to the nature of economics, economists are particularly concerned about the possibility that people inflict upon themselves the treatments they rationally think will have desired benefits; so, economists are particularly keen to find settings in which people have little control over the treatments they receive.

• Cochran, W. G. 1965. The planning of observational studies of human populations (with Discussion). Journal of the Royal Statistical Society A 128.2: 234–266.

  DOI: 10.2307/2344179

  This article, by an author of the 1964 US Surgeon General’s Report Smoking and Health, was the first to define a subfield of statistics devoted to methodology for the design and analysis of observational studies. In addition to giving structure to the main tasks that arise in observational studies, the section entitled “The Step from Association to Causation” remains one of the best discussions of this critical step.

• Hernán, M. A., and J. M. Robins. 2020. Causal inference. Boca Raton, FL: Chapman & Hall/CRC.

  This is an engaging, modern introduction to causal inference from the perspective of two leaders in epidemiological methodology. Particularly careful is the discussion of treatments given over a period of time, where early responses to treatment may affect subsequent doses or discontinuation of treatment. Preliminary drafts of the book were distributed as samizdat, so the book was influential and well known in advance of its formal publication.

• Imbens, G. W., and D. B. Rubin. 2015. Causal inference in statistics, social, and biomedical sciences. New York: Cambridge Univ. Press.

  DOI: 10.1017/CBO9781139025751

  This book by economist Guido Imbens and statistician Donald B. Rubin summarizes and extends their many contributions to causal inference. The book discusses the potential outcomes framework for causal inference, the nature and role of instruments or instrumental variables in noncompliance, and applications of natural experiments and quasi-experimental reasoning in economics.

• Rosenbaum, P. R. 2017. Observation and experiment: An introduction to causal inference. Cambridge, MA: Harvard Univ. Press.

  DOI: 10.4159/978067498269

  This book is a thorough but nontechnical introduction to causal inference, carefully developing the parallels between randomized experiments and observational studies. It presents in nontechnical terms a number of technical ideas—potential outcomes, propensity scores, sensitivity analyses, amplification of sensitivity analyses, design sensitivity, differential effects and generic biases, and instruments.

• Rutter, M., ed. 2007. Identifying the environmental causes of disease: How should we decide what to believe and when to take action? London: Academy of Medical Sciences.

  Written by a committee, this free volume surveys conceptual issues. Most interesting and unique is chapter 6, which reviews twenty major areas that have been examined using observational studies, judging ten areas to have reached correct conclusions, six areas with misleading conclusions, and four for which the jury is still out. The connection between smoking and lung cancer is judged correct, and harm from the MMR vaccine is judged misleading.

• Shadish, W., T. D. Cook, and D. T. Campbell. 2002. Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.

  This is the most recent of three books about quasi-experimental techniques by Donald T. Campbell and colleagues. Bigger and more up to date than its two predecessors, it also places more emphasis—as its title suggests—on the ability to generalize a causal inference from one setting or population to another.

• Vandenbroucke, J. P. 2004. When are observational studies as credible as randomised trials? The Lancet 363.9422: 1728–1731.

  DOI: 10.1016/S0140-6736(04)16261-2

  This article provides an overview and taxonomy of observational studies in medicine. It recalls several causal conclusions derived from observational studies that were subsequently overturned by randomized clinical trials. It considers when and why observational studies are more or less likely to err.