Protocol Analysis

by

Peter F. Delaney, Robert J. Wallander, Gretchen A. Preheim

• LAST REVIEWED: 28 August 2018
• LAST MODIFIED: 28 August 2018
• DOI: 10.1093/obo/9780199828340-0223

Introduction

Verbal protocol analysis is a set of systematic methods for collecting and analyzing peoples’ thoughts during a task. Verbal protocol analysis is primarily used to identify the strategies people use when completing a task when such strategies are not known, or to test theories of the processes people use to generate an outcome. Verbal protocols can be collected either concurrently (while doing the task) or retrospectively (after the fact). There are a standardized set of methods for collecting the reports that help to avoid altering the thought processes being observed and obtain valid reports instead of confabulation. While there are many other kinds of qualitative verbal data, including interviews, questionnaires, focus groups, asking people to explain what they know, and so on, these other methods are still verbal reports, but they are not usually referred to as verbal protocol analysis. Once they are collected, the protocols need to be segmented into meaningful units of analysis, and then these units are coded using a behavioral coding scheme. Finally, the data needs to be analyzed.

General Overviews and Basic Methods

Although protocol analysis is quite old, almost all contemporary research using verbal reports draws upon a Psychological Review paper, Ericsson and Simon 1980, and a book-length treatise on protocol analysis, Ericsson and Simon 1993. These works review the evidence supporting the idea that verbal reports could, if collected properly, provide valid data about what people are thinking and serve as the basis for theories of thinking. Their recommended procedures and instructions are available in Fox, et al. 2011. Ericsson and Simon proposed that verbal reports are not a full explanation of what people are doing, but rather a kind of behavioral data that needs to be explained (and thus akin to things like number of steps taken, observable behavior, and latency). Furthermore, they suggested that asking participants to “think aloud” by verbalizing what they were thinking—without explanation or embellishment—produced reliable and valid reports about the contents of thought (the think-aloud method). They called these “Level 1” verbalizations. In contrast, methods that encourage people to recode information from one modality to another (e.g., describing a visually processed face in words, or the experience of an emotion being converted into words) were called “Level 2” verbalizations. Anything that asked people to draw inferences or generate something were termed “Level 3” verbalizations. They also allowed for the use of retrospective verbal reports in which people are asked to recall in order all of the thoughts they had during the task, starting from the first thought. For a review of qualitative analysis methods other than verbal protocol analysis and their pitfalls, see Chi 1997. For an overview in French, see Roussel 2017, and for a Spanish-language introduction, see Requena 2003.

• Chi, M. T. H. 1997. Quantifying qualitative analyses of verbal data: A practical guide. Journal of the Learning Sciences 6:271–315.

  DOI: 10.1207/s15327809jls0603_1

  Outlines methods other than using the think-aloud or retrospective report methodology proposed by Ericsson and Simon, laying out possible problems and benefits of various qualitative methods.

• Ericsson, K. A., and H. A. Simon. 1980. Verbal reports as data. Psychological Review 87:215–251.

  DOI: 10.1037/0033-295X.87.3.215

  The original review of Ericsson and Simon’s theory of verbal reports as data.

• Ericsson, K. A., and H. A. Simon. 1993. Protocol analysis: Verbal reports as data. Rev. ed. Cambridge, MA: MIT Press.

  This landmark book provided a major review of the literature, evidence for when verbal reports altered performance, and an outline of methods for collecting and analyzing verbal reports data. Usually cited when using verbal reports as data. An earlier edition from 1984 is also available.

• Fox, M. C., K. A. Ericsson, and R. Best. 2011. Do procedures for verbal reports of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychological Bulletin 137:316–344.

  DOI: 10.1037/a0021663

  The recommended instructions along with warm-up tasks are provided in the supplementary materials.

• Requena, M. 2003. El análisis de protocolo como técnica para la comprensión de los procesos de razonamiento. Laurus 9:79–96.

  A Spanish-language review of protocol analysis.

• Roussel, K. 2017. Les protocoles verbaux (think-aloud protocols): Enjeux méthodologiques de validité pour la recherche en contexte scolaire. Revue Canadienne des Jeunes Chercheures et Chercheurs en Education 8.1 (Spring): 160–167.

  A French-language review of verbal protocol analysis. Available online.