Approaches to Multimodal Discourse Analysis

by

Jesse Pirini

• LAST REVIEWED: 27 June 2017
• LAST MODIFIED: 27 June 2017
• DOI: 10.1093/obo/9780199756841-0183

Introduction

Culture and society are produced through interactions between people, objects, and environments. Within these interactions it has become clear that the modes of spoken and written language are only some of a diverse range of modes involved in producing meaning and experience. As topics of study, the modes of spoken and written language have been joined by modes like gesture, gaze, composition, and layout. Multimodal discourse analysis names a range of approaches to studying social interaction and meaning as multimodal, that is, produced with and through multiple modes. However, multimodal discourse analysis is not about identifying and studying modes as isolated but rather about understanding the world as multimodal. This understanding is developed through theoretical and methodological developments. Three major theoretical bases are in use in multimodal discourse analysis: (a) Hallidayan systemic functional linguistics (SFL), underpinning a social semiotic and SFL approach; (b) mediated discourse analysis, underpinning a mediated action based approach; and (c) conversation analysis (CA), underpinning a turn-taking based approach. The simple idea of identifying modes beyond language belies the growing complexity of research in this area. The multimodal literature contains significantly different definitions of mode and different foci. For example, systemic functional approaches began by analyzing how meaning is embedded within images and artifacts, and these studies have been extended at times to include studies of interaction. Mediated discourse analysis focuses primarily on interaction and understands images and artifacts through how social actors interact with them, rather than seeking to decode possible meanings from the perspective of the analyst. Conversation analytic approaches also focus on interactions and only engage with images and artifacts as they appear in interaction. All three theoretical bases develop from studies of language. Researchers in CA still often argue that talk remains the most important mode, while social semiotics and mediated discourse analysis dispute this. Multimodal (inter)action analysis builds upon mediated discourse analysis but is the only approach designed specifically for the study of multimodal interaction and multimodal action. The ongoing development of frameworks within multimodal discourse analysis is indicative of intellectual diversity. For some scholars, this is a source of concern, and they attempt to unify multimodality, while for others it is desirable, especially for a research domain that offers applications to so many areas of social life. This bibliography explores central texts under the umbrella of multimodal discourse analysis in five domains: mediated discourse analysis, multimodal (inter)action analysis, social semiotics, systemic functional multimodal discourse analysis (SF-MDA), and conversation analytic style multimodality. A sixth area focusing on introductory texts begins the bibliography.

Introductory Texts

Multimodal discourse analysis defines a diverse range of approaches for studying how social actors produce meaning and how social actors interact with other social actors and their environments. Due to the variety of approaches, there are no introductory texts that are able to explore all approaches. Jewitt 2014 provides a collection of perspectives on multimodality including work in thematic areas and case studies. Jewitt, et al. 2016 introduces some of the main approaches to multimodality focusing on Systemic Functional Multimodal Discourse Analysis, Social Semiotics, and Multimodal-Type Conversation Analysis. That book is useful for students and graduate students and highlights approaches building on SFL and CA. Norris and Maier 2014 is another useful text covering a range of approaches including Multimodal (Inter)Action Analysis, Systemic Functional Multimodal Discourse Analysis, Mediated Discourse Analysis, and Social Semiotics. This reader includes sections on developing multimodal theory/methodology, carrying out multimodal studies, and example analyses providing guidance for graduates and developing researchers. Kress and van Leeuwen 2001 is a key work specifically on social semiotics, developing principles that can be applied across modes. For those interested in how texts, discourses, and objects mediate interactions, Scollon 2001 is an important starting point for introducing Mediated Discourse Analysis. Mediated discourse analysis takes action as primary and considers how texts and objects mediated actions. Norris 2004 builds upon mediated discourse analysis to develop Multimodal (Inter)Action Analysis as a suite of methodological tools for analyzing multimodal interaction. Last, Norris 2016 is a four-volume collection of work in multimodality, tracing historical developments starting in 1956. This collection is included here to highlight the long history of multimodal studies.

• Goodwin, Charles. 2000. Action and embodiment within situated human interaction. Journal of Pragmatics 32.10: 1489–1522.

  DOI: 10.1016/S0378-2166(99)00096-X

  This paper analyzes data from girls playing hopscotch and archaeologists classifying color to theorize the role of the body and material environment as in interaction. This paper has influenced many authors taking a multimodal perspective on interaction and is important for its early challenge of the separation of talk and context.

• Jewitt, Carey. 2014. The Routledge Handbook of Multimodal Analysis. London: Routledge.

  This is one of the first handbooks in multimodality, providing twenty-two chapters from various authors in four sections. The handbook includes theoretical and methodological tools and perspectives for multimodality, work in key thematic areas, and several case studies. As an early handbook in multimodality, this book surveys the work of people working specifically within multimodal frameworks and is a useful start point for students to either move backward, to earlier seminal works, or forward, to newly published work.

• Jewitt, Carey, Josephus Johannes Bezemer, and Kay L. O’Halloran. 2016. Introducing multimodality. London: Routledge.

  This introductory text includes chapters on three major approaches in multimodality: systemic functional linguistic multimodal discourse analysis, social semiotics, and conversation analytic type multimodal discourse analysis. An additional chapter briefly introduces some other approaches. Primarily a text for students, this book does a good job of introducing some approaches to multimodal discourse analysis and considers applications, strengths, and weaknesses.

• Kress, Gunther, and Theo van Leeuwen. 2001. Multimodal discourse. 1st ed. London and New York: Bloomsbury Academic.

  This book builds upon Kress and van Leeuwen 1996 (cited under Social Semiotics). This 2001 book extends the social semiotic approach beyond writing and images. An important early work in social semiotics, the authors seek to develop common principles across different modes.

• Norris, Sigrid. 2004. Analyzing multimodal interaction: A methodological framework. London: Routledge.

  This book introduces multimodal (inter)action analysis as a methodological framework for the analysis of interaction. Building on the primacy of action from Mediated Discourse Analysis, Norris specifically develops a multimodal framework with a range of novel methodological tools. This book is essential reading for those interested in the interactions between social actors and between social actors and their environments.

• Norris, Sigrid. 2016. Multimodality. 4 vols. New York: Routledge.

  This four-volume collection traces the development of studies in multimodality starting in 1956 and including work up to 2014. This important collection includes work taking a range of theoretical bases and demonstrates the long history of multimodal studies. The introductory sections to the set and to each volume are highly accessible and are a particular highlight.

• Norris, Sigrid, and Carmen Daniela Maier. 2014. Interactions, images and texts: A reader in multimodality. Vol. 11. New York: Walter de Gruyter.

  This is one of the only readers in multimodality available. It provides a highly accessible range of chapters covering multimodal (inter)action analysis, social semiotics, SF-MDA, and mediated discourse analysis. The book is in three sections, and the first section on how multimodal theory and methodology are developed is particularly engaging, providing personal insight into high level academic processes.

• Scollon, Ron. 2001. Mediated discourse?: The nexus of practice. London and New York: Routledge.

  Scollon revisits an ethnographic study during one year of a young child’s life where he previously focused on talk. In this follow-up he conducts an in-depth analysis of how the child develops the practice of handing. This text first codifies the central theoretical concepts in Mediated Discourse Analysis, highlighting the primacy of action.