Measurement for Improvement in Education

by

Sola Takahashi, Jon Norman, Kara Jackson, Marsha Ing, Starlie Chinen

• LAST REVIEWED: 22 April 2020
• LAST MODIFIED: 22 April 2020
• DOI: 10.1093/obo/9780199756810-0247

Introduction

This article focuses on “measurement for improvement,” which is the analytic work critical to making and spreading effective changes in quality improvement approaches to system transformation. Quality improvement methods aim to trigger and accelerate the learning of people within a system to make that system work better. This is accomplished through the careful attention of those who experience and enact the system at various levels and by leveraging the expertise from within and outside of the system. Measurement provides critical fuel for that learning. “Measurement for improvement” serves as an umbrella term encompassing a range of measures discussed by quality improvement scholars. These other terms include “process measures,” “practical measures,” and “pragmatic measures.” Data about system outcomes are often a key motivating factor for spurring improvement efforts, for example, low students’ graduation rates or low achievement test scores. However, outcome measures typically fall short of the data needed to inform the day-to-day work of trying changes in practice and learning from these change efforts. Instead, measures of key processes that lead to system outcomes are needed. Measures for improvement need to be closely connected to key processes, timely, and easy to collect and analyze on a regular basis by people in the system. They must also function within social processes that engender trust and transparency so that improvers can learn from failures as well as successes. The education sector has looked to quality improvement efforts in health care as a model for this work, and this article draws heavily on key texts from quality improvement in health care. However, there are some key differences in the types of data regularly available between the fields of health care and education; these differences prompt attention to certain measurement concerns, which are taken up in the references included in this article. The article begins with History and Lineage, which includes some key references that trace quality improvement ideas from industry to health care to education. The next section, Features of Measures for Improvement covers how the function of measures in a quality improvement endeavor shapes the form they take. The next section, A Set of Measures to Inform Improvement, discusses the types of measures needed in combination to inform quality improvement work in systems. The following section, Rigor of Measures to Inform Improvement, addresses what it means to have rigorous measures for improvement, in the context of the educational field where the phenomena of interest are often difficult to see or count. Another challenge of measurement for improvement is taken up in the next section, Analytic Infrastructure for Measurement for Improvement. The challenges of data collection, analysis, and consumption within the busy work lives of improvers imposes constraints and considerations to the social and technological infrastructure that enables the use of measures for improvement purposes. Finally, this article concludes with cases that illustrate measurement for improvement in educational contexts.

History and Lineage

Quality improvement in education draws on a rich heritage of continuous improvement approaches developed in other fields. Originally, continuous quality improvement grew out of efforts within industry during the early 20th century to standardize products. Deming 1982, Deming 1994, Deming 2012, and Shewhart 2015 pioneered techniques for measuring processes within industrial production systems and for analyzing those data. Deming 1982 identified the central problem as one of failure to understand variation. Deming 1994 additionally specified that the analysis of the system must distinguish between common causes of variation and special causes, allowing individuals to determine when and how to intervene. Further, Deming 2012 highlighted how measurement is key to this process: standardized measures (and measurement practices) are required in order to capture how the system works over time and across units. Deming’s and Shewhart’s work flourished in the industrial sector over the next century; individuals in the service sector began to integrate quality improvement concepts into their systems during the 1990s and 2000s. Benneyan, et al. 2003 illustrates how quality improvement ideas from Deming and Shewhart were adopted by health care to improve patient outcomes. Provost and Murray 2011 provided further specifics on the application of improvement methods to health care focusing on data use and measurement for understanding variation. Transition from industrial applications to health-care systems required additional definition of improvement ideas in this new context. Solberg, et al. 1997 suggests that measurement in health care may be divided into three types: measurement for research, measurement for accountability, and measurement for improvement. According to Solberg and colleagues, measurement for research, which aims to create generalized knowledge, and measurement for accountability, which is used to assess system performance, differ from measurement for improvement on the dimensions of who the audience is, the purpose of the measurement activity, what is actually measured, and in what manner. Quality improvement in education draws heavily on the work in health care to adapt the methods and tools used in industrial settings to a social service field. In particular, measurement for improvement in education has not only different requirements (the challenge of measuring concepts such as “reading comprehension”) but also a lack of a set of systematic practices related to data collection and measurement. Yeager, et al. 2013 tackles both of these challenges to describe “practical measurement,” or the development of measures within education improvement efforts that are timely, minimally burdensome, and strongly related to the problem of practice.

• Benneyan, J. C., R. C. Lloyd, and P. E. Plesk. 2003. Statistical process control as a tool for research and healthcare improvement. BMJ Quality & Safety 12.6: 458–464.

  DOI: 10.1136/qhc.12.6.458

  A description of the origin and application of statistical process control. Statistical process control is a method of collecting and representing data to illuminate special causes of variation in a process, enabling individuals to differentiate them from normal variance in collected data.

• Deming, W. E. 1982. Out of the crisis. Cambridge, MA: MIT Press.

  In this text Deming outlines his theory of management, which includes a discussion of the importance of proper measurement for understanding variation in performance in industrial production.

• Deming, W. E. 1994. The new economics for industry, government, education. 2d ed. Cambridge, MA: MIT Press.

  An introduction to the system of transformation that undergirds Deming’s theory of management. Key to this transformative set of processes is understanding variation and the importance of data and measurement in doing so.

• Deming, W. E. 2012. The essential Deming: Leadership principles from the father of quality. New York: McGraw Hill Professional.

  A collection of previously unpublished work by Deming, of note in the text is the focus on the use of statistical control processes in real world applications.

• Provost, L. P., and S. Murray. 2011. The health care data guide: Learning from data for improvement. San Francisco: Jossey-Bass.

  A comprehensive guide describing data analysis and visualization techniques commonly used in improvement science efforts in health care.

• Shewhart, W. 2015. Economic quality of manufactured product. New York: Martino Fine Books.

  An introduction of the concept of “statistical process controls,” including control charts, which are used to understand variation across time. Control charts and other statistical process control methods allow for identification of special causes of variation versus common causes.

• Solberg, L. I., G. Mosser, and S. McDonald. 1997. The three faces of performance measurement: Improvement, accountability, and research. The Joint Commission Journal on Quality Improvement 23.3: 135–147.

  DOI: 10.1016/S1070-3241(16)30305-4

  The authors describe how measurement for improvement differs from measurement for research and accountability purposes.

• Yeager, D., A. Bryk, J. Muhich, H. Hausman, and L. Morales. 2013. Practical measurement. Palo Alto, CA: Carnegie Foundation for the Advancement of Teaching.

  An introduction to the concept of “practical measures.” The authors outline the characteristics of such measures, noting that they differ from traditional research and accountability measures used in education.