Computer-Based Testing

by

Samuel Greiff, Romain Martin

• LAST REVIEWED: 13 January 2014
• LAST MODIFIED: 13 January 2014
• DOI: 10.1093/obo/9780199756810-0031

Introduction

With the introduction of the first computers in the 1970s, the potential of this new technology to generate not only new types of learning environments but also completely new settings for the design and administration of tests, was quickly recognized. Compared to the well-known paper-and-pencil tests, which imply a static presentation of test items with limited interactivity and which depend on complex logistics and administration procedures implemented by trained test administrators, computer-based tests seemed to offer a number of advantages. Due to the interactive testing environments and the data-processing capacities offered by the computer, these potential advantages and added values cover a range of new possibilities that might extend from standardized and automatized administration and scoring procedures through interactive and media-enriched new item types to the possibility of recording and exploiting behavioral data or the possibility of new test administration procedures such as adaptive testing. From a historical perspective, computer-based testing has also capitalized on the introduction of new psychometric models, especially the introduction of item response theory in the 1960s. Further, administering tests on computers opens the door toward the measurement of psychological constructs that have been beyond the reach of paper-and-pencil instruments, because they require interactions in complex environments and can only be accessed through observations in natural settings. The computer offers the possibility to simulate these environments, to record important interactions, and thus to administer tests for these interactive constructs in an efficient and scalable way. This would not be feasible without this new technology. In the last decades, research on computer-based testing has thus experienced a considerable development, as the empirical basis for the potential added value of computer-based tests compared to classical paper-and-pencil instruments had to be developed. However, computer-based testing did not only come with advantages. A number of technical and theoretical challenges related to these tests, such as the availability and compatibility of hardware and software or the need for large item banks have been initially underestimated. Today, one of the major challenges of the field is the development of truly interactive test formats that take full advantage of the technical possibilities offered by the computer. These developments raise a number of challenges in the field of theory development related to the new technology enriched constructs, in the field of psychometric developments related to new complex types of test data, and in the field of IT development related to the need for comparable hard- and software components.

General Overviews

The advent of computer-based testing in the 1970s and the massive introduction of computers in the following decades in professional work environments and in private households led to an almost excessive optimism toward the new possibilities offered by the computer as testing platform. The potential changes that the introduction of computer-based testing (CBT) would imply for the field of educational measurement were described in Bunderson, et al. 1989, which predicted four successive generations of computerized tests. Despite this initial optimism and despite major advances in the field over the last decades, it has to be acknowledged from today’s perspective that the field of computer-based testing has not evolved as rapidly as initially foreseen. For instance, the technological and theoretical implications of computer-based testing have a potential impact on test validity and it is thus important to confront the potential benefits of CBT with its potential threats, as done in Huff and Sireci 2001. It is today also widely acknowledged that due to their technical embedding, computer-based tests have different development and quality assurance processes as well as different life cycles than those that have been well known for decades for paper-and-pencil test instruments and batteries. These differences and the need to integrate knowledge about software life cycles into the development and maintenance of computer-based tests are described in Schegel and Gilliland 2007. Despite a certain number of common features of computer-based test instruments, reviews of existing instruments reveal that test quality has nevertheless to be evaluated on a case-by-case basis as illustrated in Wild, et al. 2008. The two last references both originate from health-related domains, thus illustrating that in this domain a large number of applications in the field of computer-based testing has been generated. Furthermore, the development and delivery of computer-based tests requires dedicated quality assurance processes, which are different in a number of ways from those described for paper-and-pencil instruments. Detailed guidelines for computer-based testing have been published by the International Test Commission 2005. The whole field of CBT is currently evolving rapidly, as the technological environment for the delivery of computer-based tests is increasingly available; but many issues related to the potential and the added value of computerized tests are only emerging today, so that ambitious research agendas for the development of the field are set up as described in Scheuermann and Guimarães Pereira 2008.

• Bunderson, Victos C., Dilkon K. Inouye, and James B. Olsen. 1989. The four generations of computerized educational measurement. Educational measurement 3:367–407.

  Outlines a framework for the potential development of computer-based testing as seen from the perspective of the late 1980s. The advent of four different generations of tests is expected: computerized testing, computerized adaptive testing, continuous measurement, and intelligent measurement. The comparison of these initial expectations with the real development of the field is compelling.

• Huff, Kirsten L., and Stephen G. Sireci. 2001. Validity issues in computer-based testing. Educational Measurement: Issues and Practice 20:16–25.

  DOI: 10.1111/j.1745-3992.2001.tb00066.x

  Based on a definition of validity, this article provides a systematic analysis of potential benefits and threats for the validity of originally paper-and-pencil test instruments when the mode of delivery of these tests is switched to computer-based test administration.

• International Test Commission. 2005. International guidelines on computer-based and Internet delivered testing. Gainesville, FL: International Test Commission.

  With this document, the International Test Commission provides an extensive set of recommendations for good practices in the field of computer-based and Internet-delivered tests. The main objective is to raise awareness among all the concerned stakeholders with regard to quality aspects related to this specific type of test delivery.

• Schegel, Robert E., and Kirby Gilliland. 2007. Development and quality assurance of computer-based assessment batteries. Archives of Clinical Neuropsychology 22:49–61.

  DOI: 10.1016/j.acn.2006.10.005

  Provides an overview of the life cycle of computer-based assessment batteries and proposes a model for the implementation of a systematic quality assurance process for this type of test batteries in the field of health-related research.

• Scheuermann, Friedrich, and Angela Guimarães Pereira, eds. 2008. Towards a research agenda on computer-based assessment: Challenges and needs for European educational measurement. Luxembourg City: Office for Official Publications of the European Communities.

  Describes from a European perspective a research agenda for the field of computer-based assessment showing the diversity of aspects that have to be considered in order to develop the entire field.

• Wild, Katherine, Diane Howieson, Frank Webbe, Adriana Seelye, and Jeffrey Kaye. 2008. Status of computerizí cognitive testing in aging: A systematic review. Alzheimer’s & Dementia: The Journal of the Alzheimer’s Association 4:428–437.

  DOI: 10.1016/j.jalz.2008.07.003

  Provides a systematic review of the potential benefit of computer-based testing for the assessment of cognitive skills in a population of elderly people. Even if it targets a specific content domain, the article provides an interesting insight into the variability of computer-based assessment batteries and stresses the need to judge these batteries on a case-by-case basis.