Jump to Content Jump to Main Navigation

Education Methodologies for Conducting Education Research
by
Marisa Cannata

Introduction

Education is a diverse field and methodologies used in education research are necessarily diverse. The reasons for the methodological diversity of education research are many, including the fact that the field of education is composed of a multitude of disciplines and tensions between basic and applied research. For example, accepted methods of systemic inquiry in history, sociology, economics, and psychology vary, yet all of these disciplines help answer important questions posed in education. This methodological diversity has led to debates about the quality of education research and the perception of shifting standards of quality research. The citations selected for inclusion in this article provide a broad overview of methodologies and discussions of quality research standards across the different types of questions posed in educational research. The citations represent summaries of ongoing debates, articles or books that have had a significant influence on education research, and guides to those who wish to implement particular methodologies. Most of the sections focus on specific methodologies and provide advice or examples for studies employing these methodologies.

General Overviews

The interdisciplinary nature of education research has implications for education research. There is no single best research design for all questions that guide education research. Even through many often heated debates about methodologies, the common strand is that research designs should follow the research questions. The following works offer an introduction to the debates, divides, and difficulties of education research. Schoenfeld 1999, Mitchell and Haro 1999, and Shulman 1988 provide perspectives on diversity within the field of education and the implications of this diversity on the debates about education research and difficulties conducting such research. National Research Council 2002 outlines the principles of scientific inquiry and how they apply to education. Published around the time No Child Left Behind required education policies to be based on scientific research, this book laid the foundation for much of the current emphasis of experimental and quasi-experimental research in education. To read another perspective on defining good education research, readers may turn to Hostetler 2005. Readers who want a general overview of various methodologies in education research and directions on how to choose between them should read Creswell 2009 and Green, et al. 2006. The American Educational Research Association (AERA), the main professional association focused on education research, has developed standards for how to report methods and findings in empirical studies. Those wishing to follow those standards should consult American Educational Research Association 2006.

  • American Educational Research Association. 2006. Standards for reporting on empirical social science research in AERA publications. Educational Researcher 35.6: 33–40.

    DOI: 10.3102/0013189X035006033Save Citation »Export Citation »E-mail Citation »

    The American Educational Research Association is the professional association for researchers in education. Publications by AERA are a well-regarded source of research. This article outlines the requirements for reporting original research in AERA publications.

    Find this resource:

  • Creswell, J. W. 2009. Research design: Qualitative, quantitative, and mixed methods approaches. 3d ed. Los Angeles: SAGE.

    Save Citation »Export Citation »E-mail Citation »

    Presents an overview of qualitative, quantitative and mixed-methods research designs, including how to choose the design based on the research question. This book is particularly helpful for those who want to design mixed-methods studies.

    Find this resource:

  • Green, J. L., G. Camilli, and P. B. Elmore. 2006. Handbook of complementary methods for research in education. Mahwah, NJ: Lawrence Erlbaum.

    Save Citation »Export Citation »E-mail Citation »

    Provides a broad overview of several methods of educational research. The first part provides an overview of issues that cut across specific methodologies, and subsequent chapters delve into particular research approaches.

    Find this resource:

  • Hostetler, K. 2005. What is “good” education research? Educational Researcher 34.6: 16–21.

    DOI: 10.3102/0013189X034006016Save Citation »Export Citation »E-mail Citation »

    Goes beyond methodological concerns to argue that “good” educational research should also consider the conception of human well-being. By using a philosophical lens on debates about quality education research, this article is useful for moving beyond qualitative-quantitative divides.

    Find this resource:

  • Mitchell, T. R., and A. Haro. 1999. Poles apart: Reconciling the dichotomies in education research. In Issues in education research. Edited by E. C. Lagemann and L. S. Shulman, 42–62. San Francisco: Jossey-Bass.

    Save Citation »Export Citation »E-mail Citation »

    Chapter outlines several dichotomies in education research, including the tension between applied research and basic research and between understanding the purposes of education and the processes of education.

    Find this resource:

  • National Research Council. 2002. Scientific research in education. Edited by R. J. Shavelson and L. Towne. Committee on Scientific Principles for Education Research. Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

    Save Citation »Export Citation »E-mail Citation »

    This book was released around the time the No Child Left Behind law directed that policy decisions should be guided by scientific research. It is credited with starting the current debate about methods in educational research and the preference for experimental studies.

    Find this resource:

  • Schoenfeld, A. H. 1999. The core, the canon, and the development of research skills. Issues in the preparation of education researchers. In Issues in education research. Edited by E. C. Lagemann and L. S. Shulman, 166–202. San Francisco: Jossey-Bass.

    Save Citation »Export Citation »E-mail Citation »

    Describes difficulties in preparing educational researchers due to the lack of a core and a canon in education. While the focus is on preparing researchers, it provides valuable insight into why debates over education research persist.

    Find this resource:

  • Shulman, L. S. 1988. Disciplines of inquiry in education: An overview. In Complementary methods for research in education. Edited by R. M. Jaeger, 3–17. Washington, DC: American Educational Research Association.

    Save Citation »Export Citation »E-mail Citation »

    Outlines what distinguishes research from other modes of disciplined inquiry and the relationship between academic disciplines, guiding questions, and methods of inquiry.

    Find this resource:

Datasets

There are several sources of data that could be used for education research. The National Center for Education Statistics (NCES) is the most comprehensive source of data on education and is part of the U.S. Department of Education. The Common Core of Data (CCD) provides basic information on all public schools and school districts in the United States and the Private School Universe Survey provides similar data for private schools. The CCD also has a financial questionnaire for public school districts to collect comparable revenue and expenditure data across districts. NCES also conducts various cross-sectional or longitudinal surveys that provide a variety of data on numerous educational issues. Datasets are available about early childhood education (such as the Early Childhood Longitudinal Study), home education, school crime and safety, teacher supply and demand (such as the Schools and Staffing Survey), career and technical education, high school experiences and programs (such as the Education Longitudinal Study of 2002 and the High School and Beyond study), and postsecondary experiences and programs (such as the National Postsecondary Student Aid Study). The National Assessment of Educational Progress serves as the nation’s report card for overall academic achievement across states and can be linked to school and teacher information. NCES also coordinates the U.S. participation in several international assessments, such as the Trends in International Mathematics and Science Study (TIMSS). The Census Bureau and the Current Population Surveys are other possible data sources for education research. Further, the increase of statewide longitudinal data systems has provided researchers with greater opportunities to study existing educational interventions or programs. Individual districts are also sometimes willing to provide access to their data to researchers who implement safeguards to protect data confidentiality.

Journals

There are numerous journals that report education research. The following journals are highlighted because they not only include original research in education, but also often publish articles that discuss educational research itself. They discuss the difficulties, dilemmas, and standards of education research and often include multiple perspectives on research methodologies. The best sources of advice on how to conduct quality research are the top journals in the field. Reading reports of high-quality research can help researchers recognize and implement high-quality research themselves. Education research may be found in journals published by professional associations focused on educational research, such as the American Educational Research Association, or educational practice, such as the National Council of Teachers of Mathematics or American Association of Colleges for Teacher Education. Thus researchers should explore professional associations related to their work for journals that publish research in their area. For example, Educational Researcher is published in association with the American Educational Research Association and is sent to every association member. It serves as an outlet for broad discussions about research from many perspectives. The Review of Educational Research also is published in association with the American Educational Research Association and publishes literature reviews and meta-analyses. Readers interested in assessment and measurement should review recent issues of Educational Measurement: Issues and Practice, which is published in association with the National Council on Measurement in Education. The various disciplines that inform education also have journals that focus on education research in that discipline. One example is Education Finance and Policy, which is focused on economics of education. These journals will provide readers with not only an outlet to publish their own work, but also examples of how to implement research designs, describe the methods, and report findings.

Experimental Research

The past decade has seen a rapid increase in the number of experimental designs. Experimental evaluations were previously viewed as either impossible to implement in educational settings or unethical as they denied treatment to some group. However, the increasing influence of economists in education research has led to the rise in emphasis placed on randomized controlled trials and experimental research. Experimental designs are useful to answer research questions related to the effectiveness of educational programs or curricula in reaching a specified outcome, such as student achievement or college enrollment. Cook 2002 and Shadish, et al. 2001 review the arguments for randomized controlled trials and counter the arguments that they are not appropriate for educational research. Raudenbush 2005 provides a contrasting perspective and argues that experimental designs are useful, but cautions that too much focus on a single design may be too limiting. For example, experimental designs are able to achieve high internal validity, but in highly controlled conditions that often limit their external validity and generalizability. Mosteller and Boruch 2002 also provides diverse perspectives on the role of experimental designs in education research. Donner and Klar 2000 and Raudenbush 1997 provide guidance on a particular type of randomized trial—the cluster randomized trial—that considers the nature of schools as systems with students nested within classrooms and schools. Betts, et al. 2006 and Borman, et al. 2005 offer examples of experimental designs.

Quasi-Experimental Research

Randomized controlled trials are not always possible. Thus rigorous methodologies using other designs are sometimes necessary to answer questions about the effectiveness of educational programs. These methodologies are called quasi-experimental because they attempt to mimic the ability of experimental designs to make causal inferences about the effect of educational interventions. Economics currently has a large role in influencing discussions of quality in education research methodology for both experimental and quasi-experimental designs. Angrist and Pischke 2009 provides an overview of various quasi-experimental methodologies from econometrics. Other works in this section highlight particular methodologies. Bloom 2003 focuses on time series analyses. Heckman 1978 develops instrumental variables. Imbens and Lemieux 2008 outlines regression discontinuity designs. Hirano and Imbens 2001 and Rosenbaum and Rubin 1983 provide perspectives on propensity score matching. McCaffrey, et al. 2003 describes value-added models that attempt to isolate the effects of teachers on student achievement. One limitation of quasi-experimental designs is that it is never completely possible to remove the possibility that there is a confounding variable that limits the ability to make causal connections between the educational intervention and outcomes. Frank 2000 is therefore useful because it provides a mechanism to evaluate and quantify the potential impact of an unidentified confounding variable.

  • Angrist, J. D., and J. -S. Pischke. 2009. Mostly harmless econometrics: An empiricist’s companion. Princeton, NJ: Princeton Univ. Press.

    Save Citation »Export Citation »E-mail Citation »

    Easy to read and provides numerous examples of many econometric techniques. Focused on applying techniques and provides solutions for problems that are frequently encountered.

    Find this resource:

  • Bloom, H. S. 2003. Using “short” interrupted time-series analysis to measure the impacts of whole-school reforms: With applications to a study of accelerated schools. Evaluation Review 27.1: 3–49.

    DOI: 10.1177/0193841X02239017Save Citation »Export Citation »E-mail Citation »

    When using secondary data, all available data are usually used. However, when designing studies to collect primary data, researchers need to know how large of a sample they should use. This article provides formulas for calculating the sample size needed to have enough power to detect program effects. It also provides a description of interrupted time series designs.

    Find this resource:

  • Frank, K. A. 2000. Impact of a confounding variable on a regression coefficient. Sociological Methods and Research 29:147–194.

    DOI: 10.1177/0049124100029002001Save Citation »Export Citation »E-mail Citation »

    Quasi-experimental methods are used to try to make a causal inference about the relationship between variables. While threats to the ability to make a causal inference are present, it can be difficult to evaluate the effect on the findings. This paper provides a useful index for quantifying the robustness of a causal inference.

    Find this resource:

  • Heckman, J. 1978. Dummy endogenous variables in a simultaneous equations system. Econometrica 47: 153–161.

    Save Citation »Export Citation »E-mail Citation »

    This author was vital to developing a method known as instrumental variable analysis. This paper is an important piece of the development and outlines why instrumental variables should be used.

    Find this resource:

  • Hirano, K., and G. W. Imbens. 2001. Estimation of causal effects using propensity score weighting: An application to data on right heart catheterization. Health Services and Outcomes Research Methodology 2:259–278.

    DOI: 10.1023/A:1020371312283Save Citation »Export Citation »E-mail Citation »

    This paper applies propensity score matching techniques. It is useful as a criticism of propensity score matching and through its description of problems encountered.

    Find this resource:

  • Imbens, G. W., and T. Lemieux. 2008. Regression discontinuity designs: A guide to practice. Journal of Econometrics 142:615–635.

    DOI: 10.1016/j.jeconom.2007.05.001Save Citation »Export Citation »E-mail Citation »

    Provides a helpful overview of regression discontinuity designs and guides for implementing them. In particular, it is useful for its discussion of choosing an appropriate bandwidth or window around the discontinuity.

    Find this resource:

  • McCaffrey, D., D. Koretz, J. R. Lockwood, and L. Hamilton. 2003. Evaluating value-added models for teacher accountability. Santa Monica, CA: RAND.

    Save Citation »Export Citation »E-mail Citation »

    Provides a useful introduction to value-added models and their applications in measuring teacher effectiveness. It includes a discussion of relevant measurement and statistical issues, validity of value-added models, and examples of recent applications in research.

    Find this resource:

  • Rosenbaum, P., and D. B. Rubin. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika 70:41–55.

    DOI: 10.1093/biomet/70.1.41Save Citation »Export Citation »E-mail Citation »

    One of the hallmark articles on propensity score matching.

    Find this resource:

Hierarchical Linear Modeling

One unique feature of education research is that data points are often related to each other. That is, students are clustered within classrooms, which are clustered within schools. Student-level measures are not randomly distributed because of this hierarchical nature of the data. Teacher-level measures are also not independent of each other as teachers’ opinions of something within their school are likely to be similar, or at least potentially more similar than opinions that stretch across multiple schools. Hierarchical linear modeling was developed to deal with the fact that assumptions of standard statistical techniques are violated due to this clustering. The works in this section describe hierarchical linear modeling (HLM) and provide examples. Raudenbush and Bryk 2002 is the foremost textbook on HLM and the authors have developed special software to make HLM easier, although this software is not necessary to do multilevel modeling. Raudenbush, et al. 2000 provides the user manual for the HLM software. Cohen, et al. 2002 provides additional perspective on the importance of multilevel models and when and how they should be used. The development of multilevel models can be traced to organizational theory and research on the organizational contexts of schools as described by Bidwell and Kasarda 1980. Rowan, et al. 1991 takes on the issue of how much variation occurs between individuals and how much variation occurs between groups of individuals. It finds that a significant portion of the variance occurs between individuals within groups. Willett 1994 extends HLM to measuring change over time by considering observations over time clustered within individuals. Lee and Burkam 2003 and Lee, et al. 1991 provide examples of studies using multilevel models.

  • Bidwell, C. E., and J. J. Kasarda. 1980. Conceptualizing and measuring the effects of school and schooling. American Journal of Education 88:401–430.

    DOI: 10.1086/443540Save Citation »Export Citation »E-mail Citation »

    This article played an important role in moving forward educational research by highlighting the importance of the difference between schools and schooling. It also was a vital development in studying the social organization of schools and pushing education research to consider the varying organizational levels and structures of schooling.

    Find this resource:

  • Cohen, P., J. Cohen, S. G. West, and L. S. Aiken. 2002. Random coefficient regression and multilevel models. In Applied multiple regression/correlational analysis for the behavioral sciences. 3d ed. Edited by P. Cohen, J. Cohen, S. G. West, and L. S. Aiken, 536–567. Mahwah, NJ: Lawrence Erlbaum.

    Save Citation »Export Citation »E-mail Citation »

    Provides a useful overview of statistical models that consider the multilevel nature of schooling. This chapter is useful for those who want to explore the impact of school-level variables on the relationship between variables measured at an individual level. It is important for its treatment of clustering within levels as not just a problem to be overcome, but as a meaningful aspect of the data.

    Find this resource:

  • Lee, V. E., and D. T. Burkam. 2003. Dropping out of high school: The role of school organization and structure. American Educational Research Journal 40.2: 353–393.

    DOI: 10.3102/00028312040002353Save Citation »Export Citation »E-mail Citation »

    Provides an example of an analysis using hierarchical linear modeling for a dichotomous outcome variable.

    Find this resource:

  • Lee, V. E., R. F. Dedrick, and J. B. Smith. 1991. The effect of the social organization of schools on teachers’ efficacy and satisfaction. Sociology of Education 64.3: 190–208.

    DOI: 10.2307/2112851Save Citation »Export Citation »E-mail Citation »

    An example of an analysis using hierarchical linear modeling on a study of teacher attitudes and school structure. It is useful for those who want to read an application of a specific methodology that considers the organizational structure and context of schools.

    Find this resource:

  • Raudenbush, S. W., and A. S. Bryk. 2002. Hierarchical linear models: Applications and data analysis methods. 2d ed. Newbury Park, CA: SAGE.

    Save Citation »Export Citation »E-mail Citation »

    The textbook on hierarchical linear models. Should be read by anyone who wants to do HLM.

    Find this resource:

  • Raudenbush, S. W., A. S. Bryk, Y. F. Cheong, and R. T. Congdon. 2000. HLM 5: Hierarchical linear and nonlinear modeling. Chicago: Scientific Software International.

    Save Citation »Export Citation »E-mail Citation »

    The user manual for the HLM software. It addresses the technicalities of using the software and guidelines on making substantive decisions in analysis.

    Find this resource:

  • Rowan, B., S. W. Raudenbush, and S. J. Kang. 1991. Organizational design in high schools: A multivariate analysis. American Journal of Education 99.2: 238–266.

    DOI: 10.1086/443980Save Citation »Export Citation »E-mail Citation »

    Explores the properties of measures of school organizational conditions from teacher surveys. It is important for highlighting that while there are noticeable differences between schools, there is also a great deal of variation among teachers within a school.

    Find this resource:

  • Willett, J. B. 1994. Measuring change more efficiently by modeling individual growth over time. In The international encyclopedia of education. 2d ed. Edited by T. Husen and T. N. Postlethwaite, 671–678. Oxford: Pergamon.

    Save Citation »Export Citation »E-mail Citation »

    While HLM is most commonly used to study individuals nested in groups, it can also be used to measure growth in learning when multiple observations over time are clustered within individuals. This chapter discusses such change models.

    Find this resource:

Survey Research

Regardless of the analytic methodology used, education research often involves the collection of new data. One form of this primary data collection is survey research. Education research may involve surveys of teachers, principals, schools, districts, parents, students, or another stakeholder in education. Teachers and schools are common participants in survey research collections. The grade level under study will influence the ability to include student surveys as the reading abilities of younger students may limit their ability to complete surveys. In some cases, it is possible to administer short verbal surveys when collecting data from young students. Collecting high-quality survey data is a complex process. Researchers will want to design a survey with unbiased questions and achieve high response rates to minimize bias in the participating sample. A response rate of 70 percent is considered acceptable, although researchers should still consider the possibility of nonresponse bias, or the possibility that a group of participants in the sample was systematically less likely to respond to the survey. The following works were selected because they provide overviews and guidance to researchers who want to conduct survey research. Dillman, et al. 2009 provides an introduction to survey research and detailed advice on all aspects of implementing surveys. It includes many examples on survey construction and recruiting participants. Converse, et al. 2008 also explores different ways to recruit participants and implement a survey. This article is relevant to the increasing use of the Internet in survey administration. Desimone and LeFloch 2004 and Sudman, et al. 1996 review the importance of understanding how participants think when they respond to surveys. The advice in these works is useful in improving the wording and format of survey questions to ensure the data collected are high quality. Fisher 1993 provides another perspective on question wording, focused on how to avoid asking questions in ways that prompt the respondents to feel pressured to respond in socially desirable ways. When participants complete surveys, they often leave some questions blank, thus producing missing data in the surveys. Allison 2000 discusses the implications of missing data and the challenges of imputing responses when data are missing. Mayer 1999 and Rowan, et al. 2002 address some specifics regarding survey research in education, particularly surveys used to capture teacher instruction and effects of teachers on student achievement.

  • Allison, P. D. 2000. Multiple imputation for missing data: A cautionary tale. Sociological Methods and Research 28:301–309.

    DOI: 10.1177/0049124100028003003Save Citation »Export Citation »E-mail Citation »

    Provides guidance on how to handle missing data from survey research. It addresses many of the challenges of using multiple imputation as a common approach to missing data.

    Find this resource:

  • Converse, P., E. W. Wolfe, X. Huang, and F. L. Oswald. 2008. Response rates for mixed-mode surveys using mail and email/web. American Journal of Evaluation 29:99–107.

    DOI: 10.1177/1098214007313228Save Citation »Export Citation »E-mail Citation »

    Study randomly assigned teachers to complete a survey either online or by mail and compares response rates. The authors recommend alternating the survey mode for the final reminder to improve response rates.

    Find this resource:

  • Desimone, L., and K. LeFloch. 2004. Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis 26.1: 1–22.

    DOI: 10.3102/01623737026001001Save Citation »Export Citation »E-mail Citation »

    Reviews the importance of conducting cognitive interviews when designing surveys. Cognitive interviews can help improve survey questions by understanding the thought processes participants undergo when responding. This can increase validity and reliability of survey data.

    Find this resource:

  • Dillman, D. A., J. A. Smyth, and L. M. Christian. 2009. Internet, mail and mixed-mode surveys: The tailored design method. 3d ed. New York: Wiley.

    Save Citation »Export Citation »E-mail Citation »

    Outlines steps to successfully implement a survey, including how to word questions, sampling concerns, and procedures to maximize response rates.

    Find this resource:

  • Fisher, R. J. 1993. Social desirability bias and the validity of indirect questioning. Journal of Consumer Research 20:303–315.

    DOI: 10.1086/209351Save Citation »Export Citation »E-mail Citation »

    Social desirability bias refers to the tendency of survey participants to answer in ways they think they should be answering, rather than reflecting their true situation. This article provides useful hints to frame questions in ways that reduce bias.

    Find this resource:

  • Mayer, D. P. 1999. Can policymakers trust survey data? Educational Evaluation and Policy Analysis 21.1: 29–45.

    Save Citation »Export Citation »E-mail Citation »

    Explores the validity and reliability of attempts to capture teachers’ instruction through teacher surveys by comparing survey responses to classroom observations. The study concludes that surveys can be useful to measure instructional practices, although it is better practice to use composite measures rather than individual indicators.

    Find this resource:

  • Rowan, B., R. Correnti, and R. J. Miller. 2002. What large-scale, survey research tells us about the effects of teachers and teaching on student achievement: Insights from the Prospects Study of Elementary Schools. Teachers College Record 104.8: 1525–1567.

    DOI: 10.1111/1467-9620.00212Save Citation »Export Citation »E-mail Citation »

    Reviews the various analytic techniques for measuring teacher effects on student achievement, explores why such effects occur and how large-scale surveys of teachers can inform educational research on the relationship between teachers and student achievement.

    Find this resource:

  • Sudman, S., N. M. Bradburn, and N. Schwarz. 1996. Thinking about answers: The application of cognitive processes to survey methodology. San Francisco: Jossey-Bass.

    Save Citation »Export Citation »E-mail Citation »

    This book was at the forefront of attempts to understand the cognitive processes of survey participants while responding to a survey. The concepts in this book are still relevant for those designing a new survey.

    Find this resource:

Assessment and Measurement

No Child Left Behind required states to test all students in grades three through eight in reading and math. The federal Department of Education has also provided grants to states to develop longitudinal data systems to link student assessment results across years. This increasing availability of student achievement data has led to an enhanced ability to analyze student assessment results as outcomes of educational programs. Researchers who wish to use student assessment data should have an understanding of how these assessments are developed and the standards that assessments (and the ways in which they are used) must meet to be valid and reliable. The works outlined in this section are intended to provide researchers with such knowledge. American Educational Research Association, American Psychological Association, and National Council on Measurement in Education 1999 describes the professional standards assessments must meet. For readers who want an overview of the multitude of complex issues involved in educational testing by recognized experts in the field, Linn 1993 will be useful. Readers who want additional information on validity in educational assessment should consult Messick 1995 and Moss, et al. 2006. Hamilton, et al. 1995 provides an application of assessment theory and practice through examining the assessment associated with a longitudinal survey of students. Allen and Yen 1979 provides a readable introduction to the theory underlying current assessment and measurement practices. Assessment and measurement are considered together in this section because much of the theory behind educational assessments also applies to other measures used in educational research, such as measures of student self-esteem or teacher efficacy. Regardless of whether the measure is of student knowledge or something else, the measure should be reliable and valid for the contexts in which it is used. Readers interested in developing non-assessment measures should find all works in this section helpful, but may want to pay particular attention to Linacre 2002 and Wright and Masters 1982.

  • Allen, M. J., and W. M. Yen. 1979. Introduction to measurement theory. Prospect Heights, IL: Waveland.

    Save Citation »Export Citation »E-mail Citation »

    Provides an overview of developing assessments and survey measures, including evaluating the reliability and validity of measures, principles of test construction, and equating and scaling measures. The topics are useful to those designing their own survey measure or assessment or who wants to evaluate established measures.

    Find this resource:

  • American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. 1999. Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    Save Citation »Export Citation »E-mail Citation »

    Outlines standards for designing and using educational assessments. It is revised from previous standards and developed jointly by three major professional associations.

    Find this resource:

  • Hamilton, L. S., E. M. Nussbaum, H. Kupermintz, J. I. M. Kerkhoven, and R. Snow. 1995. Enhancing the validity and usefulness of large-scale educational assessments: II. NELS:88 science achievement. American Educational Research Journal 32.3: 555–581.

    Save Citation »Export Citation »E-mail Citation »

    Highlights multidimensional nature of student knowledge in science. Factor analysis discovered three subscales of science achievement in tenth graders’ science knowledge on a large-scale assessment. A companion article in the same journal reports similar findings for mathematics.

    Find this resource:

  • Linacre, J. M. 2002. Optimizing rating scale category effectiveness. Journal of Applied Measurement 3.1: 85–106.

    Save Citation »Export Citation »E-mail Citation »

    Provides guidance on evaluating the effectiveness of rating scales in survey measures and assessments. It uses a Rasch measurement model and includes useful applications.

    Find this resource:

  • Linn, R. L., ed. 1993. Educational measurement. 3d ed. American Council on Education series on higher education. Phoenix, AZ: Oyrx.

    Save Citation »Export Citation »E-mail Citation »

    An overview of validity, reliability, bias, and scaling in educational assessment. It provides a general overview of issues involved in constructing, administering, and scoring assessments. This handbook should be consulted by any researcher interested in developing, using, or evaluating educational assessments.

    Find this resource:

  • Messick, S. 1995. Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. The American Psychologist 50.9: 741–749.

    DOI: 10.1037/0003-066X.50.9.741Save Citation »Export Citation »E-mail Citation »

    This article begins by providing an overview of the types of validity of educational assessments commonly considered: content, criterion, and construct validity. The author then introduces a comprehensive concept of validity that includes both what scores mean and how they are used.

    Find this resource:

  • Moss, P. A., B. J. Girard, and L. C. Haniford. 2006. Validity in educational assessment. Review of Research in Education 30.1: 109–162.

    DOI: 10.3102/0091732X030001109Save Citation »Export Citation »E-mail Citation »

    Literature review discusses theories of validity in educational assessments. It is particularly useful for those interpreting and using assessments to highlight issues of validity for the consequences of various test uses.

    Find this resource:

  • Wright, B. D., and G. N. Masters. 1982. Rating scale analysis. Chicago: Mesa.

    Save Citation »Export Citation »E-mail Citation »

    Item response theory (IRT) underlies much current practice in test construction, and Rasch models are a common version of IRT. This book provides helpful, detailed guidance on implementing Rasch measurement models for assessments and other applications of measures involving Likert-type ratings.

    Find this resource:

Qualitative Research Methodologies

Some research questions are not suitable to randomized control trials or survey research. For example, experimental or quasi-experimental approaches may provide robust estimates of whether people respond in a particular way to a specified program, intervention, or incentive, but they are limited in the ability to shed light on how or why people respond as they do. There are also important questions about the meaning people ascribe to their situations that are vital to understanding educational experiences. For these reasons and more, qualitative research methodologies remain critical to educational research. While qualitative research has been criticized for not providing “objective” answers, high-quality qualitative research also has standards for reliability, validity, and systematicity similar to quantitative research. The works in this section are intended to provide an overview of such standards. Erickson 1986 outlines several types of qualitative methodologies, and readers who want a basic description of the options when conducting fieldwork should consult this work. Readers who are designing qualitative studies should also refer to guidance in Lincoln and Guba 1985 and Patton 2002, in which the authors discuss all aspects of qualitative research design. Rubin and Rubin 1995 and Krueger and Casey 2000 focus on specifics of conducting interviews and focus groups, respectively, in qualitative work. Readers who are implementing these components in their qualitative studies will find these readings particularly helpful. Glaser 1965 was instrumental in developing methods to systematically analyze qualitative data, and contemporary readers may still find it useful as it introduces the analytic method known as “grounded theory.” Readers who want more detailed information and examples on analyzing and reporting qualitative data should consult Miles and Huberman 1994. All research involves ethical concerns; however, these concerns are more prominent in qualitative methodologies in which researchers may (and indeed try to) develop relationships with the participants in the research. Fine and Weis 1996 will be useful to readers who are struggling with how to resolve ethical dilemmas as they are presented.

  • Erickson, F. 1986. Qualitative methods in research on teaching. In Handbook of research on teaching. Edited by M. C. Wittrock, 119–161. Washington, DC: American Educational Research Association.

    Save Citation »Export Citation »E-mail Citation »

    A general overview of several qualitative research methods, including ethnography, participant observation, case study, symbolic interactionism, phenomenology, and constructivism.

    Find this resource:

  • Fine, M., and L. Weis. 1996. Writing the “wrongs” of fieldwork: Confronting our own research/writing dilemmas in urban ethnographies. Qualitative Inquiry 2.3: 251–274.

    DOI: 10.1177/107780049600200301Save Citation »Export Citation »E-mail Citation »

    The authors explore the ethical dilemmas in ethnographic research. While specific to ethnography, the article is useful for the larger discussion of responsibilities of qualitative researchers to those who participate in their research.

    Find this resource:

  • Glaser, B. G. 1965. The constant comparative method of qualitative analysis. Social Problems 12.4: 436–445.

    DOI: 10.1525/sp.1965.12.4.03a00070Save Citation »Export Citation »E-mail Citation »

    One of the early guides to systematic analysis of qualitative data. This work is historically important for its role in developing grounded theory.

    Find this resource:

  • Krueger, R. A., and M. A. Casey. 2000. Focus groups: A practical guide for applied research. 3d ed. Thousand Oaks, CA: SAGE Publications.

    Save Citation »Export Citation »E-mail Citation »

    A useful guidebook for using focus groups in qualitative fieldwork. It includes advice on designing focus group questions and logistics for implementing focus groups.

    Find this resource:

  • Lincoln, Y., and E. Guba. 1985. Naturalistic inquiry. Beverly Hills, CA: SAGE.

    Save Citation »Export Citation »E-mail Citation »

    A handbook for designing qualitative studies. This book is particularly useful for its discussion of sampling procedures.

    Find this resource:

  • Miles, M. B., and A. M. Huberman. 1994. Qualitative data analysis: An expanded sourcebook. 2d ed. Thousand Oaks, CA: SAGE.

    Save Citation »Export Citation »E-mail Citation »

    This book provides detailed guidance on how to systematically analyze qualitative data. The discussion of how to display qualitative data is particularly useful.

    Find this resource:

  • Patton, M. Q. 2002. Qualitative research and evaluation methods. 3d ed. Vol. 3. Thousand Oaks, CA: SAGE.

    Save Citation »Export Citation »E-mail Citation »

    A comprehensive handbook for all aspects of designing and implementing qualitative research projects. It includes a description of differences between types of qualitative research designs and strategies for making design decisions at all stages, including sampling, data collection, analysis, and reporting.

    Find this resource:

  • Rubin, H., and I. Rubin. 1995. Qualitative interviewing. Thousand Oaks, CA: SAGE.

    Save Citation »Export Citation »E-mail Citation »

    Can serve as a guidebook for designing and conducting fieldwork interviews. It includes useful advice for constructing questions, contacting participants, and engaging in interviews.

    Find this resource:

Program Evaluation

Program evaluation is a systematic attempt to collect and analyze data related to a specific educational program, curricula, or intervention. It may use any of the research methodologies described in the previous sections and should adhere to the standards described when implementing these methodologies. Program evaluation is described separately here for two reasons. First, there is a research community focused on program evaluation. Mertens 2005, Petrosino 2000, and Rog 1994 represent important works in program evaluation across many fields and will be useful to readers who want an introduction to program evaluation. Weiss 1997 is helpful for examining program evaluation specifically related to educational research and the use of theories behind educational interventions in designing program evaluations. The second reason program evaluation is considered in a separate section is the formation of the What Works Clearinghouse (WWC) and the response of the educational research community in the standards used by the WWC. The WWC was established by the federal Department of Education to locate and evaluate individual studies of the effectiveness of various educational programs to create summaries of the evidence collected on specific programs. The goal of this endeavor was to provide information to educational practitioners on the evidence of effectiveness for programs they are considering adopting. What Works Clearinghouse 2008 describes the evidence standards and process for evaluating programs and should be read by anyone who will use reports created by the WWC. The creation of the WWC has led to much debate within the education research community about standards for research and evidence of program effectiveness. Chatterji 2005 and Slavin 2008 provide contrasting perspectives on standards for evidence used by the WWC. Slavin 2008 in particular provides a critique of the WWC. Program evaluations should consider not only the overall impact of educational programs, but also the conditions under which program effects are achieved and whether the program is equally effective for all populations. Readers should read Honig 2009 for an examination of how to conduct evaluations that consider such issues.

  • Chatterji, M. 2005. Evidence on “What Works” An argument for extended-term mixed-method (ETMM) evaluation designs. Educational Researcher 34.5: 14–24.

    DOI: 10.3102/0013189X034005014Save Citation »Export Citation »E-mail Citation »

    An overview of extended-term mixed-method evaluation designs, with a discussion of its guiding principles and theoretical rationale. The author argues that such designs are necessary for making generalized causal inferences in educational evaluations.

    Find this resource:

  • Honig, M. I. 2009. What works in defining “What Works” in educational improvement: Lessons from education policy implementation research, directions for future research. In Handbook of education policy research. Edited by G. Sykes, B. Schneider, D. N. Plank, and T. G. Ford, 333–347. New York: Routledge.

    Save Citation »Export Citation »E-mail Citation »

    Provides guidance for evaluations in education research to move from general assessments to examinations of specific populations for whom and under what conditions programs can be effective.

    Find this resource:

  • Mertens, D. M. 2005. Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods. 2d ed. Thousand Oaks, CA: SAGE.

    Save Citation »Export Citation »E-mail Citation »

    A handbook on conducting program evaluations in education. It is useful for anyone designing program evaluations.

    Find this resource:

  • Petrosino, A. 2000. Mediators and moderators in the evaluation of programs for children. Evaluation Review 24.10: 47–72.

    DOI: 10.1177/0193841X0002400102Save Citation »Export Citation »E-mail Citation »

    An overview for defining, identifying, and assessing mediators and moderators in program evaluations.

    Find this resource:

  • Rog, D. J. 1994. Constructing natural “experiments.” In Handbook of practical program evaluation. Edited by J. S. Wholey, H. Hatry, and K. Newcomer, 119–132. San Francisco: Jossey-Bass.

    Save Citation »Export Citation »E-mail Citation »

    A discussion of how to take advantage of natural variation to evaluate program effects.

    Find this resource:

  • Slavin, R. E. 2008. What works? Issues in synthesizing educational program evaluations. Educational Researcher 37.1: 5–14.

    DOI: 10.3102/0013189X08314117Save Citation »Export Citation »E-mail Citation »

    This article begins a discussion of how to evaluate program evaluations, specifically focused on the procedures of the What Works Clearinghouse. Interested readers should also review responses to this article that were published in the same issue of Educational Researcher for a continuing dialogue on evidence-based research.

    Find this resource:

  • Weiss, C. H. 1997. How can theory-based evaluation make greater headway? Evaluation Review 21.4: 501–524.

    DOI: 10.1177/0193841X9702100405Save Citation »Export Citation »E-mail Citation »

    Theory-based evaluation uses the beliefs and assumptions that undergird a program or intervention to assess how well the program is meeting assumptions at each theorized step in a sequence. This approach is useful to understand not only whether a program was effective, but also the stage at which programs appear to be limited in creating intended results.

    Find this resource:

  • What Works Clearinghouse. 2008. Procedures and standards handbook, version 2.0. Washington, DC: Institute of Education Sciences.

    Save Citation »Export Citation »E-mail Citation »

    Outlines the procedures used in evaluating specific educational interventions for their impact on student achievement. It includes details on the review process, standards of evidence, and how summaries are created.

    Find this resource:

Research Syntheses

Most research involves original empirical work. As research on a specific topic accumulates, however, the findings may appear inconsistent or contradictory. Making statements based on rigorous scientific research requires not just high-quality individual studies, but also an assessment of what the collection of research on that area says so that knowledge accumulates and progresses. Research syntheses are particularly important in moving fields forward and accumulating knowledge to build theories that incorporate new knowledge and testing new theories through new empirical research. The following works provide insight to the process of systematically conducting syntheses of past research. Boote and Beile 2005 focuses on the role of literature reviews in designing studies and developing doctoral researchers. Readers who want a broad overview to both systematic literature reviews and meta-analyses should consult Cooper and Hedges 1994. Meta-analysis is a particular type of research synthesis in which effect sizes from individual studies are used as data points to aggregate findings. For readers who want practical advice on conducting meta-analyses, Lipsey and Wilson 2001 will be indispensable. D’Agostino and Murphy 2004 is also useful for readers who want to read an example of a complete meta-analysis. Rothstein, et al. 2005 will be helpful to readers who want to explore the impact of publication bias on meta-analyses. Finally, meta-analyses and literature are conducted with varying quality, as are the individual studies that make up the meta-analysis. Torgerson 2007 provides a useful framework for gauging the quality of research reviews and will be helpful to researchers who want advice on how to conduct high-quality reviews.

  • Boote, D. N., and P. Beile. 2005. Scholars before researchers: On the centrality of the dissertation literature review in research preparation. Educational Researcher 34.6: 3–15.

    DOI: 10.3102/0013189X034006003Save Citation »Export Citation »E-mail Citation »

    Useful guidance for predoctoral researchers in preparing literature reviews prior to engaging in the dissertation research. It reviews the purpose of the literature review in both designing studies and developing scholars.

    Find this resource:

  • Cooper, H., and L. V. Hedges, eds. 1994. The handbook of research synthesis. New York: Russell Sage Foundation.

    Save Citation »Export Citation »E-mail Citation »

    This handbook provides a basic overview and helpful guidance on conducting literature reviews and meta-analyses. It includes discussions of how to identify research, how to evaluate the quality of previous research, how to analyze and combine effect sizes, and how to report results of syntheses.

    Find this resource:

  • D’Agostino, J. V., and J. A. Murphy. 2004. A meta-analysis of reading recovery in United States schools. Educational Evaluation and Policy Analysis 26.1: 23–38.

    DOI: 10.3102/01623737026001023Save Citation »Export Citation »E-mail Citation »

    An example of a well-done meta-analysis.

    Find this resource:

  • Lipsey, M. W., and D. B. Wilson. 2001. Practical meta-analysis. Applied Social Research Methods 49. Thousand Oaks, CA: SAGE.

    Save Citation »Export Citation »E-mail Citation »

    One of the primary guides on planning and implementing meta-analyses. It includes many examples and applications.

    Find this resource:

  • Rothstein, H. R., A. J. Sutton, and M. Borenstein, eds. 2005. Publication bias in meta-analysis: Prevention, assessment, and adjustment. Chichester, UK: John Wiley.

    DOI: 10.1002/0470870168Save Citation »Export Citation »E-mail Citation »

    One challenge in conducting meta-analyses is that studies that find null effects often face difficulty getting published, thus creating a bias toward finding an effect in published work. This book provides guidance on evaluating and addressing publication bias.

    Find this resource:

  • Torgerson, C. J. 2007. The quality of systemic research reviews of effectiveness in literacy learning in English: A “tertiary” review. Journal of Research in Reading 30.3: 287–315.

    DOI: 10.1111/j.1467-9817.2006.00318.xSave Citation »Export Citation »E-mail Citation »

    This article evaluates several meta-analyses of literacy learning in the United Kingdom. It is useful for the discussion of how to evaluate and discuss the quality of research reviews and common threats to validity of research reviews.

    Find this resource:

Implementation

Implementation research is both an important research agenda in its own right and a vital complement to experimental and quasi-experimental designs that aim to measure overall program impact. Implementation research helps us understand what happens when programs interact with specific educational contexts. Elmore and McLaughlin 1988 and Fullan and Pomfret 1977 are important works in the development of implementation research. These readings were among the first to highlight that implementation of reforms vary across contexts and may impact the reform’s effectiveness. These readings also begin to detail what influences implementation. Datnow and Park 2009 provides a more contemporary account of policy implementation and an updated review of the factors the affect implementation across contexts. Cohen 1990 is particularly enlightening as it paints a picture of a reform in one classroom and the challenges encountered by a teacher trying to faithfully implement reforms. For readers who want to conduct an implementation study, Honig 2009 will be useful. Dusenbury, et al. 2003 will be helpful to readers who are conducting an implementation study in conjunction with a randomized controlled trial. It provides guidance on measuring the fidelity of program implementation across settings.

  • Cohen, D. K. 1990. A revolution in one classroom: The case of Mrs. Oublier. Educational Evaluation and Policy Analysis 12.3: 327–345.

    Save Citation »Export Citation »E-mail Citation »

    A case study of one teacher trying to implement a mathematics reform. It highlights the difficulty of curricular reforms when they hit classrooms and the challenges of assessing implementation through the eyes of the teacher or an outside observer.

    Find this resource:

  • Datnow, A., and V. Park. 2009. Conceptualizing policy implementation: Large-scale reform in an era of complexity. In Handbook of education policy research. Edited by G. Sykes, B. Schneider, D. N. Plank, and T. G. Ford, 348–361. New York: Routledge.

    Save Citation »Export Citation »E-mail Citation »

    Provides a framework for understanding implementation of educational policies and programs.

    Find this resource:

  • Dusenbury, L. R., R. Brannigan, M. Falco, and W. B. Hansen. 2003. A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research 18.2: 237–256.

    DOI: 10.1093/her/18.2.237Save Citation »Export Citation »E-mail Citation »

    Research review highlights the relationship between fidelity of implementation and estimates of program effectiveness. It also provides recommendations for measuring and analyzing fidelity.

    Find this resource:

  • Elmore, R. F., and M. W. McLaughlin. 1988. Steady work: Policy, practice, and the reform of American education. Santa Monica, CA: RAND.

    Save Citation »Export Citation »E-mail Citation »

    Highlights the importance of understanding how policies are implemented in schools in evaluating school reform. It is important for designing research to assess fidelity and implementation as it outlines factors that influence variability in implementation.

    Find this resource:

  • Fullan, M., and A. Pomfret. 1977. Research on curriculum and instruction implementation. Review of Educational Research 47 2: 335–397.

    Save Citation »Export Citation »E-mail Citation »

    Describes why we need to study implementation, how to define and measure implementation, and how to identify the determinants of implementation.

    Find this resource:

  • Honig, M. I. 2009. What works in defining “What Works” in educational improvement: Lessons from education policy implementation research, directions for future research. In Handbook of education policy research. Edited by G. Sykes, B. Schneider, D. N. Plank, and T. G. Ford, 333–347. New York: Routledge.

    Save Citation »Export Citation »E-mail Citation »

    Outlines lessons from implementation research and recommends that education research move from general assessments of effectiveness to examinations of specific populations for whom and under what conditions programs can be effective.

    Find this resource:

LAST MODIFIED: 12/15/2011

DOI: 10.1093/OBO/9780199756810-0061

back to top

Article

Up

Down