Reflections by a Research-Based Teacher Educator | Scientifically Based Research

By K.C. Holder

Summer 2004

The No Child Left Behind Act (NCLB) mandates that educational practices and federally funded programs be supported by scientifically based research (SBR). Claims about the effectiveness of practices and programs must be validated and proven reliable through "scientific" research methods. This assumes that educational practices and programs supported by SBR will have positive effects on student achievement—we will leave no child behind. However, SBR has been so narrowly defined that it excludes research methods that have historically been rich and valuable.

What has been the impetus behind the current political pendulum swing, away from the acceptance of a broad swath of research methodology, both qualitative and quantitative, to today's singular focus on quantitative research? Should the boundaries of what constitutes good research be so narrowly defined?

Any classroom teacher or administrator, and probably most parents can tell you that one instructional strategy does not and will not work for all students all of the time. But does the requirement that federally funded programs and practices be supported by scientific research direct us toward a "one size fits all" approach to teaching and educational research?

In defining "scientifically based research," the NCLB Act uses words such as "objective," "valid," "reliable" and, most important, "causal." Specifically, the Act states that, "[SBR] ...is evaluated using experimental or quasi-experimental designs." In short, experimental designs involve randomly assigning subjects into control and experimental groups, manipulating a treatment, observing an effect on dependent and/or independent variables, and determining the strength of the effect through statistical analysis. Sounds good, doesn't it? The problem is that experimental and quasi-experimental designs are based on an underlying assumption of causality.

In his article, "Causal Explanation, Qualitative Research, and Scientific Inquiry in Education," J.A. Maxwell writes that, "this claim [that experimental methods are the 'gold standard'] has been based on a restrictive and philosophically problematic model of causality, a model that is implicit in most of the advocacy for SBR" (2004, p. 3).

Maxwell acknowledges that causality, and specifically the use of experimental designs, can be very powerful when three conditions are met: 1) there is a well-developed theory, 2) causal processes are manipulable, and 3) the situation is conducive to direct investigation of causal processes. Such conditions are met in laboratory settings, such as medicine and science but, as Maxwell notes "these conditions are less often met in education, where causal processes are often complex, temporally and contextually variable, and directly observable" (p. 9).

In education, new theories are frequently being introduced, assessment of treatments is not always manipulable, and causal processes are more aligned with direct observation. Teachers know that classrooms are extremely complex environments that often are far from the controlled environments that pure experimental design demands.

What about ethnographies of exceptional classroom environments, for instance? Or comparative case studies between expert and novice teachers? I, for one, can learn a lot about teaching from examining unique and exemplary cases.

For example, ethnographies can provide descriptions of classroom culture, through which one can learn what it is like to be a member of that classroom. Understanding what it is like to be a member allows me, as a teacher, to tap into the unique needs of individual students in my classroom.

Similarly, every teacher knows that one or two students in his or her classroom will be particularly hard to reach. Comparative case studies provide descriptions of attributes that distinguish between two cases. The detailed description of the complex temporal and contextual factors that influence these attributes allows me, as a reader, to understand how and why the expert and novice are different (or how and why two students are different within a particular classroom). As an educator, understanding these differences positions me to lead my students toward becoming more like the experts.

My current research explores the nature and landscape of teacher knowledge. Although we know bits and pieces of this specialized knowledge base, as a field we have much to learn about how teachers' professional knowledge influences their practice. Pedagogical content knowledge has been described as "the ways of representing and formulating the subject that make it comprehensible to others" by expertly using analogy, illustration, explanation, and demonstration to represent ideas (Shulman, 1986, p. 9).

To be accredited through the National Council for Accreditation of Teacher Education, teacher education departments must document that, "Teacher candidates reflect a thorough understanding of pedagogical content knowledge delineated in professional, state, and institutional standards" (NCATE, 2002). However, I recently reviewed 51 journal articles that had "pedagogical content knowledge" in their titles, and no single study met all of the criteria for scientific research as defined in NCLB.

In other words, the literature base defining pedagogical content knowledge does not meet the criteria for scientifically based research, yet many in the professional discipline of teacher education acknowledge the concept and use it as an evaluation criterion for accreditation.

This is only one of the paradoxes that education researchers face. As with teaching methods, a balance is needed between research generated to explain causal relationships (typically quantitative) and research generated to be explorative (typically qualitative).

Currently, pedagogical content knowledge is undergoing theory-to-practice implementation. The implementation cycle is as follows: (a) introduce the theory; (b) use qualitative research to understand, explain, and describe the theory; and (c) use quantitative research to determine the effectiveness of the theory. As a theory is implemented into practice it progresses through stages exemplified in the following sample research questions:

Stage 1. What is going on here? Introduce the theory of pedagogical content knowledge.
Stage 2. Can the theory be used to describe novice and expert teachers? If so, how? If pedagogical content knowledge is an important component of the knowledge base of teaching, does professional education, in fact, transmit this area of professional knowledge? How do the characteristics of the lesson preparation method stimulate preservice teachers to show their ability to develop pedagogical content knowledge?
Stage 3. There will be no difference in student achievement scores where the students' teachers received training in pedagogical content knowledge, compared to those teachers who did not (null hypothesis).

In short, causal and/or experimental designs are not appropriate for the types of research questions that occur at Stage 1 and Stage 2. Only after a theory is introduced and better understood by the academic community of researchers and applied by a representative sample of teachers does the theory become "ready" for causal study.

Using evidence-based claims for important educational decisions definitely moves us one step forward; however, limiting the scope of what counts as "scientific" may only take us two steps back.

References

Creswell, J.W. (2002). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. Upper Saddle River, NJ: Merrill/Prentice Hall.

Holder, K.C. (2004, April). Pedagogical content knowledge: Can a successful program of research exist without scientific-based research? Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.

Maxwell, J.A. (2004). Causal explanation, qualitative research, and scientific inquiry in education. Educational Researcher, 33(2), 3-11.

National Council for Accreditation of Teacher Education. (2002). Professional standards for the accreditation of schools, colleges, and departments of education. Washington, DC: Author.

National Research Council. (2002). Scientific research in education. R.J. Shavelson & L. Towne (Eds.), Committee on Scientific Principles for Educational Research. Washington, DC: National Academy Press.

Shulman, L.S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4-14.

—K.C. Holder is an assistant professor of education at Eastern Oregon University.

Respond to this article