Volume 9 Number 4
The Science of Quality
Education Research in School Reform

In This Issue

The Promise of Scientific Evidence in Education

A New Day for Mathematics Education?

A Tale of Two Schools: Reading Instruction in the Real World

Teaching by the Numbers

Lessons for Investigation

Evaluation in the Classroom

Finding Out What Works for Native Students

As Perfect as It Gets

Resources: The What Works Clearinghouse

Letters

Expert Opinion: An essay by researcher Jonathan Supovitz

Web Exclusives:

Defining High Quality Research

Reflections by a Research-Based Teacher Educator

About This Issue

Previous Issues

Text Only

Feedback

 

Summer 2004

Maybe it was the wave of television programs about crime scene investigations and forensic evidence, but in the past few years scientific inquiry has really caught on. Growth management policies in the Northwest are guided by the standards of "best available evidence." Health practitioners are guided by "evidence-based practice." And now the No Child Left Behind Act (NCLB) in more than 100 places in the law, has challenged educators with the use of "scientifically based research" (SBR).

• Resources

To this end, one of the strategic goals of the U.S. Department of Education is to "transform education into an evidence-based field" by "encouraging the use of scientifically based methods within federal education programs." And rightfully so, when you consider the billions of dollars invested in public elementary and secondary education. By taking a look at the research evidence, we hope the odds will improve for doing what works.

Educators in the Northwest are concerned about becoming better consumers of evidence. In a 2002 survey of 5,250 educators conducted by the Northwest Regional Educational Laboratory, 48 percent of teachers, 68 percent of principals, and 65 percent of superintendents responding to the item indicated that "more" or "much more" effort was needed to "use classroom and school-level data and educational research to improve teaching and learning." Further, these same Northwest educators (56 percent of teachers, 65 percent of principals, and 58 percent of superintendents) asserted the need to "work collaboratively to examine the effectiveness of current school and classroom practices." These opinions reflect a desire to become more knowledgeable about the evidence used to bolster professional decisions, policies, and practices.

What did Congress have in mind when it repeatedly inserted scientifically based research requirements into the NCLB Act? First, more accountability was desired of public education to increase the academic achievement of all students, especially those most in need. To this end, SBR is all about trying to get it right—making decisions to implement programs and practices for which there is evidence of effectiveness.

The second intent was an increased attention to program integrity. Clearer theories of action and processes proven to work provide better justification for setting resource allocation priorities.

Third, taxpayers are concerned about their return on the huge investment of time and money devoted to educational improvement. What are the lost opportunity costs of unsuccessful programs?

Most important, the creation of a reliable and valid cumulative knowledge base is the hallmark of a true profession. Scientific advances develop over time and across disciplines in other fields, but less so in a highly complex and decentralized arena such as education. Education remains a highly skilled craft, where the wisdom of experience takes precedence over empirical evidence.

Yet, we would not deny that the "application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs" is a good idea. After all, good science (and good public education!) got us to the moon and Mars. So, below are a few ideas to help you become better consumers of scientifically based research:

Think Like a Scientist. Put away your lab coats and Bunsen burners. Science is about how you think about things. The National Research Council published a book in 2002 entitled Scientific Research in Education that is well worth reading. The authors identify six principles of science:

• Pose significant questions that can be investigated empirically. Science is all about efforts to answer the what, why, and how of things.
• Link research to relevant theory. Studies are bolstered by a conceptual rationale that explains why relationships likely exist.
• Use methods that permit direct investigation of the question. Objective and reliable procedures can be used for direct measurement.
• Provide a coherent and explicit chain of reasoning. Causal connections can be made in explaining how things work.
• Replicate and generalize across studies. Findings are cumulative across studies to build a more reliable and robust body of knowledge.
• Disclose findings to encourage professional scrutiny and dialogue. Findings are published and debated in refereed journals.

In reflection, these six principles are not particularly foreign to educators. We've just become both overwhelmed with the amount of purported research evidence available and lax about testing the science behind our own practices and ideas. At the 2002 meeting of the American Educational Research Association, Mary Kennedy of Michigan State University reported her research findings that less than one-third of teacher practices are directly derived from research. And when teachers do use research-based practices, they cannot clearly explain why they've used them.

Become a Better Consumer of Research. The Coalition for Evidence-Based Policy hopes that educators will place a premium on evidence generated by randomized experiments. Indeed, in December 2003, the coalition published for the Institute of Education Sciences Identifying and Implementing Educational Practices Supported by Rigorous Evidence: A User Friendly Guide. But maybe there isn't experimental evidence to address your question, or maybe your research question might not be relevant to an experiment at all. Does that mean that you should ignore other research information that may be available? The National Institute for Literacy published a guide in May 2003 written by Paula Stanovich and Keith Stanovich entitled Using Research and Reason in Education, addressing how teachers can use scientifically based research to make curricular and instructional decisions. The authors provide three straightforward sources of evidence:

• The results of formally tested student performance by your district or state—what does your own evidence say about what works?
• The published findings of empirical research on the instructional practices of others—what does the literature suggest is working for others?
• Justification by proof-of-reason, where past research theory and empirical evidence is used to show the link between practices and outcomes, but direct evidence is lacking. What arguments can you make that your "theory of instruction" is effective based upon past studies?

It's hard to argue that the more rigorous the research method, the more conclusive the evidence (when all else is equal). So we generally recognize that true experiments are less equivocal than quasi-experiments, and those are better than correlational and case studies, which in turn supersede expert opinion, which trumps anecdotal evidence. We do need to demand to see the nature of the evidence when the curriculum vendor assures us that the material is "scientifically based research that meets NCLB requirements." At education conferences throughout the region, the halls are filled with vendors who cross their hearts that their material meets the "gold standard." Be a better consumer and ask for the evidence.

Know Where and How To Find Research. The SBR provisions of NCLB imply that educators have the time and resources to seek out and review original research on topics of interest. Between 1991 and 2001 there were 334,647 abstracts entered into the ERIC Database. Only some of these entries are research, but you'll have to read them all to find out. If you took 10 minutes to review each abstract for eight hours a day, every day, that will only take 38.7 years to review the entire body of literature, which would then be out of date. If you only wanted to review the 14,477 education journal articles cited in ERIC in 2001, you could get that done in a mere one and two-thirds years.

In their review of 100,000 journal articles, the National Reading Panel found about 40 articles that had acceptable scientific rigor. As G. Reid Lyon, a researcher with the National Institute of Child Health and Human Development, pointed out to the U.S. Congress in 2000, "much of the research which has been disseminated is not scientifically sound, that which is sound is not in a useful format for teachers, and what is sound and useful is often ignored, especially by teacher colleges."

Clearly there must be a better way of vetting the research evidence. The most promising source will be the What Works Clearinghouse (www.w-w-c.org). This project is funded by the Institute of Education Sciences to systematically review and compile the best available research evidence about various educational practices and approaches.

Other sources of literature reviews, research syntheses, meta-analyses, and digests of information about best practices are available from the National Clearinghouse of Comprehensive School Reform (www.goodschools.gwu.edu); the Northwest Regional Educational Laboratory (www.nwrel.org); the Education Resources Information Center system (www.eric.ed.gov); the National Reading Panel (www.nationalreadingpanel.org); Eisenhower National Clearinghouse (www.enc.org); and the National Center for Research on Evaluation, Standards, and Student Testing (www.cse.ucla.edu).

Evidence Is Not Enough. Just because an experiment produces conclusive evidence, it doesn't necessarily mean it is information that you should adopt. The Stetler/ Marram Research Utilization Model created in the field of nurse education provides very helpful guidance to decisionmaking. Cheryl B. Stetler wrote about the model in "Refinement of the Stetler/Marram Model for Application of Research Findings to Practice," an article in the January/February 1994 issue of Nursing Outlook. The model suggests that five basic questions be satisfactorily answered in determining if you should implement scientifically based research findings:

• Is this valid research that is relevant to my work? For example, is it concerned with a content area or instructional pedagogy at issue for me?
• Is the focus of the research consistent with my audience, setting, and approach? For example, is this Philadelphia study of fourth-graders likely to be relevant to my high school kids in Wellpinit, Worley, or Weston?
• Are the potential risks, necessary resources, and organizational readiness feasible? Is this something we could actually pull off with the people, problems, and dollars we actually have?
• How do the research conclusions fit with other substantiated evidence? What do you already know about this area based on prior research, professional wisdom, and accepted practice?
• Is it applicable for taking action at this time?

Be Strategic. You can't do it all, so be strategic in determining what and how you will use evidence. In writing proposals for grants and programs under NCLB, plan on doing at least four things:

• Be clear and quantitative in documenting the needs of your school and students.
• Build a compelling case for the best available evidence of what, how, and why a proposed intervention should work in your setting.
• Include a bibliography of research references.
• Describe methods you will use to evaluate the effectiveness of the program.

Then, as you implement your new research-based practices, consider the following:

• Focus attention on full, consistent, and true implementation of the program design. Then carefully evaluate results to determine if student performance was positively affected and why or why not.
• Devote efforts to better understanding the performance of your students; the philosophy and context of the school; and the quality, scope, and intensity of your programs. In what areas would you seek additional research evidence in designing a course of action?
• In areas that seem to work, focus on building a theory of action among staff members to provide a rationale of why practices logically lead to desired results. Regularly evaluate, debrief, and gauge progress. This is the true measure of practitioner research.
• Share the results of your research with others to contribute to the growth of knowledge.

Remember that the value of research is determined by how it can be applied to actual practice. In 1601, Lancaster demonstrated the value of citrus fruit in preventing scurvy among seamen. It took 263 years for the British merchant navy to adopt the practice.

Adapted and reprinted by permission. Nelson, S. (2004). The promise of scientific evidence for improving educational results. Curriculum in Context, 31(1), 24-27. Steven Nelson is director of NWREL's Office of Planning and Service Coordination. (Photo by Denise Jarrett Weeks)

This document's URL is: Home | Up & Coming | Programs & Projects: Northwest Education | People | Products & Publications | Topics © 2002 Northwest Regional Educational Laboratory Date of Last Update: 7/18/2004 Email Webmaster Tel. 503.275.9500