As Perfect as It Gets

It is doubtful that this lyrical plea of The Young Rascals could have been satisfactorily answered by scientific research, but an expectation is increasing that science will provide the solutions to the common ills of society. This expectation comes at a time when science has sometimes failed to provide the answers—in space exploration, in medicine, in environmental conservation, and in economic recovery. This expectation is also currently being applied to an unusual field—the science of education.

In 2002, the Education Sciences Reform Act (ESRA) created the Institute for Education Sciences within the United States Department of Education with the challenging mission of "transforming education into an evidence-based field." ESRA followed on the heels of the No Child Left Behind Act's demand for the application of "scientifically based research" in undertaking a variety of initiatives to improve the quality, equality, and accountability of public education in America. Would basing educational decisions on relevant, well-designed research evidence increase the quality, efficiency, and effectiveness of schooling? Yes, but...

In an imperfect world, there is never sufficient scientific evidence to answer every question when it needs to be answered. More often than not, practitioner wisdom will prevail for several reasons:

Relevance. In the rural Northwest, remote villages and isolated rural communities create extremely diverse contexts for delivering public education. Large-scale randomized experiments in Philadelphia and Chicago don't generalize very well to either the students or the situations faced there. Research is particularly limited for Alaska Native, American Indian, Pacific Islander, Hispanic, and immigrant children.

Timeliness. School administrators do not have the luxury of tabling decisions until conclusive evidence is available. They base their judgment on the "best available evidence," which is usually a combination of some research information, prior personal experience, professional standards, state policy, and the advice of respected peers, tempered against available resources and political expedience.

"Equivocality." Research investigations can only reject null hypotheses with increasing levels of confidence. Research cannot unequivocally prove anything. Repeated high-quality investigations increase the certainty of a finding, but they can never guarantee an outcome. This is true in spite of the claims made by vendors of programs validated by scientifically based research. While scientists set the .05 level of probability as the greatest allowable level of confidence, school decisionmakers are searching for solutions that have anything greater than a .50 (coin toss) chance of succeeding!

Access. Because of the dynamic complexity of a decentralized system such as public education in America, efforts to find, evaluate, and synthesize research evidence that relates to a particular educational issue quickly become a daunting task that few educators have the time and resources to undertake. There is just too much information of uncertain quality.

Competing Criteria. Research findings often note that "all things being equal, x is superior to y." Of course, all things never are equal. Evidence of effectiveness is only one of several criteria used to weigh decisions, including cost, compatibility with the setting, utility, quality, ease of implementation, coherence with current standards and philosophy, durability, and availability. An additional criterion is "political will." If what is right is known, is there sufficient political will, social potency, and resources to make difficult and unpopular choices that may cause suffering in the short term?

Credibility. The scientific research community adheres to principles of objectivity, impartiality, and replicability in carrying out systematic inquiry. It places a premium on being accurate and reliable in ascribing confidence to research findings. However, outside the scientific research community, information is viewed with skepticism when it is couched in technical terms and constrained by endless methodological limitations that sound downright evasive. As a result, there is a gap between what researchers know and practitioners believe. This gap between research and practice is true in every field, including education.

The intent of these six points is not to make a compelling case against scientific research, but rather to apply sound principles of science in their proper context. The purpose of research is not to prove, but to improve. Attention to more objective and reliable data can help to test ideas and assumptions. It can help us to perceive what we have never seen before and to demystify the aristocracy of science.

WHAT'S "SCIENTIFIC" ABOUT IT?

The key words being: "rigorous," "systematic," "objective," "reliable," "valid," and "relevant."

The method that embodies these characteristics, say officials from the U.S. Department of Education, is the randomized field study. This method, in fact, is the Bush administration's "gold standard for determining what works." A year ago, speaking to educators and researchers at a conference of the American Educational Research Association in Chicago, Grover "Russ" Whitehurst, director of the education department's Institute of Education Sciences, said, "Randomized trials are the only sure method for determining the effectiveness of education programs and practices."

Randomized controlled trials are studies that randomly assign individuals—or entire classrooms, schools, or school districts—to an intervention group or to a control group to measure the effects of the intervention. The advantage of random assignment is that it makes it possible to evaluate whether the intervention itself, as opposed to other factors, is the cause of the outcomes.

Rigorous, Systematic, and Objective. A rigorous study, the education department says, is one that has been well designed and implemented. The report of a rigorous study will clearly describe the participants, method of random assignment, the intervention and how it differed from what the control group received, and how the intervention was supposed to affect outcomes. It will provide data showing that there were no systematic differences between the intervention and control groups at the outset of the study. And it will state the size of the difference in outcomes between the intervention and control groups. It also means that the process is so systematic and objective that it can be successfully replicated by others, with the same result.

Reliable and Valid. The authors of the book, Reading and Understanding Research (Sage, 1998), Lawrence Locke, Stephen Silverman, and Waneen Wyrick Spirduso, explain reliability and validity in this way:

Finally, reliability deals with consistency—across time, observers, and studies.

Relevant. Does the information address an issue of significance to you in a useful way? The degree to which a research report is relevant to teachers' workaday world has been one measure of the true value of education research. The assumption is that research must be relevant to the classroom teacher to bridge the gap between research and practice.

In Chicago, Whitehurst told his audience: "The people on the front lines of education do not want research minutia, or post-modern musings, or philosophy, or theory, or advocacy, or opinions from education researchers.... They want us, the research community, to provide them a way to cut through the opinion and advocacy with evidence."

Furthermore, "the people on the front lines of education want research to help them make better decisions... [about] curriculum, teacher professional development, assessment, technology, and management. These are questions of what works best for whom under what circumstances. These are questions that are best answered by randomized trials...."

Along these lines, the education department places randomized trials at the top of a hierarchy:

FOR TEACHING'S SAKE

In a study by Mary M. Kennedy, a researcher at Michigan State University, it is interesting to note that more than 100 teachers indicated they did not have a preference, when reading research reports, for one kind of method over another. What mattered to them was if the research study investigated an aspect of the link between teaching and learning.

Any assumption that "the bridge between research and practice can be found in a particular genre of research" is unfounded, Kennedy concluded. "The studies that teachers found to be most persuasive, most relevant, and most influential to their thinking were all studies that addressed the relationship between teaching and learning," whether they were experimental, correlational, narrative, or some other research method.

When teachers expressed high regard for an experimental study using randomized assignment, it was because the study had investigated this teaching-and-learning link—not because they necessarily had more faith in this experimental method.

"Teachers were more likely to find the experiment relevant because it offered a specific teaching strategy that they could try out in their own classrooms, and they were more likely to find the teacher narrative relevant because it provoked thought and raised questions for them," wrote Kennedy in her report, "A Test of Some Common Contentions About Educational Research," published in the fall 1999 edition of American Educational Research Journal.

PRACTITIONER AS RESEARCHER

Finding out "what works" is not only a matter of reading about what scientists have to say about the intricacies of education. It's also a matter of conducting your own research to find out more about how kids are learning in your own state, school district, schoolhouse, or classroom. The randomized field trial is a powerful and valuable form of research, but comprehensive data collection and analysis by schools, districts, and states to inform practices and policies can also count as scientifically based research.

This spring, in fact, the U.S. Department of Education launched the Teacher to Teacher initiative. The department will bring teachers from across the country to Washington, DC, this July for an inaugural Research to Practice Summit. At the summit, teachers who've developed successful research-based practices will share their winning strategies with other teachers and researchers.

"How can I be sure in a world that's constantly changing?" The Young Rascals ask. You can't. While there is no sure thing in education, we can improve our odds by demanding the best available evidence, applying principles of research to our development work, and having thoughtful dialogue about our own local data.

Choosing the Right Method

Generally, there are three types of questions that one might address in education research:

What is going on or what's the intervention?
Does the intervention make a systematic difference?
How and why does it make a difference?

The first question may be addressed by a wide range of research methods, including descriptive and nonexperimental studies. The second can also be addressed by various designs, but perhaps this is where randomized controlled trials are most appropriate and effective. The third is best addressed by descriptive studies, case studies, and nonexperimental studies. The point is that experimental research is neither the only nor the best way in many instances to generate useful information to improve practice and policy in education.

— Kim Yap, senior program director
NWREL Center for Research, Evaluation, and Assessment

NWREL Center for Research, Evaluation, and Assessment
www.nwrel.org/comm/centers/rea.html
This center at NWREL serves anyone who needs data-based information to make decisions, improve effectiveness, or document work. Its Research Unit provides a variety of research services, including consulting on research design, data analysis, interpretation of results, and presentation of findings. It also reviews, interprets, and synthesizes available research on important topics. The center conducts research projects that run the gamut from descriptive studies to quasi-experimental and randomized trials. For more information, visit the Web site, e-mail radera@nwrel.org, or call (503) 275-9558 or (800) 547-6339.

resources

For guidelines for assessing the rigor of a research study, or to learn more about the NCLB Act and how it requires schools to use scientifically based research, visit these Web sites:

Identifying and Implementing Educational Practices Supported by Rigorous Evidence: A User Friendly Guide
www.ed.gov/rschstat/research/pubs/rigorousevid/index.html
This guide outlines the government's standard for what constitutes high-quality evidence of effectiveness in programs and practices.

Scientifically Based Research: A Planning Tool for Educators
www.ael.org/page.htm?&pv=x&pd=1&index=778
A checklist is the key instrument in this free planning tool that can help state, district, and school leaders to evaluate research evidence supporting products or programs. The tool outlines steps to conducting your own research or contracting for research services. It also suggests what educators should look for when developers/companies conduct their own research or contract with independent researchers.

A Policymaker's Primer on Education Research
www.ecs.org/html/educationIssues/Research/primer/foreword.asp
This guide is intended to help policymakers to make evidenced-based decisions by helping them to answer three big questions: What does the research say? Is the research trustworthy? How can the research be used to guide policy? It was jointly developed by Mid-continent Research for Education and Learning and the Education Commission of the States.

U.S. Department of Education
No Child Left Behind Act
www.ed.gov/nclb/landing.jhtml
Included on this site is a link to the section, Proven Methods, which includes fact sheets, transcripts, presentations, and other documents outlining the government's position on scientific research in education. Specific reference is made to scientific research in reading, math, science, English, and professional development for teachers.

These Web sites can be useful for finding evidence-based educational interventions.

The What Works Clearinghouse
www.w-w-c.org
Still in development, this federal database will provide practitioners, researchers, and others with a source for rigorous research studies relevant to education that have been reviewed and summarized by experts.

The Promising Practices Network
www.promisingpractices.net
This site highlights programs and practices that credible research indicates are effective in improving outcomes for children, youth, and families.

The Campbell Collaboration
www.campbellcollaboration.org/Fralibrary.html
This is a registry of reviews of evidence on the effects of interventions in the social, behavioral, and educational arenas.

Social Programs That Work
www.excelgov.org/displayContent.asp?Keyword=prppcSocial
This site offers a series of papers developed by the Coalition for Evidence-Based Policy on social programs that are backed by rigorous evidence of effectiveness.

In an imperfect world, the purpose of research is not to prove, but to improve.

WHAT'S "SCIENTIFIC" ABOUT IT?

FOR TEACHING'S SAKE

PRACTITIONER AS RESEARCHER

Choosing the Right Method

resources