Defining High Quality Research | Scientifically Based Research

Summer 2004

NWREL Deputy Executive Officer Dr. Bob Blum

Northwest Education Coeditor Denise Jarrett Weeks recently sat down with NWREL CEO Carol Thomas and Deputy Executive Officer Bob Blum to talk about the changing face of educational research and technical assistance in the wake of the No Child Left Behind Act.

How are NCLB requirements for science-based research playing in the field, and what do you see as the Lab's role in helping schools meet those requirements?

Dr. Carol Thomas: Scientifically based research has been around for a long time, but the change in emphasis came about through the No Child Left Behind Act. The term was in the Act over 100 times. That got our attention, because it was slowly filtering down to the schools and they were saying, "What does that mean: scientifically based research?"

That's one term that's been used; "evidenced-based" and "best evidence" are other terms. The basic intent, from the federal perspective, of scientifically based research is to have the best evidence being used in schools to improve student achievement. That's the goal and the spirit of why they're looking at getting the results of scientifically based research into the classroom.

From the Lab's perspective, the role that we have is to help schools implement the best evidence. It may not always be a randomly controlled trial, which is what I think they're calling the "gold standard." We're trying to take schools from where they are and move them forward, using the best evidence.

Dr. Bob Blum: I think the emphasis on having a higher quality research upon which to base decisions is really healthy. It's not new. For example, at this laboratory we've synthesized research for a long time. The effort to get the best evidence has been ongoing. The difference today is the very strong emphasis on a single type of research, the "gold standard" experimental research. It pushes us toward higher quality, but it also raises expectations that that type of research is available to guide decisions in any area that school folks make decisions in, and that's not true.

CT: Another thing, depending on the question that a teacher, a superintendent, or a principal asks, the methodology that you use to answer that question will vary. I think you need to make sure that those things are aligned.

BB: I want to emphasize that the reason we need the very best research we can get is so that folks in schools and districts, who are making decisions about how to improve learning for kids, have the very best information they can get. It's not just research to do good research, it's research to support decisionmaking in schools.

CT: And that's the Lab's perspective, as opposed to the role of a university that might be doing basic research to develop theory, for example. We want to make sure that the research we do has a theoretical base, but the Lab role is more applied research and development, because we are trying to help schools in that decisionmaking.

Since the emphasis on scientifically based research, the Lab has made some decisions to try to strengthen the work that we're doing in this area. We already had good quality assurance processes in place, but we've strengthened those even more, and not only strengthened them but tried to make sure that the staff we have managing research and technical assistance projects and other kinds of projects understand the quality assurance process and what they need to do to make sure their work is of high quality.

Other things that we've tried to do as an institution are to expand and strengthen our own capacity to conduct applied research. You can look across the laboratory at our portfolio of work and that was never a huge part of it, but we always had something going on there. Not all of the labs have, but we did. So we're trying to expand that a little bit. We're doing that through the research units that we have in a few of our centers, specifically in the Center for Research, Evaluation, and Assessment, and through the Research Council. We have good, solid researchers on that council.

The official name of the council is the NWREL Educational Research Council. It's made up of nationally recognized scholars. They're going to meet twice each year, and they'll advise us on priorities and practices. The five who are on the steering committee are Ray Barnhardt (University of Alaska, Fairbanks), Bill Demmert (Western Washington University), Allen Glenn and Dan Goldhaber (University of Washington), and Sam Stringfield (Johns Hopkins University). We'll probably be identifying additional members sometime this fall. They're going to work with us both as advisers and as partners in specific research-based projects, so they're not just going to be giving us advice—they're going to be partners in finding and doing work. We also have people from higher education on our board who also bring a research perspective.

What do you think about the proposed Knowledge Utilization in Education Act?

BB: I think it's a really good mechanism to highlight the need for the connection between research and practice. It's getting some key ideas that are very central to school improvement discussed at the congressional level. So it's a good mechanism for causing that discussion to happen.

In NCLB, there's the emphasis on scientifically based research, but there's not much about what it takes to make the connection between that and practice on a day-to-day basis in schools. There's a gap in the message: "You've got to use this gold standard research, and, oh, by the way, prove to us that you're using it as the basis for your decisionmaking."

CT: The What Works Clearinghouse is an example of that idea—if the information is there, they will come and use it. That's their depository for that information. They're starting out with five or six different categories that they're going to put in there if they can find enough research. But even once they get it there—the research that we did way back in the '70s proves that that isn't enough. There really does have to be some connection with the ultimate practitioner early on, to talk about what their needs are and their context. That's why there are regional labs. We're regionally based so we can understand the context that they're operating in, and to have a dialogue about the research and how it can be adapted to fit their needs. And also what can be adapted, because some things you can adapt so much that you've lost what was originally scientifically based. That dialogue with the school practitioner is critical.

BB: It's a matter of capacity. We talk about scientifically based research as though findings in a particular topic area are always going to be congruent. Well, they're not. There are going to be conflicting findings, and for practitioners, on a day-to-day basis, to have the time and capacity to digest that, understand it, and make their decisions based on best evidence is a stretch.

CT: Even the gold standard of randomized, controlled trials—they're not all created equal either. Just because somebody does a randomized study doesn't make it a high quality randomized study.

BB: And there's the notion of replicating studies and getting the same results. You can have the good, high quality research, but getting it used in a productive way can be miles away. I think that is what the Knowledge Utilization in Education Act is about. It's putting together some assistance to fill that gap in the middle.

CT: It also helps when we go back to Washington to meet with members of Congress to discuss education policy. It gives us an anchor to talk about the work that we do and why it's important as opposed to going back and saying, "support our work, give us more money." It's a different conversation.

Researcher Jonathan A. Supovitz has written, "The danger of undue focus on randomized experiments is that there is a potential for conducting better substantiated studies of less meaningful interventions." This is an interesting challenge for the What Works Clearinghouse.

BB: I remember back in graduate school, some graduate students were doing experiments, and what they were experimenting with was an intervention that might be three hours long. They'd try it out in three different schools. You can control for variables and get the comparison data you need from that small piece of research in a complex system, but the meaning of the findings is pretty small when you talk about improving whole schools and whole districts. The bigger the piece of research, the more meaningful the findings, but the harder it is to control.

CT: If you think about scientific inquiry as a continuum, and you do some things—observations and correlational studies—that develop questions and theories, that can set a framework to build a portfolio of randomized studies that will fit into a coherent set of findings. That's one of the things I'd like to see laboratories do more of. They haven't been able to because they're funded in five-year cycles, so you're constantly responding to the change in the administration's emphasis for the scope of work. You also don't have long enough; right about the time you're getting things to the point where you can do a series of these things, you're having to write a proposal for your next five-year funding cycle. It's why I've always advocated for a 10-year funding cycle for laboratories, because it makes more sense to be able to look at research and development that way.

It's "gold standard" for a reason, so I think that part of it is correct, if you can do it. But the complexity of education is not a laboratory setting. It isn't. There are a lot more things to control for. So it's got to be a big picture of many different types of methodologies.

BB: You can name any kind of medical research and it doesn't always work—you might have a firmer understanding of your percentage of chances of it working, but for you as an individual, it may or may not. And that's true in education. It can be fairly good in some schools with certain characteristics, and other schools with the same characteristics may or may not get the same effect. It's just not absolute. I think sometimes folks are looking for the absolute answer, and it may not be there. And that's not to say that we shouldn't keep pushing for higher quality; we should.

The sense is that this policy about using scientific research is here to stay?

BB: I think the intent of NCLB is not new. I think the intent has been around for a long time. The intent is to get clear about what to teach and to hold folks accountable to teaching to standards, so that all kids learn. That's been around for a long time. It was the central core of the Effective Schools movement. It was, in a different way, the central core of the Oregon Competencies Program, which was way back in the 1970s. To get clear about it, to teach to the clear goals, and then to measure to see if kids are getting it, and then to go back and do what's necessary to bring them up.

CT: It is important to make sure that the curriculum, assessment, and instruction are aligned.

BB: In Oregon, all of this started emerging with the Oregon Competencies program in the 1970s. Now there is the Oregon 21st Century Schools Program. Some of the central elements remain the same and those central elements really are not different from the central elements of NCLB, it's just on a bigger stage. There are already moves to adapt NCLB. States are already seeking changes in some of the key elements, so through that political process NCLB is going to change. But it doesn't mean that it will lose its central concept. It will still be, "Let's get clear about what kids need to learn. Let's learn how to measure it. Let's teach to it. Get kids to learn it. Let's be accountable and see what's going on and make adjustments so that more and more kids learn."

CT: I can agree with most of NCLB's goals. A lot of the push for this really came from minority parents, because they were seeing a gap in achievement between their kids' achievement—and kids from poor families, too—and the achievement of majority kids. Even though the U.S. Department of Education has recently issued guidelines to make aspects of NCLB more flexible, we're still concerned that we don't leave behind the kids that we, the laboratory, are interested in, too. That's the core of our work.

Jonathan Supovitz surveyed principals and they overwhelmingly told him, "I wouldn't let my school get involved in a scientific experiment. I don't have time for it. I don't trust it. I don't want my kids to be guinea pigs."

BB: That's going to be an issue when you try to get schools to either randomize students within a school or to have some schools in and some schools out of a research study; some classrooms in and some classrooms out. That's going to be an issue.

CT: The first step is making sure that you have entrée. What schools don't want to be is used and then not given access to the data. Regional Laboratories don't do that; we work in partnership with the school. They own some of the work and therefore the results. They are not waiting until the end to all of a sudden get the findings. We're making sure that they get the data.

BB: The challenge of getting control schools to really be control schools is a big challenge.

CT: Other issues that we'll be dealing with are maintaining confidentiality of data and parental consent for participation in field trials.

BB: I believe that, overall, the push toward higher quality research is healthy. That's a good move. I think sometimes it's pushed too hard—further than people can get in a short period of time, but overall it's a healthy thing for us to be doing. In a way, it's like pushing schools to become better and better; it's pushing researchers to become better and better, and it's healthy.

CT: A priority for NWREL is to do strategic, high-quality research and development that responds to the needs in our region. And then to work with states and school districts to use the best evidence available to improve instruction that results in gains in achievement for all students.

Respond to this article