Northwest Education: The Science of Quality: Education Research in School Reform
Summer 2004
She can't be sure what's coming down the pike, but Susan Harrington's trying to prepare Idaho's schools anyway. However you look at it, a new day is dawning for mathematics education. She knows that much, because she's seen something similar happen over there, across the hall where her colleagues in reading are grappling with new realities in this era of the No Child Left Behind Act.
As Idaho's math curriculum coordinator, Harrington is point person for seeing that the state's math programs attend to the mandates of the No Child Left Behind (NCLB) legislation. Right now, she's in the middle of a curriculum adoption review in which she's facing one of the most stringent directives of NCLB: that any programs funded by Title I be based on scientific research.
In Idaho, curriculum adoption recommendations are made at the state level, so the burden is on Harrington to vet math programs for school districts around the state. When Harrington began the review last summer, she knew of only one other state, Mississippi, that was also examining publishers' claims about their "research-based" math curricula. Without a lot of precedence to guide them, Harrington and, presumably, her Southern peer are having to make it up as they go, with a long sideward glance at what's been happening in reading.
Harrington has observed with great interest how the NCLB Reading First initiative, now in its second year, is transforming the way reading is taught across the nation. It has tied $900 million a year in Title I funding to the stipulation that schools use reading programs that have been proven by scientific experiments to raise students' reading proficiency in five key skills (phonemic awareness, phonics, fluency, vocabulary, and comprehension). In this way, the U.S. Department of Education is galvanizing the teaching of reading around a phonics-based instructional approach.
So, when the federal government launched the Mathematics and Science Initiative in February 2003, Harrington thought she'd best read the writing on the wall. While it's not entirely clear that the initiative will affect mathematics as profoundly as the Reading First initiative affected the teaching of reading, Harrington doesn't want her state to have to play catch-up. Poised to help lead Idaho's mathematics curriculum adoption process—which occurs every five years—she realized she would need to be able to point school districts to math programs that had evidence of effectiveness that could hold up under strict scrutiny.
Yet, unlike for reading, the federal government hasn't stipulated effectiveness in what exactly. In the case of reading, there is a large body of experimental research from which the federally convened National Reading Panel could conclude there is "converging evidence" that there are five key skills involved in reading proficiency. Effectively, this has led to a focus on a phonics-based teaching approach. But in math, research has been fragmented and seldom experimental—insufficient for identifying key skills and supporting a single-approach method of teaching. Harrington is finding these gaps to be both freeing and frustrating.
"That was the toughest part. We haven't had really clear guidance yet on this issue, especially when we're in the situation of not having a 'medical model' study done very often" in math education, says Harrington. By "medical model," she's referring to double-blind studies using control groups, a research method pioneered in medicine and the "gold standard" the U.S. Department of Education hopes will transform math education into an evidence-based field.
Research studies, at present, might be insufficient to say exactly what kids should learn in math, but states' academic standards do say. While Harrington and the other reviewers certainly felt the heat of the law's mandate for scientific evidence, they kept their attentions coolly focused on their first priority: Does the program align with Idaho's academic standards in mathematics? If not, no amount of scientific rigor would be sufficient to get the program onto the adoption list.
Like many states, Idaho based its academic standards for mathematics on those identified by the National Council of Teachers of Mathematics (NCTM). Starting in 1989 with the publication of Curriculum and Evaluation Standards for School Mathematics, the NCTM has issued standards for curriculum, teaching, and assessment that have been widely endorsed by mathematicians and educators and central to education reform in mathematics. In addition to basic skills, the NCTM standards emphasize reasoning, problem solving, using technology, and connecting math to other disciplines and real-world contexts.
Yet, in its planning document, "Mathematics and Science Initiative Concept Paper," the U.S. Department of Education doesn't fall in step with proponents of such reform measures. Rather, it points to critics on both sides of the issue:
Critics have charged, for example, that many of the reforms have de-emphasized learning basic mathematics facts and mastery of standard algorithms, encouraged inappropriate dependence on calculators, and relied too heavily on student-directed, discovery learning strategies. Proponents of the reforms counter that traditional approaches over-emphasize memorization of basic mathematics facts and procedures to the neglect of children's conceptual understanding of mathematics.
The paper goes on to say:
At issue, fundamentally, are what constitutes mathematics proficiency and which teaching methods support student achievement of this proficiency. While there has been much debate, very little empirical research has been conducted to determine if one approach or another or some combination of approaches leads to improved mathematics achievement in general—or to improved algebra performance in particular—across ethnic, racial, and socioeconomic groups in our country.
For the time being, Harrington and her colleagues on the review committee feel free to consult various sources as they investigate what constitutes a quality math program. "Just for curiosity's sake," Harrington consulted a couple of pre-NCLB curriculum reviews from 1999: the U.S. Department of Education's Promising and Exemplary Programs and the American Association for the Advancement of Science's Project 2061 Textbooks Evaluation.
And she looked at another, more recent source: "Adding It Up from the National Research Council in 2001 does a very nice job, I think, of boiling it down to the components we're looking for," says Harrington, "and you can get at those components from either side of the debate."
For example, she says, Adding It Up: Helping Children Learn Mathematics identifies five strands of mathematical proficiency that every child should master, and one of those strands is "procedural fluency," which she interprets to mean practicing basic skills. Another strand is "conceptual understanding," which is emphasized, among other things, in the NCTM standards, she says.
"So I think Adding It Up has really hit the mark as recognizing that NCTM has valuable information we need to pay attention to and, also, we still need the practice and the other traditional components... not all one or the other. We're going to take the best pieces from both."
She adds: "It's like another piece of evidence. If I've got a product that's... surfacing to the top of all three studies, then I'm feeling pretty confident that this is a product we should really take a look at as a possible one for a school whose needs match what the product offers."
But the question on Harrington's mind is: Will the Mathematics and Science Initiative ultimately restrict schools to programs that emphasize basic skills and direct instruction?
"I do worry about that. I think that we're involved in the same issue in math that the reading folks have experienced. I think that the solution is to recognize that there is a sensible compromise with the open-ended versus the traditional approach," she says.
In fact, kids need both manners of learning to develop the skills they'll need in today's society, she says. "I think we are in the process of redefining the basics." Today, she says, it's essential that students develop mathematical reasoning and problem-solving skills. These are transferable from one job to another, one situation to another.
"We don't just want numbers on pages of paper. We want those numbers to mean something." When children start school, they come with a natural love of numbers, of sorting, grouping, and finding patterns, she says. "We need to build on that and keep it engaging and not teach those numbers in isolation, but keep them attached to the world and meaningful relationships and connections for that student."
Last summer, Harrington and the curriculum review committee members were seated at five tables piled high with textbooks, manipulatives, computer software, posters—all of the colorfully packaged implements of teaching mathematics today. At one table were the K-4 materials; at another, the middle school materials; and at two others, the high school materials. The fifth table was something new: "We had, for the first time ever, the 'scientifically based research' table," recalls Harrington.
Before them lay a score sheet they would use to evaluate the merits of publishers' claims about research supporting the products. Harrington had come across it almost by chance. A Title I colleague had shown her the rubric, "Continuum of Evidence of Effectiveness," developed by the U.S. Department of Education for its Comprehensive School Reform program. It was designed to help evaluators rank school reform programs in four dimensions—theory/research foundation, evaluation-based evidence of effectiveness, implementation, and replicability—according to "most rigorous," "somewhat rigorous," and "marginal." It was the closest thing to guidelines Harrington had found, and she readily made use of it.
"I feel like we're kind of going to an unknown land here. That's why I don't know exactly what's at stake, because we're still feeling our way through this process. Other states haven't done it where we could learn from their processes, their mistakes."
Few publishers produce reports of the kind of rigorous research called for in the NCLB Act. Usually, the research method publishers use is simply to compare student achievement at a school that has used their product with another school that hasn't. These kinds of comparison studies are not experimental because participants aren't randomly selected and the studies don't control for other explanations. One can't conclude with confidence that it was or wasn't the math curriculum that made the difference in student achievement.
Nevertheless, Harrington and her committee had to take a broad view when analyzing the rigor of the research supporting the math products in front of them. Sitting with them was Frank Gallant, a University of Idaho professor who they'd invited in to help them understand important distinctions in the research methods they were examining. The key question they had to ask themselves was, If a study reports that student achievement is rising in math, to what can that be attributed? Sometimes, Harrington says, "when you look at them, and you ask enough questions, you could conclude that the variable is this product."
They identified two curriculum programs that aligned with Idaho standards and had research they felt was "most rigorous." In the category of "somewhat rigorous," they identified nine programs. They listed these 11 titles on the Idaho Department of Education Web site, and it is from this list that school districts choose their math curricula.
Harrington stresses that the list is not comprehensive. Just because a program is not on the list, she says, doesn't mean that it isn't worthy. It may simply mean that the publisher is still gathering research data on the program's effectiveness and hasn't yet gotten that information out to the states.
In fact, this summer, Harrington will guide an interim review to catch newly released programs. Packages keep arriving as publishers roll new programs and materials off the presses or finish field-testing them in schools. The quality of publishers' studies continues to vary.
"Some are more impressive than others," says Harrington, but at least they're doing them. "The fact that they are looking at the effect of their material on students, that they're not anymore just saying, 'Well, we think this is good.' They are actually taking a look and seeing what effect it is having and [asking], 'Can we quantify and qualify that?'"
All of this attention on research is mostly a good thing, says Harrington. Now, when she gets a phone call from a school district asking her to recommend a good math program, she can do so with greater confidence.
"I have more indepth evidence to share with them, in addition to testimonial evidence," she says. "I think it just helps me to better guide their discussion."
Just one caveat:
"I wouldn't want school districts to expect their student achievement to automatically increase with the choice of a scientifically based program. It's how it's implemented that is more important to me. The teaching is more important, in my opinion, than the curriculum material. Instruction, to me, is the magic bullet."
postscript: If you would like to know more about Idaho's math curriculum adoption, contact Susan Harrington at Sharring@sde.state.id.us.
For eight years, Barbara Geldersma traveled the West selling reading programs to school districts for Macmillan/McGraw-Hill School Publishers. A former elementary literacy curriculum director and secondary school administrator in Las Vegas, Geldersma recently joined NWREL's Comprehensive Center team.
In an interview with Northwest Education, Geldersma shared her insights into the workings of the textbook publishing industry in this evidence-based era in education. Her comments have been edited for length and clarity.
Educational publishing is a very competitive business that attempts to meet the educational needs of states, districts, schools, teachers, and students; respond to changes in pedagogy; and, in the era of NCLB, meet the requirements for scientifically based programs. Programs are very expensive to develop and may cost upward of $50 million for a K-6 reading program. Publishers typically publish major programs on a six- to seven-year cycle, which is usually driven by the textbook adoption cycles of the three most populous states in the country: California, Texas, and Florida. States will put out a "call" for a new reading, math, science, or social studies adoption, and a new program is developed according to new criteria for the curriculum's scope and sequence and assessments.
There is a small margin of profit in educational publishing, so publishers will target as their main customers these three big states, writing their curriculum programs to address their particular review criteria. Therefore, when those big customers make curriculum choices, it affects what all publishers produce nationally.
The mandate for research-based programs is every bit as difficult for publishers as it is for schools. Doing scientific research as called for in NCLB is enormously expensive and time-consuming.
Until No Child Left Behind, there was little need to do such rigorous research, so publishers published what was demanded by the marketplace and didn't focus on doing extensive research studies prior to publication. Prior to NCLB, publishers would often do "learner verification research" (LVR). This meant that as soon as a full unit of instruction was developed for a new textbook—comprising about six to eight weeks of instruction—a publisher would find schools willing to implement it and do pre- and posttests. These pre- and posttests would form the basis of the LVR, but LVRs are not scientific because there are no control groups.
Another form of "proof" is a chart showing school test scores on the state assessments comparing scores before a textbook was adopted and after a year or two of implementation. There is no way of knowing if the textbook program made the difference or if other factors led to an improvement.
Publishers also do correlation studies comparing their program to a state's standards or a particular set of criteria. The publisher may hire a correlator who looks at the state standards and does page-by-page correlations to the standards. These correlations may show a match with the state criteria or standards but are not always accurate.
Publishers may also ask program authors to write white papers on specific topics or content in the programs. This information is called "based-on" research—meaning a program is "based on" this research but the total program hasn't actually been researched in a formal study.
For most of these textbook programs, the six- to seven-year cycle isn't enough time to do a longitudinal experimental study. So, today, publishers are hiring outside consultants to do quasi-experimental research on elements of their programs, such as studying the effectiveness of the program's key instructional strategies. It is the first time that publishers are going to third-party researchers to this extent.
Some tips to keep in mind when reviewing math programs:
|
This document's URL is: Home | Up & Coming | Programs & Projects: Northwest Education | People | Products & Publications | Topics © 2002 Northwest Regional Educational Laboratory Date of Last Update: 7/18/2004  |