Expert Opinion: Andrew Porter and Value-Added Assessment | Accountability

Expert Opinion

Andrew Porter and Value-Added Assessment

By Anne Turnbaugh Lockwood

NWREL researcher Anne Turnbaugh Lockwood spoke at length recently with Vanderbilt University scholar Andrew Porter about “value-added” accountability in K-12 education. Porter is the Patricia and Rodes Hart Professor of Educational Leadership and Policy and director of the Learning Sciences Institute at Vanderbilt University. He has published widely on psychometrics, student assessment, education indicators, and research on teaching. His current work focuses on curriculum policies and their effects on opportunity to learn. Below, Lockwood summarizes her conversation with Porter in which he explained the meaning of value-added accountability within the No Child Left Behind Act; its impact on school systems, states, and teachers; and the need for transparent reporting as value-added accountability is implemented in the states.

Andrew Porter begins by placing value-added accountability firmly within the context of the No Child Left Behind (NCLB) Act—emphasizing the place that it could play within the parameters of NCLB. Measuring annual student gains—and tying these gains to school, district, and teacher performance—is known as “value-added accountability,” he explains.

“It is important to realize the distinction between the provisions of accountability within NCLB,” he says, “and the concept of value-added accountability. NCLB doesn't focus on how much students are gaining, but on the level of their performance.

“Of course, under the provisions of NCLB, by the year 2014, 100 percent of students must be proficient. There is considerable emphasis in NCLB on school and district accountability, but not on teacher accountability.”

Porter summarizes the advantages of NCLB. “No matter how weak the students are, no matter how poor their background, the school has to get them to the same goal as everyone else.”

How Value-Added Accountability Differs

The concept of value-added accountability is quite different, Porter adds. “Instead of measuring student proficiency, it asks: How much do students gain? What part of the gain can be attributed to the school? Or to the teacher?”

He clarifies, “Currently, most of the focus is on school accountability. Since NCLB requires math and reading testing in every grade, three-eight, now people are asking how much gain there is per year—in addition to the level of student performance. In the past, since testing was done at only one grade, for each of the three levels of schooling (such as in the elementary grades) there simply wasn't any way to measure value-added.”

Porter explains, “The easiest way to think of value-added is to realize that when students progress through schooling, they improve in achievement, and then the question becomes: How much of that improvement in achievement can be attributed to the school or even the teacher?

“In the context of NCLB, I would use the two together—both the performance level and the value-added. A school that has reached proficiency for nearly 100 percent of its students might fail its adequate yearly progress (AYP). It's very difficult to get all students proficient. For that school, though, their value-added could be huge.

“The general idea,” he notes, “is how well the school does in terms of meeting its adequate yearly progress in increasing the percentage of its students who reach proficiency for subgroups of the student population. And at the same time, the missing part of NCLB is: How well is the school doing in terms of adding value? A school is performing adequately as long as it is moving decently on one or the other—or on both of those two criteria.”

Measuring Value-Added Accountability

Measuring value-added accountability quickly can become another thorny issue, Porter argues, although he emphasizes that the basic ideas are actually quite simple.

As he explains, if children are tested in the spring of the third grade, and tested again in the spring of the fourth grade with a different test, but that yields scale scores on a vertical scale, it is easy to compare the scores from one grade to another.

He notes, “While it can get technically very complicated, at its simplest it is helpful to think of it this way: If a student tests in the third grade and gets a 300 on the test, and tests again in the fourth grade and gets a 350 on the test, one can say the student gained 50 points. In another third-grade-fourth-grade situation, the students may have a 300-325 gain. These comparisons make sense.

“With data like that, one can form a simple idea of gain scores. How much did the students gain from third to fourth grade?”

The comparison is quite simple. “The students in one school gained 50 points on average,” Porter explains, “and the students in the other school gained 25 points on average. As a conclusion, one can say that there is more value-added in the first school than there is in the second school; some argue teachers can be evaluated in the same way.

“If in third grade, students get a high score, in fourth grade they have to get a yet higher score to show a value-added. Or if a student in third grade gets a low score, they don't have to get such a high score in the fourth grade to show value-added.

“This is very helpful if people control for third-grade performance,” he points out, “and it is helpful if a school has a very high student achievement average. One can ask: Is that because it is an effective school, or does it have good students in the first place? That is what value-added is trying to help decipher.

“If you have this baseline, people can say that first they will control on the basis of how good their students are and then they can see what value they added to the students they have, whether they were high achievers or low achievers.”

The benefit of controlling for prior achievement, Porter says, is to take into account prior achievement scores. “One can control simultaneously for their scores in first, second, and third grades, and then control for their scores in reading as well as mathematics. Bill Sanders's approach does that. It is similar to a gain score but technically not the same as a gain score because it is not just a baseline on math in the prior years, but it is a baseline on math in all prior years—plus a baseline on achievement in all the other tested subjects.”

This explanation, to Porter, leads to the large question: Should there be a control on a measure of socioeconomic status—such as the level of the mother's education? Controlling for socioeconomic status (SES) quickly becomes a bramble patch, he adds. “If I control for all prior achievement, some of that will control for SES because SES and student achievement are correlated, but it could conceivably not control for all SES effects. If I have a school or teacher that has, on average, higher SES students, I might expect them to gain slightly more than the school that has lower SES students. For that reason, if one wants to estimate the school's value-added, perhaps one wants to control for differences in SES as well.

“However,” he persists, “the counterargument is this: Does one hold schools to lower standards because they have students with lower SES? Which argument is correct?”

He answers himself. “I don't know. This is a debate that has been going on my entire professional career. Value-added is a new name, but it's not a new procedure. It's been around a long time.”

In addition, there is no perfect way to measure value-added, he says. “Every way of doing it carries some assumptions, and the assumptions won't be exactly right.”

School Districts and Value-Added Accountability

How should school districts deal with the complicated aspects of measuring value-added accountability—particularly in the absence of psychometricians with sophisticated knowledge?

“These decisions are made at the state level rather than at the district level,” Porter explains, “which is good because states have more capacity on average. Most states have a technical panel that can look at a district's plans and critique them. They can evaluate what makes sense, what doesn't, what's psychometrically and statistically sound, and what isn't.

“If a state purchases their testing—which they do—and also purchases the analysis of their data for value-added, they should be purchasing a procedure that is open to scrutiny. It absolutely cannot be a secret procedure.”

Porter underscores this as an especially critical point. “Results must be reported. The model ought to be laid out so that experts know exactly what was done. It ought to be laid out so that results can be replicated. It ought to be laid out so that other approaches can be used. It must be constructed, compared, and contrasted to the approach that was used. All of that should be part of the state's agreement with the vendor.”

Mobility and Value-Added

How should states and districts control for mobility when they conduct assessments for value-added accountability? Should students be enrolled in one school or district for a minimum period of days in order to be tested?

“In NCLB,” Porter says, “there is a statement that the student must be in school a minimum number of days. I've always believed this was not a good provision because some of the students we want to hold the schools accountable for are the students who are the most mobile.”

Yet, while students are disaggregated by ethnicity, race, migrant status, and English language learner (ELL) status in order to measure their progress toward AYP, there is no provision for transience within or outside a district within the same state. As part of his own work in large urban school systems, he points to the case of Philadelphia, where if a student was enrolled in two different schools in one year, both schools were given partial credit for the child, based on the number of days the student was enrolled in each school.

“In order to calculate a gain on a child at the individual level, one must have two data points for each student. If a data point is missing, the gain score cannot be calculated.

“That's a problem,” he continues, “but missing data can be estimated by using the entire data set across students, grades, and subjects tested. In this way, the schools can be held accountable for the students who are highly mobile.”

He adds, “There isn't any perfect way to do that, but it's not acceptable to throw these students out of the equation.”

Another problem that is causing some consternation, Porter reports, is the whole question about student IDs. “Many states, curiously enough, did not have school IDs when testing in grades 4, 8, and 10. They couldn't connect a student's fourth-grade score to that student's eighth-grade score.

“But in order to test for annual value-added, most states are moving to student-level IDs, and there are many arguments about whether social security numbers can be used, and also many arguments about the security of data. This is a big cloud in the background that is requiring states to move into a new arena, but even extremely conservative states are moving into what now is called a student-level database.”

Although value-added accountability presents a thicket of considerations and complications for researchers, psychometricians, and panels of experts across the nation, Porter believes in it. “Despite all the problems, am I against it?”

He answers himself. “Absolutely not.” He says with some humor, “I believe it has something to add.”

Anne Turnbaugh Lockwood is a senior program advisor for NWREL’s Center for Classroom Teaching and Learning. She holds a Ph.D. in educational psychology from the University of Illinois at Urbana–Champaign, and is the author of six books and more than 100 articles on educational issues.