---

By Melissa Steineger

The success of charters hinges largely on their ability to show results. A recent seminar in Idaho helped a group of charter school faithful find their way toward solid assessments.

BOISE, Idaho—They were shut inside a drab, windowless conference room for two snowy days in January. The dozen charter school staff members and policymakers nevertheless traveled far in their journey toward accountability. And, judging by their careful attention and comments along the way, these public school pioneers welcomed the trip — even though it included a two-hour presentation by a statistics expert; an afternoon of humbling discoveries about their shortfalls as appraisers of student writing; and a morning of lessons on the deficiencies of their portfolio assessment. That's because the Idaho Department of Education seminar on Assessment, Portfolios, and Data Analysis could, in the end, prove critical to the success of their charter schools.

Charter schools across the Northwest and throughout the nation are struggling with two crucial-and related-issues: assessment (measuring student achievement) and accountability (measuring the school's performance as a whole). Without one, you can't have the other. Yet, how do you determine if you're really doing well in either? That's what charter school founders, staffers, parents, and authorizing agencies are struggling with across the region.

In theory, charter schools are given more autonomy than other public schools because they're held more rigorously accountable for substantiating student performance. To do this, charter schools set forth goals in their agreements with the authorizing agency. The authorizing agency then measures the school's performance against those stated goals to determine whether to continue the school's charter. If a charter school does not demonstrate solid performance, the authorizing agency can close the charter school's doors.

The problem arises when a charter school's goals are vague. And the problem, it turns out, affects almost every charter school operating today. Indeed, unmeasurable performance goals have the potential for being the charter school movement's Achilles' heel.

An analysis by the state of Florida found that only six of 33 accountability agreements between the Sunshine State's charter schools and their authorizers contained any measurable goals and objectives. Studies by the Northwest Regional Educational Laboratory found that similar problems exist in Alaska and Idaho.

Solving the problem is crucial. Charter schools typically have a time limit of three to five years for achieving what they have promised. If authorizers don't see the benefits, they may decide to revoke a school's charter. The more a school can demonstrate quantifiably that it is making a difference, the better. Charter schools are well aware of the issue. An informal survey by Charter Friends National Network in early 1998 found accountability to be their top priority.

Yet, after almost a decade, educators and policymakers have still to agree on the nuts and bolts — by what methods should these publicly financed schools be judged?

One obvious way to measure charter schools, at least in the eyes of those outside the movement, is with state and national tests. Even some backers of charter schools say such tests complement the schools' aims because both the tests and the schools emphasize performance-based accountability.

But most charter schools see obvious problems with the world of high-stakes, nationally standardized tests. By relying on traditional tests, schools risk skewing instruction to match the tests. Or recreating the status quo.

Some charter advocates believe that attempts to tighten charter school regulation is an effort to rein in the entire movement. Others complain that standardized tests create a conservative influence on charter schools. Freedom from rules and regulations is supposed to give charter schools the latitude to innovate, so it would be a great irony if they had to use traditional tests to prove their worth and ensure their existence.

Yet, even if schools and authorizers can agree on the yardstick, setting performance targets for individual schools can be tricky. Schools must be able to build their own mission and goals based on their philosophies. But at the same time they must be able to assess progress toward their goals.

The problem with demonstrating achievement seems to be in large part due to difficulty in crafting data-based performance measures. Understanding the nuances of assessment and how schools could create their own assessment and accountability plans were the goals of the Idaho workshop, organized by Carolyn Mauer — who heads up the Idaho Bureau of Curriculum and Accountability in the state's Department of Education — in consultation with the Northwest Regional Educational Laboratory.

The road to meaningful assessment

The journey in Idaho started when the keynote speaker related the true story of a first-grader who had not yet learned to distinguish a T from a J — both letters were Ts to her eyes. One morning at the beginning of the school year, she took a 10-minute reading test and failed. That afternoon her parents were informed that the little girl would be moving into the special education program, despite the fact that she was doing well in all her subjects. After two days of hearing her parents discuss their efforts to have the child retested, the child told her father not to worry about it, explaining, "I'm just not very smart."

"I tell that story," explains the girl's father and workshop keynoter, Phillip Kelly of Boise State University, "to demonstrate the power of misused assessment. Anyone can use assessment to destroy a child's confidence. We must use assessment to build children's confidence."

For some students, Kelly says, assessment is about personal risk management. A child who is successful the first time gains confidence and encouragement. But imagine a child — perhaps intimidated by the school environment or simply shy — who fails the first time, second time, third time, and so forth. Why on earth, asked Kelly, would that child try his or her hardest on the 17th time, or after six years of failure, merely at a teacher's urging? That, he says, would actually be irrational. Such children would rather slouch in the back of the room, sneering, or simply tune out and eventually drop out than to be seen as failures. "For them," he adds, "assessment is about avoiding the embarrassment of being seen as incapable. If kids don't feel able to learn, there will be no learning."

The essential question, Kelly says, is how to help students want to learn. Used well, assessment can help children gain the ability to self-assess and the confidence to take risks and succeed. At the end of every video game, Kelly notes, the player "fails." "Yet, kids don't give up, they start over. Why? To better their score — in other words to raise their achievement." Video games, Kelly says, have clear goals and provide immediate feedback the player can use to measure his or her accomplishments. "Would video games be as popular," he asks, "if you finished the game and waited three months for the results to be mailed to you?"

For Kelly, the three key attributes of a good assessment tool are that it:

• Intimidates no one (because students know the target)

• Surprises no one

• Merely corroborates what kids already know (because kids understand what they will be assessed on and can reasonably predict the level of success they will achieve on the assessment)

Kelly cites the case of a high school English teacher who had students read an example of good writing and discuss why they thought it was good. The next time the class met, the teacher had students read an example of bad writing and discuss why they considered it bad. In the third class, the teacher had the students compare the two papers. Using the discussion, students developed their own scale of good, bad, and the steps in between. In the fourth class, the teacher asked students to write a paper that would be evaluated on the scale that they had discussed and developed.

That teacher, Kelly says, helped each student develop the understanding and insight to determine what was good and what was bad. Again, the teacher showed them that these were exactly the qualities they would be assessed on. That understanding of what they would be assessed on and how they would be measured gave students the confidence to take risks and succeed.

In other words, Kelly says, teachers can help students by using assessment not just as a checklist, but as a way for students to gain insight into how to improve. The English teacher, for example, could show a student that his or her writing did not contain the qualities the group had set as desirable — qualities that the student had a full and complete understanding of — and use that as a starting point for the student to improve his or her writing.

This type of good classroom assessment actually boosts test scores, Kelly says. A 1998 review of the effects of classroom assessment found that good classroom assessment provided the grade equivalent increase of one to four grades, with the biggest boost for low achievers. Yet, teachers rarely have training to develop high-quality assessments to inform future instruction. "The U.S.," Kelly says, "needs basic assessment literacy."

For starters, Kelly suggests that teachers — and schools — look at what they want to measure, whether it's content, reasoning ability, performance skills, or even values. Then, consider the best type of assessment tool to use, from multiple choice to interaction between teacher and student (see sidebar).

One form of assessment that is popular especially among charter schools is the portfolio method. Indeed, 75 percent of Idaho's charter schools use or plan to use student portfolios as part of their battery of assessments. But, in the words of Susan Seaman, a teacher at Renaissance Public Charter School, "You can call any collection a portfolio, but what is the quality of the contents?"

Renaissance Charter School in Moscow, Idaho, enrolls about 70 students in K-12 with an emphasis on the individual nature of student learning. Arts education, multiple intelligences theory, and holistic learning are part of the educational emphasis. Renaissance teachers are so interested in the topics of assessment and accountability that they came in for two days before the start of the school year to review the issues.

The education department's Mauer offers some criteria for using portfolios in assessing student work. The first step, she says, is to decide which subjects will be included in the portfolio and what the scoring criteria will be. Portfolios should incorporate the same content schoolwide, and ensure that teachers are requiring and scoring the same things in the same ways to ensure comparability and replicability. Portfolios should document a student's strengths, Mauer says, not his or her shortcomings. Portfolios should build confidence by demonstrating what students can do, not what they don't know. Finally, portfolios should be only one of multiple measures a school or teacher uses to assess student learning.

Tammy Emerich of Idaho's Meridian Charter High School near Boise presented examples of the portfolio method her school uses. At the beginning of each term, Emerich gives her students a list of things they must know by the end. Then instead of giving a final exam, she interviews each student to determine his or her true achievement level. "Life isn't multiple choice," she says. "It's a project-based world, so it's much more realistic to have students sit down and explain."

Writing is often included in portfolios. Lynette Hill, an English and language arts specialist at the state's education department, offers some tools for assessing student writing. Hill helped develop Idaho's direct writing assessment and is the state's writing competency course coordinator. Idaho conducts writing assessments in grades four, eight, and 11.

"Assessing writing is so subjective," says Hill. "Twelve experts would say 12 different things about one piece." Standards help reduce the subjectivity of scoring.

To help teachers use standards in assessing their students' writing, Idaho publishes booklets of the areas that will be measured in each of the grade levels that are officially measured, as well as three additional grade levels. The booklets contain the rationale for the testing as well as the writing terms and vocabulary students are expected to know and how they will be scored. Scoring grids list the five "grades" papers can receive, from "advanced" to "minimal," and what specific attributes each of those levels demonstrates. An advanced paper for a fourth-grader, for instance, is a uniquely developed topic with details related to time, place, characters, and plot using figurative language in a vibrant, consistent voice with varied sentence length, among other things.

No credibility without accountability

David Breithaupt, a research and evaluation analyst for the Idaho Department of Education, is an unabashed numbers zealot.

A buoyant man with a booming voice and a gusto for his topic, he strides through the conference audience, looming close to make a point at first one table, then another. He seems on the verge of grasping participants by the shoulders to heal them of their fear of statistics. Amazingly, he infuses even an after-lunch crowd with his arithmetical ebullience. While he orates, listeners' faces light up appreciatively.

"Successful assessment starts with measurable objectives," he booms. "A goal doesn't have to be measurable; an objective does." To drive home his point, he recites an objective lifted directly from the charter of one participant's school: "Demonstrate refined reading, writing, listening, speaking, and presentation skills in multiple forms of expression, using communication skills appropriate to the setting and audience." He pauses a moment to let the words sink in. As worthy as this objective may sound, he continues, "It is not measurable."

Begin crafting a measurable accountability plan, he says, by asking three simple questions:

• Where am I going?

• How shall I get there?

• How will I know whether I have arrived?

The answer to "Where am I going?" becomes your goal, he says. Next, look at where you are. The difference between the two is what you need to learn — or teach — to reach your destination.

Such goals are not necessarily measurable, he notes. So the school must next set measurable objectives to determine whether it is on the way to reaching its goals. Measurable objectives describe specific measurable tasks or steps to the goal that students are expected to achieve and answers the question, "How shall I get there?"

To be measurable an objective must pass the "Hey, Dad" test. "In other words," Breithaupt says, "'Hey, Dad. Let me show you how I can factor a polynomial.' This is measurable. 'Develop communication skills' is a worthy concept. But you can't show whether or not someone has 'developed communication skills,' so it's not measurable."

Three to four objectives per area of curriculum are easier to measure than an exhaustive laundry list, he notes. So, rather than "improve fourth-grade reading," a measurable objective might talk about vocabulary, comprehension, word-attack skills — in other words, the components of good reading. "And let me give you a tip," he adds. "Don't use the word 'and' in writing objectives because you can't measure two or more items well, such as reading and writing. If I demonstrate refined reading but not writing skills, did I pass? You can't tell."

Evaluation, he says, is simply what a student must be able to exhibit for a school to say the student has met the goal. He explains that all evaluation can be boiled down to a two-step process, whether you're evaluating fourth-grade reading or a new car model. First, you describe what you're looking at. Second, you judge. In educational parlance, "describe" is to issue a grade via the assessment tool you've chosen, and "judge" is to compare to your standards or goals.

After planning, teaching, and evaluating, it's time to report — to describe and disseminate the information to provide a foundation for additional and continued funding and as an aid for other schools to replicate success or learn from mistakes.

Jana Nichols of Meridian Charter High School has taught for 22 years in schools throughout the West and Midwest. That breadth of experience convinced her, she says, that "we could do school differently." After finishing an advanced degree, she interviewed for a position at Meridian, and the school's outside-the-box thinking clicked with Nichols' beliefs.

Meridian has some 150 students in grades nine through 11 and plans to add grade 12 in the 2001-02 school year, with a cap of 200 students. The school focuses on technology and offers career paths in computer networking, electronics, computer programming, and graphic design. Students' grades for the week are delivered to parents each Thursday, and youngsters have the opportunity during Friday study hall to either catch up or improve their classwork. If their grades are satisfactory, they can watch a movie or participate in a job shadow.

A consultant helped write the school's improvement plan objectives, which were clearly measurable. But when the improvement plan team tried on its own to write objectives for safety and discipline, they got stuck, Nichols reports. Breithaupt's presentation helped her start the process of rewriting the objectives in measurable terminology.

Trina Burns of Blackfoot Charter Community Learning Center also learned enough to begin rewriting her school's goals. Blackfoot, which opened its doors in August, has about 50 students in kindergarten through fifth grade. "Our charter doesn't have concrete goals — I've been struggling with that," Burns says. For example, one first-grade objective at Blackfoot is "read well." With the help of what she learned in the workshop, Burns expects to be able to rewrite that into a specific goal that will help her school not only demonstrate the students' learning, but also help kids learn better. Rewriting the goals will take time, she says. But in the long run, the effort is worth it.

Respond to this article