Northwest Regional Comprehensive Center
NWRCC
SEA Rapid Response
Evaluating State Accountability Systems
The documents and Web sites for this section come from Marianne Perie and Brian Gong at the Center for Assessment (http://www.nciea.org/). Marianne has done an extensive literature review of the topic and recommended these publications to review for standard criteria for evaluating accountability systems. Brian added this note in an email message to the NWRCC (March 2007):
Accountability systems always are designed to serve some purpose(s), and typically entail many interdependent design decisions. Most of these criteria do not provide direction on what the design decisions should be (e.g., what the purpose of the accountability system should be), nor do they address the details of trade-offs, which are often the most complex and important aspects of making an accountability system elegant and workable (e.g., how demanding should the accountability system be, in terms of what indicators and measurements are valued and available to the state?). Of course, criteria and rubrics are intended to specify qualities, not tell how to achieve those qualities, so a criterion might be, for example, "Evidence that the system produces acceptably reliable accountability decisions," but the criteria would not say how to achieve reliable decisions (and might not even say what is acceptable in terms of reliability since there are no professionally agreed upon standards). I think these criteria would be most useful when applied by someone who already knows quite a bit about accountability systems. If the state is trying to improve its system, not evaluate it formally, then two other resources might be useful: a) working through documenting the validity of the accountability system, and b) looking at other states' systems.
Standards for educational accountability systems [Policy Brief 5].
Baker, E., Linn, R., Herman, J., & Koretz, D. (2002). Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
This document puts forth 22 standards for educational accountability systems, divided into five domains: System components, testing, stakes, public reporting, and evaluation. These standards are intended to provide guidance to states and districts in developing or reviewing their own accountability systems. These standards represent models of practice from three perspectives: research knowledge, practical experience, and ethical considerations. [PDF]
A framework for examining validity in state accountability systems.
Forte Fast, E. & Hebbler, S. (2004). Washington, DC: Council of Chief State School Officers.
This CCSSO report lays out a framework under which one can examine the validity of a state accountability system, focusing on four core concepts: Goals, consequences, decision rules, and performance indicators. [PDF]
Designing school accountability systems: Toward a framework and process.
Gong, B. & ASR SCASS. (2002). Washington, DC: Council of Chief State School Officers.
This CCSSO report is meant to assist states in making decisions about designs for accountability systems. It lays out ten central questions that must be answered in any accountability design, and then explores possible directions states can go in answering the questions. [PDF]
Building a workable accountability system.
Ananda, S. & Rabinowitz, S. (2001). San Francisco: WestEd.
This brief for policymakers and educators identifies the questions that must be answered in designing or evaluating an accountability system. It summarizes the issues, provides options, and also identifies potential problems. [PDF]