Activity 3.6

How to Critique an Assessment

Purposes:

  1. To practice determining how well an assessment aligns with curriculum and instruction

  2. To build skill using the Chapter 3 guidelines for judging the quality of alternative assessment instruments: identifying content and skill coverage, the quality of performance tasks and criteria, equity and fairness, the consequences of assessment, and cost effectiveness

Uses:

This is an advanced level task. It can be used in Chapter 3 to practice the skills needed to be a critical consumer of alternative assessments.

Prerequisites include: knowledge of the characteristics of a quality assessment (Chapter 3 text, or Activity 1.5—Clapping Hands, Activities 3.1—Performance Tasks, Keys to Success, and 3.3—Performance Criteria, Keys to Success); understanding the role of performance assessment in instruction, learning, and assessment (Chapter 2 text or Activities 3.2—Spectrum of Assessment Activity and 2.1—Sorting Student Work).

Rationale:

Using alternative assessment in the classroom does not automatically ensure good assessment. Students have a greater chance at success when learning goals, instructional design, and assessment methods complement each other. Even when instruction, assessment, and curriculum are aligned, it is possible to write assessment tasks that are flawed. The goal of this activity is to have participants discuss the quality of the Aquarium Problem (Appendix A—Sampler, Sample A.15) and whether it aligns to the curriculum content and the instruction modeled.

Materials:

Time Required:

1-2 hours; optional extension requires an additional 30 minutes

Facilitator's Notes:

  1. (5-10 minutes) Introduction

    a. Review purposes as needed (A3.6,O1—How to Critique an Assessment Purposes). You might note that the general procedure is to discuss the importance of quality assessment and how good assessment instruments align to the curriculum and the instruction modeled. For example, if students are taught a concept using problem solving, then the assessment should reflect the problem-solving skills practiced during instruction.

    This activity also gives participants practice at reviewing assessment instruments for quality and alignment. Teachers do not have time to develop all the assessment instruments they need, so they need to be critical consumers of those developed by others.

    Then say:

    We are going to critique an assessment and our job is to decide if this is a good assessment instrument. (See Overhead A3.6,O2—Critiquing a Performance Task)

     b. Explain to the participants that a teacher has just finished a unit of study designed to build the following skills (learning targets): mathematical operations, problem-solving skills, communication skills, and data interpretation skills covered (see Overhead A3.6,O3—A Teacher Wants to Assess…).

    c. The teacher received the Aquarium Problem (Appendix A—Sampler, Sample A.15—Aquarium Problem) from the New Standards Project and he/she wants to determine if this would be a good assessment instrument to use with his/her students to measure the skills. Remind the participants that teachers do not necessarily need to develop every assessment instrument they use; they should beg, borrow, and steal assessments. Doing this requires critical review of the assessments of others to make sure the instrument is usable for their specific needs.

  1. (5-10 minutes) Introduce the aquarium task (Appendix A—Sampler, Sample A.15—Aquarium Problem.

    a. Let participants know that the Aquarium Problem is from the New Standards Project from the University of Pittsburgh and is used with fourth grade students.

    b. Have participants look at the aquarium task—the problem, the scoring guide, and the five student responses. Participants should work the problem. You might ask if anyone has any questions about the task.

  1. (5-30 minutes) Have participants review Handout A3.6,H1—Alternative Assessment Evaluation Form and point out the issues addressed. Ask the participants to read over the form and see if they have any questions about the criteria. It's best if participants have been introduced previously to notions of quality. (See prerequisites on page 1.)

  2. (15-30 minutes) Critique

    a. You may use this as an individual activity, but it is recommended that participants are divided into five discussion groups. Each group could critique the assessment using all the criteria on the evaluation form, or each group could focus on a different section of criteria. Using Overhead A3.6,O4—Critiquing the Aquarium Problem, review their task with the participants:

In your discussion groups:

b. As the groups work, walk around the room and listen to the conversations within each group.

  1. (15-30 minutes) Discussion

    a. After all groups are finished with the critiquing process, give each group time to report their findings.

    b. Once the group report is complete, allow the other groups to ask questions or add additional comments to the presenting group's findings.

Be sure all the following points are covered:

  1. (5 minutes) Summary

    Overhead A3.6,O5—Summary. Ask participants if they have any more discussion or issues to address about critiquing an assessment instrument. Reiterate the major points of the session.

    a.

  2. It is important to have experience at critiquing assessments prior to using them with students.

    b. It is okay to change or modify instruments.

    c. There are guidelines available to use for critiquing assessment instruments; however, particular circumstances might require emphasis on different things. Teacher groups could design their own critiquing instruments using the criteria they feel are most important.

    d. Good alternative assessments cover the desired content and skills, have good performance tasks and criteria, are equitable and fair, lead to positive consequences, and are cost-effective.

    Optional Extension

    The Fairness and Rater Bias section on the alternative assessment evaluation form (Handout A3.6, H1) usually creates the most concern when discussing the Aquarium Problem. The issue is that the task context may not be equally familiar to all students. Many times teachers to not realize the need to discuss assessment tasks with others to determine if they are fair for all students. This may leave them feeling isolated and frustrated. The video mentioned in the reference box can be used to address these concerns.

    Reference Box

    The video Effective Assessments—Making Use of Local Context is available from WestEd, 730 Harrison Street, San Francisco, California 94107, (415) 565-3000.

    Use video section ONE—A Fish Story. It shows how a group of Navajo teachers changed the aquarium problem to better suit the cultural background of their students while still assessing the same skills.