Activity 3.6

How to Critique an Assessment

---

Purposes:

1. To practice determining how well an assessment aligns with curriculum and instruction

2. To build skill using the Chapter 3 guidelines for judging the quality of alternative assessment instruments: identifying content and skill coverage, the quality of performance tasks and criteria, equity and fairness, the consequences of assessment, and cost effectiveness

Uses:

This is an advanced level task. It can be used in Chapter 3 to practice the skills needed to be a critical consumer of alternative assessments.

Prerequisites include: knowledge of the characteristics of a quality assessment (Chapter 3 text, or Activity 1.5—Clapping Hands, Activities 3.1—Performance Tasks, Keys to Success, and 3.3—Performance Criteria, Keys to Success); understanding the role of performance assessment in instruction, learning, and assessment (Chapter 2 text or Activities 3.2—Spectrum of Assessment Activity and 2.1—Sorting Student Work).

Rationale:

Using alternative assessment in the classroom does not automatically ensure good assessment. Students have a greater chance at success when learning goals, instructional design, and assessment methods complement each other. Even when instruction, assessment, and curriculum are aligned, it is possible to write assessment tasks that are flawed. The goal of this activity is to have participants discuss the quality of the Aquarium Problem (Appendix A—Sampler, Sample A.15) and whether it aligns to the curriculum content and the instruction modeled.

Materials:

• Overhead projectors, screen, overhead pens, chart paper, tape, and markers

• Overheads A3.6,O1—How to Critique an Assessment Purposes, A3.6,O2—Critiquing a Performance Task, A3.6,O3—A Teacher Wants to Assess…, A3.6,O4—Critiquing the Aquarium Problem, A3.6,O5—Summary

• Handouts A3.6,H1—Alternative Assessment Evaluation Form, Appendix A—Sampler, Sample A.15—Aquarium Problem

Time Required:

1-2 hours; optional extension requires an additional 30 minutes

Facilitator's Notes:

1. (5-10 minutes) Introduction

   a. Review purposes as needed (A3.6,O1—How to Critique an Assessment Purposes). You might note that the general procedure is to discuss the importance of quality assessment and how good assessment instruments align to the curriculum and the instruction modeled. For example, if students are taught a concept using problem solving, then the assessment should reflect the problem-solving skills practiced during instruction.

   This activity also gives participants practice at reviewing assessment instruments for quality and alignment. Teachers do not have time to develop all the assessment instruments they need, so they need to be critical consumers of those developed by others.

   Then say:

   We are going to critique an assessment and our job is to decide if this is a good assessment instrument. (See Overhead A3.6,O2—Critiquing a Performance Task)

   b. Explain to the participants that a teacher has just finished a unit of study designed to build the following skills (learning targets): mathematical operations, problem-solving skills, communication skills, and data interpretation skills covered (see Overhead A3.6,O3—A Teacher Wants to Assess…).

   c. The teacher received the Aquarium Problem (Appendix A—Sampler, Sample A.15—Aquarium Problem) from the New Standards Project and he/she wants to determine if this would be a good assessment instrument to use with his/her students to measure the skills. Remind the participants that teachers do not necessarily need to develop every assessment instrument they use; they should beg, borrow, and steal assessments. Doing this requires critical review of the assessments of others to make sure the instrument is usable for their specific needs.

2. (5-10 minutes) Introduce the aquarium task (Appendix A—Sampler, Sample A.15—Aquarium Problem.

   a. Let participants know that the Aquarium Problem is from the New Standards Project from the University of Pittsburgh and is used with fourth grade students.

   b. Have participants look at the aquarium task—the problem, the scoring guide, and the five student responses. Participants should work the problem. You might ask if anyone has any questions about the task.

3. (5-30 minutes) Have participants review Handout A3.6,H1—Alternative Assessment Evaluation Form and point out the issues addressed. Ask the participants to read over the form and see if they have any questions about the criteria. It's best if participants have been introduced previously to notions of quality. (See prerequisites on page 1.)

4. (15-30 minutes) Critique

   a. You may use this as an individual activity, but it is recommended that participants are divided into five discussion groups. Each group could critique the assessment using all the criteria on the evaluation form, or each group could focus on a different section of criteria. Using Overhead A3.6,O4—Critiquing the Aquarium Problem, review their task with the participants:

In your discussion groups:

• Decide if this problem will elicit the skills the teacher wants to assess.

• Using the alternative assessment form (Handout A3.6,H1), rate the quality of the assessment.

• When the group has finished critiquing the assessment, list (on chart paper) the ratings agreed on for the assessment and the reasons for them. Add the things the group liked about the assessment and ways the assessment could be improved. Post the chart paper on the wall.

• Decide on a spokesperson to report your results to the whole group.

b. As the groups work, walk around the room and listen to the conversations within each group.

5. (15-30 minutes) Discussion

   a. After all groups are finished with the critiquing process, give each group time to report their findings.

   b. Once the group report is complete, allow the other groups to ask questions or add additional comments to the presenting group's findings.

Be sure all the following points are covered:

• Did this assessment cover all the skills to be assessed? Did the task elicit the skills? Did the criteria assess them? What would you change?

• What are the possible sources of bias and distortion in this task?

• What could a teacher do with the results of this assessment?

• Would this instrument be worth the time spent by the teacher and the student?

• How could this task be modified to exhibit more authenticity?

• What type of curriculum coverage and instructional design was needed prior to this performance assessment?

6. (5 minutes) Summary

   Overhead A3.6,O5—Summary. Ask participants if they have any more discussion or issues to address about critiquing an assessment instrument. Reiterate the major points of the session.

   a.

7. It is important to have experience at critiquing assessments prior to using them with students.

   b. It is okay to change or modify instruments.

   c. There are guidelines available to use for critiquing assessment instruments; however, particular circumstances might require emphasis on different things. Teacher groups could design their own critiquing instruments using the criteria they feel are most important.

   d. Good alternative assessments cover the desired content and skills, have good performance tasks and criteria, are equitable and fair, lead to positive consequences, and are cost-effective.

   Optional Extension

   The Fairness and Rater Bias section on the alternative assessment evaluation form (Handout A3.6, H1) usually creates the most concern when discussing the Aquarium Problem. The issue is that the task context may not be equally familiar to all students. Many times teachers to not realize the need to discuss assessment tasks with others to determine if they are fair for all students. This may leave them feeling isolated and frustrated. The video mentioned in the reference box can be used to address these concerns.

   Reference Box The video Effective Assessments—Making Use of Local Context is available from WestEd, 730 Harrison Street, San Francisco, California 94107, (415) 565-3000.

   Use video section ONE—A Fish Story. It shows how a group of Navajo teachers changed the aquarium problem to better suit the cultural background of their students while still assessing the same skills.

   
   
   

   This document's URL is:

   Home | Up & Coming | Programs & Projects: Assessment | People | Products & Publications | Topics

   © 2001 Northwest Regional Educational Laboratory

   
   Email Webmaster
   Tel. 503.275.9500