Activity 1.8

Sam's Story: Comprehensive Assessment*

Here's My Best Work

Purposes:

  1. To understand the importance of using multiple measures to get a more complete picture of a student

  2. To understand that each form of assessment has strengths and weaknesses

  3. To understand the need for alignment of outcomes, instruction, and assessments (we must match student achievement targets to assessment methods)

  4. To understand the concepts of "sampling,""target-method match," matching purposes to assessment plans, and "validity"

Uses:

This is a good introductory activity to use in Chapters 1 or 3 to illustrate the ideas of "sampling," "validity" (bias and distortion), "target-method match," and alignment between assessment and instruction.

Rationale:

Participants may have heard rhetoric that suggests that one form of assessment is better than another in all circumstances and for all purposes, and that a single assessment should yield all the information needed. Actually, no single type of assessment can yield all the information needed for all users and uses. And, different types of assessment methods are better suited to gather information about different types of student achievement targets (see Chapter 1, Key 3—Target-Method Match) or for different purposes (see Key 2—Clear and Appropriate Purposes). In this activity, participants will gain perspective on the strengths and weaknesses of various types of assessments.

Materials:

Time Required:

40-60 minutes

Facilitators Notes:

  1. As needed, use A1.8,O1—Sam's Story Purposes to introduce the activity.

  2. (5-10 minutes) Ask participants to read the handout (A1.8,H1—Sam's Story) and complete the chart individually. This task requires participants to rate how much they would trust each piece of assessment evidence as:

    • A sole measure

    • One of several highly relevant measures

    • Not directly relevant, but might be useful

    • Wouldn't tell much that's useful

    Ask participants to note the reasons for their choices.

  3. (5 minutes) When participants are done, reassemble the group. For each of the 10 items, ask participants to hold up their hands to indicate how they'd trust each item. Record numbers in each box. This helps to illustrate the variability of opinion based on knowledge and experience.

    Indicate participant responses on the Overhead A1.8,O2—Sam's Story Summary Chart by drawing a line though the category(ies) that represent the majority of responses.

  4. (10 minutes) After going through all 10 items, point out (if the case) that there's nothing they'd trust as a sole measure of Sam's performance. Why is that? (Answers may include: need to know what each assessment is measuring, need to know how the assessment relates to the class curriculum, need more examples of Sam's work in math, the different tests may be measuring different things.)

    Look at the measures that were rated the best. What do these have in common? (Possible answers: they're all closely related to the target that we were making a decision about; they include multiple samples of work; they are "authentic.")

    Look at the measures that were rated the lowest. Why did they get such low ratings? (Possible answers: they may be biased, marginally relevant to the decision at hand, not "authentic" enough, or not comprehensive enough.)

  5. (20 minutes) Note that appropriate educational decisions depend on the accuracy and comprehensiveness of educational assessment. Inaccurate or partial information may cause harm. Ask participants for examples of how inaccurate or partial information may cause harm to students.

    Tie to the concept of sampling (Chapter 1, Key 4—Bias and Distortion): When we try to determine Sam's performance in math, it's obviously impractical to see how well he can solve every possible problem there is. Nonetheless, you as a teacher want to get an accurate idea of how well Sam can handle the full array of possible tasks that Sam might be expected to encounter. That full array of possible tasks is called a "domain of knowledge."

    Since it's impossible to see how well Sam can perform on the entire range of math problems, we have to select a "sample" of math problems from the entire assessment domain. We try to sample the contents of a domain in a representative fashion so that, based on how well students do on the sample, we can make inferences about the students' status with respect to the entire domain.

    Ask participants which items on the list "sample" best. (Answers: in-class math tests, portfolio.)

    Tie to the concept of validity (Chapter 1, Key 4—Eliminating Sources of Bias and Distortion): Validity has to do with the extent to which we can make good decisions such as:

    • Are we looking at the right information?

    • Are we measuring the right thing?

    • Are we paying attention to things that aren't really relevant?

    • Are we inadvertently measuring something else?

    Ask participants which items on the list they feel are the most valid for the purpose of making a mastery decision? Why?

    Tie to the concept of matching method to purpose (Chapter 2, Key 2—Clear and Appropriate Purposes): Note that the reason the teacher wanted information about Sam was to "decide about Sam's mastery of the content and to design an instructional plan." Ask teachers if one's decisions about which evidence to trust would be different if the purpose were different, for example, to decide if Sam were working up to potential? In that case, what measures might be best?

  6. (10 minutes) Conclude the discussion by asking the following questions of the group. To make a decision about Sam's mastery of math and to design an instructional plan for him, what additional information do you need to know:
    • About Sam?
    • About his last math teacher and that teacher's course content, instructional strategies, and grading criteria?
    • About the tests used to provide these results?