Chapter 4
Evaluating School-Linked Strategies

Why evaluate school-linked strategies?

How can school staff and their partners participate in evaluation efforts?

What Are the Steps in Evaluating School-Linked Strategies?

The word "Evaluation" can fill practitioners with dread if they have lived through the experience of having an outside evaluator descend on a school or program, collect a batch of data, and declare that the school or program is inadequate. But evaluation does not need to be a complicated or negative task. In fact, many people discover that their personal experiences in comprehensive strategies help them contribute to a realistic and thorough evaluation. They also realize that the evaluation process sharpens their awareness of a program's strengths and weaknesses and helps them think creatively about new linkages that could benefit children and families.

This guidebook views evaluation as:

• an activity that is done by and with families and other collaborative partners--not to them;

• a continuous process of gathering information about a program or strategies to periodically assess its progress toward shared goals--a tool for measuring how strategies are working and whether they are achieving desired results;

• a means of improving accountability to partners, children, families, communities, and funders; and

• a method for using information to modify and improve activities, strategies, and goals.

For materials that provide more in-depth discussion of the evaluation process, refer to the Suggested Resources section of this guidebook (Appendix B).

Why Evaluate School-Linked Strategies?

An evaluation tells partners whether their activities are achieving the results they want--not just at the end of an activity but also in the short term and during an intermediate period. Collaborative partners can use the evaluation process and the information it provides to:

• Build ownership of the comprehensive strategies among stakeholders. Partners and participants who are involved in evaluating comprehensive strategies become even more invested in their success. For example, the federal Even Start program encourages local partnerships to engage current and potential collaborators and policymakers in the evaluation process. These evaluations often attract community support and financial resources for the partnership.

• Guide program improvement. Evaluations enable programs to periodically reflect on their goals and measure their progress in achieving them. By comparing the activities that were envisioned to those that actually occurred--and by assessing the success of the activities that were implemented--a partnership may decide to refine or alter its programs or services.

• Build accountability. School-linked strategies require a major investment of resources and energy. Funders, policymakers, and other stakeholders want evidence their investments are producing better outcomes for children, youth, and families. Evaluation results can account for how resources were used and document program effectiveness to justify the costs of the effort. By showing that they are accountable for their investments, partnerships also show that they are responsive to the concerns of their supporters.

• Test innovative ideas. Because school-linked strategies often try new approaches--for example, building skills rather than simply treating problems--they must have some means of assessing which approaches work, and why. Clear feedback enables partnerships to build on the best ideas and discard ones that do not work.

• Support school reform and improve service systems. School-linked strategies strengthen reforms in schools and in human service delivery systems. Evaluations enable partners within these systems to pinpoint how each program enhances reform and to identify the strengths and weaknesses of connections among schools and agencies. This information helps partnerships align and coordinate multiple systems and partners.

• Ensure strong communication and organization. Evaluations help clarify communication patterns and roles among individual partners and agencies. This feedback enables programs within a comprehensive effort to improve their communication--and thus their overall effectiveness. Partnerships can also use evaluation results in organizing and structuring their activities to refine cross-agency strategies. For example, when the Savannah New Futures collaborative learned about the early failure of elementary school students, it moved to expand its focus to include school readiness activities (Center for the Study of Social Policy, 1995).

• Document the need for policy changes. Evaluation results that accurately portray the experiences of children and families participating in a program can help partnerships develop recommendations concerning policies and practices (Nissani & Nelson, 1995).

Tips for Taking Action: Preliminary Questions for Evaluators

What do we want to learn about the comprehensive school-linked strategies?

What will be different for children and families when the strategies are implemented?

What will be different for teachers, health and human services providers, and other front-line practitioners when the strategies are implemented?

How will we know the school-linked strategies are operating effectively?

How Can School Staff and Their Partners Participate in Evaluation Efforts?

In the past, some evaluations involved school or program staff only in a limited way--perhaps by assigning them the task of collecting specific data (Shaw & Replogle, 1995). This approach does not take advantage of all partners' potential to enrich evaluation (Gomby & Larson, 1992). School staff and their partners must be involved in evaluation efforts; otherwise, these key stakeholders may feel that they--not their program--are being evaluated, and that the evaluation is an ordeal imposed on them by an outside authority (Shaw & Replogle, 1995). When all partners work together to identify the purpose of an evaluation, design measurements, and interpret findings, they all learn how school-linked strategies make a difference in the community and they all develop a sense of ownership.

Tips for Taking Action: Roles for Partners in an Evaluation

School staff and their partners can:

• identify the assumptions about how people and conditions change that guide the comprehensive strategies;

• connect the evaluation to these assumptions;

• determine the purpose of the evaluation;

• design tools and measures for gathering useful information;

• share expertise and knowledge about children and families;

• offer insights about how comprehensive strategies are working and how their effectiveness can be measured; and

• suggest ways to modify and improve programs and strategies, based on evaluation information.

What Are the Steps in Evaluating School-Linked Strategies?

A useful evaluation of school-linked strategies involves:

1. identifying the purpose of the evaluation, the information needs of stakeholders, and the assumptions about change that guide practitioners and programs;

2. identifying the short-term, intermediate, and long-term indicators of progress and results that will produce useful information;

3. selecting techniques for gathering information;

4. developing a system for maintaining records and other information; and

5. developing a feedback loop for using evaluation information to improve school-linked strategies.

A description of these steps follows.

Identify the Purpose of Evaluation, the Information Needs of Stakeholders, and the Partnership's Assumptions About Change

Start by thinking about the goals that your partnership chose to guide its efforts. What were your goals, and what assumptions did you make about the best way to bring about change? What strategies did your goals encourage you to adopt? How do the strategies connect to the desired results expressed in your goals? The answers to these questions will suggest types of information you can collect that will help you evaluate your partnership's success.

"Agreement on a common set of
goals and outcome measures not
only makes collaboration easier, but
also helps promote a community-wide
'culture of responsibility' for
children and families, and fuels the
momentum for change."
--Lisbeth Schorr in Making a difference: Moving to outcome-based
accountability for comprehensive service reforms, (1994)

Next, think about the information needs of each stakeholder. Parents and families need to learn what comprehensive strategies offer them and how they can contribute to the community. School leaders need to learn how comprehensive strategies fit into the broader picture of educational improvement and student achievement. What links exist between the comprehensive strategies and school reform goals and approaches? What changes have occurred that might influence school policies? Health care providers want to know if the comprehensive strategies are improving access to health care for children and families. How many children are receiving health services, and of what type? What patterns of service use have been identified? In response, how has the partnership modified its approach? Community organizations want to know what impact the strategies are having on community capacity-building and revitalization.

Each of these information needs will suggest different types of evaluation approaches. Your evaluation may not be able to satisfy every information need, but you will want to respond to the needs of as many stakeholders as possible.

Finally, clarify the goals and expected results of your evaluation. Again, this process will help determine the information you will need to collect. The purpose of evaluation should be determined by drawing on the insights of diverse stakeholders.

Identify Useful Indicators of Progress and Results that Meet Information Needs

A good evaluation doesn't wait until a program is "finished" and then try to measure final results; instead, it is an ongoing process that measures change at many stages. You will want to choose indicators that measure:

• short-term accomplishments;

• interim or intermediate progress; and

• long-term results.

Short-term indicators measure what actually happened within the comprehensive effort: whether resources were generated, staff were hired, programs were implemented, and services were provided. Interim indicators measure what happened as a result of the strategies--for example, parents read to their children more frequently (an interim result) because they attended a school-sponsored parenting class (a short-term result). Long-term results measure the changes linked to interim results--for example, whether a child's reading ability improved because his or her parent read aloud at home. Evaluations also reveal the relationships between these different types of indicators.

Be aware that some conditions, such as high mobility of families, can make it hard to measure some indicators of progress. Be creative and selective as you choose indicators and methods of measuring change.

Select Techniques for Gathering Information

The approaches your partnership uses to collect information and to document and measure progress will depend on the purpose and design of your evaluation. A discussion of key issues follows.

What information will be most meaningful? Traditional evaluations collect quantitative and qualitative information. Quantitative information includes numerical measures, trends, and statistics. This type of information comes from attendance records, intake and eligibility forms, census reports, or state data sources. Evaluators also create questionnaires or surveys to produce their own quantitative information.

Qualitative information is more descriptive and subjective than quantitative data. Common sources are interviews, observations, and focus groups with staff and program participants. Evaluation teams often turn to qualitative data to explain trends they have found in quantitative information. For example, if records show that a new counseling center is being underutilized, interviews with families and staff may reveal why.

Both types of data are essential to evaluation because they can be used to explain and interpret each other. By using both, your partnership will build consistency and be able to assess your progress accurately. Again, different kinds of information will be meaningful to different stakeholders. A school principal may be impressed by information that more parents are reading to their children; a parent, however, might be more interested in the fact that more children feel safe walking to and from school than in knowing that the majority of parents gave high ratings to a literacy class. And some information will be meaningful to the whole partnership--especially if it can be used for continuous improvement.

Using a variety of techniques to gather several types of information will give your partnership insight into many community perspectives. For example, interviewing people to collect stories about community history not only yields useful knowledge, it provides information that is more meaningful to many community members than statistical measurements. Collecting stories also helps involve families, teachers, and other front-line stakeholders in the evaluation process. Their stories give insights into the paths families follow as they participate in comprehensive strategies and help partners better understand the connection between expected program results and the realities of families' experiences.

Which methods will capture the information you need? Many of the methods for gathering information described in Chapter 3 (Community Assessment) are also used for evaluation. These include:

• structured interviews with children, school staff, and other collaborative partners;

• surveys and questionnaires of participants;

• documentation of families' stories;

• focus groups;

• self-reporting and feedback forms completed by school staff and their partners;

• review and analysis of existing sources such as school and agency records and demographic profiles of the community; and

• observation of participants.

Consider how the methods you choose will be received by families, staff, partners, and the community. Will they be meaningful to them? Again, make sure that the language used in forms or interviews will be understandable, culturally appropriate, and respectful and that providing or collecting the information will not place undue burdens on families, school staff, or service providers. How will all stakeholders use the evaluation findings in a way that clearly supports and benefits the community?

Many school-linked partnerships are moving away from evaluations based on purely quantitative models and experimental designs (see Shaw & Replogle, 1995; Wagner et al., 1994). Instead, these partnerships document how programs operate, how families experience the programs, and what short- and long-term benefits result from the programs. Short-term and interim results indicate whether progress is being made toward the long-term goals. Long-term results indicate whether children and families have benefited from the comprehensive strategies in the ways specified by the partnership's goals--that is, whether the participants' lives or community conditions have changed as planned. If your partnership's efforts do not produce the short-term results you sought, study the information you collected to find out which factors may be inhibiting progress. Adjust your strategies and continue to assess short-term results to make sure your efforts are on track.

Linking Measurements and Results

In order to measure progress and accomplishments, you must clearly understand your partnership's goals and assumptions about change. For example, a partnership with the goal of improving academic performance may assume that an increase in student attendance will lead to improved academic performance--and that a school-based health clinic may help increase student attendance by treating the illnesses that keep children out of school. What short-term outcomes will indicate that the strategies are on track?

One indicator may be a reduction in absences caused by illness. If school attendance increases in the short term, the partnership might expect a long-term result of improved school performance. On the other hand, if school attendance does not increase in the short term, partners would not anticipate that school performance would improve.

Develop a System for Gathering Information and Maintaining Records

Evaluations require time, human and financial resources, and a structure that supports information gathering and record keeping. Without a system for collecting and managing information, evaluation responsibilities may eat into the schedules of school and agency staff and detract from the quality or quantity of their activities. A good evaluation neither overburdens teachers, service providers, and program participants with paperwork nor produces so little data that there is nothing to analyze.

Many schools and agencies have record-keeping systems in place that can be modified or revised for an evaluation of comprehensive strategies. Successful evaluations use the following guidelines to design or revise record-keeping systems:

Simplify information collection methods. School and agency staff are much more likely to complete a half-page checklist than a five-page, handwritten reporting form. If the technology is available, you can save time by having staff type information directly into the computer system instead of using written forms.

Create flexible procedures. In communities with low literacy rates, many parents who participate in comprehensive strategies may need to enter data verbally into a tape recorder rather than writing reports.

Build a two-way flow of communication. When the partners who collect information have easy, frequent access to the individuals or groups analyzing it, a feedback loop is created that informs all stakeholders.

Ensure confidentiality. It isn't easy to create a two-way flow of data and at the same time respect the confidentiality of participants. Make sure your partnership develops effective methods for maintaining confidentiality when it plans an evaluation.

Document the stories of some families. School and agency staff can provide valuable information by describing the path that some families follow as they work with a program and progress toward their goals. This information puts a human face on facts. Ask staff to explain the referrals they make and comment on families' conditions and concerns.

Review and refine record-keeping efforts. Few information systems are perfect. Sometimes programs have to gather data from past records or reconstruct a record of services; other record-keeping systems capture inadequate information or fail to produce the information needed for evaluation. These imperfections can be worked out over time if partners continually assess the changes and identify what works well in the system--and what needs to work better.

Develop a Feedback Loop for Using Evaluation Information to Improve School-Linked Strategies

As evaluation results become available, your partnership can set up a system for sharing and discussing the information so that all partners understand how programs and services relate to goals and visions. A feedback loop also enables partners to use information to make informed decisions.

Share the findings with all partners; they need to hear how their participation has made a difference for children and families. In particular, ask school and agency staff to read your evaluation findings and prepare recommendations based on their insights. Their first-hand experience can help them identify changes in practice that directly address any issues that have surfaced. Set aside time at partnership meetings to discuss evaluation findings and brainstorm improvements. And disseminate evaluation findings to the larger community to build support for school-linked strategies and increase awareness about community conditions.

Learning Opportunities

When partners help evaluate school-linked strategies, the evaluation process itself becomes a learning experience. To "demystify" evaluation, make sure that:

• any person from outside the partnership who serves as an evaluator is comfortable working and talking with parents and community members;

• all members of the partnership have an opportunity to discuss what they want to learn from the evaluation and to understand how the data they are collecting will generate the information they want;

• partners have an opportunity to discuss evaluation findings with the person who coordinated the evaluation, so they understand what the findings mean for the program and the community; and

• partners think about what, if anything, the partnership will do differently because of what it learned from the evaluation.

Finally, remember that teachers often play a central role in an evaluation by reporting information on student outcomes, such as improvements in completing homework, following directions, and staying on task (Illback, 1993). To minimize their workload, make sure that teachers develop efficient record-keeping procedures and have adequate time to complete the task.

Summary

Evaluations help build ownership of strategies among partners, guide program improvement, build accountability, test innovative ideas, support school reform, ensure strong communication and organization, and document the need for policy changes. School staff and their partners should be involved in each of the following steps: (1) identifying the purpose of evaluation, the information needs of stakeholders, and the partnership's assumptions about change; (2) identifying useful indicators of progress and results that meet information needs; (3) selecting techniques for gathering information; (4) developing a system for maintaining records and other information; and (5) developing a feedback loop for using evaluation information to improve school-linked strategies.

Table of Contents | Next Section

This document's URL is: Home | Up & Coming | Programs & Projects: Child and Family Program | People | Products & Publications | Topics © 2001 Northwest Regional Educational Laboratory Email Webmaster Tel. 503.275.9500