Center for Research, Evaluation, and Assessment
Program Evaluation: An Introduction
This brief article is an orientation to program evaluation, written for program directors or managers who are thinking of hiring a consulting institution to provide evaluation services for a new or existing program or project. The intent is to provide a basic understanding of how you can benefit from hiring an independent evaluator, how an evaluation works, and how to work with an evaluator to get the best results.
Many administrative units or funding agencies require programs or projects to hire an external evaluator to provide credible, third-party observation of program impact, and this is why many programs hire evaluators - because it is a requirement. This relationship need not be adversarial, and in fact, evaluators can contribute a great deal to the success of a new or evolving program. For this reason, many programs contract for evaluation services as they would hire any other type of consultant - in order to bring additional expertise onto their management team, and gain insight and feedback from someone with experience and a fresh, independent perspective.
Program evaluation is a specialty field, with its own advanced degree programs and professional associations. However, experienced professionals in specific content areas often provide evaluation in those fields, particularly if they have a research and policy background, a professional service background in that discipline, and familiarity with program evaluation techniques. The Evaluation Program at NWREL includes specialists in program evaluation, educational measurement and research design as well as in the fields of language arts assessment, mathematics assessment, instructional systems technology, clinical and educational psychology, educational leadership, communication, statistics, business administration, information technology and information management.
Although there is no uniform agreement about the precise definitions of the terms, most evaluation professionals agree that there are two broad purposes for program evaluation and thus two "types" of program evaluation: formative evaluation and summative evaluation. In practice, most evaluations contain both types and there is no clear dividing line between them, however, the terms are conceptually useful in understanding what you get when you hire an evaluator.
Formative evaluation produces information that helps form and transform your program. For instance, you may have a detailed plan for how you want teachers to implement a new course, but in practice we may find that teachers don't follow your plan exactly and therefore the outcome (or lack of outcome) will not be a fair test of the course you designed, since the course isn't being implemented as you intended. Checking the fidelity of the implementation is one important aspect of formative evaluation; without it, you might never know that the course you designed wasn't really being delivered.
A more thorough formative evaluation of this situation might provide you with reasons why your design wasn't being implemented properly - perhaps inadequate training, unclear materials, inappropriate staffing, or unexpected structural barriers are preventing your program from taking root as you intended. Specific research on the training, materials, staffing, or context could give you specific information about how to make the necessary improvements. This kind of feedback, if provided early enough, can help you make adjustments and corrections to get the program on track.
Summative evaluation is generally produced for policy makers, funding sources or potential adopters of your program, and is aimed at delineating clearly the benefits generated by your program as well as the costs and essential conditions necessary to gain those benefits. By hiring an independent, external evaluator to study your program, the claims you make regarding benefits and costs can be supported by a credible, professional third-party observer. A professional program researcher can often provide guidance on what claims you can reasonably make for your program and who may benefit from adopting or continuing to use your program. In addition, thorough documentation of your program can provide a sound basis for launching new projects, seeking additional funding, or creating extensions or additions to your programs.
In order for this process to work effectively, program directors and program evaluators must work as a team throughout the process, ideally beginning during planning and development and before the implementation of a program or program modification. Often the collection of adequate information must begin before a program starts, so that we can determine what changes occur once the program begins. For both formative and summative evaluation, measurement and data collection tools and data management systems must be part of the basic foundational planning of the project. Moreover, proper selection or creation of these systems requires detailed, thoughtful conceptual mapping of the program's intended outcomes, the way that these outcomes will be reached by program inputs and processes, the milestones or benchmarks that must be reached along the way, and the timeline along which these elements will unfold. Building and implementing a strong evaluation plan starts with outlining a logic model or causal map of your project, and professional evaluators can work with you to facilitate this process, so that the workflow of your project becomes very clear and measurable.
Time spent in the beginning mapping out and developing these plans and measurement strategies will be rewarded later with useful formative feedback, increasing the chances of program success. Beginning with a well-developed plan and measurement model will also allow the production of thorough, convincing evidence of how program successes were achieved and how these successes might be continued or replicated. This is exactly the kind of evidence that program directors need to justify their efforts, and it is also the type of information policy makers need in order to make good decisions.
Bottom line recommendations:
- Your evaluator is your ally. Professional evaluation can help your program run more efficiently and more effectively, and can help you demonstrate to others exactly what your program is capable of doing.
- Evaluation should begin in the planning and early implementation phases of your program, project, or program modification - NOT AT THE END! Waiting until the middle or end of your program to work on evaluation will prevent you from gaining the benefits of formative evaluation, and will severely restrict the possibility that a summative evaluation will provide credible support for your claims of effectiveness.
- Data collection for evaluation and data collection for internal program management often overlap significantly, and you can save time and money by building a strong data management system, integrating both management and evaluation, at the outset of your program.
- Time spent carefully building a theory of how your program works and what kinds of information will be useful in tracking the program will not be time wasted. In fact, taking the time to build a thorough logic model or causal map of your program will result in many savings later - savings of time, money, frustration, and regret. Work with an experienced program researcher to develop your evaluation plan, and you will reap the benefits.