Chapter 10: Types of Program Evaluation

Summary

This video from the Children's Hospital of Philadelphia explores different types of program evaluation, including formative, process, impact, and outcome evaluations. It emphasizes the importance of evaluation in improving program effectiveness and management. The video outlines when to use each type of evaluation based on the stage of the program and its goals. Additionally, it introduces various evaluation designs such as randomized experiments and pre/post studies, highlighting the importance of program fidelity and the role of both quantitative and qualitative methods in measuring impact. The video provides valuable insights into designing impactful evaluations.

Takeaways

• 😀 Program evaluation is essential for improving the management and effectiveness of organizations and programs.
• 😀 There are four main types of program evaluation: formative, process, impact, and outcome.
• 😀 Formative evaluation helps assess whether a proposed program is needed and if it's understood by the target audience.
• 😀 Process evaluation occurs during program implementation and helps identify any early issues that need fixing.
• 😀 Impact evaluation measures the program's effect on the target population's behaviors after it has been implemented.
• 😀 Outcome evaluation occurs at a set time during or after the program to assess whether program goals are being met.
• 😀 The type of evaluation design chosen should depend on the program’s purpose and available budget, with impact and outcome evaluations being common choices for stakeholders and funders.
• 😀 If the evaluation is for internal use, formative or process evaluations may be more appropriate to refine and optimize the program.
• 😀 Both qualitative and quantitative methods are used to measure program impact and outcomes, helping to understand short- and long-term effects.
• 😀 Fidelity in program evaluation answers how closely the program's results match the original plans or the original program it was adapted from.
• 😀 Various outcome evaluation designs include randomized experiments, quasi-experiments, pre/post studies, cross-sectional studies, panel studies, and time series studies.

Q & A

What are the four main types of program evaluation mentioned in the script?

The four main types of program evaluation are formative evaluation, process evaluation, impact evaluation, and outcome evaluation.

When is formative evaluation typically conducted, and why is it important?

Formative evaluation is typically conducted at the beginning of a program, either during the development of a new program or when offering a pre-existing program to a new setting or population. It is important because it helps determine if the proposed program elements are needed and how well they are understood, allowing for adjustments before full implementation.

What is the purpose of a process evaluation?

A process evaluation takes place during the implementation of a program and helps assess how well the program is functioning. It provides an early warning if there are any issues that need to be addressed.

What does an impact evaluation measure, and when is it typically conducted?

An impact evaluation is conducted after the program has made contact with at least one person, and it measures the degree to which the program is affecting the target population's behaviors and whether the program is meeting its specific, measurable objectives.

How does an outcome evaluation differ from an impact evaluation?

An outcome evaluation is typically conducted during the program’s operation at predetermined intervals or at the end of the program, and it measures the overall success of the program in achieving its goals. In contrast, impact evaluations focus more on short-term behavioral changes.

Why is the evaluation design focused on both impact and outcome evaluations for stakeholders?

Impact and outcome evaluations are often focused on for stakeholders and funders because they provide critical evidence regarding the effectiveness of the program, the achievement of program objectives, and help inform decisions on policy and funding.

What role does Fidelity play in program evaluation?

Fidelity in program evaluation answers the question of how closely the results of a program match the intended design. It helps assess whether the program was implemented as planned, and whether the actual outcomes reflect the original goals and activities.

What are some of the specific evaluation designs mentioned for outcome evaluation?

The script mentions several evaluation designs for outcome evaluation, including randomized experiments, quasi-experiments, before-and-after studies (pre-post studies), independent cross-sectional studies, panel studies, and time series studies.

How does the budget affect the choice of evaluation design?

The available budget plays a significant role in determining the evaluation design. A limited budget may restrict the ability to conduct comprehensive evaluations, whereas a larger budget may allow for more in-depth evaluations, such as impact and outcome evaluations.

What evaluation methods will be discussed in the following chapters?

The following chapters (Chapters 12 and 13) will discuss the qualitative and quantitative methods that can be used for program evaluation, focusing particularly on the approaches that work best for impact and outcome evaluations.