Chapter 11: Measuring Program Impact and Designing Evaluation Instruments

Summary

This video provides valuable insights on measuring the impact of teen driving safety programs. It focuses on key performance indicators like attitudes, knowledge, skills, behavior, and self-efficacy, which help evaluate the effectiveness of a program. It highlights both quantitative and qualitative methods for data collection, such as surveys, interviews, and record reviews. The video emphasizes the importance of SMART objectives in program evaluation to ensure measurable and time-bound results. It concludes with practical examples of how to assess behaviors like distracted driving and seatbelt use, offering a comprehensive guide for evaluating program success.

Takeaways

• 😀 Performance measures are crucial for setting specific goals, monitoring progress, and communicating the value of a program to society.
• 😀 Key performance indicators (KPIs) track program progress over time and help compare performance across different sites, programs, and populations.
• 😀 Recommended KPIs for teen driver programs include measuring attitudes, knowledge, skills, behaviors, self-efficacy, frequency of behaviors, and behavioral intentions.
• 😀 Attitudes refer to stable beliefs or feelings, and measuring them helps assess how participants feel about a certain concept, person, or behavior.
• 😀 Knowledge measures assess the awareness and familiarity of participants with certain concepts, such as safe driving practices.
• 😀 Skills assessments focus on improving participants' expertise and ability to perform specific driving-related tasks.
• 😀 Self-efficacy refers to participants' confidence in their ability to engage in desired behaviors, such as safe driving practices.
• 😀 Frequency measures track how often certain behaviors occur, such as seat belt use or speeding.
• 😀 Behavioral intentions gauge the likelihood that participants will engage in specific behaviors, like using a cell phone while driving or speeding.
• 😀 Quantitative data collection methods include surveys, record reviews, GIS data, and environmental assessments, while qualitative methods include interviews, focus groups, and observations.
• 😀 When designing performance measures for distracted driving, examples of questions might include: 'How likely are you to intervene if the driver is texting?' and 'Do you support laws banning handheld phone use while driving?'

Q & A

What is the primary purpose of measuring program impact in the context of teen driving programs?

The primary purpose of measuring program impact is to track the progress of the program over time, compare how it operates across different sites and populations, and communicate its results and value to society. It also helps in setting specific goals, allocating resources, and monitoring progress.

What are the key performance indicators (KPIs) mentioned in the script for teen driving programs?

The key performance indicators include measures of attitudes, knowledge, skills, self-efficacy, frequency of behavior, and behavioral intentions of the participants.

How can attitudes be measured in a teen driving program evaluation?

Attitudes can be measured by understanding the stable beliefs or feelings that participants have about certain concepts, objects, or behaviors. This is often inferred through observing participants' actions or responses.

What role does self-efficacy play in teen driving program evaluations?

Self-efficacy refers to an individual's judgment of their ability to perform a certain behavior. Increasing participants' self-confidence can enhance the likelihood of them engaging in the desired behaviors, such as safe driving practices.

Why is it important to choose the right performance measures for your evaluation?

Choosing the right performance measures is crucial because they guide the evaluation process by focusing on specific aspects of behavior change, providing measurable goals, and ensuring that the evaluation reflects the program's true impact on participants.

What are the two main methods for data collection in program evaluations, and how do they differ?

The two main methods for data collection are quantitative and qualitative methods. Quantitative methods involve numerical data and help answer questions like 'how much' or 'how often,' while qualitative methods involve descriptive data and focus on understanding 'how' and 'why' participants behave in certain ways.

What are some examples of quantitative data collection methods in program evaluations?

Examples of quantitative data collection methods include surveys, record reviews, indicator data, geographical information system (GIS) data, and environmental assessments.

What are some examples of qualitative data collection methods in program evaluations?

Qualitative data collection methods include open-ended surveys, in-depth interviews, focus groups, participant diaries, and observations.

How can you measure distracted driving behavior in a teen driving program?

Distracted driving behavior can be measured by assessing the frequency of participants engaging in distractions like texting while driving, their attitudes toward distracted driving (e.g., support for laws banning phone use while driving), and their behavioral intentions (e.g., willingness to give up using a phone while driving if influenced by others).

What are SMART objectives, and how do they relate to program evaluation?

SMART objectives are goals that are Specific, Measurable, Achievable, Realistic, and Time-bound. They help in defining clear and actionable targets for the program, making it easier to evaluate its impact and progress.