Assessment Bias

Summary

In this video, Dan Hickey discusses the critical principle of assessment bias, emphasizing the need for fairness in testing. He outlines two main types of bias: offensiveness and unfair penalization. Hickey advises avoiding stereotypes in assessments and using examples that might disadvantage certain student groups. He introduces the concept of disparate impact and differential item functioning (DIF) to detect bias. Hickey suggests a three-step approach for bias review, applicable to both high-stakes and classroom assessments. He also touches on considerations for students with disabilities and English language learners, stressing the importance of fairness and clear curricular aims in assessment design.

Takeaways

• 🔍 The goal of assessment is to achieve the absence of bias, ensuring fairness for all students.
• 🚫 Two main forms of bias to avoid in assessments are offensiveness and unfair penalization.
• 👨‍👩‍👧‍👦 Offensiveness can occur when assessment content is offensive to a subgroup of students, often seen in instructor-developed assessments.
• 🌐 Unfair penalization happens when test content puts a subgroup at a disadvantage, which is harder to detect than offensiveness.
• 📚 An example of unfair penalization is using culturally specific content that is unfamiliar to certain students, like international students.
• 🔄 Disparate impact refers to different ethnic groups performing differently on an assessment, which should be examined for potential bias.
• 🔍 Differential item functioning (DIF) is a technique used to detect if an item is behaving unexpectedly for a subgroup of students.
• 👥 A bias review panel can be assembled to identify and eliminate bias in assessments, with members from diverse backgrounds.
• 👨‍🏫 For classroom assessments, bias detection can be scaled down to increasing instructor awareness and careful item review.
• ♿️ Accommodations must be made for students with disabilities during testing, which requires specific knowledge and expertise.
• 🌐 Assessments should be designed to be fair and not just 'politically correct', focusing on clear curricular aims without bias.

Q & A

What is the main principle discussed in the video script?

The main principle discussed in the video script is assessment bias, which refers to the absence of unfair advantage or disadvantage in assessment practices.

What are the two forms of bias that can be found in assessments?

The two forms of bias mentioned are offensiveness and unfair penalization. Offensiveness occurs when the content of an assessment item is offensive to a subgroup of students, while unfair penalization happens when a test item disadvantages a student from a specific subgroup.

Why is offensiveness not a common problem in standardized tests?

Offensiveness is not a common problem in standardized tests because major test developers carefully analyze their test items to avoid such issues.

How can instructors avoid offensive items in their assessments?

Instructors can avoid offensive items by being mindful of stereotypes regarding gender, sexual orientation, race, ethnicity, and other variables, and by critically reviewing their assessments.

What is an example of unfair penalization in an assessment?

An example of unfair penalization is using a common American fable on a reading comprehension test for international students who are unfamiliar with it, thus putting them at a disadvantage.

What is disparate impact in the context of assessments?

Disparate impact refers to the situation where different ethnic groups perform better or worse on a particular assessment, typically in standardized achievement tests.

How is differential item functioning (DIF) used to detect bias in assessments?

Differential item functioning is a psychometric technique used to detect if a particular item is behaving in unexpected ways for a specific subgroup of students compared to the entire set of items.

What are the three steps in the judgmental approach to identify and eliminate bias in assessments?

The three steps are: 1) Assemble a bias review panel, 2) Have the panel examine each item for potential bias, and 3) Make an overall absence of bias judgment considering all items as a whole.

Why is it important to consider students with disabilities and English language learners in assessment bias discussions?

It is important because these groups may require specific accommodations or considerations to ensure fairness in assessments, and there are many rules and regulations associated with their testing needs.

What is the speaker's opinion on the criticism that focusing on bias is about being 'politically correct'?

The speaker believes that focusing on bias is not about being 'politically correct' but is fundamentally about ensuring fairness in assessments.

What advice does the speaker give for avoiding bias in assessments?

The speaker advises to have clear curricular aims, ensure assessments target those aims without offending or unfairly penalizing students, and to be careful and take the time to review assessment items for potential bias.