Assessment Bias

Summary

In this video, Dan Hickey discusses the critical principle of assessment bias, emphasizing the need for fairness in testing. He outlines two main types of bias: offensiveness and unfair penalization. Hickey advises avoiding stereotypes in assessments and using examples that might disadvantage certain student groups. He introduces the concept of disparate impact and differential item functioning (DIF) to detect bias. Hickey suggests a three-step approach for bias review, applicable to both high-stakes and classroom assessments. He also touches on considerations for students with disabilities and English language learners, stressing the importance of fairness and clear curricular aims in assessment design.

Takeaways

• 🔍 The goal of assessment is to achieve the absence of bias, ensuring fairness for all students.
• 🚫 Two main forms of bias to avoid in assessments are offensiveness and unfair penalization.
• 👨‍👩‍👧‍👦 Offensiveness can occur when assessment content is offensive to a subgroup of students, often seen in instructor-developed assessments.
• 🌐 Unfair penalization happens when test content puts a subgroup at a disadvantage, which is harder to detect than offensiveness.
• 📚 An example of unfair penalization is using culturally specific content that is unfamiliar to certain students, like international students.
• 🔄 Disparate impact refers to different ethnic groups performing differently on an assessment, which should be examined for potential bias.
• 🔍 Differential item functioning (DIF) is a technique used to detect if an item is behaving unexpectedly for a subgroup of students.
• 👥 A bias review panel can be assembled to identify and eliminate bias in assessments, with members from diverse backgrounds.
• 👨‍🏫 For classroom assessments, bias detection can be scaled down to increasing instructor awareness and careful item review.
• ♿️ Accommodations must be made for students with disabilities during testing, which requires specific knowledge and expertise.
• 🌐 Assessments should be designed to be fair and not just 'politically correct', focusing on clear curricular aims without bias.

Q & A

What is the main principle discussed in the video script?

The main principle discussed in the video script is assessment bias, which refers to the absence of unfair advantage or disadvantage in assessment practices.

What are the two forms of bias that can be found in assessments?

The two forms of bias mentioned are offensiveness and unfair penalization. Offensiveness occurs when the content of an assessment item is offensive to a subgroup of students, while unfair penalization happens when a test item disadvantages a student from a specific subgroup.

Why is offensiveness not a common problem in standardized tests?

Offensiveness is not a common problem in standardized tests because major test developers carefully analyze their test items to avoid such issues.

How can instructors avoid offensive items in their assessments?

Instructors can avoid offensive items by being mindful of stereotypes regarding gender, sexual orientation, race, ethnicity, and other variables, and by critically reviewing their assessments.

What is an example of unfair penalization in an assessment?

An example of unfair penalization is using a common American fable on a reading comprehension test for international students who are unfamiliar with it, thus putting them at a disadvantage.

What is disparate impact in the context of assessments?

Disparate impact refers to the situation where different ethnic groups perform better or worse on a particular assessment, typically in standardized achievement tests.

How is differential item functioning (DIF) used to detect bias in assessments?

Differential item functioning is a psychometric technique used to detect if a particular item is behaving in unexpected ways for a specific subgroup of students compared to the entire set of items.

What are the three steps in the judgmental approach to identify and eliminate bias in assessments?

The three steps are: 1) Assemble a bias review panel, 2) Have the panel examine each item for potential bias, and 3) Make an overall absence of bias judgment considering all items as a whole.

Why is it important to consider students with disabilities and English language learners in assessment bias discussions?

It is important because these groups may require specific accommodations or considerations to ensure fairness in assessments, and there are many rules and regulations associated with their testing needs.

What is the speaker's opinion on the criticism that focusing on bias is about being 'politically correct'?

The speaker believes that focusing on bias is not about being 'politically correct' but is fundamentally about ensuring fairness in assessments.

What advice does the speaker give for avoiding bias in assessments?

The speaker advises to have clear curricular aims, ensure assessments target those aims without offending or unfairly penalizing students, and to be careful and take the time to review assessment items for potential bias.

Outlines

00:00

🔍 Understanding Assessment Bias

Dan Hickey introduces the concept of assessment bias, emphasizing the importance of unbiased assessment practices. He outlines two main types of bias: offensiveness and unfair penalization. Offensiveness refers to assessment content that may offend a subgroup of students, often inadvertently appearing in instructor-developed assessments. Unfair penalization occurs when test content puts subgroups at a disadvantage, such as using culturally specific content that is unfamiliar to international students. Hickey also discusses disparate impact and the use of Differential Item Functioning (DIF) to detect bias in test items. He suggests a three-step judgmental approach for identifying and eliminating bias, which includes assembling a bias review panel to examine each item for potential bias against students based on personal characteristics.

05:03

📚 Accommodations for Diverse Learners

The second paragraph focuses on the need for accommodations in assessments for students with disabilities and English language learners. It highlights the importance of tailoring assessments to the specific needs of these students, recognizing that different disabilities require different accommodations. For non-native speakers, the use of word problems in content knowledge assessments can pose additional challenges. Hickey emphasizes the necessity for educators to develop expertise in creating fair assessments for diverse student populations. He concludes by addressing the misconception that addressing bias is about political correctness, asserting that it is fundamentally about fairness. He advises educators to set clear curricular aims and ensure assessments align with these aims without causing offense or unfair disadvantage to any student group.

Keywords

💡Assessment Bias

Assessment bias refers to the unfair advantage or disadvantage that certain questions or tests may pose to different groups of students due to cultural, social, or personal factors. In the video, the speaker emphasizes the importance of striving for an absence of bias in assessments to ensure fairness. An example given is the use of a common American fable in a reading comprehension test for international students, which could be biased as it may be unfamiliar to them.

💡Offensiveness

Offensiveness in the context of the video pertains to the content of an assessment item that may be offensive to a subgroup of students, often due to stereotypes or insensitive references. The speaker suggests avoiding stereotypes related to gender, sexual orientation, race, and ethnicity in assessments to prevent offending any student group. This is crucial for maintaining a respectful and inclusive educational environment.

💡Unfair Penalization

Unfair penalization occurs when an assessment item puts a subgroup of students at a disadvantage, not because of their lack of knowledge or skill but because of external factors. The video provides an example of using a culturally specific fable in a test for international students, which could penalize them unfairly because of their unfamiliarity with the content.

💡Disparate Impact

Disparate impact is a term used when different ethnic groups perform significantly better or worse on a particular assessment. The video mentions that while this occurrence might or might not indicate bias, it is essential to examine such instances closely to ensure that assessments are fair and unbiased.

💡Differential Item Functioning (DIF)

DIF is a psychometric technique used to detect if a specific test item is behaving differently for a particular subgroup of students compared to the general test population. The video highlights the use of DIF by test makers to identify and remove items that may be biased against certain subgroups, thus ensuring a fair assessment process.

💡Bias Review Panel

A bias review panel is a group of individuals assembled to examine assessment items for potential bias. The video describes that this panel, which can vary in size from a few to several members, evaluates each item to determine if it might offend or unfairly penalize any student group. This process is crucial for high-stakes assessments and can also be applied in classroom settings.

💡Validity

Validity in assessments refers to whether an assessment measures what it is intended to measure accurately and fairly. The video touches upon validity in the context of unfair penalization, noting that some instances of students being exposed to content differently can affect the validity of an assessment.

💡Accommodations

Accommodations are adjustments made to assessments to ensure that students with disabilities can participate fairly. The video points out the necessity for educators to be knowledgeable about the appropriate accommodations for students with disabilities during testing, which is a critical aspect of creating an equitable assessment environment.

💡English Language Learners

English Language Learners (ELLs) are students who are not native English speakers and are still developing their proficiency in the language. The video discusses the challenges ELLs might face in assessments, especially when content is delivered through complex language structures like word problems in math assessments, and the need for educators to consider this when designing tests.

💡Cultural Relevance

Cultural relevance in assessments means that the content of the assessment is appropriate and meaningful to the cultural backgrounds of the students taking the test. The video suggests that while there are debates about how to achieve cultural relevance, it is an important discussion to have to ensure that assessments are fair and do not disadvantage any student group.

Highlights

Assessment bias is a critical principle to achieve fairness in assessment.

There are two main forms of bias: offensiveness and unfair penalization.

Offensiveness occurs when assessment content offends a subgroup of students.

Unfair penalization happens when test content disadvantages a subgroup.

Offensiveness is less common in standardized tests but can appear in classroom assessments.

Avoid stereotypes in assessments regarding gender, sexual orientation, race, and ethnicity.

Unfair penalization is harder to detect and is more common in classroom assessments.

Using unfamiliar content, like American fables, can unfairly penalize international students.

Disparate impact refers to different ethnic groups performing differently on assessments.

Differential item functioning (DIF) is used to detect bias in test items.

Bias review panels can be assembled to identify and eliminate bias in assessments.

The panel examines each item for potential bias based on personal characteristics.

Bias detection can be scaled up or down depending on the assessment's stakes.

Instructors should increase awareness of bias in all forms of assessment.

Accommodations must be made for students with disabilities during testing.

Non-native speakers may face additional challenges in language-based assessments.

Fairness in assessment is not about political correctness but about treating all students equitably.

Clear curricular aims can help avoid bias in assessments.

It's important to carefully examine assessment items for potential bias.

Transcripts

Hi again, this is Dan Hickey.

This is one in a series short videos concerning assessment, practices,

principles, and policies.

Today we're going to talk about one

principle in assessment. A very important principle known as assessment bias

What we want to achieve an assessment is really the absence of bias.

So there's really two forms of bias that you'll probably find your self concerned

with

These are offensiveness and unfair penalization

Offensiveness occurs when the content of an

item or assessment is offensive to a subgroup students

Because most major test developers carefully analyze their test items,

this is not really a common problem in standardized tests.

However, offensiveness often creeps into instructor developed classroom

assessments.

Even when instructors try to avoid offensive items, every now and then

an offensive item may appear. Try to avoid stereotypes regarding gender

sexual orientation, race, ethnicity, or any such variables in your assessments.

This is of course going to be a bigger problem in some domains than others.

Unfair penalization occurs when the content of a test item disadvantages

a student who is a member of a subgroup. This bias is more common in

classroom assessments than offensive items because it is

so much harder to detect. An example would be using a common American fable

on a reading comprehension test that is given to US foreign students

and newly arrived international students. Because the fable is

unfamiliar to the international students. it unfairly penalizes them.

This can also happen when some students have been more directly exposed to

content

of an assessment than other students, but this actually falls under a separate

topic known as validity, which I'll discuss

in a different video. Another really important issue

concerns disparate impact. This refers to the occasion when different ethnic groups

perform better or worse

on a particular assessment, typically in the context of standardized achievement

tests.

When this happens there might or might not be bias.

But when it occurs, it's worthy of a close examination.

Differential item functioning is a psychometric technique used to detect

whether a particular item is behaving in unexpected ways for a particular

subgroup of

students relative to the entire set of items.

While disparate impact remains a controversial topic,

widespread use of DIF has allowed standardized test makers to identify and

remove

specific items that subgroups perform poorly on

relative to the rest the items. Instructors and administrators can use

the three steps in a judgmental approach to identify and eliminate bias in assessments.

These steps are usually taken for high stakes assessments, but you can also use

them in classroom settings as well.

The first step is to assemble a bias review panel

For high stakes testing. This might involve as many as 25 reviewers

who all know the content of the assessment but come from a mixed

gender and ethnic backgrounds. For lower stakes classroom assessments, the panel can

be much smaller.

The panel members examine each item in the assessment and ask the question:

"Might this item offend or unfairly penalize

any group of students on the basis of personal characteristics such as gender,

ethnicity, religion, or-race?" Finally

the panel makes an overall absence of bias-judgment. Here, the panel members look at

the items in the aggregate

or as a whole. And again, they ask the question: "Considering all of the items in

the device you just reviewed,

do the items, taken as a whole, offend or unfairly penalize

any group of students on the basis of personal characteristics?"

Bias detection can be scaled up or scale down depending on the needs.

When there are large numbers of students involved, judgmental approaches can be scaled

up

and usually will also involve psychometric approaches as well.

When there are smaller numbers of students involved, the judgmental approach can be

scaled down,

even to the level of simply increasing awareness by the classroom instructor

all the assessments used in the class.

And also, not just thinking about paper and pencil assessments but other

informal assessments as well.

The important thing is to take some time looking at the assessment items

and the assessment you carry out to search for sources of bias.

Two additional areas of consideration, concern students with disabilities

English language learners. These are very important issues,

and there's lotsa rules and regulations associated with them.

I'm not going to discuss them in depth because, frankly, they're quite specific and

because I'm not a specialist in these issues. As for disabilities,

in most countries in both K-12 settings and universities,

educators are required to make accommodations during testing for

students

with disabilities that are relevant to that assessment act.

If you're administrator or if you have students with disabilities in your

classrooms,

you really need to be knowledgeable if you want to claim to be an expert in

assessment about this important issue.

Remember, different disabilities call for different accommodations.

The second type consideration is for non-native speakers.

If you're testing content knowledge, such as in mathematics,

but using word problems, your assessment might be a lot harder for students who are

being tested in their second or third language.

Again, this varies a great deal by level and context.

If you have non-native speakers, particularly if your tests are high stakes,

you should develop significant expertise in this regard.

Let me wrap up with some opinions my own in this regard.

Over the years, I've had a few students in my class argue that this is really just

about being overly sensitive ,about being quote: "politically correct".

And that's simply not the case, it's really about fairness

plain and simple. People have lots of opinions and there are lots of debates about how

culturally relevant curriculum should be.

In my opinion, these are important discussions to have.

You can avoid many problems by having clear curricular aims

and then making sure that your assessments target those aims

without offending anyone or unfairly penalizing students.

And whatever you do, avoid falling into the trap of only worrying about the

subgroups of students that you assume

you have in your class. That's all for now on the topic of bias.

There's lots more to be learned. But remember, the important thing is to look

carefully at your items

and consider whether or not they're offense. It just takes time and you have to be

careful.

Bye now.