7 tips for good survey questions

Summary

This script offers seven essential tips for crafting effective survey questions. It emphasizes clarity and specificity to ensure consistent interpretation, the importance of asking questions respondents are willing to answer, and the need for truthful responses, addressing social desirability bias. It also advises on ensuring questions are answerable, avoiding double-barreled questions, eliminating biased language, and the crucial step of pre-testing questions. The goal is to create meaningful, non-wasteful survey questions that yield valuable insights.

Takeaways

• 🔍 Ensure questions are clear and specific to avoid varied interpretations.
• 🗣️ Craft questions that respondents are willing to answer, especially when personal topics are involved.
• 😌 Design questions to encourage truthful responses, considering social desirability bias.
• 🧠 Make sure questions are answerable, taking into account the respondent's knowledge and memory.
• 🚫 Avoid double-barreled questions that combine two distinct inquiries into one.
• 🛡️ Steer clear of biased terms or wording that could sway responses.
• 🔬 Conduct pre-tests with real people to gather feedback and identify any issues with the questions.
• 📝 The goal is to create meaningful questions, not perfect ones, to make the most of survey resources.
• ⏰ Surveys are time and cost-intensive; thus, it's crucial to avoid wasting resources on poorly designed questions.

Q & A

What is the first criterion for a good survey question according to the transcript?

A good question is one where all respondents interpret the question in a consistent manner.

Why is it important to ensure clarity and specificity in survey questions?

Ensuring clarity and specificity helps to avoid different interpretations of what constitutes a 'serious crime' or any other term that might vary among respondents.

What is an example of a personal question that might make people unwilling to answer?

Questions about sexual behavior or drug use are examples of personal questions that might make people unwilling to answer due to privacy concerns.

How can survey designers encourage truthful responses to personal questions?

Survey designers can encourage truthful responses by carefully crafting questions, explaining the need for the information, and assuring respondents of confidentiality or anonymity.

What is social desirability and how can it affect survey responses?

Social desirability is a psychological phenomenon where respondents might answer questions in a way that they believe will be viewed favorably by others, potentially leading to dishonest or exaggerated responses.

Why is it important for survey questions to be answerable by respondents?

Questions should be answerable to ensure that respondents can provide accurate information, avoiding vague answers like 'I'm not sure' that do not contribute to meaningful data collection.

What is a double-barreled question and why should they be avoided in surveys?

A double-barreled question is one that combines two different parts into a single question, making it difficult for respondents to answer if they only agree with one part. They should be avoided because they can lead to confusion and inaccurate responses.

How can biased terms or wording in survey questions affect the results?

Biased terms or wording can lead respondents to answer based on their perception of the term rather than the actual question, resulting in different answers depending on the language used.

What is the purpose of pre-testing survey questions?

Pre-testing survey questions helps to identify any confusion or unexpected responses, allowing for adjustments to be made before the full survey is conducted.

Why is it crucial to avoid wasting time and resources on bad survey questions?

Bad survey questions can lead to inaccurate or unusable data, wasting the time and resources of both the survey creators and respondents, and potentially leading to incorrect conclusions.

What is the ultimate goal of creating good survey questions as per the transcript?

The ultimate goal is to create questions that elicit meaningful responses, ensuring that the survey results are valuable and contribute to the intended research or decision-making process.

Outlines

00:00

🔍 Developing Good Survey Questions

The paragraph emphasizes the importance of crafting clear and effective survey questions. It outlines seven tips for achieving this: ensuring questions are consistently interpreted, maintaining respondent willingness to answer, encouraging truthful responses, verifying that respondents can answer the questions, avoiding double-barreled questions, steering clear of biased terms, and pre-testing questions. The paragraph uses examples to illustrate the potential issues with poorly designed questions and provides alternative phrasings to improve clarity and response quality.

Keywords

💡Survey Question

A survey question is a query posed to respondents in a survey to gather information. In the context of the video, it is emphasized that a 'good question' should be interpreted consistently by all respondents, ensuring clarity and specificity. For example, the video discusses how the question 'Have you ever been the victim of a serious crime?' could be interpreted differently by different people, highlighting the need for clear and specific survey questions.

💡Consistent Interpretation

Consistent interpretation refers to the uniform understanding of a question by all respondents. The video stresses the importance of crafting questions that are clear to avoid varied interpretations. This is illustrated by the example of defining 'serious crime,' which might differ from person to person, thus affecting the accuracy of survey responses.

💡Willingness to Answer

Willingness to answer pertains to the respondent's comfort level in responding to a question. The video points out that questions about sensitive topics like sexual behavior or drug use might deter respondents from answering truthfully. It suggests that if such questions are necessary, they should be framed carefully and respondents should be assured of confidentiality.

💡Truthfulness

Truthfulness in survey responses is the accuracy and honesty with which respondents answer questions. The video discusses 'social desirability' as a factor that can lead to dishonest answers. For instance, people might falsely claim to have a library card due to the social desirability of being well-read, which underscores the need for questions that encourage honest responses.

💡Respondent's Knowledge

Respondent's knowledge refers to the information or awareness a respondent has, which enables them to answer a survey question accurately. The video gives the example of asking about monthly grocery spending, which many might not know offhand, emphasizing the need for questions that respondents can confidently answer based on their knowledge.

💡Double Barreled Questions

Double barreled questions are those that contain two distinct parts but ask for a single response. The video explains that such questions can confuse respondents and lead to ambiguous answers. An example provided is a question about both education spending and non-violent criminals, which should be split into two separate questions for clarity.

💡Biased Terms

Biased terms are words or phrases that carry a positive or negative connotation, potentially influencing survey responses. The video advises avoiding such terms to prevent bias in answers. It gives examples like 'Obamacare' versus 'The Affordable Care Act,' highlighting how different terms can elicit different responses due to their connotations.

💡Pre-testing

Pre-testing is the process of trialing survey questions on a small group to identify and rectify any issues before full-scale deployment. The video emphasizes pre-testing as a critical step to ensure that questions are well-understood and yield meaningful responses. It suggests gathering feedback on whether questions were confusing or if responses were unexpected.

💡Meaningful Responses

Meaningful responses are those that provide valuable and relevant information to the survey's objectives. The video's theme revolves around creating questions that elicit such responses, as opposed to vague or irrelevant answers. The goal is to make surveys efficient and valuable, which is why the tips are provided to craft questions that encourage clear and useful input from respondents.

💡Confidentiality

Confidentiality in the context of surveys refers to the assurance that respondents' answers will be kept private, encouraging honesty. The video mentions that when asking personal questions, it's important to assure respondents of confidentiality to increase the likelihood of truthful responses.

💡Social Desirability

Social desirability is a psychological factor where individuals tend to answer questions in a way that they believe will be viewed favorably by others. The video uses the example of owning a library card to illustrate how social desirability can lead to respondents giving answers that they think are more socially acceptable, even if not accurate.

Highlights

A good survey question is one where all respondents interpret the question in a consistent manner.

Ensure questions are clear and specific to avoid different interpretations.

People should be willing to answer the questions in a survey.

Personal questions require careful crafting and assurance of confidentiality.

Survey questions should encourage respondents to answer truthfully.

Social desirability can influence the truthfulness of responses.

Rephrase questions to mitigate the impact of social desirability.

Ensure respondents can actually answer the questions based on their knowledge.

Avoid double-barreled questions that combine two different parts.

Split complex questions into separate, single-part questions for clarity.

Avoid biased terms or wording that could lead to different answers.

Different terminologies can significantly affect survey responses.

Pre-testing questions on real people is crucial for identifying potential issues.

Feedback from pre-testing helps refine questions for better responses.

The goal is to create questions that elicit meaningful responses.

Surveys should not waste time and resources on ineffective questions.

Following these tips can help craft clear and useful survey questions.

Transcripts

When we’re conducting a survey we would like to make sure we’re asking questions

that will collect accurate and useful information.

But it isn’t always obvious what makes a survey question a good survey question.

Here are 7 tips that can help in developing good survey questions.

ONE: A “good question” is one where all respondents interpret the question in a consistent

manner.

That is not always easy to achieve.

The question “Have you ever been the victim of a serious crime?”

might be interpreted differently by different people.

Is getting your car stolen a serious crime?

Probably.

What about your bicycle?

What about your skateboard?

Make sure your questions are as clear and as specific as possible.

TWO: A “good question” is also one where people are WILLING to answer the question.

If a survey asks about very personal things such as sexual behavior or drug use you might

find lots of people unwilling to answer such questions.

If you do need to ask personal questions you need to craft those questions carefully and

explain to the respondent why you need this information and assure them that you’ll

keep their responses confidential or even anonymous.

THREE: A “good question” is one that a respondent will answer truthfully.

Let’s take the following question as an example:

Do you own a library card?

Seems clear enough and it doesn’t seem too personal, but surprisingly lots of respondents

might say yes even when they don’t have library card.

Why is that?

It’s something called social desirability.

Reading books and being well read is socially desirable, so someone who never goes to the

library might not want to admit that they don’t have a library card for fear they

might be judged negatively.

It might be better to ask the question this way:

“Some people get books from a library, others buy them from bookstores or online, do you

currently have a library card or not?”

That might sound a little awkward but you are likely to get a more truthful answer.

FOUR: A “good question” is one that a respondent can ACTUALLY answer.

If you asked some one how much money they spend on groceries in an average month you

might be surprised how many people would say they weren’t sure.

Make sure people have the knowledge or information to answer your survey question.

FIVE: Avoid double barreled questions.

These are questions that ask for a single answer to a question that really has two different

parts.

For example, let’s say you were asked the following question:

“Do you believe the United States should spend more money on education and less money

locking up non-violent criminals?”

That’s a double-barreled question.

Respondents who agree with one part but not the other will have trouble answering this

question.

You should break that question into two separate questions.

SIX: Avoid biased terms or wording.

This is probably the hardest thing to do when writing questions.

Should you use the term Obamacare or use “The Affordable Care Act”?

Should your use the term “Welfare” or should you say “Assistance to the Poor”?

It isn’t obvious which is the better terminology, but different terms will likely lead to different

answers.

Surveys will sometimes give one question to half the sample and a slightly different worded

question to another half to see how respondents react to the different wording.

SEVEN: Pre-test your questions.

This is the most important tip.

You need to try out your questions on real people and then ask for feedback.

Were they confused by a question, did they answer in a way that surprised you.

Pre-testing will help you find out if you have any problems with your questions.

Of course there is no such thing as a perfect survey question.

Instead our goal is to create questions that elicit meaningful responses.

Surveys take time and money.

We don’t want to waste our time and the respondent’s time asking bad questions.

Following these 7 tips can help you craft clear and useful survey questions.