The primary goal of surveys is to help you gather insights into your audience’s opinions and feedback, allowing you to make informed decisions such as product improvements, new service additions, and campaign messaging adjustments. But this would only happen if it’s accurate, and one of the most common reasons why survey data don’t always give you reflect your audience’s true opinion is bias.

In this article, we’ll discuss a particular type of bias: conformity bias. Conformity bias happens when people believe there’s a particular way they should answer, and answer you based on that, even if it’s not how they feel.

What’s Conformity Bias?

Conformity bias happens almost every day; it’s an invisible peer pressure we feel when making an unpopular choice. For example, let’s say you go to get ice cream with your friends and you want to have the mint flavour, but you get to the shop and people are teasing someone for ordering mint. All your friends ask for chocolate and vanilla; chances are, you might want to order the same thing because that’s the generally accepted choice.

This is how conformity bias works. It happens when you align your attitude, beliefs, and behaviors to suit your group or community. Another name for this is groupthink.

How Does It Impact Survey Accuracy and Decision-Making

The main consequence of conformity bias is that you collect unreliable data. Here’s how it affects your decision-making:

• Flawed Decisions: If you don’t have the right data, you’re likely to launch products based on inaccurate market insights or implement ineffective strategies. For example, if your respondents agree with your proposed features because they think it would be rude to say no, using this data means you’ll end up launching a feature nobody wants. 
• Missed Opportunities: If you collect incorrect data, that means you didn’t get your audience’s true pain points, feedback ideas, or niche preferences that can help you differentiate yourself from competitors. Your data is only going to reflect mainstream opinion, and you don’t get to cater to the needs of your niche audience, which can be what changes everything.
• Erosion of Trust: If you continuously make decisions that don’t resonate with your audience because of wrong data (even if they are the ones who gave you the wrong data), they will lose faith in your data collection process. For example, stakeholders or even you will get tired of using resources to carry out research that doesn’t work.

Common Signs of Conformity Bias in Survey Responses

Here are some common red flags to detect conformity bias in your data:

• Unusual Consensus: Let’s say you asked, “To what extent do you agree with the statement: ‘It is always best to prioritize the needs of the community over individual desires’ (strongly agree, agree, neutral, disagree, strongly disagree?” If you ask people from only a specific region or a common culture, they are likely to answer similarly because it’s the socially acceptable answer.
• Lack of Nuance or Extremes:  Another indicator of conformity bias is when the majority of the responses are “middle-of-the-road,” even when the topic is something that people should either strongly agree with or strongly disagree with.
• Inconsistency with Other Data: If you look at past survey results and compare them to yours, but they contradict one another, it may be an indicator that your responses may be inaccurate or that the past results are inaccurate.
• “Socially Desirable” Answers: Also, if respondents consistently choose answers that present them in a positive light or align with perceived societal expectations, especially for sensitive topics, they may be trying to align with popular opinion and not necessarily their true opinions.

Examples of Conformity Bias in Real Survey Scenarios

• Product Satisfaction Surveys: If everyone rates a new product as “excellent,” but your product adoption and sales suggest otherwise, it may be due to conformity bias. People may rate your product as excellent because they don’t want to be the “only one” who dislikes it, especially if they have seen positive reviews or comments about your product.
• Political Polls: People can express support for a popular candidate they don’t truly want because they don’t want to be singled out as the outlier, or so they agree with the perceived winning side.
• Employee Engagement Surveys: Employees might hesitate to give negative feedback about management or company culture if they don’t trust the anonymity of the survey. So, instead of critiquing the company and giving their true opinions, they make their feedback polite to avoid negative consequences.
• Public Opinion on Controversial Topics: People are likely to temper down their true opinions on sensitive subjects to avoid appearing extreme or out of step with the majority, especially on platforms where they think their responses may not be anonymous.

How Conformity Bias Creeps Into Surveys and Polls

People who give conforming answers most time aren’t even trying to give conforming answers; they just find themselves doing it. In fact, they may not even be aware that they gave conforming answers even after the survey. Here are the major factors that trigger conformity bias:

Social Pressure and Groupthink Effects

If you don’t explicitly tell respondents that their data and responses will remain anonymous, they may conform their answers to fit societal standards. Another example is collecting opinions in a group setting, for example, group interviews where everyone can hear every other person’s answers and may change theirs to fit the “group” sentiment.

Question Wording and Survey Design Pitfalls:

• Leading Questions: If you structure your questions in a way that subtly nudge respondents toward a particular answer (e.g., “Don’t you agree that our new policy is excellent?”), Respondents may feel compelled to agree with you, even if they don’t feel that way.
• Suggestive Language: Using emotionally charged words or phrases that can influence opinions (e.g., “Do you have a secret hunch about how you will die?”). This question is emotionally charged and may trigger anxiety and fear in the respondent, so they respond with something generic (e.g., “I don’t know.”)
• Limited Answer Options: Also, if you don’t provide enough diverse answer choices, respondents may feel forced to pick an option they don’t agree with but feel is acceptable. For example, if the only options are “Yes” or “No,” respondents may feel forced to pick an option if it doesn’t truly reflect their view. However, if your options were more diverse, like a rating scale of strongly agree to strongly disagree, you give respondents room to pick the option that best fits their opinion.

Survey Environment and Timing Influences:

• Lack of Anonymity: If respondents are not assured of their anonymity, they are more likely to conform because they don’t want to be seen as the odd one, especially when discussing controversial topics. Most people don’t want the attention or even backlash that comes with being different, so they may not express their true opinion to avoid sticking out.
• Public Settings: Taking a survey in a public place, e.g., street interviews where others are around them, and even online if the video gets uploaded, may give respondents second thoughts about expressing their true feelings, especially when it’s unpopular and they could get critiqued for it.
• Time Constraints: If respondents don’t have enough time to calmly think and choose an answer that truly resonates with them, they may just opt for the easiest or most common answer while rushing.

How to Test Your Survey for Conformity Bias Before Launch

Here are some tips to help you check for conformity bias before launching your survey:

• Running Pilot Tests and Analyzing Response Patterns: Before launching full-scale, test your survey with a small group that fits your target audience for the survey. This will help you spot unusual consensus and identify the specific questions that can trigger conformity bias.
• Using Control Questions to Detect Bias: Include a few “control questions” that subtly rephrase earlier questions or ask about related, less sensitive topics. If the respondents respond differently to these questions, it could mean one of the questions was worded in a way that made respondents feel pressured to choose a certain option. For example, asking people, “Do you think an inconsistent person? Yes or No” vs “On a scale of 1-10, how likely are you to forget to do your chores?”
• Gathering Feedback from Diverse Participants: Get feedback on your survey design from different people who fit your target audience. Ask them if they felt pressured by any of the questions or if they felt compelled to answer in a certain way.

Next Steps After Collecting a Survey with Conformity Bias

We have discussed how to detect and how to prevent conformity bias, but what do you do when you’ve already had a survey with conformity bias on your hands?

Here’s what to do:

• Acknowledge the Limitation: Be transparent about the potential for bias when analyzing them to make decisions, and give a wide margin of error.
• Look for Corroborating Evidence: Cross-reference your survey data with other sources of information such as sales data, downloads, user journeys, customer support tickets, and qualitative interviews. This will help you see the extent of the conformity bias and whether the conformity bias is serious enough to warrant overhauling the results.
• Conduct Follow-Up Qualitative Research: If a particular area shows signs of bias, consider conducting in-depth interviews or focus groups to understand the underlying reasons for the responses. This way, respondents use their own words to express themselves and are likely to give you more accurate and insightful answers.
• Adjust Your Interpretation: Be more cautious in making definitive statements based solely on the affected survey results. Try to corroborate the survey data with other information, such as customer support tickets or one-on-one interviews.
• Refine Future Surveys: Use the insights gained from previous surveys with conformity bias to improve your survey design and question wording for subsequent data collection.

The Bottom Line

Understanding conformity bias helps you know how it manifests and take proactive steps to mitigate it. It also improves the accuracy and reliability of your survey data.

Using Formplus helps you create surveys with diverse question types and options like rating scales, open-ended questions, matrix scale, etc., that allow respondents to fully express their opinions; they could even upload a video or audio response.

Ready to get started? Try Formplus.