Let’s say you want to know how many people in your city like ice cream, and the only people you interviewed are first graders from a particular school. The result of this survey isn’t accurate or reliable; even if the survey results show that 90% of respondents like ice cream, this doesn’t necessarily mean everyone in your city likes it. It just means a particular demographic, first graders, likes ice cream. 

You didn’t interview everyone you were supposed to, and this phenomenon is known as undercoverage bias. It happens when you unintentionally exclude a particular part of the population in your data collection, leading to sampling bias and unreliable survey data. This could be due to not randomizing your sample, or lacking adequate equipment, resources, or tools to interview all relevant individuals.

Let’s see how to spot it and prevent it in your data collection.

What Exactly is Undercoverage Bias?

Undercoverage bias happens when you don’t represent certain groups, or they’re underrepresented in your sample. It’s like taking a group photo; you took the photo in portrait mode instead of landscape mode, so you were not able to capture everybody. The people at the edges were cut out of your picture. Saying that you have a photo of the class would be inaccurate, because you only took a picture of some of the people and not the entire class. This is how undercoverage bias works.

Undercoverage bias slips into online data collection in different ways. The most common way it happens is when only a specific demographic is interested in responding to your survey. For example, you conducted an election poll on a particular platform, and only people aged 25 to 40 responded to the survey. Their opinion doesn’t necessarily reflect what people in your city think about the candidates. 

How Undercoverage Bias Sneaks into Your Online Forms and Polls

Online forms and polls allow you to easily reach people, but they might not necessarily be your target population. Here are the most common ways undercoverage bias can slip into your forms and polls:

• Platform-Specific Distribution

If you conduct your poll on just one platform, let’s say LinkedIn, you limit your data to only people who use that platform. If your survey is about using a budgeting app and you want to ask people from 20 to 45, using only LinkedIn means you will only get responses from the platform’s primary audience 9professionals).

What about solo business owners who aren’t “professional” but are still a part of your target market?

• Device Dependency

Over 60% of global website traffic comes from mobile. If your form isn’t mobile-responsive, you’re losing many potential participants, which could exclude a significant part of your target demographic.

• Internet Restriction

Poor internet connection can be a major problem if you’re conducting your polls exclusively digitally, as you may leave out a significant part of your target population. For example, if you want to take a national census in a country where remote areas have poor internet access, and use digital forms only, people living in those areas won’t be part of the census.

• Target Audience Assumptions

While you might think certain aspects are user-friendly, respondents may struggle with them. For example, if you created a product development feedback form to understand desired features, but some users found it hard to navigate and abandoned the form.

This means the products and features you develop would only favor that responsive group, while a significant portion of your user base wouldn’t have access to the features and products they want.

• Language Barriers

If your survey is only in one language and your target population is multilingual, you will only get responses from people who understand the language the survey is written. For example, you create a survey about “Global AI adoption in everyday workflow”, but you design the survey in only English, it would be difficult to capture an accurate representation of how everyone in the world feels.

Unless your form is multilingual, people won’t understand what you’re asking, and only English speakers would answer, skewing your results and making your data unreliable and inaccurate.

• Access Barriers

Even if a form is online, complex CAPTCHAs, mandatory logins, or confusing navigation can confuse people, and they will abandon your form.

Why Undercoverage Bias Skews Your Data (And Your Decisions)

The problem with undercoverage bias isn’t just about missing a few people; it’s about making decisions based on flawed information. Here’s how it affects your decisions: 

• Inaccurate Insights: Undercoverage bias in your data collection means you only reflect the opinion of a very narrow segment. For a skincare brand developing a new product, if there was undercoverage bias in the product development form that you sent, you would only produce products that fit the goals and the needs of the people in those forms, not necessarily your entire target audience, allowing you to miss out on key market opportunities.
• Wasted Resources: Another problem is that you’ll be losing a lot of money and revenue if you only build according to skewed data. For example, let’s say you decided to launch a new UI/UX design after collecting feedback from people, but the feedback only reflected the opinion of a select few. This means the new revamp that you did to your website does not benefit the entire audience; you only benefited a few, making you waste resources, time, and work hours working on a feature nobody actually needs.
• Missed Opportunities: You miss out on market opportunities when you leave out a particular segment, so you need to reach them so that you can market to them quickly today.
• Damaged Reputation: If you don’t meet your target audience’s needs, they will automatically assume that they don’t matter to you or that you don’t understand what they need, and go to competitors, so eventually it will damage your brand.

Common Warning Signs You Might Have an Undercoverage Problem

Here’s how you can check for undercoverage bias in your data collection:

• Unexpectedly Homogeneous Responses: If you notice that all your responses are skewed towards the same thing, that everybody is accepting of something, it most likely means you are asking people with similar beliefs, similar goals, or similar needs, right? So you are leaving out an important population. No real population is ever monolithic.
• Low Response Rates from Specific Groups: Demographic data collection helps you know if people really take your survey. If you check and see that there are not many responses from a particular demographic, you are at risk for undercoverage bias.
• Discrepancies with Known Demographics: If you notice that your survey data is deviating from the standard numbers, this might suggest it’s undercoverage bias. For example, if you take census data and you notice that it is way different from what multiple researchers have taken in the past 3 to 6 months, that is probably a sign of undercoverage bias.
• Feedback Loops from Unsurveyed Groups: If people express feeling excluded, that’s also a very good sign that there is undercoverage bias in your data collection.
• “Niche” Feeling Results: If the end result you get is only tailored to a particular audience, meanwhile, the data you’re supposed to collect should have a broader reach, that’s a huge indicator of undercoverage bias.

How to Mitigate Undercoverage Bias in Your Polls and Forms

Here are some common practices to help you prevent undercoverage bias;

1. Define Your Population Precisely: Before you even start, clearly outline who your target population is, including their demographics, behaviors, and access points.
2. Demographic Questions: Include essential demographic questions (age, gender, location, income, education, primary device, etc.) in your forms. This allows you to compare your sample’s demographics against known population statistics after data collection.
3. Cross-Tabulation and Segmentation: Analyze your data by different demographic groups. Do certain groups have significantly fewer responses? Are their response patterns vastly different from the overall average, hinting at their underrepresentation impacting the overall result?
4. Pilot Testing with Diverse Groups: Before a full launch, test your form with a small, diverse group that intentionally includes representatives from all expected segments. Pay attention to feedback regarding access or usability.
5. External Data Comparison: Always cross-reference your survey results with reliable external data sources (e.g., national statistics, industry reports). If your survey shows 90% of your users are under 25, but industry reports indicate your general user base is closer to 50% under 25, you have an undercoverage issue.

Read also: Undercoverage Bias: Definition, Examples in Survey Research

How Formplus Can Help

Our form creation tool helps keep your survey design and distribution engaging and accessible, allowing everyone to participate in your surveys. Here are some Formplus features designed to help you eliminate undercoverage bias:

• Responsive Design: Automatically adjusts to any screen size (a must-have!).
• Multiple Sharing Options: Email integration, direct links, social media sharing buttons, embed codes for websites.
• Conditional Logic/Branching: While not directly for undercoverage, it allows you to create more tailored and relevant questions, improving engagement and reducing drop-off rates for diverse users.
• Pre-fill Capabilities: For known users, pre-filling data can save time and improve completion rates.
• Analytics and Reporting: Tools that show you where responses are coming from (device, location) can help you identify underrepresented areas.
• Accessibility Features: Look for platforms that adhere to WCAG (Web Content Accessibility Guidelines).
• Variety of Question Types: Beyond just text boxes, ensure you can use scales, multiple-choice, file uploads, ratings, and even signature fields, to cater to diverse ways people might want to express themselves.

To conclude,

Your insights are only as good as your data; if your data is inaccurate, your insights and recommendations are unreliable. Undercoverage bias is a silent threat to the integrity of your data. It can skew your results, leading to misinformed decisions that cost time and resources.

We hope this guide helps you better safeguard your polls and forms from undercoverage bias. Get started with Formplus to create responsive and engaging surveys, people actually want to complete. You can also check out our guide on creating viral forms.