The popular saying “When you can’t measure what is important, you make important what you can count” came from the McNamara Fallacy.

Here’s how it all began: The US Secretary of Defense from 1961 to 1968, Robert McNamara, decided to use a military strategy that he felt was best based on the data he had. He relied so much on measurable data that it led to several misguided conclusions and ultimately, the loss of the Vietnam War. 

It was from this incident that the McNamara Fallacy coined its name, also known as the qualitative fallacy. This happens when decision-makers depend solely on quantitative metrics while ignoring qualitative factors.

This type of fallacy often happens in surveys when the researcher relies solely on measurable data, assuming that what cannot be measured with numbers is nonexistent.

How the McNamara Fallacy Shows Up in Surveys

In Education Week, Linda McNeil mentioned that “measurable outcomes may be the least significant results of learning.” Honestly, it sums up the McNamara Fallacy quite nicely. We’ve unconsciously equated data to numbers, but that’s not true; qualitative and quantitative data are both very valid and have their use cases.

The McNamara Fallacy shows up subtly in surveys when we use only closed-ended questions, such as Yes/No answers or 1-5 ratings, because their statistics are very easy to analyse. 

When we rely only on quantitative data, we tend to ignore the minority groups whose experience might be significantly different from the majority group. Their opinions matter too, and it may even be what you need to differentiate your brand, product, services, or policies.

Why Falling for the McNamara Fallacy Harms Your Survey Results

One of the best examples of this fallacy remains the original. Let’s break down the approach to collecting data for informing the strategy for the Vietnam War:

• The first step taken: The measurement of whatever could have been easily measured with numbers.
• The second step taken: The things that don’t have quantitative value or cannot be measured easily were ignored.
• The third step taken: Whatever cannot be measured or quantified easily is not relevant.
• The fourth step: It was concluded that whatever can not be easily measured or valued in quantity does not exist. 

McNamara was operating on efficiency and data, which is generally a good thing. But the gap in his analysis was measuring sentiment, which would have revealed the bigger picture. So, after losing a great number of US Troops to the Vietnam War, the US withdrew its troops back home.

How It Affects Your Survey Insights

We are like McNamara more often than not, especially when the stakes are high. When conducting a survey, we tend to focus on the non-sentimental statistics and highlight the opinion of the majority while disregarding the minority.

The major problem with this approach is that it creates a blind spot that can sink years of resources. Imagine, as a business owner launching a fashion brand, the only feedback you collect is quantitative, with nothing about how your customers feel about your products. You’re actively missing the opportunity to create a brand identity that would resonate with your target audience and make them fall in love with your products.

Common Survey Mistakes Linked to the McNamara Fallacy

Here are some of the mistakes related to the McNamara Fallacy:

• Hyper-Efficiency: Focusing only on the numeric scores and ratings rather than getting the honest opinions or contexts of the respondents. It uses only what was measured while disregarding the other contexts that you can’t measure numerically.
• Single Question Type: Rather than using open-ended questions and follow-ups to gather data, you skip them and use only closed-ended questions. This may save time and help you see what the majority wants, but it still doesn’t give context to why they want what they want.

How to Detect if Your Survey is Victim to the McNamara Fallacy

Here’s how to probe your survey for fallacy:

• Question type and options: Does your survey measure quantifiable data and ask questions for context? For example, “On a scale of 1-10, how would you rate this new ice-cream flavor for quantitative data?” For qualitative data, ask, “What are three(3) that first come to mind after tasting this new ice-cream flavor?”

• Survey Researcher: You can also seek professional help from an analyst who is knowledgeable in analysing and quantifying data. They can help you look for loopholes in your data collections and help you account for them when using the data to inform your decisions.

Practical Ways to Avoid the McNamara Fallacy in Your Surveys

Here are some best practices to help you avoid this fallacy affecting your survey:

• Balance quantitative and qualitative data: You must be able to balance the measurement of your quantitative data with that of your qualitative data so that one would not be given more weight while ignoring the other.
• Using open-ended questions strategically: Ask for context by strategically asking open-ended questions. This will allow you to hear from the respondents in their tone and voice, and gain insights that can’t fit into your closed-ended questions.
• Incorporating context and narrative into analysis: When analyzing your data, ensure that you incorporate the context and narrative of each variable into your conclusion to make an informed decision.

Why Formplus Is Essential to Overcome the McNamara Fallacy

Formplus helps researchers get the full picture. Our builder helps you collect the “what” (quantitative data) and the “why” (qualitative data) behind the numbers. Here’s how you can use Formplus to overcome the McNamara Fallacy:

• Multiple Question Types and Options: Our form builder allows you to create a survey with a well-balanced mix of question types, such as multiple-choice fields for quantitative data and open-ended text fields for rich, qualitative feedback. This ensures you’re collecting both types of information from the start.
• Mobile-Friendly Forms: More than 60% of website visits are from mobile, so it’s easier to reach a wider, more diverse audience when your forms are mobile-friendly. Using Formplus enables you to collect data from participants in various locations and situations. This provides a broader range of perspectives and qualitative insights.
• Calculates and Displays Results: While Formplus can automatically calculate and display quantitative results with the analytics dashboard, it doesn’t stop there. You can also export and analyze the qualitative data collected from open-ended questions. It will help you find the context behind the numbers.
• Customizable Forms: You are not stuck with template questions. You can tailor your questions to the specific nuances of your research and collect dynamic responses. It could include media uploads like video answers and open-ended text responses. This ensures you ask the right questions to uncover qualitative data that might not fit into a standard survey template.
• Work Teams:  This helps you seamlessly analyze responses from multiple perspectives. This collaborative approach helps prevent a single person’s bias from dominating the research. Also, ensures that both quantitative and qualitative findings are considered equally.

Conclusion

The McNamara Fallacy is popularly common in surveys; it reminds us that “not everything that counts can be counted.” If you do not notice it quickly, it might jeopardize your results. You can use numbers to make informed decisions, but not to replace human judgment.

However, this fallacy can be avoided by using a form builder like Formplus that collects both quantitative metrics (e.g., ratings and scores) and qualitative feedback (e.g., open-ended responses and video answers). This ensures you gather a complete picture, allowing you to move beyond simple statistics and truly understand the “why” behind the numbers.