You have a number from a survey. How can you be sure that number means anything?

Take two surveys that both measure employee engagement. Each asks new employees to rate how engaged they are on a scale of 1 to 10. Both produce clean charts and statistics that appear to support conclusions about employee engagement.

The researchers behind one of the surveys went a step further than data collection. They followed up on what happened to the employees after they took the survey and asked:

• Did the low-engagement employees quit?
• Did the high-engagement employees stay longer and perform better than average-engagement employees in the same group?

The first survey only tells you what respondents reported. The second provides evidence that respondents’ answers are linked to something measurable in the real world. This is what is called criterion validity.

Other types of validity check whether a question measures what it is supposed to measure. Criterion validity checks whether the measure is related to an outside outcome it is expected to predict or correspond with.

This distinction matters for survey researchers, because a survey can produce statistics that look clear and well structured but turn out to have no connection to the outcomes the organization cares about.

What Does Criterion Validity Mean?

Criterion validity asks whether a survey measure relates to something outside the survey that it is expected to predict or correspond with.

The key word is external. The survey results are not checked against the survey itself. They are compared against a separate benchmark, which is called the criterion.

Example 1: A company collects customer satisfaction scores to estimate whether customers intend to stay. To test criterion validity, researchers compare satisfaction scores with actual retention. If customers who reported high satisfaction stayed and those who reported low satisfaction left, that is evidence the survey is valid for this purpose.

Example 2: A pre-hire assessment is supposed to forecast job performance. Researchers can test its criterion validity by checking whether candidates who scored high on the assessment later received higher performance ratings than those who scored low.

A question can look appropriate from a survey design point of view and still produce results that have no relationship with the outcome it was meant to predict. Criterion validity tests that assumption directly.

Concurrent vs. Predictive Validity

There are two types of criterion validity: concurrent validity and predictive validity. The difference is mainly about timing, meaning when the survey is taken compared with when the criterion is measured.

Concurrent validity checks whether a survey measure matches an existing, already-validated criterion measured at the same time.

Example: You want to test a new product loyalty survey against an established one. You have the same participants complete both surveys at the same time, then compare the results. If the new survey produces results similar to the established one, that supports its validity.

Predictive validity checks whether a survey measure forecasts an outcome that happens later.

Example: You survey users about how loyal they expect to be to your brand, then wait 6 to 12 months. Are the users who reported higher loyalty still customers? Have the users who reported lower loyalty left at a higher rate?

Predictive validity takes longer to check, but it is often the more important of the two if your organization plans to use survey responses to make predictions about the future.

In short:

• Concurrent validity asks: “Does my measure match an accepted measure taken at the same time?”
• Predictive validity asks: “Does my measure forecast an outcome that happens later?”

Why Criterion Validity Matters in Survey Research

Survey questions can be well written, clear, and easy to answer, and still produce data that has not been shown to be useful for its intended purpose. This is a particular concern for organizations trying to predict outcomes such as employee retention or customer satisfaction.

Organizations use surveys to find out whether customers will keep doing business with them, whether employees will stay with the company, whether job candidates will fit the work environment, or whether a planned course of action is likely to get a good response from customers. Measuring and reporting statistics alone does not answer these questions. The survey results need to be tested against real-world evidence.

Criterion validity also gives researchers a way to move beyond their own opinions about how good a set of questions is. They can compare survey responses against objective evidence, such as customer behavior or established measurements.

This protects against a common mistake in survey design: assuming that because a question sounds logical and appears to address what the organization wants to know, it will reflect actual real-world performance.

How to Assess Criterion Validity

Checking criterion validity requires three things:

1. The survey measure: the questions or scale being tested.
2. The criterion: an external measure that is clearly separate from the survey.
3. A method of comparison: a way to determine how strongly the survey measure and the criterion are related.

Start by choosing a criterion that fits what the survey is meant to measure, and plan how you will collect it before you run the survey. For example, it is easy to ask respondents about their purchase intent, but comparing those answers with actual sales over time requires that you can link each respondent to their later purchases. If that link is not planned ahead, the criterion data may not be usable once the survey is done.

Once both sets of data are collected, compare them using an appropriate statistical method:

• Continuous measures: Use a correlation coefficient, such as Pearson’s r, to measure how closely the two sets of values move together.
• Yes/no outcomes (for example, did the customer churn or stay?): Compare survey scores between the groups that had each outcome.

Some studies call for other techniques, but the principle stays the same: relate what respondents reported to outcomes measured outside the survey.

Common Applications of Criterion Validity in Survey Research

The concept is easiest to understand through examples:

• Employee engagement and turnover: Test how engagement scores relate to future voluntary turnover. If low scores consistently appear alongside high turnover, the measure shows predictive validity for retention.
• Net Promoter Score and customer behavior: Test how NPS responses relate to later purchases, retention, or referrals. The key is to check the relationship against observed behavior rather than assuming it exists.
• Pre-employment assessments and job performance: Correlate assessment results with later performance ratings or other job-related outcomes to see whether the assessment predicts future performance.
• Health surveys and clinical status: Compare self-reported symptoms against an established clinical diagnosis to evaluate how well survey responses agree with it.
• Customer satisfaction and renewal behavior: Satisfaction scores are expected to predict renewals and cancellations. If satisfaction scores are related to whether customers stay or leave, the measure has predictive validity.

The same logic applies in all of these examples: validity is tested against data from outside the survey, not against the survey results alone.

Criterion Validity Compared With Other Types of Validity

Criterion validity is often examined alongside other types of validity:

• Face validity: Whether a question appears, at first glance, to measure what it is intended to measure. It does not show that the measure actually works.
• Content validity: Whether a measure covers all the relevant dimensions of the concept being studied. For example, a job satisfaction survey that asks about salary, autonomy, and relationships with colleagues will have higher content validity than one that only asks about salary.
• Construct validity: Whether a measure accurately reflects the theoretical concept it is meant to capture. It is studied by checking whether the measure relates to other measures in the ways theory predicts.
• Criterion validity: Whether a measure relates to real-world outcomes or established measures outside the survey.

Each type of validity answers a different question, so a strong survey instrument often needs several kinds of validity evidence.

Challenges Associated With Criterion Validity

Criterion validity depends on real-world outcome data, which can take time and resources to obtain.

• Availability of a criterion: Some survey areas have established external measures. For new research questions or abstract concepts, it may not be clear what outcome can serve as the criterion.
• Time: Predictive validity requires waiting for outcomes to occur, which can take weeks, months, or years.
• Quality of the criterion: When no accurate external measure is available, researchers may create their own, which can be unreliable in its own right.
• Sample size: A validity study needs enough participants to produce stable results. The number required depends on the statistical test used and how strong the relationship is.
• Correlation is not causation: A correlation shows that a relationship exists between the survey measure and the criterion. It does not show that whatever the survey measures causes the outcome.

Best Practices for Applying Criterion Validity When Designing Surveys

Plan for criterion validity before you run the survey. It is difficult to add after the fact.

• Define the expected outcomes in advance. Decide which behaviors or outcomes the survey should predict. This clarifies what the scale needs to measure.
• Use concurrent validity for new or shortened scales. Comparing a new or shortened instrument against an established standard produces evidence faster than waiting for future outcomes.
• Plan how to link data. If you expect survey responses to relate to later behavior, decide upfront how you will connect each respondent’s answers to their follow-up outcomes.
• Use adequate sample sizes and suitable statistical tests. Choose methods that fit your data rather than defaulting to correlation because it is familiar.
• Treat validation as ongoing. Evidence of validity applies to a specific population at a specific point in time. Organizations may need to retest their measures as economic, environmental, or political conditions change.

Conclusion

A survey’s structure and findings alone do not establish confidence in its results. The real test is how well survey responses relate to real-world outcomes, such as customer retention or employee job performance. Researchers should account for this when designing a new survey.

When criterion validity is established, a survey can describe how people think and also provide reliable measures for understanding and predicting outcomes. Its purpose is to increase confidence that survey results are useful for predicting what will happen in practice.