How to Spot AI-Generated Survey Responses in Open-Ended Questions
By Angela Kayode-Sanni
| Last updated:
Jul 12
|
6 min read
Share
Open-ended survey questions have long been one of the most valuable sources of qualitative research. Unlike multiple-choice questions, they allow respondents to explain their experiences, opinions, and motivations in their own words. These responses often reveal the reasons behind customer behavior, uncover unexpected pain points, and provide context that numbers alone cannot.
The rapid adoption of generative AI has introduced a new challenge for researchers. Instead of answering thoughtfully, some participants now rely on tools such as ChatGPT, Claude, or Gemini to generate responses they can paste directly into surveys. While these answers are usually well-written, they may not reflect the respondent’s actual experiences or opinions.
For researchers, marketers, and organizations that depend on customer feedback to make decisions, this creates a serious data quality problem. When AI-generated responses become part of a dataset, they can distort findings, hide genuine customer concerns, and reduce confidence in the results.
Although identifying AI-written responses is becoming more difficult as these tools improve, many responses still share recognizable characteristics. By combining careful review with behavioral data and quality control measures, researchers can significantly reduce the impact of AI-generated submissions.
Generative AI has made producing polished text almost effortless. A respondent can enter a survey question into an AI assistant and receive a complete answer within seconds. For participants rushing through incentive-based surveys, this shortcut can be tempting.
Not every respondent who uses AI has dishonest intentions. Some people use it because they struggle to express themselves, while others believe AI can help them write more clearly. Regardless of the motivation, the final response may represent the AI’s interpretation rather than the respondent’s genuine thoughts.
This becomes particularly problematic in open-ended questions that are designed to capture personal experiences.
For example, consider the question:
What was the biggest challenge you experienced while using our product?
A genuine response might say:
“I couldn’t find the export button, so I spent almost ten minutes looking through the settings before giving up.”
An AI-assisted response may instead read:
“The product generally provides a positive user experience. However, improvements to navigation and usability could further enhance customer satisfaction.”
The second response is grammatically correct, but it offers little practical insight. It does not describe a real experience, identify a specific issue, or provide information that product teams can act on.
When many responses resemble this pattern, researchers may overlook genuine customer frustrations and draw inaccurate conclusions from their qualitative analysis.
No single characteristic can prove that a response was written by AI. However, several indicators together may justify closer review.
Most people write naturally. Their responses often include informal wording, occasional spelling mistakes, incomplete sentences, or conversational expressions. AI-generated responses, on the other hand, frequently appear unusually polished and consistent.
While excellent grammar alone does not indicate AI use, responses that sound exceptionally formal for a casual survey question may deserve additional attention.
AI often produces answers that sound reasonable without saying anything meaningful.
For example:
“Overall, the experience met expectations while presenting opportunities for future improvement.”
Although this sounds professional, it provides no specific details about what happened, what worked well, or what should improve.
Human respondents are generally more likely to mention concrete events, products, people, or situations.
AI models frequently rely on predictable sentence structures and transition phrases.
Examples include:
One response containing these phrases is not unusual. However, when many respondents use nearly identical writing structures, it may indicate AI assistance.
Open-ended questions are designed to capture authentic experiences.
Human responses often include details such as:
AI-generated responses tend to stay at a higher level, offering generalized observations rather than describing something that actually happened.
If multiple respondents independently submit answers that share nearly identical wording, sentence structure, or examples, further investigation may be necessary.
This is particularly true when responses appear unique at first glance but follow remarkably similar patterns throughout.
Remember that none of these indicators should be used in isolation. A combination of multiple signals provides a much stronger basis for identifying potentially AI-generated responses.
Identifying AI-written responses is most effective when researchers combine manual review with behavioral analysis instead of relying on a single detection method.
Reading responses within the context of the entire survey often reveals inconsistencies that automated systems miss.
Look for answers that:
Comparing responses side by side also makes recurring patterns easier to identify.
Behavioral data provides valuable context alongside text analysis.
Useful signals include:
While none of these behaviors proves AI use, they can help identify submissions that warrant closer review.
AI detection software can assist quality assurance by identifying responses that share characteristics commonly associated with AI-generated text.
However, these tools are not always accurate. Some respondents naturally write in a formal style, while AI-generated content can sometimes appear convincingly human.
For this reason, detection software should be treated as one piece of evidence rather than the final decision-maker. Combining automated analysis with human judgment generally produces more reliable results.
Preventing low-quality responses is often easier than identifying them after data collection.
Broad questions encourage generic answers.
Instead of asking:
What do you think about our service?
Ask:
Think about the last time you contacted our support team. What problem were you trying to solve, and how was it resolved?
Questions that require personal context are more difficult for AI to answer convincingly without detailed information from the respondent.
Attention checks help identify disengaged participants before their responses affect your dataset.
Researchers should also monitor:
These quality indicators become even more valuable when combined with manual review.
Automatically deleting responses that appear AI-generated can introduce unnecessary bias.
Instead, establish a review process that flags suspicious submissions for further evaluation. Consider the written response alongside behavioral data, survey metadata, and other quality indicators before deciding whether to exclude it.
This approach reduces the risk of removing legitimate participants while maintaining the integrity of your research.
AI writing tools continue to evolve, and survey quality control must evolve alongside them.
Regularly review your screening methods, update quality assurance procedures, and refine survey questions to reduce opportunities for AI-assisted submissions.
An adaptive approach helps maintain reliable research even as AI-generated content becomes more sophisticated.
Generative AI is changing how people complete surveys, particularly open-ended questions. Although AI-generated responses often sound polished and professional, they frequently lack the personal experiences, emotional detail, and contextual information that make qualitative research valuable.
Researchers should avoid relying on any single indicator when identifying AI-generated survey responses. Instead, evaluate language patterns, behavioral signals, response consistency, and contextual relevance together before making a decision.
Thoughtful survey design, careful manual review, and appropriate use of technology provide the strongest defense against AI-generated noise. By continuously improving these practices, organizations can protect data quality, preserve authentic customer feedback, and make decisions based on insights that accurately reflect real human experiences.
Connect to Formplus, Get Started Now - It's Free!
You may also like:
Ever had to measure something between a negative to a positive range? That’s what a staple scale survey is. Ratings and measuring...
In this article, we’ll look into what a nutritional assessment is, why it’s important and how you can easily create yours with Formplus
Guide on conducting a customer satisfaction survey, question types, examples and online survey form templates
Introduction A donor survey is a piece of research that asks people who are interested in supporting a cause or organization to answer...
