Top 5 Biases with Human Surveys–And Why We Can't Get Rid of Them

When asked, four in ten survey respondents misrepresent their intended purchase behavior.* It’s not as if they do it purposefully (though some do), but there are many factors that can influence their responses, including the framing and ordering of questions. At the same time, these biases are near impossible to eliminate from surveys for reasons we'll discuss below.

The result? Nearly all of your surveys have some level of bias. While there are some things you can do to minimize them, the bad news is they're never going away. (*Say-do-Gap among active US online shoppers 2022, Statista 2023).

Framing and Priming Biases

In a survey, have you ever included a question like: "How satisfied are you with our product?" Perhaps instead you asked: "How dissatisfied are you with our product?" These two seemingly similar questions will elicit very different responses. Furthermore, if later in the survey you follow up with a question like "Rank our product on a scale of 1-10," you will get very different results depending on whether you used satisfied or dissatisfied in the prior question.

These are examples of framing and priming biases. A keen eye and practice can often overcome them, but because we pack so many questions into surveys it's unlikely we'll be able to eradicate these concerns. At the same time, we'll never fully understand how a question could prime a respondent.

Response Bias

Survey respondents can unintentionally (or intentionally) provide inaccurate answers due to things like social desirability, misunderstanding of questions, cognitive dissonance, or other desires to look better than their actual behavior might suggest. For example, when asked in surveys, people will overwhelmingly overestimate the amount they contribute to charities and other philanthropic activities compared to what we know from actual contribution data.

Even when survey questions are carefully crafted, it’s near impossible to expect that all respondents will view the question with the same intent as the designer of the survey, making response bias one of the most persistent biases.

Fatigue or Inattention Biases

The reason TED talks are a maximum of 18 minutes is because that is our best estimate as to when mental fatigue kicks in. How many surveys have you seen go beyond this limit? For those that do, they likely all suffer from fatigue or inattention bias, as respondents lose focus, leading to less accurate responses and data validity issues.

Why is this bias so widespread? Because surveys are expensive and time-consuming, the more questions we can pack into a single survey, the less it will cost and the faster we’ll get results.

Selection Bias

Selection bias occurs when the sample of respondents fails to accurately represent the broader population of interest. This bias can arise from factors like availability, willingness, or access to survey participants, which may unintentionally exclude key demographics. As a result, businesses risk drawing conclusions that don’t reflect the views of their target audience.

Due to resource constraints, this bias persists because traditional surveys often rely on convenience sampling—selecting respondents based on availability or accessibility. Willingness to dedicate more time and money is one solution that could help remediate the problem.

Sample Size Bias

I know a market researcher who once had a client ask if she could survey millennial moms in certain districts of Pennsylvania who had at least one child with a recorded mental disability. I'm not certain it's possible to create a large enough sample to fit those requirements. Outside of this example, traditional surveys often face constraints in terms of time, budget, and logistics, limiting the number of respondents that can be reached.

Small sample sizes can produce results that are not reliable or representative enough to make broad strategic decisions (as any statistician will tell you, just look at the confidence intervals!). Again, the problem comes down to the resources (time and money) you're willing to dedicate to get a large enough sample size to make confident decisions.

The Role of AI in Overcoming Survey Challenges

Instead of asking the same panel 30 questions, think of the biases you could eliminate if after each question you could erase your panelists’ memories. Priming and fatigue biases go right out the window! By leveraging algorithms that simulate human behavior and decision-making, AI helps eliminate these biases and more. For example, we actually CAN and DO erase the memory of AI panelists, starting anew with every question. And if you're concerned about framing, AI-generated surveys allow you to ask the same question in multiple ways to get a holistic answer.

AI surveys minimize response biases because we can prompt them with data and information on actual behavior, and at the same time, they have no ulterior motivations (unless Skynet is real, yikes). Further, because we "code" instead of "recruit" the samples, we can customize the audience and make it as large as we want–selection and sample size biases be gone!

AI-powered Insights Agents, like our Jenny, enable businesses to scale research effortlessly, gather insights in minutes rather than days, and receive consistent, objective feedback without the distortions caused by human interpretation. With these capabilities, Jenny empowers businesses to make smarter, data-driven decisions quickly and cost-effectively.