New study on surveys: repetitive questions yield bad data

In a lot of research, we often work with surveys. But according to a new study, surveys that ask too many of the same type of questions tire respondents and return unreliable data. Why? People tire from questions that vary only slightly and tend to give similar answers to all questions as the survey progresses.

From the press release:

Marketers, policymakers, and researchers who rely on long surveys to predict consumer or voter behavior will have more accurate data if they craft surveys designed to elicit reliable, original answers, the researchers suggest.

“We wanted to know, is gathering more data in surveys always better, or could asking too many questions lead to respondents providing less useful responses as they adapt to the survey,” said first author Ye Li, a UC Riverside assistant professor of management. “Could this paradoxically lead to asking more questions but getting worse results?”

While it may be tempting to assume more data is always better, the authors wondered if the decision processes respondents use to answer a series of questions might change, especially when those questions use a similar, repetitive format.

The research addressed quantitative surveys of the sort typically used in market research, economics, or public policy research that seek to understand people’s values about certain things. These surveys often ask a large number of structurally similar questions.

Researchers analyzed four experiments that asked respondents to answer questions involving choice and preference.

Respondents in the surveys adapted their decision making as they answer more repetitive, similarly structured choice questions, a process the authors call “adaptation.” This means they processed less information, learned to weigh certain attributes more heavily, or adopted mental shortcuts for combining attributes.

In one of the studies, respondents were asked about their preferences for varying configurations of laptops. They were the sort of questions marketers use to determine if customers are willing to sacrifice a bit of screen size in return for increased storage capacity, for example.

“When you’re asked questions over and over about laptop configurations that vary only slightly, the first two or three times you look at them carefully but after that maybe you just look at one attribute, such as how long the battery lasts. We use shortcuts. Using shortcuts gives you less information if you ask for too much information,” said Li.

While humans are known to adapt to their environment, most methods in behavioral research used to measure preferences have underappreciated this fact.

“In as few as six or eight questions people are already answering in such a way that you’re already worse off if you’re trying to predict real-world behavior,” said Li. “In these surveys if you keep giving people the same types of questions over and over, they start to give the same kinds of answers.”

The findings suggest some tactics that can increase the validity of data while also saving time and money. Process-tracing, a research methodology that tracks not just the quantity of observations but also their quality, can be used to diagnose adaptation, helping to identify when it is a threat to validity. Adaptation could also be reduced or delayed by repeatedly changing the format of the task or adding filler questions or breaks. Finally, the research suggests that to maximize the validity of preference measurement surveys, researchers could use an ensemble of methods, preferably using multiple means of measurement, such as questions that involve choosing between options available at different times, matching questions, and a variety of contexts.

“The tradeoff isn’t always obvious. More data isn’t always better. Be cognizant of the tradeoffs,” said Li. “When your goal is to predict the real world, that’s when it matters.”

Li was joined in the research by Antonia Krefeld-Schwalb, Eric J. Johnson, and Olivier Toubia at Columbia University; Daniel Wall at the University of Pennsylvania; and Daniel M. Bartels at the University of Chicago. The paper, “The more you ask, the less you get: When additional questions hurt external validity,” is published in the Journal of Marketing Research.

Abstract of the study:

Researchers and practitioners in marketing, economics, and public policy often use preference elicitation tasks to forecast real-world behaviors. These tasks typically ask a series of similarly-structured questions. The authors posit that every time a respondent answers an additional elicitation question, two things happen: (1) they provide information about some parameter(s) of interest, such as their time preference or the partworth for a product attribute, and (2) the respondent increasingly adapts to the task—i.e., using task-specific decision processes specialized for this task that may or may not apply to other tasks. Importantly, adaptation comes at the cost of potential mismatch between the task-specific decision process and real-world processes that generate the target behaviors, such that asking more questions can reduce external validity. The authors used mouse- and eye-tracking to trace decision processes in time preference measurement and conjoint choice tasks: Respondents increasingly relied on task-specific decision processes as more questions were asked, leading to reduced external validity for both related tasks and real-world behaviors. Importantly, the external validity of measured preferences peaked after as few as seven questions in both types of tasks. When measuring preferences, less can be more.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.