Summary of Main Findings

The sample recruited via mall intercepts contained more demographic diversity than the sample recruited by a web panel
The web panel sample manifested significantly more topic knowledge and higher response quality than the mall sample
The web panel sample was, on average, in poorer health than the mall sample, but were also more nutrition conscious
Nonetheless, there was virtually no difference in experimental effects between the two samples, suggesting that the sample differences did not moderate consumer reactions to the food label experiment

Grocery shoppers are bombarded by a huge variety of messages on food packages about nutrients contained in the food item and their purported health benefits. The U.S. Food and Drug Administration (FDA) collects empirical data to understand consumer reactions to such messages in order to regulate claims on food labels.

The FDA has migrated from the traditional method of mall-intercepts to collect consumer data to web surveys, which have presented a viable alternative for presenting visual stimuli with more control and efficiency in data collection. Hence, an empirical test of mall-intercept vs. web panel samples was needed to inform the relative merits and weaknesses of each sample source and their implications for study outcomes.

To this end, we analyzed data from an FDA experiment that was deployed concurrently on a sample of consumers intercepted at shopping malls in 5 major metropolitan areas across the country vs. a sample of consumers recruited by a web panel.

Although the sample provider contracted for this work was tasked with providing a reasonably diverse spread of research participants, the charts below reveal notable differences such that the mall sample contained more demographic diversity, while the web panel sample included a disproportionately higher proportion of females and white people, and were older and better educated than the mall sample.

The most critical output from this research was to determine whether results from the food label experiment would be different between the two sample sources. Output from mixed model ANOVAs and multilevel logistic regressions repeatedly revealed that sample source rarely moderated experimental effects on the 7 dependent variables:

whether the respondent thought the food label mentioned any disease-reduction benefits
whether the respondent could identify the responsible nutrient
whether the respondent thought the responsible nutrient can be obtained from foods other than the one tested
extent to which the food label was helpful in conveying the possible health benefits of the food item
extent to which the food was likely to have the stated benefit
extent to which the food label was helpful in purchase decisions
extent to which the respondent would consider buying the food item

Sample source did not moderate the vast majority of main and interaction effects on core dependent variables. In the rare instances where significant effects emerge, sample source produced no systematic impact on experimental effects. A few examples are shown below.

The web panel sample appeared more knowledgeable than the mall sample about nutrients in foods, and also reported taking more types of dietary supplements than mall respondents, regardless of whether it was a multi-vitamin/mineral supplement, single ingredient supplement, or herbal/botanical supplement. The web panel respondents were more likely than the mall respondents to be on a low-carb diet, shop for their own groceries, and were more likely to describe themselves as overweight rather than underweight or about the right weight. The web panel sample also reported higher rates of cancer, hypertension, high cholesterol, and obesity than the mall sample. Taken together, these differences suggest that the web panel sample was, on average, in poorer health than the mall sample, but were also more nutrition conscious.

Web panel respondents took almost twice the amount of time mall participants took to view the food labels, especially those that contained the most content. In addition, almost twice as many web panel respondents as mall participants clicked on the back panel to view more information about the food labels. Because the web panel respondents should have been at least as familiar as mall respondents with visual stimuli embedded in surveys, if not more so, these consistent patterns in the paradata are not likely to be caused by their lack of familiarity with the interface. Rather, this evidence suggests that web panel respondents were more engaged and less in a hurry to complete the survey than mall respondents, on average.

Summary of Main Findings

The sample recruited via mall intercepts contained more demographic diversity than the sample recruited by a web panel
The web panel sample manifested significantly more topic knowledge and higher response quality than the mall sample
The web panel sample was, on average, in poorer health than the mall sample, but were also more nutrition conscious
Nonetheless, there was virtually no difference in experimental effects between the two samples, suggesting that the sample differences did not moderate consumer reactions to the food label experiment, thus bolstering confidence that the FDA would have arrived at the same research conclusion regardless of methodology

The research manuscript with literature review, sampling methodology, analytic approach, and detailed results with all main and interaction effects from the mixed model ANOVAs and multilevel logistic regressions can be found here <PDF>

Citation: Chang, LinChiat and Chung-Tung Jordan Lin. 2012. Comparing FDA Food Label Experiments Using Samples from Web Panels vs. Mall-Intercepts. Manuscript in Press @ Field Methods.