Summary of Main Findings
- Comparative analyses were conducted on a total of 163 measures of practice characteristics, treatment choices, choice of risk factors used in disease severity assessment, attitudes and perceptions.
- A few differences emerged on clinical practice tenure and dosing of specific agents
- But these few differences produced no convergent or systematic impact on core metrics
- No evidence that using one or the other sample source would have led to different conclusions on the key business questions
Although physician web panels are routinely used in healthcare market research, there are persistent concerns about the quality of panel samples. Concerns about panels of medical doctors tend to revolved around 3 main themes:
- Panel conditioning occurs when prior surveys change respondents’ behaviors or change the way respondents answer questions on subsequent surveys. This reactive effect of prior surveys on later responses can take different forms: repeated surveys may raise consciousness about specific domains and affect respondents actual choices and opinions; repeated surveys on the same topics may crystallize attitudes and/or result in more extreme attitudes; and unmotivated panel members become increasingly savvy about how to respond in order to finish the survey quickly to earn substantial cash honorariums in the least amount of time possible.
- Panel attrition is not a problem if attrition is random across all panel members. But a panel that is representative at the outset can deteriorate in quality when it loses panel members that disproportionately hold certain characteristics. For example, if doctors with higher patient loads are more likely to quit the panel than those with lower patient loads, or if panel members who are less satisfied with the level of cash honorariums are more likely to quit the panel than those who are happy with the current level of compensation.
- Self-selection biases can undermine the representativeness of panel samples because doctors who agree to serve on panels may have attitudes, preferences, lifestyles, and experiences that are different from doctors who are not on the panel.
We compared web panel samples of medical doctors to "fresh" random samples recruited via telephone across 3 medical specialties: Neurology, Pulmonology, and Pediatrics. All physicians, regardless of sample source, completed the surveys via the Internet.
Study One
This study was conducted to assess market share for a specific scenario in the treatment of asthma and COPD (Chronic Obstructive Pulmonary Disease). The sampling frame was a target list of high-prescribing physicians provided by the pharmaceutical company that commissioned this research.
Among Pulmonologists/Allergists, the non-panelists had a few more years in clinical practice than those on the web panel; but both specialities showed no other differences on practice characteristics such as patient volume, patient characteristics, and time spent in direct clinical care.
Because the focus of this study was to assess market share in response to a specific future scenario in the treatment of asthma and COPD, it was critical to explore whether the market share estimates would differ significantly between the two sample sources. As shown below, there was no significant difference between the two sample sources on the market share projections of pulmonologists and allergists.
In contrast, a couple differences emerged among pediatricians, such that panel pediatricians appeared to be more cautious with high dosing of Product A and more comfortable with low dosing of Product B, compared to pediatricians in the “fresh” sample. Examining the projected prescription volumes within Product A alone, it is apparent that non-panelists expected to prescribe about the same amount of high dose (29%) vs. low dose (30%) of Product A, whereas the panelists expected to prescribe only about half the amount of high dose (19%) vs. low dose (37%) of Product A. The ratios of projected prescription volumes within Product B alone did not exhibit the same dramatic gap, but slight trends suggest that non-panel pediatricians were less conservative than panel pediatricians in Product B dosing as well.
Study Two
This study was conducted to provide data for a market segmentation among high prescribers treating multiple sclerosis. The sampling frame was again a target list of high-prescribing physicians provided by the pharmaceutical company that commissioned this research.
There was no difference between the two sample sources on most practice characteristics except one - non-panel neurologists treated fewer patients per month than their counterparts on the web panel. This difference was inconsequential because the business focus of this research was on the treatment of patients with severe, relapsing forms of multiple sclerosis, which was not different between the two sample sources.
The neurologists were asked to indicate their likelihood to consider each one of 36 clinical risk factors when determining whether a newly diagnosed patient with a relapsing form of MS has a more aggressive form of the disease, using a 7-point likelihood scale. There was no significant difference between the two sample sources on all 36 clinical risk factors. In addition, the relative importance of those risk factors were comparable between sample sources, because the ratings were highly correlated with Pearson’s r = .99, p<.001.
The neurologists were also asked to rate their agreement with 76 attitudinal measures tapping their opinions on a wide range of domains such as drug dosing, drug administration, cost issues, long-term disease control, patient empowerment, aggressive vs. conservative therapy, interferon therapy, as well as their tendency to switch treatment regimens.
Out of the 76 attitude items, only 5 yielded significant differences between the two sample sources. The Bonferroni correction is appropriate in this circumstance when we are conducting a series of so many tests of significance on variables that are not entirely independent of one another and based on data from the same set of respondents. Given that we have as many as 76 attitudinal variables, if we go on testing long enough we will find a difference that reaches statistical significance simply by chance alone. Thus it is important to apply the family-wise error rate on this set of significance tests to ensure that we do not attach too much importance to a few significant results among a mass of non-significant ones.
As expected, once the Bonferroni correction was applied, there was no significant difference between the two sample sources on all 76 attitudes. In addition, the relative ratings on those attitudes were highly correlated between the two sample sources, Pearson’s r = .98, p<.001.
Summary of Main Findings
- Comparative analyses were conducted on a total of 163 measures of practice characteristics, treatment choices, choice of risk factors used in disease severity assessment, attitudes and perceptions.
- A few differences emerged on clinical practice tenure and dosing of specific agents
- But these few differences produced no convergent or systematic impact on core metrics
- No evidence that using one or the other sample source would have led to different conclusions on the key business questions
The full manuscript prepared for the JSM conference proceedings can be retrieved here <PDF>
Citation: Chang, LinChiat and Jeremy Brody. 2010. Comparing Web Panel Samples vs. Non-Panel Samples of Medical Doctors. Paper presented at the 2010 Joint Statistical Meetings in Vancouver, Canada.