Research World. May 2005. Online Panels
Market researchers have been using panels
for over half a century, but it's only in the last seven or eight
years that online panels have emerged as a viable way of conducting
market research. Today, most of the largest market research companies
either have, or are building, their own online access panel.
On the face of it, online panels offer a number of obvious benefits.
They are low-cost, compared to traditional face-to-face or CATI
methodologies. They allow market researchers and their clients
to survey large numbers of people very quickly. And the online
environment enables the researcher to serve up a variety of different
content, including photography, video and audio, for the respondents'
consideration.
But how credible is the data that comes out of an online panel
survey? Detractors argue that respondents may be motivated, not
by a desire to give honest answers, but rather, by a desire to
earn money or "reward points" for their participation
in surveys. They also question how representative an online panel
can be of the population as a whole.
ESOMAR is carrying out is own work on online access panels. An
ESOMAR Panels Project Team has spent the last eight months updating
a set of guidelines for conducting market research on the Internet.
These will be presented at an ESOMAR Panel Seminar, to be held
in Budapest in April.
The guidelines will recommend policies that panel owners should
have in place regarding everything from how often panellists will
be selected for interview, and how often panellists who ignore
or refuse requests to participate are removed from the panel,
to the panel owner's policy on detecting and dealing with duplicates.
"We are not telling people what they should do" says
John O'Brien, a consultant to ESOMAR's Professional Standards
committee, who chairs the Panels Project Team.
"The methodology is still quite new, so it will take time
for right and wrong, good and bad, to emerge. What is important
is that the panel owner should have a policy in place in each
of these areas."
One of the key issues, says O'Brien, is the degree to which an
online panel is representative of the population as a whole, given
the fact that a significant proportion of the population in most
European countries is not online. In this respect, he points to
the work being done by George Terhanian, president of Harris Interactive
(HI) Europe, to ensure that the results produced by online panels
are representative of the population as a whole. Terhanian himself
outlines the problem.
"It is entirely possible to recruit an online panel from
which you can project the findings to the general population of
the country, but it is more difficult than most people realise"
he says. "Building a panel that appears to be reasonably
representative of a country's population is straightforward, although
it is at times difficult to identify and over-recruit light online
users, such as older women. But this is the easy part.
"Because of the self-selection factors involved such
as when you decide to go online, when you decide to join a panel,
and when you decide to take part in a study for that panel, there
are lots of biases, other than socio-demographics, that a good
market research company has to take into account."
HI's answer is what Terhanian calls a multiple-mode approach,
which it adopted when it started building its 7 million-strong
online panel in 1997. Every quarter, HI runs
a parallel, nationally-representative, face-to-face or telephone-based
study, interviewing a cross-section of people within the country,
using a population-wide, probability-based approach. At the same
time, the company interviews a cross-section of its online panel,
taking into account growth, attrition and learning, using the
same questionnaire. The
responses of the participants in the nationally representative
survey serve as a check, and a means of adjusting the online sample
if necessary. This process, says Terhanian, has helped HI to identify
several factors, beyond socio-demographic, that distinguish people
who are in an online panel to those who are not. These factors
then inform the selection of future samples, and the means by
which the resulting data are weighted in order to achieve a representative
set of results.
Is there anybody out there?
The representativeness of the sample is not the only issue. Finding
people to take part in online samples is also a problem, according
to David Pring, senior vice president & general manager of
Ipsos-Insight.
"Online is like other media" says Pring. "Consumers
don't want to answer market research questions. Over the past
2-5 years, we have started to see people becoming much less co-operative
with online research companies."
In the early days of online research, he argues, consumers were
enthused by the chance to win prizes, but as the novelty has worn
off, and the majority of panel owners have moved to small monetary
or points-based reward schemes, this enthusiasm has waned.
"There is a relatively finite pool of people in the US who
want to get involved" says Pring.
Incentives clearly have a part to play in this, and while care
must be taken not to over-reward participants, Pring believes
the industry does need to reassess the way participants are compensated
for giving up their time and opinions.
"The reward structure is a big problem, and has been since
the beginning" he says. "We make multimillion-dollar
decisions, but we are disinclined to pay respondents anything
like a reasonable sum for their time and involvement. If someone
was on a panel, and at the end of the month, they got 30-50 dollars,
maybe they would feel it's kind of good and fun, but if they only
get a couple of dollars each time, so after half a year they've
made 20 dollars, then maybe not. But that's where we are today,
because clients have looked at it and said: 'I can save money',
not 'I should be giving respondents more money for giving up their
time.'"
Some practitioners, however, remain wary of monetary-based rewards.
"There are a lot of points-based and monetary incentives
in the UK and the US, but if you only use points or money to recruit
people, then you are only addressing a very active but small proportion
of the people you want" says Kees de Jong, CEO of Bloomerce
Access Panels Europe.
In other European countries, says de Jong, other mechanisms are
used successfully.
"The best way to get respondents is to address an intrinsic
interest in answering surveys on a particular topic, not just
pay them" he says.
Another tactic used by Bloomerce and gaining in popularity elsewhere,
is for the panel owner to make a contribution to a charity of
the respondents' choice. De Jong concedes, however, that there
is a limited number of reward models, and clearly, the means of
incentivising online panel members for their participation is
an issue which will challenge panel owners over the next three
to five years, as more panels are established, creating more competition
for David Pring's "finite pool of people."
The other key challenge, says Anne Hedde, CEO of Lightspeed Research,
is to ensure that these panellists provide good quality data.
She says:
"The ability for a panel member to self-select and opt in
to a survey, as opposed to the panel provider inviting the panellist
to participate, is allowing a 'professional panellist' syndrome
to be created. If a panellist can join 20 panels and if they are
allowed to opt in to surveys, that will lead to poor quality data.
"If, on the other hand, you build a good, open panel from
a lot of different sources and have good stringent business rules,
you will get good quality data."
Whether or not this happens, she says, depends on two factors.
"The first is that we set global standards and business rules
on how we recruit and how many surveys panellists can take"
she says. "But change can also come from research buyers
demanding quality. They have to know what they are buying and
they have to demand respondents who have been recruited in a variety
of ways, who cannot self select, and who are fairly paid, not
too much or too little, for their participation."
This, it seems, is where most of the research industry would like
to be. Clearly however, it's not there yet.