A good poll story begins with a good poll. At the heart of a good
poll is a randomly selected representative sample of the target
population. Unfortunately, bad polls and bad samples are everywhere, and
stories based on those flawed polls find their way on air or into print
with dismaying frequency. One reason is that it’s hard and
sometimes prohibitively expensive to collect a random or representative
sample. Instead, some researchers use convenience samples.
One common type of convenience sample produces surveys that researchers
call self-selected opinion polls, or SLOP surveys. As the name
suggests, the sample in a SLOP survey is not selected randomly. Instead,
individuals choose whether to participate. Margin of sampling error
cannot be estimated for a SLOP poll, no matter how large. The classic
example of a convenience sample is one done by interviewers who stand in
a shopping mall and ask shoppers as they walk by to fill out a survey.
That’s perhaps a good way to meet new people but a bad way to
select a representative sample of any group. The people who agree to
participate may be different than those who do not.
Researchers have learned, often to their great embarrassment, that
these types of samples often produce flawed results. Respondents who
volunteer to participate in such surveys tend to be more extreme or
otherwise very different in their views than those who do not. In no way
can they be said to be representative of the population, so the survey
results cannot be used to say anything useful about a target
The Internet is awash with SLOP polls that invite people to answer a
question and then view the results. In addition to attracting only those
with an ax to grind on a particular issue, even the best
Internet-derived convenience samples currently tend to include too few
older people, minorities and less affluent, less well educated. In
short, they tend to miss people who don’t have access to a
computer or an Internet connection. These surveys also invite
manipulation, as a number of news organizations have learned to their
Even samples that are selected at
random can be hopelessly flawed if respondents are selected from a pool
that is different than the population the researcher is attempting to
measure. Some examples:
- Sampling Lessons from an Angry, Drunken
If you’re thinking about writing a story based on an Internet poll
or a survey based on some other type of convenience sample, think of
Hank, the Angry, Drunken Dwarf.
People Magazine asked visitors to its website to vote in an online poll
for the Most Beautiful Person of 1998. The ballot included Julia
Roberts, Leonardo DiCaprio, Madonna and other usual suspects. It also
allowed write-in nominations. The temptation proved too great for radio
bad boy Howard Stern, who advised his listeners to email votes for Hank
the Angry, Drunken Dwarf, a Stern sidekick who died in 2001.
A small army of online prankster quickly took up the campaign. Hank
swamped the competition, finishing with 230,169 votes, or about 16 times
the number who supported DiCaprio, the pretty face whom People declared
the fairest of the fair and put on its cover.
The people at the People website were initially appalled and closed
their website. Then they decided to capitalize on all the free
publicity, reopened the poll and publicly embraced Hank, who was duly
crowned as the site's winner of the "Beautiful People Poll," and
pictured wearing a pink bunny suit. "It kind of made us nostalgic for
the days when all we had to contend with was sparring between the Xena
fans and the Hercules supporters," People website editors wrote in a
brief introductory note.
- Man of the Century: Mustafa Kemal
To usher in the New Millennium and say goodbye to the old, Time magazine
sponsored an online poll to name the Person of the Century. At one point
late in the balloting, Jesus Christ led the list, followed by Adolf
Hitler and Eric Cartman (a character on the "South Park" cartoon show
who finished 14th in People's Beautiful People Poll).
At a different point in the balloting, another write-in candidate topped
the list: Mustafa Kemal Ataturk, the founder of the modern, secular
Turkish Republic. It seems that it had become a matter of national pride
to get Ataturk named Time’s Person of the Century, and local
newspapers urged their readers to vote for him and provided the
site’s email address.
Time editors said they would use the results of the web survey to advise
their decision. They passed over Ataturk, Jesus Christ and Eric Cartman
to name Albert Einstein their Person of the Century in its Dec. 31, 1999
- Everybody Loves Tom!
2005 was an Annus Horribilis for actor Tom Cruise. There was that
unfortunate bit of couch-bouncing on the Oprah Winfrey show and his
publicized bashing of Brooke Shields for using physician-prescribed
drugs to treat her depression.e
Parade.com decided to conduct an online poll to see if people thought
Tom had only himself to blame for his troubles or whether it was the
media’s fault. A whopping 84% of respondents blamed the media.
Sounded a bit implausible, or at least that’s what the editors at
Parade thought. But let them tell the story, which they did in a message
to the celebrity gossip blogger Jossip: “We at Parade found this a
little bit fishy—so we did some investigating. We found out more
than 14,000 (of the 18,000+ votes) that came in, were cast from only 10
computers! Furthermore, there was one computer responsible for nearly
8,400 votes alone, all blaming the media for Tom's troubles. We also
discovered that at least two other machines were the sources of
inordinate numbers of votes….There is even a chance they wrote a
special "bot" program for the sole purpose of skewing the results,
rather than casting the votes by hand on a computer.”
- Girls Just Want to Have Too Much
In March of 2006, the American Medical Association reported disturbing
rates of binge drinking and unprotected sex among college women during
spring break. The report was based on what the researchers claimed was a
survey of “a random sample” of 644 women.
The survey results were breathlessly reported on the Today Show, the CBS
Early Show, and hundreds of reports followed on local television and
radio newscasts. The findings also were reported in the Time magazine,
and in a chart that ran in the New York Times.
One problem: The sample was not random. The results were based on only
women who volunteered to answer the question as part of an online survey
panel. Only about a quarter of these women had ever gone on a spring
break trip. The Times eventually published a correction explaining the