Bad Samples

A good poll story begins with a good poll. At the heart of a good poll is a randomly selected representative sample of the target population. Unfortunately, bad polls and bad samples are everywhere, and stories based on those flawed polls find their way on air or into print with dismaying frequency. One reason is that it’s hard and sometimes prohibitively expensive to collect a random or representative sample. Instead, some researchers use convenience samples.

One common type of convenience sample produces surveys that researchers call self-selected opinion polls, or SLOP surveys.  As the name suggests, the sample in a SLOP survey is not selected randomly. Instead, individuals choose whether to participate. Margin of sampling error cannot be estimated for a SLOP poll, no matter how large. The classic example of a convenience sample is one done by interviewers who stand in a shopping mall and ask shoppers as they walk by to fill out a survey. That’s perhaps a good way to meet new people but a bad way to select a representative sample of any group. The people who agree to participate may be different than those who do not.

Researchers have learned, often to their great embarrassment, that these types of samples often produce flawed results. Respondents who volunteer to participate in such surveys tend to be more extreme or otherwise very different in their views than those who do not. In no way can they be said to be representative of the population, so the survey results cannot be used to say anything useful about a target population.

The Internet is awash with SLOP polls that invite people to answer a question and then view the results. In addition to attracting only those with an ax to grind on a particular issue, even the best Internet-derived convenience samples currently tend to include too few older people, minorities and less affluent, less well educated. In short, they tend to miss people who don’t have access to a computer or an Internet connection. These surveys also invite manipulation, as a number of news organizations have learned to their dismay.

Even samples that are selected at random can be hopelessly flawed if respondents are selected from a pool that is different than the population the researcher is attempting to measure. Some examples:

• Sampling Lessons from an Angry, Drunken Dwarf
  If you’re thinking about writing a story based on an Internet poll or a survey based on some other type of convenience sample, think of Hank, the Angry, Drunken Dwarf.
  People Magazine asked visitors to its website to vote in an online poll for the Most Beautiful Person of 1998. The ballot included Julia Roberts, Leonardo DiCaprio, Madonna and other usual suspects. It also allowed write-in nominations. The temptation proved too great for radio bad boy Howard Stern, who advised his listeners to email votes for Hank the Angry, Drunken Dwarf, a Stern sidekick who died in 2001.
  A small army of online prankster quickly took up the campaign. Hank swamped the competition, finishing with 230,169 votes, or about 16 times the number who supported DiCaprio, the pretty face whom People declared the fairest of the fair and put on its cover.
  The people at the People website were initially appalled and closed their website. Then they decided to capitalize on all the free publicity, reopened the poll and publicly embraced Hank, who was duly crowned as the site's winner of the "Beautiful People Poll," and pictured wearing a pink bunny suit. "It kind of made us nostalgic for the days when all we had to contend with was sparring between the Xena fans and the Hercules supporters," People website editors wrote in a brief introductory note.
• Man of the Century: Mustafa Kemal Ataturk
  To usher in the New Millennium and say goodbye to the old, Time magazine sponsored an online poll to name the Person of the Century. At one point late in the balloting, Jesus Christ led the list, followed by Adolf Hitler and Eric Cartman (a character on the "South Park" cartoon show who finished 14th in People's Beautiful People Poll).
  At a different point in the balloting, another write-in candidate topped the list: Mustafa Kemal Ataturk, the founder of the modern, secular Turkish Republic. It seems that it had become a matter of national pride to get Ataturk named Time’s Person of the Century, and local newspapers urged their readers to vote for him and provided the site’s email address.
  Time editors said they would use the results of the web survey to advise their decision. They passed over Ataturk, Jesus Christ and Eric Cartman to name Albert Einstein their Person of the Century in its Dec. 31, 1999 edition.
• Everybody Loves Tom!
  2005 was an Annus Horribilis for actor Tom Cruise. There was that unfortunate bit of couch-bouncing on the Oprah Winfrey show and his publicized bashing of Brooke Shields for using physician-prescribed drugs to treat her depression.e
  Parade.com decided to conduct an online poll to see if people thought Tom had only himself to blame for his troubles or whether it was the media’s fault. A whopping 84% of respondents blamed the media.
  Sounded a bit implausible, or at least that’s what the editors at Parade thought. But let them tell the story, which they did in a message to the celebrity gossip blogger Jossip: “We at Parade found this a little bit fishy—so we did some investigating. We found out more than 14,000 (of the 18,000+ votes) that came in, were cast from only 10 computers! Furthermore, there was one computer responsible for nearly 8,400 votes alone, all blaming the media for Tom's troubles. We also discovered that at least two other machines were the sources of inordinate numbers of votes….There is even a chance they wrote a special "bot" program for the sole purpose of skewing the results, rather than casting the votes by hand on a computer.”
• Girls Just Want to Have Too Much Fun
  In March of 2006, the American Medical Association reported disturbing rates of binge drinking and unprotected sex among college women during spring break. The report was based on what the researchers claimed was a survey of “a random sample” of 644 women.
  The survey results were breathlessly reported on the Today Show, the CBS Early Show, and hundreds of reports followed on local television and radio newscasts. The findings also were reported in the Time magazine, and in a chart that ran in the New York Times.
  One problem: The sample was not random. The results were based on only women who volunteered to answer the question as part of an online survey panel. Only about a quarter of these women had ever gone on a spring break trip. The Times eventually published a correction explaining the misrepresentation.