AAPOR
The leading association
of public opinion and
survey research professionals
American Association for Public Opinion Research
It’s impractical to poll an entire population—say, all 145 million registered voters in the United States. That is why pollsters select a sample of individuals that represents the whole population. Understanding how respondents come to be selected to be in a poll is a big step toward determining how well their views and opinions mirror those of the voting population.

To sample individuals, polling organizations can choose from a wide variety of options. Pollsters generally divide them into two types: those that are based on probability sampling methods and those based on non-probability sampling techniques.

For more than five decades probability sampling was the standard method for polls. But in recent years, as fewer people respond to polls and the costs of polls have gone up, researchers have turned to non-probability based sampling methods. For example, they may collect data on-line from volunteers who have joined an Internet panel. In a number of instances, these non-probability samples have produced results that were comparable or, in some cases, more accurate in predicting election outcomes than probability-based surveys.

Now, more than ever, journalists and the public need to understand the strengths and weaknesses of both sampling techniques to effectively evaluate the quality of a survey, particularly election polls.

Probability and Non-probability Samples
In a probability sample, all persons in the target population have a known chance of being interviewed and, ideally, no one is left out. For example, in a telephone survey based on random digit dialing (RDD) sampling, there is a known probability that a particular telephone number will be selected. (A description of RDD sampling and other techniques commonly used in election surveys appears at the end of this brief.)

The major advantage of a probability-based sample is that we can calculate how likely the findings from the sample accurately represent the full population. That is, we can calculate the margin of sampling error, which is basically the price we pay for not interviewing every member of the population. This ability to estimate, within a specified range, the accuracy of survey findings has made probability-based sampling the cornerstone of modern survey research.

Non-probability sampling methods do not share this feature. Participants are included in the sample by other means—typically because they volunteer—so that a person’s chance of being in the sample is unknown. For example, in an opt-in sample a person accepts an invitation to complete a survey that is offered to all visitors to a website. The chance of that person visiting that website and then choosing to participate in the survey cannot be known. One serious consequence is that only certain types of people may choose to opt into the survey and they may be different than those who do not in ways that bias the final results. This is the critical difference between probability and non-probability sampling.

With non-probability samples is there is no simple way to calculate the “margin of error;” instead, estimates of the likely error must be based on a statistical models. As a result, AAPOR has cautioned that it may be misleading to report a margin of sampling error for surveys based on non-probability samples.

Nonresponse to polls is a big factor affecting the accuracy of poll results. In a probability sample, the respondents can be thought of as “self-selecting” into the sample. To the extent that the respondents and non-respondents differ systematically on the survey variables—for example, which candidate they support in an upcoming election--nonresponse can bias the poll results, and that is true even if the initial sample was a probability sample. Lower response rates increase the risk of compound bias due to nonresponse. In a similar way, the accuracy of non-probability samples, such as opt-in samples, can be affected by self-selection. In both types of sampling, if the people who participate in the poll are different from those who do not, results can be biased because of these differences.

In addition to sampling method, there are a number of other features of polls that affect the accuracy of the results. For example, how questions are worded or the sequence of questions presented to respondents have been shown to affect poll results and whether they reflect what people in total population really think.

For such reasons, AAPOR’s Code of Professional Ethics calls for transparency in the reporting of sample design, response rates, and the wording of the questions so that these elements can be assessed along with poll results.

Types of Sampling Techniques

Probability Samples
   
Non-probability Samples
  Download PDF Version