The Price We Pay For Not Interviewing Everyone

Publicly released polls almost always (and should always) have a margin of sampling error (MOSE) specified in any report of the results. That is the + or – 3 percentage points, or + or – 2 percentage points you generally see at the end of a method statement. (We often talk about a MOSE as if it applies to the whole survey, but note that in reality it applies to each answer in the survey as you see in all the examples.)

Basically, the margin of sampling error is the price you pay for not talking to everyone in your population group. The MOSE describes the range that the answer likely falls between if we had talked to everyone instead of just a sample. For example, for a telephone sample of 1000 randomly selected adults nationwide, the finding from the poll will be within plus or minus 3 percentage points of the answer we would have gotten had we talked to all 210 million adults. So if one of the findings of the poll was that 58% approved of the job their Governor was doing, we would know that the true value would lie somewhere between 55% and 61% if we had talk to the whole population in the state.

To be technically correct, we really only have some degree of confidence around the MOSE we calculate for probability based samples. Generally, pollsters calculate the MOSE using a 95%confidence level. That is, in 95 times out of a 100, we expect the answer we get from the survey is reflective of the true answer within the MOSE.

What’s so special about 1000 respondents?

MOSE is largely driven by the size of the sample. The larger the sample, the smaller the MOSE, while the smaller the sample the larger the MOSE. Wonder why most polls interview 1000 people? Just look at the chart below. Notice that as the sample size increases the MOSE falls, quite dramatically between small sample sizes of say 100 and larger sample sizes of say 1000. But once we get to 1000, we don’t see much change in the MOSE. In fact, note that between doing 1000 interviews and doing 2500 interviews the MOSE falls by only 1 percentage point. Again, notice that moving between 2500 and 5000 interviews the MOSE falls by only another percentage point. And since one of the main drivers of the cost of a poll is how many interviews you do, there isn’t much “bang for your buck” when you double your sample to these larger sample sizes.

Three more points to make about this chart:

1. The MOSE is largely dependent only on the sample size, NOT the population size. (This is almost always true for populations size 10,000 or more). It doesn’t matter if we are trying to represent the views of all adults in the United States, or the views of all adults in the city of Peoria. This chart doesn’t change.
2. Notice the large MOSE for a smaller sample of say 100 – which means the findings on the poll of 100 adults is between + and – 10 percentage points of the “real” answer for all adults. So if 50% of the sample conducted with 100 adults say they support Candidate A, then the “real” answer most likely lies between 40% and 60% (50% -10 percentage points and 50%+ 10 percentage points). Alternatively if 50% of the sample conducted with 1000 adults say they support Candidate A, then the “real” answer most likely lies between 47% and 53% (50% - 3 percentage points and 50% + 3 percentage points). Note how much smaller this second range is, and how much more we have learned from this larger sample about how much support really exists for Candidate A.
3. If the poll results are based on any subsample of the total sample, for example the findings include not only what all adults say but also show what women say, then the MOSE for the results for that subsample are based on the size of that group, not the total sample. For example, if 1000 adults are interviewed, but the findings show the result of women only, then the sample size to look at in the chart would be at about 500 since women and men are generally distributed about evenly. If instead the results are shown for any other subgroup, you would need to know how many people make up that subgroup in the sample to be able to know the MOSE.

Margin of sampling error is an actual quantity we can measure.  There is no such thing as a measurable margin of error for a poll. 

Poll results are subject to lots of sources of errors ranging from how well the questions were designed and asked to how well the interview was conducted to how well the sample design was implemented.  Good pollsters and researchers do everything in their power to minimize these other possible sources of errors, but they are non-measurable in any case, and one can never know the precise amount of error associated with any poll finding.