Report of the AAPOR Ad Hoc Committee on the 2008 Presidential Primary Polling

2008 Presidential Primary Polling

Contact: Michael Traugott (mtrau@umich.edu; 734. 846.4469)
OR Nancy Mathiowetz (nancym2@uwm.edu; 414.229.2216)

FOR IMMEDIATE RELEASE

Investigating the 2008 Presidential Primary Polls

LENEXA, Kansas ------ After an extensive investigation, a committee of the American Association for Public Opinion Research (AAPOR) has identified several factors that contributed to inaccuracies in 2008 Presidential Primary polls, most notably the miscalling of the New Hampshire Democratic primary.

The special AAPOR committee composed of leading academic and private sector experts in public opinion and survey research conducted the analysis. The results show that several methodological factors combined to undermine the accuracy of predictions in New Hampshire, South Carolina, Wisconsin and California. Although the limited data available made it impossible to conduct definitive tests of all likely sources of different poll performance, the following factors were identified as likely reasons for the polling errors in New Hampshire, where many polls mistakenly predicted an Obama victory:

• Given the compressed caucus and primary calendar, polls conducted before the New Hampshire primary may have ended too early to capture late shifts in the electorate™s preferences there.
• Most commercial polling firms conducted interviews on the first or second call, but respondents who required more effort to contact were more likely to support Senator Clinton. Instead of continuing to call their initial samples to reach these hard-to-contact people, pollsters typically added new households to the sample, skewing the results toward the opinions of those who were easy to reach on the phone, and who more typically supported Senator Obama.
• Non-response patterns, identified by comparing characteristics of the pre-election samples with the exit poll samples, suggest that some groups who supported Senator Clinton”such as union members and those with less education”were under-represented in pre-election polls, possibly because they were more difficult to reach.
• Variations in likely voter models could explain some of the estimation problems in individual polls. Application of the Gallup likely voter model, for example, produced a larger error than was present in the unadjusted data. The influx of first-time voters may have had adverse effects on likely voter models.

œThis analysis suggests some important explanations for the errors in the 2008 New Hampshire Presidential Primary and raises significant questions for research on pre-election polling methods, said AAPOR President Richard Kulka.

The committee also concluded that several factors were unlikely to have contributed to estimation errors in the New Hampshire pre-primary polls. Among these factors was the so-called œBradley effect in which it is hypothesized that poll respondents say they support a Black candidate in order to appear unbiased, but then cast their ballots for a white candidate in the privacy of the voting booth.

œMany New Hampshire polls predicted Barack Obama would beat Hillary Clinton in that state, said Michael Traugott, committee chair. œSo when Clinton won, some people pointed to latent racism as the reason. But in the data we have from a wide variety of New Hampshire pre-election and exit polls, we found no evidence that white respondents over-represented their support for Obama.

Other factors that the committee discounted:

• The exclusion of cell phone only individuals from samples did not seem to have an effect.
• Use of a two-part candidate preference or œtrial heat question -- intended to reduce the number of œundecided responses -- does not appear to have affected distributions of candidate preference.
• There is little evidence that Independents made a late decision to vote in the New Hampshire Republican primary.

A number of other possible explanations for the error in the New Hampshire primary polls were considered, but insufficient information was available to evaluate them. These explanations include the order of names on the ballot and the short time frame for polling in New Hampshire following the Iowa caucus.

Traugott noted that the analysis also revealed wide variation in the œtrial heat question, arguably the most important question that respondents are asked. In New Hampshire, there were 11 different question wordings used polls for the Democratic primary, and 10 different wordings used in polls concerning the Republican primary. In some versions, the candidates™ names were not mentioned at all. In others, only the œmajor candidates were named. Some polls randomized the candidates™ names, while others did not.

œWe also learned that some polling firms are buying lists of registered voters with phone numbers, and then they are contacting people with interactive voice response technology --basically computerized calls -- and that they™re taking information from the person who answers the phone, who may or may not be the person identified in the voter list. said Traugott. œAnother approach was to interview any registered voter in the household, he said. œThese practices should be the focus of further research.

For the report, supported in part by a grant from the University of Michigan Institute for Social Research (ISR), Traugott and colleagues analyzed individual, household-level response data provided by seven polling organizations. They also compared information on question wording, weighting, interviewer characteristics, sampling frames, and other methodological factors from up to 19 other firms, in many cases relying on publicly available information gleaned from the Internet.

# # # # #

The American Association for Public Opinion Research (AAPOR) is the leading professional organization of public opinion and survey research professionals in the U.S., with members from academia, media, government, the non-profit sector and private industry. AAPOR members embrace the principle that public opinion research is essential to a healthy democracy, providing information crucial to informed policymaking and giving voice to the nation™s beliefs, attitudes and desires. It promotes a better public understanding of this role, as well as the sound and ethical conduct and use of public opinion research.