Focus Groups Misused
October 16, 2000
ANN ARBOR-Several news media organizations misrepresented the results of focus groups and other non-scientific samples of public opinion after last Wednesday's presidential debates, according to officials of the American Association for Public Opinion Research (AAPOR), the leading professional association for public opinion researchers.
"As we look ahead to the next presidential debate, we urge journalists to pay special attention to how they portray the results of focus groups and other instant measures of voters' reactions to the debate," said Murray Edelman, president of AAPOR. "All too often, journalists will state correctly that the results of such samples are not scientific, then go ahead and report them and analyze them as though they were."
Fox News, for example, reported the results of a "FOXNews.com-Speakout.com instant response analysis," in which visitors to their web site were invited to register their reaction to the candidates' comments during the debate. A Fox News Channel correspondent said of the people who participated in the interactive exercise, "This is not a scientific sample, but it is an accurate representation of what Democrats, Republicans and Independents thought, who saw the debate."
"The Fox News correspondent was right when he said that this sample was not scientific, but wrong when he characterized it as an accurate representation," said Edelman. "Measurements of public opinion that are based on the views of people who visit a web site don't reflect the opinions of anyone other than those people who participated."
Various methodological problems associated with web-based surveys were elucidated in an earlier news release issued by AAPOR on September 28, 2000. It can be viewed at
Several television networks, including CNN, NBC and MSNBC, featured interviews with members of focus groups of "undecided voters" in their post-debate analyses, in an attempt to assess the debate's impact upon such voters. But in at least two instances, the correspondents made confusing or misleading statements about the groups.
On MSNBC's post-debate analysis, correspondent Sara James introduced a group of six "undecided voters" in Tampa, Florida, by saying that these people were "by no means representative of undecided voters across the United States, but are a fairly good cross-section of undecided voters in this region of the battle-ground state of Florida."
"Six people is neither a representative sample of undecided American voters nor a meaningful cross-section of any group of any size at all," said Michael Traugott, past president of AAPOR and professor of communication studies at the University of Michigan. "Terms such as these have the potential to be very misleading when not used with care. In this case, these were simply six people from Tampa, Florida, each expressing their own opinions. Nothing more can legitimately be said about who, if anybody, they might represent."
Results of focus groups can be mischaracterized in other ways as well. Based on the opinions of his focus group in Cincinnati, MSNBC analyst Frank Luntz declared that "we have a clear winner in this debate." Then, to support that headline, he asked members of his group to raise their hands in response to the question, "Who thought George Bush did better than you expected he would?" to which nearly all members raised their hands.
"This is not the same question as 'Who do you think won the debate?'" said Edelman. "It is perfectly possible for focus group participants who thought Al Gore won the debate also to agree that George Bush did better than they thought he would."
While nuances of question wording can be debated, this particular issue is compounded by the fact that, in his introduction of Luntz, MSNBC anchor Brian Williams referred to him only as a "political pollster," making no reference to Luntz's long-standing affiliation with the Republican Party.
In 1997, AAPOR found Luntz to be in violation of the Association's Code of Professional Ethics and Practices for "repeatedly refusing to make public essential facts about his research on public attitudes about the Republicans' 'Contract with America.'" The text of AAPOR's statement censuring Luntz can be found here.
Many news organizations are using focus groups as a way to supplement their political coverage by adding a human dimension to their stories. Compared to a telephone survey, a focus group is a low-cost technique for illustrating how citizens are reacting to political events, issues, andcandidates by showing or referring to real people and their opinions.
A focus group is assembled from a group of people who often share some characteristic such as the fact that they remain undecided about their vote choice, they voted for a particular candidate in a previous election, or they are of the same gender or race. The conversation in a focus group is unlike a structured interview because it is based upon responses to broad, open-end questions. Sometimes the discussion can be affected by an especially vocal participant.
The problem with focus groups arises when news organizations try to generalize from such a conversation to the attitudes or opinions of the general public or likely voters from a small hand-picked set of focus group participants. While focus groups can illustrate how some people feelabout issues or candidates, a group of 10 to 20 individuals cannot be used to represent a larger population of citizens.
Reporters should refrain from describing or characterizing the results of a focus group conversation as "representative" of or "reflecting" what the general public thinks or even what a specific subgroup thinks. A focus group conversation is just about how a particular group of people reacted to a series of questions they were asked by a specific moderator.
About AAPOR: The American Association for Public Opinion Research is the primary professional association representing public opinion researchers, and has a strong interest in protecting and strengthening the credibility of survey research. Founded in 1947, AAPOR is an organization of over 1,500 professionals from government agencies, colleges and universities, non-profit organizations, and commercial polling firms who are engaged or interested in the methods and applications of public opinion and survey research.
Back to top