AAPOR Transparency Initiative

AAPOR Transparency Initiative: Rewarding Survey Methodology Disclosure

 Image  Peter Miller, AAPOR President 2009-2010

AAPOR needs a new approach to upholding and promoting its values.

We are known as the organization that publicizes the failure of survey organizations to be open about their research methods. We are clearly on record about the crucial importance of transparency for the integrity and progress of public opinion research. We have worked with journalists for years on how to interpret polls and surveys for the public. Our efforts to study the 2008 pre-election polls and our subsequent public objection to the lack of openness by one organization during that study are recent examples of our attempts to advance the profession. 

But what we have done so far is not enough. Despite decades of work, transparency in public opinion and survey research remains an elusive goal. Often it remains too difficult to get information about how surveys are conducted. Too many researchers do not know how to document their work, or are reluctant to disclose their methods for fear of criticism by non-transparent competitors. Too many significant questions about survey practice remain unaddressed because getting information about how studies are done is onerous or impossible. Too many members of the public have become cynical about survey research because they do not understand how different methods underlie conflicting findings.

AAPOR’s Ad Hoc Committee that studied pre-primary polls in the winter and spring of 2008 intended to release its report in time for our annual meeting in May of that year. The members of the committee hoped their findings would inform polling practice in the general election. Instead, the committee issued its report in April 2009, about a year late, because many organizations that published pre-primary poll results took so long providing methodological information. In the end, the committee had to publish its findings based on partial data.

It is obvious that if an AAPOR committee cannot efficiently gather methodological information for a report commissioned in the aftermath of a significant polling failure (in New Hampshire in 2008), then transparency is not the guiding norm that it should be in our profession. 

The AAPOR Transparency Initiative is a program to place the value of openness at the center of our work. In my view, it should include:

  • our Association’s public recognition of excellence in transparency;
  • a system for collecting and archiving disclosure information;
  • education for organizations in how to make disclosure a routine part of their work;
  • outreach to survey sponsors, users of survey data and the public to reinforce the value of transparency and to publicize the organizations that participate in the initiative;
  • collaboration with other associations (e.g., ASA, CASRO, ESOMAR, NCPP, WAPOR) and academic disciplines in these activities.

Public recognition is the central part of this effort.  Complementing our traditional standards procedures, which focus on violations of our Code, we should give AAPOR’s stamp of approval to survey organizations for timely and complete methodological disclosure. 

A system for collecting and storing disclosed information in one place will offer survey organizations a convenient, standard method for meeting their transparency obligations and provide a useful research tool for the profession.

Education and assistance for survey organizations that seek AAPOR recognition will increase participation in the transparency initiative. 

Outreach to survey sponsors and data users will provide more incentive for organizations to seek AAPOR’s recognition.

Collaboration with our colleagues in other professional organizations is necessary for the effort to succeed.We need all relevant organizations operating in unison.

There is a lot of work involved here, but I think we can have the elements of the initiative in place by the time of our meeting next May in Chicago. Thus far, the Executive Council has endorsed the idea and I have recruited several AAPOR past presidents to serve as an advisory committee. The editors of Survey Practice, our e-journal for survey practitioners, have agreed to host a discussion of the practical issues raised by the initiative. I have had communications with executives at the American Statistical Association, CASRO and WAPOR, all of whom have offered their help. Our colleagues at the Roper Center for Public Opinion Research have offered to assist with information collection and archiving. I have received very useful guidance in informal discussions with some prominent public pollsters.

I now invite everyone in the Association to participate. The program needs your ideas and help.Your commitment to transparency and your understanding of how surveys work will enable us to anticipate obstacles and make the initiative strong and effective. If you commission, conduct or use data from public surveys, you have an important role to play in elevating the norm of transparency to its central place in our profession.