Report to AAPOR Standards Committee on the Status of Interviewer Falsification Practices and Policies

In the past few years, the Office of Research Integrity has taken on cases of interviewer fabrication or falsification of data. While Survey Research Organizations (SROs) had been dealing with the problem of “curbstoning” for decades, their typical response to the problem did not fit with the ORI policy on research misconduct that had been in existence for about 15 years. In June 2002 an article was published in the ORI Newsletter stating definitively that “fabrication or falsification of data by lower-level staff who conduct surveys or interviews or administer questionnaires with human subjects” constitutes research misconduct. Faced with the realization that a large portion of the SRO community was unaware of their obligation to report falsification to their institutional Research Integrity Officers (RIOs) as possible misconduct, in July 2002 Dr. Alan Price wrote letters to the heads of seven major SROs and five associations in survey research. Dr Price followed up with phone calls to two of each type of organization about the possibility of bringing together the SRO community to tackle the issue and received a positive response.

The Ann Arbor  Falsification Summit  (2003)

Led by Dr. Robert Groves of the ISR at Michigan, the SRO community initiated further discussions, which included the ORI and several institutional RIOs, of how to appropriately handle allegations of interviewer falsification. A summit conference of representatives of governmental, private and academic survey research organizations was convened in Ann Arbor, MIin April 2003 by Dr. Groves to develop a “Best Practices” statement for the prevention, detection, and repair of interviewer falsification. The resulting document, “Interviewer Falsification in Survey Research: Current Best Methods for Prevention, Detection, and Repair of Its Effects,” was subsequently adopted by the American Association of Public Opinion Research and by the Survey Research Methods Section of the American Statistical Association.

Falsification SummitII (2005)

After the initial development of the best practices document, members of the SRO community wanted to explore ways of handling minor allegations of falsification that, from their perspective, should not trigger scientific misconduct procedures or mandatory reporting to ORI. To pursue this, the CIC RIOs (the Big Ten institutions plus UI Chicago) met with Drs. Price and Grovesin April, 2004. The general consensus of the RIOs at the meeting was that many curbstoning cases had little impact on the overall research and that reviewing an allegation of curbstoning under an institution’s misconduct procedures, which often takes at least many months to complete, against a part-time, at will employee with no other connection to the research community and where the alleged curbstoning did not have a significant impact on the research was not a good use of institutional compliance resources.

Over the next year a working group of RIOs completed a draft policy somewhat similar to the de minimus ORI standard for plagiarism cases. The SRO community endorsed the proposed process and a second Ann Arbor summit conference was then convened in April 2005 to help flesh out the policy with specific attention paid to what de minimus thresholds would be acceptable to the RIOs and ORI. The implementation plan detailed a certification process for units with ongoing survey activities, using peer review verification that falsification detection and repair procedures are consistent with best practices espoused by the American Association for Public Opinion Research and the American Statistical Association. If an SRO was certified, minor interviewer falsification incidents would be handled locally by the survey organization, using industry best practices as approved by the RIO, while more serious incidents would be subject to the full scrutiny of the ORI process. The recommended threshold is that an event would exceed the de minimus standard if, in a single study, a single interviewer or a group of colluding interviewers allegedly falsify either more than 50 interviews or more than two percent of the cases. The plan to implement the policy, including the certification process was approved by the AAPOR Executive Council on May 2005.

Current Status

In April 2006 the original Best Practices Statement on Interviewer Falsification and the endorsed Implementation Policy for handling de minimus cases was presented to the Council on Government Relations for their comment and consideration. COGR endorsed the report and policy. Despite the support of COGR, to date this plan has not been formally approved or endorsed by ORI. 

Thanks to Ron Langley of the University of Kentucky for preparing this report.

Approved for posting by AAPOR Council, May 13, 2009.