AAPOR
The leading association
of public opinion and
survey research professionals
American Association for Public Opinion Research

Falsification-Certification

  1. Preface

In many research institutions covered by the Office of Research Integrity oversight there exist ongoing research activities using survey data collections. Some of these organizations are conducting hundreds of surveys each year, with hundreds of thousands of human subjects. Given the common practice of using temporary staff as interviewers, this volume of activity inevitably produces a consistent flow of inquiries about possible interviewer falsification or fabrication.

For that reason, to permit research integrity officers and ORI to utilize their resources most productively, an operationalization of the de minimus standards for ongoing survey research activities seems appropriate.

The following includes a three-step process:

a. A certification process for a unit with ongoing survey activities, using peer review verification that falsification detection and repair procedures are consistent with best practices espoused by the American Association for Public Opinion Research and the American Statistical Association

b. Notification by the certified unit to the RIO using preapproved protocols that an de minimus incident has occurred

c. Annual reporting of falsification detection activities by the unit to the RIO.

  1. Operational Definitions

Key Personnel. For purposes of this protocol, only staff not deemed “key personnel” on the research project will be in scope. Said alternatively, all key personnel will be subject to then-current RIO and ORI investigatory actions appropriate as events judged above the de minimus levels.

“Key personnel” include the principal investigator, co-investigators, and other individuals who contribute to the scientific development or execution of a study in a substantive, measurable way. Typically, these individuals have doctoral or other professional degrees, although individuals at the masters or baccalaureate level are sometimes considered key personnel depending on their involvement in a project.

Material Impact on Research. Alleged events of interviewer falsification or fabrication will be classified as falling below or above a de minimus level. The impact of an act of falsification on the scientific conclusions of a survey are a function of sample size (i.e., in a survey of 25,000 subjects, falsification of 50 cases will not typically affect any scientific conclusions). However, the seriousness of the violation of scientific integrity represented by the event is a function of numbers of falsified or fabricated records, regardless of the sample size of the survey. For that reasons, the protocol uses two criteria for the de minimus definition:

If in a single study, a single interviewer or a group of colluding interviewers allegedly falsify either more than 50 interviews (i.e., with positive verification that they were falsified) or more than 2% of the cases that would have been used in analysis if the falsified records were not removed or repaired, then the alleged event will exceed the de minimus threshold.

            Such a decision rule is illustrated in the graphic below:

 

Survey Research Organization. Since the certification process involves a formal proposal to the RIO, followed by peer review, it is expected to be used most often by semi-permanent survey research organizations. A “survey research organization” is an  institution or semi-permanent unit within an institution (e.g., a budget item, appointment of a director by higher authorities, a staff, space assignment) that has as one of its primary purposes the interviewer-assisted collection of survey data for research purposes.

  1. Certification of Protocol Compliance with Best Practices Statement

The process of certification begins with a request from a survey research organization to the research integrity officer of the institution:

To obtain certification:

a. The director of the survey research organization develops a set of procedures for falsification that are consistent with the best practices statement approved by AAPOR and ASA

b. The director and the local RIO review the procedure

c. RIO and the survey research organization will solicit peer review by external survey research experts of the procedures, who will recommend amendments for conformity with the best practices statement and submit their review to the RIO

d. RIO approves the plan

After certification is obtained:

a. After approval the survey research organization may later amend the plan, with the agreement of the RIO

b. Once a year, the survey research organization will submit to the RIO a report documenting efforts aimed at preventing and detecting falsification, as part of the documentation required under the best practices statement.

  1. Defining the Routine Reporting Requirements Below the de minimus Threshold

Once the survey organization is certified under the protocol, it will begin the process of routine reporting of de minimus events, as they occur during the course of its research activities:

  1. For events that fall below the de minimus threshold, short documentation of the event will be forwarded to the RIO at the time of their occurrence:

                                     i.This documentation would be that normally recorded as part of the verification process and personnel process involved in terminating the interviewer.

                                     ii.This documentation would describe the elimination of the falsified data and the efforts to replace the falsified data with data reported by the sample unit in follow-up attempts.

                                     iii.All records pertaining to the alleged falsification, including the suspect data records removed from the analysis file, will be retained in a form allowing for their subsequent review and/or analysis for a period of five years, or in the case of personally-identifiable human subject data, to the maximum time that may have been established by an institutional review board.

                                     iv.The RIO would review the event report and if he/she judged it below the de minimus thresholds, would retain the report.

                                     v.The documentation of the case forwarded to RIO would not include the individually identifiable information about any research subjects.

  1. Defining the Process Requirements for Events Above the de minimus Thresholds

Certified survey research organizations would be required to work closely with the RIO when it appeared that an alleged interviewer falsification or fabrication event was likely to exceed the de minimus thresholds:

a. When a case appears to be approaching the triggering level, the director of the survey research organization will contact the RIO.

b.All records pertaining to the alleged falsification, including the suspect data records removed from the analysis file, will be retained in a form allowing for their subsequent review and/or analysis for a period of five years, or to the maximum time permitted by an institutional review board in the case of linked/coded data sets.

Approved by AAPOR Council, May 2005

Endorsed by Council on Government Relations, April 2006

Not approved or endorsed by Office of Research Integrity, as of May 2009

Approved for posting by AAPOR Council, May 13, 2009