AAPOR
The leading association
of public opinion and
survey research professionals
American Association for Public Opinion Research

Disclosure Standards

Section III. of the AAPOR Code of Professional Ethics & Practice (November 2015) specifies:
 
III. Standards for Disclosure


Good professional practice imposes the obligation upon all public opinion and survey researchers to disclose sufficient information about how the research was conducted to allow for independent review and verification of research claims. Full and complete disclosure will be made at the time results are released, either publicly or to a research client, as the case may be. As detailed below, some information, if not immediately available, will be released within 30 days of any request for such materials.

A. Disclosure Items for Surveys
Broadly defined, surveys feature a predetermined set of questions and/or other measurements administered to a set of sample units with the intention of supporting inferences to a population of interest. We will include the following items in any report of survey research results or make them available immediately upon release of that report:
1. Who sponsored the survey and who conducted it. If different from the sponsor, the original sources of funding will also be disclosed.

2. The exact wording and presentation of questions and response options whose results are reported. This includes preceding interviewer or respondent instructions and any preceding questions that might reasonably be expected to influence responses to the reported results.

3. A definition of the population under study and its geographic location.

4. Dates of data collection.

5. A description of the sampling frame(s) and its coverage of the target population, including mention of any segment of the target population that is not covered by the design. This may include, for example, exclusion of Alaska and Hawaii in U.S. surveys; exclusion of specific provinces or rural areas in international surveys; and exclusion of non-panel members in panel surveys. If possible the estimated size of non-covered segments will be provided. If a size estimate cannot be provided, this will be explained. If no frame or list was utilized, this will be indicated.

6. The name of the sample supplier, if the sampling frame and/or the sample itself was provided by a third party.

7. The methods used to recruit the panel or participants, if the sample was drawn from a pre-recruited panel or pool of respondents.

8. A description of the sample design, giving a clear indication of the method by which the respondents were selected, recruited, intercepted or otherwise contacted or encountered, along with any eligibility requirements and/or oversampling. If quotas were used, the variables defining the quotas will be reported. If a within-household selection procedure was used, this will be described. The description of the sampling frame and sample design will include sufficient detail to determine whether the respondents were selected using probability or non-probability methods.

9. Method(s) and mode(s) used to administer the survey (e.g., CATI, CAPI, ACASI, IVR, mail survey, Web survey) and the language(s) offered.

10. Sample sizes (by sampling frame if more than one was used) and a discussion of the precision of the findings. For probability samples, the estimates of sampling error will be reported, and the discussion will state whether or not the reported margins of sampling error or statistical analyses have been adjusted for the design effect due to weighting, clustering, or other factors. Disclosure requirements for non-probability samples are different because the precision of estimates from such samples is a model-based measure (rather than the average deviation from the population value over all possible samples). Reports of non-probability samples will only provide measures of precision if they are accompanied by a detailed description of how the underlying model was specified, its assumptions validated and the measure(s) calculated. To avoid confusion, it is best to avoid using the term “margin of error” or “margin of sampling error” in conjunction with non-probability samples.

11. A description of how the weights were calculated, including the variables used and the sources of weighting parameters, if weighted estimates are reported. 

After survey results are reported, we will make the following items available within 30 days of any request for such materials:
12. Procedures for managing the membership, participation, and attrition of the panel, if a pool, panel, or access panel was used.

13. Methods of interviewer training, supervision, and monitoring, if interviewers were used.

14. Details about screening procedures, including any screening for other surveys that would have made sample members ineligible for the current survey must be disclosed (e.g., in the case of online surveys if a router was used).

15. Any relevant stimuli, such as visual or sensory exhibits or show cards. In the case of surveys conducted via self-administered computer- assisted interviewing, providing the relevant screen shot(s) is optimal, though not required.

16. Details of any strategies used to help gain cooperation (e.g., advance contact, compensation or incentives, refusal conversion contacts) whether for participation in a group, panel or access panel or for participation in a particular research project.

17. Procedures undertaken to ensure data quality, if any. Where applicable, this includes re-contacts to confirm that the interview occurred and/or to verify the respondent’s identity, measures taken to prevent respondents from completing the same survey more than once, and other quality control procedures (e.g., logic checks and tests for speeding and patterning). If no such efforts were undertaken, this will be disclosed.

18. Summaries of the disposition of study-specific sample records so that response rates for probability samples and participation rates for non- probability samples can be computed. If response or cooperation rates are reported, they will be computed according to AAPOR Standard Definitions. If dispositions cannot be provided, the reason(s) will be disclosed and this will be mentioned as a limitation of the study.

19. The unweighted sample size on which one or more reported subgroup estimates are based.

20. Specifications adequate for replication of indices or statistical modeling included in research reports.

B. Disclosure Items for Qualitative Research

Qualitative research includes focus groups, in-depth interviews, case studies, narrative research, and ethnography, among other approaches. This research generally involves descriptive, unstructured data. We will include the following items in any report of qualitative research or make them available immediately upon release of that report:
1. Who sponsored the research and who conducted it. If different from the sponsor, the original sources of funding will also be disclosed.

2. A definition of the population under study and its geographic location.

3. The instrumentation used (e.g., questionnaires, discussion guides), a description of the data collection strategies employed (e.g., focus groups, semi-structured interviews), and the language(s) used.

4. A description of any relevant stimuli, such as visual or sensory exhibits or show cards.

5. Dates of data collection.

6. The physical location of all data collection activities (e.g., subject home, office/workplace, clinic, focus group facility, street corner).

7. A description of subject eligibility (e.g., age or gender requirements) and the procedures employed to screen and recruit research subjects.

8. The number of research subjects, by data collection strategy.

9. Methods of interviewer and/or coder training, supervision, and monitoring, if interviewers or coders were used.

10. Duration of research participation (e.g., length of interviews, focus group sessions).

11. Any compensation/incentives provided to research subjects.

12. Information regarding whether or not data collection included audio or video recordings.

C. Disclosure Items for Content Analysis

Content analysis includes the systematic analysis of text, images or other content. Content analysis can be qualitative or quantitative and the material to be analyzed can be from any source (e.g., open-ended verbatim responses collected in surveys, newspaper and magazine articles, television and radio broadcasts, politicians' speeches, tweets, or posts on social networking sites). Content analysis can be performed by human coders and/or via automated coding software. We will include the following items in any content analysis report or make them available immediately upon release of that report:
1. Who sponsored the research and who conducted it. If different from the sponsor, the original sources of funding will also be disclosed.

2. A description of how the content analyzed was collected or obtained. This discussion will state the source(s) used, how much content was analyzed (e.g., number and average length of articles, tweets or blog postings, news broadcasts), dates analyzed, language(s) included, and criteria or decision rules used to include or exclude elements of content.

3. The sampling approach used. If a census of the target population of content was used, that will be explicitly stated.

4. A discussion of any threats or concerns about the validity or quality of the content (e.g., online commentary created by bots, fabricated social media profiles, relevant content missing from the source(s) used) and any steps taken to address them.

5. A description of how the analysis was conducted. This discussion will mention whether coding was done by software or human coders (or both) and whether quantitative or qualitative methods (or both) were used. For analysis using automated coding, this discussion will mention the software and the parameters or decision rules that were used. For analysis using human coding, this discussion will mention the number of coders and any training and instructions provided to them. If a formal coding scheme was used, it will be provided. If no formal coding scheme was used, this will be disclosed.

6. Inter-coder reliability, amount of text analyzed by multiple coders, process for resolving inconsistencies, and any steps taken to increase the level of agreement, if multiple coders were used. If only one coder was used, this will be disclosed.

7. The unit of analysis (e.g., a news article, broadcast, tweet, or blog posting).

8. A description of how the weights were calculated, including the variables used, if weights were used to develop a coding scheme and/or produce final estimates.

9. Other information that may need to be disclosed if the content analysis involves qualitative data collection (see Section III-B) or survey data collection (see Section III-A).

D. If the results reported are based on multiple samples or multiple modes, the preceding items (from Section III-A, III-B, or III-C, as applicable) will be disclosed for each.

E. Reflecting the fundamental goals of transparency and replicability, AAPOR members share the expectation that access to datasets and related documentation will be provided to allow for independent review and verification of research claims upon request. Datasets may be held without release for a period of up to one year after findings are publicly released to allow full opportunity for primary analysis. In order to protect the privacy of individual respondents, such datasets must be de-identified to remove variables that can reasonably be expected to identify a respondent. Those who commission publicly disseminated research have an obligation to disclose the rationale for why eventual public release or access to the datasets is not possible, if that is the case.

F. If any of our work becomes the subject of a formal investigation of an alleged violation of this Code, undertaken with the approval of the AAPOR Executive Council, we will provide additional information on the research study in such detail that a fellow researcher would be able to conduct a professional evaluation of the study.


Frequently Asked Questions about AAPOR’s Standards for Minimal Disclosure