Best Practices for Survey Research
"The quality of a survey is best judged not by its size, scope, or prominence, but by how much attention is given to [preventing, measuring and] dealing with the many important problems that can arise."
--"What is a Survey?", American Statistical Association
Below you will find recommendations on how to produce the best survey possible. If you would like guidance on how to assess the quality of a survey that has already been conducted, please see the AAPOR Task Force report “Evaluating Survey Quality in Today’s Complex Environment” as well as AAPOR’s Election Polling Resources.
How to produce a quality survey:
- Have specific goals.
- Consider alternatives data beyond a survey.
- Select samples that well represent the population to be studied.
- Use designs that balance costs with errors.
- Take great care in matching question format and wording to the concepts being measured and the population being studied.
- Pretest questionnaires and procedures.
- Train interviewers carefully on interviewing techniques and the subject matter of the survey.
- Check quality at each stage.
- Maximize cooperation or response rates within the limits of ethical treatment of human subjects.
- Use appropriate statistical analytic and reporting techniques.
- Develop and fulfill pledges of confidentiality given to respondents.
- Disclose all methods of the survey to allow for evaluation and replication.
1. Have specific goals for the survey.
The objectives of a high-quality survey or poll should be specific, clear-cut and unambiguous. Such surveys are carried out solely to develop statistical information about the subject, not to produce predetermined results, nor as a ruse for marketing, fund-raising, changing voters' minds, or similar activities.
Back to top
2. Consider alternative data beyond a survey.
In its initial conceptualization, the ideal survey takes seriously the important question of whether or not the information needed would best be acquired by conducting a survey or poll. A survey generally originates when an individual or institution is confronted with a need for information for which existing data appear to be insufficient. At this point, it is important to consider if the required information can even be collected by a survey or whether a survey would actually be the best way to acquire the information needed. Additionally is a survey alone sufficient or should the survey data be augmented in some way using nonsurvey data. If a survey is indeed appropriate, then careful attention must be given as to who is to be sampled and what is to be learned about those sampled.
Back to top
3. Select samples that well represent the population to be studied.
A replicable or repeatable plan is developed to randomly choose a sample capable of meeting the survey's goals. Sampling should be designed to guard against unplanned selectiveness. A survey's intent is not to describe the particular individuals who, by chance, are part of the sample, but rather to obtain a composite profile of the population. In surveys using a probability-based sample, the sample is not selected haphazardly or only from persons who volunteer to participate. It is scientifically chosen so that each person in the population will have a chance of selection and that chance, or “probability,” is known. This way, the results can be projected from the sample to the larger population with known levels of certainty/precision.
Some surveys use what’s known as nonprobability samples, where respondents opt-in or volunteer to take the survey. In this case, the probabilities of selection are unknown, hence the name “nonprobability.” Since the chances of being included are not known, typically modeling, weighting or other adjustments are necessary to project from the sample to the larger population.
In probability-based samples, critical elements in an exemplary survey are: (a) to ensure that the right population is indeed being sampled in order to address the questions of interest; and (b) to locate (or "cover") all members of the population being studied so they have a chance to be sampled. The quality of the list of such members (the "sampling frame") whether it is up-to-date and complete is probably the dominant feature for ensuring adequate coverage of the desired population to be surveyed. Where a particular sample frame is suspected to provide incomplete or inadequate coverage of the population of interest, multiple frames should be used.
Historically, most surveys taken seriously by social scientists, policy makers, and the informed media have used some form of random or probability sampling, the methods of which have been well grounded in statistical theory and the theory of probability. More recently, changes in society have challenged researchers’ ability to efficiently sample the general public using probability methods, leading researchers to consider alternative, non-probability based methods that might be more cost effective in many cases. In non-probability surveys, critical elements include (a) how the respondents are recruited (panel, river sample, etc.), (b) how respondents are selected for and/or routed to a particular survey and (c) how the data are adjusted after collection.
Back to top
4. Use designs that balance costs with errors.
For example, allocating a survey budget to support a very large sample size, but with insufficient attention to follow-up of non-respondents generally yields results that are less accurate than surveying a smaller, more representative sample. Similarly, allocating most of one's funds to provide a large sample size but with little or no resources devoted to interviewer training would not be prudent. Although sampling errors can be readily estimated using probability sampling methods, they do not reflect the total error of a survey statistic or estimate, which is a function of many different features of a given survey. Survey professionals carefully seek to balance these various types of error in the design and conduct of a particular survey, in order to minimize the total error given the budget or resources available.
Back to top
5. Take great care in matching question wording to the concepts being measured and the population studied.
Based on the goals of a survey, questions for respondents are designed and arranged in a logical format and order to create a survey questionnaire. The ideal survey or poll recognizes that planning the questionnaire is one of the most critical stages in the survey development process, and gives careful attention to all phases of questionnaire development and design, including: definition of topics, concepts and content; question wording and order; and questionnaire length and format. One must first ensure that the questionnaire domains and elements established for the survey fully and adequately cover the topics of interest. Ideally, multiple rather than single indicators or questions should be included for all key constructs.
Beyond their specific content, however, the manner in which questions are asked, as well as the specific response categories provided, can greatly affect the results of a survey. Concepts should be clearly defined and questions unambiguously phrased. Question wording should be carefully examined for special sensitivity or bias. When dealing with sensitive subject matter, techniques should be used that minimize the discomfort or apprehension of respondents or respondents and interviewers if the survey is interviewer administered. Ways should be devised to keep respondent mistakes and biases (e.g., memory of past events) to a minimum, and to measure those that cannot be eliminated. To accomplish these objectives, well-established cognitive research methods (e.g., paraphrasing and "think-aloud" interviews) and similar methods (e.g., behavioral coding of interviewer-respondent interactions) should be employed with persons similar to those to be surveyed to assess and improve all key questions along these various dimensions.
In self-administered surveys careful attention should be paid to the visual formatting of the questionnaire, whether that be the layout of a mail survey or a particular eye towards respondents completing a web survey on a mobile device. Effort should be taken to reduce respondent burden through a positive user experience in order to reduce measurement error and break offs.
Back to top
6. In interviewer-administered surveys, train interviewers carefully on interviewing techniques and the subject matter of the survey.
Insisting on high standards in the recruiting and training of interviewers is also crucial to conducting a quality survey. For high-quality data to be collected, interviewers in telephone or in-person surveys must be carefully trained to do their work properly through face-to-face ("classroom") or telephone training, self-study, or some combination of these. Good interviewer techniques should be stressed, such as how to make initial contacts, how to select respondents for the survey, how to deal with reluctant respondents, how to conduct interviews in a professional manner, and how to avoid influencing or biasing responses. Training should also involve practice interviews to familiarize the interviewers with the variety of situations they are likely to encounter. Time should be spent going over survey concepts, definitions, and procedures, including a question-by-question approach to be sure that interviewers can deal with any misunderstandings that may arise. Additionally a list of frequently asked questions and prepared responses should be provided to interviewers, as well as instruction on how to pronounce difficult names or words in the survey.
Back to top
7. Pretest questionnaires and procedures to identify problems prior to the survey.
High-quality surveys always provide adequate budget and time for pre-testing questionnaire(s) and field procedures. A pre-test of the questionnaire and field procedures is the only way of finding out if everything "works" especially if a survey employs new techniques or a new set of questions. Because it is rarely possible to foresee all the potential misunderstandings or biasing effects of different questions or procedures, it is vital for a well-designed survey operation to include provision for a pre-test. All questions should be pretested to ensure that questions are understood by respondents, can be properly administered by interviewers or rendered by web survey software and do not adversely affect survey cooperation. In circumstances where one is uncertain about the best design or any critical component of such a design, split-sample experiments, which systematically compare the effects of two or more alternatives, should be included either prior to or as part of the pre-testing process to select the most appropriate or effective design(s) or component(s).
Back to top
8. Construct quality checks for each stage of the survey.
Excellent surveys are those that collect information carefully and check and verify each step of the research process. To assure that the proper execution of a survey corresponds to its design, every facet of a survey must be looked at during implementation. Checks must be made at every step to ensure that the sample is selected according to specifications; that the interviewers do their work properly; that the information from questionnaires is edited and coded accurately; that computer data entry is done correctly; and that the computer programs used for data collection and analysis work properly.
Sloppy execution in the field can seriously undermine results. Controlling the quality of fieldwork is done by observing/monitoring, verifying and/or redoing a small sample of the interviews. At least some questionnaire-by-questionnaire checking (including interviewer "edits") and a review of frequencies to monitor questionnaire performance while in the field are also essential to detect omissions (e.g., skip errors) or other obvious mistakes in the data before it is too late to fix them.
Back to top
9. Maximize cooperation or response rates within the limits of ethical treatment of human subjects.
Non-response occurs when members of the sample cannot or will not participate in a survey. Careful sample management and control to ensure that a large proportion of sample members provide the information requested is essential to good survey practice. Proper sample management and control entails such things as adding sample in correctly formulated replicates, tracking the disposition of all cases, monitoring the sample while in the field for potential problems, and "metering" or rationing resources to ensure the collection of data from harder-to-reach respondents. Interviewers must also be carefully equipped through training with effective responses to deal with concerns that reluctant respondents might express.
Specific procedures designed explicitly to stimulate survey cooperation or participation should also be considered, such as (where possible) sending advance letters to sample households or individuals to inform them of the pending survey, offering monetary (i.e., cash) or non-monetary (some other valued reward) incentives to encourage participation, and sending reminders or making follow-up calls to those who do not respond initially. Failure to follow up non-respondents and refusals, in particular, can severely undermine an otherwise well-designed survey.
To deal with this possibility, visits, calls or invitations to individuals in the sample are scheduled with careful attention to such considerations as the best time of day. Allowance is made for repeated attempts (e.g., callbacks at different times and days, multiple reminder emails, etc.) to thoroughly work the selected sample in not-at-home and related situations. Special efforts (e.g., reworking refusals with an experienced interviewer or follow ups in a different mode) are made to persuade reluctant persons to participate.
Back to top
10. Use statistical analytic and reporting techniques appropriate to the data collected.
Excellence in survey and public opinion research requires that data analysis and interpretation be competent and clear and that findings or results be presented fully, understandably, and fairly. The information collected should be critically examined in a search for meaning. Routine reliability studies may be conducted for key measurements.
Special codes should be provided for missing items, indicating why the data are not included. Statistical tables should be clearly labeled, including identification of questionnaire source, and the (unweighted) number of cases forming the base for each cross-tabulation. Sampling errors should be included for all statistics presented, rather than only the statistics themselves, and should account for the any design effect resulting from a complex sample design and/or weighting of the data. If modeling is used, include specifications adequate for replication of indices or statistical modeling included in research reports, such as the statistical technique, variables included and any model fit statistics should be reported (see AAPOR Code of Ethics).
Findings and interpretations should be presented honestly and objectively, with full reporting of all relevant findings, including any that may seem contradictory or unfavorable. Sampling and nonsampling errors including coverage, measurement and reporting errors, response variance, interviewer and respondent bias, non-response, imputation error and errors in processing the data should acknowledged and where possible, be taken into account in the analysis of survey data and interpretation of survey results, in a comprehensive effort to assess error from each perspective. Conclusions should be carefully distinguished from the factual findings, and great care should be taken to be sure that the conclusions and the findings presented are consistent.
Back to top
11. Carefully develop and fulfill pledges of confidentiality given to respondents.
Establish clear intentions and meticulous procedures to assure the privacy of respondents and the confidentiality of the information they provide. Unless the respondent explicitly requests otherwise or waives confidentiality for specified uses, one should hold as privileged and confidential the identity of individual respondents and all information that might identify a respondent with his or her responses.
Exemplary survey research practice requires that one do whatever is possible to protect the privacy of research participants and to keep collected information they provide confidential or anonymous. One must establish clear intentions to protect the confidentiality of information collected from respondents, strive to ensure that these intentions realistically reflect one's ability to do so, and clearly state pledges of confidentiality and their realistic limitations to respondents. That is, one must ensure that the means are adequate to protect confidentiality to the extent pledged or intended, that procedures for processing and use of data conform to the pledges made, and that appropriate care is taken in dealing with directly identifying information (i.e., using such steps as destroying this type of information or removing it from the file when it is no longer needed for inquiry).
Interviewers and other research staff must be carefully trained to uphold and maintain the confidentiality of respondents' identities and the information they provide. Many organizations require staff to sign an explicit oath or pledge of confidentiality to do so before beginning work. In the verification of information, one must protect the identity of respondents from outside disclosure.
One should also assure that appropriate techniques are applied to control for potential statistical disclosure of respondent data. Individual respondents should never be identified or identifiable in reporting survey findings: all survey results should be presented in completely anonymous summaries, such as statistical tables and charts, and statistical tabulations presented by broad enough categories so that individual respondents cannot be singled out.
Back to top
12. Disclose all relevant methods of the survey to permit evaluation and replication.
Excellence in survey practice requires that survey methods be fully disclosed and reported in sufficient detail to permit replication by another researcher and that all data (subject to appropriate safeguards to maintain privacy and confidentiality) be fully documented and made available for independent examination. Good professional practice imposes an obligation upon all survey and public opinion researchers to include, in any report of research results, or to make available when that report is released, certain minimal essential information about how the research was conducted to ensure that consumers of survey results have an adequate basis for judging the reliability and validity of the results reported. Exemplary practice in survey research goes beyond such standards for "minimal disclosure," promulgated by AAPOR and several other professional associations (e.g., CASRO and NCPP) by (a) describing how the research was done in sufficient detail that a skilled researcher could repeat the study, and (b) making data available for independent examination and analysis by other responsible parties (with appropriate safeguards for privacy concerns).
A comprehensive list of the elements required for disclosure by the AAPOR Transparency Initiative are available here.
Back to top
Sources Cited in Best Practices & Survey Practices that AAPOR Condemns
In addition to numerous suggestions forwarded by Council members and other AAPOR members, the following sources were used in compiling these two lists of "practices that AAPOR condemns" and "best practices":
AAPOR (2015). Code of Professional Ethics and Practices.
American Statistical Association (ASA). Ethical Guidelines for Statistical Practice.
American Statistical Association (1995). What Is A Survey? ASA: Section on Survey Research Methods.
American Statistical Association (1995). How to Plan A Survey. ASA: Section on Survey Research Methods.
American Statistical Association (1995). How to Collect Survey Data. ASA: Section on Survey Research Methods.
American Statistical Association (1996). Judging the Quality of a Survey. ASA: Section on Survey Research Methods.
Council of American Survey Research Organizations (1996). CASRO Journal 1996: "Best Practices"What Is It and How Do We Get There? Port Jefferson, NY: The Council of American Survey Research Organizations.
Heller, Harry E. (1996). "Best Practices: A Contrary View." In CASRO Journal 1996: "Best Practices"What Is It and How Do We Get There?, Pp. 65-67. Port Jefferson, NY: The Council of American Survey Research Organizations.
National Council on Public Polls (NCPP). Twenty Questions A Journalist Should Ask About Poll Results.
Research Industry Coalition (RIC). Integrity and Good Practice in Marketing and Opinion Research. Three Industry Position Statements on: Abuses Performed Under the Guise of Research · Respondent and Data Privacy · Misuse of Call-In "Polls".
Research Industry Coalition (RIC). RIC Statement of PROFESSIONAL & ETHICAL STANDARDS for Marketing and Opinion Research.
Turner, Charles F., and Martin, Elizabeth (Eds.). (1984). Surveying Subjective Phenomena (Vol. 1). New York: Russell Sage Foundation.