"The quality of a survey is best judged not by its size, scope, or
prominence, but by how much attention is given to [preventing, measuring
and] dealing with the many important problems that can arise."
--"What is a Survey?", American Statistical Association
How to produce a quality survey:
The objectives of a high quality survey or poll should be specific, clear-cut and unambiguous. Such surveys are carried out solely to develop statistical information about the subject, not to produce predetermined results, nor as a ruse for marketing, fund-raising, changing voters' minds, or similar activities.Back to top
Consider alternatives to using a survey to collect information.
In its initial conceptualization, the ideal survey takes seriously the important question of whether or not the information needed would best be acquired by conducting a survey or poll. A survey generally originates when an individual or institution is confronted with a need for information for which existing data appear to be insufficient. At this point, it is important to consider if the required information can even be collected by a survey or whether a survey would actually be the best way to acquire the information needed. If a survey is indeed appropriate, then careful attention must be given as to who is to be sampled and what is to be learned about those sampled.
Select samples that well represent the population to be studied.
A replicable or repeatable plan is developed to randomly choose a sample capable of meeting the survey's goals. Sampling should be designed to guard against unplanned selectiveness. A survey's intent is not to describe the particular individuals who, by chance, are part of the sample, but rather to obtain a composite profile of the population. In a bona fide survey, the sample is not selected haphazardly or only from persons who volunteer to participate. It is scientifically chosen so that each person in the population will have a measurable chance of selection. This way, the results can be reliably projected from the sample to the larger population with known levels of certainty/precision.
Critical elements in an exemplary survey are: (a) to ensure that the right population is indeed being sampled (to address the questions of interest); and (b) to locate (or "cover") all members of the population being studied so they have a chance to be sampled. The quality of the list of such members (the "sampling frame")whether it is up-to-date and complete is probably the dominant feature for ensuring adequate coverage of the desired population to be surveyed. Where a particular sample frame is suspected to provide incomplete or inadequate coverage of the population of interest, multiple frames should be used.
Virtually all surveys taken seriously by social scientists, policy makers, and the informed media use some form of random or probability sampling, the methods of which are well grounded in statistical theory and the theory of probability. Reliable and efficient estimates of needed statistics can be made by surveying a carefully constructed sample of a population, provided that a large proportion of the sample members give the requested information. The latter requires that careful and explicit estimates of potential non response bias and sample representativeness be developed.
Use designs that balance costs with errors.
For example, allocating a survey budget to support a very large sample size, but with insufficient attention to follow-up of non respondents to produce a good response rate (cf. item 9, below) generally yields results that are less accurate than surveying a smaller sample with a higher response rate. Similarly, allocating most of one's funds to provide a large sample size but with little or no resources devoted to interviewer training would not be prudent. Although sampling errors can be readily estimated using probability sampling methods, they do not reflect the total error of a survey statistic or estimate, which is a function of many different features of a given survey. Survey professionals practicing at their best carefully seek to balance these various types of error in the design and conduct of a particular survey, in order to minimize the total error given the budget or resources available.
Take great care in matching question wording to the concepts being measured and the population studied.
Based on the goals of a survey, questions for respondents are designed and arranged in a logical format and order to create a survey questionnaire. The ideal survey or poll recognizes that planning the questionnaire is one of the most critical stages in the survey development process, and gives careful attention to all phases of questionnaire development and design, including: definition of topics, concepts and content; question wording and order; and questionnaire length and format. One must first ensure that the questionnaire domains and elements established for the survey or poll fully and adequately cover the topics of interest. Ideally, multiple rather than single indicators or questions should be included for all key constructs.
Beyond their specific content, however, the manner in which questions are asked, as well as the specific response categories provided, can greatly affect the results of a survey. Concepts should be clearly defined and questions unambiguously phrased. Question wording should be carefully examined for special sensitivity or bias. Techniques should be developed to minimize the discomfort or apprehension of both respondents and interviewers when dealing with sensitive subject matter. Ways should be devised to keep respondent mistakes and biases (e.g., memory of past events) to a minimum, and to measure those that cannot be eliminated. To accomplish these objectives, well- established cognitive research methods (e.g., paraphrasing and "think aloud" interviews) and similar methods (e.g., behavioral coding of interviewer-respondent interactions) should be employed with persons similar to those to be surveyed to assess and improve all key questions along these various dimensions.
Pretest questionnaires and procedures to identify problems prior to the survey.
High quality surveys and polls always provide adequate budget and time for pretesting questionnaire(s) and field procedures. A pretest of the questionnaire and field procedures is the only way of finding out if everything "works"especially if a survey employs new techniques or a new set of questions. Because it is rarely possible to foresee all the potential misunderstandings or biasing effects of different questions or procedures, it is vital for a well-designed survey operation to include provision for a pretest. All questions should be pretested to ensure that questions are understood by respondents, can be properly administered by interviewers, and do not adversely affect survey cooperation. In circumstances where one is uncertain about the best design or any critical component of such a design, split sample experiments, which systematically compare the effects of two or more alternatives, should be included either prior to or as part of the pretesting process to select the most appropriate or effective design(s) or component(s).
Train interviewers carefully on interviewing techniques and the subject matter of the survey.
Insisting on high standards in the recruiting and training of interviewers is also crucial to conducting a quality survey or poll. For high quality data to be collected, interviewers in telephone or in person surveys must be carefully trained to do their work properly through face-to-face ("classroom") or telephone training, self-study, or some combination of these. Good interviewer techniques should be stressed, such as how to make initial contacts, how to deal with reluctant respondents, how to conduct interviews in a professional manner, and how to avoid influencing or biasing responses. Training should also involve practice interviews to familiarize the interviewers with the variety of situations they are likely to encounter. Time should be spent going over survey concepts, definitions, and procedures, including a question-by- question approach to be sure that interviewers can deal with any misunderstandings that may arise.
Construct quality checks for each stage of the survey.
Excellent surveys and polls are those that collect information carefully, and check and verify each step of the research process. To assure that the proper execution of a survey corresponds to its design, every facet of a survey must be looked at during implementation. Checks must be made at every step to ensure that the sample is selected according to specifications; that the interviewers do their work properly; that the information from questionnaires is edited and coded accurately; that computer data entry is done correctly; and that the computer programs used for data analysis work properly.
Sloppy execution in the field, in particular, can seriously undermine results. Controlling the quality of fieldwork is done by observing/monitoring, verifying and/or redoing a small sample of the interviews. At least some questionnaire-by-questionnaire checking (including interviewer "edits") and a review of frequencies to monitor questionnaire performance while in the field are also essential to detect omissions (e.g., skip errors) or other obvious mistakes in the data before it is too late to fix them.
Nonresponse occurs when members of the sample cannot or will not participate in a survey. Careful sample management and control to ensure that a large proportion of sample members provide the information requested is essential to good survey practice. A low cooperation or response rate does more damage in rendering a survey's results questionable than a small sample, because there may be no valid way scientifically of inferring the characteristics of the population represented by the non respondents.
Proper sample management and control entails such things as adding sample in correctly formulated replicates, tracking the disposition of all cases, monitoring the sample while in the field for potential problems, and "metering" or rationing resources to ensure the collection of data from harder-to-reach respondents. Interviewers must also be carefully equipped through training with effective responses to deal with concerns that reluctant respondents might express.
Specific procedures designed explicitly to stimulate survey cooperation or participation should also be considered, such as (where possible) sending advance letters to sample households or individuals to inform them of the pending survey, offering monetary (i.e., cash) or non-monetary (some other valued reward) incentives to encourage participation, and sending reminders or making follow-up calls to those who do not respond initially. Failure to follow up non respondents and refusals, in particular, can severely undermine an otherwise well-designed survey.
To deal with this possibility: (a) visits or calls to sample households are scheduled with careful attention to such considerations as the best time of day to call or visit; (b) allowance is made for repeated attempts (e.g., callbacks at different times and days) to thoroughly work the selected sample in not-at home and related situations; and (c) special efforts (i.e, reworking refusals with an experienced interviewer) are made to persuade persons who are inclined not to participate to respond. In mail surveys, it is usually necessary to send reminders and conduct several follow- up mailings, and at times to contact at least a subsample of the remaining non respondents by telephone or personal visit. Where possible, specific efforts to directly observe or measure the characteristics of non respondents should also included in the overall survey design.
Use statistical analytic and reporting techniques appropriate to the data collected.
Excellence in the practice of survey and public opinion research requires that data analysis and interpretation be competent and clear, and that findings or results be presented fully, understandably, and fairly. The information collected should be critically examined in a search for meaning processed, refined, and thoroughly analyzed. Routine reliability studies should be conducted for all key measurements.
Special codes should be provided for missing items, indicating why the data are not included. And, ideally, the "filling in" or imputation of these missing data items (based on rigorous and well validated statistical methods) should be undertaken to reduce any biases arising from their absence. Statistical tables should be clearly labeled, including identification of questionnaire source, and the (unweighted) number of cases forming the base for each cross-tabulation. Sampling errors should be included for all statistics presented, rather than only the statistics themselves.
Findings and interpretations should be presented honestly and objectively, with full reporting of all relevant findings, including any that may seem contradictory or unfavorable. Sampling and nonsampling errors including coverage, measurement and reporting errors, response variance, interviewer and respondent bias, non response, imputation error and errors in processing the data should explicitly be taken into account in the analysis of survey data and interpretation of survey results, in a comprehensive effort to assess error from each perspective. Conclusions should be carefully distinguished from the factual findings, and great care should be taken to be sure that the conclusions and the findings presented are consistent.
Carefully develop and fulfill pledges of confidentiality given to respondents.
Establish clear intentions and meticulous procedures to assure the privacy of respondents and the confidentiality of the information they provide. Unless the respondent explicitly requests otherwise,or waives confidentiality for specified uses, one should hold as privileged and confidential the identity of individual respondents and all information that might identify a respondent with his or her responses.
Exemplary survey research practice requires that one literally do "whatever is possible" to protect the privacy of research participants and to keep collected information they provide confidential or anonymous. One must establish clear intentions to protect the confidentiality of information collected from respondents, strive to ensure that these intentions realistically reflect one's ability to do so, and clearly state pledges of confidentiality and their realistic limitations to respondents. That is, one must ensure that the means are adequate to protect confidentiality to the extent pledged or intended, that procedures for processing and use of data conform to the pledges made, and that appropriate care is taken in dealing with directly identifying information (i.e., using such steps as destroying this type of information or removing it from the file when it is no longer needed for inquiry).
Interviewers and other research staff must be carefully trained and indoctrinated to uphold and maintain the confidentiality of respondents' identities and the information they provide and take/sign an explicit oath or pledge of confidentiality to do so before beginning work. In the verification of information, one must protect the identity of respondents from outside disclosure.
One should also assure that appropriate techniques are applied to control for potential statistical disclosure of respondent data. Individual respondents should never be identified or identifiable in reporting survey findings: all survey results should be presented in completely anonymous summaries, such as statistical tables and charts, and statistical tabulations presented by broad enough categories so that individual respondents cannot be singled out.
Excellence in survey practice requires that survey methods be fully disclosed and reported in sufficient detail to permit replication by another researcher and that all data (subject to appropriate safeguards to maintain privacy and confidentiality) be fully documented and made available for independent examination. Good professional practice imposes an obligation upon all survey and public opinion researchers to include, in any report of research results, or to make available when that report is released, certain minimal essential information about how the research was conducted to ensure that consumers of survey results have an adequate basis for judging the reliability and validity of the results reported. Exemplary practice in survey research goes beyond such standards for "minimal disclosure," promulgated by AAPOR and several other professional associations (e.g., CASRO and NCPP) by (a) describing how the research was done in sufficient detail that a skilled researcher could repeat the study, and (b) making data available for independent examination and analysis by other responsible parties (with appropriate safeguards for privacy concerns).
A comprehensive list of the elements proposed for disclosure by one or more sources which in combination, exceed the "standards for minimum disclosure" proposed by any one of the professional organizations includes: