AAPOR
The leading association
of public opinion and
survey research professionals
American Association for Public Opinion Research

Current Knowledge and Considerations Regarding Survey Refusals

 
AAPOR Task Force on Survey Refusals
September 8, 2014
 
 
 
 
Prepared for AAPOR Council by the Task Force on Survey Refusals operating under the auspices of the AAPOR Standards Committee, with Task Force members including:
 
David Dutwin, SSRS, Task Force Co-Chair
John D. Loft, RTI International, Task Force Co-Chair
Jill Darling, Veterans Health Administration
Allyson Holbrook, University of Illinois at Chicago
Timothy Johnson, University of Illinois at Chicago
Ronald E. Langley, University of Kentucky
Paul J. Lavrakas, Independent Consultant
Kristen Olson, University of Nebraska-Lincoln
Emilia Peytcheva, RTI International
Jeffery Stec, Charles River Associates
Timothy Triplett, The Urban Institute
Andrew Zukerberg, National Center for Education Statistics
 
Table of Contents
INTRODUCTION
EXECUTIVE SUMMARY
What Is a Refusal?
Who Refuses?
Refusal Aversion
Refusal Conversions
Rights of Respondents
Recommendations
WHAT IS A REFUSAL?
Definition of Refusals
Interim Refusal Disposition Codes.
Final Assignment of a Refusal Disposition Code.
Distinguishing Between Hard versus Soft Refusals
Distinguishing Between Informant Refusals and Respondent Refusals
Differences in Defining Refusals Based on Mode of Data Collection
Special Considerations in Defining Refusals when Using a Cell Phone Sampling Frame
Temporary Refusal Codes in Interviewer Administered Surveys
Known Eligible Respondent Soft Refusal (2.112a):
Known Eligible Respondent Hard Refusal (2.112b):
Other Person Soft Refusal (No Screening for Eligibility) (2.111a):
Other Person Hard Refusal (No Screening for Eligibility) (2.111b):
Breakoff, During Introduction (Before Screening for Eligibility is Completed) (2.121):
Breakoff, After Interview Started (2.122):
Calculating Refusal Rates
Refusal Report Form (RRF) and Contact Observations in Interviewer-Administered Surveys
WHO REFUSES?
Theories for Non-Participation
Leverage-Saliency Theory.
Economic Exchange Theory.
Social Exchange Theory.
Cognitive Dissonance Theory.
Survey Refusals, Then and Now
Demographic and Attitudinal Characteristics
Patterns of Attitudes/Behaviors Associated with Refusers
REFUSAL AVERSION
Definition and Purpose
Why Employ Refusal Aversion Methods?
To Reduce Bias.
To Improve Survey Face Validity and Response Rates.
To Reduce Costs.
To Reduce Effort.
Refusal Aversion Methods: Survey Design
Survey Introductions.
Survey Fallback Statements.
Advanced Notification.
Leaving Study Information.
Incentives.
Contact Rules:
REFUSAL CONVERSION
Definition and Purpose
Why Undertake Refusal Conversion Efforts?
To Reduce Bias.
To Reduce Cost and Effort.
Concerns of Data Quality from Converted Refusals
Missing Data.
Accuracy and Completeness of Response.
Satisficing.
Attenuation.
Length of the Interview.
Refusal Conversion Methods: Survey Design
All or Some?
Selected/Qualified Initial Refusals.  Depending on the study design, i
Tracking Outcomes.
Multiple Refusal Conversion Attempts.
Refusal Conversion Letters.
Mode.
Multi-phase and Responsive Designs.
Timing.
Incentives.
Contact Rules.
Refusal Conversion Methods: Interviewer Management
Interviewer Training:
Demographic Characteristics.
Attitudes.
Experience.
Interviewer Voice Parameters.
RIGHTS OF RESPONDENTS
Refusal Conversions in the Context of Human Subject Protections
U.S. Federal Regulatory Environment.
Research that is Exempt under U.S. Federal Regulations.
Variability in IRB Interpretation of U.S. Federal Regulations.
U.S. Federal Communications Commission (FCC) Rulings & State Laws.
Associational Positions on Refusals
IRB Considerations Related to Refusals
Should Certain Initial Refusals Be Considered “Final”?
Rights of Respondents with Respect to Internal Do Not Call Lists
RECOMMENDATIONS
Appendix
Refusal Rate 1 (REF1).
Refusal Rate 2 (REF2).
Refusal Rate 3 (REF3).
GLOSSARY

INTRODUCTION
With few exceptions, participation in a survey in the U.S. is a voluntary act.  Nonparticipation in surveys occurs for basically three reasons:  either the sampled individual or entity cannot be contacted and has no opportunity to decide about participation, or a contact is made but ends before the individual is provided enough information to make a decision about participation, or a contact is made and the individual decides to decline, or refuse, participation. The consistent increase in refusal rates over time and the acceleration of refusals in the past quarter century is a concern to the survey profession for several reasons. 

When refusal rates are high, survey researchers must be ever more concerned about the impact of refusals on potential systematic bias of the estimates generated by surveys.  Higher refusal rates also directly contribute to greater costs.  These costs include the monetary impact of refusals on the survey project itself and the increased labor associated with high refusals, and the potential costs to respondents, some of whom feel potentially that they are being pressured into participation by efforts to minimize refusals.  The more often sampled persons initially don’t participate, the more likely survey researchers concerned about response rates might attempt to persuade them of the merits or benefits of participation, in efforts known in the survey research profession as refusal conversion[1].   However, to the extent that an individual’s refusal is a request for privacy, refusal conversion efforts may infringe on that privacy. Therefore, even though refusal conversion is a common and accepted practice within the survey research profession, researchers must balance the validity of survey data with the protections of persons selected as sample members.

This document is intended as a resource for survey researchers interested in learning about the current state of knowledge about refusals in survey research, their impact, techniques to minimize them, and ethical considerations for the rights of respondents with regard to survey refusals.  It is also intended as a resource for members of Institutional Review Boards (IRBs) and of the general public who wish to learn more about these important issues.  

EXECUTIVE SUMMARY
The landscape of survey research has arguably changed more significantly in the past decade than at any other time in its relatively brief history.  In that short time, landline telephone ownership has dropped from some 98 percent of all households to less than 60 percent; cell phone interviewing went from a novelty to a mainstay; address-based designs quickly became an accepted method of sampling the general population; and, surveys via internet panels became ubiquitous in many sectors of social and market research, even as they continue to raise concerns given their lack of random selection.
Among these widespread changes, it is perhaps not surprising that the substantial increase in refusal rates has received comparatively little attention.  As we will detail, it was not uncommon for a study conducted twenty years ago to have encountered one refusal for every one or two completed interviews, while today experiencing three or more refusals for every one completed interview is commonplace.  This trend has led to several concerns that motivate this Task Force. As refusal rates have increased, refusal bias (as a component of nonresponse bias) is an increased threat to the validity of survey results. Of practical concern are the efficacy and cost implications of enhanced efforts to avert initial refusals and convert refusals that do occur. Finally, though no less significant, are the ethical concerns raised by the possibility that efforts to minimize refusals can be perceived as coercive or harassing potential respondents.  Indeed, perhaps the most important goal of this document is to foster greater consideration by the reader of the rights of respondents in survey research.

In 2010, the AAPOR Standards Committee responded to these concerns about survey refusals with the development of a special subcommittee on ‘refusal’ conversions. That subcommittee quickly realized that it was necessary to address the broad issue of survey refusal in addition to conversions alone.  At the subcommittee’s recommendation, the AAPOR Executive Council formed the Task Force on Survey Refusals to produce an official AAPOR document on survey refusals. 

Invitations to join the Task Force were extended to a number of AAPOR members, with the goal to include authors of important research on refusals, a mix of representatives from academia, for-profit and non-profit organizations and IRB members.  The committee met several times throughout 2012 and 2013, and began by reviewing and editing the draft outline that was used in the proposal to the Executive Council for establishing the Task Force.  The final report contains five main sections, with working subcommittees taking responsibility for each section:
  • What is a Refusal? (Tim Triplett, Chair)
  • Who Refuses? (John Loft, Chair)
  • Refusal Aversion (Jeff Stec, Chair)
  • Refusal Conversion (Emilia Peytcheva, Andy Zukerberg, Co-Chairs)
  • Rights of Respondents (Ron Langley, Chair)
Summaries of each section in the report are provided below.

What Is a Refusal?
The use of the term “refusal” is broadly used for instances where a potential respondent or someone on behalf of that sampled respondent does not participate in the survey. It includes instances where the person explicitly refuses to participate, or more ambiguous instances of non-participation where the person hangs up before they know what the call was about or before eligibility to participate can be determined, or where the person gives a temporary reason for not being able to participate at that time. As such, a refusal is at a conceptual level a psychological aversion— an aversion to participating in a survey at a given time —and may or may not be manifested by overt behavior explicitly indicating that one does not wish to participate.  From an operational perspective, in interviewer-administered surveys, researchers (and their staff; e.g., interviewers) often must infer whether respondents are refusing to participate based on ambiguous behavior and categorize such behaviors as ‘refusals’ or not. For example, potential respondents who do nothing other than hang up on a telephone interviewer might be considered callbacks, or might otherwise be considered refusals, for operational purposes. In other cases (e.g. mail or web surveys), researchers may need to decide whether a failure to respond constitutes a refusal since most potential respondents to such surveys do not explicitly refuse (e.g., by mailing or e-mailing a response indicating their refusal).  In addition, there are two operational types of refusals, interim and final, and the guidelines for categorizing a survey case as a refusal (either interim or final) varies by type of survey, and should be clearly set out in advance by researchers. 
For finalizing survey dispositions and for response rate calculations, in surveys without screening criteria (e.g., other than needing to reach a household with at least one adult), any initial refusals that have not later resulted in a completed interview or been found to be ineligible should remain classified as refusals, even if other outcome dispositions (e.g., no answer or busy signal in the case of a telephone survey) resulted from subsequent contact attempts.  However, in surveys with specific screening criteria, refusals that have not been screened to determine eligibility should not be treated as final refusals but as instances where eligibility status remains unknown (AAPOR Standard Definitions, 2012).

Researchers also often distinguish between “hard” and “soft” refusals, though the definitions of these terms vary by organization and within organizations.  Certainly, refusals where a potential respondent asks to be “taken off the list,” explicitly says “do not call me again,” or threatens legal action if efforts are made at re-contact should be considered hard refusals and not re-contacted.[2]  A refusal that does not meet these criteria, but where a potential respondent becomes perceptibly angry or upset by the request to be interviewed, are likely candidates to be considered hard refusals as well.  The definition of soft refusals also varies from organization to organization, but they are often considered to be refusals where respondents give “temporary” reasons for non-participation such as being too busy, being uncertain as to the usefulness of participating, or an expressed lack of interest about the survey topic. Some organizations also include hang-ups as soft refusals. How “hard” or “soft” a refusal is lies on a continuum, but individual organizations make determinations of how cases are categorized as “hard” or “soft.” As a practical matter, a hard refusal means that no further contact will be made whereas a soft refusal may be considered for some type of refusal conversion.

How one defines refusals varies by survey mode; and each mode has its own special considerations when defining refusals.  While Web surveys and mail surveys encounter refusals, the vast majority of non-respondents to these modes simply never respond to the request to be interviewed, so there is little information available to determine whether these cases represent true refusals.  In the end, it is not uncommon for Web and mail surveys to have dozens, even hundreds or thousands of cases in which no response is ever received back by the survey organization. Rarely do sampled households explicitly communicate with the study organization that they do not wish to participate.  In face-to-face and telephone surveys, however, potential respondents are put “on the spot” and must choose whether to participate or not.  In these situations, it is more apparent when a refusal to participate occurs. 

Much of this report is “telephone centric” because (1) in interviewer-administered survey (telephone and in-person) it is easier to identify refusals (in contrast to Web and mail surveys), (2) telephone surveys tend to experience higher levels of refusals than other interviewer-administered modes (e.g., in-person interviews), and (3) for these reasons much of the methodological work that has been reported about refusals is based on telephone surveys  However, where appropriate, we cover special considerations for refusals in each mode.

Who Refuses?
From the perspective of survey error, the principal concern regarding refusals is the possibility that sampled refusers are systematically different on the measures of interest in a particular survey than sampled participators across a range of demographic, attitudinal, and/or behavioral variables, so that their self-exclusion may bias results to a non-negligible extent.  From an operational perspective, the concern is to minimize the consequences of refusals on survey costs. From an ethics perspective, distinguishing “refusal” from non-contact or other forms of non-participation can be significant in considerations of possible coercive or harassing procedures. Social-psychological theories which may underpin the decision to participate in a particular survey (or even in any survey) include Leverage Salience Theory, Economic Exchange Theory, Social Exchange Theory, and Cognitive Dissonance Theory. 

We then detail the substantial increase in survey refusals that repeated cross-sectional studies have witnessed in the past quarter century.  Published research, as well as original research conducted by this Task Force, finds that refusals have at least doubled, and in some cases refusals on current studies have seen a threefold or more increase compared to similar studies conducted a decade or more ago.  We also consider the issue of the proclivity to refuse on cell phones compared to landlines.

A substantial focus of the “Who Refuses” section is the potential for systematic differences by demographics, attitudinal, and behavioral measures.  Research on these topics faces a substantial hurdle--if a potential respondent has refused to be interviewed, depending on sample frame and/or study design, demographics or other attributes of that individual may not be readily available; (at least in the absence of nonresponse follow-up studies that have begun to be conducted in the past decade).  Researchers have addressed this obstacle to the extent possible by comparing “easy” interviews with interviews obtained only after a significant number of call attempts or by way of a refusal conversion.  However, these strategies cannot reveal the attributes of those who refused and never participated. 

As to the question of who refuses, we have two general observations.  First, there is no shortage of differences found between those who participate at first contact and those who participate only after repeated contacts.  Second, while there is some evidence that refusers are more likely to be non-white and score lower on measures of civic engagement compared to participators, there is little consistency in other differences from study to study. Although other findings were not consistent between studies, the fact that every study found multiple significant differences between those who easily participate to those who participate with some initial resistance is reason enough for researchers to be concerned with systematic/differential nonresponse due to refusals and to continue to investigate this topic.
 
Refusal Aversion
The section on refusal aversion begins with a review of why researchers might enact procedures to avert refusals.  As detailed in the section on Who Refuses, there is widespread, if inconsistent, evidence that sampled refusers are systematically different than sampled participators on any number of demographic, attitudinal and behavioral variables.  As such, methods to avoid refusals are done principally to try to reduce the potential for refusal-related bias.  Even without strong evidence of systematic bias, researchers may be concerned with avoiding refusals for considerations of face validity, to meet sponsor’s requirements, or to try to reduce effort and costs.  While this may seem contradictory, given that most refusal avoidance strategies come with their own costs and effort, it is possible that the cost and labor saved in converting initial refusals to completions is greater than the cost and labor on executing refusal avoidance strategies.  In addition, materials and procedures that introduce the survey and explain its features are often of particular interest to research ethics committees (IRBs) and other entities charged with evaluating whether research protocols are respectful of participants’ rights.
Refusal aversion strategies are broken down into two main classifications: survey design strategies and interviewer management strategies. 

Survey Design. With regard to survey design, we detail seven types of common strategies: survey introductions, persuasive materials, use of multiple contact modes, advance notification, leaving messages about the study, incentives, and rules of contact. 

First are survey introductions that are designed to minimize refusals as much as possible.  Introductions can occur on the doorstep in in-person surveys, on the telephone in telephone surveys, in cover letters in mail and internet surveys, or via an advance letter or e-mail.  Two key issues with regard to introductions are whether standardized introductions foster fewer refusals than customized introductions and whether an introduction will exacerbate differential refusals, for example, among those not interested in the topic mentioned in the introduction.  Also noted and detailed is the importance of an interviewer maintaining the interaction with the sampled respondent, for the longer an interviewer can engage a potential respondent, the more likely they will decide to participate.  Similarly, introduction letters or e-mails that capture and hold potential respondents’ attention may be more likely to gain participation. In this regard, we review a second technique to avoid refusals: the development and use of survey fallback statements, that is, persuasive, declarative statements or materials that are authored prior to the survey entering the field to address frequently asked questions (FAQs) posed by respondents.  A third strategy is the use of multiple modes to reach out to potential respondents.  Existing research is inconclusive, but suggests that overcoming resistance may require different modes in contacting reluctant or outright-refusing participants. Fourth, we review the impact of advance notification and fifth, the impact of leaving study information on the doorstep or on voice mails.  A sixth area has to do with incentives, and specifically with whether to offer them at all, whether to offer them on a contingent or non-contingent basis, to whom to offer them, and what amount to offer.  Finally, we review decisions one can make regarding rules of contact and repeated contact to each household or business.

Interviewer Management. Another major category of refusal aversion efforts has to do with characteristics and behaviors of interviewers themselves.  Research has explored the impact of interviewers’ experience, personality, attitudes and expectations, and voice parameters.  This section also explores interviewer training and testing, both general and specific.  The literature is diverse, but there is clear evidence that experience matters, while effects of personality are more mixed, with no difference across some personality attributes and differences across some others.  There is some evidence that interviewers who are open to and effective in utilizing persuasive techniques attain higher response rates than those who are not, though the research in this area has largely been specific to in-person surveys.  As to whether specific refusal aversion training is effective, the results are mixed but generally positive.

Refusal Conversions  
Of course, researchers are interested in refusal conversions for the same reasons they try to avert refusals in the first place: to minimize the extent of nonresponse and refusal-related nonresponse bias, to maximize survey face validity, and to potentially reduce cost and effort.  Aversion and conversion represent points on a continuum of overcoming hesitance to participate and the distinction may involve subjective judgment. A concern with refusal conversions is that the reluctant participant may provide lower quality data compared with data from initially willing participants. This observation leads to concerns of item nonresponse, lower accuracy, satisficing, attenuation, and less elaborate responses to open ended survey items. This is a trade-off of error due to refusal-related nonresponse versus a variety of response errors that may be introduced by reluctant or uncommitted respondents.

Researchers who conduct refusal conversions face a different set of survey design and interviewer management decisions compared to the concerns of refusal aversion.  For one, researchers must decide whether to try to convert all or only some of the initial refusals.  If only a subset of conversions is attempted (e.g., due to limited budget), it may be necessary to target select subpopulations for conversion.  Researchers also have to keep in mind that it is possible that a significant percent of refusals will be encountered from members of the sampling unit (typically a household) who are not necessarily the person the researcher wishes to interview, because the refusal occurred before within-household selection or specific screening criteria were carried out.
Researchers also must consider tracking the outcomes of the conversion attempts, perhaps even deciding to maintain a set of interim and final dispositions for the refusal conversion attempts.  Finally for study design considerations we discuss the practice of multiple refusal conversions, the practice of attempting to convert households that have already refused not just initially but during conversion attempts as well.  As noted in the report, we do not recommend such strategies as part of best practices in survey research.  

In addition, there are many study design considerations similar to those reviewed for refusal aversions.  These include the use of refusal conversion letters, differences by mode, timing and contact rules, and the use of refusal conversion incentives.  There are also similar decision points with regard to interviewer management, such as specific refusal conversion training, and the impact of demographics, vocal characteristics, experiences and attitudes on the conversion rates of interviewers. 

Rights of Respondents
The final main section of the report discusses the rights of respondents.  Our intention is that previous sections provide context for this discussion. This section begins with the observation that considerations of respondent rights in survey research have historically centered on confidentiality and privacy and that more recently considerations have expanded to include respondent rights with regard to refusing to participate and potential harassment of those who have already refused to participate.  This final section also discusses considerations with regard to IRB review protocols related to recruiting survey participants and procedures concerning refusal aversion and conversion.  Two recommendations are that we have well-defined rules for what constitutes a ‘hard’ refusal (cases which will not be eligible for re-contact), and that we will try to work with IRBs to develop a mutual understanding of the balance between legitimate survey methodology and respondent rights.

Recommendations
This report provides many considerations, most of which have been briefly touched upon in this executive summary.  The Task Force developed several recommendations, included in this last section of the report and classified as operational recommendations and suggestions for further research. 
Operational recommendations are intended to encourage tracking and reporting of refusals as distinct from other types of nonresponse. The distinction is important for several reasons. Ethically and legally, researchers have an obligation to respect the rights of individuals sampled for surveys. Reasonable efforts can and should be made to assure that sampled persons have the information necessary to make an informed choice to participate or not. However, these efforts must be balanced with protections from the potential harassment of repeated contact attempts. Noting refusal as an event and developing indications of whether they are true refusals as opposed to other more ambiguous forms of non-participation is important information for evaluating this tradeoff. A second reason for the distinction is to support effective allocation of data collection resources. Identifying initial refusals from other forms of nonresponse can lead to more focused and more effective follow-up efforts. Finally, reporting of refusals in survey documentation can form the basis of further inquiry about the effect of refusals on the quality and costs of surveys.

Two themes underlie suggestions for further research. First, several recommendations concern the efficacy of refusal aversion and conversion techniques and are intended to continue to build an empirical basis for best practices in survey design. Second, in the final recommendation, we note the possibility that inclusion of data from initially reluctant respondents may, in some instances, increase total survey error and suggest that it is important to continue to research systematic differences in data collected from cooperative and from reluctant respondents.
  

WHAT IS A REFUSAL?
Like any social behavior, the measurement of the act of refusing to participate in a survey requires an operational definition.  In the process of creating that definition, some ambiguities, gray areas, complexities, and potential exceptions to rules often arise.  As social researchers we typically explore phenomenon for which the understanding is largely hidden in the minds of others, making operational definitions difficult and imperfect.  Indeed, one potential respondent may be averse to the notion of  an interview, and will forcefully and angrily refuse, perhaps even threaten legal action from the intrusion, while a different prospective respondent, similarly objectionable to the idea of partaking in an interview, may only politely state that it is not a good time for him/her to participate.  Are both, operationally defined, refusals?  If so, what is the commonality that makes them so? Another example is a case where a survey is designed to interview female adults, and a male answers the phone, only to hang up on the interviewer seconds after he utters “hello.”  Given that a female did not expressly refuse to participate in the interview, and given that the interviewer does not even know whether a female adult resides in the household, do we call this case a refusal? A third example is a mail or internet survey where a specific person is sent the questionnaire or a link to the questionnaire and either loses the questionnaire or the link.  Do we count this case as a “refusal”?    Again, in many cases initial ambiguous acts of non-participation are not ‘refusals’ per se, but the industry tends to call them refusals for lack of a more clarified term.

We begin the discussion of “what is a refusal” with the assertion that both the conceptual definition of a refusal as well as the operationalization or measurement of refusals may require inference from what is observable.  It is also important to note that in most cases there are two ways that survey refusals are operationalized: the interim, in-the-field disposition of a given contact attempt with a case as a refusal (or not), and the final, end-of-field accounting of a case as a refusal (or not) in order to publish final dispositions and calculate a survey’s response rates.  These distinctions are important because there are instances where an individual may have refused to participate, but the case should not be categorized as a refusal for purposes of calculating the response rate (e.g., if the person later is found to be ineligible). In addition, there also are instances where a person might not explicitly refuse to be interviewed, but nevertheless might be counted as a final refusal (e.g., slams the door without saying a word to the interviewer).  In the end, these distinctions are important to note because survey researchers not only have to decide how to count that particular case (as a refusal or not), but also decide what do to next (e.g., whether to re-contact to try a conversion, and how and when should that conversion take place; see also, Bauman, 2008).

Definition of Refusals
The most general definition of a household refusal in a survey, from the AAPOR Standard Definitions is:  A case “… in which some contact has been made with a household and a responsible household member [or employee at a business] has declined to do the interview…” (AAPOR Standard Definitions, 7th edition, 2011, p. 14).  This may or may not occur before the eligibility of the household has been determined and temporary and final field outcome disposition codes ideally should retain this distinction. No matter how such an event is coded, as a temporary and/or as a final disposition, it remains the case that a “consenting adult” within a selected household or business has refused to participate.  However, this definition does not necessarily imply that a data collection organization should or will designate any such case as a refusal for purposes of either the remaining field period or as the final disposition of the sample case in question at the end of the field period.  Each of these requires some consideration.

Interim Refusal Disposition Codes.  For a variety of reasons, we find that survey research organizations sometimes do not choose to assign a refusal-related disposition code to a case in which a refusal has occurred, at least not on a temporary basis.  The decision whether a given act of refusing to participate is considered an interim survey refusal should be based on 1) the rationale for not participating and 2) whether the act of refusing came from a valid (that is, qualified and selected to conduct the survey) survey respondent.

For example, in telephone surveys, a potential respondent may answer “hello” but then silently hang up on the interviewer during the interviewer’s initial introduction.[3]  In such cases, the potential respondent did not explicitly and vocally express his/her desire not to participate.  Moreover, in surveys that require respondent selection, it is not clear whether the person who hung up was a qualified respondent.  Regardless which of these might be correct, we find that survey research organizations often choose to categorize such cases as an interim callback rather than a survey refusal. For example, it is sometimes the case that conversions are handled by interviewers specifically trained in conversion strategies.  By coding all initial hang-ups and the many other implicit refusals to participate as refusals, survey research organizations could accrue many more refusals than they could potentially try to convert with a small set of specially-trained refusal converters.  As such, it is not uncommon for survey research organizations to consider such cases as unscheduled callbacks rather than refusals.  These cases can then be processed by the “regular” pool of interviewers, and may receive many more callback attempts than would be the case if such cases were specifically coded as refusals.  In such cases unique dispositions codes should be created for each of the different types of refusals, which then would allow the survey organization to more effectively allocate different refusal types to different follow-up strategies based in part upon on the resources available to a particular survey project.

Final Assignment of a Refusal Disposition Code.  Consistent with the spirit of AAPOR’s Standard Definitions the final disposition given to a case that ever was a refusal, should be a refusal unless (1) it later became a completion, (2) was found to be ineligible or (3) there was some unusual circumstance that justifies assigning a different final disposition to the case than a refusal.[4]    Thus, in surveys that require eligibility screening, the screening stage of the survey or the main questionnaire portion of the survey can be the point in the overall survey where a refusal can occur. With regard to the screening stage of the survey, AAPOR Standard Definitions state: “Cases for which there is a household and it is not known if there is an eligible respondent usually crop up because of a failure to complete a needed screener. Even if this failure clearly were the result of a refusal, it would not be so classified unless the existence of an eligible respondent were known or could be inferred” (AAPOR Standard Definitions, 7th edition, 2011, p. 16).  Although this circumstance would not count as a refusal for the main questionnaire portion of the survey, it would count as a refusal for the screening portion of the survey.  That is because the response rate for the screening portion of the survey is often estimated separately from the response rate for the main questionnaire portion (AAPOR Standard Definitions, 2011).  Since the screening stage of the survey assumes that all households are eligible to be screened, any refusal encountered during screening should be assigned a disposition code as a refusal for the screening stage response rate calculations. [5] 

Distinguishing Between Hard versus Soft Refusals
A very important dimension on which refusals differ is whether they are “hard” or “soft” refusals (Deming, 1953).  For those categorized as “hard” refusals in which the refuser is known or reasonably assumed to be an adult of the household (or business) who states explicitly “never contact me/us again” this report recommends that the relevant contact information is placed on an internal Do Not Call list (see the final section of this report on Rights of Respondent for more information about Do Not Call lists) and that conversions are never attempted.   

In general, soft refusals can be refusals where respondents give reasons for not participating that may change (e.g., not having time or being too busy at that moment).  Refusals that are made by an individual other than the selected respondent may also be classified as soft refusals.  It is our understanding that some organizations also count immediate hang-ups in telephone interviews (in which the respondent doesn’t provide any reason for not participating) as soft refusals because these potential respondents may misidentify the contact attempt as a telemarketer attempting to sell something, and in any event have not heard enough information about the survey to decide whether to participate.

In the end, it is up to the survey organization to decide for each study where to draw the line between hard refusals and soft refusals. Some survey research organizations have interviewers classify refusals as hard or soft or rate them on a continuum that ranges from hard to soft (cf. Lavrakas, 1993).  Such ratings may be made on a “refusal report form” on which interviewers are asked to provide information about a refusal (see additional discussion on refusal report forms below). Refusal report forms may also ask interviewers to indicate the reason for not participating, if one is given, and to provide additional information that might be useful in later persuading the potential respondent of the merits of participation (e.g., respondent characteristics and behaviors). 

A major goal of such ratings is to identify cases to consider for conversion.   In telephone surveys, conversions have been successful in 10% to 30% of conversion attempts (see, e.g., Cominole, et al. 2008; Triplett, 2002; Retzer, Schipani and Cho, 2005; Lavrakas, 2010). The rationale for conversion attempts is that a person who initially offers resistance might have a change of mind if provided an opportunity to consider information about the sponsorship or purpose of the survey or the nature of the requested survey tasks, or offered a more convenient time to complete the interview. There is some evidence that interviewers can reliably distinguish between hard and soft refusals and that refusal conversion attempts are more successful with the latter than the former (Stoop, Billiet, Koch, and Fitzgerald, 2010).  Similarly, refusals where the respondent gave reasons that could change over time or could be addressed better by a more skilled re-contacting interviewer (e.g., being too busy, bad time, or wants more information) have been found to result in greater conversion success than refusals where the respondent gave reasons that were unlikely to change over time and more difficult for interviewers to address (e.g., no interest, don’t do surveys, and dislike subject; Dutwin and Herrmann, 2006).  Dutwin and Herrmann (2006) found that respondents rated by the interviewer as less angry were not more likely to be converted.  This suggests that classifying hard refusals more broadly to include cases where a potential respondent has displayed anger, but has not explicitly said not to contact again, may be inappropriate as these cases may not be any more difficult to convert than soft refusals.  This may in part be due to the fact that the person responding to the conversion attempt may not be the same person who responded to the initial interview request. Indeed, conversion attempts have been found to be more successful when the initial refusal came from someone other than the person reached during the conversion attempts, particularly for “hard” refusals (Stoop, Billiet, Koch, and Fitzgerald, 2010).   

Distinguishing Between Informant Refusals and Respondent Refusals
Refusals that occur before respondent selection are similar to those that occur during the screener portion of the survey.  The AAPOR Standard Definitions state that one should count such instances as a refusal for purposes of calculating the response rate, but again there is the issue of whether a refusal from the “right” household member has occurred.  This is not an inconsequential issue given that three fourths of all households in the U.S. have more than one adult (ACS, 2010), and as many as two-thirds of refusals are ones where the potential respondent does nothing other than silently hang up (Dutwin and Herrmann, 2006).  In other words, for about half of all refusals, interviewers will not know whether the refusal resulted from a respondent who would ultimately be selected to complete the survey if the respondent-selection process had been completed.  This in turn may have implications as to how conversion attempts will be enacted. In these cases, other than previously noted where screening is required, this form of nonparticipation will count as survey refusals for the final disposition of the case, regardless of whether the initial refusal is made by the selected respondent.[6] 

An advantage in dealing with these circumstances is to have separate refusal disposition codes for interviewers to use to distinguish between (1) a person known to be the designated respondent for the survey, (2) a person known not to be the designated respondent for the survey, and (3) instances where the person who refuses is not known to be either the designated respondent or someone else.

Differences in Defining Refusals Based on Mode of Data Collection
What constitutes a refusal may vary depending on the mode of data collection. Some of these differences result directly from more information being available about a contact attempt when the survey is administered by an interviewer (for most telephone and in-person interviews) than when the survey is self-administered (mail and internet surveys).  Since most of the discussion above about what constitutes a refusal is within the context of telephone interviewing, this section will discuss how defining refusals in other survey modes differs from telephone interviewing.

In-person surveys. For in-person household surveys, the interviewer can usually tell if a sampled building is a housing unit.  Thus, in a household survey with no additional screening for eligibility, if an adult answers the door and refuses, then this should be coded as a refusal, even if the person does not hear what the survey is about.  However, one potential difficulty in defining a refusal for in-person household surveys is the distinction between a request for a re-contact and a refusal.  This difficulty is one that is also problematic for telephone surveys, but may be particularly difficult for in-person interactions where the respondent may be more likely to maintain norms of politeness.  In these cases, the respondent may implicitly be refusing by politely putting off the interviewer in a way that appears that they are asking to be re-contacted (e.g., the respondent says s/he cannot participate in the survey right now or is too busy and may even suggest a convenient later time to be contacted).  Similar to examples noted earlier, these cases should be given a temporary disposition code that indicates the respondent requested to be contacted later and not a refusal code.  On the other hand, if the interviewer suspects this is really a refusal, s/he should make a notation of that along with providing justification for this supposition in the contact history for the case.

Mail surveys. For mailed surveys, it is difficult to make generalizations about explicit refusals that are conveyed to the researchers because there usually is a very small proportion of sampled respondents who send back the questionnaire indicating[7] that they are not interested.  Some people will also try to contact researchers directly (especially when a telephone number or an email address is provided) to inform the researchers that they are not willing to participate.  In other cases, some respondents will mail back a non-contingent incentive sent to them with an explanation that they are not willing to participate. But in most surveys these examples occur very infrequently. As such, there likely will be too few of these clear refusals to justify the development and deployment of any special targeted conversion procedures other than follow-up mailings.  

In contrast, the vast majority of “refusals” in mail surveys in which there is no telephone or in-person follow-up are associated with sampled addresses from which the researchers never receive any response, including any acknowledgement of ever receiving the survey.  Of course some of these are not refusals, and thus AAPOR’s Standard Definitions encourages researchers to make reasonable estimates (in calculating and applying e as part of their mail survey response rates calculations) of the proportion of these address – from which they never hear back anything – that are in fact bad addresses or ineligible.   But importantly, AAPOR’s Standard Definitions do not count these types of non-respondents in mail surveys of specifically-named persons as refusals, but rather as cases of unknown eligibility, for reasons of an unreturned questionnaire (Standard Definitions, 7th edition, 2011, p. 28). 

Internet (web) surveys. For most internet surveys, and similar to most mail surveys, the number of explicit refusals is extremely low.  Clearly, in all cases where a potential respondent replies to a survey invitation saying they do not wish to participate, such cases should be classified as refusals.  Of course, in surveys that strive to gain cooperation via the internet, many cases will “implicitly” refuse by simply choosing not to respond to the survey request.  This includes respondents who visit the survey on the internet, but then fail to complete any questions.

However, as with mail surveys, AAPOR’s current Standard Definitions make clear that such cases should be coded as having unknown eligibility, and thus not coded as refusals, since typically there is no way to determine whether a particular invitation was ever received.  Some of these unknown case outcomes will be noncontacts.  Others will be received but never opened because of the substance of the subject line.  However, depending on the quality of the email addresses used for sampling, one may assume that a large portion are implicit refusals, although, for purposes of calculating a response rate, they are treated as cases of unknown eligibility.  Again, it is the responsibility of the researchers to provide a credible basis for which to calculate “e” and then apply it to the portion of their final dispositions of unknown eligibility that are likely to be refusals (or noncontacts) and thus include these as part of the survey’s nonresponse outcomes in response rate calculations.

When the researchers have information that a potential respondent visited the survey website but failed to start any part of the screener or main questionnaire, then contact with such cases clearly has been made.  Whether these cases should be treated as refusals will depend on whether there is any question about their eligibility.  If there is no question of eligibility, then these cases are refusals.  Researchers would be well-advised to use a separate disposition code to identify those respondents who visited the survey website (if they are known to the researchers) but failed to begin the questionnaire.  In that way, researchers will be able to differentiate such “refusing” cases from the other cases for which nothing is known other than they did not respond in any way. 

A second type of internet survey covered in the Standard Definitions is probability-based internet panels.  The primary difference between internet studies of specifically-named persons and probability-based internet panels is that in most cases the latter is at least a two-stage design, starting with recruitment to the panel and then an invitation to panel members requesting participation in a specific study.  Recruitment to these internet panels can be executed using telephone, mail, and/or in-person methods, and thus refusals that occur in the recruitment stage would be defined by the guidelines specific to the methodology executed for the recruitment stage.  Once recruitment is complete, it is often the case that recruited respondents must complete an initial profile survey.  Members of the final panel will include only those who completed the profile survey. Those who do not complete the profile survey can be considered refusals to the profile stage. After that, response rates for participation in a particular survey among panel members are computed as a function of the initial response rate in recruiting the panel and the response rate for a particular survey.

In contrast, non-probability internet panels utilize recruitment methods that include open invitations, widely disseminated via various means.  Therefore, it is impossible for anyone (including the researchers) to know exactly who has been “invited” to be part of the panel, and thus there is no way to know who might have refused to join the panel, other than to know that “refusers” vastly outnumber those who do behave in some way to show an interest in joining the panel.  As such, the Standard Definitions recommends the computation of a “participation rate” rather than a response rate for non-probability-based panels.  It is typical of non-probability panels that once respondents are part of the panel, they are invited to particular surveys.  At this stage the definitions of who refuses in non-probability panels (and in probability internet panels) are the same as described above for internet surveys of specifically-named persons. 

Special Considerations in Defining Refusals when Using a Cell Phone Sampling Frame
Using a cell phone frame for sampling in the U.S. provides additional considerations when determining what constitutes a refusal.  One could argue that if an interviewer reaches a respondent on a cell phone or gets their voice mail, then by definition you have reached the working number of an eligible respondent because cell phones are largely personal devices.  However there are numerous studies that show that a non-trivial percentage of cell phones are used exclusively by minors (AAPOR Cell Phone Task Force, 2010, pp. 34-35) and thus one cannot assume with confidence that an eligible respondent has been reached, especially when an immediate refusal is encountered. As to voice mail, most cell phone users have voice mail, but the variety of operator messages an interviewer may hear (which differ by provider) are often ambiguous as to whether an eligible person/household has been reached or even whether the number is in service (AAPOR Standard Definitions, 7th edition, 2011, p. 16; AAPOR Cell Phone Task Force, 2010, p. 42). Finally, even if one reaches a person on a cell phone who hears the survey introduction and confirms they are an adult, if they hang-up before they verify they reside within the target geography for the survey, one does not know if they are eligible.  (Of note, this is not often a problem with landline surveys unless small geographies are being targeted, although as more landline numbers are ported this will likely increase as a problem).

An additional issue with cell phone interviewing is deciding how to code ‘dropped calls.’ This occurrence does not necessarily correspond to landline situations such as hang-ups before hearing the survey introduction or before respondent selection, or even break-offs during the interview.  In those cases, the respondent has actively decided not to cooperate at all or any longer.  In contrast, during a cell phone interview a “break-off” may occur as the result of a dropped call or other technical problems and may have nothing to do with the respondent actively deciding to break off from the interview (AAPOR Cell Phone Task Force, 2010, p. 50). Recent research suggests that rather than immediately treating these cases as a form of refusal, the interviewer should call back immediately to see if it was just a dropped call and ideally complete an interview at that time (Mark & ZuWallack, 2012).

Temporary Refusal Codes in Interviewer Administered Surveys
In the section “Temporary vs. Final Disposition Codes,” the AAPOR Standards Definitions suggest a number of temporary disposition codes for refusals.  These are not given any detailed explanation, but since there are a number of temporary codes listed having to do with various types of interim refusals, we offer further explication here:

Known Eligible Respondent Soft Refusal (2.112a):  As described earlier in this report, it is largely up to the researcher to define the difference between hard and soft refusals.  Generally, a respondent soft refusal occurs when the designated respondent, who is known to be eligible for a particular survey, indicates that s/he is unwilling to begin the questionnaire, but does so in a manner that suggests that s/he may not be adamant about this were contact to be made again at another time (e.g., “I’m sorry. I’m feeding the baby right now”).  If the survey’s protocol includes efforts to convert refusals, these types of refusals are prime candidates for re-contacting. 

Known Eligible Respondent Hard Refusal (2.112b):  In a hard refusal, the designated respondent indicates that s/he is adamantly unwilling to begin the questionnaire.  Cases where respondents who are known to be eligible for the particular survey have explicitly asked not to be contacted again should be placed on the study’s Do Not Call list (see section on Rights of the Respondent), and placed into a final refusal response rate code.  This code is for cases that researchers consider hard refusals but not DNC refusals. These cases should not be re-contacted.

Other Person Soft Refusal (No Screening for Eligibility) (2.111a):  In this case, contact (e.g. the person answering the telephone, or replying by mail) is made with an informant who may or may not be the designated respondent. The informant indicates that her/his household (or establishment) is unwilling to cooperate, but suggests that the decision to participate may be different if contact was made at another time.  If a survey protocol includes conversion attempts, these types of refusals are good candidates for re-contacting, since the unit-level refuser may not be the designated respondent and/or may not be the one who is contacted (e.g., answers the call or opens the mail) the next time.

Other Person Hard Refusal (No Screening for Eligibility) (2.111b): In this case, contact (e.g. the person answering the telephone, or replying by mail) is made with an informant who indicates adamantly that her/his household (or establishment) is unwilling to cooperate.   (This does not include hard refusals that should be placed onto an internal Do Not Call list.)  Since it may be uncertain if the person who has refused is in fact the designated respondent, some researchers will try to re-contact these cases on the chance that someone else will be selected as the respondent.

Breakoff, During Introduction (Before Screening for Eligibility is Completed) (2.121):  In an interviewer-administered survey in which a household or person is screened for eligibility; the potential respondent may break-off contact with the interviewer before the screening sequence is complete.  Even though this is a “refusal” for the screening part of the survey, it is unclear whether the household/person is eligible.  Many researchers re-contact these cases to attempt to determine eligibility.  If it is found that the case is ineligible, it will not count in any refusal-related response rate calculations or in any nonresponse bias investigations.

If the re-contact is attempted with the person/household and the case is never screened for eligibility, the final disposition of this case is a refusal in the “screening” stage, and would not be a part of the “interview” stage.

Breakoff, After Interview Started (2.122):  In an interviewer-administered survey, the respondent may start the questionnaire but then refuse to complete it.  In an internet survey, the respondent may only partially complete the questionnaire before breaking off.  Rarely, in a mail survey, a respondent may return a partially completed questionnaire.  For each of these instances, this break-off outcome would become a final refusal disposition even if additional re-contact attempts were made and they did not lead to a completed interview.  The exception to this is when enough data were gathered before the point of the breakoff to consider the case a partial interview, which the AAPOR Standard Definitions recommend defining before the field period has begun.

For all of the refusals described above (soft or hard, respondent or other person, breakoff or not), if a refusal case is never converted to a completion, regardless of whether an unsuccessful conversion attempt was made, the final disposition of the case is a refusal.  This holds regardless of whether subsequent contact is made with the respondent/household during the field period, unless that subsequent contact leads to a determination that the person/household/business is ineligible for the survey. (Standard Definitions, 2011, p. 10)

Calculating Refusal Rates
AAPOR recommends that a survey’s refusal rate be calculated using guidelines in the Standard Definitions.  These are provided in the appendix.

Refusal Report Form (RRF) and Contact Observations in Interviewer-Administered Surveys
The problem of gathering information about non-participants is especially challenging in survey research, and is essentially impossible with most mail and internet surveys.[8]  However, with interviewer-administered survey modes, the interviewer becomes a potential source for this information.   Thus, in in-person and telephone surveys the best proxies are interviewer estimates that are gathered on structured observation forms.

A refusal report form (RRF), also called an interview observation, contact observation, or contact history form, is a structured form used by interviewers immediately after a refusal is encountered (see Bauman, 2008; Lavrakas, Bauman, and Merkle, 1992; Lavrakas, 1993; NAEP, 2003).  The use of a RRF during a survey produces valuable paradata and has three main benefits:
  • It can provide estimates of relevant parameters (e.g., a refuser’s gender, race, age, etc.) that can be compared with the same characteristics in the final sample of responders to help determine the possible presence and impact of potential nonresponse bias.
  • It can provide valuable information that can help interviewers in subsequent contact attempts to convert an initial non-responder to a completed interview.
  • It can help researchers conduct investigations into the nature of refusals so as to plan better strategies on how to reduce their frequency of occurrence.
RRFs capture structured information about all individual refusals.  Such information most often is lost when interviewers are given only informal instructions to write down notes about the refusals if they think it is appropriate.  RRFs can be used either in person or on the telephone. Information about the refusal is recorded based on judgments that interviewers make either visually (for in-person) or audibly (for telephone and in-person).

Figure 1 shows an example of an RRF used with RDD surveying.[9]  However, there is no standardized format for an RRF.  Researchers develop RRFs with variables that are most relevant to the particular survey being conducted and that can be expected to be reasonably accurate as recorded by the interviewers in that survey.  For example, in an in-person survey, an interviewer may be able to provide information about the type of home, presence of children’s toys outside the home, orderliness of the yard, neighborhood setting, and so forth. Telephone interviewers could not collect that information from a brief phone conversation, but research indicates that interviewers can provide accurate information for certain demographic characteristics of the people contacted to participate in the telephone survey (cf. Lavrakas, Bauman, and Merkle, 1992; Lavrakas, 1993).

A RRF typically tries to capture two types of information:  Demographic information about the refuser (e.g., gender, age, race, ethnicity, educational attainment); and details about the nature of the refusal (e.g., strength of the refusal, reasons given for refusal, perceived barriers to participating, etc.). This information may help interviewers, in future contact attempts, to convert that refuser into a cooperating respondent.

Given that the vast majority of refusals in telephone surveys typically occur in the first few seconds of the interviewer–respondent interaction, the interviewer has very little time to develop rapport, anticipate potential barriers, and alleviate respondent objections.  Bauman, Merkle, and Lavrakas (1992) concluded that attentive telephone interviewers are able to accurately make estimates of gender, race, and age in a majority of cases, even when the interactions are very brief – at least in enough cases to make these data useful.  Other research, reviewed in Olson (2013), has shown variability across interviewers in their ability to estimate demographic and other characteristics in both face to face and telephone surveys.

Interviewers are also able to provide details about the refusals to help survey administrators determine which cases to contact again and how best to approach the next interactions.  For example, if a conversion interviewer is re-contacting an initial non-participating household where the reason was “we’re just sitting down to dinner,” s/he may start the next interaction with “I’m sorry we previously reached you at a bad time.  Is this time better?”  Or, if the RRF indicates that a woman refused but during the conversion attempt a man answers, the interviewer can adjust the introduction accordingly.  This information, when systematically collected for both respondents and non-respondents, can also be used for nonresponse adjustment weights (Olson 2013).

For in-person surveys, more information about the refuser is at the interviewer’s disposal because he or she actually sees the person who refused.  Thus the data about the refuser’s characteristics in an in-person survey are likely to be more reliable than the data that can be gathered about these characteristics in a telephone survey. 
 
Refusal.jpg
Figure 1. Example of a Refusal Report Form (RRF) for an RRD survey.
Source. Lavrakas (1993, p. 80)
 
 

 
WHO REFUSES?
In this section, we summarize theories related to survey non-participation, provide a brief history of refusals as a component of nonresponse, and describe some characteristics of individuals who refuse.

Given the considerable complexities noted above in defining “what is a refusal,” it is it not surprising that the historical and empirical literature does not consistently separate factors and behaviors related to refusals from other types of nonresponse. Nonetheless, we believe it is important to make the distinction. The ethical and scientific issues around understanding refusals can be distinguished from those around other types of nonresponse and it is important to consider the unique aspects of refusals in survey design. 

As is the case for all factors related to nonresponse, a principal concern regarding refusals is whether there are systematic differences with respect to survey variables of interest between sampled individuals who don’t participate in a survey and those who do. Systematic differences in these characteristics between participants and non-participants are a threat to the validity and representativeness of survey data and can contribute to survey error. To minimize this threat, researchers focus efforts for averting and/or converting refusals among sub-populations who may be more prone to not participate in survey research.  Yet as noted below, while there is some evidence of systematic bias from refusals, the findings are inconsistent from the study to study. 

Theories for Non-Participation
A number of hypotheses have been offered for why some people willingly participate in certain (or all) surveys and others do not.  These hypotheses are relevant to why a given person will participate in a survey sometimes, but not other times, and they stem from a number of classic sociological and psychological theories.
Leverage-Saliency Theory.  Groves, Singer, and Corning (2000) proposed leverage-saliency theory to explain why people participate or refuse to participate in a survey.  The theory recognizes that, in a given survey and at the particular time of the request to participate, there are a variety of positive factors that influence a sampled individual’s decision to participate in a survey as well as myriad other factors that can negatively influence the decision.  Each factor has leverage and has a different weight in the decision.   Additionally, each factor may be made differentially salient to the individual, especially in interviewer-administered surveys where the interviewer decides what to tell the respondent about the survey. If the collective weight of the salient reasons to participate outweighs the collective weight of the salient reasons to refuse, then the person will agree to participate at the time of the survey request. If, instead, the collective weight of the salient reasons to refuse outweighs the collective weight of the salient reasons to participate, then the person will refuse to participate at the time of the survey request. 
Some of the factors that may influence the decision include:
  • Importance of the survey topic to the potential respondent
  • Sponsorship of the survey and the respondent’s evaluation of that organization
  • Perceived burden of participating
  • Psychological/social rewards of participation
  • Material rewards of participation
Economic Exchange Theory.  This theory (Biner and Kidd, 1994; Porter and Whitcomb, 2003; Lavrakas, 2008) suggests that survey respondents are rational beings who weigh the value of the benefits they will receive from participating in a survey against the costs to themselves for doing so. 
As with Leverage-Saliency Theory, benefits can include intangible factors such as contributing to a public good (knowledge, science), helping researchers, or supporting a recognizable entity, as well as tangible factors such as material incentives. Costs can include the perceived value of the time necessary to complete the survey, distrust of the motives of the survey sponsor or data collection organization, or concern about the security or confidentiality of the information that might be revealed during the survey. If perceived benefits outweigh perceived costs, then the economic exchange theory hypothesizes that the sampled person will comply with the survey request. If the opposite is true, s/he will not.

Social Exchange Theory.  The theory of social exchange is among the most prominent of the theories about survey participation (for application of social exchange theory to survey research see Dillman, 1978; 2007). Social exchange theory hypothesizes that people participate in surveys because of an ongoing exchange relationship between the sampled unit and the survey research organization, the sponsor, or the interviewer (Childers and Skinner 1996; Dillman 1978; Goyder 1987; Goyder, Boyer and Martinelli 2006). The supporting materials for the survey (introductory letters, descriptions of the survey, follow-up materials) are designed to create and emphasize a sense of obligation for the sampled individual to participate in the survey (Groves and Couper, 1998). Designs following this model generally include some material (non-contingent) incentive that is given in advance of the person actually performing the survey task, but do not rely exclusively on material incentives to create the sense of obligation and a desire to reciprocate by helping the researcher. Building trust in the benefits of the research to the individual and felt obligation towards the researcher are the central facets of social exchange theory in gaining cooperation and avoiding refusals.

Cognitive Dissonance Theory.  Cognitive dissonance theory (Festinger, 1957) hypothesizes that an individual with conflicting attitudes or behaviors is motivated to resolve that conflict.  One way individuals can do this is to change either an attitude or behavior. For example, in surveys that give incentives or bring other ideas that make a respondent feel positively inclined toward a survey, an individual may choose to participate to bring their behavior in line with those positive inclinations. That is, cognitive dissonance theory would predict that survey design elements that emphasize how the survey results will benefit the respondent, either directly or indirectly, or those that emphasize how participation is consistent with the potential respondents’ attitudes may create a sense of dissonance in the mind of reluctant respondent who will need to resolve this dissonance by potentially participating.

These theories of participation in surveys are based on a similar understanding that propensity to participate in surveys results from the effects of numerous factors that operate in different ways on different individuals and at different times. More flexible approaches to reduce nonresponse such as response propensity modeling (Lavrakas, Burks, and Bennett, 2004), responsive design (Heeringa and Groves, 2006) and adaptive total design (Biemer, 2010, Mitchell, et al., 2010) draw on the idea that individuals with varying characteristics may be differentially attracted to different design features. In these models, response propensity is viewed as a characteristic of individual sample members that varies depending on the characteristics of the survey.  Thus, an objective of a “responsive” survey design is to employ different (i.e., tailored) approaches, guided in part by (1) information that can be known about each respondent before the field period begins[10] and (2) information on the survey process (paradata) to identify the particular design features that may be particularly attractive to different sampled cases.

Survey Refusals, Then and Now
Survey researchers have observed a trend toward higher refusal rates over the past three decades. Groves and Couper (1998) noted that although the nonresponse rate to the Current Population Survey (CPS) remained consistent between 1955 and the early 1990s, the refusal rate increased significantly over the same period of time.  Atriosic and colleagues (2001) found that the trend continued through the following decade with the initial refusal rate in six U.S. government household surveys – including the CPS and the National Health Interview Study – increasing between 1990 and 1999. Final refusals to the University of Michigan’s monthly Survey of Consumer Attitudes have increased by an average of about one-fifth of a percentage point each year – from 19% in 1979 to 27% in 2003 – although the increase was steeper toward the end of this period (Curtin, Presser, & Singer, 2005).  During this same period, despite the introduction of monetary incentives for those who initially declined to participate, the proportion of refusals converted to completed interviews decreased by about 0.22 percentage points each year; had the incentives to refusers not been used, the rate of decrease would have been greater.  Brick and Williams (2013), examining the National Health Interview Survey from 1997 to 2007 and the National Household Education Survey from 1996 to 2007, found that while overall refusal rates in both surveys increased over this time period, there was no consistent pattern for changes in the proportion of refusals of all non-respondents.

The problem of increasing refusal rates is not limited to studies conducted in the U.S.; for example, de Leeuw and de Heer (2002) analyzed data from longitudinal government surveys conducted in 16 countries.  For most of these countries, data were available from the 1980s through the 1990s. On average, refusal rate increased by 1% every three years, although the magnitude of the change in refusal rate differed significantly from country to country and from year to year.

While much of this research is now at least ten years old, there is no more recent published evidence that suggest these trends have changed (Holbrook, Pfent, and Krosnick, 2007).  In fact, because of the relative dearth of data on refusal rates in the past ten years, very little is known about changes in refusal rates in the past decade, and consequently, very little is known about refusal rates on cellular telephones specifically.  As such, our Task Force requested data from 15 major survey research providers on telephone call outcomes by frame (landline and telephone) of their principal repeated cross-sectional studies, or studies that have otherwise been conducted at least periodically during the past ten years.  Utilizing these field disposition codes, we calculated refusal rates (AAPOR RR2) for studies conducted from 1999 to 2012.  Below are the results.  As is shown in Table 1, refusal rates (aggregated across all the companies that provided data) have roughly doubled in the 13 year span of the table, from the high twenties to as high as fifty percent.  Cell phone refusal rates are generally the same as landline refusal rates, but interestingly trend a bit lower than for the landline samples during the past few years.
 
Table 1: Refusal Rates by Year
(Number of Surveys in Parenthesis)
Year Landline Cell
1999 27% (2)  
2000 29% (2)  
2001 30% (2)  
2002 32% (3)  
2003 29% (3)  
2004 26% (3)  
2005 30% (2)  
2006 34% (2)  
2007 36% (3)  
2008 37% (6) 40% (4)
2009 43% (5) 40% (5)
2010 47% (5) 42% (5)
2011 50% (6) 47% (6)
2012 48% (5) 45% (5)
 
 
Demographic and Attitudinal Characteristics
Assessing whether refusers to in-person and telephone surveys are different from responders is difficult because there is often little respondent-specific information available, particularly about the refusers.  For most mail and internet surveys, the comparison is difficult as well.  For mail surveys, however, one method is to match area-level demographic characteristics to the sampled addresses to learn if there are meaningful differences between responders and non-responders in the areas in which they live (but this, of course, mixes noncontacts with refusers). This approach is only an indirect measure of the characteristics of refusals because the characteristics of the area may not be consistent with the characteristics of the sampled case.  Furthermore, there are no attitudinal or behavioral population parameters at small areas that can be used for such analyses. Overall, we found no literature on characteristics of refusals in mail surveys to review in this report. 

In general population internet surveys that are not based on existing panels, there is little information available on refusers as well.  However, for research using an internet panel, there is a great amount of information (demographic and attitudinal) that can be used to compare refusers to responders.  Internet panel providers keep extensive “profile” information on all their panel members.  However, we found few examinations of refusals in internet panels.  An exception is a series of papers by Lavrakas and his colleagues at GfK, Google, and uSamp. For example, Lavrakas, Dennis, Peugh, Shand-Lubbers, Lee, and Charlebois (2012), using a very rich set of demographic data from GfK’s KnowledgePanel, found that the only demographic variables that were even marginally significant in differentiating responders from refusers were the presence of a child in the home (more likely to refuse) and being of Hispanic ethnicity (more likely to respond).  Lavrakas, Clancy, and Wheeler (2013), using data from uSamp’s internet/mobile panel, found that the only demographics to consistently differentiate refusers and responders were having a low level of educational attainment (more likely to refuse) and being  a younger aged male (more likely to refuse).

Due to the dearth of reported research on mail and internet surveys, the remainder of this section on demographic correlates of refusing is focused on research on refusals in telephone and in-person surveys.

While many studies have attempted to understand the differences between respondents and non-respondents (refusers as well as other non-responders) there are few replicated findings, particularly in studies based on random samples of households. Studies that are based on a random sample can be benchmarked against a known source (e.g. Census data) to identify differences in characteristics between respondents and the known population. However, these studies typically do not have the background data that would be required to compare respondents and non-respondents on the variables of interest (e.g. O’Neil 1979). Moreover, the benchmarking approach can confound factors related to refusals and other forms of nonresponse.

As noted above, samples of individuals from data-rich frames (e.g. internet panels, follow-up response in a longitudinal design, or sampling frames with robust demographic information) allow researchers to compare the characteristics of refusers with those of other non-respondents and participants, but observed differences in these studies may not be generalizable to the general population or other populations of interest. Furthermore, many surveys of this type employ survey modes (e.g. internet, in-person, or mail) which may have patterns of non-response that are different than RDD telephone surveys (e.g. Schwarz et al., 1991; Dillman et al., 2009).

Further confounding researchers’ ability to understand if certain types of people are more inclined to refuse to participate than others is that the decision to participate is also affected by the skill, experiences and characteristics of the interviewers themselves, which may be, purposefully or not, disproportionately distributed across a given sample (e.g. Groves, Cialdini, and Couper, 1992). As well, recent studies have found that participation decisions may also be affected by the subject matter of the survey itself (e.g. Tourangeau, Groves, and Redline, 2010) as well as respondent attitudes toward surveys and perception of the risk and benefits involved in participation (e.g. Peytchev, Peytcheva, and Groves, 2010; Couper et al., 2010). Social isolation, or disengagement from the community may also play a role (e.g., Groves and Couper, 1998; Abraham, Maitland, and Bianchi, 2006).

For all of these reasons, it is not surprising that studies of refusers’ characteristics yield mixed or inconclusive results. Studies based on samples of individuals where demographic information is known have found some differences along dimensions of socio-economic status and age.  An early study found propensity to refuse among older individuals, and those in moderate income groups, but found no differences by race or sex (DeMaio, 1980).  Massey et al. (1981) also found significantly higher refusal rates among those ages 65 or older and noted that initial refusers in this age range were also more likely than those in other age categories to complete a survey when contacted again.  Boersma et al.'s (1997) cross sectional two-stage medical study in the Netherlands found that refusals differed along socio-demographic and health dimensions (housing status, age and prevalence of dementia). Groves and Couper (1998) reported lower cooperation rates among residents who live in single-person households, are childless, and not living in single family homes. Carrol and Chong (2006) reported that those who refused to complete an in-person interview for the National Health and Nutrition Examination Surveys were younger and more likely to be non-Hispanic black.  Converted refusals in that study were similar to initial respondents with respect to education.  Dutwin and Herrmann (2006) found refusal rates in a national health survey to be significantly higher in non-white areas of the U.S., but that conversion rates were most robust in these areas as well.

Studies of refusal characteristics based on data obtained from RDD samples rely mainly on comparing the demographics of participants who initially refused and were converted with those of respondents who initially agreed to participate (reviewed in more detail in the Refusal Conversion section below). Whether initial refusers in RDD studies who later agree to participate are different from those who never agree is much harder to assess, since researchers have very little information about those who don’t participate, even after conversion attempts.   Studies that do compare respondents who never refused to respondents who initially refused but were later converted into a full interview have yielded mixed evidence.  In a study of non-response to an RDD household survey in the city of Chicago, O’Neil (1979) found higher rates of refusal among city residents who were 60 or over; white; lower income, “blue collar” or Polish, German, or Irish persons. O’Neil did not find any differences by gender or marital status.  In 1999, Triplett, Wang, Safir, et al. (2002) compared respondents, refusers, and other non-respondents sampled for a national RDD telephone survey (The National Survey of America’s Families or NSAF) and found initial refusers are more likely to be African American, renters, and members of large households. In 2002, Triplett noted that initial refusals were more likely to be African American, older, and lacking a high school diploma, and found no significant differences by gender, region, household size, or number of children. He also noted that across studies, men who initially refused were easier to convert than women.  Peytchev, Baxter, and Carley-Baxter (2009) found that individuals in the National Intimate Partner and Sexual Violence Survey (NISVS) Pilot Study who had initially refused to participate but were subsequently interviewed after refusal conversion attempts were significantly more likely to be Hispanic than those who never refused. Voigt, Koepsell, and Daling (2003) conducted in-person interviews with respondents who had initially declined to participate in RDD epidemiology studies conducted in Washington State, and found that refusers who later cooperated tended to be nonwhite and unmarried, as well as more likely to have college degrees and be employed in managerial or professional positions. This latter finding conflicts with Triplett’s study (2002). Miller and Wedeking (2006) reported that younger people were significantly less likely to be converted. Johnson et al. (2006) measured positive associations with survey refusal among households in areas of Illinois with “concentrated affluence and residential stability.”

Despite the difficultly of studying the demographic differences between participants and refusers (both converted and unconverted), and the conflicting results of some of the reported findings, there is some consistency among the findings that refusers are more ethnically diverse than participants.  However, research is still needed to gain a clearer picture of the potential systematic differences between participants and non-participants’ demographics.
Patterns of Attitudes/Behaviors Associated with Refusers.  One important question is whether potential respondents who don’t participate in a survey have different attitudes and opinions or engage in different behaviors than those who agree to be interviewed. This is important for two reasons.  First, if refusers have different attitudes, opinions, or behaviors of interest to the survey than participating respondents, this will create nonresponse bias, thus reducing the extent to which the sample is representative of the population of interest.  Second, understanding how refusers differ from those who agree to participate may also inform attempts to convert refusers as it may allow researchers and interviewers to tailor their approach to converting refusals.
Merkle and Edelman (2002) investigated whether exit poll response rates affected the discrepancy between exit poll estimates of vote choice and official estimates of vote choice at the precinct level and found no association. This finding suggests that vote choice may not consistently differ between responders and non-responders to exit polls. Kendrick, Hapgood, and Marsh (2001) found that in child health surveillance checks conducted 18 months after a baseline survey, responders to the baseline survey were more likely to own several types of child-related safety equipment than non-responders. Similarly, using a follow-up study of refusers, O’Neill, Marsden, and Silman (1995) found that those who agreed to participate in a study of vertebral osteoporosis differed in their behavior from refusers in a number of ways (e.g., exercise, smoking, calcium consumption), although the groups did not consistently differ in their risk of osteoporosis (also see Boström, Hallgvist, Haglund, Romelsjö, Svanström, and Diderichsen, 1993).

Another approach to indirectly assess whether refusers and participants differ has been to compare the distribution of respondent characteristics or responses across surveys with different response rates (Holbrook, Pfent, and Krosnick, 2003; Keeter et al., 2000; Keeter, et al. 2006). For example, Keeter et al. (2000) compared the distribution of responses from a survey conducted with a “standard” methodology (that resulted in lower response rate) to one conducted with a “rigorous” methodology (that resulted in a much higher participation rate). Only a small number of responses to substantive items showed differences across the surveys. However, this approach suffers from the same weaknesses of follow-up surveys and using reluctant respondents as a proxy for refusers and non-respondents – such techniques do not necessarily measure differences in opinions or behaviors of hardcore refusers who do not participate no matter how rigorous the methodology or follow-up.

Conceptually, the decision to participate or refuse to participate in a particular survey is likely to be related to two categories of variables.  The first category of variables has to do with attitudes and behaviors that are associated with survey participation generally.  Examples of these types of attitudes and behaviors include: 1) civic engagement and participation, since survey participation can be seen as a form of civic participation or volunteerism (Groves, Singer, and Corning, 2000); 2) altruistic or helping tendencies, since survey participation is often framed as helping researchers and/or interviewers (Porter, 2004); 3) trust (although this may vary by survey sponsor; and, perhaps, 4) a general tendency to like being asked to voice one’s opinion (although this fourth category does not seem to have been explored in the literature[11]).

A second category of attitudes and behavior associated with survey participation are survey-specific – that is to say that they are tied to specific characteristics of the survey (or an interaction between characteristics of the survey and the individual respondent).  As such, these attitudes and behaviors may be associated with participation in one survey but not another.  These include the potential respondents’ attitudes toward the sponsor of the survey (McCarthy, Johnson, and Ott, 1999), the survey organization, and the topic of the survey (e.g., Bucholz et al., 1996; Groves, Presser, and Dipko, 2004; Peytchev, Peytcheva, and Groves, 2010). The topic of the survey may also affect the willingness to participate both by leading to higher participation among respondents who are more interested in the topic than among those who are less interested (e.g., Groves, Presser, and Dipko, 2004), and by reducing participation among respondents who want to avoid having to give socially undesirable reports of attitudes or behaviors such as prejudice or having had an abortion (e.g., Benson, Booman, and Clark, 1951; Peytchev, Peytcheva, and Groves, 2010).  Because the reasons for refusing may vary across surveys, findings regarding differences between respondents and refusals may not always generalize across studies.

Included in this second category of survey-specific characteristics for why someone will participate or not are factors such as 1) how busy the person is at the time of the survey request, 2) the person’s general disposition at the time of the survey request, 3) the nature and value of the material incentives, if any, that are offered, 4) the respondent’s perception of her/his qualifications to provide answer of value about the survey topic, and 5) the perceived burden associated with participating in the survey (e.g., Lavrakas, 2001).

As this section underscores, most research on the potential differences between participants and refusers is difficult and expensive to conduct, leading to a relative dearth of data with which to assess potential systematic refusing by demographics, attitudes, knowledge or behaviors.  Other than meta-data on the sample itself (e.g., location based on area code), there is almost no information available about hardcore refusers in the vast majority of surveys. The exception is in internet panel surveys where the panel vendor accumulates a host of rich and varied variables about the panel member from previous questionnaires the panelist has completed.  However, these panels most often limit generalizability to large scale cross-sectional research As well, most panel vendors are unwilling to release information about how their refusers compare to their participators on a given survey and most clients appear unaware of the value to them of asking for those data from panel vendors.  Necessity being the mother of invention, researchers have “made do” with the data available, comparing panel respondents who are more difficult to recruit with those who more readily participate.  Thus far, this research finds relatively small differences, on average.  As such, there is a great deal left to understand with regard to potential systematic nonresponse from survey refusals.
 
REFUSAL AVERSION

Definition and Purpose
Refusal aversion is any action taken by the survey researchers and their research staff (including supervisors and interviewers in interviewer-administered surveys) to avoid a refusal or other non-participation outcome before they actually occurs  As this section will detail, these include considerations for survey design, such as effective communication in initial contacts in cover letters/emails and between interviewer and the potential respondent.  Also, we consider subsequent communication in later contact attempts, such as persuasive fallback statements for use by interviewers when a potential respondent is hesitant to participate.  Survey design features to minimize refusals also include overarching design decisions such as the use of incentives or advance notification techniques.  Survey research organizations may also attempt to avoid refusals throughinterviewer management, both by hiring only interviewers they think will succeed at avoiding refusals based on their experience, personality, attitudes, and even voice, and by training interviewers in such a way as to proactively and reactively try to minimize refusals. 

Why Employ Refusal Aversion Methods?

To Reduce Bias.  From a methodological and statistical perspective, the principal motivation to avert refusals would be to reduce or even eliminate nonresponse bias that is introduced when some respondents participate and others, who would respond differently to key survey measures, refuse. Under a deterministic model for nonresponse, nonresponse bias can be expressed for an unadjusted sample mean as a function of the nonresponse rate (M/N) (where M is the actual number of respondents and N is the total number of potential respondents sampled) and the difference in means between respondents’ and non-respondents’ answers for a survey question of interest:
Bias(YR)=(M/N)(YR-YM )

Reducing the number of refusals by aversion (or conversion) techniques affects possible nonresponse error in two ways:  1) by reducing the nonresponse rate (as M/N goes to zero, the possible nonresponse bias goes to zero) and 2) by reducing the difference in means between the respondents’ and non-respondents’ survey responses (as the difference (YR-YM) goes to zero, the possible nonresponse bias goes to zero).

An alternative expression for nonresponse error indicates that the nonresponse bias of an unadjusted respondent mean is a function of the covariance between an individual’s likelihood of participating in the survey (P) and a survey variable of interest, Y (cov(P,Y)), and the average response propensity ( P ): Bias(YR) = cov(P,Y)/P.  Under this stochastic model, refusal aversion and conversion changes the average response propensity and potentially changes the covariance between response propensity and the survey variable of interest (Bethlehem 2002; Groves 2006). 

As reviewed in the previous section of this report, it is not clear when refusals will introduce bias in a survey, whether it is consistently across surveys or idiosyncratically in certain surveys.  It is important to note, however, that research has found demographic, attitudinal and behavioral differences between refusers (or proxy measurements thereof) and participants, even though few if any consistent differences have been found from study to study.  Overall, the argument that one should try to avert refusals and therefore minimize bias, or at least the potential for bias, of survey estimates overall is a credible one. 

To Improve Survey Face Validity and Response Rates.  A second reason to attempt to avoid refusals is to improve the face validity and perceived credibility of a survey research study.  As documented in the previous section, the number of refusals in a typical opinion research study of the general population can be considerable. In many cases, refusal rates are much greater than those from as recent as 10-15 years ago.  Since it is often difficult (or impossible) to determine whether there are significant differences in survey responses from respondents and the responses that would have been given by non-participants, endeavoring to improve the response rate to a survey can be a tangible way to address nonresponse bias due to refusals.  While response rates are not the only determinant of potential nonresponse bias, the avoidance and reduction of refusals can improve the survey from a face validity perspective.

In addition, although there is no consistent link between nonresponse rates and nonresponse bias (Groves, 2006; Groves and Peytcheva 2008), many industries and fields have ‘minimum’ response rate standards that have to be met for the research to be considered to be valid.  For example, the U.S. Office of Management and Budget requires at least an 80 percent response rate, or else recommends that a study of nonresponse bias be conducted in order to gain government funding.[12] Other non-governmental studies typically have much lower response rate minimums, some as low as 10%, but nevertheless have some contractually set rate.  Averting (and converting) refusals is thus a means by which these minimum response rates can be achieved.
To Reduce Costs.  Refusals result in survey researchers having to select and contact a larger sample size than they would have in the absence or reduction of these refusals.  Thus refusals have important cost implications for a survey budget.  In surveys with a target minimum final sample size, each final refusal during the field period requires at least one additional unit to be sampled and attempted to obtain the required number of completions.  For each averted refusal, costs are reduced through minimizing potential conversion costs or costs for contacting a newly sampled case. Although this cost reduction occurs in all survey modes, cost implications for each averted refusal likely are larger for interviewer-administered surveys (in-person surveys and telephone surveys) than for self-administered surveys.  In theory, costs saved by averting a refusal, raising face validity, and possibly improving overall data quality should offset some or all of the costs of the refusal aversion strategies.

The costs associated with refusals can be broken into two categories:  direct costs and indirect costs.  Direct costs are costs related to the collection of the survey data that can be measured using reliable accounting data.  For example, the direct costs associated with refusals for a telephone survey primarily are the labor and telephone expenses associated with those refusals.  For an in-person survey, those direct costs are the labor expenses along with transportation expenses to execute the survey in-person.  The primary direct costs related to mail and internet surveys are the initial costs associated with contacting and experiencing a refusal from each uncooperative respondent.  Across all survey modes, initial incentives and additional incentives can be used to attempt to avoid refusals, which contribute to the direct costs.

Indirect costs are those costs which cannot be quantified (or quantified completely) using available accounting data.  Decline in interviewer morale that can result from handling difficult and/or numerous refusals is one such indirect cost.  This indirect cost manifests itself in lower interviewer productivity, which may be more difficult to quantify.  In addition, other indirect costs associated with refusals are a decline in data quality and a decline in survey quality measures like response rates.   

To address whether minimizing refusals makes sense from a cost perspective, any analysis should consist of an accounting of the direct and indirect costs associated with refusals.  To the extent that costs to avert refusals (sending advanced notification about the survey, leaving messages or other survey information, offering incentives, developing special interviewer training, etc.) are more than offset by the costs that would have been incurred by those refusals, refusal aversion efforts may be justified from a cost perspective.  (To the extent that refusal aversions lead to a reduction in nonresponse bias, refusal conversion efforts also may be justified from a data quality perspective.[13])

Unfortunately, to our knowledge, the cost impact of refusals has not been addressed in the literature, with one exception.  Stec and Lavrakas (2007) analyzed cost data (direct and indirect costs) from a very large RDD telephone survey, the Nielsen Diary Placement Survey.  They estimated that approximately 19.3% of the total costs of the survey could be attributed to these initial refusals.[14]

To Reduce Effort.  Reducing refusals using refusal aversion techniques requires additional effort.  This additional effort, for example, can include administering and processing incentives or advance letters, increasing the frequency and intensity of interviewing, and conducting more extensive interviewer training.  From another perspective, however, instituting successful refusal aversion methods reduces the number of sample records to be considered for refusal conversion.  Therefore, while efforts to avert refusals lead to additional resources expended, methods implemented to avert refusals can save on other resources that would have been used to address the higher incidence of refusals absent these aversion methods. 

Of course, how much averting refusals will impact the total costs of the survey depends upon the type, intensity, and efficacy of efforts to avert refusals.  These primary methods used by survey researchers to avert refusals are reviewed below.  The methods are divided into two main types of aversion strategies:   survey design strategies and  interviewer management and training strategies.

Refusal Aversion Methods: Survey Design

Survey Introductions.  The survey introduction – including advance letters/emails and what is administered by an interviewer or written in a cover letter – is the first impression that a survey researcher makes on the sampled respondents.  That first impression can, and often does, have a considerable impact on the sampled respondents’ ultimate decisions to participate in the survey.  For example, the disclosure of subject matter in a survey introduction can lead to a refusal bias.  Specifically, if subject matter disclosed in the survey introduction has a large impact on whether respondents decide to participate and that subject matter also is measured in the survey instrument, then the sample statistics based on this subject matter can suffer from nonresponse bias due in part to refusals from those who hold different views about the topics mentioned in the introduction compared to those who decide to participate in the survey once they learn of the subject matter (Groves and Peytcheva, 2008).[15] 

Survey introductions can be categorized into five parts, in the order that they are likely to occur:  1) advanced letters/emails; 2) interviewers’ introductions or cover letter/email introductions at the time the survey task is first presented; 3) the respondent selection sequence; 4) addressing respondents’ questions and concerns; and 5) setting callbacks (Campanelli, 2008).  Dillman et al. (1976) reported that the earliest moments of the interaction with a respondent are particularly crucial.  He found that of respondents contacted, very few respondents (1-4%) terminate the actual questionnaire once it was started.[16]  Groves et al. (2004a) note that decisions to participate in a survey often are made in less than the first 30 seconds after contact.   Therefore, timely tailoring of the introduction to impart information pertaining to the identification of the interviewer and research organization, explanation of the research and its importance, and promise of confidentiality, as well as persuasive reasons for the respondents to participate, is crucial to avert refusals.[17] 
In crafting survey introductions and training interviewers to use them successfully, the mode of the survey is an important consideration.  For mail and internet surveys, essentially no interactive communication takes place.  So, well-crafted, written introductions are important.  In telephone surveys and to an even larger extent with face-to-face surveys, interpersonal communication becomes more important.  In this context, interviewers need to be focused and carefully tailor communications to elicit participation (Miller and Cannell, 1982; Groves and McGonagle, 2001; Shuttles, Welch, Hoover, and Lavrakas, 2002).
Considering that refusals during the introduction typically  occur quickly and that structured introductions are not very effective at obtaining cooperation, Groves and McGonagle (2001) stressed the importance of tailoring introductions and maintaining interaction with the respondent for as long as possible.  To tailor the introduction effectively to respondents, Groves and McGonagle found that experienced interviewers “engage in a continuous search for cues about the attributes of the sample household or the person who answers the door (or phone), focusing on those attributes that may suggest their approach to the participation decision” (p. 250).  Experienced interviewers recognize these cues and use them to tailor an effective survey introduction approach.
Maintaining interaction is an important determinant of a successful introduction.  In order to gain cooperation in telephone surveys, Groves (1979) recommended delaying an interview request and to spend time explaining the survey, the survey organization, the role of the respondent, and the proposed use of the data.  In fact, it has been suggested that “soft refusals” often occur when interviewers press respondents to participate in the survey too soon after initial contact (Groves et al., 1992) or when interviewers do not adequately address respondents’ concerns (Groves and Couper, 1998).[18]  The intent with extending the conversation between the interviewer and respondent is to give the interviewer the opportunity to gain as many informational cues as possible in order to tailor the survey request most effectively as well as to invoking the “commitment principle” – i.e., the longer the interaction goes, the more difficult it becomes to dismiss the interviewer (Groves and McGonagle, 2001).

Survey Fallback Statements.  Survey fallback statements (i.e., persuasive, declarative statements that are authored prior to the survey entering the field to address FAQs posed by respondents) are brief targeted scripts that can be used by interviewers – either as they were written or tailored by the interviewer to the person to whom they are speaking – to try to overcome respondents’ initial reluctance to participate in a survey.  Often, survey researchers can anticipate many of the questions that respondents will ask of interviewers before making their decisions to participate or not (Lavrakas and Merkle, 1991) and the reasons respondents may give for not wanting to participate in a survey.  Fallback statements can be presented to interviewers as part of the training for a specific survey and/or as part of their general training.[19]  Common topics that fallback statements address include: 1) how the respondent’s address or telephone number was chosen, 2) more details about the purpose of the survey, 3) the importance of the survey, 4) how the survey data will be used, 5) more details on how the survey data will be kept confidential, 6) how and why the designated respondent (and only the designated respondent) was chosen for the survey, and, 7) the name of a contact person at the survey organization and that person’s telephone number which the respondent can use to verify the legitimacy of the survey (Lavrakas, 1987; 1993; 2008).

Multiple Modes.  Mixed mode surveys are increasingly being used in an attempt to increase response rates while attempting to control costs (de Leeuw, 2005).  Researchers hope that offering a choice of modes to a sampled respondent or household will increase the likelihood that they will participate, and thus, increase response rates overall (cf. Groves and Kahn, 1979).  Researchers also hope that mixing modes will facilitate the use of a lower cost (albeit with a possibly lower response rate) mode, such as the mail or telephone, for part of the data collection effort that brings in the easiest-to-get respondents, thereby permitting more resources to go toward a higher cost (and possibly a higher response rate) mode such as in-person for the hardest-to-get respondents. Empirical evidence on whether mixing modes yields higher response rates or lower refusal rates than a single mode study appears to depend upon (1) the modes that are deployed, (2) the order in which they are implemented, and (3) the particular recruitment procedures deployed in each mode (de Leeuw, 2005; Dillman et al., 2009; Gilbert, 2009; Levenstein, 2009; Link and Mokdad, 2006; Olson and Groves, 2012; Olson, Smyth, and Wood, 2012; Shih and Fan, 2007).

Whether mixed mode studies affect refusal rates, however, is less clear.  Limited empirical evidence exists examining whether mixed mode studies affect nonresponse bias, but to our knowledge, the work that has been reported does not examine refusals in particular. For example, in their meta-analysis examining nonresponse bias, Groves and Peytcheva (2008) found that interviewer-administered modes have larger nonresponse biases than self-administered modes, but they did not explicitly examine mixed mode surveys compared to other surveys.  Other more recent studies (e.g., Dillman et al. 2009; Beebe, McAlpine, Ziegenfuss, Jenkins, Haas, and Davern 2012; Scott, Jeon, Joyce, Humphreys, Kalb, Witt, and Leahy 2011) similarly found no consistent pattern in nonresponse biases across different modes or survey estimates. The general finding across studies is that different modes may or may not bring in different types of respondents, and this appears to depend on the specific modes that are used and how nonresponse bias is evaluated.  However, none of these studies was conducted in a way that could disentangle refusal-related nonresponse from noncontact-related nonresponse. 

Thus, a systematic review or meta-analysis is needed to examine nonresponse biases across simultaneous and sequential mixed mode surveys and in a way that can disaggregate noncontact and refusal causes to discern more clearly their independent effects on nonresponse bias. There is a great deal more research that is needed to better understand the circumstances under which using more than one mode to gain cooperation from sampled respondents, including reducing refusals, can be successful in raising response rates and possibly reducing nonresponse bias.  The literature noted here so far suggests there is reason for guarded optimism.  However, whether it also can be cost-effective, and under which circumstances, is another matter that remains unknown and needs considerable further attention.

Advanced Notification.  A pre-notification letter or advance letter is a letter sent to a sampled person or household shortly before the formal survey request is made.  The advance letter generally seeks to build anticipation for the study and legitimize the request (Camburn, Lavrakas, Battaglia, Massey, and Wright, 1995; Dillman, 2007).  Advance letters appear to work by reducing the refusal rate, as they have not been found to impact the non-contact rate.[20]  Most importantly, advance letters are an additional contact attempt with the respondent.  Additional contacts have been shown to be one of the more effective ways to boost response, and indeed, it is not clear if the advance letter is more effective than an additional questionnaire mailing or other contact strategy. (Heberlein and Baumgartner, 1978; Dillman, 2007).

Research has shown that effective advance letters are short, addressed to a specific person, emphasize the sponsorship and purpose of the study, and emphasize the importance of participation.  The letters should also invoke norms of reciprocity by offering something back to the respondent (e.g., information, summary reports, and/or incentives) (Dillman, 2007; Camburn et al., 1995; De Leeuw et al., 2007).  In telephone studies, including a non-contingent incentive with the advance letter has been shown to improve significantly the effectiveness of the mailing (of note, the letter should be sent no more than one week before the initial survey request, cf. Shuttles, Lavrakas, and Lai, 2004);

The results of individual studies using an advance letter without an incentive have been equivocal.  Some studies have reported no improvement in response rates or a negative impact.  However, reviews that have aggregated multiple studies find overall that advance letters are an effective way to boost response rates to mail, telephone and in person studies (Fox et al., 1988; Heberlein and Baumgartner, 1978; De Leeuw et al., 2007).  There also is some evidence that differential demographic groups are more likely that others to be influenced by advance letters (Groves and Peytcheva, 2008; Goldstein and Jennings, 2002; de Leeuw et al., 2007; Mann, 2005).  As well, RDD based surveys may be more susceptible to differential effects from advance letters as the characteristics of households with matched phone numbers are different than those with unmatched numbers (Link and Mokdad, 2005). 

Leaving Study Information.  Leaving study information in the form of study brochures, answering machine messages, and emails can be a way to avert refusals.  Most of the empirical research in this area focuses on the use of answering machine messages.  For respondents that use answering machines to facilitate communication when they are not home, survey researchers have hypothesized that scripted messages left on answering machines or voice mail could prepare the respondents for future attempts to contact them; and, perhaps even encourage respondents to call the survey organization to complete the survey (Weisberg et al., 1996; Link, 2008). 

Xu et al. (1993) found that the response rate was significantly higher when an answering machine message was left than when it was not.  Tuckel and Schukers (1997), however, found little difference in the contact rate or the completion rate for households by whether an answering machine message was left or not.  The researchers did find that the completion rate for a brief message was more than ten percentage points higher than for a longer message.  This finding comports with Dillman et al. (1976) where they found brevity in communication with respondents is preferred. Link et al. (2003) found only a small and statistically insignificant difference in the completion rate between households where an answering machine message was left, and refusal rates that were slightly higher for households where an answering machine message was left.  There was also almost no difference in refusal conversion rates between the answering machine message group and the no answering machine message group.

In a large national experimental study, Shuttles and Lavrakas (2005) tested several different answering machine messages by using a theory-based approach to vary the content of the message, but found no significant impact on response rates to a landline RDD survey across the different messages.  Benford, Lavrakas, Tompson and Fleury (2010), in three national studies using RDD cellphone and landline numbers, either left a voice message upon first contact with an answering machine or voice mail (treatment condition) or did not (control condition).  Although there was no consistent effect on refusal rates (AAPOR Refusal Rate 2) for the landline part of the RDD samples, leaving a voice message consistently and significantly lowered refusal rates for the cellphone part of sample in each of the three surveys by an average of 6.5 percentage points.

The implication of these more recent studies suggests that leaving answering machine messages on landline telephones has limited or no impact on averting refusals and improving completion rates.  Moreover, there are real costs to leaving answering machine messages.  This suggests that limited survey resources that might be devoted to leaving answering machine messages on landline telephones could be better utilized elsewhere.  However, more research is needed on the effect of messages left on cellphone voice mail before researchers can be confident of the wisdom of this approach to possibly lowering refusal rates and improving response rates.

Incentives.  Incentives have been used in survey research to try to decrease refusal rates and thereby increase response rates for more than 80 years (Shuttleworth, 1931); also see Armstrong (1975), Church (1993), Kulka (1994), Singer et al. (1999), Edwards et al. (2002), Cantor et al. (2008) and Holbrook et al (2008), Singer et al. (2013) for summaries and meta-analyses of incentives studies)[21].  Incentives have been used across research disciplines by both the private and public sector and for surveys of establishments, households, and persons.   The use of incentives to lower refusal rates and thereby raise response rates has been extensively researched.  However, there remains a great deal still to learn about their effects on other survey goals, in particular their potential for reducing refusal-related nonresponse error and their effects on total survey costs (cf. Lavrakas, 2012).

Past research on averting refusals and raising response rates has focused on the use of contingent and/or non-contingent incentives, the timing of the incentive offer (e.g., in an advance letter, at the time of first contact, at some later contact, or between panel waves), the application of incentives in different modes, the type of incentive offered (e.g., cash, non-cash, or non-material), the amount of incentive offered, the method of delivery of non-contingent incentives, and even the denomination of the bills with which cash incentives are given.

Fixed versus Differential Incentives and Survey Refusal Rates. Much of the research literature has focused on offering the same incentive to all respondents, and many research organizations follow this practice.  Offering the same incentive to all respondents is consistent with the principal of justice laid out in the Belmont report which encourages fair treatment of all respondents (see Oldendick, 2012 for a discussion of ethical considerations in using incentives).  Offering the same incentive to all respondents can reduce operational burden by streamlining operations and potentially reducing the overhead costs associated with incentives by allowing the organization to purchase larger quantities of a single item.  But in terms of reducing refusals, offering the same incentive to all respondents makes little theoretical sense (cf. Trussell and Lavrakas, 2004; Lavrakas, 2012), given the evidence that the same incentive has a differential impact on different respondents (cf. Trussell and Lavrakas, 2004). 

Response Propensity Modeling.  Knowing which respondents will be motived by an incentive to participate rather than to refuse is not an easy task.  That said, researchers on some studies can know much about each sampled respondent before any survey request is made, and additional information can be known about the respondent after the survey field period begins.  Often this initially known auxiliary data can be appended to the initially designated sample to predict the likelihood a given respondent will participate in the survey task.  For some surveys (e.g., panel studies, especially online panels, with multiple waves) this information is surprisingly robust.  Lavrakas and Burks and their colleagues have shown that RDD landline survey information (including paradata) can be used to compute a response propensity score for each respondent that correlates significantly with whether the respondent participated or not in a subsequent mail survey (Lavrakas, Burks, and Bennett, 2004; Burks, Lavrakas, Bennett, and Usamanova, 2004; Burks, Lavrakas, and Bennett, 2005).

In a very large national experimental study that looked at the impact of differential incentive amounts and prior contact with the respondent, Trussell and Lavrakas (2004) found that respondents in a mail survey who were initially more willing to participate were less impacted by a non-contingent incentive than those who could not be contacted during the prior telephone mode’s recruitment phase and even less so than those who had initially refused in the prior phase of recruitment.  In a two-phase mail study, McPhee and Hastedt (2012) also noted that households that required multiple contacts to respond at the initial phase were more responsive to a larger cash incentive in the second phase of the study than those that had responded to earlier initial phase mailings. 
Lack of Interest in the Survey Topic.  Other researchers have theorized and shown that incentives can offset a lack of interest in the topic area; e.g., Groves, Presser, and Dipko (2004) and Petrolia and Bhattacharjee (2009).

Providing Multiple Incentives.  Some researchers explain the diminishing returns that have been observed in studies with multiple incentive levels as an indication of a change from social exchange to economic exchange in the mind of the U.S. general population, whereby the researcher is paying the person for response (Dillman, 2007).  However, there appears to be no published evidence of these hypotheses being tested directly. 
Incentives and Survey Effort and Costs.  Incentives have been found to lower refusal rates (Eyerman et al., 2005; Singer, 2002) and thereby to reduce the number of calls and days required to complete cases (Painter et al., 2003).

Timing of Incentives:  A number of meta analyses have shown that incentives are most effective at increasing response rates when they are non-contingent in nature and thus are provided with the initial survey request in a mail survey (Church, 1993; Singer et al., 2000; Fox et al., 1988) or with an advance pre-notification letter in a telephone survey (Camburn,  et al. 1996; Shuttles and Lavrakas, 2004) or in an in-person survey; or given by the interviewer (Berlin et al., 1992; McGrath, 2006; Eyerman et al.,2005). 

Recently, several studies have shown the effectiveness of a prepaid incentive reducing implicit refusals and thereby in increasing response via the internet (Millar and Dillman, 2011; Messar and Dillman, 2011).  Generally, these studies have mailed the incentive to the respondent with a letter that provided information on completing the survey over the internet.  Offering an incentive unconditionally decreases refusal rates and increases response rates more so than a comparatively larger contingent incentive, in which the person must respond to obtain the incentive.

Incentives also can be offered between waves in a panel survey.  A number of studies have looked at the impact of incentives in an earlier wave of a longitudinal study on later waves.  The results generally indicate that prior incentives do not have a negative impact on participation in future waves with no incentives and do not set expectations for higher incentives (Colicchia et al., 2011; Singer at al., 1998 and 2000; Lengacher et al., 1995).

Impact of Incentives.  Twenty years ago a meta-analysis of mail survey incentive experiments showed that using non-contingent (prepaid) token cash incentives markedly reduced refusals and boosted response rates by an average of 19.1 percentage points (Church, 1993).  In an analysis of telephone surveys Singer et al. (1999) found that each dollar of increased incentive led to a one-third (0.0033) of a percentage point increase in the response rate.  Singer et al. (1999) also noted that the impact of incentives was lower in studies that already achieved high response rates without the incentives. 
Using data from a national mixed mode (telephone and mail) experiment (n = 113,743), Trussell and Lavrakas (2004) examined the change in cooperation with a non-contingent incentive varying it in one dollar increments.   Consistent with past research (e.g., Armstrong, 1975; Dillman, 2000; Warriner et al., 1996; Singer et al., 1999), the greatest incremental gain in cooperation came with the first $1 given.  Furthermore, Trussell and Lavrakas showed that the rate (slope) of the increase in cooperation in the mail stage of the survey per $1 additional dollar value in incentive was greatest among the cohort that had previously refused to participate in the telephone stage of the survey. Lavrakas and Shuttles (2004) found that the most cost-effective incentive value was $2, lowering refusal rates and thereby raising response rates in the first stage telephone survey by six (6) percentage points more than the non-incentive condition and also lowered refusals and raised cooperation by four (4) percentage points in the subsequent mail survey stage that followed the RDD survey.
In most comparisons, cash has been found to work much better in lowering refusals than non-cash gifts, lotteries or donations to charity (Church, 1993; Kropf et al., 1999; Singer et al., 1999; Singer, 2002).   Since the effectiveness of cash over other incentives has been established, researchers have begun to look at optimal cash amounts.  Some analyses have found a linear relationship between incentive amount and response rates (Singer et al., 1999; Church, 1993).  In contrast, Fox et al. (1998) found support for a diminishing returns theory whereby the gain in response rate for each additional dollar of incentive begins to go down.  However, other more recent studies have had findings where the difference between a small incentive and a larger incentive is greater than the difference between no incentive and the lowest incentive amount (McGrath (2006), Upchurch and Battles (2006), or where the difference is neutral (Brick et al. (2011). 

Nonresponse Bias and Incentives.  Eyerman et al. (2005) found that incentives were effective at increasing response across subgroups, but did not differentially increase response within subgroups.  McGrath (2006) found that an incentive brought in more African American respondents which increased the representativeness of the surveyed population.  Other research indicates that incentives may bring in a greater number of lower income respondents (Kulka, 1994).  In a review of published literature, Singer (2002) concluded “in all of these studies, certain kinds of dependent variables would be seriously mis-measured if incentives had not been used.”  

Contact Rules:  Prior studies investigating telephone survey calling rules in surveys of the general population have found, or at least suggested, that efficiencies may be gained through “optimizing” calling rules by reaching people when they are most available and willing to cooperate.  Much of the literature discusses which days of the week and which times of the day to call sample units.  Weekday evenings and weekends have consistently been found to have the best contact rates, whereas weekday daytime hours are least productive in terms of contact.  Increasing the amount of time between dialing attempts (also known as “lag-time”) has been found to be beneficial, providing the field period makes this feasible (Weeks, Kulka, & Pierson, 1987; Massey, Wolter, Wan, & Liu, 1996; and Odom & Kalsbeek, 2000).

Much of the research noted above is not focused solely on reducing refusals, but rather are strategies that have been investigated to reduce nonresponse rates – and in particular, noncontact rates – more generally.  With that said, given that refusals are a large part of nonresponse, if such strategies reduce overall nonresponse, they can potentially also reduce the likelihood of refusals. 

Refusal Aversion Methods: Interviewer Management
Interviewers play a key role in influencing whether a contact with a potential respondent will end as a refusal or as some other disposition.  In telephone interviewing, for example, it is not only what they say but how they say it that can lead to a refusal or the avoidance of a potential refusal. How the interviewer’s voice “sounds” to whoever has answered the incoming phone call, including the demographic characteristics of the interviewer conveyed by the interviewer’s voice, can be important.  In in-person interviewing, not only do voice characteristics impact refusal aversion but so does the information conveyed by an interviewer’s appearance. 

As noted previously in this report, the content of survey introductions  affects the propensity to participate.  However, less is known about the potential impact of an interviewer’s own behavior and characteristics.  What is clear however is that there is typically a great deal of variation across interviewers working on the same survey project in their refusal rates (cf. Lavrakas, 2010).

To understand how interviewers affect participation decisions, there are essentially two issues to consider: (1) what are the characteristics interviewers “bring to the table” when they are hired, and (2) what skills can be developed by interviewers to help them learn to avert refusals.

Interviewer Experience.  A reliable predictor of higher response rates (and lower refusal rates) is interviewer experience, or proxies for experience, such as pay grade (Campanelli, Sturgis, and Purdon, 1997; Couper and Groves, 1992; Durbin and Stuart, 1951; Durrant, Groves, Staetsky, and Steele, 2010; Groves and Couper, 1998; Hox and De Leeuw, 2002; Hughes, Chromy, Giacoletti, and Odom, 2002; Link, 2006; Lipps and Pollien, 2011; Pickery and Loosveldt, 2002; Sinibaldi, Jackle, Tipping, and Lynn, 2009, but see Blohm, Hox, and Koch, 2006).  Interviewers who have more experience tend to have lower refusals rates and thus higher response rates. As a result, the Cross-Cultural Survey Guidelines (University of Michgan Survey Research Center, 2010) recommend to “select interviewers who have previously worked on similar studies and have good recommendations based on their performance” (http://ccsg.isr.umich.edu/iwerselection.cfm). Granted, this phenomenon may be a result of attrition of poor performing interviewers from the interviewing pool, thus leaving only better interviewers among the ‘experienced’ group (“survivorship bias”). Nevertheless, experienced interviewers tend to have higher response rates than inexperienced interviewers in both face-to-face and telephone modes.

Interviewer Personality Characteristics.  When hiring experienced interviewers is not possible, some organizations have attempted to use tests at the hiring stage or at the beginning or ending of training to evaluate potential interviewer performance. One type of test identifies interviewer personality characteristics that are associated with higher response rates. Empirical evaluations of these types of tests have found little consistent success and/or counterintuitive findings.  During the mid-1980s, the U.S. Census Bureau examined a variety of personality traits using three personality assessments, including the Jackson Personality Inventory. It found that interviewers whose personality traits indicated that they had high complexity and high innovation had lower response rates, and those with high organization and ‘social adroitness’ had higher response rates(Marshall, Brown, and Page, 1989).  In the early 1990s, Groves and Couper (1998) examined whether interviewers with a self-monitoring personality trait had higher cooperation rates, but found little association.

Some organizations have tried to use the “Big 5” (see John and Srivastava, 1999) personality factors.  For example, Shuttles, Skyrme, Camayd-Freixas, Haskins, Wilkinson, Vallar, and Lavrakas (2008) examined the relationship of personality, cognitive, and language measures with actual training and objective job performance of incumbent and newly hired employees.    Regarding personality traits and their relationship with an interviewer performance metric (i.e., the ratio of the number of refusals to the number of completions an interviewer achieved), it was found that greaterConscientiousness[22] and greater Emotional Stability[23] correlated significantly with having lower refusal rates. More recently, interviewers whose ‘Big 5’ personality traits included Agreeableness (Jäckle, Lynn, Sinibaldi, and Tipping, 2011; Yu, Liu, and Yang, 2011) and Openness (Jäckle, et al., 2011, but see Yu, Liu and Yang, 2011) were found to have lower response rates, whereas Conscientious (Jäckle, et al., 2011, but see Yu, Liu and Yang, 2011) and Extroverted (Jäckle et al., 2011; Yu, Liu and Yang, 2011) interviewers have higher response rates. Across the three studies using the Big 5 personality traits, there is no single trait that is consistently associated with response rates. Conscientiousness and Extrovertedness are associated with higher response rates in two of the studies, whereas Agreeableness is associated with lower response rates in two of the studies.

Interviewer Attitudes and Expectations.  In addition to interviewer personality traits, research has investigated whether interviewer attitudes and expectations can also impact participation. Confidence or optimism in being able to convince respondents to participate has been found either to have no association with response rates (Link, 2006; Groves and Couper, 1998) or to have a positive effect (Durrant, Groves, Staetsky, and Steele, 2010; Snijkers, Hox, and de Leeuw, 1999). Interviewers who think that respondents can be persuaded or who are willing to engage in persuasion techniques have higher response rates than other interviewers (Durrant, Groves, Staetsky, and Steele, 2010; Groves and Couper, 1998; Hox and De Leeuw, 2002; Singer, Frankel, and Glassman, 1983; Sinibaldi, Jackle, Tipping, and Lynn, 2009), although this relationship does not hold uniformly across the research literature (Blohm, Hox, and Koch, 2006; Campanelli, Sturgis, and Purdon, 1997). Most of the work on interviewer attitudes and expectations has been conducted on large-scale face-to-face surveys, so little is known about interviewer attitudes and expectations in telephone surveys (see Singer, Frankel and Glassman, 1983 and Link, 2006 for exceptions).

Interviewer Testing.  Knowledge and performance tests administered at the end of training generally have been unsuccessful at predicting future response rates. Groves and McGonagle (2001) developed written skills tests to administer at the end of refusal aversion training to evaluate how well the interviewers could match potential concerns with themes and provide an effective response. These tests failed to be powerful predictors of later performance.  The same was found by Shuttles et al. (2002; 2003).  O'Brien, Mayer, Groves, and O'Neill (2002) audio-taped interactions between interviewers and fake respondents at the end of training, and then used coders to evaluate various criteria. These researchers also failed to find an association between the training outcomes and the field outcomes. This is clearly an area needing greater empirical investigation.

Interviewer Voice Parameters.  Vocal characteristics of survey interviewers’ speech, such as speed and pitch, or ratings of speech such as ‘confidence’ and ‘pleasantness’ have been examined as survey participation predictors. Early studies selected a relatively small number of high and low response rate interviewers and evaluated their speech using both raters and acoustic measurements. These older studies found that interviewers with lower refusal rates had higher pitched voices, had more variability in their speech, spoke faster, and were rated as more confident (e.g., Oksenberg and Cannell, 1988; Oksenberg, Coleman, and Cannell, 1986).

However, these early findings have not always been replicated in more recent, larger-scale investigations (e.g., Benkí, Broome, Conrad, Groves, and Kreuter, 2011; Groves, O'Hare, Gould-Smith, Benki, and Maher, 2008; van der Vaart, Ongena, Hoogendoorn, and Dijkstra, 2006). For example, Groves, et al. (2008), using sophisticated linguistic technology, found that interviewers who spoke more quickly had no difference in cooperation rates and those rated to be more confident actually had lower cooperation rates than their slower paced or less confident counterparts. Additionally, Benki, et al. (2011) found a nonlinear relationship between speech rate and response rates. Clearly, there is more work needed here before using vocal characteristics as a screening device for interviewer hiring.

General Refusal Aversion Training.  Some refusal aversion training for all interviewers generally occurs as a small part of the normal interviewer training. The length of interviewer training at different survey research organizations (and even within the same organization across different surveys) varies dramatically, lasting from about half an hour to multiple days (Tarnai and Moore, 2008), although typically with little focus on refusal aversion (Morton-Williams, 1993).  In a recent survey of telephone interviewing organizations, Tarnai and Moore (2008) found that between 85 and 90 percent of organizations “always” cover topics in training such as “addressing respondent concerns” and “explanation of refusal behavior and interactions,” but only about half cover tailoring of introductions, and between 50 and 60 percent cover “advanced” techniques such as refusal conversion and “recognition of respondents concerns” (p. 380-1).

In most organizations, interviewers are provided with a list of sampled persons’ frequently asked questions (e.g., Lavrakas, 1987; Lavrakas, 1993; Maynard and Schaeffer, 2002), and refusal aversion training essentially consists of covering those questions (e.g., O’Brien, et al., 2006). In some cases this training consists of providing information to interviewers which they can use to help persuade curious and/or reluctant respondents who may be getting ready to refuse.

It is unknown whether this kind of standard practice is effective relative to no training at all, as empirical evidence is sorely lacking. Durand, Gagnon, Doucet, and Lacourse (2006) implemented a one-hour training session on sampling and effects of nonresponse bias for low performing interviewers and new hires. Although the trained group was not randomly assigned to the training, the limited training improved interviewers’ performance relative to a comparable group that did not receive training.

Specific Refusal Aversion Training.  In contrast to the general question-answer model of survey interviewer refusal aversion training, Groves and Couper (1998) found two main behaviors taken ‘at the doorstep’ by successful interviewers – tailoring and maintaining interaction. These behaviors require interaction over multiple “conversational turns” (i.e., each time the person speaking in a conversation changes), without pushing the interviewer to receive a ‘yes’ response to cooperating. Groves and McGonagle observed “unfortunately, only rarely do interviewer recruits receive training in the multi-turn repartee inherent in maximizing the odds of ‘yes’ over all contacts” (2001, p. 251).  In fact, between about 5 percent (Dahlhamer, Simile, and Taylor, 2006; Dahlhamer, Simile, and Taylor, 2008) to around 50 percent (Groves and Couper, 1998, p. 237) of interactions between sampled households and interviewers involve questions that are consistent with the list of questions provided to interviewers about the study.

In addition, Groves et al. (2004) found that decisions to participate in a survey are often made in less than 30 seconds of being contacted.   In telephone surveys of households in particular, respondents often refuse to participate within the first few seconds (Oksenberg and Cannell, 1988). Therefore, interviewers have to be thoroughly trained and well-practiced on delivering relevant fallback statements quickly.  When an interviewer prolongs the conversation with a respondent, which includes correctly identifying the reason(s) for the respondent’s reluctance to participate, the interviewer can glean as many informational cues as possible to tailor the request for participation.  In addition, the longer the interaction with the respondent, the more difficult it becomes to dismiss the interviewer (Groves and McGonagle, 2001).

In reaction to this observation, Groves and McGonagle developed a training protocol focused on refusal aversion methods.  The protocol consisted of three parts: (1) obtaining a comprehensive list of potential concerns that may be voiced for a particular study, (2) categorizing and developing responses to these concerns, usually in concert with experienced, successful interviewers, and (3) training with repeated practice for interviewers to develop fast, appropriate responses to the types of concerns that may be voiced on the doorstep.

Refusal aversion training is significantly longer and much more cognitively intense than “normal” training.  For example, Mayer and O'Brien (2001) used two full days of training; O'Neill (2007) used two half-day trainings; O'Brien, Mayer, Groves, and O'Neill (2002) used a one day (8 hour) training; Shuttles, Welch, Hoover, and Lavrakas (2002; 2003) used three four-hour sessions delivered over consecutive days with class sizes limited to 10 experienced interviewers, and Cantor, Allen, Schneider, Hagerty-Heller, and Yuan (2004) added two hours of telephone-assisted training to an existing 12 hour study training. Link, Armsby, Hubal, and Guinn (2006) developed an avatar-based computer simulation method for refusal avoidance training, but to our knowledge, never evaluated the effectiveness of this method in the field.  In a survey of telephone interviewing organizations, Tarnai and Moore (2008) found that 19 percent reported “always” using this method of rapid-fire drills to practice addressing respondent concerns.

Does the intensive refusal aversion training as described by Groves and McGonagle work to decrease refusal rates over standard practice?  The answer appears to be a qualified “yes,” depending on the previous success of the interviewer, the length of time from training to measuring cooperation rates, and how training is administered.  Studies have found that cooperation rates increase by five percentage points (Cantor et al., 2004; Karlsson, 2010; Mayer and O'Brien, 2001) to 10 percentage points (Groves and McGonagle, 2001; O'Brien, Mayer, Groves, and O'Neill, 2002; Schultz, 2011) to about 15 percentage points (Groves and McGonagle, 2001; Mayer and O'Brien, 2001) as a result of this training, depending on the specifics of the study. The experiments by Mayer and O’Brien (2001), O’Brien et al. (2002), and Cantor et al. (2004) all used a control group and showed improved cooperation rates due to the training. 
Groves and McGonagle (2001) found that the effect of the training was wholly concentrated among telephone establishment survey interviewers who had the lowest response rates before the training. Mayer and O’Brien found greater gains were achieved the longer the time elapsed after the training.  Cantor et al. (2004) found that success in implementing refusal aversion training through automated telephone systems depended heavily on supervisor feedback after the training. Specifically, a group who had inconsistent supervisor feedback actually experienced lower response rates as a result of the training.  O’Neill (2007) showed mixed results initially, but longer term improvement in response rates for ‘clerks’ in an establishment study.  However, Shuttles, Welch, Hoover, and Lavrakas (2002; 2003) found no positive or negative effects resulting from their refusal aversion training.

Summary
As this section has detailed, there are a wide range of strategies (albeit some that are costly) available to survey researchers who wish to reduce the number of initial refusals encountered in a survey research project.  These include study design features such as the content and implementation of effective survey introductions and fallback/persuasive statements; the strategic offering of multiple modes of participation; pre-notification; incentives; and best practices with regard to contact rules.  Though the results are mixed, refusals can also be avoided through interviewer management, including the use of experienced interviewers, interviewers with certain personality and vocal attributes that have been shown on average to minimize refusals, and structured interviewer training design to avoid refusals.  Unfortunately, even with all of these strategies at our disposal, encountering refusals in survey research is inevitable.  As such, we turn now to strategies to overcome “refusals” that already have occurred into completions, that is, strategies of refusal conversion.

REFUSAL CONVERSION
Definition and Purpose
Refusal conversions are a common and accepted practice in the survey research industry.  While there may be a question as to why researchers would willing ever re-contact a person who has already refused to participate, it is important to note, as we have covered in this report, that the term “refusal” typically includes cases for which there was no verbal “refusing” to cooperate (as noted it is typical for over half of “refusals” on telephones to have simply hung up without a word) and in many other cases a researcher is looking for a person in the household who in fact has yet to be reached in that the initial refusal was made by someone else in the household.  And finally in perhaps the majority of initial refusals, respondents were not fully screened, were not given a full description of the purpose of the phone call, and were not yet assured confidentiality and anonymity. For these reasons, it has long been considered acceptable practice for researchers to attempt to persuade households to reconsider their initial reluctance, and indeed in a consistent plurality of cases in published studies (noted elsewhere), these households do reconsider and participate in the interview.

In addition to wanting to avoid survey refusals in the first place, the previous section also provided some insights into why one might want to conduct refusal conversions.  As with refusal aversions, there are two main motivations, namely, concerns over data quality and as part of an attempt to reduce total survey cost and effort.  Also as with refusal aversions, the principal goal of conversions is to reduce nonresponse rates and potential nonresponse error. 
Why Undertake Refusal Conversion Efforts?

To Reduce Bias.  The effect of conversions on nonresponse error is a function of how many refusals are converted, how this affects the response rate and how different the converted refusals are from respondents who never refused and remaining non-respondents. If converted refusals are similar to respondents on key survey estimates, then the refusal conversion will do little to change the survey estimates or reduce potential nonresponse bias.  If converted refusals are similar to all non-respondents, then refusal conversion has the potential to change survey estimates and reduce potential nonresponse bias. 

In order to determine whether conversion can lead to a reduction in nonresponse bias, researchers must compare the refusers’ characteristics and answers to key survey measures to the never-refused respondents’ characteristics and answers.  Unfortunately, many studies do not have information on unconverted refusers.  Consequently, in the vast majority of the literature, research on potential differences between refusers and respondents focuses on differences between converted refusals and initial cooperators (see “Who Refuses?”).  In this section we primarily consider research that focuses on comparisons with initial cooperators and converted refusals.  Comparison of survey data from converted refusals and initial cooperators can yield meaningful comparisons that give insights into the potential effectiveness of refusal conversion.  Housing characteristics—We start with environmental and housing characteristics, which have been examined by surprisingly few studies. Converted refusals are more likely to live in urban areas than initial cooperators (DeMaio, 1980; McDermott and Tan, 2008; Smith, 1983), but this appears to depend on the country (Billiet, Philippens, Fitzgerald, and Stoop, 2007).  Persons living in multi-unit structures such as apartment buildings are more likely to be initial cooperators than converted refusals (Bates, Dahlhamer, and Singer, 2008; O'Neil, 1979).  Whether an individual is a home owner or renter has shown mixed results, with both renter status (McDermott and Tan, 2008) and homeowner status (Lynn, Clarke, Martin, and Sturgis, 2002; O'Neil, 1979) associated with being a converted refusal.

Sex, Age and Race—Most of the investigations of nonresponse error and refusal conversion focus on the primary demographic characteristics of the individual such as sex, race and age.  The results of these studies have been inconsistent.  For example, the gender of the individual has been found to be significantly associated with refusal conversion status, but not in a consistent direction (Billiet, et al., 2007; Carroll and Chong, 2006; Currivan, 2005; DeMaio, 1980; McDermott and Tan, 2008; Miller and Wedeking, 2006; Peytchev, Baxter, and Carley-Baxter, 2009; Robins, 1963; Fuse and Xie, 2007; Schoenman, et al., 2003; Stoop 2005; Connelly, et al. in Lavrakas, et al., 1992). 
 
Similarly, race and age have shown mixed results. Research has found no difference across racial groups, non-whites to be more likely than whites to be converted refusals, and whites to be more likely to be converted refusals than non-whites (Currivan, 2005; DeMaio, 1980; Lynn, et al., 2002; Smith, 1983; Lavrakas, et al., 1992; O’Neil, 1979; Carroll and Chong, 2006; McDermott and Tan, 2008; Peytchev, et al., 2009; Safir, et al., 2002).  Age has also shown mixed associations with refusal conversion status, with converted refusals found to be older, younger, not different in age, or finding a curvilinear relationship between age and refusal conversion (Billiet, et al., 2007; Carroll and Chong, 2006; DeMaio, 1980; Keeter, et al., 2006; Lynn, et al., 2002; Safir, et al., 2002; Schoenman, et al., 2003; Smith, 1983; Miller and Wedeking, 2004; Peytchev, et al., 2009; Billiet, et al., 2007; Fuse and Xie, 2007; O’Neil, 1979; Lavrakas, et al., 1992; McDermott and Tan, 2008). Thus, for these important demographic characteristics, there is not a consistent single “profile” of a converted refusal across studies.

Socio-Economic Status (SES)—The socioeconomic variables of education, income and employment status have received less attention than age, race and sex. Persons with lower levels of education consistently have been found to be more likely to be a converted refusal than their more educated counterparts (Carroll and Chong, 2006, Connelly, et al. in Lavrakas, et al., 1992; McDermott and Tan, 2008; O’Neil, 1979; Robins, 1963; Currivan, 2005; Miller and Wedeking, 2004; Stoop, 2005; Keeter, et al., 2006).  Findings on income are mixed with studies finding that lower income persons are more likely, less likely, and no difference in the rate of being converted refusals than initial cooperators or that a curvilinear relationship exists between income and refusal conversion (McDermott and Tan, 2008; DeMaio, 1980; O’Neil, 1979; Stoop, 2005; Smith, 1983; Schoenman, et al., 2003; Safir, et al., 2002).  Employment status has received very limited examination, with recent work finding converted refusals being more likely to be employed (Lynn, et al. 2002).  In studies more than three decades old, differences in refusal conversion status across types of occupations have been noted (O’Neil 1979; Robins 1963).  O’Neil for example found that whereby professionals were less likely to be found in conversions, skilled laborer were more likely to be among the converted.
Non-demographics—Of course, most studies are not conducted to measure only demographic characteristics. Furthermore, nonresponse bias on demographic characteristics does not necessarily indicate nonresponse bias on other characteristics (Peytcheva and Groves, 2009).  When health characteristics are examined (e.g., smoking status, BMI, blood pressure, drinking alcohol), converted refusals are either no different from or more likely to have health problems than cooperative respondents (Lynn, et al., 2002; Bates, et al., 2008; Currivan, 2005).  In terms of attitudes, trust in various institutions (political, social, governmental) tends to be lower among converted respondents than among initial cooperators (Billiet, et al., 2007; Keeter, et al., 2006), although this varies by characteristic and country.  There is no consistent relationship between party identification and conversion status (e.g., Smith, 1983; Lavrakas, et al., 1992; Connelly, et al., 1990 in Lavrakas, et al., 1992; Keeter, et al., 2006).  One study found differences in estimates of crime and victimization between converted refusals and cooperative respondents among males (Peytchev, et al., 2009). Other estimates of survey variables of interest are mixed.

Given these differences, does the inclusion of converted refusals change survey estimates?  In almost all instances, the estimates that have been examined are overall means and proportions, not subclass means or measures of association or variability (see O’Neil, 1979 for an exception). Additionally, in the studies we have found, the proportion of converted refusals as part of the respondent pool is quite small, generally less than one-quarter of the total respondents and often a great deal less. With these important limitations on the inference that can be made about the effect of conversion on nonresponse error, the relative change in survey estimates with only the initial cooperators compared to the full sample is generally quite small (e.g., Lavrakas, et al., 1992; Lynn, et al., 2002; Montaquila, et al., 2002).  This does not mean that conversion is not worth the effort in terms of nonresponse error, however.  Although the change in overall estimates of means and proportions from including converted refusals is minimal, this does not indicate what would happen to estimates for subgroups—for whom the share may be found perhaps even by over half within conversion interviews—nor to measures of association.  Overall, additional research is needed to better understand the impact of refusal conversion on reduction of bias in estimates. 

To Reduce Cost and Effort.  The primary rationale to conduct refusal conversions may be to increase response rates and possibly reduce nonresponse bias.  But are refusal conversions also cost-effective?  In other words, are the costs of refusal conversion efforts offset by the benefits of those efforts?  Or, could the resources expended on refusal conversion efforts better serve the researchers’ interest by being allocated for other error-reducing efforts?
As we saw with refusal aversion efforts, the costs associated with refusal conversions can be broken into two categories:  direct costs and indirect costs – and they are the same as those discussed for refusal aversion.

In principle it is certainly possible that interviews from refusal conversions may be less costly to obtain than interviews from “fresh” (previously uncalled) sample or “active” sample (i.e., sample that has been dialed and is in an active disposition such as answering machine, no answer, and callback).  Although initial refusers might not be as likely to participate as respondents in households that have not yet been contacted, initial refusals in telephone surveys are sample records that are highly likely to be actual households. Thus, it is possible for initial refusals to yield more completes per dial than fresh sample in many RDD telephone surveys, where a significant percent of fresh sample will be non-working numbers, businesses, no answers, and initial refusals themselves.  Indeed, in studies in which refusal aversion was particularly poor and thus refusals rates were very high, it might be more likely that reworking refusals via conversion attempts would be more productive than working fresh sample, particularly if the researchers have at their disposal effective interviewers trained specifically in conversion. 

Furthermore, refusal conversion sample is logically more productive compared to fresh or active sample when a screening survey is being conducted and the initial refusals have been screened and identified as eligible.  In a study requiring screening of a rare population (for example populations whose incidence among the general population is 15 percent or less), the dialing of qualified refusals is likely far more productive than the dialing of fresh phone numbers with unknown eligibility status.

Ideally, cost analyses of refusal conversions should consist of an accounting of both the direct and indirect costs associated with these efforts.  Unfortunately, to our knowledge, the cost impact of refusal conversions has not been addressed in the literature, with one exception. Stec and Lavrakas (2007) examined the total costs incurred for completing surveys without any initial refusals to the total costs incurred for completing surveys with respondents who initially refused in a large U.S. national RDD survey.  Comparing these costs on a completed survey basis, the researchers found that the average cost of completing an interview was about three times higher than the average cost of completing a refusal conversion interview.  Their research suggests that, independent of any reduction in nonresponse bias due to refusal conversions, there is justification from a cost perspective to conduct refusal conversions. 

Concerns of Data Quality from Converted Refusals
Refusal conversion efforts may affect the quality of survey data by bringing in respondents who by their very nature to be reluctant survey participants provide poor quality data.  And in fact, some research has in fact explored whether converted initial refusals provide lower quality data even after controlling for demographic and attitudinal or behavioral differences between them and those respondents who never refused.  In particular, converted refusals may be more likely to fail to answer particular questions (i.e., more missing data), to satisfice (e.g., more straight-lining), to give incomplete answers to open-ended items, and not to give as much thought to their answers--which could result in attenuation of the associations between variables—than  respondents that agreed to participate without first refusing.

Missing Data.  Research finds that attempting to convert reluctant respondents clearly increases item nonresponse rates, although this effect varies over the type of item nonresponse, across items, and across survey topics.  In general, item nonresponse rates are higher for converted refusals than for respondents who have never refused, and for respondents who at some point during the recruitment effort stated that they were “not interested” in participating in the survey compared to those who never stated a lack of interest (Olson, 2012). This association holds at aggregate levels as well. In a time series analysis of 20 years of an ongoing telephone survey, Yan, Curtin, and Jans (2010) found a positive association between aggregate refusal conversion rates and aggregate item nonresponse rates to income questions. When converted, respondents who stated that they were “not interested” in the survey also had higher partial interview rates (Dahlhamer, et al., 2006).  However, the relationship between refusal conversion and other indicators of data quality is less clear.

Accuracy and Completeness of Response.  There is limited information about the accuracy and completeness of survey responses and refusal conversion.  In three studies that have records available with which to evaluate response accuracy, there is no evidence of less accurate reports for converted refusals (Olson and Kennedy, 2006; Robins, 1963; Kreuter, Müller and Trappmann, 2010).  However, using a “more is better” heuristic for quality of reports, converted refusals reported fewer activities in a time diary (Triplett et al. 1996) and fewer items and expenditures in an expenditure survey (McDermott and Tan 2008) than respondents who had initially cooperated.  Measures of political knowledge show notable differences between those who are “not interested” on the doorstep versus those who do not say this in political studies (Couper, 1997; Campaneli, Purdon and Sturgis, 1997), but there is no difference between converted refusals and cooperative respondents in science knowledge questions (Yan, Tourangeau and Arens, 2004).

Satisficing.  There is mixed empirical evidence of increased satisficing behaviors among converted refusals.  When refusal conversion indicators are examined independently, there is no consistent statistical difference between converted refusals and cooperative respondents in non-differentiation (Miller and Wedeking, 2006; Yan, et al. 2004), in acquiescence (Yan, et al., 2004), middle or extreme responses (Yan, et al. 2004) or in primacy or regency effects (Blair and Chun, 1992; Jans, 2007).

Attenuation.  There is limited evidence of attenuation of “theoretically expected relationships” for converted refusals (Miller and Wedeking, 2004, p. 8).  When refusal conversion is combined with interviewer assessments of effort put forth by the respondent during the interview, a relationship between reluctance and data quality emerges, but it is difficult to discern how much of the relationship here arises because it is a converted refusal versus the end-of-interview interviewer assessments of lack of effort (Kaminska, McCutcheon, and Billiet, 2010). In general, differences in data quality outcomes such as these for converted refusals versus cooperative respondents need more empirical research before appropriate conclusions can be drawn.

Length of the Interview.  In interviewer-administered surveys, converted refusals tend to have somewhat shorter interviews than cooperative respondents, but this has only been examined in a very few studies.  Specifically, converted refusals have been found to have shorter interviews in the Consumer Expenditure Survey (McDermott and Tan, 2008) and in the American National Election Studies (Miller and Wedeking, 2004), but not in the mixed-mode National Postsecondary Student Aid Study (Cominole, et al., 2008).  In the National Health Interview Survey, persons who initially said that they were “not interested,” “too busy” or had “privacy concerns” had shorter interviews than those who did not make those statements, a finding that held after multivariate controls (Dahlhamer, Simile and Taylor, 2008).[24] 

Refusal Conversion Methods: Survey Design
Researchers face a number of survey design decisions when deciding to convert initial refusals.  These are briefly detailed below.

All or Some?  Many survey research organizations attempt to convert as many soft refusals as possible, requiring no decisions about which of these cases should receive refusal conversion efforts. Researchers should avoiding conversion attempts on hard refusals entirely, as related to ethical consideration and as noted elsewhere in this report.[25]   But organizations can, and possibly should, selectively choose to try to convert some soft refusals and not others for some studies.  Selectively choosing to convert only some refusals might be utilized in studies that require certain sample sizes for various subgroups, or in studies where nonresponse of particular groups is a concern.  It may also be utilized in light of a limited budget where conversion attempts on all initial refusals are too costly. 

An additional consideration is the degree to which, in dual-frame surveys, cell phone refusal conversions should be treated differently than landline refusal conversions.  Since cells are personal devices, it is very unlikely that a person different than the one who initially refused will be reached during a conversion attempt, and typically there is no household level screening on cell phones, though one ordinarily must screen out the significant number of cell owners who are under the age of 18.  Overall, then, there is some argument to be made that refusal conversions on cell phones are more problematic since it is unlikely that the initial respondent refused “for the household.”  That said, our sense as a committee is that refusal conversions are being made on cell phones and that while they may not have the same rate of success as comparable landline conversion attempts, they nevertheless are successful in many cases.

Selected/Qualified Initial Refusals.  Depending on the study design, in households with more than one resident, there is no guarantee that whoever initially refused to participate will be the household member who receives a conversion attempt (again noting that this consideration may not be applicable on cell phones).  Given that a significant percentage of refusals occur before any screening (well over half of all non-“do not call” refusals by telephone, for example, can often occur without the potential respondent uttering anything other than “hello” before the hang up) or before a within-unit selection of a designated respondent has been completed, those who are contacted during a refusal conversion attempt may be a previously un-contacted household member.  In part because of this, some research that has tracked the success rate of conversions in general population surveys have failed to find a relationship between the intensity of refusal and successful conversion (Dutwin and Herrmann, 2006).

Tracking Outcomes.  The basic refusal conversion rate, that is, the percentage of all conversion cases that result in a completed interview or a successful eligibility screening (regardless of whether the case was found to be eligible or not), can be an important statistic in order to evaluate the efficacy of the refusal conversion efforts. 

Typical rates for successful conversion efforts in landline telephone surveys range from about 10 to 30% (e.g., Cominole, et al. 2008; Triplett, 2002; Retzer, Schipani and Cho, 2005; Lavrakas, 2010). In contrast, little is known about the refusal conversion success rate in in-person surveying. Furthermore, the equivalent of a converted refusal in a mail survey or internet survey generally takes place when follow-up contact is made with non-responders and some of those cases proceed to complete and return the questionnaire.  However, experience shows that most mail survey and internet survey researchers do not appear to characterize these cases as “converted refusals.”  Overall, the reporting of the results of conversion attempts and success is uncommon among mail and internet studies.

Based on the AAPOR Standard Definitions, sampled cases that have undergone refusal conversion attempts should be coded with one of three final dispositions – as an interview, not eligible (if found to be ineligible during re-contact), or as a refusal. 

Although not yet addressed in AAPOR’s Standard Definitions (as of 2013) three rates would be informative about a survey’s refusal conversion efforts. In each case, the numerator for the rate would be the number of converted refusals that led to a completed interview.  The respective three denominators would be:
  • The number of cases that were initial refusals minus cases that ended up being a disposition other than a refusal or a completion;
  • The number of initial refusals that were attempted to be converted regardless of whether they were later found to be ineligible; and
  • The total number of completed interviews, including those coming from converted refusals and those that did not.
Multiple Refusal Conversion Attempts.  Although we have no reliable empirical evidence,  anecdotally it appears that it is not uncommon for some survey organizations to make multiple conversion attempts to prior refusals who do not complete an interview.  Simply put, this means that some households refuse, then decline to participate in a later conversion attempt, and then after some time, are contacted yet again to see if they would reconsider.
When this happens, the outcome from each conversion attempt should be recorded in a study’s call records. One open question is how effective each successive conversion attempt is at obtaining a completed interview. As of now, we know of no evidence on this topic.

The practice of multiple conversion attempts also raises the issue of whether a respondent has the right to refuse and not be contacted again.  Is it appropriate for each organization to ask and answer to its own satisfaction what line should be drawn between trying to minimize nonresponse rates and the potential harassment of the respondent?  Not only does this have clear ethical implications that are addressed in the final section of this report, but it also has potentially severe public relations consequences for organizations that do this and for our profession.  Given concerns of rights of respondents, the Task Force does not recommend the practice of multiple conversion attempts.

Refusal Conversion Letters.  When a respondent or household informant has indicated reluctance to participate in a study, many organizations send a follow up letter to the household.  These are often called refusal conversion letters.  The purpose of such a letter is to allay concerns about the study and gain cooperation.  Dillman’s Tailored Design Method (2007) emphasizes the importance of using varying contact approaches.  These persuasion letters can function as an alternate contact mode and approach.  They can also serve to build trust under social exchange theory by legitimizing the survey request (Dillman, 2007).  Such letters may be tailored to specific concerns raised by the respondent, providing these concerns were captured at the time of the refusal (see previous section on refusal report forms).  Responding in this way can be seen to show regard for the respondent, which will be construed by some as a “reward” of sorts.  Under Leverage Saliency Theory, these letters can be viewed as an opportunity to explain the benefits of participation to the individual respondent based on the specific concerns they raised or other data observed during the initial contact (Olson et al., 2011).  In mail surveys, persuasion letters can be sent to households that have not explicitly refused, but have not responded.  Usually these letters include a replacement questionnaire.  For example, Heberlein and Baumgartner (1978) hypothesized that the repeated mailings increase the psychological cost of not responding by increasing guilt or leading the respondent to conclude that it is easier to respond than to keep ignoring the request.  In this context, Dillman (2007) recommends varying the content of the cover letter with each subsequent mailing to be more directed and address respondent feedback that has been received from earlier mailings.

Some research indicates that including an incentive with the refusal conversion letter (Zukerberg et al., 2007) and using priority mail (Brick et al., 2004) can increase the effectiveness of the letter.  However, tailoring the content of the letter was found to be less effective than sending a generic letter that appealed to the respondent’s sense of community (Olson et al., 2011).  More research is needed to determine optimal letter content and timing for refusal conversion letters.

While there is little empirical evidence that such persuasion letters work, in many cases they represent an opportunity to secure participation from a unit that would not be contacted again under normal survey protocols. Additionally, use of refusal conversion incentives typically requires sending a letter to households (although it is becoming more common to send an electronic incentive via email).  For example, Zukerberg et al. (2007) noted that a conversion letter sent to units coded as “final refusal” resulted in an increased response rate of approximately 2 percentage points among those sent only a letter, with a significantly larger response rate for households sent both a letter and an incentive in a mixed mode (mail with telephone nonresponse follow up) study.  More research is needed to determine empirically the effectiveness of letters over other contact approaches. Likewise, more information is needed to determine the impact on nonresponse error by sending the letters.  Furthermore, conversion letters in RDD telephone surveys are particularly complicated when they can only be sent to addresses that have been “matched” with a landline phone number, limiting the scope of their effectiveness.     

Mode.  Miller and Cannell (1982) outlined the differences between face-to-face and telephone interviews.  Social distance between the interviewer and the respondent, use of visual cues, and the perception of the survey request as a sales call are design features that have to be taken into consideration.  Face-to-face and telephone surveys also differ in the techniques that can be applied for refusal conversion.  Face-to-face interviewing offers great opportunities for refusal conversion as the norms of conversation permit multi-turn conversations (Groves and McGonagle, 2001). In contrast, the interaction length in telephone interviews is drastically reduced and the information exchange is restricted (Groves, 1979).  During the past two decades, survey interviewers have begun to be trained to maintain the interaction with the sampled person because longer interactions can provide more cues for tailoring are more difficult to end in a refusal, given the general norms of social interaction (Groves and Couper, 1992).  The nature of the interaction between the respondent and interviewer in telephone surveys has necessitated the addition of a refusal category that rarely exists in face-to-face surveys – i.e., partials or break-offs.
Refusal conversion is a standard practice in telephone surveys because repeat contact attempts to non-respondents are typically inexpensive (Stoop et al, 2010).  Refusal conversion in face-to-face surveys is often more expensive but here as well specific efforts are made to convert refusals, including the use of specially trained refusal converters.  An additional complication in face-to-face surveys is the greater concern of respondent harassment – e.g., in some countries it is illegal to re-approach sampled individuals who explicitly refuse to participate.  The same is true in the U.S. where certain states have research-related harassment laws that apply to telephone calls, such as Hawaii, Missouri, Montana, and Utah.

Refusals in mail surveys are typically not explicit since sampled persons simply do not return the questionnaire.  In addition, unreturned questionnaires do not necessarily reflect refusals – they can be a result of an address mix up, lost mail, or other misdirection of the survey.  Following the Tailored Design Method (Dillman, 2000), typically a week after mailing the initial survey, a thank you postcard is sent to sampled households as a reminder to non-respondents to complete their questionnaire.  About four weeks after the original mailing, a replacement package is sent to non-respondents. This sequencing of the distribution of materials is a common feature in self-administered modes (when used as stand-alone modes or in mixed-mode designs) and has been shown to maximize response rates in mail surveys (see Dillman, 1978; Dillman, 2000).

It is known that mixed mode data collection designs that use a combination of modes to different groups of respondents, different phases in the data collection, or to different parts of the questionnaire (de Leeuw, Hox, & Dillman, 2008; Martin, 2011) may reduce nonresponse rates (Sala & Lynn, 2009).  In fact, some of these surveys offer alternative modes of data collection at the refusal conversion stage.  For example, the Swiss Household Panel, administered initially via telephone (using CATI), offers CAPI and Web modes to initial refusals (Voorpostel and Ryser, 2011) to try to again obtain a converted completion.

Multi-phase and Responsive Designs.  Multi-phase and responsive designs sometimes are used to accomplish a form of “refusal conversion” by attempting to re-contact a sample of refusers (and possibly other types of non-respondents) after the original field period for a survey has been closed.
Typically, multi-phase designs are associated with a different survey protocol in the subsequent phases of data collection.  Multi-phase designs can involve double-sampling (cf. Deming, 1953; Birnbaum and Sirken, 1950), such as when a probability sample of non-respondents is selected after the main data collection has ended and resources are focused on measuring non-respondents in order to study possible nonresponse bias.  Double-sample designs are most successful when tailoring within the main study was not possible.  For example, if mode switches were not feasible during data collection, the double sample can use an alternative mode; thus, this kind of design may disproportionately cover sample persons comfortable only with that alternative mode (Groves and Couper, 1998).  One example of a multi-phase design focused on non-respondents was the Chicago Mind and Body Survey, a face-to-face survey of about 3,100 persons. The second stage of data collection involved subsampling non-final cases based on interviewers’ assessment of the likelihood that a case will become a completed interview under the new design protocol (an increase in the incentive from $60 to $100 and a fixed number of call-back attempts).  At the third stage, the incentive was raised to $150 for all remaining non-respondents and the effort was limited to one contact only.  While not a random sample in its second phase, the Behavioral Risk Factor Surveillance System Survey (BRFSS) has enacted a multi-phase design in some of the states where the survey is fielded.  After phone field period ends CDC researchers invite all initial refusers (except hard refusers) and eligible non-responding households to do the survey by mail, and more recently by web.

A two-phase sampling design for nonresponse is the simplest example of a responsive design (Groves and Heeringa, 2006) that aims at reducing nonresponse bias by adjusting the survey design to make it more appealing to non-respondents.  Responsive designs are growing in practice and range from simple to highly complex implementations.  At the simplest level, studies might collect just a few elements of paradata and use them to inform decisions during nonresponse follow-up, such as which nonresponse cases to target.  At the most complex level, studies may use a number of phases (e.g., a pretest phase, a main interview phase in a given mode, and follow-up of non-respondents in an alternative mode), different cost models, or large numbers of paradata elements to modify continually data collection procedures while in the field. 

Timing.  Once the decision to invest survey resources into conversions is made, an important consideration is what the timing should be for the re-introduction of initial refusal cases back into the sample.  A good deal of research has been done on the optimal scheduling of initial and subsequent attempts to contact a sample member (e.g., Weeks et al., 1980, 1987; Stokes et al., 1990; Odom et al., 2000; Stec et al., 2007).  However, less work has been done on the optimal time to attempt a conversion.

Survey research organizations often use rules of thumb for determining when refusal conversion attempts should begin after the initial refusals.  For example, Foote et al. (2005a, 2005b) described a RDD survey in which the telephone protocol was to wait one week before attempting refusal conversions on initial refusals.  Other researchers described refusal conversion protocols where initial refusals would remain dormant as long as possible before being addressed by a refusal conversion team (Schoenman, 2003).   Still others have conducted unpublished empirical research using very large RDD paradata sets to seek an optimal time to allow passing before attempting a refusal conversion.[26]  As part of these decisions, survey researchers take into account the remaining length of the field period, which tends to have the effect of shortening the wait time between the initial refusal and subsequent conversion attempts.  For example, Triplett et al. (2001) found that it took, on average, almost 5 telephone call attempts to convert an initial refusal.  This could lead researchers to begin conversions more quickly, especially when there are tight time constraints on the survey field period. 

If survey researchers use refusal conversion letters with or without monetary incentives, some amount of time (e.g., seven days) must elapse between the initial refusal and subsequent conversion attempts for these methods to be effective.  For example, Brick et al. (2005) report, since 1999, the National Household Education Survey (NHES) has used follow-up letters sent to households that initially refuse to participate in NHES collections (see also Sloan et al., 2007).  This would tend to increase the waiting period between the initial refusal and subsequent conversion attempts. 

Much of the research on the length of time before beginning refusal conversions has been done in the area of telephone surveys.  Lavrakas et al. (1992) found that refusal conversion attempts done with more call attempts over a longer time period after the original refusal were more successful than refusal conversion attempts done with fewer call attempts over a shorter time period.  While these researchers found that more calling attempts and longer periods of time to make calls were more likely to lead to successful refusal conversions, it was not clear whether the timing of when to start refusal conversion attempts made any difference achieving refusal conversions.

Subsequent research primarily finds that a longer waiting period between the initial refusals and the refusal conversion attempts generally leads to more successful outcomes.  However, the effect may be slight.  For the European Social Survey (ESS), a cross-national survey of attitudes and values conducted face-to-face with respondents in 20 to 30 European countries, Buellens et al. (2010) found that longer elapsed times between the initial refusal and the conversion attempt has a positive effect on refusal rates.

This finding is similar to research on conversion in telephone surveys.  For example, Triplett et al. (2001) examined nine national studies conducted at the University of Maryland’s Survey Research Center from 1995 to 2000.  The researchers found that the conversion rate was worst during the six days immediately following the initial refusal.[27]  After waiting seven days, the success rate for conversions was higher and fairly stable as far out as 18 days after the initial refusals.  After 18 days, the conversion success began to decline until it was no better than the lower success rates at four to six days. 
Edwards et al. (2003, 2004) examined refusal conversion in the 2001 California Health Interview Survey, an RDD telephone survey.  In that work, these researchers conducted a controlled experiment in which initial refusals were randomly assigned to holding period groups of one week, two weeks, and three weeks.  The results indicated that there was essentially no difference among the groups, but with slightly higher success rates occurring with the lengthier holding period groups. 

Incentives.  Although there is extensive literature exploring the use of incentives to avert refusals, there is far less literature that looks at the use of incentives to convert reluctant cases or outright refusals after initial contact attempts have been made.  This may be in part due to factors such as ethical concerns about offering differential amounts to people in a study or rewarding people who are less willing to participate in the research.[28]
Research that has looked at using incentives to convert reluctant respondents has differentiated between explicit refusals, where the respondent has verbally indicated an unwillingness to participate, and cases where no contact has been made with the respondent (e.g. ring no answer or messages left on answering machines in telephone studies).  Incentives have been found to be effective at increasing response rates among households that expressed a reluctance to participate.  However, the results are mixed on the effectiveness of increasing response rates among households that were in a noncontact status prior to the incentive offer (cf. Cantor et al., 2003; Trussell and Lavrakas, 2004; Zukerberg et al., 2007; Colicchia et al., 2011; Brick et al., 2005).

McPhee and Hastedt (2012) compared $5 and $15 incentives with non-responding households and noted that $15 was more effective than $5.  Brick et al. (2005) found no difference between $2 and $5 dollars incentives in refusal conversion.  Similar to the literature on refusal aversion, Kropf et al. (1999) found that the promise of a $5 contribution to charity was not an effective refusal conversion tool.  As well, a few recent studies provide evidence that offering unconditional incentives upfront increases overall response more than offering incentives only to refusal or reluctant cases (Cantor et. al., 2003, McPhee and Hastedt, 2012).  With regard to the impact of refusal incentives on nonresponse, Singer et al. (2000) reported that refusal conversion incentives changed the response distribution for a limited set of items.  Otherwise the literature appears silent on whether incentives used with refusal conversions affect nonresponse bias.

Contact Rules.  Very little publically disseminated research of which we are aware has focused on calling rules and outcomes for calls made after the initial refusal.  Ahmed and Kalsbeek (1998) saw a decline in refusal conversion success rates when the initial refusal is encountered later in the call history.  However, that result could be sensitive to the small sample sizes observed at larger numbers of call attempts.  In a logistic regression model that controlled for the gender of the interviewer, whether scripts were provided for refusal conversions, and whether interviewers had undergone refusal conversion training, these researchers found as the number of call attempts increased and the elapsed time between the initial call attempt and the final call attempt increased, the likelihood of achieving a refusal conversion actually declined. 

Refusal Conversion Methods: Interviewer Management

Interviewer Training:  Telephone and in-person survey organizations may opt to assign a limited cadre of special interviewers when attempting a refusal conversion.  Usually, more experienced interviewers (or those who tend to have lower refusal rates) will be utilized for conversion attempts.  However, some survey organizations believe that all their interviewers should be refusal conversion “specialists,” although there is no reported evidence that all interviewers are equal to the task.  Rather, the limited anecdotal evidence suggests there is considerable variation across interviewers in their abilities to convert refusals.
Basson and Chronister (2006) advise limiting refusal conversion training in telephone surveys to interviewers who have some experience and maintain good response rates.  They recommend small scale conversion training (four interviewers at a time) consisting of one-hour discussions with the project manager and phone room supervisor, after which each interviewer discusses additional conversion techniques and engages in mock interviews with a supervisor.  Trainees listen to an experienced refusal conversion interviewer for about two hours before they can start calling refusals themselves.  Interviewers do not call back their own initial refusals. 

Gwartney (2007) recommends a careful review of contact history records to allow interviewers to come up with a good refusal conversion strategy for each attempted case.  Listening to the audio file of the initial refusal does not seem to help refusal callers, and if anything, decreases the probability of a complete interview and increases the probability of a second refusal compared to not listening to the audio file (Basson and Chronister, 2006).

Demographic Characteristics.  Certain demographic characteristics may be important in gaining respondent cooperation.  There is evidence of higher cooperation when certain attributes, such as when gender (Durrant et al., 2011) or race (Moorman, 1999) between the interviewer and respondent match.  Using a proxy for the education of the person who answers the door, Durrant, et al. (2011) found lower refusal rates when the interviewer and the respondent both had lower education, or both had college degrees.  Additional characteristics, such as ethnicity or age can also be matched to achieve higher cooperation rates (Durrant et al., 2009).  Some members of this task force believe that it is best to match the demographic characteristics of interviewers to refusing cases, but no valid empirical evidence has been reported to support this hypothesis.

Attitudes.  As reviewed in the refusal aversion section, interviewer self-confidence and belief in the legitimacy of their work have been found to be correlated with success in gaining cooperation (Groves and Couper, 1998; Hox and de Leeuw, 2002; Mierzwa et al, 2002; Singer, Frankel and Glassman, 1983; Durant et a., 2010; Groves and Fultz, 1985; Hansen, 2007).  Expectation about the ease of persuading respondents has been found to have an effect on response rates (Singer, Frankel and Glassman, 1983).   These correlations may well hold for success in refusal conversion, but to our knowledge there is no evidence as yet reported about this.

Of additional note, experience has shown that certain interviewers report that they actually “like” the challenge of converting refusals.   But there is no empirical evidence that shows whether this is in fact true and, if so, whether it is related to the ability of interviewers to succeed in converting refusals.  Logic dictates, however, that interviewers who actively dislike trying to convert refusals should not be assigned to this task.

Experience.  Experience (or interviewer tenure) is a well-known correlate with respondent cooperation.  More experienced interviewers (or higher pay-grade interviewers) generally produce higher response rates (Campanelli, Sturgis and Purdon 1997; Couper and Groves 1992; Durbin and Stuart 1951; Link 2006; Hox and de Leeuw 2002; Groves and Couper 1998; Pickery and Loosveldt 2002; Durrant, et al. 2010; Lipps and Pollien 2011; Sinibaldi, et al. 2009; Hughes, et al. 2003, but see Blohm, Hox and Koch 2006).  Several researchers suggest that successful interviewers focus on respondents’ voiced concerns (Morton-Williams, 1993) and tailor their behavior to individual respondents (Groves and Couper, 1998).  Interviewer experience has also been related to strategies interviewers employ to address respondents’ concerns.  For example, Snijkers, Hox and de Leeuw (1999) found that more experienced (and more successful) interviewers rely more on tailoring.

The degree of interaction between the interviewer and the sample member has been identified as an important factor in gaining initial cooperation (Groves, 1979; Groves et al., 1992; Groves and McGonagale, 2001; Miller and Cannell, 1982).  This seems to be the case at the refusal conversion stage as well, suggesting that an interviewing approach that focuses on maintaining the interaction with the respondent (Mishler, 1986) as opposed to the traditional standardized interview, is more effective in successfully converting refusals (Fuse and Xie, 2007).

Interviewer Voice Parameters.  Various voice characteristics such as pitch, pitch variation and vocal loudness, have also been associated with success in gaining respondent cooperation, as reviewed earlier (e.g., van der Vaart, et al., 2005; Oksenberg, Coleman and Cannell, 1984; Oksenberg and Cannell, 1988; Groves and Benski, 2006; Groves et al., 2008).  To our knowledge, however, there are no studies that investigate the relationship between interviewer voice characteristics and conversion success.

Interviewer Compensation. In some survey organizations interviewers who work on conversions earn additional compensation for their efforts. In some other organizations, being qualified to work on conversions is part of the requirements for promotion to a higher grade of interviewer (which includes a raise in hourly wage). Lavrakas (1987; 1993) recommended that interviewers earn a stipend to their hourly base wage for each success they achieve, including each time they convert a refusal.  However, to our knowledge, the effect of this has never been tested in an un-confounded controlled experiment.
 
RIGHTS OF RESPONDENTS
Although we recognize the value of attempting “refusal” conversions with initially reluctant respondents in order to reduce the potential risk of nonresponse error, the Task Force is also mindful of the ethical responsibility to respect the rights of sampled respondents who do not want to participate in voluntary surveys.  To that end, this section discusses ethical, legal, and regulatory issues primarily in the United States involving refusal conversions and other survey-refusal-related considerations.  These issues vary in other countries and it was beyond the scope of this Task Force to make what is written here a comprehensive international review. The Task Force does acknowledge that compared to the U.S., in some countries these matters are more highly articulated and researchers are more restricted in what they are permitted to do concerning survey refusals, and in some other countries the exact opposite is the case.

We believe this section also is of value related to overcoming the lack of knowledge (misunderstanding) of many sampled respondents about their “rights” when sampled for legitimate research purposes. We also believe this section will help many Institutional Review Board (IRB) members better understand the complex issues that underlie the motivations of survey researchers to minimize refusals and thereby achieve higher response rates, with the ultimate goal of achieving higher quality data for the purposes their surveys are meant to serve – many of which are directly related to raising the quality of life for the general public.

Refusal Conversions in the Context of Human Subject Protections
Federal regulations which are relevant to survey research and refusal conversion, as well as elements of AAPOR’s Code of Professional Ethics and Practices, emphasize the voluntary nature of most survey participation, and the need to respect and protect respondent autonomy, privacy, and confidentiality. For example, the AAPOR Code of Professional Ethics and Practices notes that ethical conduct of survey research requires (a) voluntary participation (except where specified by law); (b) avoidance of practices or methods that can harm or seriously mislead potential respondents about intent or content; and (c) protection of respondent privacy and confidentiality.  Federal regulations govern whether IRB oversight is required, but interpretation of the regulations by individual IRBs leads to variability in the application of those requirements.

U.S. Federal Regulatory Environment.  Federal regulations pertaining to survey research fall under the aegis of Human Subjects Protections established by Title 45 Code of Federal Regulations Part 46, subpart A. These regulations, established in 1991, were an expansion of existing regulations and grew out of a report published in 1978 by the U.S. Department of Health and Human Services (HHS) entitled Ethical Principles and Guidelines for the Protection of Human Subjects of Research,  known as the Belmont Report. In the early 1990s, U.S. federal departments and agencies adopted subpart A of 45 CFR part 46 as a common Federal Policy for the Protection of Human Subjects, known as the “Common Rule.” 

Common Rule regulations apply to all research funded by HHS and to institutions that have agreed to follow Common Rule regulations regardless of funding source.  Common Rule regulations require researchers to obtain prior review of projects involving human subjects by an Institutional IRB (45 CFR 46.103) and for the IRB to monitor the ethical conduct of that research over the life of the project. An IRB has authority to terminate approval of a project due to non-compliance, or if the level of anticipated risk increases (45 CFR 46.113).   IRBs use the Belmont Report to guide their assessments, focusing on its three basic ethical principles for research:
  1. Respect for Persons (protect individual autonomy, particularly vulnerable populations with “diminished autonomy”),
  2. Beneficence (minimize risk in relation to benefit), and
  3. Justice (fairly distribute benefits and burdens of research).
Research that is Exempt under U.S. Federal Regulations. There are six categories of research that are considered “exempt” from these regulations, with the 2nd category (reproduced below) being the most applicable to general survey research:

Research involving the use of educational tests (cognitive, diagnostic, aptitude, achievement), survey procedures, interview procedures or observation of public behavior, unless: (i) information obtained is recorded in such manner that human subjects can be identified, directly or through identifiers linked to the subjects; and (ii) any disclosure of the human subjects’ responses outside the research could reasonably place the subjects at risk of criminal or civil liability or be damaging to the subjects’ financial standing, employability, or reputation. (Underlining Added)
45 CFR 46.101(b)(2)

The first part of this exemption applies to survey, interview, and observational research that does not include identifiable information in the response data file.  Identifiers include codes (such as Case ID numbers assigned by a CATI system) if they can be linked to the data via the sample file, contact list or any other “crosswalk file” that associates individual identifiers with response data.  Examples of data collection files that would not be considered exempt under 45 CFG 46.101(b)(2)i are: a voter list augmented with a response ID (it is a crosswalk file that links identifiable information with responses), or a response data file that includes the respondent’s telephone number.  In general, identifiers include phone numbers, addresses, and social security numbers, as well as other information that can directly link responses to individuals. In some cases information such as zip code, facility name, or work role is also an identifier.  What constitutes identifiable information varies by size and type of population and, as alluded to in part ii of 45 CFG 46.101(b)(2), level of risk.  Individual health information, for example, is protected by stringent HIPAA regulations, and a data file that includes any of the 18 identifiers listed under those regulations is considered identifiable data.

Variability in IRB Interpretation of U.S. Federal Regulations.  According to the HHS’ Office for Human Research Protection’s (OHRP’s) Institutional Review Board Guidebook, survey and interview research involving children and other vulnerable populations is never exempt, but rather requires full IRB review [Federal Policy §___.101(b)(2); 45 CFR 401(b)].   It also indicates that survey and interview research with adults is exempt from the federal regulations - unless the information obtained is recorded in such a manner that the subjects can be identified, and the information obtained could reasonably place the subjects at risk of criminal or civil liability or be damaging to the subjects' financial standing, employability, or reputation [Federal Policy §___.101(b)(2)]. However, the Guidebook falls short of advocating an actual position on the exempt nature of most survey research, ending the paragraph by saying:  “Furthermore, some IRBs review all research involving human subjects, even where the research is exempt under the federal regulations.”
Some IRBs may place restrictions on the number of attempts used to reach participants, and in the absence of standardized guidance on the number of attempts which constitute harassment, may do so in a way that limits the ability of researchers to locate busy or harder to reach individuals.

There is a high degree of variability in whether IRBs (and other relevant officials) will classify an individual project as non-research, exempt, or needing IRB review (Lindenauer et al., 2002). As well, IRBs have a wide variation of concerns about the use of incentives (Brown et al., 2006; VanderWalde, 2005) – including their use to avert or convert refusals – which mainly center on the ethical principles of fairness, and avoidance of coercion. Refusal conversion can also be a concern for IRB staff and members. For example, an IRB or even an individual IRB member may consider contrary to the prescribed voluntary nature of participation, to try to re-contact (within the same study) someone who has previously “refused” to participate. However, as discussed earlier in the report, our use of the term refusal, especially in the context of conversions, broadly covers ambiguous situations of non-participation.

U.S. Federal Communications Commission (FCC) Rulings & State Laws.  Survey and marketing research telephone contact with the public is regulated through various acts under the Federal Communications Commission (FCC). The Telephone Consumer Protection Act of 1991 (TCPA) (47 U.S.C. § 227) is concerned mainly with privacy and it governs the ways in which telemarketers can contact consumers. While most rules apply only to market research and sales calls (e.g. establishment of a do-not-call-list, restrictions on times of day), some of the rules governing cell phone contact (e.g. requirement for hand-dialing if there is not an established prior relationship), and “robo-calling” apply to all types of survey research, including scholarly research and public opinion polling.

The AAPOR Cell Phone Task Force Report section on legal and ethical issues in RDD cell phone surveys includes a summary of FCC regulations and best practices for cell phone surveys (p.77), as well as a summary of state laws that address the legal and ethical considerations of harassment for the number of times a survey organization calls a household (for cell and landline phone surveys):

There are various state level harassment laws in the U.S. that need to be considered when determining the placing of callbacks to a cell phone respondent. For example, under current Utah law, it is illegal for anyone to cause a telephone to ring “repeatedly” or “continuously” (the law is not more specific). Under Missouri law, it is considered harassment for anyone to make "repeated" telephone calls; (in one case brought under the law, four call attempts to an answering machine were sufficient to constitute harassment). In Hawaii, it is illegal to “repeatedly” make a communication anonymously or at an extremely inconvenient hour; and in Montana one cannot use a telephone to disturb, by repeated telephone calls, the peace, quiet or right of privacy of a person. Although a matter of interpretation, multiple callback attempts to a respondent runs the risk of violating any one of these state laws.
With the advent of Caller ID, even though a cell phone respondent may not hear the phone ring all of the times a survey organization calls, the respondent often will have a record of how many times calls from a given number have been made to her/his cell phone. Thus, in addition to being a possible ethical violation of what many might construe as “harassment,” and regardless of whether it also is a legal violation, those planning to conduct cell phone surveys need to think carefully about how multiple callbacks may affect their final response rates if they alienate cell phone owners throughout the survey field period with “too many” (e.g., more than 10) and/or “too frequent” (e.g. several calls within a 24-hour period) callbacks. -- 2010 AAPOR Cell Phone Task Force Report

Several more laws have been enacted since that Task Force report was released. For example, New Hampshire’s statute Section 664:16-a r  restricts push polling, defined as calling “voters on behalf of, in support of, or in opposition to any candidate for public office and asking questions relative to opposing candidates which state, imply, or convey information about the candidate’s character, status, or political stance or record.” As defined, the statute also covers campaign message testing which is generally distinguished from push polling. However, without a clear distinction, campaigns are now required to identify the campaign in question up front, which can bias a legitimate survey.[29]

Associational Positions on Refusals

A review of a sampling of professional associations’ codes of conduct and/or best practices statements indicates that these documents discuss the importance of responsibility to respondents or respondent rights.  However, they are typically referring to respondent confidentiality, not refusal-related rights.  For example, the ICC/ESOMAR International Code on Market and Survey Research discusses privacy rights and the responsibility of researchers to behave ethically, but nowhere in the 36-page document are the words “refuse” or “refusal.”  The Marketing Research Association’s Code of Marketing Research Standards states that members “will respect the respondent’s right to withdraw or to refuse to cooperate at any stage of the study…” but does not mention refusal conversion.  CASRO’s Code of Standards and Ethics for Survey Research states that: “Survey Research Organizations must respect the right of individuals to refuse to be interviewed or to terminate an interview in progress” and also that SROs can make “…reasonable efforts to obtain an interview including…. re-contacting an individual at a different time if the individual is unwilling or unable to participate during the initial contact.” Finally, AAPOR’s Best Practices for Survey Research refers to “…follow-up calls to those who do not respond initially” and to “…special efforts (i.e. reworking refusals with an experienced interviewer)…made to persuade persons who are inclined not to participate to respond.” 
 
IRB Considerations Related to Refusals

As noted previously, IRBs have wide latitude in how they interpret the Common Rule when reviewing survey research, and survey researchers may be subject to misapplication of regulations if an IRB has members who are not knowledgeable about survey methods.  As mentioned, some IRBs may have concerns with the number of call attempts made and refusal conversion in general, and anecdotal information suggests others may not even be aware that multiple call attempts and conversion attempts are made as a routine part of legitimate survey research.  Part of the reason for this may be that the survey research profession has not provided IRBs with much guidance regarding refusal conversion practices. While it is our intent that the final report of this task force will provide guidance to survey and market researchers about refusal conversion practices, it is likely that without proactive efforts to educate them most IRBs will be unaware of it. Thus the task force recommends that at minimum, survey and market researchers make an effort to educate the IRBs that review their projects about the importance of multiple attempts and refusal conversion efforts when those are prudently implemented so as not to conflict with research ethics.  IRBs should be made aware of standard survey research methodology that involves multiple call attempts and conversion attempts, and why these are important to preserving data quality.  Survey researchers should explain to their IRBs that the use of “soft” refusal conversions are not intended as harassment, and rather that they increase the likelihood that respondents will be provided with an opportunity to consider fully and make an informed participation decision. (A similar case may be made for including multiple attempts to contact respondents in the case of non-contacts.) These standard, evidence-based survey methodologies increase the scientific validity of the findings and enhance the fairness of the study approach by ensuring that participants are provided with the information they need to make their decision to participate or not. The Task Force believes that limiting re-contact to “soft” refusals and gatekeeper refusals does not increase the risks that respondents will feel coerced to participate. 

Researchers can make the case to an IRB that by following survey best practices, they are behaving ethically and responsibly to protect the rights of respondents. Best practices include required training in human subjects’ protection for all key personnel, including interviewers[30].  Risks in survey research are generally limited to inadvertent harm that may arise from accidental breaches of privacy and confidentiality through human error. As such we need to make sure that IRBs are aware of the training our interviewers undergo to recognize and properly dispose of “hard” refusals which will not be contacted again, and how we handle those that ask to be “taken off the list.” The next section provides more detail about final refusals and internal DNC lists.

Should Certain Initial Refusals Be Considered “Final”?

As discussed in the first section of this report, most survey firms classify refusals as being either a “hard” refusal, which should not be contacted again, or a “soft” refusal, which may be re-contacted at a later date.  While this distinction is not necessary in calculating final response rates, it is a very useful internal (i.e., operational) distinction made by survey organizations in order to schedule their work load, decide on the size of future sample releases, and obtain an estimate of a project’s final response rate while the project is still in the field.  Although researchers have attempted to develop objective, operational definitions of what constitutes a “hard” vs. “soft” refusal, reliance on a trained interviewer’s subjective assessments will always remain an important element of the process by which these various types of refusals are differentiated.  Organizational definitions are also likely to vary considerably.  While it is inevitable that there is variance in how “hard” vs. “soft” refusals are distinguished, both between interviewers within organizations, as well as across organizations, we recommend that organizations err on the side of caution and strive to foster consistency across interviewers within the organization.   Overall, the most important guiding criteria is to consider the rights of respondents when deciding when to consider an initial refusal as being final and therefore not subject to re-contact.  

If efforts to convert refusals increase the probability of participation among those who have a lower propensity to respond to a survey, then researchers have some rationale to avoid “finalizing” too many initial refusals.  Following this logic, a “hard” refusal would be a refusal that should not be re-contacted because the propensity of obtaining a response is close to zero and any re-contact efforts are highly unlikely to increase the propensity to respond to the survey. In cases where persons are reportedly angry or hostile toward the interviewer and those who specifically ask not to be re-contacted or who ask to be taken off the list, it is straightforward to classify these instances as hard refusals. The challenge, of course, is how to know when there is little or no chance of obtaining an interview following an initial refusal when there are no clear-cut indications (implicit or explicit) that it is a hard refusal.  But from an ethical standpoint, the Task Force believes that in the case of ambiguity it is better for survey organizations to err on the side of not trying to re-contact a refuse than to take the chance of upsetting the refuser by trying to convert her/him.

Beyond the more adamant refusals described above, determining whether other initial refusals should also be considered final requires more consideration.   The relevant context is whether the person refused to participate (including an immediate hang-up or door-slam) before an interviewer had the opportunity to explain the purpose of the study or determine that the person who refused would be the designated respondent.  If the person who refuses was told the purpose of the study and is the designated respondent then it is unlikely that s/he would agree to participate upon a re-contact. Therefore, it may be ethically appropriate to finalize these types of initial refusals.[31]   However, it may be justifiable to re-contact a person, if the person hasn’t been fully informed regarding the purpose of the study or may not be the target respondent.  The final point is that for self-administered surveys, such as mail or web surveys, potential respondents rarely indicate they do not want to respond because there is no direct interaction with a live interviewer.[32]  Thus, the ethics of sending a follow-up questionnaire in the mail (or a follow-up email reminder to non-responders (most of whom can be assumed to be implicit refusers) is essentially never considered to be an ethical issue. 

Rights of Respondents with Respect to Internal Do Not Call Lists

On October 1, 2003, the U.S. Federal Trade Commission started the National Do Not Call (DNC) Registry based on the Telemarketing Sales Rule.  Telemarketers are forbidden to dial numbers on this list, and there are now in excess of 50 million unique telephone numbers on the DNC Registry.  Only telemarketing companies and organizations that solicit sales of goods or services are banned from dialing these numbers.  Survey research firms are still permitted to call numbers on the DNC Registry.  Research thus far finds an inconclusive effect of the establishment of the DNC Registry on response rates (Link, Mokdad, Kulp, and Hyon, 2006), although Lavrakas (2004) reported that those whose sampled number was on the DNC were significantly more likely to complete an interview in a very large national RDD landline study conducted in November 2003, compared to those whose sampled number was not on the list. 

Even though the law only limits solicitations, respondents may not perceive a difference between a telemarketing call and a request to conduct a survey.  It may be that for some respondents, any unsolicited telephone call is an equally unwelcome invasion of their privacy and they confuse the DNC list with also preventing these types of calls.  As such, the Marketing Research Association strongly urges each survey research organization to establish and use an internal Do Not Call List.  As stated by the MRA (2012): “Establishing, communicating and maintaining voluntary internal do not contact lists is an essential aspect of respondent cooperation.  In addition to the benefit to respondent cooperation, such opt-out capabilities will assist in assuaging many state attorneys general who are beginning to expect them from any business/organization, sales-related or not.” 

It is unclear to what degree survey research and market research firms maintain their own DNC lists.  However it is important to protect the rights of respondents and the Task Force believes that every survey firm should maintain such lists and consider the following:
  1. What should be put on the list? While a DNC list is meant to list persons who do not wish to be “called,” the overarching goal of such company-based internal lists are to delineate persons or households that do not wish to be bothered again by a request to conduct a survey by a specific survey research firm.  As such, persons or households who request never to be contacted again in a face-to-face setting (e.g., an interviewer on the doorstep), or via internet (an e-mailed request) or mail (a letter) should be added to the list.                                                                                                                                                                                                                                                      However, this raises the question what exactly should be on the list, since for none of the three examples above would the survey research firm necessarily know the telephone number of the person or household.  A firm can consider including not just phone numbers but any information available for each household requesting to be placed on the list, including possibly name, address, and e-mail.  There are, however, a number of issues with the creation of such a detailed list.  Perhaps most importantly, placing personal, identifying information on a list outside the purview of a specific study may violate the confidentiality agreements for that specific study.  Secondly, the maintenance of a comprehensive list necessarily requires much more labor than a basic phone list, as it would require survey firms to pull multiple fields of information and merge them into one master list.  An alternative would be mode-specific lists, such as an e-mail DNC (as in Do Not Contact) list for any person or household that has asked for such via e-mail, a phone list for phone exclusion requests, etc.  In many cases requests to be placed on a DNC list may be almost exclusively received via telephone. In that case, maintaining a company-wide telephone list will cover the vast majority of people requesting to be placed on a do not call list. If no project-specific information is collected, such a list is not likely to violate confidentiality agreements for most studies. 
  2. Should the request be specific to the person who refused or the entire household?   It can be argued that a refusal is a personal act, and so a request to be placed on a do not call list would involve only the rights of the person making the request.  However, in practice it is difficult, if not operationally impossible, to place an individual on an internal do not call list but still allow other members of the household respondents to be eligible to reply to surveys, particularly if they share a telephone.  Without data on who among the household members are on the list and who is not, there is no way to maintain DNC records at the level of the individual for telephone surveys at the household level.
  3. Should survey organizations maintain study-specific lists or organization-wide lists? It is unlikely (although not out of the question) that a respondent who is requesting to be placed on a DNC list is anticipating that such a list would only apply to a single study. It is more likely the respondent believes that placement on the DNC list would apply to all studies conducted by a research organization.  Yet, for larger organizations with multiple divisions and projects, it may not be feasible to maintain an organization-wide list.  It is important however to continue to consider the confidentiality requirements of each study.
  4. How long will a number remain on the list? Research on the mobility of landline telephone numbers suggests that as much as 15 percent of numbers “change hands” every year (MSG correspondence, 2012).  There is no reliable estimate of turnover for cell phone numbers, however. Given phone number turnover, it is certainly possible that an individual might change telephone numbers several times over a period of time. This creates the difficulty that phone numbers originally placed on a DNC list may be reassigned to other potential respondents that have not asked for the numbers to be included there. 
  5. How soon will the respondent be placed on the internal DNC list?  Federal regulations for telemarketers require that lists be updated at least every 30 days.  Research organizations might also consider similar practices for their internal DNC lists.
  6. Should internal DNC lists cover not just the survey firm but the clients and organizations of the principal investigators as well?  The question here is whether it is the responsibility of survey research firms to provide DNC lists compiled from other studies to the survey sponsors so that they can relay such lists to survey research firms who are perhaps conducting other unrelated studies.  One must be careful with such a practice to not violate confidentiality agreements for any specific study.
  7.  How should numbers on the DNC list be treated for response rate calculations if they   are selected in a sample?   It may not always be appropriate to count members of a DNC list who are sampled to participate in a survey as refusals when calculating the final response rate. However, and in line with the spirit of AAPOR’s Standard Definitions guidelines, in surveys of the general population, one should consider these cases as valid households who have refused to participate, but not necessarily 100 percent of them for reasons such as number turnover.                                                                                                                                                                                                                                                                                                                                                                                                       However there is a problem with this approach in screening studies where some respondents who requested to be placed on a DNC list may not be eligible for that survey.  As such survey organizations may consider DNC refusals in screening studies in the same manner as other refusals before a necessary screener, that is, a “no screener completed” (AAPOR’s 3.21 disposition code).

In conclusion, with rare exceptions such as the decennial Census and the American Community Survey, participation in a survey is a voluntary act and we as a profession have an ethical responsibility to respect the rights of potential respondents to refuse to participate, especially after they have been fully informed about what the survey organization is asking of them. Aside from legal obligations such as harassment laws, ethical principles such as those laid out in the Belmont Report must guide our decisions about whether to re-contact a reluctant or refusing respondent. At a minimum, survey organizations should adopt clear guidelines for training interviewers as to what constitutes a “hard” refusal, and these cases should never be re-contacted for conversion attempts. The Task Force also recommends that refusal conversion of soft refusals should be attempted only by veteran interviewers who have been trained in conversion practices. Survey organizations also should carefully consider the ethical implications before engaging in some of the practices discussed in this report (such as offering differential incentives to research participants being asked to do the same task) and recognize the IRBs may have good reasons to be uncomfortable approving such practices.  Finally, researchers should be cognizant that IRBs, in principle, are their “partners” in human subjects’ protection and that IRB members need to be educated about the methods that survey researchers use and why these methods are used. As a reminder to the reader, AAPOR has guidance available to educate IRBs about survey research and survey researchers about IRBs and federal regulations regarding human subjects’ research. (http://www.aapor.org/Full_AAPOR_IRB_Statement1.htm).

 
RECOMMENDATIONS
Operational Considerations for Survey Researchers
From the Section “What is a Refusal?”
  1. Summarizing the above discussion, in most surveys there are two ways that survey refusals are operationalized: the interim, in-the-field disposition of a given contact attempt with a case as a refusal (or not), and the final, end-of-field accounting of a case as a refusal (or not) in order to publish final dispositions and calculate a survey’s response rates.  These distinctions are important because there are instances where an individual may have initially declined to participate, but the case should not be categorized as a refusal for purposes of calculating the response rate. There also are instances where a person might not explicitly refuse to be interviewed, but nevertheless might be counted as a final refusal. These distinctions are important to note because, during the field period, survey researchers must decide what to do about temporary refusals (e.g., whether to try a refusal conversion, and how and when should that conversion take place) and, at the end of the field period, determine a final disposition (as a refusal or another disposition). The discussion on a generic Refusal Report Form provides useful suggestions for developing paradata to facilitate both types of decisions.
  2. In surveys with no eligibility criteria, temporary refusals that are never converted should have a final disposition of refusal.
  3. In surveys with eligibility criteria, final disposition categories should distinguish cases where eligibility has been determined from cases where eligibility is unknown (i.e., the latter should not be treated as a final refusal).
From the Section “Refusal Aversion”
  1. Survey design should, whenever possible, tailor introductory letters, interviewer scripts, and other materials describing the identity of the interviewer, the research organization, explanation of the research and its significance, and assurances of confidentiality. Recent research indicates that, compared with highly structured introductions, lower refusal rates are obtained when interviewers can tailor the survey introduction to present information relevant to the sampled person’s specific concerns and questions.
  2. In interviewer administered surveys, it is recommended that interviewer training emphasize refusal aversion, including techniques to maintain interaction with the sampled person and gain informational cues to tailor the survey request effectively.
  3. While available empirical research on impact is mixed, survey researchers should, where possible, incorporate the following design features to avert and thereby minimize initial refusals:
    • Survey Introductions and fallback statements
    • Offer multiple modes of participation
    • Pre-notification contact
    • Leave-behind material describing the survey (for example, study brochures, answering machine messages, e-mail)
    • Incentives, but consider:
      • Timing of the incentive (e.g., in advance letter, first contact, post-survey)
      • Amount of incentive
      • Form of incentive (for example, denomination of bills if a cash incentive)
      • Fixed or differential incentives
    • Optimize contact rules to make attempts when sampled persons are most likely to be available and responsive to the survey request.
    • Interviewer selection, training, and management.
From the Section “Refusal Conversion”
  1. While research findings are mixed, researchers should consider concerns about the quality of data obtained from converted refusals:
    1. Item nonresponse
    2. Accuracy and completeness of responses
    3. Satisficing
    4. Attenuation
    5. Length of completed interview
  2. Though very little empirical data on impact exist, operational considerations for refusal conversion tactics should include:
    • All soft refusals or targeted to certain types of soft refusals
    • Track outcomes of conversion attempts
    • Avoiding the practice of multiple conversion attempts Refusal conversion letters and possible incentives
    • Mode differences
    • Multi-phase designs
    • Responsive design
    • Contact rules for conversion, including timing of conversion attempts
    • Interviewer selection, training, and management
From the Section “Rights of Respondents”
  1. The task force recommends that survey and market researchers make an effort to educate members of Internal Review Boards (IRBs) about the importance of high response rates and the effectiveness of refusal aversion and conversion attempts in achieving high response rates.
    1. Standard survey research methodology involves multiple call attempts and conversion attempts for soft refusals.
    2. Soft refusal conversions are not intended to be harassment and increase the likelihood that potential respondents will be provided an opportunity to consider participation fully and make an informed decision.
    3. By following survey best practices, survey researchers are behaving ethically and responsibly to protect the rights of study members. Best practices include:
      1. Required training in human subjects’ protections for the research staff, including interviewers
      2. Security protections for personally identifiable information (PII) and private health information (PHI).
  2. Survey organizations should maintain “Do Not Call” lists on a project-specific or organization-wide basis. When doing so, key questions to be answered include:
    1. What contact information should be placed on the list?
    2. Should a DNC request be specific to a household or individual?
    3. Is an organization-wide DNC list feasible?
    4. How long does the information remain on the list?
    5. How quickly will the information be added to the list?
    6. In a business survey, is the entire business placed on the list or only the specific unit?
    7. Should DNC lists be conveyed to clients of the survey organization?
    8. How should entries on the DNC list be treated for response rate calculation?
 
Recommendations for Further Research
From the Section “What is a Refusal?”
  1. Hang-ups without comments (early hang-ups, during introduction). We suggest further research into outcomes of cases that have an initial or early disposition of hang-up without comment.
    1. What are the final dispositions of these cases as compared with other early dispositions?
    2. What resources do these cases require for final disposition?
    3. Are there strategies for follow-up that are more or less successful in terms of outcomes (for example, varying the times of follow-up calls, sending conversion letters, assignment to more experienced interviewers)?
    4. Is there a point of diminishing returns for additional attempts?
    5. Can we gain a better understanding of the percent and degree to which voice mail screening is, in the minds of respondents, an actual avoidance of soft refusal to conduct an interview?
  2. Distinguishing “hard” and “soft” refusals. Refusal conversions should not be attempted on hard refusals. 
  3. We suggest that more research is needed concerning the applications and benefits of the Refusal Report Form (RRF). For example, what types of information about initial contacts should be recorded to inform subsequent attempts?
From the Section “Who Refuses?”
  1. Continue research into differences between refusers and responders with respect to potential nonresponse bias. As noted in the section on “Who Refuses,” to the extent that refusers have different attitudes, opinions, or behaviors than participants in surveys, their absence from analysis of survey can bias results.
  2. Another reason for continued study of refusers is to develop a better understanding of factors related to refusal that may form the basis for improved techniques in refusal aversion and conversion. For example, the section distinguishes two set of reasons for refusal—a set related to participation in surveys in general and a set related to specific features of a particular survey. Developing techniques to classify initial refusals as one or the other set could lead to improvements in follow-up approaches and materials.
From the Sections on “Refusal Aversion” and “Refusal Conversion”
  1. The section on Refusal Aversion calls for a systematic review or meta-analysis of refusals and nonresponse bias across simultaneous and sequential mixed mode surveys to disentangle noncontact and refusal as causes of nonresponse and discern more clearly the effects of each on bias.
  2. More research is necessary concerning the context and content of initial and follow-up contacts, including:
    1. Effect of advance letters and other survey materials (e.g., brochures or “leave behind” material) on refusal rates.
    2. Effect of messages left on voice mail and answering machines—research suggests that messages left in voice-mail on landline phones have no effect on refusal, but may be different for cell phones.
    3. Effect of fixed versus differential incentives and survey refusal rates (based on response propensity).
    4. Prior research suggests efficiencies in optimizing calling rules or patterns to reach individuals when they are most likely to available and willing to cooperate. Published reports have tended to focus on minimizing overall nonresponse (that is, from both noncontacts and refusals). Extending this literature to concentrate on minimizing refusals might reveal contact patterns that are especially effective in aversion and conversion.
    5. Efficacy of refusal conversion letters, that is, identify elements that are effective and how to match the content of these letters to an individual’s reasons for refusal.
  3. Prior research indicates that interviewer characteristics and behaviors can affect cooperation of sampled persons and also that training programs can be effective in training interviewers to avoid refusals. Research in this area should continue in order to enable more effective and efficient techniques for interviewer selection, training, and management.
  4. Much of the research summarized above concerns techniques to minimize refusal. Further research is also necessary to consider effects on total survey error from errors that might be introduced by reluctant or converted respondents. The question is whether there are conditions where total survey error may be increased by the inclusion of respondents who have initially refused to participate and have been converted.
  5. The Task Force recommends that research specific to refusals on cell phones be executed, at all levels and on all pertinent topics.  The vast majority of research summarized on this report is not just telephonic, but limited entirely to landline telephones.  As such, much of the research noted in this report could simply be replicated, the goal being to see comparative differences between landline and cell phone interviews and interviewing.
 

 
Appendix

Table 1
Abbreviations Used in Refusal Rate Formulae
Abbreviation Meaning
REF Refusal Rate
I Completed Interview
P Partial Interview
R Refusal or Break-off
NC Noncontact
O Other Known Eligible
U Unknown Status
e Estimated proportion of unknown eligibility that are eligible
 
Refusal Rate 1 (REF1). This refusal rate is the number of refusals divided by the interviews (complete and partial), plus all non-respondents (refusals, noncontacts, and others), plus all the cases of unknown eligibility. This rate will be the lowest as the denominator is the largest of the refusal rate formulae.
                                     R
REF1 = –––––––––––––––––––––––––––––––
                    (I + P) + (R + NC + O) + U

Refusal Rate 2 (REF2).
 This refusal rate includes only the estimated eligible cases among the unknown cases. Researchers who report this rate always should include their rationale for estimating “e”.  If only one refusal rate is reported, REF2 will best convey the proportion of the eligible sample that ended as a refusal, assuming the estimation of “e” is a reasonably unbiased one.
                                      R
REF2 = –––––––––––––––––––––––––––––––
                   (I + P) + (R + NC + O) + e(U)
 
Refusal Rate 3 (REF3). This refusal rate excludes cases of unknown eligibility.  If this rate is reported, the researcher should disclosure her/his rationale for excluding all cases of unknown eligibility. This rate will be the highest as the denominator is the smallest of the refusal arte formulae.
                                     R
REF3 = –––––––––––––––––––––––––––––––
                       (I+ P) + (R + NC + O)
 
See http://www.aapor.org/AM/Template.cfm?Section=Standard_Definitions2&Template=/CM/ContentDisplay.cfm&ContentID=3156

GLOSSARY
Address based sample (ABS): Sample for survey research generated at the level of address, typically from the U.S. Postal Service’s Computerized Delivery Sequence File (CDSF).
Attenuation: Underestimating the relationship between two or more measures due to measurement error.
Bias:  When a sample statistic is systematically different than the true value of the parameter in the overall target population.
Breakoffs: Interviews in which the respondent discontinues the interviewing process at some point.  If a sufficient number of questions has been answered, researchers may consider these partial interviews; otherwise, they serve as incomplete interviews (breakoffs).
Contingent incentive: A payment to complete an interview, provided after completion of the interview.
Converted refusals: Interviews in which the respondent has initially refused to participate, but then later is persuaded to complete a full interview.
Cross-sectional survey: A survey that occurs at one point in time for one sample.  In other words, a survey that is not re-administered at some later point in time, with either a new sample or by attempting to re-contact respondents from the first survey.
Do not call list (DNC): A list maintained by a survey organization which includes all persons that have requested to never be called again to participate in surveys.
Hard refusal: Refusals in which the respondent vehemently declines to be interviewed (see section of What is a Refusal for further explication).
Initial cooperators: Persons who agree to conduct a survey on the very first request to do so.
Institutional Review Board (IRB): A board to professionals meant to review any research that deals with human subjects, in part to assess whether such research is respecting the rights of respondents.
Internet panels: Pre-recruited persons who form a pool of potential respondents willing to conduct surveys at a later date.  Typically, but not exclusively, such members are recruited using non-probabilistic methods.
Longitudinal survey (Panel survey): Surveys in which respondents are re-contacted at least once to be re-interviewed, often for surveys that are identical or contain sections of similarity at each point in time.
Non-contingent incentive: Payment for participation in surveys that are provided as an inducement before a potential respondent has even agreed to participate in the survey.
Ported telephone number: A telephone number that originates from one type of phone service (typically landline) and is then transferred to the other type of service (typically cell phone).
Probability-based sample/panel: The recruitment of persons to participate in future surveys, in which the recruitment process is done with some method of random sampling.
Refusal aversion: An effort taken by a survey research firm to employ techniques that will avoid persons refusing to participate in survey research.
Refusal conversion: An effort taken by a survey research firm to re-contact persons who initially don’t participate in a survey in order to persuade them to in fact participate.
Refusal report form (RRF): A form filled out by an interviewer after the occurrence of a refusal, documenting information such as the passion to which the person refused, their likely gender, age, and other information, which could be used to tailor and maximize the potential success of a later refusal conversion attempt.
Representativeness: The degree to which a sample accurately portrays a target population.
Responsive survey design: Utilizing information attained during the survey process in order to analyze survey costs and errors, and to then execute mid-course decisions, survey design and project operational alterations to maximize efficiency and data validity.
Sample dispositions: A typology of various results that can occur from attempting to conduct survey research.  Examples include a completed interview, refusal, bad number, answering machine, busy signal, ring but no answer, etc.
Sampled respondent: A respondent selected for participation in a survey.
Satisficing:  The answering of survey questions without executing the optimal cognitive effort to arrive at the most accurate answer, typically due to not working to fully understand the meaning of the question, to retire all possibly relevant information from memory, to integrate all information into judgment, or to then properly contextualize judgment onto the available responses of a given question.
Soft refusal: An event in which a person initially declines to participate in a survey, but rather than outright refusing, may quietly hang up, or say they don’t have time at the moment, etc.
Survey error/Total survey error: Refers to all sources of variance and bias, that is, random and systematic variance, which can impact the overall accuracy of survey estimates to the targeted population.
 

REFERENCES
Abraham, K., Maitland, A., & Bianchi, S. (2006). Nonresponse in the American Time Use Survey: Who is missing from the data and how much does it matter? Public Opinion Quarterly, 70(5), 676-703.
Ahmed, W. M. & Kalsbeek, W. D. (1998). An analysis of telephone call history data from the behavioral risk factor surveillance system. Proceedings of the Section on Survey Research Methods of the American Statistical Association, 224-9.
American Association for Public Opinion Research (2010). New considerations for survey researchers when planning and conducting RDD telephone surveys in the U.S. with respondents reached via cell phone numbers.  Deerfield, IL: Paul Lavrakas, Chair. 
American Association for Public Opinion Research (2011). Standard definitions: Final dispositions of case codes and outcome rates for surveys. 7th edition. Deerfield, IL.
American Association for Public Opinion Research website. http://www.aapor.org/Best_Practices1.htm#best9
Armstrong, J. S. (1975). Monetary incentives in mail surveys.  Public Opinion Quarterly, 39(1), 111-116.
Atrostic, B. K., Bates, N., Burt, G., and Silberstein, A. (2001). Nonresponse in U.S. government household surveys: Consistent measures, recent trends, and new insights. Journal of Official Statistics 17, 209-226.
Basson, D. & Chronister, M. (2006). Recordings of prior refusals: Do they improve later conversion attempts?  Paper written in preparation for the Methods on Longitudinal Surveys.  Essex, England.
Bates, N., Dahlhamer, J., & Singer, E. (2008). Privacy concerns, too busy, or just not interested: Using doorstep concerns to predict survey nonresponse. Journal of Official Statistics, 24, 591-612.
Bauman, S. L. (2008). Refusals. In P.J. Lavrakas (Ed.), Encyclopedia of Survey Research Methods. Newbury Park: Sage Publications.
Bauman, S. L., Merkle, D. M., & Lavrakas, P. J. (1992). Interviewer estimates of refusers’ gender, age, and race in telephone surveys. Paper presented at the 16th annual conference of the Midwest Association for Public Opinion Research, Chicago, IL.
Beckler, D. G. & Ott, K. (2006).  Indirect monetary incentives with a complex agricultural establishment survey.  Proceedings of the Survey Research Methods Section, American Statistical Association, 2741-2748
Benford, R., Lavrakas, P., Thompson, T., & Fleury, C. (2010). Nonresponse in RDD cell phone surveys.  Prepared for the AAPOR Cell Phone Task Force Report. Deerfield, IL.
Benson, S., Booman, W. P., & Clark, K. E. (1951). A study of interview refusals. Journal of Applied Psychology, 35(2), 116-119.
Benkí, J., Broome, J., Conrad, F., Groves, R., & Kreuter, F. (2011). Effects of speech rate, pitch, and pausing on survey participation decisions.  Paper presented at the American Association for Public Opinion Research annual conference, Phoenix, AZ.
Berlin, M., Mohadjer L., Waksberg, J., Kolstad, A., Kirsch, I., Rock, D., & Yamamoto, K. (1992).  An experiment in monetary incentives.  Proceedings of the Survey Research Methods Section of the American Statistical Association 393-398.
Bethlehem, J. (2002). Weighting nonresponse adjustments based on auxiliary information.  In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey Nonresponse.  New York: John Wiley & Sons, Inc.
Beullens, K., Billiet, J., & Loosveldt, G. (2010).  The effect of the elapsed time between the initial refusal and conversion contact on conversion success: Evidence from the 2nd round of the European social survey.  Qual Quant, 44, 1053-1065.
Biemer, P. (2010). Total survey error: Design, implementation, and evaluation. Public Opinion Quarterly, 74(5), 1-35.
Billiet, J., Philippens, M., Fitzgerald, R., & Stoop, I. (2007). Estimation of nonresponse bias in the European Social Survey: Using information from reluctant respondents. Journal of Official Statistics, 23, 135-162.
Biner, P. M. & Kidd, H. J. (1994). The interactive effects of monetary incentive justification and questionnaire length on mail survey response rates. Psychology & Marketing, 11(5), 483-492.
Birnbaum, Z. W. & Sirken, M. G. (1950). Bias due to non-availability in sample surveys. Journal American Statistical Association, 45, 98-111.
Blohm, M., Hox, J., & Koch, A. (2006). The influence of interviewers' contact behavior on the contact and cooperation rate in face-to-face household surveys. International Journal of Public Opinion Research, 19, 97-111.
Blumberg, S.J., Luke, J.V. (2012). Wireless substitution: Early release of estimates from the National Health Interview Survey, January–June 2012. National Center for Health Statistics. Available from: http://www.cdc.gov/nchs/nhis.htm. Boersma F., Eefsting J. A., van den Brink, W., van Tilburg, W. (1997). Care services for dementia patients: Predictors for service utilization.  International Journal of Geriatric Psychiatry, 12(11), 1119-26.
Boersma, F., Eefsting, J.A., van den Brink, W. & van Tilburg, W. (1997). Characteristics of non-responders and the impact of non-response on prevalence estimates of dementia. International Journal of Epidemiology. 26(5), 1055-1062.
Brick, J. M. & Williams, D. (2013). Explaining rising nonresponse rates in cross-sectional surveys. Annals of the American Academy of Political and Social Science, 645, 36-59.
Brick, J. M., Williams, D., & Montaquilla, J. (2011). Address-based sampling for subpopulation surveys. Public Opinion Quarterly, 75, 409-428.
Brick, J. M., Hagedorn, M. C., Montaquila, J., Roth, S. B., & Chapman, C. (2004). Using an experiment to design an RDD survey. Paper presented at the American Association for Public Opinion Research Annual Conference, Phoenix, AZ.
Brick, J. M., Montaquila, J., Hagedorn, M. C., Roth, S. B., & Chapman, C. (2005). Implications for RDD design from an incentive experiment. Journal of Official Statistics, 21(4), 571–589.
Brown, G. G., Brown, J. S., Kratochvil C., Schonfeld T. L., & Prentice E. P.  (2006). Paying research subjects In Bankert, E.A. and Amdur, R.J., (Eds), Institutional Review Board Management and Function. Sudbury MA.
Bucholz, K. K., Marion, S. L., Shayka, .J J., Marcus, S.C., & Robins, L. N. (1996).  A short computer interview for obtaining psychiatric diagnoses. Psychiatric Services, 47, 293-297.
Camburn, D., Lavrakas, P. J., Battaglia, M. P., Massey, J. T., & Wright, R. A. Using advance respondent letters in random-digit-dialing telephone surveys. Proceedings of the Section on Survey Research of the American Statistical Association, 969-974.
Campanelli, P., Sturgis, P., & Purdon, S. (1997). Can you hear me knocking: An investigation into the Impact of interviewers on survey response rates. Survey Methods Centre at SCPR. London.
Campanelli, P. (2008).  Introduction. In P. J. Lavrakas (Ed.),  Encyclopedia of survey research methods.  Thousand Oaks, CA: Sage Publications.
Cantor, D., O’Hare, B., & O’Connor, K. (2007). The use of monetary incentives to reduce non-response in random digit dial telephone surveys, In J. M. Lepkowski, C. Tucker, J. M. Brick, E. De Leeuw,  L. Japec, P. J. Lavrakas, M. W. Link, & E. Sangster, (Eds.), Advances In telephone survey methodology (pp. 471-498).  New York: J. W. Wiley and Sons, Inc.
Cantor, D., Allen, B., Schneider, S., Hagerty-Heller, T., & Yuan, A. (2004). Testing an automated refusal avoidance training methodology. Paper presented at annual meeting of the American Association for Public Opinion Research, Phoenix, AZ.
Cantor, D., Wang, K., & Abi-Habib, N. (2003). Comparing promised and pre-paid incentives for an extended interview on a random digit dial survey.  Proceedings of the Survey Research Methods Section of the American Statistical Association, , 37-43.
Carpenter, E. H. (1974).  Personalizing mail surveys: A replication and reassessment. Public Opinion Quarterly, 38, 614-20.
Carroll, M. & Chong, Y. (2006). Are refusal conversions different from willing respondents with respect to item non-response, demographics and selected health characteristics? The National Health and Nutrition Examination Survey, 1999-2002. Proceedings of Survey Research Methods Section of the American Statistical Association, 2795-2802.
Church, A. (1993). Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly, 57(1), 62-79.
Colicchia, M., Czaplewki, M., & Jaszcak, A. (2011).  Refusal conversion and participation in a longitudinal study of older adults.  Paper presented at the 2011 Annual Conference of the American Association for Public Opinion Research, Phoenix, AZ.
Collins, M., Sykes, W., Wilson, P., & Blackshaw, N. (1988). Nonresponse: The UK experience. In R. M. Groves, P. P. Biemer, L. E. Lyberg, J. T. Massey, W. L. Nicholls, II, and J. Waksberg (Eds.), Telephone Survey Methodology, 213-231.  New York: Wiley.
Cominole, M., Franklin, J., & Siegel, P. (2008). Combined strategies to increase early response rates.  Presented at the annual meeting of the American Association for Public Opinion ResearchNew Orleans, LA.
Cook, S., LeBaron, P., Flicker, L., & Flanigan, T. S. (2009). Applying incentives to establishment surveys: A review of the literature. Proceedings of the Survey Research Methods Section of the American Statistical Association, 5649 – 5657.
Council of American Survey Research Organizations website. http://www.casro.org/pdfs/10CodeOfStandards.pdf
Council of Professional Associations on Federal Statistics (2008).  Seminar on survey respondent incentives: research and practice.  Washington, DC.  Meeting notes available at: http://www.copafs.org/seminars/use_of_incentives_five_questions.aspx.
Couper, M. P., Singer, E., Conrad, F. G., & Groves, R. M. (2010). Experimental studies of disclosure risk, disclosure harm, topic sensitivity, and survey participation. Journal of Official Statistics, 26(2), 287-300.
Couper, M. P. & Groves, R. M. (1992). The role of the interviewer in survey participation. Survey Methodology, 18, 263-278.
Currivan, D. (2005). The impact of providing incentives to initial telephone survey refusers on sample composition and data quality. Presented at the annual meeting of the American Association for Public Opinion Research,Miami, FL.
Curtin, R., Presser, S., & Singer, E. (2005). Changes in telephone survey nonresponse over the past quarter century. Public Opinion Quarterly, 69(1), 87-98.
Dahlhamer, J. M., Simile, C. M., & Taylor, B. (2006). Exploring the impact of participant reluctance on data quality in the national health interview survey (NHIS). In Proceedings of Statistics Canada Symposium 2006: Methodological Issues in Measuring Population Health.
Dahlhamer, J. M., Simile, C. M., & Taylor, B. (2008). Do you really mean what you say? Doorstep concerns and data quality in the National Health Interview Survey (NHIS). Proceedings of Survey Research Methods Section of the American Statistical Association, 1484-1491.
de Leeuw, E. D. & De Heer, W. (2002). Trends in household survey nonresponse: A longitudinal and international comparison. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds). Survey nonresponse. (pp. 41-54). New York: Wiley.
de Leeuw, E., Callegaro, M., Hox, J., Korendijk, E., & Lensvelt-Mulders, G. (2007). The influence of advance letters on response in telephone surveys. A meta-analysis. Public Opinion Quarterly, 71, 413-443.
de Leeuw, E. (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics 21, 233-255.
de Leeuw, E. D., Hox, J. J., & Dillman, D. A. (2008). Mixed mode surveys: When and why. In E. D. De Leeuw, J. J. Hox & D. A. Dillman (Eds.), International handbook of survey methodology, (pp. 299-316).  New York: Lawrence Erlbaum.
DeMaio, T. J. (1980). Refusals: Who, where and why. The Public Opinion Quarterly, 44, 223-233.
Deming, W. E. (1953). On a probability mechanism to attain an economic balance between the resultant error or response and the bias of nonresponse. JASA, 48, 743-72.
Dillman, D. A. (1978). Mail and Telephone Surveys: The Total Design Method. New York: John Wiley and Sons.
Dillman, D. A. & Frey, J. H. (1974). The contribution of personalization to mail questionnaire response as an element of a previously tested method. Journal of Applied Psychology, 59, 297-301.
Dillman, D., Gallegos, J., & Frey, J. (1976).  Reducing refusal rates for telephone interviews.  Public Opinion Quarterly, 40, 66-78.
Dillman, D. A., Phelps, G., Tortora, R., Swift, K., Kohrell, J., Berck, J., & Messer, B. L. (2009). Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the internet. Social Science Research, 38, 1-18.
Dillman, D. A. (2000). Mail and Internet Surveys: The Tailored Design Method.  New York: Wiley.
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Internet, Mail and Mixed-Mode Surveys: The Tailored Design Method, Third Edition. Hoboken, NJ: John Wiley & Sons.
Dillman, D. A. (2007). Mail and Internet Surveys. The Tailored Design Method, 2nd edition. New York: Wiley.
Dipko, S. & Freedman, K. (2009). Cell phone reverse directories – promise or peril? Proceedings of the Section on Survey Methods of the American Statistical Association , 6167-9173.
Durand, C., Gagnon, M. E., Doucet, C., & Lacourse, E. (2006). An inquiry into the efficacy of a complementary training session for telephone survey interviewers. Bulletin de Methodologie Sociologique, 92, 5-27.
Durrant, G. B. & Steele, F. (2009). Multilevel modelling of refusal and noncontact in household surveys: Evidence from six uk government surveys. Journal of the Royal Statistical Society Series A (Statistics in Society), 172(2), 361-381.
Durrant, G, B., D'Arrigo, J., & Steele, F. (2011). Using field process data to predict best times of contact conditioning on household and interviewer influences. Journal of the Royal Statistical Society: Series A (Statistics in Society), 174(4), 1029-1049.
Durbin, J. & Stuart, A. (1951). Differences in response rates of experienced and inexperienced interviewers. Journal of the Royal Statistical Society, 114, 163-205.
Durrant, G. B., Groves, R. M., Staetsky, L., & Steele, F. (2010). Effects of interviewer attitudes and behaviors on refusal in household surveys. Public Opinion Quarterly, 74, 1-36.
Dutwin, D. & Herrmann, M. (2006). Finding converts: Who becomes a refusal conversion? Paper presented at the annual conference of the American Association for Public Opinion Research, Montreal, Canada.
Edwards, P., Roberts, I., Clarke, M., DiGuisepi, C., Pratap, S., Wentz, R., & Kwan, I. (2002). Increasing response rates to postal questionnaire: Systematic review. BMJ, 324, 1183.
Edwards, S., Martin, D., DiSogra, C., & Grant, D. (2004). Altering the hold period for refusal conversion cases in an RDD survey. Proceedings of the Section on Survey Research Methods of the American Statistical Association, 3440-3.
Edwards, W. S., DiSogra, C., & Yen, W. (2003) Scheduling calls for refusal conversion in an RDD survey.  Presentation at the annual conference of the American Association for Public Opinion Research, St. Petersburg Beach.
ESOMAR website.  http://www.esomar.org/uploads/public/knowledge-and-standards/codes-and-guidelines/ICCESOMAR_Code_English_.pdf
Eyerman, J., Bowman, K., Butler, D., & Wright, D. (2005). The differential impact of incentives on refusals: Results from the 2001 National Household Survey on Drug Abuse incentive experiment. Journal of Economic and Social Measurement, 30(2-3), 157-169.
Festinger, L. (1957). A Theory Of Cognitive Dissonance. Stanford, CA: Stanford University Press.
Foote R., Schipani, K., Schipani D., & Cho, Y. I. (2005b). Refusal conversion: Monitoring the trends. American Association for Public Opinion Research Section on Survey Research Methods, 4984-90.
Fox, R., Crask, M. R., & Kim, J. (1988). Mail survey response rate:  A meta-analysis of selected techniques for inducing response.  Public Opinion Quarterly, 52, 467-491.
Fuse, K. & Xie, D. (2007). A successful conversion or double refusal: A study of the process of refusal conversion in telephone survey research. The Social Science Journal, 44, 434-446.
Gilbert, T. (2009). Mode preferences in the 2008 national survey of college graduates. Paper presented at the annual conference of the American Association of Public Opinion Research, Hollywood, FL.
Goldstein, K. & Jennings, M. K. (2002). The effect of advance letters on cooperation in a list assisted sample telephone survey. Public Opinion Quarterly, 66, 608-617.
Goyder, J. (1987). The silent minority: Nonrespondents on sample surveys.  Boulder, CO: Westview Press.
Goyder, J., Boyer, L., & Marinelli, G. (2006). Integrating exchange and heuristic theories of nonresponse. Bulletin of Sociological Methodology, 92(1), 28-44.
Groves, R. M. & McGonagle, K. A. (2001). A theory-guided interviewer training protocol regarding survey participation. Journal of Official Statistics, 17(2), 249-265.
Groves, R. M. (1979). Actors and questions in telephone and personal interview surveys. Public Opinion Quarterly, 43, 190–205.
Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70(5), 646-675.
Groves, R. M., O'Hare, B. C., Gould-Smith, D., Benki, J., & Maher, P. (2008). Telephone interviewer voice characteristics and the survey participation decision.  In J. M. Lepkowski, C. Tucker, J. M. Brick, E. D. De Leeuw, L. Japec, P. J. Lavrakas, M. W. Link, and R. L. Sangster (Eds.),Advances in Telephone Survey Methodology, (pp. 385-400).  Hoboken, NJ: John Wiley & Sons.
Groves, R. M., Cialdini, R. B., & Couper, M. P. (1992). Understanding the decision to participate in a survey. Public Opinion Quarterly, 56, 475-495.
Groves, R., Fowler, F., Couper, M., Lepkowski, J., Singer, E., & Tourangeau, R. (2004). Survey Methodology. New York: Wiley.
Groves, R. M. & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias – a meta-analysis, Public Opinion Quarterly, 72(2), 167–189.
Groves, R. M. & Kahn, R. L. (1979). Surveys by telephone. New York: John Wiley & Sons, Inc.
Groves, R. M., Singer, E., Corning, A. D., & Bowers, A. (1999). A laboratory approach to measuring the effects on survey participation of interview length, incentives, differential incentives, and refusal conversion.  Journal of Official Statistics, 15(2), 251-268.
Groves, R. M., Presser, S., & Dipko, S. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68(1), 2-31.
Groves, R. M. & Couper, P. M., (1998). Non-response in household interview surveys. New York: John Wiley and Sons.
Groves, R. M. & Heeringa, S. G., (2006). Responsive design for household surveys: Tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society: Series A (Statistics in Society), 169(3), 439-457.
Groves, R. M. & Benki, J. (2006). 300 hellos: Acoustic properties of initial respondent greetings and response propensities in telephone surveys.  17th International Workshop on Household Survey Nonresponse.  Omaha, NB.
Groves, R. M. & Fultz, N. H., (1985). Gender effects among telephone interviewers in a survey of economic attitudes. Sociological Methods & Research, 14(1), 31-52.
Groves, R. M., Singer, E., & Corning, A. D. (2000). Leverage-saliency theory of survey participation: description and an illustration. The Public Opinion Quarterly, 64(3), 299-308.
Gwartney, P. A. (2007). The Telephone Interviewer's Handbook: How to Conduct Standardized Conversations. San Francisco: Jossey-Bass.
Hansen, K.M. (2007). The effects of incentives, interview length, and interviewer characteristics on response rates in a CATI-study.  International Journal of Public Opinion Research, 19(1), 112-121
Heberlein, T. & Baumgartner, R. (1978). Actors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature. American Sociological Review, 43, 447-462.
Holbrook, A., Pfent, A., & Krosnick, J. A. (2003). Response rates in recent surveys conducted by non-profits and commercial survey agencies and the news media. American Association for Public Opinion Research annual conference.  Nashville, TN.
Holbrook, A., Krosnick, J. A., & Pfent, A. (2007). The causes and consequences of response rates in surveys by the news media and government contractor survey research firms. Advances in Telephone Survey Methodology, 499-678.
Houtkoop-Streenstra.,& van den Bergh, H.. (2000). Effects of Introductions in large-scale telephone survey interviews. Sociological Methods and Research. 28(3), 281-300.
Hox, J. & De Leeuw, E. (2002). The influence of interviewers' attitude and behavior on household survey nonresponse: An international comparison. In R. M. Groves, D. A. Dillman, J. L. Eltinge, and R. J. A. Little (Eds.), Survey Nonresponse, (pp. 103-120).  New York: Wiley.
Hughes, A., Chromy, J., Giacoletti, K., & Odom, D. (2002). Impact of interviewer experience on respondent reports of substance use.  In J. Gfroerer, J. Eyerman, and J. Chromy (Eds.), Redesigning an Ongoing National Household Survey, (pp. 161-184).  Washington, DC: Substance Abuse and Mental Health Services Administration.
Jäckle, A., Lynn, P., Sinibaldi, J., & Tipping, S. (2011). The effect of interviewer personality, skills and attitudes on respondent co-operation with face-to-face surveys.  In ISER Working Paper Series, 2011(14).
James, J. & Bolstein, R. (1990). The effect of monetary incentives and follow-up mailings on the response rate and response quality in mail surveys. Public Opinion Quarterly, 54, 346-361.
John, O. P. & Srivastava, S. (1999). The big-five trait taxonomy: history, measurement, and theoretical perspectives. In L. A. Pervin & O. P. John (Eds.), Handbook of Personality: Theory and Research, 2, (pp. 102–138).  New York: Guilford Press.
Johnson, T.P., Cho, Y.I., Campbell, R.T., & Holbrook, A.L. (2006). Using community-level correlates to evaluate nonresponse effects in a telephone survey. Public Opinion Quarterly 70(5), 704-719.
Karlsson, A. Ö. (2010). Recent developments in interviewer training at Statistics Iceland: minimizing interviewer effects and reducing refusal rates. Nordisk Statistikermøde, , 1-7.
Kaminska, O., McCutcheon, A. L., & Billiet, J. (2010). Satisficing among reluctant respondents in a cross-national context.  Public Opinion Quarterly, 74(5), 956-985.
Keeter, S., Kennedy, C., Dimock, M., Best, J., & Craighill, P. (2006). Gauging the impact of growing nonresponse on estimates from a national RDD telephone survey. Public Opinion Quarterly 70, 759-779.
Keeter, S., Miller, C., Kohut, A., Groves, R. M., & Presser, S. (2000). Consequences of reducing nonresponse in a national telephone survey. Public Opinion Quarterly, 64(2), 125-148.
Kendrick, D., Hapgood, R., & Marsh, P. (2001). Do safety practices differ between responders and non-responders to a safety questionnaire?. Injury Prevention, 7(2), 100-103.
Kohut, A., Keeter, S., Dimrock, M., Doherty, C., & Christian, L. M. (2112). Assessing the representativeness of public opinion surveys, Pew Research Center, 52.
Kropf , M. E., Scheib, J., & Blair, J. (1999). The effect of alternative incentives on cooperation and refusal conversion in a telephone survey. Paper presented at the 54th annual conference of the American Association for Public Opinion Research,  St. Pete Beach, FL.
Kulka, R. A. (1994). The use of incentives to survey “hard-to-reach” respondents: A brief review of empirical research and current practice. Paper prepared for Seminar on New Directions in Statistical Methodology,  Bethesda, MD.
Lavrakas, P. J. (1987). Telephone Survey Methods: Sampling, Selection, and Supervision (1st edition). Thousand Oaks, CA: Sage Publications.
Lavrakas, P. J. (1993). Telephone survey methods: Sampling, selection, and supervision (2nd edition). Thousand Oaks, CA: Sage Publications.
Lavrakas, P. J. (2001). Improving data quality in the Nielsen media research diary and meter samples. Paper presented at the  Advertising Research Foundation annual conference,  Chicago, IL
Lavrakas, P. J. (2004). Will a perfect storm of cellular forces sink RDD sampling?  Presented at the annual conference of the American Association for Public Opinion Research,  Phoenix, AZ.
Lavrakas, P. J. (2008). Fallback statements. In Paul J. Lavrakas (Ed.). Encyclopedia of survey research methods.  Thousand Oaks, CA: Sage Publications.
Lavrakas, P. J., Clancy, R., & Wheeler, J. (2013). Online vs. mobile survey mode comparison. In Measuring the Unmeasured Conference of the Advertising Research Foundation, New York, NY.
Lavrakas, P. J., Dennis, J. M., Peugh, J., Shand-Lubbers J., Lee, E., & Charlebois, O. (2012).  Investigating nonresponse bias in a nonresponse bias study.  Paper presented at the annual meeting of the American Association of Public Opinion Research, Orlando, FL.
Lavrakas, P. J., & Merkle, D. M. (1991). A reversal of roles: When respondents question interviewers. In Midwest Association for Public Opinion Research Conference, Chicago, IL.
Lavrakas, P. J. & Shuttles, C. D. (2004). Two advance letter experiments to raise survey responses rates in a two-stage mixed mode survey. Paper presented at the 2004 Joint Statistical Meetings, Toronto, Canada.
Lavrakas, P. J., Bauman, S. L., & Merkle, D. M. (1992). Refusal report forms (RRFs), refusal conversions, and non-response bias. Working paper. Northwestern University Survey Laboratory.
Lavrakas, P. J.  (2010). Non-response bias mini-summit: Study design.  Paper presented to the Council for Research Excellence.
Lavrakas, P. J.  (2012). Recent developments and emerging issues in dual frame RDD surveying. Paper presented at the SRC Workshop: Telephone Surveys and the Mobile Phone Only Populations.  Melbourne, Australia.
Lavrakas, P. J., Burks, & Bennett. (2004). Predicting samples respondents’ likelihood to cooperate.   Paper presented at the annual MAPOR Conference,  Chicago, IL.
Lavrakas, P. J., Trussell, N., Daily,G., Yancey, T., Watkins, R. (2006).  Improving Response Rates among Targeted Demographic Subgroups Using Large Cash Incentives.  Paper presented at the annual conference of the American Association for Public Opinion Research Conference,  Montreal, Canada.
Levenstein, R. (2009). Mode preference, mode choice, and data quality in the Army Defense, Resilience, and Retention Study.  Paper presented at the annual conference of the American Association for Public Opinion Research, Hollywood, FL.
Lengacher, J. E., Sullivan, C., Couper, M. P., & Groves, R. (1995). Once reluctant, always reluctant? Effects of differential incentives on later survey participation in a longitudinal study.  Proceedings of the Survey Research Methods Section of the American Statistical Association, 1029-1034.
Lindenauer, P.K., Benjamin, E. M., Naglieri-Prescad, D., Fitzgerald, J., & Pekow, P. ( 2002). The role of the institutional review board in quality improvement: A survey of quality officers, institutional review board chairs, and journal editors. American Journal of Medicine, 113, 575–579.
Link, M. W. (2006). Predicting the persistence and performance of newly recruited telephone interviewers. Field Methods, 18, 305-320.
Link, M. W. (2008). Introduction. In P. J. Lavrakas (Ed.). Encyclopedia of Survey Research Methods.  Thousand Oaks, CA: Sage Publications.
Link, M. W., Mokdad, A., Town, M., Weiner, J., & Roe, D. (2003). Improving response rates for the BRFSS: Use of lead letters and answering machine messages. Proceedings of the Joint Statistical Meetings of the American Statistical Association, 141-148.
Link, M. W. & Mokdad, A. H. (2005). Effects of survey mode on self-reports of adult alcohol consumption: A comparison of mail, web and telephone approaches. Journal of Studies on Alcohol and Drugs66(2), 239.
Link, M. W. & Mokdad, A. (2006). Can web and mail survey modes improve participation in an RDD-based national health surveillance? Journal of Official Statistics, 22, 293-312.
Link, M., Mokdad, A. H., Kulp, D., & Hyon, A. (2006). Has the National Do Not Call Registry helped or hurt state-level response rates? A time series analysis. Public Opinion Quarterly, 70 (5), 794-809.
Link, M. W., Armsby, P. P., Hubal, R. C., & Guinn, C. I. (2006). Accessibility and acceptance of responsive virtual human technology as a survey interviewer training tool. Computers in Human Behavior, 22, 412-426.
Lipps, O. & Pollien, A. (2011). Effects of interviewer experience on components of nonresponse in the European Social Survey. Field Methods, 23, 156-172.
Lynn, P., Clarke, P., Martin, J., & Sturgis, P. (2002). The effects of extended interviewer efforts on nonresponse bias.  In R. M. Groves, D. A. Dillman, J. L. Eltinge, and R. J. Little (Eds.), Survey Nonresponse, (pp. 135-147).  New York: Wiley.
Mann, C. (2005). Do advance letters improve preelection forecast accuracy? Public Opinion Quarterly, 69(4) 561-571.
Manzo, A. & Burke, J. (2012).  Increasing response rate in web-based/internet surveys. In L. Gideon (ed.), Handbook of Survey Methodology for the Social Sciences, (pp. 327-343).  Springer Science+Business Media: New York.
Mark, A. & ZuWallack, R. (2012). Cell phone operational efficiencies for a survey of young adults.  Presented at the annual conference of the American Association for Public Opinion Research,  Orlando, Florida.
Marketing Research Association website, accessed July 10, 2012. http://mra.cms.memberfuse.com/sites/mra/files/CodeMRStandards.pdf
Marketing Research Association website, accessed July 10, 2012. http://www.marketingresearch.org/developing-internal-do-not-contact-listshttp://www.marketingresearch.org/developing-internal-do-not-contact-lists.                              
Marshall, J. W., Brown, Y. P., & Page, W. G. (1989). Maintaining workforce quality in a changing survey environment.  Bureau of the Census Fifth Annual Research Conference, 615-630.  Arlington, VA: U.S. Department of Commerce, Bureau of the Census.
Martin, P. (2011). What makes a good mix? Chances and challenges of mixed mode data collection in the ESS. London: Centre for Comparative Social Surveys. Working Paper Series.
Massey, J. T., Wolter, C., Wan, S. C., & Liu, K. (1996). Optimum calling patterns for random digit dialed telephone surveys. Proceedings from Section on Survey Research Methods of the American Statistical Association, 485-490.
Massey, J.T., Barker, P.R., & Hsiung, S. (1981). An investigation of response in a telephone survey. Proceedings from Section on Survey Research Methods of the American Statistical Association, 426-431.
Mayer, T. S. & M. O'Brien, E. (2001). Interviewer refusal aversion training to increase survey participation. Paper presented at the Joint Statistical Meetings of the American Statistical Association, Atlanta, GA.
Maynard, D. W. & Schaeffer, N. C. (2002). Refusal conversion and tailoring.  In D. W. Maynard, H. Houtkoop-Steenstra, N. C. Schaeffer, and J. van der Zouwen (Eds.), Standardization and Tacit Knowledge: Interaction and Practice in the Survey Interview, (pp. 219-239).  New York: John Wiley & Sons, Inc.
McCarthy, J. S., Johnson, J. V., & Ott, K. (1999). Exploring the relationship between survey participation and survey sponsorship: What do respondents and nonrespondents think of us.  Paper presented at the International Conference on Survey Nonresponse, Stockholm
McDermott, N. & Tan, L. (2008). The effect of refusal conversion on data quality in the consumer expenditure interview survey. Consumer Expenditure Survey Anthology, 23-32.
McGrath, D. (2006). An incentives experiment in the U.S. Consumer Expenditure Quarterly Survey. Proceedings of the Survey Research Methods Section of the American Statistical Association.
McPhee, C. & Hastedt, S. (2012).  More money? The impact of larger incentives on response rates in a two-phase mail survey.  Paper presented at the Federal Committee on Statistical Methods Research Conference, Washington, DC.
Merkle, D. M. & Edelman, M. (2002). Nonresponse in exit polls: A comprehensive analysis. In R.M. Groves, D.A. Dillman, J.L. Eltinge and R.J.A. Little, (Eds.), Survey nonresponse. Wiley, New York.
Messer, B. L. & Dillman, D. A. (2011). Surveying the general public over the internet using address-based sampling and mail contact procedures. Public Opinion Quarterly, 75(3), 429-457.
Mierzwa, F. J., McCracken, R., Vandermaas-Peeler, R., & Tronnier, C. D. (2002). The impact of interviewer characteristics on survey participation: Findings from NSCAW. Proceedings of the American Association for Public Research Section on Survey Research Methods ,, 2343-2348.
Millar, M. M. & Dillman, D. A. (2011). Improving response to web and mixed-mode surveys. Public Opinion Quarterly, 75(2), 249-269.
Miller, J. M. & Wedeking, J. (2003). The harder we try, the worse it gets? Examining the impact of refusal conversions and high callback attempts on the quality of survey data. Paper presented at annual conference of the American Association for Public Opinion Research, , Nashville, TN.
Miller, J. M. & Wedeking, J. (2006). Examining the impact of refusal conversions and high callback attempts on measurement error in surveys. Paper presented at the annual conference of the American Association for Public Opinion Research, Montreal, Canada.
Miller, P. & Cannell, C. (1982). A study of experimental techniques for telephone interviewing. Public Opinion Quarterly, 46, 250–269.
Mishler, E. G. (1986). Research interviewing: Context and narrative.  Cambridge: Harvard University Press.
Mitchell, S., Hunter, D., Carley-Baxter, L., & Keating, M. (2012). Using paradata to monitor survey production, cost, and quality within an adaptive total design framework. Paper presented at the European Conference on Quality in Official Statistics.  Athens, Greece.
Montaquila, J. M., Brick, J. M., Hagedorn, M.C., Kennedy, C., & Keeter, S. (2008). Aspects of nonresponse bias in RDD telephone surveys. In J. M. Lepkowski, C. Tucker, J. M. Brick, E. D. De Leeuw, L. Japec, P. J. Lavrakas, M. W. Link, and R. L. Sangster (Eds.), Advances in Telephone Survey Methodology, (pp. 561-586).  Hoboken, NJ: John Wiley & Sons.
Moorman, P., Newman, B., Millikan, R., Tse, C. K., & Sandler, D. (1999). Participation rates in a case-control study: The impact of age, race, and race of interviewer. Annals of Epidemiology, 9(3), 188-195.
Morton-Williams, Jean.  (1993). Interviewer Approaches. Brookfield, VT: Dartmouth.
National Assessment of Educational Progress (2003). Supervisors Manual. Retrieved August 13, 2012 from http://nces.ed.gov/nationsreportcard/pdf/about/2004_fl_ft_supervisor_manual.pdf
National Postsecondary Student Aid Study (2008).  (NPSAS:08) Field test methodology report, U.S. Department of Education, Working Paper Series.
O’Brien, E. M., Black, M. C., Carley-Baxter, L. R., & Simon, T. R. (2006). Sensitive topics, survey nonresponse, and considerations for interviewer training. American Journal of Preventive Medicine, 31, 419-426.
O'Brien, E. M., Mayer, T. S., Groves, R. M., & O'Neill, G. E. (2002). Interviewer training to increase survey participation.Presented at the Joint Statistical Meetings of the American Statistical Association, New York.
O'Neill, T. W., Marsden, D., & Silman, A. J. (1995). Differences in the characteristics of responders and non-responders in a prevalence survey of vertebral osteoporosis. Osteoporosis International, 5(5), 327-334.
O'Neil, M. J. (1979). Estimating the Nonresponse bias due to refusals in telephone surveys. Public Opinion Quarterly, 43, 218-232.
O’Neil, M. J., Groves, R. M., & Cannell, C. (1979). Telephone interview introductions and refusal rates: Experiments in increasing respondents’ cooperation. Proceedings of the Survey Research Methods Section of the American Statistical Association, 252-255.
O'Neill, G. E. (2007). Improving the effectiveness of interviewer administered surveys through refusal avoidance training.  American Statistical Association Third International Conference on Establishment Surveys, 233-239. Odom, D. M. & Kalsbeek, W. D. (2000). Further analysis of telephone call history data from the behavioral risk factor surveillance system. Proceedings from Section on Survey Research Methods of theAmerican Statistical Association, 398-403.
Odom, D. M. & Kalsbeek, W. D. (2000). Further analysis of telephone call history data from the behavioral risk factor surveillance system. Proceedings from Section on Survey Research MethodsAmerican Statistical Association, 398-403.
Oksenberg, L. & Cannell, C. F. (1988). Effects of interviewer vocal characteristics on nonresponse.  In R. M. Groves, P. Biemer, L. Lyberg, J. T. Massey, W. L. Nicholls II, and J. Waksberg (Eds.), Telephone Survey Methodology, (pp. 257-269).  New York: John Wiley & Sons, Inc.
Oksenberg, L., Coleman, L., & Cannell, C. F. (1986). Interviewers' voices and refusal rates in telephone surveys. Public Opinion Quarterly, 50, 97-111.
Oldendick, R. (2012). Survey research ethics. In L. Gideon (Ed.), Handbook of Survey Methodology for the Social Sciences, (pp. 23-35).  Springer Science+Business Media: New York.
Olson, K., Smyth, J. D., & Wood, H.(2014). Does giving people their preferred survey mode actually increase survey participation rates? Public Opinion Quarterly, 76, 611-635
Olson, K., Lepkowski, J. M., & Garabrant, D. H. (2011). An experimental examination of the content of persuasion letters on nonresponse rates and survey estimates in a nonresponse follow-up study. Survey Research Methods, 5(1), 21-26
Olson, K. & Groves, R. M. (2012). An examination of within-person variation in response propensity over the data collection field period. Journal of Official Statistics, 28, 29–51.
Olson, K., & Kennedy, C. (2006). Examination of the relationship between nonresponse and measurement error in a validation study of alumni. In Proceedings of the Survey Research Methods Section (pp. 4181-88).
Olson, K. (2013). Paradata for Nonresponse Adjustment. The Annals of the American Academy of Political and Social Science, 645(1), 142-170.
Painter, D., Chromy, J. R., Meyer, M., Granger, R. A., & Clarke, A. (2003).  Effects of incentives on data collection: a record of calls analysis of national survey on drug use and health. Proceedings of the Survey Research Methods Section of the American Association for Public Opinion Research, 170-176.
Petrolia, D. R. & Bhattacharjee, S. (2009).  Revisiting incentive effects: evidence from a random-sample mail survey on consumer preferences for fuel ethanol.  Public Opinion Quarterly, 73(3), 537-550.
Peytchev, A., Baxter, R. K., & Carley-Baxter, L. R. (2009). Not all survey effort is equal. Public Opinion Quarterly, 73, 785-806.
Peytchev, A., Peytcheva, E., & Groves, R. M. (2010). Measurement error, unit nonresponse, and self-reports of abortion experiences. Public Opinion Quarterly, 74(2), 319–327.
Peytcheva, E. & Groves, R. M. (2009). Using variation in response rates of demographic subgroups as evidence of nonresponse bias in survey estimates. Journal of Official Statistics, 25, 193–201.
Pickery, J. & Loosveldt, G. (2002). A multilevel multinomial analysis of interviewer effects on various components of unit nonresponse. Quality and Quantity, 36, 427-437.
Porter, S. R. (2004). Raising response rates: what works? New Directions for Institutional Research, (121), 5-21.
Porter, S. R. & Whitcomb, M. E. (2003). The impact of lottery incentives on student survey response rates. Research in Higher Education, 44(4), 389-407.
Rapoport, R., Lavrakas, P. J., Ben-Porath, E., & Herrmann, M. (2013). Multiple approaches for evaluating non-response bias in a short-field-period survey. Paper presented at the annual conference of the American Association of Public Opinion Research,  Boston MA.
Robins, L. N. (1963). The reluctant respondent. Public Opinion Quarterly, 27, 276-286.
Safir, A., Steinbach, R., Triplett, T., & Wang, K. (2002). Effects on survey estimates from reducing nonresponse. Proceedings of the American Association for Public Opinion ResearchSurvey Research Methods Section, 3024-3029.
Sala, E. & Lynn, P. (2009). The potential of a multi-mode data collection design to reduce non-response bias: The case of a survey of employers. Quality and Quantity, 43, 123-136.
Schoenman, J. A., Berk, M. L., Feldman, J. J., & Singer, A. (2003). Impact of differential response rates on the quality of data collected in the CTS physician survey.  Evaluation & the Health Professions, 26, 23-42.
Schultz, R. (2011). Refusal aversion training for CATI interviewers. Paper presented at International Field Directors and Technologies Conference, Scottsdale, AZ.
Schwarz, N., Strack, F., Hippler, H. J., & Bishop, G. (1991). The impact of administration mode on response effects in survey measurement. Applied Cognitive Psychology, 5,193,212.
Schwartz, L. K., Goble, L., & English, E. M. (2006).  Counterbalancing topic interest with cell quotas and incentives: Examining leverage-salience theory in the context of the poetry in the America Survey. Proceedings of the Survey Research Methods Section of the American Association for Public Opinion Research, 4229-4235.
Scott, A., Jeon, S. H., Joyce, C. M., Humphreys, J. S., Kalb, G., Witt, J., & Leahy, A. (2011). A randomised trial and economic evaluation of the effect of response mode on response rate, response bias, and item non-response in a survey of doctors. BMC Medical Research Methodology, 11, 126-150.
Shih, T. H. & Fan, X. (2007). Response rates and mode preferences in web-mail mixed-mode surveys: A meta-analysis. International Journal of Internet Science 2, 59-82.
Shuttles, C. D., Lavrakas, P. J., & Lai, J. (2004).  Getting respondents to stay on the phone: A large-scale experiment of combining an incentive with a pre-contact letter.  Paper presented at annual conference of the American Association of Public Opinion Research, Phoenix, AZ.
Shuttles, C. D., Welch, J. S., Hoover, J. B., & Lavrakas, P. J. (2002). The development and experimental testing of an innovative approach to training telephone interviewers to avoid refusals. Paper presented at the annual meeting of the American Association of Public Opinion Research, St. Petersburg Beach, Florida.
Shuttles, C. D., Welch, J. S., Hoover, J. B., & Lavrakas, P. J. (2003). Countering nonresponse through interviewer training: avoiding refusals training ART II.  Paper presented at annual conference of the American Association of Public Opinion Research.  Nashville, TN.
Shuttles, C. D., Skyrme, P. Y., Camayd-Freixas, E., Haskins, C. E., Wilkinson, D. L., Vallar, K. D., & Lavrakas, P. J. (2008). Hiring the right stuff:  Development of a skills and personality assessment system for hiring effective telephone research interviewers. Paper presented at Telephone Survey Methods II Conference, Miami, FL.
Singer, E., Frankel, M. R., & Glassman, M. B. (1983). The effect of interviewer characteristics and expectations on response. Public Opinion Quarterly, 47, 68-83.
Singer, E., Van Hoewyk, J., & Mayer, M. (1998). Does the payment of Incentives create expectation effects? Public Opinion Quarterly, 62, 152-164.
Singer, E., Van Hoewyk, J., Gebler, N., Raghunathan, T., & McGonagle, K. (1999). The effect of incentives in interviewer-mediated surveys. Journal of Official Statistics, 15, 217-230.
Singer, E., Van Hoewyk, J., & Mayer, M. (2000). Experiments with incentives in telephone surveys.  Public Opinion Quarterly, 64, 171-188
Singer, E. (2002). The use of incentives to reduce nonresponse in household surveys. Survey Nonresponse, 51, 163-177.
Shuttleworth, F. K. (1931). A study of questionnaire technique. Journal of Educational Psychology, 22(9), 652.
Sinibaldi, J., Jackle, A., Tipping, S., & Lynn, P. (2009). Interviewer characteristics, their doorstep behaviour, and survey co-operation. in an annual conference of the American Association of Public Opinion Research, Hollywood, FL.
Skalland, B., Wolter, K., Shin, H. C., & Blumberg, S. J. (2006). A non-response bias analysis to inform the use of incentives in multistage RDD telephone surveys.  Proceedings of the Survey Research Methods Section, American Statistical Association, 3705-3712.
Sloan, M. & Cidade, M. (2007).  A comparison of two methods of refusal conversion in an establishment survey. Paper presented at the annual conference of the American Association for Public Opinion Research, Anaheim, CA.
Smith, T. W. (1983). The hidden 25 percent: An analysis of nonresponse on the 1980 General Social Survey. Public Opinion Quarterly, 47, 386-404.
Snijkers, G., Hox, J., & de Leeuw, E. D. (1999). Interviewers' tactics for fighting survey nonresponse. Journal of Official Statistics, 15, 185-198.
Stec, J. A. & Lavrakas, P. J. (2007). Costs of refusals in large RDD national surveys.  Paper presented at annual conference of the American Association of Public Opinion Research, Anaheim, CA.
Stec, J. A., Lavrakas, P. J., Shuttles, C. D., Daily, G., Yancey, T., & Watkins, R. (2007).  Gaining efficiencies in scheduling callbacks in large RDD national surveys.  Paper presented at the annual conference of the American Association of Public Opinion Research Conference, Anaheim, CA.
Stokes, S .L. & Greenberg, B. S. (1990).  A priority system to improve callback success in telephone surveys. Proceedings of the Survey Research Methods Section of theAmerican Statistical Association, 742-7.
Stoop, I. A. L. (2005). The hunt for the last respondent. the hague: Social and cultural planning office of the Netherlands. Nonresponse. New York: Wiley.
Stoop, I., Billiet, J., Koch, A., & Fitzgerald, R. (2010). Improving survey response: Lessons learned from theEeuropean Social Survey. Chichester, West Sussex, UK: John Wiley and Sons.
Survey Research Center. (2010). Guidelines for best practice in cross-cultural surveys. Ann Arbor, MI: Survey Research Center, Institute for Social Research, University of Michigan.
Tarnai, J. & Moore, D. L. (2008). Measuring and improving telephone interviewer performance and productivity. In J. M. Lepkowski, C. Tucker, J. M. Brick, E. D. De Leeuw, L. Japec, P. J. Lavrakas, M. W. Link, and R. L. Sangster (Eds.), Advances in Telephone Survey Methodology, (pp. 359-384).  Hoboken, NJ: John Wiley & Sons, Inc.
Tourangeau, R., Rips, L. J., & Rasinski, K. A. (2000). The Psychology of Survey Response. New York: Cambridge University Press.
Tourangeau, R., Groves, R. M., & Redline, C.D. (2010). Sensitive topics and reluctant respondents: Demonstrating a link between nonresponse bias and measurement error. Public Opinion Quarterly74(3), 413-432.
Triplett, T., Blair, J., Hamilton, T., & Kang, Y. C. (1996, August). Initial cooperators vs. converted refusers: are there response behavior differences. In Proceedings of the Joint Statistical Meetings (pp. 1038-1041).
Triplett, T. (2002). What Is gained from additional call attempts & refusal conversion and what are the cost implications?  Urban Institute, November 2002.
Triplett, T., Scheib, J., & Blair, J. (2001).  How long should you wait before attempting to convert a telephone refusal?  Proceedings of the American Statistical Association,.
Trussell, N. & Lavrakas, P. J. (2004). The influence of incremental increases in token cash incentives on mail survey response: Is there an optimal amount?  Public Opinion Quarterly, 68(3), 349-367.
Tuckel, P. & Shukers, T. (1997). The effect of different introductions and answering machine messages on response rates..  Proceedings of the Section on Survey Research Methods of the American Statistical Association, 1047-1051.
Upchurch, H. L. & Battles, S. J. (2006). The effect of progressive incentives on response rates. Proceedings of the American Statistical Association section of the American Association for Public Opinion Research, , 4250-4253
U.S. Census Bureau. (2010). DP02 Selected social characteristics in the United States [Data]. 2010 American Community Survey. Boston, Massachusetts. Retrieved from http://factfinder2.census.gov
van der Vaart, W., Ongena, Y., Hoogendoorn, A., & Dijkstra, W. (2006). Do interviewers’ voice characteristics influence cooperation rates in telephone surveys? International Journal of Public Opinion Research, 18, 488-499.
van der Vaart, A. W. & van Zanten, J. H. (2005). Donsker theorems for diffusions: Necessary and sufficient conditions. Annals of Probability, 33, 1422-1451.
VanderWalde, A. (2005). Undue inducement: The only objection to Payment? American Journal of Bioethics, ,5(5), 25-27.
Voorpostel, M. & Ryser, V. A. (2011).  Mixed mode data collection as a strategy to decrease panel attrition in the Swiss household panel.  Working paper 2_11. Swiss Foundation For Research in Social Sciences.  http://aresoas.unil.ch/workingpapers/WP2_11.pdf
Voigt, L.F., Koespsell, T.D. & Daling, J.R. (2003). Characteristics of telephone survey respondents according to willingness to participate. American Journal of Epidemiology. 157(1), 66-73.
Warriner, K., Goyder, J., Gjertsen, H., Hohner, P., & McSpurren, K. (1996). Charities, no; lotteries, no; cash, yes: Main effects and interactions in a Canadian incentives experiment. Public Opinion Quarterly, 60, 542–62.
Weeks, M. F., Jones, B. L., Folsom, Jr., R. E., & Benrud, C. H. (1980). Optimal times to contact sample households.  Public Opinion Quarterly, 44, 101-14.
Weeks, M. F., Kulka, R. A., & Pierson, S. A. (1987). Optimal call scheduling for a telephone survey. Public Opinion Quarterly, 51(4), 540-9.
Weisberg, H. F., Krosnick, J. A., & Bowen, B. D. (1996). An Introduction to Survey Research, Polling, And Data Analysis. Thousand Oaks: Sage Publications.
Xu, M., Bates, B. J., & Schweitzer, J. C. (1993). The impact of messages on survey participation in answering machine households. Public Opinion Quarterly, 57, 232-237.
Yan, T., Tourangeau, R., & Arens, Z. (2004). When less is more: Are reluctant respondents poor reporters? Proceedingsof the Survey Research Methods Section of the American Stiatistcal Association, 4632-4651.
Yu, J. & Cooper, H. (1983). A quantitative review of research design effects on response rates to questionnaires. Journal of Marketing Research, 20, 36-44.
Yu, R. R., Liu, Y. S., & Yang, M. L. (2011). Does interviewer personality matter for survey outcomes? Evidence from a face-to-face panel study of Taiwan. Presented at World Association for Public Opinion Research Annual Conference, Amsterdam, The Netherlands.
Zukerberg, A., Hall, D., & Henly, M. (2007). Money can buy me love: Experiments to increase response through the use of monetary incentives.  Paper presented at the Joint Statistical Meetings of the American Statistical Association, Salt Lake City, UT.
 
[1] It should be stated that we use the term ‘refusal’ broadly as that is the practice of our profession. In truth many of the cases we refer to as initial refusals that might be targets for conversion efforts are actually cases where people have not explicitly refused or have not heard what the call is about before contact is broken. We do not advocate attempting to convert clear refusals where a fully informed individual says they will not participate.
[2] Survey and opinion research is not covered by the Federal Trade Commission “Do Not Call” rules that govern telemarketing. However, as discussed later in this report, many survey organizations maintain internal do not call lists.
[3] Indeed, some find that over half of all telephone initial refusals are cases where the potential respondent does nothing other than initially say “hello” and then silently hangs up (Dutwin and Herrmann, 2006).
[4] However, even with the straightforward cases mentioned above, some would argue that if the respondent was (in the interviewer’s judgment) intoxicated or impaired it should not be considered a refusal because an impaired person cannot give ‘informed’ consent (Gwartney, 2007). Likewise, we do not believe it would be ethical to accept agreement to participate from a temporarily or permanently impaired person.
[5] The AAPOR Standard Definitions note that there are some households in which there is no adult ages 18 and older, and therefore in a study of adults, there is no eligible household member.  However, the Standard Definitions make clear that such households are so few that one should assume that every household is valid for purposes of categorizing refusals and calculating response rates.
[6] AAPOR’s Standard Definitions encourages researchers after the field period is completed to estimate the proportion of such refusals that are likely to be something other than a refusal (i.e., calculating “e” for these outcomes) and to (1) take this into account when computing response rates and (2) disclose the basis for the magnitude of e that they applied.
[7] The Standard Definitions state that for mail surveys of specifically named persons that “entirely blank questionnaires are sometimes mailed back in the return envelope without any explanation as to why the questionnaire was returned blank. Unless there is good reason to do otherwise, this should be treated as an ‘implicit refusal’” (Standard Definitions, 7th edition, 2011, p. 27). 
[8] In the case of mail surveys, it is possible to match information about the housing unit to the addresses of refusers which may be valuable in helping researchers to understand and deal with refusing cases.
[9] Lavrakas (1993) appears to have been the first to formally describe the use and value of such a form and to show an example of it in his book on telephone survey methods.
[10] In in-person, mail, and telephone surveys, there is a surprising amount of useful information that can be matched to sampled telephone numbers and addresses, that survey researchers can append to their initially designated sample of cases to potentially help in tailoring the approach taken to gain cooperation at the level of the individual.
[11] Of the one study of which we are aware (Lavrakas, 2001), in a national RDD study it was found that when asked a long series of reasons about why they might not want to participate in a survey, one of the four factors that explained the various reasons for not wanting to participant concerned not wanting/liking to express one’s opinion because of a lack of self-esteem that one’s opinions were worth expressing.
[12] http://www.whitehouse.gov/sites/default/files/omb/inforeg/statpolicy/standards_stat_surveys.pdf
[13] See intra section on Refusal Conversion and Nonresponse Error.
[14] These total costs are the direct costs associated with effort expended as well as indirect costs related to the misclassification of refusals as callbacks and the loss of productivity associated with lower interviewer morale.
[15] However, to the extent that survey participation is driven by multiple influences, besides the survey introduction, e.g., mixed modes, incentives, etc., survey participants will likely be more varied thereby making nonresponse bias less of an issue (Groves et al., 2004b). 
[16] This low rate of breakoffs has changed considerably with regards to internet questionnaires that are started but not completed.
[17] Other methods to try to avoid refusals are sending advanced notification about the survey (possibly including a non-contingent incentive or mention of a contingent incentive), leaving messages or other survey information, and developing special interviewer training.  These other methods of refusal aversion are addressed later in this section.
[18] Mayer and O’Brien (2001) noted that questions asked by respondents can be quite beneficial in the context of eliciting survey participation because it shows the respondent is motivated and attending to the conversation; it allows the conversation to continue; and it provides the interviewer information to use to tailor the interaction.
[19] In self-administered mail and internet surveys, this type of information can be provided to respondents at the time they are recruited in the form of a series of answers to FAQs; or this information can be posted on a webpage to which interviewers can refer respondents.
[20] However, more research is needed, as in current times so many persons are screening their telephone calls or letting them ring through to voice mail and answering machines, that it is reasonable to hypothesizes that if a survey research organization’s name appears on Caller ID or in the substance of a message left on voice mail, that may increase the proclivity of some people to answer their phone the next time they see the organization’s name on their Caller ID.
[21] Of note, few of these meta-analyses reviewed the considerable research on incentive effects that has been reported in conference papers during the past decade and beyond.  This report contains as much of that literature as we could locate.
[22] Conscientiousness is comprised of subscales measuring trustworthiness, being detail-orientated, being driven, and being highly organized. 
[23] Emotional Stability is comprised of subscales measuring the amount of anxiety, insecurity, despondency, and belligerence one has.
[24] At an aggregate level, Holbrook, Krosnick and Pfent (2008) found that longer RDD surveys have lower cooperation rates, but found no association between interview length and refusal rates (they did not examine refusal conversion rates).
[25] See the section on “What is a Refusal” for more details about hard and soft refusals.
[26] While one of the authors of this report (Lavrakas) was at Nielsen Media Research in the mid-2000s, an internal investigation using millions of RDD landline call attempts indicated that 72 hours was the minimum elapsed time before a refusal conversion should be attempted.
[27] Triplett (2002) notes that the University of Maryland’s Survey Research Center used a callback rule of thumb for refusal conversions of at least one week.  The exception to this rule of thumb was if the refusals occurred near the end of a field period so that there was not enough remaining time to wait one week prior to attempting refusal conversions.  To the extent that there are other factors that would impact refusal conversions success rates, other than the time elapsed since the initial refusals, which occur at the end of a survey field period, these factors could explain, at least in part, the differential success rates found by Triplett.
[28] See http://www.copafs.org/seminars/use_of_incentives_five_questions.aspx for a discussion of researcher concerns and Groves et al. (1999) for respondent attitudes toward differential incentives.
[29] At the time of this writing, New Hampshire was considering revision of this statute to more precisely define push polling and to exempt “bona fide survey and opinion research” which would specifically include message testing.
[30] See: (http://www.aapor.org/Report_on_the_Status_of_Human_Subjects_Protection_Training_Requirements1.htm)
[31] The exception for these “clear” but not “adamant” initial refusals would be if the project plans include offering incentives to people who refuse which gives the respondent additional information to consider in deciding whether or not to participate.
[32] For both mail and web surveys, a best practice could be to provide a method (e.g., toll-free number, email address, etc.) for respondents to contact the survey researcher with questions or concerns.