Costs

Previous: Operational Issues
in RDD Cell Phone Surveys
 

 Table of Contents

 Next: Conclusions and Disclosure
Recommendations

COSTS IN RDD CELL PHONE SURVEYS

It has been clear from the first studies with RDD cell phone surveying in the U.S. that in most cases its cost is substantially greater than the cost of similar surveying of persons sampled and interviewed via landline RDD. During the past decade, the survey industry has accumulated sufficient experience with RDD cell phone surveys that it is now possible to provide a reasonably reliable assessment of the magnitude of the cost differential between RDD cell phone and RDD landline surveying and to gain a reasonable, albeit preliminary, insight into the factors that drive RDD cell phone interviewing costs. 

For this report, specific cost data from more than 30 dual frame RDD surveys were gathered from four academic and four commercial survey organizations. Using these data, analyses were conducted comparing RDD cell phone and RDD landline costs in dual frame surveys where each type of sample was pursued under otherwise similar constraints and conditions. In this section, general cost factors that create a cost differential between RDD and cell phone interviewing are first considered. Then a review is provided of the data that were gathered to assess average cost differentials and to identify some of the conditions that cause the cost differential to be lesser or greater in different research applications.

 

Factors that Create Cost Differentials between RDD Landline and RDD Cell Phone Surveying

If one calculates the cost per completion for the cell phone part of an RDD survey and divides that by the cost per completion of the landline RDD arm, the resulting quotient yields the cost ratio of cell phone to landline RDD data collection. This cost ratio is affected by several factors, including: (1) interviewer time; (2) the cost of sample numbers; (3) the use of remuneration and/or incentives; and (4) possible mailings sent to respondents. Some of these factors operate on cost by affecting interview productivity, whereas others operate more directly on total costs.

Additional Interviewer Time Differentially Increases Cell Phone Survey Costs.The primary reason for the greater cost of RDD cell phone surveying compared to RDD landline surveying is the lower productivity of cell phone sampling/interviewing. Simply put, it takes more hours of interviewer time to achieve a given number of completions from an RDD cell phone sample than it does from a traditional list-assisted RDD sample drawn from working blocks of phone numbers assigned to landlines. Interviewing time is the fundamental cost unit of telephone survey budgets: an hour of an interviewer's time in the CATI center entails not only that employee's wages and benefits, but a portion of the time of the supervisors, an hour of phone dialing and accrual of telephone charges, and other infrastructure overhead.1 

Below, some of the factors that contribute to the lower hourly productivity of cell phone interviewing are considered, but first attention is given to some other cost components.

Cost of Sample Numbers Differentially Increases Cell Phone Survey Costs. Irrespective of the interviewing hours a survey requires, the sampled phone numbers are acquired at a cost, usually purchased at a fixed per-number rate from a commercial sampling vendor. As discussed in the Nonresponse section of this report, RDD cell phone calling is generally less productive than calling RDD landline numbers, for various reasons including the screening often needed to determine respondent eligibility. Thus considerably more numbers must be purchased and attempted to achieve a given number of completed cell phone interviews. 

If RDD cell phone and RDD landline numbers are purchased at the same price, the cost for a cell phone RDD designated sample will be correspondingly higher than the cost for a landline RDD designated sample. However, since there is no effective prescreening service available for U.S. cell phone numbers, cell phone samples in the U.S. do not carry the additional charges associated with automated pre-screening for business and nonworking numbers.2 Of course, this inability to prescreen cell phone samples greatly increases the total size of the sample of RDD cell phone numbers a given survey must process manually with its interviewers, which in turn further increases total costs for processing the cell phone sample.

Remuneration/Incentives Differentially Increase Cell Phone Survey Costs. Another set of costs apart from telephone interviewing costs are the costs of cash or the other remuneration and/or incentives offered to respondents, either prior to or after a cell phone interview completion, as well as the costs of other triggers that may be used to encourage response, such as advance letters or refusal-conversion mailings, which are possible with landline samples. 

It is fairly common in dual frame RDD telephone surveys to offer a small ($5 or $10) cash or gift-card remuneration to cell phone respondents, usually conditional upon interview completion. The ethical rationale for these remunerations, which traditionally have not been offered to landline RDD respondents, is that they serve to “compensate†a respondent for charges incurred as a result of using the cell phone to complete the interview. (See more discussion in the Legal and Ethical Issues and the Operational Issues sections of this report.) 

The offering of these monetary gifts may increase rates of response in cell phone interviewing (Brick et al., 2007; Diop et al., 2008; and Diop et al., 2008), but the effects reported in the research literature have not been consistently positive (Pew, 2008; Oldendick and Lambries, 2010). Monetary gifts may be more effective in promoting survey participation among cell-phone-only respondents because some of these persons are thought to be more “cost-conscious†and are likely to fit the demographic groups (e.g., young, unattached, lower income, renters) for which gifts of material value tends to carry more “leverage and salience†than other, less tangible rewards of participating in a survey (cf. Groves, Singer and Corning, 2000). However, other cell-only persons likely have calling plans with unlimited or large amounts of monthly minutes, and may not be affected by offers of remuneration. 

Furthermore, when cash or gift cards are sent to respondents, there are additional postal and administrative processing costs. Since the remuneration is given to cell phone respondents and not to the landline RDD respondents, they increase the cost differential between the two types of telephone surveying.

Possible Mailings Differentially Increase Landline Survey Costs. Advance letters and refusal-conversion letters, whether mailed with or without a token cash incentive, can be sent only to those for whom addresses are known. Therefore, they cannot be sent in advance to the vast majority of cell phone RDD cases, nor can they be sent to those landline RDD cases that have telephone numbers that fail to match to the databases used by matching vendors. If these postal communications are used in a survey for the matched landline RDD cases, the per-completion cost of the landline RDD side of a dual-frame survey is raised accordingly, which serves to somewhat diminish the cost differential between cell phone RDD and landline RDD interviewing.

 

Factors That Affect the Differential in Hourly Production Rates for Cell Phone RDD Surveys Compared to Landline RDD Surveys

As noted above, the biggest factors affecting the cost ratio in dual frame surveys are those that affect interview productivity. The hourly productivity (i.e., Hours per Completion or HPC) for any telephone sample is a product of the following factors:

  • Working number rate,
  • Contact rate,
  • Eligibility rate,
  • Cooperation rate,
  • Interview length, and
  • Dialing method.

Each of these may differ between RDD cell phone and RDD landline samples, most often in ways that yield lower productivity on the cell phone side.

Working Number Rate. The working number rate is a function of working number density in the number blocks from which sample is drawn. The differential in the working number rate is dependent on the relative density of working numbers within the cell phone and landline exchanges in use in a given sampling area; and these working-density ratios may vary from one area to another.

In most cases, list-assisted landline RDD samples select only "working" number blocks into the sampling frame, resulting in greater efficiency. In contrast, no such selection is possible for cell phone RDD samples because there are no publicly available directories or other sources listing cell phone subscribers. In addition, landline RDD samples can be prescreened to eliminate nonworking and business numbers, yielding a significant gain in calling efficiency, but this prescreening is not possible for RDD cell phone numbers in the U.S. The overall result is a significantly lower working number rate for cell phone RDD samples.

Contact Rate. The contact rate is affected by cultural and technical differences in how people in the U.S. use cell phones as contrasted with how they use household landline phones. Many people use cell phones as a supplementary communication device. Many cell phones that interviewers call are turned off when called. Call screening/Caller ID technology is essentially universal on cell phones, as is voice mail; and both are thought to promote greater screening of incoming calls by respondents. The result may be a lower live contact rate for the cell phones numbers that are sampled. Furthermore, the contact rate is in part a result of the calling effort, and not all telephone surveys will apply equal effort to both sides of a dual frame survey. If the cell phone effort is lower (e.g., fewer call attempts per sampled cell phone number), then the size of cost differential would be reduced.

Eligibility Rate.These rates are lower in cell phone samples for several reasons and have major effects on survey costs for cell phone sampling:

1. Not being eligible due to age. Many more minors (persons under age 18) have their own cell phone compared to minors who have their own landline. Minors usually are not eligible for telephone interviewing in general population surveys, and they cannot as easily "hand off" the call to an eligible adult when reached on a cell phone, compared to what happens when ineligible young people are reached via a landline household telephone; nor is it often appropriate, due to the specifics of a cell phone survey sampling design, to have a minor hand off a cell phone to an adult.

2. Not being eligible due to geography. Cell phones can be purchased in one location and used in another, and often are. Thus, in non-national telephone surveys in the U.S., there are people reached by cell phone who reside outside the survey area. (This can occur also with landline phones, due to number portability, but currently this is not as prevalent as it is with cell phones.) For local area samples such as counties or cities, landline RDD can approximate the geographic area far more accurately and efficiently than cell phone RDD, which must rely on telephone company rate centers as the sampling unit for localized sampling. Thus, the cost differential between the cell phone part of the sample and the landline part due to these eligibility rate differences usually will be greater when a dual frame survey is conducted in the U.S. within a non-national geography.

3. Not being eligible due to type of telephone service screening. A major factor in the eligibility rate differential between cell and landline samples is the type of dual frame design chosen by the researcher. Dual frame surveys that screen cell phone respondents and interview only those who are cell phone only will have considerably lower rates of eligibility than “all-cell†designs. If cell phone mostly respondents also are included as eligible, this will reduce the cost differential of this type of screening design, but nevertheless the cell phone component will remain more expensive than the landline component due at least in part to this screening. 

4. Not being eligible for other reasons. Other study-specific screening procedures also may create differences in eligibility rates between the landline and cell phone parts of a dual frame survey. For example, a survey seeking young Hispanic males likely will reach more eligible cases on the cell phone side than on landlines; the opposite would be true for a survey of married female retirees. Many dual frame surveys use methods for random selection within the household on the landline side, but omit these procedures on the cell phone side, thereby accepting whoever answers a cell phone as the eligible respondent. This difference in determining who is eligible reduces the productivity differential somewhat between cell phone and landline RDD. 

In sum, in most U.S. dual frame designs the cell phone sampling will have a lower overall eligibility rate, resulting in more time spent on screening and recruiting for the cell phone side, a higher HPC, and a larger cost ratio.

Cooperation Rate. The cooperation ratemay or may not be different for cell phone and landline RDD surveying. As noted in the section of this report on Nonresponse, recent experience suggests less difference in the cooperation rates than had been experienced earlier in the previous decade when cell phone interviewing was beginning to be deployed in the U.S. If a monetary gift is offered to cell phone respondents and not to landline respondents, that can increase the cell phone cooperation rate and thus reduce the productivity differential. On the other hand, if mailed inducements such as advance letters, token cash incentives, or refusal conversion letters are used on the landline side it will differentially increase productivity on the landline side, and thereby increasing the productivity differential. 

Interview Length. If the length of the survey questionnaire differs between cell phone and landline questionnaires, this will have a differential influence on productivity for each sample. Yet, even if the two questionnaires are otherwise identical, cell phone interviews, on average, will take a minute or so longer due to the extra questions that may be needed about telephone service and usage, and time to gather information for distribution of any remuneration and/or incentive. 

However, some studies use a shortened cell phone interview – in particular, some states participating in the U.S. Centers for Disease Control and Prevention's 2009 cell phone trials for BRFSS have chosen to use shortened interviews on the cell phone side. This may have the effect of increasing the cell phone response rate compared to what it would be with a longer interview. However, with topics that are interesting (e.g., health) and when conducted for "the public good," cell phone surveys with interviews as long as 30 to 35 minutes have been found to not suffer in their response rate (cf. Brick et al., 2007). If response rate is unaffected by the length of the interview and if the cell phone questionnaire is shortened, this will reduce the cost differential between the cell phone and landline surveys.

Dialing Method. The method of dialing that isused affects productivity. The required manual dialing of cell phones in the U.S. slows down the interviewing process and contributes to the size of the productivity differential. However, the degree of impact this has will depend on the dialing method used on the landline RDD side. Some telephone survey call centers (especially academic survey organizations) use autodialers to call numbers one by one while interviewers listen to the calls ringing. This process is faster than manual dialing, but certainly slower than a predictive dialer for RDD landline sample that “finds†a potential respondent on the line and serves the connection up to the next "available" interviewer. The average dialing times for cell phones also will differ if interviewers are instructed to let them ring longer (e.g., at least eight rings) before coding a RNA compared to landline dialing (e.g., at least six rings), or if cell phone dialing requires the interviewer to more often spend time leaving messages on the respondent's voice mail.

Summary of Cost Factors. The cost per completion for either part of a dual frame RDD survey (cell phone or landline) can be thought of as a sum of:

  • The cost per completion of remuneration/incentives and/or advance mailings, if any;
  • The cost per completion of the purchased sample phone numbers; and
  • The interviewing costs per completion. 

This third term is by far the largest factor in most telephone surveys and can be calculated as a product of the billing rate (or full cost) for an interview hour and the HPC. HPC can, in turn, be thought of as a sum of the interview length and the hours spent (per completion) on screening and recruiting – i.e., all interviewer time that is not devoted directly to completing the interview.

It is this last cost component "Screening and Recruiting Hours per Completion (SRHPC)" that is markedly higher for cell phone interviewing. Any differentials in the productivity factors listed above (e.g., working number rate, contact rate, eligibility rate, or cooperation rate) have a direct, multiplicative effect on the ratio of SRHPC in cell phone interviewing to SRHPC in landline interviewing. As is shown by the data presented below, the SRHPC ratio (cell phone SRHPC divided by landline SRHPC) drives the HPC ratio and the overall cost ratio as well.

 

Methodology for Gathering Cost and Productivity Data from Recent Dual Frame Surveys

During late 2009, a survey was conducted by the Cost subcommittee of the AAPOR Cell Phone Task Force to gather data from a select group of telephone survey organizations in the U.S. (Guterbock, Lavrakas, Tompson and ZuWallack, 2010). This survey used a purposive nonprobability sample of eight nationally known survey organizations (four commercial and four academic).3 Telephone interviews were conducted by members of the Cost subcommittee with a senior researcher at each organization who was knowledgeable about the cost information that was to be gathered. Each of those senior researchers then had a spreadsheet assembled containing available information about each of the dual telephone frame surveys that the organization had conducted. These spreadsheets were shared in confidence with the Cost subcommittee members.

Information was provided about 38 separate dual frame RDD surveys. These surveys represented a mix of national, state and local surveys. The type of information that was gathered about the RDD cell and RDD landline samples in these surveys included: (1) number of completions, (2) average length of a completion in minutes, (3) geography covered, (4) screening criteria, (5) completes per hour (CPH), (6) incentive amounts, and (7) cost per interview (CPI). These data were used to generate various ratios for the analyses reported below.

Although it is acknowledged that the findings from this survey may not be representative of all recent dual frame RDD telephone surveying in the U.S., it appears to be the first time such cost data have been gathered from a wide set of survey organizations and about a relatively large number of dual frame telephone surveys. Thus, it is believed that the findings will do much more to inform, than misinform, the reader about the relative costs of cell phone RDD surveying in the U.S. compared to landline RDD surveying. Nevertheless, the reader is cautioned not to place an undue amount of importance on these findings until findings from a much larger and more representative cost survey become available. 

 

Productivity and Cost Ratios in Current Dual Frame RDD Telephone Surveys in the U.S.

The cost and productivity data gathered about the 38 dual frame RDD surveys support the conclusion that RDD cell phone surveying achieves lower productivity than RDD landline surveying. 

As shown in Table 3, the ratio of time devoted to screening and recruiting respondents (i.e., the SRHPC ratio) averages 2.5 times higher for the RDD cell phone samples than for RDD landline samples across the 26 surveys for which these data were available. No survey reported better productivity on the cell phone side compared to the landline side (i.e., this would be an SRHPC of less than 1.0); and the lowest ratio was 1.2. The standard deviation of 1.0 indicates that, if the sample is representative, about two-thirds of dual frame RDD surveys would have SRHPC ratios between 1.8 and 3.8. The maximum SRHPC ratio among the 26 surveys shown in Table 3 was 5.4.4 

Table 3
Productivity Statistics and Cost Ratios for                                          Dual Frame RDD Surveys

 

Screening and Recruiting Hours per Completion

Hours per Completion

Overall Cost per Interview

Ratio (cell/landline)

SRHPC Ratio

HPC Ratio

Cost Ratio

Mean

2.53

2.00

2.05

Minimum

1.21

1.17

1.35

Maximum

5.37

3.47

3.97

Std. deviation

1.02

0.63

0.77

N

26

26

20

The differential in hours per completion (the HPC ratio) takes into account the time devoted to the actual interview. The productivity differential as measured by HPC lessens somewhat compared to SRHPC, with an average HPC ratio of 2.0. 

Thus, on average across all the surveys, cell phone RDD surveying took twice as long per completion as the RDD landline; i.e., the completions per hour in the cell phone surveys came in at half the rate of the landline surveys. 

The cost ratios in Table 3 take into account the cost of the phone number sample and cash incentives, generally used on the cell phone side only. With these added per-complete cost increments (small for most surveys), the average cost ratio rises slightly to 2.1.

Effects of a Cell Phone Only Design. As noted above, eligibility rates in the cell phone part are notably lower if a dual-frame design requires screening for cell phone only cases and dropping those reached via the cell phone sample that have dual phone service. As shown in Table 4, all three ratios (SRHPC, HPC, and overall cost) are substantially lower when there is no screening for type of telephone service and all cell phones are considered to be qualified for inclusion on this factor. The SRHPC ratio for cell-only surveys was 3.0, contrasting with 2.3 for all-cell surveys. The average HPC ratio was 2.4 for the cell phone side in cell-only survey compared to the companion landline survey, but 1.9 in all-cell surveys. The overall cost ratios varied in similar fashion, with the cell phone completions being roughly double the cost of landline completions in all-cell surveys, whereas they were 2.6 times the cost in cell-only surveys.

Table 4
Means for All-Cell (No Screening) versus Cell Phone Only Surveys

Type of Dual Frame Design

 

Screening and Recruiting Hours per Completion

Hours per Completion

Overall Cost per Interview

 

 

SRHPC Ratio

HPC Ratio

Cost Ratio

All-cell designs

Mean

2.38

1.87

1.96

 

N

20

20

17

 

Cell phone only designs

Mean

3.03

2.45

2.60

 

N

6

6

3

Cost Effects of the Geographic Area Covered. As previously discussed, the survey eligibility rate also is affected by the geographical location of the target population. When the study area is the entire nation, people who purchase their phone in one state and move to another remain eligible, at least from a geographic standpoint. When the survey geography is an entire state, then the eligible phone exchanges can be identified readily by area code alone. However, when the study area is a county, metro area, or another location not coincident with area code or prefix boundaries, then both landline and cell phone samples are less efficient for reaching persons in the defined study area. However, the decrease in efficiency is greater for the cell phone RDD samples than for the list-assisted landline RDD samples. 

Table 5 compares state and national dual frame surveys, on the one hand, and local dual frame surveys on the other, while still separating cell-only surveys from all-cell surveys. The differences in the productivity and cost ratios are relatively small, but are in the expected directions: i.e., local all-cell surveys have higher ratios than state or national all-cell surveys. The one survey that undertook cell-only calling in a local area experienced a cost ratio of 4 to 1, indicating clearly the difficulty of finding qualified respondents on the cell phone side under such a sampling design.

Table 5
Means for Local Surveys versus National/State Surveys, by Design Type

Dual Frame Design

Geography

 

Screening and Recruiting Hours per Completion

Hours per Completion

Overall Cost per Interview

 

 

 

SRHPC Ratio

HPC Ratio

Cost Ratio

All-cell designs

National & statewide

Mean

2.36

1.84

1.86

 

 

N

14

14

12

 

Local

Mean

2.42

1.92

2.19

 

 

N

6

6

5

 

Cell phone only designs

National & statewide

Mean

3.03

2.45

1.92

 

 

N

6

6

2

 

Local

Mean

--

--

3.97

 

 

N

0

0

1

Summary of Cost Issues in U.S. RDD Cell Phone Surveys

The survey industry in the United States now has had sufficient experience with dual frame telephone surveys combining RDD cell phone and RDD landline for production and cost ratios to be estimated from empirical data. Based on reports from a heterogeneous set of more than 30 recent dual frame RDD surveys, it can now be said with some confidence that in a typical dual frame survey using an “all-cell†design (i.e., without screening for type of telephone service), that cell phone RDD interviewing will be, on average, about half as productive (per hour) as the landline RDD interviewing, and hence about double in cost per interview. If a cell phone only design is used as the cell phone component in a dual frame design, the cell phone completions will cost, on average, approximately two-and-a-half times more than the landline completions. 

Various features of a dual frame RDD survey, especially those that impact the working number rate, contact rate, eligibility rate, cooperation rate, type of dialing that can be deployed, use of advance mailings, or interview length, may cause the cost ratios between the cell phone and landline samples in that survey to vary from the ratios reported above. In particular, dual frame surveys in local areas will experience higher productivity and cost ratios than those using statewide or national sampling frames. 

Putting Costs into Their Proper Context. The overall cost increment for converting a landline RDD survey to a dual frame RDD survey that includes cell phone interviewing will depend, of course, on the number of interviews that are attempted by cell phone. If a greater proportion of the interviewing effort is allocated to the cell phone RDD frame, then the overall ratio of the dual frame survey’s cost (relative to the cost of a landline-only RDD design) will be higher. As has been discussed in the Coverage and Sampling section of this report, there is no current agreement on the optimal allocation of the final sample between landline RDD and cell phone RDD samples. 

Readers should keep in mind that the cost of cell phone sampling must be considered within the context of optimizing the dual frame design. RDD cell phone samples in the U.S. are undertaken to improve coverage and to bring into the final sample proportionally members of groups that would (increasingly) be underrepresented in a RDD landline-only design; thereby enhancing the survey’s face validity, as well as providing a more representative unweighted final sample. However, if a researcher chooses to minimize the number of cell phone completions out of a desire to minimize field costs, the survey may pay a penalty in statistical precision, because the realized sample will need to be adjusted with fairly large post-stratification weights. With a large design effect, the effective sample size may be reduced to the point where more completions are needed to achieve the desired level of precision. Interested readers should consult Benford, Tompson, Fleury, Feinberg, Feinberg, Speulda and Weber (2009) for more details on these issues and a discussion of the costs of various dual frame designs measured against the resulting effective sample sizes. (Appendix C of this report contains an updated summary presentation of this work by Benford and his colleagues.)

 

Electronic White Pages Sampling and Telephone Survey Costs

One intriguing alternative that is being explored by some U.S. researchers is the possibility of substituting directory-listed landline phone numbers (also called Electronic White Pages or EWP sample) for some or all of the landline RDD sample in a dual (landline and cell phone) frame design. It is well known that EWP landline samples underrepresent certain demographic groups, but by fortuitous circumstances these generally are the same groups that are fairly easily reached via a cell phone sample, so they appear to be fairly well covered in a dual frame design that combines cell phone and EWP landline samples.

A dual frame study with a EWP landline sample and a cell phone RDD sample (without screening) would cover all telephone households with the exception of those that have an unlisted landline and no cell phone available.An analysis of NHIS data by Guterbock and colleagues (forthcoming) suggests that coverage error from exclusion of the “unlisted-landline-only†households is likely to be very small for most survey purposes. Since interviewing from a EWP landline sample is far more cost-efficient than is interviewing from a landline RDD sample, the savings from substituting EWP landline sample for landline RDD sample can substantially offset the incremental cost of including the cell phone RDD sample. A series of recent surveys by the University of Virginia Center for Survey Research has been testing this approach in “triple-frame†studies that combine landline RDD, EWP landline, and cell phone RDD samples, with promising results.5 In these surveys, only 2 percent or fewer of interviewed telephone households in the combined and weighted landline RDD and cell phone RDD samples report themselves to be in the “unlisted-landline-only†segment.

Previous: Operational Issues
in RDD Cell Phone Surveys

 Table of Contents

 Next: Conclusions and Disclosure
Recommendations


1 The fixed charges of running a CATI research facility typically are billed back to clients/sponsors as an overhead charge applied to the hourly interviewing rate. Any factor that increases the Hours per Completion (HPC) "and correspondingly lowers the Completes per Hour (CPH)" will have a large direct effect on the overall study cost. These costs are nearly always the major portion of data collection costs in the production phase of an RDD telephone survey. 

2 Thanks to Frank Markowitz of SSI for clarifying sample cost issues in a personal communication.

3 The survey organizations were promised their names would not be disclosed.

4 Two studies that reported even higher ratios were excluded as "too extreme" outliers.

5 This three-sample approach affords some cost savings over the more usual dual frame design, while not fully abandoning the more expensive, traditional landline RDD sampling frame. This design has allowed direct comparison of survey estimates drawn from combining the EWP and cell phone samples with those obtained from combining landline RDD and cell phone samples (Guterbock et al., 2009).