AAPOR
The leading association
of public opinion and
survey research professionals
American Association for Public Opinion Research

Appendix C

 Table of Contents

Considerations for Sample Design, Estimates, Weighting and Costs1

 

Dual frame RDD landline and RDD cell sample designs, whether overlapping or screened for cell phone only (CPO) have two types of costs associated with them:

  • Financial costs, that is, the sum of the costs of landline interviews plus the sum of the costs of the cell phone interviews; and
  • Statistical “cost” in the precision of estimates due to the effects of weighting. 

This Appendix is based on the work of Benford et al. (2009), using the results of polls conducted by GfK for the Associated Press (AP) – the AP-GfK Poll. It is presented here as documentation of a sample cost allocation model when using dual frame landline RDD and cell phone RDD surveys. The purpose of the Appendix is to illustrate the implications of sampling design decisions concerning the proportion of final sample that comes from each RDD frame when predetermined levels of precision are important.

 

Cost Allocation Model

Conventional thinking depicts the cost of completing interviews via a cell phone frame at some ratio to the cost of a landline interview. For example, the cost per interview (CPI) by cell phone might be two times that of the landline CPI. But it needs to be recognized that this ratio of costs varies. For example, a 10-minute interview might be twice the CPI in the cell frame versus the landline CPI but this ratio is generally smaller for a 15-minute interview. This is because the costs of cell phone interviews are incremental in nature.

Assuming that the core questionnaire content is the same regardless of frame, incremental costs are found in the cost of additional screening questions necessary for those contacted on a cell phone; and the cost of additional questions that are needed to gather information for the distribution of a reimbursement, the cost of the reimbursement, and the other cost differentials in the cell phone versus landline samples. The choice to screen for CPO persons is an additional incremental cost. Contact rates, cooperation and other sample disposition rates differ by cell frame and landline as well, and these may be thought of in terms of a ratio. Thus, a cost model that covers variants in interview length or population members is complex with core fixed costs, incremental costs assigned to the cell frame interviews, and variable costs between the two sample types. However, the ratio in one design to another makes understanding sample frame design decisions easier to comprehend. 

As an example, AP-GfK polls typically average about 15 minutes for the landline interview. Additional costs associated with cell phone completions result in a ratio of 1.8:1, cell phone CPI to landline CPI. Further, we also estimated the incremental costs of screening for CPO persons and get a ratio of 2.9:1. These ratios lend themselves to allocation models for these two types of designs.

If we start with a landline RDD interview, the cost can be expressed as the final sample size times the cost of obtaining each interview with n as the final sample size and C as the cost per interview:

            nlandline * Clandline

Costs for obtaining interviews from a cell phone RDD frame are higher due to higher sample cost, manual dialing, asking additional questions to ascertain safety and age, and offering a re-imbursement along with collecting the relevant information and processing reimbursements.2 Our experience indicates that the cost of a cell phone interview is a little less than double a landline interview. Conceptually then the total cost of the dual frame design is:

(nlandline * Clandline) + (nCell phone * 1.8Clandline)

Although this might be useful to compute the total cost, another approach is to understand the difference in cost from a base of landline cost. Total cost then is dependent on the allocation of sample3. To understand the relative cost of our design decision we substitute n for each sample frame with the portion of sample allocation. This is A for landline and 1-A for cell frame:

((A)(Clandline)) + ((1-A)(1.8Clandline))

If A = 1 then the cost is entirely that of a landline sample and, conversely, if A = 0 then the cost is entirely that of a cell phone sample. 

In 2010, AP-GfK polls are allocated as 70 percent landline and 30 percent cell phone. This dual frame design then is 24 percent more expensive than landline only sample of similar size would cost. This is shown by:

(0.70Clandline) + (0.30) * (1.8Clandline) = 1.24Clandline

Similarly, if we screen for CPO, we estimate that the CPO design is approximately three times the cost of a landline interview.4 If we set CPO at 13 percent of the total final sample, then the cost of this design decision is 25 percent greater than a landline only sample of similar size:

(0.87Clandline) + ((0.13) 2.9Clandline) = 1.25Clandline

A 50 percent landline, 50 percent cell phone frame design without screening is 40 percent more than a similar-sized landline only design:

(0.50Clandline) + ((0.50) 1.8Clandline) = 1.4Clandline

A comparable coverage solution to the 50/50 dual frame is to sample CPO proportionate to population estimates at 21.6 percent5 which is:

(0.784Clandline) + ((0.216) 2.9Clandline) = 1.41Clandline

This may be a useful way to understand the relative cost of sample design decisions in contrast to traditional landline designs. 

However, these decisions may also need to be put in the context of efficiency, as shown in the Table C-1. This table, based on six AP-GfK polls in 2010 with dual frame overlapping sample design, is an updated version of Table 7, in Benford et al. (2009). 

Table C-1
Comparison of Efficiencies, Relative Costs, Effective Sample (n=1,0006 )

 

Efficiency7

Relative Cost to Landline

Effective
Sample
Size

Ratio of Effective Sample to Landline

Weighted Cost for Effective n=1,000

Landline only

0.456

1.00

456

1.00

2.15

CPO w/landline

0.514

1.25

514

1.12

2.43

Dual 70/30

0.491

1.24

491

1.08

2.53

Dual 50/50

0.605

1.40

605

1.35

2.31

CPOproportional

0.522

1.41

522

1.18

2.70

The updated table shows the importance of accounting for precision needs or tolerances around estimates based on design decisions and when weights are used to approximate unbiased estimates. Although landline samples are the least expensive, they result in the least amount of effective sample size.8 

If, for example, precision is needed at +/-3.1 percent at the 95 percent confidence level, then effectively, a final sample of 1,000 is needed. Aside from dampening the variability through trimming weights to lift the effective sample size, which would affect each design comparably, collecting a larger unweighted sample to achieve the desired effective sample size can be considered. The last column on the right in the table shows the adjusted costs to achieve this goal, computed as 1,000 divided by the effective sample size times the relative cost to landline. Although the landline, with all its coverage issues, is still the least expensive, the “best buy” is a dual 50/50 design given that design has the next lowest weighted cost to achieve an effective
n = 1,000.

 Table of Contents


1 The Task Force thanks Robert Benford of GfK Custom Research North America for contributing this updated summary of the cost information provided in Benford et al. (2009).

2 Reimbursement in the AP-GfK survey is $5.

3 Costs in these examples do not include project management and statistical support in developing weights and estimation.

4 A large part of the cost of screening for CPO is related to how simple or how complex the approach to the screening of this population is, i.e., number of questions to define a person as CPO.

5 MRI Fall 2009 estimate of cell phone only

6 Data are actual and modeled from six AP-GfK Polls in 2010 reflecting the dual 70 percent landline RDD and 30 percent cell RDD design currently in use.

7 Efficiency is 1 divided by the design effect or can be computed as the sum of the weights divided by the sum of the squared weights.

8 The effects of weighting on effective sample size vary between these surveys and will vary across other types of surveys.  Results will vary for many reasons, including the choice of variables used in the weighting scheme and the degree to which various groups are underrepresented in the realized subsamples. However, it is expected that similar sample designs to those discussed here would produce similar directional differences in effective sample size due to landline RDD's undercoverage of younger adults and other demographic subgroups. Details of the weighting in this analysis can be found at http://ap-gfkpoll.com/poll-methodology. As a quick overview, weights are computed using a pre-weight that includes probabilities of selection and a mixing parameter and then raked to age by sex, race as black and all other races, Hispanic and non-Hispanic, and educational attainment all determined from the CPS and census region by phone service per Media Research & Intelligence's fall 2009 wave