Non-Probability Sampling

¹ Webographics are attitudinal variables thought to account for the differences between people who do surveys online and those who do not.  They generally measure lifestyles issues such as the types of activities people engage and their frequency, media use, attitudes toward privacy, and openness to innovation.
 

7. MEASURES OF QUALITY

Measuring the quality of data from non-probability samples is a new and challenging task.  For decades survey researchers have relied on measures for assessing data quality based in the probability sampling paradigm.   Many survey organizations and government agencies have established standards and guidelines to produce these measures, and they are often documented in quality profiles² that summarize what is known about the sources and magnitude of error in data for the surveys they conduct.  Because they are grounded in probability theory and have been used for decades, they are widely accepted as useful measures of quality.

Unfortunately, non-probability samples violate three key assumptions on which many of these measures are based.  Those assumptions are: (1) a frame exists for all units of the population; (2) every unit has a positive probability of selection; and (3) the probability of selection can be computed for each unit.  The standard quality metrics are designed to measure the degree to which a specific sample violates these assumptions due to such real-life constraints as incomplete coverage and unit nonresponse.

The quality standard guidelines issued by Statistics Canada, the U.S. Office of Management and Budget, and the U.S Census Bureau contain little commentary on methods for assessing data quality in non-probability samples.  For example, Statistics Canada (2009) acknowledges that non-probability samples can be an easy, fast and inexpensive way to conduct preliminary design studies, focus groups and follow-up surveys.  But they also go on to say, “The ability to make reliable inferences about the entire population and to quantify the error in the estimates makes probability sampling the best choice for most statistical programs.”

The U.S. Office of Management and Budget (2006) has issued standards and guidelines for Federal statistical surveys.  The document describes 20 standards that cover in great detail all stages of a survey including design, pretesting, data collection, data processing and editing, data analysis, and data dissemination.  However, they offer almost no guidance on non-probability samples.  Agencies are instructed to select samples using accepted statistical methods, for example, “probabilistic methods that can provide estimates of sampling error” (OMB 2006, pg. i). The use of non-probability samples must be “justified statistically and be able to measure estimation error”. To wit:

When a nonprobabilistic sampling method is employed, include the following in the survey design documentation: a discussion of what options were considered and why the final design was selected, an estimate of the potential bias in the estimates, and the methodology to be used to measure estimation error. In addition, detail the selection process and demonstrate that units not in the sample are impartially excluded on objective grounds in the survey design documentation.    (OMB 2006, p. 7).
 
The statistical quality standards issued by the U.S. Census Bureau are equally unhelpful.  Requirement A3.3: reads “Sampling frames that meet the data collection objectives must be developed using statistically sound methods” (italics added).   The requirement goes on to say, “Statistically sound sample designs require a probability sample” (U.S. Census Bureau 2011, pg.25).

As we have noted elsewhere in this report, virtually all non-probability methods must overcome three obstacles: the exclusion of large numbers of people from the sampling process; reliance on volunteers or referrals; and, in many instances, high nonresponse.  Some methods, such as convenience sampling, largely ignore these problems while others, such as sample matching and post-survey adjustments of various kinds, go to considerable length to identify the right set of auxiliary variables that can be used to adjust the sample so that it is representative (or at least approximately representative) of the target population. However, as of this writing there are no widely-accepted measures or practices for validating the assumptions and the efficacy of those adjustments.

The absence of these measures and practices may be the primary reason why survey researchers, especially those charged with producing accurate national benchmarks, have been reluctant to accept non-probability methods.  And indeed, AAPOR recently issued a statement noting that the difficulty of validating the assumptions that underlie claims of representativeness for opt-in panels causes it to continue to recommend probability sampling when precise estimates are needed (Baker et al. 2011, p. 758).

Without an accepted framework for assessing data quality in non-probability samples survey researchers might still find it useful to conceptualize the problem within the familiar framework of Total Survey Error (Groves, 1989).  This framework groups errors into two broad categories: errors of non-observation and errors of observation.  The former describe errors that affect the representativeness of the sample and the latter errors that affect the measures of interest because of how the survey was designed and executed.  Given that the most frequent criticism of non-probability samples is that they are not representative, we consider errors of non-observation first.

7.1 Errors of Non-Observation
Errors of non-observation derive from gaps between the target population, the sampling frame, and the sample. They generally are grouped into three categories: coverage error, sampling error, and nonresponse error. 

Coverage Error. Coverage error measures how well a sample frame covers the target population. Under ideal circumstances every member of the target population is listed in the sampling frame and therefore has a chance to be selected for the sample.   Groves et al. (2009, pg. 84) have argued that “without a well-defined sampling frame, the coverage error of resulting estimates is completely unknowable.”  Since in non-probability samples units generally are not sampled from a well-defined sampling frame, the task of assessing coverage error seems impossible. 

However, this assessment is deeply rooted in the probability sampling paradigm and may not be directly applicable to non-probability samples.  In probability samples, coverage ratios are the most commonly used metric of coverage error. A coverage ratio is the ratio of the estimated total from the sample computed by using only the inverse of the probability of selection weights (perhaps adjusted for nonresponse) to the known population total. In household surveys, these are typically demographic ratios where the denominators are data from recent censuses.

Non-probability samples can compute similar ratios, but they do not begin with inverse sampling weights so the computations must be different. It is possible to compare relative ratios from a non-probability sample to population totals from a census.  For example, suppose the target population of a study is intravenous drug users within a county and a respondent-driven sample is used to locate and recruit respondents for this hard-to-reach population.  If the health clinic records that serve that county’s population are assumed to provide reasonably good population controls, then demographic ratios such as sex ratios by age group and race/ethnicity category could be computed.  If the demographic ratios of the survey are vastly different from those reflected in the records, there may be evidence of coverage error. If the ratios are similar, however, it does not imply that the coverage is complete. It only implies that the coverage is similar across the different age and race/ethnicity groups.

The principal problem is that non-probability samples that attempt to estimate totals must rely on external population totals to produce the equivalent of sampling weights (since they are not sampled from a frame). This makes the computation of absolute rather than relative coverage ratios difficult.  Researchers have used different ways to compute absolute coverage ratios, but with at best mixed success. For example, one approach for online surveys using sample from opt-in panels is to estimate the proportion of the population that has access to the Internet as a coverage measure.  While this is somewhat informative, it makes an assumption that all those who do have access to the Internet are exposed to the survey request and therefore eligible to participate. This simply is not true in almost all cases we can think of.  More direct measures of coverage are needed for non-probability samples.

Of course, coverage ratios, much like response rates, are imperfect measures of biases due to non-coverage even in probability samples. As an extreme example, suppose the sampling frame of addresses in the postal service delivery files contains the address of 98 percent of the entire U.S. population and it is used to draw a probability sample. The coverage ratio is very high, but this sample would have dreadful coverage for estimating characteristics of the homeless.

A recent article by Blair and Conrad (2011) discusses the notion of “problem” coverage in the context of cognitive interview pretests, and focuses on how to select a non-probability sample size that is adequate to detect questionnaire problems.  For probability samples, the notion of sample size is directly related to the level of sampling error one can expect – the relationship for non-probability samples is much less clear.  Blair and Conrad provide a first attempt to quantify and give practical guidelines for sample size determination in this setting. They demonstrate that different sizes of quota samples (ranging from as few as 5 to as many as 90) are needed to observe different outcomes (e.g., mean number of problems discovered, likelihood of high impact problem discovery). One of their conclusions is that “small samples sizes may miss a substantial percentage of problems, even if concern is limited to those problems with a serious impact on measurement error” (Blair and Conrad 2011, pg. 654).  More empirical studies such as these are needed to guide quality considerations for quota and other non-probability samples.

Sampling Error. The next component of the total survey error model, sampling error, measures variations in the survey estimates over all possible samples under the same design utilizing the same sample frame.   When the sample frame is assumed to provide full or nearly full coverage of the entire population, then it is reasonable to assume that such measures are at least an approximate estimate of the precision of the survey’s estimates. However, full coverage is seldom a viable assumption with non-probability samples.  The case is especially clear with non-probability panels and so AAPOR has long maintained “reporting margin of sampling error with opt-in or self-identified samples is misleading” (Baker et al. 2011, p. 773). 

We agree that margin of sampling error in surveys has an accepted meaning and that this measure is not appropriate for non-probability samples.  However, the broader statistics literature does use terms that are directly comparable to sampling error as a measure of variation of the estimates that is not tied to the idea of all possible samples. For example, most elementary statistics texts assume a model (suppose a random sample is selected from a Normal distribution) and estimate the standard error of a statistic with respect to that model as a measure of the uncertainty due to sampling.

We believe that users of non-probability samples should be encouraged to report measures of the precision of their estimates, but suggest that, to avoid confusion, the set of terms be distinct from those currently used in probability sample surveys.  The precision of estimates from non-probability samples is not the average deviation over all possible samples, but rather is a model-based measure of deviation from the population value.  Ipsos, for example has proposed the credibility interval (Ipsos, 2012) for their estimates from an opt-in panel survey.  As noted in Section 6, the credibility interval is measure of uncertainty that is used with Bayesian methods, and Ipsos described their procedure as Bayesian. Other model-based approaches also produce estimates of precision such as standard errors that could be used and do not refer to the average over all possible samples (the accepted terminology for design-based inferences used in probability samples).

Although the research base does not exist to endorse this particular measure or to urge its adoption across the industry, we believe the industry needs constructive attempts to develop measures that fills the gap created by the unsuitability of the standard margin of error calculation with non-probability samples.   Treating estimates as though they had no error at all is not a reasonable option.  At this point, it falls to individual researchers to judge the usefulness of this particular measure.  Such judgments are only possible when organizations using them fully disclose the full range of information specified in the AAPOR Code of Professional Ethics and Practice along with a detailed description of how the underlying model was specified, its assumptions validated, and the measure calculated.  

There are, of course, many methods for selecting non-probability samples and so no single approach to estimate a measure of error is likely to be appropriate.  For example, Sudman (1966) proposed a technique for quota sampling and showed that if the sample is selected using the procedures he outlined and certain assumptions are satisfied estimates of sampling error like those used with probability samples also can be used with quota samples selected under his method. Other methods of estimating sampling error have been developed for respondent-driven sampling (see Section 4).  Once again, because the methods used to draw non-probability samples are so diverse, it is incumbent upon those who use these methods to clearly document their methods and the assumptions underlying their computations of variability or uncertainty.

It is important that non-probability sampling methods be evaluated more closely with respect to any claims about their measures of precision. One well-established approach to accomplish this is to use replicate samples, an idea that traces back to the early days of probability sampling (Mahalanobis, 1946; Deming,). This approach is also used in observational studies and experiments.

In its purest form, multiple samples are selected and the variation in the estimates across these samples is a measure of the standard error of the estimate. Pseudo-replication methods take this a bit further and take one sample and divide it into replicate samples. The theory for pseudo-replication in probability samples is well established (Wolter, 2007), and developments for non-probability methods are feasible. Replication can be economical, relatively robust, and understood by many practitioners. It does not require larger sample sizes and can be executed relatively quickly.  However, like traditional margin of error computations in probability samples, this approach only estimates variance; biases in the estimates cannot be measured.

Nonresponse error. The response rate in a probability sample is the ratio of eligible units measured to the total number of eligible units in sample (AAPOR 2009).  The response rate is probably the most recognized quality measure for probability samples, even though in recent years its limitations as a predictor for nonresponse bias of estimates have been made clear (Groves 2006).

In non-probability samples, the denominator for the ratio may not be known, therefore it is not always possible to produce response rates as traditionally defined by AAPOR and other professional standards bodies.  Consequently, as with margin of error calculations, researchers reporting on non-probability samples should avoid the term “response rate” and instead use another term.  ISO 20252: 2008 recommends the term participation rate, which it defines as “the number of respondents who have provided a usable response divided by the total number of initial personal invitations requesting participation” (ISO, 2008).  This term has been adopted by AAPOR and is included the 2011 revision of its Standard Definitions.  Eysenbach (2004) suggests a number of other related measures including view rate and completion rate. 

Opt-in panels often collect detailed profiles as part of the recruitment stage that researchers might use to assess differences between the responders and non-responders to specific studies. They might make use of this information to assess potential nonresponse bias and data quality, but as of this writing we are not aware of any instances in which this is being done.

Other response metrics specific to opt-in panels are covered in detail by Callegaro and DiSogra (2008).  Some, but not all, apply to non-probability panels. These include:

• The absorption rate measures the degree to which email invitations are not delivered to members because of wrong addresses, full mailboxes, or a network delivery error.  This measure is one indicator of how well a panel provider updates and communicates with its members.
• The break off rate is the number of surveys that do not meet a preset threshold of answered questions. High break-off rates can be an indicator data quality by signifying questionnaire design problems (e.g. poorly designed formatting or navigation or indication the survey is too long).  
• The screening completion rate/study-specific eligibility rate is the number of people who complete the screener and qualify plus those who complete the screener but do not qualify divided by the total number of survey invitations. If screening and eligibility rates are very different from well-established external benchmarks, this may indicate fraudulent/incorrect reporting by panel members who are purposefully self-selecting into studies.   
• The attrition rate measures the percentage of members that drop out over a set period of time. It is measured by counting the number of panel members that remain active month after month (Clinton 2001).  High attrition rates may signal poorly designed questionnaires (e.g. surveys that are too long) that result in panel fatigue and high dropout rates.

 
The relatively few measures that have been developed to evaluate online samples mostly assume an opt-in panel of the sort that has dominated online market research over about the last decade.  But the panel model is rapidly falling into obsolescence as the demand for online respondents grows, clients look for greater demographic diversity in their online samples and interest in studying low incidence populations increases.   Providers of online samples are increasingly relying on a range of sources that expands beyond their proprietary panels to the panels of competitors, social networks, and the use of general survey invitations placed on a variety of websites across the Internet, much like river sampling.   These respondents may no longer receive invitations to do a specific survey on a specific topic.  Instead they receive a general solicitation to do a survey that directs them to a website where they are screened and then routed to a waiting survey for which they already have qualified.  The software that controls this process is called a router.  Its goal is to ensure that anyone willing to do a survey online gets one.  As of this writing there is a good deal of variation in how routers are designed, how they operate, and what impacts, if any, they have on the data.  Unfortunately, they also make it impossible to calculate a participation rate as we have discussed it above.

Nonresponse bias studies have become part of most government-sponsored probability sample surveys with OMB playing a key role in the development of nonresponse bias analysis approaches.  The use of subsamples of nonresponders may be applicable to specific types of non-probability sample surveys.  Techniques that make use of individual/geographic internal characteristics available in the sampling frame are widely used in probability sampling and may be applicable to non-probability samples that collect information on the pool of potential respondents.  In some cases external benchmarks are available for one or more key substantive variables included in the survey. When little internal or external information is available it may still be possible to assess nonresponse bias using face validity methods.  For example, the medical literature may indicate that roughly half of the persons with a specific condition are females.  If one uses a non-probability sampling technique to produce a sample of persons with the condition, using say nonrandom seeds, and 80% of the respondents are females, then it is clear that in the process of generating the sample males were much less likely to be nominated and/or participate in the survey.  In this situation it is not clear that weighting the sample by gender and other socio-demographic can account for the 30 percent of males that are missing from the sample.

Reinforcing the complexity of non-response in non-probability samples, Gile, Johnston and Salganik (2012) introduced a three-level measure of non-response for respondent-driven sampling.  Based on respondents’ reports of the numbers of coupons refused by their contacts, and the numbers of coupons they distributed, the authors computed a coupon-refusal rate, and a coupon non-return rate, which combine to form a total nonresponse rate.  

7.2 Measurement Error
The other broad category of errors accounted for in the TSE framework describes errors in observation, commonly referred to as measurement errors.  These errors are generally seen to arise from four sources: the questionnaire, the interviewer (if there is one), the respondent, and the mode of data collection (e.g., face to face, mail, telephone or web).   Here we might argue that data coming from non-probability samples likely have the same error properties as those coming from probability samples. Both are prone to observational gaps: between the constructs intended to be measured and the measurements devised to quantify them; between the application of the measurements and the responses provided; and between the response provided and the data ultimately recorded and edited.  Whether the survey data come from a probability or non-probability sample should not matter --we should be able to use the same type of measurement error quality indicators. 

The Questionnaire.  One of the most common quality indicators associated with the survey questionnaire is construct validity, that is, the degree to which survey items that measure the properties of key constructs correlate in expected ways or the degree to which the survey has measured the true value of a construct.  For example, if we wanted to assess data privacy or confidentiality concerns we would expect that respondents answering that they are “very concerned” about data privacy might also express concern about computer identity theft, hackers and belief that government agencies routinely share data.  If we found these items to be correlated in the expected direction, then we would have some reassurance that the underlying construct is being measured.   We might also compare key items across demographic or other defined groups that we would expect to respond differently on those items. For example, in a study of political attitudes on social issues such as same-sex marriage, gun control, and limits on abortion we would expect to see consistent differences between Republicans and Democrats.

A more direct measure of construct validity uses multiple items within the questionnaire specifically designed to replicate key items.  The National Household Survey on Drug Use uses this technique to validate respondent reports of marijuana use (Biemer and Lyberg 2003).   The first questionnaire asks, “How long has it been since you last used marijuana or hashish?”  Later in the same survey, respondents are asked, “On how many days in the past 12 months did you use marijuana or hashish?”  If a respondent answers “a day or more” to the second item then we would expect the answer to the earlier item to be something less than one year. High levels of inconsistency between these items might signal problems in the survey’s measurement of marijuana use.  

The Interviewer.  The interviewer does not play a role in many of the newer non-probability sample surveys since self-administered paper or web surveys are commonly used.  The interviewer as a source of measurement error is however relevant when non-probability sample surveys are conducted by telephone or in person.  Again, there is an extensive literature on the interviewer as a source of measurement error in probability sample surveys (Groves 1989) and much or all of this literature is relevant to non-probability sample surveys that use interviewers.  These techniques include embedded designs for measuring interviewer variance, examining the relationship between interviewer characteristics survey responses, taking advantage of the feasibility of randomizing interviewers to experimental survey conditions in centralized telephone interviewing facilities, measuring interviewer compliance  with training materials, and examining paradata to identify potential cases on interviewer falsification.

The Respondent.  All non-probability sample surveys rely on a set of respondents generated in some nonrandom fashion and those respondents are a potential source of survey measurement error.  Kahn and Cannell (1957) and Groves (1989) identified five stages of respondent reporting:

1. Encoding of information
2. Comprehension
3. Retrieval
4. Judgment of appropriate answer, and
5. Communication

Virtually all of the techniques to assess respondent reporting errors developed for probability sample surveys are relevant.  Reverse record check studies can be used to compare respondent reports with administrative/program/medical records.  Reverse record checks are also potentially very useful at the design stage to help determine the length of recall period for the reporting of events from memory.  Cognitive interviewing can also be very helpful in assessing measurement error related to one of more of the above steps.  Cognitive interviewing can be useful for testing respondent comprehension of eligibility screening criteria, especially if the eligibility characteristics of the cognitive interviewing participants can be accurately determined external to the screening questions being tested.

 The widespread use of the web as a data collection modality allows the use of technology to obtain detailed analyzable information on the respondent interaction with the questionnaire.  In addition to recording information on break-off points, well designed web questionnaires can also allow for the measurement of time spent on each question, forward and backwards movement within the questionnaire, frequency of selection of “don’t know” response categories as a measure of item nonresponse, and length (i.e., number of words) of open ended responses.

Finally, there are a set of quality issues that have their roots in the opt-in panel paradigm. These are discussed in 7.4.

The Mode of Data Collection. Groves (1989) among others has described the response effects  most affected by the mode of data collection.  Today, surveys are conducted in-person, in-person with the respondent provided with a communication device to call into a centralized telephone interviewing facility, using mail administered surveys, by telephone using decentralized telephone interviewing consisting of one interviewer making calls, by telephone from centralized telephone interviewing facilities, and using the web.  The picture is even more mixed when we consider various hybrids such as in-person interviewing where the respondent completes a self-administered interview using a computer-assisted device.

Over the past 10 years there has been considerable survey modality research conducted for a wide range of probability sample surveys.  The growth of the various types of multi-modality surveys has allowed for mode experiments to be embedded in probability sample surveys.  The primary objective of these experiments is to determine whether the mode of data collection leads to mode effects.  The literature on mode effects for mail surveys and web surveys is particularly relevant to non-probability sample surveys because these seem to be the two primary modes of data collection.  Some types of non-probability sample surveys are amenable to randomization of mode to mail or web or the use of mode choice experiments whereby the respondent is given the option to select among two or more modes.  It is also important to examine break-off rates in multi-modality surveys and to determine where break-offs are occurring in the screening/main questionnaire.

7.3 External Validity
The idea of external validity is central to much of the work done thus far that attempts to assess the quality of survey data collected from opt-in panels.  A literature has developed over the last decade describing attempts to assess the validity of samples from these sources as compared to probability samples of varying quality (mostly on the telephone), benchmark data such as censuses, electoral outcomes and other data collected by non-survey methods such as administrative or sales data.  This literature was reviewed in the earlier AAPOR task force on opt-in panels (Baker et al. 2011) and will not be reprised here.  Arguably the most frequently cited work is the Yeager et al. (2011) evaluation of five opt-in panels which compared a small set of variables to two probability samples, a high quality government survey and administrative records. 

On the face of it this would seem to be an effective way to assess the validity a specific non-probability sample.  In practice it is problematic for at least three reasons.

First, it’s not always practical because the external measures needed for comparison may not exist. This can be done in carefully designed experiments for this purpose, but this is not the standard approach to doing surveys.  If the target population and the survey topic are frequently studied or monitored it may be more practical to find sources for comparison.  But as target populations become more narrowly defined and topics more esoteric, such sources may be difficult if not impossible to find.

Second, even very high quality probability surveys that purport to measure the same thing sometimes don’t agree.  For example, the number and percent of persons who are uninsured is estimated in at least four U.S. federal government surveys, all of which have high response rates. Nevertheless, there are differences in the estimates that far exceed sampling error and those appear to be due to measurement and other nonsampling errors (Davern et al. 2011). Another example is the Survey of Income and Program Participation (SIPP) estimates of the number of persons with General Education Development (GED) test high school equivalencies to be about 70 percent higher than the estimate from the Current Population Survey (CPS) School Enrollment Supplement (Crissey and Bauman 2012). Both sources of these estimates are high quality surveys conducted by the U.S. Census Bureau.

Finally, administrative records and other kinds of data collected by non-survey means are not error-free. For example, Smith et al. (2010) described a situation in which the misclassification of ethnicity and race in administrative records occurred in nearly 25% of the records, the majority due to missing information. This is not an unusual situation in administrative records that are not developed to support statistical uses. Brackstone (1987) reviewed the advantages and disadvantages of administrative records for these purposes. Boruch (2012) also assessed the uses of these types of records and provided a different and thought-provoking discussion.

 The bottom line would seem to be that in many cases it is very difficult to determine the reason the estimates from different surveys differ and to assign some proportion of the difference to a specific cause like the sampling mechanism. Further, gold standards for comparison do not exist in most cases.  Nonetheless, we can make judgments about validity assuming we understand something about the quality of the comparison data that are available or when multiple sources for comparison exist, but precise estimates of error due to the sampling mechanism are very difficult to come by.

7.4 Quality Measures Unique to Opt-in Panels
Over about the last 10 years the market research industry has come to rely on a variety of non-probability methods used in conjunction with online access panels.  Some of these methods have been described in previous sections and results from studies using opt-in panels have frequently been challenged on grounds of both bias and variance (Yeager et al. 2011; Walker and Petit 2009).  However, these and other studies like them generally have not looked at the specific sampling methods used.  Their focus has tended to be on the panels themselves rather than the techniques used to draw samples from them.  From that perspective it seems entirely legitimate to focus on coverage and in turn point out the variety of ways in which individual panels are recruited and maintained.   Likewise, calculating estimates of coverage error and examining the techniques that may be used to adjust for it are essential parts of evaluating the quality of samples from opt-in panels. 

However, it is now widely accepted across the market research industry that there are three additional problems that are unique and probably endemic to the panel model.  They are:

• The tendency for people to sign up on the same panel using different identities in order to increase their chances of being selected for surveys.
• The possibility that when more than one panel is used to get a significant number of lower-incidence respondents that some people may be sampled and complete the same survey more than once.  A more malevolent form involves the use of web robots (or simply bots) to automatically complete the same survey multiple times.
• High rates of satisficing as evidenced by very short completion times, straightlining in matrix style questions, frequent selection of non-substantive responses, skipping questions, and inconsistent or nonsensical responses.

The magnitude of these problems across the industry and the degree to which they affect survey estimates is a matter of some debate, and one suspects that there is a good deal of variation from panel to panel if not survey to survey.  Nonetheless, they are now widely acknowledged across the industry as significant quality problems to be addressed by industry and professional associations, software developers, panel companies and individual researchers.  For example, both ISO 20252 –Market, Opinion and Social Research and ISO 26362 – Market, Access Panels in Market, Opinion and Social Research specify requirements to be met and reported on.  The market now has two software solutions, RelevantID and TrueSample with the functionality to identify potentially problematic respondents in online samples.  Many individual researchers have implemented post survey editing procedures designed to isolate duplicate respondents and potential satisficers (Le Guin and Baker, 2007).

But these sample quality procedures are not without problems of their own.  For example, the procedures commonly used to validate the identity of prospective panelists have been shown to reject people under 30, the less affluent, the less well-educated and non-whites at a much higher rate than other demographic groups (Courtright and Miller 2011).  Likewise, there are no clear standards for measuring survey engagement or separating out poor respondent performance from poor questionnaire design.

Nonetheless, researchers interested in assessing the quality of samples drawn from opt-in panels are wise to insist on reports of the specific steps taken to ensure that all respondents are real people with the characteristics they claim to have; that no respondent was allowed to complete more than once; and that unengaged respondents have been identified and the measures used to engage them clearly defined.    The measures used and the extent to which they provide a workable assessment of sample quality are not yet established standards, but they can provide the researcher another quality assessment tool when working with opt-in panels.

7.5 Other Metrics on the Horizon
As we noted at the outset, the two main measures used in probability sampling to evaluate the representation component of TSE, coverage ratios and response rates, may not be very good indicators of bias. Researchers who routinely work with probability samples have been investigating alternative approaches to deal with this shortcoming, especially with respect to response rates. For example, R-indicators are one approach to replace or supplement response rates (Schouten, Cobben, and Bethlehem 2009). Särndal (2011) also addressed this problem and suggested balancing the sample and the set of respondents to reduce nonresponse bias. Essentially, the R-indicators and some of the indicators proposed by Särndal and his colleagues involve comparing response rates for subgroups identified from auxiliary data.  If the response rates from all the subgroups are relatively consistent, then there is little evidence of nonresponse bias (nonresponse bias occurs when response rates co-vary with the characteristics being estimated). The indicators quantify this variation in response rates across the subgroups.

Other methods have also been proposed that are more directly applicable to non-probability samples. Frank and Min (2007) have suggested a testing approach to assess whether causal inferences can be generalized when the data are not a probability sample of the entire population. Fan (2011) proposed a method for constructing tolerance “maps” to provide users different tolerances of non-representativeness in different studies.  Using data from Louis Harris and Pew Internet and American Life Project surveys, he demonstrated how representative responses might be obtained from a non-probability Internet sample recruited exclusively from politically conservative websites.  In concept, this new method is consistent with the fit for purpose ideas discussed in the next section since it allows researchers to “devise samples capable of giving responses that are less than truly representative but that are still within the researcher’s tolerance”.

7.6 Summary
The general lack of well-defined measures for assessing the quality of non-probability samples is due in part to the lack of a single framework within which all non-probability methods fall and, perhaps more importantly, the willingness of practitioners historically to accept these methods at face value.   Survey researchers, on the other hand, are accustomed to using widely accepted quality measures that they believe substantiate and qualify the inferences they make from probability samples.  It seems clear that if non-probability methods are to be embraced as valid for surveys then similar measures or methods are needed.

The TSE model may offer some help.  Errors of non-observation and especially coverage and nonresponse error pose the biggest challenge. Sampling frames for non-probability samples are seldom defined in ways that cover the full target population and give every member of that population the opportunity to be sampled.  Measures of relative coverage may be possible in some instances, but because they are generally based on variables available for comparison (such as demographics) there is no guarantee that they measure the characteristics that matter most. 

In the end, it may be more important to validate the steps taken to overcome coverage and nonresponse error in a non-probability sample than it is to measure it, at least for routine practice.

Establishing external validity or representativeness through comparison of survey estimates to benchmark data such as that from censuses, high quality probability sample surveys or administrative records is sometimes possible, but it’s not clear that the data for comparisons always exist and are at least reasonably error-free.  Replication might also be helpful to assess the variability of the estimates and it can be relatively easy to do with at least some non-probability methods.

We think it reasonable to approach the measurement side of the model with pretty much the same tools as we use for probability samples.  Where errors of observation are concerned we probably can expect that non-probability samples have error properties that are similar to probability samples using similar modes.

As we have said at different points and in different ways in this report, the inherent risk in model-based methods is that the assumptions underlying any given model do not hold and the outcomes are sensitive to those assumptions.  When this happens, the estimates are biased, possibly severely biased.  If we are to make informed judgments about the quality of a non-probability sample the underlying assumptions of any adjustments made must be clearly stated, empirical validation of those assumptions described and the auxiliary variables used defined.  We welcome the development of new approaches and measures such as the credibility interval, but note that migrating such measures from the proposal stage to the acceptance stage requires more validation than we have seen to date.  Even then, researchers accustomed to empirical measures rooted in strong theory and decades of practice must become comfortable making judgments that may not be black and white.

Transparency of method has long been championed by AAPOR as the only route to careful evaluation of the quality of a survey.  This is especially true when the survey relies on a non-probability sample.  However, there is no apparent consensus as to what should be disclosed.  The AAPOR Code has a set of items that are appropriate to probability-based surveys but may not work well for sources such as opt-in panels.  The Public Works and Government Services Canada (2008) has proposed a disclosure standard for online surveys and two ISO standards (20252 and 26362) have done so as well.

The bottom line is that the multiplicity of approaches and methods along with their continued evolution make judgments about the quality of non-probability samples difficult. And in most cases individual researchers and data users are forced to make case-by-case decisions about the quality and utility of estimates from studies that use these kinds of samples.  To make progress, procedures for certain types of non-probability samples like opt-in panels need to stabilize and estimates from these samples across a broad range of applications need to be evaluated. If such results are consistent with a priori theoretical assessments of the accuracy or stability of the estimates, then support for these methods will be greatly enhanced.  

8.  FIT FOR PURPOSE

A key reason for the predominance of probability sampling over about the last 60 years has been its accuracy, that is, its demonstrated ability when its key assumptions are met to generate population estimates that are within a calculable range of the true values for that population.  During much of that time, discussions about survey data quality generally have been framed using statistical concepts such as bias and variance with varying attempts to define the types of errors in survey design and implementation that produce them.  Groves (1989) summarized decades of statistical and social science literature on total survey error (TSE), described the principal categories of error and their impact on bias and variance, and tied these to the costs of conducting the survey. 

As discussed in the previous section, the TSE model is an excellent lens through which to view the sources of errors in probability-based surveys, with some potential usefulness in non-probability surveys as well.  Nonetheless, Groves and Lyberg (2010) argue that TSE should be but one approach within a much broader quality framework.  First, costs are important and must be considered in a practical sense. Second, a broader framework is needed to assess survey results in light of how those data are to be used, a set of criteria expressed in terms such as fit for purpose or fitness for use.

8.1 The Changing Definition of Quality
In their book, Biemer and Lyberg (2003) linked changing definitions of survey data quality to the broader quality revolution of the 1980s and 1990s.  In manufacturing settings the definition of quality once amounted to something like “free of defects” or “conformance to specifications.”   With advent of the Total Quality Management (TQM) movement this changed and one begins to see a stronger emphasis on the somewhat subjective needs of the customer and the use to which a product was to be put.  As Deming (1982) writes “Quality should be aimed at the needs of the consumer.”  Juran (1992) is often credited with having invented the term, fitness for use, by which he meant that how a product would be used and the price a customer is willing to pay should be important factors in the design process and therefore an essential part of the concept of quality. 

ISO, the International Organization for Standardization, has as its mission the development and dissemination of international standards for products and services.  Working through a network of national institutes in 162 countries, ISO has developed over 19,000 global standards to “ensure desirable characteristics of products and services such as quality, environmental friendliness, safety, reliability, efficiency and interchangeability - and at an economical cost.” 

Thus the world at large has come to define quality not in absolute terms but rather in the context of the expectations of the customer, the purpose for which a product or service is acquired and how well it fits that purpose.

8.2 The Concept of Quality in Survey Research
Deming is mostly identified in the popular mind with the so-called “quality revolution” and the TQM movement, first in postwar Japan and then here in the US.  Within the survey profession he is more likely thought of first and foremost as an eminent statistician who in the early 1940s was instrumental in the development of sampling procedures at the Bureau of the Census (Aguayo 1990).  It was during his time at Census that Deming (1944) argued that accuracy should not be the sole criteria for evaluating survey data.  It is at least equally important that survey results be “useful” by which he meant, “helping to provide a rational basis for action.” (p. 369).

The link between quality and usefulness has been periodically reinforced throughout the literature on sampling and surveys.  For example, Sudman (1976) tied sample quality to how the information will be used by describing two extremes.  One of those extremes he calls “exploratory data gathering” whose main purpose is to generate initial hypotheses to be used in a later study.  At the other extreme are large-scale government data collections to support policy development and program implementation.  For Sudman the needs of the latter dictate a higher level of precision than the former.  Thus, he seemed to adhere to a fitness for use test in which one important measure is the accuracy of the estimate.  He offered 10 specific examples of studies in which sample designs were compromised to one degree or another, mostly due to cost or general feasibility.  The emphasis, however, in virtually all cases is on whether the potential loss of accuracy that flows from those compromises will substantially affect the usefulness of the estimates.

Kish (1987), in a similar vein, acknowledged that “statistical designs always involve compromises between the desirable and the possible.” (p.1)  For Kish those compromises are driven by the practicalities of feasibility and resources while being attuned to the purpose for which the research is designed. He listed three main categories or dimensions in which compromises typically are made: (1) representation having to do mostly with incomplete sampling frames and low nonresponse; (2) randomization meaning finding ways to account for the effects of confounding variables; and (3) realism which has to do with the degree to which survey variables measure the constructs they are meant to describe.  He presented 10 research designs in an ordered list.  At one end of the list were surveys where representation is essential.  At the other end were experiments where randomization is critical.   He placed observational studies, where the need for realism drives compromise, in the middle.  This list of designs is meant to capture the differences between what Kish calls “enumerative” or descriptive studies on the one hand and “experimental” or analytical studies on the other.

Groves (1989) made a similar point when he noted how the needs of what he called “describers” differ from those of “modelers.”  The former look for survey data that fully reflect the population they want to describe and are especially concerned with errors of non-observation.   The latter seek data with measures that fully capture the concepts needed to test their theories and worry less about coverage error and nonresponse.  Government agencies are mostly describers because their principal interest is in a precise estimate of some specific characteristic(s) of a target population.  Academics and sometimes market researchers, on the other hand, tend to be modelers who are interested in how personal characteristics interact to produce a specific behavior such as voting for one candidate over another or choosing product A rather than product B.  Describers and modelers each have their own primary areas of concern and therefore different ideas about what constitutes high quality data. 

O’Muircheartaigh (1997) also linked the needs of the data user and the reasons for which the data were collected to quality. 

The concept of quality, and indeed the concept of error, can only be defined satisfactorily in the same context as that in which the work is conducted.  To the extent that the context varies, and the objectives vary, the meaning of error will also vary. . . Rather than specify an arbitrary (pseudo-objective) criterion, this redefines the problem in terms of the aims and frame of reference of the researcher.  It immediately removes the need to consider true value concepts in any absolute sense, and forces consideration of the needs for which the data are being collected. (p.1).         

8.3 Fit for purpose in Government Statistical Agencies
Government statistical agencies are the classic example of describers for whom accuracy is the primary attribute of quality, often because the estimates typically play a major role in making financial and policy decisions.   Yet increasingly, a considerable body of statistical agency requirements includes additional quality dimensions and criteria by which fit for purpose design decisions might be made.  In 2002, Statistics Canada released a set of quality guidelines that specified six “elements of quality . . . to be considered and balanced in the design and implementation of the agency’s statistical programs.” (p.4) The six are:

• Relevance –the degree to which the data are needed for some agency or policy purpose.
• Accuracy – the degree to which estimates correctly measure what they are designed to measure within an acceptable margin of error.
• Timeliness – the likelihood that the data will be available when it is needed to support an intended action.
• Accessibility – the availability of the data in a form that is useful to those needing them.
• Interpretability – the ease with which users of the data can understand the design and data collection processes so that judgments can be made about its usefulness.
• Coherence – the degree to which the data fit within a statistical program and are consistent with other similar data across time and geography.

 
The Australian Bureau of Statistics (2009) has developed a similar framework.  It adds a seventh element of quality that it calls “the Institutional Environment.”  This refers to the brand of the agency or organization associated with a statistical product and the degree to which it engenders confidence by virtue of a history of objectivity, independence, and protection of confidentiality of research participants.   

Other agencies have frameworks that are variations on this same set of themes.  They include Eurostat (2003), Statistics Sweden (Rosén and Evers 1999), the International Monetary Fund (Carson 2001), and the Organization for Economic Coordination and Development (OECD 2002).

8.4 Fit for purpose in Market Research
Market researchers sometimes behave as describers and other times as modelers.  Tracking studies that continually measure phenomena such as product satisfaction or use over time are in some ways similar to data collections by government statistical agencies.  Media measurement services that rack viewership and readership are another example of where precise estimates are desired.  In these instances the fitness for use criteria applied are not unlike those described above for government agencies although an emphasis on cost, timeliness and accessibility sometimes predominate.  At other times market researchers are more like modelers whose main focus is on data collections that can support development and testing of statistical models that describe, for instance, how personal characteristics and product features interact to make some products successful while others fail.

So market researchers, whether implicitly or explicitly, have adopted the concept of fitness for use.  Nowhere is this clearer than in the specific context of opt-in panels (e.g., Bain, 2011).  However, relatively little has been written to describe a specific framework of the sort used by the government statistical agencies.  ESOMAR, the global professional organization for market, opinion, and social research, regularly issues guidelines to help researchers make fit to purpose decisions about specific research methodologies.  For example, their “26 Questions to Help Research Buyers of Online Samples” (ESOMAR, 2008) suggests that a prospective online sample provider be vetted on seven criteria: the company’s profile, its sample sources, recruitment methods, panel management practices, compliance with industry policy and applicable law, partnerships with other suppliers, and methods of data quality and validation.  Their “36 Questions to Help Commission Neuroscience Research” (ESOMAR, 2011) use a somewhat different set of criteria.  As their titles suggest, both documents suggest questions to be asked, describe why each question may be important, but leave it to the researcher to evaluate how the answers may affect the quality of a specific research design.

Smith and Fletcher (2004) have offered a much clearer framework.  They started from the premise that “there is an often unavoidable gap between the quality of the raw data and the kind of market research evidence to which we aspire” (p. 61).   The challenge that falls primarily to the analyst is to bridge the gap between the ideal and the real, between the data we wish we had and the data that we have.  To that end Smith and Fletcher offered eight methodological questions for analysts to ask, most of which might be interpreted in terms of the error categories in the standard TSE model.  One of the eight involves a judgment about whether the study design was fit to purpose, which they characterized as a tradeoff among five key variables:

• The required accuracy/precision
• The depth or level of detail in the data
• The practical and ethical constraints
• The time when the data are needed

• The budget

8.5 Summary
Compromises in survey design are commonplace in surveys of all kinds and in all sectors of the research industry.  Advocates of probability samples have come to accept what once they may have thought of as unacceptably high levels of nonresponse.  The dramatic rise in the use of opt-in panels has been premised on a willingness to accept overwhelming coverage and selection error.   Those compromises are mostly practical and increasingly accepted, but seldom explicitly set in a fitness for use framework.  Yet even in idealized circumstances where the classic constraints of budget, time, and feasibility may not require compromise in design the degree to which survey results are usable by decision makers is now widely recognized as a key driver of design.  To quote Groves and Lyberg (2010), “. . . statistics are of little importance without their use being specified.” (p. 873).

The notion of fit for purpose as a definition of quality and the key driver of survey design has some history.  In their brief review of that history Biemer and Lyberg (2003) arrived at fitness for use as the basic definition of survey quality.  They defined the three key dimensions of survey quality as accuracy, timeliness, and accessibility.  In their words, for survey data to be considered of high quality it must be “as accurate as necessary to achieve their intended purposes, be available at the time it is needed (timely) and be accessible to those for whom the survey was conducted.” (p.13).   A number of government statistical agencies have further elaborated that basic framework, although it is our impression that these organizations have struggled with implementation.
Arguably the most compelling framework is the distinction between the needs and purposes of describers versus those of modelers.  The needs of the former fit rather neatly into the TSE framework that has been widely adopted by survey researchers.  Describers especially focus on minimizing coverage error and nonresponse as a way of ensuring representation and accuracy.  They generally favor probability-based sample designs.

Modelers, on the other hand, are much more focused on measuring all of the concepts that they expect play a significant role in explaining the behavior that is at the core of their research.  Their interest is in the relationships among a broad set of characteristics rather than precise measurement of those characteristics in the population of interest.  They don’t ignore errors of non-observation, but they generally assume no meaningful relationship among the phenomenon they are studying, the dependent variable(s), and the likelihood of an individual being selected or responding.  Put another way, their primary emphasis tends to be on internal validity.   Modelers make extensive use of non-probability samples because they often are less expensive than those favored by describers. 

As useful as these two distinctions might be they are less a dichotomy than opposite ends of a continuum.  Kish’s ordered list of designs is probably the best framework for thinking about how we might accommodate the purpose of the research and the precision required to the practicalities of time, budget, accessibility and general feasibility.  

In some cases, the choice may come down to a non-probability sample or no survey at all.  If some level of confidence that the assumptions of the model hold sufficiently for the purposes of the research, then the choice of a non-probability sample is justified. If not, the estimates from a non-probability sample might lead to poor decisions and no survey may be a better choice.

9. CONCLUSIONS

We believe that the foregoing review highlights the major issues survey researchers face when considering non-probability sampling.  Due to the unsettled landscape that is non-probability sampling for surveys, our treatment of these issues may not be as definitive as would be possible in more developed areas of survey research.  Many of the methods we described have long been used in other disciplines, but they remain unfamiliar to many in the survey profession.  We hope that this report is the beginning of a much broader exploration of those methods and, to that end, summarize below what we believe to be the key issues.

Unlike probability sampling, there is no single framework that adequately encompasses all of non-probability sampling. Although there are variants within probability sampling, the design-based approach has a common theoretical basis – select a relatively large sample where the chance of selecting each unit is known and weight the sample by the inverse of its selection probability (adjusting for missing data as needed) to estimate population characteristics.  By comparison, there are a variety of non-probability sampling methods, each with different approaches to sampling and estimation. The statistical properties and empirical performance of these methods vary considerably. Thus, non-probability sampling is a collection of methods rather than a single method, and it is difficult if not impossible to ascribe properties that apply to all non-probability sampling methodologies. Before using a non-probability method, a researcher should understand the specific approach being used and its limitations, especially with regard to limitations on the inferences that can be made.

Researchers and other data users may find it useful to think of the different non-probability sample approaches as falling on a continuum of expected accuracy of the estimates.  At one end are uncontrolled convenience samples that produce estimates assuming that respondents are a random sample of the population. These surveys generally have little or no evidence to support them. Inferences from these samples are extremely risky. At the other end are methods that select respondents based on criteria related to the survey subject matter and adjust the results using variables that are correlated with the key outcome variables. When studies also have evidence to support their assumptions, inferences from these samples are less risky.³
The difficulty arises in placing surveys between the two extremes.  This is largely uncharted territory for social, opinion and market research surveys. The risk assessment often depends on both substantive knowledge of the population being studied and technical features of the methods being used. Substantively, understanding the variability of the characteristic(s) being studied within the target population and the planned use of the estimates are critical to assessing the risk of the inference. The most common error in studies of human behavior and attitudes is to assume more homogeneity than exists. This error leads to biases and understatements of the variability of the estimates. Technically, sampling and adjustment control may be even harder to evaluate. We believe that some control at sample selection is very important; probability sampling is an example of a highly controlled sampling mechanism. Rubin (2008) expresses a similar sentiment in another setting. The importance of controlling the sample is at the core of another conclusion (below) that sample matching is the most promising non-probability approach for surveys. Combining control of the sampling with the use of good auxiliary variables at the adjustment stage should make inferences from non-probability samples less risky.

Transparency is essential. Whenever non-probability sampling methods are used, there is a higher burden than that carried by probability samples to describe the methods used to draw the sample, collect the data, and make inferences.  Once again, non-probability sampling is not a single method and differences in methods may be very important. Therefore, a clear description of methods and assumptions is essential for understanding the usefulness of the estimates. When the assumptions are clearly identified and there is evidence to support those assumptions, then the user has the opportunity to make an informed assessment of the risks associated with their use of the survey estimates. That assessment is not simple and it is a hurdle that non-probability samples must clear if they are to be generally accepted in scientific circles.  Black box methodologies must be opened up and made transparent. Genuine proprietary information may not have to be disclosed, but the methodology must be clear and the effort made to assess key assumptions summarized. Too many online surveys, in particular, consistently fail to include information that is adequate to assess their methodology.  This must change.

Making inferences for any probability or non-probability survey requires some reliance on modeling assumptions. Those assumptions must be clearly stated and evidence of the effect that departures from those assumptions might have on the accuracy of the estimates identified to the extent possible. While our focus has been on non-probability sampling methods, users of probability samples should also recognize that the inability to obtain 100% coverage and response rates means that inferences from these samples also often rely on modeling assumptions.

The most promising non-probability methods for surveys are those that are based on models that attempt to deal with challenges to inference in both the sampling and estimation stages.  Although probability sampling is the standard paradigm for making statistical inferences from surveys, it is not the only way to make inferences or even the most common one in general statistical practice. Other approaches, called model-based when applied in the survey field (Valliant, Royall, and Dorfmann 2000), are used in most non-survey applications. These approaches typically assume that responses are generated according to a statistical model (e.g., the observations come from a process that has a common mean and variance), and that by accounting for important auxiliary variables it is possible to improve the fit and usability of these models. Once the model is formulated, standard statistical estimation procedures such as likelihood-based or Bayesian techniques are used to make inferences about the parameters being estimated. Of course, simple model assumptions that ignore the complexity of the issues may not be sufficient. Even with model-based methods the challenges to inference remain.

One of the reasons model-based methods are not used more frequently in surveys may be that developing the appropriate models and testing their assumptions is difficult and time-consuming, requiring significant statistical expertise.  Assumptions should be evaluated for all the key estimates, and a model that works well for some estimates may not work well for others.  One of the key attributes of probability sampling is that it has a standard approach to produce the vast array of estimates often needed from surveys. Although probability sampling requires assumptions to deal with missing data, the assumptions are relatively standard and the approaches to adjustment have become routine. Achieving the simplicity of probability sampling methods for producing multiple estimates is a hurdle for non-probability sampling methods to overcome. In essence, even though non-probability samples can collect data much more cheaply than probability samples in many circumstances, the less controlled sampling approach requires more effort at the analysis stage.

Fit for purpose is an important concept for judging survey data quality, but its application to survey design requires further elaboration.  In discussing complex modeling, a report of the National Academy of Science (2012) wrote, “the level of rigor employed should be commensurate with the importance and needs of the application and decision context. Some applications involve high-consequence decisions and therefore require a substantial …effort; others do not” (p. 96). There is an emerging consensus among national statistical agencies around the key elements that should make up a quality framework to support the development and execution of survey designs that are fit to purpose.  They generally involve balancing the key elements of relevance, accuracy, timeliness, accessibility, interpretability, and consistency.  The logical next step is to transition those frameworks from the theoretical to the practical, refining them accordingly. Non-probability sampling needs to be evaluated within this same framework.

Sampling methods used with opt-in panels have evolved significantly over time, and, as a result, research aimed at evaluating the validity of survey estimates from these sample sources should focus on sampling methods rather than the panels themselves.  There is a tendency to lump all online samples from opt-in panels into the single bucket of online, as if opt-in panels were a sampling method. They are not.  Users of opt-in panels may employ different sampling, data collection, and adjustment techniques.  Research evaluations of older methods of non-probability sampling from panels may have little relevance to the current methods being used. Research should shift from a focus on opt-in panels to one that evaluates the different sampling and estimation strategies researchers might employ with these sample sources.

If non-probability samples are to gain wider acceptance among survey researchers there must be a more coherent framework and accompanying set of measures for evaluating their quality.  One of the key advantages of probability sampling is the toolkit of measures and constructs (such as TSE) developed for it that provides ways of thinking about quality and error sources.  Using that toolkit to evaluate non-probability samples is not especially helpful because the framework for sampling is different.  Arguably the most pressing need is for researched aimed at developing better measures of the quality of non-probability sampling estimates that include bias and precision.  

Although non-probability samples often have performed well in electoral polling, the evidence of their accuracy is less clear in other domains and in more complex surveys that measure many different phenomena.   Surveys designed to yield only a handful of estimates on a related set of outcomes may require the control of only a small set of covariates.  Frequent repetition, as in tracking surveys, and the availability of external benchmarks (such as election results) make experimentation possible and may make model development easier. However, many surveys do not have these advantages.  A survey often produces many estimates across a broad array of subject areas and domains, requiring a larger set of covariates. 

Non-probability samples may be appropriate for making statistical inferences, but the validity of the inferences rests on the appropriateness of the assumptions underlying the model and how deviations from those assumptions affect the specific estimates. Throughout the report, we have emphasized the need for further development of a theoretical basis for any non-probability sampling method to be followed by empirical evaluation of that method. The evaluation should assess the appropriateness of the assumptions under various circumstances and for different estimates. Our review identified sample matching as one of method that already has a theoretical basis constructed for evaluation studies that could be modified and amplified for use with surveys. Several researchers have begun this effort already. The post-survey adjustment methods applied to non-probability sampling have largely mirrored efforts in probability samples. Although this may be appropriate and effective to some extent, further consideration of selection bias mechanisms may be needed. We believe an agenda for advancing a method must include these attributes.