3 In practice, few contemporary surveys, not even probability samples, might be classified as low risk using these criteria
REFERENCES
AAPOR (American Association for Public Opinion Research). 2012. “Understanding a ‘Credibility Interval’ and How it Differs from the ‘Margin of Sampling Error’ in a Public Opinion Poll.” Downloaded from http://aapor.org/AM/Template.cfm?Section=Understanding_a_credibility_interval_and_how_it_differs_from_the_margin_of_sampling_error_in_a_publi&Template=/CM/ContentDisplay.cfm&ContentID=5475
AAPOR. (American Association for Public Opinion Research). 2009.
Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. Revised 2011.
Abate, Tom. 1998. “Accuracy of Online Surveys May Make Phone Polls Obsolete.”
The San Francisco Chronicle, D1.
Alvarez, R. Michael, and Carla VanBeselaere. 2005. “Web-Based Surveys.
” The Encyclopedia of Measurement. California Institute of Technology. http://www.mta.ca/~cvanbese/encyclopedia_new2.pdf.
Asur, Sitaram, and Bernardo A. Huberman. 2010. “Predicting the Future with Social Media.” Downloaded from http://arxiv.org/pdf/1003.5699v1.
Australian Bureau of Statistics 2009. ABS Data Quality Framework, Downloaded from http://www.abs.gov.au/ausstats/
[email protected]/mf/1520.0 on 4/30/2013.
Baker, Reg, Stephen J. Blumberg, J. Michael Brick, Mick P. Couper, Melanie Courtright, J. Michael Dennis, Don Dillman, Martin R. Frankel, Philip Garland, Robert M. Groves, Courtney Kennedy, Jon Krosnick, Paul J. Lavrakas, Sunghee Lee, Michael Link, Linda Piekarski, Kumer Rao, Randall K. Thomas, and Dan Zahs. 2010. “AAPOR Report on Online Panels.”
Public Opinion Quarterly 74(4):711–81.
Banks, David. 2011. “Reproducible Research: A Range of Response Statistics.”
Politics, and Policy, 2, DOI: 10.2202/2151–7509.1023.
Berinsky, Adam J. 2006. “American Public Opinion in the 1930s and 1940s: The Analysis Of Quota-Controlled Sample Survey Data.”
Public Opinion Quarterly 70(4):499–529.
Bernhardt, Annette, Ruth Milkman, Nik Theodore, Douglas Heckathorn, Mirabai Auer, James DeFilippis, Ana Luz González, Victor Narro, Jason Perelshteyn, Diane Polson, and Michael Spiller. 2009. “Broken Laws, Unprotected Workers: Violations of Employment and Labor Laws in America’s Cities.” Downloaded from http://www.nelp.org/page/-/brokenlaws/BrokenLawsReport2009.pdf?nocdn=1.
Berzofsky, Marcus E., Rick L. Williams, and Paul P. Biemer. 2009. “Combining Probability and Non-Probability Sampling Methods: Model-Aided Sampling and the O*NET Data Collection Program.”
Survey Practice August: 1-5. http://surveypractice.files.wordpress.com/2009/08/berzofsky.pdf.
Bethlehem, Jelke. 2010. “Selection Bias in Web Surveys.”
International Statistical Review 78(2):161–88.
Bethlehem, Jelke, and Silvia Biffignandi. 2012.
Handbook of Web Surveys. Hoboken, New Jersey: John Wiley & Sons Inc.
Bethlehem, Jelke, Fannie Cobben, and Barry Schouten. 2011.
Handbook of Nonresponse in Household Surveys. Hoboken, NJ: John Wiley & Sons.
Biemer, Paul P., and Lars E. Lyberg. 2003.
Introduction to Survey Quality. New York: Wiley.
Biernacki, Patrick, and Dan Waldorf. 1981. “Snowball Sampling: Problem And Techniques Of Chain Referral Sampling.”
Sociological Methods and Research 10(2):141–63.
Birnbaum, Zygmunt William, and Monroe G. Sirken. 1965. “Design of Sample Surveys to Estimate the Prevalence of Rare Diseases: Three Unbiased Estimates.” National Center for Health Statistics.
Vital and Health Statistics 2(11).
Blair, Johnny, and Frederick Conrad. 2011. “Sample Size for Cognitive Interview Pretesting.”
Public Opinion Quarterly 75(4):636–58.
Brick, J. Michael. 1990. “Multiplicity Sampling in an RDD Telephone Survey.” Sampling Design Issues section of the
American Statistical Association 296–301.
Brick, J. Michael. 2011. “The Future Of Survey Sampling.”
Public Opinion Quarterly 75(5):872–88.
Brick, J. Michael, and Douglas Williams. 2013. “Explaining Rising Nonresponse Rates in Cross-Sectional.”
The ANNALS of the American Academy of Political and Social Science 645(1):36–59.
Bryson, Maurice C. 1976. “The Literary Digest Poll: Making of a Statistical Myth.”
American Statistical Association 30(4):184–85.
Callegaro, Mario, and Charles DeSogra. 2008. “Computing Response Metrics for Online Panels.”
Public Opinion Quarterly 72(5):1008–32.
Campbell, Donald T., and Julian C. Stanley. 1963.
Experimental And Quasi-Experimental Designs For Research. Chicago, IL: Rand-McNally.
Carlson, Robert G., Jichuan Wang, Harvey A. Siegal, Russel S. Falck, Jie Guo. 1994. “An Ethnographic Approach To Targeted Sampling: Problems And Solutions In AIDS Prevention Research Among Injection Drug And Crack-Cocaine Users.”
Human Organization 53:279–86.
Centers for Disease Control and Prevention. 2005. “Statistical Methodology of the National Immunization Survey 1994-2002.”
Vital and Health Statistics Series 2, 138.
Chang, Linchiat, and Jon A. Krosnick. 2009. “National Surveys via RDD Telephone Interviewing Versus the Internet: Comparing Sample Representativeness and Response Quality.”
Public Opinion Quarterly 73(4):641–78.
Chui. Michael, Markus Löffler, and Roger Roberts. 2010. “The Internet of Things.”
McKinsey Quarterly.
Clinton, Joshua D. 2001. “Panel Bias from Attrition and Conditioning: A Case Study of the Knowledge Networks Panel.” Paper presented at 56
th Annual Conference of the American Association for Public Opinion Research, May, Montreal, Canada.
Cochran, William G. 1965. “The Planning of Observational Studies of Human Populations.”
Journal of the Royal Statistical Society Series A, 128(2):234–66.
Coleman, James S., 1958. “Relational Analysis: The Study of Social Organizations with Survey Methods.”
Human Organization 17:28–36.
Copas, John B. and H. G. Li. 1997. “Inference for Non-random Samples.”
Journal of the Royal Statistical Society Series B, 59:5-95.
Cornfield, Jerome. 1971. “The University Group Diabetes Program: A Further Statistical Analysis of the Mortality Findings.”
Journal of the American Medical Association 217(12):1676–87.
Couper, Mick P., Arie Kapteyn, Matthias Schonlau, and Joachim Winter. 2007. “Noncoverage and Nonresponse in an Internet Survey.”
Social Science Research 36:131–48.
Couper, Mick P. 2000. “Web Surveys: A Review of Issues and Approaches.”
Public Opinion Quarterly 64(4):464–94.
Couper, Mick P. 2007. “Issues of Representation in eHealth Research with a Focus on Web Surveys.”
American Journal of Preventive Medicine 32:S83–S89.
Couper, Mick P., and Michael Bosnjak. 2010. “Internet Surveys.”
Handbook of Survey Research Chapter 16, P.V. Marsden and J.D. Wright editors, Bingley, UK: Emerald Group Publishing Limited.
Curtin, Richard, Stanley Presser, and Eleanor Singer. 2005. “Changes in Telephone Survey Nonresponse Over the Past Quarter Century.”
Public Opinion Quarterly 69(1):87–98.
Dawber, Thomas R., Gilcin F. Meadors, and Felix E. Moore. 1951. “Epidemiological Approaches to Heart Disease: The Framingham Study.”
American Journal of Public Health 41:279–86.
de Munnik, Daniel, David Dupuis, and Mark Illing. 2009. “Computing the Accuracy of Complex Non-Random Sampling Methods: The Case of the Bank of Canada’s Business Outlook Survey.” Bank of Canada Working Paper 2009–10, March 2009. http://www.bankofcanada.ca/wp-content/uploads/2010/02/wp09-10.pdf.
Dever, Jill A., Ann Rafferty, and Richard Valliant. 2008. “Internet Surveys: Can Statistical Adjustments Eliminate Coverage Bias?”
Survey Research Methods 2:47–62.
Deville, J.C., and Särndal, C.E. (1992). “Calibration estimators in survey sampling.”
Journal of the American Statistical Association, 87, 376-382.
Deville, J.C.. 1991. “A Theory of Quota Surveys.”
Survey Methodology 17:163–81.
Diamond, Shari S. 2000. “Reference Guide on Survey Research.” In
Reference Manual on Scientific Evidence 2
nd Edition. Washington, DC: Federal Judicial Center.
DiSogra, Charles. 2008. “River Samples: A Good Catch for Researchers?” In GfK Knowledge Networks http://www.knowledgenetworks.com/accuracy/fall-winter2008/disogra.html
Duffield, Nick. 2004. “Sampling for Passive Internet Measurement: A Review.”
Statistical Science 19(3):472–498.
Duffy, Bobby, Kate Smith, George Terhanian, and John Bremer. 2005. “Comparing Data from Online and Face-to-Face Surveys.”
International Journal of Market Research 47:615–39.
Duncan, G. 2008. “When to Promote, and When to Avoid, a Population Perspective.”
Demography 45(4):763–84.
Efron, Brad and Rob Tibshirani. 1993.
An Introduction to the Bootstrap. CRC Press.
Elliott, Marc, and Amelia Haviland. 2007. “Use of a Web-Based Convenience Sample to Supplement a Probability Sample.”
Survey Methodology 33(2):211–15.
Elliott, Michael R. 2009. “Combining Data from Probability and Non-Probability Samples Using Pseudo-Weights.”
Survey Practice. August. http://surveypractice.files.wordpress.com/2009/08/elliott.pdf.
Elżbieta, Getka-Wilczyńska. 2009. “Mathematical Modeling of the Internet Survey.” Warsaw School of Economics, Poland. http://www.intechopen.com/source/pdfs/8946/InTech-Mathematical_modeling_of_the_internet_survey.pdf.
Erikson, Robert S. and Christopher Wlezien. 2008. “Are Political Markets Really Superior to Polls as Election Predictors?”
Public Opinion Quarterly 72(2):190–215.
Eysenbach, Gunther. 2004. “Improving the Quality of Web Surveys: The Checklist for Reporting Results from Internet E-Surveys (CHERRIES).”
Journal of Medical Internet Research 6(3):e34.
Fan, David P. 2011. “Representative Responses from Non-Representative Survey Samples.” Paper presented at the Midwest Association of Public Opinion Research, Chicago.
Field, Lucy, Rachel A. Pruchno, Jennifer Bewley, Edward P. Lemay Jr, Norman G. Levinsky. 2006. “Using Probability vs. Nonprobability Sampling to Identify Hard-to-Access Participants for Health-Related Research: Costs and Contrasts.”
Journal of Aging and Health 18(4):565–83.
Felix-Medina, Martin H., and Steven K. Thompson. 2004. “Combining Link-Tracing Sampling and Cluster Sampling to Estimate the Size of Hidden Populations.”
Journal of Official Statistics 20(1):19–38.
Frank, Ove. 1971.
Statistical Inference in Graphs. Ph.D. thesis, Stockholm.
Frank, Ove. 1995. “Network Sampling and Model Fitting.” [Carrington, Peter J., John Scott, and Stanley Wasserman, eds.].
Models and Methods in Social Network Analysis. Cambridge University Press, 2005, 31–56. Cambridge Books Online. http://dx.doi.org/10.1017/CBO9780511811395.003.
Frankel, Martin R., and Frankel, Lester R. 1987. “Fifty Years of Survey Sampling in the United States.”
Public Opinion Quarterly 51 Part 2:S127–38.
Fricker, Scott, and Roger Tourangeau. 2010. “Examining the Relationship Between Nonresponse Propensity and Data Quality in Two National Household Surveys.”
Public Opinion Quarterly 74(5):934–55.
Fuller, Wayne A. 1987.
Measurement Error Models. New York, NY: John Wiley & Sons Inc.
Gile, Krista J. 2011. “Improved Inference for Respondent-Driven Sampling Data with Application to HIV Prevalence Estimation.”
Journal of the American Statistical Association 106:135–46.
Gile, Krista J., and Mark S. Handcock. 2010. “Respondent-Driven Sampling: An Assessment of Current Methodology.”
Sociological Methodology 40:285–327.
Gile, Krista J., and Mark S. Handcock. 2011. “Network Model-Assisted Inference from Respondent-Driven Sampling Data.” ArXiv Preprint.
Gile, Krista J., Lisa G. Johnston, and Matthew J. Salganik. 2012. “Diagnostics for Respondent-Driven Sampling.” arXiv:1209.6254. Under Review.
Gini, Corraco, and Luigi Galvani. 1929. “Di una Applicazione del Metodo Representative.”
Annali di Statistica 6(4):1–107.
Gittelman, Steven H. and Elaine Trimarchi. 2009. “Consistency: The New Quality Concern.” Marketing Research Association’s
Alert! Magazine, October, 49(10):19–21.
Gittleman, Steven H. and Elaine Trimarchi 2010. “Online Research…and All that Jazz! The Practical Adaptation of Old Tunes to Make New Music.” Online Research 2010. Amsterdam: ESOMAR.
Gjoka, Minas, Maciej Kurant, Carter T. Butts, and Athina Markopoilou. 2011. “Practical Recommendations on Crawling Online Social Networks.”
JSAC special issue on Measurement of Internet Topologies 29(9):1872–92.
Glasser, Gerald J., and Gale D. Metzger. 1972. “Random-Digit Dialing as a Method of Telephone Sampling.”
Journal of Marketing Research 9:59–64.
Goel, Sharad, and Matthew J. Salganik. 2010. “Assessing Respondent-Driven Sampling.”
Proceedings of the National Academy of Science of the United States of America 107(15):6743–47.
Goel, Sharad, and Matthew J. Salganik. 2009. “Respondent-Driven Sampling as Markov Chain Monte Carlo.”
Statistics in Medicine 28(17):2202–29.
Goodman, Leo A. 1961. “Snowball Sampling.”
Annals of Mathematical Statistics 32:148–70.
Groves, Robert M. 1989.
Survey Errors and Survey Costs. New York, NY: John Wiley & Sons Inc.
Groves, Robert M. 2006. “Nonresponse Rates and Nonresponse Bias in Household Surveys.”
Public Opinion Quarterly 70:646–75.
Groves, Robert M., and Lars Lyberg. 2010. “Total Survey Error: Past, Present, and Future.”
Public Opinion Quarterly 74(5):849–79.
Groves, Robert M., Eleanor Singer, and Amy Corning. 2000. “Leverage-Saliency Theory of Survey Participation: Description and an Illustration.”
Public Opinion Quarterly 64:299–308.
Groves, Robert M., Floyd Fowler, Mick P. Couper, James Lepkowski, Eleanor Singer, and Roger Tourangeau. 2009.
Survey Methodology. Wiley: New York.
Groves, Robert M., Stanley Presser, and Sarah Dipko. 2004. “The Role of Topic Interest in Survey Participation Decisions.”
Public Opinion Quarterly 681:2–31.
Häder, S., and Gabler, S. 2003. “Sampling and Estimation.” In
Cross-Cultural Survey Methods, Janet A. Harkness, Fons J.R. van de Vijver, and Peter Ph. Mohler. (eds.). New York Wiley, pp. 117–34.
Handcock, Mark S., and Krista J. Gile. 2011. “On the Concept of Snowball Sampling.”
Sociological Methodology 41(1):367–71.
Hansen, Morris H., William N. Hurwitz. 1943. “On the Theory of Sampling from Finite Populations.”
Annals of Mathematical Statistics 14 (4):333–62.
Hansen, Morris H., William G. Madow, and Benjamin J. Tepping. 1983. “An Evaluation of Model Dependent and Probability-Sampling Inferences in Sample Surveys.”
Journal of the American Statistical Association 78(384):776–93.
Harris Interactive. 2004. “Final Pre-Election Harris Polls: Still Too Close to Call but Kerry Makes Modest Gains.” The Harris Poll #87, November 2, 2004. http://www.harrisinteractive.com/harris_poll/index.asp?pid=515.
Harris Interactive. 2008. “Election Results Further Validate Efficacy of Harris Interactive’s Online Methodology.” Press Release from Harris Interactive, November 6, 2008.
Heckathorn, Douglas D. 1997. “Respondent-Driven Sampling: A New Approach to the Study of Hidden Populations.”
Social Problems 44:174–99.
Heckman, James J. 1979. “Sample Selection Bias as a Specification Error.”
Econometrica 47:153–62. http://vanpelt.sonoma.edu/users/c/cuellar/econ411/heckman.pdf.
Henrich, Joseph, Steven J. Heine, and Ara Norenzayan. 2010. “Beyond WEIRD: Towards a Broad-Based Behavioral Science.”
Behavioral and Brain Sciences 33:111–35.
Intrade. 2012. http://www.intrade.com/v4/home/
Ipsos. 2012. “Ipsos Poll Conducted for Reuters.” Downloaded from www.ipsos-na.com/download/pr.aspx?id=11637
Isaksson, Annica and Sunghee Lee. 2005. “Simple Approaches to Estimating the Variance of the Propensity Score Weighted Estimator Applied on Volunteer Panel Web Survey Data - A Comparative Study.” SRMS proceedings. 3143–3149 http://www.amstat.org/sections/srms/proceedings/y2005/Files/JSM2005-000172.pdf
Pasek, Josh, and Krosnick, Jon A. 2010. “Measuring Intent to Participate and Participation in the 2010 Census and Their Correlates and Trends: Comparisons of RDD Telephone and Non-Probability Sample Internet Survey Data.” Census.gov Study Series: Survey Methodology #2010-15, Statistical Research Division, U.S. Census Bureau. https://www.census.gov/srd/papers/pdf/ssm2010-15.pdf
Kalton, Graham. 1993.
Sampling Rare and Elusive Populations. Department of Economic and Social Information and Policy Analysis Statistics Division, United Nations. New York.
Kalton, Graham. 2003. “Practical Methods for Sampling Rare and Mobile Populations.”
Statistics in Transition 6:491–501.
Kalton, Graham. 2009. “Methods of Oversampling Rare Population in Social Surveys
.”
Survey Methodology December 2009, 35(2):125–41.
Kalton, Graham, and Dallas W. Anderson. 1986. “Sampling Rare Populations.”
Journal of the Royal Statistical Society Series A, 149(1):65–82.
Kalton, Graham, and Ismael Flores-Cervantes. 2003. “Weighting Methods.”
Journal of Official Statistics 19(2):81–97.
Kaplan, Charles D., Dirk Korf, Claire Sterk. 1987. “Temporal and Social Contexts of Heroin-Using Populations: An Illustration of the Snowball Sampling Technique.”
The Journal Nervous and Mental Disease 175(9):566–74.
Kendall, Carl, Ligia Kerr, Rogerio Gondim, Guilherme Werneck, Raimunda Macena, Marta Pontes, Lisa Johnston, Keith Sabin, and Willi McFarland. 2008. “An Empirical Comparison of Respondent-Driven Sampling, Time Location Sampling, and Snowball Sampling for Behavioral Surveillance in Men Who Have Sex with Men, Fortaleza, Brazil.”
AIDS and Behavior, 12(1):97–104.
Kiaer, Anders Nicolai. 1895-6. “Observations et Experiences Concernant des Denobrements Represenatifs.”
Bulletin of the International Statistical Institute 9, Liv. 2:176–83.
Kind, Allison. 2012. Tweeting the News, Case Study: News Organizations’ Twitter Coverage of the 2011 State of the Union Address. http://www.american.edu/soc/communication/upload/Allison-Kind.pdf
Kish, Leslie. 1987.
Statistical Design for Research. John Wiley & Sons, New York.
Kish, Leslie. 1995. “The Hundred Years’ War of Survey Sampling.”
Statistics in Transition 2(5):813–30. Reprinted in 2003, Graham Kalton and Steven Heeringa, eds.
Leslie Kish: Selected papers. New York, Wiley.
Kish, Leslie. 1965. “Selection Techniques for Rare Traits.”
Genetics and the Epidemiology of Chronic Diseases, Public Health Service Publication No. 1163.
Kish, Leslie. 1965.
Survey Sampling. New York: John Wiley & Sons, Inc.
Kleinbaum, David G., Hal Morgenstern, and Lawrence L. Kupper. 1981. “Selection Bias in Epidemiologic Studies.”
American Journal of Epidemiology 113(4):452–63.
Klovdahl, Alden S., John J. Potterat, Donald E. Woodhouse, John B. Muth, Stephen Q. Muth, and William W. Darrow. 1994. “Social Networks and Infectious Disease: The Colorado Springs Study.”
Social Science & Medicine 38(1):79–88.
Kogan, Steven M., Cyprian Wejnert, Yi-fu Chen, Gene H. Brody, and LaTrina M. Slater. 2011. “Respondent-Driven Sampling with Hard-to-Reach Emerging Adults: An Introduction and Case Study with Rural African Americans.”
Journal of Adolescent Research 26(1):30–60.
Kott, Phillip S. 2006. “Using Calibration Weighting to Adjust for Nonresponse and Coverage Errors.”
Survey Methodology 32(2):133–142.
Kreuter, Frauke, Kristen Olson, James Wagner, Ting Yan, Trena M. Ezzati-Rice, Carolina Casas-Cordero, Michael Lemay, Andy Peytchev, Robert M. Groves, and Trivelore Raghunathan. 2011. “Using Proxy Measures and Other Correlates of Survey Outcomes to Adjust for Non-Response: Examples from Multiple Surveys.”
Journal of the Royal Statistical Society Series A 173(2):389–407.
Kruskal, William, and Frederick Mosteller. 1980. “Representative Sampling, IV: The History of the Concept in Statistics, 1895-1939.”
International Statistical Review 48:169–95.
Kruskal, William, and Frederick Mosteller. 1981. “Ideas of Representative Sampling.”
New Directions for Methodology of Social and Behavioral Science: Problems with Language Imprecision 3–24.
Lagakos, Stephen W., and Louise M. Ryan. 1985. “On the Representativeness Assumption in Prevalence Tests of Carcinogenicity.”
Applied Statistics 34:54–62.
Lavrakas, Paul J., Charles D. Shuttles, Charlotte Steeh, and Howard Fienberg. 2007. “The State of Surveying Cell Phone Numbers in the United States–2007 Special Issue.”
Public Opinion Quarterly 71(5):840–854.
Lee, Sunghee. 2006. “An Evaluation of Nonresponse and Coverage Errors in a Web Panel Survey.”
Social Science Computer Review 2(4):460–75. http://ssc.sagepub.com/content/24/4/460.abstract.
Lee, Sunghee. 2006. “Propensity Score Adjustment as a Weighting Scheme for Volunteer Panel Web Surveys.”
Journal of Official Statistics 22(2):329–49.
Lee, Sunghee, and Richard Valliant. 2009. “Estimation for Volunteer Panel Web Surveys using Propensity Score Adjustment and Calibration Adjustment.”
Sociological Methods and Research 37(3):319-43. http://smr.sagepub.com/content/37/3/319.full.pdf.
Lee, Sunghee, and Richard Valliant. 2008. “Weighting Telephone Samples Using Propensity Scores.” In
Advances in Telephone Survey Methodology, edited by J. M. Lepkowski, C. Tucker, J. M. Brick, E. D. de Leeuw, L. Japec, P. J. Lavrakas, M. W. Link, and R. L. Sangster, 170–83. Hoboken, NJ: John Wiley & Sons, Inc.
Lensvelt-Mulders, Gerty J. L. M., Peter J. Lugtig, and Marianne Hubregtse. 2009. September. “Separating Selection Bias and Non-Coverage in Internet Panels Using Propensity Matching.”
Survey Practice, 2 [e-journal].
Lesser, Virginia. “Advantages and Disadvantages of Probability and Non-Probability-Based Surveys of the Elderly and Disabled.” [online presentation]. http://ncat.oregonstate.edu/pubs/TRANSED/1081_Surveys.pdf
Levy, Paul S., and Stanley Lemeshow. 2008.
Sampling of Populations: Methods and Applications. 4th ed. Hoboken, NJ: John Wiley & Sons, Inc.
Little, Roderick J.A., and Donald Rubin. 2002.
Statistical Analysis with Missing Data. 2nd ed. Hoboken, NJ: John Wiley & Sons, Inc.
Little, Roderick J.A., and Sonia L. Vartivarian. 2004. “Does Weighting for Nonresponse Increase the Variance Of Survey Means?” April 2004. Working Paper 35.
The University of Michigan Department of Biostatistics Working Paper Series.
Little, Roderick J.A. 1986. “Survey Nonresponse Adjustments for Estimates of Means.”
International Statistical Review 54(2):139–57.
Little, Roderick J.A. 1988. “Missing-Data Adjustments in Large Surveys.”
Journal of Business & Economic Statistics 63:287–296.
Lohr, Sharon L. 1999.
Sampling: Design and Analysis. Pacific Grove, CA: Brooks/Cole Publishing Company.
Loosveldt, Geert, and Nathalie Sonck. 2008. “An Evaluation of the Weighting Procedures for an Online Access Panel Survey.”
Survey Research Methods 2:93–105.
Lynn, Peter. 2004. “The Use of Substitution in Surveys.”
The Survey Statistician 49:14–6.
Lynn, Peter, and Roger Jowell. 1996. “How Might Opinion Polls Be Improved? The Case for Probability Sampling.”
Journal of the Royal Statistical Society Series A 15:21–8.
MacKellar, Duncan, Kathleen M. Gallagher, Teresa Finlayson, Travis Sanchez, Amy Lansky, and Patrick S. Sullivan. 2007. “Surveillance of HIV Risk and Prevention Behaviors of Men Who Have Sex With Men—A National Application of Venue-Based, Time-Space Sampling.”
Public Health Reports 122(1):39–47.
Malhotra, Neil, and Jon A. Krosnick. 2007. “The Effect of Survey Mode and Sampling on Inferences about Political Attitudes and Behavior: Comparing the 2000 and 2004 ANES to Internet Surveys with Nonprobability Samples.”
Political Analysis 15:286–323.
Manyika, James, Michael Chui, Brad Brown, Jacques Bughin, Richard Dobbs, Charles Roxburgh, and Angela Hung Byers. 2011.
Big Data: The Next Frontier for Innovation, Competition, and Productivity. Seoul, San Francisco, London, and Washington, DC: McKinsey Global Institute. http://www.mckinsey.com/insights/mgi/research/technology_and_innovation/big_data_the_next_frontier_for_innovation.
Matthews, Vince. 2008. “Probability or Nonprobability: A Survey is a Survey–or Is It?” National Agricultural Statistics Service (NASS) white paper. http://www.nass.usda.gov/Education_and_Outreach/Understanding_Statistics/Statistical_Aspects_of_Surveys/survey_is_survey.pdf
Mayes, Linda C., Ralph I. Horwitz, and Alvan R. Feinstein. 1988. “A Collection of 56 Topics with Contradictory Results in Case-Control Research.”
International Journal of Epidemiology 17:680–5.
McKenzie, David. J, and Johan Mistiaen. 2009. “Surveying Migrant Households: A Comparison of Census-Based, Snowball and Intercept Point Surveys.”
Journal of the Royal Statistical Society Series A, 172:339–60.
McPherson, Miller, Lynn Smith-Lovin, and James M. Cook. 2001. “Birds of a Feather: Homophily in Social Networks.”
Annual Review of Sociology 27:415–44.
Moser, Claus Adolf, and Alan Stuart. 1953. “An Experimental Study of Quota Sampling.”
Journal of the Royal Statistical Society: Series A, 116:349–405.
Mosteller, Frederick, Herbert Hyman, Philip J. McCarthy, Eli S. Marks, David B. Truman, et al. 1949. “The Pre-Election Polls of 1948.” Report to the Committee on Analysis of Pre-Election Polls and Forecasts. Bulletin 60. New York: Social Science Research Council.
Muhib, Farzana B., Lillian S. Lin, Ann Stueve, Robin L. Miller, Wesley L. Ford, Wayne D. Johnson, and Philip J. Smith. 2001. “A Venue-Based Method for Sampling Hard-To-Reach Populations.”
Public Health Reports 116(1):216–22.
Mutz, Diana. 2011.
Population-Based Survey Experiments. Princeton, NJ: Princeton University Press.
National Research Council. 2012. Assessing the Reliability of Complex Models: Mathematical and Statistical Foundations of Verification, Validation, and Uncertainty Quantification. Washington, D.C.: The National Academies Press.
New York Times. 2008. “Google Uses Searches to Track Flu’s Spread.” November 11. http://www.nytimes.com/2008/11/12/technology/internet/12flu.html?_r=2.
New York Times. 2012. “Which Polls Fared Best and Worse in the 2012 Presidential Race.” November 10. http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and-worst-in-the-2012-presidential-race/.
Neyman, Jerzy. 1934. “On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection.”
Journal of the Royal Statistical Society 97:558–625.
O’Connor, Brendan, Ramnath Balasubramanyan, Bryan R. Routledge, and Noah A. Smith. 2010. “From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series.” In
Proceedings of the Fourth International Association for the Advancement of Artificial Intelligence Conference on Weblogs and Social Media, Washington, DC, May 2010. www.aaai.org.
Office of Management and Budget (OMB). 2006.
Standards and Guidelines for Statistical Surveys. http://www.whitehouse.gov/sites/default/files/omb/inforeg/statpolicy/standards_stat_surveys.pdf. (DUPLICATE—SEE U.S. OFFICE OF MANAGEMENT AND BUDGET page 7)
Olivier, Lex. 2011. “River Sampling Non-Probability Sampling in an Online Environment.” [Web log, November 13, 2011.] Center for Information-Based Decision Making and Marketing Research. http://lexolivier.blogspot.com/2011/11/river-sampling-non-probability-sampling.html
Olson, Kristen. 2006. “Survey Participation, Nonresponse Bias, Measurement Error Bias, and Total Bias.”
Public Opinion Quarterly 70(5):737–58.
Peracchi, Franco, and Finis Welch. 1995. “How Representative are Matched Cross-Sections? Evidence from the Current Population Survey.”
Journal of Econometrics 68:153–79.
Pew Research Center. 2011. “Muslim Americans: No Signs of Growth in Alienation or Support for Extremism.” Survey Report. Washington, DC: Pew Research Center. http://www.people-press.org/2011/08/30/muslim-americans-no-signs-of-growth-in-alienation-or-support-for-extremism/
Peytchev, Andy, Sarah Riley, Jeffrey Rosen, Joe Murphy, and Mark Lindblad. 2010. “Reduction of Nonresponse Bias in Surveys Through Case Prioritization.”
Survey Research Methods 4(1):21–9.
Pfeffermann, Danny, and C.R. Rao, editors. 2009.
Sample Surveys: Inference and Analysis. Handbook of Statistics, Vol 29B. Oxford: Elsevier B.V.
Poynter, Ray. 2010.
The Handbook of Online and Social Media Research. Chichester, United Kingdom: Wiley.
Presser, Stanley. 1984. “The Use of Survey Data in Basic Research in the Social Sciences.” In
Surveying Subjective Phenomena, vol. 2, edited by C. F. Turner and E. Martin, 93–114. New York: Russell Sage.
Public Works and Government Services Canada. 2008. “The Advisory Panel on Online Public Opinion Survey Quality —Final Report.” http://www.tpsgc-pwgsc.gc.ca/rop-por/rapports-reports/comiteenligne-panelonline/tdm-toc-eng.html
Rao, J.N.K. and C.F.J. Wu. 1988. “Resampling Inference with Complex Survey Data.”
Journal of the American Statistical Association 83:231–41.
Rao, John N.K. 2003.
Small Area Estimation. Hoboken, NJ: John Wiley & Sons, Inc.
Ribeiro, Bruno, and Don Towsley. 2010. “Estimating and Sampling Graphs with Multidimensional Random Walks.” In Proceedings of the 10th ACM SIGCOMM conference on Internet measurement. http://arxiv.org/pdf/1002.1751.pdf.
Rivers, Douglas. 2007. “Sampling for Web Surveys.” White paper prepared from presentation given at the 2007 Joint Statistical Meetings, Salt Lake City, Utah, July-August. https://s3.amazonaws.com/yg-public/Scientific/Sample+Matching_JSM.pdf.
Rivers, Douglas, and Delia Bailey. 2009. “Inference from Matched Samples in the 2008 U.S. National Elections.” Paper presented at the 64th Annual Conference of the American Association for Public Opinion Research, Hollywood, Florida, May.
Rivers, Douglas. 2007. “Sample Matching for Web Surveys: Theory and Application.” Paper presented at the 2007 Joint Statistical Meetings, Salt Lake City, Utah, July-August.
Robinson, William T., Jan M.H. Risser, Shanell McGoy, Adam B. Becker, Hafeez Rehman, Mary Jefferson, Vivian Griffin, Marcia Wolverton, and Stephanie Tortu. 2006. “Recruiting Injection Drug users: A Three-Site Comparison of Results and Experiences with Respondent-Driven and Targeted Sampling Procedures.”
Journal of Urban Health 83(1):29–38.
Rosenbaum, Paul R. 2005. "Observational Study." in Everitt & Howell, eds.l
Encyclopedia of Statistics in Behavioral Science. Vol 3, 1451–1462.
Rosenbaum, Paul, and Donald B. Rubin. 1983. “The Central Role of the Propensity Score in Observational Studies for Causal Effects.”
Biometrika 70:41–55.
Rosenbaum, Paul R., and Donald B. Rubin. 1984. “Reducing Bias in Observational Studies Using Subclassification on the Propensity Score.”
Journal of the American Statistical Association 79:516–24.
Rothman, Kenneth J., and Sander Greenland. 1998.
Modern Epidemiology. 2nd ed. Philadelphia: Lippincott Williams & Wilkins.
Rothman, Kenneth J., Sander Greenland, and Timothy L. Lash. 2008.
Modern Epidemiology. 3rd ed. Philadelphia: Lippincottt Williams & Wilkins.
Rothschild, David. 2009. “Forecasting Elections: Comparing Prediction Markets, Polls, and Their Biases.”
Public Opinion Quarterly 73(5):895–916.
Royall, Richard. 1970. “On Finite Population Sampling Theory Under Certain Linear Regression Models.”
Biometrika 57:377-87.
Rubin, Donald B. 2008. "For objective causal inference, design trumps analysis."
The Annals of Applied Statistics, 2, 808-840.
Rubin, Donald B. 1979. “Using Multivariate Matched Sampling and Regression Adjustment to Control Bias in Observational Studies.”
Journal of the American Statistical Association 74:318–28.
Salganik, Matthew J. 2006. “Variance Estimation, Design Effects, and Sample Size Calculations for Respondent-Driven Sampling.”
Journal of Urban Health 83:98–112.
Salganik, Matthew J., and Douglas D. Heckathorn. 2004. “Sampling and Estimation in Hidden Populations Using Respondent-Driven Sampling.”
Sociological Methodology 34:193–239.
Särndal, Carl-Erik, Bengt Swensson, and Jan Wretman. 1992.
Model-Assisted Survey Sampling. New York: Springer-Verlag, Inc.
Savage, Mike, and Roger Burrows. 2007. "The coming crisis of empirical sociology."
Sociology, 41, 885-899.
Schield, Milo. 1994. “Random Sampling Versus Representative Samples.”
American Statistical Association Proceedings Of The Section On Statistical Education 107–10. Downloaded from www.StatLit.org/pdf/1994SchieldASA.pdf.
Schillewaert, Niels, Tom De Ruyck, and Annelies Verhaeghe. 2009. “Connected Research – How Market Research Can Get the Most Out of Semantic Web Waves.”
International Journal of Market Research 51(1):11–27.
Schonlau, Matthias, Arthur van Soest, and Arie Kapteyn. 2007. “Are ‘Webographic’ or Attitudinal Questions Useful for Adjusting Estimates from Web Surveys Using Propensity Scoring?”
Survey Research Methods 1:155–63.
Schonlau, Matthias, Arthur van Soest, Arie Kapteyn, and Mick Couper. 2009. “Selection Bias in Web Surveys and the Use of Propensity Scores.”
Sociological Methods & Research 37:291–318.
Schonlau, Matthias, Kinga Zapert, Lisa Payne Simon, Katherine Sanstad, Sue Marcus, John Adams, Mark Spranca, Hongjun Kan, Rachel Turner, and Sandra Berry. 2004. “A Comparison Between Responses from a Propensity-Weighted Web Survey and an Identical RDD Survey.”
Social Science Computer Review 22:128–38.
Sears, David O. 1986. “College Sophomores in the Laboratory: Influences of a Narrow Data Base on Social Psychology’s View of Human Nature.”
Journal of Personality and Social Psychology 51:515–30.
Shadish, William R., Thomas D. Cook, and Donald T. Campbell. 2001.
Experimental and Quasi-Experimental Designs for Generalized Causal Inference. 2nd ed. Stamford, CT: Cengage Learning/Wadsworth Publishing.
Shadish, William R., M. H. Clark, and Peter Steiner. 2008. “Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments.”
Journal of the American Statistical Association 103:1334–44.
Silver, Nate. 2012. (See New York Times 2012 above.)
Smith, Aaron. 2011.
Trends in Cell Phone Usage and Ownership. Washington, D.C.: Pew Research Center. http://www.pewinternet.org/Presentations/2011/Apr/FTC-Debt-Collection-Workshop-Cell-Phone-Trends.aspx
Smith, T. M. F. 1983. “On The Validity of Inferences From Non-Random Sample.”
Journal of the Royal Statistical Society Series A, 146(4):394–403.
Snell, Laurie, J., Peterson, Bill and Grinstead, Charles. (1998). “Chance News 7.11.” Downloaded from http://www.dartmouth.edu/~chance/chance_news/recent_news/chance_news_7.11.html on August 31, 2009. Stirton Stirton
Snow, Rob E., John D. Hutcheson, James E. Prather. 1981. “Using Reputational Sampling to Identify Residential Clusters of Minorities Dispersed in a Large Urban Region: Hispanics in Atlanta, Georgia.” Proceedings of the section on Survey Research Methods,
American Statistical Association 101–6.
Squire, Peverill. 1988. “Why the 1936 Literary Digest Poll Failed
.”
Public Opinion Quarterly 52:125–33.
Statistics Canada 2002. Statistics Canada’s Quality Assurance Framework. Available from http://www5.statcan.gc.ca/bsolc/olc-cel/olc-cel?catno=12-586-X&CHROPG=1&lang=eng.
Statistics Canada. 2009.
Statistics: Power from Data! Nonprobability sampling. http://www.statcan.gc.ca/edu/power-pouvoir/ch13/nonprob/5214898-eng.htm
Steckler, Allan, and Kenneth R. McLeroy. 2008. “The Importance of External Validity.”
American Journal of Public Health 98:9–10.
Steiner, Peter M., Thomas D. Cook, and William R. Shadish. 2011. “On the Importance of Reliable Covariate Measurement in Selection Bias Adjustments Using Propensity Scores.”
Journal of Educational and Behavioral Statistics 36:213–36.
Stephan, Franklin F., and Philip J. McCarthy. 1958.
Sampling Opinions: An Analysis of Survey Procedure. Oxford: John Wiley & Sons, Inc.
Stolzenberg, Ross M., and Daniel A. Relles. 1997. “Tools for Intuition About Sample Selection Bias and its Correction.”
American Sociological Review 62:494–507.
Stone, Mary Bishop, Joseph L. Lyon, Sara Ellis Simonsen, George L. White, and Stephen C. Alder. 2007. “An Internet-Based Method of Selecting Control Populations for Epidemiological Studies.”
Practice of Epidemiology 165:109–12.
Strauss, Murray A. 2009. “Validity of Cross-National Research Using Unrepresentative Convenience Samples.”
Survey Practice 43(3).
Sudman, Seymour. 1966. “Probability Sampling with Quotas.”
Journal of the American Statistical Association 20:749–71.
Sudman, Seymour, and Brian Wansink. 2002.
Building a Successful Convenience Panel. Chicago, IL: American Marketing Association.
Sugden, R.A. and Smith, T.M.F. 1984. “Ignorable and Informative Designs in Survey Sampling Inference.”
Biometrika 71(3):495–506.
Taylor, Humphrey. 2007. “The Case for Publishing (Some) Online Polls.” Polling Report.
Terhanian, George, and John Bremer. 2012. “A Smarter Way to Select Respondents for Surveys?”
International Journal of Market Research 54(6):751–780.
Terhanian, George, and John Bremer. 2000. “Confronting the Selection-Bias and Learning Effects Problems Associated with Internet Research.” Research paper: Harris Interactive. http://growingupwithmedia.com/pdf/Confronting_Selection_Bias.pdf.
Terhanian, George, and John Bremer. 1995. “Creative Applications of Selection Bias Modeling in Market Research.” ISI paper.
Terhanian, George, Jonathan W. Siegel, Cary Overmeyer, John Bremer, and Humphrey Taylor. 2001. “The Record of Internet-Based Opinion Polls in Predicting the Results of 72 Races in the November 2000 U.S. Elections.”
International Journal of Market Research 43(2):127–135.
The Telegraph. 2012. “Wisdom Index Poll Puts Labour Eight Points Ahead of Conservatives.” http://www.telegraph.co.uk/comment/9307396/Wisdom-index-poll-puts-Labour-eight-points-ahead-of-Conservatives.html
Thompson, Steven K. 1990. “Adaptive Cluster Sampling.”
Journal of the American Statistical Association 85:1050–59.
Thompson, Steven K. 1992.
Sampling. Wiley, New York.
Thompson, Steven K., Linda M. Collins. 2002. “Adaptive Sampling in Research on Risk-Related Behaviors.”
Drug and Alcohol Dependence 68(1):S57–67.
Thompson, Steven K., and Ove Frank. 2000. “Model-Based Estimation with Link-Tracing Sampling Designs.”
Survey Methodology 26:87–98.
Thompson, Steven K., George A.F. Seber. 1996.
Adaptive Sampling. Wiley, New York.
Thompson, Steven K. 2006a. “Targeted Random Walk Designs.”
Survey Methodology 32:11–24.
Thompson, Steven K. 2006b. “Adaptive Web Sampling.”
Biometrics 62:1224–34.
Thompson, Steven K. 1990. “Adaptive Cluster Sampling.”
Journal of the American Statistical Association 85(412):1050–59.
Thompson, Steven K. 2002.
Sampling. New York: John Wiley & Sons, Inc.
Tighe, Elizabeth, David Livert, Melissa Barnett, and Leonard Saxe. 2010. “Cross-Survey Analysis to Estimate Low-Incidence Religious Groups.”
Sociological Methods & Research 39:56–82.
Tourangeau, Roger, Frederick G. Conrad, and Mick P. Couper. 2013.
The Science of Web Surveys. New York: Oxford University Press.
Trow, Martin. 1957.
Right-Wing Radicalism and Political Intolerance. Arno Press, New York, Reprinted 1980.
Twyman, Joe. 2008. “Getting It Right: Yougov and Online Survey Research In Britain.”
Journal of Elections, Public Opinion & Parties 18:343-354.
U.S. Census Bureau. 2011.
U.S. Census Bureau Statistical Quality Standards. Washington, D.C.
U.S. Department of Agriculture (USDA). 2006. “The Yield Output Forecasting Program of NASS.” Downloaded from http://www.nass.usda.gov/Education_and_Outreach/Understanding_Statistics/yldfrcst2006.pdf
U.S. Office of Management and Budget. 2006.
Standards and Guidelines for Statistical Surveys. Washington, D.C. http://www.whitehouse.gov/sites/default/files/omb/inforeg/statpolicy/standards_stat_surveys.pdf
Valliant, Richard and Jill A. Dever. 2011. “Estimating Propensity Adjustments for Volunteer Web Surveys.”
Sociological Methods & Research 40(1):105–137. http://smr.sagepub.com/content/40/1/105.
Valliant, Richard, Alan H. Dorfman, and Richard M. Royall. 2000.
Finite Population Sampling and Inference. New York: John Wiley & Sons, Inc.
Vavreck, Lynn and Rivers, Douglas. 2008. “The 2006 Cooperative Congressional Election Study.”
Journal of Elections, Public Opinion & Parties 18(4):355–66.
Vehovar, Vasja. 1995. “Field Substitutions in Slovene Public Opinion Survey.” In
Contributions to Methodology and Statistics Metodološki zvezki, 10. A. Ferligoj and A. Kramberger eds. Ljubljana: FDV, 39–66.
Vehovar, Vasja. 1999. “Field Substitution and Unit Nonresponse.”
Journal of Official Statistics 15(2):335–50.
Volz, Erik, and Douglas Heckathorn. 2008. “Probability Based Estimation Theory for Respondent Driven Sampling.”
Journal of Official Statistics 24:79–97.
Walker, Robert, and Raymond Pettit. 2009.
ARF Foundations of Quality: Results preview. New York: The Advertising Research Foundation.
Watters, John K., and Patrick Biernacki. 1989. “Targeted Sampling: Options for the Study of Hidden Populations.”
Social Problems 36(4):416–30.
Wejnert, Cyprian, and Douglas D. Heckathorn. 2007. “Web-Based Network Sampling: Efficiency and Efficacy of Respondent-Driven Sampling for Online Research.”
Sociological Methods and Research 37(1):105–34.
Welch, Susan. 1975. “Sampling by Referral in a Dispersed Population.”
Public Opinion Quarterly 39:237–45.
Winship, Christopher, and Robert D. Mare. 1992. “Models for Sample Selection Bias.”
Annual Review of Sociology 18:327–350.
Wolter, Kirk M. 2007.
Introduction to Variance Estimation. New York: Springer Science+Business Media, LLC.
Yates, F. 1946. “A Review of Recent Statistical Developments in Sampling and Sampling Surveys.”
Journal of the Royal Statistical Society 109:12–43.
Yeager, David S., Jon A. Krosnick, LinChiat Chang, Harold S. Javitz, Matthew S. Levendusky, Alberto Simpser, and Rui Wang. 2011. “Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples.”
Public Opinion Quarterly 75:709–47.
Zea, Maria Cecilia. 2010. “Reaction to the Special Issue on Centralizing the Experiences of LGB People of Color in Counseling Psychology.”
Counseling Psychologist 38(3):425–33.
Zukin, Cliff, Jessica Godofsky, Carl Van Horn, Wendy Mansfield, and J. Michael Dennis. 2011. “Can a Non-Probability Sample Ever Be Useful for Representing a Population?: Comparing Probability and Non-Probability Samples of Recent College Graduates.” Research presented at the 2011 Annual Conference of the American Association for Public Opinion Research. http://www.knowledgenetworks.com/ganp/docs/aapor2011/aapor11-Can-a-Non-Probability-Sample.pdf
APPENDIX A: AAPOR NON-PROBABILITY TASK FORCE MISSION STATEMENT
November 22, 2011
Survey researchers routinely conduct studies that use different methods of data collection and inference. Some employ a probability sampling framework, while others use non-probability designs. There are a wide range of these non-probability designs including case-control studies, clinical trials, evaluation designs, intercept surveys, and volunteer panels. Researchers may also work with other datasets such as administrative records or surveillance systems in which the researcher does not have control of part or all of the system of collection. While not surveys per se, they nonetheless may provide insight into methods of working with data not collected using probability-based methods.
The mission of the task force is to examine the conditions under which various survey designs that do not use probability samples might still be useful for making inferences to a larger population. We assume that a large probability sample with complete coverage of the target population and complete response meets this requirement, but this rarely occurs in practice. Non-probability samples may be an acceptable alternative, depending on the purpose for which the data are being collected.
Non-probability sampling and inference designs are less well developed and explored. The assumptions required to make valid inferences from non-probability samples are not well known. Hence it is beneficial to examine non-probability methods and assumptions to assess the conditions under which they might be expected to provide estimates that are fit for the planned use.
The task force is charged with examining the strengths and weaknesses of different non-probability methods. Both theoretical and empirical support for these methods will be considered. Since online sampling, whether from access panels or intercepts, has become so prevalent, these designs will be a major but not exclusive focus of the task force.