A Meta-analysis of Within-Household Selection Impact on Survey Outcome Rates, Demographic Representation and Sample Quality in the European Social Survey

• Authors: Piotr Jabkowski
• Languages of publication: EN

Abstracts

EN

Face-to-face surveys of the general population often start with a probability sample of the households or addresses. However, even if a probability sample of households or addresses has been drawn, random selection of the target respondent within the selected household is crucial for obtaining a probability sample of individuals comprising the population. Over a dozen procedures of within-household selection have been described in survey literature. This article is concerned only with the two most popular of these, i.e., with the Kish grid procedure and the class of birthday procedures (i.e., next-birthday, last-birthday and closestbirthday methods). The main goal of this paper is to address the question of whether the Kish grid and birthday methods differ in their impact on: (1) refusal and cooperation rates, and (2) demographic representation of the survey sample, as well as (3) the degree of interviewers’ influence on the selection process. Based on 98 different surveys from all seven rounds of the European Social Survey, a meta-analysis was conducted to generate quantitative measures indicating the size of the overall impact of Kish grid and birthday procedures. Several conclusions can be formulated based on the analysis. Firstly, Kish grid samples (compared to birthday samples) result in significantly higher odds of receiving refusals and significantly lower odds of obtaining cooperation. Secondly, both Kish grid and birthday samples have a similar and significant impact on gender and age imbalance. Finally, birthday procedures give interviewers greater opportunity to influence the selection process. The latter means that the use of Kish grid samples is usually associated with a higher quality of the within-household selection, even though in such samples refusal rates are significantly higher and cooperation rates are significantly lower compared to those obtained in birthday samples.

Keywords

EN

probability samples; within-household selection; Kish grid; birthday procedures; internal and external criteria of representativeness; European Social Survey

• Publisher: Institute of Philosophy and Sociology, Polish Academy of Sciences
• Journal: ASK. Research and Methods
• Year: 2017
• Volume: 26
• Issue: 1
• Pages: 31-60

Contributors

author

Piotr Jabkowski

• Adam Mickiewicz University in Poznan

References

• Bakker, J., I., (Hans). (2007). Meta-analysis. In G. Ritzer (Ed.), The Blackwell Encyclopedia of Sociology (pp. 2963–2964). Malden: Blackwell Publishing Ltd.
• Bethlehem, J., Cobben, F., & Schouten, B. (2011). Handbook of Nonresponse in Household Surveys. Hoboken: John Wiley & Sons, Ltd. https://doi.org/10.1002/9780470891056
• Biemer, P. P. (2010). Total Survey Error. Design, Implementation, and Evaluation. Public Opinion Quarterly, 74(5), 817–848. https://doi.org/10.1093/poq/nfq058
• Binson, D., Canchola, J. A., & Catania, J. A. (2000). Random Selection in a National Telephone Survey: A Comparison of the Kish, Next-Birthday, and Last-Birthday Methods. Journal of Official Statistics, 16(1), 53–59.
• Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to Meta-Analysis. Hoboken: John Willey & Sons, Ltd. https://doi.org/10.1002/9780470743386
• Brogan, D. J., Denniston, M. M., Liff, J. M., Flagg, E. W., Coates, R. J., & Brinton, L. A. (2001). Comparison of telephone sampling and area sampling: Response rates and within-household coverage. American Journal of Epidemiology, 153(11), 1119–1127. https://doi.org/10.1093/aje/153.11.1119
• Brooks, C. A., & Bailar, B. (1978). Statistical Policy Working Paper 3 – An Error Profile: Employment as Measured by the Current Population Survey. Washington: Office of Federal Statistical Policy and Standards, U.S. Department of Commerce.
• Bryant, B. E. (1975). Respondent Selection in a Time of Changing Household Composition. Journal of Marketing Research,12(2), 129–135. https://doi.org/10.2307/3150434
• Card, N. A. (2015). Applied meta-analysis for social science research. New York: Guilford Publications.
• Church, A. H. (1993). Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly, 57(1), 62–79. https://doi.org/10.1086/269355
• Czaja, R., Blair, J., & and Sebestik, J. P. (1982). Respondent Selection in a Telephone Survey: A Comparison of Three Techniques. Journal of Marketing Research, 19(3) 381–385. https://doi.org/10.2307/3151573
• Denk, C. E., & Hall, J. W. (2000). Respondent Selection in RDD Surveys: A Randomized Trial of Selection Performance. Paper presented at the annual meeting of the American Association for Public Opinion Research, Portland, OR.
• Denk, C. E., Guterbock, T. M., & Gold, D. B. (1996). Modelling Selection of Respondents Within Household in Telephone Surveys. Paper presented at the annual meeting of the American Association for Public Opinion Research, Salt Lake City, UT.
• ESS Round 1: European Social Survey (2016): ESS-1 2002 Documentation Report. Edition 6.5. Bergen, European Social Survey Data Archive, NSD – Norwegian Centre for Research Data for ESS ERIC.
• ESS Round 4: European Social Survey (2016): ESS-4 2008 Documentation Report. Edition 5.4. Bergen, European Social Survey Data Archive, NSD – Norwegian Centre for Research Data for ESS ERIC.
• ESS Round 5: European Social Survey (2016): ESS-5 2010 Documentation Report. Edition 4.1. Bergen, European Social Survey Data Archive, NSD – Norwegian Centre for Research Data for ESS ERIC.
• ESS Round 6: European Social Survey (2016): ESS-6 2012 Documentation Report. Edition 2.3. Bergen, European Social Survey Data Archive, NSD – Norwegian Centre for Research Data for ESS ERIC.
• ESS Round 7: European Social Survey (2016): ESS7- 2014 Documentation Report Edition 3.1. Bergen, European Social Survey Data Archive, NSD – Norwegian Centre for Research Data for ESS ERIC.
• Eurostat (2014). Quality report of the European Union Labour Force Survey 2013. Eurostat Statistical Working Papers.
• Gaziano, C. (2005). Comparative Analysis of Within-Household Respondent Selection Techniques. Public Opinion Quarterly, 69(1), 124–157. https://doi.org/10.1093/poq/nfi 006
• Groves, R. M. (1989). Survey Errors and Survey Costs, Hoboken: John Wiley & Sons, Inc. https://doi.org/10.1002/0471725277
• Groves, R. M. (2006). Nonresponse Rates and Nonresponse Bias in Household Surveys. Public Opinion Quarterly, 70(5), 646–675. https://doi.org/10.1093/poq/nfl 033
• Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167–189. https://doi.org/10.1093/poq/nfn011
• Groves, R. M., & Couper, M. P. (1998). Nonresponse in household interview surveys. Hoboken: John Wiley & Sons, Ltd. https://doi.org/10.1002/9781118490082
• Groves, R. M., & Kahn R., L. (1979). Surveys by Telephone: A National Comparison with Personal Interviews. New York: Academic Press.
• Hagan, D. E., & Collier C., M. (1983). Must Respondent Selection Procedures for Telephone Surveys Be Invasive? Public Opinion Quarterly, 47(4), 547–556. https://doi.org/10.1086/268811
• de Heer, W. (1999). International Response Trends: Results of an International Survey. Journal of Official Statistics, 15(2), 129–142.
• Hopkins, K. D., & Gullickson, A. R. (1992). Response rates in survey research: A metaanalysis of the effects of monetary gratuities. Journal of Experimental Education, 61(1), 52–62. https://doi.org/10.1080/00220973.1992.9943849
• Hox, J. J., & de Leeuw, E. (1994). A Comparison of Nonresponse in Mail, Telephone, and Face-to-Face Surveys. Applying Multilevel Modeling to Meta-Analysis. Quality and Quantity, 28(4), 329–344. https://doi.org/10.1007/BF01097014
• Keeter, S., & Fisher, K. (1997). A Comparison of ‘Last Birthday’ and ‘Youngest Male/Oldest Female’ Respondent Selection Procedures. Paper presented at the annual meeting of the American Association for Public Opinion Research, St. Petersburg, FL.
• Kish, L. (1949). A Procedure for Objective Respondent Selection within the Household. Journal of the American Statistical Association, 44(247), 380–387. https://doi.org/10.1080/01621459.1949.10483314
• Koch, A. (2016). Assessment of socio-demographic sample composition in ESS Round 6. Mannheim: European Social Survey, GESIS.
• Kohler, U. (2007). Survey from Inside: An Assessment of Unit Nonresponse Bias With Internal Criteria. Survey Research Methods, 2(1), 55–67.
• Korns, A. (1977). Coverage Issues Raised by Comparisons Between CPS and Establishment Employment. In American Statistical Association, Proceedings of the Social Statistics Section, Part I (pp. 60–69).
• Lavrakas, P. L., Stasny, E. A., & Harpuder, B. (2000). A Further Investigation of the Last-Birthday Respondent Selection Method and Within-Unit Coverage Error. In American Statistical Association, Proceedings of the Survey Research Methods Section (pp. 890–895).
• Lavrakas, P. L., Bauman, S. L., & Merkle, D. M. (1993). The Last-Birthday Selection Method & Within-Unit Coverage Problems. In American Statistical Association, Proceedings of the Survey Research Methods Section (pp. 1107–1112).
• Le, K. T., Brick, M., Diop, A., & Alemadi, D. (2013). Within-household Sampling Conditioning on Household Size. International Journal of Public Opinion Research, 25(1), 108–118. https://doi.org/10.1093/ijpor/eds008
• de Leeuw, E. D., Callegaro, M., Hox, J., Korendijk, E., & Lensvelt-Mulders, G. (2007). The influence of advance letters on response in telephone surveys: A meta-analysis. Public Opinion Quarterly, 71(3), 413–443. https://doi.org/10.1093/poq/nfm014
• Lynn, P., Gabler, S., Häder, S., & Laaksonen, S. (2007). Methods for Achieving Equivalence of Samples in Cross-National Surveys: The European Social Survey Experience. Journal of Official Statistics, 27(1), 107–124.
• Manfreda, K. L., Bosnjak, M., Berzelak, J., Hass, I., & Vehovar, V. (2008). Web surveys versus other survey modes: A meta-analysis comparing response rates. International Journal of Market Research, 50(1), 79–104.
• Martin, E. (1999). Who Knows Who Lives Here? Within-Household Disagreements as a Source of Survey Coverage Error. Public Opinion Quarterly, 63(2), 220–236. https://doi.org/10.1086/297712
• Medway, R. L., & Fulton, J. (2012). When More Gets You Less: A Meta-analysis of the Effect of Concurrent Web Options on Mail Survey Response Rates. Public Opinion Quarterly, 76(4), 733–746. https://doi.org/10.1093/poq/nfs047
• Menold, N. (2014). The Influence of Sampling Method and Interviewers on Sample Realization in the European Social Survey. Survey Methodology, 40(1), 105–123.
• O’Rourke, D., & Lakner, E. (1989). Gender bias: Analysis of factors causing male underrepresentation in surveys. International Journal of Public Opinion Research, 1(2), 164–176. https://doi.org/10.1093/ijpor/1.2.164
• O’Rourke, D., and Blair, J. (1983). Improving Random Respondent Selection in Telephone Surveys. Journal of Marketing Research, 20(4), 428–432. https://doi.org/10.2307/3151446
• Oldendick, R. W., Bishop, G. F., Sorenson, S. B. & Tuchfarber, A. F. (1988). A Comparison of the Kish and Last Birthday Methods of Respondent Selection in Telephone Surveys. Journal of Official Statistics, 4(4), 307–318.
• Paisley, W. J., & Parker, E. B. (1965). A Computer-Generated Sampling Table for Selecting Respondents Within Households. Public Opinion Quarterly, 29(3), 431–436. https://doi.org/10.1086/267343
• Salmon, C. T., & Nichols, J. S. (1983). The Next-Birthday Method of Respondent Selection. Public Opinion Quarterly, 47(2), 270–276. https://doi.org/10.1086/268785
• Shih, T., & Fan, X. (2007). Response Rates and Mode Preferences in Web-mail Mixed Mode Surveys: A Meta-analysis. International Journal of Internet Science, 2(1), 59–82.
• Smith, W. (2007). Survey Non-Response Procedures in Cross-National Perspective: The 2005 ISSP Non-Response Survey. Survey Research Methods, 1(1), 45–54.
• Smith, W., Chey, T., Jalaludin, B., Salkeld, G., & Capon, T. (1995). Increasing Response Rates in Telephone Surveys: A Randomized Trial. Journal of Public Health, 17(1), 33–38.
• Sodeur, W. (1997). Interne Kriterien zur Beurteilung von Wahrscheinlichkeitsauswahlen. ZA-Information / Zentralarchiv für Empirische Sozialforschung, 41, 58–82.
• Stoop, I., Billiet, J., Koch, A., & Fitzgerald R. (2010). Improving Survey Response. Lessons learned from the European Social Survey. Hoboken: John Wiley & Sons, Ltd. https://doi.org/10.1002/9780470688335
• Stoop, I. (2004). Surveying Nonrespondents. Field Methods, 16(1), 23–54. https://doi.org/10.1177/1525822X03259479
• Stoop, I. (2005). The Hunt for the Last Respondent. Nonresponse in Sample Surveys. The Hague: Social and Cultural Planning Office.
• Scherpenzeel, A., & Saris, W. E. (1997). The validity and reliability of survey questions: A meta-analysis of MTMM studies. Sociological Methods and Research, 25 (3), 341–383. https://doi.org/10.1177/0049124197025003004
• The American Association for Public Opinion Research (2016). Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 9th edition. AAPOR.
• Tourangeau, R., Shapiro, G., Kearney, A., & Ernst, L. (1997). Who lives here? Survey undercoverage and household roster questions. Journal of Official Statistics, 13(1), 1–18.
• Troldahl, V. C., & Carter Jr, R. E. (1964). Random Selection of Respondents Within Household in Phone Surveys. Journal of Marketing Research, 1(2), 71–76. https://doi.org/10.2307/3149926
• Valentine, C. A., & Valentine, B. L. (1971). Missing Men: A Comparative Methodological Study of Underenumeration and Related Problems. Washington: The U.S. Census Bureau.
• Yan, T. (2009). A Meta-analysis of Within-Household Respondent Selection Methods. Paper presented at the annual meeting of the American Association for Public Opinion Research, Hollywood, FL.
• Ziniel, S. (2008). Within-Unit Coverage Error. In P. Lavrakas (Ed.), Encyclopaedia of Survey Research Methods (pp. 962–963). Thousand Oaks: SAGE Publications, Inc.