How (Not) to Estimate the Design Effect of a Complex Sampling Scheme: A Case Study of the Polish Section of the European Social Survey, Round 5
Selected contents from this journal
Languages of publication
Design effect (DEFF) is a measure used to assess the effectiveness of a particular sampling scheme. Even though its deﬁnition is remarkably simple (cf. Kish 1965: 258), its practical implementation turns out to be problematic. Researchers therefore usually simplify the estimation of DEFF by independently determining the values of three components, namely, the clustering effect (DEFFc), the stratiﬁcation effect (DEFFs) and the effect of unequal sampling probabilities (DEFFp) and by multiplying these partial measures to obtain a measure of overall effect. However, the validity of such a simpliﬁed version depends on strict formal requirements which are met only in a few sampling schemes. The subject of the analysis presented here is the sampling scheme in the Polish section of round 5of the European Social Survey (ESS). It will be shown that the method of DEFF estimation applied by the Polish coordinators of the project, which is compatible with the methodological recommendations of ESS (cf. Lynn et al. 2007: 114),does not satisfy the formal criteria that would validate its use. The author proposes two other ways of estimating the size of DEFF (cf. Gabler et al. 2006: 116-117) appropriate for the sampling scheme in ESS5-PL. Empirical analyses indicate that the use of the simpliﬁed procedure of DEFF prediction leads to signiﬁcant underestimation of variance inﬂation in the sample design of ESS5-PL and, in turn, to overestimation of effective sample size.
- Barnett, Vic. 1982. Elements of Sampling Theory. Warszawa: Państwowe Wydawnictwo Naukowe.
- Biemer, Paul P. 2011 Latent Class Analysis of Survey Error. New Jersey: John Wiley & Sons, Inc.
- Dorofeev, Sergey and Peter Grant. 2006. Statistics for Real-Life Sample Surveys. Non-Simple-Random Samples and Weighted Data. Cambridge: Cambridge University Press.
- Gabler, Siegfried, Matthias Ganninger, Sabine Häder, and Ralf Munnich. 2008. “Design effect (deff).”In: Encyclopaedia of Survey Research Methods, edited by Paul J. Lavrakas. PLACE: SAGE Publications Inc., pp. 193-197.
- Gabler, Siegfried, Sabine Häder and Partha Lahiri. 1999. “A model based justiﬁ cation of Kish’s formula for design effects for weighting and clustering.” Survey Methodology Vol. 25(1): 105-106.
- Gabler, Siegfried, Sabine Häder and Peter Lynn. 2006. “Design effects for multiple design samples.”Survey Methodology Vol. 32(1): 115-120.
- Ganninger, Mathias. 2013. The ESS Sample Design Data File (SDDF). Documentation of the European Social Survey.
- Groves, Robert M.1989. Survey Errors and Survey Costs. New Jersey: John Wiley & Sons, Inc.
- Groves, Robert M., Floyd J. Fowler, Mick P. Couper, James M. Lepkowski, Elanor Singer and Roger Tourangeau. 2004. Survey Methodology. New Jersey: John Wiley & Sons, Inc.
- Kendall, Maurice G. and Alan Stuart. 1979. The Advanced Theory of Statistics. Vol. 2: Inference and Relationship. 4th ed. London: Grifﬁn.
- Kish, Leslie. 1965. Survey Sampling. New Jersey: John Wiley & Sons, Inc.
- Kish, Leslie. 1987. Statistical Design for Research. New Jersey: John Wiley & Sons, Inc.
- Kish, Leslie and Martin R. Frankel. 1974. “Inference from complex samples.”Journal of the Royal Statistical Society. Series B (Methodological) Vol. 36(1): 1-37.
- Kohler, Urlich. 2007. “Survey from inside: an assessment of unit nonresponse bias with internal criteria.”Survey Research Methods Vol. 2(1): 55-67.
- Kuha, Jouni and David Firth. 2011. “On the index of dissimilarity for lack of ﬁ t in loglinear and log-multiplicative models.”Computational Statistics and Data Analysis Vol. 55(1):375-388.
- Lee, Hyunshik. 2012. “How should one ﬁnd out the contributions to the design effect (variance) made by each of the design components (stratiﬁ cation, clustering, weighting) of a complex sample design?”Survey Statistician Vol. 66: 16-20.
- Lynn, Peter and Siegfried Gabler. 2005. “Approximation of b* in the prediction of design effects due to clustering.”Survey Methodology Vol. 31(1): 101-104.
- Lynn, Peter, Siegfried Gabler, Sabine Häder and Seppo Laaksonen. 2007. “Methods for achieving equivalence of samples in cross-national surveys.”Journal of Ofﬁcial Statistics Vol. 27(1): 107-124.
- Mulekar, Madhuri S., John C. Knutson and Jyoti A. Champanerkar. 2008. “How useful areapproximations to mean and variance of the index of dissimilarity?” Computational Statistics & Data Analysis Vol. 52(4): 2098-2109.
- Park, Inho and Hyunshik Lee. 2004. “Design effects for the weighted mean and total estimators under complex survey sampling.”Survey Methodology Vol. 30(2): 183-193.
- Paul, Sudhir R, Krishna K. Saha and Uditha Balasooriya. 2003. “An empirical investigation of different operating characteristics of several estimators of the intraclass correlation in the analysis of binary data.”Journal of Statistical Computation & Simulation Vol. 73(7): 507-523.
- Sampling design in ESS5-PL. 2010. Warszawa: Ośrodek Realizacji Badań Społecznych IFiS PAN.
- Sampling for the European Social Survey Round I. 2002. Bergen: European Social Data Archive, Norwegian Social Science Data Service.
- Sampling for the European Social Survey Round II. 2004. Bergen: European Social Data Archive, Norwegian Social Science Data Service.
- Sampling for the European Social Survey Round V: Principles and Requirements. 2010. Mannheim: The Sampling Expert Panel of the ESS. GESIS.
- Sawiński, Zbigniew. 2011. “Intra-cluster homogeneity in survey samples: a neglected tool.”Paper presented at the 4th Conference of the European Survey Research Association (ESRA), Lausanne, 18-22 July 2011.
- Ukoumunne, Obioha C. 2002. “A comparison of conﬁdence interval methods for the intraclass correlation coefﬁcient in cluster randomized trials.”Statistics in Medicine Vol. 21(24): 3757-3774.
- Vehovar, Vasja. 2007. “Non-response bias in the European Social Survey.”In: Measuring Meaningful Data in Social Research, edited by G. Loosveldt, M. Swyngedouw, and B. Cambr. Leuven: ACCO, pp. 335–356.
Publication order reference