Representation of Southeast European Countries in International Survey Projects: Assessing Data Quality

• Authors: Marta Kołczyńska
• Languages of publication: EN

Abstracts

EN

This paper deals with the problem of unequal representation of countries in international surveys, and the differences in data quality between survey projects, both obstacles to cross-national comparative research. The first part of the paper investigates international surveys on general population samples conducted in South-East Europe in the period between 1990 and 2010. Documentation of country participation in both general and region- or theme-specific survey projects shows that some countries are systematically excluded from surveys. Consequently, from comparative perspective, the generalizability of research results is not only limited but also potentially biased, omitting atypical cases. The second part of the paper focuses on the quality of surveys. It finds that the most problematic element of surveys is survey documentation, an essential component of the data. Without documentation the value of datasets, analyses using them and conclusions drawn on their basis are questionable. The proposed synthetic measure of data quality, the Survey Quality Index, could lead to setting standards for the documentation of the survey process, and thus facilitate cross-national research and allow for meaningful integration of existing survey data.

Keywords

EN

Survey methodology; cross-national research; data quality; survey documentation

• Publisher: Institute of Philosophy and Sociology, Polish Academy of Sciences
• Journal: ASK. Research and Methods
• Year: 2014
• Volume: 23
• Pages: 57-78

Contributors

author

Marta Kołczyńska

• The Ohio State University, Columbus, OH 43210, USA
• Institute of Philosophy and Sociology, Polish Academy of Sciences, ul. Nowy Świat 72, 00-330 Warszawa

References

• Bagić, Dragan. 2011. Financial Literacy in Bosnia and Herzegovina: analytical report. Prepared for World Bank. http://microdata.worldbank.org/index.php/catalog/1025/download/20735 (Accessed 10.08.2013).
• Bian, Yanjie & Lulu Li. 2012. The Chinese General Social Survey (2003–8): Sample Designs and Data Evaluation. Chinese Sociological Review 45(1): 70–97.
• Biemer, Paul P. 2010. Total Survey Error - Design, Implementation, and Evaluation. Public Opinion Quarterly 74(5): 817–848.
• Biemer, Paul. P. & Lars. E. Lyberg, 2003. Introduction to Survey Quality. Hoboken, NJ: Wiley.
• Bracewell, Wendy & Alex Drace-Francis. 1999. South-Eastern Europe: History, Concepts, Boundaries. Balkanologie 3(2): 47-66.
• Coenders, Marcel, Marcel Lubbers & Peer Scheepers. 2005b. Majorities’ attitudes towards minorities in (former) Candidate Countries of the European Union: Results from the Eurobarometer in Candidate Countries 2003. Report 3 for the European Monitoring Centre on Racism and Xenophobia. Ref. No. 2003/04/01.
• DeBell, Matthew. 2012. Computaion of Survey Weights: Bridging Theory and Practice. Presentation at Conference 1 of The Future of Survey Research: A Pair of Conferences At The National Science Foundation. October 3-4, 2012, Arlington (US). https://iriss.stanford.edu/sites/all/files/EDGE/DeBellOctober.pdf (Accessed 10.08.2013).
• Deming, W. Edwards. 1944. “On Errors in Surveys,” American Sociological Review 9(4): 359-369.
• Eppler, Martin J. 2006. Managing Information Quality Increasing the Value of Information in Knowledge-intensive Products and Processes. Heidelberg/New York: Springer.
• European Commission Public Opinion website. http://ec.europa.eu/public_opinion/index_en.htm (Accessed 10.08.2013).
• Eurostat. 2000.Assessment of the quality in statistics. General standard report, Luxembourg, April 2000 at http://www.unece.org/stats/documents/2000/11/metis/crp.3.e.pdf (Accessed 13.05.2013).
• Gallie, Duncan & Serge Paugam. 2002. Social precarity and social integration. Report for the European Commission, based on Eurobarometer 56. European Commission, Brussels.
• Granda, Peter & Meinhard Moschner. 2013. Analysis of Comparative Surveys – What Has Changed? Tenth or Eleventh International Workshop on Comparative Survey Design and Implementation (CSDI). Stockholm Universitet, 21-23 March. http://csdiworkshop.org/v2/images/2013/CSDI10-Granda.pdf (Accessed 10.08.2013).
• Groves, Robert M. & Lars Lyberg.2010. Total survey error: Past, present, and future. Public Opinion Quarterly 74(5): 849–879.
• Groves, Robert M. 2006. Non‑response rates and non‑response bias in household surveys. Public Opinion Quarterly 70(5): 646-675.
• Groves, Robert. M. & Emilia Peytcheva. 2008. The impact of non‑response rates on nonresponse bias. Public Opinion Quarterly 72(2): 167-189.
• Halman, Loek. 2001. The European Values Study: A Third Wave. Source book of the 1999/2000 European Values Study surveys. Tilburg University. http://info1.gesis.org/dbksearch18/download.asp?id=19002 (Accessed 10.04.2014).
• Harkness, Janet A. 1999. In pursuit of quality: issues for cross-national survey research. International Journal of Social Research Methodology 2(2): 125-140. http://www.ingentaconnect.com/content/routledg/tsrm/1999/00000002/00000002/art00004 (Accessed 10.08.2013).
• Haselden, Lucy & Amanda White. 2001. Developing new quality indicators in social surveys. Paper for the International Conference on Quality in Official Statistics. Stockholm, May 14 – 15. http://www.scb.se/Grupp/Produkter_Tjanster/Kurser/Tidigare_kurser/q2001/Session_33.pdf (Accessed 10.08.2013).
• Heath, Anthony, Stephen Fisher & Shawna Smith. 2005. The Globalization of Public Opinion Research. Annual Review of Political Science 8: 297-333.
• Hyman, Herbert H. 1954. Interviewing in social research. Chicago: University of Chicago Press.
• ILO. 2012. International Standard Classification of Occupations: ISCO-08. Structure, group definitions and correspondence tables. http://www.ilo.org/wcmsp5/groups/public/-dgreports/-dcomm/-publ/documents/publication/wcms_172572.pdf (Accessed 10.08.2013).
• International Organization for Standardization. 2005. ISO 9000 Quality management systems - Fundamentals and vocabulary. Geneva, Switzerland: International Organization for Standardization (ISO).
• Jagodzinski, Wolfgang & Christof Wolf. 2009. Access to and Documentation of Publicly Financed Survey Data. RatSWD Working Paper Series. WP 84 (May).
• Juran, Joseph M. & A. Blanton Godfrey (Eds.). 1988. Juran’s quality control handbook (5th ed.). New York: McGraw-Hill.
• Lampe, John R. 2006. Balkans into Southeastern Europe: A Century of War and Transition. New York: Palgrave. Macmillan.
• Lupri, Eugen. 1969, Theoretical and Methodological Problems in Cross-National Research. Sociologia Ruralis 9: 99-113.
• Matsuo, Hideko & Geert Loosveldt. 2013. Report on quality assessment of contact data files in Round 5: Final report 27 countries. Working paper Centre for Sociological Research (CeSO) Survey Methodology CeSO/SM/2013-3. http://ess.nsd.uib.no/ess/round5/ (Accessed 10.08.2013).
• Mohorko, Anja, Edith de Leeuw & Joop Hox. 2013. Coverage Bias in European Telephone Surveys: Developments of Landline and Mobile Phone Coverage cross Countries and over Time. Survey Methods: Insights from the Field. Retrieved from http://surveyinsights.org/?p=828
• Mohler, Peter Ph. 2013. Comparative Surveys: Fit for Analysis?. Presentation prepared for the 11th CSDI March 21-23, 2013, Stockholm. http://csdiworkshop.org/v2/images/2013/mohler%20cdsi%202013%20fit%20for%20use%203.pdf (Accessed 10.08.2013).
• Petrakos, Michalis, Maria Kleideri & Antonia Ieromnimon. 2010. Quality Assessment of the 2nd European Quality of Life Survey. Eurofound. http://www.eurofound.europa.eu/pubdocs/2011/60/en/1/EF1160EN.pdf (Accessed 10.08.2013).
• Schoene, Matthew & Marta Kolczynska. 2014. “Survey Data Harmonization and the Quality of Data Documentation in Cross-National Surveys.” CONSIRT Working Papers Series 3 (CONSIRT Labs: Methodology of Survey Data Harmonization) at consirt.osu.edu.
• Słomczyński, Kazimierz M. & Irina Tomescu-Dubrow. 2006. Representation of Post-Communist European Countries in Cross-National Public Opinion Surveys. Problems of Post-Communism 53 (4): 42-52.
• Smith, Shawna N., Stephen D. Fisher & Anthony Heath. 2011. Opportunities and challenges in the expansion of cross-national survey research, International Journal of Social Research Methodology 14: 485-502.
• TED. 2011. Tenders Electronic Daily. Supplement to the Official Journal of the European Union. http://ted.europa.eu/udl?uri=TED:NOTICE:251658-2011:TEXT:EN:HTML (Accessed 10.08.2013).
• TNS. 2011. TNS is awarded EU Contract to Conduct Flash Eurobarometers. tnsglobal.com. http://www.tnsglobal.com/press-release/tns-awarded-eu-contract-conductflash-eurobarometers (Accessed 10.08.2013).
• Todorova, Maria. 1997. Imagining the Balkans. New York: Oxford University Press.
• UNESCO. 2006. International Standard Classification of Education: ISCED 1997. http://www.uis.unesco.org/Library/Documents/isced97-en.pdf (Accessed 10.08.2013).
• UNESCO. 2012. International Standard Classification of Education: ISCED 2011. http://www.uis.unesco.org/Education/Documents/isced-2011-en.pdf (Accessed 10.08.2013).
• Warner, Uwe & Jürgen H.P. Hoffmeyer-Zlotnik. 2005. Measuring Income in Comparative Social Survey Research; in: Jürgen H.P. Hoffmeyer-Zlotnik & Janet A. Harkness (Eds.). Methodological Aspects in Cross-National Research. ZUMA-Nachrichten Spezial 11. Mannheim: ZUMA: 203-221.
• Wang, Richard. Y. & Diane M. Strong. 1996. Beyond Accuracy: What Data Quality Means to Data Consumers. Journal of Management Information Systems 12(4): 5-34.