Using Web Probing to Elucidate Respondents' Understanding of 'Minorities' in Cross-Cultural Comparative Research

• Authors: Braun Michael , Behr Dorothée , Raiber Klara , Repke Lydia
• Languages of publication: EN

Abstracts

EN

Due to the growing significance of international studies, the need for tools to assess the equivalence of items is pressing. Web probing, which is implementing verbal probing techniques traditionally used in cognitive interviewing in online surveys, is a method to complement quantitative techniques to establish equivalence of items in crosscultural research. We illustrate this approach by assessing the question of 'how important it is that government authorities respect and protect the rights of minorities', which was originally used in the International Social Survey Program, for respondents in five countries (Germany, Britain, the U.S., Mexico, and Spain). First, participants answered this question using a 7-point Likert scale. Then they wrote freely what types of minorities they had thought of. Whether country differences in the response patterns can be interpreted substantially depends partially on how similarly the term 'minorities' is understood across these five contexts. Our results show that people in the participating countries have slightly different kinds of 'minorities' in mind.

Keywords

EN

web probing; minorities; comparability; cross-cultural research

• Publisher: Institute of Philosophy and Sociology, Polish Academy of Sciences
• Journal: ASK. Research and Methods
• Year: 2019
• Volume: 28
• Issue: 1
• Pages: 3-20

Contributors

author

Braun Michael

author

Behr Dorothée

author

Raiber Klara

author

Repke Lydia

References

• Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J. M., Zahs, D. (2010). AAPOR report on online panels. Public Opinion Quarterly, 74, 711–781. https://doi.org/10.1093/poq/nfq048
• Beatty, P.C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71, 287–311. https://doi.org/10.1093/poq/nfm006
• Benítez, I., & Padilla, J. L. (2014). Analysis of nonequivalent assessments across different linguistic groups using a mixed methods approach: Understanding the causes of differential item functioning by cognitive interviewing. Journal of Mixed Methods Research, 8, 52–68. https://doi.org/10.1177/1558689813488245
• Behr, D., Meitinger, K., Braun, M., & Kaczmirek, L. (2017). Web probing – implementing probing techniques from cognitive interviewing in web surveys with the goal to assess the validity of survey questions. GESIS – Survey Guidelines. GESIS – Leibniz-Institute for the Social Sciences. https://doi.org/10.15465/gesis-sg_en_023
• Brannen, J. (2005). Mixing methods: The entry of qualitative and quantitative approaches into the research process. International Journal of Social Research Methodology, 8, 173–184. https://doi.org/10.1080/13645570500154642
• Coenders, M., Lubbers, M., & Scheepers, P. (2004). Majority populations’ attitudes towards migrants and minorities. Report for the European Monitoring Centre on Racism and Xenophobia. Wien: EUMC. Ref. no. 2003/04/01. http://fra.europa.eu/en/publication/2005/majorities-attitudes-towards-migrants-and-minorities-keyfindings-eurobarometer-a-0
• Davidov, E., Dülmer, H., Schlüter, E., Schmidt, P., & Meuleman, B. (2012). Using a multilevel structural equation modeling approach to explain cross-cultural measurement noninvariance. Journal of Cross-Cultural Psychology, 43, 558–75. https://doi.org/10.1177/0022022112438397
• Davidov, E., Meuleman, B., Cieciuch, J., Schmidt, P., & Billiet, J. (2014). Measurement equivalence in cross-national research. Annual Review of Sociology, 40, 55–75. https://doi.org/10.1146/annurev-soc-071913-043137
• European Monitoring Centre on Racism and Xenophobia (2005). Majorities’ attitudes towards minorities. Key findings from the Eurobarometer and the European Social Survey – Summary. Brussels, Belgium: European Monitoring Centre on Racism and Xenophobia. Summary. http://fra.europa.eu/en/publication/2005/majoritiesattitudes-towards-migrants-and-minorities-key-findings-eurobarometer-a-0
• European Social Survey (2012). ESS round 6 source questionnaire. London: Centre for Comparative Social Surveys, City University London.
• Fitzgerald, R., Widdop, S., Gray, M., & Collins, D. (2011). Identifying sources of error in cross-national questionnaires: Application of an error source typology to cognitive interview data. Journal of Official Statistics, 27, 569–599.
• ISSP Research Group (2016). International Social Survey Programme. Citizenship II – ISSP 2014. GESIS Data Archive, Cologne. ZA6670 Data file Version 2.0.0. https://doi.org/10.4232/1.12590
• Lee, J. (2014). Conducting cognitive interviews in cross-national settings. Assessment, 21, 227–240. https://doi.org/10.1177/1073191112436671
• Miller, K., Fitzgerald, R., Padilla, J.-L., Willson, S., Widdop, S., Caspar, R., Schoua-Glusberg, A. (2011). Design and analysis of cognitive interviews for comparative multinational testing. Field Methods, 23, 379–396. https://doi.org/10.1177/1525822X11414802
• Meitinger, K. (2017). Necessary but insufficient: Why measurement invariance tests need online probing as a complementary tool. Public Opinion Quarterly, 81, 447–472. https://doi.org/10.1093/poq/nfx009
• Meitinger, K., & Behr, D. (2016). Comparing cognitive interviewing and online probing: Do they find similar results? Field Methods, 28, 363–380. https://doi.org/10.1177/1525822X15625866
• Meitinger, K., Braun, M., & Behr, D. (2018). Sequence matters in online probing: The impact of the order of probes on response quality, motivation of respondents, and answer content. Survey Research Methods, 12, 103–120.
• Prüfer, P., & Rexroth, M. (2005). Kognitive Interviews. ZUMA How-to-Reihe, 15. Retrieved from http://www.gesis.org/fileadmin/upload/forschung/publikationen/ gesis_reihen/howto/How_to15PP_MR.pdf?download=true
• Schuman, H. (1966). The random probe: A technique for evaluating the validity of closed questions. American Sociological Review, 31, 218–222. https://doi.org/10.2307/2090907
• Seyranian, V., Atuel, H., & Crano, W. D. (2008). Dimensions of majority and minority groups. Group Processes & Intergroup Relations, 11, 21–37. https://doi.org/10.1177/1368430207084843
• Thrasher, J. F., Quah, A. C. K., Dominick, G., Borland, R., Driezen, P., Awang, R., Boado, M. (2011). Using cognitive interviewing and behavioral coding to determine measurement equivalence across linguistic and cultural groups: An example from the International Tobacco Control Policy evaluation project. Field Methods, 23, 439–460. https://doi.org/10.1177/1525822X11418176
• Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices and recommendations for organizational research. Organizational Research Methods, 3, 4–70. https://doi.org/10.1177/109442810031002
• Vijver, F. J. R. van de, & Chasiotis, A. (2010). Making methods meet: Mixed designs in cross-cultural research. In J. A. Harkness, M. Braun, B. Edwards, T. P. Johnson,
• L. Lyberg, P. P. Mohler, B.-E. Pennell, & T. Smith (Eds.), Survey methods in multinational, multiregional, and multicultural contexts (pp. 455–473). Hoboken, NJ: Wiley. https://doi.org/10.1002/9780470609927.ch24
• Widdop, S., Fitzgerald, R., & Gatrell, L. (2011). European Social Survey round 6 cognitive pre-testing report. London: Centre for Comparative Social Surveys.
• Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781412983655
• Willis, G. B. (2015). The practice of cross-cultural cognitive interviewing. Public Opinion Quarterly, 79, 359–395. https://doi.org/10.1093/poq/nfu092