Mixed Methods in Value Research: An Analysis of the Validity of the Russian Version of the Schwartz Value Survey (SVS) Using Cognitive Interviews, Multidimensional Scaling (MDS), and Confirmatory Factor Analysis (CFA)

• Authors: Maria Efremova , Tatiana Panyusheva , Peter Schmidt , Florian Zercher
• Languages of publication: EN

Abstracts

EN

The aim of this study is the validation of the Russian version of the Schwartz Value Survey (SVS) using qualitative (cognitive interviewing) and quantitative (multidimensional scaling, confirmatory factor analysis) methods. Specifically, this paper explores the issue of whether respondents interpret the values measured in the survey in the way intended by Shalom Schwartz. The analysis revealed problems with participants’ understanding of the wording, which are due both to subtle differences in translation and to the cultural context. In addition, multidimensional scaling and confirmatory factor analysis were used to investigate whether the problems found in the qualitative analysis could also be detected by these quantitative methods. The findings of the qualitative analysis allowed a better understanding of the findings of the quantitative analysis and vice versa. However, the results differed considerably. In conclusion, several recommendations are given for use of such a mixed method approach.

Keywords

EN

values; SVS; cognitive pretests; mixed methods; formal and construct validity; confirmatory factor analysis; multidimensional scaling; cross-loadings and error correlations

• Publisher: Institute of Philosophy and Sociology, Polish Academy of Sciences
• Journal: ASK. Research and Methods
• Year: 2017
• Volume: 26
• Issue: 1
• Pages: 3-30

Contributors

author

Maria Efremova

• National Research University Higher School of Economics Moscow

author

Tatiana Panyusheva

• National Research University Higher School of Economics Moscow

author

Peter Schmidt

• University of Giessen and Humboldt Research Fellow of the Polish Foundation for basic Research

author

Florian Zercher

• University of Giessen

References

• Alemán, J., Woods, D. (2016) Value Orientations From the World Values Survey. How Comparable Are They Cross-Nationally? Comparative Political Studies. Vol 49, Issue 8, 2016.
• Adcock, R. & Collier, D. (2001). Measurement validity: A shared standard for qualitative and quantitative research. American Political Science Review, 95(3), 529–546. https://doi.org/10.1017/S0003055401003100
• Beatty, P. C. & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71(2), 287–311. https://doi.org/10.1093/poq/nfm006
• Behr,D., Kaczmirek L., Bandilla W. & Braun, M. (2012) Asking probing questions in web surveys: Which factors have an impact on the quality of the responses, Social Science Computer Review, 30(4), 487–498. https://doi.org/10.1177/0894439311435305
• Beierlein, C., Davidov, E., Schmidt, P., Schwartz, S. H. & Rammstedt, B. (2012). Testing the discriminant validity of Schwartz’ Portrait Value Questionnaire items – A replication and extension of Knoppen and Saris (2009). Survey Research Methods, 6(1), 25–36. https://doi.org/10.18148/srm/2012.v6i1.5092
• Bilsky, W., Janik, M. & Schwartz, S. H. (2011). The structural organization of human values: Evidence from three rounds of the European Social Survey. Journal of Cross-Cultural Psychology, 42(5), 759–776. https://doi.org/10.1177/0022022110362757
• Blair, J., Conrad, F., Ackermann, A. C. & Claxton G. (2006). The effect of sample size on cognitive interview findings. Paper presented at the American Association for Public Opinion Research Conference, Montreal, Canada, May 18–21. Retrieved from http://www.abtassociates.com/presentations/aapor06_sample_size_cognitive_interviews.pdf
• Boeije, H. & Willis G. B. (2013). The Cognitive Interviewing Reporting Framework (CIRF). Towards the harmonization of cognitive testing reports. Methodology, 9(3), 87–95. https://doi.org/10.1027/1614-2241/a000075
• Borg, I. & Groenen P. (1997). Modern multidimensional scaling: Theory and applications. New York: Springer. https://doi.org/10.1007/978-1-4757-2711-1
• Brannen, J. (2009). Prologue: Mixed methods for novice researchers: Reflections and themes. International Journal of Multiple Research Approaches, 3(1), 8–12. https://doi.org/10.5172/mra.455.3.1.8
• Braun, M., Behr, D. & Kaczmarek, L. (2012). Assessing cross-national equivalence of measures of Xenophobia. Evidence from probing in web surveys. International Journal of Public Opinion Research, 25(3), 383–395. https://doi.org/10.1093/ijpor/eds034
• Brown, T. A. (2015). Confirmatory factor analysis for applied research. Second edition. London: Guilford Press.
• Bryman, A. (2007). Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research, 1(1), 8–22. https://doi.org/10.1177/1558689806290531
• Castro, F. G., Kellison, J. G., Boyd, S. J. & Kopak, A. (2010). A methodology for conducting integrative mixed methods research and data analyses. Journal of Mixed Methods Research, 4(4), 342–360. https://doi.org/10.1177/1558689810382916
• Cieciuch, J. & Schwartz, S. H. (2012). The number of distinct basic values and their structure assessed by PVQ-40. Journal of Personality Assessment, 94(3), 321–328. https://doi.org/10.1080/00223891.2012.655817
• Datler, G., Jagodzinski, W. & Schmidt, P. (2013). Two theories on the test bench: Internal and external validity of the theories of Ronald Inglehart and Shalom Schwartz. Social Science Research, 42, 906–925. https://doi.org/10.1016/j.ssresearch.2012.12.009
• Fontaine, R. J., Poortinga, Y. H., Delbeke, L. & Schwartz, S. H. (2008). Structural equivalence of the values domain across cultures: Distinguishing sampling fluctuations from meaningful variation. Journal of Cross-Cultural Psychology, 39(4), 345–365. https://doi.org/10.1177/0022022108318112
• Inglehart, R. (1997). Modernization and postmodernization: Cultural, economic, and political change in 43 societies. Princeton: Princeton University Press.
• Hak, T., van der Veer, K., & Jansen, H. (2008). The Three-Step Test-Interview (TSTI): An observation-based method for pretesting self-completion questionnaires. Survey Research Methods, 2(3), 143–150.
• Harkness, J. & van de Vijver F.J.R. & Mohler P.(2003). Cross-Cultural Survey Methods. New York: J. Wiley.
• Kluckhohn, C. & Strodtbeck, F. (1961). Variations in value orientations. Evanston, IL: Row, Peterson.
• Knoppen, D. & Saris, W. E. (2009). Do we have to combine values in the Schwartz’ human values scale? A comment on the Davidov studies. Survey Research Methods, 3(2), 91–103.
• Latcheva, R. (2011). Cognitive interviewing and factor-analytic techniques: A mixed method approach to validity of survey items measuring national identity. Quality & Quantity, 45(6), 1175–1199. https://doi.org/10.1007/s11135-009-9285-0
• Lebedeva, N. M. (2001). Value-motivational personality structure in Russian culture. Psychological Journal, 22(3), 26–36.
• Lebedeva, N. M., Tatarko, A. N. (2009) Kultura kak faktor obschestvennogo progressa. Moscow: Yustitsinform.
• Lebedeva, N. M., Osipova, E. & Cherkasova, L. (2013). Values and social capital as predictors of attitudes towards innovation. Research Working Paper WP 5/09, BRP Series: Sociology, М: NRU HSE.
• Luyt, A. (2012). A framework for mixing methods in quantitative measurement development, validation and revision: A case study. Journal of Mixed Methods Research, 6(4), 294–316. https://doi.org/10.1177/1558689811427912
• Meitinger,K.(2017) Necessary but Insufficient: Why Measurement Invariance Tests Need Online Probing as a complimentary Tool, Public Opinion Quarterly, 81(2), 447–472. https://doi.org/10.1093/poq/nfx009
• Ozhegov, S. I. (2010). Dictionary of Russian language. Moscow: Onyx.
• Petros, S. G. (2012). Use of a mixed methods approach to investigate the support needs of older caregivers to family members affected by HIV and AIDS in South Africa. Journal of Mixed Methods Research, 6(4), 275–293. https://doi.org/10.1177/1558689811425915
• Pruefer, P. (2011). Cognitive interviewing. A short overview. GESIS – Leibniz-Institute for the Social Sciences, Mannheim, Germany.
• Robinson, S. & Mendelson, A. L. (2012). A qualitative experiment: Research on mediated meaning construction using a hybrid approach. Journal of Mixed Methods Research, 6(4), 332–347. https://doi.org/10.1177/1558689812444789
• Rogozin, D. M. (2002). Cognitive analysis of the Questionnaire Tool. Moscow: Institute of Public Opinion.
• Rokeach, M. (1973). The nature of human values. New York: Free Press.
• Różycka-Tran, J., Khanh Ha, T.T., Cieciuch, J., Schwartz, S. H. (2017). Universals and Specifi cs of the Structure and Hierarchy of Basic Human Values in Vietnam.
• Behavioral Research Vol. 42 p. 1–32. Health Psychology Report. 5(3) 193–204. https://doi.org/10.5114/hpr.2017.65857
• Saris, W. E., Satorra, A. & Sörbom, D. (1987). The detection and correction of specification errors in structural equation models. Sociological Methodology, 17, 105–129. https://doi.org/10.2307/271030
• Saris, W. E., Satorra, A. & van der Veld, W. (2009). Testing structural equation models or detection of misspecifications? Structural Equation Modeling, 16(4), 561–582. https://doi.org/10.1080/10705510903203433
• Saris, W. E. (2011) Preface. Published in W. E. Saris, Oberski, D., Revilla, M., Zavala-Rojas, D., Lilleoja, L., Gallhofer, I. and Gruner, T. The Development of the Program SQP 2.0 for the Prediction of the Quality of Survey Questions. RECSM.
• Schmidt, P., Bamberg, S., Davidov, E., Hermann, J. & Schwartz, S. H. (2007). Die Messung von Werten mit dem “Portraits Value Questionnaire” [The measurement of values with the “Portraits Value Questionnaire”]. Zeitschrift für Sozialpsychologie, 38(4), 261–275. https://doi.org/10.1024/0044-3514.38.4.261
• Schüßler, A. & Schmidt, P. (2008, September). Does cognitive interviewing matter? Testing the effects of cognitive methods on the data quality. Paper presented at the 7th International Conference on Social Science Methodology, Campus di Monte Sant’ Angelo, Naples, Italy.
• Schwartz, S. H. (1992). Universals in the content and structure of values: Theory and empirical tests in 20 countries. In M. Zanna (Ed.), Advances in experimental social psychology (Vol. 25, pp. 1–65). New York: Academic Press.
• Schwartz, S. H. (1994). Are there universal aspects in the content and structure of values? Journal of Social Issues, 50, 19–45. https://doi.org/10.1111/j.1540-4560.1994.tb01196.x
• Schwartz, S. H. (2005a). Basic human values: Their content and structure across countries. In A. Tamayo & J. B. Porto (Eds.), Valores e comportamento nas organizações [Values and behavior in organizations] (pp. 21–55). Petrópolis, Brazil: Vozes.
• Schwartz, S. H. (2005b). Robustness and fruitfulness of a theory of universals in individual human values. In A. Tamayo & J. B. Porto (Eds.), Valores e comportamentonas organizações [Values and behavior in organizations] (pp. 56–95). Petrópolis, Brazil: Vozes.
• Schwartz, S. H. (2006). Les Valeurs de Base de la Personne: Théorie, Mesures et Applications [Basic human values: Theory, measurement, and applications]. Revue Française de Sociologie, 42, 249–288. https://doi.org/10.3917/rfs.474.0929
• Schwartz, S. H. (2014). National culture as value orientations: Consequences of value differences and cultural distance. In V. Ginsburgh & D. Throsby (Eds.), Handbook of the economics of art and culture (Vol. 2, pp. 547–586). North Holland: Elsevier. https://doi.org/10.1016/B978-0-444-53776-8.00020-9
• Schwartz, S. H. & Bilsky, W. (1987). Toward a universal psychological structure of human values. Journal of Personality and Social Psychology, 53, 550–562. https://doi.org/10.1037/0022-3514.53.3.550
• Schwartz, S. H. & Boehnke, K. (2004). Evaluating the structure of human values with confirmatory factor analysis. Journal of Research in Personality, 38, 230–255. https://doi.org/10.1016/S0092-6566(03)00069-2
• Schwartz, S. H., Cieciuch, J., Vecchione, M., Davidov, E., Fischer, R., Beierlein, C., … Konty, M. (2012). Refining the theory of basic individual values. Journal of Personality and Social Psychology, 103(4), 663–668. https://doi.org/10.1037/a0029393
• Smith, T. W. (2011). Refining the Total Survey Error Perspective. International Journal of Public Opinion Research, 23(4): 464–484. https://doi.org/10.1093/ijpor/edq052
• Tourangeau, R. (1984). Cognitive sciences and survey methods. In T. Jabine, M. Straf, J. Tanur & R. Tourangeau (Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73–100). Washington, DC: National Academy Press.
• Willis, G. B. (2005). Cognitive interviewing. A tool for improving questionnaire design. Thousand Oaks, CA: Sage Publications. https://doi.org/10.4135/9781412983655
• Wolf, C., Joye, Ch. D., Smith, T.W., Fu, Y. (2016). The SAGE Handbook of Survey Methodology. pp. 503–524. Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781473957893