Judgements of research co-created by Generative AI: Experimental evidence

• Authors: Paweł Niszczota , Paul Conway

Abstracts

EN

The introduction of ChatGPT has fuelled a public debate on the appropriateness of using Generative AI (large lan guage models; LLMs) in work, including a debate on how they might be used (and abused) by researchers. In the current work, we test whether delegantig parts of the research process to LLMs leads people to distrust researchers and devalues their sciencfti work. Parctiipants ( N = 402) considered a researcher who delegates elements of the research process to a PhD student or LLM and rated three aspects of such delegation. Firstly, they rated whether it is morally appropriate to do so. Secondly, they judged whether-after deciding to delegate the research process-they would trust the scientist (that decided to delegate) to oversee future projects. Thirdly, they rated the expected accuracy and quality of the output from the delegated research process. Our results show that people judged delegating to an LLM as less morally acceptable than delegating to a human (d = -0.78). Delegation to an LLM also decreased trust to oversee future research projects (d = -0.80), and people thought the results would be less accurate and of lower quality (d = -0.85). We discuss how this devaluation might transfer into the underreporntig of Generavtie AI use.

• Publisher: Uniwersytet Ekonomiczny w Poznaniu
• Journal: Economics and Business Review
• Year: 2023
• Volume: 9
• Issue: 2
• Pages: 101-114

Dates

published

2023

Contributors

author

Paweł Niszczota

author

Paul Conway

References

• Alper, S., & Yilmaz, O. (2020). Does an abstract mind-set increase the internal consistency of moral atitudes and strengthen individualizing foundations? Social Psychological and Personality Science, 11(3), 326-335. htps://doi. org/10.1177/1948550619856309
• American Psychological Association. (2019). Publication manual of the American Psychological Association (7th ed.). APA.
• Bates, D., Mächler, M., Bolker, B. M., & Walker, S. C. (2015). Fiting linear mixedeefcts models using lme4. Journal of Statistical Software , 67(1). htps://doi. org/10.18637/jss.v067.i01
• Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspecvties on Psychological Science , 6(1), 3-5. htps://doi.org/10.1177/1745691610393980
• Cargill, M., & O'Connor, P. (2021). Writing scientifc research articles: Strategy and steps. John Wiley & Sons.
• Cha, Y. J., Baek, S., Ahn, G., Lee, H., Lee, B., Shin, J., & Jang, D. (2020). Compensating for the loss of human distinctiveness: The use of social creativity under HumanMachine comparisons. Computers in Human Behavior, 103, 80-90. htps://doi. org/10.1016/j.chb.2019.08.027
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.
• Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms aeftr seeing them err. Journal of Experimental Psychology: General, 144(1), 114-126. htps://doi.org/10.1037/xge0000033
• Dowling, M., & Lucey, B. (2023). ChatGPT for (finance) research: The Bananarama conjecture. Finance Research Letters, 103662. https://doi.org/10.1016/j. frl.2023.103662
• Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Baabdullah, A. M., Koohang, A., Raghavan, V., Ahuja, M., Albanna, H., Albashrawi, M. A., Al-Busaidi, A. S., Balakrishnan, J., Barleet, Y., Basu, S., Bose, I., Brooks, L., Buhalis, D., … Wright, R. (2023). “So what if ChatGPT wrote it?” Multidisciplinary perspectives on oppor - tunities, challenges and implications of generative conversational AI for research, pracctie and policy. Internaotinal Journal of Informaotin Management , 71, 102642. htps://doi.org/10.1016/j.ijinfomgt.2023.102642
• Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023). GPTs are GPTs: An early look at the labor market impact potenatil of large language models (arXiv:2303.10130). arXiv. htps://doi.org/10.48550/arXiv.2303.10130
• Funder, D. C., & Ozer, D. J. (2019). Evaluating eefct size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science , 2(2), 156-168. htps://doi.org/10.1177/2515245919847202
• King, M. (2023). Can GPT-4 formulate and test a novel hypothesis? Yes and no. TechRxiv. htps://doi.org/10.36227/techrxiv.22517278.v1
• Korinek, A. (2023). Language models and cognitive automation for economic re - search. Working Paper, 30957. National Bureau of Economic Research. htps:// doi.org/10.3386/w30957
• Korzynski, P., Mazurek, G., Altmann, A., Ejdys, J., Kazlauskaite, R., Paliszkiewicz, J., Wach, K., & Ziemba, E. (2023). Generative Artifcial Intelligence as a new context for management theories: Analysis of ChatGPT. Central European Management Journal, 31(1). htps://doi.org/10.1108/CEMJ-02-2023-0091
• Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., Leon, L. D., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V. (2022). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. medRxiv. htps://doi.org/10.1101/2022.12.19. 22283643
• Kuznetsova, A., Brockho,f P. B., & Christensen, R. H. (2017). lmerTest package: Tests in linear mixed eefcts models. Journal of Statistical Software , 82(13), 1-26.
• OpenAI. (2022, November 30). ChatGPT: Optimizing language models for dialogue . OpenAI. htps://openai.com/blog/chatgpt/
• OpenAI. (2023). GPT-4 technical report (arXiv:2303.08774). arXiv. htps://doi. org/10.48550/arXiv.2303.08774
• Palan, S., & Schietr, C. (2018). Prolicfi.ac -A subject pool for online experiments.Journal of Behavioral and Experimental Finance, 17, 22-27. htps://doi.org/10.1016/j. jbef.2017.12.004
• Peer, E., Brandimarte, L., Samat, S., & Acquis,ti A. (2017). Beyond the Turk: Alternavtie plaotfrms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153-163. htps://doi.org/10.1016/j.jesp.2017.01.006
• Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2022). Data quality of plaotfrms and panels for online behavioral research. Behavior Research Methods, 54(4), 1643-1662. htps://doi.org/10.3758/s13428-021-01694-3
• Satariano, A. (2023, March 31). ChatGPT is banned in Italy over privacy concerns. The New York Times. htps://www.nytimes.com/2023/03/31/technology/chatg - pt-italy-ban.html
• Stokel-Walker, C. (2023). ChatGPT listed as author on research papers: Many scienitsts disapprove. Nature, 613(7945), 620-621. htps://doi.org/10.1038/d41586- 023-00107-z
• Thorp, H. H. (2023). ChatGPT is fun, but not an author. Science, 379(6630), 313-313. htps://doi.org/10.1126/science.adg7879
• Tools such as ChatGPT threaten transparent science; here are our ground rules for their use. (2023). Nature, 613(7945), 612-612. htps://doi.org/10.1038/d41586- 023-00191-1
• Wach, K., Duong, C. D., Ejdys, J., Kazlauskaitė, R., Korzynski, P., Mazurek, G., Paliszkiewicz, J., & Ziemba, E. (2023). The dark side of Generative Artifcial Intelligence: A critical analysis of controversies and risks of ChatGPT. Entrepreneurial Business and Economics Review, 11(2), 7-24. hpts://doi.org/10.15678/EBER.2023.110201
• Wang, S. H. (2023). OpenAI-explain why some countries are excluded from ChatGPT. Nature, 615(7950), 34-34. htps://doi.org/10.1038/d41586-023-00553-9
• Wu, Y., Mou, Y., Li, Z., & Xu, K. (2020). Investigating American and Chinese subjects' explicit and implicit percepotins of AI-generated arsticti work. Computers in Human Behavior, 104, 106186. htps://doi.org/10.1016/j.chb.2019.106186

Identifiers

DOI

10.18559/ebr.2023.2.744

Biblioteka Nauki

2231661