REGISTERED REPORTS AS A METHOD TO INCREASE CREDIBILITY OF SCIENCE – EXPERIMENTAL STUDY AMONG PSYCHOLOGICAL STUDENTS

• Authors: Piotr Sorokowski , Agata Groyecka , Kamil Błaszczyński , Tomasz Frąckowiak , Aleksander Kobylarek
• Languages of publication: EN

Abstracts

EN

Aims: The modern scientific publishing system suffers from many problems, amongst which one of the most important  is the pressure to publish positive results. A potentially simple way to mitigate publication bias are reviews of manuscript, prior to the data collection and analyses, as well as results presentation and discussion (i.e. Registered Reports). Methods: To test this prediction we conducted a quasi-experiment: two groups of students (n=38), as a part of their academic classes, were asked to design and conduct research projects. They were divided in two groups – Registered Reports- and control group. In both groups students have been encouraged to publish their papers in a local scientific journal. Results: Analyses revealed significant differences in p levels between groups, suggesting more reliable scores for Registered Reports group. Conclusions: Our study partially confirmed the  stated hypothesis and suggested, that registered reports might mitigate publication bias. Future recommendations are advised.

• Publisher: Fundacja Pro Scientia Publica
• Journal: Journal of Education Culture and Society
• Year: 2019
• Volume: 10
• Issue: 2
• Pages: 67-75

Dates

published

2019-09-02

Contributors

author

Piotr Sorokowski

• Institute of Psychology, University of Wrocław Dawida 1, Wrocław, Poland

• https://orcid.org/0000-0001-9225-9965

author

Agata Groyecka

• Institute of Psychology, University of Wrocław Dawida 1, Wrocław, Poland

• https://orcid.org/0000-0002-1932-4828

author

Kamil Błaszczyński

• Faculty of Psychology, University of Wrocław Ul. Dawida 1, Wrocław, Poland

• https://orcid.org/0000-0003-4896-8371

author

Tomasz Frąckowiak

• Faculty of Psychology, University of Wrocław Ul. Dawida 1, Wrocław, Poland

• https://orcid.org/0000-0002-0244-2879

author

Aleksander Kobylarek

• Institute of Pedagogy, University of Wrocław, , ul. Dawida 1/3, Poland

• https://orcid.org/0000-0002-4562-9035

References

• Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483(7391), 531–533. https://doi.org/10.1038/483531a
• Bem, S. L. (1974). The measurement of psychological androgyny. Journal of consulting and clinical psychology, 42(2), 155.
• Callaway, E. (2011). Report finds massive fraud at Dutch universities. Nature, 479(7371), 15.
• Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., ... & Altmejd, A. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637.
• Enserink, M. (2012). Final Report on Stapel Also Blames Field As a Whole. Science, 338(6112), 1270–1271. https://doi.org/10.1126/science.338.6112.1270
• Erlich, A. (2018). Pre-acceptance as a method to combat publication bias in area studies: a pilot in the Caucasus. Caucasus Survey, 6(3), 224-229.
• Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4(5), e5738.
• Fanelli, D. (2010). “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE, 5(4), e10068. https://doi.org/10.1371/journal.pone.0010068
• Findley, M. G., Jensen, N. M., Malesky, E. J., & Pepinsky, T. B. (2016). Can results-free review reduce publication bias? The results and implications of a pilot study. Comparative Political Studies, 49(13), 1667-1703.
• Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The Extent and Consequences of PHacking in Science. PLOS Biology, 13(3), e1002106. https://doi.org/10.1371/journal.pbio.1002106
• Jackson, C. J., Levine, S. Z., Furnham, A., & Burr, N. (2002). Predictors of Cheating Behavior at a University: A Lesson From the Psychology of Work. Journal of Applied Social Psychology, 32(5), 1031–1046. https://doi.org/10.1111/j.1559-1816.2002.tb00254.x
• John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953
• Kuczyńska, A. (1992). Inwentarz do oceny płci psychologicznej. Podręcznik. [The Psychological Sex Inventory] Warszawa: Pracownia Testów Psychologicznych PTP.
• Lakens, D. (2015). On the challenges of drawing conclusions from p-values just below 0.05. PeerJ, 3, e1142. https://doi.org/10.7717/peerj.1142
• Leimu, R., & Koricheva, J. (2004). Cumulative meta–analysis: a new tool for detection of temporal trends and publication bias in ecology. Proceedings of the Royal Society of London. Series B: Biological Sciences, 271(1551), 1961-1966.
• Levine, T. R., Asada, K. J., & Carpenter, C. (2009). Sample Sizes and Effect Sizes are Negatively Correlated in Meta-Analyses: Evidence and Implications of a Publication Bias Against NonSignificant Findings. Communication Monographs, 76(3), 286–302. https://doi.org/10.1080/03637750903074685
• Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7(6), 537-542.
• Morgan, G. A. (2000). Quasi-Experimental Designs. Journal of the American Academy of Child & Adolescent Psychiatry. 39. 794–796
• McCabe, D. L., Treviño, L. K., & Butterfield, K. D. (2001). Cheating in academic institutions: A decade of research. Ethics & Behavior, 11(3), 219–232.
• Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Du Sert, N. P., Simonsohn U., Wagenmakers E.J., Ware J.J. & Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 21. https://doi.org/10.1038/s41562-016-0021
• Nosek, B. A., & Lakens, D. (2014). Registered Reports: A Method to Increase the Credibility of Published Results. Social Psychology, 45(3), 137–141. https://doi.org/10.1027/1864-9335/a000192
• Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115, 2600-2606
• Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science, 7(6), 615–631. https://doi.org/10.1177/1745691612459058
• Oleszkiewicz, A., Karwowski, M., Pisanski, K., Sorokowski, P., Sobrado, B., & Sorokowska, A. (2017a). Who uses emoticons? Data from 86 702 Facebook users. Personality and Individual Differences, 119, 289-295.
• Oleszkiewicz, A., Frackowiak, T., Sorokowska, A., & Sorokowski, P. (2017b). Children can accurately recognize facial emotions from emoticons. Computers in Human Behavior, 76, 372-377.
• Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716–aac4716. https://doi.org/10.1126/science.aac4716
• Powell, K. (2016, November 4). Hard work, little reward: Nature readers reveal working hours and research challenges. Retrieved form http://www.nature.com/news/hard-work-little-reward-nature-readers-reveal-working-hours-and-research-challenges-1.20933
• Probst, T. M., & Hagger, M. S. (2015). Advancing the Rigour and Integrity of Our Science: The Registered Reports Initiative: Editorial. Stress and Health, 31(3), 177–179. https://doi.org/10.1002/smi.2645
• Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638
• Saunders, R., & Savulescu, J. (2008). Research ethics and lessons from Hwanggate: what can we learn from the Korean cloning fraud? Journal of Medical Ethics, 34(3), 214–221. https://doi.org/10.1136/jme.2007.023721
• da Silva, J. A. T. (2016). Does the removal of results from a submitted paper reduce publication bias?. Pacific Science Review B: Humanities and Social Sciences, 2(1), 29-30
• Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
• Simonsohn, U. (2013). Just Post It: The Lesson From Two Cases of Fabricated Data Detected by Statistics Alone. Psychological Science, 24(10), 1875–1888.https://doi.org/10.1177/0956797613480366
• Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014a). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534–547. https://doi.org/10.1037/a0033242
• Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014b). p-Curve and effect size correcting for publication bias using only significant results. Perspectives on Psychological Science, 9(6), 666–681.
• Sorokowski, P., Kulczycki, E., Sorokowska A., Pisanski, K. (2017). Predatory journals recruit fake editor. Nature, 543, 481-483.
• Stahl, D., & Pickles, A. (2018). Fact or fiction: reducing the proportion and impact of false positives. Psychological Medicine, 48(7), 1084-1091
• Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49(1), 108-112.
• Stroebe, W., & Strack, F. (2014). The Alleged Crisis and the Illusion of Exact Replication. Perspectives on Psychological Science, 9(1), 59–71. https://doi.org/10.1177/1745691613514450
• Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition. Perspectives on Psychological Science, 4(3), 274–290. https://doi.org/10.1111/j.1745-6924.2009.01125.x
• Wagenmakers, E., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100(3), 426–432. https://doi.org/10.1037/a0022790
• Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results. PLoS ONE, 6(11), e26828. https://doi.org/10.1371/journal.pone.0026828
• Wicherts, J., Veldkamp, C., Augusteijn, H., Bakker, M., van Aert, R., & van Assen, M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies A checklist to avoid p-hacking. Frontiers in Psychology, 7, 1832. https://doi.org/10.3389/fpsyg.2016.01832
• Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P., Dijkstra, K., & Zeelenberg, R. (2017). Participant nonnaiveté and the reproducibility of cognitive psychology. Psychonomic Bulletin & Review, 25, 1968-1972

Identifiers

DOI

10.15503/jecs20192.67.75