Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

Raising the value of research studies in psychological science by increasing the credibility of research reports : the transparent Psi project

Kekecs, Zoltan LU orcid ; Palfi, Bence ; Szaszi, Barnabas ; Szecsi, Peter ; Zrubka, Mark ; Kovacs, Marton ; Bakos, Bence E. ; Cousineau, Denis ; Tressoldi, Patrizio and Schmidt, Kathleen , et al. (2023) In Royal Society Open Science 10(2).
Abstract

The low reproducibility rate in social sciences has produced hesitation among researchers in accepting published findings at their face value. Despite the advent of initiatives to increase transparency in research reporting, the field is still lacking tools to verify the credibility of research reports. In the present paper, we describe methodologies that let researchers craft highly credible research and allow their peers to verify this credibility. We demonstrate the application of these methods in a multi-laboratory replication of Bem's Experiment 1 (Bem 2011 J. Pers. Soc. Psychol. 100, 407-425. (doi:10.1037/a0021524)) on extrasensory perception (ESP), which was co-designed by a consensus panel including both proponents and opponents... (More)

The low reproducibility rate in social sciences has produced hesitation among researchers in accepting published findings at their face value. Despite the advent of initiatives to increase transparency in research reporting, the field is still lacking tools to verify the credibility of research reports. In the present paper, we describe methodologies that let researchers craft highly credible research and allow their peers to verify this credibility. We demonstrate the application of these methods in a multi-laboratory replication of Bem's Experiment 1 (Bem 2011 J. Pers. Soc. Psychol. 100, 407-425. (doi:10.1037/a0021524)) on extrasensory perception (ESP), which was co-designed by a consensus panel including both proponents and opponents of Bem's original hypothesis. In the study we applied direct data deposition in combination with born-open data and real-time research reports to extend transparency to protocol delivery and data collection. We also used piloting, checklists, laboratory logs and video-documented trial sessions to ascertain as-intended protocol delivery, and external research auditors to monitor research integrity. We found 49.89% successful guesses, while Bem reported 53.07% success rate, with the chance level being 50%. Thus, Bem's findings were not replicated in our study. In the paper, we discuss the implementation, feasibility and perceived usefulness of the credibility-enhancing methodologies used throughout the project.

(Less)
Please use this url to cite or link to this publication:
author
; ; ; ; ; ; ; ; and , et al. (More)
; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; and (Less)
organization
publishing date
type
Contribution to journal
publication status
published
subject
keywords
credibility, metascience, real-time procedures, replication, research methods, transparency
in
Royal Society Open Science
volume
10
issue
2
article number
191375
publisher
Royal Society Publishing
external identifiers
  • scopus:85147579818
  • pmid:36756055
ISSN
2054-5703
DOI
10.1098/rsos.191375
language
English
LU publication?
yes
id
7b4a453d-ec1a-4a97-9d50-c148ac134b3b
date added to LUP
2023-02-21 14:17:51
date last changed
2024-12-13 13:37:33
@article{7b4a453d-ec1a-4a97-9d50-c148ac134b3b,
  abstract     = {{<p>The low reproducibility rate in social sciences has produced hesitation among researchers in accepting published findings at their face value. Despite the advent of initiatives to increase transparency in research reporting, the field is still lacking tools to verify the credibility of research reports. In the present paper, we describe methodologies that let researchers craft highly credible research and allow their peers to verify this credibility. We demonstrate the application of these methods in a multi-laboratory replication of Bem's Experiment 1 (Bem 2011 J. Pers. Soc. Psychol. 100, 407-425. (doi:10.1037/a0021524)) on extrasensory perception (ESP), which was co-designed by a consensus panel including both proponents and opponents of Bem's original hypothesis. In the study we applied direct data deposition in combination with born-open data and real-time research reports to extend transparency to protocol delivery and data collection. We also used piloting, checklists, laboratory logs and video-documented trial sessions to ascertain as-intended protocol delivery, and external research auditors to monitor research integrity. We found 49.89% successful guesses, while Bem reported 53.07% success rate, with the chance level being 50%. Thus, Bem's findings were not replicated in our study. In the paper, we discuss the implementation, feasibility and perceived usefulness of the credibility-enhancing methodologies used throughout the project.</p>}},
  author       = {{Kekecs, Zoltan and Palfi, Bence and Szaszi, Barnabas and Szecsi, Peter and Zrubka, Mark and Kovacs, Marton and Bakos, Bence E. and Cousineau, Denis and Tressoldi, Patrizio and Schmidt, Kathleen and Grassi, Massimo and Evans, Thomas Rhys and Yamada, Yuki and Miller, Jeremy K. and Liu, Huanxu and Yonemitsu, Fumiya and Dubrov, Dmitrii and Röer, Jan Philipp and Becker, Marvin and Schnepper, Roxane and Ariga, Atsunori and Arriaga, Patrícia and Oliveira, Raquel and Põldver, Nele and Kreegipuu, Kairi and Hall, Braeden and Wiechert, Sera and Verschuere, Bruno and Girán, Kyra and Aczel, Balazs}},
  issn         = {{2054-5703}},
  keywords     = {{credibility; metascience; real-time procedures; replication; research methods; transparency}},
  language     = {{eng}},
  month        = {{02}},
  number       = {{2}},
  publisher    = {{Royal Society Publishing}},
  series       = {{Royal Society Open Science}},
  title        = {{Raising the value of research studies in psychological science by increasing the credibility of research reports : the transparent Psi project}},
  url          = {{http://dx.doi.org/10.1098/rsos.191375}},
  doi          = {{10.1098/rsos.191375}},
  volume       = {{10}},
  year         = {{2023}},
}