Let’s Talk about Science Reform! Introducing a Workshop on Theoretical and Methodological Approaches to Investigating the Open Science Movement
By Sheena F. Bartscherer and Sven Ulpts
Background
Sparked by highly publicised cases of scientific misconduct as well as the identification of issues surrounding questionable research practices and irreproducibility of research findings, a scandalization process ensued (Penders, 2024). This scandalization initially started around narratives of a crisis concerning the research (processes) in some psychological and biomedical (sub)disciplines. However, without much attention to local (epistemic and practical) conditions of the varying research communities implicated (see new localism, Guttinger 2020), their crisis narrative quickly spread to other disciplines, and ultimately engulfed pretty much the entire research system, with narratives of a fundamentally ‘broken science’. Using these scandalous narratives as justifications, a scientific reform movement with aspirations of universal applicability burned across scientific disciplines and still continues to gain more ground every day (Hesselmann & Reinhart, 2024; Peterson & Panofsky, 2023). The proclaimed goal of science reformers has been and continues to be the improvement of science overall by making findings more verifiable and reproducible, primarily through opening up research processes and making them more transparent (to the public and scientists alike).
While these Open Science reforms and the associated communities have garnered quite a bit of positive attention from researchers, the media, funders, and policymakers, they have also received some fundamental criticism over the years. Critics have, for instance, pointed to a rather narrow understanding of what constitutes ‘good science’, often accusing science reformers of promoting or even demanding positivist and quantitative perspectives, which have the potential to produce new mechanisms of exclusion rather than inclusion (see e.g., Bazolli, 2022; Guzzo et al., 2022; Malich & Rehmann-Sutter, 2022; Penders et al., 2019, 2020; Peterson & Panofsky, 2023; Ulpts et al., 2025). This issue becomes particularly apparent in the (online) discourse culture associated with the Open Science movement, wherein, for example, epistemologies that do not fit in with their programme are frequently the target of demeaning comments for not meeting their standards of reproducibility and replication (often true for qualitative research and most of the humanities) (Ulpts et al., 2025).
Critical voices have thus described their tone as often being abrasive, ripe with public call-outs, moralizing tendencies, and social media posts that shame scholars not living up to reformers’ ideals of ‘good science’ (Penders, 2022). This conduct seems to often culminate in a with-us-or-against-us mentality (Walkup, 2021). A direct consequence of such an isolationist mentality apparently fuelled by a conviction of a moral and intellectual superiority, is that Open Science advocates rarely, if ever, seriously engage with public criticism of their ideals, work, or proposed reforms. Critics are instead frequently denounced as ‘bullies’, ridiculed as being ignorant, or simply not acknowledged at all (see “brOpen Science” and “the tone debate”) (Derksen & Field, 2021; Flis, 2022; Penders, 2022; Rubin, 2024b; Whitaker & Guest, 2020).
Similarly, other critics have questioned whether the implementations of Open Science measures really fulfil the promises of openness and increased participation (especially of marginalized research communities). Instead, they have suggested that Open Science measures – amongst other things – feed into neoliberalism and shift existing or create new barriers, thus casting doubt over the promise of Open Science benefitting all stakeholders of the global science system equally and truly ‘opening up’ science (Bahlai et al., 2019; Beigel, 2024; Haustein et al., 2024; Hostler, 2024; Leonelli, 2023; Mirowski, 2018). In this context, whether deliberately or not the implementation of Open Science is, contrary to its claimed ideals, largely a process of subordination to the existing market systems which are dominated by monetary incentives and profit margins, especially from non-academic actors (Bartscherer & Reinhart, forthcoming). In other words, critics suspect that Open Science actually perpetuates and worsens already existing inequalities instead of alleviating them.
Another common concern has been that some of the demands made by Open Science reformers are unrealistic or simply impractical. For instance, the (monetary) resources and time available to researchers can vary starkly between individuals, institutes, and nations, but are ultimately always limited. Nevertheless, the implementation of Open Science practices often requires a dedicated amount of resources (e.g., ensuring reproducibility and conducting replications can require a lot of time) thereby ending up competing with tasks and schedules already in place in academia. Instead of reducing researchers’ workloads, Open Science often adds onto it (Callard, 2022; Hostler, 2023; Leonelli, 2022; Peterson & Panofsky, 2021).
Ironically, despite publicly hailing methodological rigour and standardizations of research procedures, reformers have already become the target of methodological criticism themselves. For instance, the problem diagnoses and proposed methodological solutions of reformers have been accused of lacking theoretical and statistical background knowledge (see e.g., Devezer et al., 2019, 2021). Moreover, critics have pointed towards apparent limits of proposed Open Science solutions such as replications and larger replication initiatives (Coyne, 2016; Feest, 2019; Haig, 2022; Rubin, 2024b), as well as pre-registrations (Rubin, 2024a; Szollosi et al., 2020). Lastly, critics have accused reformers of not adhering to their own standards when conducting or reporting their research, culminating in the recent retraction of an article that was previously heralded as evidence that the Open Science ‘revolution’ has indeed improved science (see Bak-Coleman, 2024; Bak-Coleman & Devezer 2024; Center for Open Science, 2023; Nosek et al., 2018; Protzko et al., 2024a, 2024b; Vazire, 2018).
As demonstrated with this brief overview of our own observations as well as already existing scholarly explorations of some of the more salient issues in the context of Open Science reform(er)s, Open Science reform has itself become the object of critical inquiry. In fact, the body of critical research on Open Science principles, their implementations, and the social environment created by those communities promoting them is continuously growing (Anderson-González & Gruson-Daniel, 2023; Field, 2022; Ottolini & Noel, 2024; Penders, 2022; Peterson & Panofsky, 2023; Rubin, 2024b).
The Workshop
The growing corpus of critical reflections on Open Science reform has been the inspiration for us to explore these critical debates around Open Science and science reform in more detail. We thus decided to bring together prominent scholars who contributed to these debates in the past and present, to be part of a one-day in-person workshop in Berlin this May. The idea is to have each of our invited speakers briefly present their own research on the varying aspects of the Open Science reform movement and to elaborate on how they theoretically and/or methodologically address the issues they sought to investigate.
To start off the workshop, Sheena F. Bartscherer will present her ongoing investigations into the value systems and narratives underlying the Open Science movement. In her talk, she will explore the self-perception and public communication of reformers’ activism, highlighting international replication initiatives and their surrounding community.
Sven Ulpts will proceed with a talk on agreements and discrepancies between researchers’ perspectives and the ideals promoted by Open Science reformers on issues like the practice and role of preregistration, replication as well as reproducibility. Hence, in his talk he will explore how current notions of Open Science reform relate to the actual realities and aims of researchers, who are supposed to live by the standards proposed or even demanded by reformers.
Sven’s presentation will be followed by Fernanda Beigel who will highlight the social and technical conditions underlying the advancements, controversies and resistances to Open Science reform in the social sciences and humanities. In particular, she will compare the ideals of Openness as propagated by current reform movements with implemented forms of Openness in Latin America and the historical conditions as well as developments that such local (indigenous) forms of Openness are contingent on.
Stefanie Haustein will explore the publishing side of Open Science in her talk, discussing her research on the financial dimensions underlying the actual implementation of Open Science. In particular, she will speak on open access models and how large academic publishers are able to make an even larger profit with new formats such as APCs.
Afterwards, Bart Penders will talk about how current science reform relates to the War on Science. Based on theories of politics of quantification, he will explore how the Open Science reform movement elevates specific epistemologies to be the guiding principle for all of science and how the roles of statistics, fraud, reliability and retractions are used and sometimes even weaponized.
Lastly, David Peterson will conclude the round of talks by presenting his work on theoretically approaching Metascience as a social movement, reflecting on how this conceptualization of the Metascience community behind the Open Science reform has been taken up by other researchers since, to characterize and investigate Open Science reform.
We will wrap up the workshop with an open panel discussion involving all six speakers, encouraging the audience to participate in a communal discussion and reflection of the day’s proceedings. The aim of the event is to provide a forum for the critical reflection of theoretical and methodological approaches to investigating modern science reform and Open Science more specifically.
We invite scholars interested in the social and epistemic dimensions of Open Science as well as those curious about studying science reform more broadly, to attend this workshop. Everyone, whether critical or enthusiastic about Open Science, is invited and welcome to join!
For registration please send an email directly to Sven Ulpts via: su [at] ps.au.dk
The deadline for registrations is Sunday, April 20th 2025.
Citation
Bartscherer, S. F., & Ulpts, S. (2025). Let’s talk about science reform! Introducing a workshop on theoretical and methodological approaches to investigating the open science movement. Critical Metascience. https://markrubin.substack.com/p/lets-talk-about-science-reform
References
Anderson-González, M., & Gruson-Daniel, C. (2023). Mapping contemporary “research on research” and “science studies”: How new methods change the traditional academic landscape and inform public open science policies. https://doi.org/10.31222/osf.io/p4mca
Bahlai, C., Bartlett, L., Burgio, K., Fournier, A., Keiser, C., Poisot, T., & Whitney, K. (2019). Open science isn’t always open to all scientists. American Scientist, 107(2), 78. https://doi.org/10.1511/2019.107.2.78
Bak-Coleman, J. (2024, September 24). I did not want to write this blog post: The Preregistration Revelation. https://joebakcoleman.com/blog/2024/protzko/
Bak-Coleman, J., & Devezer, B. (2024). Claims about scientific rigour require rigour. Nature Human Behaviour, 8(10), 1890–1891. https://doi.org/10.1038/s41562-024-01982-w
Bartscherer, S. F. & Reinhart, M. (unpublished manuscript). The community forming around replications. A cartography of international replication initiatives in the behavioural, social, and cognitive sciences.
Bazzoli, A. (2022). Open science and epistemic pluralism: A tale of many perils and some opportunities. Industrial and Organizational Psychology, 15(4), 525–528. https://doi.org/10.1017/iop.2022.67
Beigel, F. (2024). Cartographies for an inclusive Open Science. https://doi.org/10.1590/SciELOPreprints.10286
Callard, F. (2022). Replication and reproduction: Crises in psychology and academic labour. Review of General Psychology, 26(2), 199–211. https://doi.org/10.1177/10892680211055660
Center for Open Science. (2023, November 9). The reforms are working: Evidence that the credibility of social-behavioral sciences can be improved. https://www.cos.io/blog/the-reforms-are-working
Coyne, J. C. (2016). Replication initiatives will not salvage the trustworthiness of psychology. BMC Psychology, 4(1), 28. https://doi.org/10.1186/s40359-016-0134-3
Derksen, M., & Field, S. (2021). The tone debate: Knowledge, self, and social order. Review of General Psychology, 26(2), 172–183. https://doi.org/10.1177/10892680211015636
Devezer, B., Nardin, L. G., Baumgaertner, B., & Buzbas, E. O. (2019). Scientific discovery in a model-centric framework: Reproducibility, innovation, and epistemic diversity. PloS one, 14(5), e0216125. https://doi.org/10.1371/journal.pone.0216125
Devezer, B., Navarro, D. J., Vandekerckhove, J., & Ozge Buzbas, E. (2021). The case for formal methodology in scientific reform. Royal Society Open Science, 8(3), 200805. https://doi.org/10.1098/rsos.200805
Feest, U. (2019). Why replication is overrated. Philosophy of Science, 86(5), 895–905. https://doi.org/10.1086/705451
Field, S. M. (2022). Charting the constellation of science reform [Preprint]. Open Science Framework. https://doi.org/10.31219/osf.io/udfw4
Flis, I. (2022). The function of literature in psychological science. Review of General Psychology, 26(2), 146–156. https://doi.org/10.1177/10892680211066466
Guttinger, S. (2020). The limits of replicability. European Journal for Philosophy of Science, 10(2), 10. https://doi.org/10.1007/s13194-019-0269-1
Guzzo, R. A., Schneider, B., & Nalbantian, H. R. (2022). Open science, closed doors: The perils and potential of open science for research in practice. Industrial and Organizational Psychology, 15(4), 495–515. https://doi.org/10.1017/iop.2022.61
Haig, B. D. (2022). Understanding replication in a way that is true to science. Review of General Psychology, 26(2), 224–240. https://doi.org/10.1177/10892680211046514
Haustein, S., Schares, E., Alperin, J. P., Hare, M., Butler, L.-A., & Schönfelder, N. (2024). Estimating global article processing charges paid to six publishers for open access between 2019 and 2023. ArXiv. https://doi.org/10.48550/ARXIV.2407.16551
Hesselmann, F., & Reinhart, M. (2024). From scandal to reform: Approaches to research integrity at a turning point. Journal of Responsible Innovation, 11(1), 2414491. https://doi.org/10.1080/23299460.2024.2414491
Hostler, T. J. (2023). The invisible workload of open research. Journal of Trial and Error, 4(1), 21–36. https://doi.org/10.36850/mr5
Hostler, T. J. (2024). Open research reforms and the capitalist university: Areas of opposition and alignment. Collabra: Psychology, 10(1), 121383. https://doi.org/10.1525/collabra.121383
Leonelli, S. (2022). Open science and epistemic diversity: Friends or foes? Philosophy of Science, 89(5), 991–1001. https://doi.org/10.1017/psa.2022.45
Leonelli, S. (2023). Philosophy of Open Science (1st ed.). Cambridge University Press. https://doi.org/10.1017/9781009416368
Malich, L., & Rehmann-Sutter, C. (2022). Metascience is not enough – A plea for psychological humanities in the wake of the replication crisis. Review of General Psychology, 26(2), 261–273. https://doi.org/10.1177/10892680221083876
Mirowski, P. (2018). The future(s) of open science. Social Studies of Science, 48(2), 171–203. https://doi.org/10.1177/0306312718772086
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
Ottolini, L., & Noel, M. (2024, June 26). Opening up science as a work: An international comparison of openness to society and openness of publication - NECSUS. https://necsus-ejms.org/opening-up-science-as-a-work-an-international-comparison-of-openness-to-society-and-openness-of-publication/
Penders, B. (2022). Process and bureaucracy: Scientific reform as civilisation. Bulletin of Science, Technology & Society, 027046762211263. https://doi.org/10.1177/02704676221126388
Penders, B. (2024). Scandal in scientific reform: The breaking and remaking of science. Journal of Responsible Innovation, 11(1), 2371172. https://doi.org/10.1080/23299460.2024.2371172
Penders, B., De Rijcke, S., & Holbrook, J. B. (2020). Science’s moral economy of repair: Replication and the circulation of reference. Accountability in Research, 27(2), 107–113. https://doi.org/10.1080/08989621.2020.1720659
Penders, B., Holbrook, J. B., & De Rijcke, S. (2019). Rinse and repeat: Understanding the value of replication across different ways of knowing. Publications, 7(3), 52. https://doi.org/10.3390/publications7030052
Peterson, D., & Panofsky, A. (2021). Self-correction in science: The diagnostic and integrative motives for replication. Social Studies of Science, 51(4), 583-605. https://doi.org/10.1177/03063127211005551
Peterson, D., & Panofsky, A. (2023). Metascience as a scientific social movement. Minerva, 61(2), 147–174. https://doi.org/10.1007/s11024-023-09490-3
Protzko, J., Krosnick, J., Nelson, L., Nosek, B. A., Axt, J., Berent, M., Buttrick, N., DeBell, M., Ebersole, C. R., Lundmark, S., MacInnis, B., O'Donnell, M., Perfecto, H., Pustejovsky, J. E., Roeder, S. S., Walleczek, J., & Schooler, J. W. (2024a). High replicability of newly discovered social-behavioural findings is achievable. Nature human behaviour, 8(2), 311–319. https://doi.org/10.1038/s41562-023-01749-9 (Retraction published Nat Hum Behav. 2024 Oct;8(10):2067. doi: 10.1038/s41562-024-01997-3.)
Protzko, J., Krosnick, J., Nelson, L., Nosek, B. A., Axt, J., Berent, M., Buttrick, N., DeBell, M., Ebersole, C. R., Lundmark, S., MacInnis, B., O’Donnell, M., Perfecto, H., Pustejovsky, J. E., Roeder, S. S., Walleczek, J., & Schooler, J. W. (2024b). Retraction note: High replicability of newly discovered social-behavioural findings is achievable. Nature Human Behaviour, 8(10), 2067–2067. https://doi.org/10.1038/s41562-024-01997-3
Rubin, M. (2024a). Preregistration does not improve the transparent evaluation of severity in Popper’s philosophy of science or when deviations are allowed. ArXiv. https://doi.org/10.48550/ARXIV.2408.12347
Rubin, M. (2024b). Questionable metascience practices. Journal of Trial and Error, 4(1), 5–20. https://doi.org/10.36850/mr4
Szollosi, A., Kellen, D., Navarro, D. J., Shiffrin, R., Van Rooij, I., Van Zandt, T., & Donkin, C. (2020). Is preregistration worthwhile? Trends in Cognitive Sciences, 24(2), 94–95. https://doi.org/10.1016/j.tics.2019.11.009
Ulpts, S., Bartscherer, S. F., Field, S. M., & Penders, B. (2025). The social replication of replication: Moving replication through epistemic communities. https://doi.org/10.31235/osf.io/pqc4v_v2
Vazire, S. (2018). Implications of the credibility revolution for productivity, creativity, and progress. Perspectives on Psychological Science, 13(4), 411–417. https://doi.org/10.1177/1745691617751884
Walkup, J. (2021). Replication and reform: Vagaries of a social movement. Journal of Theoretical and Philosophical Psychology, 41(2), 131–133. https://doi.org/10.1037/teo0000171
Whitaker, K., & Guest, O. (2020, September 28). #bropenscience is broken science. The Psychologist. https://www.bps.org.uk/psychologist/bropenscience-broken-science
 | A guest post by
|
 | A guest post by
|