What's Special About Metascience?
What's Special About Metascience?
Comments on Romero (2023)
Felipe Romero presented his work - “The conceptual origins of metascience: Fashion or revolution?” - in The Popper Seminar at the London School of Economics on 30th May 2023. Here, I highlight a few points that he raised and provide some comments along the way.
What is Metascience?
Metascience is often described as the “science of science.” However, as Peterson and Panofsky (2023) recently observed, contemporary metascience is something more than this. To illustrate, they used a Venn diagram to show how metascience may be conceived as falling at the intersection of science of science, open science, and methodological activism.
How do we Demarcate Metascience?
After providing some examples of the various aspects of metascience, Romero considers some different ways of distinguishing metascience from related disciplines such as (a) the history and philosophy of science, (b) sociology, and (c) science, technology, and society (STS) studies. He notes that metascientists see themselves as quite separate from these other disciplines. Indeed, he notes that some metascientists seem to show a resistance to engaging with these disciplines by (a) failing to engage in interdisciplinary discussions, (b) not citing interdisciplinary work, and (c) generally adopting “an adversarial attitude.”
But what distinguishes metascience from these other disciplines? Romero runs through several possibilities and considers the pros and cons in each case.
Metascience as Empirical Research
Romero notes that metascientists tend to stress the empirical nature of their work as a distinctive feature. As Jonathon Schooler explained in Peterson and Panofsky’s (2023) article, metascientists champion “less armchair, less qualitative distillation of impressions and more hard-nosed empiricism” (p. 167).
However, as Romero points out, one problem with emphasising the empirical character of metascience is that it runs the risk of concealing and/or overlooking important philosophical issues. What is a replication? What counts as evidence? What is progress? Empirical work must adopt philosophical positions on these questions, even if they’re only implicit positions. As Romero explains, “you might think that the work is empirical, but you are already making commitments about all of these notions.”
The philosopher Daniel Dennett (2013) famously argued that “there is no such thing as philosophy-free science, just science that has been conducted without any consideration of its underlying philosophical assumptions” (p. 20). Similarly, it might be argued that there is no such thing as a philosophy-free metascience, and greater discussion of the philosophy of metascience may help to build mutually beneficial bridges between metascience and the philosophy of science.
Romero accepts that metascientists do conceptual work around issues such as replication, evidence, and progress. However, he suggests that, without paying closer attention to previous work that’s been done in other disciplines, “there is a risk of reinventing the wheel and repeating mistakes.” To be fair, some metascientists couch their work within a Popperian philosophy of science (Derksen, 2019). Nonetheless, there remain concerns about the lack of integration with other positions in the philosophy of science and in other related fields (e.g., Gervais, 2021; Morawski, 2019, p. 234; Wiggins & Christopherson, 2019). For example, Gervais (2021) explained that:
Metascience and methodological reform movements—despite noble intentions—risk needlessly reinventing other fields’ time-tested wheels or, worse, reproducing the problematic norms, institutions, and incentives of status quo science (p. 827).
To illustrate his point, Romero notes that some metascientists focus on “methodological individualism,” according to which “(bad) statistics and methodology are the primary factors affecting science.” From this perspective, many of the problems lie with individuals, and many of the proposed solutions are correspondingly individualistic in nature. Hence, as Field (2022) observed, “the current crisis prominently features finger-pointing and ad-hominem attacks between researchers, where direct accusations of incompetence and data fabrication are hurled around between people and fields” (p. 13).
However, Romero observes that other metascientists are rediscovering the idea that we need to look at social structures and organization to affect scientific progress (for further discussions, see Flis, 2019, p. 175; Flis, 2022; Wiggins & Christopherson, 2019, p. 210). I think this broader and more abstract level of analysis is more befitting of a genuinely metascientific approach. In my view, scientists should critique the work of other scientists, and metascientists should critique scientific processes and approaches.
Metascience as a Focus on Methods
Romero also notes that the “dominant narrative…that metascience is good because it is empirical…could lead to methodological fetishism.” In my view, “methodological fetishism” occurs when people use methodological checklists and heuristics in ways that ignore the local goals of specific research studies (see also Devezer et al., 2021, p. 3). For example, depending on the function(s) that you think it serves, preregistration may not be a useful approach in certain research situations (e.g., some qualitative studies; Steltenpohl et al., 2023). Nonetheless, Spitzer and Mueller (2023) found that some psychological researchers suggest making preregistration mandatory, and Aguinis et al. (2020, p. 29) have argued that we should “require” preregistration for both quantitative and qualitative studies. In these cases, the overapplication of preregistration may represent a form of methodological fetishism.
Furthermore, the use of checklists and heuristics may mislead researchers, reviewers, and their readers into a false sense of security about their approach when a deeper, more philosophical consideration would reveal important uncertainties and limitations. As Chamberlain (2000) explained, “it is much less worrying to concern oneself with the correct details of method and methodology than to tangle with the philosophy of science” (p. 293).
Metascience as Fashion
Metascience has seen a remarkable rise in popularity in the wake of the replication crisis. Hence, Romero asks whether metascience may be a “fashion,” although he makes it clear that this term is not used in a pejorative manner.
To illustrate, Romero considers how metascience and open science might fit with a fashion model that has been applied previously in science (Crane, 1969; Starbuck, 2009). Consistent with this model, he notes that metascience contains:
Elites: Small groups of scholars who set fashions
Conformity: Scholars who adopt practices they don’t endorse
Performativeness: Scientists who pursue a practice/theory to elicit approval from others
“What’s the problem?” asks Romero. “I think that some scientists are adopting open science as a fashion and this leads to performative open science.”
As an example, Romero considers “performative preregistration,” which might manifest in a couple of ways: First, the “underspecification problem” is that researchers might go through the motions of preregistration, but preregister plans that are vague and, therefore, relatively useless (Bakker et al., 2020). Second, the “undisclosed deviations problem” is that researchers may not indicate deviations from their preregistered plan (Akker et al., 2022). Again, this approach ticks the preregistration box, but it makes the potential benefits of preregistration relatively inaccessible.
Romero concludes that a risk associated with metascience being a fashion is that fashions can have poor epistemic benefits and, when scientists undertake a cost-benefit analysis, this may result in a decline in the uptake of research practices like preregistration.
Metascience as Activism
According to Romero, the best way to demarcate metascience is to refer to its normative, activist component: Metascientists tend to be activists who are trying to change scientific systems and culture (see also Peterson & Panofsky, 2023, p. 147).
In Romero’s view, it is the activism aspect of metascience that provides the sharpest distinction with the history and philosophy of science, sociology, and STS, because these other disciplines focus on describing and understanding science, but not necessarily improving it. Importantly, however, if metascience is to be distinguished by its goal of “improving science,” then I think the metascience community should include an adequate representation of viewpoints about what “science” is and how it can best be “improved.” In this sense, the current overrepresentation of psychologists in this community may be problematic (see also Flis, 2019).
Final point from me: As I’ve noted elsewhere (Rubin, 2023), there’s a danger of homogenizing metascience in these discussions (Field, 2022). Like science, metascience may be many things to many people, and it may be done in many different ways, with different metascientists occupying different positions in Peterson and Panofsky’s (2023) Venn diagram as well as having different interdisciplinary overlap with the history and philosophy of science, sociology, and STS. There is strength in diversity here, and we should be careful not to lose sight of this diversity during efforts to categorize and demarcate metascience.
Further Info
Website: https://www.feliperomero.org/
Publications: https://www.feliperomero.org/publications.html
References
Aguinis, H., Banks, G. C., Rogelberg, S. G., & Cascio, W. F. (2020). Actionable recommendations for narrowing the science-practice gap in open science. Organizational Behavior and Human Decision Processes, 158, 27-35. https://doi.org/10.1016/j.obhdp.2020.02.007
Bakker, M., Veldkamp, C. L., van Assen, M. A., Crompvoets, E. A., Ong, H. H., Nosek, B. A.,…& Wicherts, J. M. (2020). Ensuring the quality and specificity of preregistrations. PLoS Biology, 18(12), e3000937. https://doi.org/10.1371/journal.pbio.3000937
Chamberlain, K. (2000). Methodolatry and qualitative health research. Journal of Health Psychology, 5(3), 285-296. https://doi.org/10.1177/135910530000500306
Crane, D. (1969). Fashion in science: Does it exist? Social Problems, 16(4), 433-441. https://doi.org/10.2307/799952
Derksen, M. (2019). Putting Popper to work. Theory & Psychology, 29(4), 449-465. https://doi.org/10.1177/0959354319838343
Dennett, D. C. (2013). Intuition pumps and other tools for thinking. WW Norton & Company.
Devezer, B., Navarro, D. J., Vandekerckhove, J., & Ozge Buzbas, E. (2021). The case for formal methodology in scientific reform. Royal Society Open Science, 8(3), Article 200805. https://doi.org/10.1098/rsos.200805
Field, S. M. (2022, July 13). Charting the constellation of science reform. https://doi.org/10.31219/osf.io/udfw4
Flis, I. (2019). Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis. Theory & Psychology, 29(2), 158-181. https://doi.org/10.1177/0959354319835322
Flis, I. (2022). The function of literature in psychological science. Review of General Psychology, 26(2), 146-156. https://doi.org/10.1177/10892680211066466
Gervais, W. M. (2021). Practical methodological reform needs good theory. Perspectives on Psychological Science, 16(4), 827-843. https://doi.org/10.1177/1745691620977471
Morawski, J. (2019). The replication crisis: How might philosophy and theory of psychology be of use? Journal of Theoretical and Philosophical Psychology, 39(4), 218–238. https://doi.org/10.1037/teo0000129
Peterson, D., & Panofsky, A. (2023). Metascience as a scientific social movement. Minerva https://doi.org/10.1007/s11024-023-09490-3
Romero, F. (2023, May 30) The conceptual origins of metascience: fashion or revolution? YouTube. https://youtu.be/AKWCCdK-jgc
Rubin, M. (2023). Questionable metascience practices. Journal of Trial & Error. https://doi.org/10.36850/mr4
Spitzer, L., & Mueller, S. (2023). Registered report: Survey on attitudes and experiences regarding preregistration in psychological research. Plos One, 18(3), e0281086. https://doi.org/10.1371/journal.pone.0281086
Starbuck, W. H. (2009). The constant causes of never-ending faddishness in the behavioral and social sciences. Scandinavian Journal of Management, 25(1), 108-116. https://doi.org/10.1016/j.scaman.2008.11.005
Steltenpohl, C. N., Lustick, H., Meyer, M. S., Lee, L. E., Stegenga, S. M., Reyes, L. S., & Renbarger, R. L. (2023). Rethinking transparency and rigor from a qualitative open science perspective. Journal of Trial & Error. https://doi.org/10.36850/mr7
Van den Akker, O., van Assen, M. A. L. M., Enting, M., de Jonge, M., Ong, H., Rüffer, F. F.,…Bakker, M. (2022, October 18). Selective hypothesis reporting in psychology: comparing preregistrations and corresponding publications. PsyArXiv. https://doi.org/10.31222/osf.io/nf6mq
Wiggins, B. J., & Christopherson, C. D. (2019). The replication crisis in psychology: An overview for theoretical and philosophical psychology. Journal of Theoretical and Philosophical Psychology, 39(4), 202–217. https://doi.org/10.1037/teo0000137
Reference for this Post
Rubin, M. (2023, June 13). What’s special about metascience: Comments on Romero (2023). Critical Metascience. https://markrubin.substack.com/p/whats-special-about-metascience
Subscribe to this Blog and/or Share this Article