Prediction vs Accommodation: Which is Better and When?
Prediction vs Accommodation: Which is Better and When?
Summary of Syrjänen (2023)
In an article published last week in Synthese, philosopher of science Pekka Syrjänen asked “does a theory become better confirmed if it fits data that was not used in its construction versus if it was specifically designed to fit the data?” The first approach is called prediction, and the second approach is called accommodation. The debate over the epistemic advantages of prediction and accommodation has been bubbling away for many years (for a recent overview, see Dellsén, 2023), but Syrjänen breathes new life into the discussion by contextualising it in relation to concerns about questionable research practices such as HARKing and science reforms such as preregistration and open research data and materials.
In his article, Syrjänen considers three research practices that are thought to decrease the epistemic value of accommodation: (1) overfitting, (2) hypothesis hunting, and (3) fudging. After consider these issues, he concludes that accommodation may have roughly the same epistemic value as prediction. Indeed, he argues that accommodation may have greater epistemic value than prediction in some situations. I’ll briefly summarise his key points below.
(1) Overfitting
Overfitting occurs when models are created to fit systematic effects, but they also unintentionally fit random sampling error. Syrjänen notes that prediction avoids overfitting because it tests models that have been created independent from the research results and their random sampling error. However, he also notes that, during accommodation, adding more data can reduce the potential for overfitting by reducing random variation (Yarkoni & Westfall, 2017).
To illustrate, imagine that you either (A1) split your data into two parts, generate a model based on the first part via accommodation and then (A2) predict results based on the second part or (B) generate a model by accommodating all of the data across the whole data set. Syrjänen acknowledges that the risk of overfitting is greater in the case of (B)’s accommodation than (A2)’s prediction. However, he also notes that the risk of overfitting is greater in (A1)’s accommodation than in (B)’s accommodation because (B) uses more data and so is less likely to be influenced by random sampling error. Furthermore, in cases in which (A2)’s prediction is based on an independent sample, its smaller sample size relative to (B) will make it less representative of the target population, leading to lower predictive accuracy. Finally, although prediction has the advantage of avoiding overfitting to random noise, accommodation has the advantage of avoiding underfitting of the systematic effect. Hence, Syrjänen concludes that overfitting is not a winning argument against accommodation and that, in some situations, accommodation is preferable.
(2) Hypothesis Hunting
Based on the work of Mayo (1996), Syrjänen also considers a practice known as “hypothesis hunting.” As he explained,
in hypothesis hunting, researchers collect a large dataset with many variables, and then sift through the data looking for any statistically significant effects between the variables. If such associations are found, the researchers dismiss the non-significant results and present the significant results as if they had been targeting those from the outset.
Hypothesis hunting is sometimes described as hypothesising after the results are known or HARKing (Kerr, 1998).1 A key point here is that HARKed effects are not necessarily spurious; they may also be real effects. Furthermore, researchers can use extant background knowledge to screen out effects that are less likely to be true. For example, after engaging in hypothesis hunting, a researcher might find a positive correlation between generalized self-efficacy and indecision. However, background knowledge in psychology suggests that this correlation should be negative, not positive. With this knowledge in hand, the researcher (and their prospective reviewers) may be more likely to discount this effect as a “false positive.”
Hence, Syrjänen concludes that hypothesis hunting may only be problematic when a field’s theory-based screening process is relatively lax. In contrast, “in fields where the standards for hypothesis construction are stricter, the chances that researchers discover and publish spurious but credible findings is substantially diminished.”
(3) Fudging
Finally, Syrjänen considers Lipton’s (2004) work on “fudging.” Here, theorists make forced or unnatural changes to their theories in order to accommodate their results (i.e., another issue connected with HARKing). These changes lead to potentially inferior (convoluted) explanations that may warrant lower confidence. According to Lipton, this undisclosed fudging is problematic even when people are able to acknowledge that the resulting theory is of low quality. Syrjänen rightly queries Lipton’s argument here, asking:
Do the consequences of fudging—e.g. convoluted theories and constructs—speak for themselves, or do scientists need to resort to the distinction between novel prediction and accommodation to avoid the negative epistemic consequences of fudging?
In response, Syrjänen points out that there seems to be no evidence that people need to know that fudging has occurred in order for them to be able to recognise its deleterious outcomes vis-a-vis low quality theory. In other words, knowing that a theory has been adapted to fit a result doesn’t seem to contribute to an evaluation of that theory over and above an evaluation of that theory’s actual content.
So Which is Better? Prediction or Accommodation?
On the basis of the above arguments and more, Syrjänen concludes that, “overall, novel prediction and accommodation appear roughly on a par in their epistemic consequences, or novel prediction could even be associated with greater epistemic issues than accommodation.” However, in practice, the answer to the “which is better?” question depends on a variety of complex contextual factors. Hence, as Syrjänen explains:
Whether either novel predictions or accommodations are more compelling in scientific practice is itself a contingent issue. Depending on the contingent methodological choices that are made by researchers in practice, either predictivist or accommodationist advantages could come to effect in different contexts in science.
Implications for Metascience and Open Science Reforms
Syrjänen’s work has implications for several issues discussed in the areas of metascience and open science reform. For example, he considers the distinction between confirmatory and exploratory research and concludes that
it remains unclear to what extent there are overall epistemic differences between prediction-based confirmatory research and accommodation-based exploratory research (cf. Oberauer & Lewandowsky, 2019; Szollosi & Donkin, 2021; Rubin & Donkin 2022).
Syrjänen also has a nuanced evaluation of preregistration. He argues that preregistration may help to bolster the epistemic value of prediction, giving it a potential advantage over accommodation. However, practical difficulties with implementing preregistration may limit the size of this advantage. In particular, low uptake in the scientific community, vague research specifications, and frequent deviations from the research plan may reduce the degree of support provided by preregistration (for more on these last two issues, see my post on the “preregistration prescriptiveness trade-off”). Furthermore, other open science practices, such as open data and open research materials, may reduce any putative advantage of prediction over accommodation even further.
Finally, Syrjänen calls for more transparency in research, but he calls for a specific type of transparency called “contemporary transparency,” which is something I’ve previously discussed:
Based on the results of this investigation, it appears that in terms of transparency, only what Rubin (2020) calls ‘contemporary transparency’ may be required in theory evaluation, i.e. researchers should disclose their current hypotheses, methods, and analyses, and justify these over alternative possibilities. What Rubin (2020) calls ‘historical transparency,’ i.e. transparency about differences between what the researchers originally planned to do and what they ended up doing, may not ultimately count as much from the epistemic point of view, as long as contemporary transparency is provided.
The Article
Syrjänen, P. (2023). Novel prediction and the problem of low-quality accommodation. Synthese, 202, Article 182, 1-32. https://doi.org/10.1007/s11229-023-04400-2 (Open Access)
Reference for this Post
Rubin, M. (2023, December 1). Prediction vs accommodation: Which is better and when? Summary of Syrjänen (2023). Critical Metascience. https://markrubin.substack.com/p/prediction-vs-accommodation
This process of hypothesis hunting can also be described as “p-hacking.” Syrjänen notes that the terminology in this area can be confusing, with different people using the same term to mean different things. Here, he follows Oberauer and Lewandowsky (2019) and uses “HARKing” in the context of accommodation (i.e., generating a hypothesis to fit the results) and “p-hacking” in the context of prediction (i.e., finding results that fit the hypothesis).