Hi Mark, great post! A lot of this resonates with my own experiences with preregistration.
I'd tend to think that the two scenarios of a deviation from a strict preregistration or an unanticipated decision after a vague preregistration tend to have pretty similar consequences for the credibility of the analysis subsequently presented. I.e., in both cases there is the possibility (albeit not the certainty) that knowledge about the results produced by different analysis choices affected which analysis the researcher decided to report.
Personally, I edge towards the strict preregistration route, but for pretty banal reasons. My experience has been that when a preregistration leaves a particular decision unstated, it's pretty easy for that to fall between the cracks in a communication sense. The researcher might not "click" that they actually had to make a decision, or they might forget to write down that this happened, and the reviewers may not realise that a decision needed making. In contrast, when a preregistration said X but the study did Y, it's a little more obvious to all concerned.
By the way, the idea of trade-offs makes me think of the one applying to data analysis plans created before versus after we've seen the data. When we create the analysis plan first (as in a preregistration), this has the advantage of ruling out the possibility of the observed values of the statistics from affecting which statistics we choose to report, this being a potential source of bias. However, this plan can't use other information we obtain after collecting data (e.g., knowledge about distributional assumption violations). In contrast, a plan created or modified after collecting data can take into account useful information from the data itself, but comes with the possibility that analysis decisions are consciously or unconsciously affected by knowledge of what substantive results they produce.
I don't think one could make a principled argument that either of these options is *in general* better than the other (unless one were a rabid predictivist or something). It depends on the situation. But when peer reviewing a preregistered study I do quite like being able to read about *both*!
Thanks very much, and thanks for your thoughts on this. Yes, I see what you mean about things being a bit clearer from a communication perspective in the case of the stricter, more prescriptive preregistration.
And it’s interesting you bring up the issue of taking account of information in the data (e.g., knowledge about distributional assumption violations). Nosek et al. (2018) discuss this issue a bit in their Preregistration Revolution paper under the subheading “Discovery of Assumption Violations During Analysis.” I give a nod to this point in my post (“researchers can preregister “if…then” contingencies (i.e., decision trees)…”). But, unless you have “God-like planning abilities” (Navarro, 2020), you won’t be able to cover every eventuality using that approach.
So, I agree that it’s always useful to adapt the analysis to the actual data you have in front of you, rather than the data that you planned for. Like you say, “it depends on the situation.” So, to make everyone happy, we can report the results that depend on the situation as well as those that depend on the preregistered plan! :-)
Hello Mark, I liked your post and agree with most of it. My only disagreement is with the notion that preregistered confirmatory tests become exploratory whenever researchers deviate from their preregistration. To me, there is a difference between a planned but deviated test and an unplanned test. I agree with you that confirmation and exploration are distinguished based on whether the test was planned, but I think that a deviated plan is still a plan, and so deviated preregistrations are still confirmatory.
However, not all confirmatory tests are equal. Firstly, a stricter preregistration prohibits more possible scenarios and thus can confirm more than a vaguer preregistration. Secondly, a confirmatory test’s confirmation decreases as the amount of deviations increase. Thus, I would rephrase the preregistration trade-off you have identified as one involving confirmation: assuming that the goal of researchers is to confirm their theories as much as they can, they are faced with a trade-off between 1) strict preregistrations that has more potential confirmation which can decrease if deviations are made, and 2) vague preregistrations that has less potential confirmation which can also decrease if deviations are made but such deviations are less likely. Basically, researchers can choose to make big bets (strict preregistrations) with potentially big pay-offs (strong confirmation) or small bets (vague preregistrations) with potentially small pay-offs (weak confirmation).
Glad you liked the post, and thanks very much for your comments.
For me a “planned but deviated” test is functionally equivalent to an “unplanned test” because the deviation is unplanned. Obviously, if the deviation was planned, then it wouldn’t be a deviation; it would part of the plan! So, I wouldn’t agree that “a deviated plan is still a plan,” because the deviation is, by definition, a deviation from the plan!
You explained that “researchers can choose to make big bets (strict preregistrations) with potentially big pay-offs (strong confirmation) or small bets (vague preregistrations) with potentially small pay-offs (weak confirmation).” This sounds like you’re arguing that strict preregistrations entail *severe* tests (Mayo, 2018) that produce strong confirmation, whereas vague preregistrations entail non-severe tests that don’t. But, in my view, it’s better to consider the strict-vague dimension as orthogonal to the severe-nonsevere dimension because some tests that arise from vague preregistrations may be more severe than tests that are prescribed by strict preregistrations. For example, imagine you preregister a very strict and prescriptive test that explains in great detail which methods and analyses you *must* use. However, these methods and analyses are actually highly unreliable and invalid! In this case, following the strict preregistration will lead to a nonsevere test. In contrast, if you preregister a vaguer set of methods and analyses, then you can use your discretion to pick and choose an approach that *increases* the severity of your test (for a related discussion, see Rubin & Donkin, 2022, https://doi.org/10.1080/09515089.2022.2113771). So, from my perspective, neither the prescriptiveness of your plan, the content of your plan, or the extent to which you deviate from that plan affects the severity of your test. The severity of your test is based on what you *actually did,* not what you planned to do.
However, as I pointed out in the Endnote in my post, there are many different points of view on this issue, and I'm sure some people share your perspective (e.g., perhaps Fife & Rodgers, 2019?). So, again, I’m grateful for your thoughts on this.
Hi Mark, great post! A lot of this resonates with my own experiences with preregistration.
I'd tend to think that the two scenarios of a deviation from a strict preregistration or an unanticipated decision after a vague preregistration tend to have pretty similar consequences for the credibility of the analysis subsequently presented. I.e., in both cases there is the possibility (albeit not the certainty) that knowledge about the results produced by different analysis choices affected which analysis the researcher decided to report.
Personally, I edge towards the strict preregistration route, but for pretty banal reasons. My experience has been that when a preregistration leaves a particular decision unstated, it's pretty easy for that to fall between the cracks in a communication sense. The researcher might not "click" that they actually had to make a decision, or they might forget to write down that this happened, and the reviewers may not realise that a decision needed making. In contrast, when a preregistration said X but the study did Y, it's a little more obvious to all concerned.
By the way, the idea of trade-offs makes me think of the one applying to data analysis plans created before versus after we've seen the data. When we create the analysis plan first (as in a preregistration), this has the advantage of ruling out the possibility of the observed values of the statistics from affecting which statistics we choose to report, this being a potential source of bias. However, this plan can't use other information we obtain after collecting data (e.g., knowledge about distributional assumption violations). In contrast, a plan created or modified after collecting data can take into account useful information from the data itself, but comes with the possibility that analysis decisions are consciously or unconsciously affected by knowledge of what substantive results they produce.
I don't think one could make a principled argument that either of these options is *in general* better than the other (unless one were a rabid predictivist or something). It depends on the situation. But when peer reviewing a preregistered study I do quite like being able to read about *both*!
Hi Matt,
Thanks very much, and thanks for your thoughts on this. Yes, I see what you mean about things being a bit clearer from a communication perspective in the case of the stricter, more prescriptive preregistration.
And it’s interesting you bring up the issue of taking account of information in the data (e.g., knowledge about distributional assumption violations). Nosek et al. (2018) discuss this issue a bit in their Preregistration Revolution paper under the subheading “Discovery of Assumption Violations During Analysis.” I give a nod to this point in my post (“researchers can preregister “if…then” contingencies (i.e., decision trees)…”). But, unless you have “God-like planning abilities” (Navarro, 2020), you won’t be able to cover every eventuality using that approach.
So, I agree that it’s always useful to adapt the analysis to the actual data you have in front of you, rather than the data that you planned for. Like you say, “it depends on the situation.” So, to make everyone happy, we can report the results that depend on the situation as well as those that depend on the preregistered plan! :-)
m
Hello Mark, I liked your post and agree with most of it. My only disagreement is with the notion that preregistered confirmatory tests become exploratory whenever researchers deviate from their preregistration. To me, there is a difference between a planned but deviated test and an unplanned test. I agree with you that confirmation and exploration are distinguished based on whether the test was planned, but I think that a deviated plan is still a plan, and so deviated preregistrations are still confirmatory.
However, not all confirmatory tests are equal. Firstly, a stricter preregistration prohibits more possible scenarios and thus can confirm more than a vaguer preregistration. Secondly, a confirmatory test’s confirmation decreases as the amount of deviations increase. Thus, I would rephrase the preregistration trade-off you have identified as one involving confirmation: assuming that the goal of researchers is to confirm their theories as much as they can, they are faced with a trade-off between 1) strict preregistrations that has more potential confirmation which can decrease if deviations are made, and 2) vague preregistrations that has less potential confirmation which can also decrease if deviations are made but such deviations are less likely. Basically, researchers can choose to make big bets (strict preregistrations) with potentially big pay-offs (strong confirmation) or small bets (vague preregistrations) with potentially small pay-offs (weak confirmation).
Thanks Bob,
Glad you liked the post, and thanks very much for your comments.
For me a “planned but deviated” test is functionally equivalent to an “unplanned test” because the deviation is unplanned. Obviously, if the deviation was planned, then it wouldn’t be a deviation; it would part of the plan! So, I wouldn’t agree that “a deviated plan is still a plan,” because the deviation is, by definition, a deviation from the plan!
You explained that “researchers can choose to make big bets (strict preregistrations) with potentially big pay-offs (strong confirmation) or small bets (vague preregistrations) with potentially small pay-offs (weak confirmation).” This sounds like you’re arguing that strict preregistrations entail *severe* tests (Mayo, 2018) that produce strong confirmation, whereas vague preregistrations entail non-severe tests that don’t. But, in my view, it’s better to consider the strict-vague dimension as orthogonal to the severe-nonsevere dimension because some tests that arise from vague preregistrations may be more severe than tests that are prescribed by strict preregistrations. For example, imagine you preregister a very strict and prescriptive test that explains in great detail which methods and analyses you *must* use. However, these methods and analyses are actually highly unreliable and invalid! In this case, following the strict preregistration will lead to a nonsevere test. In contrast, if you preregister a vaguer set of methods and analyses, then you can use your discretion to pick and choose an approach that *increases* the severity of your test (for a related discussion, see Rubin & Donkin, 2022, https://doi.org/10.1080/09515089.2022.2113771). So, from my perspective, neither the prescriptiveness of your plan, the content of your plan, or the extent to which you deviate from that plan affects the severity of your test. The severity of your test is based on what you *actually did,* not what you planned to do.
However, as I pointed out in the Endnote in my post, there are many different points of view on this issue, and I'm sure some people share your perspective (e.g., perhaps Fife & Rodgers, 2019?). So, again, I’m grateful for your thoughts on this.
Mark