The authors do a good job for the most part of explaining each method's possible threats to validity and demonstrating an example of these threats. However, the corresponding assumptions that need to hold to avoid such threats are not as clearly presented in all cases. Instead, they're usually spread between multiple subsections, sometimes listed explicitly as assumptions and sometimes implied through the descriptions of threats to validity. For example, Section 4 (The Observational Approach) has discussion of the exchangeability assumption and a mention of positivity and SUTVA under "Internal Validity and Confounding", then has an Experiment section, then has a "Further Threats" section, where Assumption 5 (external validity) is referenced. If the assumptions were listed in a more organized way in the appendix, this would be fine. I was hoping for a straightforward list of assumptions per research method, or even a table showing which assumptions applied. However, the appendix is instead grouped by validity type, so the threats for a single research approach are spread out between multiple sections.
If the checklist, at least, had clearer guidance and lists of assumptions to check, this also wouldn't be a big deal (since then, it would mainly act as a reference for the checklist). However, I fear that too much of the checklist is less a 'checklist' and more a light guidance for a large number of free response fields. Section A seems reasonable enough - mainly just listing out what research methods are being used. But Section B is far too open ended. Ideally, I would want Section B to be at least one level lower. If I have a paper I'm assessing that uses proxy models, I'd like the checklist to point me towards which assumptions I need to check in that case. As it is, the current version requires a reader to have a deep understanding of all possible validity threats and what research questions they apply to, so they know which ones are relevant based on their answers to Section A. And doing that mapping from the appendix isn't easy, since all of the assumption lists are titled based on the section number from the main paper, not their research method, and the descriptions of many of them are not standalone. This reads like a checklist that would be extremely valuable to have already filled out for you when assessing paper, but also one that would be extremely daunting to fill out for yourself (especially in the context of reading someone else's paper), to the point where I don't see many people wanting to put in the effort to learn the process to do it.
Some of the examples in the main paper could be stronger. For example, in Section 2 when describing "Internal Validity", the final sentence of that bullet presents an "example" that feels incomplete. I assume the intent is that a new model that comes out, that happens to have a different architecture, likely also has other aspects of the design that are different, so we can't claim that the change in outcome is due solely to the change in architecture. But you don't actually mention the key piece that "other aspects of the design are likely different", so it comes across a bit unclear.
Figure 4, you appear to be showing results between "plain-text" and "8-token", and between "proxy treatment" and "pretraining". In Section 3.2, when talking about Figure 4, it seems to only talk about the right (*8-token) side of the figure. Your terminology also doesn't quite match between the two (the text references "heavily repeated random token canaries", while the figure has "plain-text" and "8-token").
In Section 5, the description of a single-run experiment could be clearer and would really benefit from an example of how you can form treatment and control groups in a single run. We finally get a form of example a page later in the "Experiment" sub-section, but that context would be helpful to have when reading that page.
The description of the experiment in Section 5 looks like it may have swapped variable names. It describes run T as the default 'from scratch' run and run C as an identical run with some sequences swapped out. I assume T and C are intended to be "treatment" and "control", and the references later in this paragraph (e.g., "run T yields yi(ai) and run C yields yi(0i)") make me think that you just swapped the letters when initially defining them.
Author Response
Thank you for your efforts in reviewing our paper and for appreciating its contribution!
Great suggestion. Previously, the paper discussed this in the section on related work. We now additionally discuss it in the introduction.
We would be happy to answer any additional questions during the rebuttal period.