acceptodds
Under review as a conference paper at ICLR 2027

A Framework and Checklist for Validity Threats in Foundation Model Research

Abstract

Controlled experiments are the backbone of machine learning research, but at the scale of modern foundation models, they have become prohibitively expensive. Instead, the community increasingly relies on research strategies that approximate the ideal experiment at a fraction of the cost: proxy experiments and scaling laws, observational studies with publicly available models, and single-run designs that leverage variation within individual training runs. In this work, we show that these research strategies perform causal inference in disguise. Specifically, they aim to measure treatment effects conditional on a specific training recipe. To estimate this treatment effect, efficient research strategies replace various parts of the ideal experiment with cheaper substitutes. Based on this analysis, we draw on concepts from the empirical social sciences and propose a unified framework to identify the implicit assumptions made by different research strategies. We find that each comes with a characteristic validity profile: proxy experiments gain statistical and internal validity at the expense of external and construct validity; observational studies face confounding and effect heterogeneity; and single-run designs are strained by interference between treated units. In case studies with recent papers, we find that these assumptions often receive insufficient attention. To overcome this problem, we propose a checklist that can help authors and reviewers identify threats to the validity of foundation model research designs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
foundation model, experiment design, causal inference, validity, framework

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_LXEfOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a unified framework for understanding the validity of research designs used to study foundation models when full-scale controlled experiments are prohibitively expensive. It frames these designs as attempts to estimate recipe-specific treatment effects. It uses concepts from the empirical social sciences and evaluates the research designs along four types of validity: statistical, internal, external, and construct validity. Concretely, the paper analyzes three common research strategies: the proxy, observational, and single-run approach, and characterizes their respective validity risks and the required assumptions for validity. Experiments are performed to demonstrate key threats. One useful contribution is a practical checklist to help researchers and reviewers identify and assess validity threats in foundation model research. Case studies of recent papers show that the assumptions for the validity of research results often receive insufficient attention.

Critical strengths

-Overall, the paper is well-presented, and, to my understanding, methodologically sound.

-The paper provides clear and systematic analysis of three widely used research strategies -- the proxy, observational, and single-run approach -- and characterizes their underlying assumptions and validity risks.

-A key strength of the paper is its practical checklist for identifying and assessing validity threats in foundation model research. By creating concrete questions for authors and reviewers, the paper provides a useful tool for improving the design and evaluation of foundation model research.

Critical weaknesses

None. I do not identify any material weaknesses that would warrant rejection. However, I'm not familiar with the topic of foundation models research design, therefore, the novelty and practical significance of the proposed framework are not entirely clear to me.

Questions for authors

Not a question but a suggestion: It would be helpful to further clarify the extent to which the identified assumptions and validity threats are already recognized in foundation model research.

1 comment
AuthorsOpenReview ↗

Author Response

Thank you for your efforts in reviewing our paper and for appreciating its contribution!

Not a question but a suggestion: It would be helpful to further clarify the extent to which the identified assumptions and validity threats are already recognized in foundation model research.

Great suggestion. Previously, the paper discussed this in the section on related work. We now additionally discuss it in the introduction.

We would be happy to answer any additional questions during the rebuttal period.

Reviewer_7CV2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on understanding threats to validity in commonly-used methods for researching the behavior of foundation models. Since state-of-the-art foundation models can take a lot of time and be very expensive to train, it's often not realistic to train multiple versions of them with different parameters to understand performance with different settings. In practice, many proxy techniques are used to infer performance on the full model, such as experimenting with a proxy model and inferring model performance with scaling laws or performing meta-analysis on observational data from multiple runs. The authors point out that these methods are applications of causal effect estimation methods, a class of methods that has been well-studied and where each method depends on a set of often un-testable assumptions. The authors formalize this framing, provide examples, and demonstrate how violating these assumptions can lead to incorrect conclusions. This culminates in a checklist of necessary assumptions that can be used by foundation model researchers, either while designing experiments or while assessing the validity of existing experiments

Critical strengths

I think the setup of this paper is great. Formalizing the causal framing and assumptions behind experiments that are often performed fairly ad-hoc is very valuable and can provide helpful insight to practitioners. The way the paper is presented is also very clear. Laying out the types of validity up-front and then stepping through examples of each research strategy, while explaining the threats to validity it introduces and demonstrating with an experiment, makes the paper a very easy read and helps solidify the reader's understanding of each research strategy. (as opposed to waiting to do all the experiments at the end)

I really appreciate the authors' commitment to scientific rigor. Work that helps the community understand common pitfalls and how to do science better is challenging to do right but extremely important and is exactly the kind of work we need more of.

Critical weaknesses

The authors do a good job for the most part of explaining each method's possible threats to validity and demonstrating an example of these threats. However, the corresponding assumptions that need to hold to avoid such threats are not as clearly presented in all cases. Instead, they're usually spread between multiple subsections, sometimes listed explicitly as assumptions and sometimes implied through the descriptions of threats to validity. For example, Section 4 (The Observational Approach) has discussion of the exchangeability assumption and a mention of positivity and SUTVA under "Internal Validity and Confounding", then has an Experiment section, then has a "Further Threats" section, where Assumption 5 (external validity) is referenced. If the assumptions were listed in a more organized way in the appendix, this would be fine. I was hoping for a straightforward list of assumptions per research method, or even a table showing which assumptions applied. However, the appendix is instead grouped by validity type, so the threats for a single research approach are spread out between multiple sections.

If the checklist, at least, had clearer guidance and lists of assumptions to check, this also wouldn't be a big deal (since then, it would mainly act as a reference for the checklist). However, I fear that too much of the checklist is less a 'checklist' and more a light guidance for a large number of free response fields. Section A seems reasonable enough - mainly just listing out what research methods are being used. But Section B is far too open ended. Ideally, I would want Section B to be at least one level lower. If I have a paper I'm assessing that uses proxy models, I'd like the checklist to point me towards which assumptions I need to check in that case. As it is, the current version requires a reader to have a deep understanding of all possible validity threats and what research questions they apply to, so they know which ones are relevant based on their answers to Section A. And doing that mapping from the appendix isn't easy, since all of the assumption lists are titled based on the section number from the main paper, not their research method, and the descriptions of many of them are not standalone. This reads like a checklist that would be extremely valuable to have already filled out for you when assessing paper, but also one that would be extremely daunting to fill out for yourself (especially in the context of reading someone else's paper), to the point where I don't see many people wanting to put in the effort to learn the process to do it.

Some of the examples in the main paper could be stronger. For example, in Section 2 when describing "Internal Validity", the final sentence of that bullet presents an "example" that feels incomplete. I assume the intent is that a new model that comes out, that happens to have a different architecture, likely also has other aspects of the design that are different, so we can't claim that the change in outcome is due solely to the change in architecture. But you don't actually mention the key piece that "other aspects of the design are likely different", so it comes across a bit unclear.

Figure 4, you appear to be showing results between "plain-text" and "8-token", and between "proxy treatment" and "pretraining". In Section 3.2, when talking about Figure 4, it seems to only talk about the right (*8-token) side of the figure. Your terminology also doesn't quite match between the two (the text references "heavily repeated random token canaries", while the figure has "plain-text" and "8-token").

In Section 5, the description of a single-run experiment could be clearer and would really benefit from an example of how you can form treatment and control groups in a single run. We finally get a form of example a page later in the "Experiment" sub-section, but that context would be helpful to have when reading that page.

The description of the experiment in Section 5 looks like it may have swapped variable names. It describes run TT as the default 'from scratch' run and run CC as an identical run with some sequences swapped out. I assume TT and CC are intended to be "treatment" and "control", and the references later in this paragraph (e.g., "run T yields yi(ai)y_i(a_i) and run C yields yi(0i)y_i(0_i)") make me think that you just swapped the letters when initially defining them.

Questions for authors

No specific questions

2 comments
AuthorsOpenReview ↗

Author Response

Thank you for your great comments and efforts in reviewing our paper. We are glad to hear that you like the setup of the paper!

the corresponding assumptions [...] are not as clearly presented in all cases.

Appendix Table 1 now provides the straightforward list of assumptions per research method that the reviewer was asking for. The table also provides links to the formal discussion of each assumption, and to the corresponding bullet points in the revised checklist.

To further improve presentation, we revise the structure of the Appendix.

much of the checklist is less a 'checklist' and more a light guidance for a large number of free response field [...] the current version requires a reader to have a deep understanding of all possible validity threats and what research questions they apply to, so they know which ones are relevant based on their answers to Section A.

Thank you for this insightful observation. We address this in two parts.

First, we completely revise the instructions in Part B of the checklist. The instruction-bullets in the revised checklist are now much more concrete and clearly linked to the answers in Part A and the list of assumptions in Appendix Table 1.

Second, we release an online version of the checklist that is easier to use. The online version asks specifically about the assumptions that are relevant for the chosen research strategy. It is available here.

Some of the examples in the main paper could be stronger.

We use the additional page to add more examples to the main paper. We also add Appendix Section D, which illustrates the different validity types with examples.

In Section 5, the description of a single-run experiment could be clearer and would really benefit from an example of how you can form treatment and control groups in a single run.

The revised Section 5 has an improved description and an example early on.

Your terminology also doesn't quite match [Figure 4] / swapped variable names [Section 5]

The revised paper fixes these concerns.

Thank you again for your great comments. We would be happy to answer any additional questions during the rebuttal period.

AuthorsOpenReview ↗

Follow-up by Authors

Thank you again for your careful review and great comments on our paper. If our response and revision resolve your concerns, we would kindly ask you to consider increasing your score.

Reviewer_5VuAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors propose a framework to reframe — in aggregate — how the research community thinks about its research on modern foundation models, drawing a direct connection to causal inference. As modern foundation models often require prohibitively large/expensive experiments in order to demonstrate conclusive results, and our researchers questions about foundation models are typically causal questions, the authors suggest that the research community is really performing “causal inference in disguise,” and, crucially, doing so without adequately stating or accounting for the necessary assumptions for valid causal inference. To address this, the authors provide an explicit framing of the causal inference problem (treatment effect estimation) that researchers are typically engaged in, involving one or more training recipes, treatments, and outcomes. This effect estimation problem is then connected to four key types of validity — statistical, internal, external, and construct — and the authors detail several representative categories of experiments (using simpler/easier proxy approaches, doing an observational study, approximating from a single training run) and discuss the main threats to validity for each category of experiment.

Critical strengths
  • The paper is well-written and well-motivated. Apt examples are used throughout to connect the claims about foundation model research to the relevant papers doing such research.
  • The examples and unified formalism of the causal inference setup draws a clear (and helpful) arc to reason about foundation model research, and where it might be lacking, as a whole.
  • The callouts of specific validity threats, tied to each type of experiment, are usefully summarized.
Critical weaknesses
  • The discussion of interference is the weakest/least clear part to me. The argument seems to be: training on one document/item may have a causal effect on the outcome of another document (e.g., the presence of canary i may affect the outcome for canary j). But isn’t this true for every datapoint in general? What is it that directly connects this form of interference to the causal claim being made? Would this still be an issue if the treatment were defined at the dataset level instead of the datapoint level? Alternatively, could this be an issue of ‘multiple versions of treatment’ (also part of SUTVA) rather than interference? In the canary example, perhaps adding canary i is one version of treatment, and subsequently adding canary j is a second version of treatment? Some clarification on this issue, and probably a second example of an experiment where such interference can occur, would be helpful.
  • The checklist itself is not discussed in much detail in the main body of the text. As this is the main proposed solution to the identified problem, some more discussion of its design and important features/choices could help at the conclusion.
Questions for authors

My main questions are around interference (described also above in the weaknesses).

  • Isn’t interference as framed in this work a natural feature of any learning process that is trained on multiple units? What is it that directly connects this form of interference to the (causal) research claim being made?
  • Would this still be an issue if the treatment were defined at the dataset level instead of the datapoint level?
  • Alternatively, could this be an issue of ‘multiple versions of treatment’ (also part of SUTVA) rather than interference? In the canary example, perhaps adding canary i is one version of treatment, and subsequently adding canary j is a second version of treatment? Some clarification on this issue, and probably a second example of an experiment where such interference can occur, would be helpful.
4 comments
AuthorsOpenReview ↗

Author Response

Thank you for the thoughtful review and efforts in reviewing our paper. We are happy that the reviewer finds the paper well-written and well-motivated!

The discussion of interference is the weakest/least clear part to me.

We agree that this discussion could be improved and revise it in the paper.

Could this be an issue of ‘multiple versions of treatment’ (also part of SUTVA) rather than interference? / What is it that directly connects this form of interference to the (causal) research claim being made?

Let us clarify the difference between interference and multiple versions of treatment by drawing an analogy to a medical setting. In our setting, the goal is to estimate the average effect of training on a document on the document’s loss under the trained model. For this treatment effect, the documents are the units that form the population, training on the document is the treatment, and the per-document loss is the outcome. In analogy, the documents are the patients, training on the document is giving the patient a drug, and the per-document loss is the patient’s health outcome.

Now, no interference asks whether treating one patient affects the potential outcomes of another patient. For example, when many people get vaccinated, that can affect the potential outcomes for all patients, because all patients share the same public space. Here, assuming no interference may lead to an underestimation of the total effect of introducing vaccinations. In contrast, multiple versions of treatment would be, for example, that different people get their vaccination in different doses.

Returning to the memorization example, the fact that all documents share the same model weights, and thus treating document i may affect the outcomes for all documents is an example of interference. In this setting, multiple versions of treatment would be, for example, that some documents would be included several times in training while others would be included only once.

Would this still be an issue if the treatment were defined at the dataset level instead of the datapoint level?

If we understand the question correctly, it would not be an issue. This is because it would mean that we ask a different causal question. Concretely, if we asked, “What is the effect of training on this pretraining dataset?”, then treated unit would be the entire training run. The single run strategy would therefore be unsuitable because the examples within the dataset could no longer serve as independently treatable units. Interference would then arise only if training choices in one run affected outcomes in another run.

Isn’t interference as framed in this work a natural feature of any learning process that is trained on multiple units

This is exactly the point that we want to make: interference is an inherent threat to the internal validity of single-run designs. How strongly different units interfere and to what degree that invalidates the research claim must be discussed on a case-by-case basis.

The checklist itself is not discussed in much detail in the main body of the text.

We agree that this could be expanded and use much of the additional page for an expanded discussion of the checklist.

Thank you again for your careful review. We would be happy to answer any additional questions during the rebuttal period.

AuthorsOpenReview ↗

Follow-up by Authors

Thank you again for your careful review. If our response and revision resolve your concerns, we would kindly ask you to consider increasing your score.

Thank you for the additional explanation in answer to my questions. I still find the setup of the interference claim to be a bit messy. From the above, we have "the documents are the units that form the population, training on the document is the treatment, and the per-document loss is the outcome." In this case, treating document i means training on document i, and treating document j means training on document j, so for any documents that aren't identical, the treatments themselves are different (corresponding perhaps to the other part of SUTVA). To mirror the medical analogy above, document i and document j could be considered akin to different "doses" of the same treatment (even with the same model weights, we would expect each to have a different effect). For me, this muddies the claim that this is equivalent to interference.

AuthorsOpenReview ↗

Author Response

Thank you for the additional comment, we believe that we understand your point much better now.

First, we completely agree that training on document ii and training on document jj are doing different things to the model. In this sense, one might say that adding document ii and document jj to the training data are two different kinds of treatments that we can perform. Indeed, this framing fits right into the framework of our paper.

Now, the trick in the setting in Section 5 is that we are not measuring a single outcome at the level of the entire model, but that we have an outcome for every document. We also consider the treatment to be "adding the document to the training data", which is the same possible treatment for every document. In this setup, we only have a single treatment, and the fact that adding different documents to the training data has different effects on the model corresponds to treatment effect heterogeneity, not to multiple versions of treatment.

Importantly, this framing is inherited from the literature (as our goal is to analyze existing research strategies). For concrete examples of this setup in the literature, we refer to Lesci et al. (2024, Definitions 1 and 2). For another example, see Bordt and Pawelczyk (2026).

Formal definition of the setting and the assumptions

As an additional clarification, let us provide the formal definition of the setting and the assumptions:

  • Setting. Each document receives a binary treatment ai∈0,1a_i \in \\{0, 1\\} and has its own outcome yiy_i. For example, the treatment could be whether a benchmark question (the "document") is part of the training data or held-out, and the outcome could be whether the model is able to answer the benchmark question (data contamination).

  • Interference. In this setup, interference asks whether the outcome of document ii depends on the treatment of another document jj. Formally, let yi(ai,a−i)y_i(a_i, \mathbf{a}_{-i}) be the potential outcome for document ii given the treatment assignment for all documents (ai,a−i)(a_i, \mathbf{a}_{-i}). Then no interference requires

    yi(ai,a−i)=yi(ai,a−i′)for all ai,a−i,a−i′.y_i(a_i, \mathbf{a}_{-i}) = y_i(a_i, \mathbf{a}'_{-i}) \quad \text{for all } a_i, \mathbf{a}_{-i}, \mathbf{a}'_{-i}.

    Consequently, if training on document jj changes the loss of document ii, that is,

    yi(ai,aj=1,a−ij)≠yi(ai,aj=0,a−ij),y_i(a_i, a_j = 1, \mathbf{a}_{-ij}) \neq y_i(a_i, a_j = 0, \mathbf{a}_{-ij}),

    this is, by definition, a violation of no interference. In the example with benchmark questions, interference could happen, for example, if two questions are about a similar topic.

  • No multiple versions of treatment. This part of SUTVA is a condition within a unit. In other words, for each document ii, ai=0a_i=0 and ai=1a_i=1 must have a clear meaning, so that yi(1)y_i(1) is well defined. It would, for example, be violated if ai=1a_i=1 could equally mean adding the document once or hundreds of times.

  • Treatment effect heterogeneity. The fact that documents ii and jj differ, and therefore have different effects yi(1)−yi(0)≠yj(1)−yj(0)y_i(1) - y_i(0) \neq y_j(1) - y_j(0), is treatment effect heterogeneity.

Changes to the paper

Based on the comments by the reviewer, we now realize that the original submission did not give the specific setting and the formal definitions in Section 5 enough room. Over the course of the rebuttal, we have already restructured the appendix so that the reader can find the formal assumptions more easily, and we also added an overview table of all assumptions. For the final version, we additionally commit to further clarify the within-run approach in Section 5.

Thank you again for engaging in the discussion, we hope that this resolves your remaining concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We appreciate all the reviewers' insightful feedback, which has helped us improve the paper further. Here we summarize the most important changes to the paper during the rebuttal. Please also refer to the individual responses.

  • We use the additional page in the main paper to add more examples, clarify the discussion in Section 5 (Reviewer 5VuA, Reviewer 7CV2), and expand the discussion of the checklist (Reviewer 5VuA).
  • We re-organize the Appendix and now provide a straightforward list of all assumptions in Appendix Table 1 (Reviewer 7CV2). The novel Appendix Section D illustrates the four validity types with examples.
  • We revise the checklist to be less open-ended. Part B is now directly based on the list of assumptions (Reviewer 7CV2).
  • We release an (anonymized) online version of the checklist here. This format can more easily point towards the specific assumptions that need to be addressed under different research strategies, and might generally be easier to use than the plain-text version (Reviewer 7CV2).

The revised paper highlights these changes in color. We would be happy to answer any additional questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.