acceptodds
Under review as a conference paper at ICLR 2027

Beyond OOD: Evaluating Large Foundation Models via Out-of-World Distribution

Abstract

Large Foundation Models (LFMs) have shown strong performance across diverse tasks, yet it remains unclear whether such performance reflects genuine cross-distribution generalization or interpolation over web-scale pretraining data. Conventional out-of-distribution (OOD) evaluation is ineffective, since the boundaries of LFM pretraining corpora are opaque and data contamination is difficult to rule out. Building on a world-model view of data generation, we make explicit a world variable that governs the input–output semantics and use its regular-world support to characterize the pretraining boundary. This leads to Out-of-World Distribution (OOWD), an OOD-like evaluation framework that first intervenes on this world variable and then evaluates cross-distribution transfer under the induced counter-world semantics. We instantiate OOWD across diverse generalization settings, including symbolic-to-verbal mathematical, program-to-natural-language, cross-style visual, and text-to-image transfer. Across these settings, we find that LFMs consistently transfer counter-world rules from the source domain to unseen target domains, whereas small models fail despite also fitting the source-domain data. Mechanistic analyses associate transfer with deeper layers and sparse causally relevant components. We further convert the principle into algorithmic design, OOWDAlign, for improving the generalization of scratch models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
out-of-domain generalization, large foundation models evaluation, data contamination, world model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6NuROpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary
  • The paper proposes a new evaluation paradigm to test the generalization of large models (e.g. LLMs or vision-and-language models) beyond their training distribution. It is kindof an extension of the OOD paradigm, applicable to large models whose training distribution is not known. Their key insight is to "characterize the boundary [between in- and out-of-distribution] in terms of world semantics rather than observations". The framework is very general ("OOWD").

  • They then apply the framework to evaluate small and large models in 3 domains, and find that the large models show stronger generalization.

  • They then perform mechanistic analyses on these models to understand why they generalize better.

  • They finally propose a method to improve their form of generalization ""OOWDAlign".

Critical strengths

(S1) The evaluation of generalization capabilities is an important topic, and the issues of data contamination/data coverage (the core motivation for this paper) are very important concerns in the evaluation of ML systems in general.

(S2) The framework is applied to several domains.

(S3) The experiments/results are analyzed under multiple angles (behavioral, mechanistic).

Critical weaknesses

Main weaknesses

(W1) The first main issue is the clarify of the presentation and what the proposed framework is about. I had to go through the whole paper until I could make sense of what was initially stated in the abstract and intro. I now understand that the paper is about evaluating models after fine-tuning them on new tasks that have been handcrafted to tbe incompatible with the similar tasks from the (supposed) training data. I don't understand why this is not stated as such more explicitly upfront in the paper. My advice for the authors (maybe for next time) is to get feedback on their manuscript from a friend who doesn't know at all about the project. Right now it feels like the paper is written for someone who already knows everything about it.

(W2) The second main concern is whether the proposed framework is really doing what the authors claim it's doing. I understand that the initial motivation is to fix the issue with "traditional" OOD evaluations that they can't be guaranteed to be OOD because we don't exactly know the training distribution of large models. The goal of these traditional OOD evaluations is to evaluate the robustness of models to inputs that they haven't been extensively exposed to during training. So I assume that this is the goal here as well (I can't find a clear statement of a different goal in the paper). But after reading the paper, I understand that, what s being tested, is how robust the model is on new tasks after being fine-tuned on data from these new tasks. These experiments are interesting, but I think that the findings are answering very different questions, namely how much knowledge is reused across related-but-conficlting tasks, and across pretraining/fine-tuning phases.


Detailed or minor comments

(C1, related to W1) The first 4 pages of the paper are very vague and abstract. It's very difficult to understand concretely what the paper proposes. A concrete example (and possibly a "teaser" figure in the intro) to anchor the discussion from the start would be very helpful. The critical questions that the reader needs to know about are only answered much later in the text, like what does the framework prescribe exactly and what does it take to apply it to a new task/domain/model? What does iit entail exactly, like creating a new dataset? metrics? fine-tuning models? etc.

(C2, related to W2) I'm still trying to understand what the OOWD framework does exactly. Taking the frist (math) domain for example, I understand that the task is: "answer arithmetic questions". The pretrained model is fine-tuned on symbolic examples, and evaluated on natural language example. The trick of OOWD is simply to assign different meanings to the digits and operators. Is this correct? If so, for a given pretrained model with a given (fixed) arithmetic proficiency, I understand that the proposed setting is evaluating how well the model can learn (through fine-tuning) new meanings of the digits/operators that conflict with its pretrained knowledge. (This is my understanding but not really what is stated in the paper.)

(C3) Sentences like this in the formal description are really unhelpful unless you already understand the whole paper "/(...) then evaluates cross-distribution transfer under the same induced semantics" (across what distribution? same as what? induced by what?).

(C4) "conventional OOD evaluation requires source and target distributions to lie outside the training distribution": I don't understand, I think the source usually refers to the training distribution.

(C5) "scratch models": I don't know what this means. I suppose the authors mean "models trained from scratch", but this is not a good qualifier; all models are initially trained from scratch.

(C6) In the section on related work: "Existing literature often characterizes OOD generalization as a two-stage process: a model first recovers task-relevant latent variables from inputs, and then predicts the correct output based on these." I'm familiar with the OOD generalization literature but I don't see how this description fits the series of citations that follow this sentence. Maybe the citations on causal representation learning are related, but OOD generalization is kindof a second point to these works.

Questions for authors

Did I misunderstand the paper (in relation to W2/C2 above)?

Prospects w.r.t. a possible update of my rating:

  • Addressing W1 requires an extensive revision of the first ~4 pages of the paper, which I'm not sure can be addressed solely with the promise of an improved final version.

  • If the concern in W2 is correct, fixing W2 means changing the framing of the paper, but this would require an extensive rewrite of (at least) the first few pages of the paper. The experiments themselves are valuable and may be sufficient. My comments are rather about what the proposed setting/experiments address, and what the observations mean.

  • If the concern in W2 is correct, this also means that the main strength of the paper (evaluating capabilities with possible data contamination) does not hold.

(No need to reply to the minor comments C3-C6.)

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6NuR

Thank you for taking the time to review our paper carefully. We especially appreciate such attentive engagement with our work in today's research environment. Thank you also for identifying the presentation problems so directly. Below, we present our revisions and responses.

W1 / C1: Making the procedure explicit in the first four pages.

We have substantially revised the opening presentation, with changes marked in blue in the revised manuscript. In particular, we added Figure 1 to the introduction to show the full procedure: construct a counter-world rule, fine-tune on source examples following that rule, and test the adapted model on a target domain with fixed weights. We replaced the abstract, formalization-led explanation in the introduction with a concrete account aligned with this figure, and reorganized Section 3 to make the distributional logic explicit. These revisions are already present in the revised manuscript, rather than being deferred to the final version.

Concretely, applying OOWD requires constructing or relabeling data under a specified counter-world rule, choosing different source and target input distributions, and adapting the model on source data only. For example, we can redefine addition as a+b↦a+b+3a+b\mapsto a+b+3, teach this rule using symbolic expressions, and test it using natural-language word problems. Held-out source accuracy measures whether the rule was learned; target accuracy measures whether it transfers to the different input distribution. The OOWD ratio ρ=Acc-T/Acc-S\rho=\mathrm{Acc\text{-}T}/\mathrm{Acc\text{-}S} summarizes that transfer, while the two accuracies are reported separately to expose failures of source fitting.

W2 / C2: Is this merely learning a new task after fine-tuning?

Your description of the arithmetic experiment is correct. We agree that the goal is to evaluate generalization to distributions that were not extensively represented during training. Where our interpretation differs is that learning the source examples and transferring to the target are distinct requirements. Symbolic expressions and natural-language word problems share computational structure, but their input distributions differ. Treating them as one undifferentiated new task would obscure the cross-distribution transfer being measured. Sharing an underlying task rule is also a standard feature of OOD generalization settings.

The revised Figure 1 makes this relationship explicit. In a conventional setting, a model is trained on AA and evaluated on a different distribution A′A'. With an LFM, we cannot reliably determine whether the proposed target input distribution is already covered by pretraining. OOWD therefore introduces a controlled change in the input–output rule. Starting from pretraining on AA, we train on a limited counter-world source distribution BB, then evaluate on a target distribution B′B' that uses the same counter-world rule but has a different input distribution. Relative to the task's regular-world pretraining reference and the added source distribution, respectively, the intended shifts are

\qquad p_{B'}(x)\neq p_B(x).$$ Here, pretraining and source fine-tuning together form the training stage. So we can describe OOWD as an operational approximation to conventional OOD testing through a controlled and mild extension of training. It does evaluate an adapted model, and we have made that fact explicit. However, adaptation does not remove the generalization question: models with similar source accuracy can differ sharply in target transfer. In the updated operator-overloading results, configurations with source accuracy of at least 90% yield almost identical mean source accuracy for Tiny GPT-2 and Qwen3-4B, **95.45% versus 95.23%**, but target accuracy of **46.89% versus 92.96%**. This difference cannot be explained simply by whether the new source rule was learned. We also agree that reusing pretrained knowledge is part of this process. Such reuse is compatible with OOD generalization: transfer generally depends on carrying useful structure from training to a different distribution. Our claim is therefore that OOWD exposes differences in cross-distribution rule transfer under controlled adaptation, while reducing the contamination ambiguity of ordinary benchmark testing. **C3–C6: Terminology and minor clarifications.** We have corrected and clarified the wording in the manuscript in response to these comments. By “scratch models,” we specifically mean the baselines initialized randomly and trained on our limited regular-world datasets before counter-world adaptation, as opposed to starting from broadly pretrained checkpoints. We welcome any further questions or comments and would be happy to continue the discussion. If our revisions and responses address your concerns, we would be grateful if you would consider raising your score.
Reviewer_6NuROpenReview ↗

Reviewer's response

Thanks for the response. It mostly confirmed my initial concerns. I tried to read the new version with a set of fresh eyes but the description is about as inscrutable as the first version. I put anyone to the challenge of understanding what is going on from figure 1 alone. I reiterate my suggestion to the authors to test out their paper first on colleagues who do not know about this project, and see if they can understand it correctly.

This paper is clearly below the bar for the main conference hence I have updated my score to a strong rejection.

Reviewer_TRALOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses a fundamental and timely problem: how to reliably evaluate the generalization of Large Foundation Models (LFMs) when the boundaries of their web-scale pretraining corpora are opaque and data contamination cannot be ruled out. The authors argue that conventional out-of-distribution (OOD) evaluation is ill-suited to this regime, since a nominally "shifted" benchmark may already lie within the pretraining support.

To address this, the paper proposes Out-of-World Distribution (OOWD), an OOD-like evaluation framework grounded in a world-model view of data generation. The key idea is to make explicit a latent world variable w that governs input–output semantics, characterize the pretraining boundary in terms of world space rather than observation space, then intervene on w to induce a counter-world semantics and test whether a rule learned from a source distribution transfers to a distinct target distribution. The framework is instantiated across four diverse settings — symbolic-to-verbal mathematical transfer, program-to-natural-language rule transfer, cross-style visual rule transfer, and text-to-image arithmetic transfer.

Critical strengths

Conceptually novel and well-motivated problem formulation. The reframing of the contamination problem from observation space to world space is elegant and provides a structural guarantee that purely observational counterfactual benchmarks lack. This is a genuinely fresh angle on a well-known problem.

Clean experimental design. The source/target distinction is carefully controlled, and the contrast between LFMs and small models (both fitting source data) isolates the phenomenon of cross-distribution transfer rather than mere memorization. The four transfer settings span language, code, vision, and multimodal reasoning, giving broad empirical coverage.

Mechanistic depth. The paper goes beyond a descriptive benchmark by providing interpretability evidence (layer localization, sparse path patching) that offers insight into why LFMs transfer. This elevates the contribution from "another benchmark" to a diagnostic tool.

Actionable algorithmic payoff. Deriving OOWDAlign from the exposed failure mode of scratch models demonstrates that the framework is not only a test but also an inspiration for algorithm design — a rare and valuable property for an evaluation paper.

Strong relevance and potential impact. The question of whether LFM capabilities reflect genuine generalization or interpolation over pretraining data is of broad and growing importance to the community.

Critical weaknesses

Contamination mitigation is structural, not proven. The framework reduces the likelihood of contamination but does not eliminate it. Since LFMs are trained on web-scale corpora, one cannot rule out that some counter-world conventions (e.g., base-9 arithmetic, reversed rules) have appeared online. The paper should more explicitly discuss this residual risk and whether any of the constructed counter-worlds could plausibly occur in pretraining data.

Limited discussion of the "world variable" identifiability. The notion of a latent world variable w governing input–output semantics is intuitive but under-formalized in places. Clarifying what assumptions are needed for the counter-world intervention to be well-defined (and how violations would bias conclusions) would strengthen the theoretical grounding.

Scope of small-model baselines. The small models are evaluated in only the first three of the four settings. Explaining this omission (or extending coverage) would make the LFM-vs-small-model contrast more complete.

OOWDAlign is somewhat preliminary. While promising, the algorithmic contribution appears secondary and its gains, though consistent, are comparatively modest. The paper would benefit from a more thorough analysis of when OOWDAlign helps versus when it does not.

Broader societal/robustness implications could be expanded. Given the paper's motivation in healthcare, autonomous driving, and scientific discovery, a deeper discussion of the practical limits of OOWD-style evaluation (e.g., cost, applicability to proprietary models without weight access) would be valuable.

Questions for authors

Please see the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer TRAL

Thank you for the thoughtful review and for recognizing the motivation, experimental design, and diagnostic value of OOWD. We agree that the assumptions and practical scope deserve a clearer discussion.

W1 / W2: Residual pretraining overlap and the interpretation of the world variable.

We agree that the counter-world construction does not prove that all conflicting rules or examples are absent from pretraining. Web-scale corpora may contain erroneous arithmetic, reversed conventions, or explicit discussions of alternative rules. The distinction we rely on is between the dominant regular-world conditional for the task and a distribution that systematically follows the counter-world rule, rather than between absolute presence and absence of individual examples.

Our working assumption is that regular-world answers dominate the relevant pretraining conditional for familiar tasks such as elementary arithmetic. At the population optimum, next-token prediction fits the training distribution; assuming the model approximates the relevant conditional well, its reliable regular-world performance is consistent with that assumption. This does not identify exact corpus frequencies or establish zero exposure to conflicting examples. In our constructed data, however, the conflicting rule is applied systematically. A small amount of exposure to incorrect arithmetic need not make the counter-world task distribution identical to the regular-world reference. We will make these assumptions explicit in the final version.

W3: Why is there no small-model baseline in the fourth setting?

Text-to-image arithmetic simultaneously requires visual parsing, arithmetic, and an interface between modalities. A meaningful small-model comparison would first need to establish these basic capabilities and then test counter-world transfer. Otherwise, a low target score could reflect a failure to read the image rather than a failure to transfer the learned rule. Training such a multimodal model from scratch is beyond our data and compute budget, as stated in Section 4. We therefore restrict the interpretation of this setting to text-to-image rule transfer in the evaluated VLMs and the comparison between their two sizes; it does not provide a foundation-model-versus-scratch-model comparison.

W4: When does OOWDAlign help, and when does it not?

The objective of OOWDAlign encourages the same counter-world relabeling to produce compatible classifier updates across visible domains. The associated model selector asks whether a virtual update on one visible domain reduces loss on another. Neither procedure uses the held-out evaluation domain. The existing ablations already reveal useful limits. The table below separates the alignment objective from transfer-based model selection. Values are held-out-domain accuracy percentages, averaged over three trials.

MethodPACS, R18PACS, R50OfficeHome, R18OfficeHome, R50Average
ERM + source-IID selection50.1251.3629.4129.9740.21
ERM + transfer selection50.2650.9129.4329.6740.07
OOWDAlign + source-IID selection51.7450.3129.8030.5440.59
OOWDAlign + transfer selection53.4252.3429.7031.0441.63

With source-IID selection, adding the alignment objective improves the average by 0.38 percentage points, but reduces PACS ResNet-50 accuracy from 51.36% to 50.31%. Combining the objective with transfer selection increases the average improvement over ERM to 1.42 points. The selector is not uniformly beneficial even for OOWDAlign: on OfficeHome with ResNet-18, it changes accuracy from 29.80% to 29.70%. Moreover, applying the same selector to ERM checkpoints changes the average by approximately −0.15 points, as reported from the underlying results; the displayed table entries are rounded.

This suggests that the gains do not come from a universally better checkpoint selector. They may depend on the representations and checkpoint pool produced by alignment training. The present evidence is limited to two datasets and two scratch-trained backbones. How the benefit depends on pretraining, the degree of domain shift, and regularization strength remains an important direction for further study.

W5: Practical cost, proprietary models, and robustness implications.

OOWD measures rule transfer after controlled adaptation. Success on these tasks does not directly establish reliability or safety in healthcare, autonomous driving, or scientific applications. Its applicability depends on constructing meaningful counter-world rules and validating the source–target distribution difference. Compared with inference-only benchmarks, it incurs additional costs for rule and dataset construction, source fine-tuning, and evaluation across configurations. Our LoRA experiments show that full-parameter updates are unnecessary in the tested arithmetic setting, although adaptation still requires computation.

For proprietary models, direct weight access is not necessarily required if the provider offers a suitable fine-tuning interface. An inference-only interface, however, cannot reproduce the current protocol; prompt-based adaptation would constitute a different evaluation setting and should be validated separately. Finally, the observation that an induced rule can propagate across domains suggests that post-adaptation evaluation should also examine unintended behavior outside the fine-tuning distribution. We therefore view OOWD as a controlled diagnostic that complements application-specific validation, and will expand the discussion of these practical limits.

Reviewer_TRALOpenReview ↗

Thank for the rebuttal.

The authors have carefully addressed the mentioned issues. I keep my positive score.

Reviewer_B4JFOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

The paper proposes OOWD, an evaluation protocol for testing whether large foundation models generalize beyond their pretraining data. A counter-world rule, such as a redefined arithmetic operator or a permutation of class labels, is injected by fine-tuning on one source domain. The model is then tested on a target domain with a different surface form. Across math, code, vision and text-to-image settings, pretrained LFMs transfer the rule much better than small models trained from scratch. The authors also localize the transfer to a few components and propose OOWDAlign for domain generalization.

Critical strengths
  1. Using counter-world rules is a simple and clean way to reduce contamination. The experiments also cover many settings, including math, code, vision and cross-modal transfer.
  2. The layer freezing and targeted tuning results show where the shared rules are stored in the model, which I find interesting.
Critical weaknesses
  1. My main concern is that the main finding seems close to what is already widely accepted, that pretraining builds shared representations across different input forms such as languages, code and images. The paper adds evidence that a rule injected through one form can spread through these shared representations. This is interesting, but it is a much narrower question than whether the models generalize beyond pretraining, which is what the paper sets out to answer.
  2. I also have some concerns about the details. The large and small models differ in both size and pretraining, so the gap cannot be attributed to scale alone. In most settings, the small models reach much lower source accuracy than the LFMs, except for traversal overloading (Fig. 2), although the abstract says they also fit the source domain. The formal definition does not guarantee that the counter-worlds are really outside pretraining. Finally, in Table D.8, the fully fine-tuned Llama-3.1-8B on operator overloading loses almost all target transfer when it uses chain-of-thought.
Questions for authors
  1. I agree that the target answers cannot be memorized, so the transfer is not simple interpolation. However, it seems to rely on concepts such as addition that pretraining already covers. I would like to hear how the authors see the difference between this and generalizing within the pretraining distribution.
  2. Could the authors report each model separately, including pretrained GPT-2, and compare large and small models at similar source accuracy?
  3. I am curious why the rule disappears when the fully fine-tuned model uses chain-of-thought (Table D.8). Could the authors comment on this?
  4. Although the authors have cited papers such as [1, 2], it would be better if authors could position themselves properly against these works.

[1] Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks. NAACL 2024.

[2] Exploring the hidden reasoning process of large language models by misleading them. EMNLP 2025.

5 comments
AuthorsOpenReview ↗

Response to Reviewer B4JF

Thank you for the careful review and for recognizing the value of the counter-world construction and the mechanistic analyses. Your comments help us clarify both the scope of OOWD and the interpretation of the model comparisons.

W1 / Q1: Shared representations, familiar concepts, and generalization beyond pretraining.

We agree that shared representations are a widely held explanation for learning and generalization, and that better representations acquired through pretraining may support stronger transfer. However, whether LFMs actually exhibit stronger cross-distribution generalization remains contested, as discussed in our introduction. This motivates an evaluation that can reduce the ambiguity caused by pretraining coverage. The existence of shared representations alone does not resolve this evaluation problem.

Our work does not claim to have definitively established generalization beyond the entire, unknown pretraining distribution. Indeed, the difficulty of identifying that distribution is precisely our starting point. OOWD takes a step toward this goal through a controlled extension of the model's training experience. Using the notation of the revised Figure 1:

  • Conventional OOD evaluation trains on a distribution AA and tests on a different distribution A′A'. For LFMs, it is difficult to establish whether a proposed test distribution is already covered by pretraining.
  • OOWD starts with pretraining on AA, introduces counter-world rules through training on a source distribution BB, and tests on a target distribution B′B'. The target differs from the regular-world pretraining reference in p(y∣x)p(y\mid x) and from the added source distribution in p(x)p(x), while retaining the source's counter-world rule.

Pretraining and source fine-tuning therefore jointly constitute training. The source provides the evidence needed to learn an arbitrary new rule; the target tests whether that rule transfers to an input distribution absent from counter-world fine-tuning. Through this controlled and mild extension of training, OOWD provides an operational approximation to conventional OOD testing without requiring full knowledge of pretraining coverage. We have revised the introduction and Section 3 to make this interpretation explicit.

Reusing concepts such as addition is consistent with this goal. OOD generalization concerns a difference between training and test distributions; it does not require every concept needed at test time to be absent from training. For example, recognizing that a word problem involves addition does not by itself supply the counter-world answer: the model must apply the rule learned through symbolic source examples to that different input form. Thus, our evaluation is compatible with, and complementary to, studies of generalization within the pretraining distribution.

W2 / Q2: Model size, pretraining, and comparisons at similar source accuracy.

We agree that the gap between LFMs and small models cannot be attributed to parameter count alone, because these models also differ in the amount and breadth of pretraining. However, model capacity and training data are coupled in practice (Kaplan et al., 2020; Hoffmann et al., 2022; Muennighoff et al., 2023): training a large model on a small dataset, or reproducing foundation-model pretraining for every small baseline, would introduce predictable limitations. Our main comparison therefore concerns the broad-pretraining and limited-data training regimes, rather than an isolated causal effect of model size.

Within the feasible experimental scope, the visual experiments provide a complementary control: the same ResNet or ViT architecture is evaluated with and without ImageNet pretraining. These comparisons hold parameter count fixed and show that additional pretraining improves counter-world transfer. We report the individual models below rather than relying only on the pooled regression lines.

To address insufficient source fitting, we substantially extended training steps and data for the small language model. Figure 3 has been updated with these results.

The following table compares the original and updated Tiny GPT-2 results. All values are accuracy percentages. Each entry averages all 48 configurations shown for that task.

TaskOriginal sourceUpdated sourceOriginal targetUpdated target
Digit overloading59.5274.316.6015.83
Operator overloading64.9992.193.0442.71
Set-operation overloading60.6591.773.769.76

The stronger training substantially improves source fitting, and also improves target accuracy, particularly for operator overloading. Nevertheless, a large transfer gap remains. The next table reports each language model in Figure 3 separately, including pretrained GPT-2. Each cell is source / target accuracy (%), averaged over the same three-seed configuration means. The Tiny GPT-2 row uses the updated results above; traversal results are unchanged.

ModelDigit overloadingOperator overloadingSet-operation overloadingTraversal overloading
Tiny GPT-2, trained from scratch (19.25M)74.31 / 15.8392.19 / 42.7191.77 / 9.76100.00 / 12.10
Pretrained GPT-2 (124M)97.28 / 95.7297.63 / 97.1197.81 / 58.27100.00 / 90.21
Qwen3-4B92.38 / 81.4695.23 / 92.96100.00 / 92.9599.95 / 99.87
Qwen3-8B95.28 / 85.7696.48 / 95.7099.99 / 92.1899.99 / 99.48

For a more direct comparison at similar source accuracy, we additionally restrict the operator-overloading configurations to those with three-seed mean source accuracy of at least 90%. The 11 retained Tiny GPT-2 configurations average 95.45% source and 46.89% target accuracy, whereas the 16 Qwen3-4B configurations average 95.23% source and 92.96% target accuracy. Thus, almost identical mean source accuracy is accompanied by a 46.07-percentage-point target gap. For set-operation overloading, the eight qualifying Tiny GPT-2 configurations average 95.92% / 9.71%, compared with 97.81% / 58.27% for pretrained GPT-2. Traversal provides an even closer comparison: both Tiny GPT-2 and pretrained GPT-2 achieve 100.00% source accuracy, but their target accuracies are 12.10% and 90.21%, respectively. We acknowledge that source fitting remains weaker for digit overloading despite the additional training, and think greater task difficulty is a plausible explanation.

The individual visual models in Figure 3 are reported below using the same source / target convention.

Model and initializationPACSOfficeHome
ResNet-18, scratch82.80 / 52.1156.01 / 26.87
ResNet-18, ImageNet pretrained93.57 / 59.6270.59 / 45.65
ResNet-50, scratch80.62 / 44.8353.50 / 23.47
ResNet-50, ImageNet pretrained94.22 / 63.7374.13 / 52.27
ViT-S/16, scratch71.07 / 42.0142.35 / 19.25
ViT-S/16, ImageNet pretrained93.55 / 74.0775.52 / 55.59
ViT-B/16, scratch66.98 / 37.8440.72 / 16.92
ViT-B/16, ImageNet pretrained79.83 / 60.1757.26 / 41.09
Qwen3-VL-2B98.78 / 94.9690.42 / 85.28
Qwen3-VL-8B98.94 / 94.6392.00 / 87.36

For PACS and OfficeHome, the datasets are fixed, and our additional hyperparameter exploration did not yield higher small-model source accuracy, so these results are unchanged.

W2: Can counter-world rules already occur in pretraining?

We agree that the formal construction does not guarantee that every counter-world example is absent from pretraining. A web-scale corpus may contain incorrect arithmetic or examples of alternative conventions. Our argument concerns a systematic difference in the conditional rule, rather than the absence of every individual example.

The underlying assumption is that, for familiar tasks such as elementary arithmetic, regular-world answers dominate the relevant pretraining conditional. At the population optimum, next-token prediction fits the training distribution; if a model approximates that conditional well, its reliable regular-world behavior is consistent with this assumption. In contrast, our counter-world data systematically follow a deliberately conflicting rule. Occasional conflicting examples in pretraining therefore do not by themselves make these two task distributions identical.

We will further clarify the assumptions and residual contamination risk in the final version. The supported conclusion is strong source-to-target transfer under a controlled rule change, not a proof that the target has zero probability under the entire pretraining corpus.

W2 / Q3: Why does full fine-tuning fail under chain-of-thought prompting?

Our interpretation is that full-parameter fine-tuning on direct-answer supervision disrupts the model's ability to follow a chain-of-thought output format. The fine-tuning examples explicitly require the final answer without intermediate reasoning, so the CoT evaluation also changes the response format relative to adaptation. The targeted-adaptation results in Table D.8 support this explanation. Updating the selected attention heads preserves general capabilities much better and retains the benefit of CoT prompting:

Adaptation of Llama-3.1-8BSourceTarget, direct answerTarget, CoTMMLUCEVAL
Full-parameter fine-tuning99.7598.122.3560.5047.18
Top four attention heads99.2582.1988.6268.8654.90
Top eight attention heads99.1293.5097.0068.4554.16

Q4: Relationship to the two cited counterfactual-evaluation studies.

In Reasoning or Reciting?, the counterfactual condition is supplied through the prompt, for example by asking for addition in base 11. The paper explicitly states that its counterfactual conditions need not be absent from pretraining. Our distinction is that, when the base instruction is part of the full input xx, a correct base-11 answer remains compatible with ordinary instruction-conditioned arithmetic; the prompt changes the task specification without necessarily contradicting the regular-world conditional for that full input. Such tests can probe uncommon task conditions, but do not by themselves establish separation from pretraining. OOWD instead introduces a conflicting rule through source training and measures its transfer to a different input distribution.

We do build on the fine-tuning intervention and dataset-construction ideas in Exploring the Hidden Reasoning Process of Large Language Models by Misleading Them. That work studies the hypothesis that LLMs abstract task structure before reasoning by testing the transfer of deliberately incorrect rules. Our contribution develops this intervention into a more general OOD evaluation framework: we formalize the distinct shifts relative to pretraining and source adaptation, systematically compare foundation models with limited-data baselines, extend evaluation settings, and add component-level causal interventions and targeted adaptation. OOWDAlign further translates the evaluation principle into an algorithm for domain generalization.

We welcome any further questions or comments and would be happy to continue the discussion. If our revisions and responses address your concerns, we would be grateful if you would consider raising your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

PDF update

We thank all three reviewers for your careful feedback. We have updated the manuscript PDF, with revisions highlighted in blue. The main changes include a new overview figure and clearer introduction to the OOWD protocol, a revised formalization in Section 3, and stronger small-model baselines in Figure 3 with updated training details. We have also corrected and clarified wording throughout the relevant sections in response to your comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.