acceptodds
Under review as a conference paper at ICLR 2027

Generalization Dynamics of LM Pre-training

Abstract

People typically assume that LMs stably mature from pattern-matching parrots to generalizable intelligence during pre-training. We build a toy eval suite and show this mental model is wrong: throughout pre-training, LMs frequently and suddenly hop between parrot-like and intelligence-like computations. We call this mode-hopping. Across our suite, LMs suddenly latch onto memorized or in-context patterns instead of in-context learning, use System 1 instead of System 2 thinking, pick up what sounds true instead of what is true, fail at multi-hop persona QA, out-of-context reasoning, and emergent misalignment — then just as suddenly revert and generalize. Mode-hopping is not explained by standard optimization dynamics: it is locally stable and cannot be fixed by checkpoint averaging. We instead think of it as a capacity allocation problem: in a capacity-bounded model, generalizable circuits must compete with the shallow ones learned early in training, and the data in each pre-training window decides which circuits win. Our suite provides a new efficient lens on generalization. We demonstrate two concrete applications: (i) select intermediate pre-training checkpoints that strongly generalize reasoning and alignment, better than the final pre- or mid-training checkpoints, and (ii) select pre-training data that controls and stabilizes generalization dynamics.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
generalization, pre-training, alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XNosOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds a toy behavioral eval suite that tells pattern-matching "parrots" apart from generalizing intelligence, and uses it to track the intermediate checkpoints of OLMo3 and Apertus. The authors find that models frequently and abruptly switch between pattern-matching and generalization throughout pre-training (mode-hopping), even while loss and standard benchmarks improve smoothly. They argue that this is not caused by eval noise, metric choice, or standard optimization dynamics, and attribute it to capacity allocation between generalizable and shallow circuits. Larger models mitigate mode-hopping but do not eliminate it. The authors also show two applications: selecting intermediate checkpoints that generalize better after post-training, and selecting pre-training data to stabilize generalization dynamics.

Critical strengths
  1. Extensive experiments. The eval suite covers six behavioral evals plus two fine-tuning-based tests, spanning many datasets. It is applied to dense intermediate checkpoints of two fully open model families at four scales (OLMo3 7B/32B, Apertus 8B/70B), with results averaged over 4 seeds. The authors also carefully rule out several alternative explanations: eval noise (stable curves on 10 standard benchmarks), metric artifacts (soft probabilities and a FLOPs x-axis), and optimization dynamics (single-step local stability tests and checkpoint averaging). Together, these make the core phenomenon convincing.
  2. Novel perspective. The paper challenges the common assumption that generalization improves steadily during pre-training. It shows that generalization behavior can switch abruptly between modes even after trillions of tokens and far beyond Chinchilla-optimal budgets, while loss and standard benchmarks improve smoothly. This adds a dimension that loss-based scaling analyses and downstream benchmarks do not capture.
  3. Practical relevance of checkpoint selection (Application 1). The authors show that an intermediate checkpoint selected with the cheap toy suite (4.5T) beats the final pre- and mid-training checkpoints after post-training, both in math SFT → GPQA transfer (36.3% vs. 29.8%) and in robustness to prefilling attacks (53% vs. 21%). This suggests that the common practice of post-training only the final checkpoint may be suboptimal, and that a lightweight behavioral probe could guide checkpoint choice in practice.
Critical weaknesses
  1. Limited size and diversity of some evals. Several evals have few test examples or little variety. Truthy Answer has fewer than 300 claims per subset. Each Intuitive Answer subset consists of 200 templated variants of a single CRT problem. Multi-hop Persona QA has only 5 test questions per persona. With such narrow item pools, part of the observed oscillation may reflect sensitivity to a few specific items or templates rather than a general shift in generalization behavior. Given the overall scale of the experiments, this is understandable, but the conclusions drawn from these evals should be stated more carefully.
  2. Section 5.2 is underspecified. The paper only states that subsets are selected "based on whether they enhance pattern-matching or generalization in the original OLMo3 pre-training run", without describing the actual procedure. It is unclear how a subset is labeled as pattern-encouraging or generalization-encouraging (e.g., by the accuracy change across a checkpoint window, or by document-level scoring), which windows are used, and what training hyperparameters are used. If the data are labeled by how target-eval accuracy changed after training on them in the original run, the result is partly circular. In addition, the experiment covers a single eval (answer+1) and only about 6.7B tokens, with no check on other evals or general capabilities. As it stands, this section is a feasibility demonstration rather than a usable data-selection method.
  3. Mode-hopping may not be a single, global phenomenon. Figure 12 shows low cross-dataset correlations within the same eval (about 0.15–0.50). This means that a checkpoint that generalizes on one dataset often fails on another. This raises the question of whether mode-hopping reflects a global "generalization prior", as the paper frames it, or a collection of largely independent task-specific fluctuations. It also weakens the case for using the suite as a general checkpoint selector.
Questions for authors
  1. Is the math SFT in Section 5.1 trained long enough? The authors follow the practice of Ren et al. (2026) [1], which shows that reasoning SFT can generalize as well as RL under multi-epoch training. Ren et al. also report a dip-and-recovery pattern: out-of-domain performance first drops early in SFT, then recovers and improves with continued training up to 8 epochs. In Figure 13(a), GPQA accuracy of the 4.9T checkpoint keeps declining over the roughly 200 post-training steps shown, which looks like it could be the "dip" phase. Could the authors report the number of epochs and total training steps, and say whether training was run to convergence? Specifically, if SFT continues further, does the 4.9T checkpoint (or the final pre-/mid-training checkpoints) recover on GPQA, and does the gap with the 4.5T checkpoint remain? If the gap shrinks after full training, the advantage of the selected checkpoint would be a difference in recovery speed rather than in final generalization, which would substantially change the conclusion of Application 1.

[1] Qihan Ren, Peng Wang, Ruikun Cai, Shuai Shao, Dadi Guo, et al. Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability. arXiv preprint arXiv:2604.06628, 2026.

3 comments

1. Part of the observed oscillation may reflect sensitivity to a few specific items or templates rather than a general shift in generalization behavior.

We observe mode-hopping across our eval suite, which in total consists of tens of thousands of test examples covering diverse distributions.

Each individual test set has hundreds to 1K examples. This is a common size for test sets. For example, GPQA, a widely adopted benchmark, has ~500 test examples. Multi-hop persona QA is indeed an outlier with only 5 test questions. We sample hundreds of rollouts per question to stabilize the results.

We would also like to clarify that we never optimize our test set against model behaviors to inflate mode-hopping. All test sets are built before testing models on them.

2. Section 5.2 is underspecified. It is unclear how a subset is labeled as pattern-encouraging or generalization-encouraging (e.g., by the accuracy change across a checkpoint window, or by document-level scoring)

Yes it’s based on the accuracy change across a checkpoint window.

3. In addition, the experiment covers a single eval (answer+1) and only about 6.7B tokens, with no check on other evals or general capabilities. As it stands, this section is a feasibility demonstration rather than a usable data-selection method.

Our toy experiments in Sec 5.2 aims to show that pre-training generalization dynamics can be controlled by controlling data. This experiment is quite expensive, since we need to run continued pre-training on OLMo3 32B (a dense model). We leave the exploration of controlling pre-training dynamics on more diverse, challenging tasks to future work.

4. Is the math SFT in Section 5.1 trained long enough?

There are our training hyperparameters:

Epochs8
Learning rate5e-5, cosine scheduler
LoRAr = 64, α = 128
Batch size256

The step count may look small because of our large batch size. In preliminary runs with 20 epochs, training converged by epoch 8.

The authors have responded to your reviews. Please read their response and reply. Please specify which concerns are resolved and which remain. Please update your review and score if the assessment has changed.

AC

Thank you for the clarifying response and the additional experiment. The authors have addressed most of my concerns, while I believe the test sets like Multi-hop persona QA should be augmented beyond the five provided examples for its credibility. Accordingly, I maintain my weak accept assessment.

Reviewer_uTC9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to study how LMs' generalization ability evolves during pre-training. Most evaluations, however, track different aspects of ICL performance across pre-training checkpoints, without a clear generalization setting relative to pre-training data. The tasks include arithmetic, factual reasoning, and flipped-label classification (eg sentiment analysis), often with ambiguous demonstrations, and the authors find that performance can fluctuate sharply even when pre-training loss and standard benchmark performance remain stable. They call this phenomenon "mode-hopping" and hypothesize that shallow pattern-matching circuits and "generalizable" circuits compete for limited model capacity.

Critical strengths
  1. Section 3 is clearly written, with intuitive and nice figures that make the observed performance fluctuations easy to follow.
  2. The experiments cover different models, scales, and tasks, with additional analyses and ablations.
  3. The paper also explores potential applications of the observed phenomenon to checkpoint and pre-training data selection.
Critical weaknesses
  1. Problem/motivation and eval are mismatched. The paper is motivated by studying pre-training generalization, but most evaluations just measure different aspects of ICL performance across checkpoints. There is no clear ID/OOD relative to pre-training. "Train Examples" in Table 1 are actually in-context demonstrations (misleading term).
  2. Ambiguous ICL tasks. In Repetitive Answer and Successive Answer, both hypotheses (a) solving the task and (b) continuing the answer pattern fit all demonstrations. The prompt never clarifies which one to follow. Especially for a next-token pre-trained LM, continuing the pattern is perfectly natural, while solving arithmetic is not explicitly required by the training objective.
  3. The authors try to explain mode-hopping as competition between shallow and "generalizable" circuits, but never identify these circuits or provide causal evidence for such switching.
  4. Many of these phenomena have already been studied. eg Flipped Answer is almost identical to the flipped-label ICL setting in Larger Language Models Do In-Context Learning Differently 2023. The main difference seems to be plotting across checkpoints.
Questions for authors
  • Could the authors discuss the novelty of each evaluation relative to existing literature?
  • I suggest reframing the paper to focus on ICL and reasoning performance across pre-training checkpoints, rather than pre-training generalization.
  • The paper covers a lot of ground. Section 3 is easy to follow, but Sections 4 and 5 feel rather rushed. In particular, I struggled to understand how Section 4 supports the proposed two-circuit competition hypothesis, and some experimental settings in Section 4&5 are not clearly explained. I suggest expanding the discussion on WHY THIS HAPPENS (Section 4) and what evidence supports the proposed explanation (eg mechanistic interpretability or targeted ablations). For a paper focused on observing a phenomenon, given the space limit, I believe understanding why it happens is always more valuable than exploring potential applications.

A note on my rating: I don't like the 4-point scale. I gave a reject because I think substantial revisions are needed, but I don't think this is a 1-point paper. I appreciate the effort that went into it and would be happy to raise my score if my concerns are addressed.

8 comments

1. The paper is motivated by studying pre-training generalization…but there is no clear ID/OOD relative to pre-training.

In our paper, generalization means intelligence-like computation, not performance under distribution shift. The latter is arguably impossible to study cleanly for pre-trained LMs, since we cannot fully characterize their training distribution.

2. Ambiguous tasks. In Repetitive Answer and Successive Answer, both hypotheses (a) solving the task and (b) continuing the answer pattern fit all demonstrations. The prompt never clarifies which one to follow.

Ambiguity is central to our eval suite by design: when multiple computations fit the context, which one does the LM use? how does this change during pre-training?

3. Especially for a next-token pre-trained LM, continuing the pattern is perfectly natural, while solving arithmetic is not explicitly required by the training objective.

This is true for LMs pretrained on random data but not web data, which reflects the distribution of human-defined truth. As an extreme case, consider the prompt “1+1=” with no demonstrations. Infinitely many answers are plausible, since we never specify the task. Still, we expect a next-token pre-trained LM to answer “2”.

Our results in Sec 3 also show that a next-token pre-trained LM clearly prefers generalizable circuits over shallow ones, even if we never clarify the task. More pre-training tokens and more parameters often strengthen this preference (i.e. higher accuracy on our evals), which aligns better with human-defined truth.

4. Many of these phenomena have already been studied. The main difference seems to be plotting across checkpoints. Could the authors discuss the novelty of each evaluation relative to existing literature?

Yes our main contribution is tracking pre-training dynamics and find the counterintuitive mode-hopping phenomenon. We never emphasize the novelty of each eval. And we cite prior work wherever a related eval exists.

5. I suggest reframing the paper to focus on ICL and reasoning performance across pre-training checkpoints, rather than pre-training generalization.

We frame the paper around generalization because our evals target which computation an LM uses, not just how well it performs on some random tasks.

As a side note, we run both ICL and fine-tuning experiments. Fine-tuning is more realistic, but we like ICL because 1) it’s very efficient and 2) it’s a common way to evaluate base models.

6. In particular, I struggled to understand how Section 4 supports the proposed two-circuit competition hypothesis.

Section 4 does not test the two-circuit hypothesis directly. It rules out null hypotheses that cannot explain mode-hopping.

7. I suggest expanding the discussion on WHY THIS HAPPENS (Section 4) and what evidence supports the proposed explanation (eg mechanistic interpretability or targeted ablations).

As stated at the beginning of our intro, this is not a mech interp paper. "The distinction between parrots and intelligence is computational…This distinction, however, can be probed behaviorally"

We think this is a strength, not a weakness: our paper finds a simple yet effective lens to probe pre-training dynamics, without being bottlenecked by the slow progress of mech interp.

We shared the reviewer’s concern when writing the conjecture about capacity competition among internal components. Existing interp tools do not let us run solid mechanistic analysis on LLMs; inventing new tools is beyond the scope of our paper.

So why did we still include this conjecture? Besides being intuitively appealing, it’s supported by three pieces of empirical evidence: Multiple prior papers observe competing internal components in LMs, mainly about knowledge recall/storing ([1][2][3][4]). We observe that larger models suffer less from mode-hopping. We observe preliminary interp results on the competition between copying and task learning in our suite.

Given the reviewer’s interests, we present result #3 below and will add it to the appendix.

Mechanistic analysis on the competition between copying and task learning

Our repetitive answer eval tests whether LMs copy in-context patterns or learn in-context tasks. For example, all 8 demonstrations in the context answer "10", but the query has a different answer. In OLMo3 7B, we find two types of attention heads, one for each computation. Copy heads read and copy the repeated answers. Task heads read the query and help produce the correct answer.

1. How we find the heads

We score each head in two ways and keep the top 20 heads per score.

  • Copy score: we block the head's attention to the in-context answers and measure how much logit(correct) − logit(10) rises.
  • Task score: we zero the head's output and measure how much logit(correct) − logit(10) falls.

Under direct logit attribution, copy heads write 90% of logit(10), and task heads write 82% of logit(correct).

2. The two components compete

Amplifying copy heads suppresses logit(correct), even more than it raises logit(10). Amplifying task heads mainly raises logit(correct).

InterventionΔ logit(correct)Δ logit(10)
Amplifying copy head outputs by 1.3−0.67+0.34
Amplifying task head outputs by 1.3+0.58−0.02
Amplifying random head outputs by 1.3+0.01−0.07

3. Reliance on each component tracks mode-hopping

A layer's attention output o is the sum of its heads' outputs hᵢ. We define a head's share as hᵢ · o / ‖o‖². The copy − task share is the total share of our copy heads minus that of our task heads.

At the 8 checkpoints below, the copy − task share is consistently higher at dips than at peaks. Across 71 checkpoints, it correlates with logit(correct) − logit(10) at r = −0.93.

CheckpointModeAccuracyCopy head share − task head share
349kPeak0.474−1.40
380kDip0.031−0.24
410kPeak0.624−1.26
445kDip0.023−0.35
494kPeak0.476−1.00
542kDip0.094−0.89
600kPeak0.490−1.12
660kDip0.033−0.08

Removing the copy heads eliminates mode-hopping: accuracy stays above 0.95 at every checkpoint.

CheckpointAccuracy, originalAccuracy, copy heads removed
350k0.2840.952
380k0.0320.988
410k0.6080.988
411k0.4600.984
490k0.3160.988
495k0.3000.956
540k0.4640.956

[1] Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals. ACL2024

[2] Copy Suppression. ACL2024

[3] Characterizing Mechanisms for Factual Recall in Language Models. EMNLP2023

[4] Data Mixing Can Induce Phase Transitions in Knowledge Acquisition. NeurIPS2025

The authors have responded to your reviews. Please read their response and reply. Please specify which concerns are resolved and which remain. Please update your review and score if the assessment has changed.

AC

Reviewer_brrnOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3(+1)
Soundness
3
Read review
Paper summary

The paper suggests the novel concept, Mode Hopping, which is described by the oscillation between parrot like, shallow pattern matching answer and intelligent, functionally generalized answer during LLM pre-training stages. By evaluating various checkpoints of open source LMs with well-established diagnostic suites for isolating superficial acting from functional context execution, the authors flip the traditional idea that LLMs acquire high-level reasoning gradually and steadily during pre-training. They assume that the phenomenon came from a capacity competition among internal circuits, providing empirical evidence that mid-training checkpoints can yield better performance than the final checkpoint during post-training. Furthermore, their application shows the possibility of data filtering which can help stabilize generalization of learning dynamics during pre-training.

Critical strengths
  • The concept of Mode Hopping has high novelty supported by rigorous experimental design. By combining diagnostic probes with null-hypothesis testing across targeted and controlled evaluation suites, the study successfully rules out potential confounding factors.
  • The empirical proof that generalization does not increase monotonically, and that intermediate checkpoints can outperform the final checkpoint, reverses conventional thought of pre-training interestingly.
  • The motivation and introduction is well written.
  • Demonstration of high-frequency diagnostic probing which guide intermediate checkpoint selection can offer practical insight for efficient and effective real-world LLM training.
Critical weaknesses
  • I think the biggest missing point of the paper is the lack of mechanistic verification for the interpretable insight. While attributing Mode Hopping to capacity competition among internal components, such as Induction Heads, Successor Heads and Function Vectors, is appealing, the paper provides no internal interpretability experiments like activation patching [1] and circuit analysis [2]. So, the claim remains circumstantial rather than clearly proven.
  • Sec 5.1 of the post training evaluation seems a little bit limited. They bypassed RL in favor of SFT based solely on a single prior work [3], which leaves unexamined effect how generalized checkpoints respond to RL reward dynamics. Additionally, the training details for reproducibility such as lr and optimization hyperparameters are quite omitted.
  • Sec 5.2 presents data filtering as a potential solution for Mode Hopping based on simplified experiments, but I think they should also report possible trade-offs. Unlike the naturally emergent generalization in Sec 5.1 (where mid-training checkpoint with Math SFT successfully transfers to GPQA), forcible generalization through data filtering might risk some side effects, including degraded factual recall, overfitting to diagnostic probes and lost of diversity.

[1] Todd, E., et al., "Function Vectors in Large Language Models"

[2] Olsson, C., et al., "In-context Learning and Induction Heads"

[3] Ren Q., et al., "Rethinking generalization in reasoning SFT: A conditional analysis on optimization, data, and model capability"

Questions for authors
  1. How does Mode Hopping manifest in MoE (Mixture of Experts) architecture, where conditional routing dynamics may differ from dense models? Do the authors offer any insights or preliminary experimental results regarding this?
  2. Could the authors explicitly define the scope of "mid-training"? I think it is a little bit ambiguous as clarification is needed on whether it simply refers to arbitrary pre-final checkpoints or denotes a distinct, formally bounded training regime.
4 comments

1. While attributing Mode Hopping to capacity competition among internal components is appealing, the paper provides no interp experiments.

Yes. As stated at the beginning of our intro, this is not a mech interp paper. "The distinction between parrots and intelligence is computational…This distinction, however, can be probed behaviorally"

We think this is a strength, not a weakness: our paper finds a simple yet effective lens to probe pre-training dynamics, without being bottlenecked by the slow progress of mech interp.

We shared the reviewer’s concern when writing the conjecture about capacity competition among internal components. Existing interp tools do not let us run solid mechanistic analysis on LLMs; inventing new tools is beyond the scope of our paper.

So why did we still include this conjecture? Besides being intuitively appealing, it’s supported by three pieces of empirical evidence: Multiple prior papers observe competing internal components in LMs, mainly about knowledge recall/storing ([1][2][3][4]). We observe that larger models suffer less from mode-hopping. We observe preliminary interp results on the competition between copying and task learning in our suite.

Given the reviewer’s interests, we present result #3 below and will add it to the appendix.

Mechanistic analysis on the competition between copying and task learning

Our repetitive answer eval tests whether LMs copy in-context patterns or learn in-context tasks. For example, all 8 demonstrations in the context answer "10", but the query has a different answer. In OLMo3 7B, we find two types of attention heads, one for each computation. Copy heads read and copy the repeated answers. Task heads read the query and help produce the correct answer.

1. How we find the heads

We score each head in two ways and keep the top 20 heads per score.

  • Copy score: we block the head's attention to the in-context answers and measure how much logit(correct) − logit(10) rises.
  • Task score: we zero the head's output and measure how much logit(correct) − logit(10) falls.

Under direct logit attribution, copy heads write 90% of logit(10), and task heads write 82% of logit(correct).

2. The two components compete

Amplifying copy heads suppresses logit(correct), even more than it raises logit(10). Amplifying task heads mainly raises logit(correct).

InterventionΔ logit(correct)Δ logit(10)
Amplifying copy head outputs by 1.3−0.67+0.34
Amplifying task head outputs by 1.3+0.58−0.02
Amplifying random head outputs by 1.3+0.01−0.07

3. Reliance on each component tracks mode-hopping

A layer's attention output o is the sum of its heads' outputs hᵢ. We define a head's share as hᵢ · o / ‖o‖². The copy − task share is the total share of our copy heads minus that of our task heads.

At the 8 checkpoints below, the copy − task share is consistently higher at dips than at peaks. Across 71 checkpoints, it correlates with logit(correct) − logit(10) at r = −0.93.

CheckpointModeAccuracyCopy head share − task head share
349kPeak0.474−1.40
380kDip0.031−0.24
410kPeak0.624−1.26
445kDip0.023−0.35
494kPeak0.476−1.00
542kDip0.094−0.89
600kPeak0.490−1.12
660kDip0.033−0.08

Removing the copy heads eliminates mode-hopping: accuracy stays above 0.95 at every checkpoint.

CheckpointAccuracy, originalAccuracy, copy heads removed
350k0.2840.952
380k0.0320.988
410k0.6080.988
411k0.4600.984
490k0.3160.988
495k0.3000.956
540k0.4640.956

[1] Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals. ACL2024

[2] Copy Suppression. ACL2024

[3] Characterizing Mechanisms for Factual Recall in Language Models. EMNLP2023

[4] Data Mixing Can Induce Phase Transitions in Knowledge Acquisition. NeurIPS2025

2. Sec 5.1 runs SFT instead of RL. Miss training details.

We indeed tried RL first. Since we focus on pre-training generalization dynamics, we ran RL-zero directly on pre-training checkpoints. We skipped SFT before RL, since it would then be unclear whether our SFT data or the checkpoint drives generalization after RL.

We first verified our RL-zero infra on Qwen models, which have been mid-trained on lots of math data. We saw clean RL dynamics: reward increased while CoT length decreased.

However, RL-zero was hard to run on OLMo’s intermediate pre-training checkpoints, which have not been mid-trained. Reward increased slowly while CoT length grew quickly, which made RL very slow.

We hence ran SFT following [1] We indeed replicated their finding that long SFT unlocks generalization. Here are our training hyperparameters:

Epochs8
Learning rate5e-5, cosine scheduler
LoRAr = 64, α = 128
Batch size256

[1] Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability. Arxiv 2026

3. Sec 5.2 should report possible trade-offs.

Good point! Our toy experiments in Sec 5.2 mainly aim to show that pre-training generalization dynamics can be controlled through data. Future work that runs this at production scale should check whether it introduces unintended biases.

4. How does Mode Hopping manifest in MoE (Mixture of Experts) architecture, where conditional routing dynamics may differ from dense models?

Good question! We would love to run this analysis once MoE models release intermediate checkpoints. Marin might do this :)

5. Could the authors explicitly define the scope of "mid-training"?

We follow the definition of mid-training in the OLMo3 tech report (https://arxiv.org/abs/2512.13961), which is a clearly separate stage in their training pipelines.

Reviewer_brrnOpenReview ↗

Official Comment by Reviewer brrn

Thank you for the clarifying response and the additional experiment. The authors have addressed most of my concerns, and strengthened their credibility by filling in missing details.

While I acknowledge that this is not interpretability research, I think invoking interpretability concepts to explain the model's behavior during training requires empirical validation rather than relying solely on concepts from previous research. The newly added experiment substantiates the authors' claims.

Regarding these points, which are not just interpretability result, but also the reason for why SFT was necessary instead of RL, I believe the revised manuscript will further improve the overall completeness and clarity of the paper.

Assuming these revisions are incorporated, I will raise my presentation score and continue to support the acceptance of this paper. While a brief discussion on trade-offs would have been a nice addition, I look forward to seeing these explored in future work.

thank you for your great feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.