acceptodds
Under review as a conference paper at ICLR 2027

Measuring Damage, Not Learning: Step-0 Baselines for Continued Pretraining

Abstract

Data-recipe comparisons in continued pretraining share one design: continue from one checkpoint under competing rules, rank by held-out loss. Our case is latent-augmented pretraining, which inserts model-generated reasoning into training text uniformly: could a per-chunk signal place it better? We ran that comparison carefully — twelve conditions, up to three seeds, two domains, two model scales, a frozen truncation-safe metric — and obtained a clean, seed-stable ordering: uniform beat an equal-budget random control by nats/token, and demand-based routing lost to that control by . Then we scored the warmstart itself. Every condition, uniform included, is worse than the checkpoint it started from, so the ordering ranks damage, not learning. A tenfold lower learning rate removes of the headline contrast; with a warmstart that privileges no condition by density, both signs survive at – of their magnitude. Only replay plus a low rate puts runs back below their start, and there uniform loses its lead: random routing is ahead of it, significantly only when uniform alone matches the warmstart density, and no signal beats random. Across three public replications that share no code with ours ( runs), the learning-rate and two-axis step-0 checks reproduce every time; weight displacement, which tracked our ranking at , does not generalise. The mechanism, distribution shift and learning-rate annealing, is known (Wang et al., 2025); what is new is a documented case in which a ranking stable across seeds, domains and scales was a property of the recipe rather than of the allocations. Separately, length-corrected reasoning demand, a posterior-to-prior KL from a fixed model, predicts per-chunk latent benefit (; incumbent signal ). Of twenty-five published continued-pretraining data-recipe comparisons, most report the start, but loss-ranked ones more often as a figure point than as a number to difference against. Step 0 on two axes and a rate check cost an evaluation each; we argue they belong in the standard protocol.

Primary area:
foundation or frontier models, including LLMs
Keywords:
continued pretraining, data allocation, experimental methodology, latent-augmented pretraining, step-0 baseline, learning-rate sensitivity, empirical rigor, language model pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QmZmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continued-pretraining comparisons for latent-augmented reasoning. It first proposes reasoning demand, a posterior-to-prior KL signal that predicts per-chunk latent benefit, but finds that routing latents by this signal does not improve training over random allocation. More importantly, the authors show that their initially strong and seed-stable allocation ranking occurred entirely above the starting checkpoint’s loss: the comparison was ranking relative damage rather than learning. After correcting warmstart-density and learning-rate confounds, the effects shrink substantially, and when replay plus a lower learning rate finally yields improvement over step 0, the original uniform-routing winner no longer leads.

Critical strengths
  1. The paper is unusually careful about empirical validity: it revises several initially attractive conclusions after additional seeds and controls, distinguishes evaluation-set from training-seed uncertainty, and includes clean-room replications.
  2. The step-0 counterexample is useful: a relative ordering can remain stable across seeds, domains, and model scale while every method is worse than the checkpoint it started from.
Critical weaknesses
  1. Limited novelty. The underlying mechanisms—continued-pretraining degradation after re-warming, sensitivity to learning rate, distribution shift, replay as a remedy, and instability of data-recipe rankings across hyperparameters—are already acknowledged by the paper as established in prior work. The main methodological recommendation, explicitly comparing each recipe against its starting checkpoint, is sensible but not particularly surprising; the paper’s own survey also finds that 24/25 sampled CPT papers already report the starting checkpoint in some form.
  2. The substantive latent-allocation contribution is weak. Reasoning demand predicts per-chunk latent benefit, but this does not translate into a useful routing rule. Under the only regime where continued pretraining actually improves over step 0, no tested signal beats random routing, and the remaining allocation differences are extremely small. Thus the paper provides a careful negative result rather than a compelling new training method.
  3. “Damage” is metric-specific. The central claim is established primarily through held-out suffix NLL; downstream GSM8K/MATH performance is near floor and is explicitly not used as evidence. Thus the results demonstrate degradation on the evaluated language-modeling axis, not necessarily broader capability damage.
Questions for authors
  1. What do the authors view as the key conceptual novelty beyond prior work on CPT stability gaps, learning-rate sensitivity, replay, and recipe-ranking instability?
  2. How broadly should the term damage be interpreted given that the main evidence is held-out NLL and downstream task accuracy is not informative in this training regime?
  3. Minor presentation issue: what do the undefined terms “ThinkGap” and “BET 1/BET 3” refer to? They appear to be internal or previous terminology and should be clarified or removed.
Reviewer_iGQ9OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper first proposes 'Reasoning Demand', and 'Exceed Demand' signals to measure how latent thought would change model's reasoning and how it compares with length controlled average. The paper shows that 'Exceed Demand' signal could predict per-trunk benefits well on a frozen model, but it fails to transfer to improvement in training when the signal is being used to select when to allocate the latent budget. The authors debug this and find that all of the baselines degrade from the warmstart checkpoint. After carefully adjusting their training recipe, such as adjusting the learning rate and incorporating replay, the results improve with each fix but their proposed method still underperform the random baseline. Given these bitter lessons, the authors advocate for more rigorous evaluation of continued pretraining, including reporting step-0 checkpoint and checking learning rate.

Critical strengths
  1. The proposed signals are potentially useful, although they do not transfer to training improvement.
  2. The bitter lesson from the authors and the advocations for more rigorous evaluation, such as reporting the step-0 baseline, are valuable to the community.
Critical weaknesses
  1. The paper is poorly written and presented. The narrative repeatedly moves among the initial results, the revised settings, and keeps on narrowing down and withdrawing the previous interpretations. The paper includes term such as 'phase-4' that are used repeatedly but never defined. The paper also frequently refer to json files(Line 165, 240, 356, 365) in the main text, which is not a common practice in manuscript. The way this paper is presented, especially the trial-and-error formation and words like 'phase-4', makes it more of an internal experiment log rather than an academic publication with clear contributions.

  2. Despite rounds of debugging and fixes, the signal's relevance to training utility remains insufficiently established as it still underperforms the random baseline, and the paper only provides limited evidence explaining why the predictive signal cannot transfer to training utility.

Overall, although I do appreciate the sharing of negative results and how the authors are keep trying and debugging their recipe, the way this manuscript is presented is still at distance with a paper that is suitable for publication at ICLR.

Questions for authors

Unfortunately, due to the current quality, formation and presentation of this paper, no question could materially affect my assessment about this paper.

Reviewer_oN9zOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper is about adding model-written "thoughts" into pretraining text, as in BoLT. The authors propose a score for each chunk of text to decide which chunks should get a thought. The score does predict which chunks benefit (ρ = 0.48), but picking chunks by it works worse than picking at random. Then they find that every model they trained ends up worse than the model they started from. So the paper turns into an argument that people should always report the starting model and check the learning rate. All experiments take TinyLlama-1.1B (plus a few other small models) and train it a little more, 600 steps or about 39M tokens, on math and web text.

Critical strengths
  1. Adding model-generated reasoning to pretraining data is an interesting topic. Since generating these thoughts is costly, it makes sense to ask which parts of the text actually need them.

  2. The authors run many checks (several seeds, two domains, two model sizes, and replications on public models) and release code and result files for every reported number.

Critical weaknesses

W1. The experiments are too small to support a general claim about continued pretraining. The paper recommends its checks for all continued-pretraining studies, but all results come from 2023 models (TinyLlama-1.1B, Sheared-LLaMA-2.7B, Pythia), and no recent model such as Qwen, Llama-3 or OLMo-2 is tested. Each run is 600 steps, about 39M tokens (p.11), roughly 300 times less than BoLT's 13B tokens (p.16). At this scale the models score near zero on GSM8K and MATH (p.21), so only loss is reported and reasoning ability is not evaluated. It is unclear whether the findings hold for current models and realistic training budgets.

W2. All trained models end up worse than the model they started from, and this seems to come from the authors' own training setup. The authors restart a fully trained model with a high learning rate and a new optimizer, which they note is "where we departed from BoLT" (p.5). BoLT's own setup is not tried. The fixes they find, mixing the original training data back in and lowering the learning rate, are already standard practice (Ibrahim et al., 2024). The training also behaves unusually: test loss on the same kind of data keeps going up with more training (from 1.31 to 2.21 with 5x more steps, and to 3.44 with 3x more data; p.6), and the authors say they cannot explain this (p.23). Since the paper's main conclusion is built on this drop, its cause should be understood first.

W3. Several conclusions contradict each other. The starting model was trained on thoughts written by GPT, while all later runs use thoughts written by the model itself. The paper says this switch does not explain why the models get worse (p.5). But going back to the GPT-written data (the "100% replay" run in Table 9) "removes 89% of the damage" (p.7), and Appendix C says the two effects cannot be separated (p.16). The abstract opens with the results of the first experiment (0.49 and 0.21), which the paper later says are inflated by its setup and "should not be quoted" (p.8). Appendix J calls the finding that the proposed score does worse than random "the most stable allocation result in the paper", but the same appendix says this finding "is not established" in the fairer, matched comparison (p.29). This makes it hard to tell which claims are final.

W4. What remains is small or already known. In the only setup where training actually improves the model, the differences between methods are just 0.001-0.006 in loss (p.9), about as large as what you get by re-drawing the random selection (up to 0.004; p.29). The learning rate and the amount of mixed-in data behind these results were chosen on the test set (p.6, p.29). The abstract itself says the reason the models get worse is already known (Wang et al., 2025b), 24 of the 25 papers the authors surveyed already report the starting model (p.9), and the authors say the learning-rate check is not their own finding (p.31). So the main recommendation adds little to current practice.

Questions for authors

Q1. Please see W1-W4.

Q2. Several statements in the paper seem to contradict each other:

  1. Does switching from GPT-written to self-written thoughts explain why the models get worse? p.5: "Corpus provenance therefore does not explain the step-0 result." vs. p.7: "So full replay removes 89% of the damage in this sweep." (the same setup, "100% replay" in Table 9) vs. p.16: "we cannot separate 'continued pretraining degrades this checkpoint' from 'iteration-1 latents are weaker training data than the GPT-seeded ones' with the runs we have."
  2. Can the headline numbers be used? Abstract: "uniform beat an equal-budget random control by 0.49 nats/token, and demand-based routing lost to that control by 0.21." vs. p.8: "its magnitudes are recipe-inflated and should not be quoted."
  3. Is the problem in the standard design or in your own setup? p.2: "Comparing allocation rules by continued pretraining from a shared warmstart — the standard design — gave us a clean result" vs. p.5: "the second is also where we departed from BoLT".
  4. Is the recommendation new? Abstract: "we argue they belong in the standard protocol" and p.2: "which the design does not require and which we had not done" vs. p.9: "twenty-four report the starting checkpoint in some form — our earlier 'omitted by default' was wrong and is withdrawn" and p.31: "we do not present learning-rate sensitivity as our finding".
  5. Does the proposed score really do worse than random? p.29: "which makes it the most stable allocation result in the paper" vs. p.29: "So the density-matched demand harm seen on the evaluation set is not established."
  6. How was the main result's setup chosen? p.6: "it is what we ask readers to take as the paper's allocation finding" vs. p.6: "so the rate is test-set-informed".

Some text also looks like leftovers from earlier drafts:

  • p.30: "We position ThinkGap against the nearest lines of work". ThinkGap appears nowhere else in the paper.
  • Figure 3 (p.15): "(BET 3)", "(BET 1)", "39% reduction, but BET1 fails". BET is never defined.
  • p.21: "This appendix has been rewritten twice". p.27: "We report this paragraph twice over, because the first version was wrong". p.25: "Precision, found after submission review."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.