- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies continued-pretraining comparisons for latent-augmented reasoning. It first proposes reasoning demand, a posterior-to-prior KL signal that predicts per-chunk latent benefit, but finds that routing latents by this signal does not improve training over random allocation. More importantly, the authors show that their initially strong and seed-stable allocation ranking occurred entirely above the starting checkpoint’s loss: the comparison was ranking relative damage rather than learning. After correcting warmstart-density and learning-rate confounds, the effects shrink substantially, and when replay plus a lower learning rate finally yields improvement over step 0, the original uniform-routing winner no longer leads.
Critical strengths
- The paper is unusually careful about empirical validity: it revises several initially attractive conclusions after additional seeds and controls, distinguishes evaluation-set from training-seed uncertainty, and includes clean-room replications.
- The step-0 counterexample is useful: a relative ordering can remain stable across seeds, domains, and model scale while every method is worse than the checkpoint it started from.
Critical weaknesses
- Limited novelty. The underlying mechanisms—continued-pretraining degradation after re-warming, sensitivity to learning rate, distribution shift, replay as a remedy, and instability of data-recipe rankings across hyperparameters—are already acknowledged by the paper as established in prior work. The main methodological recommendation, explicitly comparing each recipe against its starting checkpoint, is sensible but not particularly surprising; the paper’s own survey also finds that 24/25 sampled CPT papers already report the starting checkpoint in some form.
- The substantive latent-allocation contribution is weak. Reasoning demand predicts per-chunk latent benefit, but this does not translate into a useful routing rule. Under the only regime where continued pretraining actually improves over step 0, no tested signal beats random routing, and the remaining allocation differences are extremely small. Thus the paper provides a careful negative result rather than a compelling new training method.
- “Damage” is metric-specific. The central claim is established primarily through held-out suffix NLL; downstream GSM8K/MATH performance is near floor and is explicitly not used as evidence. Thus the results demonstrate degradation on the evaluated language-modeling axis, not necessarily broader capability damage.
Questions for authors
- What do the authors view as the key conceptual novelty beyond prior work on CPT stability gaps, learning-rate sensitivity, replay, and recipe-ranking instability?
- How broadly should the term damage be interpreted given that the main evidence is held-out NLL and downstream task accuracy is not informative in this training regime?
- Minor presentation issue: what do the undefined terms “ThinkGap” and “BET 1/BET 3” refer to? They appear to be internal or previous terminology and should be clarified or removed.