acceptodds
Under review as a conference paper at ICLR 2027

Can Old Dogs Learn New Tricks? Capability Dynamics Along Pretraining

Abstract

When teams continue training a language model, they have to choose which data to train on, which checkpoint to start from, and how much old data to mix back in. These choices are usually guided by the training loss and by benchmark scores based on answer likelihood. We test whether these signals lead to the right choices. We take public OLMo-2 checkpoints and continue each one several times. Each run changes a single setting and keeps all training batches identical. We judge the model by the answers it writes. Likelihood scores can pick the wrong data. Training a 1B model on math data raises its accuracy on word problems by 47 percentage points while code data does not, yet a likelihood score computed on the final answer alone, with no written reasoning, favors code. The starting checkpoint also matters. At 7B, with our fine-tuning recipe, checkpoints from before mid- training learn a new task far more slowly than later ones. Finally, runs that look the same on loss and benchmark averages differ in which questions they get right. On GSM8K the difference is mostly a missing stop token.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Capabilities, Continued Pre training, Post training, Token costs, Developmental Learning, Replay, Data-curriculum

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HpAmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors examine whether training loss and likelihood scores reliably guide continued LLM training. Controlled experiments show that these signals can misrank training data and hide differences across checkpoints and training settings. The findings support evaluating generated reasoning and individual answers while distinguishing capability changes from output-format effects.

Critical strengths
  • The paper addresses an interesting question about whether commonly used evaluation signals can reliably guide decisions during continued language-model training.

  • The longitudinal analysis covers 107 public OLMo-2-1B checkpoints, providing a detailed view of how evaluation signals change throughout training.

  • The evaluation includes mathematical reasoning, code execution, relational reasoning, and a synthetic learning task, examining several forms of model behavior.

Critical weaknesses
  • The study identifies training choices that improve particular tasks, but does not establish whether these choices improve overall model performance after accounting for gains and losses across different capabilities.

  • The reasoning-conditioned control changes both the reasoning context and the answer format. Consequently, it does not fully separate the contribution of reasoning from that of formatting.

  • The paper reports settings where likelihood scoring performs well and others where generation is strongly affected by formatting. These results establish task-dependent weaknesses, but provide limited guidance about which readout to trust in a new setting.

  • The reported cooldown gains are unclear. At the 4,001B checkpoint, the text first reports a gain of 0.430, but later describes a second data draw changing the gain from 0.442 to 0.326.

Questions for authors

Please refer to the Critical Weaknesses.

Reviewer_p7QgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how capabilities of the model change during continued training, mainly on Olmo-2 checkpoints. It compares evaluation based on likelihood vs generated-answer accuracy. It also measures performance after finetuning different checkpoints or investigates forgetting under different replay and optimizer settings. The authors show that cheap evaluation proxy can favor wrong training data.

Critical strengths

None

Critical weaknesses

First of all, the paper is very difficult to read and understand, the writing needs a significant improvement. The paper covers many things -- evaluation, checkpoint selection, replay, optimizer settings, activation interventions -- without clearly explaining how these investigations fit together or address one main research question. The models, tasks, training setups, and scoring rules change frequently, making the comparisons difficult to track. The tables and figures present many results, while their role is often unclear.

As for the paper's substance, I find the novelty of the main result very limited. Disagreement between likelihood-based and generation-based evaluation is already discussed in prior work, for example in the "Beyond Probabilities" paper by Lyu et al.

Questions for authors

none

Reviewer_oe6POpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

I am not an expert in this area, so I am uncertain how substantial the paper’s contribution is relative to prior work. My assessment focuses primarily on the experimental evidence and whether it supports the stated conclusions.

The paper investigates how evaluation choices affect decisions about continued language-model training, including data selection, starting checkpoints, and replay. Using controlled continuations of public checkpoints, primarily from OLMo-2, it compares training loss, answer-likelihood scores, generated answers, and per-item evaluation patterns. Its central finding is that these measurements can lead to different decisions: answer-only likelihood scoring can miss improvements in mathematical problem solving, apparent forgetting can largely reflect changes in output formatting, and benchmark averages can conceal systematic differences between runs. The paper also studies checkpoint-dependent learning costs and replay behavior, although these findings are more sensitive to the training configuration.

Critical strengths
  • The experiments address concrete training decisions. The data-selection experiment provides a particularly clear example: relative to web-only training, mathematics data improves generated one-step accuracy by 46.5 percentage points, whereas code data produces no improvement, despite increasing answer-likelihood accuracy. Evaluation on SVAMP and a longer training budget provides additional support for the practical relevance of this discrepancy. (38649_Can_Old_Dogs_Learn_New_T, n.d.)

  • The controlled branching design strengthens comparisons. Continuations from a shared parent use matched training batches while varying the condition of interest. For the main replay comparison, the authors also use a parent-level test rather than treating all related forks as independent observations. These choices improve the interpretability of the results.

  • The analysis distinguishes task performance from output-format compliance. On one GSM8K continuation, accuracy falls from 13.3% to 2.0% when an immediate end-of-sequence token is required, while numerical-answer accuracy remains at 11.3%. Activation interventions and restricted-layer updates further support a localized change in stopping behavior. This is a useful demonstration of how an evaluation rule can exaggerate apparent capability loss.

  • The paper reports several informative negative results and boundaries. These include reversals of the replay effect across models, incomplete replication of the layer-localization result, and an explicit distinction between tested mechanisms and proposed explanations. This transparency helps readers assess which conclusions are well supported.

Critical weaknesses
  • Training-level replication is limited for several central comparisons. The main data-source comparisons generally use one run per source, and their confidence intervals primarily capture evaluation-item uncertainty rather than training variability. Reported changes from additional data draws are comparable to the likelihood-score preference for code. Consequently, the evidence is stronger that mathematics data improves generated performance than that likelihood scoring reliably prefers code across repeated training runs.

  • The explanation for the likelihood–generation discrepancy is not fully isolated. Scoring after the model’s reasoning improves agreement with generation, but this comparison also changes the output format. The original candidate scores include a closing JSON delimiter, whose conditional probability can depend on the candidate answer. A factorial comparison of reasoning availability and formatting, with otherwise matched prompts and scoring, would more directly identify the source of the discrepancy.

  • The replay finding has a narrow demonstrated scope. One-percent replay produces worse minimum and mean performance in the main OLMo-2-7B setting, but the final-performance difference remains unresolved, and the effect reverses for a fully trained Qwen2.5-7B parent and the tested OLMo-2-13B parent. The proposed overshoot explanation also lacks the decisive comparison between replay and no replay at the same smaller learning rate.

Questions for authors

Refer to weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.