acceptodds
Under review as a conference paper at ICLR 2027

Measure Before Reuse: Checkpoint Deltas as Post-Training Routing Signals

Abstract

Open-model ecosystems increasingly require users to choose among post-training actions before the relevant evaluations are known: accept a new checkpoint or adapter, search over merges, run targeted regression tests, reuse a prior capability delta , or avoid reuse. Existing delta methods mainly consume checkpoint deltas as construction objects. We argue that they should first be measured as data-free routing signals. Our two-stage workflow compares a prior capability delta with an update delta : Stage I uses checkpoint geometry alone to route to an action family, while Stage II uses benchmark deltas and sample-wise forgetting/backward-transfer transitions to confirm, revise, or reject candidates within that family. The geometry combines a norm ratio for displacement risk with module-level alignment for directional context. Across 20 post-training variants in our suite, norm ratio is the strongest coarse risk indicator, MLP and attention alignment add useful directionality, and global alignment does not. Case studies route a weak-conflict S1 regime to merge search, an adverse S2 regime away from prior reuse, and a statistically non-null but practically near-null SC LoRA case to targeted evaluation. In a fixed-candidate, branch-level routing study, geometry-guided action-family routing reaches mean Overall after Stage-II evaluation, compared with for always-prior and for never-prior. This is branch-level evidence over precomputed candidates, not prospective intervention selection or evaluation-cost reduction. Checkpoint deltas are evidence before they are ingredients.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language and multimodal language models

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_ZjPhOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the relationship between checkpoint geometry and performance degradation after post-training. Using 20 checkpoints from Qwen3-8B and Llama-3.1-8B, the authors analyze correlations between relative update norms, cosine similarity with reference checkpoint differences against benchmark changes. The main finding is negative: that these correlations are sensitive to model settings and the composition of the evaluated checkpoints and are not really a good predictors of the benefit of a given model update.

Critical strengths
  • The paper topic is very interesting: whether checkpoint geometry can provide useful signals about performance degradation following fine-tuning. This can save substantial cost when it comes to deciding which models to train or which weight udpates to perform.
  • While the main finding is negative, it can still be relevant in some settings.
Critical weaknesses
  • Limited novelty. The main findings are unsurprising: larger updates can lead to greater performance degradation (i.e., catastrophic forgetting). The paper does not introduce a new method or enable substantially new insights beyond existing work.
  • The analysis covers only 20 related checkpoints from two 8B model families. Several of these are actually related variants, making it difficult to draw general conclusions-- especially to larger scale models where the value of these techniques is more pronounced and relevant. So it remains unclear whether the findings generalize to larger models, different architectures, or other post-training settings.
  • The paper does not evaluate whether these measurements can predict regressions on unseen checkpoints. No held-out evaluations or comparisons against simple baselines.
  • The paper is unnecessarily difficult to follow, with convoluted terminology and extensive repetition of relatively straightforward observations. I advise the authors to re-write the whole paper to improve clarity.
Questions for authors
  • How sensitive are the findings to model scale, particularly for models substantially larger than 8B parameters?
Reviewer_GuYxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper retrospectively analyzes 20 post-training results from Qwen3-8B and Llama-3.1-8B to examine how the magnitude and direction of weight updates relate to benchmark changes. Across all candidates, larger updates correlate with poorer average retention on three shared benchmarks. After additive and merged candidates are excluded, the association reverses within model settings. The correlation between retention and MLP alignment with a reference update nearly disappears after accounting for setting differences. The results also show that average gains can mask substantial regressions on individual benchmarks.

Critical strengths
  1. The paper shows a practical limitation of using checkpoint geometry for model selection. Larger relative update norms correlate with poorer mean retention when candidates are pooled across model settings, but this association can disappear or even reverse within a setting. This highlights the need to validate a diagnostic on the candidates it will actually be used to rank.
  2. The paper clearly illustrates post-training trade-offs. Reporting target-task gains alongside changes on individual shared benchmarks shows how an improvement in mean performance can mask a substantial regression on one task.
Critical weaknesses
  1. The practical use case remains unclear. When both base and SFT checkpoints are available, it is unclear how the geometric measurements would guide the choice among those checkpoints and candidate merges. With only a base checkpoint, the findings offer little guidance on which update direction to pursue.
  2. The endpoint comparisons provide limited causal evidence. They do not isolate the effects of update magnitude and direction or explain the observed gains. For example, if an interpolation between base and SFT checkpoints improved performance, what evidence would attribute that gain to better use of knowledge already encoded in the model?
Questions for authors
  1. Given the large changes on GSM8K and MMLU, which types of questions switch from incorrect to correct, or from correct to incorrect, relative to the corresponding pre-update baseline? Do these patterns vary with alignment to the reference delta?
  2. At a fixed update norm, would applying the update in the opposite direction cause a substantial benchmark regression? If no such regression occurred, how would that affect your interpretation of reference alignment?
  3. If a practitioner has only a base checkpoint, what concrete next step, if any, do your findings suggest?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.