- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies the relationship between checkpoint geometry and performance degradation after post-training. Using 20 checkpoints from Qwen3-8B and Llama-3.1-8B, the authors analyze correlations between relative update norms, cosine similarity with reference checkpoint differences against benchmark changes. The main finding is negative: that these correlations are sensitive to model settings and the composition of the evaluated checkpoints and are not really a good predictors of the benefit of a given model update.
Critical strengths
- The paper topic is very interesting: whether checkpoint geometry can provide useful signals about performance degradation following fine-tuning. This can save substantial cost when it comes to deciding which models to train or which weight udpates to perform.
- While the main finding is negative, it can still be relevant in some settings.
Critical weaknesses
- Limited novelty. The main findings are unsurprising: larger updates can lead to greater performance degradation (i.e., catastrophic forgetting). The paper does not introduce a new method or enable substantially new insights beyond existing work.
- The analysis covers only 20 related checkpoints from two 8B model families. Several of these are actually related variants, making it difficult to draw general conclusions-- especially to larger scale models where the value of these techniques is more pronounced and relevant. So it remains unclear whether the findings generalize to larger models, different architectures, or other post-training settings.
- The paper does not evaluate whether these measurements can predict regressions on unseen checkpoints. No held-out evaluations or comparisons against simple baselines.
- The paper is unnecessarily difficult to follow, with convoluted terminology and extensive repetition of relatively straightforward observations. I advise the authors to re-write the whole paper to improve clarity.
Questions for authors
- How sensitive are the findings to model scale, particularly for models substantially larger than 8B parameters?