- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LP-SFT, a supervised fine-tuning objective that preserves the relative probabilities of plausible non-target tokens from the pretrained model. Specifically, it removes the ground-truth token from the base model's top-K candidates and minimizes a locally normalized KL divergence alongside standard cross-entropy. The authors motivate this design through an analysis of multimodal next-token entropy distributions and demonstrate improvements in aggregate performance, capability retention, and pass@k across several models and domains.
Critical strengths
- This paper is simple and easy to implement. The regularization objective can be computed using the logits from the standard forward pass.
- The idea is intuitive and follows a wide array of recent SFT literature in that preserving the "on-policiness" helps with fine-tuning.
Critical weaknesses
- My biggest concern with this paper is novelty. Preserving the pretrained model's distribution during SFT is already a well-established direction in the literature. The proposed method largely amounts to a minor modification of existing KL-based regularization and knowledge distillation objectives. I do not find sufficiently novel conceptual or methodological insights that justify its contribution at ICLR.
- The proposed method introduces additional design choices, including top-K candidate selection, target-token exclusion, and local normalization. While these choices are individually motivated, the overall framework appears unnecessarily complicated for what is essentially distribution-preserving regularization. The paper does not convincingly demonstrate why such a specialized formulation is necessary compared to simpler, well-tuned KL-based alternatives. Overall, I find it to contain unnecessary methodological complexity.
- This proposed method is essentially a subset of [1]. However, I think even without this paper, the novelty doesn't sufficiently justify itself.
[1] Decoupled Knowledge Distillation. CVPR 2022
Questions for authors
What hyperparameter search was conducted for the baselines? Could the gains of LP-SFT stem from its additional hyperparameters rather than the proposed method itself?