acceptodds
Under review as a conference paper at ICLR 2027

LP-SFT: Keeping Plausible Alternatives Alive in Supervised Fine-Tuning

Abstract

Supervised fine-tuning (SFT) is widely used to adapt pretrained language models to downstream domains, but can over-specialize the model and degrade its pre-existing capabilities. Standard cross-entropy drives probability mass onto the observed target token and can make plausible alternatives that the pretrained model itself endorsed vanish. Using Shannon and Rényi entropies, we show that pretrained next-token distributions exhibit a regular multimodal structure, with entropy peaks corresponding to different numbers of plausible alternatives. Motivated by this observation, we propose LP-SFT, a Local-Preserving Supervised Fine-Tuning objective with two key design principles: it removes the supervised token from a local top-K candidate set to avoid conflict with cross-entropy, and applies a locally normalized KL loss to preserve relative preferences among the remaining non-label alternatives. Across mixed-domain and domain-specific fine-tuning experiments, LP-SFT consistently outperforms standard SFT and recent baselines in aggregate performance while keeping base-endorsed alternatives from vanishing, achieving a favorable balance between single-sample accuracy and finite-budget solution accessibility, as measured by pass@1 and pass@k, respectively, with only a modest increase in training cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, supervised fine-tuning, post-training, catastrophic forgetting, knowledge preservation, KL regularization, entropy, next-token distribution

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5GfpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LP-SFT, a supervised fine-tuning objective that preserves the relative probabilities of plausible non-target tokens from the pretrained model. Specifically, it removes the ground-truth token from the base model's top-K candidates and minimizes a locally normalized KL divergence alongside standard cross-entropy. The authors motivate this design through an analysis of multimodal next-token entropy distributions and demonstrate improvements in aggregate performance, capability retention, and pass@k across several models and domains.

Critical strengths
  1. This paper is simple and easy to implement. The regularization objective can be computed using the logits from the standard forward pass.
  2. The idea is intuitive and follows a wide array of recent SFT literature in that preserving the "on-policiness" helps with fine-tuning.
Critical weaknesses
  1. My biggest concern with this paper is novelty. Preserving the pretrained model's distribution during SFT is already a well-established direction in the literature. The proposed method largely amounts to a minor modification of existing KL-based regularization and knowledge distillation objectives. I do not find sufficiently novel conceptual or methodological insights that justify its contribution at ICLR.
  2. The proposed method introduces additional design choices, including top-K candidate selection, target-token exclusion, and local normalization. While these choices are individually motivated, the overall framework appears unnecessarily complicated for what is essentially distribution-preserving regularization. The paper does not convincingly demonstrate why such a specialized formulation is necessary compared to simpler, well-tuned KL-based alternatives. Overall, I find it to contain unnecessary methodological complexity.
  3. This proposed method is essentially a subset of [1]. However, I think even without this paper, the novelty doesn't sufficiently justify itself.

[1] Decoupled Knowledge Distillation. CVPR 2022

Questions for authors

What hyperparameter search was conducted for the baselines? Could the gains of LP-SFT stem from its additional hyperparameters rather than the proposed method itself?

Reviewer_5udHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies how supervised fine-tuning can suppress plausible next-token alternatives that the base model assigned meaningful probability to. The authors analyze the base model’s token distributions and propose preserving the relative probabilities of these alternatives to reduce forgetting.

Their method, LP-SFT, selects the base model’s top tokens at each position, removes the supervised target, and adds a locally normalized KL penalty over the remaining alternatives. The penalty preserves their relative probabilities without fixing their total probability weight. Reference logits are cached before training, avoiding a second model in memory during finetuning, which is a positive. Experiments cover chat, code, mathematics, and mixed-domain training across several model sizes, measuring benchmark performance, alternative token collapses, and computational cost.

Critical strengths

1. The method is simple and clearly motivated. The theoretical analysis explains the roles of target-token removal and local normalization. Table 20 shows that removing either component reduces aggregate performance in the test setting. Caching reference logits also avoids keeping a second model in memory. Table 19 reports nearly identical fine-tuning cost to CE, although cache construction adds a one-time cost, but is still minimal.

2. The paper measures the behavior it aims to change. Figure 5 measures how often alternatives endorsed by the base model become nearly impossible after fine-tuning. The reported collapse rate falls from 3.7–13.3% under CE to 0.8–1.8% under LP-SFT. This connects the motivation to an observable effect beyond benchmark averages.

Critical weaknesses

1. The results do not clearly establish preservation without sacrificing adaptation, and they rely on one training seed. In the Qwen3-14B mixed-domain experiment, every fine-tuned method remains below the base model: the reported average is 66.79 for the base, 58.55 for CE, and 65.32 for LP-SFT. This demonstrates reduced degradation, but does not establish the desired trade-off when learning new capabilities. In fact it showcases something that I think undermines the method.

LP-SFT does exceed the base in other settings, so this concern is not universal. However, aggregate improvements can hide weaker performance in the training domain. After mathematics fine-tuning, Llama-3.1-8B achieves 31.67% MATH-500 pass@1 with standard CE versus 25.42% with LP-SFT, despite LP-SFT having the higher overall average.The experiments specify one training seed, 1234, without results across repeated training runs. On NuminaMath, the aggregate gains over CE are 0.90 points for Qwen3-4B and 0.79 points for Llama-3.1-8B. Their reliability is therefore unclear, and very shakey.

2. The paper does not rule out less aggressive training as an explanation. A smaller learning rate or earlier stopping might also reduce forgetting by limiting adaptation. LP-SFT’s penalty does not freeze the model: it constrains relative probabilities within a non-target subset, while allowing the target probability to increase. Nevertheless, the practical effect could still include restricting how much the model adapts.

The missing comparison is whether LP-SFT preserves more capability at matched target-task improvement. The LoRA experiment and preservation-weight ablation do not answer this question. The collapsing rate ablation should also be compared with gentler CE training to establish whether it identifies a benefit specific to LP-SFT. This is missing.

Questions for authors

These questions are listed in order of importance to my assessment.

1. Does LP-SFT outperform gentler CE training at matched target-task performance? Please compare against a focused sweep of CE learning rates and training duration. Plotting target benchmark improvement against degradation on the other benchmarks would help distinguish a better learning–retention tradeoff from reduced adaptation. Reporting collapse rates for these runs would also strengthen the interpretation of Figure 5. Right it is very unclear to me how the tradeoff is established.

2. How reliable are the gains across training seeds? The reported experiments use seed 1234. Could you repeat the key comparisons across at least three training seeds, particularly in settings where CE improves over the base? Please report target performance and retention separately, rather than only the aggregate average. Also, is there an issue with the data curation for Qwen3-14B? If every SFT method is worse than the baseline, does this not undermine the method?

3. Can you explain the target-domain regression in the Llama mathematics experiment? MATH-500 pass@1 falls from 31.67% with CE to 25.42% with LP-SFT. The absence of a direct logit penalty does not guarantee unchanged learning through shared model parameters. Visualizing CE loss or target token probability curves would help clarify whether LP-SFT learns the training answers as effectively, alongside an explanation of the held-out mathematics result.

Reviewer_7wCpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed LP-SFT (Local-Preserving Supervised Fine-Tuning), where instead of predicting a single token in next-token prediction, Top-K tokens are selected at each position. Then LP-SFT removes the supervised token and renormalizes the base probability distributions on the selected tokens. LP-SFT is evaluated over several code and math benchmarks and argued that their proposed method surpasses existing baselines.

Critical strengths
  • The paper proposes a new fine-tuning approach that actually keeps those valid alternative paths alive instead of flattening them out.

  • It formulates a principled local-preservation loss via target-token removal and local KL normalization, theoretically ensuring that alternative token preferences are protected without conflicting with standard target-token convergence.

Critical weaknesses
  • The claim introduced in Line 161 lacks both supporting citations from prior literature and formal theoretical proof.

  • Although the authors distinguish LP-SFT from traditional distillation, the proposed methodology closely relates to token-level distillation between the frozen base model and the current model. Despite this similarity, the evaluation only benchmarks against standard fine-tuning approaches rather than methods sharing this structural property.

  • Because of the structural parallels to token-level distillation, the lack of comparisons against established techniques such as EOPD[1] and AlignDistil[2] leaves a gap in understanding how LP-SFT performs relative to existing distillation baselines.

  • The evaluation misses out on testing how well LP-SFT handles out-of-distribution scenarios and cross-dataset in-domain tasks, making it hard to tell if keeping those alternative token paths actually helps with broader transfer learning.

References:

[1] Entropy-Aware On-Policy Distillation of Language Models

[2] AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation

Questions for authors
  • What does the author mean by “multimodal structure of next token distributions”?

  • How sensitive is the training stability and performance of LP-SFT to its loss-balancing hyperparameter when scaling across models ranging from 7B to 72B parameters?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.