LP-SFT: Keeping Plausible Alternatives Alive in Supervised Fine-Tuning

Supervised fine-tuning (SFT) is widely used to adapt pretrained language models to downstream domains, but can over-specialize the model and degrade its pre-existing capabilities. Standard cross-entropy drives probability mass onto the observed target token and can make plausible alternatives that the pretrained model itself endorsed vanish. Using Shannon and Rényi entropies, we show that pretrained next-token distributions exhibit a regular multimodal structure, with entropy peaks corresponding to different numbers of plausible alternatives. Motivated by this observation, we propose LP-SFT, a Local-Preserving Supervised Fine-Tuning objective with two key design principles: it removes the supervised token from a local top-K candidate set to avoid conflict with cross-entropy, and applies a locally normalized KL loss to preserve relative preferences among the remaining non-label alternatives. Across mixed-domain and domain-specific fine-tuning experiments, LP-SFT consistently outperforms standard SFT and recent baselines in aggregate performance while keeping base-endorsed alternatives from vanishing, achieving a favorable balance between single-sample accuracy and finite-budget solution accessibility, as measured by pass@1 and pass@k, respectively, with only a modest increase in training cost.