- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Ambient SFT, which combines standard SFT with high-temperature distribution matching against a frozen pre-trained model. The method aims to improve data efficiency and reduce overfitting and catastrophic forgetting. The authors provide theoretical analyses of temperature scaling and regularized estimation, and report improvements across domain adaptation, instruction tuning, and reasoning tasks.
Critical strengths
The method achieves consistent improvements over vanilla SFT across multiple model families and downstream tasks, particularly in low-data settings. The experimental results suggest that incorporating additional regularization during SFT can improve generalization.
Critical weaknesses
- Unconvincing motivation and insufficient theoretical justification.
The central motivation is not well justified. Standard SFT learns from observed target tokens, whereas the proposed method additionally fits the full vocabulary distribution of an imperfect teacher. If the goal is to mitigate overfitting, simpler approaches such as token perturbation or label smoothing could provide regularization without requiring the model to fit potentially inaccurate teacher distributions. It is unclear why the proposed approach is preferable, or whether it may instead encourage the student to reproduce the teacher's errors.
The theoretical results do not adequately resolve this concern. Theorem 3.1 establishes distribution recoverability rather than improved supervision quality, while Eq. (6) describes a generic contraction toward uniformity. Moreover, Theorem 3.2 does not specifically guarantee the benefit of high temperatures and relies on restrictive assumptions about teacher error. The connection between these results and the claimed reduction in overfitting and forgetting remains weak.
- Limited novelty and insufficient experimental comparisons.
The method closely resembles existing distillation and regularization approaches. Its effectiveness also depends on additional hyperparameters, particularly temperature and mixing probability, whose appropriate values may vary substantially across teacher models and target domains. The paper provides no clear principle for selecting them, limiting the practical applicability of the method.
More importantly, the experiments lack comparisons with simpler and well-established regularization baselines, including token perturbation, label smoothing, parameter regularization, and adequately tuned standard KL distillation. Although some temperature ablations are provided, the main instruction-tuning and reasoning experiments primarily compare against vanilla SFT. Consequently, it remains unclear whether the reported improvements are specific to the proposed approach or could be achieved by simpler alternatives.
Questions for authors
-
Motivation: If the objective is to reduce SFT overfitting, why is fitting an imperfect teacher's full vocabulary distribution preferable to simpler methods such as token perturbation or label smoothing? Could full-distribution matching instead introduce additional bias from incorrect teacher predictions?
-
Theory and hyperparameters: What theoretical justification specifically supports (T>1), and how should (T) and (p_c) be selected for different teacher qualities and target domains? Can the authors demonstrate that the method is robust to these choices?
-
Baselines: Can the authors provide comparisons with token perturbation, label smoothing, parameter regularization, and independently tuned standard KL distillation, particularly on the main instruction-tuning and reasoning benchmarks?
10 comments
Rebuttal part 2: Evidence for reduced forgetting, addressing theory and novelty concerns
W1 (motivation). Comparison with simpler regularizers (continued)
The same holds for reasoning, where the benefit of Ambient SFT becomes even wider. The two strongest baselines (tuned KD and L2-SP) fall below Vanilla SFT for reasoning while Ambient SFT improves on it. Reasoning results are shown below:
| Method | GSM8K | AIME24 | AIME25 | GPQA | IFEval | HumanEval+ | Avg. rel. Δ vs. SFT |
|---|---|---|---|---|---|---|---|
| Vanilla SFT | 94.2 | 39.8 | 33.2 | 39.3 | 34.6 | 75.6 | — |
| Tuned KD (T=1) | 90.7 | 36.2 | 30.8 | 33.0 | 34.0 | 70.7 | -7.3% |
| L2-SP (α=1) | 81.7 | 9.9 | 14.0 | 28.6 | 33.8 | 40.3 | -37.1% |
| Ambient SFT | 93.0 | 44.5 | 34.1 | 40.0 | 36.0 | 76.9 | +3.5% |
We have added these results (both for instruction tuning and reasoning) to the revised submission. We thank the Reviewer for suggesting these experiments that strengthened our submission.
W1 (motivation). Does the student reproduce the teacher's errors?
On reproducing the teacher's errors: this is exactly what the temperature controls. Ambient SFT does not ask the student to copy the teacher. The cross-entropy term fits the observed target tokens, and the Temperature Matching term asks the student to match the teacher only at a high temperature, where the teacher's confident (and possibly wrong) preferences are flattened. At T=1 the method reduces to knowledge distillation, which copies the teacher, and as it reduces to Vanilla SFT, which ignores it. Theorem 3.2 makes this trade-off precise: the teacher's error enters the risk as a bias term that shrinks as T grows, and there is an intermediate T ≠ 1 that strictly beats both maximum likelihood (Vanilla SFT) and the best knowledge-distillation run. The experiments agree: tuned KD at T=1 reaches +17.0%, below Ambient SFT at T=2.5 (+20.3%).
W1 (theory). Connection between our theory and the method
[...] Eq. (6) describes a generic contraction toward uniformity. Moreover, Theorem 3.2 does not specifically guarantee the benefit of high temperatures and relies on restrictive assumptions about teacher error. The connection between these results and the claimed reduction in overfitting and forgetting remains weak.
Indeed, Eq. (6) shows contraction towards the uniform distribution. However, the uniform distribution does not provide good supervision. This is consistent with the label smoothing experiment, where we show that if you regularize towards the uniform instead of the base model's distribution results deteriorate significantly. What instead our method attempts to do is to add the least amount of temperature such that the teacher bias becomes small. This is because there is a tension between keeping the knowledge of the teacher (information preservation) and reducing the bias (distribution alignment). Temperature controls exactly that trade-off. Theorem 3.2 exactly analyses this tradeoff for a simple model.
On the assumption about the teacher's error: Theorem 3.2 assumes that the teacher's error is of the same order as the statistical error of maximum likelihood, . This is the regime in which the question of whether to use the teacher is non-trivial. If the teacher's error is much larger than the statistical error, the bias term dominates and the optimal λ moves toward 0, i.e., toward ignoring the teacher (Vanilla SFT). If it is much smaller, the frozen teacher alone is already good enough. The interesting case, where a weak teacher and a small dataset each carry information the other lacks, is exactly the case the theorem analyzes.
On overfitting and forgetting: Theorem 3.2 measures the estimation error on the target distribution, so it speaks to overfitting (through the variance term) rather than to forgetting, which it does not model. We agree that the evidence for reduced forgetting is empirical rather than theoretical. Our new evaluations of the base models provide this evidence (see our answer to Reviewer Potc and the Base rows of Table 2 in the revised submission): Vanilla SFT falls below the base model on 6 or 7 of the 7 non-IFEval benchmarks for every model, while Ambient SFT is above both the base model and Vanilla SFT on average for every model, and mostly preserves the base model's knowledge on MMLU, PopQA and TriviaQA.
W2 (novelty). Comparison with knowledge distillation and LwF
The method closely resembles existing distillation and regularization approaches. Its effectiveness also depends on additional hyperparameters, particularly temperature and mixing probability, whose appropriate values may vary substantially across teacher models and target domains. The paper provides no clear principle for selecting them [...].
We thank the Reviewer for these comments. We address novelty first and the choice of hyperparameters second.
Rebuttal part 3: Addressing novelty concerns and robustness to different hyperparameters
W2 (novelty). Comparison with knowledge distillation and LwF (continued)
Novelty. Our method is closest to two existing approaches: knowledge distillation toward the base model (KD), which is our objective at T=1, and Learning without Forgetting (LwF).
Comparison with KD. During the rebuttal period, we tuned KD toward the same frozen base model (T=1, cross-entropy weight ∈ {0.25, 0.5, 0.75, 0.9}) and compared it with Ambient SFT (T=2.5) in every setting where we could run both (average relative improvement over Vanilla SFT):
- Instruction tuning, Qwen3-1.7B (Table 2 setup): +17.0% for the best KD run vs +20.3% for Ambient SFT. At the same mixing weight (=0.75), Ambient SFT beats KD on 7 of the 8 benchmarks.
- Instruction tuning at larger scales, with KD at its best 1.7B setting: −0.9% vs +8.0% at 4B, +7.8% vs +10.2% at 8B, and −2.7% vs +8.7% at 14B.
- Reasoning: KD is below Vanilla SFT on all six benchmarks (−7.3%), while Ambient SFT improves on it (+3.5%).
- Domain adaptation: the temperature sweep of Figure 3a shows that validation loss falls from T=1 to its minimum at T=2.5.
This is what our theory predicts. In Theorem 3.2, the curvature of the matching penalty at T=1 equals the Fisher information of the data (), so KD shrinks the estimate hardest along exactly the directions the data already determines well. Changing the temperature changes the shape of the penalty: as T grows, the teacher adds curvature where the data supplies none. The theorem shows that some T ≠ 1 strictly beats the best T=1 run, as well as maximum likelihood (Vanilla SFT) and the frozen teacher.
Comparison with LwF. Mechanically, our approach is similar to LwF: in both cases, a model is adapted to learn a new capability and is regularized to stay close to its smoothed prior predictions on inputs from the new task. There are, however, some important differences:
- In LwF, the new task has a separate output layer (only trained for the new task alone). Distillation acts on the old task’s output head while the new task trains a separate head. The two losses interact only through shared features. In our case, all the parameters are shared.
- More importantly, the output space is shared. In our method, the “new task” (i.e. learning to follow instructions) competes directly with the old tasks as they share not only some parameters, but the same output space. The parameter sharing and output space sharing change the role of temperature (see next point).
- In LwF, the temperature (T=2) is a fixed softening choice. In Ambient SFT, both loss terms act on the same next-token distribution, so the temperature sets how much the student trusts the base model: T=1 is knowledge distillation toward the base model, and is Vanilla SFT. As shown above, this choice matters.
- LwF is motivated empirically. Our method comes with a theoretical analysis that studies (for a simplified model) the bias-variance trade-off the temperature controls (Theorem 3.2).
- Finally, our method is applied to modern LLMs and its gains are not limited to retaining existing capabilities. Our method improves the target-domain loss (Table 1), increases performance on MATH through learning to follow instructions, and improves reasoning benchmarks such as AIME24 (44.5 vs 39.8 for Vanilla SFT).
We have revised our submission to discuss LwF prominently in Section 3.1.
W2 (hyperparameters). Robustness and selection of the temperature
Hyperparameters. We used a single setting, T=2.5 and =0.75, for all of our instruction-tuning and reasoning experiments (all the results of Table 2 and Figures 4 and 5). The only place where we ablated the temperature was our domain adaptation experiments. Even there, while different temperatures were optimal for different datasets, using T=2.5 instead of the optimal temperature leads to at most a 1.3% drop in validation loss (0.6% on average). Hence, T=2.5 works robustly across all our settings (domain adaptation, instruction tuning and reasoning).
That said, the temperature controls a bias-variance trade-off (Theorem 3.2), so in general we should not expect a single value to be optimal everywhere. The optimal temperature depends on:
- How many datapoints we have. With infinite data, no supervision from the biased teacher is needed, so the temperature should go to infinity.
- How good the teacher is. Powerful teachers call for lower temperatures and weaker teachers for higher ones.
- How good the student is. When the teacher and the student have different sizes (as in some of our domain adaptation experiments), the optimal temperature also depends on the gap in capability between them.
Rebuttal part 1: Comparison to other regularizers
We thank the Reviewer for their valuable time and input. We are glad the Reviewer found that our method achieves consistent improvements over vanilla SFT across model families and downstream tasks, particularly in low-data settings.
We believe that the Reviewer has been unfairly harsh to our submission by giving it a clear reject. The Reviewer raised concerns about the motivation of our method compared to simpler regularizers, the connection between our theory and the method, the novelty of our method compared to distillation and regularization approaches, the choice of hyperparameters, and the comparisons with simpler baselines. We address all these points below, with new experimental evidence and clarifications. We have updated our paper to reflect those changes (with the changes denoted in blue).
W1 (motivation). Comparison with simpler regularizers
We thank the Reviewer for this question. We agree that a regularizer is only worth using if it beats simpler ones. During the rebuttal period, we therefore compared Ambient SFT against exactly the alternatives the Reviewer mentions and additional baselines, each tuned over a grid whose best value lies inside the grid:
Results for Instruction tuning are shown below:
Ambient SFT outperforms all these baselines.
Label smoothing never improves on Vanilla SFT (-3.5% to -0.3%) and NEFTune barely changes it (+0.1% to +0.8%), while Ambient SFT gains +20.3%. Label smoothing also trains on soft targets, so the gains of Ambient SFT do not come from softer targets alone: they come from smoothing toward the base model's tempered distribution, which carries what the model learned during pre-training. The strongest baselines are tuned KD (which is our method for T=1) and L2-SP which does weight space regularization. Ambient SFT outperforms both of these methods by a significant margin for instruction-tuning.