acceptodds
Under review as a conference paper at ICLR 2027

Ambient LLMs: Data-Efficient Supervised Finetuning via Temperature Matching

Abstract

Collecting large-scale, diverse, and high-quality post-training datasets for autoregressive language models is challenging. Consequently, Supervised Fine-Tuning (SFT) is typically applied to datasets significantly smaller than those used for pre-training, which leads to overfitting and the catastrophic forgetting of the tails of the data distribution. To prevent this collapse, we introduce \methodname, a principled framework for leveraging imperfect predictions to augment the fine-tuning set. In our method, a “frozen” version of the pre-trained model serves as a noisy teacher that the fine-tuned model must match at a coarse level. In particular, we require that the high-temperature versions of the conditional distributions produced by the post-training model should match the high-temperature versions of the pre-training ones. The temperature hyperparameter controls the level of trust in the pre-trained model, with the limit reverting back to regular SFT and to knowledge distillation. We demonstrate that our approach outperforms standard SFT across domain adaptation, instruction-tuning, and reasoning.

Primary area:
generative models
Keywords:
supervised finetuning, data-efficient learning, knowledge distillation, ambient

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nhXKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Ambient SFT, which combines standard SFT with high-temperature distribution matching against a frozen pre-trained model. The method aims to improve data efficiency and reduce overfitting and catastrophic forgetting. The authors provide theoretical analyses of temperature scaling and regularized estimation, and report improvements across domain adaptation, instruction tuning, and reasoning tasks.

Critical strengths

The method achieves consistent improvements over vanilla SFT across multiple model families and downstream tasks, particularly in low-data settings. The experimental results suggest that incorporating additional regularization during SFT can improve generalization.

Critical weaknesses
  1. Unconvincing motivation and insufficient theoretical justification.

The central motivation is not well justified. Standard SFT learns from observed target tokens, whereas the proposed method additionally fits the full vocabulary distribution of an imperfect teacher. If the goal is to mitigate overfitting, simpler approaches such as token perturbation or label smoothing could provide regularization without requiring the model to fit potentially inaccurate teacher distributions. It is unclear why the proposed approach is preferable, or whether it may instead encourage the student to reproduce the teacher's errors.

The theoretical results do not adequately resolve this concern. Theorem 3.1 establishes distribution recoverability rather than improved supervision quality, while Eq. (6) describes a generic contraction toward uniformity. Moreover, Theorem 3.2 does not specifically guarantee the benefit of high temperatures and relies on restrictive assumptions about teacher error. The connection between these results and the claimed reduction in overfitting and forgetting remains weak.

  1. Limited novelty and insufficient experimental comparisons.

The method closely resembles existing distillation and regularization approaches. Its effectiveness also depends on additional hyperparameters, particularly temperature and mixing probability, whose appropriate values may vary substantially across teacher models and target domains. The paper provides no clear principle for selecting them, limiting the practical applicability of the method.

More importantly, the experiments lack comparisons with simpler and well-established regularization baselines, including token perturbation, label smoothing, parameter regularization, and adequately tuned standard KL distillation. Although some temperature ablations are provided, the main instruction-tuning and reasoning experiments primarily compare against vanilla SFT. Consequently, it remains unclear whether the reported improvements are specific to the proposed approach or could be achieved by simpler alternatives.

Questions for authors
  1. Motivation: If the objective is to reduce SFT overfitting, why is fitting an imperfect teacher's full vocabulary distribution preferable to simpler methods such as token perturbation or label smoothing? Could full-distribution matching instead introduce additional bias from incorrect teacher predictions?

  2. Theory and hyperparameters: What theoretical justification specifically supports (T>1), and how should (T) and (p_c) be selected for different teacher qualities and target domains? Can the authors demonstrate that the method is robust to these choices?

  3. Baselines: Can the authors provide comparisons with token perturbation, label smoothing, parameter regularization, and independently tuned standard KL distillation, particularly on the main instruction-tuning and reasoning benchmarks?

10 comments
AuthorsOpenReview ↗

Rebuttal part 1: Comparison to other regularizers

We thank the Reviewer for their valuable time and input. We are glad the Reviewer found that our method achieves consistent improvements over vanilla SFT across model families and downstream tasks, particularly in low-data settings.

We believe that the Reviewer has been unfairly harsh to our submission by giving it a clear reject. The Reviewer raised concerns about the motivation of our method compared to simpler regularizers, the connection between our theory and the method, the novelty of our method compared to distillation and regularization approaches, the choice of hyperparameters, and the comparisons with simpler baselines. We address all these points below, with new experimental evidence and clarifications. We have updated our paper to reflect those changes (with the changes denoted in blue).

W1 (motivation). Comparison with simpler regularizers

[...] If the goal is to mitigate overfitting, simpler approaches such as token perturbation or label smoothing could provide regularization without requiring the model to fit potentially inaccurate teacher distributions. It is unclear why the proposed approach is preferable, or whether it may instead encourage the student to reproduce the teacher's errors.

We thank the Reviewer for this question. We agree that a regularizer is only worth using if it beats simpler ones. During the rebuttal period, we therefore compared Ambient SFT against exactly the alternatives the Reviewer mentions and additional baselines, each tuned over a grid whose best value lies inside the grid:

  • Label smoothing: mixes the one-hot target with the uniform distribution with weight ε. We sweep ε ∈ {0.05, 0.1, 0.2, 0.4}.
  • NEFTune: adds uniform noise to the input embeddings during fine-tuning, a form of token perturbation. We sweep the noise scale α ∈ {5, 10, 15}.
  • L2-SP: penalizes the distance of the weights from the pre-trained weights. We sweep the penalty strength α ∈ {0.001, 0.01, 0.1, 1, 10}.
  • Knowledge distillation: Vanilla SFT plus a KL term to the frozen base model at temperature T=1, i.e., our objective with T=1. We sweep the cross-entropy weight pcp_c ∈ {0.25, 0.5, 0.75, 0.9}.

Results for Instruction tuning are shown below:

MethodSettingMATHGSM8KHumanEval+MMLUPopQATriviaQAIFEvalBBHAvg. rel. Δ vs. SFT
Base (no SFT)—33.853.578.465.613.945.637.954.2+9.0%
Vanilla SFT—24.361.068.061.912.236.358.845.2—
NEFTuneα=523.760.066.861.611.737.860.846.8+0.1%
α=1019.758.969.262.212.438.465.648.6+0.8%
α=1518.361.971.762.211.839.063.048.7+0.2%
Label smoothingε=0.0519.457.968.861.112.036.059.747.5-2.7%
ε=0.125.361.464.262.310.037.357.744.7-2.3%
ε=0.225.462.964.360.912.136.455.147.0-0.3%
ε=0.423.562.556.861.012.034.954.246.9-3.5%
L2-SPα=0.00120.256.072.561.412.237.158.445.8-2.1%
α=0.0121.657.984.563.815.343.154.949.0+7.1%
α=0.130.372.683.065.014.845.350.152.7+14.8%
α=135.072.383.264.914.545.641.852.8+15.3%
α=1035.169.481.865.213.945.638.853.7+13.5%
Tuned KD (T=1)pcp_c=0.2531.872.980.565.015.644.443.155.2+14.9%
pcp_c=0.535.171.581.364.815.743.849.252.8+17.0%
pcp_c=0.7537.567.979.064.313.642.655.552.4+15.6%
pcp_c=0.932.463.878.462.913.340.961.951.7+12.0%
Ambient SFTT=2.5, pcp_c=0.7540.472.383.764.914.943.852.353.3+20.3%

Ambient SFT outperforms all these baselines.

Label smoothing never improves on Vanilla SFT (-3.5% to -0.3%) and NEFTune barely changes it (+0.1% to +0.8%), while Ambient SFT gains +20.3%. Label smoothing also trains on soft targets, so the gains of Ambient SFT do not come from softer targets alone: they come from smoothing toward the base model's tempered distribution, which carries what the model learned during pre-training. The strongest baselines are tuned KD (which is our method for T=1) and L2-SP which does weight space regularization. Ambient SFT outperforms both of these methods by a significant margin for instruction-tuning.

AuthorsOpenReview ↗

Rebuttal part 2: Evidence for reduced forgetting, addressing theory and novelty concerns

W1 (motivation). Comparison with simpler regularizers (continued)

The same holds for reasoning, where the benefit of Ambient SFT becomes even wider. The two strongest baselines (tuned KD and L2-SP) fall below Vanilla SFT for reasoning while Ambient SFT improves on it. Reasoning results are shown below:

MethodGSM8KAIME24AIME25GPQAIFEvalHumanEval+Avg. rel. Δ vs. SFT
Vanilla SFT94.239.833.239.334.675.6—
Tuned KD (T=1)90.736.230.833.034.070.7-7.3%
L2-SP (α=1)81.79.914.028.633.840.3-37.1%
Ambient SFT93.044.534.140.036.076.9+3.5%

We have added these results (both for instruction tuning and reasoning) to the revised submission. We thank the Reviewer for suggesting these experiments that strengthened our submission.

W1 (motivation). Does the student reproduce the teacher's errors?

On reproducing the teacher's errors: this is exactly what the temperature controls. Ambient SFT does not ask the student to copy the teacher. The cross-entropy term fits the observed target tokens, and the Temperature Matching term asks the student to match the teacher only at a high temperature, where the teacher's confident (and possibly wrong) preferences are flattened. At T=1 the method reduces to knowledge distillation, which copies the teacher, and as T→∞T\to\infty it reduces to Vanilla SFT, which ignores it. Theorem 3.2 makes this trade-off precise: the teacher's error enters the risk as a bias term that shrinks as T grows, and there is an intermediate T ≠ 1 that strictly beats both maximum likelihood (Vanilla SFT) and the best knowledge-distillation run. The experiments agree: tuned KD at T=1 reaches +17.0%, below Ambient SFT at T=2.5 (+20.3%).

W1 (theory). Connection between our theory and the method

[...] Eq. (6) describes a generic contraction toward uniformity. Moreover, Theorem 3.2 does not specifically guarantee the benefit of high temperatures and relies on restrictive assumptions about teacher error. The connection between these results and the claimed reduction in overfitting and forgetting remains weak.

Indeed, Eq. (6) shows contraction towards the uniform distribution. However, the uniform distribution does not provide good supervision. This is consistent with the label smoothing experiment, where we show that if you regularize towards the uniform instead of the base model's distribution results deteriorate significantly. What instead our method attempts to do is to add the least amount of temperature such that the teacher bias becomes small. This is because there is a tension between keeping the knowledge of the teacher (information preservation) and reducing the bias (distribution alignment). Temperature controls exactly that trade-off. Theorem 3.2 exactly analyses this tradeoff for a simple model.

On the assumption about the teacher's error: Theorem 3.2 assumes that the teacher's error is of the same order as the statistical error of maximum likelihood, ∥B∥=O(n−1/2)\|B\| = O(n^{-1/2}). This is the regime in which the question of whether to use the teacher is non-trivial. If the teacher's error is much larger than the statistical error, the bias term dominates and the optimal λ moves toward 0, i.e., toward ignoring the teacher (Vanilla SFT). If it is much smaller, the frozen teacher alone is already good enough. The interesting case, where a weak teacher and a small dataset each carry information the other lacks, is exactly the case the theorem analyzes.

On overfitting and forgetting: Theorem 3.2 measures the estimation error on the target distribution, so it speaks to overfitting (through the variance term) rather than to forgetting, which it does not model. We agree that the evidence for reduced forgetting is empirical rather than theoretical. Our new evaluations of the base models provide this evidence (see our answer to Reviewer Potc and the Base rows of Table 2 in the revised submission): Vanilla SFT falls below the base model on 6 or 7 of the 7 non-IFEval benchmarks for every model, while Ambient SFT is above both the base model and Vanilla SFT on average for every model, and mostly preserves the base model's knowledge on MMLU, PopQA and TriviaQA.

W2 (novelty). Comparison with knowledge distillation and LwF

The method closely resembles existing distillation and regularization approaches. Its effectiveness also depends on additional hyperparameters, particularly temperature and mixing probability, whose appropriate values may vary substantially across teacher models and target domains. The paper provides no clear principle for selecting them [...].

We thank the Reviewer for these comments. We address novelty first and the choice of hyperparameters second.

AuthorsOpenReview ↗

Rebuttal part 3: Addressing novelty concerns and robustness to different hyperparameters

W2 (novelty). Comparison with knowledge distillation and LwF (continued)

Novelty. Our method is closest to two existing approaches: knowledge distillation toward the base model (KD), which is our objective at T=1, and Learning without Forgetting (LwF).

Comparison with KD. During the rebuttal period, we tuned KD toward the same frozen base model (T=1, cross-entropy weight pcp_c ∈ {0.25, 0.5, 0.75, 0.9}) and compared it with Ambient SFT (T=2.5) in every setting where we could run both (average relative improvement over Vanilla SFT):

  • Instruction tuning, Qwen3-1.7B (Table 2 setup): +17.0% for the best KD run vs +20.3% for Ambient SFT. At the same mixing weight (pcp_c=0.75), Ambient SFT beats KD on 7 of the 8 benchmarks.
  • Instruction tuning at larger scales, with KD at its best 1.7B setting: −0.9% vs +8.0% at 4B, +7.8% vs +10.2% at 8B, and −2.7% vs +8.7% at 14B.
  • Reasoning: KD is below Vanilla SFT on all six benchmarks (−7.3%), while Ambient SFT improves on it (+3.5%).
  • Domain adaptation: the temperature sweep of Figure 3a shows that validation loss falls from T=1 to its minimum at T=2.5.

This is what our theory predicts. In Theorem 3.2, the curvature of the matching penalty at T=1 equals the Fisher information of the data (H1=FH_1 = F), so KD shrinks the estimate hardest along exactly the directions the data already determines well. Changing the temperature changes the shape of the penalty: as T grows, the teacher adds curvature where the data supplies none. The theorem shows that some T ≠ 1 strictly beats the best T=1 run, as well as maximum likelihood (Vanilla SFT) and the frozen teacher.

Comparison with LwF. Mechanically, our approach is similar to LwF: in both cases, a model is adapted to learn a new capability and is regularized to stay close to its smoothed prior predictions on inputs from the new task. There are, however, some important differences:

  1. In LwF, the new task has a separate output layer (only trained for the new task alone). Distillation acts on the old task’s output head while the new task trains a separate head. The two losses interact only through shared features. In our case, all the parameters are shared.
  2. More importantly, the output space is shared. In our method, the “new task” (i.e. learning to follow instructions) competes directly with the old tasks as they share not only some parameters, but the same output space. The parameter sharing and output space sharing change the role of temperature (see next point).
  3. In LwF, the temperature (T=2) is a fixed softening choice. In Ambient SFT, both loss terms act on the same next-token distribution, so the temperature sets how much the student trusts the base model: T=1 is knowledge distillation toward the base model, and T→∞T\to\infty is Vanilla SFT. As shown above, this choice matters.
  4. LwF is motivated empirically. Our method comes with a theoretical analysis that studies (for a simplified model) the bias-variance trade-off the temperature controls (Theorem 3.2).
  5. Finally, our method is applied to modern LLMs and its gains are not limited to retaining existing capabilities. Our method improves the target-domain loss (Table 1), increases performance on MATH through learning to follow instructions, and improves reasoning benchmarks such as AIME24 (44.5 vs 39.8 for Vanilla SFT).

We have revised our submission to discuss LwF prominently in Section 3.1.

W2 (hyperparameters). Robustness and selection of the temperature

Hyperparameters. We used a single setting, T=2.5 and pcp_c=0.75, for all of our instruction-tuning and reasoning experiments (all the results of Table 2 and Figures 4 and 5). The only place where we ablated the temperature was our domain adaptation experiments. Even there, while different temperatures were optimal for different datasets, using T=2.5 instead of the optimal temperature leads to at most a 1.3% drop in validation loss (0.6% on average). Hence, T=2.5 works robustly across all our settings (domain adaptation, instruction tuning and reasoning).

That said, the temperature controls a bias-variance trade-off (Theorem 3.2), so in general we should not expect a single value to be optimal everywhere. The optimal temperature depends on:

  • How many datapoints we have. With infinite data, no supervision from the biased teacher is needed, so the temperature should go to infinity.
  • How good the teacher is. Powerful teachers call for lower temperatures and weaker teachers for higher ones.
  • How good the student is. When the teacher and the student have different sizes (as in some of our domain adaptation experiments), the optimal temperature also depends on the gap in capability between them.
Reviewer_uzhZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a method to prevent overfitting to a post-training dataset during Supervised Fine Tuning (SFT). The method regularizes post-training using the imperfect predictions of the frozen pre-trained model: during post-training, the loss is sampled stochastically between the standard cross-entropy SFT (samples with e.g. 75% probabability), and a new "temperature matching" loss for which we run a forward pass on the frozen pre-trained teacher to compute "teacher" logits, which are then used to compute KL divergence loss against the "student" being trained. The intention is to enable model alignment (main goal of SFT) and improve generalization, while reducing forgetting (a common pitfall of SFT).

The method is simple and practical. It needs only the fine-tuning data and the starting model, without pretraining data or a stronger external teacher. It adds no inference-time cost.

The empirical evidence is rather broad:the experiments cover multiple training settings, multiple model families, and sizes. Some gains are signifiant: for example, Qwen3-8B’s MATH score increases from 38.7 to 53.5.

The ablation are useful to interpret the method. Experiments on dataset size, training duration, temperature, and teacher capacity help identify when the method works. Figure 4 provides useful evidence that the method contributes to reducing overfitting.

Critical strengths

The method is simple and practical. It needs only the fine-tuning data and the starting model, without pre-training data or a stronger external teacher. It adds no inference-time cost.

The empirical evidence is rather broad: the experiments cover multiple training settings, multiple model families, and sizes. Some gains are significant: for example, Qwen3-8B’s MATH score increases from 38.7 to 53.5.

The ablation are useful to interpret the method. Experiments on dataset size, training duration, temperature, and teacher capacity help identify when the method works. Figure 4 provides useful evidence that the method contributes to reducing overfitting.

Critical weaknesses

The novelty of the method is very limited, particularly when compared against "Learning without Forgetting". The paper is acknowledged in "Related Work" section, but this comes a bit late in the appendix.

The paper does not provide a temperature setting that works robustly across datasets. This introduces another hyperparameter to tune, which could in fact increase training cost significantly.

The benchmarks (Table 2) and training trajectories (Figure 4) hint at reduced forgetting. However, they do not provide the pre-SFT baseline, it is thus difficult to conclude whether the method helps prevent forgetting.

Gains are not systematic: Table 2 shows lower IFEval scores for five of the six model configurations.

Questions for authors

Could you elaborate on how Ambient LLMs fundamentally differ from "Learning Without Forgetting"?

Can you suggest a way to systematically determine optimal temperature?

Could you provide evidence of reduce forgetting (compared to vanilla SFT)?

5 comments

We thank the Reviewer for their valuable time and input. We are glad the Reviewer found our method simple and practical, our empirical evidence broad, and our ablations useful. The Reviewer raised questions about the novelty of our method compared to Learning without Forgetting, the choice of the temperature, evidence of reduced forgetting, and the lower IFEval scores of Ambient SFT. We address all these points below, with new experimental evidence and clarifications. We have updated our paper to reflect those changes (with the changes denoted in blue).

Novelty compared to Learning without Forgetting

The novelty of the method is very limited, particularly when compared against "Learning without Forgetting". [...] Could you elaborate on how Ambient LLMs fundamentally differ from "Learning Without Forgetting"?

We agree with the Reviewer that the paper “Learning without Forgetting” (LfW) is relevant to our work and we have highlighted it more prominently in our revision. Mechanically, our approach is similar to the approach in this prior work: in both cases, a model is adapted to learn a new capability and is regularized to stay close to its smoothed prior predictions on inputs from the new task. There are, however, some important differences:

  1. In the prior work (LwF), the new task has a separate output layer (only trained for the new task alone). Distillation acts on the old task’s output head while the new task trains a separate head. The two losses interact only through shared features. In our case, all the parameters are shared.

  2. More importantly, the output space is shared. In our method, the “new task” (i.e. learning to follow instructions) competes directly with the old tasks as they share not only some parameters, but the same output space. The parameter sharing and output space sharing change the role of temperature (see next point).

  3. In LwF, the temperature (T=2) is a fixed softening choice. In Ambient SFT, both loss terms act on the same next-token distribution, so the temperature sets how much the student trusts the base model: T=1 is knowledge distillation toward the base model, and T→∞T\to\infty is vanilla SFT. Our experiments show this choice matters. Tuned distillation at T=1 gains +17.0% over Vanilla SFT, while Ambient SFT at T=2.5 gains +20.3%. The role of temperature is further shown in Figure 3a.

  4. LwF is motivated empirically. Our method comes with a theoretical analysis that studies (for a simplified model) the bias-variance tradeoff the temperature controls (Theorem 3.2).

  5. Finally, our method is applied to the modern LLMs and its gains are not limited to retaining existing capabilities. Our method improves the target-domain loss (Table 1), increases performance on MATH through learning to follow instructions and improves reasoning benchmarks such as AIME24 (44.5 vs 39.8 for Vanilla SFT).

We hope this answer clarifies the novelty of our approach. Despite the aforementioned differences, LwF is indeed highly relevant to our work and we have revised our submission to highlight it prominently.

Robustness of the temperature

The paper does not provide a temperature setting that works robustly across datasets. This introduces another hyperparameter to tune, which could in fact increase training cost significantly.

We want to clarify that we used a fixed temperature T=2.5 for all of our instruction following and reasoning experiments (see L422). This includes all the results of Table 2 and Figures 4, 5. The only place where we ablated the temperature was our domain adaptation experiments.

Even there, while it is true that different temperature values were optimal for different datasets (see Tables 3-7), using T=2.5 instead of the optimal temperature leads to at most 1.3% drop in validation loss (and only 0.6% drop on average). Hence, the value T=2.5 works robustly across all our settings (domain adaptation, instruction tuning and reasoning).

That said, as explained in our answer above, contrary to the LfW paper, the temperature controls a bias-variance trade-off and in general, we should not expect a single temperature value to be optimal across all settings.

New: evidence of reduced forgetting (base models before fine-tuning)

The benchmarks and training trajectories hint at reduced forgetting. However, they do not provide the pre-SFT baseline [...]. Could you provide evidence of reduce forgetting (compared to vanilla SFT)?

We thank the Reviewer for raising this excellent question. We evaluated for all models in Table 2 their base versions as well. The results are shown below:

ModelMethodMATHGSM8KHumanEval+MMLUPopQATriviaQAIFEvalBBHAvg. rel. Δ vs. SFT
Qwen3-1.7B-BaseBase (no SFT)33.853.578.465.613.945.637.954.2+9.0%
Vanilla SFT24.361.068.061.912.236.358.845.2—
Ambient SFT40.472.383.764.914.943.852.353.3+20.3%
Qwen3-4B-BaseBase (no SFT)43.581.991.275.618.460.445.570.5+5.7%
Vanilla SFT36.480.284.572.914.851.476.363.4—
Ambient SFT42.282.990.174.318.157.571.068.9+8.0%
Qwen3-8B-BaseBase (no SFT)48.585.893.178.824.870.756.276.1+9.1%
Vanilla SFT38.780.482.576.218.265.277.171.4—
Ambient SFT53.587.591.578.322.069.271.772.0+10.2%
Qwen3-14B-BaseBase (no SFT)51.281.593.782.228.977.068.076.9+6.3%
Vanilla SFT42.886.786.980.921.871.678.676.2—
Ambient SFT58.989.392.681.124.775.578.678.5+8.7%

A few findings:

  1. Both Ambient SFT and Vanilla SFT achieve large improvements in IFEval to the Base model.
  2. Vanilla SFT hurts the base model outside IFEval. It falls below the base on 6 or 7 of the other 7 tasks for every model, and the base model beats Vanilla SFT on average for every model. This is evidence for catastrophic forgetting.
  3. On the other hand, Ambient SFT consistently outperforms both Base model and Vanilla SFT on average across all models.
  4. Ambient SFT mostly preserves base knowledge on MMLU, PopQA and TriviaQA.
  5. New capabilities: Ambient SFT improves on the base model in math, where Vanilla SFT does not. MATH is above the base at 1.7B (+6.6), 8B (+5.0), 14B (+7.7). GSM8K is also above the base for every model.

We have included these results in the revised submission (Base rows in Table 2 and the results paragraph of Section 4.3). We believe that these results show evidence for both development of new capabilities (math and instruction following) and preservation of existing knowledge through Ambient SFT.

New: IFEval scores and per-source temperatures

Gains are not systematic: Table 2 shows lower IFEval scores for five of the six model configurations.

We agree with the Reviewer: Ambient SFT trails Vanilla SFT on IFEval for most models with the difference becoming smaller with scale, while it improves on the other seven benchmarks. During the rebuttal period, we found a simple fix: setting the temperature separately for each of the 21 source datasets that make up Dolci-Instruct. For each source, we measure how well the base model fits it (the gap between the base model's cross-entropy and its entropy on that source's tokens) and raise the temperature where the fit is poor, so the student relies less on the base model there. Sources the base model already fits (math, code, tool use) keep T=2.5, while poorly fit sources (e.g., CoCoNot, WildChat, Precise IF) get a higher temperature. Per-source temperatures remove the IFEval drop at every model size we tested: IFEval comes back to the level of Vanilla SFT (e.g., at Qwen3-8B, 78.6 vs 77.1 for Vanilla SFT and 71.7 for Ambient SFT with a single temperature). The gains of Ambient SFT on the other benchmarks are kept, and the average improvement over Vanilla SFT is the same as with a single temperature or slightly higher. We have added these results to the revised submission (Table 3, with the per-source rule in Appendix D.5).

(Table in next comment)

The full results are shown below:

ModelMethodIFEvalMATHGSM8KHumanEval+MMLUPopQATriviaQABBHAvg. rel. Δ vs. SFT
Qwen3-1.7BVanilla SFT58.824.361.068.061.912.236.345.2—
Ambient SFT52.340.472.383.764.914.943.853.3+20.3%
Ambient SFT (per-source T)58.240.973.283.164.514.642.853.2+21.1%
Qwen3-4BVanilla SFT76.336.480.284.572.914.851.463.4—
Ambient SFT71.042.282.990.174.318.157.568.9+8.0%
Ambient SFT (per-source T)77.143.485.390.673.716.755.669.8+8.2%
Qwen3-8BVanilla SFT77.138.780.482.576.218.265.271.4—
Ambient SFT71.753.587.591.578.322.069.272.0+10.2%
Ambient SFT (per-source T)78.654.585.292.777.220.768.171.5+10.1%

Selecting the temperature

Can you suggest a way to systematically determine optimal temperature?

The honest answer is that we do not have such a method. This is because the optimal temperature value depends on a lot of different factors:

  1. How many datapoints you have. With infinite datapoints, no supervision from the biased teacher should be needed, so temperature should be set to infinity.
  2. How good the teacher is. For powerful teachers, we should expect lower temperature values and for less powerful teachers, higher temperature values.
  3. How good the student is. In cases whether we use different model sizes for teachers and students (as in some of our domain adaptation experiments), the optimal temperature also depends on the gap of capabilities between student and teacher.

That said, (i) setting T=2.5 seems to be a very robust choice (see above) and (ii) the heuristic of setting temperature based on the gap between the base model's cross-entropy and its entropy on that source's tokens seems to work pretty well (as shown above).

To further check this idea, we measure the Spearman rank correlation between the teacher's gap on each domain and the optimal T for our domain adaptation experiments as well. The correlation is ρ = 0.82 for the 1.5B model and ρ = 0.36 for the 124M model. With only five domains these estimates are noisy, but the sign is consistent: domains the base model fits worse prefer a higher temperature.

Reviewer_PotcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Ambient SFT, a novel framework designed to alleviate catastrophic forgetting in post training. It proposes a regularization mechanism that forces the model being fine-tuned to match the high-temperature probability distribution of its frozen pre-trained teacher to preserve foundational knowledge while still efficiently adapting to new target data. Ambient SFT is mathematically proven to offer superior statistical efficiency and empirically outperforms standard SFT.

Critical strengths
  1. The insights in Theorem 3.2 are interesting. The author analyzes the trade-off between teacher bias and estimation variance. It provides a sound connection between temperature matching and learning from teacher-generated samples. Under realizability and full support, its global optimum recovers the original teacher distribution at any finite positive temperature, clarifying why smoothing preserves identifiability.

  2. The experimental results are sufficient. Table 2 reports improvements across Qwen-3, OLMo-3, and Llama, with particularly large gains in the smallest-data regime.

Critical weaknesses

For Theoretical Results:

  1. There is a gap between the theoretical results and practical solutions. Theorem 3.1 uses a different prefix distribution from the algorithm. It averages over prefixes generated autoregressively by the tempered teacher, while Eq. 4 uses fine-tuning-data prefixes. It does not directly equate the actual objective with training on complete teacher rollouts.

  2. Theorem 3.1 takes the exact global population optimum at finite temperature. It provides no sample-complexity, approximate-recovery, or optimization guarantee as high-temperature distributions approach uniformity, which means it does not establish finite-sample data efficiency.

For Practical Solution:

  1. The instruction-tuning results inTable 2 lack baselines without finetuning. Table 2 reports several general benchmarks after vanilla and Ambient SFT, but does not report scores for the same base models before finetuning. It cannot quantify each method's change from the common starting point or distinguish preservation of existing capabilities from better generalization.

  2. Lack of existing baselines for instruction and reasoning fine-tuning. Table 2 only contains the results for Vanillia SFT, and comparisons with parameter anchoring such as L2-SP [1] and a well-tuned SFT regularizer such as NEFTune [2] would clarify whether the gains are specific to temperature matching.

Reference:

  1. Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit Inductive Bias for Transfer Learning with Convolutional Networks. ICML, 2018. (L2-SP)
  2. Neel Jain et al. NEFTune: Noisy Embeddings Improve Instruction Finetuning. ICLR, 2024.
Questions for authors

Please check my weakness section.

4 comments

We thank the Reviewer for their valuable input. We are glad the Reviewer found the insights of Theorem 3.2 interesting and our experimental results strong. The Reviewer raised questions about the prefix distribution in Theorem 3.1, the finite-sample guarantees of our theory, the missing scores of the base models, and the missing comparisons with baselines such as L2-SP and NEFTune. We address all these points below, with new experimental evidence and clarifications. We have updated our paper to reflect those changes (with the changes denoted in blue).

Theorem 3.1 and the prefix distribution

[...] Theorem 3.1 uses a different prefix distribution from the algorithm. It averages over prefixes generated autoregressively by the tempered teacher, while Eq. 4 uses fine-tuning-data prefixes. [...]

We agree with the Reviewer that the way that Theorem 3.1 is stated it analyzes prefixes coming from the tempered teacher. But:

  1. The proof never uses the prefix distribution and naturally generalizes to any prefix distribution. For any prefix distribution μ, the population Temperature Matching loss equals, up to a constant, the tempered cross-entropy on next tokens drawn from the tempered teacher at prefixes drawn from μ. Changing μ also does not change what Temperature Matching converges to: on every context that μ covers, the minimizer matches the teacher at all temperatures, i.e. its logits equal the teacher's up to a constant. μ only decides on which contexts this is enforced.
  2. Theorem 3.2, the one that analyzes the behavior of our Temperature Matching algorithm, already uses fine-tuning-data prefixes (lines 258-261).

In the revised submission, we have written Theorem 3.1 in its general form (for general prefix distributions), with the prefixes coming from the tempered distribution as a corollary (Corollary A.1). We thank the Reviewer for raising this.

Finite-sample guarantees

Theorem 3.1 [...] provides no sample-complexity guarantee. [...] It does not establish finite-sample data efficiency.

We agree that Theorem 3.1 is a population statement; it is an identifiability result showing that the temperature-matching term is unbiased (its minimizer is the teacher, up to a per-context constant, at every finite T). Finite-sample efficiency is the subject of Theorem 3.2, which gives the estimation risk in a log-linear model as bias$^2$ plus an explicit 1/n1/n variance term, and shows that some (λ, T) with T ≠ 1 strictly beats maximum likelihood, the frozen teacher, and the best knowledge-distillation run. Theorem 3.2 also addresses the high-temperature regime directly: as T grows, the matching curvature behaves as Σ0⊗G/T2\Sigma_0\otimes G/T^2, which shrinks but never degenerates, while the teacher's bias decays as Δ2/(8T2)\Delta^2/(8T^2) (Eq. 6). The temperature therefore trades bias against added curvature, giving an interior optimum, consistent with the empirical optimum T = 2.5 (Fig. 3a), far from uniform. Finally, Eq. 4 matches the full tempered distribution rather than sampled tokens, so there is no sampling noise from near-uniform distributions.

New: scores of the base models before fine-tuning

Instruction-tuning results lack baselines without finetuning. [...] cannot distinguish preservation of existing capabilities from better generalization.

We thank the Reviewer for raising this excellent question. We evaluated for all models in Table 2 their base versions.

ModelMethodMATHGSM8KHumanEval+MMLUPopQATriviaQAIFEvalBBHAvg. rel. Δ vs. SFT
Qwen3-1.7B-BaseBase (no SFT)33.853.578.465.613.945.637.954.2+9.0%
Vanilla SFT24.361.068.061.912.236.358.845.2—
Ambient SFT40.472.383.764.914.943.852.353.3+20.3%
Qwen3-4B-BaseBase (no SFT)43.581.991.275.618.460.445.570.5+5.7%
Vanilla SFT36.480.284.572.914.851.476.363.4—
Ambient SFT42.282.990.174.318.157.571.068.9+8.0%
Qwen3-8B-BaseBase (no SFT)48.585.893.178.824.870.756.276.1+9.1%
Vanilla SFT38.780.482.576.218.265.277.171.4—
Ambient SFT53.587.591.578.322.069.271.772.0+10.2%
Qwen3-14B-BaseBase (no SFT)51.281.593.782.228.977.068.076.9+6.3%
Vanilla SFT42.886.786.980.921.871.678.676.2—
Ambient SFT58.989.392.681.124.775.578.678.5+8.7%

A few findings:

  1. Both Ambient SFT and Vanilla SFT achieve large improvements in IFEval to the Base model.
  2. Vanilla SFT hurts the base model outside IFEval. It falls below the base on 6 or 7 of the other 7 tasks for every model, and the base model beats Vanilla SFT on average for every model. This is evidence for catastrophic forgetting.
  3. On the other hand, Ambient SFT consistently outperforms both Base model and Vanilla SFT on average across all models.
  4. Ambient SFT mostly preserves base knowledge on MMLU, PopQA and TriviaQA.
  5. New capabilities: Ambient SFT improves on the base model in math, where Vanilla SFT does not. MATH is above the base at 1.7B (+6.6), 8B (+5.0), 14B (+7.7). GSM8K is also above the base for every model.

We have included these results in the revised submission (Base rows in Table 2 and the results paragraph of Section 4.3). We believe that these results show evidence for both development of new capabilities (math and instruction following) and preservation of existing knowledge through Ambient SFT.

New: comparison with existing baselines (instruction tuning)

Lack of existing baselines for instruction and reasoning fine-tuning. [...] comparisons with parameter anchoring such as L2-SP and a well-tuned SFT regularizer such as NEFTune would clarify whether the gains are specific to temperature matching.

We thank the Reviewer for this excellent suggestion. We agree that more baselines were needed to strengthen our submission. To that end, we add the following baselines:

  • L2-SP: penalizes the distance of the weights from the pre-trained weights [1]. We sweep the penalty strength α ∈ {0.001, 0.01, 0.1, 1, 10}.
  • NEFTune: adds uniform noise to the input embeddings during fine-tuning [2]. We sweep the noise scale α ∈ {5, 10, 15}, the values used in [2].
  • Knowledge Distillation: Vanilla SFT plus a KL term to the frozen base model at temperature T=1, i.e., our objective with T=1. We sweep the cross-entropy weight pcp_c ∈ {0.25, 0.5, 0.75, 0.9}.
  • Label smoothing: mixes the one-hot target with the uniform distribution with weight ε. We sweep ε ∈ {0.05, 0.1, 0.2, 0.4}.

We make sure that the best settings of all baselines lie inside their search grids so the baselines are not limited by the range we searched. Optimal parameters are: tuned knowledge distillation (pcp_c=0.5), L2-SP (α=1), NEFTune (α=10) and label smoothing (ε=0.2).

Instruction Tuning results (Table 2 setup, Qwen3-1.7B) are shown below:

MethodSettingMATHGSM8KHumanEval+MMLUPopQATriviaQAIFEvalBBHAvg. rel. Δ vs. SFT
Base (no SFT)—33.853.578.465.613.945.637.954.2+9.0%
Vanilla SFT—24.361.068.061.912.236.358.845.2—
NEFTuneα=523.760.066.861.611.737.860.846.8+0.1%
α=1019.758.969.262.212.438.465.648.6+0.8%
α=1518.361.971.762.211.839.063.048.7+0.2%
Label smoothingε=0.0519.457.968.861.112.036.059.747.5-2.7%
ε=0.125.361.464.262.310.037.357.744.7-2.3%
ε=0.225.462.964.360.912.136.455.147.0-0.3%
ε=0.423.562.556.861.012.034.954.246.9-3.5%
L2-SPα=0.00120.256.072.561.412.237.158.445.8-2.1%
α=0.0121.657.984.563.815.343.154.949.0+7.1%
α=0.130.372.683.065.014.845.350.152.7+14.8%
α=135.072.383.264.914.545.641.852.8+15.3%
α=1035.169.481.865.213.945.638.853.7+13.5%
Tuned KD (T=1)pcp_c=0.2531.872.980.565.015.644.443.155.2+14.9%
pcp_c=0.535.171.581.364.815.743.849.252.8+17.0%
pcp_c=0.7537.567.979.064.313.642.655.552.4+15.6%
pcp_c=0.932.463.878.462.913.340.961.951.7+12.0%
Ambient SFTT=2.5, pcp_c=0.7540.472.383.764.914.943.852.353.3+20.3%

As shown, Ambient SFT outperforms all these baselines. We have added these results to the revised submission (Table 5, with the full sweeps in Table 21, Appendix D.4).

A few findings:

  • The strongest baseline is tuned knowledge distillation. It uses the same base-model teacher as Ambient SFT, with the temperature fixed to T=1. Its best setting (pcp_c=0.5) reaches +17.0%, and at Ambient's mixing weight (pcp_c=0.75) it reaches +15.6%, while Ambient SFT with T=2.5 reaches +20.3%. As predicted by our theory, the optimal temperature is not T=1, and raising it improves the result.
  • The second best baseline is L2-SP with α=1 (+15.3%). It outperforms Vanilla SFT, but it does so by staying close to the base model: its knowledge and reasoning scores match the base model (MATH 35.0 vs 33.8, TriviaQA 45.6 vs 45.6, BBH 52.8 vs 54.2), and it gives up what SFT is meant to teach. Its IFEval drops monotonically with α, from 58.4 at α=0.001 to 38.8 at α=10, back to the base model's 37.9. Ambient SFT is above L2-SP on MATH (40.4) and IFEval (52.3) at the same time.
  • Label smoothing never improves on Vanilla SFT (-3.5% to -0.3%). Label smoothing also trains on soft targets, but it smooths toward the uniform distribution. The gains of Ambient SFT therefore come from smoothing toward the base model's tempered distribution, not from softer targets alone.
  • NEFTune changes Vanilla SFT very little (+0.1% to +0.8%): it raises IFEval (up to 65.6) but lowers MATH (down to 18.3).

New: baselines on reasoning

Finally, we take the two strongest baselines (Tuned KD and L2-SP) and we apply them for the reasoning experiments. Results are shown below:

MethodGSM8KAIME24AIME25GPQAIFEvalHumanEval+Avg. rel. Δ vs. SFT
Vanilla SFT94.239.833.239.334.675.6—
Tuned KD (T=1)90.736.230.833.034.070.7-7.3%
L2-SP (α=1)81.79.914.028.633.840.3-37.1%
Ambient SFT93.044.534.140.036.076.9+3.5%

As shown, the baselines fail to improve upon Vanilla SFT on Reasoning, while Ambient leads to an average 3.5% increase across the scores. We have added these results to the revised submission (Table 6).

New: per-source temperatures

Finally, during the rebuttal period, we found a simple refinement of Ambient SFT: setting the temperature separately for each of the 21 source datasets that make up Dolci-Instruct. For each source, we measure how well the base model fits it (the gap between the base model's cross-entropy and its entropy on that source's tokens) and raise the temperature where the fit is poor: Ts=2.5 (1+max⁡(gs,0)/0.1)T_s = 2.5\,(1 + \max(g_s, 0)/0.1). Sources the base model already fits (math, code, tool use) keep T=2.5, while poorly fit sources (e.g., CoCoNot, WildChat, Precise IF) get a higher temperature, so the student relies less on the base model there. Results are shown below:

ModelMethodIFEvalMATHGSM8KHumanEval+MMLUPopQATriviaQABBHAvg. rel. Δ vs. SFT
Qwen3-1.7BVanilla SFT58.824.361.068.061.912.236.345.2—
Ambient SFT52.340.472.383.764.914.943.853.3+20.3%
Ambient SFT (per-source T)58.240.973.283.164.514.642.853.2+21.1%
Qwen3-4BVanilla SFT76.336.480.284.572.914.851.463.4—
Ambient SFT71.042.282.990.174.318.157.568.9+8.0%
Ambient SFT (per-source T)77.143.485.390.673.716.755.669.8+8.2%
Qwen3-8BVanilla SFT77.138.780.482.576.218.265.271.4—
Ambient SFT71.753.587.591.578.322.069.272.0+10.2%
Ambient SFT (per-source T)78.654.585.292.777.220.768.171.5+10.1%

As shown, per-source temperatures remove the IFEval drop of Ambient SFT at every model size (58.2 / 77.1 / 78.6, vs 52.3 / 71.0 / 71.7 for Ambient SFT and 58.8 / 76.3 / 77.1 for Vanilla SFT), while keeping the gains of Ambient SFT on the other tasks. The average improves at 1.7B (+21.1% vs +20.3%) and 4B (+8.2% vs +8.0%) and is unchanged at 8B (+10.1% vs +10.2%). We have added these results to the revised submission (Table 3, with the per-source rule in Appendix D.5).

We hope that our rebuttal addresses the concerns of the Reviewer. If so, we would appreciate it if the Reviewer considered raising their score in light of this new evidence. Once again, we thank the Reviewer for their input that helped us significantly improve our work.

AuthorsOpenReview ↗

Looking forward to your response

Dear Reviewer, as the discussion period is approaching its end, we would love to know if the rebuttal addresses your concerns. We remain at your availability and we thank you once again for your time and feedback.

Comments on the paper

1 comment

Summary of the rebuttal for the AC and the Reviewers

We thank the AC and the Reviewers for their time and feedback. During the discussion period we ran a substantial set of new experiments and revised the paper (changes in blue). Below, we list each concern raised by the Reviewers and the experiment or revision that addresses it.

1. Concern: no pre-SFT baseline, so it is unclear whether Ambient SFT reduces forgetting (Potc, uzhZ). We evaluated the base version of every model in Table 2 on all eight benchmarks.

  • Vanilla SFT falls below the base model on 6 or 7 of the 7 non-IFEval benchmarks for every model, and the base model beats Vanilla SFT on average for every model. This is direct evidence of catastrophic forgetting.
  • Ambient SFT is above both the base model and Vanilla SFT on average for every model. It mostly preserves knowledge (MMLU, PopQA, TriviaQA) and adds capabilities: MATH is above the base at 1.7B (+6.6), 8B (+5.0) and 14B (+7.7), and GSM8K is above the base for every model.

2. Concern: missing comparisons with simpler regularizers (label smoothing, token perturbation, parameter anchoring, tuned KD), so the gains may not be specific to Temperature Matching (Potc, nhXK). We added all four baselines, each tuned over a grid whose best value lies inside the grid: label smoothing, NEFTune (token perturbation), L2-SP (parameter anchoring) and tuned knowledge distillation toward the same frozen base model (our objective at T=1). Average relative improvement over Vanilla SFT:

Instruction tuning (Qwen3-1.7B)Reasoning (OLMo-3-7B)
Label smoothing (best)−0.3%
NEFTune (best)+0.8%
L2-SP (best)+15.3%−37.1%
Tuned KD, T=1 (best)+17.0%−7.3%
Ambient SFT+20.3%+3.5%

Tuned KD also stays below Ambient SFT at 4B, 8B and 14B. Ambient SFT is the only method that improves on Vanilla SFT for reasoning. Reviewer nhXK confirmed that these comparisons address the concern about missing baselines.

3. Concern: gains are not systematic, since Ambient SFT has lower IFEval than Vanilla SFT in most configurations (uzhZ). Setting the temperature per source dataset, higher where the base model fits the data poorly, removes the IFEval drop at every size we tested (58.2 / 77.1 / 78.6 vs. 58.8 / 76.3 / 77.1 for Vanilla SFT at 1.7B / 4B / 8B) and keeps the gains elsewhere (+21.1% / +8.2% / +10.1% over Vanilla SFT).

4. Concern: no principled way to choose the temperature, which adds a hyperparameter to tune (uzhZ, nhXK).

  • A single setting (T=2.5, p_c=0.75) is used for all instruction-tuning and reasoning experiments. In domain adaptation, using T=2.5 instead of the per-domain optimum costs at most 1.3% in validation loss.
  • The per-source rule above sets the temperature from how well the base model fits the data. In domain adaptation, this fit correlates with the optimal temperature (Spearman ρ = 0.82 at 1.5B).

5. Concern: the extra compute of the teacher, and whether the gains come from it (nhXK). Ambient SFT costs about 1.1× Vanilla SFT and adds about 12.5% to parameter memory. At matched compute (Qwen3-1.7B, Table 2 setup), Vanilla SFT with an extra epoch gets slightly worse, while Ambient SFT is +24.0% better and wins 7 of 8 benchmarks. With about 35% less compute, Ambient SFT is still +20.9% better.

6. Concern: early stopping may be enough to limit forgetting (nhXK). In the epoch sweep (OLMo-3-7B), vanilla SFT trades instruction following for knowledge: from 2 to 6 epochs it gains 9.8 points on IFEval and loses on every other benchmark. Ambient SFT at 4 epochs beats the early-stopped vanilla model (its best checkpoint on all seven non-IFEval benchmarks) on all 8 benchmarks, IFEval included.

7. Concern: gap between the theory and the algorithm (Potc, nhXK). Theorem 3.1 now holds for any prefix distribution, including fine-tuning-data prefixes, which is the setting of the algorithm. Finite-sample efficiency is the subject of Theorem 3.2, which shows that some T ≠ 1 strictly beats maximum likelihood, the frozen teacher and the best T=1 distillation run.

8. Concern: limited novelty compared to knowledge distillation and Learning without Forgetting (uzhZ, nhXK). We added a section positioning our work against both and discuss the differences: a weaker teacher (the model before fine-tuning), a shared output space, and the temperature as a bias-variance knob for trust in the teacher. Empirically, tuned KD (T=1) is below Ambient SFT in every setting we ran (item 2).

We believe these additions address the Reviewers' concerns, and we thank them again for suggestions that strengthened the paper. Unfortunately, at the time of this writing, only Reviewer nhXK has acknowledged the rebuttal. The Reviewer informed us that their concerns about missing baselines are addressed and their remaining concern is the extra computational cost. Our compute-matched experiment address this concern directly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.