acceptodds
Under review as a conference paper at ICLR 2027

TOFU: Fixing Supervised Fine-Tuning induced diversity collapse in Large Language Models

Abstract

Supervised Fine-Tuning (SFT) is essential for aligning Large Language Models (LLMs) with user intent, yet it is believed to suppress generative diversity. Although this reduction is frequently referenced, formal empirical testing of the phenomenon remains limited. The expressiveness of LLMs by itself was addressed by multiple prior methods. Their varying perspectives suggest that deeper investigation could yield further improvements. In this study, we attribute the decline to two primary drivers: the neglect of low-frequency patterns within fine-tuning datasets and the forgetting of preexisting knowledge. Motivated by our theoretical analysis, we develop Tempered Focal (TOFU) loss, a novel objective that addresses both stated challenges simultaneously. Our extensive evaluation confirms at scale that generation breadth narrows after SFT and strengthens the hypothesis explaining this effect. Across multiple models and benchmarks, we demonstrate that TOFU enhances output diversity while preserving high response quality, offering a principled approach to SFT.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, SFT, Instruct tuning, Diversity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AAtGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reduction in generative diversity caused by Supervised Fine-Tuning (SFT) in large language models. The authors identify two primary factors contributing to this issue: the underrepresentation of low-frequency patterns in fine-tuning datasets and the forgetting of pretrained knowledge. To address these challenges, they propose Tempered Focal (TOFU) loss, a novel training objective designed to preserve generation diversity while maintaining response quality. Through theoretical analysis and extensive experiments across multiple models and benchmarks, the authors demonstrate that TOFU effectively mitigates diversity degradation after SFT, offering a promising approach to improving LLM fine-tuning.

Critical strengths
  1. The motivation of this work is interesting, as it addresses the important issue of reduced generative diversity during the fine-tuning of large language models.

  2. This paper is clear in writing and easy to follow.

  3. In the experimental results, this paper improves results in MATH500, Minerva, GSM8K and OlympiadBench compared with the baselines.

Critical weaknesses
  1. The effectiveness of the proposed method on modern long-context tasks deserves further investigation. For example, modern LLMs have demonstrated strong reasoning capabilities. However, this work primarily focuses on relatively easy math tasks and lacks further exploration of more challenging long-context / response tasks, particularly those requiring complex agent loop / reasoning. This can be written in the limitations.

  2. The experiments primarily focus on relatively older models, such as Qwen2.5, and lack sufficient evidence demonstrating the effectiveness of the proposed method on more recent frontier models.

Questions for authors

N/A

3 comments

Thank you very much for your feedback!

Limitations and Future Work

We agree that it is interesting to investigate the performance of TOFU in more complex systems beyond a single model, e.g. agentic loops. We see it as an opportunity for future work rather than a limitation. We hope that the community possessing larger computational resources will extend our findings to a greater scale.

Models Selection

Our choice of models was driven by the need for robust, highly capable open-source backbones that enable controlled, reproducible comparisons. Additionally, we wanted to have base models unaffected by post-training procedures, a criterion that many recent open-source releases do not satisfy.

We are happy to participate further in the discussion! Please let us know if any points remain unclear or if there are specific details we can provide to address your concerns and merit a score increase.

Dear reviewer AAtG, thank you again for your time and effort in reviewing our work! As the discussion period is ending soon, please do let us know if you have any additional questions or concerns with the paper. We'd be happy to address them.

Thank you for the detailed rebuttal and clarifications. I have carefully reviewed your responses and the additional explanations. While I appreciate the efforts made to address my concerns, my overall assessment remains unchanged, and I will maintain my original rating.

Reviewer_pVBiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper says SFT makes LLM outputs less diverse because the model ignores rare patterns in the SFT data ("ignorance") and forgets what it knew before SFT ("forgetting"). It shows that GEM is a temperature-scaled cross-entropy in gradient terms. It then proposes TOFU, which scales this gradient by the focal-loss weight. On six general LLMs, TOFU gives the best Self-BLEU among the methods that keep quality. On math and code, it solves more problems when many samples are allowed, but its single-sample accuracy is lower than CE's in almost all cases.

Critical strengths
  1. Simple method, useful theory. Theorem 3.1 is a clean and useful result. TOFU then adds only one weight per token, so it is easy to use.
  2. Broad, consistent results. TOFU beats GEM on Self-BLEU in all 24 cells of Tables D.1–D.2, and its judge score is never more than 0.1 below GEM's. It has the best math coverage in all 16 cells of Table 1. Its settings were chosen once and kept for all models.
Critical weaknesses
  1. The two causes are not tested directly. The only direct check is Figure D.2, which uses one prompt and does not name the model.

    • Rare patterns. "Pattern" is not clearly defined, and the paper never measures how often patterns appear in the SFT data. The focal weight depends only on the token's probability (Prop. 3.2). But low probability is not the same as rare: it can also mean that many words fit.

    • Forgetting. Table D.5 (ARC/MMLU) has no base-model scores, and TOFU and CE are within about one point (Pythia aside). So these numbers give no evidence that TOFU forgets less.

    So "This validates our hypothesis" (l. 411) is too strong. Please test the causes, or present them as a hypothesis.

  2. The gain over GEM is not measured in an equal setting. The focal weight is TOFU's only new part: with γ=0\gamma = 0, TOFU matches GEM at the same β\beta.

    • TOFU uses β=0.8\beta = 0.8. The original GEM paper uses β=0.7\beta = 0.7, and this paper does not say which β\beta its GEM runs used.

    • TOFU was tuned on ARC and NoveltyBench, which are also test benchmarks (App. C). No tuning is reported for GEM.

    • In math, CE is sampled at T=1.0T = 1.0 but GEM and TOFU at T=0.3T = 0.3, for stability (App. B.5). This mixes the effect of the loss with the effect of temperature.

    • I found no repeated training runs, and some gaps are small (NoveltyBench Distinct: 0.0 to 0.3 points, Table D.3).

Overall: The gains look real and useful, so I see the paper as borderline. I would raise my score if the authors test the two causes (or present them as a hypothesis) and compare with GEM in an equal setting.

Questions for authors
  1. Direct testing of causes: Can you test the two causes directly? For example, make some clearly defined patterns rare in the SFT data on purpose, and check whether TOFU learns them better than CE. For forgetting, please add base-model ARC/MMLU scores, and count the questions the base model gets right but each SFT model gets wrong.

  2. Controlled comparison with GEM: Can you add GEM at β=0.8\beta = 0.8 (TOFU with γ=0\gamma = 0), compare all methods at the same sampling temperature, and repeat training with 2–3 seeds for one or two models?

  3. Single-sample accuracy drops and majority voting: Single-sample accuracy drops. On GSM8K with Qwen-2.5-Math-7B, mean success falls from 60.1 (CE) to 46.8 (Table 1), and HumanEval pass@1 falls by 5.7 to 9.2 points (Table D.9). How does TOFU do with majority voting at the same number of samples?

  4. Inconsistent ARC scores for Mistral-12B: For Mistral-12B, the chosen TOFU setting gets ARC 75.6 in Table C.1 but 73.9 in Table D.5 (Alpaca). Were these runs set up the same way? This gap is larger than the 0.2-point lead used to pick that setting.

  5. Clarification on weight distribution (Section 3): Section 3 says the naive combination (Cor. 3.3) puts too much weight on very small probabilities (l. 231–235). But gg peaks at a small pp (about 0.09 for γ=3\gamma = 3) and falls back to 1 as pp goes to 0 (Prop. 3.2, Figure 2). So below the peak, a smaller pp gets a smaller weight. Could you clarify this?

  6. ASR and HarmBench findings: On Malicious Instruct, ASR rises from CE to TOFU for both safe models (Table D.6), and HarmBench is mixed. So "does not lead to an increase in ASR" (l. 446–447) seems too strong.

3 comments

Thank you very much for your thoughtful feedback! Please find our answer below.

Direct testing of causes

We view the attribution of diversity collapse to forgetting and the neglect of rare samples as the working hypothesis upon which our method is built. Therefore, we agree that our method's performance does not serve as strict proof of the hypothesis itself.

In addition, originally describing the ARC/MMLU results as “no evidence of catastrophic forgetting” was imprecise, as base model performance on these benchmarks is quite low to begin with. A more accurate phrasing would be “no degradation relative to CE”.

However, we do observe evidence that TOFU mitigates forgetting in Figure D.2, where TOFU “forgets” the answer distribution less than CE.

In the revised manuscript, we will adjust our wording to convey that while our empirical results (for Focal Loss and GEM) reinforce our working assumption, they should not be taken as a complete proof of it.

Controlled comparison with GEM

In our main experiments with GEM we used a default value for β=0.7\beta=0.7 (l.366-368), found through ablations in its original paper. Please find the results of using three different training seeds and training GEM with β=0.8\beta=0.8 in Table R.2. As one can see, GEM with β=0.8\beta=0.8 performs slightly worse than default GEM.

Table R.2, The evaluation is conducted on Alpaca Eval, averaged on 3 different training seeds

MethodLlama-3.1-8BOLMo-2-13B
GEM β=0.7\beta=0.727.9 ± 1.9, 4.1 ± 0.129.4 ± 3.0, 4.1 ± 0.1
GEM β=0.8\beta=0.831.1 ± 1.4, 4.1 ± 0.130.8 ± 3.2, 4.1 ± 0.1
TOFU21.7 ± 2.9, 4.0 ± 0.124.6 ± 2.8, 4.1 ± 0.1

Single-sample accuracy drops and majority voting

With the majority voting, TOFU performs on par with other methods on the MATH500 benchmark, as demonstrated by the results in Table R.3.

Table R.3, Majority Voting for k=16 on MATH500.

MethodMathstral-7BQwen2.5-1.5BQwen2.5-7BDeepSeek-7B
CE59.864.268.650.2
GEM60.864.168.447.2
TOFU61.664.568.447.4

Curious to investigate whether the drop in mean accuracy has a negative effect on subsequent RL training stages or if high coverage is sufficient by itself, we conducted GRPO on the models fine-tuned with CE, GEM, and TOFU. As shown in Table R.4, models fine-tuned with TOFU achieve greater gains in both mean accuracy (with the exception of the DeepSeek model) and coverage compared to CE and GEM.

Inconsistent ARC scores for Mistral-12B

Thank you for spotting the inconsistencies! Ablation table C.1 contains the correct values, and the only value affected by a misprint in Table D.5 is for Mistral-12B, tuned on Alpaca with TOFU and tested on ARC.

Clarification on weight distribution (Section 3)

While probabilities to the left of the peak (argmax g(p,γ)\mathrm{argmax} \ g(p,\gamma)) receive smaller weights under temperature scaling with β<1\beta < 1, those to the right, up to a threshold defined by the specific shape of the p(x)p(x) distribution, move closer to the peak and are assigned larger weights. Because the peak itself is not much greater than 1 and g(p,γ)g(p,\gamma) peaks at a low value, we expect hard examples to be overemphasized on average. We agree that this argument needs refinement and will make it more precise in the revised manuscript.

ASR and HarmBench findings

We agree that the dispersion present in the ASR results measured on Malicious Instruct and HarmBench, makes more accurate statement to be that TOFU does not lead to noticeable increase in ASR on average. That statement is reinforced by the fact that observed dispersion is way smaller than the difference between safe and unsafe methods.

We hope these answers clarify your concerns, and we would be happy to answer any further questions or provide additional details if needed!

Table R.4, Math performance after GRPO.

BenchMethodQwen-Math-1.5BQwen-Math-7BDeepSeek-Math-7BMathstral-7B
MATH500CE+GRPO81.2 (56.0)83.9 (62.8)73.1 (35.6)77.6 (50.6)
GEM+GRPO81.0 (56.8)84.8 (62.8)72.0 (36.8)79.2 (50.9)
TOFU+GRPO84.5 (57.3)85.5 (65.9)74.5 (37.0)82.1 (50.6)
MinervaCE+GRPO32.3 (15.2)34.2 (19.4)35.2 (13.4)33.6 (14.5)
GEM+GRPO36.7 (15.0)37.8 (19.2)38.6 (14.2)39.3 (15.2)
TOFU+GRPO39.3 (15.8)43.8 (22.4)42.7 (15.9)39.7 (15.6)
GSM8KCE+GRPO81.0 (62.6)81.4 (62.9)77.7 (52.1)78.0 (61.4)
GEM+GRPO79.4 (61.7)77.2 (62.5)76.4 (51.8)78.9 (61.6)
TOFU+GRPO84.8 (62.9)88.3 (65.9)80.9 (51.4)85.0 (61.3)
OlympiadBenchCE+GRPO62.7 (33.1)65.0 (39.8)51.3 (18.0)59.0 (27.4)
GEM+GRPO62.6 (32.8)67.3 (39.8)51.9 (18.2)59.5 (26.5)
TOFU+GRPO63.7 (33.9)67.7 (41.1)55.8 (21.6)62.0 (27.0)

Dear reviewer pVBi, thank you again for your time and effort in reviewing our work! As the discussion period is ending soon, please do let us know if you have any additional questions or concerns with the paper. We'd be happy to address them.

Reviewer_bHU3OpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work proposes TOFU, a new SFT loss that retains more response diversity then classic CE. It combines GEM (against forgetting pretrain knowledge) and Focal Loss (against ignoring low-frequency SFT patterns). Tested on 6 models x 2 SFT datasets x 2 benchmarks, TOFU shows lower Self-BLEU than CE, GEM and Focal Loss in all cases, suggesting wider response breadth. But its score by LLM Judge is 0.1-0.4 (out of 5) lower than CE.

Critical strengths
  • The method is well motivated and the theorem is introduced clearly. GEM targets forgetting and Focal Loss targets ignorence on low-frequency SFT patterns. Theorem 3.1 states that GEM is equivalent to a temperature-scaled CE, which makes the combining of the two losses simple.
  • The extensive evaluations indeed confirm that SFT with CE reduces response diversity. In 23 out of 24 cases, CE has higher Self-BLEU score than the base models.
Critical weaknesses

My main concern is that the proposed objective TOFU is never compared with the most naive baseline, aka. CE but with a higher sampling temperature.

  • According to Theorem 3.1, GEM is equivalent to a temperature-scaled CE, and Sec 3 says TOFU keeps "the learned distribution from becoming overly concentrated", which is also what a higher temperature can do.
  • The results also look like that of a higher temperature. In Table D.1. and D.2., TOFU's Self-BLEU is 43-56% lower than CE, with a slightly degraded LLM Judge score. But all use the same T=1 temperature.
  • So it's unclear whether TOFU has its own unique value or it's similar to just use CE with a higher temp.
Questions for authors

Could you either explain what unique values TOFU has over CE with higher temperature or re-run the CE model in Table D.1. at higher temperatures (like T=[1.25, 1.5]) and report the Self-BLEU (D) and LLM Judge score (Q)? There is no training needed.

4 comments

Thank you for your question!

While Theorem 3.1 states equivalence between GEM training objective and temperature scaled Cross Entropy, we argue that, due to limited updates in SFT and inherent optimization noise, training with LCEβ(θ)=−βEqlog⁡pθβ\mathcal{L}^\beta_{\mathrm{CE}}(\theta) = {-}\beta \mathbb{E}_{q} \log p^\beta_\theta is not equivalent to training with LCE(θ)=−Eqlog⁡pθ\mathcal{L}_{\mathrm{CE}}(\theta) = {-} \mathbb{E}_{q} \log p_\theta and scaling it post-hoc with β−1\beta^{-1} (l.191-194). This hypothesis is confirmed empirically by the results presented in Table R.1.

While for GEM we expect to obtain different solutions from post-hoc scaled CE due to difference in training dynamics, TOFU by design differs from LCEβ\mathcal{L}^\beta_{\mathrm{CE}} by an additional dynamic factor, the focal term.

As one can see from Table R.1, post-hoc temperature scaling falls short in comparison with GEM and TOFU both in terms of quality and diversity, thus proving unique value of both training objectives.

We will make the distinction between temperature scaled objective and post-hoc technique clear in the revised version of the manuscript.

We hope this clarifies your concern, and we would be happy to answer any further questions or provide additional details if needed!

Table R.1, Llama-3.1-8B, Alpaca Eval, β=0.7\beta=0.7

LossDiversityQuality
CE (T=1βT=\frac{1}{\beta})30.6 ± 12.13.8 ± 0.5
CE (T=1.25T=1.25)37.3 ± 13.34.1 ± 0.6
CE (T=1.5T=1.5)28.4 ± 12.03.8 ± 0.5
CE (T=1T=1)44.5 ± 14.54.2 ± 0.7
GEM (T=1T=1)26.0 ± 11.34.0 ± 0.7
TOFU (T=1T=1)20.1 ± 7.94.0 ± 0.6

Dear reviewer bHU3, thank you again for your time and effort in reviewing our work! As the discussion period is ending soon, please do let us know if you have any additional questions or concerns with the paper. We'd be happy to address them.

Thank you. Table R.1 resolves my concern, and I have raised my rating to 3. Please add this comparison to the paper, as the current PDF does not include it yet.

Thank you for your positive feedback and for raising your score! We have added the comparison to Section 3 / Appendix D of the revised manuscript as requested.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.