- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the reduction in generative diversity caused by Supervised Fine-Tuning (SFT) in large language models. The authors identify two primary factors contributing to this issue: the underrepresentation of low-frequency patterns in fine-tuning datasets and the forgetting of pretrained knowledge. To address these challenges, they propose Tempered Focal (TOFU) loss, a novel training objective designed to preserve generation diversity while maintaining response quality. Through theoretical analysis and extensive experiments across multiple models and benchmarks, the authors demonstrate that TOFU effectively mitigates diversity degradation after SFT, offering a promising approach to improving LLM fine-tuning.
Critical strengths
-
The motivation of this work is interesting, as it addresses the important issue of reduced generative diversity during the fine-tuning of large language models.
-
This paper is clear in writing and easy to follow.
-
In the experimental results, this paper improves results in MATH500, Minerva, GSM8K and OlympiadBench compared with the baselines.
Critical weaknesses
-
The effectiveness of the proposed method on modern long-context tasks deserves further investigation. For example, modern LLMs have demonstrated strong reasoning capabilities. However, this work primarily focuses on relatively easy math tasks and lacks further exploration of more challenging long-context / response tasks, particularly those requiring complex agent loop / reasoning. This can be written in the limitations.
-
The experiments primarily focus on relatively older models, such as Qwen2.5, and lack sufficient evidence demonstrating the effectiveness of the proposed method on more recent frontier models.
Questions for authors
N/A
3 comments
Dear reviewer AAtG, thank you again for your time and effort in reviewing our work! As the discussion period is ending soon, please do let us know if you have any additional questions or concerns with the paper. We'd be happy to address them.
Thank you for the detailed rebuttal and clarifications. I have carefully reviewed your responses and the additional explanations. While I appreciate the efforts made to address my concerns, my overall assessment remains unchanged, and I will maintain my original rating.
Thank you very much for your feedback!
Limitations and Future Work
We agree that it is interesting to investigate the performance of TOFU in more complex systems beyond a single model, e.g. agentic loops. We see it as an opportunity for future work rather than a limitation. We hope that the community possessing larger computational resources will extend our findings to a greater scale.
Models Selection
Our choice of models was driven by the need for robust, highly capable open-source backbones that enable controlled, reproducible comparisons. Additionally, we wanted to have base models unaffected by post-training procedures, a criterion that many recent open-source releases do not satisfy.
We are happy to participate further in the discussion! Please let us know if any points remain unclear or if there are specific details we can provide to address your concerns and merit a score increase.