- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces the FUEL framework, which combines "coreset fine-tuning" with "non-coreset unlearning" to mitigate the degradation of capabilities in large language models (LLMs) during the adaptation process. FUEL leverages training samples excluded from the coreset to construct gradient-ascent perturbations; it then performs gradient descent on the coreset, starting from these perturbed parameters. Experimental results demonstrate that this approach is more effective at preserving non-target capabilities and reducing performance degradation in safety-related domains.
Critical strengths
(1) The motivation is clear: preserving previously acquired model capabilities when data for non-target tasks is inaccessible is a significant challenge in LLM fine-tuning. This "target-task-data-only" setting is highly relevant in practice and provides a solid justification for the proposed method.
(2) The idea of repurposing excluded training samples to construct corrective perturbations is ingenious. The "leader–follower" architecture employed in the paper offers an elegant way to integrate this mechanism into the fine-tuning process. Experiments show that the method significantly improves the retention of non-target task capabilities while maintaining high accuracy on the target tasks.
Critical weaknesses
(1) The methodological innovation is not entirely clear; the proposed optimization strategy bears a strong resemblance to Sharpness-Aware Minimization (SAM). The "non-coreset gradient ascent" described in the paper essentially constructs a temporary perturbation, while the actual parameter updates are still performed on the coreset. Consequently, it remains ambiguous whether the performance gains stem from "unlearning" in the true sense or simply from standard perturbation-based regularization.
(2) The comparison with existing capability-preservation methods is insufficient. Although the primary goal is to mitigate "catastrophic forgetting," the core experiments only compare FUEL against standard fine-tuning and coreset-based fine-tuning. The study lacks comparisons with established regularization or continual learning techniques, such as TALR and FLOW.
(3) The primary "capability-preservation" experiments in the paper are limited to two relatively small models; their effectiveness across different model architectures and with larger parameter counts has not been sufficiently verified.
Questions for authors
Q1: Could the authors provide a direct comparison with SAM (Sharpness-Aware Minimization), along with relevant ablation studies, to demonstrate the unique contribution of the "non-coreset unlearning" mechanism?
Q2: Under identical training conditions, how does FUEL perform compared to existing regularization-based methods and continual learning approaches—such as TALR and FLOW?
Q3: Could the authors provide evaluation results using more competitive baselines and metrics for training overhead, based on more diverse and larger-scale models?
[1] Foret P, Kleiner A, Mobahi H, et al. Sharpness-aware minimization for efficiently improving generalization[J]. arXiv preprint arXiv:2010.01412, 2020.
[2] Lin J, Wang Z, Qian K, et al. Sft doesn’t always hurt general capabilities: Revisiting domain-specific fine-tuning in llms[C]//International Conference on Learning Representations. 2026, 2026: 46954-46992.
[3] Sanyal S, Prairie H, Das R, et al. Upweighting easy samples in fine-tuning mitigates forgetting[J]. arXiv preprint arXiv:2502.02797, 2025.
5 comments
Response to Reviewer jFT2 (Part 2)
A1 (continued): As shown in Tables R1 and R2, FUEL consistently outperforms all four SAM variants in preserving non-target capabilities (measured by ) and model safety, while maintaining competitive target-task performance. In contrast, the SAM variants do not exhibit comparable improvements in capability control or safety preservation. These results suggest that FUEL's advantage cannot be explained solely by generic weight perturbations or sharpness-aware regularization; rather, the non-coreset-derived unlearning direction plays a critical role.
We believe that distinguishing coreset from non-coreset data enables a complementary interaction between outer minimization (target-task learning) and inner maximization (non-coreset unlearning). This coupling provides a corrective signal that is absent from conventional SAM variants, helping mitigate unintended changes to non-target capabilities. Our results thus point to an important yet largely overlooked principle: data excluded from fine-tuning can still provide valuable corrective signals through unlearning-induced optimization.
Table R1: Ablation on the source of the perturbation in FUEL for general capability preservation, with Llama-3-8B fine-tuned on MNLI and OpenBookQA. The result format follows Table 2, with the target-task accuracy and the average accuracy over the 19 non-target tasks. Results are mean ± std over four seeds.
| Method | MNLI | MNLI | OpenBookQA | OpenBookQA |
|---|---|---|---|---|
| Base (w/o FT) | 47.5 | 64.2 | 33.4 | 65.0 |
| Standard SAM | 86.8 ± 0.9 | 54.4 ± 0.6 | 55.8 ± 1.3 | 55.2 ± 0.7 |
| Coreset-based SAM | 87.1 ± 0.6 | 54.4 ± 0.5 | 55.7 ± 1.4 | 54.0 ± 0.7 |
| SAM (separate-batch perturbation) | 87.1 ± 0.9 | 54.0 ± 0.9 | 56.0 ± 1.8 | 54.2 ± 1.1 |
| SAM (random perturbation) | 85.2 ± 1.2 | 53.4 ± 1.0 | 54.5 ± 1.9 | 53.8 ± 1.2 |
| FUEL (ours) | 87.2 ± 1.1 | 61.1 ± 0.6 | 56.8 ± 1.4 | 60.8 ± 0.6 |
Table R2: Ablation on the source of the perturbation in FUEL for safety preservation, with Llama-2-7B-Chat fine-tuned on GSM8K. The result format follows Table 3, with GSM8K accuracy for the target task and HarmBench ASR and StrongREJECT score for safety. Results are mean ± std over four seeds.
| Method | GSM8K Acc ↑ | HarmBench ASR ↓ | StrongREJECT Score ↓ |
|---|---|---|---|
| Base (w/o FT) | 27.1 | 1.0 | 2.1 |
| Standard SAM | 32.6 ± 0.5 | 34.0 ± 2.1 | 15.9 ± 0.6 |
| Coreset-based SAM | 32.8 ± 0.5 | 32.0 ± 1.9 | 15.8 ± 0.4 |
| SAM (separate-batch perturbation) | 31.5 ± 1.3 | 33.7 ± 2.5 | 14.5 ± 1.2 |
| SAM (random perturbation) | 32.8 ± 1.6 | 33.2 ± 4.1 | 16.1 ± 2.2 |
| FUEL (ours) | 33.0 ± 0.9 | 23.5 ± 2.3 | 11.9 ± 0.9 |
Response to Reviewer jFT2 (Part 3)
W2 (Q2): The core experiments only compare FUEL against standard fine-tuning and coreset-based fine-tuning. The study lacks comparisons with established regularization or continual learning techniques, such as TALR and FLOW.
A2: Following your suggestion, we compare FUEL against three anti-forgetting methods in Tables R3 and R4, which correspond to the capability preservation and safety preservation experiments in our main paper, respectively. Among the studied anti-forgetting methods, SAM [1,4] formulates training as a min-max optimization problem, where the inner maximization and outer minimization operate on the same dataset, unlike FUEL, which uses the inner maximization to perform unlearning on non-coreset samples. FLOW [5] upweights samples with low loss under the pretrained model, while TALR [6] downweights hard tokens. All methods share the same base model, learning rate, batch size, and number of training steps.
As shown in Tables R3 and R4, FUEL consistently outperforms all baselines in preserving both general capabilities and safety, while maintaining comparable target accuracy.
Table R3: Comparison with anti-forgetting methods for general capability preservation, with Llama-3-8B fine-tuned on MNLI and OpenBookQA. The result format follows Table R1.
| Method | MNLI | MNLI | OpenBookQA | OpenBookQA |
|---|---|---|---|---|
| Base (w/o FT) | 47.5 | 64.2 | 33.4 | 65.0 |
| Standard FT | 84.7 ± 0.8 | 54.0 ± 0.6 | 54.2 ± 1.3 | 53.8 ± 0.6 |
| Coreset FT | 87.4 ± 0.8 | 53.8 ± 0.5 | 57.0 ± 1.4 | 54.0 ± 0.7 |
| Standard SAM | 86.8 ± 0.9 | 54.4 ± 0.6 | 55.8 ± 1.3 | 55.2 ± 0.7 |
| FLOW | 86.3 ± 0.5 | 54.6 ± 0.4 | 57.1 ± 1.2 | 54.5 ± 0.4 |
| TALR | 87.4 ± 0.7 | 56.1 ± 0.5 | 56.6 ± 1.2 | 56.2 ± 0.6 |
| FUEL (ours) | 87.2 ± 1.1 | 61.1 ± 0.6 | 56.8 ± 1.4 | 60.8 ± 0.6 |
Table R4: Comparison with anti-forgetting methods for safety preservation, with Llama-2-7B-Chat fine-tuned on GSM8K. The result format follows Table R2.
| Method | GSM8K Acc ↑ | HarmBench ASR ↓ | StrongREJECT Score ↓ |
|---|---|---|---|
| Base (w/o FT) | 27.1 | 1.0 | 2.1 |
| Standard SAM | 32.6 ± 0.5 | 34.0 ± 2.1 | 15.9 ± 0.6 |
| FLOW | 33.8 ± 0.4 | 36.5 ± 1.5 | 18.0 ± 0.8 |
| TALR | 29.4 ± 1.1 | 31.0 ± 1.6 | 12.6 ± 0.8 |
| FUEL (ours) | 33.0 ± 0.9 | 23.5 ± 2.3 | 11.9 ± 0.9 |
Q3: Could the authors provide metrics for training overhead?
A3: Thank you for the suggestion. We measure the wall-clock training time of each method in the setting of Table R3, fine-tuning Llama-3-8B on MNLI and OpenBookQA, and report the average over the two tasks relative to Standard FT. TALR adds negligible overhead (≈1.0×), and FLOW costs ≈1.2× because it needs one forward pass of the base model over the training set to compute sample weights. SAM and FUEL both require two forward–backward passes per step, costing ≈2.0× and ≈2.2×, respectively. FUEL's extra 0.2× comes from a similar one-time forward pass to select the coreset.
Response to Reviewer jFT2 (Part 1)
We thank Reviewer jFT2 for recognizing the clear motivation and practical relevance of the target-task-data-only setting, the idea of repurposing excluded training samples through the leader–follower framework, and the improved retention of non-target capabilities while maintaining high target-task accuracy. Below, we address each of the reviewer's weaknesses (W) or questions (Q) and provide our corresponding responses (A).
W1 (Q1): Could the authors provide a direct comparison with SAM (Sharpness-Aware Minimization), along with relevant ablation studies, to demonstrate the unique contribution of the "non-coreset unlearning" mechanism?
A1: We thank the reviewer for this insightful question. We agree that FUEL reduces to SAM [1] when the follower unlearns the same data used by the leader (i.e., Dncf=Dcf in Eq. (3)). However, the key distinction lies in the role of non-coreset unlearning in constructing the perturbation, rather than perturbation-based regularization alone.
Methodologically, as shown in Eq. (4), FUEL's perturbation corresponds to a gradient-ascent step on the non-coreset data Dncf, explicitly introducing an unlearning-induced perturbation into the leader's optimization. In contrast, standard SAM constructs its perturbation using the same data batch as the outer minimization to seek flat minima. This choice matters since existing theoretical analyses of SAM's sharpness-reduction properties [2, 3] rely on the perturbation being computed from the gradient of the loss being minimized. FUEL's perturbation instead comes from the non-coreset loss, reflecting a different mechanism.
Empirically, to isolate the contribution of non-coreset unlearning from generic perturbation-based regularization, we conduct additional ablation studies in Tables R1 and R2, comparing FUEL against four SAM-related variants using different perturbation-based regularization schemes:
These comparisons disentangle two critical factors: (i) the importance of selecting non-coreset data for the inner maximization, through comparisons with standard SAM, coreset-based SAM, and SAM (separate-batch perturbation); and (ii) the importance of the unlearning direction itself, through comparison with SAM (random perturbation).