acceptodds
Under review as a conference paper at ICLR 2027

Turning Excluded Data into FUEL: Non-Coreset Unlearning Aids Coreset Fine-Tuning in Preserving Model Capabilities

Abstract

Fine-tuning is the dominant approach for adapting large language models (LLMs) to downstream tasks, yet it can degrade previously acquired capabilities, including general knowledge, reasoning, and safety alignment. We show that machine unlearning, conventionally used to remove unwanted data or capabilities, can instead play a constructive role in preserving model capabilities during fine-tuning. Specifically, we study capability control in a target-data-only setting, where the capabilities to be preserved are unknown a priori. We first show that coreset fine-tuning alone is insufficient: even an oracle-selected coreset cannot prevent non-target capability degradation. However, we find that the otherwise discarded non-coreset data can provide a corrective signal when its fine-tuning objective is reversed. Building on this finding, we introduce FUEL (Fine-tuning through Unlearning-Enhanced Learning), a leader–follower framework that fine-tunes on the coreset while unlearning the non-coreset. Across multiple tasks and model scales, FUEL maintains strong target-task performance while substantially reducing degradation on non-target tasks. Moreover, FUEL helps preserve the robustness of safety-aligned models under subsequent benign fine-tuning. Overall, these results reveal a constructive role for unlearning beyond data deletion.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Training, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_jFT2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the FUEL framework, which combines "coreset fine-tuning" with "non-coreset unlearning" to mitigate the degradation of capabilities in large language models (LLMs) during the adaptation process. FUEL leverages training samples excluded from the coreset to construct gradient-ascent perturbations; it then performs gradient descent on the coreset, starting from these perturbed parameters. Experimental results demonstrate that this approach is more effective at preserving non-target capabilities and reducing performance degradation in safety-related domains.

Critical strengths

(1) The motivation is clear: preserving previously acquired model capabilities when data for non-target tasks is inaccessible is a significant challenge in LLM fine-tuning. This "target-task-data-only" setting is highly relevant in practice and provides a solid justification for the proposed method.

(2) The idea of ​​repurposing excluded training samples to construct corrective perturbations is ingenious. The "leader–follower" architecture employed in the paper offers an elegant way to integrate this mechanism into the fine-tuning process. Experiments show that the method significantly improves the retention of non-target task capabilities while maintaining high accuracy on the target tasks.

Critical weaknesses

(1) The methodological innovation is not entirely clear; the proposed optimization strategy bears a strong resemblance to Sharpness-Aware Minimization (SAM). The "non-coreset gradient ascent" described in the paper essentially constructs a temporary perturbation, while the actual parameter updates are still performed on the coreset. Consequently, it remains ambiguous whether the performance gains stem from "unlearning" in the true sense or simply from standard perturbation-based regularization.

(2) The comparison with existing capability-preservation methods is insufficient. Although the primary goal is to mitigate "catastrophic forgetting," the core experiments only compare FUEL against standard fine-tuning and coreset-based fine-tuning. The study lacks comparisons with established regularization or continual learning techniques, such as TALR and FLOW.

(3) The primary "capability-preservation" experiments in the paper are limited to two relatively small models; their effectiveness across different model architectures and with larger parameter counts has not been sufficiently verified.

Questions for authors

Q1: Could the authors provide a direct comparison with SAM (Sharpness-Aware Minimization), along with relevant ablation studies, to demonstrate the unique contribution of the "non-coreset unlearning" mechanism?

Q2: Under identical training conditions, how does FUEL perform compared to existing regularization-based methods and continual learning approaches—such as TALR and FLOW?

Q3: Could the authors provide evaluation results using more competitive baselines and metrics for training overhead, based on more diverse and larger-scale models?

[1] Foret P, Kleiner A, Mobahi H, et al. Sharpness-aware minimization for efficiently improving generalization[J]. arXiv preprint arXiv:2010.01412, 2020.

[2] Lin J, Wang Z, Qian K, et al. Sft doesn’t always hurt general capabilities: Revisiting domain-specific fine-tuning in llms[C]//International Conference on Learning Representations. 2026, 2026: 46954-46992.

[3] Sanyal S, Prairie H, Das R, et al. Upweighting easy samples in fine-tuning mitigates forgetting[J]. arXiv preprint arXiv:2502.02797, 2025.

5 comments
AuthorsOpenReview ↗

Response to Reviewer jFT2 (Part 1)

We thank Reviewer jFT2 for recognizing the clear motivation and practical relevance of the target-task-data-only setting, the idea of repurposing excluded training samples through the leader–follower framework, and the improved retention of non-target capabilities while maintaining high target-task accuracy. Below, we address each of the reviewer's weaknesses (W) or questions (Q) and provide our corresponding responses (A).

W1 (Q1): Could the authors provide a direct comparison with SAM (Sharpness-Aware Minimization), along with relevant ablation studies, to demonstrate the unique contribution of the "non-coreset unlearning" mechanism?

A1: We thank the reviewer for this insightful question. We agree that FUEL reduces to SAM [1] when the follower unlearns the same data used by the leader (i.e., Dncf=DcfD_{\mathrm{ncf}} = D_{\mathrm{cf}} in Eq. (3)). However, the key distinction lies in the role of non-coreset unlearning in constructing the perturbation, rather than perturbation-based regularization alone.

Methodologically, as shown in Eq. (4), FUEL's perturbation corresponds to a gradient-ascent step on the non-coreset data DncfD_{\mathrm{ncf}}, explicitly introducing an unlearning-induced perturbation into the leader's optimization. In contrast, standard SAM constructs its perturbation using the same data batch as the outer minimization to seek flat minima. This choice matters since existing theoretical analyses of SAM's sharpness-reduction properties [2, 3] rely on the perturbation being computed from the gradient of the loss being minimized. FUEL's perturbation instead comes from the non-coreset loss, reflecting a different mechanism.

Empirically, to isolate the contribution of non-coreset unlearning from generic perturbation-based regularization, we conduct additional ablation studies in Tables R1 and R2, comparing FUEL against four SAM-related variants using different perturbation-based regularization schemes:

  1. Standard SAM: Performs inner maximization and outer minimization on the same training batch, without distinguishing coreset and non-coreset samples.
  2. Coreset-based SAM: Applies standard SAM to the coreset only, performing inner maximization and outer minimization on the same coreset batch. This controls for the effect of training on the coreset.
  3. SAM (separate-batch perturbation): Computes the inner-maximization perturbation using a separate, randomly sampled coreset batch, while retaining coreset-based outer minimization. This controls for the effect of using different batches for perturbation and optimization within the coreset.
  4. SAM (random perturbation): Replaces the gradient-based inner-maximization direction with a random direction of the same norm, isolating the effect of perturbation-based regularization without an unlearning signal.

These comparisons disentangle two critical factors: (i) the importance of selecting non-coreset data for the inner maximization, through comparisons with standard SAM, coreset-based SAM, and SAM (separate-batch perturbation); and (ii) the importance of the unlearning direction itself, through comparison with SAM (random perturbation).

AuthorsOpenReview ↗

Response to Reviewer jFT2 (Part 2)

A1 (continued): As shown in Tables R1 and R2, FUEL consistently outperforms all four SAM variants in preserving non-target capabilities (measured by Accu\mathrm{Acc}_{\mathrm{u}}) and model safety, while maintaining competitive target-task performance. In contrast, the SAM variants do not exhibit comparable improvements in capability control or safety preservation. These results suggest that FUEL's advantage cannot be explained solely by generic weight perturbations or sharpness-aware regularization; rather, the non-coreset-derived unlearning direction plays a critical role.

We believe that distinguishing coreset from non-coreset data enables a complementary interaction between outer minimization (target-task learning) and inner maximization (non-coreset unlearning). This coupling provides a corrective signal that is absent from conventional SAM variants, helping mitigate unintended changes to non-target capabilities. Our results thus point to an important yet largely overlooked principle: data excluded from fine-tuning can still provide valuable corrective signals through unlearning-induced optimization.

Table R1: Ablation on the source of the perturbation in FUEL for general capability preservation, with Llama-3-8B fine-tuned on MNLI and OpenBookQA. The result format follows Table 2, with Accf\mathrm{Acc}_{\mathrm{f}} the target-task accuracy and Accu\mathrm{Acc}_{\mathrm{u}} the average accuracy over the 19 non-target tasks. Results are mean ± std over four seeds.

MethodMNLI Accf\mathrm{Acc}_{\mathrm{f}}MNLI Accu\mathrm{Acc}_{\mathrm{u}}OpenBookQA Accf\mathrm{Acc}_{\mathrm{f}}OpenBookQA Accu\mathrm{Acc}_{\mathrm{u}}
Base (w/o FT)47.564.233.465.0
Standard SAM86.8 ± 0.954.4 ± 0.655.8 ± 1.355.2 ± 0.7
Coreset-based SAM87.1 ± 0.654.4 ± 0.555.7 ± 1.454.0 ± 0.7
SAM (separate-batch perturbation)87.1 ± 0.954.0 ± 0.956.0 ± 1.854.2 ± 1.1
SAM (random perturbation)85.2 ± 1.253.4 ± 1.054.5 ± 1.953.8 ± 1.2
FUEL (ours)87.2 ± 1.161.1 ± 0.656.8 ± 1.460.8 ± 0.6

Table R2: Ablation on the source of the perturbation in FUEL for safety preservation, with Llama-2-7B-Chat fine-tuned on GSM8K. The result format follows Table 3, with GSM8K accuracy for the target task and HarmBench ASR and StrongREJECT score for safety. Results are mean ± std over four seeds.

MethodGSM8K Acc ↑HarmBench ASR ↓StrongREJECT Score ↓
Base (w/o FT)27.11.02.1
Standard SAM32.6 ± 0.534.0 ± 2.115.9 ± 0.6
Coreset-based SAM32.8 ± 0.532.0 ± 1.915.8 ± 0.4
SAM (separate-batch perturbation)31.5 ± 1.333.7 ± 2.514.5 ± 1.2
SAM (random perturbation)32.8 ± 1.633.2 ± 4.116.1 ± 2.2
FUEL (ours)33.0 ± 0.923.5 ± 2.311.9 ± 0.9
AuthorsOpenReview ↗

Response to Reviewer jFT2 (Part 3)

W2 (Q2): The core experiments only compare FUEL against standard fine-tuning and coreset-based fine-tuning. The study lacks comparisons with established regularization or continual learning techniques, such as TALR and FLOW.

A2: Following your suggestion, we compare FUEL against three anti-forgetting methods in Tables R3 and R4, which correspond to the capability preservation and safety preservation experiments in our main paper, respectively. Among the studied anti-forgetting methods, SAM [1,4] formulates training as a min-max optimization problem, where the inner maximization and outer minimization operate on the same dataset, unlike FUEL, which uses the inner maximization to perform unlearning on non-coreset samples. FLOW [5] upweights samples with low loss under the pretrained model, while TALR [6] downweights hard tokens. All methods share the same base model, learning rate, batch size, and number of training steps.

As shown in Tables R3 and R4, FUEL consistently outperforms all baselines in preserving both general capabilities and safety, while maintaining comparable target accuracy.

Table R3: Comparison with anti-forgetting methods for general capability preservation, with Llama-3-8B fine-tuned on MNLI and OpenBookQA. The result format follows Table R1.

MethodMNLI Accf\mathrm{Acc}_{\mathrm{f}}MNLI Accu\mathrm{Acc}_{\mathrm{u}}OpenBookQA Accf\mathrm{Acc}_{\mathrm{f}}OpenBookQA Accu\mathrm{Acc}_{\mathrm{u}}
Base (w/o FT)47.564.233.465.0
Standard FT84.7 ± 0.854.0 ± 0.654.2 ± 1.353.8 ± 0.6
Coreset FT87.4 ± 0.853.8 ± 0.557.0 ± 1.454.0 ± 0.7
Standard SAM86.8 ± 0.954.4 ± 0.655.8 ± 1.355.2 ± 0.7
FLOW86.3 ± 0.554.6 ± 0.457.1 ± 1.254.5 ± 0.4
TALR87.4 ± 0.756.1 ± 0.556.6 ± 1.256.2 ± 0.6
FUEL (ours)87.2 ± 1.161.1 ± 0.656.8 ± 1.460.8 ± 0.6

Table R4: Comparison with anti-forgetting methods for safety preservation, with Llama-2-7B-Chat fine-tuned on GSM8K. The result format follows Table R2.

MethodGSM8K Acc ↑HarmBench ASR ↓StrongREJECT Score ↓
Base (w/o FT)27.11.02.1
Standard SAM32.6 ± 0.534.0 ± 2.115.9 ± 0.6
FLOW33.8 ± 0.436.5 ± 1.518.0 ± 0.8
TALR29.4 ± 1.131.0 ± 1.612.6 ± 0.8
FUEL (ours)33.0 ± 0.923.5 ± 2.311.9 ± 0.9

Q3: Could the authors provide metrics for training overhead?

A3: Thank you for the suggestion. We measure the wall-clock training time of each method in the setting of Table R3, fine-tuning Llama-3-8B on MNLI and OpenBookQA, and report the average over the two tasks relative to Standard FT. TALR adds negligible overhead (≈1.0×), and FLOW costs ≈1.2× because it needs one forward pass of the base model over the training set to compute sample weights. SAM and FUEL both require two forward–backward passes per step, costing ≈2.0× and ≈2.2×, respectively. FUEL's extra 0.2× comes from a similar one-time forward pass to select the coreset.

Reviewer_abtSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FUEL, a framework for mitigating capability degradation during LLM fine-tuning using only target-task data. The method partitions the training dataset into a coreset and a non-coreset, then combines gradient descent on the coreset with gradient-ascent-based perturbations derived from the non-coreset through a leader–follower optimization framework. Experiments on Llama models demonstrate improved preservation of non-target capabilities while maintaining target-task performance. Additional experiments show that FUEL can also mitigate safety degradation caused by benign fine-tuning.

Critical strengths
  1. The paper addresses capability degradation during fine-tuning in a practical setting where non-target capabilities and their corresponding training data are unavailable.

  2. Reusing excluded training examples to construct corrective gradient perturbations offers an interesting approach to capability preservation without requiring auxiliary datasets.

  3. FUEL consistently improves non-target performance across two model scales and four fine-tuning tasks while maintaining strong target-task performance. The safety experiments further demonstrate improvements in preserving alignment compared with standard fine-tuning.

Critical weaknesses
  1. The proposed algorithm closely resembles Sharpness-Aware Minimization (SAM), where the perturbation is computed using non-coreset gradients rather than the training batch itself. Since the perturbation is temporary and not directly incorporated into the final model weights, it is unclear whether FUEL actually induces unlearning of non-coreset examples or primarily benefits from perturbation-based regularization.

  2. Missing comparisons with important baselines: a). The experiments mainly compare FUEL with standard and coreset fine-tuning, without evaluating SAM or related perturbation-based methods. Comparisons with standard SAM, coreset-based SAM, and alternative gradient perturbations would help establish whether non-coreset gradients provide benefits beyond conventional sharpness-aware optimization. b) There are quite a few related exisiting works on the same/similar topics and it will greatly strengthen this draft if the author can compare with these existing works, e.g. https://arxiv.org/abs/2509.20758.

  3. The paper assumes that non-target capabilities are unavailable during fine-tuning. However, Figure 4 analyzes the choice of perturbation strength using non-target benchmark performance. It remains unclear how an appropriate perturbation strength can be selected in practice without access to such information.

Questions for authors
  1. How is FUEL fundamentally different from SAM with perturbations computed on a separate subset of training data? Is there empirical evidence that the non-coreset examples are actually unlearned?
  2. Can the authors provide additional evidence that coreset selection alone is insufficient, beyond the gradient-alignment-based oracle and the specific task pair evaluated?
6 comments
AuthorsOpenReview ↗

Response to Reviewer abtS (Part 1)

We thank Reviewer abtS for recognizing the practical setting in which non-target capabilities and their training data are unavailable, the idea of reusing excluded training examples to construct corrective perturbations without auxiliary datasets, and the consistent improvements of FUEL in preserving both non-target capabilities and safety alignment. Below, we address each of the reviewer's weaknesses (W) or questions (Q) and provide our corresponding responses (A).

W1 (Q1): The proposed algorithm closely resembles Sharpness-Aware Minimization (SAM), where the perturbation is computed using non-coreset gradients rather than the training batch itself. It is unclear whether FUEL actually induces unlearning of non-coreset examples or primarily benefits from perturbation-based regularization.

A1: We thank the reviewer for this insightful question. We agree that FUEL reduces to SAM [1] when the follower unlearns the same data used by the leader (i.e., Dncf=DcfD_{\mathrm{ncf}} = D_{\mathrm{cf}} in Eq. (3)). However, the key distinction lies in the role of non-coreset unlearning in constructing the perturbation, rather than perturbation-based regularization alone.

Methodologically, as shown in Eq. (4), FUEL's perturbation corresponds to a gradient-ascent step on the non-coreset data DncfD_{\mathrm{ncf}}, explicitly introducing an unlearning-induced perturbation into the leader's optimization. In contrast, standard SAM constructs its perturbation using the same data batch as the outer minimization to seek flat minima. This choice matters since existing theoretical analyses of SAM's sharpness-reduction properties [2, 3] rely on the perturbation being computed from the gradient of the loss being minimized. FUEL's perturbation instead comes from the non-coreset loss, reflecting a different mechanism.

Empirically, to isolate the contribution of non-coreset unlearning from generic perturbation-based regularization, we conduct additional ablation studies in Tables R1 and R2, comparing FUEL against four SAM-related variants using different perturbation-based regularization schemes:

  1. Standard SAM: Performs inner maximization and outer minimization on the same training batch, without distinguishing coreset and non-coreset samples.
  2. Coreset-based SAM: Applies standard SAM to the coreset only, performing inner maximization and outer minimization on the same coreset batch. This controls for the effect of training on the coreset.
  3. SAM (separate-batch perturbation): Computes the inner-maximization perturbation using a separate, randomly sampled coreset batch, while retaining coreset-based outer minimization. This controls for the effect of using different batches for perturbation and optimization within the coreset.
  4. SAM (random perturbation): Replaces the gradient-based inner-maximization direction with a random direction of the same norm, isolating the effect of perturbation-based regularization without an unlearning signal.

These comparisons disentangle two critical factors: (i) the importance of selecting non-coreset data for the inner maximization, through comparisons with standard SAM, coreset-based SAM, and SAM (separate-batch perturbation); and (ii) the importance of the unlearning direction itself, through comparison with SAM (random perturbation).

AuthorsOpenReview ↗

Response to Reviewer abtS (Part 2)

A1 (continued): As shown in Tables R1 and R2, FUEL consistently outperforms all four SAM variants in preserving non-target capabilities (measured by Accu\mathrm{Acc}_{\mathrm{u}}) and model safety, while maintaining competitive target-task performance. In contrast, the SAM variants do not exhibit comparable improvements in capability control or safety preservation. These results suggest that FUEL's advantage cannot be explained solely by generic weight perturbations or sharpness-aware regularization; rather, the non-coreset-derived unlearning direction plays a critical role.

We believe that distinguishing coreset from non-coreset data enables a complementary interaction between outer minimization (target-task learning) and inner maximization (non-coreset unlearning). This coupling provides a corrective signal that is absent from conventional SAM variants, helping mitigate unintended changes to non-target capabilities. Our results thus point to an important yet largely overlooked principle: data excluded from fine-tuning can still provide valuable corrective signals through unlearning-induced optimization.

Table R1: Ablation on the source of the perturbation in FUEL for general capability preservation, with Llama-3-8B fine-tuned on MNLI and OpenBookQA. The result format follows Table 2, with Accf\mathrm{Acc}_{\mathrm{f}} the target-task accuracy and Accu\mathrm{Acc}_{\mathrm{u}} the average accuracy over the 19 non-target tasks. Results are mean ± std over four seeds.

MethodMNLI Accf\mathrm{Acc}_{\mathrm{f}}MNLI Accu\mathrm{Acc}_{\mathrm{u}}OpenBookQA Accf\mathrm{Acc}_{\mathrm{f}}OpenBookQA Accu\mathrm{Acc}_{\mathrm{u}}
Base (w/o FT)47.564.233.465.0
Standard SAM86.8 ± 0.954.4 ± 0.655.8 ± 1.355.2 ± 0.7
Coreset-based SAM87.1 ± 0.654.4 ± 0.555.7 ± 1.454.0 ± 0.7
SAM (separate-batch perturbation)87.1 ± 0.954.0 ± 0.956.0 ± 1.854.2 ± 1.1
SAM (random perturbation)85.2 ± 1.253.4 ± 1.054.5 ± 1.953.8 ± 1.2
FUEL (ours)87.2 ± 1.161.1 ± 0.656.8 ± 1.460.8 ± 0.6

Table R2: Ablation on the source of the perturbation in FUEL for safety preservation, with Llama-2-7B-Chat fine-tuned on GSM8K. The result format follows Table 3, with GSM8K accuracy for the target task and HarmBench ASR and StrongREJECT score for safety. Results are mean ± std over four seeds.

MethodGSM8K Acc ↑HarmBench ASR ↓StrongREJECT Score ↓
Base (w/o FT)27.11.02.1
Standard SAM32.6 ± 0.534.0 ± 2.115.9 ± 0.6
Coreset-based SAM32.8 ± 0.532.0 ± 1.915.8 ± 0.4
SAM (separate-batch perturbation)31.5 ± 1.333.7 ± 2.514.5 ± 1.2
SAM (random perturbation)32.8 ± 1.633.2 ± 4.116.1 ± 2.2
FUEL (ours)33.0 ± 0.923.5 ± 2.311.9 ± 0.9
AuthorsOpenReview ↗

Response to Reviewer abtS (Part 3)

W2: There are quite a few related exisiting works on the same/similar topics and it will greatly strengthen this draft if the author can compare with these existing works.

A2: We thank the reviewer for pointing this out. Following your suggestion, we compare FUEL against three anti-forgetting methods in Tables R3 and R4, which correspond to the capability preservation and safety preservation experiments in our main paper, respectively. Among the studied anti-forgetting methods, SAM [1,4] formulates training as a min-max optimization problem, where the inner maximization and outer minimization operate on the same dataset, unlike FUEL, which uses the inner maximization to perform unlearning on non-coreset samples. FLOW [5] upweights samples with low loss under the pretrained model, while TALR [6] downweights hard tokens. All methods share the same base model, learning rate, batch size, and number of training steps.

As shown in Tables R3 and R4, FUEL consistently outperforms all baselines in preserving both general capabilities and safety, while maintaining comparable target accuracy.

Table R3: Comparison with anti-forgetting methods for general capability preservation, with Llama-3-8B fine-tuned on MNLI and OpenBookQA. The result format follows Table R1.

MethodMNLI Accf\mathrm{Acc}_{\mathrm{f}}MNLI Accu\mathrm{Acc}_{\mathrm{u}}OpenBookQA Accf\mathrm{Acc}_{\mathrm{f}}OpenBookQA Accu\mathrm{Acc}_{\mathrm{u}}
Base (w/o FT)47.564.233.465.0
Standard FT84.7 ± 0.854.0 ± 0.654.2 ± 1.353.8 ± 0.6
Coreset FT87.4 ± 0.853.8 ± 0.557.0 ± 1.454.0 ± 0.7
Standard SAM86.8 ± 0.954.4 ± 0.655.8 ± 1.355.2 ± 0.7
FLOW86.3 ± 0.554.6 ± 0.457.1 ± 1.254.5 ± 0.4
TALR87.4 ± 0.756.1 ± 0.556.6 ± 1.256.2 ± 0.6
FUEL (ours)87.2 ± 1.161.1 ± 0.656.8 ± 1.460.8 ± 0.6

Table R4: Comparison with anti-forgetting methods for safety preservation, with Llama-2-7B-Chat fine-tuned on GSM8K. The result format follows Table R2.

MethodGSM8K Acc ↑HarmBench ASR ↓StrongREJECT Score ↓
Base (w/o FT)27.11.02.1
Standard SAM32.6 ± 0.534.0 ± 2.115.9 ± 0.6
FLOW33.8 ± 0.436.5 ± 1.518.0 ± 0.8
TALR29.4 ± 1.131.0 ± 1.612.6 ± 0.8
FUEL (ours)33.0 ± 0.923.5 ± 2.311.9 ± 0.9

W3: The paper assumes that non-target capabilities are unavailable during fine-tuning. However, Figure 4 analyzes the choice of perturbation strength using non-target benchmark performance. It remains unclear how an appropriate perturbation strength can be selected in practice without access to such information.

A3: We thank the reviewer for raising this question. We apologize for the lack of clarity in our presentation of Figure 4. In practice, we indeed select the perturbation strength ρ solely based on target-task validation performance, without accessing any non-target data or benchmarks. Specifically, we choose the largest ρ that does not degrade target-task validation accuracy, consistently across both general-capability and safety-preservation experiments. This follows the same principle as our coreset selection strategy, where target-task performance serves as the sole accessible criterion (Lines 286–292).

Figure 4 reports non-target accuracy only to illustrate the trade-off across different strengths of ρ, rather than to determine the hyperparameter selection. In fact, the results support the effectiveness of our target-only selection criterion. On QNLI, target accuracy remains nearly unchanged for ρ ≤ 0.005 while non-target accuracy steadily improves as ρ increases. Beyond this threshold, target accuracy drops sharply with marginal additional gains in non-target accuracy. The criterion detects this drop from target accuracy alone and gives ρ = 0.005.

Reviewer_5X2sOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies capability control in LLM fine-tuning in a target-data-only setting: only the target-task data Df\mathcal{D}_\mathrm{f} and the base model θ0\theta_0 are available, and the non-target capabilities to preserve are not known in advance.

  • Oracle study (Section 3): On Llama-3-8B with OpenBookQA (OBQA) as the target and SciQ as the non-target task, a coreset built by excluding the examples whose gradients conflict most with the SciQ loss at θ0\theta_0 still loses SciQ accuracy at high coreset ratios (Figure 3). The paper concludes that data selection alone is insufficient.
  • FUEL (Section 4): The leader fine-tunes on a coreset Dcf\mathcal{D}_\mathrm{cf} (95% of Df\mathcal{D}_\mathrm{f}), and the follower performs gradient-ascent (GA) unlearning on the non-coreset Dncf\mathcal{D}_\mathrm{ncf} (the remaining 5%) under ∥δ∥2≤ρ\|\delta\|_2 \le \rho (Eq. 3). Linearizing the lower level gives δ=ρ∇θℓf(θ;Dncf)/∥∇θℓf(θ;Dncf)∥2\delta = \rho \nabla_\theta \ell_\mathrm{f}(\theta; \mathcal{D}_\mathrm{ncf}) / \|\nabla_\theta \ell_\mathrm{f}(\theta; \mathcal{D}_\mathrm{ncf})\|_2 (Eq. 4). Algorithm 1 computes this δt\delta_t on a non-coreset batch and then takes a gradient step on a coreset batch, evaluated at θt+δt\theta_t + \delta_t. The coreset is chosen among random, easy-example and hard-example selection (M1–M3, Table 1) by target-task performance.

Experiments fine-tune Llama-3-8B and Llama-3.2-1B on QNLI, MNLI, MultiRC and OBQA, and report the target accuracy Accf\mathrm{Acc}_\mathrm{f} and the average accuracy Accu\mathrm{Acc}_\mathrm{u} over the other 19 tasks. On average, FUEL raises Accu\mathrm{Acc}_\mathrm{u} over Standard FT and Coreset FT by about 6.6 points on Llama-3-8B and 4.5 points on Llama-3.2-1B, with Accf\mathrm{Acc}_\mathrm{f} within 0.8 points of Coreset FT (Table 2). On Llama-2-7B-Chat fine-tuned on SQL-Gen and GSM8K, FUEL gives the lowest HarmBench attack success rate (ASR) and StrongREJECT score of the three methods (Table 3).

Critical strengths
  1. FUEL gives consistent gains in non-target accuracy at almost no cost in target accuracy, using only the target data.

    • Capability control. In Table 2, FUEL has the highest Accu\mathrm{Acc}_\mathrm{u} in all eight settings (four target tasks, two models), 4.1 to 7.3 points above Coreset FT. Its Accf\mathrm{Acc}_\mathrm{f} is within 0.8 points of Coreset FT, and at or above Standard FT in seven of the eight settings.
    • The gains are broad. Among the target–non-target pairs where Standard FT loses at least 10 points, FUEL reduces the loss in 33 of 36 pairs on Llama-3-8B and 27 of 28 on Llama-3.2-1B (Appendix A).
    • The effect follows ρ\rho. In Figure 4, as ρ\rho grows from 0 to 0.005, Accu\mathrm{Acc}_\mathrm{u} rises from 53.9 to 59.5 while QNLI accuracy stays at 92.2–92.3.
    • Safety. Compared with Standard FT, FUEL lowers HarmBench ASR from 39.0 to 30.5 (SQL-Gen) and from 33.5 to 23.5 (GSM8K), with target performance within 0.6 points (Table 3).

    The method needs one extra gradient computation per step, no auxiliary data and no knowledge of which capabilities to keep. This is the main reason the paper is of practical interest.

  2. The setting is realistic, and the paper shows clearly that coreset selection alone does not help.

    • In the target-data-only setting, the capabilities at risk are not known in advance, which matches how fine-tuning is usually done. Figure 5 and Appendix A show that these capabilities differ by target task and by model: CSQA and SST-2 drop strongly on Llama-3-8B but much less on Llama-3.2-1B.
    • Table 2 supports the negative result of Section 3: in all eight settings, the Accu\mathrm{Acc}_\mathrm{u} of Coreset FT is within 0.3 points of Standard FT.
    • On the pair used in the oracle study, FUEL keeps SciQ at about 91 (a 6-point loss, Figure 5) with OBQA at 56.8. This is better on both metrics than any oracle coreset in Figure 3, for example about 88.5 and 49 at a 40% ratio.

    This makes the research question well posed and shows that FUEL does more than data selection.

Critical weaknesses
  1. The gains are not shown to come from unlearning the non-coreset. Algorithm 1 is a SAM update, and the paper does not compare it with SAM or with other perturbations.

    • The update is the SAM update. Lines 3–8 of Algorithm 1 are the SAM update (Foret et al., 2021), except that the ascent gradient is computed on a non-coreset batch rather than on the coreset batch used for descent. Section 4 notes that SAM uses the same linearization, but SAM is not a baseline. To first order, gt≈∇θℓf(θt;Btcf)+Htδtg_t \approx \nabla_\theta \ell_\mathrm{f}(\theta_t; \mathcal{B}^\mathrm{cf}_t) + H_t \delta_t, where HtH_t is the Hessian of the coreset batch loss. This is the same curvature term that SAM adds, and SAM has already been proposed to reduce forgetting in LLM fine-tuning (Li et al., 2024, "Revisiting Catastrophic Forgetting in Large Language Model Tuning", Findings of EMNLP).
    • The returned model is never unlearned. Algorithm 1 outputs θT\theta_T. The perturbation δt\delta_t only sets the point where the coreset gradient is evaluated (line 7) and is never added to the weights (line 8). So the returned model is trained only by descent on coreset batches. The paper does not report whether its loss on Dncf\mathcal{D}_\mathrm{ncf} is higher than that of Coreset FT, which is what "unlearning the non-coreset" would imply.
    • A random non-coreset already gives most of the gain. With M1, Dncf\mathcal{D}_\mathrm{ncf} is a random 5% of the same task, so its gradient is another estimate of the same target-task gradient, and FUEL becomes SAM with separate batches for the ascent and descent steps. Even so, on Llama-3-8B, FUEL with M1 has Accu\mathrm{Acc}_\mathrm{u} 3.2 to 6.8 points above Coreset FT in Table 2 (Figure A2). On MultiRC, the three strategies give almost the same Accu\mathrm{Acc}_\mathrm{u} (55.8–56.2).
    • Where the choice of non-coreset matters, the pattern reverses between tasks. For QNLI and MNLI, the best non-coreset is the 5% of examples with the highest loss under θ0\theta_0 (coreset M2), and the 5% with the lowest loss (coreset M3) is the worst. For OBQA it is the other way round (Figure A2). The paper explains this only by noting that the ranking matches that of Coreset FT, based on single runs.

    Without SAM, a random direction of norm ρ\rho, and a perturbation computed on coreset data as controls, the second contribution and the claim that this "constructive role of unlearning has not been previously explored" (Section 2) are not supported. The gains may instead come from the known effect of SAM-type training on forgetting. This point decides whether the paper shows a new role for unlearning or applies a known flatness-seeking update.

  2. The empirical evidence is narrow, and key parts of the protocol are not described.

    • No other methods are compared. The only baselines are Standard FT and Coreset FT. Section 2 lists methods that need only the target data and the base model, such as parameter-efficient adaptation and interpolation with the base model (Wortsman et al., 2022). The paper sets them aside as "potentially limiting adaptation or relying on the base model as a trusted anchor". But FUEL also relies on θ0\theta_0 (M2 and M3 rank examples by their loss under θ0\theta_0), and a limit on adaptation can be checked at matched Accf\mathrm{Acc}_\mathrm{f}. In Section 5.2, the capability to keep (safety) is known in advance, so the paper's argument that the capabilities to preserve must be specified a priori does not apply there. The constrained fine-tuning objective of Qi et al. (2025) and the prompt-template strategy of Lyu et al. (2024), both cited, need only the fine-tuning data and the aligned model, but neither is compared.

    • Single runs, and choices made on the reported numbers. No seeds or variances are reported. The ρ=0.005\rho = 0.005 point in Figure 4 equals the Table 2 entry for QNLI (92.2 and 59.5), so ρ\rho appears to be chosen on the reported numbers. This point also lies just before a sharp drop: QNLI accuracy is about 90.6 at ρ=0.006\rho = 0.006 and about 88.0 at ρ=0.007\rho = 0.007. The same ρ\rho is used for all tasks and both models, but the paper does not show where the drop lies for them. For the coreset strategy, Section 4 says it is chosen by validation performance, while Section 5.1 says it is the one "attaining the highest Accf\mathrm{Acc}_\mathrm{f}". Several conclusions rest on small gaps, such as the M1–M3 ranking on MultiRC used in Appendix B (84.0/83.6/82.8 and 56.2/56.0/55.8), and the ASR gaps of 5.5 to 6.0 points over Coreset FT on 400 HarmBench behaviors (22 to 24 behaviors).

    • It is unclear what Accu\mathrm{Acc}_\mathrm{u} measures. The paper does not say how the 20 tasks are evaluated (prompt, number of shots, scoring by log-likelihood or by generation, split). Several of the largest drops end near or below chance level.

      • After fine-tuning on QNLI, BoolQ falls from about 82 to 41 (Figure 2), below the 50% of random guessing on a yes/no task.
      • After fine-tuning on MNLI, CSQA falls by 52 points from about 71, to about 19 on a five-choice task, and SST-2 falls by 35 points from about 87, to about 52 (Figures 2 and 5).

      Such values suggest that the model now prefers one answer option, rather than that it has lost the knowledge. This matters because part of FUEL's advantage lies in these cells. For QNLI on Llama-3-8B, BoolQ and CSQA account for 66 of the 113 summed points by which FUEL's per-task changes exceed those of Standard FT (Figure 5).

    • The first contribution rests on one oracle experiment. Figure 3 uses one model, one target–non-target pair, and a selection score computed only at θ0\theta_0. It has points at 0, 40, 60, 80 and 100% only, so the statement about "ratios above 90%" is not backed by an oracle point (the 100% point is Standard FT). At 80%, the oracle coreset keeps OBQA at about 53.5 (54.2 with full data) and raises SciQ from about 80 to 82.5. The paper does not say whether the number of training steps is fixed across ratios. These results show that data selection is insufficient in this case, but not the general claim that "even an oracle coreset constructed with privileged preservation knowledge cannot prevent capability degradation" (Section 1).

    These gaps make it hard to judge how large and how general the benefit is.

Questions for authors
  1. Under the same coreset, learning rate and number of steps, with ρ\rho tuned for each, how does FUEL compare with the following?

    • Standard SAM, with δt\delta_t computed on the coreset batch itself.
    • δt\delta_t computed on a random 5% of Dcf\mathcal{D}_\mathrm{cf} instead of on Dncf\mathcal{D}_\mathrm{ncf}.
    • A random direction of norm ρ\rho.
    • δt\delta_t with the opposite sign, i.e., descent instead of ascent on the non-coreset.

    Please also clarify:

    • Is the evaluated model θT\theta_T (Algorithm 1) or θT+δ∗\theta_T + \delta^* (Eq. 3)?
    • What is the loss of the final FUEL model on Dncf\mathcal{D}_\mathrm{ncf}, compared with Coreset FT?
    • Why is the best non-coreset the highest-loss 5% for QNLI and MNLI, but the lowest-loss 5% for OBQA?

    If SAM or a perturbation computed on coreset data matches FUEL, I would read the gains as a flatness effect rather than a role of unlearning. If the non-coreset gradient clearly beats these controls, I would raise my score.

  2. Can the authors report the mean and standard deviation over at least three seeds for Tables 2 and 3, and add baselines at matched Accf\mathrm{Acc}_\mathrm{f}?

    • For Section 5.1: LoRA, interpolation with θ0\theta_0 (Wortsman et al., 2022), SAM, and self-distillation fine-tuning (Yang et al., 2024, ACL), which also uses only the target data and the model itself.
    • For Section 5.2: the constrained fine-tuning objective of Qi et al. (2025) and the prompt-template strategy of Lyu et al. (2024).

    Please also clarify:

    • On which split were ρ\rho and the coreset strategy chosen, and is it separate from the split used to report Accf\mathrm{Acc}_\mathrm{f}?
    • Does the trade-off in Figure 4 look the same for the other target tasks and for Llama-3.2-1B?
    • Which ρ\rho is used in Section 5.2, where the learning rate (3×10−53 \times 10^{-5}) is six times larger than in Section 5.1?

    Gains that hold across seeds and against these baselines would raise my score.

  3. (Clarification)

    • How are the 20 tasks evaluated: prompt, number of shots, log-likelihood or generation, split, and number of examples? For pairs where accuracy ends near chance level (e.g., QNLI→BoolQ, MNLI→CSQA, MNLI→SST-2), does the model choose the same answer option for almost all inputs?
    • What are the training details: full-parameter or LoRA fine-tuning, number of epochs or steps, batch sizes of Btncf\mathcal{B}^\mathrm{ncf}_t and Btcf\mathcal{B}^\mathrm{cf}_t, and optimizer?
    • What is the training cost of FUEL per step relative to Standard FT (it needs two forward and backward passes per step)? What is the total cost of choosing among M1–M3, given that M1 is itself the best of several random trials?
    • In Figure 3, is the number of training steps fixed across coreset ratios, and what are the results at 90% and 95%?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 5X2s (Part 1)

We thank Reviewer 5X2s for the detailed and constructive review, and for recognizing the consistent gains of FUEL in non-target capabilities and safety at almost no cost in target accuracy using only the target data, the practical appeal of requiring no auxiliary data or prior knowledge of which capabilities to preserve, and the clear evidence that coreset selection alone is insufficient. Below, we address each of the reviewer's weaknesses (W) or questions (Q) and provide our corresponding responses (A).

Q1: The gains are not shown to come from unlearning the non-coreset. Algorithm 1 is a SAM update, and the paper does not compare it with SAM or with other perturbations.

A1: We thank the reviewer for this insightful question. We agree that FUEL reduces to SAM [1] when the follower unlearns the same data used by the leader (i.e., Dncf=DcfD_{\mathrm{ncf}} = D_{\mathrm{cf}} in Eq. (3)). However, the key distinction lies in the role of non-coreset unlearning in constructing the perturbation, rather than perturbation-based regularization alone.

Methodologically, as shown in Eq. (4), FUEL's perturbation corresponds to a gradient-ascent step on the non-coreset data DncfD_{\mathrm{ncf}}, explicitly introducing an unlearning-induced perturbation into the leader's optimization. In contrast, standard SAM constructs its perturbation using the same data batch as the outer minimization to seek flat minima. This choice matters since existing theoretical analyses of SAM's sharpness-reduction properties [2, 3] rely on the perturbation being computed from the gradient of the loss being minimized. FUEL's perturbation instead comes from the non-coreset loss, reflecting a different mechanism.

Empirically, to isolate the contribution of non-coreset unlearning from generic perturbation-based regularization, we conduct additional ablation studies in Tables R1 and R2, comparing FUEL against four SAM-related variants using different perturbation-based regularization schemes:

  1. Standard SAM: Performs inner maximization and outer minimization on the same training batch, without distinguishing coreset and non-coreset samples.
  2. Coreset-based SAM: Applies standard SAM to the coreset only, performing inner maximization and outer minimization on the same coreset batch. This controls for the effect of training on the coreset.
  3. SAM (separate-batch perturbation) [4]: Computes the inner-maximization perturbation using a separate, randomly sampled coreset batch, while retaining coreset-based outer minimization. This controls for the effect of using different batches for perturbation and optimization within the coreset.
  4. SAM (random perturbation): Replaces the gradient-based inner-maximization direction with a random direction of the same norm, isolating the effect of perturbation-based regularization without an unlearning signal.

These comparisons disentangle two critical factors: (i) the importance of selecting non-coreset data for the inner maximization, through comparisons with standard SAM, coreset-based SAM, and SAM (separate-batch perturbation); and (ii) the importance of the unlearning direction itself, through comparison with SAM (random perturbation).

AuthorsOpenReview ↗

Response to Reviewer 5X2s (Part 2)

A1 (continued): As shown in Tables R1 and R2, FUEL consistently outperforms all four SAM variants in preserving non-target capabilities (measured by Accu\mathrm{Acc}_{\mathrm{u}}) and model safety, while maintaining competitive target-task performance. In contrast, the SAM variants do not exhibit comparable improvements in capability control or safety preservation. These results suggest that FUEL's advantage cannot be explained solely by generic weight perturbations or sharpness-aware regularization; rather, the non-coreset-derived unlearning direction plays a critical role.

We believe that distinguishing coreset from non-coreset data enables a complementary interaction between outer minimization (target-task learning) and inner maximization (non-coreset unlearning). This coupling provides a corrective signal that is absent from conventional SAM variants, helping mitigate unintended changes to non-target capabilities. Our results thus point to an important yet largely overlooked principle: data excluded from fine-tuning can still provide valuable corrective signals through unlearning-induced optimization.

Table R1: Ablation on the source of the perturbation in FUEL for general capability preservation, with Llama-3-8B fine-tuned on MNLI and OpenBookQA. The result format follows Table 2, with Accf\mathrm{Acc}_{\mathrm{f}} the target-task accuracy and Accu\mathrm{Acc}_{\mathrm{u}} the average accuracy over the 19 non-target tasks. Results are mean ± std over four seeds.

MethodMNLI Accf\mathrm{Acc}_{\mathrm{f}}MNLI Accu\mathrm{Acc}_{\mathrm{u}}OpenBookQA Accf\mathrm{Acc}_{\mathrm{f}}OpenBookQA Accu\mathrm{Acc}_{\mathrm{u}}
Base (w/o FT)47.564.233.465.0
Standard SAM86.8 ± 0.954.4 ± 0.655.8 ± 1.355.2 ± 0.7
Coreset-based SAM87.1 ± 0.654.4 ± 0.555.7 ± 1.454.0 ± 0.7
SAM (separate-batch perturbation)87.1 ± 0.954.0 ± 0.956.0 ± 1.854.2 ± 1.1
SAM (random perturbation)85.2 ± 1.253.4 ± 1.054.5 ± 1.953.8 ± 1.2
FUEL (ours)87.2 ± 1.161.1 ± 0.656.8 ± 1.460.8 ± 0.6

Table R2: Ablation on the source of the perturbation in FUEL for safety preservation, with Llama-2-7B-Chat fine-tuned on GSM8K. The result format follows Table 3, with GSM8K accuracy for the target task and HarmBench ASR and StrongREJECT score for safety. Results are mean ± std over four seeds.

MethodGSM8K Acc ↑HarmBench ASR ↓StrongREJECT Score ↓
Base (w/o FT)27.11.02.1
Standard SAM32.6 ± 0.534.0 ± 2.115.9 ± 0.6
Coreset-based SAM32.8 ± 0.532.0 ± 1.915.8 ± 0.4
SAM (separate-batch perturbation)31.5 ± 1.333.7 ± 2.514.5 ± 1.2
SAM (random perturbation)32.8 ± 1.633.2 ± 4.116.1 ± 2.2
FUEL (ours)33.0 ± 0.923.5 ± 2.311.9 ± 0.9

Q2: Is the evaluated model θT\theta_T (Algorithm 1) or θT+δ∗\theta_T + \delta^{\ast} (Eq. 3)? The returned model is never unlearned, since the perturbation is never added to the weights.

A2: We evaluate θT\theta_T from Algorithm 1. FUEL does not aim to make the final model forget DncfD_{\mathrm{ncf}}. The unlearning instead acts through the coupling between the follower's inner maximization on the non-coreset and the leader's outer minimization on the coreset, which provides a corrective signal that helps mitigate unintended changes to non-target capabilities.

AuthorsOpenReview ↗

Response to Reviewer 5X2s (Part 3)

Q3: No other methods are compared. The only baselines are Standard FT and Coreset FT.

A3: Following your suggestion, we compare FUEL against three anti-forgetting methods in Tables R3 and R4, which correspond to the capability preservation and safety preservation experiments in our main paper, respectively. Like FUEL, all of them require only the target data and the base model. Among the studied anti-forgetting methods, SAM [1,5] formulates training as a min-max optimization problem, where the inner maximization and outer minimization operate on the same dataset, unlike FUEL, which uses the inner maximization to perform unlearning on non-coreset samples. FLOW [6] upweights samples with low loss under the pretrained model, while TALR [7] downweights hard tokens. Both were reported to achieve a better trade-off between target performance and forgetting than LoRA and interpolation with the base model [6, 7], two of the baselines suggested. All methods share the same base model, learning rate, batch size, and number of training steps.

As shown in Tables R3 and R4, FUEL consistently outperforms all baselines in preserving both general capabilities and safety, while maintaining comparable target accuracy.

Table R3: Comparison with anti-forgetting methods for general capability preservation, with Llama-3-8B fine-tuned on MNLI and OpenBookQA. The result format follows Table R1.

MethodMNLI Accf\mathrm{Acc}_{\mathrm{f}}MNLI Accu\mathrm{Acc}_{\mathrm{u}}OpenBookQA Accf\mathrm{Acc}_{\mathrm{f}}OpenBookQA Accu\mathrm{Acc}_{\mathrm{u}}
Base (w/o FT)47.564.233.465.0
Standard FT84.7 ± 0.854.0 ± 0.654.2 ± 1.353.8 ± 0.6
Coreset FT87.4 ± 0.853.8 ± 0.557.0 ± 1.454.0 ± 0.7
Standard SAM86.8 ± 0.954.4 ± 0.655.8 ± 1.355.2 ± 0.7
FLOW86.3 ± 0.554.6 ± 0.457.1 ± 1.254.5 ± 0.4
TALR87.4 ± 0.756.1 ± 0.556.6 ± 1.256.2 ± 0.6
FUEL (ours)87.2 ± 1.161.1 ± 0.656.8 ± 1.460.8 ± 0.6

Table R4: Comparison with anti-forgetting methods for safety preservation, with Llama-2-7B-Chat fine-tuned on GSM8K. The result format follows Table R2.

MethodGSM8K Acc ↑HarmBench ASR ↓StrongREJECT Score ↓
Base (w/o FT)27.11.02.1
Standard SAM32.6 ± 0.534.0 ± 2.115.9 ± 0.6
FLOW33.8 ± 0.436.5 ± 1.518.0 ± 0.8
TALR29.4 ± 1.131.0 ± 1.612.6 ± 0.8
FUEL (ours)33.0 ± 0.923.5 ± 2.311.9 ± 0.9

Q4: Choices made on the reported numbers. The ρ = 0.005 point in Figure 4 equals the Table 2 entry for QNLI, so ρ appears to be chosen on the reported numbers, and this point lies just before a sharp drop.

A4: We thank the reviewer for raising this question. We apologize for the lack of clarity in our presentation of Figure 4. In practice, we indeed select the perturbation strength ρ solely based on target-task validation performance, without accessing any non-target data or benchmarks. Specifically, we choose the largest ρ that does not degrade target-task validation accuracy, consistently across both general-capability and safety-preservation experiments. This follows the same principle as our coreset selection strategy, where target-task performance serves as the sole accessible criterion (Lines 286–292). For both choices, the validation split is held out from the training data and is separate from the split used to report Accf\mathrm{Acc}_{\mathrm{f}}.

The ρ = 0.005 point in Figure 4 comes from the same runs as Table 2, but Figure 4 reports non-target accuracy to illustrate the trade-off across different strengths of ρ, rather than to select ρ. In fact, the results support the effectiveness of our target-only selection criterion. On QNLI, target accuracy remains nearly unchanged for ρ ≤ 0.005 while non-target accuracy steadily improves as ρ increases. Beyond this threshold, target accuracy drops sharply with marginal additional gains in non-target accuracy. The criterion detects this drop from target accuracy alone and gives ρ = 0.005.

Reviewer_wavqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies capability preservation during large language model fine-tuning when only target-task data are available and the non-target capabilities to preserve are unspecified. The proposed method, FUEL, partitions the data into a coreset and a non-coreset, constructs a bounded parameter perturbation using the loss-ascent direction on the non-coreset, and computes the coreset training gradient at the perturbed parameters. Experiments report improved non-target performance relative to standard and coreset fine-tuning while largely maintaining target-task performance. Additional experiments examine safety preservation under benign fine-tuning.

Critical strengths

1.The paper addresses a practically relevant problem: fine-tuners may have access only to target-task data and may not know which other capabilities need protection. 2.The writing is generally fluent, and the overall argument is clear. The progression from fine-tuning-induced degradation to the limitations of coreset selection and the proposed method is easy to follow. 3.The reported results show consistent improvements in average non-target accuracy across two model scales and several target tasks, while largely maintaining target-task performance. The safety experiments provide an additional perspective on the method’s potential utility.

Critical weaknesses

1.Limited comparisons with existing anti-forgetting methods. This is my main concern. The experiments mainly compare FUEL with Standard FT and Coreset FT. These comparisons demonstrate benefits over ordinary fine-tuning and data selection alone, but do not sufficiently establish its incremental value over existing capability-preservation approaches. I recommend adding at least one representative anti-forgetting method under comparable data-access assumptions and training conditions, reporting both target and non-target performance alongside computational costs. 2.Missing split-ratio ablation. The capability and safety experiments use 95%/5% and 90%/10% splits, respectively, but also differ in models and tasks. Since data partitioning is central to FUEL, comparing several ratios while holding the model, task, and selection strategy fixed would clarify the rationale for these choices and the method’s sensitivity. 3.Incomplete experimental details. The paper should specify whether it uses full-parameter or parameter-efficient fine-tuning, along with the optimizer, batch sizes, and training duration for the general-capability experiments. Since these choices can directly affect the extent of forgetting, I recommend explicitly documenting them in the paper.

Questions for authors

1.How does FUEL compare with a representative anti-forgetting or capability-preservation method under comparable conditions? 2.How sensitive are the results to the coreset/non-coreset ratio, and how were the default ratios selected? 3.Could the authors provide the missing training configurations and clarify whether non-target evaluation results were used for hyperparameter or checkpoint selection?

4 comments
AuthorsOpenReview ↗

Response to Reviewer wavq (Part 1)

We thank Reviewer wavq for recognizing the practical relevance of the target-data-only setting, the clarity of the writing, and the consistent improvements in non-target performance across model scales and target tasks, together with the additional perspective offered by the safety experiments. Below, we address each of the reviewer's weaknesses (W) or questions (Q) and provide our corresponding responses (A).

W1 (Q1): Limited comparisons with existing anti-forgetting methods. I recommend adding at least one representative anti-forgetting method under comparable data-access assumptions and training conditions, reporting both target and non-target performance alongside computational costs.

A1: Thanks for raising this insightful question. Following your suggestion, we compare FUEL against three anti-forgetting methods in Tables R1 and R2, which correspond to the capability preservation and safety preservation experiments in our main paper, respectively. Among the studied anti-forgetting methods, SAM [1,2] formulates training as a min-max optimization problem, where the inner maximization and outer minimization operate on the same dataset, unlike FUEL, which uses the inner maximization to perform unlearning on non-coreset samples. FLOW [3] upweights samples with low loss under the pretrained model, while TALR [4] downweights hard tokens. All methods share the same base model, learning rate, batch size, and number of training steps.

Table R1: Comparison with anti-forgetting methods for general capability preservation, with Llama-3-8B fine-tuned on MNLI and OpenBookQA. The result format follows Table 2, with Accf\mathrm{Acc}_{\mathrm{f}} the target-task accuracy and Accu\mathrm{Acc}_{\mathrm{u}} the average accuracy over the 19 non-target tasks. Results are mean ± std over four seeds.

MethodMNLI Accf\mathrm{Acc}_{\mathrm{f}}MNLI Accu\mathrm{Acc}_{\mathrm{u}}OpenBookQA Accf\mathrm{Acc}_{\mathrm{f}}OpenBookQA Accu\mathrm{Acc}_{\mathrm{u}}
Base (w/o FT)47.564.233.465.0
Standard FT84.7 ± 0.854.0 ± 0.654.2 ± 1.353.8 ± 0.6
Coreset FT87.4 ± 0.853.8 ± 0.557.0 ± 1.454.0 ± 0.7
Standard SAM86.8 ± 0.954.4 ± 0.655.8 ± 1.355.2 ± 0.7
FLOW86.3 ± 0.554.6 ± 0.457.1 ± 1.254.5 ± 0.4
TALR87.4 ± 0.756.1 ± 0.556.6 ± 1.256.2 ± 0.6
FUEL (ours)87.2 ± 1.161.1 ± 0.656.8 ± 1.460.8 ± 0.6

Table R2: Comparison with anti-forgetting methods for safety preservation, with Llama-2-7B-Chat fine-tuned on GSM8K. The result format follows Table 3, with GSM8K accuracy for the target task and HarmBench ASR and StrongREJECT score for safety. Results are mean ± std over four seeds.

MethodGSM8K Acc ↑HarmBench ASR ↓StrongREJECT Score ↓
Base (w/o FT)27.11.02.1
Standard SAM32.6 ± 0.534.0 ± 2.115.9 ± 0.6
FLOW33.8 ± 0.436.5 ± 1.518.0 ± 0.8
TALR29.4 ± 1.131.0 ± 1.612.6 ± 0.8
FUEL (ours)33.0 ± 0.923.5 ± 2.311.9 ± 0.9

As shown in Tables R1 and R2, FUEL consistently outperforms all baselines in preserving both general capabilities and safety, while maintaining comparable target accuracy. Regarding computational cost, we measure the wall-clock training time of each method in the setting of Table R1, fine-tuning Llama-3-8B on MNLI and OpenBookQA, and report the average over the two tasks relative to Standard FT. TALR adds negligible overhead (≈1.0×), and FLOW costs ≈1.2× because it needs one forward pass of the base model over the training set to compute sample weights. SAM and FUEL both require two forward–backward passes per step, costing ≈2.0× and ≈2.2×, respectively. FUEL's extra 0.2× comes from a similar one-time forward pass to select the coreset.

AuthorsOpenReview ↗

Response to Reviewer wavq (Part 2)

A1 (continued): We attribute FUEL's advantage to the corrective signal provided by unlearning the excluded data, which is absent from all competing baselines. This highlights a simple yet largely overlooked principle in model fine-tuning: leveraging excluded data through unlearning to mitigate forgetting. To verify this, we keep Algorithm 1 unchanged and only change how the perturbation is computed:

  1. Standard SAM: Performs inner maximization and outer minimization on the same training batch, without distinguishing coreset and non-coreset samples.
  2. Coreset-based SAM: Applies standard SAM to the coreset only, performing inner maximization and outer minimization on the same coreset batch. This controls for the effect of training on the coreset.
  3. SAM (separate-batch perturbation): Computes the inner-maximization perturbation using a separate, randomly sampled coreset batch, while retaining coreset-based outer minimization. This controls for the effect of using different batches for perturbation and optimization within the coreset.
  4. SAM (random perturbation): Replaces the gradient-based inner-maximization direction with a random direction of the same norm, isolating the effect of perturbation-based regularization without an unlearning signal.

These comparisons disentangle two critical factors: (i) the importance of selecting non-coreset data for the inner maximization, through comparisons with standard SAM, coreset-based SAM, and SAM (separate-batch perturbation); and (ii) the importance of the unlearning direction itself, through comparison with SAM (random perturbation).

As shown in Tables R3 and R4, FUEL consistently outperforms all four SAM variants in preserving non-target capabilities (measured by Accu\mathrm{Acc}_{\mathrm{u}}) and model safety, while maintaining competitive target-task performance. In contrast, the SAM variants do not exhibit comparable improvements in capability control or safety preservation. These results suggest that FUEL's advantage cannot be explained solely by generic weight perturbations or sharpness-aware regularization; rather, the non-coreset-derived unlearning direction plays a critical role.

Table R3: Ablation on the source of the perturbation in FUEL for general capability preservation, with Llama-3-8B fine-tuned on MNLI and OpenBookQA. The result format follows Table R1.

MethodMNLI Accf\mathrm{Acc}_{\mathrm{f}}MNLI Accu\mathrm{Acc}_{\mathrm{u}}OpenBookQA Accf\mathrm{Acc}_{\mathrm{f}}OpenBookQA Accu\mathrm{Acc}_{\mathrm{u}}
Base (w/o FT)47.564.233.465.0
Standard SAM86.8 ± 0.954.4 ± 0.655.8 ± 1.355.2 ± 0.7
Coreset-based SAM87.1 ± 0.654.4 ± 0.555.7 ± 1.454.0 ± 0.7
SAM (separate-batch perturbation)87.1 ± 0.954.0 ± 0.956.0 ± 1.854.2 ± 1.1
SAM (random perturbation)85.2 ± 1.253.4 ± 1.054.5 ± 1.953.8 ± 1.2
FUEL (ours)87.2 ± 1.161.1 ± 0.656.8 ± 1.460.8 ± 0.6

Table R4: Ablation on the source of the perturbation in FUEL for safety preservation, with Llama-2-7B-Chat fine-tuned on GSM8K. The result format follows Table R2.

MethodGSM8K Acc ↑HarmBench ASR ↓StrongREJECT Score ↓
Base (w/o FT)27.11.02.1
Standard SAM32.6 ± 0.534.0 ± 2.115.9 ± 0.6
Coreset-based SAM32.8 ± 0.532.0 ± 1.915.8 ± 0.4
SAM (separate-batch perturbation)31.5 ± 1.333.7 ± 2.514.5 ± 1.2
SAM (random perturbation)32.8 ± 1.633.2 ± 4.116.1 ± 2.2
FUEL (ours)33.0 ± 0.923.5 ± 2.311.9 ± 0.9
AuthorsOpenReview ↗

Response to Reviewer wavq (Part 3)

W2 (Q2): Missing split-ratio ablation. How sensitive are the results to the coreset/non-coreset ratio, and how were the default ratios selected?

A2: Thank you for the suggestion. We ran the requested ablation in the application of preserving safety robustness against GSM8K fine-tuning in Sec. 5.2. The model (Llama-2-7B-Chat), task and selection strategy (M2) are fixed, and only the coreset ratio varies, from 30% to 100%. Figure R1 shows HarmBench ASR (i.e., safety measurement) vs. GSM8K fine-tuning accuracy for each ratio. The 100% point corresponds to Standard FT.

Figure R1: Sensitivity of FUEL to the coreset ratio for safety preservation, with Llama-2-7B-Chat fine-tuned on GSM8K. Only the coreset ratio varies and is shown in percent next to each point, while all other settings follow Sec. 5.2 with M2 selection. The x-axis shows the HarmBench ASR, where lower is safer, and the y-axis shows the GSM8K accuracy. The 100% point corresponds to Standard FT.

As we can see, FUEL is not quite sensitive to the ratio over a broad range. From 90% to 98%, GSM8K accuracy remains comparable to the Standard FT, while HarmBench ASR is substantially reduced. Below 90%, targeted GSM8K performance starts to decline more sharply, which makes 90% the knee of the trade-off without losing the target-task performance.

As demonstrated above, we selected default ratios using target-task performance as an indicator. We keep the coreset large enough that target accuracy stays on par with Standard FT (the lossless criterion in Sec. 4), and we do not use any non-target metric.

W3 (Q3): Incomplete experimental details. The paper should specify whether it uses full-parameter or parameter-efficient fine-tuning, along with the optimizer, batch sizes, and training duration for the general-capability experiments. Could the authors provide the missing training configurations and clarify whether non-target evaluation results were used for hyperparameter or checkpoint selection?

A3: All experiments use full-parameter fine-tuning with the AdamW optimizer on 4 NVIDIA H200 GPUs. In the general-capability experiments, all methods are trained for a fixed 500 steps with a learning rate of 5×10⁻⁶ and a batch size of 16. For FUEL, the coreset and non-coreset batches both have size 16. The safety experiments use the same configuration, except that models are trained for one epoch with a learning rate of 3×10⁻⁵.

Non-target results are not used for any hyperparameter or checkpoint selection. All methods are evaluated at the final checkpoint of the fixed training budget, and all hyperparameters are chosen using target-task validation performance only. As discussed in A2, the coreset ratio is kept large enough that target validation accuracy stays on par with Standard FT. Also, we select the coreset strategy (M1–M3) with the best target validation accuracy and set ρ to the largest value that does not reduce it. For example, on QNLI the target accuracy stays nearly unchanged up to ρ = 0.005 and drops beyond it (Fig. 4), so this criterion alone gives ρ = 0.005.

References

[1]. Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-Aware Minimization for Efficiently Improving Generalization. ICLR, 2021.

[2]. Hongyu Li, Liang Ding, Meng Fang, and Dacheng Tao. Revisiting Catastrophic Forgetting in Large Language Model Tuning. Findings of EMNLP, 2024.

[3]. Sunny Sanyal, Hayden Prairie, Rudrajit Das, Ali Kavis, and Sujay Sanghavi. Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting. ICML, 2025.

[4]. Jiacheng Lin, Zhongruo Wang, Kun Qian, et al. SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs. ICLR, 2026.

AuthorsOpenReview ↗

Look forward to your feedback!

Dear Reviewer wavq,

Thank you for your insightful comments and for taking the time to review our paper.

We submitted a detailed rebuttal addressing the key questions and concerns you raised, together with additional comparisons with anti-forgetting methods (Tables R1 and R2), ablation studies (Tables R3 and R4), and a sensitivity analysis of the coreset ratio (Figure R1).

As the author–reviewer discussion period is approaching its conclusion, we would be grateful to know whether there are any remaining concerns or questions that we have not yet adequately addressed.

Thank you again for your consideration.

Kind regards,

Authors

Comments on the paper

1 comment

We thank the reviewers for their constructive comments. Based on their feedback, we have revised the manuscript and incorporated some of the additional experimental results and clarifications to address the concerns raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.