acceptodds
Under review as a conference paper at ICLR 2027

When Unlearning Meets the Harness: Helpful Assistance, Harmful Recovery

Abstract

Large language model (LLM) unlearning aims to mitigate privacy and safety risks by selectively removing targeted knowledge or capabilities while preserving useful behavior. In deployment, LLM agents rely on harnesses that provide reasoning guidance and output repair to improve task execution. However, the security issues of this assistance remain insufficiently understood, leaving a substantial gap between model-level unlearning and robustness in deployed agents. Using tool unlearning as a concrete setting, we identify a failure mode harness-mediated recovery: harness assistance intended to improve task performance restores apparently suppressed tool-use capabilities without changing model parameters, optimizing adversarial prompts, or retrieving forgotten content. Experiments on ToolAlpaca and the Berkeley Function Calling Leaderboard (BFCL) reveal two mechanisms for this recovery: reasoning assistance before generation elicits suppressed tool calls, while schema repair after generation reconstructs executable calls from malformed outputs. Therefore, we formulate unlearning robustness at the level of the complete model–harness system and introduce the Harness Recovery Rate (HRR) to quantify recovery among forget-set examples that fail under the default interface. We further propose Harness-Aware Robust Unlearning (HARU), which incorporates harness-recoverable behaviors into the unlearning objective. Across benchmarks, HARU strengthens robustness to harness-mediated recovery and achieves strong forgetting while preserving substantial retained-tool utility. Our findings further demonstrate that harness environments constitute a critical and underexplored security dimension, where deployment-time interactions can fundamentally alter intended security properties.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent, Unlearning, Harness

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_1tgSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the robustness of tool unlearning in LLM agents equipped with deployment harnesses. The authors identify a failure mode termed harness-mediated recovery, where reasoning assistance or schema repair can restore tool-use capabilities that appear to have been suppressed by unlearning. To evaluate this phenomenon, the paper introduces the Harness Recovery Rate (HRR) and proposes Harness-Aware Robust Unlearning (HARU), which incorporates harness-recoverable behaviors into the unlearning process. Experiments on ToolAlpaca and BFCL demonstrate that harness assistance can substantially increase the success rate of forgotten tool calls, while HARU reduces this recovery under the evaluated settings.

Critical strengths
  1. This paper studies an aspect of LLM unlearning that is often overlooked in existing evaluations.
  2. This paper distinguishes between reasoning assistance before generation and schema repair after generation.
  3. Formulating unlearning robustness at the model–harness level is a useful perspective. The proposed HRR metric provides an intuitive way to measure how frequently harness assistance recovers previously unsuccessful target-tool calls.
Critical weaknesses
  1. For the preprocessing harness, the main idea is to collect harness-recoverable examples and perform additional unlearning, which resembles hard-negative mining and data augmentation. For the postprocessing harness, the method introduces an explicit no-tool training target. The paper would benefit from a clearer explanation of what is technically novel beyond these existing training strategies.
  2. The experimental comparison does not fully control for training data and optimization budgets. In Appendix A.1, HARU uses 3,794 augmented trajectories and 295 optimization updates, while the baselines use 635 original forget trajectories and 50 updates. The authors also acknowledge that this comparison cannot isolate augmentation from additional optimization.
  3. This paper lacks a systematic analysis of the individual components of HARU. In particular, it is unclear how much improvement comes from harness-guided data augmentation, additional unlearning steps, and the safe-destination loss. Comparisons with standard NPO using matched training budgets, random augmentation, or ordinary hard-negative mining would help establish the effectiveness of the proposed method.
  4. The experiments focus on two relatively specific configurations: a ToolAlpaca actor assisted by an external reasoning model and an xLAM actor with deterministic schema repair. Although additional reasoning providers are evaluated, they are all fine-tuned on ToolAlpaca. The paper does not examine more realistic agent workflows involving multiple interacting tools, iterative feedback, or different types of harness assistance. It is therefore unclear how broadly the conclusions generalize to deployed LLM agents.
Questions for authors
  1. Can the authors provide training-budget-matched comparisons to better isolate the contribution of HARU?
  2. How does HARU generalize to harnesses and tool-use scenarios that differ more substantially from those used during training?
2 comments

For the preprocessing harness, the main idea is to collect harness-recoverable examples and perform additional unlearning, which resembles hard-negative mining and data augmentation. For the postprocessing harness, the method introduces an explicit no-tool training target. The paper would benefit from a clearer explanation of what is technically novel beyond these existing training strategies.

Response: HARU is not just a combination of hard-negative mining and no-tool supervision; it provides a systematic framework for maintaining suppression of designated tool-use capabilities when an agent is equipped with a harness. Its contribution has two parts:

  1. Conceptually, HARU focuses on unlearning robustness when the agent runs with a harness. Prior tool unlearning checks forgetting only without a harness, whereas HARU requires the forgotten tools to stay suppressed even when a helpful harness is attached. Its training shares the idea of adversarial training for robust unlearning [1, 2]: the harness acts as a recovery probe that exposes the forgotten calls it can still recover, and HARU trains on exactly these calls to make forgetting robust.

  2. Methodologically, HARU formalizes tool unlearning with a harness and systematically addresses two recovery mechanisms. Preprocessing elicits suppressed calls through reasoning assistance, while postprocessing reconstructs calls from residual argument payloads. HARU addresses these mechanisms by applying NPO to harness-augmented contexts and adding a safe-destination loss that encourages no-tool outputs without repairable argument payloads. Together, these strategies approximate the harness-level unlearning objective (Eq. 2).

Can the authors provide training-budget-matched comparisons to better isolate the contribution of HARU? This paper lacks a systematic analysis of the individual components of HARU. In particular, it is unclear how much improvement comes from harness-guided data augmentation, additional unlearning steps, and the safe-destination loss. Comparisons with standard NPO using matched training budgets, random augmentation, or ordinary hard-negative mining would help establish the effectiveness of the proposed method.

Response: We thank the reviewer for this suggestion. We show that HARU's gains come from harness-guided augmentation, not from additional unlearning steps or ordinary hard-negative mining. We conducted a training-budget-matched ablation on ToolAlpaca to isolate the source of HARU's gains.

We compare three groups that start from the same initial NPO checkpoint and share the same objective, hyperparameters, and training budget. They differ only in the forget examples used for refinement:

  • A. Matched training budget (continued NPO): continues unlearning on the original forget data with a budget matched to HARU, testing whether additional unlearning steps alone explain the gains.
  • B. Ordinary hard-negative mining: samples responses without the harness and trains on those that still call a target tool, testing whether ordinary mined data suffices.
  • C. HARU (harness-guided augmentation): uses the same sampling budget and selection rule as B, but samples with the reasoning harness.
ToolAlpaca groupStepsForget ↓Retain ↑Forget with harness ↓Retain with harness ↑HRR ↓
NPO (standard, reference)5033.053.053.062.032.8
A. Matched training budget19344.061.060.068.035.7
B. Ordinary hard-negative mining17635.056.049.064.023.1
C. HARU18611.059.014.069.04.5

The matched budget covers training steps and excludes the cost of generating augmentation data. Our results show that:

  1. A vs. C (additional unlearning steps): With a matched budget, continued NPO still has an HRR of 35.7 and a forget accuracy with the harness of 60.0, versus 4.5 and 14.0 for HARU, while retained-tool accuracy with the harness is similar (68.0 vs. 69.0). Additional training alone therefore does not explain HARU's gains.
  2. B vs. C (ordinary hard-negative mining): Mining without the harness lowers HRR only to 23.1, and its forget accuracy with the harness stays at 49.0, versus 14.0 for HARU. Ordinary hard negatives are therefore not enough; the gain comes from harness-guided augmentation, which exposes the calls that the harness can actually recover.

How does HARU generalize to harnesses and tool-use scenarios that differ more substantially from those used during training?

Response: We thank the reviewer for raising the generalization concern. We already discussed this in Section 5.3 of the original submission: HARU generalizes to unseen harnesses, consistently yielding strong unlearning performance. We compare HARU (unlearned with ToolAlpaca-13B as the harness) and the strongest unlearned baseline, NPO, using four unseen harnesses from different model families to provide reasoning. HARU keeps HRR below 8% on every unseen harness, whereas NPO (strongest unlearning baseline) stays above 24%. Results are in the following table:

Reasoning harnessNPO HRR ↓HARU HRR ↓
ToolAlpaca-13B (used in unlearning)32.84.5
Qwen2.5-7B (unseen)28.65.6
Mistral-7B (unseen)27.15.6
Llama-3.1-8B (unseen)34.37.8
Gemma-4-E2B (unseen)24.34.4

References [1] Zhang et al. Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models. NeurIPS 2024.
[2] Sheshadri et al. Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs. TMLR 2025.

Reviewer_hARPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether harness assistance can recover tool-use behavior that suppressed by unlearning in inference. It compares two mechanisms that reasoning guidance before generation and deterministic schema repair after generation. This paper introduces Harness Recovery Rate (HRR), which assesses recovery forget-set examples that fail under the default interface. It also proposes Harness-Aware Robust Unlearning (HARU). Experiments on ToolAlpaca and BFCL demonstrate substantial recovery for unlearning methods and the reduction of recovery under HARU.

Critical strengths
  1. The paper identifies harness-mediated recovery as a failure mode of tool unlearning. It points out that ordinary assistance can restore apparently suppressed tool-use behavior without modifying model parameters, which reveals a relevant gap between model-level forgetting and deployed-agent behavior.

  2. This paper proposed Harness Recovery Rate HRR metric that provides a useful way to quantify recovery among examples that fail under the default interface.

  3. The paper proposes Harness-Aware Robust Unlearning (HARU) to account for harness-recoverable behavior during unlearning. The experimental results on ToolAlpaca and BFCL demonstrate promising improvements in target-tool suppression while retaining useful non-target tool capabilities.

Critical weaknesses
  1. The reported implementation is different from the proposed algorithm. In Section 4, initial unlearning is said following by iterative mining of harness-recoverable responses from the current actor with retain-set training. However, Appendix A.1 states that ToolAlpaca HARU starts from the original actor, uses a fixed augmented dataset and has no separate retain stream. Such differences make it unclear that the results validate and whether the gains arise from harness-specific augmentation, additional data, or additional training.

  2. BFCL primarily involves additional training to make the model outputing empty tool calls, which does not necessarily require the model to be "harness-aware" (i.e., recognizing the external evaluation system). To validate this, NPO (Negative Preference Optimization) should be used to compare with baselines with an equivalent computational budget, as well as standard supervision for empty tool calls. Without matched-budget NPO and empty-call supervision baselines, the results cannot distinguish the gain of harness-aware learning from ordinary suppression through a safe target.

  3. HRR evaluates recovery of complete benchmark correctness instead of any invocation of a designated tool. Responses that invokes a forbidden tool but has an incorrect argument or another erroneous step may therefore count as unsuccessful forgetting-set behavior. Furthermore, ToolAlpaca includes examples that the original actor already fails, so recovery does not always indicate reversal of unlearning. Low HRR also does not imply low absolute leakage. For example, HARU still achieves 41.0% forget-set accuracy after repair on BFCL.

Questions for authors

For the rebuttal, I encourage the authors to further clarify these points mentioned in the weakness. I am open to revisiting my score if these concerns are adequately addressed.

2 comments

The reported implementation is different from the proposed algorithm. In Section 4, initial unlearning is said following by iterative mining of harness-recoverable responses from the current actor with retain-set training. However, Appendix A.1 states that ToolAlpaca HARU starts from the original actor, uses a fixed augmented dataset and has no separate retain stream.

Response: We thank the reviewer for carefully checking these details. Our ToolAlpaca implementation is a single-iteration instance of HARU, and we will align Appendix A.1 with Section 4. We use only one mining-and-training iteration, so there is a single augmented dataset generated through harness-guided mining. This dataset is generated by our procedure rather than fixed or predefined; it remains unchanged during that iteration’s training. We use one iteration because it already achieves strong forgetting, while avoiding the additional cost of repeated mining and training.

We additionally ran three iterations following the full procedure, including initial unlearning, mining from the current actor, and retain-set training. From iteration 1 to iteration 3, assisted forget accuracy decreases from 33.0% to 26.0% and HRR drops from 22.4% to 10.8%, while maintaining strong retain performance. These results show that additional iterations further reduce harness-mediated recovery while preserving retained-tool utility.

ToolAlpaca training trajectoryForget ↓Retain ↑Forget with harness ↓Retain with harness ↑HRR ↓
Iterative variant: after iteration 133.055.046.063.022.4
Iterative variant: after iteration 234.056.042.064.016.7
Iterative variant: after iteration 326.059.033.070.010.8

Such differences make it unclear that the results validate and whether the gains arise from harness-specific augmentation, additional data, or additional training.

Response: We thank the reviewer for this question. We show that HARU's gains come from harness-specific augmentation, not from additional training or additional data. We conducted an ablation on ToolAlpaca to isolate the source of HARU's gains.

We compare three groups that start from the same initial NPO checkpoint and share the same objective, hyperparameters, and training budget. They differ only in the forget examples used for refinement:

  • A. Additional training: continues unlearning on the original forget data with a matched training budget, testing whether additional training alone explains the gains.
  • B. Additional data without the harness: samples responses without the harness and trains on those that still call a target tool, testing whether additional mined data suffices.
  • C. HARU (harness-specific augmentation): uses the same sampling budget and selection rule as B, but samples with the reasoning harness.
ToolAlpaca groupStepsForget ↓Retain ↑Forget with harness ↓Retain with harness ↑HRR ↓
A. Additional training19344.061.060.068.035.7
B. Additional data without the harness17635.056.049.064.023.1
C. HARU (harness-specific augmentation)18611.059.014.069.04.5

Our results show that:

  1. A vs. C (additional training): Additional training with a matched budget still has an HRR of 35.7, versus 4.5 for HARU. Additional training alone therefore does not explain HARU's gains.
  2. B vs. C (additional data): Additional data without the harness lowers HRR only to 23.1, and its forget accuracy with the harness stays at 49.0, versus 14.0 for HARU. Additional data alone is therefore not enough; the gain comes from harness-specific augmentation.

BFCL primarily involves additional training to make the model outputing empty tool calls, which does not necessarily require the model to be "harness-aware" (i.e., recognizing the external evaluation system). To validate this, NPO (Negative Preference Optimization) should be used to compare with baselines with an equivalent computational budget, as well as standard supervision for empty tool calls. Without matched-budget NPO and empty-call supervision baselines, the results cannot distinguish the gain of harness-aware learning from ordinary suppression through a safe target.

Response: We thank the reviewer for this point. We want to clarify that, on BFCL, HARU still requires the model to be "harness-aware." Since we know the harness provides schema repair, which can reconstruct executable calls from residual arguments, we leverage this knowledge to design the safe target {"tool_calls":[]}, which leaves no arguments for the harness to repair. NPO does not know about this mechanism; it only suppresses the original call, which can leave arguments that schema repair still recovers. The evaluation with the harness shows this difference: under schema repair, NPO's forget accuracy rises from 42.0% to 88.4%, whereas HARU's changes only from 40.2% to 41.0%. This comparison shows that HARU's gain comes from accounting for the harness's behavior, not from ordinary suppression.

HRR evaluates recovery of complete benchmark correctness instead of any invocation of a designated tool. Responses that invokes a forbidden tool but has an incorrect argument or another erroneous step may therefore count as unsuccessful forgetting-set behavior.

Response: We thank the reviewer for this point. We want to clarify that such responses are counted as not forgotten once the harness is applied. Without the harness, a target-tool call with an incorrect argument or erroneous step is counted as a failure. The harness is designed to correct exactly these errors (reasoning assistance fixes the erroneous step, and schema repair fixes the arguments), so the call becomes correct with the harness and is counted as recovered by HRR.

Furthermore, ToolAlpaca includes examples that the original actor already fails, so recovery does not always indicate reversal of unlearning.

Response: We thank the reviewer for this point. Including examples that the original model already fails is allowed in unlearning evaluation. Many standard unlearning benchmarks include forget examples that the original model does not answer correctly. For example, on WMDP [1], the original models are far from 100% accurate on the forget set, and TOFU [2] and tool unlearning [3] likewise do not require a perfect original model. This is because unlearning evaluation does not focus on what the original model knows, but on whether the unlearned model has completely forgotten the target. Our setting follows the same principle: if the unlearned model with the harness still executes the target tool correctly, the target capability is exposed in deployment, regardless of the original model's outcome.

Low HRR also does not imply low absolute leakage. For example, HARU still achieves 41.0% forget-set accuracy after repair on BFCL.

Response: We thank the reviewer for this point. We agree that a low HRR does not imply low absolute leakage. Therefore, we use HRR to measure robustness to the harness, and forget and retain accuracy to measure absolute performance. On BFCL, HARU and NPO have the same level of absolute leakage without the harness (40.2% vs. 42.0% forget accuracy). At this same level, HARU is much more robust: under schema repair, NPO's forget accuracy rises to 88.4%, whereas HARU's stays at 41.0%. Forget accuracy without the harness cannot distinguish the two methods, whereas HRR directly reveals the difference in robustness (1.5% for HARU vs. 80.0% for NPO). This is why we report HRR alongside forget and retain accuracy.

References [1] Li et al. The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning. ICML 2024.
[2] Maini et al. TOFU: A Task of Fictitious Unlearning for LLMs. COLM 2024.
[3] Cheng & Amiri. Tool Unlearning for Tool-Augmented LLMs. ICML 2025.

Reviewer_pmRROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces an unlearning method for LLM tool-calling agents, with the aim of removing the ability of a model to call a specific list of tools, without sacrificing general tool calling ability. The authors argue that harness information restores suppressed tool-use capabilities ("harness mediated recovery"), and describe Harness-Aware Robust Unlearning (HARU) as a solution to strengthen unlearning performance. On two different tool unlearning settings (ToolAlpaca, BFCL), the authors show that their method achieves better performance than four tested baselines (GradDiff, NPO, and two versions of ToolDelete). The authors check for stability across different seeds, and include both generalization experiments and ablation studies. Overall, the study shows the fragility of the generalization of current unlearning methods.

Critical strengths

Research about agent harnesses is relevant and important. The writing is overall clear and the proposed method is properly described. The evaluations also show the fragility of current tool call unlearning methods, and the HRR is a useful metric for evaluating whether the items that appear suppressed under the default interface are really unlearned.

Critical weaknesses
  1. The research problem and threat models are not well motivated

Unlearning is relevant when the goal is to remove information located in a model's parameters, rather than capabilities that can be controlled externally. In tool-calling scenarios, the tool call is handled to a separate software or API outside the model weights. There is not much security reason to unlearn a tool call directly in the model weights, as you can simply remove the access to the tool to the agent by deleting it from the API. In most deployments, models have a list of tools in their system prompt, or can search for which tools to use, and this is a much more practical way to enforce which tools a model can use. As such, simply deleting relevant tools from the API would technically be better than all proposed methods.

Moreover, Figure 1.a shows that the successful unlearning calls the wrong tool which would result in an incorrect response from the model (outputting global statistics instead of local statistics), which is a problematic unintended effect. The proposed method's target for unlearned tools ({"tool_calls":[]}) might also similarly lead to unwanted and erroneous responses from a model.

2: Potential Evaluation Overfitting

The proposed method first unlearns the tool calls (Line 256), then constructs a dataset in cases when the harness-processed execution leads to an unwanted action (Line 262). The model is trained specifically with these hard negatives, meaning it was explicitly trained on cases where unlearning did not work on a prior evaluation. It is less fair to directly compare it with other techniques which have not been trained with this additional evaluation information. Overall, the proposed method can seen more as an unlearning training data collection technique on top of prior methods, rather than a novel unlearning objective method.

3: Baselines are not properly trained

No hyperparameter search was made for baselines, which leads to situations such as GradDiff getting 1.8 forget score and 1.6 retain score, and ToolDelete-DPO having 98 forget score (Table 4). Additional hyperparameter sweeps should be included.

Questions for authors

Beyond the remarks listed in the above paragraphs, I would like the ask the authors:

  • Can you propose realistic scenarios where unlearning tools inside model parameters is necessary, and preferable to simply disabling access to the tools, or post-training the model to use the correct alternatives? Can you evaluate on these scenarios and describe the threat model?

  • (Clarification) If a tool has been unlearned, but a harness answers with a complete description of the unlearned tool and provides it to the model, should the unlearned model be able to call it? A model that has perfectly unlearned a tendency to use a tool might in principle still need to be able to read and follow a harness and use it as a "new tool". How to distinguish failure to unlearn with legitimate instruction following of a harness-provided tool, especially since a unlearned model would see it a new, never-before seen tool?

6 comments

Can you propose realistic scenarios where unlearning tools inside model parameters is necessary, and preferable to simply disabling access to the tools, or post-training the model to use the correct alternatives? Can you evaluate on these scenarios and describe the threat model?

Response: Weight-level unlearning is needed whenever the deployer does not control the tool registry, e.g., after an open-weight release. Tool unlearning is useful when disabling access or teaching an alternative does not reliably prevent unwanted tool use. Our introduction motivates this need through the privacy and safety risks associated with sensitive knowledge and undesirable capabilities (Lines 38–44). We elaborate on this motivation with two concrete scenarios based on the insecure-API and deprecated-tool examples in [1]:

  1. An unsafe healthcare API. After model weights are released, users control their own deployments and can re-enable a locally disabled API if it remains accessible. Disabling it in the developer’s registry therefore cannot prevent downstream use of an insecure API that could expose patient data; safeguards must therefore reside in the weights [2].
  2. An outdated software API. Teaching a model the replacement API may leave old calling patterns intact. Users deploying the updated weights may still receive obsolete calls, motivating explicit unlearning of the old API [1].

Threat model. The task specifies disjoint sets of tools to forget and retain. Users can configure the harness but cannot change the unlearned model’s weights. The harness helps the agent complete tasks through reasoning assistance or schema repair, improving retained-tool performance (Tables 3 and 4). The threat is that this helpful assistance can also recover forgotten tool calls.

If a tool has been unlearned, but a harness answers with a complete description of the unlearned tool and provides it to the model, should the unlearned model be able to call it? A model that has perfectly unlearned a tendency to use a tool might in principle still need to be able to read and follow a harness and use it as a "new tool". How to distinguish failure to unlearn with legitimate instruction following of a harness-provided tool, especially since a unlearned model would see it a new, never-before seen tool?

Response: We thank the reviewer for this interesting question, and we want to clarify that the unlearned agent should not call a forgotten tool, even when the harness provides its complete description. The goal of tool unlearning is to completely remove the capability of the forgotten tools from the model's weights [1]. At deployment, a robustly unlearned agent should still not call the forgotten tools even when a harness supplies instructions and full tool descriptions that present the forgotten tool as a "new tool". Prior work evaluates unlearning robustness in the same way, by supplying the forgotten knowledge back in context: Shumailov et al. [3] reintroduce forgotten knowledge in context, and Lynch et al. [4] use in-context relearning, where providing related context to the unlearned model resurfaces the forgotten knowledge. Both require the unlearned model not to recall the forgotten knowledge even when the knowledge is provided again.

The proposed method first unlearns the tool calls (Line 256), then constructs a dataset in cases when the harness-processed execution leads to an unwanted action (Line 262). The model is trained specifically with these hard negatives, meaning it was explicitly trained on cases where unlearning did not work on a prior evaluation. It is less fair to directly compare it with other techniques which have not been trained with this additional evaluation information.

Response: We thank the reviewer for raising this concern. HARU's advantage does not come from simply having additional evaluation information. To show this, we give a baseline the same kind of information: hard negatives mined from the unlearned model's own failures. The only difference from HARU is whether the harness is used to select them. Both methods start from the same initial NPO checkpoint:

  • Augmentation without harness-based selection: samples the unlearned model's responses without the harness and trains on responses that still call a forgotten tool.
  • HARU (harness-based selection): uses the same sampling budget and selection rule, but samples with the harness. It therefore trains on the calls that the harness recovers.
ToolAlpaca groupForget ↓Retain ↑Forget with harness ↓Retain with harness ↑HRR ↓
Augmentation without harness-based selection35.056.049.064.023.1
HARU (harness-based selection)11.059.014.069.04.5

With the same kind of additional information, the baseline still leaves HRR at 23.1, whereas HARU reduces it to 4.5 with higher retain accuracy. The gain therefore comes from harness-based selection, not from the additional information itself.

Overall, the proposed method can seen more as an unlearning training data collection technique on top of prior methods, rather than a novel unlearning objective method.

Response: HARU is not just a training-data collection technique; it provides a systematic framework for maintaining suppression of designated tool-use capabilities when an agent is equipped with a harness. Its contribution has two parts:

  1. Conceptually, HARU focuses on unlearning robustness when the agent runs with a harness. Prior tool unlearning checks forgetting only without a harness, whereas HARU requires the forgotten tools to stay suppressed even when a helpful harness is attached. Its training shares the idea of adversarial training for robust unlearning [5, 6]: the harness acts as a recovery probe that exposes the forgotten calls it can still recover, and HARU trains on exactly these calls to make forgetting robust.

  2. Methodologically, HARU formalizes tool unlearning with a harness and systematically addresses two recovery mechanisms. Preprocessing elicits suppressed calls through reasoning assistance, while postprocessing reconstructs calls from residual argument payloads. HARU addresses these mechanisms by applying NPO to harness-augmented contexts and adding a safe-destination loss that encourages no-tool outputs without repairable argument payloads. Together, these strategies approximate the harness-level unlearning objective (Eq. 2).

No hyperparameter search was made for baselines, which leads to situations such as GradDiff getting 1.8 forget score and 1.6 retain score, and ToolDelete-DPO having 98 forget score (Table 4). Additional hyperparameter sweeps should be included.

Response: We thank the reviewer for this suggestion. We agree that the original GradDiff configuration exhibited excessive degradation of retained-tool utility. To address this concern, we conducted a 12-configuration hyperparameter sweep on xLAM-1B/BFCL, varying the learning rate (10−610^{-6}, 3×10−63\times10^{-6}, 10−510^{-5}) and forget-loss coefficient (0.1, 0.25, 0.5, 1.0), while keeping the retain-loss coefficient fixed at 1.0. The results reveal a clear forget–retain trade-off.

Among the evaluated configurations, GradDiff with learning rate 10−510^{-5} and forget coefficient 0.5 provides a reasonable compromise, achieving 55.4% forget accuracy and 53.2% retain accuracy, compared with 2.7% and 1.8%, respectively, under the original configuration in this sweep. However, this tuned configuration remains vulnerable to schema-repair assistance. Forget accuracy increases from 55.4% to 69.6%, corresponding to an HRR of 32.0%. In comparison, HARU achieves 41.1% forget accuracy, 69.1% retain accuracy, and 0.0% HRR under schema repair.

Table: GradDiff-1B Hyperparameter Sweep on BFCL-v4

Learning RateForget Coef.Forget ↓Retain ↑Forget + Harness ↓Retain + Harness ↑HRR ↓Recovery
1e-60.10100.083.9100.084.3N/AN/A
1e-60.25100.083.3100.083.7N/AN/A
1e-60.50100.083.3100.083.6N/AN/A
1e-61.00100.083.4100.083.9N/AN/A
3e-60.1096.478.096.478.10.00/4
3e-60.2596.478.096.478.20.00/4
3e-60.50100.082.7100.083.0N/AN/A
3e-61.00100.083.5100.083.9N/AN/A
1e-50.1092.982.292.982.60.00/8
1e-50.2579.575.083.975.721.75/23
1e-50.5055.453.269.657.232.016/50
1e-51.002.71.82.72.20.00/109

References [1] Cheng & Amiri. Tool Unlearning for Tool-Augmented LLMs. ICML 2025.
[2] Tamirisa et al. Tamper-Resistant Safeguards for Open-Weight LLMs. ICLR 2025.
[3] Shumailov et al. UnUnlearning: Unlearning is not Sufficient for Content Regulation in Advanced Generative AI. arXiv:2407.00106, 2024.
[4] Lynch et al. Eight Methods to Evaluate Robust Unlearning in LLMs. arXiv:2402.16835, 2024.
[5] Zhang et al. Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models. NeurIPS 2024.
[6] Sheshadri et al. Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs. TMLR 2025.

Reviewer_CD4pOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

the reviewer cannot fully understand the paper because the central concepts and explanations are difficult to follow. as far as the reviewer understands, the paper studies whether reasoning assistance or output repair can recover tool-use behavior suppressed by unlearning. it proposes HARU to suppress such recovery through additional training and reports results on ToolAlpaca and BFCL. however, what is being unlearned and how the proposed training addresses this remain unclear to the reviewer.

Critical strengths
  • the model–harness system seems an interesting topic for LLM unlearning.
  • an honest report of the evaluation results, despite that the proposed method does not achieve the absolute best in every scenario.
Critical weaknesses
  • the paper is hard to read. one of its key concept, i.e., 'harness-mediated', has a very strange word choice (mediated, a word that the reviewer rarely sees in this field), and the sentence explaining it at L95 is very cumbersome. the reviewer cannot understand this explanation sentence after going over it several times. also, “Forget tool-calling accuracy on ...” in Figure 1 is grammatically incorrect; if the authors mean tool-calling accuracy on the forget set, they should say so explicitly. these wording problems affect a central concept and the main result figure. additionally, Figure 2 appears before Figure 1 in the PDF; please correct the figure ordering.

  • it is unclear what knowledge the agent should unlearn. Section 2.1 targets the ability to invoke selected tools, yet Section 5.1 keeps registered tool schemas available to the repair harness. how do the authors distinguish retained tool-use knowledge from reconstructing a call using the supplied specification? additionally, Section 4 trains toward an explicit no-tool response for the post-processing case. this demonstrates behavioral suppression, but does not by itself establish removal of the underlying knowledge. the paper should clarify which claim it intends to support.

  • the main-body experiments do not isolate the proposed training contribution. Section 4 combines initial unlearning with additional refinement, using recovered examples for reasoning assistance and a no-tool target for schema repair. Tables 2 and 4 compare complete methods, without separating these changes. ideally, include matched-budget continued unlearning, augmentation without recovery-based selection, and direct no-tool supervision as controls. this would establish whether the gains require the proposed harness-aware construction.

  • the BFCL counts are inconsistent: Table 1 specifies 112 forget examples, whereas Table 4 gives GradDiff 0/115 recoveries among default failures. please reconcile these counts. also, Section 3 measures complete-call correctness, so an incorrect argument can make target-tool use count as unsuccessful. reporting any target-tool invocation would help distinguish suppressed tool selection from degraded argument accuracy.

Questions for authors

can the authors clarify the forgetting target and provide controlled ablations of the refinement procedure?

4 comments

the paper is hard to read. one of its key concept, i.e., 'harness-mediated', has a very strange word choice (mediated, a word that the reviewer rarely sees in this field), and the sentence explaining it at L95 is very cumbersome.

Response: Thank you for identifying this ambiguity. We will clarify the definition: Harness-mediated recovery means that tool use suppressed by unlearning returns when the model runs with an agent harness. The harness may guide the unlearned agent to elicit a correct call through reasoning assistance or reconstruct one through output repair, without changing the model weights. Similar wording appears in the agent-security literature, including “tool-mediated attack surface” [1] and “tool-mediated knowledge extraction” [2].

the reviewer cannot understand this explanation sentence after going over it several times. also, “Forget tool-calling accuracy on ...” in Figure 1 is grammatically incorrect; if the authors mean tool-calling accuracy on the forget set, they should say so explicitly. these wording problems affect a central concept and the main result figure. additionally, Figure 2 appears before Figure 1 in the PDF; please correct the figure ordering.

Response: We will change the caption to "Tool-calling accuracy on the forget set", fix the same wording everywhere else in the paper, and place Figure 1 before Figure 2.

it is unclear what knowledge the agent should unlearn. Section 2.1 targets the ability to invoke selected tools, yet Section 5.1 keeps registered tool schemas available to the repair harness.

Response: The knowledge to be unlearned is the capability to invoke designated tools [3]. In the training stage, we use the objective in Section 2.1 to suppress that tool-use capability in the model weights. Section 5.1, however, concerns the deployment and evaluation setting: we test whether forgotten tools are called again when the unlearned model is deployed with a general-purpose harness. Such harnesses (e.g., schema repair) are supplied to improve the agent's general capability without knowing what was forgotten during training, and we show that this benign assistance can still restore the forgotten capability. This shares the similar concern in conventional unlearning that benign deployment or downstream processing, such as benign relearning, can bring forgotten knowledge back [4].

how do the authors distinguish retained tool-use knowledge from reconstructing a call using the supplied specification?

Response: Our ablation (Table 5 in the original paper) supports retained tool-use knowledge as the basis for recovery, not the supplied specification. Under identical registry, harness, and decoding settings, the tool-free reference model almost never succeeds despite receiving the specification, whereas the NPO-unlearned model improves from 33 to 53 successful calls. This shows that the harness cannot create tool-calling ability from the specification alone: recovery appears only when the model still retains the tool-use knowledge that unlearning failed to remove.

ModelWithout harnessWith harness
Vicuna-7B without tool finetuning [3]1/1000/100
NPO-unlearned ToolAlpaca-7B33/10053/100
HARU11/10014/100

Additionally, this distinction is not required for robust unlearning, which requires that the forgotten capability cannot be elicited, regardless of the cue's source. Prior work likewise tests forgetting with rephrased and adversarial prompts, which may themselves supply informative cues, and counts any successful elicitation as a failure to forget [5]. Thus, a call recovered with the supplied specification still shows that the unlearning was not robust.

additionally, Section 4 trains toward an explicit no-tool response for the post-processing case. this demonstrates behavioral suppression, but does not by itself establish removal of the underlying knowledge. the paper should clarify which claim it intends to support.

Response: We intend to support removal of the targeted tool-use knowledge, not only a change in surface behavior. The no-tool target in Section 4 is a training signal, not our evidence of removal. Instead, we test removal by asking whether the capability can be recovered: if the knowledge were still present, a harness could elicit it, as in Table 5, where the NPO-unlearned model recovers from 33 to 53 successful calls. After HARU training, the same harness recovers far less (11 to 14 out of 100), and HRR drops from 35.7 (continued NPO) to 4.5.

the main-body experiments do not isolate the proposed training contribution. Section 4 combines initial unlearning with additional refinement, using recovered examples for reasoning assistance and a no-tool target for schema repair. Tables 2 and 4 compare complete methods, without separating these changes. ideally, include matched-budget continued unlearning, augmentation without recovery-based selection, and direct no-tool supervision as controls. this would establish whether the gains require the proposed harness-aware construction.

Response: We thank the reviewer for this suggestion. We show that HARU's gains come from harness-aware construction, not from additional training budget or from augmentation without recovery-based selection. We conducted an ablation on ToolAlpaca to isolate the source of HARU's gains.

We compare three groups that start from the same initial NPO checkpoint and share the same objective, hyperparameters, and training budget. They differ only in the forget examples used for refinement:

  • A. Matched-budget continued unlearning: continues unlearning on the original forget data, testing whether additional training alone explains the gains.
  • B. Augmentation without recovery-based selection: samples responses without the harness and trains on those that still call a target tool, testing whether mining the model's own failures suffices.
  • C. HARU: uses the same sampling budget and selection rule as B, but samples with the reasoning harness.
ToolAlpaca groupStepsForget ↓Retain ↑Forget with harness ↓Retain with harness ↑HRR ↓
A. Matched-budget continued unlearning19344.061.060.068.035.7
B. Augmentation without recovery-based selection17635.056.049.064.023.1
C. HARU18611.059.014.069.04.5

Our results show that:

  1. A vs. C (matched-budget continued unlearning): Matched-budget continued unlearning still has an HRR of 35.7, versus 4.5 for HARU. Additional training alone therefore does not explain HARU's gains.
  2. B vs. C (augmentation without recovery-based selection): Augmentation without recovery-based selection lowers HRR only to 23.1, and its forget accuracy with the harness stays at 49.0, versus 14.0 for HARU. Augmentation alone is therefore not enough; selecting the examples that the harness recovers is what matters.

the BFCL counts are inconsistent: Table 1 specifies 112 forget examples, whereas Table 4 gives GradDiff 0/115 recoveries among default failures. please reconcile these counts.

Response: We thank the reviewer for checking the details; we have corrected it in the revised version.

also, Section 3 measures complete-call correctness, so an incorrect argument can make target-tool use count as unsuccessful. reporting any target-tool invocation would help distinguish suppressed tool selection from degraded argument accuracy.

Response: We thank the reviewer for this suggestion. We additionally report any executed target-tool call, and HARU keeps both target-tool invocation and correctness low, with and without the harness. HARU calls the target tool on only 14 of the 100 ToolAlpaca forget examples without the harness and 19 with it. In contrast, the baselines call it on 63–100 examples without the harness and on all 100 with it, so their correctness is reduced mainly by degraded arguments rather than suppressed invocation. HARU's low forget accuracy therefore comes from suppressing target-tool invocation, not only from degrading argument accuracy.

MethodCorrect w/o harness ↓Correct w/ harness ↓Any target-tool call w/o harness ↓Any target-tool call w/ harness ↓
Original7585100100
GradDiff6576100100
NPO335385100
ToolDelete-SFT467263100
ToolDelete-DPO45535879
HARU11141419

References [1] Zhang et al. AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purification. arXiv:2602.22724, 2026.
[2] Zang et al. Extracting Knowledge from Tools in LLM Agents. arXiv:2608.30288, 2026.
[3] Cheng & Amiri. Tool Unlearning for Tool-Augmented LLMs. ICML 2025.
[4] Hu et al. Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning. ICLR 2025.
[5] Ye et al. LLM Unlearning Should Be Form-Independent. IEEE S&P 2026.

Thank you for the detailed response. The clarifications and additional experiments alleviate some of my concerns, but are not sufficient to change my overall assessment. I therefore maintain my original rating.

Thank you for your response and for taking the time to consider our clarifications. While we understand that your recommendation remains unchanged, since there is still time for discussion, could you let us know if there are any remaining concerns we could further clarify or address? We are happy to discuss and address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.