- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the robustness of tool unlearning in LLM agents equipped with deployment harnesses. The authors identify a failure mode termed harness-mediated recovery, where reasoning assistance or schema repair can restore tool-use capabilities that appear to have been suppressed by unlearning. To evaluate this phenomenon, the paper introduces the Harness Recovery Rate (HRR) and proposes Harness-Aware Robust Unlearning (HARU), which incorporates harness-recoverable behaviors into the unlearning process. Experiments on ToolAlpaca and BFCL demonstrate that harness assistance can substantially increase the success rate of forgotten tool calls, while HARU reduces this recovery under the evaluated settings.
Critical strengths
- This paper studies an aspect of LLM unlearning that is often overlooked in existing evaluations.
- This paper distinguishes between reasoning assistance before generation and schema repair after generation.
- Formulating unlearning robustness at the model–harness level is a useful perspective. The proposed HRR metric provides an intuitive way to measure how frequently harness assistance recovers previously unsuccessful target-tool calls.
Critical weaknesses
- For the preprocessing harness, the main idea is to collect harness-recoverable examples and perform additional unlearning, which resembles hard-negative mining and data augmentation. For the postprocessing harness, the method introduces an explicit no-tool training target. The paper would benefit from a clearer explanation of what is technically novel beyond these existing training strategies.
- The experimental comparison does not fully control for training data and optimization budgets. In Appendix A.1, HARU uses 3,794 augmented trajectories and 295 optimization updates, while the baselines use 635 original forget trajectories and 50 updates. The authors also acknowledge that this comparison cannot isolate augmentation from additional optimization.
- This paper lacks a systematic analysis of the individual components of HARU. In particular, it is unclear how much improvement comes from harness-guided data augmentation, additional unlearning steps, and the safe-destination loss. Comparisons with standard NPO using matched training budgets, random augmentation, or ordinary hard-negative mining would help establish the effectiveness of the proposed method.
- The experiments focus on two relatively specific configurations: a ToolAlpaca actor assisted by an external reasoning model and an xLAM actor with deterministic schema repair. Although additional reasoning providers are evaluated, they are all fine-tuned on ToolAlpaca. The paper does not examine more realistic agent workflows involving multiple interacting tools, iterative feedback, or different types of harness assistance. It is therefore unclear how broadly the conclusions generalize to deployed LLM agents.
Questions for authors
- Can the authors provide training-budget-matched comparisons to better isolate the contribution of HARU?
- How does HARU generalize to harnesses and tool-use scenarios that differ more substantially from those used during training?
2 comments
How does HARU generalize to harnesses and tool-use scenarios that differ more substantially from those used during training?
Response: We thank the reviewer for raising the generalization concern. We already discussed this in Section 5.3 of the original submission: HARU generalizes to unseen harnesses, consistently yielding strong unlearning performance. We compare HARU (unlearned with ToolAlpaca-13B as the harness) and the strongest unlearned baseline, NPO, using four unseen harnesses from different model families to provide reasoning. HARU keeps HRR below 8% on every unseen harness, whereas NPO (strongest unlearning baseline) stays above 24%. Results are in the following table:
| Reasoning harness | NPO HRR ↓ | HARU HRR ↓ |
|---|---|---|
| ToolAlpaca-13B (used in unlearning) | 32.8 | 4.5 |
| Qwen2.5-7B (unseen) | 28.6 | 5.6 |
| Mistral-7B (unseen) | 27.1 | 5.6 |
| Llama-3.1-8B (unseen) | 34.3 | 7.8 |
| Gemma-4-E2B (unseen) | 24.3 | 4.4 |
References
[1] Zhang et al. Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models. NeurIPS 2024.
[2] Sheshadri et al. Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs. TMLR 2025.
Response: HARU is not just a combination of hard-negative mining and no-tool supervision; it provides a systematic framework for maintaining suppression of designated tool-use capabilities when an agent is equipped with a harness. Its contribution has two parts:
Conceptually, HARU focuses on unlearning robustness when the agent runs with a harness. Prior tool unlearning checks forgetting only without a harness, whereas HARU requires the forgotten tools to stay suppressed even when a helpful harness is attached. Its training shares the idea of adversarial training for robust unlearning [1, 2]: the harness acts as a recovery probe that exposes the forgotten calls it can still recover, and HARU trains on exactly these calls to make forgetting robust.
Methodologically, HARU formalizes tool unlearning with a harness and systematically addresses two recovery mechanisms. Preprocessing elicits suppressed calls through reasoning assistance, while postprocessing reconstructs calls from residual argument payloads. HARU addresses these mechanisms by applying NPO to harness-augmented contexts and adding a safe-destination loss that encourages no-tool outputs without repairable argument payloads. Together, these strategies approximate the harness-level unlearning objective (Eq. 2).
Response: We thank the reviewer for this suggestion. We show that HARU's gains come from harness-guided augmentation, not from additional unlearning steps or ordinary hard-negative mining. We conducted a training-budget-matched ablation on ToolAlpaca to isolate the source of HARU's gains.
We compare three groups that start from the same initial NPO checkpoint and share the same objective, hyperparameters, and training budget. They differ only in the forget examples used for refinement:
The matched budget covers training steps and excludes the cost of generating augmentation data. Our results show that: