acceptodds
Under review as a conference paper at ICLR 2027

ARC-Unlearn: Blast-Radius-Aware LLM Unlearning via Agentic Retain-Set Synthesis and Adjacent-Retain NPO

Abstract

Machine unlearning for large language models aims to remove unwanted target knowledge while preserving the related knowledge around it. However, most existing approaches primarily evaluate how effectively target information is forgotten, potentially overlooking damage to semantically related but legitimate knowledge. We refer to this localized collateral damage as the blast radius. We introduce BlastBench, an adjacency-aware benchmark for quantifying collateral damage during machine unlearning through Adjacent Retain Rate (ARR). We further present ARC-Unlearn, a multi-agent framework that identifies target-specific adjacent concepts, constructs an adjacent-retain set, and protects this knowledge during unlearning. Its optimization component, Adjacent-Retain NPO (AR-NPO), combines NPO-based forgetting, adjacent-retain cross-entropy, and general-knowledge KL regularization. Across three 7B-scale models and 45 targets, standard forgetting baselines achieve high forget efficacy but substantially degrade adjacent knowledge. Among the evaluated methods, AR-NPO is the only one that achieves non-trivial forget efficacy () while improving adjacent retention relative to the original model (). An -sweep reveals a controllable FE–ARR trade-off between target forgetting and adjacent retention. On a matched 9-target subset at 32B scale, AR-NPO reaches and , with all nine targets satisfying our surgical-zone criteria. Full-parameter fine-tuning further achieves near-complete measured forgetting () together with above-baseline adjacent retention (). Finally, held-out and cross-generator audits show that the adjacent-retention gains persist beyond the original generated QA.

Primary area:
generative models
Keywords:
Machine Unlearning, LLM unlearning, Blast Radius, Adjacent Knowledge Preservation, Knowledge Collateral Damage, Benchmark, Adjacent Retain Rate, Multi-Agent Framework, Negative Preference Optimization, Adversarial Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oa9TOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies that LLM unlearning methods could cause damage to the semantically adjacent knowledge to the forget target. To define this, author use the blast radius. The paper has three contributions:

(1) BlastBench, a benchmark with 45 forget targets across three domains with about 13.8K QA pairs generated by an LLM. The authors also propose new metrics. A two-agent pipeline to construct the adjacent set where a prober agent that finds adjacent concepts based on an LLM-rated a score, and a synthesizer agent that generates retain QA pairs for these concepts without mentioning the forget target.

(2) An algorithm is exploited, AR-NPO, combining the NPO forget loss with a cross-entropy loss on the adjacent set and a KL loss on WikiText. The authors evaluate AR-NPO against GA, GD, NPO, and SimNPO on three 7B models, and claim that AR-NPO is the only method that achieves non-trivial forgetting while keeping adjacent knowledge above the original model.

Critical strengths

This paper considers an interesting problem in LLM unlearning. Collateral damage to neighboring knowledge is an important failure mode where the unlearning safety intervention might miss.

Critical weaknesses

This paper suffers from two following major concerns: (1) The proposed metrics, FE, ARR, DRR, FP, RR, are not well motivated and defined. Where is the motivation? Why the previous metrics like regular accuracy, verbmem, and knowmem on the forget and retain sets are not working here? In the definition, acc(·) is denoted as the mean per-token log-probability of the gold answer. Log-probabilities are negative. Also, the authors define ARR as acc(A)/acc_0(A). When the unlearned model gets worse on A, acc(A) becomes more negative and ARR rises above 1. Similar issue for FE = 1 − acc/acc_0 metric, where for strong forgetting mode, this metric becomes negative. On the other hand, the SimNPO method achieves 0.000 in every run, do we have any clipping operation here?

(2) I find the proposed method, AR-NPO, the same as NPO but with a new set added. In the original NPO paper (Zhang et al., 2024), various similar approaches are already included like retain-set cross-entropy (NPO+RT) and KL (NPO+KL). AR-NPO is the combination of both, with the adjacent set A as the retain set. So, there is no novelty in terms of the algorithm. The paper just considers/constructs a new training set. A fair comparison requires well-defined metrics and accounting for the trade-off in training time.

Questions for authors

Please check the Critical Weaknesses section.

Reviewer_eWvbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies collateral damage to semantically adjacent knowledge during LLM unlearning. It introduces BlastBench, an adjacency-aware benchmark, and ARC-Unlearn, which uses LLM-based probing to identify neighboring concepts and generate targeted retain examples. The proposed AR-NPO objective combines NPO with adjacent-retain cross-entropy and general KL regularization. Experiments across several models report improved adjacent knowledge retention while maintaining varying degrees of forgetting efficacy.

Critical strengths

The paper addresses an important practical concern in LLM unlearning: preserving legitimate knowledge related to the information being removed.

The experimental evaluation covers multiple models and domains, with additional ablations and scaling experiments that provide useful empirical observations.

Critical weaknesses
  1. The central problem of preserving neighboring knowledge while modifying targeted knowledge has been extensively studied in knowledge editing, including Unveiling the Pitfalls of Knowledge Editing, Neighboring Perturbations of Knowledge Editing, and RippleEdits. More directly, Mechanistic Unlearning already investigates reducing unintended side effects in both editing and unlearning. The paper does not sufficiently distinguish its conceptual contribution from this literature. Moreover, AR-NPO primarily combines existing NPO with an adjacent-retain objective, making the methodological novelty somewhat limited.

  2. Insufficient evidence supporting the claimed advantages. The main comparisons are not conducted at matched forgetting efficacy (e.g., AR-NPO achieves FE=0.566 versus NPO's 0.992), making the retention improvements difficult to interpret. Additionally, the 32B results show that GD achieves strong adjacent retention without explicit adjacent-retain supervision. Finally, Eq. (1) appears inconsistent with the stated definition of accuracy as mean log-probability, potentially affecting the interpretation of the core metrics.

Questions for authors

What is the fundamental difference between the proposed blast-radius formulation and the locality/neighboring-knowledge preservation problems extensively studied in knowledge editing? What is the main methodological novelty beyond applying targeted retain-set regularization to NPO?

Can the authors provide comparisons at matched forgetting efficacy and clarify the implementation of FE/ARR in Eq. (1), particularly given the use of potentially negative log-probabilities?

Reviewer_jH5LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses collateral damage to knowledge adjacent to an unlearning target. It introduces BlastBench, covering 45 targets across three domains, and ARC-Unlearn, a pipeline that identifies adjacent concepts and synthesizes retain examples. Its optimization component, AR-NPO, combines a forgetting objective, adjacent-retain cross-entropy, and general-retain KL regularization. Experimental results demonstrate that explicitly protecting knowledge adjacent to the unlearning target can improve its retention while maintaining a reasonable level of forgetting effectiveness. However, the definitions of the key evaluation metrics require further clarification, and the experimental comparisons do not fully establish the retention advantages of AR-NPO under comparable forgetting strengths.

Critical strengths
  1. Focuses on a meaningful problem in machine unlearning. Evaluating adjacent knowledge separately from general utility can reveal damage overlooked by conventional evaluations. The distinction between forget, adjacent-retain, distal-retain, and general-retain sets provides a useful framework for assessing the forgetting–retention trade-off.
  2. Provides comprehensive experimental evaluations. This paper covers multiple domains and model families, including human validation, held-out and cross-generator evaluations, and experiments beyond the main LoRA setting. These experiments help assess the robustness of the proposed method across different data-generation and model adaptation settings.
Critical weaknesses
  1. The definition of the central evaluation metrics needs clarification before the results can be interpreted reliably. Section 3 defines acc as mean per-token log-probability, but Equation (1) uses ratios of this quantity with an interpretation appropriate to a positive performance score. For example, a change from −1 to −3 would give ARR = 3 despite reduced answer likelihood. This could be a notation or description error rather than an implementation error. However, because FE and ARR underpin the main conclusions, their actual computation needs to be made explicit.
  2. The main comparison does not establish the retention advantage at matched forgetting strength. In Table 2, AR-NPO has reported FE = 0.566, while GA, GD, and NPO have FE close to 0.99. Thus, part of the retention difference could reflect different operating points. The AR-NPO alpha sweep is useful, but the paper would be more convincing with baseline comparisons at similar forgetting levels under a common evaluation scope.
Questions for authors
  1. What exact quantity is used as acc in the implementation: mean log-probability, exponentiated mean log-probability, mean probability, or another score? Please provide the precise aggregation rule and confirm whether correcting the definition changes any reported results.
  2. Once the metric definition is clarified, how do the strongest baseline operating points compare with AR-NPO at similar FE, for example near the reported value of 0.56? Could existing checkpoints or a limited comparison under matched evaluation conditions address this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.