acceptodds
Under review as a conference paper at ICLR 2027

Scope Before Update: Graph-Guided Paired Supervision for Entity Unlearning

Abstract

Large language models can reproduce sensitive or copyrighted information, motivating machine unlearning to remove selected knowledge while preserving other capabilities. Existing pipelines commonly require users to supply a forget set, which can re-expose sensitive content and complicate request auditing. We study a user-minimized interface in which the requester identifies a target entity and the service constructs the supervision needed for the update. This shifts the challenge to recovering useful target associations and separating behavior to change from nearby knowledge to preserve. We introduce AERIS (Association-guided Entity Removal with Integrated Supervision), which organizes repeatedly elicited target-model associations into a weighted local graph and converts its paths into paired forget and neighbor data. The constructed supervision requires no original training passages and can be reused with standard unlearning objectives. Experiments on RWKU and TOFU show stronger forgetting than direct self-generation across the evaluated update objectives. With forget data fixed, graph-local neighbor supervision improves nearby knowledge preservation over Wikipedia retain data under both regularized objectives. On eight shared RWKU targets, AERIS reduces Forget-All by 20.2% relative to DirectQA under GA+GD.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, LLM Unlearning, Privacy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_o613OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AERIS, a framework for constructing entity-level unlearning supervision from a minimal entity identifier without access to original training passages. It repeatedly queries the target model to build a weighted association graph, then uses graph-guided sampling to generate target-focused forget examples and neighboring knowledge retention examples. These datasets are used with existing unlearning objectives. Experiments on RWKU and TOFU evaluate forgetting effectiveness and knowledge preservation, with reported improvements over direct QA generation and evidence that graph-local retention data help reduce collateral damage.

Critical strengths
  1. This paper is well-written with clear structure.
  2. The topic is timing and may provide insights for the community.
Critical weaknesses

The incremental technical contribution over prior work remains unclear. Prior work [1] has already proposed a two-stage framework that generates and self-verifies entity-related QA pairs using the target model, then applies existing unlearning algorithms to update it. Reveal and Release [2] also studies unlearning through model-generated forget data and iterative updates. Relative to these works, the main addition is the use of an association graph to jointly construct forget and retain data. While this design has practical value, the paper does not sufficiently demonstrate what capabilities the graph enables that previous methods cannot readily achieve. Its distinct contribution therefore requires more convincing validation.

The experiments do not sufficiently establish the necessity of the graph structure itself. The current results primarily show that the complete pipeline improves forgetting, but it remains unclear whether the gains come from graph guidance or more extensive data generation. A direct comparison would preserve the same mined content and training budget while replacing graph-guided sampling with a simple sampling baseline. Without such a controlled experiment, the paper’s central technical design has not been independently validated.

An alternative explanation for the forgetting improvements remains unresolved. The method primarily trains on QA pairs whose answers identify the target entity, while the paper also finds that varying the factual correctness of these pairs has relatively little effect on the results. This raises the concern that the improvements may partly arise from suppressing the probability of generating the target’s name. The current evaluation does not sufficiently distinguish name suppression from the unlearning of associated knowledge. The authors should further examine whether the model can still recover facts about the target when the expected answers do not contain its name.

Stronger forgetting does not yet establish a better overall trade-off. Under GA+GD in Table 2, AERIS reduces Forget-All from 0.6092 to 0.4861 compared with DirectQA, but Neighbor-All also drops from 0.6709 to 0.5835. Thus, improved forgetting is accompanied by a substantial loss of neighboring knowledge. The local retention ablation provides encouraging evidence, but it is insufficient to establish a better overall trade-off. Comparing retention at matched levels of forgetting would help determine whether the gains extend beyond the effects of more aggressive updates.

[1] Weitao Ma et al. Unveiling Entity-Level Unlearning for Large Language Models: A Comprehensive Analysis.

[2] Linxi Xie et al. Reveal and Release: Iterative LLM Unlearning with Self-generated Data.

Questions for authors
  1. What benefit does the graph provide over existing self-generated-data approaches when the mined content and training budget are held fixed?

  2. How can the authors distinguish genuine knowledge unlearning from target-name suppression, particularly when the expected answers do not contain the target’s name?

  3. Does AERIS preserve neighboring knowledge better than DirectQA when both methods achieve comparable levels of forgetting?

Reviewer_Ze6XOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this paper the authros present AERIS, a graph guided framework that automatically constructs forget and neighbor supervision from a target model’s elicited associations. Experiments on RWKU and TOFU show better forgetting than direct QA generation while graph-local neighbor supervision improves knowledge preservation over generic retain data.

Critical strengths

The authors addresses a practical limitation of entity-level unlearning by eliminating the need for user-provided forget corpora. Constructing both forget and neighbor supervision from a shared association graph is a reasonable approach to handling interconnected entity knowledge. The framework is also compatible with multiple unlearning objectives, and the controlled retain-source ablation.

Critical weaknesses
  1. The necessity of graph-based construction is not well established. AERIS combines multiple components but their individual contributions are not separated. Comparisons with simpler graph free multi-hop QA generation or randomized graph sampling would help prove the benefit of proposed graph structure.
  2. The forgetting–preservation trade-off is still concerning. In Table 2 AERIS has stronger forgetting than DirectQA but severely degrades neighboring knowledge; similar degradation of utility appears on TOFU where it is unclear whether the method enables selective forgetting or more aggressive suppression of knowledge.
Questions for authors
  • Can the authors compare AERIS to a graph-free multi-hop QA generation baseline (same data sizes and generation budgets)? That would show whether improvements come from graph structure or broader association coverage.
  • How does AERIS compare with DirectQA at matched forgetting strengths? In particular, can the authors demonstrate improved neighbor preservation under comparable Forget-All scores?
Reviewer_UicXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AERIS, a graph-based approach for constructing entity-level unlearning data. AERIS builds an entity association graph by repeatedly querying the target model and automatically generates Forget and Neighbor Sets through path sampling, without requiring access to the original training data. Experiments on RWKU and TOFU show that AERIS achieves stronger forgetting than direct QA generation, while graph-local neighbor supervision helps preserve neighboring knowledge.

Critical strengths

The paper is well organized. The method and its steps are easy to understand. It also provides many experiments, ablation studies, and additional analyses. These provide useful evidence for evaluating the effectiveness of the method.

Critical weaknesses
  1. Different LLMs produce clearly different Memory Graphs. This means AERIS may need to rebuild the Forget/Neighbor Set for each model. Compared with external datasets that can be reused across models, this may bring extra construction costs. The paper lacks an analysis of how well the generated data can transfer across models and how such transfer affects Forgetting and Utility Performance.

  2. Although AERIS achieves better Forgetting Performance, it seems to come at a large cost to Utility. In several experiments, AERIS shows stronger Forgetting, but also worse Neighbor Preservation or General Utility. In addition, the Correctness Mixture experiment shows that even when the Forget Set contains many incorrect facts, or even when all QA pairs are incorrect, the Forgetting Performance remains similar. This raises further concerns about whether such forgetting may cause unnecessary damage to the model's normal abilities.

Questions for authors
  1. In the Correctness Mixture experiment, does the ratio of incorrect QA pairs affect the model's General Utility? Could the authors also report the corresponding Utility Performance?

  2. Could the authors try using the Forget/Neighbor Set generated by model A directly on model B with a different architecture, and compare the Forgetting and Utility Performance after transfer?

  3. Could the authors report the end-to-end time needed to build the complete Forget/Neighbor QA dataset for different models, so that the extra construction cost can be evaluated?

  4. The data construction process of AERIS seems to mainly depend on manually set values, such as the number of queries, Graph Depth, and Sampling Budget. How do the authors decide when to stop the construction process? How do they know whether the generated Forget Set is complete enough?

  5. Since the method relies on predefined Prompts to extract knowledge from the target model, some long-tail knowledge that is difficult to generate may be missed. Have the authors considered this problem? How can they ensure that the Forget Set covers enough knowledge about the target?

Reviewer_kc2qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses service-side supervision construction for entity-level LLM unlearning, where the user only specifies a target entity name and the provider generates both the forget and retain sets without external corpora or original training data. The proposed AERIS iteratively queries the target model to build a local association graph around the target entity and samples graph paths to synthesize target-forgetting and neighbor-retaining QA pairs.

Critical strengths
  • Constructing both forget and retain supervision on the service side from a minimal entity anchor is well-motivated and avoids requiring users to collect sensitive forget corpora or exposing providers to unverified update data.
  • Using a single elicited local association graph to jointly derive target-forgetting QAs and graph-local neighbor retention QAs is modular and compatible with standard downstream unlearning objectives.
Critical weaknesses
  • Iteratively probing a target LLM across multiple hops around an anchor, filtering by model consistency/confidence, and separating target paths from local neighbor paths to preserve locality closely parallels Wei et al., 2026, which is not discussed in the manuscript.
  • AERIS forces the target entity to be the answer in every forget QA. The study of Figure 3 shows that forgetting is insensitive to the factual correctness of the generated QAs indicates that optimization mainly suppresses emitting the target's surface name rather than erasing underlying relational knowledge.
  • While gains are notable on the small 8-entity subset, improvements over standard supervision across the full 200-entity RWKU benchmark are marginal, and DirectQA is omitted at full scale. Meanwhile, AERIS suffers severe utility degradation on near-domain TOFU splits (Real Authors and World Facts) and RWKU TriviaQA.

Wei, Rongzhe, et al. "Do llms really forget? evaluating unlearning with knowledge correlation and confidence awareness." NeurIPS 2025.

Questions for authors
  1. How does AERIS's multi-hop association recovery conceptually and empirically differ from the confidence-aware supporting subgraph extraction and local consistency formulation in Wei et al., 2026?
  2. Given that every forget QA places the target entity in the answer span and GA performance is invariant to QA factual correctness, does the model merely learn to suppress the target entity's name tokens while still leaking facts when the entity is queried in the prompt or via indirect multi-hop paths?
  3. How is entity canonicalization implemented in practice, and how does AERIS prevent unmerged target aliases from entering the neighbor retain set and reinforcing target knowledge, such as "Taylor Alison Swift" in Figure 5?
  4. Why does AERIS cause near-total collapse on TOFU's Real Authors and World Facts splits (Tables 12–14) and sharp drops on RWKU TriviaQA (Table 2) even with retention regularization, and how does AERIS compare against DirectQA on the full 200-entity RWKU split as well as in end-to-end generation cost?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.