- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AERIS, a framework for constructing entity-level unlearning supervision from a minimal entity identifier without access to original training passages. It repeatedly queries the target model to build a weighted association graph, then uses graph-guided sampling to generate target-focused forget examples and neighboring knowledge retention examples. These datasets are used with existing unlearning objectives. Experiments on RWKU and TOFU evaluate forgetting effectiveness and knowledge preservation, with reported improvements over direct QA generation and evidence that graph-local retention data help reduce collateral damage.
Critical strengths
- This paper is well-written with clear structure.
- The topic is timing and may provide insights for the community.
Critical weaknesses
The incremental technical contribution over prior work remains unclear. Prior work [1] has already proposed a two-stage framework that generates and self-verifies entity-related QA pairs using the target model, then applies existing unlearning algorithms to update it. Reveal and Release [2] also studies unlearning through model-generated forget data and iterative updates. Relative to these works, the main addition is the use of an association graph to jointly construct forget and retain data. While this design has practical value, the paper does not sufficiently demonstrate what capabilities the graph enables that previous methods cannot readily achieve. Its distinct contribution therefore requires more convincing validation.
The experiments do not sufficiently establish the necessity of the graph structure itself. The current results primarily show that the complete pipeline improves forgetting, but it remains unclear whether the gains come from graph guidance or more extensive data generation. A direct comparison would preserve the same mined content and training budget while replacing graph-guided sampling with a simple sampling baseline. Without such a controlled experiment, the paper’s central technical design has not been independently validated.
An alternative explanation for the forgetting improvements remains unresolved. The method primarily trains on QA pairs whose answers identify the target entity, while the paper also finds that varying the factual correctness of these pairs has relatively little effect on the results. This raises the concern that the improvements may partly arise from suppressing the probability of generating the target’s name. The current evaluation does not sufficiently distinguish name suppression from the unlearning of associated knowledge. The authors should further examine whether the model can still recover facts about the target when the expected answers do not contain its name.
Stronger forgetting does not yet establish a better overall trade-off. Under GA+GD in Table 2, AERIS reduces Forget-All from 0.6092 to 0.4861 compared with DirectQA, but Neighbor-All also drops from 0.6709 to 0.5835. Thus, improved forgetting is accompanied by a substantial loss of neighboring knowledge. The local retention ablation provides encouraging evidence, but it is insufficient to establish a better overall trade-off. Comparing retention at matched levels of forgetting would help determine whether the gains extend beyond the effects of more aggressive updates.
[1] Weitao Ma et al. Unveiling Entity-Level Unlearning for Large Language Models: A Comprehensive Analysis.
[2] Linxi Xie et al. Reveal and Release: Iterative LLM Unlearning with Self-generated Data.
Questions for authors
-
What benefit does the graph provide over existing self-generated-data approaches when the mined content and training budget are held fixed?
-
How can the authors distinguish genuine knowledge unlearning from target-name suppression, particularly when the expected answers do not contain the target’s name?
-
Does AERIS preserve neighboring knowledge better than DirectQA when both methods achieve comparable levels of forgetting?