- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This research proposes an unlearning method that addresses a limitation of previous approaches: directly unlearning a target fact may still leave it recoverable through multi-hop reasoning over related knowledge. The proposed approach examines both explicit model responses and latent representations to identify recovery paths and selectively unlearn supporting facts that interrupt these paths. Experimental results demonstrate more effective forgetting than the baselines while limiting the loss of unrelated knowledge.
Critical strengths
- The research is clearly motivated, addressing superficial unlearning by expanding the search to include multi-hop reasoning paths that can reconstruct forgotten knowledge.
- The confidence-aware minimum-cut approach appears sound and practical, although its effectiveness remains limited by the coverage of the discovered supporting knowledge.
- The experimental results support the effectiveness of the proposed method, showing substantial improvements over the baselines.
Critical weaknesses
- One of my major concerns is the limited scope of the experimental setting. The evaluation relies on a single fictional domain with relatively bounded knowledge, which may make the relevant information easier to identify and control than in real-world settings. This could partly explain the substantial performance gains. Experiments on more diverse, real-world knowledge would help establish the method’s generalizability and practical effectiveness.
- The paper also omits a closely related work: Yang et al., “FAITHUN: Toward Faithful Forgetting in Language Models by Investigating the Interconnectedness of Knowledge” (EMNLP 2025). The authors should discuss its relationship to their work, particularly its treatment of superficial unlearning and interconnected knowledge. FAITHUN also provides a relevant real-world evaluation setting that could strengthen the experimental validation.
Questions for authors
.
4 comments
We are keen to discuss further with you
Dear Reviewer hpEu,
We sincerely thank you again for your time and constructive comments.
In our response, we have: (1) added experiments on YAGO3-10, a real-world knowledge graph, to evaluate generalization beyond Harry Potter; and (2) clarified our relationship to FAITHUN: while FAITHUN examines whether forgetting extends to related questions that depend on the target fact, our work discovers and interrupts alternative supporting paths that can reconstruct the target fact from other retained knowledge.
As the discussion period is approaching its end, we would greatly appreciate it if you could kindly let us know whether our response addresses your concerns or whether there are any further questions. We would be happy to address them.
Best wishes,
Authors
We are keen to discuss further with you
Dear Reviewer hpEu,
We sincerely thank you again for your time and constructive comments.
In our response, we have added experiments to evaluate generalization beyond Harry Potter and clarified the distinction between our work and FAITHUN.
As the discussion period is approaching its end, we would greatly appreciate it if you could kindly let us know whether our response addresses your concerns or whether you have any further questions. We would be happy to address them.
Best wishes,
Authors
Thank you for conducting the additional experiments. While I still think the evaluation is not comprehensive, these results would be a valuable addition to the main paper. Please also clarify the experimental setup. In particular, what do GA and NPO refer to here? Are they combined with your proposed method, as in Table 1?
We thank Reviewer hpEu for the constructive feedback and for recognizing the clear motivation of our work, the soundness and practicality of the confidence-aware minimum cut, and the effectiveness shown in our experiments.
[W1. Limited experimental scope] Thank you for this constructive suggestion. Our choice of Harry Potter is motivated by the need for rigorous evaluation. Access to the complete Harry Potter original texts allows us to construct a comprehensive reference from raw text as an approximate upper bound on the model’s knowledge within this domain. Our model-specific filtering then aligns this reference with the knowledge actually retained by each evaluated model, minimizing omissions of relevant supporting knowledge. The same procedure can be applied to other domains, especially when model holders have access to their training corpora. We focus on Harry Potter because, for real-world knowledge domains such as Wikipedia, the evaluated models’ complete training corpora are unavailable, making it difficult to construct a comparable upper bound. Evaluations in these domains remain useful, but missing supporting knowledge introduces uncertainty into path discovery and evaluation.
To examine generalization beyond the fictional Harry Potter domain, we additionally conducted experiments on YAGO3-10, a knowledge graph dataset containing real-world facts about entities and their relationships. Using Qwen3.5-4B with LoRA fine-tuning, we randomly selected 100 target triples for evaluation.
These results demonstrate that our framework can discover supporting paths and perform unlearning beyond Harry Potter, with target accuracy decreasing from 0.42 to 0.29 with GA and 0.23 with NPO.
[W2. Relationship to FAITHUN] Thank you for pointing out this relevant work. We will add a discussion of FAITHUN [1] and clarify its relationship to our paper. FAITHUN tests whether forgetting extends to related questions, including multi-hop questions whose reasoning requires the target fact. Our work examines whether alternative paths composed of other retained facts can reconstruct the target fact itself. We therefore develop a framework to discover and verify these paths, select facts to unlearn through a confidence-aware minimum cut, and evaluate the remaining paths after unlearning. This also differs from FAITHUN’s KLUE method, which selectively updates neurons associated with target knowledge.
[1] Yang et al. "FAITHUN: Toward Faithful Forgetting in Language Models by Investigating the Interconnectedness of Knowledge." EMNLP, 2025. arXiv:2502.19207.
We hope these responses address your concerns and are happy to discuss further.