- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper conducted a series of experiments to test the limitations of existing benchmarks and their evaluation methods. A typical benchmark setting contains a set of target facts to remove (target) and a set of facts that need to be preserved (retain). However, as many unlearning methods retrain the model on the “retrain set” to preserve or restore the knowledge of the facts in the retain set, certain facts that are not in the target set might have been forgotten as collateral damage in the original unlearning process. As a result, the authors argue that reporting the effectiveness of an unlearning method on these two sets is not sufficient, and should report the “untargeted-holdout drop” against matched retraining. In addition to validating this core argument, the authors also show that on the widely used benchmark, TOFU, one of the “standard” methods (Gradient Difference) expresses the same issue of forgetting untargeted facts.
Critical strengths
- The problem is clearly defined, with an operational test of "selectivity": forget the targets, keep the retain set, and damage untargeted facts no more than retraining would.
- The authors conducted a great number of experiments testing different configurations of unlearning methods, using their synthetic data and a number of public benchmarks.
Critical weaknesses
- The presentation of this paper is far from ideal. For example, the structure and flow make it difficult to follow the important settings and the significance of the results of each experiment. Some terminology is introduced before formally defining it (e.g., Forget gap, Protocol U) and the use is not always consistent.
- The main conclusion, while interesting, is not too surprising. Some aspects of the key message have also been reported previously. For example, Peleg et al. (2026) highlights the “over-forgetting” issue and Ko et al. (2025) reports that unlearned models fail up to 98.7% of test cases that were answerable by the original pretrained models.
Questions for authors
- It appears that the paper only uses the ROUGE-L score from TOFU experiments. What do other metrics proposed in TOFU show?
- How practical is the proposed evaluation protocol (Section 6)? For example, if the targeted forget set of facts exists in pretraining data, does this mean that the model pretraining is needed?