- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce UnlearnBench to address the limited coverage of compositional reasoning, multi stage memorization, and attribute level forgetting in previous benchmarks, which mainly focus on single hop QA and entity or document level unlearning. The benchmark allows precise control over knowledge provenance, memorization stages, and forget and retain splits. Experiments evaluate several unlearning methods and identify different failure modes across settings.
Critical strengths
Overall, the paper is easy to follow. It provides a useful benchmark for studying selective forgetting of attributes and unlearning in multi hop reasoning settings.
Critical weaknesses
- The paper could be clearer with more figures and fewer typos.
- The experiments need clearer training details and tests on more models.
Questions for authors
- The benchmark design could be explained more clearly with an overview figure showing how UnlearnBench differs from existing benchmarks. Figures summarizing the main experimental findings would also help readers follow the analysis.
- There are several typos and formatting issues, including:
- “allen-zhu2023storage” in L155–156.
- “qwen3technicalreport” in L161.
- “BLUR (?)” in Table 2.
- The number of methods stated in L262 appears inconsistent with the methods listed.
- Many experimental results fall into two extremes: the model loses both target and retained knowledge, or it barely forgets the target knowledge. This makes the methods difficult to compare and raises the question of whether the results are due to limitations of the methods or unsuitable training settings. Although Appendix D.7 includes some results with different learning rates and training steps, the tuning budget, search ranges, and checkpoint selection criteria remain unclear.
- Most experiments focus on Qwen3 0.6B. Although the appendix includes additional Llama results, the main findings on compositional reasoning and multiple training stages are not sufficiently tested across models. Evaluating these settings on another model would help establish whether the findings hold more broadly.