- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates a mismatch between common LLM unlearning objectives and reference-based evaluation. One-sided objectives continue suppressing forget-answer probabilities even after the model approaches the behavior of a model that never learned the forgotten data. The authors propose NGREF, which minimizes token-level KL divergence from a frozen pretrained checkpoint on forget examples while maintaining a cross-entropy objective on retain examples. The paper provides a distribution-space analysis and evaluates the method across 24 TOFU configurations, supplemented by training trajectories, controlled objective comparisons, preconditioner ablations, and a secondary extraction experiment.
Critical strengths
- The distinction between suppressing an answer and recovering the behavior of an ignorant reference is well motivated.
- The evaluation covers two models, three forget splits, and four LoRA ranks.
Critical weaknesses
- The figures use inconsistent formatting and have poor readability. Please standardize their presentation and improve the legibility of labels and annotations.
- NGREF matches the pretrained checkpoint, whereas TOFU evaluates against a retain-only model. The suitability of this surrogate needs stronger empirical justification.
- The baseline performs well before over-suppression. Comparisons under a shared stopping protocol would better establish NGREF’s practical advantage.
- The objective uses standard KL distillation and retain-set training. Its distinction from existing reference-based unlearning methods should be clarified.
Questions for authors
see weakness