- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces TRIAGE, a framework that complements behavioral evaluation of LLM unlearning with model-internal diagnostics. CoFi and CHess use diagonal approximations of the empirical Fisher and Hessian to measure changes in parameter sensitivity and local curvature. A Forget / Adjacent-Retain / Generic-Retain partition helps describe whether these changes are concentrated on the target or extend to retained content. TRIAGE assigns each checkpoint to one of four classes using CoFi. Experiments cover 12 methods, four language models, and WMDP, TOFU, and MUSE, although not every model is tested on every benchmark. The results suggest that similar behavioral forgetting can accompany different internal changes. The paper also includes sensitivity and robustness analyses and a fixed-budget relearning attack.
Critical strengths
-
The adjacent-retain control is useful. Tables 1 and 2 show that relatively preserved general utility can coexist with large internal sensitivity changes on semantically adjacent retained content. Bringing this control into model-internal evaluation adds useful information beyond a standard Forget / Generic-Retain comparison.
-
The experimental coverage is broad, and the interpretation is mostly careful. The paper examines several methods, models, and benchmarks, with additional checks on sample size, probe count, and diagnostic subspace. The authors explicitly acknowledge that internal change does not establish knowledge removal. This makes the results useful as complementary diagnostics rather than a certificate of successful unlearning.
Critical weaknesses
-
The diagnostic interpretation of AdjGap needs stronger validation. The authors acknowledge that structural change does not imply behavioral damage. Still, disagreement between structural and behavioral signals suggests non-redundancy, but does not by itself validate AdjGap as an indicator of collateral damage. The relearning results mainly connect CoFi changes with recovery; they do not clearly establish the incremental diagnostic value of AdjGap beyond the overall magnitude of CoFi changes. Labels such as “collateral dominant” may suggest that actual damage has been demonstrated.
-
Uncertainty around class boundaries remains important. The authors provide some stability analyses and acknowledge uncertain boundary cases, but classification still mainly follows point estimates. For example, RMU's AdjGap in Table 1 is only 1.01 percentage points. A confidence interval for AdjGap itself, accounting for the joint uncertainty of its components, would help determine whether the sign used for classification is reliable. The separate intervals for Forget and Adjacent-Retain do not answer this directly.
Questions for authors
- After accounting for initial behavioral forgetting and general utility, does AdjGap add information about relearning recovery or damage on Adjacent-Retain content?
- Which class assignments remain reliable when using paired confidence intervals for AdjGap and repeated unlearning runs with different training seeds?
2 comments
Most of my concern has been addressed, and I have raised my score.
Polite ping
Dear reviewer,
We believe we have addressed all concerns raised in your review. Please let us know if there are any outstanding issues you would like us to address. We would sincerely appreciate your engagement and time for engagement is running out.
Again, thanks a lot for your valuable feedback.
Sincerely,
The authors