- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces RUMOR to examine unlearning metrics under changes in knowledge-expression form and post-unlearning interventions. It also proposes Unlearning Depth (UD), a weighted measure of disagreement between separate unlearning runs using different forms of the same knowledge. The study covers nine metrics and five unlearning methods across privacy, copyright, and toxic-content settings, with additional analyses of metric choice and evaluation anchors.
Critical strengths
The data-side study raises a useful evaluation issue: strong performance on the MCQA training examples is not reliably reflected in QA-based measurements. Appendix F helps rule out the simple explanation that those models failed to fit their MCQA examples. This is a concrete reason to question evaluations tied to one expression form.
I also appreciate the supplementary checks. Method-conditional results, the Phi experiments, alternative core metrics, and the anchor analysis expose meaningful variation rather than presenting the aggregate ranking as universal. In particular, reporting both GA's utility collapse and the poor discriminability of non-QA anchors makes the limitations easier to assess
Critical weaknesses
UD's interpretation as depth is not sufficiently supported by its construction or validation.** Equation (4) measures agreement with another unlearning run, without anchoring that agreement to successful forgetting. An identity procedure that returns the unchanged model for every form obtains UD = 0 while retaining the target knowledge and its utility. I recognize that the paper explicitly says UD is not a standalone certificate and reports utility in Table 5; this counterexample nevertheless shows why the subsequent depth rankings require a controlled comparison at established forgetting levels. Table 11 adds to the concern: NPO and RMU have large QA-anchored scores but approximately zero MCQA-anchored scores. Choosing the more discriminative anchor, or obtaining closer agreement with Prob through weighting, does not independently validate which ordering reflects deeper removal.
Questions for authors
See weakness.
4 comments
Looking Forward to Your Feedback
Dear Reviewer Arx1,
Thank you again for your thoughtful review and for identifying the central limitation in UD’s interpretation. In our response, we provide comparisons at matched forgetting levels and clarify that UD measures cross-expression consistency, to be interpreted jointly with forgetting efficacy and retained utility. We also commit to removing unconditional depth rankings. To address your concern about anchor dependence, we have completed the four-by-four evaluation matrix and recomputed anchor-specific UD with uniform weights. These results support presenting Table 11 as an anchor-sensitivity analysis rather than evidence of an anchor-independent ordering of removal depth.
We would greatly appreciate your feedback on whether these results and proposed revisions address your concerns. Please let us know if any aspect needs further clarification. Thank you for your time and consideration.
Thanks for your rebuttal. My concerns have been resolved. I will maintain my score.
Thank You for Your Feedback
Dear Reviewer Arx1,
Thank you for reviewing our response and confirming that your concerns have been resolved. Your comments helped us sharpen the interpretation of UD and clarify the limitations arising from anchor dependence.
Thank you again for your thoughtful review and constructive discussion.
Response to Reviewer Arx1
We thank the reviewer for identifying the central limitation in the original interpretation of UD.
1. UD is conditional on established forgetting and retained utility
The reviewer’s identity counterexample exposes an important limitation of Equation (4): it measures cross-expression agreement but cannot by itself certify forgetting.
We will therefore report an outcome in the constrained form
Δf(UD)@Ur≥τu,
where Δf is the absolute QA-reference forgetting reduction, UD is the cross-expression discrepancy, and Ur≥τu specifies the retained-utility requirement. UD plays the role of a dispersion measure within this report: it describes whether form-specific outcomes agree after forgetting and utility have been established.
The identity procedure obtains
0.0000(0.0000)@Ur=0.8705,
meaning zero forgetting reduction, zero expression discrepancy, and high utility. It is therefore correctly identified as ineffective despite its UD of zero.
We also compare the five methods at a common QA residual probability of pf=0.5. The values are piecewise-linear estimates from the observed checkpoint trajectories.
At the same forgetting level, UD remains method-dependent and does not track utility. At the stronger level pf=0.1, GA has utility 0.0989 and UD 0.0006, whereas RMU has utility 0.7780 and UD 0.7653. Thus, GA is more expression-consistent but causes much greater utility loss.
We will remove claims such as “GradAscent is the deepest method” and use the following interpretation:
2. Anchor dependence and Table 11
The corrected evaluation matrix confirms the reviewer’s concern about anchor dependence. Table 11 cannot establish an anchor-independent ordering of removal depth.
The original non-QA-anchor analysis omitted the QA-unlearned run under non-QA evaluation forms. We completed the matrix by evaluating all four unlearning runs under all four evaluation forms, requiring 15 additional evaluations and no retraining. To isolate anchor sensitivity from the effect of difficulty weighting, we recomputed the anchor-specific UD using uniform weights: for each evaluation anchor, we take the mean absolute score difference between the anchor-form run and the other three form-specific runs, with all four runs evaluated under the same anchor.
The corrected values are:
The complete evaluation rows explain the near-zero non-QA values:
MCQA evaluation therefore compresses the four outcomes into a narrow near-zero range. A low MCQA-anchored UD means that the runs receive similar MCQA scores; it does not prove that they retain equal knowledge or achieve deeper removal.
QA remains the pre-specified primary anchor because it is the benchmark’s original form and the setting in which Probability was meta-evaluated by RUMOR. This choice does not establish QA as a universal ground-truth anchor. Indeed, Probability’s MCQA data-side correlation is only 0.360 despite its overall RUMOR score of 0.847.
We will revise the paper accordingly: