acceptodds
Under review as a conference paper at ICLR 2027

Towards Substantive LLM Unlearning: Robust Evaluation and Expression-Insensitive Metric

Abstract

Large language model (LLM) unlearning aims to remove private, copyrighted, or harmful knowledge from trained models while preserving general utility. Despite recent algorithmic progress, reliably determining whether target knowledge has been removed remains challenging. Existing metrics primarily assess immediate post-unlearning behavior, often overlooking robustness and thus becoming susceptible to superficial behavioral changes and unreliable residual-knowledge measurements. This work proposes RUMOR, a comprehensive framework that evaluates unlearning-metric reliability from data-side and model-side robustness perspectives. RUMOR tests whether metrics can reliably distinguish different degrees of residual knowledge across expression forms and remain stable under model-level red-team unlearning attacks. We further introduce Unlearning Depth (UD), a robustness-oriented metric that measures whether target knowledge is consistently forgotten across expression forms. Across diverse unlearning tasks and methods, existing metrics can be misled by superficial behavioral changes, yielding unreliable residual-knowledge measurements. These findings reveal critical limitations in current unlearning evaluation and provide an extensible framework for assessing whether unlearning genuinely removes target knowledge.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, LLM Unlearning, Unlearning Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Arx1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RUMOR to examine unlearning metrics under changes in knowledge-expression form and post-unlearning interventions. It also proposes Unlearning Depth (UD), a weighted measure of disagreement between separate unlearning runs using different forms of the same knowledge. The study covers nine metrics and five unlearning methods across privacy, copyright, and toxic-content settings, with additional analyses of metric choice and evaluation anchors.

Critical strengths

The data-side study raises a useful evaluation issue: strong performance on the MCQA training examples is not reliably reflected in QA-based measurements. Appendix F helps rule out the simple explanation that those models failed to fit their MCQA examples. This is a concrete reason to question evaluations tied to one expression form.

I also appreciate the supplementary checks. Method-conditional results, the Phi experiments, alternative core metrics, and the anchor analysis expose meaningful variation rather than presenting the aggregate ranking as universal. In particular, reporting both GA's utility collapse and the poor discriminability of non-QA anchors makes the limitations easier to assess

Critical weaknesses

UD's interpretation as depth is not sufficiently supported by its construction or validation.** Equation (4) measures agreement with another unlearning run, without anchoring that agreement to successful forgetting. An identity procedure that returns the unchanged model for every form obtains UD = 0 while retaining the target knowledge and its utility. I recognize that the paper explicitly says UD is not a standalone certificate and reports utility in Table 5; this counterexample nevertheless shows why the subsequent depth rankings require a controlled comparison at established forgetting levels. Table 11 adds to the concern: NPO and RMU have large QA-anchored scores but approximately zero MCQA-anchored scores. Choosing the more discriminative anchor, or obtaining closer agreement with Prob through weighting, does not independently validate which ordering reflects deeper removal.

Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Arx1

We thank the reviewer for identifying the central limitation in the original interpretation of UD.

1. UD is conditional on established forgetting and retained utility

The reviewer’s identity counterexample exposes an important limitation of Equation (4): it measures cross-expression agreement but cannot by itself certify forgetting.

We will therefore report an outcome in the constrained form

Δf(UD)@Ur≥τu,Δ_{f} (UD) @ Uᵣ≥τᵤ,

where ΔfΔ_{f} is the absolute QA-reference forgetting reduction, UD is the cross-expression discrepancy, and Ur≥τuUᵣ≥τᵤ specifies the retained-utility requirement. UD plays the role of a dispersion measure within this report: it describes whether form-specific outcomes agree after forgetting and utility have been established.

The identity procedure obtains

0.0000(0.0000)@Ur=0.8705,0.0000 (0.0000) @ Uᵣ=0.8705,

meaning zero forgetting reduction, zero expression discrepancy, and high utility. It is therefore correctly identified as ineffective despite its UD of zero.

We also compare the five methods at a common QA residual probability of pf=0.5p_{f}=0.5. The values are piecewise-linear estimates from the observed checkpoint trajectories.

MethodForgetting reduction ΔfΔ_{f} ↑Retain utility ↑UD ↓
GradAscent0.38050.49430.0003
DPO0.38050.52100.2447
GradDiff0.38050.53040.2844
NPO0.38050.53950.3126
RMU0.38050.82540.3731

At the same forgetting level, UD remains method-dependent and does not track utility. At the stronger level pf=0.1p_{f}=0.1, GA has utility 0.0989 and UD 0.0006, whereas RMU has utility 0.7780 and UD 0.7653. Thus, GA is more expression-consistent but causes much greater utility loss.

We will remove claims such as “GradAscent is the deepest method” and use the following interpretation:

At a specified forgetting level and retained-utility condition, lower UD indicates a more expression-invariant measured forgetting outcome.

2. Anchor dependence and Table 11

The corrected evaluation matrix confirms the reviewer’s concern about anchor dependence. Table 11 cannot establish an anchor-independent ordering of removal depth.

The original non-QA-anchor analysis omitted the QA-unlearned run under non-QA evaluation forms. We completed the matrix by evaluating all four unlearning runs under all four evaluation forms, requiring 15 additional evaluations and no retraining. To isolate anchor sensitivity from the effect of difficulty weighting, we recomputed the anchor-specific UD using uniform weights: for each evaluation anchor, we take the mean absolute score difference between the anchor-form run and the other three form-specific runs, with all four runs evaluated under the same anchor.

The corrected values are:

MethodQA anchorCloze anchorKG anchorMCQA anchor
GA0.0000830.0000000.0000010.000000
GD0.6883390.0004850.0005290.000000
NPO0.6636960.0335260.0380540.016620
DPO0.4683970.0884000.0905100.022938
RMU0.8596250.0272640.0457480.001108

The complete evaluation rows explain the near-zero non-QA values:

  • Under QA evaluation, NPO’s four form-specific runs receive 0.0923, 0.7117, 0.7535, and 0.8028. Under MCQA evaluation, the same runs receive 0.0499, 0.000002, 0.000001, and approximately zero.
  • Under QA evaluation, RMU receives 0.000046, 0.8537, 0.8612, and 0.8642. Under MCQA evaluation, it receives 0.000003, 0.000323, 0.003282, and 0.000277.

MCQA evaluation therefore compresses the four outcomes into a narrow near-zero range. A low MCQA-anchored UD means that the runs receive similar MCQA scores; it does not prove that they retain equal knowledge or achieve deeper removal.

QA remains the pre-specified primary anchor because it is the benchmark’s original form and the setting in which Probability was meta-evaluated by RUMOR. This choice does not establish QA as a universal ground-truth anchor. Indeed, Probability’s MCQA data-side correlation is only 0.360 despite its overall RUMOR score of 0.847.

We will revise the paper accordingly:

  1. Present Table 11 as an anchor-sensitivity analysis.
  2. Include all four unlearning runs for every anchor.
  3. Describe difficulty weighting as heuristic scale normalization and include uniform-weight sensitivity results.
AuthorsOpenReview ↗

Looking Forward to Your Feedback

Dear Reviewer Arx1,

Thank you again for your thoughtful review and for identifying the central limitation in UD’s interpretation. In our response, we provide comparisons at matched forgetting levels and clarify that UD measures cross-expression consistency, to be interpreted jointly with forgetting efficacy and retained utility. We also commit to removing unconditional depth rankings. To address your concern about anchor dependence, we have completed the four-by-four evaluation matrix and recomputed anchor-specific UD with uniform weights. These results support presenting Table 11 as an anchor-sensitivity analysis rather than evidence of an anchor-independent ordering of removal depth.

We would greatly appreciate your feedback on whether these results and proposed revisions address your concerns. Please let us know if any aspect needs further clarification. Thank you for your time and consideration.

Thanks for your rebuttal. My concerns have been resolved. I will maintain my score.

AuthorsOpenReview ↗

Thank You for Your Feedback

Dear Reviewer Arx1,

Thank you for reviewing our response and confirming that your concerns have been resolved. Your comments helped us sharpen the interpretation of UD and clarify the limitations arising from anchor dependence.

Thank you again for your thoughtful review and constructive discussion.

Reviewer_FtxfOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RUMOR, a framework for evaluating unlearning metrics under changes in the expression of target knowledge and post-unlearning interventions. It studies nine metrics across personal-information, copyright, and toxic-knowledge scenarios. It further proposes Unlearning Depth (UD), which measures differences between checkpoints unlearned using alternative expression forms and a fixed evaluation anchor. The experiments find substantial sensitivity to multiple-choice training forms and relearning, and compare five unlearning methods using UD alongside utility measurements.

Critical strengths
  • The paper develops unlearning evaluation framework by considering both data-side and model-side robustness, and evaluate the experiments on 9 existing related metrics, 3 attacking methods.
Critical weaknesses
  • What's the implication of the UD score proposed in the paper? In the paper, it states UD measures the cross-expression consistency of an unlearning method, but I don't know how it connects to the unlearning?
  • Limited model families evaluation, only check the Llama family. it would be nice to extend to other model family.
Questions for authors
  • In this paper, UD score is developed, I have one concerns when using UD score
MethodAfter QA unlearningAfter MCQA unlearningAfter Cloze unlearningAfter KG unlearning
A: Consistently lowers target-answer likelihood0.100.100.100.10
B: Leaves all parameters unchanged0.900.900.900.90

Both methods obtain UD = 0, although Method B retains the original model and its target knowledge.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ftxf (1/2)

We thank the reviewer for the constructive comments. We address the concerns as follows.

W1. What does UD imply, and how does it connect to unlearning?

The reviewer’s counterexample highlights why the implication of UD must be stated more precisely. UD measures the expression sensitivity of an established forgetting effect: it tests whether changing the expression form used during unlearning changes the measured outcome. It does not independently establish that forgetting has occurred.

We will therefore report an unlearning outcome in the constrained form

Δf(UD)@Ur≥τu,Δ_{f} (UD) @ Uᵣ≥ τᵤ,

where ΔfΔ_{f} is the absolute reduction in residual target probability, UD is the cross-expression discrepancy, and Ur≥τuUᵣ≥τᵤ specifies a retained-utility condition. Similar to reporting precision at a specified recall level, this form evaluates consistency at a defined forgetting and utility operating point.

Under the 10% forget-ratio setting on Llama-3.2-1B-Instruct, the following table reports all three dimensions. Residual pfp_{f} is measured on the QA-reference run. Following Table 5, retained Probability and ES are averaged over the Cloze-, KG-, and MCQA-unlearned runs and across the three scenarios.

MethodResidual forget Prob. pfp_{f} ↓Prob. utility ↑ES utility ↑Avg. UD ↓
GA0.0000.0010.1120.001
GD0.0010.6900.4520.668
NPO0.0940.7200.5170.620
DPO0.3370.7250.5290.367
RMU0.0010.8080.6020.787

These results show why the three quantities must be considered jointly. GA combines low residual probability and low UD with almost complete utility collapse. DPO preserves utility and has lower UD than NPO, GD, and RMU, but retains substantially more target knowledge. RMU achieves strong forgetting and the highest utility, but its outcome is more expression-sensitive.

We will therefore replace statements that interpret low UD alone as “deeper forgetting” with:

Among outcomes with comparable forgetting efficacy and acceptable retained utility, lower UD indicates a less expression-sensitive forgetting effect.

W2. Evaluation across model families

We would like to clarify that our experiments are not restricted to Llama. Appendix D also evaluates Phi-3.5-mini, covering all nine RUMOR metrics and all five unlearning methods across the three scenarios.

When aggregating over all three backbones, Probability achieves the highest overall RUMOR score among the nine evaluated metrics (0.816); MCQA remains the most challenging expression form, and relearning causes the largest model-side degradation.

The average Phi UD values are 0.0266 for GA, 0.2337 for DPO, 0.3906 for NPO, 0.4586 for RMU, and 0.4829 for GD. This additional model family preserves the broad grouping, although GD and RMU exchange order locally. We will highlight the Phi results more clearly in the main text.

AuthorsOpenReview ↗

Response to Reviewer Ftxf (2/2)

Q1. Both successful forgetting and an unchanged model can obtain UD=0

The reviewer’s example illustrates precisely why UD must be conditioned on an established forgetting effect. Suppose the original residual target probability is 0.90:

  • Method A reduces it to 0.10 in every form: ΔfΔ_{f}=0.80, UD =0.
  • Method B leaves it at 0.90 in every form: ΔfΔ_{f}=0, UD =0.

The two procedures have the same UD but different forgetting reductions. In our constrained reporting form, Method B is immediately identified by its zero ΔfΔ_{f}.

The corresponding negative control in the Personal information setting is:

ProcedureResidual pfp_{f} ↓Forgetting reduction ΔfΔ_{f} ↑Retain utility ↑UD ↓
Identity0.88050.00000.87050.0000

We further compare the methods at common QA residual-probability levels using their checkpoint trajectories:

Matched pfp_{f}MethodForgetting reduction ΔfΔ_{f} ↑Retain utility ↑QA-anchored UD ↓
0.5GA0.38050.49430.0003
0.5DPO0.38050.52100.2447
0.5GD0.38050.53040.2844
0.5NPO0.38050.53950.3126
0.5RMU0.38050.82540.3731
0.1GA0.78050.09890.0006
0.1GD0.78050.17280.5834
0.1RMU0.78050.77800.7653

These are piecewise-linear trajectory estimates. NPO and DPO are omitted at pf=0.1p_{f}=0.1 because their observed trajectories do not reach that level.

At the same forgetting level, UD captures a separate property from utility. For example, at pf=0.1p_{f}=0.1, GA has near-zero UD but retains only 0.0989 utility, whereas RMU retains 0.7780 utility but has higher expression discrepancy.

We will therefore present UD only together with forgetting reduction and retained utility.

AuthorsOpenReview ↗

Looking Forward to Your Feedback

Dear Reviewer Ftxf,

Thank you again for your constructive comments, particularly the counterexample illustrating that both successful forgetting and an unchanged model can yield UD = 0. In our responses, we clarify that UD measures cross-expression consistency and must be interpreted jointly with forgetting efficacy and retained utility. We have provided an identity negative control and comparisons at matched forgetting levels to make this distinction concrete. We have also highlighted the existing Phi-3.5-mini experiments in Appendix D, which extend our evaluation beyond the Llama family.

We would greatly appreciate your feedback on whether these explanations and results address your concerns about UD’s interpretation and model-family coverage. Please let us know if any points need further clarification. Thank you for your time and consideration.

Reviewer_gx7UOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper makes two contributions. The first is RUMOR, a framework for meta-evaluating unlearning metrics along two axes. Data-side robustness fine-tunes models on 10%–100% of a forget set written in an alternative form (MCQA, Cloze, KG triples), scores each checkpoint on the original QA form, and reports the Spearman correlation with the ideal monotone order. Model-side robustness follows Wang et al. (2024c) and computes the Pearson correlation between metric values before and after jailbreak, relearning, and quantization attacks. The authors evaluate nine metrics (Prob, ROUGE, EM, ES and five MIA variants) on TOFU, NarrativeQA and PKU-SafeRLHF with Llama-3.2-1B and Llama-3.1-8B, plus Phi-3.5-mini in the appendix, and conclude that Probability is the most robust overall (0.847 in Table 1). The second contribution is Unlearning Depth (UD). UD unlearns the same facts in different forms and measures how far each alternative-form run lands from standard QA-form unlearning on QA evaluation, with difficulty weights estimated from the data-side runs. Applied to GA, GD, NPO, DPO and RMU, UD ranks GA and DPO as the "deepest" and RMU as the shallowest (Tables 2, 4, 5 and Figure 3).

Critical strengths
  1. The question is a real one, and separating where a metric fails is a sensible way to frame it. Most unlearning papers report forget-set scores under one fixed format and one checkpoint. Section 3 argues clearly that format sensitivity and post-hoc recovery are different failure sources and should be diagnosed separately. Even if the execution has issues, a data-side vs. model-side breakdown is a useful lens for the community.

  2. The empirical coverage is broad for a meta-evaluation paper: 9 metrics, 3 scenarios, 5 unlearning methods, 3 attack families (5 variants), 2 Llama scales plus Phi-3.5-mini (Appendix D), and per-model, per-scenario breakdowns in Tables 13–18. Some findings are useful in their own right and seem robust across the tables. Relearning is by far the most discriminative attack, while jailbreak and quantization barely move rankings (Table 1). MIA-GradNorm is consistently the least stable metric. Metric robustness depends strongly on the unlearning method (Table 6, where EM and ROUGE drop to 0.071 and −0.102 for GD).

  3. The paper is unusually transparent about its own caveats in the appendix. It reports the anchor dependence of UD (Table 11), checks in-form MCQA accuracy to rule out a trivial explanation (Table 12), explains why pooled correlations differ from averages of the per-pair tables (Appendix K), and even footnotes a suspicious duplicate row (Table 16). I could check several numbers by hand, and they line up. The MCQA column of Table 1 is the mean of Tables 13–18 (e.g., Prob: 0.360), the Avg. column of Table 2 matches its entries, and Table 5's Avg. UD matches the 1B columns of Table 2.

Critical weaknesses
  1. UD is an absolute gap from the reference run, so it rewards collapse rather than depth. GA destroys the model (retain-set Prob 0.001, Table 5) and gets UD ≈ 0 (Table 2), but scores 0.466 at a 1% forget ratio (Table 4). Methods aren't matched on forget quality or utility, and every non-QA anchor gives UD ≈ 0 for all methods (Table 11).

  2. The data-side test has no ground truth for residual knowledge. A low MCQA Spearman may just mean that MCQA training never produced QA-form knowledge; Appendix F only shows in-form accuracy. The difficulty weighting then gives this least reliable form 0.74–0.81 of UD's weight (Table 3), which amplifies noise instead of calibrating for it.

  3. The novelty over Wang et al. (2024c), OpenUnlearning, and the multi-form probing already in RWKU and TOFU is modest. No seeds or CIs are reported, and the headline gaps are small (Prob 0.847 vs. ES 0.828, Table 1). Pooling across methods also inflates Pearson correlations: EM on GD is 0.071 method-conditional vs. 0.907 pooled (Table 6).

Questions for authors
  1. Could you report, for each method and scenario, the forget-set Prob of the standard QA-unlearned reference model and the retain/utility numbers, and recompute UD normalized by the reference drop, e.g., |E_alt − E_std| / (E_before − E_std)? Does GA still rank first once methods are tuned to comparable utility?

  2. Figure 3's first continual round (one 1% request, PI, Llama-3.2-1B) seems to match the 1% column of Table 4, yet the values differ a lot (RMU ≈ 0.27 vs. 0.795, NPO ≈ 0.18 vs. 0.503, GD ≈ 0.41 vs. 0.582). What differs between these two settings? Also, how many seeds underlie the "second request is a critical transition point" claim?

  3. What exactly is the starting model for UD and the data-side runs? Is it a model first fine-tuned on QA-form forget+retain and then unlearned with Cloze, KG or MCQA forget data? For the data-side runs, which base is fine-tuned on 10–100% of the alt-form data, and is the retain set included? The main text says Llama-3.2-1B, while Appendix F/K say -Instruct.

6 comments
AuthorsOpenReview ↗

Response to Reviewer gx7U (1/3)

We thank the reviewer for the detailed and constructive assessment. We address the three weaknesses and three questions below.

W1. UD, model collapse, and unmatched operating points

We agree that low UD alone does not establish successful unlearning: all form-specific runs may forget consistently, retain the target consistently, or collapse consistently. We therefore reinterpret UD as a conditional measure of cross-expression consistency and report it jointly with the absolute forgetting reduction ΔfΔ_{f} and retained utility:

Δf(UD)@Ur≥τu,Δ_{f} (UD) @ Uᵣ≥τᵤ,

where Δf=pf(M0)−pf(MuQA)Δ_{f}=p_{f}(M₀)-p_{f}(Mᵤ^{QA}). An unchanged model has Δf=0Δ_{f}=0, while a collapsed model fails the utility requirement even if it has low UD. Because the appropriate utility threshold is application-dependent, we report Uᵣ explicitly rather than impose a universal threshold.

The new comparisons in Q1 show that GA’s low UD at the original checkpoints coincides with near-zero retained utility. After matching retained utility, GA no longer has the lowest UD; after matching forgetting efficacy, its low UD still coincides with substantially lower utility than RMU. We therefore remove unconditional claims that GA is the “deepest” or best method. At matched forgetting levels, the trade-off remains. At pf=0.5p_{f}=0.5, GA and RMU achieve the same forgetting reduction of 0.3805, but GA has utility 0.4943 and UD 0.0003, whereas RMU has utility 0.8254 and UD 0.3731. At pf=0.1p_{f}=0.1, GA has utility 0.0989 and UD 0.0006, while RMU has utility 0.7780 and UD 0.7653.

The completed uniform-weight evaluation matrix also confirms anchor dependence. NPO changes from 0.6637 under QA to 0.0166 under MCQA, while RMU changes from 0.8596 to 0.0011. These low MCQA-anchored values result from score compression and do not establish deeper removal. We will present Table 11 only as an anchor-sensitivity analysis.

W2. Data-side ordering and difficulty weighting

We agree that in-form MCQA accuracy alone cannot establish that the acquired knowledge is accessible through QA. We therefore evaluate the 100% MCQA-trained checkpoint on the same 400 QA prompts used in the data-side experiment.

ScenarioUntuned QA token-F1 ↑100% MCQA-trained QA token-F1 ↑Increase
Personal information0.20000.4385+0.2385
Copyright content0.02660.1098+0.0832
Toxic knowledge0.12140.3275+0.2061

QA-form token-F1 increases in all three scenarios, providing direct evidence of cross-form transfer. The data-side ordering itself comes from strictly nested target-training subsets, with the retain data and training protocol fixed. We will clarify that this is an intervention-derived ordinal reference rather than an absolute ground-truth measurement of latent knowledge.

The Difficulty Weighting Module is a heuristic scale normalization, rather than a reliability weight. It assigns wij∝1/dijwᵢⱼ∝1/dᵢⱼ, where dijdᵢⱼ is the average adjacent score change along the nested sequence. This prevents compressed score ranges from contributing negligibly to UD, but may amplify noise when rankings are unstable. We will state this limitation and report uniform-weight results as a sensitivity analysis.

W3. Novelty, statistical uncertainty, and aggregation

Our contribution is the controlled separation of data-side and model-side failures in metric meta-evaluation. Wang et al. (2024c) study robustness under post-hoc interventions, while RUMOR additionally tests whether metrics recover a ten-level checkpoint ordering across structurally different expression forms. OpenUnlearning primarily studies two knowledge states constructed from paraphrased QA pairs or merged biographies on TOFU. RWKU and TOFU use multiple probes to evaluate unlearning methods, whereas RUMOR makes the evaluation metric itself the object of study. We will clarify these distinctions without claiming that multi-form probing or post-hoc testing is individually new.

We added 95% percentile bootstrap confidence intervals using 5,000 resamples of the stored evaluation outputs. Data-side intervals resample checkpoint levels, model-side intervals resample paired pre-/post-intervention scores, and aggregate intervals use hierarchical resampling.

Robustness quantityEstimate95% bootstrap CI
Overall Probability0.847[0.766, 0.919]
Overall ES0.828[0.728, 0.892]
GD-conditional EM0.071[−0.684, 1.000]
AuthorsOpenReview ↗

Response to Reviewer gx7U (2/3)

Probability performs better on the data side than ES (0.787 vs. 0.734), whereas ES performs better on the model side (0.921 vs. 0.908). The overlapping aggregate intervals do not support statistically significant superiority. We therefore describe Probability and ES as having the two highest observed aggregate scores and report the complete intervals for all nine metrics in the supplement. These intervals quantify uncertainty over the stored evaluation units rather than variation across independent training runs.

Pooled and method-conditional correlations address different auditing tasks. The pooled result tests whether comparisons across methods remain stable, for which between-method variation is relevant; the conditional result tests stability among checkpoints within one method. EM’s pooled model-side correlation of 0.907 and GD-conditional correlation of 0.071 therefore indicate good cross-method but weak and uncertain within-GD robustness. Under method-conditional aggregation, the mean scores are 0.874 for Probability, 0.972 for ES, and 0.730 for EM. We will report both scopes.

Q1. Forgetting, utility, reference normalization, and utility matching

Let E_{before} be the pre-unlearning Probability on the QA forget set and Mᵤ^{QA} the QA-unlearned reference. Following the reviewer’s suggestion, we compute

UDnorm=UDEbefore−E(MuQA,Kf,QA).\mathrm{UD}_{\mathrm{norm}} = \frac{\mathrm{UD}} {E_{\mathrm{before}}-E(M_u^{QA},K_f,QA)}.

The following table reports forgetting, utility, and UD from the same QA-reference checkpoint.

MethodScenarioForget Prob ↓Retain Prob ↑Retain ES ↑Raw UD ↓Ref. drop ↑Norm. UD ↓
GAPersonal information0.0000000.0000000.0347790.0002020.8804820.000229
GDPersonal information0.0000000.3696450.1136310.7186330.8804820.816181
NPOPersonal information0.0923130.3043550.0838620.6982510.7881690.885915
DPOPersonal information0.3038240.4875640.2067000.4611130.5766580.799630
RMUPersonal information0.0000460.8331480.6194070.8630190.8804360.980218
GACopyright content0.0000000.0000000.2609250.0000000.9605400.000000
GDCopyright content0.0000130.2345180.4045390.8159890.9605270.849522
NPOCopyright content0.0020840.6201290.6441250.8595900.9584560.896848
DPOCopyright content0.4021760.6207200.7736590.4706120.5583640.842840
RMUCopyright content0.0030540.8868270.9346690.9280530.9574860.969260
GAToxic knowledge0.0000000.0000000.0183630.0024250.6322360.003836
GDToxic knowledge0.0033090.4038390.0956650.4681950.6289270.744434
NPOToxic knowledge0.1884640.3613550.0804610.3011800.4437720.678681
DPOToxic knowledge0.3058120.3995400.0986970.1679410.3264240.514487
RMUToxic knowledge0.0001830.6245670.2132620.5711670.6320530.903669

Reference normalization does not change GA’s numerical rank at these checkpoints, but its low UD coincides with severe utility collapse.

We next select each method’s optimization horizon to approach common QA-reference retain-Probability targets: 0.865 for Personal information and Copyright content, and 0.600 for Toxic knowledge. All achieved utilities are within 0.04 of their targets.

AuthorsOpenReview ↗

Response to Reviewer gx7U (3/3)

ScenarioMethodMatched retain Prob ↑QA forget Prob ↓Raw UD ↓Norm. UD ↓
Personal informationGA0.88370.87910.0016261.1855
Personal informationGD0.88440.87960.0011441.3702
Personal informationNPO0.88370.87920.0015691.2198
Personal informationDPO0.88360.87960.0005340.6399
Personal informationRMU0.84630.00120.8673790.9865
Copyright contentGA0.83540.78600.1423340.8155
Copyright contentGD0.88810.78830.1350000.7836
Copyright contentNPO0.89400.84520.1034550.8970
Copyright contentDPO0.84720.83450.0887290.7041
Copyright contentRMU0.83480.01580.9392520.9942
Toxic knowledgeGA0.61770.51700.1107220.9611
Toxic knowledgeGD0.63520.47360.1941081.2238
Toxic knowledgeNPO0.62710.52880.0664090.6423
Toxic knowledgeDPO0.56370.49080.0935720.6615
Toxic knowledgeRMU0.58380.00340.6121570.9734

After utility matching, DPO has the lowest raw and normalized UD for Personal information and Copyright content, while NPO has the lowest values for Toxic knowledge. GA therefore no longer ranks first. However, these operating points still differ substantially in forgetting efficacy; for example, in Personal information, RMU reduces the forget Probability to 0.0012, while the other four methods remain near their pre-unlearning level. Normalized UD is also sensitive when the reference drop is close to zero. We therefore avoid deriving an unconditional method ranking from either raw or normalized UD.

Q2. Difference between Figure 3 and Table 4

The two results come from different experimental settings:

  • Table 4, 1% ratio: unlearns on the standard forget01 set, uses retain99, and evaluates on the same 40 examples.
  • Figure 3, continual step 1: unlearns on forget01_step1, uses retain95, and evaluates on the complete 200-example forget05 set.

The standard forget01 is equivalent to forget01_step5 and is disjoint from forget01_step1. Figure 3 therefore describes the first request toward a cumulative 5% target, while Table 4 reports an independent 1% experiment.

The original Figure 3 used seed=0; we repeated it with seed=1 and seed=2. The table reports the mean ± standard deviation across seeds and the 95% confidence interval of the paired seed-wise change.

MethodStep-1 UDStep-2 UDΔUD95% CI of ΔUD
GradAscent0.463 ± 0.0390.404 ± 0.097−0.059[−0.218, 0.100]
GradDiff0.435 ± 0.0360.693 ± 0.038+0.258[0.141, 0.375]
NPO0.171 ± 0.0060.527 ± 0.023+0.355[0.311, 0.400]
DPO0.174 ± 0.0070.572 ± 0.014+0.398[0.345, 0.451]
RMU0.252 ± 0.0110.507 ± 0.021+0.255[0.232, 0.279]

UD increases at the second request for GradDiff, NPO, DPO, and RMU across all three seeds, with confidence intervals excluding zero. GradAscent is the exception. We will therefore describe the second request as a transition observed for four of the five methods, rather than as a universal critical point.

Q3. Starting models and data-side training

  • UD experiments: Within each scenario, all 20 method-form runs independently start from the same scenario-specific checkpoint trained on the full QA-form data. For the personal-information 1B setting, this is open-unlearning/tofu_Llama-3.2-1B-Instruct_full.
  • Data-side experiments: For the 1B and 8B settings, runs start from meta-llama/Llama-3.2-1B-Instruct and meta-llama/Llama-3.1-8B-Instruct, respectively. Each run is trained on the fixed QA-form retain90 set and a 10%–100% subset of the forget data expressed as Cloze, KG, or MCQA. Every percentage is independently initialized from its corresponding base model.

We will use the full Instruct model names consistently throughout the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviewer Feedback and Rebuttal

We sincerely thank the Area Chair and all reviewers for the time and care devoted to evaluating our manuscript. We especially appreciate the detailed examination of our experiments, the thoughtful counterexamples, and the constructive suggestions provided throughout the discussion. This feedback has helped us sharpen our claims, strengthen the empirical analysis, and communicate the scope and limitations of our work more clearly. We are also grateful for the reviewers’ recognition of the value of our research question and evaluation framework.

The key strengths highlighted by the reviewers are:

  • Useful evaluation perspective: Reviewer gx7U described the separation of data-side and model-side failures as “a useful lens for the community.” Reviewer Arx1 recognized the data-side findings as “a concrete reason to question evaluations tied to one expression form.”
  • Broad empirical coverage: Reviewer gx7U noted that “the empirical coverage is broad for a meta-evaluation paper,” spanning nine metrics, three scenarios, five unlearning methods, and multiple model backbones.
  • Transparent analysis: Reviewer Arx1 appreciated that the supplementary analyses “expose meaningful variation rather than presenting the aggregate ranking as universal,” while gx7U highlighted the paper’s transparency about its caveats.

Guided by these suggestions, we clarified that UD measures cross-expression consistency and must be interpreted jointly with forgetting efficacy and retained utility. We provided an identity negative control, reference-normalized UD, and comparisons at matched forgetting or utility levels, and committed to removing unconditional depth rankings. We also completed the anchor-sensitivity analysis, added cross-form transfer evidence, bootstrap confidence intervals, and three-seed continual-unlearning results, and clarified the experimental protocols and existing Phi evaluation.

We appreciate the reviewers’ time in reconsidering our work following the rebuttal. Reviewers gx7U and Arx1 explicitly confirmed that their concerns were resolved. Reviewers gx7U and Ftxf increased their ratings from Weak Rejection to Weak Acceptance.

We will incorporate the additional results, clarifications, and explicit limitations into the revised manuscript. We thank the reviewers again for their constructive engagement and the Area Chair for coordinating the discussion and considering our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.