- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper recasts targeted factual unlearning as a joint constraint problem: each prompt or internal readout of an old fact is treated as an “access” whose old-answer score responds linearly to a weight update, and the update is confined to a null-space-protected subspace under a covariance-weighted norm. Within this local model, the authors use a Farkas-type certificate to characterize whether all required decreases can be achieved by one shared update, and a dual formulation to obtain the unique minimum-cost update, whose rank is bounded by the number of active constraints. They turn this into U2E, a pipeline that constructs multiple external and internal accesses, selects layers by transfer to a held-out prompt view, applies a trust-region step, validates candidates on the full nonlinear model with relinearization, and abstains on requests it cannot satisfy safely. On MultiCounterFact with Llama-3.2-3B and Llama-3.1-8B, U2E reports lower old-answer accuracy than ZeroUnlearn together with much smaller perplexity drift, along with improved though still limited robustness on unseen access routes.
Critical strengths
- Clean, well-motivated framing. The paper treats factual unlearning as joint suppression across several accesses under shared knowledge protection. This is a natural and compelling way to pose the problem, and it clarifies why single-query edits can leave the old fact recoverable through other routes. Making the shared-parameter constraint explicit, with feasibility as a prerequisite, is a useful conceptual contribution.
- Rigorous statistical protocol. The evaluation uses paired seed bootstraps, exact sign-flip tests, and Holm correction, with seeds treated as the replication unit. The authors are candid about limitations: they report a cross-model audit (Appendix L) in which U2E underperforms ZeroUnlearn on average suppression, and they describe the hyperparameter sensitivity study as descriptive rather than confirmatory.
- Well-designed, carefully executed experiments. The experiments are chosen to probe both the strengths and the weaknesses of the method. They include held-out access families, black-box extraction, component ablations, scaling to a second model size, and a preservation-calibrated comparison. Behavioral metrics count abstained requests in the denominator, which keeps the reported results honest.
Critical weaknesses
-
Baselines appear untuned, which weakens the main comparison.
- The primary comparator, ZeroUnlearn, barely changes the model (FE 17.6 vs. 18.2 for Base), so outperforming it is a low bar.
- Several loss-based baselines show signs of misconfiguration rather than a fair comparison. GA, NPO, and UNDIAL collapse retention to roughly 0-5%. NPO behaves completely differently across scales (FE 0.8 on 3B vs. 22.8 on 8B). SimNPO raises accuracy on retain data far above Base (RE 59.8 vs. 21.7), which suggests a degenerate run. Using released-implementation defaults is not equivalent to a tuned comparison.
- The most relevant baselines are editing methods that handle multiple prompts or targets (AlphaEdit, MEMIT-style editors, the refusal and empty-set editors of Li et al.). AlphaEdit appears only in an appendix, and the preservation-calibrated comparison (App. K) omits GA, NPO, RMU, and UNDIAL.
-
Retention and generalization claims are stronger than the evidence supports.
- Retained knowledge drops from 20.68 to 18.24 (about 12% relative) and FS from 19.6 to 15.8, so “near-base” retention is generous.
- The utility-panel result (62.087 vs. 62.084) is reported to three decimals on 3,500-example panels, where sampling noise is on the order of ±0.8 points. This shows the absence of large degradation, not preservation to within 0.003.
- The original-query view is part of the construction set, so FE is measured in-sample. Only FG and the nine held-out families in the robustness study test transfer.
- Transfer to unseen routes is modest: 73% of facts remain recoverable under Extraction@20, worst-route residual improves only from .724 to .654, and only 47.3% vs. 36.3% of facts are robust at τ = 0.75.
-
The cost analysis does not match the reported profile. Sec. 5.4 argues that U2E’s optimization scale stays far below that of optimization-based unlearning, but Table 4 shows 191.5 s for U2E vs. 15.7 s (ZeroUnlearn) and 10.0 s (RMU), with 43 GiB vs. 17 GiB of GPU memory and 45/50 vs. 50/50 successes. An m × m dual solve should not take 164 s, so the cost evidently comes from repeated full-model passes, which the complexity discussion does not reflect.
-
Evaluation is limited to a single small-scale setting. TOFU, RWKU, and WMDP are cited but never used, and the paper evaluates only 50 forget facts on one dataset with no study of how the method scales with the number of requests. A side-by-side comparison with well-established methods on a standard unlearning benchmark such as RWKU, TOFU, or WMDP is needed to support the claim that U2E is a practical framework for selective factual unlearning.
-
The code is not provided, so this weakens reproducibility.
Minor:
-
I would rephrase this line in the abstract “and covariance and null-space protection”, it does not sound very good.
-
line 121: behavioral factual unlearning, there is some ambiguity in this phrase, can the authors elaborate more?
-
Appendix E contains only a Table so it ends up rendering weirdly, making it seem like it is empty. I would fix that for reading clarity purposes.
-
Algorithm 1: after acceptance, S is never updated, so the loop as written doesn’t terminate.
-
The “residual” metric is not defined in the main text.
Questions for authors
-
Can you show the old fact is weakened (e.g., probing, or suppression with the neutral target excluded from the contrast), not just the output contrast?
-
Why does SimNPO raise accuracy above Base, and how were baselines configured?
5 comments
2. Does U2E genuinely weaken old facts while preserving unrelated knowledge? (W2, Q1)
Thank you for your careful comments on factual suppression, knowledge preservation, and generalization. Following your suggestion, we conducted additional experiments to directly examine old-answer suppression and clarified the preservation and transfer results.
Genuine old-answer suppression (Q1). We measured the teacher-forced sequence log-probability of the original answer without involving the neutral target. This directly evaluates the model's remaining access to the original fact.
Table 4. Neutral-free old-answer suppression (six paired seeds, 300 facts).
| Metric | Base | U2E | ZeroUnlearn |
|---|---|---|---|
| Rewrite old-answer decrease ↑ (nats) | 0.0000 | 3.0045 | 0.2411 |
| FG old-answer decrease ↑ (nats) | 0.0000 | 1.5342 | 0.1331 |
| Exact-sequence hit ↓ (%) | 24.00 | 2.00 | 20.33 |
U2E consistently suppresses the original answer across rewrite and paraphrase prompts, with improvements over ZeroUnlearn supported by paired 95% confidence intervals ([2.3219, 3.2013] and [1.1261, 1.6717] nats, respectively). The reduction in exact-sequence recovery provides complementary evidence. These findings establish genuine old-answer weakening independently of the neutral-target contrast.
Knowledge preservation (W2). Your observation about retained knowledge and FS motivates a closer examination of the trade-off between targeted suppression and preservation. We report these effects together with general capability performance.
Table 5. Target forgetting and knowledge preservation (submitted 3B study).
| Metric | Base | U2E | Change |
|---|---|---|---|
| Target old-answer access ↓ (%) | 19.25 | 4.65 | −14.60 |
| Retained knowledge ↑ (%) | 20.68 | 18.24 | −2.44 |
| FS ↑ (%) | 19.60 | 15.80 | −3.80 |
| General utility macro | 62.084 | 62.087 | +0.003 |
| Per-task utility change (points) | — | — | −0.42 to +0.16 |
The results show substantial targeted suppression with measurable changes in retained knowledge and FS. General capability performance remains broadly stable across the evaluated tasks. We have refined the preservation claims to reflect the observed forgetting–preservation trade-off.
Generalization beyond editing prompts (W2). We have clarified the evaluation protocol to distinguish construction-view effectiveness (FE) from transfer measured through FG and nine unseen access families, including black-box extraction attacks.
Table 6. Held-out factual access and extraction robustness.
| Metric | U2E | ZeroUnlearn |
|---|---|---|
| FG ↓ (%) | 6.30 | 18.10 |
| Worst-route residual ↓ | 0.654 | 0.724 |
| Extraction@20 ↓ | 0.730 | 0.840 |
| Robust facts at ↑ | 0.473 | 0.363 |
U2E consistently improves held-out suppression and extraction resistance over ZeroUnlearn, showing that its effect extends beyond the construction prompts. Together, these experiments provide direct evidence of old-answer weakening, quantify the preservation trade-off, and establish improved robustness across unseen factual accesses.
3. Where does U2E's computational cost come from? (W3)
Thank you for highlighting the gap between theoretical complexity and measured runtime. U2E uses localized model editing to achieve targeted unlearning with explicit knowledge protection, avoiding full-model retraining.
Table 7. Same-GPU warm-start profile for 50 facts.
| Metric | U2E | ZeroUnlearn | RMU | Mechanistic |
|---|---|---|---|---|
| Total time (s) ↓ | 191.54 | 15.70 | 9.96 | 171.80 |
| Time per successful fact (s) ↓ | 4.256 | 0.314 | 0.199 | 3.436 |
| Peak GPU memory (GiB) ↓ | 43.45 | 17.41 | 42.95 | 17.54 |
| Successful requests / 50 ↑ | 45 | 50 | 50 | 50 |
Additional profiling attributes 25.29 seconds to access construction and 164.15 seconds to the combined solve/training phase, including full-model computations. The low-dimensional dual formulation controls the cost of constrained optimization, while the remaining computation supports multi-access coordination, preservation checks, and iterative editing.
The additional computation enables U2E to coordinate multiple factual accesses while preserving unrelated knowledge, yielding stronger unseen-access robustness than ZeroUnlearn. We have clarified the distinction between low-dimensional optimization complexity and end-to-end editing cost in Section 5.4.
4. How broadly does U2E generalize across models and request scales? (W4)
Thank you for suggesting broader evaluations, particularly the scaling study with increasing unlearning requests. We examined U2E across model sizes, request scales, and additional datasets and architectures.
We evaluated U2E on Llama-3.2-3B and Llama-3.1-8B, using ten seeds per model.
Table 8. Results across model sizes (ten seeds per model).
| Model | Method | FE ↓ (%) | FG ↓ (%) | |ΔPPL| ↓ |
|---|---|---|---|---|
| Llama-3.2-3B | U2E | 3.00 | 6.30 | 0.0395 |
| ZeroUnlearn | 17.60 | 18.10 | 0.9964 | |
| Llama-3.1-8B | U2E | 7.60 | 13.70 | 0.0546 |
| ZeroUnlearn | 23.20 | 22.40 | 0.8944 |
U2E achieves stronger factual suppression with smaller PPL changes on both models, indicating that its effectiveness extends to the larger 8B setting.
Following your suggestion, we conducted additional experiments with 10, 25, and 50 requests across three paired seeds. Each run starts independently from the same base model, using nested fact sets and a fixed evaluation protocol.
Table 9. Request-scale evaluation (three paired seeds per size).
| Requests | Method | FG ↓ (%) | |ΔPPL| ↓ | Coverage ↑ | Time (s) ↓ | Memory (GiB) ↓ |
|---|---|---|---|---|---|---|
| 10 | U2E | 1.67 | 0.0057 | 1.000 | 13.87 | 27.79 |
| ZeroUnlearn | 15.00 | 0.4587 | 1.000 | 24.05 | 18.61 | |
| 25 | U2E | 6.00 | 0.0145 | 0.973 | 55.82 | 35.80 |
| ZeroUnlearn | 14.00 | 0.8839 | 1.000 | 43.81 | 18.62 | |
| 50 | U2E | 8.33 | 0.0400 | 0.987 | 129.08 | 45.25 |
| ZeroUnlearn | 19.67 | 0.4146 | 1.000 | 80.17 | 18.63 |
The scaling results show that U2E maintains effective factual suppression, low model drift, and high edit acceptance as the number of requests increases. This supports the scalability of protected multi-access editing within the evaluated request range.
We also evaluated Llama-3.2-3B on ZsRE and DeepSeek-R1-Distill-Qwen-1.5B on MCF. Both settings show low PPL drift under U2E, while forgetting effectiveness and edit coverage vary across factual distributions and architectures. These experiments further characterize the behavior of U2E beyond the primary setting.
We appreciate your suggestion to broaden the benchmark coverage. We have extended the evaluation to ZsRE and an additional model architecture, alongside the new request-scaling study. These experiments provide further evidence on the effectiveness, preservation behavior, and computational characteristics of U2E across the evaluated settings.
5. How is reproducibility supported? (W5)
Thank you for emphasizing reproducibility. We have expanded the appendix with detailed implementation settings, experimental configurations, evaluation protocols, and baseline specifications to facilitate reproduction. The source code will be publicly released upon acceptance.
6. Clarifications and presentation corrections (M1–M5)
Thank you for these helpful suggestions. We have made the following revisions:
- M1 — Abstract wording: Revised the sentence to clearly distinguish covariance-weighted preservation costs from null-space protection.
- M2 — Behavioral factual unlearning: Clarified the task as suppressing specified factual associations across external prompts and internal readouts, with explicit preservation constraints.
- M3 — Appendix E: Organized the implementation settings, per-request procedure, and acceptance criteria into a coherent description.
- M4 — Algorithm 1: Explicitly removed accepted requests from the active set and clarified the termination argument in Appendix D.
- M5 — Residual metric: Defined worst-access residual and its aggregation across facts and access families in the main text.
Response to Reviewer nxFh
We sincerely appreciate your thorough and constructive review. Your suggestions have helped us strengthen the baseline comparisons, clarify the forgetting–preservation trade-off, improve the computational analysis, and expand the experimental evaluation. We hope the revised manuscript and additional evidence satisfactorily address your concerns, and we thank you again for your valuable feedback.
1. Are the baseline comparisons fairly configured and calibrated? (W1, Q2)Thank you for pointing out the baseline configuration issues and suggesting stronger editing comparisons. Following your suggestions, we calibrated the baselines, added controlled multi-access comparisons, and examined SimNPO's unusual retention results.
We tested 0.5×/1×/2× strengths on two development seeds, selected each method's closest FS/RE/RG/RS preservation match to U2E, and froze the settings before six paired confirmation seeds. The selected strengths were 0.5× (ZeroUnlearn), 1× (AlphaEdit), 2× (SimNPO), and 1× (Mechanistic). The corresponding matching distances were 1.125, 0.575, 41.400, and 36.450, respectively.
Table 1. Baseline comparison (six paired seeds).
Your concern about ZeroUnlearn being a weak comparator is particularly relevant. Against AlphaEdit, which achieves lower FE/FG, U2E still reduces worst-route residual from 0.738 to 0.654 and Extraction@20 from 0.863 to 0.730, with smaller PPL drift (0.0415 versus 0.0535). This provides stronger evidence for U2E's advantage on unseen factual accesses.
Following your suggestion, we also include MEMIT-style editors and a controlled joint-editing baseline.
Table 2. Additional editing comparisons. P1 and P2 use different evaluation panels; comparisons are made within each protocol.
Protected-Joint-LS uses identical accesses, protection constraints, trust radii, and acceptance checks, changing only the joint solver. U2E achieves higher accepted coverage (97.0% versus 3.67%) and lower Extraction@20 (0.730 versus 0.853), supporting the contribution of protected common descent beyond simple joint fitting.
Regarding SimNPO (Q2), thank you for highlighting its unusually high retain accuracy. SimNPO optimizes an auxiliary retain-QA objective, which helps explain this behavior. However, its FE/FG of 42.33/47.67 and PPL change of 2.7186 show that higher retain accuracy does not necessarily indicate better knowledge-preserving unlearning. Its RE/RG/RS reach 61.42/54.10/54.87 in the six-seed confirmation.
Table 3. SimNPO configuration.
We have made the baseline settings and relevant editing comparisons more explicit in the revision, including the original GA, NPO, and UNDIAL configurations. The additional AlphaEdit and controlled joint-editing results support U2E's advantage in unseen-access robustness beyond the original ZeroUnlearn comparison.