acceptodds
Under review as a conference paper at ICLR 2027

U2E: Protected Common Descent for Targeted Behavioral Factual Unlearning

Abstract

Targeted factual unlearning seeks to suppress a small set of obsolete or sensitive associations in a deployed language model while preserving its remaining knowledge and capabilities. Knowledge editing offers a local intervention, but its formulation around a replacement value does not determine whether one protected update can suppress the old fact across several prompts and internal representations whose required directions may conflict. Building on the key–value view of transformer memory and covariance and null-space protection, we formulate factual unlearning as protected common descent. This formulation characterizes whether multiple access-specific decreases admit a shared protected update and, when feasible, the minimum cost of satisfying the local constraints. The associated quadratic problem has a unique solution with an active-support rank bound. Based on this formulation, U2E constructs low-rank candidate updates using local progress estimates and validates them on the full nonlinear model before commitment. Experiments demonstrate favorable tradeoffs between old-answer suppression and measured retention under the evaluated protocols, supporting protected common descent as a practical framework for selective factual unlearning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model unlearning, model editing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nxFhOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper recasts targeted factual unlearning as a joint constraint problem: each prompt or internal readout of an old fact is treated as an “access” whose old-answer score responds linearly to a weight update, and the update is confined to a null-space-protected subspace under a covariance-weighted norm. Within this local model, the authors use a Farkas-type certificate to characterize whether all required decreases can be achieved by one shared update, and a dual formulation to obtain the unique minimum-cost update, whose rank is bounded by the number of active constraints. They turn this into U2E, a pipeline that constructs multiple external and internal accesses, selects layers by transfer to a held-out prompt view, applies a trust-region step, validates candidates on the full nonlinear model with relinearization, and abstains on requests it cannot satisfy safely. On MultiCounterFact with Llama-3.2-3B and Llama-3.1-8B, U2E reports lower old-answer accuracy than ZeroUnlearn together with much smaller perplexity drift, along with improved though still limited robustness on unseen access routes.

Critical strengths
  1. Clean, well-motivated framing. The paper treats factual unlearning as joint suppression across several accesses under shared knowledge protection. This is a natural and compelling way to pose the problem, and it clarifies why single-query edits can leave the old fact recoverable through other routes. Making the shared-parameter constraint explicit, with feasibility as a prerequisite, is a useful conceptual contribution.
  2. Rigorous statistical protocol. The evaluation uses paired seed bootstraps, exact sign-flip tests, and Holm correction, with seeds treated as the replication unit. The authors are candid about limitations: they report a cross-model audit (Appendix L) in which U2E underperforms ZeroUnlearn on average suppression, and they describe the hyperparameter sensitivity study as descriptive rather than confirmatory.
  3. Well-designed, carefully executed experiments. The experiments are chosen to probe both the strengths and the weaknesses of the method. They include held-out access families, black-box extraction, component ablations, scaling to a second model size, and a preservation-calibrated comparison. Behavioral metrics count abstained requests in the denominator, which keeps the reported results honest.
Critical weaknesses
  1. Baselines appear untuned, which weakens the main comparison.

    • The primary comparator, ZeroUnlearn, barely changes the model (FE 17.6 vs. 18.2 for Base), so outperforming it is a low bar.
    • Several loss-based baselines show signs of misconfiguration rather than a fair comparison. GA, NPO, and UNDIAL collapse retention to roughly 0-5%. NPO behaves completely differently across scales (FE 0.8 on 3B vs. 22.8 on 8B). SimNPO raises accuracy on retain data far above Base (RE 59.8 vs. 21.7), which suggests a degenerate run. Using released-implementation defaults is not equivalent to a tuned comparison.
    • The most relevant baselines are editing methods that handle multiple prompts or targets (AlphaEdit, MEMIT-style editors, the refusal and empty-set editors of Li et al.). AlphaEdit appears only in an appendix, and the preservation-calibrated comparison (App. K) omits GA, NPO, RMU, and UNDIAL.
  2. Retention and generalization claims are stronger than the evidence supports.

    • Retained knowledge drops from 20.68 to 18.24 (about 12% relative) and FS from 19.6 to 15.8, so “near-base” retention is generous.
    • The utility-panel result (62.087 vs. 62.084) is reported to three decimals on 3,500-example panels, where sampling noise is on the order of ±0.8 points. This shows the absence of large degradation, not preservation to within 0.003.
    • The original-query view is part of the construction set, so FE is measured in-sample. Only FG and the nine held-out families in the robustness study test transfer.
    • Transfer to unseen routes is modest: 73% of facts remain recoverable under Extraction@20, worst-route residual improves only from .724 to .654, and only 47.3% vs. 36.3% of facts are robust at τ = 0.75.
  3. The cost analysis does not match the reported profile. Sec. 5.4 argues that U2E’s optimization scale stays far below that of optimization-based unlearning, but Table 4 shows 191.5 s for U2E vs. 15.7 s (ZeroUnlearn) and 10.0 s (RMU), with 43 GiB vs. 17 GiB of GPU memory and 45/50 vs. 50/50 successes. An m × m dual solve should not take 164 s, so the cost evidently comes from repeated full-model passes, which the complexity discussion does not reflect.

  4. Evaluation is limited to a single small-scale setting. TOFU, RWKU, and WMDP are cited but never used, and the paper evaluates only 50 forget facts on one dataset with no study of how the method scales with the number of requests. A side-by-side comparison with well-established methods on a standard unlearning benchmark such as RWKU, TOFU, or WMDP is needed to support the claim that U2E is a practical framework for selective factual unlearning.

  5. The code is not provided, so this weakens reproducibility.

Minor:

  1. I would rephrase this line in the abstract “and covariance and null-space protection”, it does not sound very good.

  2. line 121: behavioral factual unlearning, there is some ambiguity in this phrase, can the authors elaborate more?

  3. Appendix E contains only a Table so it ends up rendering weirdly, making it seem like it is empty. I would fix that for reading clarity purposes.

  4. Algorithm 1: after acceptance, S is never updated, so the loop as written doesn’t terminate.

  5. The “residual” metric is not defined in the main text.

Questions for authors
  1. Can you show the old fact is weakened (e.g., probing, or suppression with the neutral target excluded from the contrast), not just the output contrast?

  2. Why does SimNPO raise accuracy above Base, and how were baselines configured?

5 comments
AuthorsOpenReview ↗

Response to Reviewer nxFh

We sincerely appreciate your thorough and constructive review. Your suggestions have helped us strengthen the baseline comparisons, clarify the forgetting–preservation trade-off, improve the computational analysis, and expand the experimental evaluation. We hope the revised manuscript and additional evidence satisfactorily address your concerns, and we thank you again for your valuable feedback.


1. Are the baseline comparisons fairly configured and calibrated? (W1, Q2)

Thank you for pointing out the baseline configuration issues and suggesting stronger editing comparisons. Following your suggestions, we calibrated the baselines, added controlled multi-access comparisons, and examined SimNPO's unusual retention results.

We tested 0.5×/1×/2× strengths on two development seeds, selected each method's closest FS/RE/RG/RS preservation match to U2E, and froze the settings before six paired confirmation seeds. The selected strengths were 0.5× (ZeroUnlearn), 1× (AlphaEdit), 2× (SimNPO), and 1× (Mechanistic). The corresponding matching distances were 1.125, 0.575, 41.400, and 36.450, respectively.

Table 1. Baseline comparison (six paired seeds).

MethodFE ↓FG ↓Worst residual ↓Extraction@20 ↓∣ΔPPL∣\lvert\Delta\mathrm{PPL}\rvert ↓Coverage ↑
U2E2.008.330.6540.7300.04150.970
ZeroUnlearn20.3319.670.7240.8400.90791.000
AlphaEdit0.006.330.7380.8630.05351.000
SimNPO42.3347.670.8680.8132.71861.000
Mechanistic0.0034.170.7260.8174.23711.000

Your concern about ZeroUnlearn being a weak comparator is particularly relevant. Against AlphaEdit, which achieves lower FE/FG, U2E still reduces worst-route residual from 0.738 to 0.654 and Extraction@20 from 0.863 to 0.730, with smaller PPL drift (0.0415 versus 0.0535). This provides stronger evidence for U2E's advantage on unseen factual accesses.

Following your suggestion, we also include MEMIT-style editors and a controlled joint-editing baseline.

Table 2. Additional editing comparisons. P1 and P2 use different evaluation panels; comparisons are made within each protocol.

ProtocolMethodFE ↓FG ↓Extraction@20 ↓Accepted coverage ↑
P1, 10 seedsMEMIT0.407.40—1.000
P1, 10 seedsQuery-Merged MEMIT18.2020.30—1.000
P2, 6 seedsU2E2.008.330.7300.970
P2, 6 seedsProtected-Joint-LS21.0019.500.8530.0367

Protected-Joint-LS uses identical accesses, protection constraints, trust radii, and acceptance checks, changing only the joint solver. U2E achieves higher accepted coverage (97.0% versus 3.67%) and lower Extraction@20 (0.730 versus 0.853), supporting the contribution of protected common descent beyond simple joint fitting.

Regarding SimNPO (Q2), thank you for highlighting its unusually high retain accuracy. SimNPO optimizes an auxiliary retain-QA objective, which helps explain this behavior. However, its FE/FG of 42.33/47.67 and PPL change of 2.7186 show that higher retain accuracy does not necessarily indicate better knowledge-preserving unlearning. Its RE/RG/RS reach 61.42/54.10/54.87 in the six-seed confirmation.

Table 3. SimNPO configuration.

ParameterOriginalConfirmation
Learning rate1e-51e-5
Batch size / accumulation8 / 48 / 4
Epochs / optimizer steps10 / 2010 / 20
Maximum length128128
Beta / Delta4.5 / 04.5 / 0
Forget weight0.1250.250
Retain weight1.01.0
Retain QA examples256256

We have made the baseline settings and relevant editing comparisons more explicit in the revision, including the original GA, NPO, and UNDIAL configurations. The additional AlphaEdit and controlled joint-editing results support U2E's advantage in unseen-access robustness beyond the original ZeroUnlearn comparison.

2. Does U2E genuinely weaken old facts while preserving unrelated knowledge? (W2, Q1)

Thank you for your careful comments on factual suppression, knowledge preservation, and generalization. Following your suggestion, we conducted additional experiments to directly examine old-answer suppression and clarified the preservation and transfer results.

Genuine old-answer suppression (Q1). We measured the teacher-forced sequence log-probability of the original answer without involving the neutral target. This directly evaluates the model's remaining access to the original fact.

Table 4. Neutral-free old-answer suppression (six paired seeds, 300 facts).

MetricBaseU2EZeroUnlearn
Rewrite old-answer decrease ↑ (nats)0.00003.00450.2411
FG old-answer decrease ↑ (nats)0.00001.53420.1331
Exact-sequence hit ↓ (%)24.002.0020.33

U2E consistently suppresses the original answer across rewrite and paraphrase prompts, with improvements over ZeroUnlearn supported by paired 95% confidence intervals ([2.3219, 3.2013] and [1.1261, 1.6717] nats, respectively). The reduction in exact-sequence recovery provides complementary evidence. These findings establish genuine old-answer weakening independently of the neutral-target contrast.

Knowledge preservation (W2). Your observation about retained knowledge and FS motivates a closer examination of the trade-off between targeted suppression and preservation. We report these effects together with general capability performance.

Table 5. Target forgetting and knowledge preservation (submitted 3B study).

MetricBaseU2EChange
Target old-answer access ↓ (%)19.254.65−14.60
Retained knowledge ↑ (%)20.6818.24−2.44
FS ↑ (%)19.6015.80−3.80
General utility macro62.08462.087+0.003
Per-task utility change (points)——−0.42 to +0.16

The results show substantial targeted suppression with measurable changes in retained knowledge and FS. General capability performance remains broadly stable across the evaluated tasks. We have refined the preservation claims to reflect the observed forgetting–preservation trade-off.

Generalization beyond editing prompts (W2). We have clarified the evaluation protocol to distinguish construction-view effectiveness (FE) from transfer measured through FG and nine unseen access families, including black-box extraction attacks.

Table 6. Held-out factual access and extraction robustness.

MetricU2EZeroUnlearn
FG ↓ (%)6.3018.10
Worst-route residual ↓0.6540.724
Extraction@20 ↓0.7300.840
Robust facts at τ=0.75\tau=0.75 ↑0.4730.363

U2E consistently improves held-out suppression and extraction resistance over ZeroUnlearn, showing that its effect extends beyond the construction prompts. Together, these experiments provide direct evidence of old-answer weakening, quantify the preservation trade-off, and establish improved robustness across unseen factual accesses.


3. Where does U2E's computational cost come from? (W3)

Thank you for highlighting the gap between theoretical complexity and measured runtime. U2E uses localized model editing to achieve targeted unlearning with explicit knowledge protection, avoiding full-model retraining.

Table 7. Same-GPU warm-start profile for 50 facts.

MetricU2EZeroUnlearnRMUMechanistic
Total time (s) ↓191.5415.709.96171.80
Time per successful fact (s) ↓4.2560.3140.1993.436
Peak GPU memory (GiB) ↓43.4517.4142.9517.54
Successful requests / 50 ↑45505050

Additional profiling attributes 25.29 seconds to access construction and 164.15 seconds to the combined solve/training phase, including full-model computations. The low-dimensional dual formulation controls the cost of constrained optimization, while the remaining computation supports multi-access coordination, preservation checks, and iterative editing.

The additional computation enables U2E to coordinate multiple factual accesses while preserving unrelated knowledge, yielding stronger unseen-access robustness than ZeroUnlearn. We have clarified the distinction between low-dimensional optimization complexity and end-to-end editing cost in Section 5.4.

4. How broadly does U2E generalize across models and request scales? (W4)

Thank you for suggesting broader evaluations, particularly the scaling study with increasing unlearning requests. We examined U2E across model sizes, request scales, and additional datasets and architectures.

We evaluated U2E on Llama-3.2-3B and Llama-3.1-8B, using ten seeds per model.

Table 8. Results across model sizes (ten seeds per model).

ModelMethodFE ↓ (%)FG ↓ (%)|ΔPPL| ↓
Llama-3.2-3BU2E3.006.300.0395
ZeroUnlearn17.6018.100.9964
Llama-3.1-8BU2E7.6013.700.0546
ZeroUnlearn23.2022.400.8944

U2E achieves stronger factual suppression with smaller PPL changes on both models, indicating that its effectiveness extends to the larger 8B setting.

Following your suggestion, we conducted additional experiments with 10, 25, and 50 requests across three paired seeds. Each run starts independently from the same base model, using nested fact sets and a fixed evaluation protocol.

Table 9. Request-scale evaluation (three paired seeds per size).

RequestsMethodFG ↓ (%)|ΔPPL| ↓Coverage ↑Time (s) ↓Memory (GiB) ↓
10U2E1.670.00571.00013.8727.79
ZeroUnlearn15.000.45871.00024.0518.61
25U2E6.000.01450.97355.8235.80
ZeroUnlearn14.000.88391.00043.8118.62
50U2E8.330.04000.987129.0845.25
ZeroUnlearn19.670.41461.00080.1718.63

The scaling results show that U2E maintains effective factual suppression, low model drift, and high edit acceptance as the number of requests increases. This supports the scalability of protected multi-access editing within the evaluated request range.

We also evaluated Llama-3.2-3B on ZsRE and DeepSeek-R1-Distill-Qwen-1.5B on MCF. Both settings show low PPL drift under U2E, while forgetting effectiveness and edit coverage vary across factual distributions and architectures. These experiments further characterize the behavior of U2E beyond the primary setting.

We appreciate your suggestion to broaden the benchmark coverage. We have extended the evaluation to ZsRE and an additional model architecture, alongside the new request-scaling study. These experiments provide further evidence on the effectiveness, preservation behavior, and computational characteristics of U2E across the evaluated settings.


5. How is reproducibility supported? (W5)

Thank you for emphasizing reproducibility. We have expanded the appendix with detailed implementation settings, experimental configurations, evaluation protocols, and baseline specifications to facilitate reproduction. The source code will be publicly released upon acceptance.


6. Clarifications and presentation corrections (M1–M5)

Thank you for these helpful suggestions. We have made the following revisions:

  • M1 — Abstract wording: Revised the sentence to clearly distinguish covariance-weighted preservation costs from null-space protection.
  • M2 — Behavioral factual unlearning: Clarified the task as suppressing specified factual associations across external prompts and internal readouts, with explicit preservation constraints.
  • M3 — Appendix E: Organized the implementation settings, per-request procedure, and acceptance criteria into a coherent description.
  • M4 — Algorithm 1: Explicitly removed accepted requests from the active set and clarified the termination argument in Appendix D.
  • M5 — Residual metric: Defined worst-access residual and its aggregation across facts and access families in the main text.
Reviewer_TjWeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes U2E, a framework for targeted factual unlearning that jointly suppresses multiple external and internal accesses to an old fact while preserving unrelated knowledge. The method formulates unlearning as a protected common-descent problem, derives feasibility conditions and minimum-cost low-rank updates, and uses full-model verification for adaptive refinement. Experiments on Llama models demonstrate improved forgetting–retention tradeoffs compared with existing baselines.

Critical strengths

Principled formulation. The paper provides an interesting perspective on factual unlearning by explicitly considering the compatibility of multiple suppression requirements under knowledge-preservation constraints.

U2E achieves stronger suppression than several baselines while largely preserving retained knowledge and general model utility. The evaluation also includes unseen access routes, extraction attacks, and component ablations.

Critical weaknesses

There is insufficient justification of the optimization procedure. Equation (10) determines feasible progress under trust-region constraints, but the final update is obtained through Equation (8), which does not explicitly retain these constraints. It is unclear whether the resulting update is guaranteed to remain within the trusted region. This gap affects the theoretical justification of the proposed algorithm.

Multi-prompt editing and null-space knowledge protection have been explored in prior work, including CAE and AlphaEdit. The current ablations do not sufficiently isolate the benefit of the proposed feasibility-aware optimization over simpler joint editing approaches. Furthermore, 73% of targeted facts remain recoverable under Extraction@20, and the cross-model results in Appendix L show substantially weaker suppression.

Questions for authors

Does the final minimum-cost update necessarily satisfy the trust-region constraints in Equation (10)? If not, how is this handled in practice?

Can the authors isolate the contribution of common-descent optimization by comparing against simpler joint multi-access editing with identical access sets and preservation constraints?

How do the authors explain the high remaining extraction success and the weaker results on the Qwen-based model?

6 comments
AuthorsOpenReview ↗

Response to Reviewer TjWe

We sincerely thank you for identifying the key technical issues and suggesting a more direct evaluation of our contribution. We have addressed the trust-region concern through an explicit finite constrained formulation and post-composition verification, and added a controlled joint-editing comparison under identical access and preservation conditions. We hope these revisions resolve your principal concerns and would greatly appreciate your reconsideration of the initial rating.


1. Does the final minimum-cost update remain inside the trust region? (W1, Q1)

Thank you for pointing out this gap in our explanation. Eq. (10) finds the largest attainable progress t∗t^* under the trust-region constraints, while Eq. (8) characterizes the minimum-cost solution in the local protected geometry without explicitly imposing these constraints. As you correctly noted, the solution of Eq. (8) is not necessarily within the trust region. The minimum-cost guarantee therefore applies to the local optimization problem, whereas the trust-region guarantee applies to the final accepted update.

In practice, U2E enforces this distinction through full-model verification. After composing the candidate low-rank update, we check its value movement at every constrained layer and access, together with the forgetting and preservation requirements. The update is committed only if all checks pass; otherwise, it is refined or rejected. Thus, every committed update satisfies the prescribed trust-region constraints, without assuming that the radius-free minimum-cost solution is itself feasible.

Table 1. Trust-region verification after composing the update. A violation occurs when the maximum radius ratio exceeds 1+10−41+10^{-4}.

PanelSeedsRequestsAcceptedNo-opAbstainedMaximum ratioViolations
Submitted P11050044720330.98000
Independent P26300291630.99650

Across 738 accepted edits, we observe no trust-region violations, with maximum radius ratios of 0.9800 and 0.9965, both below one. We have revised Algorithm 1 to make the post-composition verification explicit and clarified how the local minimum-cost solution is combined with finite-step validation to ensure that every committed update respects the prescribed trust-region and preservation constraints.


2. Why does protected common descent add more than multi-prompt editing? (W2, Q2)

Thank you for suggesting a more direct comparison. As you noted, CAE and AlphaEdit have already explored multi-prompt editing and knowledge protection. Our contribution is to address conflicts between different factual accesses under preservation constraints. When these accesses favor different update directions, combining their objectives may improve some while leaving others unchanged or even strengthened. U2E instead determines whether a single protected update can reduce all specified accesses and finds a minimum-cost solution when such an update exists.

Following your suggestion, we added Protected-Joint-LS, which uses the same access directions, protected keys, trust-region limits, relinearization budget, acceptance criteria, facts, and evaluator as U2E. The only difference is the optimization: Protected-Joint-LS fits the access responses jointly using protected least squares, whereas U2E explicitly enforces a decrease for each access.

Table 2. Comparison with a simpler joint editor (six paired seeds).

MetricU2EProtected-Joint-LS
FE ↓ (%)2.0021.00
FG ↓ (%)8.3319.50
Blind residual ↓0.11770.1425
Extraction@20 ↓0.7300.853
Accepted coverage ↑0.9700.0367
Absolute PPL change ↓0.04150.0064

Under identical conditions, U2E accepts 97.0% of requested edits, compared with 3.67% for Protected-Joint-LS, and achieves lower FE/FG, blind residual, and Extraction@20. The smaller PPL change of Protected-Joint-LS reflects its much lower acceptance rate (1.83 versus 48.5 edits per 50 requests). These results show that explicitly coordinating the decrease of each access is more effective than minimizing their combined fitting error, even when both methods use the same access information and preservation constraints.

3. How should Extraction@20 and the Qwen results be interpreted? (W2, Q3)

Thank you for raising these concerns. Both results highlight why we evaluate forgetting together with its effects on the rest of the model.

Regarding Extraction@20, the reported 73% is the fraction of facts recovered within 20 attack attempts, not the success rate of a single prompt. This is a challenging setting because an attacker can try different ways to retrieve the same fact. Nevertheless, U2E reduces extraction success from 84% to 73%, lowers worst-route residual from 0.724 to 0.654, and increases the fraction of facts that meet the robustness criterion across all nine unseen routes from 36.3% to 47.3%. We also measured the old answer's probability directly, without using the neutral target: U2E reduces its log-probability by 3.0045 nats on rewrite prompts and 1.5342 nats on paraphrases. These results show that the improvement extends beyond the prompts and contrastive scores used during editing.

Table 3. Robustness on unseen accesses, Llama-3.2-3B (six paired seeds).

MetricU2EZeroUnlearn
Worst-route residual ↓0.6540.724
Extraction@20 ↓0.7300.840
Robust facts at τ=0.75\tau=0.75 ↑0.4730.363
Absolute PPL change ↓0.0410.908

Regarding Qwen, ZeroUnlearn achieves stronger forgetting, but this comes with much larger changes to the model. Its absolute PPL change reaches 102.2240, compared with 0.0615 for U2E, and its worst-path amplification is 0.7669 versus 0.0303. The PPL difference holds across all six paired seeds (95% CI [31.4588, 218.9605]).

Table 4. Forgetting and preservation on DeepSeek-R1-Distill-Qwen-1.5B (six paired seeds).

MetricU2EZeroUnlearn
Route-balanced log-survival change ↓-0.0226-3.4983
Extraction@20 ↓0.49670.1700
Worst-path amplification ↓0.03030.7669
Absolute PPL change ↓0.0615102.2240

The Qwen results illustrate an important risk of aggressive forgetting. A method can strongly suppress the targeted fact while also changing the model's behavior on unrelated inputs. Here, ZeroUnlearn's stronger suppression is accompanied by a very large PPL increase and substantially greater worst-path amplification. This is why U2E does not pursue forgetting strength at any cost. Instead, it seeks updates that weaken multiple factual accesses while keeping changes to other model behavior under explicit control.

AuthorsOpenReview ↗

Follow-up on Rebuttal and Invitation for Further Discussion

Dear Reviewer TjWe,

As the ICLR rebuttal discussion period is approaching its end, we would like to kindly follow up on our responses. We sincerely appreciate your valuable suggestions and would be grateful for the opportunity to hear your thoughts and engage in further discussion if you have any remaining concerns.

Thank you very much for your time and consideration.

Reviewer_M3KaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes U2E for targeted behavioral factual unlearning. It constructs a minimum-cost parameter edit that jointly decreases multiple output and internal old-answer scores under knowledge-preservation constraints, followed by full-model validation and adaptive refinement.

Critical strengths
  • The paper addresses the important problem of suppressing multiple access paths to a fact while preserving unrelated knowledge.
  • The local constrained formulation is mathematically clear, and the experiments include extraction attacks and preservation measurements.
Critical weaknesses
  • Connection between formulation and unlearning. The theoretical results characterize a local approximation, but decreasing selected old-answer scores relative to a neutral answer does not necessarily remove the information.
  • Evaluation primarily relies on MultiCounterFact, and main results rely on LLama models. The Qwen-based evaluation shows substantially weaker average suppression than ZeroUnlearn.
  • The full procedure involves layer selection, access construction, validation thresholds, relinearization, and request removal, but several steps are insufficiently specified. Appendix E 'implementation details' is empty although being referenced multiple times in the paper.
  • Novelty and presentation. The method combines established covariance-weighted editing, null-space protection, and classical constrained optimization. The contribution beyond this combination needs clearer positioning. The presentation repeatedly emphasizes the abstract formulation while leaving the practical algorithm and the scope of its guarantees difficult to follow.
Questions for authors

Equation (10) selects attainable progress under trust-region constraints, but the subsequent minimum-cost solve appears to omit those constraints. What ensures that the resulting update remains within the trust region?

7 comments
AuthorsOpenReview ↗

Response to Reviewer M3Ka

We sincerely appreciate your constructive feedback, which has helped us substantially improve the theoretical motivation, empirical validation, and clarity of the manuscript. We hope that the additional evidence on direct old-answer weakening, unseen-access robustness, and controlled joint editing, together with the clarified trust-region formulation and implementation details, adequately addresses your main concerns. We would be grateful if you could reconsider your initial assessment in light of these revisions.

1. How does protected common descent lead to factual unlearning? (W1)

Thank you for raising this important question. Our formulation is motivated by how factual knowledge is accessed in language models: the same fact can be retrieved through different prompts and internal representations, but these accesses may respond differently to the same parameter update. Suppressing one access may therefore leave another unchanged or even strengthen it. This is why we formulate factual unlearning as protected common descent: we seek a single update that decreases multiple access-specific old-answer scores while preserving unrelated knowledge, rather than replacing the answer to one prompt.

Theoretical connection to factual unlearning. For each access ii, the local response to a parameter update is ⟨ΔW,ai⟩C\langle\Delta W,a_i\rangle_C. Our formulation requires this response to be negative across all specified accesses. Theorem 1 characterizes when such a shared update exists under knowledge-preservation constraints and identifies its minimum-cost realization. More importantly, Appendix C connects these constructed accesses to unseen retrieval routes. If an unseen access direction satisfies anew=∑iwiai+ea_{\mathrm{new}}=\sum_i w_i a_i+e, with wi≥0w_i\geq0, then

\leq -t\sum_i w_i b_i+\|\Delta W\|_C\|e\|_C.$$ This bound shows that suppression can transfer to an unseen access when its response is sufficiently covered by the constructed directions. It also explains why coordinating multiple access directions matters: the shared update can weaken retrieval routes beyond those explicitly optimized. We account for local approximation errors through full-model verification and adaptive refinement. Does the old answer itself become weaker? To distinguish factual weakening from a mere increase in the neutral answer's probability, we additionally measure the absolute teacher-forced sequence log-probability of the old answer, without using the neutral target in the evaluation. We use six paired seeds comprising 300 facts. **Table 1. Neutral-free old-answer weakening (six paired seeds).** Lower old-answer log-probability and exact-sequence hit rate indicate stronger suppression. | Metric | Base | U2E | ZeroUnlearn | AlphaEdit | U2E−Zero advantage | |---|---:|---:|---:|---:|---:| | Rewrite `log p(old)` ↓ | -4.6731 | -7.6775 | -4.9142 | -9.3407 | 2.7633 | | Rewrite old-answer decrease ↑ | 0.0000 | 3.0045 | 0.2411 | 4.6676 | 2.7633 | | FG `log p(old)` ↓ | -4.9099 | -6.4440 | -5.0429 | -6.4766 | 1.4011 | | FG old-answer decrease ↑ | 0.0000 | 1.5342 | 0.1331 | 1.5667 | 1.4011 | | `Δ log p(neutral)` | 0.0000 | 0.6319 | -0.0296 | 21.8215 | 0.6615 | | Exact-sequence hit (%) ↓ | 24.00 | 2.00 | 20.33 | 0.00 | 18.33 | As shown in Table 1, U2E decreases the old answer's log-probability by **3.0045 nats on rewrite prompts and 1.5342 nats on paraphrases**, compared with 0.2411 and 0.1331 for ZeroUnlearn. The paired advantages are 2.7633 nats (95% CI [2.3219, 3.2013]) and 1.4011 nats ([1.1261, 1.6717]), respectively. Meanwhile, the neutral target increases by only 0.6319 nats, and the exact-sequence hit rate drops from 24.00% to 2.00%, compared with 20.33% for ZeroUnlearn. These results show that U2E reduces the probability of the old answer itself, rather than merely changing its preference relative to the neutral target. Does this weakening extend to unseen retrieval routes? We further evaluate nine held-out access families and a 20-attempt black-box extraction attack. These tests examine whether the same fact remains accessible through routes not used to construct the update. **Table 2. Robustness across nine held-out access families (six seeds).** Lower residual, extraction, and PPL change are better; higher coverage and robust-fact rate are better. | Metric | U2E | ZeroUnlearn | AlphaEdit | SimNPO | Mechanistic | |---|---:|---:|---:|---:|---:| | Worst-route residual ↓ | 0.654 | 0.724 | 0.738 | 0.868 | 0.726 | | Extraction@20 ↓ | 0.730 | 0.840 | 0.863 | 0.813 | 0.817 | | Absolute PPL change ↓ | 0.041 | 0.908 | 0.053 | 2.719 | 4.237 | | Coverage ↑ | 0.970 | 1.000 | 1.000 | 1.000 | 1.000 | | Facts robust at `τ=0.75` ↑ | 0.473 | 0.363 | — | — | — |

U2E achieves the lowest worst-route residual (0.654) and Extraction@20 (0.730) among the compared methods, improving over ZeroUnlearn by 0.070 and 0.110, respectively. The fraction of facts meeting the robustness criterion across all nine unseen routes increases from 36.3% to 47.3%. These improvements are achieved with an absolute PPL change of only 0.041, compared with 0.908 for ZeroUnlearn, showing that stronger cross-route suppression can be achieved with limited changes to the model's overall behavior.

Together, these results establish the connection between our formulation and targeted behavioral factual unlearning. Protected common descent addresses the compatibility of multiple factual access paths, while the transfer bound explains how their joint suppression can extend to unseen routes. The neutral-free and extraction evaluations further confirm that the resulting updates weaken access to the old fact itself, rather than merely optimizing an old-versus-neutral contrast.


2. Does U2E generalize beyond MultiCounterFact and Llama? (W2)

We thank you for raising this concern. We evaluate U2E across model scales, architectures, and unseen retrieval routes to examine whether its forgetting–preservation behavior extends beyond the primary setting.

On ten paired MultiCounterFact seeds, U2E consistently outperforms ZeroUnlearn on both Llama-3.2-3B and Llama-3.1-8B, achieving stronger factual suppression with substantially smaller model drift.

Table 1. Results across Llama model scales (ten paired seeds). Lower is better.

| Model | Method | FE ↓ | FG ↓ | |ΔPPL| ↓ | |---|---|---:|---:|---:| | Llama-3.2-3B | U2E | 3.00 | 6.30 | 0.0395 | | Llama-3.2-3B | ZeroUnlearn | 17.60 | 18.10 | 0.9964 | | Llama-3.1-8B | U2E | 7.60 | 13.70 | 0.0546 | | Llama-3.1-8B | ZeroUnlearn | 23.20 | 22.40 | 0.8944 |

U2E improves FE/FG by 14.6/11.8 percentage points on 3B and 15.6/8.7 points on 8B, confirming consistent suppression–preservation advantages across model scales.

We further evaluate DeepSeek-R1-Distill-Qwen-1.5B using six paired seeds with identical fact IDs, model revisions, and evaluation fixtures. This experiment examines whether U2E's protection mechanism continues to control collateral changes under a different architecture.

Table 2. Protected-update behavior on DeepSeek-R1-Distill-Qwen-1.5B (six paired seeds).

MetricU2EZeroUnlearnU2E advantage
Absolute PPL change ↓0.0615102.2240102.1625
Worst-path amplification ↓0.03030.76690.7366
PPL-stable seeds ↑6/60/66/6

ZeroUnlearn achieves stronger average suppression on this checkpoint, but with substantially larger model drift and worst-path amplification. U2E limits the absolute PPL change to 0.0615 and worst-path amplification to 0.0303. The paired PPL advantage is consistent across all six seeds (95% CI [31.4588, 218.9605]). These results show that U2E's protected-update mechanism remains effective at controlling model drift and preventing excessive amplification of alternative access routes under a different architecture.

Beyond the primary prompts, we also evaluate nine unseen access families, including multilingual queries and black-box extraction, to assess cross-route robustness. Together, the two-scale Llama results and Qwen evaluation show how U2E coordinates factual suppression with knowledge preservation across different model settings.


3. Is the practical procedure sufficiently specified? (W3, W4)

Thank you for pointing out the missing implementation details. Appendix E in the submitted PDF contains the configuration table, but its placement made the appendix appear empty. We have corrected the layout and consolidated the implementation details in the revised Appendix E.

To make the procedure easier to follow, we now describe U2E in the order it is executed. For each fact, we first construct three prompt views and fixed internal readouts, then select an editable layer based on transfer to a held-out prompt. Using these accesses, we build covariance-weighted directions with protected keys and solve for a shared update. We then check the candidate on the full model against the forgetting requirements, public KL limits (mean ≤ 0.002, max ≤ 0.01), and neighboring-fact preservation (drop ≤ 0.1). If forgetting is insufficient, we relinearize and refine the update for up to three rounds. Finally, we commit updates that pass all checks and remove processed requests from the active set; otherwise, we reduce the request set or abstain.

We have updated Algorithm 1 accordingly and added the exact prompt templates, layer-selection rules, solver settings, validation thresholds, and stopping criteria to Appendix E. This also makes the distinction between the local theoretical solution and the verified final update explicit.

4. What is new beyond existing editing components? (W4)

We appreciate your concern about the novelty of our approach. While covariance-weighted editing and null-space protection are established techniques, they do not determine whether multiple factual accesses can be suppressed simultaneously under the same preservation constraints. In fact, updates that suppress individual accesses may conflict once restricted to the protected parameter space. Our contribution is to make this joint feasibility problem explicit and use its solution to guide factual unlearning.

Specifically, protected common descent provides (1) a necessary and sufficient condition for the existence of a shared protected descent update, (2) a minimum-cost solution with an active-support rank bound under the local response model, and (3) a practical mechanism that verifies candidate updates on the full model and abstains when the specified requirements cannot be satisfied. The novelty lies in connecting these results into a feasibility-aware editing procedure, rather than introducing new covariance or projection operators.

To isolate this contribution, we added Protected-Joint-LS, a simpler joint multi-access editor using identical access directions, protection geometry, trust radii, refinement budget, acceptance checks, and evaluation protocol. Only the feasibility-aware solver is replaced by a protected stacked least-squares fit.

Table 3. Same-condition comparison of joint editing strategies (six paired seeds).

MetricU2EProtected-Joint-LS
FE ↓ (%)2.0021.00
FG ↓ (%)8.3319.50
Blind residual ↓0.11770.1425
Extraction@20 ↓0.7300.853
Accepted coverage ↑0.9700.0367
Absolute PPL change ↓0.04150.0064

Despite using the same accesses and protection constraints, Protected-Joint-LS accepts only 3.67% of requests, compared with 97.0% for U2E. The latter also achieves substantially lower FE/FG, blind residual, and Extraction@20. The smaller PPL change of Protected-Joint-LS accompanies its very low acceptance rate. These results show that the feasibility-aware common-descent solve is essential for turning protected multi-access requirements into effective, committable edits, rather than merely combining existing editing components.


5. How does the finite update respect the trust region? (Q1, W4)

Thank you for highlighting the need to clarify trust-region enforcement. Eq. (10) determines the maximum attainable common progress t∗t^* under the value-movement constraints ∥ΔWℓkiℓ∥2≤ρℓ\|\Delta W_\ell k_{i\ell}\|_2 \leq \rho_\ell, while Eq. (8) characterizes the minimum-cost update in the local protected geometry. Since the latter does not independently enforce these radius constraints, U2E checks the composed candidate update before committing it.

Specifically, after constructing the low-rank update, we evaluate its value-movement radius at every constrained layer and access. The candidate is committed only if all radius constraints and full-model forgetting and preservation checks are satisfied. Otherwise, it is refined within the prescribed relinearization budget or rejected. Thus, trust-region compliance is enforced on the actual finite update, rather than inferred from the radius-free local solution.

Table. Post-composition trust-region audit. A violation is defined as a maximum radius ratio greater than 1+10−41+10^{-4}.

PanelSeedsRequestsAcceptedNo-opAbstainedMaximum radius ratioViolations
Submitted P11050044720330.98000
Independent P26300291630.99650

Across 738 committed edits, we observe no trust-region violations. The maximum realized ratios remain below one in both panels. We have made the post-composition radius check explicit in Algorithm 1 and clarified the distinction between the local minimum-cost characterization and the finite-update acceptance conditions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary: Key Revisions and Additional Evidence

We sincerely thank all reviewers for their thoughtful and constructive feedback. In response, we have strengthened the manuscript through additional theoretical clarifications, controlled experiments, broader evaluations, and implementation details. The main improvements are summarized below.

  1. Direct evidence of factual weakening. Neutral-free evaluation confirms that U2E reduces the original answer's log-probability by 3.00 nats on rewrite prompts and 1.53 nats on paraphrases, compared with 0.24 and 0.13 for ZeroUnlearn. The exact-sequence hit rate decreases from 24.0% to 2.0%.

  2. Robustness beyond construction prompts. Across nine unseen access families, U2E improves worst-route residual (0.654 vs. 0.724), Extraction@20 (0.730 vs. 0.840), and the fraction of robust facts (47.3% vs. 36.3%) relative to ZeroUnlearn.

  3. Isolating the contribution of protected common descent. We introduced Protected-Joint-LS under identical access, protection, trust-region, and validation conditions. U2E achieves 97.0% accepted coverage versus 3.67%, with stronger factual suppression and extraction resistance.

  4. Clarifying theoretical and practical guarantees. We distinguished the local minimum-cost characterization from finite-update acceptance conditions. Post-composition verification across 738 accepted updates confirms zero trust-region violations. We also clarified the feasibility conditions, implementation procedure, and stopping criteria.

  5. Broader and fairer evaluation. We added preservation-calibrated baseline comparisons, additional editing baselines, request-scaling experiments (10–50 requests), and evaluations across Llama model sizes, ZsRE, and a Qwen-based architecture. We also clarified computational costs and the limitations of cross-model generalization.

Together, these revisions provide stronger evidence that protected common descent offers a principled and practically verifiable approach to coordinating factual suppression across multiple access paths while explicitly controlling changes to unrelated model behavior.

We sincerely appreciate the reviewers' time and consideration. We hope these additional results and clarifications help address the concerns raised during the review process, and we would welcome any further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.