acceptodds
Under review as a conference paper at ICLR 2027

Rotation-First Unlearning: Selective Weight Damage in a Data-Dependent Basis

Abstract

After training, a language model may need to forget specific content without losing the capabilities learned from the rest of its data. Most LLM unlearning methods try to achieve this through additional optimization, using objectives that suppress the forget set while regularizing behavior on retain data. We propose Rotation-First Unlearning (RFU), a training-free weight update built on a simple geometric premise: if forget and retain activations concentrate energy in different subspaces, the forget-heavy subspaces can be identified and attenuated directly. RFU runs one forward pass over forget and retain data, forms activation moment matrices at each target layer, and solves a generalized eigenproblem that orders directions by forget-over-retain activation energy. It then converts this ordering into a Euclidean-orthonormal rotation that preserves the leading generalized eigenspaces, shrinks the corresponding weight coordinates with a closed-form rule, and maps the weights back to their original parameterization. The RFU update is deterministic, label-free, and gradient-free. We analyze RFU with a first-order bound on the retain damage, and derive a spectral concentration diagnostic that predicts when the RFU update would be selective before any evaluation. Across TOFU, MUSE, and WMDP on Llama-, Qwen-, Gemma-, and Mistral-family models, standalone RFU improves forget-retain tradeoff in spectrally separable settings, and consistently enables deeper forgetting at matched or near-matched retention when used as a warm start for other training-based unlearning methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
machine unlearning, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GAjqOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper proposes RFU, an unlearning framework that performs rotation-first spectral shrinkage to remove target knowledge while preserving retained capabilities. RFU estimates forget and retain activation covariances, solves a retain-regularized generalized eigensystem to identify forget-dominant directions, rotates weight updates into this basis, attenuates high-risk directions with closed-form shrinkage, and maps them back. The method is gradient-free, label-free, and efficient, requiring only forward passes over forget and retain data. Experiments on LLM unlearning benchmarks show competitive forgetting, stronger utility preservation, and improved robustness against relearning and quantization attacks.

Critical strengths
  • RFU provides a clean linear-algebraic view of unlearning by contrasting forget and retain activation covariances. The generalized eigenvalue formulation gives an interpretable criterion for identifying forget-dominant directions, making the method more principled than heuristic parameter damping or purely loss-based unlearning.

  • The method is gradient-free and label-free, requiring only forward passes to estimate activation statistics before applying a closed-form weight edit. This makes RFU computationally attractive for large language models, especially compared with iterative fine-tuning or adversarial unlearning methods.

Critical weaknesses
  • The model coverage is not fully sufficient. First, RFU is mostly evaluated on small-to-mid dense models up to roughly 8B parameters, leaving unclear whether the method remains stable on same-family 14B–30B models such as Qwen2.5-14B, Gemma-2/3-27B, or similar mid-scale Llama/Qwen/Gemma variants. Second, the paper does not evaluate MoE architectures, where token-dependent expert routing may change activation covariance structure and affect RFU’s rotation-based edit. Testing models such as Mixtral-8x7B, Qwen1.5-MoE-A2.7B, or DeepSeekMoE-16B would better establish architectural generality.

  • The paper emphasizes RFU’s efficiency through forward-only statistics and closed-form editing, but it lacks a detailed runtime and memory analysis. A stronger evaluation should report wall-clock comparisons, stage-wise cost breakdowns, and scaling behavior with model size, hidden dimension, edited layers, and forget/retain set size.

Questions for authors
  • Could the authors clarify what exactly is being “rotated” in RFU: the weight matrix, the activation/input basis, or only the coordinate system used to express the shrinkage operation?

  • How do the authors select which layers or weight matrices to edit with RFU, and how sensitive are the forgetting–retention trade-offs to this layer-selection strategy across different model architectures?

  • Could you demonstrate some failure cases of RFU? This would help clarify the boundaries of RFU’s capabilities and provide insights for future work.

5 comments
AuthorsOpenReview ↗

Response to questions (1/2)

Thank you for your careful review of our paper. We are glad you find RFU clean and principled, with additional benefits for interpretability. Your questions get at the heart of the RFU method; we address them below.

1. What is being rotated?

Briefly, RFU re-expresses each edited weight matrix in a rotated input basis, shrinks forget-heavy directions in that basis, and then maps the edited weights back to the original coordinates.

To discuss this in more detail, let us use the paper's notation where WW is a linear layer's weight matrix, aa its input, VV an orthonormal basis of input directions, and DβD_\beta a diagonal matrix of attenuation factors. RFU performs

\ \longrightarrow\ WVD_\beta V^\top.$$ Multiplication by $V$ first expresses the weights in the new input coordinates. Multiplication by $D_\beta$ then shrinks their responses along those coordinates. Finally, multiplication by $V^\top$ expresses the edited weights back in the original coordinates. Equivalently, on an input, $$W_{\mathrm{RFU}}a=(WV)D_\beta(V^\top a).$$ Here, $V^\top a$ is the input expressed in the RFU basis. The rotation determines which input directions receive each attenuation factor, and helps align the shrinkage with forget-heavy directions. Mapping back then expresses this edit in the model's original coordinates. **2. Why these layers, and how sensitive is the choice of layers** Across all evaluated model architectures, we apply RFU to all linear projections in every transformer block except for the MLP down projections, which we exclude for efficiency reasons. Thus, we edit the attention projections $q,k,v,o$ and the MLP gate/up projections in every transformer block. We exclude MLP down projections because their inputs are much wider than the model's hidden state. Recall that RFU uses forget and retain activation covariance matrices to determine the rotation for each layer. Each covariance matrix has one row and one column per input dimension, so solving the RFU generalized eigenvalue problem has a computational cost that grows cubically with that dimension. Thus, the wider inputs of the down projections make this step very expensive. We discuss this lightly in Appendix F.2, where we report that including the down projections for Llama-3.2-3B TOFU forget01 reduces forget recall from $0.471$ to $0.414$ but increases the preprocessing cost $\approx4\times$. **New layer selection ablations.** To better answer your question on how sensitive forget-retention is to layer selection, we test different layer selection strategies on TOFU forget01. First, we test how layer depth affects forget-retain: we apply the default selection to all blocks (excluding MLP down projections), edit layers in the first half only, and edit layers in the second half only. Results are reported below, where each cell shows forget recall ↓ / model utility ↑: | Model | All blocks | First half only | Second half only | |---|---:|---:|---:| | Llama-3.2-1B | **0.430** / 0.571 | 0.477 / 0.594 | 0.521 / 0.583 | | Llama-3.2-3B | **0.471** / 0.673 | 0.516 / 0.678 | 0.673 / 0.665 | | Qwen2.5-1.5B | **0.440** / 0.491 | 0.466 / 0.492 | 0.521 / 0.494 | | Qwen2.5-3B | **0.477** / 0.542 | 0.549 / 0.544 | 0.580 / 0.550 | | Gemma-3-4B | **0.428** / 0.538 | 0.495 / 0.568 | 0.568 / 0.546 | This table shows that editing all the blocks produces the most forgetting on every model. In comparison, editing only the first half produces less forgetting but preserves more model utility. Finally, editing only the first half produces more forgetting than editing only the second half. Next, we compare the effects of editing the default selection, editing MLP gate/up projections only, and editing attention projections $q,k,v,o$ only. | Model | All modules (default) | MLP projections only | Attention projections only | |---|---:|---:|---:| | Llama-3.2-1B | **0.430** / 0.571 | 0.445 / 0.586 | 0.477 / 0.580 | | Llama-3.2-3B | **0.471** / 0.673 | 0.521 / 0.673 | 0.645 / 0.675 | | Qwen2.5-1.5B | **0.440** / 0.491 | 0.464 / 0.497 | 0.540 / 0.496 | | Qwen2.5-3B | **0.477** / 0.542 | 0.487 / 0.558 | 0.683 / 0.541 | | Gemma-3-4B | **0.428** / 0.538 | 0.471 / 0.555 | 0.692 / 0.560 | Again, the default selection forgets the most on every model, but editing only the MLP projections comes close---its forget recall is within $0.01$ to $0.05$ of the default selection, with equal or slightly higher model utility. Editing only the attention projections forgets less on every model, by $0.05$ to $0.26$ on forget recall.
AuthorsOpenReview ↗

Response to questions (2/2)

[...Continued from earlier comment]

3. Capabilities / failure cases of RFU

You asked us to "clarify the boundaries of RFU's capabilities". We particularly appreciate this framing, as it is difficult to just say a method is simply "weak" or "strong"---all methods seem to have their own unlearning characteristics, or their own "boundaries" within which some tasks are better suited.

For RFU, two questions are especially relevant to delineating its boundary: (1) Can the activation statistics distinguish forget and retain? (2) Do the retain examples cover the capabilities we actually want to protect?

(1) Separation between forget and retain. All methods that aim for selective unlearning face the challenge of separating forget behavior from retain behavior. RFU measures separation with activation covariances, so it looks for linear input directions that are used much more by forget data than by retain data. In the limiting case where every direction has the same ratio of forget-to-retain (Af=cAρA_f=cA_\rho), RFU reduces to WRFU=W/(1+βc)W_{\mathrm{RFU}}=W/(1+\beta c), and there is no special direction for a rotation to isolate.

This is unlikely to occur in practice, but it hopefully gives you an idea of how RFU relies on the forget-to-retain ratio for its selective edit. Other methods approach this separation differently. DU constructs its forget and retain filters separately, then suppresses forget-associated components after reducing their overlap with retain. When forget directions are prominent and distinct from retain, DU can remove them more aggressively than RFU, as we see on TOFU forget01. However, when the dominant directions overlap, DU can miss weaker forget directions that RFU can still reach. Consistent with this, DU forgets less than RFU on the larger forget05 and forget10 sets, as larger forget sets cover more content and are more likely to share activation structure with retain.

(2) Retain examples vs capabilities to protect. Sometimes, a direction is important for broader capabilities but is rarely activated by the given retain set. Again, this is a general unlearning problem, and not in itself specific to RFU. However, in RFU, this problem has a direct effect on how strongly each direction is shrunk.

Recall that RFU uses the retain covariance ArA_r in the denominator of its forget-to-retain ratio, u⊤Afuu⊤(Ar+ρI)u.\frac{u^\top A_f u}{u^\top(A_r+\rho I)u}. If the retain examples barely activate a direction, the denominator becomes small, making that direction appear more forget-specific than it really is. RFU may therefore shrink it too strongly, damaging broader capabilities that were not represented in the retain data. (The ridge ρ\rho can limit this effect, but cannot recover information about capabilities missing from the retain set.)

This motivates the question we investigate in Appendix D.5: can we better preserve general capabilities by giving RFU a broader retain set? We test this by mixing the benchmark retain covariance with generic-text statistics, using 30% generic and 70% benchmark statistics (γ=0.3\gamma=0.3) while keeping β\beta fixed.

ModelForget recall before → afterMMLU loss before → after
Llama-3.2-3B0.471 → 0.4534.3 → 1.8
Gemma-3-4B0.428 → 0.4464.0 → 1.0
Llama-3.1-8B0.450 → 0.4436.9 → 3.0

We see that broadening the retain statistics significantly reduces how much MMLU is lost across all three models, while forgetting remains nearly unchanged. Thus, better retain coverage can improve RFU's forgetting-retention tradeoff by giving it a more complete picture of which directions should be protected.


Additional model coverage. We are actively working to extend our evaluation to larger models, following your suggestion. We have limited compute, so these results are not available yet, but we hope to share results later during the discussion period.

We thank you for your time and engagement!

AuthorsOpenReview ↗

Additional model coverage and runtime and memory analysis

Following your suggestion, we have run RFU on larger dense models and mixture-of-experts (MoE) architectures.


Larger dense models.

We now include additional runs of RFU on Qwen-2.5-14B and Qwen-2.5-32B for TOFU. Since we do not have enough compute to fully fine-tune the 32B model, we train it on TOFU using LoRA, merge the adapters into the model weights, then apply RFU to the merged model. We also train a 14B model using the same LoRA procedure, alongside a fully fine-tuned 14B model, to understand whether the training procedure affects the unlearning results.

The table shows forget recall (fRf_R) before and after RFU, where lower is better, as well as the change in model utility (MU) relative to the unedited model.

Model (TOFU training)Forget setForget recall before → afterMU change
Qwen2.5-14B (full)forget010.672 → 0.282−0.044
forget050.647 → 0.472−0.012
forget100.647 → 0.499−0.012
Qwen2.5-14B (LoRA)forget010.933 → 0.322−0.018
forget050.904 → 0.518−0.008
forget100.900 → 0.570−0.006
Qwen2.5-32B (LoRA)forget010.976 → 0.356−0.034
forget050.989 → 0.601−0.004
forget100.986 → 0.694−0.002

We see that, as with the smaller models in the paper, RFU lowers forget recall while largely preserving MU. The smallest forget set (forget01) has the largest reduction in forget recall; larger forget sets have less forgetting, but also forget information with almost no change in MU.

Comparing the fully fine-tuned and LoRA-trained 14B models, we see that the LoRA-trained model starts with substantially higher forget recall. RFU removes more recall from the LoRA models, but its final forget recall also remains higher. Still, it leaves both the fine-tuned and LoRA-trained models with much less forget recall.


MoE architectures.

To understand how RFU extends to MoE architectures, we also ran RFU on Qwen1.5-MoE-A2.7B and OLMoE-1B-7B.

In MoE, some experts receive relatively few forget or retain tokens, so their covariance estimates may be less reliable. RFU normally determines β\beta using the eigenvalue scales of all edited layers. However, for MoE, we compute β\beta using only layers that process every token, such as the attention projections and shared experts when present. We still apply the RFU weight edit to each expert using the expert's activation statistics and spectral decomposition.

ModelForget setForget recall before → afterMU change
Qwen1.5-MoE-A2.7Bforget011.000 → 0.439−0.012
forget050.981 → 0.604−0.002
forget100.982 → 0.646−0.002
OLMoE-1B-7Bforget010.956 → 0.411−0.012
forget050.921 → 0.555+0.004
forget100.912 → 0.604+0.003

Much like with dense models, RFU substantially lowers forget recall while largely preserving MU. The largest reductions are on forget01, but even larger forget sets have significant forgetting. These results demonstrate that RFU remains effective on MoE architectures.


Runtime and memory.

Finally, we profile RFU on the 14B and 32B models on TOFU forget01 and forget10. We break the process down into 3 stages: collecting forget and retain activation statistics, constructing the RFU basis, and applying the weight edit. The times for running RFU on a single RTX PRO 6000 are provided below, in seconds.

ModelForget statistics (forget01 / forget10)Retain statisticsRFU basisWeight editPeak GPU memory
Qwen2.5-14B16 / 52632451456 GiB
Qwen2.5-32B27 / 1461803262474 GiB

Time complexity. For nn total forget/retain tokens and an edited weight matrix of shape m×dm\times d, where dd is the input dimension and mm is the output dimension, computing the covariance matrices costs O(nd2)O(nd^2), constructing the RFU basis costs O(d3)O(d^3), and applying the weight edit costs O(md2)O(md^2).

Note that within each Transformer block, the query, key, and value projections share the same input activations, as do the gate and up projections; RFU can therefore reuse the covariance matrices and rotation basis across these projections. Retain statistics can also be reused for different forget requests on the same model. When changing the edit strength β\beta for the same forget/retain data, RFU can also reuse both the statistics and rotation basis, and requires only another weight edit.

Memory. Each covariance matrix requires O(d2)O(d^2) memory. We update these matrices as we process the data, rather than storing all the activations, so their memory requirements do not increase with dataset size. RFU also avoids the additional memory required for backpropagation and optimizer states.

Reviewer_jX1ZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Rotation-First Unlearning (RFU), a gradient-free weight-editing method for language models. RFU solves a generalized eigenproblem using forget and retain activation covariances, orthonormalizes the ordered eigenvectors, and applies spectrum-dependent weight shrinkage. The spectrum also determines edit strength and provides a separability score. Experiments evaluate standalone edits and warm starts across TOFU, MUSE, and WMDP, with additional analyses of alternative bases, relearning, quantization, and sequential deletion.

Critical strengths
  1. RFU provides a closed-form edit with explicit control over directional attenuation and a valid Euclidean weight-change bound.
  2. The paper evaluates multiple model families and benchmarks, reports paired five-seed warm-start experiments, and includes meaningful comparisons with alternative bases and the exact objective minimizer.
  3. The appendix examines general-capability degradation, retain-covariance coverage, paraphrased answers, and recovery attacks rather than relying exclusively on the primary forgetting metric.
Critical weaknesses
  1. Too much essential content is relegated to the appendix, while Table 2 occupies disproportionate space in the main paper.
  2. Equation (2) yields shrinkage factors in the generalized eigenbasis, but RFU applies them to QR-orthonormalized directions. As Appendix A.4 acknowledges, these directions generally no longer have Rayleigh quotients equal to their paired eigenvalues. Preserving nested subspaces establishes neither individual directional energy ratios nor optimality for the original objective. The norm bound remains valid, but the interpretation of each applied factor as calibrated forget-to-retain attenuation requires qualification.
Questions for authors

see weakness

2 comments

Thank you for your thoughtful review of our paper. We are glad that you like the explicit control that RFU provides, and that you find the experiments meaningful. We address your concerns below.

1. Adding appendix content to the main paper

We agree that the appendix has a lot of investigations that are not just extensions of main paper content, but show different (and important) facets of the unlearning problem beyond just performance on the forget set. We are happy to compress Table 2 and bring more content to the main paper, as you suggest. We can also use some of the recovered space to discuss the comparison with the exact objective minimizer you asked about, as discussed below:

2. Clarifying the relationship between RFU and the exact minimizer

We agree that QR orthonormalization generally changes the individual Rayleigh quotients, and that preserving the leading generalized eigenspaces does not mean RFU remains the exact minimizer of eq. (2). Nevertheless, the eigenvalue-based shrinkage factors retain two useful properties after QR: (A) They are exactly the diagonal entries of the matrix defining the minimizer, expressed in the QR basis, and (B) the eigenvalues remain lower bounds on the forget-to-retain energy ratios of the corresponding QR directions.

(A) Let Aρ=Ar+ρIA_\rho=A_r+\rho I, D=diag⁡(αi)D=\operatorname{diag}(\alpha_i), and αi=(1+βλi)−1\alpha_i=(1+\beta\lambda_i)^{-1}. The exact minimizer of eq. (2) is W⋆=WMW^\star=WM, where M=Aρ(Aρ+βAf)−1.M=A_\rho(A_\rho+\beta A_f)^{-1}.

Writing the ordered, unpivoted QR factorization as U=VTU=VT, we get V⊤MV=T−⊤DT⊤=D+NV^\top MV=T^{-\top}DT^\top=D+N where NN is strictly lower triangular.

Thus, the eigenvalue-derived factors are exactly the diagonal entries of the editing operator MM in the QR basis, although MM also contains off-diagonal mixing. RFU retains these diagonal entries and removes the mixing, yielding P=VDV⊤P=VDV^\top.

Moreover, among symmetric operators that preserve every leading QR subspace, PP has the smallest Frobenius-norm distance to MM. This follows because such operators must be diagonal in VV, and DD is the closest diagonal matrix to D+ND+N.

(B) Although the individual QR directions generally have different Rayleigh quotients from their paired eigenvalues, ordered QR preserves their leading subspaces. For every ii, writing vi=∑j≤icjujv_i=\sum_{j\le i}c_ju_j, we have

vi⊤Afvivi⊤Aρvi=∑j≤icj2λj∑j≤icj2≥λi.\frac{v_i^\top A_fv_i}{v_i^\top A_\rho v_i} = \frac{\sum_{j\le i}c_j^2\lambda_j}{\sum_{j\le i}c_j^2} \geq \lambda_i. Thus, λi\lambda_i is a lower bound on the forget-to-retain energy ratio of the QR direction. Since the actual ratio may be larger, using λi\lambda_i gives no more shrinkage than using the direction's actual ratio.

We can also establish a guarantee relating the changes to layer outputs on forget and retain inputs. Let hi=1−αih_i=1-\alpha_i and define the leading edit component Hk=∑i=1khivivi⊤.H_k=\sum_{i=1}^k h_i v_iv_i^\top.

Since HkH_k maps into the leading generalized eigenspace Sk\mathcal S_k, for any layer weight matrix WW,

\geq \lambda_k\left( \mathbb E_R\|WH_ka\|_2^2 +\rho\|WH_k\|_F^2 \right).$$ This follows because $A_f-\lambda_kA_\rho$ is positive semidefinite on $\mathcal S_k$. Moreover, the remaining weight edit satisfies $$\|W(I-P) - W H_k\|_F \leq h_{k+1} \|W\|_F.$$ Thus, when $\lambda_k$ is large and $h_{k+1}$ is small, the leading component of the edit produces larger squared changes to layer outputs on forget inputs than on retain inputs, while the remaining weight edit is small. If you find this helpful, we can clarify these details in Section 3.3; that is, we can explicitly distinguish the exact generalized eigenvector ratios from their QR lower bounds, explain the diagonal approximation to the exact minimizer, and give the result on output-level selectivity. --- We appreciate your feedback and will incorporate it into our draft. If you have any additional concerns, we are happy to discuss further. We thank you for your time and engagement!

Dear Reviewer jX1Z,

Thank you again for engaging with our paper! In our response, we hope we were able to clarify why RFU's eigenvalue-based shrinkage has meaningful guarantees after QR orthonormalization, in the sense that the shrinkage factors match the diagonal of the exact solution in the RFU basis, while the eigenvalues bound the forget-to-retain energy ratios of the QR directions.

If you have any remaining concerns, let us know and we would be happy to discuss. We are grateful for your support, and glad that our method resonated with you!

Reviewer_t5KpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Rotation-First Unlearning (RFU), a gradient-free weight-editing method that identifies forget-heavy directions through generalized eigendecomposition and applies selective shrinkage in an orthonormal basis. It also introduces a spectral separability score to predict unlearning effectiveness. Experiments on TOFU, MUSE, and WMDP demonstrate its standalone performance and its benefits as an initialization for training-based unlearning methods.

Critical strengths

Simple and practical method. RFU provides a deterministic, gradient-free unlearning approach with a bounded weight update. The orthonormalization step offers a reasonable solution to the instability of editing directly in the generalized eigenbasis.

The paper evaluates multiple models and benchmarks, with useful basis ablations and robustness analyses. The improvements when combined with existing training-based unlearning methods are particularly encouraging.

Critical weaknesses

There is limited theoretical justification. The idea of identifying forget-specific representation subspaces for selective unlearning is closely related to existing work, including Deep Unlearning, AI Engram, and ReGLU (ACL 2026). The paper does not sufficiently distinguish its contributions from these approaches. Moreover, the shrinkage factors are derived in the generalized eigenbasis but applied after QR orthonormalization, where the original eigenvalues no longer directly characterize individual directions. The provided norm bound does not establish selective forgetting.

Deep Unlearning achieves stronger forgetting in several settings, while standalone RFU shows limited effectiveness on WMDP. Stronger matched-retention comparisons are needed. Additionally, the separability score is primarily validated across forget sets of different sizes, making it difficult to determine whether it genuinely predicts unlearning effectiveness independently of dataset characteristics.

Questions for authors

Can the authors clarify the main methodological differences from ReGLU and related subspace-based approaches, and explain why eigenvalue-based shrinkage remains justified after QR orthonormalization?

How does RFU compare with properly tuned baselines under matched retention constraints? Can the authors validate the predictive power of the separability score using different forget sets of the same size?

3 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for your thoughtful review of our paper; we are glad you find RFU simple and practical, and the orthonormalization step effective for bounding the weight update. In our reading, you would like a clearer explanation on when to choose RFU, methodologically and/or experimentally. We hope to clarify this in our response.

Methodology 1: RFU compared to DU, AI Engram, and ReGLU

Let WW be a linear layer, aa its input, and Af=𝔼F[aa⊤]A_f=𝔼_F[aa^⊤], Ar=𝔼R[aa⊤]A_r=𝔼_R[aa^⊤]. Then\begin{align} \text{RFU:}~~ &W'=WV\text{diag}_i\left(\frac{1}{1+βλ_i}\right)V^⊤\\\\ \text{DU:}~~ &W'=W[I-(I-P_r)P_f]\\\\ \text{AI Engram:}~~ &W'=W[I-α_EpA_f C^†]\\\\ \text{ReGLU:}~~ &Z_k=\text{TopEig}_k((1-ω)B_f-ω B_r). \end{align}For RFU, Afui=λi(Ar+ρI)uiA_fu_i=λ_i(A_r+ρ I)u_i, so λiλ_i measures how strongly direction uiu_i is used by forget inputs relative to retain. VV contains these directions after orthonormalization in decreasing λiλ_i order.

Like Af,ArA_f,A_r for RFU, Pf,PrP_f,P_r in DU summarize the activation directions in each dataset, but with eigenvalues scaled to [0,1][0,1]. The term (I−Pr)Pf(I-P_r)P_f extracts forget components and reduces their overlap with retain directions before removal. This directly suppresses forget directions when the spaces are distinct, but when the major directions overlap, DU can underweight weaker forget-specific features.

AI Engram subtracts forget component αEWpAfC†α_EWpA_fC^†, where pp is the forget-token fraction and C=pAf+(1−p)ArC=pA_f+(1-p)A_r. When ArA_r is positive definite and αE=1α_E=1, ρ=0ρ=0, β=p/(1−p)β=p/(1-p), Engram gives W′=WMW'=WM, where M=Ar(Ar+βAf)−1M=A_r(A_r+β A_f)^{-1} is the exact minimizer of RFU's quadratic objective. In general, however, AfC†A_fC^† need not equal C†AfC^†A_f, so Engram's operator need not be symmetric and can have off-diagonal terms in the orthonormal basis VV. These terms can make W′viW'v_i include other responses WvjWv_j, so the output may change direction rather than only become weaker. RFU instead gives W′vi=(1+βλi)−1WviW'v_i=(1+βλ_i)^{-1}Wv_i with no mixing, and its orthonormal basis also prevents the weight edit from being inadvertently amplified by recombining nonorthogonal components after shrinkage.

For ReGLU, Bf=WAfWTB_f=WA_fW^T and Br=WArWTB_r=WA_rW^T are output second moments. Thus, ReGLU selects directions with a large weighted difference between forget and retain output energies, while RFU selects input directions using a forget-to-retain ratio. In addition, ReGLU uses the selected directions to initialize and train a LoRA. RFU avoids this additional training stage as it uses activation statistics to determine what directions to shrink, and how much to shrink them.

Methodology 2: why eigenvalue shrinkage remains justified after QR

Eigenvalue-based shrinkage remains justified after QR orthonormalization for two reasons: (A) RFU keeps the exact minimizer's diagonal shrinkage in the QR basis, and (B) ordered QR preserves the forget-heavy subspaces that justify assigning these shrinkage factors to the new directions.

(A) Recall that the objective in eq. 2 penalizes changes to retain outputs and remaining forget output. With Aρ=Ar+ρIA_ρ=A_r+ρ I, ordered QR U=VTU=VT, and Dii=(1+βλi)−1D_{ii}=(1+βλ_i)^{-1}, the minimizer satisfies W⋆=WM,  M=Aρ(Aρ+βAf)−1.W^⋆=WM,~~ M=A_ρ(A_ρ+β A_f)^{-1}. In the QR basis, the editing operator is V⊤MV=T−⊤DT⊤=D+N,V^⊤ MV=T^{-⊤}DT^⊤=D+N, where TT is upper triangular and NN is strictly lower triangular. RFU keeps DD and drops the mixing NN, giving P=VDV⊤P=VDV^⊤, with∣I−P∣2=βλ11+βλ1<1.|I-P|_2=\frac{βλ_1}{1+βλ_1}<1.Thus, RFU uses the shrinkage factors derived from the objective while removing cross-direction mixing, so each QR direction is attenuated independently and the size of the weight change is bounded.

(B) Importantly, ordered QR preserves the leading generalized eigenspaces. For every kk, letting Sk=span(v1,…,vk)=span(u1,…,uk)S_k=\text{span}(v_1,…,v_k)=\text{span}(u_1,…,u_k), we have z⊤Afz≥λkz⊤Aρz    ∀z∈Sk.z^⊤A_fz≥λ_kz^⊤A_ρz~~~~∀z∈S_k.In particular, because vi∈Siv_i∈S_i,vi⊤Afvivi⊤Aρvi≥λi.\frac{v_i^⊤A_fv_i}{v_i^⊤A_ρv_i}≥λ_i. Thus, the eigenvalues remain valid lower bounds on the forget-to-retain energy ratios after QR. Directions assigned strong shrinkage are therefore guaranteed to be forget-heavy in the orthonormal basis.

This also gives an output-level selectivity result beyond the norm bound. Let hi=βλi/(1+βλi)h_i=βλ_i/(1+βλ_i) and Hk=∑i=1khivivi⊤H_k=∑_{i=1}^kh_iv_iv_i^⊤ denote the part of the edit I−PI-P acting on SkS_k. Then, for any layer weight matrix WW,𝔼F∣WHka∣22≥λk(𝔼R∣WHka∣22+ρ∣WHk∣F2).𝔼_F|WH_ka|_2^2≥λ_k(𝔼_R|WH_ka|_2^2+ρ|WH_k|_F^2).

Each row of WHkW H_k lies in SkS_k, so applying the preceding subspace inequality to each row and summing proves the bound. Thus, for the leading kk directions, the mean squared output change is at least λkλ_k times as large on forget inputs as on retain, even with the ridge penalty. Moreover, for k<dink<d_{in}, ∣W(I−P)−WHk∣F≤hk+1∣W∣F.|W(I-P)-WH_k|_F≤h_{k+1}|W|_F.

This means that when hk+1h_{k+1} is small, the edit outside the leading kk directions is also small. Together, the two bounds show both how the leading part of the edit affects forget inputs relative to retain, and also how much of the edit remains outside those directions.

AuthorsOpenReview ↗

Response (2/2)

[...Continued from earlier comment]

Experiment 1: matched-retention comparisons

We agree that DU achieves stronger forgetting in several settings. However, we wish to clarify a few points on where RFU offers an advantage.

  • Recall that DU constructs its forget and retain filters separately, while RFU selects directions by their forget-to-retain ratio. This means that when major forget and retain directions overlap, DU can underweight weaker forget-specific features. In practice, this means that DU does worse on larger forget sets that cover more content, which we see on TOFU, and it also does worse when the forget text is closely related to retain text, which we see on MUSE-Books. This is all to say that DU and RFU have complementary strengths, and whether RFU should be preferred over DU depends on the specific unlearning setting.
  • DU is tuned over its forget and retain scaling coefficients αf\alpha_f and αr\alpha_r, whereas RFU does not use any per-setting tuning. Thus, RFU can be seen as a simpler/more efficient alternative, where the strength of the edit is calibrated from the spectrum rather than from searching over and evaluating multiple candidate edits.
  • When we add tuning to RFU, tuned RFU does better than DU in most TOFU settings. We present these results below.

Following your request, we compare RFU, tuned RFU, and (tuned) DU on TOFU under matched retention constraints. Analogously to how DU is tuned over αf\alpha_f and αr\alpha_r, we tune RFU over strength β\beta and ridge ρ\rho, controlling how much forget-heavy directions are shrunk and how much directions that retain inputs barely use are protected from the edit, respectively.

MethodTuned overForget-recall reduction, MU loss ≤0.01\le 0.01Forget-recall reduction, MU loss ≤0.02\le 0.02
RFUnone0.1420.163
Tuned RFUβ\beta, ρ\rho0.3170.384
DUαf\alpha_f, αr\alpha_r0.2930.355
AI EngramαE\alpha_E0.2530.336

Recall that we run TOFU on 3 forget sets (forget01, forget05, forget10) with 5 models each (Llama-1B, Llama-3B, Qwen-1.5B, Qwen-3B, Gemma-4B). Each number in the table is therefore averaged over 3x5=15 settings.

For each tuned method and setting, we evaluate multiple choices of hyperparameters and select the choice that gives the largest forget-recall reduction while keeping MU loss within budget. If no evaluated choice satisfies the budget, we count a reduction of 0. Note that, since default RFU does not search over hyperparameters, it uses the same edit for both columns; however, RFU satisfies the 0.02 budget on more settings (11) than it satisfies the 0.01 budget (8), so the averages differ between the two budgets.

We see on average over all models and forget sets sizes, RFU does forget less than the tuned methods under matched retention, but it also works without any per-setting hyperparameter search. With tuning, RFU reaches the largest reduction in mean forget recall of the methods at both MU loss budgets.

Experiment 2: separability score

Since the original separability score validation varied the size of the forget set, the concern is that SsepS_{sep} might simply reflect forget set size. To investigate this, we construct 32 equally sized forget sets, each with 40 TOFU questions, while varying the authors represented. For each set, we compute SsepS_{sep} before editing, then measure the reduction in forget-answer probability that can be achieved with MU loss ≤0.01\le 0.01. As in our paraphrase analysis, we use answer probability as a continuous measure of forgetting. We then group sets by their score within each model and report their mean absolute probability reduction (higher means more forgetting), as well as the Spearman correlation across all 32 sets.

ModelLow scoreMid scoreHigh scoreSpearman (95% CI)
Llama-3.2-1B0.2880.3350.4070.80 (0.59, 0.90)
Llama-3.2-3B0.6750.7210.7520.80 (0.60, 0.90)
Qwen2.5-1.5B0.1070.1310.2620.58 (0.27, 0.78)
Qwen2.5-3B0.1090.1560.2480.72 (0.41, 0.91)
Gemma-3-4B0.0500.0580.0840.44 (0.08, 0.70)

Across all five models, higher scoring sets tend to achieve larger reductions under the same retain cost budget, with positive Spearman correlations ranging from 0.440.44 to 0.800.80. The strength of the association does vary across models, but these results suggest that SsepS_{sep} still provides useful information about achievable forgetting, even when the size of the forget set is held fixed.


We appreciate the opportunity to answer your questions, which are important to understanding the contribution of RFU. We hope these responses help to provide some more clarification. If you have any additional concerns, we are happy to discuss further.

Thank you for your kind engagement with our work!

Dear Reviewer t5Kp,

Thank you again for your engagement and thoughtful feedback. In our response, we hope we clarified how RFU differs from other weight editing methods: primarily, by focusing on forget-to-retain activation ratios, RFU can identify and shrink forget-heavy directions, even when forget and retain representations overlap. We also hope we were able to clarify why eigenvalue-based shrinkage remains justified after QR orthonormalization: the shrinkage factors match the diagonal of the exact solution in the RFU basis, while the eigenvalues bound the forget-to-retain energy ratio of the orthonormalized directions, supporting a selectivity guarantee beyond the norm bound.

We also hope the additional experiments help clarify the practical utility of RFU. Unlike other weight editing methods, RFU requires no per-setting tuning, but it also forgets less on TOFU when averaged over all the forget sets. However, when RFU is also tuned, we find that tuned RFU achieves stronger forgetting than the other tuned baselines under matched-retention constraints. Separately, our experiments on different forget sets of the same size suggest that the separability score remains informative even when the size of the forget set is held fixed.

Let us know if you have any remaining concerns and we would be happy to discuss. In the meantime, we thank you sincerely for your time and feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.