- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
This paper proposes RFU, an unlearning framework that performs rotation-first spectral shrinkage to remove target knowledge while preserving retained capabilities. RFU estimates forget and retain activation covariances, solves a retain-regularized generalized eigensystem to identify forget-dominant directions, rotates weight updates into this basis, attenuates high-risk directions with closed-form shrinkage, and maps them back. The method is gradient-free, label-free, and efficient, requiring only forward passes over forget and retain data. Experiments on LLM unlearning benchmarks show competitive forgetting, stronger utility preservation, and improved robustness against relearning and quantization attacks.
Critical strengths
-
RFU provides a clean linear-algebraic view of unlearning by contrasting forget and retain activation covariances. The generalized eigenvalue formulation gives an interpretable criterion for identifying forget-dominant directions, making the method more principled than heuristic parameter damping or purely loss-based unlearning.
-
The method is gradient-free and label-free, requiring only forward passes to estimate activation statistics before applying a closed-form weight edit. This makes RFU computationally attractive for large language models, especially compared with iterative fine-tuning or adversarial unlearning methods.
Critical weaknesses
-
The model coverage is not fully sufficient. First, RFU is mostly evaluated on small-to-mid dense models up to roughly 8B parameters, leaving unclear whether the method remains stable on same-family 14B–30B models such as Qwen2.5-14B, Gemma-2/3-27B, or similar mid-scale Llama/Qwen/Gemma variants. Second, the paper does not evaluate MoE architectures, where token-dependent expert routing may change activation covariance structure and affect RFU’s rotation-based edit. Testing models such as Mixtral-8x7B, Qwen1.5-MoE-A2.7B, or DeepSeekMoE-16B would better establish architectural generality.
-
The paper emphasizes RFU’s efficiency through forward-only statistics and closed-form editing, but it lacks a detailed runtime and memory analysis. A stronger evaluation should report wall-clock comparisons, stage-wise cost breakdowns, and scaling behavior with model size, hidden dimension, edited layers, and forget/retain set size.
Questions for authors
-
Could the authors clarify what exactly is being “rotated” in RFU: the weight matrix, the activation/input basis, or only the coordinate system used to express the shrinkage operation?
-
How do the authors select which layers or weight matrices to edit with RFU, and how sensitive are the forgetting–retention trade-offs to this layer-selection strategy across different model architectures?
-
Could you demonstrate some failure cases of RFU? This would help clarify the boundaries of RFU’s capabilities and provide insights for future work.
5 comments
Response to questions (2/2)
[...Continued from earlier comment]
3. Capabilities / failure cases of RFU
You asked us to "clarify the boundaries of RFU's capabilities". We particularly appreciate this framing, as it is difficult to just say a method is simply "weak" or "strong"---all methods seem to have their own unlearning characteristics, or their own "boundaries" within which some tasks are better suited.
For RFU, two questions are especially relevant to delineating its boundary: (1) Can the activation statistics distinguish forget and retain? (2) Do the retain examples cover the capabilities we actually want to protect?
(1) Separation between forget and retain. All methods that aim for selective unlearning face the challenge of separating forget behavior from retain behavior. RFU measures separation with activation covariances, so it looks for linear input directions that are used much more by forget data than by retain data. In the limiting case where every direction has the same ratio of forget-to-retain (), RFU reduces to , and there is no special direction for a rotation to isolate.
This is unlikely to occur in practice, but it hopefully gives you an idea of how RFU relies on the forget-to-retain ratio for its selective edit. Other methods approach this separation differently. DU constructs its forget and retain filters separately, then suppresses forget-associated components after reducing their overlap with retain. When forget directions are prominent and distinct from retain, DU can remove them more aggressively than RFU, as we see on TOFU forget01. However, when the dominant directions overlap, DU can miss weaker forget directions that RFU can still reach. Consistent with this, DU forgets less than RFU on the larger forget05 and forget10 sets, as larger forget sets cover more content and are more likely to share activation structure with retain.
(2) Retain examples vs capabilities to protect. Sometimes, a direction is important for broader capabilities but is rarely activated by the given retain set. Again, this is a general unlearning problem, and not in itself specific to RFU. However, in RFU, this problem has a direct effect on how strongly each direction is shrunk.
Recall that RFU uses the retain covariance in the denominator of its forget-to-retain ratio, If the retain examples barely activate a direction, the denominator becomes small, making that direction appear more forget-specific than it really is. RFU may therefore shrink it too strongly, damaging broader capabilities that were not represented in the retain data. (The ridge can limit this effect, but cannot recover information about capabilities missing from the retain set.)
This motivates the question we investigate in Appendix D.5: can we better preserve general capabilities by giving RFU a broader retain set? We test this by mixing the benchmark retain covariance with generic-text statistics, using 30% generic and 70% benchmark statistics () while keeping fixed.
| Model | Forget recall before → after | MMLU loss before → after |
|---|---|---|
| Llama-3.2-3B | 0.471 → 0.453 | 4.3 → 1.8 |
| Gemma-3-4B | 0.428 → 0.446 | 4.0 → 1.0 |
| Llama-3.1-8B | 0.450 → 0.443 | 6.9 → 3.0 |
We see that broadening the retain statistics significantly reduces how much MMLU is lost across all three models, while forgetting remains nearly unchanged. Thus, better retain coverage can improve RFU's forgetting-retention tradeoff by giving it a more complete picture of which directions should be protected.
Additional model coverage. We are actively working to extend our evaluation to larger models, following your suggestion. We have limited compute, so these results are not available yet, but we hope to share results later during the discussion period.
We thank you for your time and engagement!
Additional model coverage and runtime and memory analysis
Following your suggestion, we have run RFU on larger dense models and mixture-of-experts (MoE) architectures.
Larger dense models.
We now include additional runs of RFU on Qwen-2.5-14B and Qwen-2.5-32B for TOFU. Since we do not have enough compute to fully fine-tune the 32B model, we train it on TOFU using LoRA, merge the adapters into the model weights, then apply RFU to the merged model. We also train a 14B model using the same LoRA procedure, alongside a fully fine-tuned 14B model, to understand whether the training procedure affects the unlearning results.
The table shows forget recall () before and after RFU, where lower is better, as well as the change in model utility (MU) relative to the unedited model.
| Model (TOFU training) | Forget set | Forget recall before → after | MU change |
|---|---|---|---|
| Qwen2.5-14B (full) | forget01 | 0.672 → 0.282 | −0.044 |
| forget05 | 0.647 → 0.472 | −0.012 | |
| forget10 | 0.647 → 0.499 | −0.012 | |
| Qwen2.5-14B (LoRA) | forget01 | 0.933 → 0.322 | −0.018 |
| forget05 | 0.904 → 0.518 | −0.008 | |
| forget10 | 0.900 → 0.570 | −0.006 | |
| Qwen2.5-32B (LoRA) | forget01 | 0.976 → 0.356 | −0.034 |
| forget05 | 0.989 → 0.601 | −0.004 | |
| forget10 | 0.986 → 0.694 | −0.002 |
We see that, as with the smaller models in the paper, RFU lowers forget recall while largely preserving MU. The smallest forget set (forget01) has the largest reduction in forget recall; larger forget sets have less forgetting, but also forget information with almost no change in MU.
Comparing the fully fine-tuned and LoRA-trained 14B models, we see that the LoRA-trained model starts with substantially higher forget recall. RFU removes more recall from the LoRA models, but its final forget recall also remains higher. Still, it leaves both the fine-tuned and LoRA-trained models with much less forget recall.
MoE architectures.
To understand how RFU extends to MoE architectures, we also ran RFU on Qwen1.5-MoE-A2.7B and OLMoE-1B-7B.
In MoE, some experts receive relatively few forget or retain tokens, so their covariance estimates may be less reliable. RFU normally determines using the eigenvalue scales of all edited layers. However, for MoE, we compute using only layers that process every token, such as the attention projections and shared experts when present. We still apply the RFU weight edit to each expert using the expert's activation statistics and spectral decomposition.
| Model | Forget set | Forget recall before → after | MU change |
|---|---|---|---|
| Qwen1.5-MoE-A2.7B | forget01 | 1.000 → 0.439 | −0.012 |
| forget05 | 0.981 → 0.604 | −0.002 | |
| forget10 | 0.982 → 0.646 | −0.002 | |
| OLMoE-1B-7B | forget01 | 0.956 → 0.411 | −0.012 |
| forget05 | 0.921 → 0.555 | +0.004 | |
| forget10 | 0.912 → 0.604 | +0.003 |
Much like with dense models, RFU substantially lowers forget recall while largely preserving MU. The largest reductions are on forget01, but even larger forget sets have significant forgetting. These results demonstrate that RFU remains effective on MoE architectures.
Runtime and memory.
Finally, we profile RFU on the 14B and 32B models on TOFU forget01 and forget10. We break the process down into 3 stages: collecting forget and retain activation statistics, constructing the RFU basis, and applying the weight edit. The times for running RFU on a single RTX PRO 6000 are provided below, in seconds.
| Model | Forget statistics (forget01 / forget10) | Retain statistics | RFU basis | Weight edit | Peak GPU memory |
|---|---|---|---|---|---|
| Qwen2.5-14B | 16 / 52 | 63 | 245 | 14 | 56 GiB |
| Qwen2.5-32B | 27 / 146 | 180 | 326 | 24 | 74 GiB |
Time complexity. For total forget/retain tokens and an edited weight matrix of shape , where is the input dimension and is the output dimension, computing the covariance matrices costs , constructing the RFU basis costs , and applying the weight edit costs .
Note that within each Transformer block, the query, key, and value projections share the same input activations, as do the gate and up projections; RFU can therefore reuse the covariance matrices and rotation basis across these projections. Retain statistics can also be reused for different forget requests on the same model. When changing the edit strength for the same forget/retain data, RFU can also reuse both the statistics and rotation basis, and requires only another weight edit.
Memory. Each covariance matrix requires memory. We update these matrices as we process the data, rather than storing all the activations, so their memory requirements do not increase with dataset size. RFU also avoids the additional memory required for backpropagation and optimizer states.
Response to questions (1/2)
Thank you for your careful review of our paper. We are glad you find RFU clean and principled, with additional benefits for interpretability. Your questions get at the heart of the RFU method; we address them below.
1. What is being rotated?
Briefly, RFU re-expresses each edited weight matrix in a rotated input basis, shrinks forget-heavy directions in that basis, and then maps the edited weights back to the original coordinates.
To discuss this in more detail, let us use the paper's notation where W is a linear layer's weight matrix, a its input, V an orthonormal basis of input directions, and Dβ a diagonal matrix of attenuation factors. RFU performs
\ \longrightarrow\ WVD_\beta V^\top.$$ Multiplication by $V$ first expresses the weights in the new input coordinates. Multiplication by $D_\beta$ then shrinks their responses along those coordinates. Finally, multiplication by $V^\top$ expresses the edited weights back in the original coordinates. Equivalently, on an input, $$W_{\mathrm{RFU}}a=(WV)D_\beta(V^\top a).$$ Here, $V^\top a$ is the input expressed in the RFU basis. The rotation determines which input directions receive each attenuation factor, and helps align the shrinkage with forget-heavy directions. Mapping back then expresses this edit in the model's original coordinates. **2. Why these layers, and how sensitive is the choice of layers** Across all evaluated model architectures, we apply RFU to all linear projections in every transformer block except for the MLP down projections, which we exclude for efficiency reasons. Thus, we edit the attention projections $q,k,v,o$ and the MLP gate/up projections in every transformer block. We exclude MLP down projections because their inputs are much wider than the model's hidden state. Recall that RFU uses forget and retain activation covariance matrices to determine the rotation for each layer. Each covariance matrix has one row and one column per input dimension, so solving the RFU generalized eigenvalue problem has a computational cost that grows cubically with that dimension. Thus, the wider inputs of the down projections make this step very expensive. We discuss this lightly in Appendix F.2, where we report that including the down projections for Llama-3.2-3B TOFU forget01 reduces forget recall from $0.471$ to $0.414$ but increases the preprocessing cost $\approx4\times$. **New layer selection ablations.** To better answer your question on how sensitive forget-retention is to layer selection, we test different layer selection strategies on TOFU forget01. First, we test how layer depth affects forget-retain: we apply the default selection to all blocks (excluding MLP down projections), edit layers in the first half only, and edit layers in the second half only. Results are reported below, where each cell shows forget recall ↓ / model utility ↑: | Model | All blocks | First half only | Second half only | |---|---:|---:|---:| | Llama-3.2-1B | **0.430** / 0.571 | 0.477 / 0.594 | 0.521 / 0.583 | | Llama-3.2-3B | **0.471** / 0.673 | 0.516 / 0.678 | 0.673 / 0.665 | | Qwen2.5-1.5B | **0.440** / 0.491 | 0.466 / 0.492 | 0.521 / 0.494 | | Qwen2.5-3B | **0.477** / 0.542 | 0.549 / 0.544 | 0.580 / 0.550 | | Gemma-3-4B | **0.428** / 0.538 | 0.495 / 0.568 | 0.568 / 0.546 | This table shows that editing all the blocks produces the most forgetting on every model. In comparison, editing only the first half produces less forgetting but preserves more model utility. Finally, editing only the first half produces more forgetting than editing only the second half. Next, we compare the effects of editing the default selection, editing MLP gate/up projections only, and editing attention projections $q,k,v,o$ only. | Model | All modules (default) | MLP projections only | Attention projections only | |---|---:|---:|---:| | Llama-3.2-1B | **0.430** / 0.571 | 0.445 / 0.586 | 0.477 / 0.580 | | Llama-3.2-3B | **0.471** / 0.673 | 0.521 / 0.673 | 0.645 / 0.675 | | Qwen2.5-1.5B | **0.440** / 0.491 | 0.464 / 0.497 | 0.540 / 0.496 | | Qwen2.5-3B | **0.477** / 0.542 | 0.487 / 0.558 | 0.683 / 0.541 | | Gemma-3-4B | **0.428** / 0.538 | 0.471 / 0.555 | 0.692 / 0.560 | Again, the default selection forgets the most on every model, but editing only the MLP projections comes close---its forget recall is within $0.01$ to $0.05$ of the default selection, with equal or slightly higher model utility. Editing only the attention projections forgets less on every model, by $0.05$ to $0.26$ on forget recall.