- Rating
- 4(+2)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes Causal Routing for Unlearning (CRU). It selects hidden-state dimensions using activation variance on forget data, then trains small routers to suppress these dimensions depending on the input. The base model remains frozen. Experiments on TOFU and RWKU show promising forgetting–utility trade-offs and low training memory costs.
Critical strengths
-
The method is simple and memory-efficient. Training only small routers makes the approach practical. On Phi, the reported TOFU training memory drops from about 71 GiB for the evaluated weight-update baselines to 14 GiB.
-
The empirical results are promising. TOFU experiments cover four model families. On RWKU, CRU achieves adversarial-probe recall of 0.052, compared with 0.250 for the strongest evaluated baseline. The random-selection ablation also supports the value of the localization step.
Critical weaknesses
-
The claim of knowledge removal is not supported by the evidence. Section 5.1 claims that CRU guarantees removal rather than hiding. However, Appendix A.2 explicitly states that removing the routers exactly restores the original model. The RWKU results show that the tested prompts extract less knowledge. They do not establish that the knowledge is gone or that recovery is difficult. This distinction is central to the paper’s motivation and conclusions.
-
The comparisons do not establish the benefit of learned routing. NeuMuter, CAST, DSG, and GUARD-IT are discussed but not evaluated, despite being closely related to CRU. Most comparisons instead use standard weight-update methods, several of which suffer utility collapse. These results do not show whether learned routing improves over existing gating methods at comparable utility. Matched comparisons with the closest alternatives are needed to establish the main contribution.
Questions for authors
-
What access is the attacker assumed to have? With the routers kept active, can target knowledge be recovered through prompts designed to bypass the gates or a small amount of fine-tuning?
-
Figure 5 reports a membership-inference AUC of 0.058. Under the standard ROC-AUC definition, reversing the scores gives 0.942, indicating strong separation. How should this result be interpreted? Why is the increased vulnerability described as only slight?
-
The evaluated model families are relatively old. Qwen2.5 was released in September 2024, about two years ago. Can you provide results on Qwen3.5, especially on RWKU, to show that the main findings extend to current models?
-
Several closely related works explore similar ideas of localization, routing, and gating:
- Gradient Routing: Masking Gradients to Localize Computation in Neural Networks (arXiv 2024)
- Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearning (ICLR 2026)
- Preemptive LLM Unlearning against Forbidden Capability Acquisition via Gradient Sealing (arXiv 2026)
Please cite these works and clarify how CRU differs from them in both motivation and methodology, particularly in how target-related neurons are identified and how routing or gating is used to control their influence. These distinctions are important for understanding the actual novelty of CRU.
I would be happy to raise my score if the authors address these concerns with clear explanations and supporting results.
3 comments
W2. NeuMuter, CAST, DSG, and GUARD-IT are discussed but not evaluated
In the revised paper we compare with GUARD-IT and NeuMuter on both TOFU settings. The table below shows that CRU wins across all axes for both settings on Llama and Qwen, forgets the most on both settings for DeepSeek, and for Forget10 on Phi.
| Model | Method | Forget01 Forgetting | Forget01 MU | Forget10 Forgetting | Forget10 MU |
|---|---|---|---|---|---|
| Llama | GUARD-IT | 0.11 | 0.51 | 0.15 | 0.51 |
| NeuMuter | 0.00 | 0.51 | 0.01 | 0.51 | |
| CRU | 0.67 | 0.55 | 0.21 | 0.53 | |
| Qwen | GUARD-IT | 0.84 | 0.48 | 0.78 | 0.47 |
| NeuMuter | 0.01 | 0.48 | 0.01 | 0.48 | |
| CRU | 0.98 | 0.50 | 0.99 | 0.53 | |
| DeepSeek | GUARD-IT | 0.47 | 0.49 | 0.49 | 0.49 |
| NeuMuter | 0.39 | 0.48 | 0.38 | 0.47 | |
| CRU | 0.57 | 0.35 | 0.66 | 0.36 | |
| Phi | GUARD-IT | 0.86 | 0.46 | 0.71 | 0.45 |
| NeuMuter | 0.71 | 0.17 | 0.42 | 0.46 | |
| CRU | 0.98 | 0.14 | 0.97 | 0.23 |
However, CAST cannot be benchmarked fairly. In the paper, the authors claim that they "manually search for the appropriate intervention strength and layers" and that "there is not a universally applicable range for the grid search". They have not released any setting for the models we test. Similarly, DSG needs a pretrained sparse autoencoder for the exact model it runs on, and there are no publicly available ones for the majority of our models.
With these baselines, CRU is now compared against 9 SoTA methods, that is substantially stronger and more comprehensive than any previous setting in the literature.
| Method | Pareto-optimal panels |
|---|---|
| CRU | 8/8 |
| GA | 5/8 |
| GUARD-IT | 4/8 |
| NPO | 3/8 |
| SimNPO | 2/7 |
| GD, ICU, NeuMuter | 0 |
Finally, as the table shows, CRU is the only method that is never pareto-dominated by any method, even with the addition of GUARD-IT and NeuMuter.
Q3. provide results on Qwen3.5, especially on RWKU
RWKU publicly releases forget corpora only for Llama-3-8B and Phi-3, which is why we and prior work use them. However, following your suggestion, we are doing our best effort to regenerate RWKU's forget corpus for Qwen3.5-4B using the benchmark's own scripts and run all methods with the same tuning protocol.
Re-running every method on a new model is very costly and takes a lot of time. We are still waiting for results; we will update you when they are done.
Q4. Several closely related works explore similar ideas
Thanks for these references. The revised paper adds 2 out of 3 of the suggested references; the third, published on arXiv, is concurrent to our work as per ICLR's policy ("Papers published on or after July 17, 2026 are considered contemporary relative to the September 16, 2026 deadline"). While they share vocabulary with CRU (localization, routing, gating), they differ in the problem addressed and in what is routed or gated.
-
Gradient Routing [1] masks gradients during training so that a user-specified capability is localized in a pre-chosen subregion; it does not act on an already trained model. CRU instead keeps a trained model frozen and gates its forward activations per token at inference time.
-
Ssiuu [2] adds an attribution-based regularizer to full-parameter unlearning so that methods erase knowledge rather than hide it behind inhibitory neurons that retraining can disrupt; it involves no localization-then-gating step. CRU is also a suppression mechanism, but an explicit one separate from the weights. Since Ssiuu targets retraining attacks (weight access) while CRU is designed for black-box deployment, we state this scope explicitly and list robustness to retraining as a limitation in the revised paper (lines 538–539).
-
GSU [3] retrains all parameters to saturate selected SwiGLU gates, so that an open-weight model cannot acquire a capability through future fine-tuning; it does not remove knowledge already present and adds no gating module.
In summary, none of these methods keeps the base model frozen and learns an input-conditional gate that decides at query time whether target-related units pass, which is the axis along which we position CRU (§1, Fig. 1). We add this discussion to the related work section (lines 141-143, 150-153, Figure 1 has the 2 new methods).
[1] Gradient Routing: Masking Gradients to Localize Computation in Neural Networks.
[2] Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearning
[3] Preemptive LLM Unlearning against Forbidden Capability Acquisition via Gradient Sealing
Thank you for the additional experiments and the thoughtful responses. I appreciate the effort and have raised my score to 3. The added NeuMuter and GUARD-IT comparisons largely address my concern about missing close baselines, and the clarifications on the threat model and related work are helpful.
Two points remain open for me: (1) The results support suppression under the tested black-box probes, but not necessarily the stronger claim that the knowledge is “gone”. More cautious wording would help, especially since attacks tailored to the routers remain untested. (2) The MIA interpretation could use further clarification: reversing the scores turns an AUC of 0.058 into 0.942. While this does not establish knowledge recovery, the privacy implications of distinguishing forgotten targets still deserve consideration, even under black-box access.
Good luck!
Thanks for the positive assessment on the readability of the paper and the highlighted strengths. As some of the weaknesses identified are closely related to some of the questions, we answer them together for clarity. All changes discussed here are implemented in the revised paper. Specifically, for changes spanning from reviewer
Bpyf, the edits are highlighted in ORANGE.We indeed train only a fraction of the parameters to perform unlearning: i.e., the routers of Fig. 2. It is also true that if we remove the trained routers (just like steering approaches do) one can recover the base model prior to the unlearning process (as per Appendix A.2) for auditing purposes only. Nevertheless, our threat scenario is black-box: i.e., the routers are deployed alongside the backbone model and a potential malicious attacker can only perform queries to elicit knowledge, which we show does not work. This is visible in Figure 5 (top row, L1/L2/L3) where CRU resists even more sophisticated (higher level) attacks, whereas all baselines, besides SimNPO in some cases, show vulnerabilities to the same attacks. This also answers your question Q1, to which we add the following.
For fairness of comparison, we fully comply with RWKU's attack schema. RWKU gives the attacker full query access and per-token likelihoods with nine families of adversarial prompts. The gate-bypassing prompts you describe are exactly the Level 3 probes test from RWKU (comprising adversarial prompts and probes, e.g., cross-lingual, synonym, reverse-query, in-context...). We note that CRU's forget retention does not rise under these attacks. For contrast, SimNPO's rises significantly on both models. To our knowledge, no benchmark tests attacks adapted to an inference-time component, and none of TOFU or RWKU evaluates fine-tuning recovery, which would be an interesting research avenue.
To avoid any confusion, in lines 363-366 of the revised paper we clarify this attack schema introduced in the RWKU benchmark.
In RWKU's MIA, the attacker scores text about the target (FM) and that of unrelated subjects (RM) by Min-K% likelihood, and the AUC measures how well the two sets can be separated. If CRU merely hides the knowledge while still encoding it, the target text would remain more likely than unrelated text, giving an AUC at or above the original model's ($\sim$0.45). We observe the opposite, namely that CRU has lower likelihood for the target text than text about subjects it was never asked to forget. The distance from chance therefore reflects suppression of the target, not residual knowledge. It makes the forgotten subject detectable, but it is not evidence that the knowledge is still present.
Although MIA can tell that the forget set was part of the unlearning process with CRU, we argue that in a black-box scenario this is irrelevant. Moreover, we want to emphasize that, on forget prompts, CRU produces non-fluent output (gibberish) that makes no claims about the target, whereas the SoTA baselines produce fluent but fabricated statements on such a target.
To further support this interpretation, we used Claude Opus 5.5 to classify each answer to the RWKU forget probes (on Phi-3) as leaked (still revealing knowledge about the target), hallucinated, or gibberish. Every baseline that reduces leakage does so by replacing the leaked answers with fabricated ones: when GA, PURGE, and NPO do not leak, the answer is hallucinated 96-97% for GA, PURGE, and NPO, and 88% for SimNPO. CRU leaks least (6.7%, vs. 67.6% for the original model and 15.3–45.9% for the baselines), and it is the only method that stops revealing the target without fabricating: only 26% of its non-leaking answers are hallucinated, while the rest are non-fluent output (67.1% of all answers). CRU's absolute hallucination rate (24.6%) is also lower than the original model's (31.3%). We consider fluent fabrication the most harmful failure in practice, because a user cannot distiguish it from a true answer, whereas CRU's output is conspicuously not an answer. We added this analysis to the paper (Table 1 and lines 370-378) and clarify that the vulnerability is slight because of our black-box threat scenario.