acceptodds
Under review as a conference paper at ICLR 2027

Causal Routing for Unlearning

Abstract

LLMs cannot forget the way we delete a file. Strangely, we are asked to remove something that was never put anywhere in particular. What the model took from a piece of text is now smeared across billions of weights. Existing methods rewrite all of them to change one thing, and none of them say which part produced that change. To address this, we introduce Causal Routing for Unlearning (CRU) by asking where the concept is expressed in the model and suppressing only that part. One untrained forward pass over the forget set ranks neurons by how their activations vary. Then, small routing modules on those neurons gate and suppress only the concepts that need to be forgotten. In CRU, the base model is frozen, and any change in behavior is caused only by the gated neurons; hence, why the routing is causal. Due to our parameter efficiency (only % as many parameters as the base model), unlearning a concept costs GiB, whereas the baselines require GiB. On TOFU, CRU is indistinguishable from the retained model (, KS test) and is never Pareto-dominated, whereas every compared baseline matches its forgetting on the larger-forget batches only by collapsing utility. On RWKU, it achieves an adversarial-probe recall of , compared to for the strongest baseline, meaning the knowledge is gone, not merely harder to reach. Thus, deciding on the intervention at query time, rather than fixing it beforehand, is the axis along which we argue that unlearning should proceed.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine unlearning, large language models, knowledge localization, conditional computation, parameter-efficient methods

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_BpyfOpenReview ↗
Rating
4(+2)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes Causal Routing for Unlearning (CRU). It selects hidden-state dimensions using activation variance on forget data, then trains small routers to suppress these dimensions depending on the input. The base model remains frozen. Experiments on TOFU and RWKU show promising forgetting–utility trade-offs and low training memory costs.

Critical strengths
  1. The method is simple and memory-efficient. Training only small routers makes the approach practical. On Phi, the reported TOFU training memory drops from about 71 GiB for the evaluated weight-update baselines to 14 GiB.

  2. The empirical results are promising. TOFU experiments cover four model families. On RWKU, CRU achieves adversarial-probe recall of 0.052, compared with 0.250 for the strongest evaluated baseline. The random-selection ablation also supports the value of the localization step.

Critical weaknesses
  1. The claim of knowledge removal is not supported by the evidence. Section 5.1 claims that CRU guarantees removal rather than hiding. However, Appendix A.2 explicitly states that removing the routers exactly restores the original model. The RWKU results show that the tested prompts extract less knowledge. They do not establish that the knowledge is gone or that recovery is difficult. This distinction is central to the paper’s motivation and conclusions.

  2. The comparisons do not establish the benefit of learned routing. NeuMuter, CAST, DSG, and GUARD-IT are discussed but not evaluated, despite being closely related to CRU. Most comparisons instead use standard weight-update methods, several of which suffer utility collapse. These results do not show whether learned routing improves over existing gating methods at comparable utility. Matched comparisons with the closest alternatives are needed to establish the main contribution.

Questions for authors
  1. What access is the attacker assumed to have? With the routers kept active, can target knowledge be recovered through prompts designed to bypass the gates or a small amount of fine-tuning?

  2. Figure 5 reports a membership-inference AUC of 0.058. Under the standard ROC-AUC definition, reversing the scores gives 0.942, indicating strong separation. How should this result be interpreted? Why is the increased vulnerability described as only slight?

  3. The evaluated model families are relatively old. Qwen2.5 was released in September 2024, about two years ago. Can you provide results on Qwen3.5, especially on RWKU, to show that the main findings extend to current models?

  4. Several closely related works explore similar ideas of localization, routing, and gating:

    • Gradient Routing: Masking Gradients to Localize Computation in Neural Networks (arXiv 2024)
    • Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearning (ICLR 2026)
    • Preemptive LLM Unlearning against Forbidden Capability Acquisition via Gradient Sealing (arXiv 2026)

    Please cite these works and clarify how CRU differs from them in both motivation and methodology, particularly in how target-related neurons are identified and how routing or gating is used to control their influence. These distinctions are important for understanding the actual novelty of CRU.

I would be happy to raise my score if the authors address these concerns with clear explanations and supporting results.

3 comments

Thanks for the positive assessment on the readability of the paper and the highlighted strengths. As some of the weaknesses identified are closely related to some of the questions, we answer them together for clarity. All changes discussed here are implemented in the revised paper. Specifically, for changes spanning from reviewer Bpyf, the edits are highlighted in ORANGE.

W1. The claim of knowledge removal is not supported by the evidence.

We indeed train only a fraction of the parameters to perform unlearning: i.e., the routers of Fig. 2. It is also true that if we remove the trained routers (just like steering approaches do) one can recover the base model prior to the unlearning process (as per Appendix A.2) for auditing purposes only. Nevertheless, our threat scenario is black-box: i.e., the routers are deployed alongside the backbone model and a potential malicious attacker can only perform queries to elicit knowledge, which we show does not work. This is visible in Figure 5 (top row, L1/L2/L3) where CRU resists even more sophisticated (higher level) attacks, whereas all baselines, besides SimNPO in some cases, show vulnerabilities to the same attacks. This also answers your question Q1, to which we add the following.

For fairness of comparison, we fully comply with RWKU's attack schema. RWKU gives the attacker full query access and per-token likelihoods with nine families of adversarial prompts. The gate-bypassing prompts you describe are exactly the Level 3 probes test from RWKU (comprising adversarial prompts and probes, e.g., cross-lingual, synonym, reverse-query, in-context...). We note that CRU's forget retention does not rise under these attacks. For contrast, SimNPO's rises significantly on both models. To our knowledge, no benchmark tests attacks adapted to an inference-time component, and none of TOFU or RWKU evaluates fine-tuning recovery, which would be an interesting research avenue.

To avoid any confusion, in lines 363-366 of the revised paper we clarify this attack schema introduced in the RWKU benchmark.

Q2. Figure 5 reports a membership-inference AUC of 0.058 ... Why is the increased vulnerability described as only slight?

In RWKU's MIA, the attacker scores text about the target (FM) and that of unrelated subjects (RM) by Min-K% likelihood, and the AUC measures how well the two sets can be separated. If CRU merely hides the knowledge while still encoding it, the target text would remain more likely than unrelated text, giving an AUC at or above the original model's ($\sim$0.45). We observe the opposite, namely that CRU has lower likelihood for the target text than text about subjects it was never asked to forget. The distance from chance therefore reflects suppression of the target, not residual knowledge. It makes the forgotten subject detectable, but it is not evidence that the knowledge is still present.

Although MIA can tell that the forget set was part of the unlearning process with CRU, we argue that in a black-box scenario this is irrelevant. Moreover, we want to emphasize that, on forget prompts, CRU produces non-fluent output (gibberish) that makes no claims about the target, whereas the SoTA baselines produce fluent but fabricated statements on such a target.

To further support this interpretation, we used Claude Opus 5.5 to classify each answer to the RWKU forget probes (on Phi-3) as leaked (still revealing knowledge about the target), hallucinated, or gibberish. Every baseline that reduces leakage does so by replacing the leaked answers with fabricated ones: when GA, PURGE, and NPO do not leak, the answer is hallucinated 96-97% for GA, PURGE, and NPO, and 88% for SimNPO. CRU leaks least (6.7%, vs. 67.6% for the original model and 15.3–45.9% for the baselines), and it is the only method that stops revealing the target without fabricating: only 26% of its non-leaking answers are hallucinated, while the rest are non-fluent output (67.1% of all answers). CRU's absolute hallucination rate (24.6%) is also lower than the original model's (31.3%). We consider fluent fabrication the most harmful failure in practice, because a user cannot distiguish it from a true answer, whereas CRU's output is conspicuously not an answer. We added this analysis to the paper (Table 1 and lines 370-378) and clarify that the vulnerability is slight because of our black-box threat scenario.

MethodLeakedHallucinatedGibberishHallucinated / non-leaked
Base67.6%31.3%0.5%97%
GD67.6%31.3%0.5%97%
GA45.9%52.1%1.6%96%
PURGE44.0%53.6%1.6%96%
NPO37.8%60.6%1.3%97%
SimNPO15.3%74.1%10.1%88%
CRU6.7%24.6%67.1%26%

W2. NeuMuter, CAST, DSG, and GUARD-IT are discussed but not evaluated

In the revised paper we compare with GUARD-IT and NeuMuter on both TOFU settings. The table below shows that CRU wins across all axes for both settings on Llama and Qwen, forgets the most on both settings for DeepSeek, and for Forget10 on Phi.

ModelMethodForget01 ForgettingForget01 MUForget10 ForgettingForget10 MU
LlamaGUARD-IT0.110.510.150.51
NeuMuter0.000.510.010.51
CRU0.670.550.210.53
QwenGUARD-IT0.840.480.780.47
NeuMuter0.010.480.010.48
CRU0.980.500.990.53
DeepSeekGUARD-IT0.470.490.490.49
NeuMuter0.390.480.380.47
CRU0.570.350.660.36
PhiGUARD-IT0.860.460.710.45
NeuMuter0.710.170.420.46
CRU0.980.140.970.23

However, CAST cannot be benchmarked fairly. In the paper, the authors claim that they "manually search for the appropriate intervention strength and layers" and that "there is not a universally applicable range for the grid search". They have not released any setting for the models we test. Similarly, DSG needs a pretrained sparse autoencoder for the exact model it runs on, and there are no publicly available ones for the majority of our models.

With these baselines, CRU is now compared against 9 SoTA methods, that is substantially stronger and more comprehensive than any previous setting in the literature.

MethodPareto-optimal panels
CRU8/8
GA5/8
GUARD-IT4/8
NPO3/8
SimNPO2/7
GD, ICU, NeuMuter0

Finally, as the table shows, CRU is the only method that is never pareto-dominated by any method, even with the addition of GUARD-IT and NeuMuter.

Q3. provide results on Qwen3.5, especially on RWKU

RWKU publicly releases forget corpora only for Llama-3-8B and Phi-3, which is why we and prior work use them. However, following your suggestion, we are doing our best effort to regenerate RWKU's forget corpus for Qwen3.5-4B using the benchmark's own scripts and run all methods with the same tuning protocol.

Re-running every method on a new model is very costly and takes a lot of time. We are still waiting for results; we will update you when they are done.

Q4. Several closely related works explore similar ideas

Thanks for these references. The revised paper adds 2 out of 3 of the suggested references; the third, published on arXiv, is concurrent to our work as per ICLR's policy ("Papers published on or after July 17, 2026 are considered contemporary relative to the September 16, 2026 deadline"). While they share vocabulary with CRU (localization, routing, gating), they differ in the problem addressed and in what is routed or gated.

  1. Gradient Routing [1] masks gradients during training so that a user-specified capability is localized in a pre-chosen subregion; it does not act on an already trained model. CRU instead keeps a trained model frozen and gates its forward activations per token at inference time.

  2. Ssiuu [2] adds an attribution-based regularizer to full-parameter unlearning so that methods erase knowledge rather than hide it behind inhibitory neurons that retraining can disrupt; it involves no localization-then-gating step. CRU is also a suppression mechanism, but an explicit one separate from the weights. Since Ssiuu targets retraining attacks (weight access) while CRU is designed for black-box deployment, we state this scope explicitly and list robustness to retraining as a limitation in the revised paper (lines 538–539).

  3. GSU [3] retrains all parameters to saturate selected SwiGLU gates, so that an open-weight model cannot acquire a capability through future fine-tuning; it does not remove knowledge already present and adds no gating module.

In summary, none of these methods keeps the base model frozen and learns an input-conditional gate that decides at query time whether target-related units pass, which is the axis along which we position CRU (§1, Fig. 1). We add this discussion to the related work section (lines 141-143, 150-153, Figure 1 has the 2 new methods).

[1] Gradient Routing: Masking Gradients to Localize Computation in Neural Networks.

[2] Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearning

[3] Preemptive LLM Unlearning against Forbidden Capability Acquisition via Gradient Sealing

Thank you for the additional experiments and the thoughtful responses. I appreciate the effort and have raised my score to 3. The added NeuMuter and GUARD-IT comparisons largely address my concern about missing close baselines, and the clarifications on the threat model and related work are helpful.

Two points remain open for me: (1) The results support suppression under the tested black-box probes, but not necessarily the stronger claim that the knowledge is “gone”. More cautious wording would help, especially since attacks tailored to the routers remain untested. (2) The MIA interpretation could use further clarification: reversing the scores turns an AUC of 0.058 into 0.942. While this does not establish knowledge recovery, the privacy implications of distinguishing forgotten targets still deserve consideration, even under black-box access.

Good luck!

Reviewer_tjDnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Causal Routing for Unlearning (CRU), a method that targets unlearning in frozen LLM backbones by gating the localized concepts at later layers. CRU operates in 2 stages: concept localization, which aims to determine relevant activations via a high-variance surrogate over the forget set, and learnable gating factors, which suppress these activations to induce unlearning. Since the gating is input-dependent and localized rather than fixed globally, CRU reduces impact on model utility over the retain set. CRU is evaluated on 2 datasets, TOFU and RWKU, over 4 model families and demonstrates strong performance over existing methods while also reducing parameter usage.

Critical strengths

I think the overall idea of CRU is sufficiently novel by generalizing the gating to be input-dependent and localized, which addresses some limitations of existing methods. For example, the assumption that only specific activation positions encode certain knowledge could be quite strong. The experiments and performance seem favorable for CRU, as it can retain model utility while inducing significant unlearning across several model families and datasets compared with existing methods.

Critical weaknesses

W1. I don't find the paper well-written, nor does it express the idea clearly and consistently. I will list some apparent points here. However, this is not an exhaustive list.

  1. Some points in the abstract and introduction seem to conflict with the methodology. For example, the abstract states "piece of text is now smeared across billions of weights" (line 11) and the introduction states " cannot forget (unlearn) the way we delete a file" (line 31), but then the authors proceed to localize the fact to the hidden states (line 64-77) and claim "indistinguishable" (line 21) unlearning from the retained model and "knowledge is gone" (line 24).
  2. The motivation is not clearly explained. For example, what specific use cases make small/medium-sized lab/company want to unlearn concepts from the LLMs, which makes retraining infeasible (line 39). The GDPR example is intuitive, but this one is not.
  3. What does it mean to say "concept that merely passes nearby"? (line 55). What does it mean to "intervene on where the fact is expressed/retrieved"? (line 68-70) I suppose this means the weight and activation (or hidden states). But then it would also suggest CRU does not target general neurons (line 70-74) but rather the activation values of the output neurons only, i.e., the output MLP of the block. Fig. 2 also suggests that CRU modifies the activations.
  4. In related work (line 132), DPO is a preference optimization method that can be used to train a refusal pattern, but not an unlearning method by itself. The sentence makes it look like an unlearning method.

W2. I don't find that some of the claims are correct or sufficiently backed. Similarly, below is an inexhaustive list of observations:

  1. The author mentions NeuMuter, CAST, and DSG as a special case of their formulation (lines 775-785), but does not provide comparisons with any of these methods or provide any justification. These methods seem to be the most closely related to CRU instead of the compared methods, i.e., GA, NPO, etc.
  2. The term "causal" in the name seems to be forced. There are many techniques where we modify inputs/weights and observe output changes, but we don't usually term these procedures causal. The localization also uses a surrogate and thresholding, which makes isolating causal effects less sound.
  3. Proposition 1 does not seem really useful as it provides little new insight. Also, ww seems to be the output of a sigmoid in Eq. 7, which cannot be smaller than 0. I suppose the authors' argument can be demonstrated more convincingly by a simple histogram of the gate value between the forget set and retain set.
  4. The claim of guaranteed knowledge removal (line 323) requires stronger evidence. The result on membership inference seems to suggest CRU effectively hides the knowledge rather than remove it.
  5. The computation comparison only includes methods that require fine-tuning, which should generally incur higher cost, and does not compare against frozen backbone methods. For example, NeuMuter is mentioned in the paper but is not compared against in terms of computation.
  6. The ablation claims the number of shared neurons is small (line 420-429), but the numbers do not look small to me, i.e., 27% to 59% with Spearman correlations from 0.18 to 0.46.

While the proposed method seems to be performant, it appears to me that the manuscript is not ready due to many clarity and consistency issues, as well as mischaracterized claims. However, I may miss some context or misunderstand details that justify some of the authors' arguments. I will gladly correct my evaluation upon the authors' valid clarifications.

Questions for authors

Please take a look at the listed concerns in the Critical Weaknesses section. If the majority of the listed concerns come from my misunderstanding of the paper, then I will gladly correct my evaluation. Otherwise, the required modifications seem to be quite substantial, and I cannot suggest an acceptance score in this case.

4 comments

Thanks for engaging with our work so substantially and for the in-depth review. We want to clarify that all changes discussed here are made available in the revised PDF. Specifically, for changes spanning from reviewer tjDn, the edits are highlighted in RED.

In this response, we highlight the main misunderstandings and discuss how the revision takes them into account.

W1.1: Some points in the abstract and introduction seem to conflict with the methodology.

Our intention is to state that specific knowledge or facts of modern LLMs (during pre-training) are spread over billion parameters. Identifying where such knowledge is located is challenging. On the other hand, deleting a file is easy because we can search for it in a filesystem. Yet, we agree that our previous phrasing is exagerated. As such, we amended the text in lines 25-26.

W1.2: what specific use cases make small/medium-sized lab/company want to unlearn concepts from the LLMs, which makes retraining infeasible (line 39)

Retraining an LLM requires significant hardware and financial resources; no small/medium-size lab or company can afford such resources. As such, unlearning, i.e., modifying already pre-trained models, is the only viable solution.

W1.3: What does it mean ... "concept that merely passes nearby"? ... "intervene on where the fact is expressed/retrieved"? (line 68-70) ... it would also suggest CRU does not target general neurons (line 70-74) but rather the activation

  1. Passes nearby. We mean a prompt that is topically or lexically related to the forget target (e.g. Paris) without explicitly requesting the fact to be forgotten (e.g., a question about Paris neighborhoods). Input-independent masks suppress on all inputs, and threshold rules fire whenever similarity crosses τ\tau. CRU's gate is learned and evaluated per token. We rephrased line 55 (now 56-58) accordingly.

  2. Retrieved vs. expressed. This distinction concerns depth, not weights vs. activations. Causal tracing (Meng et al., 2022) localizes recall of factual associations in middle-layer MLPs. CRU acts in the last kk blocks, where that information is carried toward the output head. Independently, CRU acts on activations, not weights (θu=θo\theta_u = \theta_o). Figure 14 supports the depth choice: gating the first or middle blocks forgets equally well but reduces model utility from 0.53 to 0.20/0.16. We separated these two points in the text, see lines 68–73.

  3. Neurons. You are right. CRU's units are coordinates of each block's output hidden state, i.e., dimensions of the residual stream (dd = model width), not MLP intermediate neurons. This matches Eqs. 4, 7, and 8 and our implementation, where both the variance statistic and the gate are computed on the decoder layer's output via a forward hook (see concepts.py and routing.py). Note that this is not the block's MLP output: hlh_l is the full residual stream after block ll, so the gate scales everything written to the selected directions up to that point. We gate there deliberately, because it makes the construction independent of each family's feed-forward design. This is why a single implementation runs unchanged across all four model families. NeuMuter, by contrast, masks FFN intermediate neurons. We keep the term "neuron" for readability (lines 207–208) but now define it explicitly at first use as a hidden-state dimension, and clarify that the localization literature we cite (Dai et al., 2022; Geva et al., 2021) motivates our choice of layers rather than of units.

W1.4: DPO is a preference optimization method

Thanks. We fixed it (lines 45-46, 132-135).

W2.1: The author mentions NeuMuter, CAST, and DSG as a special case of their formulation (lines 775-785), but does not provide comparisons

Thanks for the suggestion. We added experiments for GUARD-IT and Neumuter on TOFU Forget01 and TOFU Forget10. We present the results in the response W2 to reviewer Bpyf. In a nutshell, CRU wins across all metrics and settings for both Llama and Qwen, and forgets the most on Deepseek for both settings.

MethodPareto-optimal panels
CRU8/8
GA5/8
GUARD-IT4/8
NPO3/8
SimNPO2/7
GD, ICU, NeuMuter0

As the table shows, CRU is the only method that is never pareto-dominated by any method, even with the addition of GUARD-IT and NeuMuter.

However, CAST cannot be benchmarked fairly. In the paper, the authors claim that they "manually search for the appropriate intervention strength and layers" and that "there is not a universally applicable range for the grid search". They have not released any setting for the models we test. Similarly, DSG needs a pretrained sparse autoencoder for the exact model it runs on, and there are no publicly available ones for the majority of our models.

The addition of these two baselines raises our experimental evaluation to 10 methods, including CRU. This is substantially stronger than what we have seen in the literature, and provides a robust evaluation of CRU against the SoTA.

W2.2 The term "causal" in the name seems to be forced.

We agree. We requested changing the title from Causal Routing for Unlearning to Conditioned Routing for Unlearning after abstract submission, since we concluded that the name could mislead researchers in the field of causal reasoning or causal graphs. Yet, we received no answer. As such, we modified the title in the revision.

W2.3: Proposition 1 does not seem really useful as it provides little new insight.

Proposition 1 was put as a sanity check to analytically show that there must be a difference between the retain and forget ww-s. As suggested, we added per-token gate histograms (Fig. 3). The two distributions sit at opposite ends, with almost no mass near the 0.5 optimum. We replaced the old Figure 3 since it just showed what we're currently showing with the new histogram (we keep however the derived averages ±\pm stds since we refer to them in other places in the paper).

W2.4: The claim of guaranteed knowledge removal (line 323) requires stronger evidence.

We substituted "guarantees" with prevents recovery of the target under black-box access (line 360-369), since we do not provide a formal guarantee.

The result on membership inference seems to suggest CRU effectively hides the knowledge rather than remove it.

We believe the membership-inference result supports a different interpretation than the one provided by the reviewer.

In RWKU's MIA, the attacker scores text about the target (FM) and text about unrelated subjects (RM) by Min-K% likelihood, and the AUC measures how well the two sets can be separated. If CRU merely hid the knowledge while the model still encoded it, the target text would remain more likely than unrelated text, giving an AUC at or above the original model's ($\sim$0.45). Instead, we observe the opposite: an AUC of 0.058, which reveals that the model now assigns target text markedly lower likelihood than text about subjects it was never asked to forget. The distance from chance therefore reflects suppression of the target, not residual knowledge of it.

This difference also shows in what the models generate. On forget prompts, CRU produces non-fluent output (gibberish) that makes no claims about the target, whereas the SoTA baselines produce fluent but fabricated statements about it.

To further support this interpretation, we used Claude Opus 5.5 to classify each answer to the RWKU forget probes (on Phi-3) as leaked (still revealing knowledge about the target), hallucinated, or gibberish. When GA, PURGE, and NPO do not leak, the answer is hallucinated 96-97% for GA, PURGE, and NPO, and 88% for SimNPO. CRU leaks least (6.7%, vs. 67.6% for the original model and 15.3–45.9% for the baselines), and it is the only method that stops revealing the target without fabricating: only 26% of its non-leaking answers are hallucinated. CRU's absolute hallucination rate (24.6%) is also lower than the original model's (31.3%). We consider fluent fabrication the most harmful failure in practice, because a user cannot distiguish it from a true answer. We added this analysis, including the following table, to the paper (Table 1 and lines 370-378).

MethodLeakedHallucinatedGibberishHallucinated / non-leaked
Base67.6%31.3%0.5%97%
GD67.6%31.3%0.5%97%
GA45.9%52.1%1.6%96%
PURGE44.0%53.6%1.6%96%
NPO37.8%60.6%1.3%97%
SimNPO15.3%74.1%10.1%88%
CRU6.7%24.6%67.1%26%

W2.5 The computation comparison ... does not compare against frozen backbone methods.

We compare our running time on TOFU against the frozen-backbone baselines. The results, reported in the table below, show that CRU is much faster than Neumeter on all models and settings, and remain competitive towards GUARD-IT while scoring significantly higher in Utility and Forgetting on almost all models (see Table of results to W2 of reviewer Bpyf.).

ModelMethodForget01 · Minutes ↓Forget10 · Minutes ↓
DeepSeekGUARD-IT1.51.5
NeuMuter50.8507.4
CRU15.815.0
LlamaGUARD-IT1.61.6
NeuMuter57.5577.6
CRU16.216.6
PhiGUARD-IT1.11.1
NeuMuter32.2321.8
CRU9.79.7
QwenGUARD-IT1.41.4
NeuMuter54.7546.0
CRU14.414.5

W2.6 The ablation claims the number of shared neurons is small (line 420-429)

We agree that our wording overstates the result. At p=95p = 95, each selection keeps 5% of the dimensions per layer, so two unrelated selections would overlap by about 5%. The observed overlap is 5–12$\times$ higher:

Model∣Cf∣\lvert C^f \rvert∣Cf∩Cr∣\lvert C^f \cap C^r \rvertOverlap× chance
Llama8202210.275.4$\times$
DeepSeek8203150.387.7$\times$
Phi6162800.469.2$\times$
Qwen 7B7204220.5911.7$\times$

Counts are summed over the four gated layers (TOFU forget01, p=95p = 95).

A substantial fraction of the selected dimensions therefore reflects generic, input-independent variance rather than the forget set. We removed "only a minority" and "the vast majority are specific," and now report the overlap together with its chance level. This does not undermine the method, because CRU does not rely on Phase 1 for specificity. As stated in §1, Phase 1 only restricts where suppression may occur; when a selected dimension is suppressed is decided per token by the router, whose objective penalizes closing gates on retain inputs (Eq. 9). Shared dimensions therefore cost utility only if the router closes them on retain inputs, and it largely does not: mean gate values are 0.01–0.04 on forget inputs versus 0.79–0.85 on retain inputs (Fig. 3). The overlap analysis thus shows that variance-based selection alone is not concept-specific, which is precisely why the gating stage is needed. See lines 459-461, 474-485, 531-533, and D3.1.

Thank you, authors, for your significant effort in revising the paper and addressing my concerns. While I appreciate the effort in the revision, based on the authors' response, I'm afraid my initial concerns about the weaknesses are justified, i.e., not a misunderstanding. The changes to the revision are therefore quite substantial and cannot warrant an accept score, as mentioned in the initial review. For W1, the concerns are partially addressed, although I still find that the paper has some inconsistent or exaggerated claims. For W2, I appreciate the addition of NeuMuter and GUARD-IT in several results. I do not agree with the authors on their interpretation on the membership inference result, as the removal of knowledge should have similar behavior to a model trained without that data, instead of generating gibberish on the data. The "CRU is cheap" claim is still not sufficiently backed up, as the memory consumption is only compared to non-frozen-backbone methods.

Reviewer_AWMnOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a machine unlearning method, Causal Routing for Unlearning, which aims to conditionally suppress the output of neurons associated with the forget set based on the activations computed during inference. This has two steps: first, they measure which neurons are most related to the forget set using output variation, selecting the top pp percentile with highest variation for suppression. Second, they train a small MLP for each affected block, which is applied to each vector in the activation's input sequence and produces a gating score g∈[0,1]g \in [0,1] which is multiplied with each target neuron. All MLPs are jointly trained with the model frozen according to a loss function which seeks to minimize the mean gating values on the forget set and maximize the mean gating values on the retain set.

The primary contribution is the conditional intervention based on the activations observed during inference. CRU is evaluated on two widely-used unlearning benchmarks, TOFU and RWKU, and is favorably compared with eight baseline unlearning methods.

Critical strengths
  • CRU's conditional activation suppression is novel, and tying it to the inputs at inference time is appealing.
  • CRU has strong unlearning performance on both benchmarks, which is good to see from any proposed method.
Critical weaknesses

None

Questions for authors

Do you have any insight as to why the runtime of CRU is slow compared to most other baseline methods? Given how few trainable parameters each routing module has, I would expect it to be faster.

Reviewer_Uvw2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presented Causal Routing for Unlearning (CRU), a parameter-efficient method for suppressing targeted knowledge in LLMs while keeping the base model frozen. CRU identifies candidate neurons using activation variance fist. Then, CRU trains small, input-dependent routing modules to selectively suppress their activations. Experiments demonstrate a good balance between forgetting and retained utility. CRU also improved resistance to adversarial knowledge-recovery probes. Requiring approximately 0.01% additional parameters, CRU reduces reported a clear memory usage reduce.

Critical strengths
  • Parameter-efficient: Trains small routing modules while keeping the base model frozen
  • Memory-efficient: Requires substantially less GPU memory than the compared training-based baselines
  • Strong results: Shows a favorable forgetting–utility balance
Critical weaknesses
  • Section 2 is very redundant.
  • The dataset used in this paper is also very limited and out-dated.
  • There are some critical references missed. "Learning-Time encoding shapes Unlearning LLMs", "Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness".
Questions for authors

I am still concerned about why this approach works. Maybe more experiments can help. Also please read the recommended two papers as references: "Learning-Time encoding shapes Unlearning LLMs", "Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness"

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment

General comment

We thank the reviewers for their engagement with our work. We have uploaded a revised PDF of our paper that includes the suggested changes marked using the following color codes:

  • teal for reviewer Uvw2
  • blue for reviewer AWMn
  • red for reviewer tJdn
  • orange for reviewer Bpyf
  • brown for reviewers tjDn and Bpyf combined, since some changes were shared.

We would also like to note that experiments for LLM Unlearning are time-consuming and computationally intensive. Yet, with the implemented suggestions, CRU is now tested across 4 models, totaling 9 families, 2 benchmarks, and 4 scenarios. This is a substantially more comprehensive evaluation than any other paper in the LLM unlearning literature.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.