acceptodds
Under review as a conference paper at ICLR 2027

Inference-Time Machine Unlearning via Gated Activation Redirection

Abstract

Large Language Models (LLMs) memorize vast amounts of training data, raising concerns regarding privacy, copyright infringement, and safety. Machine unlearning seeks to remove the influence of a targeted forget set D_f while preserving model performance, ideally approximating a model retrained from scratch without D_f. Once an LLM is in use, every new request to make it forget specific content demands updating its weights. However, unlearning through parameter updates is expensive, hard to audit, and can be undone by quantization. We show that unlearning can be enforced entirely at inference time, without training, gradients, or weight changes. We introduce Inference-Time Unlearning via Gated Activation Redirection (GUARD-IT), a training- and gradient-free method that unlearns via input-dependent activation steering at inference time. guard stores the content to be forgotten as a small library of activation directions, and during inference, it routes each query through a similarity gate that activates only for relevant directions and applies them as a norm-preserving rotation of the residual stream, while unrelated queries pass through the unmodified model. The same design carries across three model families and nine checkpoints from 0.8B to 8B parameters, and new forget requests are absorbed by one offline pass of forward passes. On TOFU, against 16 gradient-based and inference-time baselines, and on MUSE and WMDP, GUARD-IT forgets without breaking the model, and on TOFU it is the only method that suppresses memorization in every Llama configuration without collapsing: it keeps utility and fluent generation in every configuration, moves the model's output distribution closest to a model that never saw the forgotten data on the forget01 split, survives 4- and 8-bit quantization and ten sequential forget requests, and holds under jailbreak attacks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine unlearning, inference-time adaptation, activation steering, representation editing and engineering, residual stream intervention, gated networks, model privacy, data deletion, LLM safety, interpretability, activation space geometry, activation engineering, parameter-efficient methods, quantization robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2Qs8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Paper Summary

This paper proposes GUARD-IT, a training- and gradient-free method that performs LLM unlearning entirely at inference time via input-dependent activation steering, without modifying model weights.

  • Offline phase. The forget corpus is embedded with a sentence encoder (all-MiniLM-L6-v2) and partitioned by k-means, with the number of clusters selected by silhouette score. For each cluster, a Prototype Steering Vector (PSV) is computed as the mean residual-stream activation at an early layer (first quartile of the stack), and a reference direction is computed from the retain corpus.
  • Online phase. A similarity gate embeds each query with the same encoder and activates the clusters whose centroids exceed a cosine threshold. The active PSVs are averaged, projected orthogonally to the retain reference, rescaled to the activation norm, and subtracted from the hidden state as a norm-preserving rotation with strength α\alpha. Queries that do not activate any cluster pass through the unmodified model.

Claimed contributions:

  • a formulation of unlearning as norm-preserving, input-gated steering whose strength reduces to a single scalar calibrated per model;
  • a similarity gate over clustered PSVs that enables input-dependent, compositional unlearning;
  • support for continual unlearning and robustness to quantization and jailbreak attacks;
  • a large-scale evaluation, together with newly released TOFU checkpoints for Gemma 4 and Qwen 3.5.

Validation. The main evaluation is on TOFU (forget01 and forget05) with Llama-3.2-1B, Llama-3.2-3B, and Llama-3.1-8B. It compares against 12 weight-editing baselines and 4 inference-time baselines, plus GRUN and a gated-refusal control that shares the same gate, using five seeds. Results are reported in terms of memorization, model utility, a gibberish score, and Forget Quality. The authors report that GUARD-IT is the only method that suppresses memorization without collapse in every Llama configuration, and that it achieves the highest Forget Quality on forget01. Additional experiments cover:

  • Gemma 4 and Qwen 3.5;
  • membership inference and seven jailbreak templates;
  • ten rounds of continual unlearning;
  • 4- and 8-bit quantization;
  • MUSE and WMDP-Cyber, compared against published baseline numbers;
  • ablations on the gate, clustering, pooling, steering-vector construction, layer, and coefficient.
Critical strengths
  1. Practical and appealing design for deployed models. GUARD-IT requires no training, gradients, or weight changes. The offline cost is small (minutes on a single GPU), and new forget requests are absorbed with forward passes only. Because forget requests are stored as external vectors, each request can be inspected, updated, or rolled back, which is attractive for closed-weight serving.
  2. Rigorous and large-scale evaluation. The TOFU comparison covers 16 baselines run under a single OpenUnlearning protocol, with five seeds and reported standard errors. It spans three Llama scales and two additional model families, and the authors release new TOFU checkpoints for Gemma 4 and Qwen 3.5, which is a useful contribution to the community.
  3. Informative controls and ablations. The gated-refusal control, which keeps the same gate but replaces the intervention, directly addresses whether the gains come from the gate alone. The ablations cover removing the gate and clustering, diff-means versus orthogonal projection, token pooling, a BM25 gate, paraphrase robustness, and layer and coefficient sweeps. Together they clarify the role of each component.
  4. Consistent results across settings. GUARD-IT avoids collapse in every reported configuration and maintains utility at the level of the original model. It achieves strong Forget Quality on forget01, and its behavior is stable across continual-unlearning rounds and quantization levels.
  5. Candid discussion of scope. The paper acknowledges that the forgotten content remains in the parameters, that the method targets closed-weight serving, and that the vector store must itself be protected. It also reports that an informed attacker can partially recover the membership-inference signal by inverting the decision rule.
Critical weaknesses
  1. The method performs gated inference-time suppression rather than unlearning, and the evaluation does not clearly separate the two.

    • The knowledge is never removed. The forgotten content remains fully intact in the parameters, and any query that does not fire the gate receives the original model's answer. The paper acknowledges this in its limitations. Nevertheless, the title, abstract, and comparisons frame GUARD-IT as machine unlearning on par with weight-editing methods.
    • The metrics measure the gated system. Mem and FQ are computed only on forget queries, with the gate active. They therefore measure the behavior of the gated system, not the knowledge in the model.
    • The gated-refusal control is weak. It only prepends an instruction to the prompt, which is not expected to shift the likelihood of the ground-truth answer. Its low Mem (0.13 / 0.14) therefore says little about whether the specific PSV directions matter.
    • Stronger controls are needed. Under the same gate, two controls would be more informative: (i) a norm-preserving rotation along a random direction of matched magnitude, and (ii) direct logit suppression of the answer tokens. These would show whether GUARD-IT's gains in Mem and FQ come from the learned forget directions or simply from any likelihood-altering intervention applied when the gate fires.
  2. The robustness of the gate is not adequately tested, and some robustness claims hold by construction.

    • Template jailbreaks already evade the gate. Wrapping queries in the seven jailbreak templates lowers the fire rate from 0.86 to 0.74. About a quarter of the wrapped forget queries therefore bypass the gate entirely and receive the original model's answer.
    • Natural evasion strategies are untested. These include cross-lingual queries (the gate uses an English sentence encoder), indirect references to the target (e.g., referring to an author through one of their books), splitting a question across conversation turns, obfuscated encodings, and adaptive attacks that optimize inputs to fall below the gate threshold.
    • Membership leakage is inverted, not removed. The MIA results show that the leakage signal is inverted rather than eliminated: an informed attacker recovers an AUC of about 0.75. This makes the claim that the method "holds under jailbreak attacks" too strong.
    • Quantization robustness is not a fair comparison. Robustness to quantization follows directly from not modifying the weights. Comparing it against weight-editing baselines whose edits can be undone by quantization does not demonstrate an advantage of the method itself.
  3. The evaluation design favors the gate.

    • Utility preservation is structural rather than measured. Model utility is computed on retain, real-author, and world-fact queries on which the gate almost never fires (false-positive rate 0.01). Utility therefore remains at the original level by construction, as the paper itself notes. What is not evaluated is the collateral damage when the gate does fire, for example on queries that combine a forget entity with general knowledge, or on retain authors closely related to forget authors.
    • Continual unlearning may suppress content that was never requested. Each round is evaluated on the full forget05 split, and Avg and Gibb are identical at rounds 6, 8, and 10 (0.62 / 0.81 at 1B, 0.61 / 0.83 at 3B). This implies that, by round 6, authors whose removal had not yet been requested were already suppressed. This sits uneasily with the reported 0.01 false-positive rate on retain authors from the same domain. Mem should be reported separately on requested and not-yet-requested subsets at each round.
    • Tuning effort is uneven. The layer and α\alpha are selected by sweeping on the forget01 evaluation metrics, and they are recalibrated for Gemma, Qwen, MUSE (α\alpha = 1.0 / 0.8), and WMDP. The baselines use OpenUnlearning's released configurations.
  4. Several results are less favorable than presented.

    • Fluency is a relative weakness. GUARD-IT's gibberish score (0.80–0.84) is below that of the Finetuned and Retain models (0.86–0.92) and below several non-collapsed baselines on forget01 at 1B (e.g., NPO 0.91, RMU 0.89, DPO 0.92). The claim of fluent generation relies on the paper's own collapse threshold (Gibb < 0.50).
    • Steered outputs contain fabricated facts. Many steered outputs in Table 9 contain fabricated specifics, such as an invented book title or a wrong parental occupation. This raises honesty concerns that the metrics do not capture.
    • MUSE results indicate over-unlearning. KnowMem is 0.7 (News) and 0.5 (Books), compared with the retrained reference of 33.1 and 28.9. Under MUSE's retrain-referenced criterion, this is over-unlearning, which the paper instead presents as suppressing knowledge "past" the reference. The main text reports News PrivLeak (−1.0) but not Books PrivLeak (−25.5, comparable to the baselines).
    • The WMDP evaluation is limited. It covers only WMDP-Cyber, where GUARD-IT's 26.47% accuracy is close to chance, and uses published baseline numbers. MMLU is reported only in aggregate. Overall MMLU is expected to be unaffected because the gate rarely fires; the relevant subsets (e.g., computer security, college computer science) are not reported.
    • Forget Quality does not pass on forget05. On forget05, Forget Quality remains below TOFU's threshold (1.4e-3 at 1B, 4.3e-3 at 3B), ranking fourth and third among the sixteen configurations.
  5. Limited novelty and missing related work.

    • The components are established. Input-conditioned steering (CAST, SADI), classifier-gated intervention (ECO), gated representation editing (GRUN), and norm-preserving rotation (Angular and Spherical steering) are all prior work. The contribution lies mainly in their combination and in the evaluation.
    • The most direct baselines are missing. Dynamic SAE Guardrails [1] combines a dynamic classifier with conditional activation-level intervention and reports robustness in sequential unlearning. This is essentially the same paradigm and is the most direct missing baseline. DRAGON [2] leaves the base model untouched, uses a lightweight detector to identify forget-relevant prompts, and explicitly targets continual unlearning. MLLMEraser [3] performs input-aware, training-free test-time unlearning via activation steering.
    • Positioning on suppression versus erasure. Since the paper explicitly leaves the forgotten content in the parameters, it should position itself within the discussion of suppression versus erasure [4, 5].
    • Other relevant work. The paper should also engage with work on honesty in unlearning [6], which bears on the fabricated outputs above, and on extracting unlearned information via activation steering [7].

[1] Aashiq Muhamed, Jacopo Bonato, Mona Diab, Virginia Smith. SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMs. COLM 2025.

[2] Yaxuan Wang, Chris Yuhao Liu, Quan Liu, Jinglong Pang, Wei Wei, Yujia Bao, Yang Liu. DRAGON: Guard LLM Unlearning in Context via Negative Detection and Reasoning. ICLR 2026.

[3] Chenlu Ding, Jiancan Wu, Leheng Sheng, Fan Zhang, Yancheng Yuan, Xiang Wang, Xiangnan He. MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steering. arXiv:2510.04217, 2025.

[4] Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, Mor Geva. Intrinsic Test of Unlearning Using Parametric Knowledge Traces. EMNLP 2025.

[5] Puning Yang, Junchi Yu, Qizhou Wang, Philip Torr, Bo Han, Xiuying Chen. Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning. ICML 2026.

[6] Renjie Gu, Jiazhen Du, Yihua Zhang, Sijia Liu. Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning. ACL 2026.

[7] Atakan Seyitoğlu, Aleksei Kuvshinov, Leo Schwinn, Stephan Günnemann. Extracting Unlearned Information from LLMs with Activation Steering. arXiv:2411.02631, 2024.

Questions for authors
  1. Controls for the source of the gains. Under the same gate and threshold, how do the following compare with GUARD-IT on Mem, FQ, and Gibb: (i) a norm-preserving rotation along a random direction of matched magnitude, and (ii) direct logit suppression of the ground-truth answer tokens? This would clarify whether the learned PSV directions are necessary, or whether any likelihood-altering intervention applied when the gate fires yields similar results.

  2. Gate evasion. How does the gate fire rate, and the resulting leakage, change under:

    • cross-lingual queries;
    • indirect references to the target (e.g., asking about "the author of [book title]" without naming the author);
    • questions split across multiple conversation turns;
    • obfuscated or encoded inputs;
    • adaptive attacks that optimize the input to fall below the similarity threshold?

    For the roughly 26% of jailbroken forget queries that bypass the gate, what is the leakage (e.g., ROUGE-L against the forgotten answer)?

  3. Collateral effects when the gate fires. Could you evaluate queries on which the gate fires but which also require retained knowledge? Examples include questions that combine a forget author with general facts, comparisons between a forget author and a retain author, and retain authors that are semantically close to forget authors. Could you also report the gate's false-positive rate separately for retain authors with high embedding similarity to the forget set?

  4. Continual unlearning. At each round, what is Mem on the requested subsets versus the not-yet-requested subsets of forget05? Given that Avg is identical at rounds 6, 8, and 10, does the gate already fire on authors whose removal had not yet been requested, and how is this reconciled with the 0.01 false-positive rate on retain authors?

  5. Hyperparameter selection and additional results.

    • Were the layer and α\alpha selected on the same forget set and metrics used for evaluation? How would results change with selection on a held-out split?
    • Were the baselines' hyperparameters tuned with comparable effort?
    • Could you report Books PrivLeak in the main text?
    • Could you report WMDP-Bio, as well as MMLU subsets closely related to the forget domain (e.g., computer security and college computer science)?
    • Could you compare against Dynamic SAE Guardrails [1] and DRAGON [2]?

Minor comments

  • Outdated references. Several cited works have since been published and should be cited by their venues, for example TOFU (COLM 2024), WMDP (ICML 2024), SimNPO (NeurIPS 2025; currently cited as a 2024 workshop paper), LUNAR (NeurIPS 2025), and ECO (NeurIPS 2024).
  • Honesty of steered outputs. It would help to quantify how often steered outputs contain fabricated specifics rather than non-committal answers (Table 9), and to discuss whether such fabrication is a desirable post-unlearning behavior [6].
  • Wording of distributional claims. The phrase "statistically closest to a model that never saw them" holds on forget01 but not on forget05, where FQ remains below the threshold. The main text should state this qualification explicitly.
5 comments

Thank you for the detailed review. It asked for many new controls, attacks, and comparisons, and we ran all of them, so this reply is longer than we would like. Each point is answered below, in the order of your review; each question is answered under the matching weakness. The revised PDF has the complete tables (your requests in purple, new tables numbered R1, R2, …).

1. Suppression rather than unlearning

We agree that GUARD-IT leaves the forgotten content in the weights, as the paper states in its limitations, but this does not place it outside machine unlearning. The standard benchmarks define unlearning through the model's behavior (TOFU's goal is "a model that is indistinguishable from one trained exclusively on the retain set"; Maini et al., 2024, Sec. 2.2.3), and Mem and FQ are computed from the model's outputs on forget queries for every method. For a weight-editing baseline, they measure the edited model; for GUARD-IT, they measure the model with its gate, which is what a provider serves. The same metrics evaluate all 16 baselines and the weight-preserving methods published as unlearning, including DRAGON and Dynamic SAE Guardrails, which you suggest as baselines. Whether unlearning should also be verified inside the parameters is a fair question; however, answering this question is akin to reframing the unlearning research area, which is outside the scope of this study.

Editing the weights does not guarantee removing the knowledge either. Your reference [4] finds that existing unlearning methods "minimally impact concept vectors and mostly suppress them during inference"; [7] extracts unlearned information through activation steering, and quantization restores suppressed content (Zhang et al., 2024b). In our experiments (App. N), 4-bit quantization lowers the Mem of UNDIAL from 0.60 to 0.40 and that of RMU from 0.15 to 0.03 on 8B forget05, while GUARD-IT reaches 0.60 at 4-bit (0.53 at full precision). Removal from the parameters would matter only against someone who holds the weights, and for an open-weight model the original checkpoint is already public: releasing an edited one does not withdraw the copies already downloaded, so the content stays retrievable; however, the new weights are modified. Removal can only be enforced where the provider holds the weights and serves the model; there, the served behavior, which Mem and FQ measure, is all users can reach. This is the setting GUARD-IT targets, as the introduction and Sec. 5 state, and Sec. 2 now positions it as suppression rather than erasure [4, 5].

We agree that the gated-refusal control only shows that the gate alone does not lower Mem, and that it cannot tell whether the PSV directions matter. We therefore ran both controls you proposed, with the same gate, centroids, and threshold as GUARD-IT (App. F.1). Util is the same for all edits in every cell, so the tables below report Mem, Gibb, and FQ.

Same strength (random direction with the PSV's norm):

Mem ↑ GUARD-ITMem ↑ randomGibb ↑ GUARD-ITGibb ↑ randomFQ ↑ GUARD-ITFQ ↑ random
1B forget01.68.50.80.767.7e-013.1e-01
1B forget05.63.52.81.671.4e-032.4e-02
3B forget01.66.43.84.779.2e-012.7e-01
3B forget05.57.42.83.784.3e-033.9e-04

Logit suppression (its strength raised until it reaches GUARD-IT's Mem, within 0.01):

Gibb ↑ GUARD-ITGibb ↑ logitFQ ↑ GUARD-ITFQ ↑ logit
1B forget01.80.817.7e-013.8e-01
1B forget05.81.811.4e-032.7e-02
3B forget01.84.759.2e-012.1e-01
3B forget05.83.814.3e-035.1e-04

GUARD-IT (Table 1) and random direction: means over five seeds, standard errors ≤ 0.02 (Table R4). Logit suppression: one run per strength, with Gibb and FQ interpolated at GUARD-IT's Mem.

The gate decides when to intervene, and the direction decides how much is forgotten and how cleanly. At the same strength, the PSV forgets more than a random direction in all four settings (Mem +0.11 to +0.23), produces more fluent answers (Gibb +0.04 to +0.14), and reaches a higher FQ in three of the four. The exception is 1B forget05, where the higher FQ of the random direction comes with lower Mem (0.52 vs. 0.63) and lower fluency (Gibb 0.67 vs. 0.81). The gains therefore come from the learned forget direction, and not merely from applying a likelihood-altering intervention whenever the gate fires. Logit suppression is a stronger reference than any deployable method, since it needs the gold answer at inference time, which a deployed system does not have. GUARD-IT reaches the same Mem without that information, with the same fluency at 1B, higher fluency at 3B, and a higher FQ in three of the four settings.

2. Robustness of the gate

We agree that the gate is the component an attacker has to evade, since a query that does not fire it reaches the unmodified model. Under the seven jailbreak templates, the gate still fires on 74% of the wrapped forget questions, and leakage, measured as ROUGE-L against the forgotten answer, falls from 0.63 with the gate off to 0.37 with it (Table 7). The 26% that bypass the gate are answered as by the original model (ROUGE-L 0.62 vs. 0.63), as you anticipated. We also ran the natural evasion strategies you listed and report leakage in free generation as ROUGE-L recall of the gold answer (lower is better; 1B forget05; App. D.3–D.4):

QueryFinetunedRetainGUARD-IT
Translated (mean of ES/DE/IT).25.22.15
Indirect reference.31.26.29
Split over two turns.57.36.33
One conversation per author (turns 2–20).51.39.17
Obfuscated (leetspeak, homoglyphs, typos, spacing).27.24.21
Encoded (base64).10.09.07
Adaptive paraphrases.41.32.41

GUARD-IT leaks less than the Retain model, which never saw the forget set, on five of the seven strategies. Three of these rows required adapting the gate, and none required retraining. For the translated queries, we translated the 200 forget05 questions and their answers into Spanish, German, and Italian; the default English encoder does not fire on them, so we rebuilt the library from the translated forget set with a multilingual encoder. With it, leakage falls below Retain in all three languages, Util stays at 0.60, and Avg rises from 0.55 to 0.59 (App. D.4). For the obfuscated and encoded queries, your question led us to add input normalization: the gate also checks a normalized copy of the query, and Table 7 and the 1B forget05 results of Table 1 are unchanged. The two remaining strategies show the limits of the gate. On indirect references, leakage stays between Retain and Finetuned (0.29 vs. 0.26 and 0.31). Natural paraphrases keep the gate firing (85% of TOFU's paraphrased questions vs. 86% of the original ones), but paraphrases optimized to fall below the threshold evade it and leak as much as the Finetuned model. Like translations, the paraphrases found this way can be added to the library, and Sec. 5 states this limitation.

We also agree that membership leakage is inverted rather than removed. The rotation raises the loss of forget queries above the holdout level, so an attacker who knows the defense and reverses the decision rule recovers an AUC of about 0.75, against 1.00 before unlearning. Sec. 3.5 now states this, and the abstract now says that GUARD-IT reduces jailbreak leakage instead of saying that it "holds" under jailbreak attacks.

Finally, we respectfully disagree that the quantization comparison is unfair. Quantization is common practice when serving LLMs, so evaluating unlearning on quantized models is a realistic test, and it is what a provider would face in deployment. That GUARD-IT's robustness comes from keeping the weights untouched does not make it less of an advantage: it follows from the method's design, and weight-editing methods lack it, since rounding partly undoes their edits (point 1). GUARD-IT keeps its effect at 4 and 8 bits, and at 8B it has the highest Avg at both precisions and on both splits (App. N). One full-precision extraction also serves the 4- and 8-bit models, with Avg within 0.01 of re-extraction at 1B and 3B (App. N.1). The fact that GUARD-IT is inherently robust to quantization is a non-trivial feature of our method: practitioners need explicit evidence that unlearning will work in realistic quantized scenarios, as we show in App. N (Tables 25–27).

3. Evaluation design

GUARD-IT's utility is measured the way unlearning benchmarks define it. TOFU and MUSE, two of the most widely used unlearning benchmarks, compute utility on queries kept apart from the forget set, and we evaluated GUARD-IT on both, on the same sets as every baseline. The weight-editing baselines lose utility on these sets, for example, NPO on 1B forget05 (Util 0.45 vs. 0.60; Table 1) and every gradient-based baseline on MUSE Books (at most 52.7, against 74.5 for the retrained model and 69.6 for GUARD-IT; Table 29). GUARD-IT keeps utility because its gate rarely fires on these queries, which is the mechanism we propose.

Neither benchmark covers the case you raise: a query on which the gate fires but which also needs retained knowledge. TOFU and MUSE contain no such queries, so none of the methods in Tables 1, 3, and 29 are evaluated on them. This is a limitation of how the benchmarks are built, so we added the two tests you suggest (App. D.2).

On the 10% of retain authors most similar to the forget centroids, the gate fires on 12.5% of their questions and on none of the other retain questions, and the probability of their gold answer drops by 4.4 points, against 0.6 for the other retain authors. On 150 compound questions that pair a forget author with a world fact, a retain fact, or a comparison, GUARD-IT answers the retained part correctly in 37% of cases, against 43% for the original model and 19% for ECO, the other gated method.

You were right to question the identical values in Table 5: they were an error, as GUARD-IT's row repeated the same values at rounds 6, 8, and 10. Re-run with five seeds, its Avg grows as requests accumulate (rounds 6, 8, 10: 0.56, 0.60, 0.62 at 1B; 0.54, 0.59, 0.61 at 3B) and stays the highest among non-collapsed methods. As you suggested, we report Mem after each round separately for requested and not-yet-requested authors (seed 0; App. Q):

Mem / round123456789
1B, requested.61.58.63.55.63.68.66.66.67
1B, not yet requested.09.11.10.11.11.09.10.12.26
3B, requested.55.54.59.49.58.62.61.60.63
3B, not yet requested.03.04.04.05.04.04.05.06.15

Without steering (α = 0 in the same runs), Mem is 0.09 at 1B and 0.03 at 3B. Authors not yet requested stay close to that level through round 8, so the gate does not suppress them, consistent with its 0.01 false-positive rate. At round 9, when a single author remains, their Mem rises to 0.26 and 0.15, still far below that of the requested authors, and at round 10 every author has been requested (Table 1).

We agree that the tuning effort should be comparable, and the two procedures are closer than they may appear. Layer and α were chosen on forget01; on the eight forget05 authors never used for that choice, the α ranking is the same and Mem is 0.05 lower than on the forget01 authors, so the selection carries over to held-out authors. The baselines use OpenUnlearning's configurations, which their authors tuned on TOFU, and for each one we kept the better of 5 and 10 epochs per model and split (Table 13). On a new model or benchmark, GUARD-IT recalibrates α (and, on WMDP, the layer) with forward passes only, one evaluation per candidate, and on Gemma and Qwen the layer follows the first-quartile rule without a new sweep (App. M, O). The MUSE and WMDP baselines are the published numbers on the same checkpoints, with the configurations chosen by their authors.

Reviewer_KRoJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors explore the idea of unlearning through activation steering. What makes their approach unique is that it requires no training or gradient updates. They leverage much of the recent work on activation steering (in things like sentiment, style and refusal control) and apply it to unlearning. The appeal is that the approach does not need any re-training and can be applied to existing models. This is also the drawback, as the information is essentially still there in the weights. Steering vectors are formed from the mean hidden activations associated with the information to be forgotten. They employ a gating mechanism (essentially k-means) on the forget set. Empirical results look interesting, especially on TOFU but drop off as the forgetting tasks get more complex.

Critical strengths

One of the big strengths of the paper is that it does not need training or gradient updates (unlike GRUN). It also incorporates some interesting ideas around semantic clustering, which allows for input-dependent combinations of multiple forgetting directions. It also employs an orthogonalization and norm-preserving activation steering which seems to help. Results seem quite promising especially on TOFU with less complicated tasks. Another interesting angle is their insensitivity to quantization, which is known to be an issue gradient-based weight unlearning.

Critical weaknesses

A negative to the paper is that it seems to be going down the well trodden path of activation steering for LLMs, with a modest improvement for the task of unlearning. I do agree that the lack of any training is nice, but the benefit seems small. Also, the approach is full of heuristics that are largely unprincipled. This is fine if it leads to significant empirical improvement, but that is not always the case. Their exploration of quantization effects and unlearning are interesting, but steering seems to give the same problem. You can essentially just remove the steering from the network and then have full access to the unlearned content. The authors do flag that their method is a suppression approach rather than permanent erasure, but it does weaken the quantization angle of the paper.

Questions for authors

It would be great if the authors could do a better job on what the quantization robustness could be practically used for, since their approach is essentially a suppression rather than permanent erasure technique? Also, the authors claim their approach does not need any training, but I would contend that k-means clustering etc. is a type of training just not gradient-descent. The big advantage for that they are doing is computational efficiency, not just that they do not perform training. It would be good to clarify if the authors agree with this, and if it could be made clearer in the paper motivation?

1 comment

We thank the reviewer for the careful review. Changes are in green in the revised PDF.

Improvement over activation steering We respectfully disagree that GUARD-IT is a modest improvement over activation steering. Table 3 already includes CAST, a conditional steering method: it has a lower Avg than GUARD-IT in all six Llama settings (e.g., 0.34 vs. 0.66 on 3B forget01) and lower utility (Util 0.39–0.59 vs. 0.60–0.67). Gating alone is not enough either: GUARD-IT's gate with a refusal instead of steering leaves Mem at 0.11–0.15, against 0.53–0.68. Against the 16 baselines in Tables 1 and 3, GUARD-IT is the only method that suppresses memorization in every Llama configuration without collapsing. Five of the 15 gradient-based configurations collapse on forget05 at all three scales, while GUARD-IT's Avg changes by at most 0.05 between splits (Table 1). This comes from how GUARD-IT selects what to suppress: one direction per cluster, from the model's own activations, minus the component shared with retained content, applied as a norm-preserving rotation only when the gate matches that cluster.

Gradient-free, and the benefit is cost (Q2) We agree that k-means is a form of training, so "gradient-free" is the accurate term; the revision uses it throughout, and the introduction now states cost as the benefit. At 8B on forget05, GUARD-IT's offline phase takes 10.6 minutes on one GPU with 20.3 GB of peak memory; the median gradient-based baseline takes 85.3 minutes on two GPUs with 45.6 GB (Table 11). Ten such requests would cost about 28 GPU-hours with that baseline and under 2 with GUARD-IT. Changing α needs no new run (App. C), and the gate adds under 5% per question (App. H.1).

Design choices Each component addresses one failure mode, and the revision ablates every one. Removing one at a time (Llama-3.2-1B forget05; App. D.1, Table R1):

Component removedUtil ↑Mem ↑Avg ↑Gibb ↑What breaks
Gate: steer every input0.06±.010.62±.000.11±.010.54±.02utility collapses
Clusters: one centroid, one PSV0.600.200.300.83the gate misses most forget questions
None (GUARD-IT, Table 1)0.600.63±.000.62±.000.81±.01–
  • Direction (App. F, F.1, J.1): diff-means, which keeps the retain component, loses fluency at high α; behind the same gate, a random direction forgets less and is less fluent (Mem 0.52 vs. 0.63, Gibb 0.67 vs. 0.81), and MLLMEraser's direction loses fluency.
  • Rotation (App. F.2): addition with the same direction and step loses fluency (Gibb 0.64 vs. 0.81).
  • Encoder, layer, and token position (App. B–D): BM25 matches the dense gate, and the layer and α are set by a sweep of forward passes.

The number of clusters is set by the silhouette score, with no tuning. These choices pay off: against the 21 baseline configurations of Tables 1 and 3, GUARD-IT has the higher Avg, or the baseline collapses, in 121 of 126 comparisons.

Suppression versus erasure, and quantization robustness (Q1) We agree that GUARD-IT suppresses rather than erases, but this is not the problem quantization creates for weight-editing methods. Quantization is a routine serving step, and it partly restores content that gradient-based unlearning removed (Zhang et al., 2024b; Abitante et al., 2026), so failure can happen without any attack. Removing GUARD-IT's steering requires a deliberate attack on the serving stack, where the vector store is protected like the weights. Sec. 5 states this setting, which GUARD-IT shares with other inference-time methods such as ECO, in-context unlearning, and DRAGON.

Anyone with the original weights recovers the forgotten content under any method, so benchmarks define unlearning by behavior. TOFU's Forget Quality asks whether the model is distinguishable from one never trained on the forget data; on forget01, GUARD-IT is the only non-collapsed method in Table 19 that passes this test (p = 0.77 at 1B, 0.92 at 3B).

We also respectfully disagree that suppression weakens the quantization result. In practice, providers routinely serve quantized models, and they need unlearning that survives quantization. GUARD-IT acts on the activations of the model actually served, so there is no weight perturbation for rounding to erase. Its 4- and 8-bit Avg agree within 0.01 in five of six settings, and at 8B it has the highest Avg at both precisions and on both splits (e.g., 0.65 vs. 0.51 for the best baseline at 4 bits on forget01; App. N, Tables 25–27). One full-precision extraction also serves both: on the 4- and 8-bit 1B and 3B models, the same vectors change Mem and Avg by less than 0.01 relative to re-extraction (App. N.1). Robustness to quantization is therefore a feature of the method, and App. N gives practitioners explicit evidence that unlearning holds in realistic quantized deployments.

If any point is still open, please let us know.

Reviewer_8sGTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an inference-time machine unlearning method that leaves model weights unchanged and redirects activations only for inputs related to the forget set. The method combines several modules such as semantic clustering, a similarity gate, and norm-preserving activation steering. Its main practical motivation is that new forgetting requests can be handled without repeated gradient-based model updates. This paper conducts extensive experiments covering multiple model families, inference-time baselines, continual forgetting, quantization, and adversarial inputs.

Critical strengths
  1. The paper provides several empirical findings. An interesting result is that simple topic-level routing can perform similarly to the learned semantic gate, suggesting that precise semantic understanding may not be necessary for selective activation steering. The gated-refusal experiment is also insightful because it separates suppressing an answer at the interface from changing the model distribution used by the unlearning metric.
  2. GUARD-IT offers a practical and modular design. Its clustered steering directions and norm-preserving intervention provide a lightweight mechanism for targeted forgetting without modifying model weights.
Critical weaknesses
  1. My main concern is the conceptual novelty. Prior work has already explored parameter-free unlearning through in-context inputs and verifier-guided inference-time refinement [1-2]. For example, MLLMEraser already combines test-time unlearning with input-aware activation steering, which is conceptually close to GUARD-IT's gated activation intervention. These works narrow the novelty from reframing to the specific design. The paper should therefore clarify which of these specific mechanisms provides an advantage over prior inference-time approaches.

[1] Inference-time Unlearning Using Conformal Prediction

[2] MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steering

  1. The key claims are stronger than the ablation study. The method is presented as addressing several distinct failure modes through clustering, gating, orthogonal projection, and norm-preserving rotation. However, the ablation in Appendix D, Table 6 removes the gate and clustering together. This shows that the full conditional design matters, but it does not establish whether clustering itself contributes beyond gating. A gate-only, single-vector condition and a clustered-but-unconditional condition would close this gap.

  2. Also, the evidence for norm-preserving rotation is incomplete. In Sec 2.2, the paper claims that preserving activation norm avoids downstream instability. But the closest ablation in Fig 10 compares orthogonal projection with diff-means, not norm-preserving rotation. The experiments do not isolate whether the reported fluency advantage comes from the rotation itself or from the choice of steering direction.

  3. In Sec 3.5 and App D, adversarial rewriting reduces gate activation, and an adaptive attacker recover part of the membership signal by reversing the decision rule. That is to say, the intervention can change the form of the leakage rather than eliminate it. This is somewhat counterintuitive, but it weakens any broad interpretation that inference-time steering provides robust unlearning.

Questions for authors
  1. Can the authors clarify the novelty of GUARD-IT relative to prior inference-time unlearning methods? Which component of GUARD-IT do the authors view as the main technical novelty? I may raise my score if the authors can convince me about the conceptual novelty.

  2. Can the authors disentangle the contributions of clustering and gating? The current ablation removes both components together (See my W2).

  3. What evidence isolates norm preservation as the source of improved utility or generation stability? The current comparison changes the steering construction in addition to the intervention geometry (See W3).

1 comment

We thank the reviewer for the careful reading and for pointing us to Conformal inference-time unlearning and MLLMEraser. New results are in orange in the revised PDF.

W1 / Q1. Novelty The main technical novelty is the forget-specific direction (PSV). It is extracted from the model's own activations on the data to forget, once per cluster, and applied as a norm-preserving rotation only when the gate matches that cluster. With it, GUARD-IT is the only method among 16 baselines, 4 of them at inference time (including in-context unlearning), that suppresses memorization in every Llama configuration without collapsing the model. On forget01, it is also the only non-collapsed method in Table 19 that passes TOFU's indistinguishability test (FQ 0.77 at 1B, 0.92 at 3B). Both works differ:

  • Conformal unlearning does not steer activations: it resamples outputs until an LLM judge accepts them. GUARD-IT generates once, adding under 5% latency per query (App. H.1).
  • MLLMEraser applies one refusal direction, estimated from harmful prompts, to every input. We ran it alone and behind our gate (Llama-3.2-1B; App. J.1):
forget01 Util ↑forget01 Mem ↑forget01 Avg ↑forget01 Gibb ↑forget05 Util ↑forget05 Mem ↑forget05 Avg ↑forget05 Gibb ↑
MLLMEraser0.510.400.450.780.440.480.460.70
Its direction + our gate, at our Mem0.600.680.640.630.600.63±.010.61±.000.58±.01
GUARD-IT (Table 1)0.600.68±.010.64±.000.80±.000.600.63±.000.62±.000.81±.01

Alone, MLLMEraser has a lower Avg on both splits. Behind our gate, it keeps utility but loses fluency in every setting, also at 3B (Gibb 0.61/0.58 vs. 0.84/0.83; best of four variants, some built from the forget questions: 0.60–0.73). So the gain comes from the PSV, applied only when the gate matches.

W2 / Q2. Clustering and gating ablation We added both conditions you proposed (App. D.1, Table R1). The clusters enter the gate (K centroids decide when to steer) and the edit (the matched cluster's PSV); each row removes them from one place (Llama-3.2-1B forget05, α = 0.2):

ConditionUtil ↑Mem ↑Avg ↑Gibb ↑Fires, forget ↑Fires, retain ↓
Clustered, unconditional0.06±.010.62±.000.11±.010.54±.0211
Gate-only, single vector0.600.73±.010.66±.000.61±.010.860.01
One centroid, single vector0.600.200.300.83≤ 0.05≤ 0.01
GUARD-IT (Table 1)0.600.63±.000.62±.000.81±.010.860.01
  • Without the gate, utility collapses.
  • Without clusters in the edit, one vector averaged over all authors over-steers at the paper's α: Mem rises, but Gibb falls to 0.61.
  • Without clusters in the gate, one centroid misses most forget questions (Mem 0.20); widened to the same coverage, it fires on 79% of retain questions.

So the clusters give the gate its coverage and selectivity, and keep the edit fluent.

W3 / Q3. Norm preservation We agree, and we added this ablation too (App. F.2, Table R5). It keeps the direction and the step fixed and changes only the renormalization (Llama-3.2-1B forget05):

ConditionUtil ↑Mem ↑Avg ↑Gibb ↑
Addition, same direction and step0.600.65±.000.62±.000.64±.01
Rotation (GUARD-IT, Table 1)0.600.63±.000.62±.000.81±.01

Without the renormalization, addition forgets about as much but loses fluency. It raises the norm of the steered activations (about 1.3× at the paper's α); the rotation keeps it fixed at any strength. This matches reports that additive edits need careful tuning to stay fluent (Vu & Nguyen, 2025; Pham & Nguyen, 2024).

W4. Form of the leakage We respectfully disagree that the inversion is counterintuitive. Loss-based attacks label low-loss inputs as members, and the rotation raises the loss of forget queries above the holdout level, so the attacks read them as non-members. This is over-unlearning, which weight-editing methods show too, so it does not weaken inference-time steering in particular: on MUSE Books, every gradient-based baseline has a negative PrivLeak (−22.6 to −30.3; GUARD-IT −25.5). Even against an attacker who flips the decision rule, GUARD-IT halves the leakage |AUC − 0.5| (0.50 → 0.25). Evading the gate does not undo the unlearning either: jailbreak rewrites lower the fire rate from 0.86 to 0.74, yet the forgotten answers stay suppressed (ROUGE-L 0.37 vs. 0.63 without the gate). Sec. 3.5 now states the inversion, and the abstract no longer says GUARD-IT "holds" under attack.

We believe the new ablations and comparison address your concerns, especially on novelty. If any point is still open, please let us know.

Comments on the paper

1 comment

We thank the reviewers and the AC. We answered the reviewers' concerns with new experiments, reported in ten new tables (R1 to R10) of the revised PDF.

Novelty (8sGT, KRoJ, 2Qs8). The contribution is the forget-specific direction (PSV). Behind the same gate and at the same strength, it forgets more than a random direction in all four settings (App. F.1). MLLMEraser and DRAGON, suggested by the reviewers, have a lower Mem and Avg than GUARD-IT in every setting (App. J.1 and J.2).

Size of the gain (KRoJ). Against the 21 baseline configurations of Tables 1 and 3, GUARD-IT has the higher Avg, or the baseline collapses, in 121 of 126 comparisons, without gradient updates and at up to eight times lower cost (Table 11).

Design choices (8sGT, KRoJ). We showed that each component is needed: without the gate, Util falls from 0.60 to 0.06; without the clusters, Mem falls from 0.63 to 0.20; without the norm-preserving rotation, Gibb falls from 0.81 to 0.64 (App. D.1 and F.2).

Suppression and robustness (2Qs8, KRoJ, 8sGT). Weight editing does not guarantee removal either: on 8B forget05, 4-bit quantization undoes part of UNDIAL's forgetting (Mem 0.60 to 0.40), while GUARD-IT reaches 0.60 (App. N). Against seven new evasion strategies, GUARD-IT leaks less than the Retain model, which never saw the forget set, on five (App. D.3 and D.4). Over ten sequential requests, its Avg stays the highest among non-collapsed methods (Table 5).

GUARD-IT remains the only method among 16 baselines that suppresses memorization in every Llama configuration on TOFU without collapsing, and it carries to MUSE, WMDP, Gemma, and Qwen, whose first TOFU checkpoints we release. We respectfully ask the AC and the reviewers to consider these results in the final recommendation.

Changes requested by 8sGT are in orange, by KRoJ in green, and by 2Qs8 in purple; other changes are in blue. Results without a standard error come from one run; the others are means ± standard error over five seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.