acceptodds
Under review as a conference paper at ICLR 2027

Forget One Sense, Preserve Another: Sense-Level Unlearning in Large Language Models

Abstract

When a name or word has multiple interpretations, an unlearning request may require a large language model (LLM) to suppress answers about one while preserving correct answers about another. Evaluating such a request requires testing the retained interpretation directly, since retention measured on other items cannot establish whether that interpretation is preserved. We formulate this task as sense-level unlearning. We introduce SensePairs, to our knowledge the first benchmark for LLM sense-level unlearning that separately probes two distinct, specified interpretations and requires their shared name or word in every question in its paired evaluation. Human evaluation measures retained-answer correctness and distinguishes refusals and degenerate outputs from well-formed, non-abstaining responses that omit the queried target fact. We propose an inference-time method SenseClamp that selects queries by interpretation and, within a frozen model, sets hidden-state coordinates along a contrast direction to layer-specific reference values fitted from retained contexts. Across three model families and two evaluation sets, we compare SenseClamp with existing unlearning approaches and identify retained-interpretation damage that other-item retention misses. This work provides a task formulation, paired benchmark, and inference-time intervention for unlearning across interpretations of a shared name or word.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kGRZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates sense-level behavioral unlearning, which aims to suppress responses associated with one registered interpretation of a shared name or word while preserving responses related to another. It introduces SensePairs and proposes SenseClamp, an inference-time method that combines interpretation routing with hidden-state clamping toward layer-specific retained-reference values in a frozen model. Experiments across three model families demonstrate the effectiveness of SenseClamp and its ability to identify degradation in retained interpretations.

Critical strengths
  1. The paper introduces a novel task of sense-level unlearning and constructs a paired dataset.By directly evaluating two interpretations of the same surface form, it addresses the limitations of conventional retention evaluations that focus only on unrelated items.
  2. The paper proposes SenseClamp, an inference-time method for sense-level unlearning, and provides comprehensive experimental evaluations to assess its effectiveness.
Critical weaknesses
  1. Although sense-level unlearning is a meaningful problem, similar cases may already exist in conventional unlearning benchmarks. The authors should evaluate SenseClamp on standard benchmarks to further demonstrate its advantages over existing methods.
  2. The comparison with GUARD-IT does not control intervention sites or strength, making it difficult to isolate the contribution of the clamping mechanism. A fairer comparison with matched intervention settings is recommended.
  3. The refusal baseline achieves higher target-answer flip rates on Llama and Qwen with comparable retention damage. The authors should clarify the practical benefits of SenseClamp over selective refusal.
  4. Figure 2 does not clearly illustrate the method's workflow, particularly the meanings of different arrows. The figure should better align with the methodological descriptions. Figures 3 and 4 also need improvements in clarity and presentation.
Questions for authors
  1. Given that the selective refusal baseline achieves better target-answer flip rates with comparable retention damage, what are the practical advantages of SenseClamp over selective refusal?
  2. Could the authors provide a fairer comparison between clamping and rotation under the same intervention sites and comparable perturbation strengths? Otherwise, it is difficult to determine whether the improvements come from the clamping mechanism itself.
4 comments
AuthorsOpenReview ↗

W1 (Part 1 of 3)

Although sense-level unlearning is a meaningful problem, similar cases may already exist in conventional unlearning benchmarks. The authors should evaluate SenseClamp on standard benchmarks to further demonstrate its advantages over existing methods.

Response

Thank you for suggesting an evaluation on a standard unlearning benchmark. We evaluated SenseClamp and five baselines on the 13 RWKU [1] targets that have a same-name referent meeting our criteria. On these requests, SenseClamp gave the best overall trade-off between target suppression and preservation across the three models.

Consider a request to suppress facts about the actor Steve McQueen. RWKU’s original neighbor questions ask about people connected to the actor, such as his co-star James Garner. A model could still answer correctly about Garner even if it could no longer answer questions about the director also named Steve McQueen. Each target and its other referent share a name with identical spelling and capitalization. We required example passages for fitting both referents and at least five separately verifiable facts about the retained referent. We kept RWKU’s original questions and added five questions per name about the other referent, for a total of 65 same-name questions.

We measured how often a previously correct answer became incorrect and averaged that rate across eligible requests. Gated prompting on Qwen illustrates what the original neighbor questions miss. Its damage rate was 0.85% on RWKU’s original neighbor questions and 37.50% on the added same-name questions. On each model, SenseClamp selected none of the 373 original neighbor questions for editing, so all 373 received the same answers as before the intervention.

Human evaluation on 13 RWKU requests (%). For each model, S ↑ is substantive target suppression on original RWKU target questions. Related ↓ and Same-name ↓ are damage rates on original neighbor questions and added same-name questions, respectively. Table 41 in the revised paper gives the eligible question counts. The same-gate refusal control replays SenseClamp’s routing decisions, returning a fixed refusal on selected questions and the stored base answer on the rest. On previously correct target questions, this gives a substantive suppression rate of 0.00%.

MethodGemma S ↑Related ↓Same-name ↓Qwen S ↑Related ↓Same-name ↓Llama S ↑Related ↓Same-name ↓
SenseClamp59.670.000.0067.990.006.2545.050.000.00
GUARD-IT46.180.004.1723.292.590.0022.4011.5527.78
LUNAR27.2329.7218.7566.1646.8018.7518.9325.6833.33
Gated prompting14.180.9056.2510.190.8537.508.900.0027.78
Partial-refusal prompting2.200.9068.750.000.0050.000.000.4350.00
Same-gate refusal0.000.000.000.000.0012.500.000.000.00

On every model, SenseClamp had the highest substantive suppression of the six methods on the original RWKU target questions, and it preserved every previously correct answer to the original neighbor questions. On the added same-name questions, its damage was zero on Gemma and Llama and 6.25% on Qwen.

Reference

[1] Zhuoran Jin et al. 2024. RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models. NeurIPS 2024, Datasets and Benchmarks Track.

AuthorsOpenReview ↗

W2 & Q2 (Part 2 of 3)

W2

The comparison with GUARD-IT does not control intervention sites or strength, making it difficult to isolate the contribution of the clamping mechanism. A fairer comparison with matched intervention settings is recommended.

Response

We ran the matched comparison you recommend (Table 38 of the revised paper). We applied both clamping and GUARD-IT-style rotation at block 7 of Qwen2.5-7B-Instruct, GUARD-IT’s own edit block (Table 29). For each request, both edits used a direction refitted at that block, identical routing decisions and the same token-position rule. The clamp set the activation coordinate along that direction to the retained reference. The rotation updated the activation along the shared direction and restored its original norm.

Q2

Could the authors provide a fairer comparison between clamping and rotation under the same intervention sites and comparable perturbation strengths? Otherwise, it is difficult to determine whether the improvements come from the clamping mechanism itself.

Response

Yes. We calibrated strength on the development set using mean decoding displacement. At each decoding position, we measured the size of the hidden-state change relative to its pre-edit norm. We averaged these relative changes within each request and then equally across requests. Rotation’s mean was 1.52% above the clamp’s, within the prespecified ±10% tolerance. With the coefficient fixed, rotation’s mean decoding displacement was 3.28% larger on Test.

New same-layer experiment on Qwen2.5-7B-Instruct, SensePairs-Test

Edit at block 7Mean decoding displacement (clamp = 100)Substantive target suppression rate ↑
Retained-reference clamp100.0037.28%
GUARD-IT-style rotation103.2819.91%

At block 7, clamping achieved 37.28% substantive target suppression on Qwen Test, compared with 19.91% for rotation. The paired 95% interval for the 17.37-percentage-point difference excludes zero. The edit site was identical and mean decoding displacement was approximately matched. This supports a contribution from the clamping rule itself.

AuthorsOpenReview ↗

W3, Q1 & W4 (Part 3 of 3)

W3

The refusal baseline achieves higher target-answer flip rates on Llama and Qwen with comparable retention damage. The authors should clarify the practical benefits of SenseClamp over selective refusal.

Q1

Given that the selective refusal baseline achieves better target-answer flip rates with comparable retention damage, what are the practical advantages of SenseClamp over selective refusal?

Response

The practical advantage of editing is that it can preserve a correct answer that a fixed refusal would block. The refusal baseline in Table 7 relies on SenseClamp to decide which questions to refuse. On Gemma-Ext, SenseClamp mistakenly selected 31 questions about the meaning to preserve that the base model had answered correctly. The refusal baseline blocked all 31. SenseClamp still answered 16 of them correctly after editing. Averaged equally across requests, editing reduced retained-side damage relative to refusal by 15.3 percentage points on Gemma Test and 18.5 points on Ext. Both paired 95% intervals exclude zero.

On Llama and Qwen, SenseClamp rarely selected questions about the meaning to preserve (Table 18). The refusal baseline inherits that selectivity because it refuses only the questions SenseClamp selects. Retained-side damage was at most 2.2% for both editing and refusal across Test and Ext. The two prompt-based baselines also check for the shared word or name but rely on instructions to tell its meanings apart. With gated prompting on Test and partial-refusal prompting on Ext, retained-side damage ranged from 9.7% to 24.0% on these models (Tables 3 and 4).

As you note, refusal produced more target-answer flips than editing on Llama and Qwen on both Test and Ext, with comparable retained-side damage. Whenever the base model answered a target question correctly, a refusal counts as a flip. Substantive suppression is measured on those same previously correct target questions. To count, the edited answer must be well-formed and non-refusing, with the queried fact absent throughout. The model must also assign the correct reference answer a lower likelihood than before editing. On SensePairs-Test, editing achieved 57.1% substantive suppression on Llama and 68.5% on Qwen (Table 3). We expanded the "Response policy" paragraph of Section 6.2 in the revised paper to state this comparison.


W4. Figure revisions

Figure 2 does not clearly illustrate the method's workflow, particularly the meanings of different arrows. The figure should better align with the methodological descriptions. Figures 3 and 4 also need improvements in clarity and presentation.

Response

Thank you for the specific feedback on the figures. We revised Figures 2, 3 and 4 in the updated PDF.

Figure 2. The dashed arrow passes the fitted parameters to the clamp, the yes and no arrows show whether a query is edited, and the inset arrow shows how one hidden state is updated.

Figure 3. It is now a scatter plot of substantive suppression against preservation of the retained interpretation.

Figure 4. It is now a paired bar chart of the two edits with every value printed.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer kGRZ,

Just checking in, since the discussion closes on November 18. Both experiments you asked for are in the revised PDF. Table 41 has six methods on 13 RWKU targets, and Table 38 compares clamping and rotation at the same block with roughly the same displacement. We also redrew Figures 2 to 4. We'd really like to hear whether this settles your concerns.

Reviewer_A9j6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors address an issue that is often overlooked in traditional LLM unlearning evaluations: when the same word or name has multiple interpretations, the model should forget only the specified interpretation while preserving correct knowledge about the others. To investigate this problem, the authors introduce the SensePairs benchmark and propose SenseClamp, an inference-time method that combines interpretation-aware routing with retained-reference hidden-state clamping to achieve conditional answer suppression without updating model parameters.

Critical strengths
  1. The benchmark reveals failures overlooked by traditional evaluations. SensePairs provides a more targeted evaluation of selective unlearning by explicitly measuring whether knowledge associated with the retained interpretation is preserved.

  2. The experimental evaluation is relatively comprehensive and rigorous. The authors evaluate three model families using 20 primary interpretation pairs and 27 additional pairs. They also employ independent human annotation by two annotators, blind adjudication, paired hierarchical bootstrap confidence intervals, and sensitivity analyses for unclear labels.

  3. The experimental results support the effectiveness of the proposed intervention. Under the specified evaluation protocol, SenseClamp effectively modifies model generation behavior, and the ablation studies suggest that the retained-reference design makes a meaningful contribution rather than serving as a redundant component.

Critical weaknesses
  1. The substantive suppression metric may reward irrelevant or hallucinated responses rather than meaningful unlearning. To avoid treating simple refusals as successful unlearning, the authors introduce substantive suppression. However, this metric still considers fluent but incorrect or irrelevant responses successful, provided that they do not reveal the target fact, are not classified as refusals or degenerate outputs, and exhibit increased reference-answer negative log-likelihood (NLL). This raises concerns about whether the reported improvements reflect genuine selective unlearning or merely answer corruption.

  2. Simpler refusal-based strategies remain competitive and can even outperform SenseClamp on the primary behavioral objective. Since the task explicitly permits selective refusal, a straightforward refusal strategy can achieve comparable or better target-answer suppression while preserving the retained interpretation. This raises questions about the practical advantages of SenseClamp over simpler alternatives.

  3. The methodological novelty is somewhat limited, and the current comparisons do not fully isolate the contribution of the clamping mechanism. Contrastive directions, activation routing, and affine projections have all been explored in prior work. Moreover, the fixed-routing comparison with GUARD-IT simultaneously changes the intervention sites, layer locations, and intervention strength. Consequently, the observed improvements cannot be attributed solely to the proposed clamping formulation.

Questions for authors

None

5 comments
AuthorsOpenReview ↗

W1 & W2 (Part 1 of 2)

W1. Selective suppression

The substantive suppression metric may reward irrelevant or hallucinated responses rather than meaningful unlearning. To avoid treating simple refusals as successful unlearning, the authors introduce substantive suppression. However, this metric still considers fluent but incorrect or irrelevant responses successful, provided that they do not reveal the target fact, are not classified as refusals or degenerate outputs, and exhibit increased reference-answer negative log-likelihood (NLL). This raises concerns about whether the reported improvements reflect genuine selective unlearning or merely answer corruption.

Response

We appreciate your question about whether the edit simply corrupts answers. MUSE [1] evaluates target knowledge and retained utility separately. SensePairs applies that distinction to two meanings of the same word. On every model in SensePairs-Test, human-assessed substantive suppression exceeded 57% on previously correct target questions. Retained-side damage measures how often previously correct answers about the other meaning become incorrect. This rate was at most 6.9% on every model (Table 3). If the edit broadly disrupted answers containing the shared word, we would expect much higher retained-side damage.

A fluent hallucination can count as suppression of a particular target fact under the conditions you list. Those conditions establish that the queried fact is absent from the answer and that the model assigns the correct reference answer a lower likelihood than before editing. Section 6.2 discusses what these responses express, with the semantic breakdown in Appendix C.5.

References

[1] Weijia Shi et al. 2025. MUSE: Machine Unlearning Six-Way Evaluation for Language Models. ICLR 2025.


W2. Selective refusal

Simpler refusal-based strategies remain competitive and can even outperform SenseClamp on the primary behavioral objective. Since the task explicitly permits selective refusal, a straightforward refusal strategy can achieve comparable or better target-answer suppression while preserving the retained interpretation. This raises questions about the practical advantages of SenseClamp over simpler alternatives.

Response

The practical advantage of editing is that it can preserve a correct answer that a fixed refusal would block. The refusal baseline in Table 7 relies on SenseClamp to decide which questions to refuse. On Gemma-Ext, SenseClamp mistakenly selected 31 questions about the meaning to preserve that the base model had answered correctly. The refusal baseline blocked all 31. SenseClamp still answered 16 of them correctly after editing. Averaged equally across requests, editing reduced retained-side damage relative to refusal by 15.3 percentage points on Gemma Test and 18.5 points on Ext. Both paired 95% intervals exclude zero.

On Llama and Qwen, SenseClamp rarely selected questions about the meaning to preserve (Table 18). The refusal baseline inherits that selectivity because it refuses only the questions SenseClamp selects. Retained-side damage was at most 2.2% for both editing and refusal across Test and Ext. The two prompt-based baselines also check for the shared word or name but rely on instructions to tell its meanings apart. With gated prompting on Test and partial-refusal prompting on Ext, retained-side damage ranged from 9.7% to 24.0% on these models (Tables 3 and 4).

As you note, refusal produced more target-answer flips than editing on Llama and Qwen on both Test and Ext. Whenever the base model answered a target question correctly, a refusal counts as a flip. Substantive suppression is measured on those same previously correct target questions. To count, the edited answer must be well-formed and non-refusing, with the queried fact absent throughout. The model must also assign the correct reference answer a lower likelihood than before editing. On SensePairs-Test, editing achieved 57.1% substantive suppression on Llama and 68.5% on Qwen (Table 3).

AuthorsOpenReview ↗

W3 (Part 2 of 2)

The methodological novelty is somewhat limited, and the current comparisons do not fully isolate the contribution of the clamping mechanism. Contrastive directions, activation routing, and affine projections have all been explored in prior work. Moreover, the fixed-routing comparison with GUARD-IT simultaneously changes the intervention sites, layer locations, and intervention strength. Consequently, the observed improvements cannot be attributed solely to the proposed clamping formulation.

Response

Our methodological contribution is to use the two meanings in each request to determine when to edit and where to move the activation. SenseClamp uses the question’s context to decide whether it asks about the target meaning. Examples of the meaning to preserve set the reference value at each layer. The edit moves the activation to that value along the fitted direction. To our knowledge, we are the first to formally define sense-level unlearning in LLMs and to build a dedicated benchmark (SensePairs) and method (SenseClamp) for it. We tested this choice by replacing the request’s own reference value with zero or a reference value from another request (Tables 2 and 14). The request’s own reference value gave the strongest suppression on Qwen, Gemma and Llama. All 15 paired 95% intervals for that advantage lie above zero (Appendix H.1, Table 37).

To separate the contribution of the clamping rule from the effects of edit site and strength, we ran a new experiment on Qwen. Both edits used block 7, GUARD-IT’s own edit block (Tables 29 and 38). For each request, we refitted one direction at that block and used it for both edits, with identical routing and token positions. We calibrated rotation on development data to match the clamp’s mean decoding displacement, which measures how much the edit changes the hidden state during decoding. On Test, rotation produced 3.28% greater mean decoding displacement, yet clamping achieved 37.28% substantive target suppression versus 19.91% for rotation. The paired 95% interval for the 17.37-percentage-point difference excludes zero. This supports a contribution from the clamping rule itself.

New same-layer experiment on Qwen2.5-7B-Instruct, SensePairs-Test

Edit at block 7Mean decoding displacement (clamp = 100)Substantive target suppression rate ↑
Retained-reference clamp100.0037.28%
GUARD-IT-style rotation103.2819.91%

The earlier comparison under identical routing points the same way. The full SenseClamp edit suppressed more target facts than GUARD-IT’s rotation on Llama and Qwen on both Test and Ext, with all four paired 95% intervals above zero (Table 11 and Figure 4).

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer A9j6,

The discussion closes on November 18, so we wanted to check in. Section 6.2 now compares editing with selective refusal on the same selected questions, and the numbers are in Table 7. On isolating the clamp, Table 38 compares clamping and rotation at the same block with roughly the same displacement. Table 37 repeats the reference-substitution experiment on all three models. Does this answer your concerns? We're happy to clarify anything that doesn't.

Reviewer_4dhhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates multiple interpretations of word as sense-level unlearning by introducing SensePairs that separately probes two distinct, specified interpretations and requires their shared name or word in every question in its paired evaluation. Human evaluation measures retained-answer correctness and distinguishes refusals and degenerate outputs from well-formed, non-abstaining responses that omit the queried target fact. The proposed inference-time method SenseClamp selects queries by interpretation and, within a frozen model, sets hidden-state coordinates along a contrast direction to layer-specific reference values fitted from retained contexts.

Critical strengths
  1. The gap in the task formulation is real, and the evidence is clean. Existing benchmarks (TOFU/MUSE/WMDP/RWKU) only measure retention on other items. The paper shows this systematically misses harm: Qwen gated prompting damages only 0.2% of other-item answers but 20.2% of answers about the preserved sense (Table 8, p. 17). This D ret≫C cross finding alone carries a paper's worth of value.

  2. Evaluation rigor is top-tier for this subfield. Two independent annotators over 6,316 deduplicated units with blind adjudication (Table 19); Scott's π reported throughout (0.708–1.000, Table 23); every estimate recomputed under both unclear-label scenarios; paired hierarchical bootstrap (5,000 replicates, seed 0, resampling requests then probes); dataset SHA-256 and code commits all frozen (Tables 26/28). More unusual still, the authors volunteer unfavorable information: the abstention detector was widened from s1.1 to s1.2 during the confirmatory phase and is self-declared "not pre-registered" (C.4); a pre-specified damage rule contained a sign error and was therefore abandoned (B.2.2); the Joint judge prompt had a role-reversal bug that was found and corrected (F.2). I would explicitly commend this honesty in the review.

Critical weaknesses
  1. The method is ACE plus a classifier gate; the delta is too small. The paper itself calls it a "reference-based affine intervention." Relative to Marshall et al. 2024 (refusal is an affine function), Lee et al. 2025 (conditional activation steering), and DSG, only two things are new: the reference value is fitted from retained contexts rather than zero, and routing uses a sense discriminator. The strongest novelty evidence is the reference-substitution ablation (retained vs. zero vs. donor, +44–50 points, Table 14) — but it was run only on Qwen; Gemma got only the destination substitutions.

2.The success mode is fluent hallucination, and this is buried in Appendix C.5. Of 343 human-verified successes, the judge labels 223 as unrelated hallucination and only 35 as consistent with the retained sense. That is, the method moves the model from "answering correctly" to "confidently making things up," not to "declining gracefully." Table 25 is blunt about it — asked what ground-contact system a crawler crane uses (reference: Tracks), the output is "Grasshopper crane." Whether this is better or worse than refusal goes undiscussed, and it is the single question the paper most needs to confront head-on.

3 Baseline configurations are open to challenge. On Qwen, no GUARD-IT coefficient met the admission constraints and the fallback picked the "least violating" c=0.25 — while the Gemma sweep shows c=0.25 → 23.6 vs. c=0.5 → 69.6 (Table 36), a 46-point swing from that single choice. The DSG adaptation also departs from the published configuration (importance-ratio percentile 90 vs. 95, feature counts {20,50,100,200} vs. {10,20,30}) and fell back after all four candidates violated the constraint. The edit comparison is explicitly not strength-matched (single-block rotation at 0.25L vs. all-block clamp at 0.5L). NPO/RMU/ECO cover Gemma/Test only.

Questions for authors

see the above weakness

4 comments
AuthorsOpenReview ↗

W1 (Part 1 of 3)

The method is ACE plus a classifier gate; the delta is too small. The paper itself calls it a "reference-based affine intervention." Relative to Marshall et al. 2024 (refusal is an affine function), Lee et al. 2025 (conditional activation steering), and DSG, only two things are new: the reference value is fitted from retained contexts rather than zero, and routing uses a sense discriminator.

Response

Thank you for identifying the two new elements of SenseClamp in your careful comparison with ACE, conditional activation steering and DSG. To our knowledge, we are the first to formally define sense-level unlearning in LLMs, introduce a dedicated benchmark for it (SensePairs), and design a method for it (SenseClamp). Because the two meanings share the same word or name, SenseClamp uses the question’s context to decide whether to intervene. Once a question is selected, examples of the meaning to preserve determine where the edit moves the activation along the fitted direction at each layer. On SensePairs-Test, SenseClamp achieved higher human-assessed substantive target suppression than every baseline in Table 3 on all three models. On each model, it preserved more than 90% of the base model’s correct answers about the other meaning.

The strongest novelty evidence is the reference-substitution ablation (retained vs. zero vs. donor, +44–50 points, Table 14) — but it was run only on Qwen; Gemma got only the destination substitutions.

Response

We extended the reference-substitution experiment in Table 14 from Qwen to Gemma and Llama. We held the routing, edit direction and edit sites fixed and compared the request’s own retained reference with two zero-reference settings and three references from other requests. In the primary analysis, the request’s own reference increased substantive suppression by more than 29 percentage points in every one of the 15 comparisons (Table 37 of the revised paper). All 15 paired 95% intervals lie above zero under both treatments of unclear labels (Appendix H.1).

Human-assessed substantive target suppression on SensePairs-Test (%, ↑).

ModelOwn retained referenceZero (α = 1)Calibrated zero (α = 2)Donor 0Donor 1Donor 2
Qwen68.520.424.318.717.819.4
Gemma70.429.640.822.728.825.3
Llama57.114.315.614.613.910.8

Unclear labels were set to no, as in the paper’s primary analysis.

In six of the nine substitutions, using another request’s reference produced an average change in the model’s activations more than 10% larger than that produced by the request’s own reference on the development set. All six suppressed fewer target facts on Test: a larger edit during calibration did not compensate for using a different reference (Appendix H.1). Gemma and Llama confirm the finding on Qwen. On all three models, suppression was strongest when the reference came from examples of the meaning the current request asks the model to preserve.

AuthorsOpenReview ↗

W2 (Part 2 of 3)

The success mode is fluent hallucination, and this is buried in Appendix C.5. Of 343 human-verified successes, the judge labels 223 as unrelated hallucination and only 35 as consistent with the retained sense. That is, the method moves the model from "answering correctly" to "confidently making things up," not to "declining gracefully." Table 25 is blunt about it — asked what ground-contact system a crawler crane uses (reference: Tracks), the output is "Grasshopper crane." Whether this is better or worse than refusal goes undiscussed, and it is the single question the paper most needs to confront head-on.

Response

Your comment led us to add an explicit discussion of refusal to the "Output content" paragraph of Section 6.2 in the revised paper. For the crawler-crane question in Table 25, a refusal is preferable to the incorrect answer “Grasshopper crane.” SenseClamp selected this question, so the same-gate refusal control returns a fixed refusal for it (Table 7). The same paragraph of the original submission summarized the auxiliary judge’s classifications, with the counts in Appendix C.5.

Our task permits selective refusal. Table 7 compares two actions on the same selected questions. SenseClamp edits the hidden state and lets the model answer. Same-gate refusal returns a fixed refusal without applying the edit. A target-answer flip occurs when a previously correct target answer is lost. Both a refusal and a fluent error count, so a flip alone does not tell us whether the queried fact appears elsewhere in the response. Gu et al. [1] found that a model could refuse open-ended questions yet retain high multiple-choice accuracy on the same material. We score target questions that the base model answered correctly. To count as substantive suppression, the edited answer must be well-formed, non-refusing and free of the queried fact throughout. The model must also assign the correct reference answer a lower likelihood than before editing.

More generally, choosing a response policy requires weighing the risk of an unrelated answer to a target question against the risk of withholding a correct answer to a retained question. Under the same unlearning request, a question about a crane’s diet still calls for a correct answer about the bird. Same-gate refusal withholds that answer if routing selects the question. Averaged equally across requests, Gemma’s retained-side damage was 15.3 percentage points lower on Test and 18.5 points lower on Ext with SenseClamp than with same-gate refusal (Table 7). The paired 95% intervals for the damage differences exclude zero on both sets. On Gemma-Ext, SenseClamp mistakenly selected 31 retained questions that the base model had answered correctly. After editing, SenseClamp still answered 16 of those 31 questions correctly.

Reference

[1] Renjie Gu et al. 2026. Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning. ACL 2026.

AuthorsOpenReview ↗

W3 (Part 3 of 3)

Baseline configurations are open to challenge. On Qwen, no GUARD-IT coefficient met the admission constraints and the fallback picked the "least violating" c=0.25 — while the Gemma sweep shows c=0.25 → 23.6 vs. c=0.5 → 69.6 (Table 36), a 46-point swing from that single choice.

Response

Thank you for checking the baseline settings so closely. On Qwen Test, the selected coefficient gave GUARD-IT more than twice the automatic target suppression of the alternative, 35.9% at 0.25 versus 17.7% at 0.5 (Table 40). On Qwen development data, no coefficient met the admission constraints. Degenerate outputs were 8.2% at 0.25 and at least 73.9% at every larger coefficient tested, so the fallback selected 0.25 (Appendix E.1). On Gemma, the development rule selected 0.5 (Table 29), which gave the higher of the two suppression rates on Test (Table 36).

The DSG adaptation also departs from the published configuration (importance-ratio percentile 90 vs. 95, feature counts {20,50,100,200} vs. {10,20,30}) and fell back after all four candidates violated the constraint.

Response

On Gemma Test, SenseClamp achieved higher substantive target suppression and lower retained-side damage than both the development-selected DSG setting and the published WMDP-Bio setting [2]. For each of these two DSG settings, the paired 95% intervals exclude zero for both differences. We repeated selection over both importance-ratio percentiles and the published feature-count grid. Table 39 reports the human evaluation of the three DSG settings and SenseClamp.

Method · DSG importance-ratio percentile / max. featuresConfiguration sourceTarget flips ↑Substantive suppression ↑Retained-side damage ↓
DSG 90/20Baseline used in our submitted Table 190.47.038.3
DSG 90/10Selected on our SensePairs development set for this rebuttal79.48.323.6
DSG 95/20DSG authors’ published WMDP-Bio setting [2]84.517.746.9
SenseClampOur method evaluated on the same Test set86.870.46.9

The edit comparison is explicitly not strength-matched (single-block rotation at 0.25L vs. all-block clamp at 0.5L).

Response

At block 7 on Qwen Test, clamping achieved 37.28% substantive target suppression, nearly twice the 19.91% achieved by rotation (Table 38). The paired 95% interval for the 17.37-percentage-point difference excludes zero. Table 11 compared GUARD-IT’s single-block rotation with all-block clamping. Here both updates used block 7, GUARD-IT’s own edit block on Qwen (Table 29). For each request, they shared the direction refitted at that block, the routing decisions and the token-position rule. At each decoding position, we measured the size of the hidden-state change relative to its pre-edit norm. We averaged over positions within each request and then equally across requests. On the development set, rotation’s mean decoding displacement was within the prespecified ±10% tolerance of the clamp’s. On Test, rotation produced 3.28% greater mean decoding displacement and still suppressed fewer target facts.

NPO/RMU/ECO cover Gemma/Test only.

Response

We evaluated ECO on SensePairs-Test on all three models. On every model, SenseClamp had higher automatic target suppression and lower retained-side damage than ECO. ECO’s retained-side damage was at least 18.2% on each model, while SenseClamp’s was at most 6.9%. NPO and RMU were evaluated on Gemma-2-2B Test. There, SenseClamp also had higher automatic target suppression and lower retained-side damage than both (Table 36). Tables 3 and 4 report the human evaluation of SenseClamp, GUARD-IT and LUNAR on all three models on Test and Ext, respectively.

Reference

[2] Aashiq Muhamed et al. 2025. SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMs. COLM 2025.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 4dhh,

A quick note before the discussion closes on November 18. You asked whether the reference-substitution result holds beyond Qwen. It does on both Gemma and Llama, and the numbers are in Table 37. Table 38 compares clamping and rotation at the same block with roughly the same displacement. Tables 39 and 40 cover the other DSG and GUARD-IT settings, and Section 6.2 now discusses refusal against an unrelated answer. If anything still looks off to you, we'd be happy to go through it.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of revisions

Thank you for the careful reviews. We have marked the changes in red in the revised PDF.

  • Appendix H.1 (Table 37) extends the reference-substitution experiment from Qwen to Gemma and Llama. (4dhh W1, A9j6 W3)
  • Appendix H.2 (Table 38) compares clamping and rotation at the same Qwen block with approximately matched mean decoding displacement. (4dhh W3, A9j6 W3, kGRZ W2/Q2)
  • Appendix H.3 (Tables 39–40) tests alternative DSG settings on Gemma and two GUARD-IT coefficients on Qwen. (4dhh W3)
  • Appendix H.4 (Table 41) evaluates six methods on all 13 eligible RWKU targets with added same-name retention questions. (kGRZ W1)
  • Section 6.2 (Table 7) discusses editing and selective refusal on the same questions selected by SenseClamp. (4dhh W2, A9j6 W2, kGRZ W3/Q1)
  • Sections 1 and 2 clarify the task, benchmark and method contributions and how SenseClamp relates to prior work. (4dhh W1, A9j6 W3)
  • We redrew Figures 2 - 4 and revised the captions. (kGRZ W4)

Thank you for taking the time to read our response and the new experiments.

On sample size, the RWKU evaluation uses every RWKU target that meets our criteria. There are only 13, and RWKU's own retention questions are about related people, not about the other person with the same name. So a standard benchmark has few such cases and does not test them. This is exactly why a dedicated benchmark is needed and important. No existing work formally defined this problem, and SensePairs is the first benchmark built for it.

On selective refusal, the same-gate refusal baseline in Table 7 is also our method. It uses SenseClamp's routing to pick the questions and only swaps the final edit for a fixed refusal. We included it to show what happens if you refuse instead of edit. When routing picks a question it should not, a refusal always loses that answer and the edit can still keep it. So this compares our method with a variant of our own method and they are all better than baselines.

On generality, names and words with more than one meaning are everywhere, so any unlearning request about a name or a word can run into this problem. SenseClamp is designed for exactly this case. On SensePairs it had the highest substantive suppression in five of the six model and test-set combinations, showing a large advantage over existing baseline methods.

We hope this helps with your remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.