acceptodds
Under review as a conference paper at ICLR 2027

UNBIND: UNlearning By INference-time Directional Steering for Code LLMs

Abstract

Code large language models acquire programming capabilities from large code corpora, but can also memorize implementations that later require removal. Code unlearning is needed to control their continued reproduction when copyright or security concerns arise. However, targeted and retained code share computational patterns, creating a tension between forgetting specific implementations and preserving general programming ability. We propose UNBIND, a code unlearning framework that separately considers which hidden states correspond to the target code and how to suppress its reproduction. By constructing separate directions for these objectives, UNBIND achieves selective unlearning at inference time while keeping model weights fixed. Our evaluation covers fourteen baselines across two code models and two corpora. UNBIND achieves the highest joint forgetting and utility score in every setting. It reduces target code reproduction by 97.3% to 99.1% as measured by F-BLEU, with at most two fewer HumanEval+ and six fewer MBPP+ problems solved than the original models. In repeated extraction tests under a fixed budget, the number of targets yielding exact spans of at least 50 tokens falls from 188–262 to 0–2 out of 300 per setting. No extracted span reaches 100 tokens, and the mean best recovery ratio ranges from 0.43% to 6.45%. Multilingual and related-code evaluations further show effective forgetting with limited impact on useful programming capabilities, supporting UNBIND as a practical approach to selective code unlearning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Code Language Models, Activation Steering, Inference-Time Intervention, Code Memorization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tHa4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a training-free method to unlearn certain code data that was seen during training, through inference-time steering of hidden states. Given a forget dataset and a retain dataset, a three-stage solution is proposed. First, the subset of important hidden states that need steering are determined by a projection taking into account the variance of both forget and retain groups, followed by comparing against the mean of the retain group. Next, the direction along which to move the hidden states is determined by the forget group. Finally, the intervention strength is calibrated to minimize degradation on generation utility. Experiments are performed on two corpora, each of 300 forget examples and 8280 retain examples. Results show that the proposed method effectively reduces reproduction of the forget examples, while largely maintaining generation accuracy.

Critical strengths
  • The proposed method intuitively makes sense and is effective under the paper's experiment setup.
Critical weaknesses
  • The practical usefulness of the proposed method is unclear. The motivation mentioned in the introduction is to unlearn samples that have copyright or security concerns. Per my understanding, those mainly target pre-training data. The problem is that the method does not scale to pre-training-level data, in which case the retain set would contain trillions of tokens. It only works on small datasets, but retraining in such cases is cheap enough and more robust than inference-time patching.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer tHa4 Part1

Pretraining data and trillion-token retain sets. We directly tested unlearning of pretraining memorization in two base models, with no fine-tuning.

  • How we mined memorized code.
    • We sampled 200k Python files from The Stack and prompted each base model with 64-token prefixes (342,653 probes for Qwen).
    • A hit is a greedy continuation that reproduces ≥ 50 contiguous tokens of the true file.
    • We kept function-implementation hits and removed license text and duplicates.
    • For Qwen, we also removed hits explained by copying from the prompt context. This filter was set from an LLM-assisted inspection of 50 random hits, 10 of which were re-checked by an author.
    • We then unlearned 300 targets.

Qwen2.5-Coder-7B-Instruct (pretraining memorization, 300 targets):

MethodF-BLEU ↓Extraction N50 ↓N100 ↓HE+MBPP+FU-H ↑
Base model0.72829868121245—
UNBIND0.035505118582.42
DD0.208——10521377.96
TaskVector0.6832745511824911.64
SimNPO-KL0.693284581232449.22
FLAT0.702283611202446.93
PROD0.720294651212452.29
GSSn/a (requires a pre-memorization model)

StarCoderBase-7B (pretraining memorization, 300 targets):

MethodF-BLEU ↓N50 ↓N100 ↓HE+MBPP+FU-H ↑
Base model0.6992855737149—
UNBIND0.2171723214780.16
MemFree (output filter)0.3470*0*3714966.87
DD0.151——201856.33
FLAT / PROD / SimNPO-KL / TaskVector0.695–0.721284–29053–5936–39151–1540.00–1.26

(*MemFree blocks 10-gram matches by construction; after identifier normalization, 38 targets still leak ≥ 32 tokens.)

What these results show:

  • No retrained model exists in this setting. Pretraining memorization is the case where retraining is infeasible, and the case the reviewer considers most important.
  • The LoRA unlearning baselines barely change memorization with their standard configurations.
  • UNBIND has the best forgetting–utility trade-off (FU-H) on both models.
  • The Qwen utility cost has a traceable cause. On Qwen, the utility cost is larger than in the fine-tuning settings, and we traced it to the calibration data. The retain calibration set here consists of raw repository files. The gate fires on 0% of these tokens but on ~10% of HumanEval+ completion tokens, so the retain budget did not constrain the intervention strength. Because the budget is defined on the operator's retain set, including instruction-style code in it lets the budget govern this cost directly (Appendix G.2). On StarCoderBase the budget was binding, and utility is essentially preserved (HE+ 37 → 32, MBPP+ 149 → 147).
AuthorsOpenReview ↗

Response to Reviewer tHa4 Part2

(continued from Part 1)

Does the method scale with the retain set? UNBIND does not train on the retain set. It only needs (i) first- and second-moment statistics of retain hidden states for the detector and (ii) a small calibration set for the budget. Both saturate quickly. Varying the retain fit data from 60 rows to the full pool, FU-H saturates at about 1,920 rows (Qwen-CSN 98.98 vs. 98.99 with the full pool; StarCoderBase 82.70 vs. 82.37). The statistics can be accumulated in a streaming fashion, so the cost is linear in the number of retain tokens processed. In our experiments, the required amount saturated at about 1,920 rows instead of growing with the size of the pool (Section 5.7, Table 5, Appendix G.2).

Is retraining cheaper and more robust?

  • Pretraining memorization. Retraining a base model is out of reach, while UNBIND needs only a few GPU-hours of calibration.
  • Fine-tuning memorization. A single UNBIND calibration costs less than one retraining (1.44 vs. 1.80 GPU-h on Qwen-CSN). Later requests are handled incrementally in about 13 min each, while retraining must be repeated for every request.
  • Robustness. UNBIND's FU-H is higher than the retrained model's in all four settings (96.27–98.79 vs. 62.03–86.63). Under eight extraction attacks, merged N50 is 0–10 for UNBIND, vs. 95 for M_retrain on Qwen-Stack (Appendices G.4, G.9, G.11).

Thank you for the response. To clarify my concern with scalability, the proposed method needs to work with matrices such as HcH_c or ∑c\sum_c , with c∈f,rc\in \\{f, r\\}. Those matrices grow linearly or quadratically with the size of forget or retain sets. At pre-training scale, both sets can grow up to million or billion level, making those matrices unmanageable.

The Part 2 rebuttal seems suggesting that we only need a small subsample of HrH_r. It's not straightforward to me why a few thousand samples can effectively represent the full pre-training distribution which should be retained. Furthermore, this does not address the case where we want to unlearn millions of samples.

Reviewer_FTLuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UNBIND, an inference-time machine unlearning framework for Code LLMs that selectively suppresses forgotten knowledge without modifying model parameters. UNBIND operates by steering hidden states at a selected Transformer layer and consists of three key components.

First, a Fisher-based target state detector identifies hidden states associated with the forget set by learning a discriminative direction from the mean and covariance statistics of forget and retain representations. Second, a gradient-guided suppression direction is derived through an eigenvalue optimization of a symmetric cross-moment matrix constructed from forget-set hidden states and target-token loss gradients. Third, a calibrated gating mechanism determines when and how strongly to intervene, with its hyperparameters selected under constraints on retain-set KL divergence, negative log-likelihood, and free-generation BLEU.

During inference, UNBIND applies a directional update only when the detector's activation exceeds a calibrated threshold, leaving other hidden states unchanged. The framework is evaluated against 14 baseline methods on two Code LLMs (Qwen2.5-Coder-7B and CodeLlama-7B) and two code corpora (CodeSearchNet and The Stack). The experiments assess unlearning effectiveness, retained utility, and resistance to several prompt-based extraction strategies.

Critical strengths

1. A parameter-free and selective unlearning mechanism.

A key strength of UNBIND is that it performs unlearning entirely through inference-time hidden-state interventions without modifying the original model parameters. This avoids the parameter drift and potential catastrophic forgetting associated with fine-tuning-based unlearning methods. Moreover, the calibrated gating mechanism restricts interventions to hidden states identified as forget-related, providing a principled way to balance forgetting effectiveness and the preservation of general code-generation capabilities. This makes UNBIND an interesting alternative for scenarios where modifying or retraining the underlying model is undesirable.

2. Comprehensive empirical comparison and evaluation.

The paper evaluates UNBIND against 14 baseline methods across two representative Code LLMs and two large code corpora. The evaluation considers both forgetting effectiveness and retained utility, and additionally examines performance under several prompt-based extraction conditions. This relatively broad experimental coverage provides useful evidence for assessing the effectiveness of the proposed framework and its trade-offs against existing unlearning approaches.

Critical weaknesses

1. The inference-time intervention does not provide persistent or irreversible unlearning, limiting its security guarantees and practical applicability.

UNBIND leaves the original model parameters entirely unchanged and suppresses forgotten knowledge only through an inference-time intervention. Consequently, the targeted information remains encoded in the underlying model and may still be accessible if the intervention mechanism is disabled or bypassed. In particular, once the original model weights are distributed, an attacker with access to them can simply remove the intervention hook and recover the original model behavior.

This raises a fundamental concern regarding whether UNBIND achieves machine unlearning in the conventional sense or primarily implements an inference-time access-control mechanism. While the latter can be useful in controlled deployment environments, it does not provide persistent forgetting guarantees for open-weight models or adversaries with control over the inference pipeline. This limitation substantially narrows the practical applicability of the method, especially in scenarios involving sensitive or copyrighted training data.

2. The single-direction intervention raises concerns about scalability, generalization, and robustness against targeted attacks.

UNBIND relies on a single linear detector direction ww and a single rank-one intervention direction vv at one selected Transformer layer. This design assumes that the hidden representations associated with the entire forget set can be effectively characterized and suppressed through a single pair of directions. However, this assumption may not hold for large-scale or heterogeneous forget sets containing thousands of code snippets with diverse semantics, programming languages, and structural patterns. In such settings, a single linear detector may fail to identify all forget-related states, while a shared suppression direction may inadvertently affect unrelated code-generation behavior.

The current experiments do not sufficiently establish the scalability or generalization of this assumption. In particular, the forget sets used in individual experiments appear relatively small, and the paper does not report results on an independent held-out forget test set. Furthermore, the multilingual experiments fit separate intervention configurations for individual languages, rather than evaluating whether a single detector and suppression direction can handle a large, mixed-language forget set.

This concern is further compounded by the limited adversarial robustness evaluation. Although Section 5.5 considers six extraction conditions, including prefix injection, instruction rephrasing, and code fences, these conditions primarily represent standard prompt variations rather than targeted attacks against the intervention mechanism. Since gating decisions depend on projecting hidden states onto a fixed direction ww, an adaptive adversary might construct prompts that shift forget-related hidden states below the activation threshold while still eliciting the targeted content.

These limitations directly affect the central claim that UNBIND can selectively suppress forgotten knowledge while preserving general model utility. Without stronger evidence on larger and more heterogeneous forget sets, held-out examples, and adaptive attacks, it remains unclear whether the proposed single-direction design can generalize beyond the relatively constrained experimental settings.

Questions for authors

1. What threat model and unlearning guarantees does UNBIND assume?

Since the original model parameters remain unchanged, users with access to the model weights or inference pipeline can remove the intervention module and restore the original behavior. Do the authors explicitly assume a trusted, controlled inference environment where the intervention cannot be disabled? How do they distinguish UNBIND from inference-time output suppression or access control, particularly in the context of machine unlearning requirements?

2. How does UNBIND scale to larger and more heterogeneous forget sets?

Could the authors provide results showing how forgetting effectiveness, retain utility, and detection false-positive/false-negative rates change as the forget set grows substantially? In particular, can a single pair of directions (w,v)(w,v) reliably handle a large mixed-language forget set, rather than fitting separate intervention configurations for individual programming languages?

3. Does UNBIND generalize to unseen forget-related examples?

Are the forget-set examples used for fitting the detector and suppression direction disjoint from those used for evaluating forgetting effectiveness? If not, could the authors evaluate UNBIND on an independent held-out forget test set, including semantically equivalent but syntactically different code snippets? This would help distinguish genuine generalization of the unlearning mechanism from suppression of representations closely resembling its fitting data.

4. How robust is the Fisher detector against adaptive adversarial attacks?

Section 5.5 evaluates several prompt-based extraction conditions, but these do not appear to explicitly target the gating mechanism. Have the authors considered adaptive attacks designed to reduce the detector activation w⊤h−srefw^\top h-s_{\mathrm{ref}} below the gating threshold while preserving the model's ability to generate forgotten content? Evaluating such attacks would help clarify whether the fixed linear detector is robust to deliberate attempts to evade intervention.

5. [Clarification] What is the offline computational cost of selecting the intervention layer?

The proposed framework requires identifying a suitable Transformer layer before constructing the detector and suppression direction. Could the authors report the GPU-hours or computational overhead associated with layer selection, ideally separating this cost from detector fitting, eigenvector computation, and calibration? How is this cost expected to scale with model size and the number of candidate layers? Since UNBIND emphasizes avoiding parameter updates, a clearer accounting of its offline preparation cost is important for assessing its practical efficiency.

2 comments
AuthorsOpenReview ↗

Response to Reviewer FTLu Part1

W1 / Q1 — Threat model and persistence. UNBIND targets deployments where the provider controls the inference stack: API serving, hosted models, and compliance takedowns. These are the settings where deletion requests for copyrighted or sensitive code are actually received and must be acted on quickly. In this setting, UNBIND gives three things that weight-level unlearning does not:

  • Fast response to new requests. A request is handled in minutes, incrementally (about 13 min per batch of 200).
  • An auditable, reversible artifact. The intervention can be inspected and rolled back.
  • No parameter drift. The ability to generate ordinary code is unaffected (see Q1 of MmoF).

Section 3 now states this threat model explicitly. Open-weight release, where the user controls the weights, is out of scope; distilling the intervention into weights is a natural extension.

W2 / Q2 — Larger and mixed-language forget sets. We trained a model that memorizes 2,000 CodeSearchNet functions, 500 each in Python, Java, JavaScript and Go. We then unlearned nested requests of 100 ⊂ 300 ⊂ 1,000 ⊂ 2,000 targets with a single (w, v) pair:

Request sizeF-BLEU (requested)N50 (orig. model → UNBIND)HE+ / MBPP+FU-HCalibration
1000.004235 → 071 / 17797.441.30 h
3000.0043113 → 067 / 19298.231.44 h
1,0000.0069374 → 072 / 18898.312.15 h
2,0000.0061745 → 073 / 18597.763.18 h
10 × 200 incremental0.0049745 → 069 / 18597.4213 min / batch

(Original model: HE+ 70, MBPP+ 195.)

  • Detection error rates. The detector fires on 90–98% of requested-target tokens (false negatives 2–10%). It fires on 0.09–0.17% of retain-test tokens and 0.2–1.1% of HumanEval/MBPP generation tokens (false positives).
  • Comparison. Baselines on the same model: SimNPO-KL reaches FU-H 63.4 at 300 targets and 87.1 at 2,000. GSS collapses the model at 300 targets (HE+ 1, FU-H 0).
  • Fallback not needed. We had pre-registered a multi-direction (k-means) fallback, to be triggered if FU-H fell 3 points below the 300-target level. It was never triggered (Section 5.7, Appendix G.4).

Q3 — Held-out targets and semantically equivalent variants.

  • Held-out targets. The paper's F-BLEU includes fit targets. We added five-fold held-out evaluation: each fold fits on 240 targets and evaluates on the other 60. Held-out F-BLEU is 0.0036–0.0123 across the four settings, the same as in-fit (0.0029–0.0124). Held-out N50 is ≤ 0.4 per fold, vs. 19–37 for the original model, and FU-H is 95.9–99.3.
  • Semantic variants. Tested on the two CSN settings:
VariantQwen-CSN, F-BLEU / N50 (orig. → UNBIND)CL-CSN, F-BLEU / N50 (orig. → UNBIND)
Original0.410 / 119 → 0.0053 / 00.448 / 186 → 0.0041 / 0
Identifiers renamed (AST)0.192 / 18 → 0.0051 / 00.209 / 31 → 0.0037 / 0
Docstring paraphrased0.276 / 60 → 0.0054 / 00.340 / 111 → 0.0036 / 0
  • Full reformatting. After complete reformatting (comments removed, black multi-line layout), the original model itself barely reproduces the targets (N50 5–13), and UNBIND leaves 3–6.
  • Comparison with output filtering. UNBIND is robust to rewrites that defeat output-level filtering: after identifier normalization, 0–2 targets leak ≥ 32 tokens, vs. 36–78 for MemFree (Appendices G.3, G.10).
AuthorsOpenReview ↗

Response to Reviewer FTLu Part2

(continued from Part 1)

Q4 — Adaptive attacks against the gate.

  • Black-box attacks. We ran eight black-box extraction attacks: identifier renaming, comment/whitespace injection, 1k/2k-token distractors, cross-language prompts, few-shot priming and chunked extraction. Each attack makes 1 greedy and 4 sampled attempts.
Qwen-CSNQwen-StackCL-CSNCL-Stack
Merged N50 over all attacks (orig. model / UNBIND)151 / 0184 / 10219 / 0228 / 4

On Qwen-Stack, even the retrained model that never saw these targets reaches merged N50 95, because much of that corpus is common code. UNBIND's 10 is far below this level.

  • White-box attack. We also ran a white-box GCG attack with a sigmoid-relaxed gate, optimizing 20 adversarial tokens to minimize the NLL of the memorized continuation. It lowers the gate firing rate from 0.97 to 0.50 but recovers nothing: N50 stays 0 and BLEU 0.006. Refitting w on attacked states restores the firing rate to 0.92 (Appendix G.11).

Q5 — Cost of layer selection. Stage-wise, a full layer sweep under the paper's protocol (one all-layer extraction plus a fixed (q, κ) evaluation per layer) costs an extrapolated 12.8 GPU-h. Layer choice is not critical: FU-H forms a plateau in deep layers, and simply taking the last layer costs 0.44 FU-H on average. This rule needs no sweep at all. Detector fit and the eigen solve together take < 0.01 h; see MmoF W2a for the per-stage breakdown (Appendix G.9).

Reviewer_MmoFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UNBIND, a code unlearning framework that separates recognizing target-related representations from suppressing target predictions. It combines a Fisher detector, a gradient-guided output direction, and a calibrated gate while keeping model and adapter weights fixed. Experiments on CodeSearchNet and The Stack show reduced target code reproduction with limited utility loss.

Critical strengths
  1. The paper addresses a practically meaningful problem: suppressing specific code implementations without broadly degrading programming ability. The reported forgetting-utility tradeoff offer useful reference value.
  2. The paper is well organized and easy to follow. It explains the experiments in detail and provides open-source code, making it easier to reproduce the results.
Critical weaknesses
  1. The novelty and design need clearer explanation.

    a. Existing method [1] already constructs separate detection and intervention directions and performs selective updates via gating. Therefore, this paper needs to further clarify the specific design advantages of the Fisher detector, the output-direction construction, and constraint calibration.

    b. Sec. 4.1 and Appendix B.2 provide the statistical motivation and derivation for the Fisher discriminant, but this does not directly amount to “being able to accurately locate the positions requiring intervention”; more direct mechanistic explanations or experimental evidence are still needed.

    c. Sec. 4.2 and Appendix B.2 establish the optimality of the selected output axis under a given auxiliary update and local first-order approximation. However, the actual deployed update magnitude is determined by the Fisher detector and the gate, and the two are not the same optimization problem. Therefore, the current derivation cannot directly explain why this design is superior to more direct control of the deployment objective.

  2. The experimental analysis could be more complete.

    a. Preparation cost, inference efficiency, and their relationship with data scale are not sufficiently quantified. The paper reports the sizes of the forget and retain data under the current setting, but it is necessary to distinguish offline costs from online inference overhead and to analyze how the number of forgetting targets and changes in content distribution affect calibration time, inference speed, and experimental results. Stating only that the deployment-stage computation is small is insufficient to judge overall efficiency and scalability.

    b. Hyperparameters are not sufficiently analyzed. Fig. 3 shows the relationship between w and v and the candidate ranges. Given that the intervention layer, gate threshold, and intervention strength may all significantly affect results, and that constraint calibration is itself part of the method, the current evidence is insufficient to distinguish whether the gains come from the method design or from configuration choices.

    c. Component effectiveness and robustness require more targeted validation. Existing ablations support the roles of the detector, independent output direction, and selective gating, but related work already has different implementations; further comparison is needed to determine the independent contribution of the components designed in this paper.

[1] Zhang X, Shang H, Li X. GSS: Gated Subspace Steering for Selective Memorization Mitigation in LLMs[J]. arXiv preprint arXiv:2602.08901, 2026.

Questions for authors
  1. What exactly does “target code reproduction” cover, and what conclusions can the evaluation support? Is the goal limited to lexical reproduction of designated forget continuations, or does it also cover underlying algorithmic logic, semantically equivalent implementations, or the use of specific APIs? These objectives have different success criteria and should be clearly distinguished. The independent programming tasks used for utility evaluation allow multiple correct implementations. A model could become less likely to generate one implementation while maintaining functional correctness by producing another. Therefore, the overall pass counts alone do not reveal whether capabilities directly associated with the forget targets have been affected. Meanwhile, F-BLEU and exact-span recovery mainly measure reproduction of designated target continuations. Could the authors provide examples or experiments that distinguish suppression of particular code-token sequences from broader changes to the algorithmic knowledge or computational patterns shared by target and retained code?
  2. Could the authors clarify the retain calibration split? The paper reports 120 retain calibration source rows for measuring the KL, NLL, and BLEU constraints during constrained (q,κ)(q,\kappa) selection. However, the split_calibration_by_row_id function in fresh_fit/calibration.py, around line 93, further splits the retain data using floor(0.8*n). The retain calibration-validation subset actually used for candidate selection may therefore contain fewer source rows than the reported retain calibration set.
3 comments
AuthorsOpenReview ↗

Response to Reviewer MmoF

W1a — Design advantages over GSS. We ran a controlled 4×4×2 component swap: probe {Fisher, logistic regression, mean-diff, GSS probe} × output direction {eigen, GSS b₁, direct, w} × gate {UNBIND, GSS}. Every cell uses the same calibration search, budget and selection rule.

  • The discriminative detector is the key advantage. Over two full grids, the mean FU-H is 98.45 / 98.49 with the Fisher probe and 98.05 / 98.52 with the logistic probe. With the GSS probe it is 8.58 / 9.49. At token level, the Fisher detector separates held-out forget states from retain, utility, semantically related code and other memorized code with AUC 0.992–1.000 (GSS probe 0.42–0.60). At a matched firing rate, its false-trigger rate is ≤ 0.6%, vs. 45–92% for the GSS probe.
  • Budget-constrained calibration gives a retain guarantee that GSS lacks. Under the same retain budget, GSS has no feasible configuration in any of the four settings. Its native configuration already exceeds the budget (retain KL 0.39 / 0.56 vs. 0.01).
  • No reference model needed. GSS splits tokens using a pre-memorization model. UNBIND only needs forget and retain data, so it applies to pretraining memorization, where GSS cannot be fitted (see tHa4).
  • The one-sided, activation-scaled gate matters in harder settings. Replacing it with the GSS gate lowers FU-H from 96.57 to 52.71 on Qwen-Stack.

We now state the contribution as "discriminative detector + budget-calibrated one-sided gate" (Introduction, Related Work, Appendix G.5).

W1b — Does the detector locate where to intervene? We tested this directly. The detector fires on 90–98% of forget-target tokens, 0.09–0.17% of retain-test tokens and 0.2–1.1% of HumanEval/MBPP generation tokens, with token-level AUC 0.992–1.000 against every negative family. Section 4.1 now states that the detector identifies hidden states belonging to the forget targets (Appendix G.6).

W1c — Auxiliary objective vs. deployment objective. We agree, and Section 4.2 now presents the eigen derivation as a gate-free auxiliary criterion rather than as optimality for the deployed update. Empirically, more direct control of the deployment objective does not help. A direction learned end-to-end through the real hard gate (300 Adam steps, an upper-bound reference) reaches FU-H 98.29 on Qwen-CSN vs. 98.68 for eigen. On Qwen-Stack it reaches 98.24 vs. 98.42–98.96 for eigen. The closed-form direction is therefore as good as optimizing the deployment objective, without any iterative optimization (Appendix G.7).

W2a — Offline and online cost, and their dependence on scale.

  • Offline, one layer (RTX 3090). Fit plus constrained calibration takes 1.44 GPU-h on Qwen-CSN, vs. 1.80 GPU-h to retrain M_retrain for every request. By stage: generation-BLEU check 0.98 h, per-target VJP 0.40 h, teacher scoring 0.04 h, Fisher solve and eigh < 0.01 h.
  • Scale. From 100 to 2,000 targets, calibration time grows from 1.30 h to 3.18 h and FU-H stays at 97.4–98.3. With incremental requests (10 × 200 targets), each batch takes about 13 min. The incrementally updated (w, v) matches a full refit (cos ≥ 0.9999999).
  • Online. Per-step decoding overhead is 0.3–3.7% across batch {1, 8, 32} × prompt {512, 2048}. Prefill overhead is ≤ 0.6%, with no extra memory (Appendices G.4, G.9).
AuthorsOpenReview ↗

Response to Reviewer MmoF Part2

W2b — Method design vs. configuration choices. The gains come from the design:

  • Component swap. Under an identical search budget, the probe choice dominates (see W1a).
  • Calibration set. Enlarging the calibration validation set changes FU-H by ≤ 0.3 on three settings. On Qwen-Stack, the larger set raises FU-H from 96.27 to 98.42–98.96.
  • Ridge coefficient. λ from 10⁻⁴ to 10⁻¹ gives FU-H 98.42–98.71, and the selected point does not change with the budget ε on Qwen-CSN.
  • Retain data volume. Results saturate at about 1,920 retain rows.
  • Layer choice. FU-H forms a plateau in deep layers; taking the last layer costs only 0.44 FU-H on average (Appendices G.2, G.5, G.8, G.9).

W2c — Independent contribution of each component. See W1a. The probe is decisive; the gate is second (swapping in the GSS gate costs 0.3–44 FU-H); the output direction has a small effect.

Q1 — Lexical reproduction vs. algorithmic knowledge. Our target is verbatim and near-verbatim reproduction of designated code, and the revision states this scope explicitly. Within that scope, suppression is not merely lexical:

  • Similarity at every level falls. CodeBLEU drops from 0.53–0.71 to 0.01–0.09, AST match from 0.66–0.74 to 0.02–0.07, called-API Jaccard from 0.52–0.65 to ≤ 0.008, and UniXcoder similarity from 0.75–0.82 to 0.07–0.11.
  • Near-verbatim rewrites are also blocked. After identifier normalization, at most 0–2 targets still leak ≥ 32 contiguous tokens, vs. 36–78 for an output-level n-gram filter (MemFree).
  • General programming ability is untouched. Of 542 HumanEval+/MBPP+ tasks, only 7 change pass/fail status on Qwen-CSN and none on Qwen-Stack. For M_retrain the numbers are 94 and 58. This is the smallest change among all methods, and the tasks most similar to the forget targets show no concentrated loss (permutation p ≥ 0.18).
  • Functional ability tied to the targets stays at the level of the retrained model. When prompted only with the signature and docstring of a forgotten function, UNBIND's pass@1 equals that of the retrained model, the gold standard for unlearning (0.111 vs. 0.111) (Appendix G.10).

Q2 — Retain calibration split. Thank you for catching this. The reviewer is correct: the 0.8 split leaves 24 (CSN) / 25 (Stack) retain validation rows, and we have corrected the text. Expanding the validation set to 120 and 480 rows leaves the conclusions unchanged (W2b). See Table 5, Appendices A.1, B.3 and G.8.

Thank you very much for answering my questions. These additions are helpful, but three issues remain insufficiently clarified:

  1. Design advantages. The Fisher detector performs similarly to logistic regression, while the eigen output direction has a limited independent contribution (Figure 5). Direct also achieves a higher FU-H than Eigen on Qwen-CSN (Table 16). The specific advantages of these design choices therefore remain insufficiently established.

  2. Efficiency and intervention novelty. Single-layer fitting and constrained calibration cost 1.44 GPU-h on Qwen-CSN and 2.64 GPU-h on Qwen-Stack, while the full layer sweep has an extrapolated cost of 12.8 GPU-h. The strong last-layer results also leave me unconvinced about the necessity and distinctive contribution of the proposed intervention mechanism relative to simpler output-level reproduction control.

  3. Functional preservation. The evaluation in Appendix G.10 covers 36 targets with valid generated tests, with UNBIND and the retrained model both achieving a pass@1 of 0.111 (approximately 4/36). These matching scores provide limited evidence of preserved target-related functionality.

Therefore, I maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.