acceptodds
Under review as a conference paper at ICLR 2027

SAE Interventions Can Be Unreliable: Post-Intervention Recovery of Suppressed Behavior

Abstract

Sparse autoencoders (SAEs) decompose residual-stream activations into interpretable features and increasingly support control mechanisms that monitor or clamp features associated with unwanted behavior. When a clamp suppresses its target behavior, the intervention is commonly treated as successful. This evaluation leaves a stronger question unresolved: has the behavior become inaccessible, or has one visible route merely been blocked? We introduce post-intervention recovery, a constrained residual-space audit of whether an SAE intervention forms a complete behavioral bottleneck. Starting from a successfully defended state, the audit searches for bounded residual perturbations that restore the suppressed behavior while the exact clamp remains active throughout optimization and generation. Encoder-orthogonal and feature-map Jacobian projections limit direct movement of defended readouts. Across Targeted Probe Perturbation (TPP), unlearning, Indirect Object Identification (IOI), and refusal steering, the audit repeatedly recovers behaviors that successful feature interventions had suppressed. In the safety-critical AdvBench setting, 20.7% of the tested valid flips become HarmBench-classified harmful under the active clamp. Frozen recovery directions transfer to held-out prompts and substantially outperform equal-norm random controls, while matched additive-steering defenses withstand the same search. Together, these results provide counterexamples to treating selected SAE features as complete behavioral bottlenecks under the evaluated perturbation classes. Behavioral completeness is therefore a distinct validation target for SAE-based control.

Primary area:
interpretability and explainable AI
Keywords:
Sparse Autoencoders, Mechanistic Interpretability, AI Safety, Latent-Space Interventions, Model Steering, Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u6saOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes post-intervention recovery to investigate whether an SAE intervention forms a complete behavioral bottleneck. The method searches for bounded perturbations that maximally restore the suppressed behavior after successful SAE clamps. Experimental results show that multiple suppressed behaviors can be successfully recovered across different task scenarios, indicating an important finding that SAE interventions are not necessarily complete behavioral bottlenecks.

Critical strengths
  • Clear problem formulation: the paper clearly distinguishes SAE interventions from complete behavioral bottlenecks, which is an important but frequently neglected point.
  • Strong theoretical support: the proposed behavior optimization is mathematically grounded.
Critical weaknesses
  • Limited real-world practicability: the method requires white-box access to residuals and gradients. Such recovery cannot be applied through adversarial inputs, making the distinction practically less critical.
  • Over-reliance on fallback: the proportion 20.7% is not quite impressive. Moreover, the successful examples are mostly achieved by the fallback plan. The effectiveness of the optimization strategy is not clear.
Questions for authors
  • Can recovery be achieved without white-box support? Otherwise, the distinction between SAE intervention and complete behavioral bottleneck is less practically significant.
  • How can the method prove that recovery is achieved by re-activating the suppressed operations rather than creating a new adversarial path?
  • It seems that most successful attempts do not meet the optimization requirements. Can the authors provide more concrete evidence that the optimization strategy is reasonable?
4 comments

Thank you for distinguishing the practical access assumptions from the theoretical claim and for asking us to account for fallback explicitly.

>>> W1&Q1 White-box access and practical scope

>>> A1 Thank you for asking about the practical access assumptions. We clarified the purpose and access model in Section 4, Section 5, and Appendix H. Post-intervention recovery is a defense diagnostic for model developers with access to internal activations and gradients, rather than an attack method. It tests whether a fixed SAE intervention sustains behavioral suppression under bounded residual perturbations. The result establishes recovery under this internal access model. Input-only recovery concerns the feasibility of adversarial attacks, which falls outside the scope of our defense diagnostic study.

>>> Q2 Behavioral recovery and circuit identity

>>> A2 Thank you for distinguishing behavioral recovery from circuit identity. We clarified the claim in Section 6.4, Appendix C.3, and Appendix H, and report the IOI controls in Appendix D.2. The audit measures whether suppressed behavior returns while the intervention remains active. The IOI controls show that recovery persists under joint encoder–decoder orthogonality and that freezing the reconstruction error reduces the recovered margin.

>>> W2&Q3 Fallback selection and hard drift constraints

>>> A3 Thank you for asking how checkpoint selection affects the reported recovery rate. We clarified the selection rules in Section 6.1, Table 1, and Appendix D.3. At a perturbation budget of 50, the fixed SAE intervention remained active and 6/29 eligible prompts recovered. All six recoveries used fallback checkpoint selection and exceeded the preferred relative drift threshold of 0.03.

Evaluation settingHarmful recovery
Budget 50, fallback enabled, drift measured after selection6/29 (20.7%)
Five budgets, retrospective count of distinct prompts with recovery at measured drift ≤ 0.033/29 (10.3%)
Budget 50, hard drift cap of 0.05, fallback disabled0/29 (0.0%)
Budget 50, hard drift cap of 0.10, fallback disabled1/29 (3.4%)

With fallback disabled, recovery still occurred on 1 of 29 eligible prompts at a hard drift cap of 0.10. At a cap of 0.05, the search found no feasible candidate. The 3/29 figure counts distinct prompts that recovered with measured drift at or below 0.03 across five budget runs.

The responses address most of my concerns. I maintain my original rating (3).

Dear Reviewer u6sa,

Thank you for your feedback! We are glad that our responses addressed your concerns. If there is anything else we can clarify, please do let us know. Thank you for your support!

Best regards, The authors

Reviewer_GDEjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether an SAE clamp makes a suppressed behavior inaccessible. Its audit searches for bounded activation perturbations while keeping the clamp active. Single-layer encoder constraints are exact under the stated assumptions. Multi-layer experiments use approximate Jacobian projections and measure feature drift. Recovery is evaluated only where the intervention first suppressed the behavior. Experiments cover TPP, unlearning, IOI, and refusal steering. Held-out transfer and random controls support the existence of recovery beyond independent per-prompt search.

Critical strengths
  • A useful test beyond clean suppression. The distinction between changing behavior and making that behavior inaccessible is a strength of the paper. The audit keeps the full clamp active during optimization and generation. It therefore tests an important property that clean suppression alone does not establish.

  • Transfer and negative controls strengthen the evidence. Two-fold evaluation with frozen recovery directions yields 11/29 harmful outputs. Each direction is trained only on the other fold. Equal-norm random controls produce at most 1/29 under that protocol. Measured feature drift is nonzero, ranging from 0.035 to 0.048. The same search also fails against the evaluated CAA and mean-shift defenses. These results support recovery beyond individual optimization of each evaluated prompt.

Critical weaknesses
  • The behavioral evidence remains narrow. The main harmfulness evaluations contain 29 valid prompts on Gemma-2-2B and eight on Gemma-2-9B. The transfer experiment reuses the same 29 prompts. The unlearning experiment has six underlying questions, despite its 91 answer-choice permutations. Cross-distribution harmful recovery is also mixed: 7/16 on JailbreakBench and 0/14 on MaliciousInstruct. The authors acknowledge these limits and report confidence intervals. These results establish failure cases, but leave uncertainty about how consistently recovery occurs across prompts and intervention settings. A larger evaluation on independent examples would strengthen the practical significance of the findings.

  • The perturbation budget needs a more relevant reference. Let hh be the activation and vv the clamp's edit. Equation 1 can be written as I(h)=h+vI(h)=h+v. Under the exact single-layer assumptions of Proposition 1,

    I(h+Δ)=h+v+Δ.I(h+\Delta)=h+v+\Delta.

    Appendix D.3 already includes budget sweeps and reports ∥Δ∥/∥h∥≈0.009\|\Delta\|/\|h\|\approx0.009 for the strict refusal audit. This does not show how large the perturbation is relative to the defense's own modification (vv). Please report ∥Δ∥/∥v∥\|\Delta\|/\|v\| across edited sites. For CAA and mean-shift, report edit magnitudes and retained benign utility. The paper acknowledges that benign utility was not measured. These measurements would contextualize the comparison. Recovery with the clamp active shows a failure under the tested budget. The missing comparison with vv limits how demanding I can judge this test to be.

Questions for authors

Q1. Can you extend the behavioral evaluation to more independent prompts and intervention settings? Please distinguish additional examples from repeated evaluations of the same examples.

Q2. How large is Δ\Delta relative to the clamp edit vv? Please report the distribution of ∥Δ∥/∥v∥\|\Delta\|/\|v\| across edited sites. What are the corresponding edit magnitudes and benign-utility results for CAA and mean-shift?

C1. The order comparison needs clarification. The paper compares adding Δ\Delta before the clamp with adding it after the clamp. These orders are provably equivalent under the stated single-layer assumptions. However, the experiment is described as using one layer in one place and several layers in another. Table 7 also reports harmful-output judgments for only one order. Similar rates of non-refusal do not establish similar rates of harmful answers. Please clarify the setup and evaluate harmful outputs for both orders. This is a secondary clarification and does not invalidate the main recovery results.

As I read the paper, the following controls would strengthen the mechanism discussion. They are suggestions, not prerequisites for the empirical counterexamples.

E1. Test compensation along decoder directions. The encoder reads features; the decoder writes the clamp's edit. Orthogonality to selected ENCODER-ONLY directions need not prevent Δ\Delta from opposing part of vv. The existing unsigned decoder-alignment diagnostic does not measure this opposition. Measure the signed alignment with the actual edit vv. In the exact single-layer setting, adding orthogonality to the selected decoder directions would test whether recovery persists without that direct compensation.

E2. Freeze the reconstruction error rather than remove it. Removing the error in F.9 eliminates harmful recovery, but makes all 24 outputs degenerate. The authors acknowledge this. C.3 supplies complementary replay evidence, without identifying a unique mechanism.

Let EE and DD be the SAE encoder and decoder. Let z~\tilde z denote the encoded features after clamping. Equation 1 at a perturbed activation gives

+\big[h+\Delta-D(E(h+\Delta))\big].$$ A proposed control keeps the bracketed error at its unperturbed value: $$I_{\mathrm{frozen}}(h+\Delta;h)=D(\tilde z(h+\Delta)) +\big[h-D(E(h))\big].$$ At $\Delta=0$, this equals the original defended operator. Re-optimize recovery and report output quality. Local changes then pass through non-clamped decoded features. For multiple layers, use an unperturbed reference on the same token prefix at every defended layer. This would clarify the role of error changes without requiring complete mechanistic identification.
4 comments

Thank you for identifying the distinction between intervention completeness and clean suppression, and for proposing controls that separate geometric compensation from behavioral recovery.

>>> W1&Q1a Independent prompts and effective sample size

>>> A1 Thank you for distinguishing independent examples from repeated evaluations. We clarified the refusal screening funnel in Section 6.3 and Appendix H, the underlying unlearning questions in Appendix D.2, and the transfer denominator and metric in Section 6.2 and Appendix F.1. The Gemma 2 2B AdvBench evaluation starts with 520 prompts. The base model produces HarmBench-classified harmful completions on 31 of those prompts. The SAE intervention turns 29 of the 31 into non-harmful completions. All 29 eligible prompts enter the recovery audit. The screening defines the denominator because a recovery test requires both a harmful base completion and successful suppression by the intervention. The two transfer folds reuse those 29 prompts, and the 91 WMDP answer permutations come from six underlying questions. Appendix F.1 reports 11/29 recoveries with held-out learned directions and at most 1/29 with the equal-norm random controls. The archived transfer records identify the 0.0349–0.0476 range as the perturbation norm relative to the original hidden-state norm. Section 6.2 and Appendix F.1 now use that metric label.

>>> W1&Q1b Consistency across distributions and intervention settings

>>> A2 Thank you for asking how the results vary across settings. We distinguish fixed-intervention transfer in Appendix F.2 from the separate 9B intervention in Appendix F.7 and report the added ABBA IOI cohort in Appendix D.2.

EvaluationEligibleHarmful recoveryWilson 95% CI
JailbreakBench, fixed 2B intervention16/10043.8% (7/16)23.1–66.8%
MaliciousInstruct, fixed 2B intervention14/1000.0% (0/14)0.0–21.5%
AdvBench, separate 9B intervention8 suppressed of 14 base-harmful87.5% (7/8)52.9–97.8%

The fixed-intervention results show heterogeneity across the two external cohorts. A normalized exact-text check found no shared prompts across these cohorts and the 2B AdvBench archive. This check measures text overlap. The cohort-level percentages describe recovery conditional on successful suppression in each benchmark. The 9B result establishes a failure case under a separate intervention, with a wide interval. The new ABBA cohort extends IOI coverage. The 2B cohorts used the same AdvBench-selected intervention without retuning. The 9B intervention was selected separately.

>>> W2&Q2a Perturbation magnitude relative to the intervention edit

>>> A3 Thank you for proposing the realized intervention edit as the reference. We report the same-site and per-prompt edit-relative distributions in Appendix D.3, using saved per-example perturbations from an archived rerun on the same 29 AdvBench prompts. The feature set, layers, and combined norm cap match the primary refusal protocol. At each layer and token in the saved support, we reconstruct v=I(h0)−h0v=I(h^0)-h^0 on the unperturbed defended trajectory with the same archived target-token prefix. The perturbations are compared offline, with no new generation or behavioral labels.

MeasurementSite level, nonzero editConcatenated within each prompt
Observations3,59429
∥Δ∥2/∥v∥2\|\Delta\|_2/\|v\|_2, median0.0420.089
Interquartile range0.021–0.0830.073–0.145
Full range0.00003–2.0560.037–0.668

Of 15,631 layer–token sites in the saved support, 12,037 have numerically zero clean edit. Using a 10−1010^{-10} denominator threshold, their edit-relative ratios are undefined and are counted separately. These are clean-reference ratios for the archived rerun. They contextualize the perturbation budget without imposing a new edit-relative constraint.

A separate single-layer IOI experiment yields 32 valid flips from 64 ABBA prompts, with all 32 recovered. Its same-token ratio has median 0.290, interquartile range 0.230–0.382, and range 0.184–0.670. All 32 edit denominators are nonzero, and measuring against the edit realized in the perturbed pass also gives a median of 0.290 to three decimals.

>>> W2&Q2b Additive defense identity and edit magnitude

>>> A4 Thank you for asking us to quantify the baseline edits. We consolidated the duplicate entries in Section 6.2 and report the nominal and realized edit magnitudes in Appendix F.4. The archived Contrastive Activation Addition (CAA) and mean-shift runs used the same mean-difference direction construction, layers, and scaling. They represent one linear steering baseline. The historical calibration implies nominal per-token edit norms of approximately 4.18–13.24 across edited layers. Repeating the original calibration matched the logged layerwise residual scales within 0.006. On 250 safe XSTest prompts, we measured the L2L_2 norm of the realized residual edit at the final input token during prefill. The 11 defended layers yielded 2,750 measurements per intervention. The linear baseline had median edit norm 7.68 (IQR 6.41–10.26, range 4.15–13.26). The SAE intervention had median 0.259 (IQR 0.120–32.26, range 0.048–600.03).

>>> W2&Q2c Retained benign utility at the tested strengths

>>> A5 Thank you for asking about retained benign utility. We added the matched utility comparison in Section 6.2 and Appendix F.4. We evaluated all three conditions on the same 128 GSM8K questions and 250 safe XSTest prompts, with identical decoding within each dataset.

ConditionGSM8K correctNumeric answer parsedXSTest prefix heuristic flagsXSTest independent judge full / partial refusals
Base model56.3% (72/128)108/1283.2% (8/250)70/250 / 12/250
SAE intervention50.8% (65/128)101/12824.8% (62/250)120/250 / 16/250
Linear steering baseline21.1% (27/128)39/12855.2% (138/250)185/250 / 38/250

Relative to base, GSM8K score changes were −5.5 percentage points for SAE (paired bootstrap 95% CI [−12.5, 0.8]) and −35.2 for linear steering ([−44.5, −26.6]). The fixed scoring rule credits a correct, parseable numeric answer. Both automated XSTest scoring methods rank the linear baseline highest in refusal frequency. The linear baseline has more refusal flags than SAE in 9 of 10 prompt categories under the official prefix heuristic and more full refusals in all 10 categories under the independent judge. The absolute estimates differ, and spot checks identified semantic judge errors. At the evaluated intervention strengths, the lower recovery observed for linear steering accompanies a larger measured utility cost.

>>> C1a Single-layer proposition and Table 7 setup (C1)

>>> A6 Thank you for flagging the discrepancy in layer count. We clarified the multi-layer setup in Appendix C.2 and Table 7, and state the fixed intervention rule in Sections 3 and 4. The ordering comparison in Table 7 uses a multi-layer refusal intervention. Proposition 1 establishes equivalence for a single-layer residual-preserving operator when the selected encoder readouts are unchanged by the perturbation. The main refusal results apply the intervention after adding the perturbation at every defended site. Refusal steering uses a fixed feature-amplification rule, and the invariant is execution of that rule at each forward pass. The rule's multiplier is fixed while its input activations can vary.

>>> C1b Matched harmfulness for both insertion orders (C1)

>>> A7 Thank you for requesting harmfulness judgments for both orders. We updated Table 7 and Appendix C.2 with matched HarmBench results from the two archived runs. The same classifier scored all 24 paired prompts and classified 5/24 completions as harmful when the perturbation was added before the intervention and 6/24 when it was added afterward. Three prompts yielded harmful completions in both runs, two only in the pre-intervention run, and three only in the post-intervention run (two-sided exact McNemar p=1.0p=1.0).

The base completions were identical across runs. HarmBench classified 12 of them as harmful and classified every unperturbed defended completion as non-harmful in both runs. On these 12 commonly eligible prompts, harmful recovery occurred on 3/12 before and 6/12 after the intervention (two-sided exact McNemar p=0.25p=0.25). The pre-intervention result therefore demonstrates harmful recovery under the active multi-layer intervention.

>>> E1a Signed alignment with the actual edit (E1)

>>> A8 Thank you for asking for signed alignment with the actual edit. We added site-level and per-prompt signed cosine measurements against the realized clean edit in Appendix D.3. Across the 3,594 sites with nonzero denominators, the signed cosine has median 0.004 and interquartile range −0.047 to 0.059. After concatenating matched sites within each prompt, the median is −0.026, with interquartile range −0.037 to −0.011, and alignment is negative on 26/29 prompts. The two aggregations summarize different distributions. Appendix D.3 reports their definitions.

In single-layer IOI, the corresponding median is −0.095, interquartile range −0.130 to −0.058, and range −0.192 to 0.044, with negative alignment in 29/32 cases. These signed measurements quantify opposition to the clean edit. The joint constraint in E1b tests whether a component along the selected decoder span is necessary in IOI.

>>> E1b Joint encoder–decoder orthogonality (E1)

>>> A9 Thank you for suggesting the joint orthogonality control. We report the encoder-only, joint, and matched-random conditions in Appendix D.2. Recovery persists under joint orthogonality. We held the prompts, intervention, perturbation budget, and optimization steps fixed in single-layer IOI. We compared encoder-only orthogonality, joint encoder–decoder orthogonality, and encoder orthogonality with an additional random span matched to the decoder's added rank. All three recovered 37/37 eligible BABA prompts and 28/28 eligible ABBA prompts. The maximum normalized constraint residual was below 6×10−86\times10^{-8}. Mean paired logit-margin differences between the joint and matched random conditions were −0.01 on BABA and +0.27 on ABBA, with 95% bootstrap intervals of [−1.43, 1.34] and [−1.78, 2.43]. The result rules out the need for a perturbation component along the selected decoder directions in these IOI cohorts.

>>> E2 Freeze the reconstruction error (E2)

>>> A10 Thank you for suggesting that we freeze the reconstruction error. We added the paired IOI recovery control and the multi-layer GSM8K utility comparison in Appendix D.2. In single-layer IOI, we froze the reconstruction error at the unperturbed reference. The zero-perturbation logits matched exactly. We re-optimized both conditions on the same prompts with the same budget and steps.

IOI cohortLive error, recoveredFrozen error, recoveredMean recovered logit margin, live → frozen
BABA37/3737/3771.73 → 28.26
ABBA28/2827/2870.00 → 27.21

Freezing the error lowers the recovered IOI margin, while recovery persists on 64 of 65 eligible prompts. The mean L2L_2 drift of the re-encoded activations increases from approximately 0.003 to 4.508 on BABA and 4.424 on ABBA. The refusal control in Appendix F.9 removes the error altogether and yields degenerate outputs on all 24 prompts.

We also measured benign utility on 128 GSM8K prompts under the same 11-layer refusal intervention. For each prompt, both arms received the same random perturbation, fixed before generation. The frozen arm used each layer's unperturbed reconstruction error during prompt processing. Both arms used the live error during token generation. The next-token logits matched exactly at zero perturbation on every prompt. Freezing reduced answer accuracy from 67/128 to 56/128 and the number of parseable answers from 95/128 to 80/128. The paired bootstrap 95% intervals for the changes were [-14.8, -1.6] and [-20.3, -3.1] percentage points, respectively. Both arms produced nonempty answers on all 128 prompts. The frozen error therefore carries a measurable utility cost in this multi-layer setting.

Reviewer_rgYAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper tackles a problem "does clamping a linear feature really suppress the behavior or there is a second path in the model, which restores this behavior?". They propose to use projected gradient descent in activation space to make the model restore the clamped behavior. Authors find that in several settings (such as TPP, unlearning, IOI, refusal direction) their method restores the suppressed behavior.

Critical strengths

The chosen approach is robust only with the hypothesis, that features are exactly linear. What happens if the feature is some kind of subspace?

Critical weaknesses

Did you analyse the optimized states Δ\Delta? There might be a case where a feature in the model (say, refusal) is not an exact vector but more like a subspace. Suppressing the SAE direction corresponding to it and then optimizing against this suppression might result in finding a similar feature, which is directed in the same subspace and which is essentially the same feature. Some qualitative analysis of the Δ\Delta states might help uncover such cases

The writing is decent, but the narrative is not. The paper mentions a lot of details where it should not (and includes 3 appendices per section) and does not where it should (like the optimization loss, the model/SAE taken, how the optimization features are chosen, etc.). Also, the figures are of poor quality, as are their captions.

Questions for authors

Definitions:

Line 110 - whats TT and dd (i assume seq length and model dim)

Line 140 - whats the dimension of Δ\Delta? Is it a vector or a matrix? In line 110 you define hh as matrix, so I assume delta is a matrix too? Then what’s the size of Δj\Delta_j? j is a token position or a dimension in activation?

Line 156 - what's Al,SA_{l, S}?

Line 157 - what’s ToptT_{opt}?

\ Line 202 - what's Lrec\mathcal{L}_{rec}? If it is defined in the appendix, please move it to the main text.

2 comments

Thank you for raising the subspace interpretation and identifying where the paper needs clearer definitions and figures.

>>> W1 Behavioral subspaces and the geometry of recovery

>>> A1 Thank you for raising the possibility that refusal is represented in a broader subspace. We clarified the scope of our claim and added a calibration-based analysis in Appendix F.5 and a matched IOI control in Appendix D.2. Our audit tests whether intervening on the selected SAE features makes the suppressed behavior inaccessible. If recovery uses another direction in the same behavioral subspace, it still demonstrates that the selected intervention is an incomplete behavioral bottleneck.

At rank 16 per layer, the calibration spans captured a median 1.84% of the saved prompt-anchor perturbation energy, compared with 0.70% for rank-matched random spans. In IOI, recovery persisted on all eligible prompts under joint encoder–decoder orthogonality. Together, these results support a claim about the selected intervention and quantify how the recovered directions relate to the measured spans.

>>> W2a Narrative and placement of essential details

>>> A2 Thank you for identifying the setup details readers need. We moved the key task settings into Section 5 and defined the intervention and recovery loss in Section 4. TPP uses the Gemma 2 2B base model. Unlearning and refusal steering use Gemma 2 2B Instruct. IOI uses GPT-2 Small with a residual-stream SAE, selects features by IOI attribution, and optimizes the correct-name versus competing-name logit difference at the answer position. Section 4 introduces the intervention rule and recovery loss before the diagnostics. Appendix G provides the detailed configurations.

>>> W2b Figures and captions

>>> A3 Thank you for flagging the figures and captions. We revised the schematic and caption in Figure 1 and clarified the metrics and comparisons in Figures 2 and 3. Figure 1's caption now distinguishes clean eligibility screening from the recovery forward pass and identifies the single-layer illustration and cross-layer Jacobian projection. The updated schematic shows a fresh residual entering recovery and uses the fixed intervention rule consistently. The result captions identify the axes, metrics, and conditioned recovery denominators. Figure 2 explains its paired markers and dataset means, and Figure 3 identifies its held-out and target-specificity comparisons.

>>> Q1 Sequence length and model dimension

>>> A4 Thank you for flagging these symbols. We defined both at first use in Section 3. TT is the sequence length for that forward pass and dd is the width of the residual stream. Each of the TT rows of hℓ(x)∈RT×dh_\ell(x)\in\mathbb R^{T\times d} is the residual vector for one token.

>>> Q2 Perturbation shape and index

>>> A5 Thank you for asking about the perturbation shape. We defined its dimensions and token index in Section 4, before Equation 2. At a single layer, Δ∈RT×d\Delta\in\mathbb R^{T\times d} has the same shape as hℓ(x)h_\ell(x). The index jj denotes a token position and Δj∈Rd\Delta_j\in\mathbb R^d is its perturbation vector. Rows outside the permitted positions are zero.

>>> Q3 Selected encoder directions

>>> A6 Thank you for asking how the directions are arranged. We defined the matrix and its columns in the notation paragraph of Section 4, before Equation 3. Aℓ,S∈Rd×∣S∣A_{\ell,S}\in\mathbb R^{d\times |S|} stacks the encoder directions for features in SS at layer ℓ\ell as columns. Equation 3 specifies orthogonality to each of these directions. The exact IOI control projects the realized perturbation after each Adam update and verifies the constraint.

>>> Q4 Permitted token positions

>>> A7 Thank you for flagging the token support. We defined the permitted positions in Section 4, summarized the task-specific support in Section 5, and report the full configurations in Appendix G. Topt⊆{1,…,T}\mathcal T_{\mathrm{opt}}\subseteq\{1,\ldots,T\} is the set of token positions at which perturbations may be optimized. TPP optimizes cached post-ablation sequence activations before probe pooling. WMDP-Bio and IOI perturb the answer-readout position. Per-prompt refusal recovery edits the prompt anchor and continuation positions, while the shared-direction experiment edits the prompt anchor.

>>> Q5 Recovery loss

>>> A8 Thank you for pointing out where the recovery loss needed a definition. We defined it before Equation 7 in Section 4, with the task objectives in Section 5 and full configurations in Appendix G. Lrec\mathcal L_{\mathrm{rec}} is the task-specific behavioral loss in Equation 7. For IOI it is the negative logit difference between the correct and competing names.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on Author Responses (For the Area Chair)

Dear Area Chair,

As the discussion period is approaching its end, we would like to briefly follow up on our author responses. We have carefully addressed the reviewers' concerns and provided additional clarifications and experimental evidence in the discussion threads.

We would greatly appreciate the opportunity to engage in further discussion and hear additional feedback, which would help us better address any remaining concerns and further improve the quality of our work.

Thank you for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.