acceptodds
Under review as a conference paper at ICLR 2027

CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuits

12 views

Abstract

Controlling the behavior of large language models (LLMs) at inference time is central to alignment, but existing steering methods such as Contrastive Activation Addition (CAA) rely on fixed single-layer interventions derived from aggregate activation differences. A single direction is applied to semantically diverse inputs, and the induced change is not sustained as representations propagate through later layers, which limits both the strength and the reliability of the resulting steering. We introduce CircuitSteer, a framework that uses Sparse Autoencoders (SAEs) to identify and manipulate coherent semantic circuits distributed across multiple layers of the residual stream. We construct an aligned co-activation circuit from feature co-activation and the geometric alignment of decoder directions, isolating the multi-layer subcircuits that jointly mediate a given target behavior. We then synthesize dense steering vectors from these sparse features and apply coordinated multi-point interventions along the circuit, guiding the model's internal semantic trajectory rather than perturbing a single site. We evaluate CircuitSteer with contrastive examples on three distinct behaviors, toxicity, emotion intensity, and sycophancy, using Gemma-2-2B and Llama-3.1-8B-Instruct. CircuitSteer is the only method that significantly reduces the target behavior while preserving fluency in every configuration; each baseline either degrades text quality or fails to shift the behavior on at least one configuration. These results show that multi-layer circuit steering, enabled by enforcing geometric alignment among selected features, yields more robust and more consistent behavioral control than static single-point interventions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
activation steering, sparse autoencoders, LLM alignment, inference-time intervention, representation engineering, multi-layer steering, controllable generation, toxicity mitigation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_w7Z4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a new method called CircuitSteer.
They challenge the idea of single-layer steering, commonly found in contrastive activation steering or SAE-based methods.
The authors employ SAEs to identify circuits distributed across multiple layers inside different LLMs.
They construct dense steering vectors from isolated subcircuits.
To evaluate their method, they try to influence three different kinds of model behavior: toxicity, emotion intensity, and sycophancy.
According to their empirical evaluation, CircuitSteer can reduce unwanted target behavior while maintaining model fluency under their chosen criterion. They also investigate the impact on other model capabilities.

Critical strengths

The motivation and goal of the paper are very clear and understandable overall. Especially Sec. 2 and the accompanying Fig. 2 make it easy to understand and follow the reasoning behind why this method is needed. In the broad sense, the experimental design supports the story well and makes the subsequent validation of the claims made in the beginning of the paper reasonable. In particular, the ablations investigating the influence of geometric alignment are interesting. The additional scaling experiment with Qwen3.5-27B, even if only on one dataset, is a very good addition, especially with the ever increasing model sizes.

Critical weaknesses
  1. Unexplained poor SAE-SSV performance in Tab. 1. The results somewhat contrast with the findings in the original SAE-SSV paper as well as the results on Gemma-2-2B. I think the high normalized PPLs are especially worth discussing further, as the authors state: "For every method and configuration, we report a single coefficient λ\lambda, chosen to maximize validation Δ\Delta subject to PPL ≤2.0\le 2.0". Which leads to the next uncertainty: CircuitSteer gets a range of λ\lambda extending from −16-16 to −0.25-0.25 on Llama and from −16-16 to −1-1 on Gemma, whereas the other methods only receive a range of λ∈−4,...,−1\lambda \in {-4,...,-1}. As the ranges differ quite substantially, the reasoning behind those decisions would be helpful. The exact values used for the experiments would also be interesting. Tab. 17 reports some of the tested lambda values. However, it does not show any values smaller than 1 in magnitude for the baselines. I wonder whether smaller coefficients could lead to better results, especially for methods that otherwise exceed the PPL threshold by quite a bit.

  2. Potential disadvantage for FeatureFlow baseline. FeatureFlow is designed for consecutive layers as it tracks features across them. The authors of FeatureFlow state that tracking those directly across wide ranges is difficult because representations change too much, a finding similar to CircuitSteer. However, Tab. 4 states that larger layer gaps are used instead of consecutive layers. If done so, I think a new cosine similarity threshold sweep is in order, as the 0.5 used by CircuitSteer is the same as FeatureFlow reports in their paper. Otherwise, I am not sure how meaningful the comparison between CircuitSteer and FeatureFlow actually is, as FeatureFlow might terminate its feature chains much earlier than intended.

Overall, I think the proposed method is interesting, but at the moment I am not entirely convinced by the comparison to the existing methods. If the authors can clarify these points, especially regarding the baseline implementations and their evaluation, this could change my assessment.

Questions for authors

Clarifications

  • In Tab. 18, the CAA results for k=1k=1 and k=3k=3 seem quite different from the results in Tab. 1. Especially on Llama, where Δ\Delta is around 0.1 compared to around 0.02 in Tab. 1. I am not sure where this large difference comes from. Are both experiments using the same evaluation setup and data? It would be helpful to clarify which layers and λ\lambda values were used here.
  • I am still a bit confused about the SAE-SSV results in Tab. 1. The authors state that the λ\lambda values are selected on validation data with a normalized PPL threshold of 2.0. However, on Llama the reported PPLs reach values between 14 and 28. Is there really such a large difference between validation and test performance? In general, it would be helpful if the authors could report the exact λ\lambda values selected for all methods in Tab. 1, along with the corresponding validation PPLs.

Suggestions for Future Work

  • I was also wondering how this work relates to Härle et al. (Measuring and Guiding Monosemanticity, NeurIPS 2025). They show that standard SAEs do not necessarily learn monosemantic features and that this can also affect steering. As CircuitSteer already selects features based on their geometric alignment, I wonder if these features are also more monosemantic compared to the ones selected without alignment. I think it would be interesting to look into this, for example using the Feature Monosemanticity Score (FMS) from Härle et al. Could this also explain some of the improvements from geometric alignment?
11 comments
AuthorsOpenReview ↗

Response to reviewer w7Z4 (1/2)

We thank the reviewer for such a careful review. The reviewer's questions about the baselines led us to new experiments that made the comparison fairer, and we are grateful for that.

1. SAE-SSV on Llama. We thank the reviewer for catching this; it was a real inconsistency on our side. No λ in our grid kept SAE-SSV under PPL 2, even on validation (ratios 16–72), so our selection fell back to the gentlest λ. We never stated this rule; the revision now states it and marks those cells. To find the cause, we applied SAE-SSV's intervention with a zero steering vector, so that only the SAE round trip remains (the activation is replaced by its reconstruction). On held-out prompts, this alone gives PPL ratios of 0.75–1.5 on Gemma-2-2B, 3.7–5.6 on base Llama-3.1-8B, and 16–32 on Llama-3.1-8B-Instruct. The fluency loss therefore comes from replacing the activation, not from λ, and it is much larger on the Instruct model even though the SAE's reconstruction error at this layer is similar on base and Instruct. The SAE-SSV paper evaluates base models and measures quality by lexical diversity rather than perplexity, so this cost may not have been visible there. CircuitSteer only adds decoder directions and never replaces the activation. While checking this, we also found that our SAE-SSV vector was trained toward the target behavior and applied with a negative coefficient, whereas the original trains it toward the desired behavior and applies a positive scale. We have re-run it on Gemma in the original orientation (scales 1–10, as in that paper). Coverage is unchanged, with reductions of 0.156, 0.131 and 0.073 on Emotion, Jigsaw and RTP (previously 0.162, 0.107 and 0.123). On Llama the orientation cannot change the outcome, since the round trip alone already exceeds the fluency window.

2. Coefficient grids. We agree that the different ranges need a justification. λ multiplies vectors of very different norms: CircuitSteer's vector is a sum of SAE decoder directions, while CAA's is a raw difference of mean activations. The same λ therefore pushes each method by a different amount, so equal grids would not mean equal tuning. The principle we should have stated is that each method's grid must bracket the fluency bound: it must contain a λ that stays fluent (PPL ≤ 2) and a stronger one that does not. Checking this, two kinds of baseline cells fell short. (i) The method was still fluent at the grid edge λ = −4, so a stronger λ might do better; we extended these grids to −16 (ITI, SRPS, CAA-1L). (ii) CAA (multi-layer) already broke fluency at −1; as the reviewer suggested, we added smaller values down to −0.25. λ is then selected exactly as in Table 1, and eight cells change their selected λ:

MethodCellλΔ / PPL (Table 1)Δ / PPL (extended)
ITILlama Jigsaw−4 → −120.068 / 1.060.035 / 1.80
SRPSGemma Emotion−4 → −60.037 / 0.460.051 / 1.13
SRPSGemma RTP−4 → −80.025 / 0.550.099 / 0.90
SRPSLlama Jigsaw−4 → −60.062 / 0.880.032 / 1.28
SRPSLlama RTP−4 → −120.036 / 1.000.012 / 2.84
CAA-1LGemma Sycophancy−4 → −160.039 / 1.210.122 / 1.64
CAA (multi-layer)Llama Emotion−1 → −0.50.072 / 2.240.008 / 2.18
CAA (multi-layer)Llama Sycophancy−1 → −0.750.025 / 3.630.017 / 1.99

Baseline cells whose selected λ changes when the grid is extended. λ is chosen on validation with the Table 1 rule (largest Δ with PPL ≤ 2.0). Δ (behavior reduction) and PPL (median ratio to unsteered) are test values over 3 seeds, so test PPL can exceed 2.0. All other cells keep their Table 1 values.

Coverage (significant reduction while fluent, out of 8) before and after extending the baseline grids:

Coverage (/8)CircuitSteerITICAA-1LCAA (multi-layer)SRPS
Table 1 grids86552
Extended grids86663

The extended grids raise both CAA variants from 5 to 6 and SRPS from 2 to 3, but no baseline exceeds 6, and CircuitSteer remains the only method at 8/8. A new appendix table lists every method's exact grid, selected λ and validation PPL, as the reviewer requested.

3. FeatureFlow. The reviewer was right. With its 0.5 cutoff over our widely spaced layers, FeatureFlow's chains often broke early, and four of eight cells steered a single layer. We now report FeatureFlow traced through all consecutive layers, as in the original paper, and with lower cutoffs, keeping the same seed feature. For comparability, every variant steers at the same layers as the other methods, whereas the original steers at every chain layer.

FeatureFlowOriginal setupConsecutive0.30.20.1
Coverage (/8)34457

Its strongest variant, with cutoff 0.1 (the same value as our τ_sim), reaches 7/8, against 8/8 for CircuitSteer. CircuitSteer's reduction is also higher in 7 of 8 cells (e.g., Gemma RTP 0.128 vs 0.055).

AuthorsOpenReview ↗

Response to reviewer w7Z4 (2/2)

4. Table 18 vs Table 1. Apologies for the confusion. Table 18 is a separate study, and the two are not directly comparable. It uses a fixed λ = −3 with no selection, places CAA at CircuitSteer's layers rather than at the layer CAA-1L selects, uses a different prompt sample, and reports a mean rather than median PPL ratio; the mean ratio is what produces the very low Llama ratios. The revised caption states this protocol, and the text no longer compares the two tables directly.

5. Monosemanticity. We thank the reviewer for the pointer to Härle et al. The question is whether geometric alignment selects features that are more specific to the target concept, which could explain part of its benefit. FMS measures how exclusively a feature encodes a single concept. We have not computed FMS itself, but we ran a closely related test on held-out prompts (RTP, Jigsaw and Emotion; five seeds) with three measures:

  • Single-feature AUROC: how well each selected feature's activation, on its own, separates target prompts (e.g., toxic) from contrast prompts (non-toxic); 0.5 is chance.
  • Steering-vector AUROC: the same separation for the summed steering vector, using the activations' projection onto it.
  • Share on target: the fraction of the selected features' total activation that falls on target prompts.

We compare the features chosen by the full CircuitSteer rule (co-activating, aligned and specific edges) with three ablations of it, as in Table 3: the same rule without the alignment requirement, with anti-aligned edges, and random edges from the candidate pool.

Full ruleNo alignmentNegative alignmentRandom edges
Gemma, single-feature AUROC0.5830.5430.5310.535
Gemma, steering-vector AUROC0.7070.6070.6250.662
Gemma, share on target0.7350.6210.6060.668
Llama, single-feature AUROC0.5400.5270.5230.522
Llama, steering-vector AUROC0.6490.6150.6220.629
Llama, share on target0.6840.6350.6510.659

The full rule scores highest on all three measures for both models. The gap is clear on Gemma and small on Llama, and single features remain only weakly specific (AUROC 0.54–0.58). So alignment does select more concept-specific features, which supports the reviewer's suggestion that this explains part of its benefit. It also agrees with Fig. 5, where alignment raises the share of toxicity-related features on Gemma/RTP from 23% to 35%. The revision adds this analysis as an appendix and discusses FMS itself as future work.

Changes to the paper

  • Point 1 (SAE-SSV). Sec. 4.1 and Table 4 state the selection rule, including the fallback to the gentlest λ when no coefficient is fluent on validation; Table 1 marks those cells (†). SAE-SSV's three Gemma cells in Table 1 are replaced by the original-orientation rerun, and Tables 8 and 11 and Fig. 1 are updated to match; Table 4 describes the original orientation and scales. A new appendix reports the SAE round trip (reconstruction error, cosine and PPL ratio on Gemma, base Llama and Llama-Instruct), and Sec. 4.2 points to it.
  • Point 2 (grids). Sec. 4.1 states the grid rule. A new appendix reports every baseline under its extended grid (old and new λ, Δ and PPL for each changed cell, and coverage), with a table of every method's selected λ and validation PPL. Table 4 corrects two grid entries, and the λ paragraph of App. A.2 is updated. All other Table 1 numbers are unchanged.
  • Point 3 (FeatureFlow). Sec. 4.1 says how FeatureFlow is traced in Table 1. A new appendix reports chain lengths and coverage for consecutive-layer tracing and cutoffs 0.3, 0.2 and 0.1, and states where each variant steers.
  • Point 4 (Table 18). The caption states its protocol (fixed λ = −3, no selection, CAA at CircuitSteer's layers, mean PPL ratio), and the text no longer compares it with Table 1.
  • Point 5 (monosemanticity). A new appendix reports the concept-specificity analysis above; we cite Härle et al. and discuss FMS as future work.

The reviewer noted that clarifying the baseline implementations and their evaluation could change the assessment. In summary: the selection rule is now stated in full, SAE-SSV runs in its original orientation, FeatureFlow is also traced as in the original paper, and every baseline grid reaches the edge of its fluency window. Under all of this, CircuitSteer remains the only method that covers all 8 configurations.

We will upload the revised manuscript toward the end of the rebuttal period. We believe these changes address the reviewer's concerns, and we would be glad to answer any further questions during the discussion. If the concerns are addressed, we would be grateful if the reviewer would consider updating the score.

Thank you for the rebuttal and additional experiments.

As I understand the rebuttal, using consecutive-layer tracing in FeatureFlow improves coverage from 3/8 to 4/8, while additionally reducing the similarity threshold to 0.1 increases it to 7/8, compared to CircuitSteer's 8/8. Could the authors clarify how the similarity threshold was selected and which intervention layers and coefficients were used? I understand the motivation for matching the intervention layers, but FeatureFlow also evaluates cumulative interventions across its feature chain. It would be interesting to see how the original intervention strategy compares.

Given the substantial changes to the baseline evaluations, including the corrected SAE-SSV implementation, extended λ\lambda ranges, and revised FeatureFlow tracing and similarity thresholds, I would like to see the complete updated results for all affected methods. This should include the selected λ\lambda values, validation and test PPL values, behavioral reductions, and statistical comparisons with CircuitSteer.

Reviewer_fwqSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CircuitSteer, a framework that uses sparse autoencoders (SAEs) to control LLM behavior through interventions at multiple layers. The method identifies related SAE features across layers based on their activation patterns and geometric similarity. It then combines these features into steering vectors to guide model outputs during inference.

Experiments on toxicity, emotion intensity, and sycophancy using Gemma-2-2B and Llama-3.1-8B-Instruct show that CircuitSteer provides more consistent behavioral control than existing methods while maintaining fluency. The authors also conduct ablation studies to examine the effects of feature selection, geometric alignment, and intervention strength.

Critical strengths
  1. The paper explores how SAE features across different layers can be combined to control model behavior. This offers a useful alternative to traditional single-layer steering methods.

  2. CircuitSteer combines feature co-activation, geometric similarity, and contrastive analysis to construct steering vectors. The method uses existing SAEs and does not require additional model training.

  3. The paper compares CircuitSteer with several baselines across different models and behaviors. The ablation studies also help explain the effects of geometric alignment, feature selection, and intervention strength.

Critical weaknesses
  1. The method builds cross-layer circuits using feature co-activation and geometric similarity, but these do not necessarily indicate causal relationships. More direct experiments are needed to support the claim that the discovered circuits explain model behavior.

  2. CircuitSteer and the baselines use different steering coefficient ranges, which may affect the results. Comparisons under the same tuning settings and fluency constraints would help establish the benefits of the proposed method.

  3. The method does not consistently outperform the strongest baselines, and its results depend on the fluency threshold. Table 2 also shows drops in GSM8K accuracy, suggesting that maintaining fluency does not guarantee preserved reasoning ability.

Questions for authors
  1. Can the authors provide stronger evidence that the discovered circuits represent causal relationships rather than feature correlations? Would interventions on individual features or connections help validate this?

  2. How do different steering coefficient ranges affect the results? Does CircuitSteer still outperform baselines under the same tuning settings and fluency constraints?

  3. How does steering strength affect reasoning accuracy on GSM8K? Can the method maintain effective behavioral control without significantly reducing reasoning performance?

8 comments
AuthorsOpenReview ↗

Response to reviewer fwqS

We thank the reviewer for three precise questions on causality, equal tuning and the reasoning cost of steering. Each now has a direct experiment.

1. Are the circuits causal? We agree that co-activation and decoder alignment are proxies (Sec. 3.1), and we have replaced "mediate" with "steer" where the text overclaimed. Our evidence is interventional: at matched size and λ, replacing the selected edges with unaligned, anti-aligned, random or non-SAE ones lowers the reduction (Table 3), and rotating only the angle between layers turns it negative (App. D.1). As the reviewer suggested, we also intervened on the circuit directly. In the unsteered model, we zero-ablate the circuit's SAE features at every circuit layer (keeping the SAE error term), against a control of the same number of random features from the circuit's own candidate pool (three draws). If the circuit carries the behavior, removing it should reduce the behavior more than removing random ones:

Ablated featuresEmotionJigsawRTPSycophancy
Circuit0.0440.1250.0450.047
Random (same number)0.0400.061−0.0100.018

Behavior reduction Δ on held-out prompts after zero-ablation in the unsteered Gemma-2-2B (mean over 3 seeds).

On Gemma-2-2B, ablating the circuit reduces the behavior more than the control in 11 of 12 cell-seeds (sign test p = 0.006), without hurting fluency. This is direct interventional evidence that the selected features contribute to the target behavior on Gemma, although it does not establish the causality of individual cross-layer edges. On Llama-3.1-8B-Instruct, whose circuits come from a much smaller candidate pool (Sec. 4.3), the effect is weaker: clear on Jigsaw (3 of 3 seeds), absent on RTP and Sycophancy.

2. Equal tuning and the fluency threshold. As in our reply to Reviewer w7Z4 (point 2), λ scales vectors of different norms, so we also report every baseline with its grid extended to the edge of its fluency window, under one selection rule. The strongest baseline then covers 6/8 configurations (significant reduction while fluent). With the additional corrected FeatureFlow tracing and cutoff sweep (Reviewer w7Z4, point 3), FeatureFlow reaches 7/8, while CircuitSteer remains at 8/8. On the threshold, Table 8 recounts coverage under other fluency windows. CircuitSteer covers the most configurations under every window with an upper bound of 2.0 or below: 8 of 8 at ≤ 2.0 and at [0.5, 2.0], and 6 at the stricter [0.5, 1.5], against 5 for ITI. Only at a bound of 3.0, where just 9% of steered texts are judged as coherent as unsteered ones (Table 7), does ITI also reach 8. The ordering holds under the extended grids. The reviewer is right that CircuitSteer does not consistently exceed the strongest baseline; our claim is reliability across configurations rather than a higher peak. Head-to-head (Table 11), CircuitSteer is significantly stronger in 20 of 68 comparisons, weaker in 2 and tied in the rest, and it matches or exceeds the strongest fluent baseline in 6 of 8 configurations, also under the extended grids.

3. GSM8K and steering strength. We agree that fluency does not guarantee reasoning, so we extended Table 2 into a strength sweep with the same protocol (500 problems, 8-shot, greedy), which reproduces Table 2 to within about one point. On Gemma-2-2B, RTP (unsteered 27.0%):

MethodλToxicity ΔGSM8K (change)
CircuitSteer−20.09825.2 (−1.8, n.s.)
ITI−20.11615.8 (−11.2)
CAA, multi-layer−10.09416.9 (−10.1)
CAA, single-layer−20.0817.4 (−19.6)

At roughly matched toxicity reduction, CircuitSteer incurs a substantially smaller GSM8K accuracy drop than ITI and both CAA variants; its 1.8-point reduction is not statistically significant (McNemar p = 0.36), whereas the baseline reductions are. CircuitSteer's drop also stays non-significant for the emotion circuit up to λ = −8 (−3.8 points) and on Llama up to λ = −2 (Emotion) and −1 (RTP). The cost grows with strength for every method, and reasoning degrades at the strongest λ (e.g., Gemma RTP at λ = −12).

Changes to the paper

  • Point 1. "Mediate" becomes "steer" where it overclaims; new circuit-ablation appendix (both models), with the weaker Llama result in the main text.
  • Point 2. State the selection and grid rules (Sec. 4.1); the extended-grid appendix shared with Reviewer w7Z4 gives each method's selected λ and validation PPL; Sec. 4.2 frames the claim as reliability across configurations.
  • Point 3. New appendix table with GSM8K accuracy and McNemar tests at every λ (CircuitSteer on both models; ITI and both CAA variants on Gemma RTP); Table 2 is unchanged.

We will upload the revised manuscript toward the end of the rebuttal period. We believe these changes address the reviewer's concerns, and we would be glad to answer any further questions during the discussion. If the concerns are addressed, we would be grateful if the reviewer would consider updating the score.

Thank you for the rebuttal and the additional experiments. I appreciate the authors' efforts. The new results help clarify some of my concerns, but I still have a few questions.

First, the ablation results show that geometric alignment is useful, but I am not sure they prove that the discovered connections are causal. I think testing a few individual features or edges would make this claim more convincing.

Second, I am still concerned about the baseline comparison. CircuitSteer uses a wider coefficient range, and Table 17 shows that some baselines perform better with larger coefficients. Could the authors explain why the same tuning range was not used for all methods?

Finally, Table 2 shows a clear drop in GSM8K accuracy. I would like to see whether this drop can be reduced by using smaller steering strengths while still keeping a reasonable steering effect.

These are my main remaining concerns. I appreciate the additional experiments, but I am not fully convinced that these issues have been resolved.

AuthorsOpenReview ↗

Response to reviewer fwqS (1/4)

We thank the reviewer for the follow-up. In short:

  • Causality. As suggested, we ablated single circuit features: every source feature of every circuit, one at a time. Removing a source feature lowers the downstream features the circuit selected for it in all 80 runs, while unaligned features change little. The selected edges therefore reflect a dependence under intervention, not only co-occurrence. This does not establish path-specific mediation, which the revised paper does not claim.
  • Coefficient ranges. ITI, SRPS and single-layer CAA are now searched from −1 to −16, a range that contains every λ CircuitSteer selects, and multi-layer CAA down to −0.25. At the paper's bound of 2.0, CircuitSteer remains the only method that covers all eight configurations. This advantage is specific to that bound, and we correct one statement of our first reply.
  • GSM8K. A weaker λ shrinks each significant drop of Table 2 to 0.6–1.8 points, none significant, while keeping at least 94% of the Table 2 reduction. At comparable reduction on Gemma RTP, ITI and multi-layer CAA lose 10 to 11 points, against 1.8 for CircuitSteer.

1. Causality: single-feature ablation

Design. For each source feature of the circuit, we zero-ablate that feature alone at its SAE layer. We then measure the relative change of every active feature at the next SAE layer on held-out target prompts, grouped by their decoder cosine with the source: targets of a selected edge, unaligned features and anti-aligned features. This covers both models and all four datasets, in 80 runs, with at least 3 seeds per configuration. If a selected edge reflects a real dependence, ablating its source should lower its target:

Feature at the next layerGemmaLlama
Target of a selected edge−19% to −40%−18% to −28%
Unaligned with the source−4% to −7%−2% to −5%
Anti-aligned with the source+8% to +20%+6% to +25%

Cells give the range over the four datasets of the mean relative change in activation. In every one of the 80 runs, edge targets drop by more than 10% and by more than unaligned features, while anti-aligned features rise. The effect is specific to the selected edges and directional, not a general loss of activity.

What this shows. Removing a source feature changes the features the circuit selected for it. Because every selected edge joins decoder directions with cosine above τ_sim, part of this change is expected from the residual stream's direct path: the removed component reaches the next SAE layer and is read by the target's encoder, which is also why anti-aligned features rise. The test does not separate this path from computation in the intervening layers, so we do not claim path-specific mediation. For steering, the total effect is what matters, since an intervention at one layer reaches the next through both paths. This is the claim the revised paper will make: following our first reply ("mediate" becomes "steer"), Sec. 3.1 will define edges by co-activation, alignment and specificity, and state the dependence this test shows.

The test that would separate the two paths is a matched control: next-layer features with the same decoder cosine to the source and similar baseline activation, but no selected edge. It approximately equalizes the direct-path contribution, so it measures how much of the drop goes beyond decoder geometry. We will add it in the final version.

Circuit-level tests from our first reply point the same way, at the level of behavior:

  • Removing all circuit features lowers the behavior more than removing as many random features from the circuit's own candidate pool: on Gemma in 11 of 12 runs (sign test p = 0.006). On Llama, whose candidate pool is much smaller, the effect is weaker: clear on Jigsaw (3 of 3 seeds) and absent on RTP and Sycophancy.
  • Removing the alignment test, reversing it, or choosing random edges lowers the reduction in all 18 point estimates of Table 3, from 0.085 on average to 0.038, 0.035 and 0.058.
  • Turning later layers' steering vectors against the first, with features and vector lengths fixed, makes the reduction negative (RTP: 0.082 to −0.030 on Gemma, 0.042 to −0.132 on Llama, App. D.1).
Reviewer_nP7JOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose to use SAE-driven circuit discovery to select a per-layer feature direction for steering. They provide compelling experiments which show that the method outperforms on a number of tasks in two realistic LLMs.

Critical strengths
  • The algorithm is intuitive and well-explained. The authors take per-layer SAEs, and then construct circuits where the nodes are SAE features, retaining the (forward-direction-only) edges in the graph which maximize a notion of contrast for a given behaviour and its negative. The per-layer feature direction is then constructed by weighting the source nodes for each trait in that layer.
  • The empirical results are strong and very fairly presented. The authors look at the two axes I would intuitively care about most in this area, fluency and behaviour reduction (both measured in the standard way). The authors do several important ablations as well.
Critical weaknesses
  1. I think the explanation of "Aligned Co-activation Circuits" is useful background, but it would be good to add a sentence in that section that you do not use this definition directly to construct your graph -- rather, your contrast specificity score (which makes much more sense!) is used to construct the graph (which I guess could be viewed as a subtraction of the co-activation circuits under the behaviour and non-behaviour conditions, if I've grokked the algebra correctly?).

  2. It is somewhat surprising that λ\lambda serves as a "universal" coefficient across layers, given e.g. norm growth across layers of many LLM architectures (not sure if that's the case for those considered in this work) and the fact that per-feature activation strength does not seem to have been used to weight the direction vectors. Although this simplicity is appealing, it would be interesting to at least consider whether e.g. mean activation-weighted direction vectors or per-layer λ\lambda improve performance, in my opinion.

Questions for authors
  1. Why do you think a single λ\lambda works well in this setting? Is it because the SAEs use preserve the per-layer norm?
  2. If not, is it worth tuning λ\lambda per-layer? Is this feasible with the sample sizes already used in the paper?
  3. Do average feature (node) magnitudes vary significantly? If so, is it plausible/feasible that a method which weights SAE directions somehow could improve performance?
3 comments
AuthorsOpenReview ↗

Response to reviewer nP7J

We thank the reviewer for the positive review, and especially for the per-layer idea. We tested both suggestions, and we add per-layer scaling to the paper as a variant.

1. Definition of the circuit. The reviewer's reading is right. We add one sentence to Section 3: the steering circuit is built from the contrastive specificity score among aligned edges, the difference between an edge's co-activation frequency on target and on contrast prompts.

2. Why does one λ work across layers? It depends on the SAE family. Llama Scope decoder norms grow with depth at the same rate as the residual stream (median decoder norm 0.09 at layer 3 and 0.77 at layer 24, mean residual norm 3.3 and 29), so one λ already gives a similar relative push at every layer. Gemma Scope decoders have unit norm while the residual norm grows from about 100 at layer 6 to about 290 at layer 18, so one λ pushes layer 6 about three times harder, relative to the residual stream, than layer 18.

3. Per-layer λ. Yes, and it needs no extra tuning. A separate λ for each layer would multiply the search, so we followed the reviewer's norm argument instead: per-layer scaling sets each layer's vector norm in proportion to that layer's mean residual norm, keeps the total push the same, and adds no hyperparameter. As the norms in point 2 predict, it helps mainly on Gemma, where one shared λ over-steers the early layers. We add it to the paper as a variant, credited to this review, alongside the submitted method.

4. Feature magnitudes and activation weighting. Yes, they vary. Within one circuit, the mean activation of the selected features differs between layers by a median of 2.4 times on Gemma and 3.3 times on Llama, and up to 7 times. Weighting each feature by its mean activation on target prompts covers all four Gemma configurations but does not carry over to Llama, so per-layer scaling is the more general fix.

How we will use it. Per-layer scaling uses no test data and no tuning, since the scale comes from residual norms on the prompts already used to fit the circuit. The revision reports it in a new appendix section; all other CircuitSteer numbers in the paper remain the method as submitted.

Changes to the paper

  • Point 1. Sec. 3 gains one sentence: the graph is built from the contrastive specificity score among aligned edges, not from the co-activation definition directly.
  • Point 2. A new appendix section reports per-layer decoder and residual norms for both SAE families and explains why one λ transfers across layers.
  • Points 3 and 4. The same appendix section defines per-layer scaling and activation weighting, credited to this review, and reports both. All main-text CircuitSteer numbers remain the method as submitted.

We thank the reviewer again for the suggestions, which led to a simple and useful variant of the method. We will upload the revised manuscript toward the end of the rebuttal period, and we would be glad to answer any further questions during the discussion.

I thank the authors for the detailed reply. I would urge them to consider clarifying in the main text that λ\lambda has different per-layer behaviour depending on whether the underlying SAEs have unit-normalized decoder directions, as this seems like a substantial source of variation in the steering method's behaviour across models. It may even be considered a strength of the approach given the apparent robustness to the parameter.

I maintain my recommendation for acceptance.

Thank you for reviewing our responses and for maintaining your recommendation for acceptance. We have incorporated your suggestion and clarified in the main text how the shared λ behaves across layers depending on whether the SAEs have unit-normalized decoder directions. Your thoughtful comments substantially improved the clarity of the paper, and we are grateful for your support throughout the discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary: Responses to All Reviewer Concerns

Dear Area Chair,

As the discussion period closes, we summarize how we addressed the questions raised in the three reviews. The reviewers identified the multi-layer use of SAE features, the training-free construction of steering vectors, and the breadth of the comparisons as strengths. Reviewer nP7J maintains a rating of clear acceptance, and Reviewer w7Z4 acknowledged that CircuitSteer remains the only method achieving 8/8 coverage. Reviewer fwqS raised three follow-up questions on 16 Nov, on causality, tuning ranges and GSM8K; each is answered with a new experiment below, posted after the reviewer's last comment. All results are in the revised PDF, with changes highlighted in blue.

  1. Causality. In 80 single-feature ablation runs, removing a source feature lowers its selected downstream features by 18% to 40% on average, against 2% to 7% for unaligned features (Sec. 3.1). Removing, reversing or randomizing the alignment test lowers the steering effect in all 18 cases of Table 3.
  2. Equal tuning. ITI, SRPS and single-layer CAA are now searched up to −16, a range covering every λ CircuitSteer selects, and all methods use the same validation-based selection rule. CircuitSteer is the only method that significantly reduces the target behavior with PPL ≤ 2.0 in all 8 configurations; no baseline exceeds 6 of 8. Head to head, it is statistically tied with or stronger than the strongest in-window baseline in 6 of 8 configurations (Table 1, Sec. 4.2).
  3. Reasoning cost. A weaker λ reduces every significant GSM8K drop to 0.6 to 1.8 points, none statistically significant, while retaining at least 94% of the behavioral reduction. At comparable reduction on Gemma RTP, CircuitSteer loses 1.8 points, against 10.1 for multi-layer CAA and 11.2 for ITI (Sec. 4.3).
  4. Baseline details and per-layer λ. The appendix reports every grid, the selected λ with validation and test PPL, SAE-SSV in both orientations, all FeatureFlow tracings, and head-to-head confidence intervals. Following Reviewer nP7J's suggestion, Sec. 3.2 explains how the shared λ behaves across layers depending on whether the SAEs have unit-normalized decoder directions.

Each concern raised in the three reviews is answered above. We thank the reviewers for questions that made the paper stronger, and the Area Chair for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.