- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce DSI, a method to steer a model by adding offsets to a sparse set of individual coordinates in attention-head outputs, spread across layers. It picks starting coordinates using an LRP-modified gradient multiplied by the mean activation difference between positive and negative examples. Then, the authors refine which coordinates to use and how strongly using a sparse optimizer on cross-entropy loss. On 12 function-vector tasks, 4-64 coordinates often match 10-shot in-context learning.
Critical strengths
The paper has very solid empirical results. The authors report Wilson confidence intervals on the test sets and report ablations of iterative refinement. They also consistently do held-out evaluation and have their code released. The AxBench extension and the cross-concept neuron-sharing analysis are good attempts to go beyond the naive next-token prediction task. Although the implementation and analysis practices are all reasonable and sound, several issues need careful consideration before publication (see below).
Critical weaknesses
-
So my first concern is on novelty. Stripped down, DSI seems a mechanical combination of (a) a mean-difference initialization, as in CAA and function vectors, (b) an attribution score from relevance patching i.e., attribution patching with LRP gradients (Jafari et al., 2025, which you cited), and (c) an optimization of additive offsets with a cap on how many coordinates are non-zero. The highlighted "non-linear interactions" part just means it optimizes through the network rather than replying on a first-order estimate, which also makes it way more computationally costly than standard approach.
-
Based on my prior comment, I think the implemented baselines in the paper cannot convince me. There are many related prior work, such as LoFiT (Yin et al., 2024; cited but not compared against), Feng et al., 2026 (cited but not compared against as well). I believe you should at least add the following comparison baselines: (a) a dense steering vector optimized with the same loss; (b) LoFiT and ReFT-style learned interventions; (c) randomly chosen neurons with optimized scales.
-
I also don't think it is appropriate to call "neuron steering" because the basis isn't privileged. Any rotation of a head's value space can be absorbed into W_V and W_O without changing the model, so individual neurons in this space are an arbitrary basis. Each selected coordinate also writes a fixed. dense vector (a row of W_O) into the residual stream. So your own critique of SAE steering on its dense nature applies here too. Actually, Arora et al., 2026 work you cited do have privileged basis. Why not use those?
Questions for authors
-
I encourage the authors to add the suggested baseline comparisons and to more clearly distinguish their approach from existing methods.
-
Please justify the term “neuron-based intervention.” Is it necessary and accurate? If the neuron basis is not privileged, I recommend rephrasing it.
5 comments
Rebuttal - 2
2) LoFiT and ReFT
LoFiT [Yin26] identifies sparse interventions by first identifying a set of attention heads and then learning an additive bias per head. This structurally limits the sparsity level LoFiT can achieve: the attention head dimension of our models is at least 128. The authors of LoFiT recommend using about ~3% of all attention heads for parameter-efficient settings [Yin26]. On Qwen 3 (8B), this corresponds to ~4352 neurons, roughly the 4,096-dimensional residual stream and the number of coordinates of a densely optimized steering vector. Since DSI interventions are typically effective starting at around 8 neurons, we can not compare to LoFiT at the sparsity level achieved by DSI. However, we can compare at LoFiT's maximum sparsity levels of 1 and 2 heads, and at the number of neurons matched to the residual stream (this closely corresponds to the recommended ~3%).
Table R4 compares LoFiT and DSI at matched neuron budgets, showing nearly identical performance. However, across models and tasks, DSI consistently identifies sparser interventions with similar performance (typically around 8-16 neurons); as shown in the newly added Appendix F - "Comparison to LoFiT" in the revised manuscript. We also added LoFiT as a baseline to Table 1 of the revised manuscript. These results demonstrate that (1) LoFiT can, by design, not match the sparsity level of DSI, (2) DSI and LoFiT performance is very close at matched neuron sizes, and (3) DSI consistently estimates sparser interventions with similar performance to the sparsest interventions possible under LoFiT.
Furthermore, in Section 5, Figure 5, we already include a comparison against ReFT-r1 [Wu25], a rank-1 version of ReFT, where DSI performs better at a comparable budget of units (see Fig. 5A). We therefore chose to, as also suggested by Reviewer YWkC, compare to AUSteer [Fen26], showing that DSI performs substantively better (accuracy difference around ~65% on average), details can be found in our reply to Reviewer YWkC.
Table R4: DSI in comparison to LoFiT.
| # Neurons | model | LoFiT | DSI |
|---|---|---|---|
| 128 (1 head) | Llama 3.2 (3B) | 0.928 | 0.945 |
| 128 (1 head) | Qwen 3 (8B) | 0.867 | 0.932 |
| 256 (1 head) | Gemma 3 (4B) | 0.914 | 0.924 |
| 256 (2 heads) | Llama 3.2 (3B) | 0.944 | 0.947 |
| 256 (2 heads) | Qwen 3 (8B) | 0.914 | 0.938 |
| (10 heads) | Gemma 3 (4B) | 0.938 | 0.938 |
| (24 heads) | Llama 3.2 (3B) | 0.940 | 0.944 |
| (32 heads) | Qwen 3 (8B) | 0.948 | 0.941 |
3) Randomly chosen neurons with optimized scales
We implemented this baseline by fixing a random neuron set and optimizing the interventions on the fixed neurons with the same optimizer settings as used for DSI. Table R5 summarizes the average accuracy at set sizes 4 and 64: Especially at small set sizes, DSI identifies more effective interventions (about 40-60% gain in accuracy on average), while at larger set sizes, the gap to the random set variant is narrowing. We added this baseline to Table 1 of the paper, as well as Appendix D - "Additional baselines for fixed neuron sets", which shows that across tasks and models, DSI performs better by a wide margin at smaller sets, while the lead over the random-set baseline gets smaller for larger set sizes. This suggests that sparse interventions need carefully selected neuron sets, that DSI successfully identifies these sets.
Table R5: Average 0-shot accuracy of DSI against a random neuron set baseline.
| size | model | random neurons | DSI |
|---|---|---|---|
| 4 | Gemma 3 (4B) | 0.167 | 0.624 |
| 4 | Llama 3.2 (3B) | 0.188 | 0.628 |
| 4 | Qwen 3 (8B) | 0.090 | 0.675 |
| 64 | Gemma 3 (4B) | 0.857 | 0.909 |
| 64 | Llama 3.2 (3B) | 0.850 | 0.936 |
| 64 | Qwen 3 (8B) | 0.863 | 0.925 |
Rebuttal - 3
W3+Q2 Usage of the term "neuron-based intervention" and applicability of DSI to MLP neurons
This comment raises two questions: (1) why is DSI not applied to the MLP neurons since those form a privileged basis, and (2) is it appropriate to call intermediate activations "neurons" if they are not transformed by element-wise nonlinearities?
Regarding (1)
DSI is agnostic to the intervened-upon model component. We focused on attention-head outputs for comparability to baselines: Function Vectors [Tod24] are extracted from the attention head output, and the newly added LoFiT [Yin26] baseline operates in the same space, as well as AUSteer-Head [Fen26]. However, we agree that evaluating DSI on MLP neurons can further strengthen our paper. We therefore now include DSI interventions on the MLP neurons (after the gate; before projection to the residual stream dimension) and added the corresponding results to Table 1 and Appendix C "Additional results for DSI on MLP neurons" of the manuscript, as well as a clarification that DSI is agnostic to the intervention site. Table R2 shows the average performance of DSI on the attention head outputs and MLP neurons for two set sizes. Consistent with the privileged basis hypothesis, very sparse interventions are even more effective on MLP neurons. This shows that DSI reliably transfers to MLP neurons as well.
Table R2: Average DSI performance applied to the attention head outputs vs. MLP neurons
| size | model | DSI AH outputs | DSI MLP neurons |
|---|---|---|---|
| 4 | Gemma 3 (4B) | 0.624 | 0.774 |
| 4 | Llama 3.2 (3B) | 0.628 | 0.786 |
| 4 | Qwen 3 (8B) | 0.675 | 0.742 |
| 64 | Gemma 3 (4B) | 0.909 | 0.919 |
| 64 | Llama 3.2 (3B) | 0.936 | 0.930 |
| 64 | Qwen 3 (8B) | 0.925 | 0.924 |
Regarding (2)
In the context of artificial neural networks, there are two common uses of the word "neuron": one generally refers to any intermediate activation within the model's computational graph (how we use the term), and a second one refers only to activations involving an element-wise nonlinearity. To avoid confusion, we now added the following clarifying footnote to section 3: "We use the term "neuron" to refer to any intermediate activation in the model's computational graph (e.g., outputs of attention heads); not only activations with an element-wise nonlinear activation function (e.g., MLP neurons)". By (1) demonstrating that DSI also works on individual MLP neurons, and (2) explicitly clarifying our use of the term "neuron" in the paper, we now clarified and substantiated the use of neuron throughout the manuscript.
Please let us know if any questions or concerns remain, we would be happy to address them during the remaining discussion period.
References
[Tod24] Todd et al., "Function vectors in large language models." ICLR 2024.
[Wu25] Wu et al., "Axbench: Steering llms? Even simple baselines outperform sparse autoencoders." ICML 2025.
[Fen26] Feng et al., "Fine-grained activation steering: Steering less, achieving more." ICLR 2026.
[Yin26] Yin et al., "LoFiT: Localized fine-tuning on llm representations." NeurIPS 2026.
Rebuttal - 1
Rebuttal Summary
Thank you for the thorough review of our work, we are glad that you highlight "very solid empirical results", and find that "the implementation and analysis practices are all reasonable and sound". To address your comments, we added 4 additional baselines, showing that DSI provides novel capabilities over existing work. We also clarified the use of the term "neuron" and applied DSI to MLP neurons, demonstrating that it transfers reliably to different model components.
W1.1 Novelty of DSI
Thank you for this comment. It prompted us to more clearly establish DSI's novelty through additional comparisons with stronger baselines (see updated Table 1 and Section 4 in the manuscript), and to clarify the core contributions of DSI in Section 1, especially in comparison to other sparse steering approaches.
Motivated by a theoretical analysis and empirical observation of neuron effect nonlinearities and interactions, DSI allows us to consistently control task execution with reliably high accuracy by intervening on as few as 1-64 neurons in recent language models. To substantiate that this is a novel finding uniquely facilitated by DSI, we made the following comparisons:
W1.2 Computational costs of DSI
We added a discussion of DSI's computational costs to Appendix J of the manuscript (see our reply to reviewer LMoj for details):
W2+Q1 Additional baselines and comparisons to prior work
We agree that more clearly distinguishing DSI from existing methods will help readers understand its novelty and contributions. Thank you for your suggestions; we have now added the following baselines:
1) A dense steering vector optimized with the same loss
We optimize a dense steering vector via the same initialization, loss and optimization procedure as DSI, added to all coordinates of the residual stream. As in the Function Vector [Tod24] baseline, the vector is added at one-third of the model depth, following the transformer block. Table R3 contains the average performance across tasks, in comparison to DSI with the same coordinate budget (the residual stream dimension) and a budget of 64 coordinates (~ 1.5%-3% of the residual stream dimension). DSI can identify considerably sparser interventions with similar accuracy. We added Appendix E - "Additional baselines for optimized steering vectors" to the revised manuscript with the exact accuracies for all task / model combinations. Across settings, DSI typically achieves similar performance to the dense steering vector with set sizes of only 4-16.
Table R3: Average steering performance of DSI vs. an optimized steering vector