acceptodds
Under review as a conference paper at ICLR 2027

Distributed Sparse Interventions in Language Models

Abstract

Language models can perform a wide range of tasks at various levels of abstraction. They flexibly infer tasks from context, execute multiple tasks simultaneously, and select among competing goals. Understanding and controlling how models represent these tasks is central to aligning them with human objectives. Researchers can change model behaviour in many ways, but model steering, which manipulates intermediate task representations, links representation to behaviour most directly. Current steering methods typically intervene densely and assume linear effects, e.g., shifting every activation along a global direction in the residual stream. Because such interventions change thousands of neurons at once, they leave researchers to separate relevant from irrelevant neurons. We introduce Distributed Sparse Interventions (DSI), which steers models through a sparse set of individual neurons distributed across layers and attention heads. Finding such a set requires accounting for the selected neurons' interactions, even across layers; DSI achieves that by iteratively refining which neurons to intervene on and how strongly. Across 12 tasks and three instruction-tuned models, DSI activates task behaviour by identifying and intervening on fewer than 0.1% of neurons, outperforming dense steering vectors and often matching 10-shot in-context learning. Because DSI identifies explicit sets of neurons, it connects neuron-level computation to changes in latent representations and, ultimately, to desired outputs, allowing us to study how distributed computational components interact to produce steered behaviour. Our results demonstrate that steering can be viewed not only as a linear shift in representation space, but also as a targeted intervention on model computation, realised by adjusting a small set of interacting neurons whose coordinated activity spans multiple layers and contributes to behaviour.

Primary area:
foundation or frontier models, including LLMs
Keywords:
steering, LLMs, sparsity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fmUQOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce DSI, a method to steer a model by adding offsets to a sparse set of individual coordinates in attention-head outputs, spread across layers. It picks starting coordinates using an LRP-modified gradient multiplied by the mean activation difference between positive and negative examples. Then, the authors refine which coordinates to use and how strongly using a sparse optimizer on cross-entropy loss. On 12 function-vector tasks, 4-64 coordinates often match 10-shot in-context learning.

Critical strengths

The paper has very solid empirical results. The authors report Wilson confidence intervals on the test sets and report ablations of iterative refinement. They also consistently do held-out evaluation and have their code released. The AxBench extension and the cross-concept neuron-sharing analysis are good attempts to go beyond the naive next-token prediction task. Although the implementation and analysis practices are all reasonable and sound, several issues need careful consideration before publication (see below).

Critical weaknesses
  1. So my first concern is on novelty. Stripped down, DSI seems a mechanical combination of (a) a mean-difference initialization, as in CAA and function vectors, (b) an attribution score from relevance patching i.e., attribution patching with LRP gradients (Jafari et al., 2025, which you cited), and (c) an optimization of additive offsets with a cap on how many coordinates are non-zero. The highlighted "non-linear interactions" part just means it optimizes through the network rather than replying on a first-order estimate, which also makes it way more computationally costly than standard approach.

  2. Based on my prior comment, I think the implemented baselines in the paper cannot convince me. There are many related prior work, such as LoFiT (Yin et al., 2024; cited but not compared against), Feng et al., 2026 (cited but not compared against as well). I believe you should at least add the following comparison baselines: (a) a dense steering vector optimized with the same loss; (b) LoFiT and ReFT-style learned interventions; (c) randomly chosen neurons with optimized scales.

  3. I also don't think it is appropriate to call "neuron steering" because the basis isn't privileged. Any rotation of a head's value space can be absorbed into W_V and W_O without changing the model, so individual neurons in this space are an arbitrary basis. Each selected coordinate also writes a fixed. dense vector (a row of W_O) into the residual stream. So your own critique of SAE steering on its dense nature applies here too. Actually, Arora et al., 2026 work you cited do have privileged basis. Why not use those?

Questions for authors
  1. I encourage the authors to add the suggested baseline comparisons and to more clearly distinguish their approach from existing methods.

  2. Please justify the term “neuron-based intervention.” Is it necessary and accurate? If the neuron basis is not privileged, I recommend rephrasing it.

5 comments
AuthorsOpenReview ↗

Rebuttal - 1

Rebuttal Summary

Thank you for the thorough review of our work, we are glad that you highlight "very solid empirical results", and find that "the implementation and analysis practices are all reasonable and sound". To address your comments, we added 4 additional baselines, showing that DSI provides novel capabilities over existing work. We also clarified the use of the term "neuron" and applied DSI to MLP neurons, demonstrating that it transfers reliably to different model components.

W1.1 Novelty of DSI

Thank you for this comment. It prompted us to more clearly establish DSI's novelty through additional comparisons with stronger baselines (see updated Table 1 and Section 4 in the manuscript), and to clarify the core contributions of DSI in Section 1, especially in comparison to other sparse steering approaches.

Motivated by a theoretical analysis and empirical observation of neuron effect nonlinearities and interactions, DSI allows us to consistently control task execution with reliably high accuracy by intervening on as few as 1-64 neurons in recent language models. To substantiate that this is a novel finding uniquely facilitated by DSI, we made the following comparisons:

  1. LoFiT [Yin26]: We explain that DSI achieves effective interventions that are an order of magnitude sparser than LoFiT (details in the reply to your next comment).
  2. AUSteer [Fen26]: We show that DSI achieves substantively higher performance than AUSteer under the same neuron budget (details in the reply to reviewer YWkC).
  3. We show that combined optimization of neuron strength and set is necessary for effective interventions under a very sparse neuron budget. DSI applies proximal algorithms to allow for this joint optimization, which is, to our knowledge, novel in the context of model steering (details in the reply to reviewer YWkC).

W1.2 Computational costs of DSI

We added a discussion of DSI's computational costs to Appendix J of the manuscript (see our reply to reviewer LMoj for details):

  1. Steering methods can be divided into methods that require numerical opimization (DSI, ReFT, LoFiT) as well as methods that do not (CAA, FV, AUSteer). Methods from the first class are generally computationally more costly as they require a model backward pass for optimization.
  2. DSI has similar computational costs to other optimization-based steering methods: computation time is similar to LoFiT; on our GPU architecture, it takes ~16.4 minutes to estimate interventions per setting.

W2+Q1 Additional baselines and comparisons to prior work

We agree that more clearly distinguishing DSI from existing methods will help readers understand its novelty and contributions. Thank you for your suggestions; we have now added the following baselines:

1) A dense steering vector optimized with the same loss

We optimize a dense steering vector via the same initialization, loss and optimization procedure as DSI, added to all coordinates of the residual stream. As in the Function Vector [Tod24] baseline, the vector is added at one-third of the model depth, following the transformer block. Table R3 contains the average performance across tasks, in comparison to DSI with the same coordinate budget (the residual stream dimension) and a budget of 64 coordinates (~ 1.5%-3% of the residual stream dimension). DSI can identify considerably sparser interventions with similar accuracy. We added Appendix E - "Additional baselines for optimized steering vectors" to the revised manuscript with the exact accuracies for all task / model combinations. Across settings, DSI typically achieves similar performance to the dense steering vector with set sizes of only 4-16.

Table R3: Average steering performance of DSI vs. an optimized steering vector

modeloptimized vector (dmodeld_\text{model})DSI (64)DSI (dmodeld_\text{model})
Gemma 3 (4B)0.9330.9090.938
Llama 3.2 (3B)0.9400.9360.944
Qwen 3 (8B)0.9320.9250.941
AuthorsOpenReview ↗

Rebuttal - 2

2) LoFiT and ReFT

LoFiT [Yin26] identifies sparse interventions by first identifying a set of attention heads and then learning an additive bias per head. This structurally limits the sparsity level LoFiT can achieve: the attention head dimension of our models is at least 128. The authors of LoFiT recommend using about ~3% of all attention heads for parameter-efficient settings [Yin26]. On Qwen 3 (8B), this corresponds to ~4352 neurons, roughly the 4,096-dimensional residual stream and the number of coordinates of a densely optimized steering vector. Since DSI interventions are typically effective starting at around 8 neurons, we can not compare to LoFiT at the sparsity level achieved by DSI. However, we can compare at LoFiT's maximum sparsity levels of 1 and 2 heads, and at the number of neurons matched to the residual stream (this closely corresponds to the recommended ~3%).

Table R4 compares LoFiT and DSI at matched neuron budgets, showing nearly identical performance. However, across models and tasks, DSI consistently identifies sparser interventions with similar performance (typically around 8-16 neurons); as shown in the newly added Appendix F - "Comparison to LoFiT" in the revised manuscript. We also added LoFiT as a baseline to Table 1 of the revised manuscript. These results demonstrate that (1) LoFiT can, by design, not match the sparsity level of DSI, (2) DSI and LoFiT performance is very close at matched neuron sizes, and (3) DSI consistently estimates sparser interventions with similar performance to the sparsest interventions possible under LoFiT.

Furthermore, in Section 5, Figure 5, we already include a comparison against ReFT-r1 [Wu25], a rank-1 version of ReFT, where DSI performs better at a comparable budget of dmodeld_\text{model} units (see Fig. 5A). We therefore chose to, as also suggested by Reviewer YWkC, compare to AUSteer [Fen26], showing that DSI performs substantively better (accuracy difference around ~65% on average), details can be found in our reply to Reviewer YWkC.

Table R4: DSI in comparison to LoFiT.

# NeuronsmodelLoFiTDSI
128 (1 head)Llama 3.2 (3B)0.9280.945
128 (1 head)Qwen 3 (8B)0.8670.932
256 (1 head)Gemma 3 (4B)0.9140.924
256 (2 heads)Llama 3.2 (3B)0.9440.947
256 (2 heads)Qwen 3 (8B)0.9140.938
dmodeld_\text{model} (10 heads)Gemma 3 (4B)0.9380.938
dmodeld_\text{model} (24 heads)Llama 3.2 (3B)0.9400.944
dmodeld_\text{model} (32 heads)Qwen 3 (8B)0.9480.941

3) Randomly chosen neurons with optimized scales

We implemented this baseline by fixing a random neuron set and optimizing the interventions on the fixed neurons with the same optimizer settings as used for DSI. Table R5 summarizes the average accuracy at set sizes 4 and 64: Especially at small set sizes, DSI identifies more effective interventions (about 40-60% gain in accuracy on average), while at larger set sizes, the gap to the random set variant is narrowing. We added this baseline to Table 1 of the paper, as well as Appendix D - "Additional baselines for fixed neuron sets", which shows that across tasks and models, DSI performs better by a wide margin at smaller sets, while the lead over the random-set baseline gets smaller for larger set sizes. This suggests that sparse interventions need carefully selected neuron sets, that DSI successfully identifies these sets.

Table R5: Average 0-shot accuracy of DSI against a random neuron set baseline.

sizemodelrandom neuronsDSI
4Gemma 3 (4B)0.1670.624
4Llama 3.2 (3B)0.1880.628
4Qwen 3 (8B)0.0900.675
64Gemma 3 (4B)0.8570.909
64Llama 3.2 (3B)0.8500.936
64Qwen 3 (8B)0.8630.925
AuthorsOpenReview ↗

Rebuttal - 3

W3+Q2 Usage of the term "neuron-based intervention" and applicability of DSI to MLP neurons

This comment raises two questions: (1) why is DSI not applied to the MLP neurons since those form a privileged basis, and (2) is it appropriate to call intermediate activations "neurons" if they are not transformed by element-wise nonlinearities?

Regarding (1)

DSI is agnostic to the intervened-upon model component. We focused on attention-head outputs for comparability to baselines: Function Vectors [Tod24] are extracted from the attention head output, and the newly added LoFiT [Yin26] baseline operates in the same space, as well as AUSteer-Head [Fen26]. However, we agree that evaluating DSI on MLP neurons can further strengthen our paper. We therefore now include DSI interventions on the MLP neurons (after the gate; before projection to the residual stream dimension) and added the corresponding results to Table 1 and Appendix C "Additional results for DSI on MLP neurons" of the manuscript, as well as a clarification that DSI is agnostic to the intervention site. Table R2 shows the average performance of DSI on the attention head outputs and MLP neurons for two set sizes. Consistent with the privileged basis hypothesis, very sparse interventions are even more effective on MLP neurons. This shows that DSI reliably transfers to MLP neurons as well.

Table R2: Average DSI performance applied to the attention head outputs vs. MLP neurons

sizemodelDSI AH outputsDSI MLP neurons
4Gemma 3 (4B)0.6240.774
4Llama 3.2 (3B)0.6280.786
4Qwen 3 (8B)0.6750.742
64Gemma 3 (4B)0.9090.919
64Llama 3.2 (3B)0.9360.930
64Qwen 3 (8B)0.9250.924

Regarding (2)

In the context of artificial neural networks, there are two common uses of the word "neuron": one generally refers to any intermediate activation within the model's computational graph (how we use the term), and a second one refers only to activations involving an element-wise nonlinearity. To avoid confusion, we now added the following clarifying footnote to section 3: "We use the term "neuron" to refer to any intermediate activation in the model's computational graph (e.g., outputs of attention heads); not only activations with an element-wise nonlinear activation function (e.g., MLP neurons)". By (1) demonstrating that DSI also works on individual MLP neurons, and (2) explicitly clarifying our use of the term "neuron" in the paper, we now clarified and substantiated the use of neuron throughout the manuscript.


Please let us know if any questions or concerns remain, we would be happy to address them during the remaining discussion period.


References

[Tod24] Todd et al., "Function vectors in large language models." ICLR 2024.

[Wu25] Wu et al., "Axbench: Steering llms? Even simple baselines outperform sparse autoencoders." ICML 2025.

[Fen26] Feng et al., "Fine-grained activation steering: Steering less, achieving more." ICLR 2026.

[Yin26] Yin et al., "LoFiT: Localized fine-tuning on llm representations." NeurIPS 2026.

Reviewer_LMojOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies Distributed Sparse Interventions (DSI), which identifies and uses a sparse set of neurons in the value projection of the attention heads to control the behaviour of an LLM. Existing steering methods are not sparse enough and ignore nonlinear effects, and the paper adopts an iterative method: minimises cross-entropy, repeatedly selecting task-relevant neurons and adjusting the intervention strength, to arrive at an optimal sparse intervention for controlling model behaviour. On Qwen3-8B, Gemma3-4B, Llama-3.2-3B and 12 FV tasks, intervening on fewer than 0.1% of all neurons is sufficient to clearly improve downstream task performance. In addition, the authors extend DSI to steering open-ended generation, and find that the neuron sets selected purely by whether they can induce a target concept have already encoded a certain amount of conceptual structure themselves.

Critical strengths
  1. The DSI proposed in this paper uses an EM-like iterative scheme to overcome the limitation of relying on a first-order effect approximation, which ignores nonlinearity and the interactions between neurons. DSI successfully finds the task-critical neurons more accurately and more sparsely.
  2. The neurons found by DSI can be used to steer generation and carry clear semantic meaning, which is helpful for the mechanistic interpretability of the model.
  3. The paper is well written and easy to follow, and the experiments substantiate its conclusions convincingly.
Critical weaknesses
  1. To locate the neurons, DSI needs to construct two sets D+ and D− on a specific dataset (according to the experimental setup, 10-shot prompts and 0-shot prompts respectively). This approach is similar to the one used in Function Vectors [1], and it still relies on specific simple tasks and carefully constructed prompts, so its applicability to more general task settings is likely to be limited.
  2. Step 6 of Algorithm 1 involves backpropagation with the cross-entropy objective. Since the original FV does not involve backpropagation, DSI is likely to require considerably more overhead to locate a sparse set of neurons on larger models.

[1] Todd et al., ICLR 2024. Function vectors in large language models.

Questions for authors
  1. I found a paper titled "Sparse Interventions in Language Models with Differentiable Masking" [2] whose title is very close to the title of this paper, but it does not appear in the reference list. Could you discuss the similarities and differences between your work and that paper?
  2. Following W2, could you report the memory and time overhead of DSI? I would like to see a comparison with ReFT-r1 [3] as used in Sec. 5, and with LoFiT [4] from the reference list, which also applies additive interventions.
  3. This paper only experiments with neurons in the MHA and does not cover neurons in the FFN projection matrices. (When I read the word "neuron", what comes to my mind first is usually a neuron in MLP up/gate/down) Have the authors tried applying DSI to neurons in the FFN to see whether the results agree with the current ones on the MHA? The SAE-based method in the AxBench experiment of Sec. 5 should make a good comparison.

[2] De Cao et al., BlackboxNLP 2022. Sparse Interventions in Language Models with Differentiable Masking.
[3] Wu et al., ICML 2025. Axbench: Steering LLMs? even simple baselines outperform sparse autoencoders.
[4] Yin et al., NeurIPS 2024. Lofit: Localized fine-tuning on LLM representations.

4 comments
AuthorsOpenReview ↗

Rebuttal - 1

Rebuttal Summary

Thank you for providing valuable feedback on our work. We are glad to hear that you highlight that neurons identified by DSI "carry clear semantic meaning, which is helpful for the mechanistic interpretability of the model", and you found our paper "well written and easy to follow" while "the experiments substantiate its conclusions convincingly".

To address your comments and questions, we described the memory and time overhead of DSI in comparison to other steering methods, clarified the data requirements in comparison to other steering methods and discussed DSI's similarities and differences to related work. Additionally, we showed that DSI interventions can be applied to MLP neurons, with similar accuracy in larger set sizes, and even higher performance in very small neuron set sizes.

W1. Need for task-specific datasets

We thank you for raising this limitation. DSI does need positive and negative examples of the target behaviour. We clarify below that this requirement is (1) shared by all supervised steering methods and (2) not tied to ICL and is more generally applicable.

  1. Most major supervised steering methods derive their intervention from examples of the target behaviour: Function Vectors [Tod24] use ICL prompts, CAA [Rim24] uses contrastive multiple-choice pairs, LoFiT [Yin26] and ReFT-r1 [Wu25] use labelled positive/negative data. This general need for training examples is a given across methods.
  2. Our AxBench experiment (Sec. 5) does not use ICL - using prompt templates for LLMs, the AxBench authors successfully generate positive and negative datasets across a wide range of concepts (hundreds of automatically labelled SAE concepts such as "expressions of vulnerability and honesty about personal struggles" or "technical terms and phrases related to scientific measurements and research methodologies"). Here, we demonstrated that DSI can effectively be used in open-ended generation settings across a wide range of steered concepts.

W2. & Q2. Computational overhead of DSI in comparison to other steering methods

Thank you for raising this concern; we are happy to provide details regarding the computational overhead of DSI. After interventions are trained, all methods require negligible overhead, since they only need to store intervention vectors and add them during the forward pass.

Since ReFT-r1 [Wu25], LoFiT [Yin26], and DSI all optimize their interventions numerically, they all require backward passes through the model, leading to similar memory and time requirements. This is, as you point out, computationally more costly than function vectors.

We estimated both LoFiT and DSI interventions on an AMD MI300A GPU. Per condition (one model, one task, one set size for DSI/head count for LoFiT), DSI takes on average ~16.4 min, while LoFiT takes ~14.3 min. The memory requirement is driven by the size of the model itself and the backward pass; in both cases, the optimizer itself adds negligible memory overhead. Since we compute DSI gradients in batches of 50, and LoFiT (as in the paper) in batches of 8, DSI has a higher memory footprint; if memory is of concern, DSI gradients can be computed in smaller batches. As ReFT-r1 on AxBench [Wu25] only trains for 3 epochs and uses small batch sizes of 6 at fixed hyperparameters, it needs less computation time than both DSI and LoFit, while the exact runtime under a new setting depends on the amount of hyperparameter optimization.

In conclusion, DSI needs similar computation time and memory as other optimization-based steering methods, while the exact details depend on the hyperparameters and how they are tuned. Non-iterative methods such as steering vectors need considerably less compute. We added a discussion of DSI's computational costs compared to other methods to Appendix J of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal - 2

Q1. Similarities and differences to prior work

Thank you for this relevant pointer; we added this work to our discussion of related work. De Cao et al. [Cao22] share our goal of identifying small neuron sets whose intervention changes a model’s output. The methods differ in the type of intervention, how sparsity is enforced, the training objective, and the type/size of model:

  1. While DSI learns an additive per-neuron offset, modifying the input-dependent activation, De Cao et al. completely replace neuron activations with learned constants.
  2. Unlike DSI, which uses the ZeroFPR algorithm, allowing us to enforce an exact neuron budget, De Cao et al.'s approach relaxes the binary neuron mask with a stochastic Hard Concrete distribution and the constraint is implemented via a Lagrangian. In their approach, sparsity is controlled only in expectation, and the exact number of intervened-upon neurons varies.
  3. De Cao et al. optimizes the probability ratio between a correct and incorrect (predicted) token, plus a KL regularizer, whereas DSI optimizes the correct token likelihood on negative examples.
  4. De Cao et al. validated their approach on 2-layer LSTMs with 1.3k hidden units and used 2.7k-11k training sentences for their tasks, whereas we show that DSI is applicable to 8B-parameter transformer models, using only 100 examples per task.

In short, while both works seek sparse neuron-level interventions, DSI targets task activation in billion-parameter transformers under a low-data regime, employs additive interventions, allows for exact neuron budgets, and uses attribution-guided proximal optimisation. We now included De Cao et al. in our related work section, together with a brief description of their contribution.

Q3. Applicability of DSI to MLP neurons

DSI is agnostic to the model component, and can be used to estimate interventions in the residual stream, attention heads, and MLP neurons. Since the Function Vectors [Tod24] baseline is constructed from attention head outputs, we chose this component to improve comparability.

We added an evaluation of DSI interventions on MLP neurons (after the gate and before projection to the residual-stream dimension). Table R2 compares performance across attention-head outputs and MLP neurons at two set sizes. Consistent with the privileged basis hypothesis, very sparse interventions are particularly effective on MLP neurons. These results demonstrate that DSI can identify effective sparse interventions on MLP neurons as well. We have added the results to Table 1 and Appendix C ("Additional Results for DSI on MLP Neurons") and clarified that DSI is agnostic to the choice of intervention site.

Table R2: Average DSI performance applied to the attention head outputs vs. MLP neurons

sizemodelDSI AH outputsDSI MLP neurons
4Gemma 3 (4B)0.6240.774
4Llama 3.2 (3B)0.6280.786
4Qwen 3 (8B)0.6750.742
64Gemma 3 (4B)0.9090.919
64Llama 3.2 (3B)0.9360.930
64Qwen 3 (8B)0.9250.924

Please let us know if further questions or comments are still open, we remain available to address them during the discussion period.


References

[Cao22] De Cao et al., "Sparse Interventions in Language Models with Differentiable Masking." BlackboxNLP 2022.

[Tod24] Todd et al., "Function vectors in large language models." ICLR 2024.

[Rim24] Rimsky et al., "Steering llama 2 via contrastive activation addition." ACL 2024.

[Wu25] Wu et al., "Axbench: Steering llms? Even simple baselines outperform sparse autoencoders." ICML 2025.

[Yin26] Yin et al., "LoFit: Localized fine-tuning on llm representations." NeurIPS 2026.

Thank you for the rebuttal, I see the efforts authors have made to improve this work. Most of my concerns have been addressed. Although I still believe that this work is mainly an extension of the FV method (novelty), the authors' additional contributions, and experiments on FFNs provide sufficient value to be published. I decide to maintain my positive score.

Thank you for acknowledging our efforts, we are glad that we could address your comments, and that you maintain your positive evaluation of our work.

Reviewer_YWkCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper proposes Distributed Sparse Interventions (DSI), a steering method for activating task behaviour in LLMs. This method applies to a small set of neurons spread across different layers and heads, instead of adding one dense vector at a layer. Candidate neurons are allocated a score by a first-order estimate of their influence on the correct-answer logit (the activation difference x an LRP-modified gradient). The top-n neurons are used as initializations, and both the selected set and the per-neuron scales are then refined with ZeroFPR under a fixed neuron budget. On 12 tasks from Todd et al. (2024) and three instruction-tuned LLMs (Qwen3-8B, Gemma-3-4B, Llama-3.2-3B), intervening on 64 neurons gives 0.86–0.98 avg zero-shot accuracy. While function vectors give 0.33–0.63 at their best layer. At small budgets it is also shown that spreading neurons across layers is more beneficial than restricting them to one layer. On AxBench, DSI with 1024 neurons outperforms ReFT-r1. Clustering concepts by the overlap of their neuron sets is shown to produce semantically coherent groups.

Critical strengths

S1. The method is simple and well motivated. Figure 2 and Appendix A establish that the single-neuron effects are non-monotone and that pairs are actually interacting, which provides justification for per-neuron scales and re-selection instead of having one global scale.

S2. The evaluations are broad across three model families and all 12 tasks reported at every set size with confidence intervals. Figure 9 in the Appendix shows that difference in task performance between first-order and iterative interventions do not stem from merely different total intervention strengths. Figure 10 shows that the refinement really changes the neuron set rather than just rescaling it.

S3. The ablations in Table 1 are convincing and the effects are substantial and consistent across different models. Additionally, the paper is upfront in mentioning that DSI costs more compute than non-iterative methods.

Critical weaknesses

W1. The paper does not sufficiently establish that jointly optimizing the set of intervened neurons and their strengths provides an advantage over simpler alternatives that identify and individually optimize a small number of intervention coordinates.

W2. Earlier work (Feng et al. (2026)) has already established important benefits of fine-grained neuron-level steering. The paper would benefit from stronger comparisons to these related approaches. The authors should state precisely which components of DSI are not present in that method and compare against it directly. A direct comparison with Feng et al. on the same tasks is needed, especially the size of the margin.

W3. The authors motivate sparse interventions by arguing that dense steering methods modify many potentially irrelevant neurons, making it difficult to identify task-relevant components. While the paper demonstrates that sparse interventions are sufficient for effective steering, it does not test the suggestion that targeting specific neurons may enable more precise and robust steering than global shifts. More specifically, it remains unclear whether sparsity itself leads to reduced interference with unrelated behaviours or improved robustness, beyond enabling effective steering with fewer modified neurons.

W4. It is not clear why the authors restrict candidates to attention-head outputs. Are these locations more effective or more interpretable than MLP neurons or residual-stream neurons? The paper does provide evidence that the locations are structured and behaviourally meaningful. But it does not establish that the selected locations reveal a mechanistic explanation of the task. Once we identify a sparse set of effective intervention neurons - what have we learned about the underlying task computation beyond identifying useful control points?

Minor (on clarity)

C1. The paper seems to use "linear" in two senses, sometimes for the effect of an intervention and sometimes for the intervention itself, which can make it hard for the reader to keep track on the first pass which assumption DSI relaxes. The authors can clarify this.

Questions for authors

Please check the weakness section.

5 comments

Rebuttal Summary

Thank you for your constructive review, we are glad that you found our method well motivated, highlight our broad evaluations and found our effects substantial and consistent in comparison to convincing ablations.

To answer the points and questions you raised, we implemented two additional baselines:

  1. We show the benefit of jointly optimizing the set of intervened-upon neurons and their strengths by comparing against a setting where the neuron set is fixed before optimization, with higher performance especially in sparse settings.
  2. We evaluate AUSteer on the same tasks, showing substantively better performance of DSI.

We further improve clarity of the manuscript by disambiguating the use of "linear" and better distinguishing the contributions of DSI and avenues for future research in the discussion. Additionally, we showed that DSI interventions achieve high accuracy on MLP neurons, providing evidence for the component-agnostic applicability of DSI.

We provide detailed point-by-point responses below.

W1. Comparison to separate intervention set and intervention strength optimization

Thank you for this suggestion. To better isolate the contribution of jointly optimising the intervention set and strengths, we added an ablation where (1) the set of neurons is chosen first by their first-order effect approximation, and (2) the intervention strength is optimised on this fixed set. We agree that this is important for assessing the benefits of joint neuron selection and strength optimisation over a sequential approach.

Table R1 shows the average 0-shot accuracy between joint optimisation (DSI) and separate optimisation (fixed support) for two set sizes (4 and 64). For the smaller set size, joint optimisation yields sizeable benefits (~20% accuracy increase on average), while for the larger set size, the difference is negligible.

Table R1: Average performance of DSI vs the fixed support baseline.

sizemodeltop-nDSI
4Gemma 3 (4B)0.4450.624
4Llama 3.2 (3B)0.4180.628
4Qwen 3 (8B)0.4600.675
64Gemma 3 (4B)0.8980.909
64Llama 3.2 (3B)0.9330.936
64Qwen 3 (8B)0.9120.925

We added the averaged results to Table 1, with full results in Appendix D "Additional baselines for fixed neuron sets", demonstrating that DSI with joint optimisation consistently matches or outperforms separate optimisation across tasks and models, with larger gains in sparser settings.

AuthorsOpenReview ↗

Rebuttal - 2

W2. Comparison to AUSteer: methodological differences and performance

Thank you for the suggestion to include additional fine-grained steering baselines. We added comparisons to AUSteer [Fen26]. (As suggested by other reviewers, we also added comparisons to LoFit [Yin26]: DSI achieves higher sparsity that LoFiT at maintained performance; typically, ~8-16 neurons for DSI achieve similar performance to 128 neurons for LoFiT; details are in our rebuttal to Reviewer fmUQ.)

Methodological differences

DSI and AUSteer are both motivated by the goal of determining sparse interventions on the neuron level, but their methodology differes substantively:

  1. Neuron scoring: AUSteer scores each neuron via "activation momentum", the fraction of contrastive pairs on which its activation difference has the dominant sign, to determine neuron-specific interventions. The score does not measure the effect of the selected neurons on the model output. DSI chooses the initial neuron set by a first-order approximation of the causal effect of an intervention on the neuron.
  • Set selection: AUSteer chooses the top-n scoring neurons. DSI iteratively refines the chosen neuron set. We find that, especially in very sparse settings, refining the neuron set contributes substatively to DSI's performance (see our answer to the first point above).
  • Intervention strength: AUSteer does not estimate additive interventions, but rescales the neurons' activation during the forward pass: activation aia_i is set to a←(1+γi)aia \gets (1+\gamma_i)a_i. These scaling factors are neuron-specific, but they do not take neuron interactions into account.

Performance comparison

We implemented AUSteer following the authors’ public code and evaluated it for all model and task combinations. We estimated both the attention-head variant (AUSteer-Head), which intervenes on the same model component as DSI, and the MLP variant (AUSteer-FFN). The same training samples as for DSI were used to construct the contrastive pairs, with the same training set size of 100 samples (in contrast do the 700-1k examples used in [Fen26]). The number of intervened upon neurons as well as the steering strength α were tuned over 1-100, covering the range used in the original study. Since DSI uses a fixed neuron set size while AUSteer sweeps over different sizes, we can not report accuracies for exactly matched set sizes, since that would be disadvantegous for AUSteer. We therefore report DSI at 8 neurons against the best performing hyperparameter combination of AUSteer on the test set per task and model (i.e., the best performing neuron set size up to 100). Table R6 shows the average performance of DSI in comparison to AUSteer; we find that DSI performs better by a wide margin (around ~65% accuracy difference against AUSteer-head).

Table R6: DSI in comparison to AUSteer.

modelAUSteer-FFN (≤100 neurons)AUSteer-Head (≤100 neurons)DSI (8 neurons)
Gemma 3 (4B)0.1090.1590.808
Llama 3.2 (3B)0.1880.2280.872
Qwen 3 (8B)0.1020.1570.804

W3. Effect of sparse interventions on steering precision and robustness

Thank you for raising this point. Our primary focus in this work is to establish the methodological foundations for sparse steering, and we agree that we do not systematically investigate whether sparse interventions improve steering robustness. Nevertheless, we take a first step towards assessing these aspects through our AxBench [Wu25] experiments in Section 5. AxBench measures the harmonic mean of concept presence, instruction following, and fluency, such that a high steering score requires inducing the target concept while preserving instruction following and response quality. Even though this is not a measure for the broader side effects of steering interventions, it provides a relevant measure of behavioral robustness.

These results led us to hypothesize that sparsity may contribute to more precise and robust steering in the Discussion. We have revised the Discussion to make clear that this is not a claim of the manuscript, but an important direction for future research.

AuthorsOpenReview ↗

Rebuttal - 3

W4.1 Choice of intervention sites

Thank you for pointing out that our paper currently does not explain the selection of intervened-upon model components sufficiently. DSI is agnostic regarding the model component and can be flexibly used to estimate interventions in the residual stream, attention heads and MLP neurons. Since the Function Vectors [Tod24] baseline is constructed from attention head outputs, we chose this component to ensure comparability. The newly added LoFiT [Yin26] baseline also operates on attention head outputs.

We added an evaluation of DSI interventions on MLP neurons (after the gate and before projection to the residual-stream dimension). Table R2 compares performance across attention-head outputs and MLP neurons at two set sizes. Consistent with the privileged basis hypothesis, very sparse interventions are particularly effective on MLP neurons. These results demonstrate that DSI can identify effective sparse interventions on MLP neurons as well. We added the results to Table 1 and Appendix C ("Additional Results for DSI on MLP Neurons") and clarified that DSI is agnostic to the choice of intervention site.

Table R2: Average DSI performance applied to the attention head outputs vs. MLP neurons

sizemodelDSI AH outputsDSI MLP neurons
4Gemma 3 (4B)0.6240.774
4Llama 3.2 (3B)0.6280.786
4Qwen 3 (8B)0.6750.742
64Gemma 3 (4B)0.9090.919
64Llama 3.2 (3B)0.9360.930
64Qwen 3 (8B)0.9250.924

W4.2 Mechanistic insights from sparse intervention neurons

Thank you for this comment. We view the reliable identification of useful control points as an important first step towards mechanistic explanations, and our work provides a methodological framework for reliably identifying and characterising these locations.

Our work focuses on establishing DSI as a robust and effective method for identifying sparse interventions in the neuron basis across models and tasks. To our knowledge, this is a new finding (i.e., LoFiT operates on attention heads while the effectiveness of AUSteer interventions is comparably limited). Section 5 provides evidence that the identified interventions can be useful for interpretability studies: (1) interventions cluster into specific layers, and (2) interventions cluster meaningfully by the concept they activate. This suggests that semantically related tasks share common control points in the neuron basis and provides an entry point for further study. DSI thus offers a useful tool for the community that can be combined with other interpretability methods, such as neuron labelling methods [Bil23], to achieve an even more fine-grained understanding by studying which neuron representations are shared across tasks, and how sub-parts of tasks are distributed across neuron representations. However, we agree that the current manuscript does not contain a detailed interpretability study, and we leave a systematic investigation of these cross-task relationships to future work, as it represents a substantial research direction in its own right.

C1. Use of "linear" in two different senses

Thank you for highlighting this potential for misunderstanding. We disambiguated both senses and removed mentions of "linear" where it referred to the form of the intervention. "Linear" now refers only to the effect of an intervention (approximately linear and additive), while the intervention itself is described as dense and direction-based. The changes are in the Abstract, the Introduction, the Related Works section, and the Discussion. This helped us to clarify the distinction and state more precisely which assumption DSI relaxes.


We appreciate the helpful suggestions, especially the comparisons to further baselines helped us strengthen our work. Please let us know if there are any questions or comments still open.


References

[Bil23] Bills, et al., "Language models can explain neurons in language models", 2023.

[Tod24] Todd, et al. "Function vectors in large language models." ICLR 2024.

[Wu25] Wu, et al. "Axbench: Steering llms? even simple baselines outperform sparse autoencoders." ICML 2025.

[Fen26] Feng, et al. "Fine-grained activation steering: Steering less, achieving more." ICLR 2026.

[Yin26] Yin, et al. "LoFit: Localized fine-tuning on llm representations." NeurIPS 2026.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. We are glad that all reviewers highlight the quality of our experiments: Reviewer YWkC finds that "evaluations are broad" and "the effects are substantial and consistent", Reviewer LMoj that the "paper is well written and easy to follow, and the experiments substantiate its conclusions convincingly", and Reviewer fmUQ that "the paper has very solid empirical results".

We carefully addressed all comments. In summary:

  1. Comparisons to prior work: We added comparisons to LoFiT [Yin26] and AUSteer [Fen26], showing that DSI is far more effective at identifying sparse interventions than prior methods (see below).
  2. Improved clarity: We disambiguated our use of the terms "linear" and "neuron", and clarified our contributions in the introduction and discussion.
  3. Applicability of DSI to MLP neurons: We showed that DSI transfers reliably to MLP neurons, demonstrating applicability across different model components.
  4. Additional baselines: We added random neuron sets, fixed neuron sets, and fully optimized steering vectors, validating the importance of DSI's components.
  5. Computational overhead of DSI: We clarified that DSI's computational cost is similar to other supervised steering methods, with a one-time estimation cost of about 16 minutes per task while inference adds only negligible overhead.

While all Reviewers acknowledged that we effectively addressed most of their comments, Reviewer YWkC increased their score, Reviewer LMoj maintained their positive evaluation, and Reviewer fmUQ remains undecided regarding the novelty of our approach, noting that DSI builds upon a combination of prior work. We clarify that DSI differs methodologically from previous sparse steering approaches and achieves substantially higher performance. We consider it a strength of DSI to rely on state-of-the-art methods (ZeroFPR [The18], RelP [Jaf25], CAA [Rim24]) to achieve that goal, as validated by our experiments: Compared with the suggested alternatives, DSI uses ~8% of LoFiT’s neurons and achieves an average accuracy of ~83% on 8 neurons compared to AUSteer’s ~18% on up to 100 neurons. Additionally, through a series of four ablation studies, we demonstrate the importance of the individual steps that comprise DSI.

In summary, our rebuttal addressed concerns about DSI's applicability to other model components, its capabilities in comparison to prior works and stronger baselines, and its computational overhead. We also clarified DSI's distinctions from prior work and improved the submission's overall clarity. At this point, all major concerns raised by the reviewers are addressed in our updated manuscript, and we remain committed to incorporating any further suggestions.


References

[The18] Themelis et al., "Forward-backward envelope for the sum of two nonconvex functions: Further properties and nonmonotone linesearch algorithms." SIOPT 2018.

[Rim24] Rimsky et al., "Steering Llama 2 via contrastive activation addition." ACL 2024.

[Jaf25] Jafari et al., "Relp: Faithful and efficient circuit discovery in language models via relevance patching.", 2025.

[Fen26] Feng et al., "Fine-grained activation steering: Steering less, achieving more." ICLR 2026.

[Yin26] Yin et al., "LoFiT: Localized fine-tuning on llm representations." NeurIPS 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.