acceptodds
Under review as a conference paper at ICLR 2027

Where Features Are Written and Read: Localizing Native Transformer Channels for Residual Directions

Abstract

Sparse autoencoders, probes, and steering methods find directions in a language model's activations that track concepts, and editing along them can steer the model's behavior. A direction does not say which of the model's own components build the feature or which carry it to behavior, so it is unclear whether an edit along the direction reflects how the model itself computes. We introduce feature-direction localization, which maps a supplied direction to the model's own multilayer perceptron (MLP) channels on both sides of it: writers, the channels that construct the feature, and consumers, the channels through which it reaches behavior. Across interpretable features in several language models, as little as 0.1% of the candidate MLP channels construct 90% of what the MLP layers contribute to a feature, and 0.06% of downstream channels account for 91% of the behavioral effect of steering along it. Connecting the two maps shows that the writers' behavioral effect depends on the same consumers that carry steering: holding those consumers at their clean activations removes much of it. An edit along the direction matched to the writers' change in the feature tests whether steering reproduces their behavioral effect: in one model the matched edit reproduces little of it, while the part of the writers' change that leaves the feature's coordinate unchanged reproduces 85%.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic interpretability, feature representation, Sparse autoencoders

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UqAcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes to bridge the gap between representation-level interpretability methods, which identify semantic directions in the residual stream of transformers, and component-level circuit studies, which identify the model’s own units responsible for a behavior. The authors identify 'writer' and 'consumer' MLP neurons, that construct or process those residual directions, and find that just a few neurons are responsible for most of the MLP contribution to a direction and for most of the behavioral effect of steering along it.

Critical strengths

Although my understanding of this paper was severely limited by its clarity, the fundamental research question (whether a restricted set of MLP neurons produce and consume semantic directions in the residual stream of LLMs) appears interesting.

Critical weaknesses

The major weakness of this paper is its clarity. Despite working in interpretability, I had extreme difficulties in understanding the methods, claims and results of the paper. The writing is heavily compressed, many newly coined terms are used without definition, and the notation is hard to track. Below I list some concrete examples, so that the authors have actionable points to address:

  • "Steering along a direction shows that the model responds to an edit, but it does not test whether the edit does what the model’s own components do." This is the central motivation of the paper, yet it is unclear: why and how would an edit do what a component does?
  • "The interventions act on the original model, so the localized sets describe the model as trained rather than a surrogate." It is unclear which surrogate the authors refer to, and the contrast seems also misleading, since surrogate-based methods also aim to describe the original model.
  • "Consumers preferentially restore their own feature’s edit." This claim appears twice in the Introduction and it cannot be parsed before going to the results.
  • "direction without a separate readout" What does this refer to?
  • "Budgets from the first at which it changes sign overshoot the clean endpoint, so we take the smallest earlier K within one standard error of the smallest remaining effect among them." This sentence is very complex and hard to understand.
  • "channels", "features", "coordinates", "edits" are never properly defined. Specifically, "channel" seems to refer to MLP neurons but this is nonstandard in the field, and the paper itself does employ the term neuron in some cases.
  • What is e\mathbf{e} in line 90 and how does it relate to d\mathbf{d}?
  • What are "low-coordinate inputs"? How are they defined?
  • What does the "confidence probes" task refer to?

The lack of clarity of the submission is not limited to these points, and I believe a thorough rewriting of the paper would be needed to make it accessible to readers. In its current form, I cannot assess its technical soundness.

Questions for authors

Please see the weaknesses for some requests of clarification.

Reviewer_oxwgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper examines the claim that features (extracted by SAEs) carry the relevant information, which can be used to steer the transformer's behavior. It does so by defining new circuit components that surround the feature: the writers (upstream MLP neurons) and consumers (downstream MLP neurons) that propagate that feature. Their experiments demonstrate that a sparse set of writer neurons carry the dependence of the feature from input, and a sparse set of consumer neurons manifest that feature in the output. However, their main contribution is to contest the claim that the residual stream features allow you to directly steer the LLM. In other words, imagine you are a passenger in a self-driving car, and you see the steering wheel turn when the car turns. You might infer that you can control the car by turning the steering wheel, but when you grab the wheel, the car barely moves. The mechanism of steering mostly bypasses the wheel.

Their experiments, when applied to open-weight models like Gemma-3 IT, find that significant information sometimes flows from the writer neurons to the output while bypassing the feature in question. This is a highly model-dependent phenomenon, though it seems that the residual stream feature captures more of the information flow from input to output for models like GPT-2 that are strictly pre-trained on prediction, while models like Gemma-3 IT that rely on post-training seem to find bypasses of the feature. They do this by swapping out the writers with a corresponding edit in the feature direction, and find that the latter sometimes is not sufficient to capture the downstream effects. Despite bypassing the features, the consumer neurons are found to carry most of the writers' effect.

Critical strengths
  1. The paper clearly identifies a weakness in focusing on features as a core interpretability tool. The fact that they can be almost completely bypassed in some open-weight models indicates that much more is necessary to understand the mechanisms that underlie a transformer's behavior and how it can be steered.

  2. In addition to clearly demonstrating a problem with a well-established technique for interpretability, they also propose a clear partial solution which seems to work, in part. The writers and consumers seem to give a partial structure for the circuit architecture that may explain the inner workings of transformers. In this way, they clearly identify a problem and propose a partial solution.

Critical weaknesses
  1. Since the consumer and writer neurons are limited to MLP layers, ignoring the attention mechanism, it is unclear what features these elements are missing. There is some information that bypasses the writers and consumers, and it is unclear whether we should expect this because there are other computational elements in the attention mechanism that carry that information, or whether there is something fundamentally missing with this approach. Appendix C shows attention outputs move the coordinate several times more than the MLP on Pythia and Qwen, which shows that the authors are aware of this weakness.

  2. The writing in this manuscript is extremely difficult to read. The results appear as a list of numbers in the text and tables. There are so many different cases (different models, measures, parameters, etc.), and the units are not always specified. Table 2 reports endpoint units that differ across checkpoints; Table 2 uses signed effects while Table 5 uses absolute effects, and Fig 2 uses two different units. For this reason, it is very difficult to interpret the numbers and know how significant they are. As a reader, I don't know which numbers I should be excited about, and which numbers are just reporting technical details of the simulation. The important numbers should be reported with context that gives the reader a baseline for what to expect. Right now, it reads like numerical mush. The paper badly needs a synthetic rewrite that organizes the numerical results into a more digestible form.

Questions for authors
  1. Why is it appropriate to focus exclusively on the MLP neurons as implementing the writers and consumers, rather than including the attention heads in this mechanism?

  2. Is there a principled reason for using the ratio of expected bias differences 1- E(|B^2-B^0|)/E(|B^1-B^0|) in Eq. 3 rather than another measure that distinguishes probability distributions, like cross-entropy?

Answers to 1 and 2, along with a more polished mathematical presentation, would raise my score.

  1. Are the authors suggesting through this research that rather than a feature being identified in a single layer, it is most reasonably stored and steerable through multi-layer interventions?
2 comments
Reviewer_oxwgOpenReview ↗

Response to authors

I would like to thank the authors for answering each of my questions and for the revision. The justification for the consumer score is convincing, and I appreciate the authors' patience. I consider that question settled.

On restricting to MLP neurons, I'm fine with it as a stated limitation, but the authors respond by explaining why MLPs are convenient. This doesn't explicitly recognize what's missing. By the appendix, attention often matters more than MLPs in some models, and those are the same models where the results are weakest. That suggests attention is a missing piece of this story.

The presentation is better at the level of individual equations, figures, and examples. The overall structure is the same. There are still many tables and checkpoints per claim, and the headline isn't clear.

On steering, the new experiment is welcome but more mixed than the abstract makes it sound. Writers help in the models where editing the direction doesn't work, but where the direction does work, it wins. The writer looks like a fallback, not a better handle in general for control.

I'm raising presentation from 1 to 2 and keeping my overall rating.

Thank you for the follow-up and for raising the presentation score.

Attention. We prioritize MLP neurons because it writes directly into the residual stream, whereas attention mainly moves information between positions. You are right that our earlier response did not explain what MLP-only localization leaves out. On Pythia and Qwen, attention contributes much of the feature value, and on Qwen it also carries much of the edit's effect on behavior. Our writers and consumers therefore capture only part of the circuit in these models. However, missing attention does not explain why direction edits fail to reproduce the writers' effects on Gemma-2 IT and Gemma-3 IT. There, the difference comes from what the MLP writers change outside the supplied direction. Conversely, Pythia relies heavily on attention, but editing its direction wodo yrks well. The final revision makes this distinction clear in the Limitations.

Presentation. Our headline is that a feature direction can be traced to a few MLP neurons that build it and a few that carry it to behavior, and comparing the direction with these neurons shows whether it is a faithful handle. The final revision states this in the contributions and groups the checkpoints in the results by whether editing the direction works.

Steering. We agree with your interpretation. Where direction editing works well, it steers about as effectively as the writers or better. Where it is weak, the writers provide a stronger handle. Our method distinguishes these cases by comparing writer ablation with a direction edit that produces the same change in feature value, separately from the steering comparison.

Reviewer_NpVdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents a systematic study of matching writers and consumers, the subcomponents of an LLM responsible for generating and exploiting a feature, respectively. While previous work usually focuses on only one side, either finding a feature direction or locating where it is consumed, the proposed method connects the two in a principled manner. The overall concept is reasonable, but the draft has some unnatural transitions: for instance, the Introduction states that MLPs are a natural target of analysis, while the Method and Experiments sections mostly regard SAEs as the target. The experiments show that only sparse sets of writers and consumers (around 0.1% and 0.06% of the candidates, respectively) are in effect for feature steering.

Critical strengths
  • Presents a reasonable view that connects a writer and a consumer, rather than considering only one of them.
Critical weaknesses
  • The writing could be improved. Transitions between consecutive sentences are often unnatural because connecting details are omitted. For instance, the distinction between the encoder readout and the decoder edit direction in Section 2.1 only becomes meaningful after Section 4.5.
  • Writer localization targets the SAE encoder coordinate, a post hoc artifact rather than a model component. The identified writers thus construct the SAE’s coordinate, not a model intrinsic feature, and inherit any misspecification of the SAE.
  • The practical value of the protocol is asserted rather than demonstrated. Section 6 gives procedural guidance, but no experiment shows that a direction passing the faithfulness test steers more reliably, or that writer channels serve as a better handle than the direction itself.
Questions for authors

The paper intervenes on native MLP channels but defines the target feature through an SAE, which is trained post hoc on the model’s activations rather than being part of the model. In what sense, then, are the identified writers writers of an original feature of the base LLM, rather than of a coordinate defined by the SAE?

Comments on the paper

1 comment

We thank the reviewers and the area chair for their time and feedback. The final revision incorporates the discussion with the reviewers.

The reviews identified clarity and structure as major weaknesses of the submission. Key terms are now defined before they are used, and the method opens with an overview of the pipeline. The contributions state the main finding, and the results group checkpoints by whether a direction edit is clearly stronger than a random edit. This also makes clearer when steering through the writers helps.

We added the published refusal direction as a second non-SAE case, strengthening the evidence that localization does not depend on an SAE. The Limitations show that missing attention does not explain the cases where direction edits fall short. The appendix also adds three checks supporting the main findings.

The original method and results remain unchanged. We hope the final revision addresses the remaining concerns about clarity and structure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.