- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes to bridge the gap between representation-level interpretability methods, which identify semantic directions in the residual stream of transformers, and component-level circuit studies, which identify the model’s own units responsible for a behavior. The authors identify 'writer' and 'consumer' MLP neurons, that construct or process those residual directions, and find that just a few neurons are responsible for most of the MLP contribution to a direction and for most of the behavioral effect of steering along it.
Critical strengths
Although my understanding of this paper was severely limited by its clarity, the fundamental research question (whether a restricted set of MLP neurons produce and consume semantic directions in the residual stream of LLMs) appears interesting.
Critical weaknesses
The major weakness of this paper is its clarity. Despite working in interpretability, I had extreme difficulties in understanding the methods, claims and results of the paper. The writing is heavily compressed, many newly coined terms are used without definition, and the notation is hard to track. Below I list some concrete examples, so that the authors have actionable points to address:
- "Steering along a direction shows that the model responds to an edit, but it does not test whether the edit does what the model’s own components do." This is the central motivation of the paper, yet it is unclear: why and how would an edit do what a component does?
- "The interventions act on the original model, so the localized sets describe the model as trained rather than a surrogate." It is unclear which surrogate the authors refer to, and the contrast seems also misleading, since surrogate-based methods also aim to describe the original model.
- "Consumers preferentially restore their own feature’s edit." This claim appears twice in the Introduction and it cannot be parsed before going to the results.
- "direction without a separate readout" What does this refer to?
- "Budgets from the first at which it changes sign overshoot the clean endpoint, so we take the smallest earlier K within one standard error of the smallest remaining effect among them." This sentence is very complex and hard to understand.
- "channels", "features", "coordinates", "edits" are never properly defined. Specifically, "channel" seems to refer to MLP neurons but this is nonstandard in the field, and the paper itself does employ the term neuron in some cases.
- What is in line 90 and how does it relate to ?
- What are "low-coordinate inputs"? How are they defined?
- What does the "confidence probes" task refer to?
The lack of clarity of the submission is not limited to these points, and I believe a thorough rewriting of the paper would be needed to make it accessible to readers. In its current form, I cannot assess its technical soundness.
Questions for authors
Please see the weaknesses for some requests of clarification.
Response to authors
I would like to thank the authors for answering each of my questions and for the revision. The justification for the consumer score is convincing, and I appreciate the authors' patience. I consider that question settled.
On restricting to MLP neurons, I'm fine with it as a stated limitation, but the authors respond by explaining why MLPs are convenient. This doesn't explicitly recognize what's missing. By the appendix, attention often matters more than MLPs in some models, and those are the same models where the results are weakest. That suggests attention is a missing piece of this story.
The presentation is better at the level of individual equations, figures, and examples. The overall structure is the same. There are still many tables and checkpoints per claim, and the headline isn't clear.
On steering, the new experiment is welcome but more mixed than the abstract makes it sound. Writers help in the models where editing the direction doesn't work, but where the direction does work, it wins. The writer looks like a fallback, not a better handle in general for control.
I'm raising presentation from 1 to 2 and keeping my overall rating.