- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work studies the phenomenon of self-repair in LLMs: the behavior of a model appears to compensate for ablated internal components. They do so by defining a systematic study whose conclusion is somehow in contrast with our current understanding: instead of considering self-repair as a noisy, difficult to explain phenomenon, the authors find that the causal repair response of an LLM abides to a simple law, with one important "slope" parameter. The slope parameter influences how the model reacts to a removed "signal". Their work helps in identifying the role of LLM internal blocks, and which and how closely they follow the simple law. They also demonstrate that, in some cases, the slope parameter can be inferred by model weights alone.
In conclusion, this paper argues that the so called self-repair might be merely a usual operation of counter-weighting reacting to a contrastive signal emerging from the model.
Critical strengths
- This work is very interesting, and sets off to study an important phenomenon, self-repair, that hinders ablation stuides on LLMs. The authors posit that self-repair is not an unreachable mechanism to understand: on the contrary, it abides to a simple affine law, with a slope coefficient that is not only interpretable, but also predictable. This slope coefficient can be readily used to understand the effects of individual components of an LLM, in terms of counteracting or reinforcing a removed signal.
- The "do-calculus" implied by this work is used to study a variety of models, which caters to the generality of the conclusions discussed by the authors. The task used to study self-repair is simple and clear, which is a virtue in this kind of empirical analyses of causation.
Critical weaknesses
- The text is very hard to read, sentences are terse, mixing terms that I find colloquial (that technically prepared devs would use to argue in a discussion) with mathematical notation that is not always explicit, and in places confusing.
- As discussed by the authors themselves, the single task used in this work, despite its virtues discussed in the strengths, might not be sufficient to state that the methodology and the findings (the affine law) apply to more complex tasks.
Questions for authors
- Figure 1: why the dose axis is inverted w.r.t. the usual convention of having negative values on the left of 0, and positive values to the right?
- In section 1 you present equation 1, but is not defined (yet). In the introduction, you define "directions" as internal units, such as a neuron, and indicate that you perturb an upstream neuron and check the opposition a downstream neuron excerts. Are you modifying the neuron activation output? Overall, for a reader that parses sequentially your paper, the introduction appears vague. Would it make sense to be a bit more precise on the "mechanics" rather than using a narrative that is reminiscent of biological interventions?
- In section 2: concerning weight-space bases, by calling Neuron directions and SVD basis in both cases, I think the notation might be confusing. Similarly, the "single interface" is difficult to grasp. What is (it is defined much later, and turns out to be a clamp value)? Overall, a pass on this section to clarify and explicit the notation would be useful to improve the readabiliy of the paper.
- Section 3, which is the most important, is somehow elusive to me, technically. I have an overall understanding of the various quantities described here, and ultimately the estimand is clear. My main issue is with the notation: I think the authors made an effort to "clean it up" such that it is not too heavy, but I'm frequently lost in double checking past definitions to make sure everything "adds up".
- Section 4: the intro to this section really reads like Claude before Opus 5.5 release. I suggest a pass to remove "Claudisms": it's this, not that. here we do this, it certifies, the verdict, ...
- Section 4: What do you mean by "pole"? It's defined in sec 2, and used in sec 4. A pole is the product of a direction by the unembedding matrix . You say a pole sets its direct effect. Its who? The pole?
- Section 4: What is a bench? "A direction enters the bench"...
- In figure 3, authors say that in the shaded band, "each crosses zero on both classes at a sufficient dose". This seems not to be true for Qwen and Mistral. Why? (see also Figure 4).
2 comments
Thank you for your rebuttal
Dear Authors, thank you for your rebuttal. Some of my questions have been sufficiently addressed, but the revised paper still has the same weaknesses presented in my first review. Section 3 is still challenging to parse, despite the additional example and algorithm.
I will maintain my score.
Response to Reviewer FU3U
We greatly appreciate Reviewer FU3U’s thoughtful engagement. The reviewer found the paper hard to follow due to flaws in the presentation. Nevertheless, they found it very interesting, reconstructed its arguments, examined sections, and provided valuable feedback.
The central idea of the paper is that conventional ablations (zero, mean, resample) can cause a semblance of self-repair due to pre-existing coupling interactions between the upstream core(s) (e.g., a neuron, Table 1) and downstream counterweights (e.g., neuron, singular direction, Tables 6-9). Any ablation is an uncalibrated point that this coupling interaction surfaces. This was measured by using the continuous dose axis and a nested counterfactual.
Conventional ablations are single-point deletions and are ill-equipped to capture this coupling interaction. The approximate affine law is one such concrete interaction that quantifies this receptiveness and, surprisingly, generalizes across four different models and to the IOI circuit identified by [Wang et al. 2023]. This dose-response could show up as a different curve, but the main thesis will still stand; it's not contingent on a single curve form. The affine law is a parsimonious approximation that consistently showed up, so we thought it was worth reporting.
A1: The dose axis λ is ordered by corruption fraction of upstream core(s), moving from clean (left) to counterfactual (right). Any intervention ν on a core(s) neuron can be written as ν=m−2λΔ, where m and Δ are the midpoint and difference between counterfactual and clean scalar activation of the neuron.
A2: Yes, we modify the neuron's scalar activation in the upstream core; more precisely, the scalar that multiplies the fixed output column. We have defined ownr in the introduction now.
A3: We chose this notation to make clear that for any component c, its output could be expressed in either of the two bases (neuron directions and SVD basis), i.e., oc=∑jzcjwcj=∑kαckvck. Yes, ν is any scalar intervention clamp on a neuron or singular value, it should be ν and not v. It was a mistake.
A4: We have added an algorithm box (Box 1, pg 5) containing a concrete example and procedure. Figure 2 can also help in encapsulating the whole procedure - there are two kinds of interventions working here:
The dose (λ) intervention is applied to the core, and the downstream component is swapped with the activation value it will assume in the same mirror-dosed model (See Box 1, pg 5).
A5: Taken. We have revised it.
A6: The directions (ωr) ‘pole’ is a set of tokens that are promoted or suppressed at extremes of unembedding projection ωr⊤WU. Examples in Tables 6-9.
A7: Bench is the set of directions in weight space. See Tables 6-9 for all such chosen directions; we chose them based on the interpretability of the poles for the relevant task.
A8: The reviewer is right, and the caption was unclear. By sufficient dose, we meant any point on the whole tested range λ∈{+1,+0.5,0,−0.5,−1,−3,−5,−7}.
Thank you again for your detailed engagement, and your questions made the revision clear. Please let us know if we can clarify anything further.
Reference. Wang, Variengien, Conmy, Shlegeris, and Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. ICLR 2023.