- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The authors use a gauge theory formalism to describe which transformations of the heads leave the outputs of the attention mechanism unchanged. The papers main theoretical contribution is Theorem 4.1 which claims that the set of symmetries they identify are all the possible symmetries of the head parameters that leave the output unchanged. They identify how changes in the architecture (RoPE, QK-normalisation, grouped query attention etc.) change the symmetry group. Experimentally they do two things, they test these symmetries by pairing predicted symmetries with plausible but non-predicted "random control" symmetries and then "editing" the weights of several LLMs and seeing whether they can distinguish between outputs of edited and non-edited models for each of the symmetries. Secondly they determine if these symmetries can be used to match heads between training runs.
Critical strengths
- I think a complete enumeration of the symmetries inherent in transformer models and developed theory on how to find these for different architectures, as well as a consideration more generally for whether mechanistic interpretability methods are robust to the sorts of across training runs and transformations of the weights is valuable for the field.
The field has been vulnerable to interesting sounding results that end up not surviving controls or that oversell, so an attempt to put this on a more rigorous footing is welcome.
- The paper gives concrete recommendations in section 4.4 on how practitioners should interpret and utilise this work.
Ultimately the goal of MI is applied in that we want to use these methods to understand LLMs and (usually) make them safer. Without this the work would be a bit too abstract.
Critical weaknesses
- I found this paper quite hard to follow.
I, and I imagine most other MI researchers, don't have much knowledge of gauge theory, and this paper isn't especially sensitive to this. At numerous points notation is introduced without any introduction or explanation. There are a few minor examples of this like using before it is clear you mean the in RMSnorm, or not defining , but the worst offender is not defining what or are. I understand these are common gauge theory terms but since most will not know much about that I think the paper would benefit from a bit more explanation and intuition. The paper by Godfrey et al. that you cite is a good example of giving intuition for work like this.
More generally the paper has some odd structural choices. Why is the rounding pitfall so prominently discussed? This only becomes clear towards the end of the paper, and even then bringing it up right at the beginning is odd since it seems like a minor correctness point? This has something to do with how numerical precision interacts with the symmetries you are studying but this isn't adequately explained.
Questions for authors
N/A
3 comments
Reviewer PJau: where your points are in the revised PDF
In the revised PDF, the points of your review are addressed at the following places.
Notation (Sec. 3, p. 3). , , and are defined at first use. The words "gauge", "orbit" and "generic" are explained in plain terms, and no gauge theory is needed. is introduced as the stabilizing constant of RMSNorm, and it no longer appears in the introduction.
Intuition (Sec. 4.1, p. 4). The paragraph "Intuition" before Thm 4.1 explains, with Figure 1, why inserting inside a head changes the weights and not the function. Two shorter paragraphs do the same for rotary embeddings (after Prop. 4.3, p. 5) and for the residual basis (after Prop. 4.9, p. 7).
Rounding pitfall. It is no longer in the introduction or in the contributions. The protocol of Sec. 5.1 (p. 8) explains why it matters: a broken symmetry could look exact. Details are in App. B.1 (p. 19).
What you found useful
- The enumeration of symmetries: Sec. 4.2-4.3 (p. 5-7), and Table A (p. 8), which shows which symmetry holds on which architecture.
- The recommendations: Sec. 4.4 (p. 7), which now includes a numerical test of a claim on three models, and Table 9 (p. 25), which gives an invariant replacement for common objects, including sparse autoencoder features.
- Robustness across training runs: Sec. 5.2 and Figure 3.
Thank you for taking the time to address my concerns. While you've addressed the lack of sufficient background knowledge which has improved the legibility of the manuscript, having read the other reviews I agree that this paper needs further work before it is ready for acceptance. As such I maintain my score.
Reply to Reviewer PJau: presentation
Thank you. We agree that the paper assumed too much background. The revised version makes three changes.
Notation is defined where it first appears (Sec. 3). GL(k) is the group of invertible k×k matrices, and the gauge has one GL(dk) and one GL(dv) factor per head. SH is the group of permutations of the H heads, and ⋊ records that relabelling the heads also relabels their matrices. θ denotes all weights, and ε, the RMSNorm stabilizer, is introduced before it is used. The word "gauge" needs no gauge theory here. It only means a change of weights that leaves the function unchanged, and the text now says this.
A short paragraph before Thm 4.1 gives the intuition. Inserting ShSh−1 between the query and key projections of a head changes both matrices but not the attention scores, so a statement that depends on the individual matrices cannot describe the function. Thm 4.1 and Cor. 4.2, which the revision credits to Tran et al. (2025) and Kim & Kim (2026), show that for generic weights head relabelling and such insertions are the only changes of this kind.
The rounding pitfall is a point of method for anyone running such checks, and we no longer list it as a contribution. The test has to separate exact symmetries from controls. Some implementations compute normalizations in float32 inside a float64 model. A control that changes one weight by a relative 10−10 then gives bit-identical logits, so a broken symmetry looks exact. The revision moves this point from the introduction to the method paragraph of Sec. 5 and explains it there.