- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies state tracking in transformers and uses group composition as a controlled setup. The authors leverage the group structure of the problem, primarily abelianization, (normal) subgroups and cosets, to mechanistically study the transformer's internal computation and connect it to partial accuracy. In particular, they find that transformers do not distinguish different members of the same coset in a group's abelianization.
Critical strengths
- The overall question of how do the algebraic properties of a particular problem determine certain mechanisms internal to a model is interesting.
- The synthetic group composition task is an appropriate controlled scenario to investigate state tracking.
Critical weaknesses
- The paper is very difficult to read and opaque, with many critical choices not sufficiently supported. For example, in section 3.1 the paper exposes two 'recipes', A and B, with which experiments are run. However, it is not clear why do the authors make this choice and what are the authors trying to control for by having both setups. I attempted to resolve this by reading the referenced appendices H and I, but these appendices simply list the specifications of these recipes; while this is great for reproduction, the scientific objective of this control is unclear. Furthermore, appendix I contains many tables with different setups, once again without explaining why they are relevant. If the authors are trying to assess whether their conclusions hold for different hyperparameters, a series of plot would be a better presentation. Additionally, the table contains a column "Status; Location" and reports 'pilot' and 'exploratory' experiments not included in the paper. I believe that the paper needs a major rewrite to make it clearer and understandable.
- Overclaims. The authors findings about transformers coset tracking seem limited to a particular training and data sampling regime. As acknowledged by the authors in Appendix G, transformers can indeed achieve perfect state tracking (in distribution), and their "order-blind" results and prediction may only apply when a particular sampling and training process is applied. This introduces confounding factors and makes it less clear when do these findings actually apply. More care is needed to assess what conditions are necessary for the author's predictions to be true.
- Theory/Practice mismatch. Following the above, the theory should be adapted to reflect these further limitation. Additionally, a transformer is not an order-blind function, even when positional encoding is removed, despite this property being an assumption of the propositions in Section 4. Therefore, more care is needed to justify why the theory is relevant for the considered cases when the experiments do not satisfy the same theory's assumptions.
Questions for authors
- Can the authors clarify what is the significance of Recipe A vs Recipe B? What is this trying to control/ablate? The same question applied to the tables in Appendix H.
- Can the authors clarify what are the conditions needed for the theory to apply, and make important requirements (such as the data generating distribution) more explicit and justified? Do these further requirements hinder the significance of the results
- The geometric analysis from Section 6 currently covers a GRU. Can it be extended to transformers?
9 comments
Scope, theory, and geometry
W2 / Q2 (scope and conditions). The plateau prediction is specific to uniform i.i.d. full-group inputs. Our framework also makes distinct predictions for other input sets.
- Where the plateau applies? It applies under uniform i.i.d. inputs from the full group, our main setting (recipe A, §3.1), and Theorem 2 states this assumption. As a robustness check, we evaluated 20 architecture and training configurations on D4 with full-group inputs, and none leaves the plateau (Fig. S7).
- Other settings are predicted as well. An algebraic criterion on the input set says when counting can solve the task.Recipe B reaches exact tracking on {r, s}, and in a preregistered test it stays at the plateau on {r, s, e}, as the criterion predicts (§5.3). Recipe B with generator inputs also reproduces the exact tracking of Liu et al. (2023) (Table 7).
- Why this does not reduce the contribution? In every setting we tested, the same readout identifies the tracked state, and the class size gives its accuracy (§3.4, §5.2, §5.3).
W3 (a Transformer is not order-blind). Thank you for raising this point. We would like to clarify that the theorems are about order-blind predictors, and a new corollary (App. S) extends the bound to models that are nearly order-blind, which the trained Transformers are beyond the exact frontier.
- New corollary (§4, App. S). If reordering the prefix changes a model's outputs by ε in total variation on average, its mean probability on the true state is at most the order-blind optimum plus ε.
- Measurement (App. S). Beyond the frontier, reordering changes the outputs by .011–.051, against .22–.50 inside it, and the bound holds in all 28 runs. The bound is on probability, and the accuracy plateau is the measurement in Table 1.
Q3 (geometric analysis for Transformers). Yes. Appendix R analyzes 40 recipe-A Transformer endpoints across 11 groups. In 38 out of 40, we find that the class means have the minimum dimension expected for the abelian quotient, and the fitted transformations approximately match its symmetries. The exceptions are a low-accuracy Q8 endpoint and an H₃(ℤ/5) endpoint with a higher-dimensional representation. Subspace swaps transfer the donor class at the intervention position, while separate single-position interventions generally do not sustain this transfer at later positions. These results establish a causal role in the current quotient readout, while leaving its sequential implementation unresolved (§6.2).
We hope these additions clarify the scope of the theoretical claims and the corresponding empirical evidence.
Follow-up: summary of answers to W1–W3 and Q1–Q3
Dear Reviewer y7zo,
Thank you again for your review. As the discussion period ends on November 18, we would be grateful to know whether our response and the revision address your concerns. Below is a detailed summary.
(W1, Q1): Recipe A is our main recipe for studying the abelianization plateau in Transformers. Recipe B, adapted from Liu et al. (2023), tests what it takes to go beyond the abelianization. The initial submission introduced both in §3.1 (“Recipe B, adapted from Liu et al. (2023), uses a 3-layer, width-512 GPT-2 transformer”) and used recipe B in §3.4 (“Quotient tracking also extends beyond the abelianization. With recipe B and odd-permutation inputs, …”). §8 also stated that “Training recipes can determine whether models reach exact tracking or remain at a partial resolution,” and App. G opened with the aim: “We now investigate how models recover state distinctions beyond the abelianization. […] We therefore use generator-restricted inputs to study how models refine quotient-level solutions.”
We also ablated recipe B in App. O of the initial submission: removing position embeddings gives chance accuracy, switching to full-group inputs produces two plateaus and one partial solution, and replacing the optimization block leaves only a short exact prefix (frontiers 12–22), while changing the MLP width keeps exact tracking. Table 7 (App. D) then crosses the two recipes with the two input types: among these cells, only recipe B with generator inputs leaves the 1/2 plateau (99.999% vs. 50.0%). We kept these results in the appendix because recipe B serves as a control, while the main text tests the plateau law under recipe A.
We agree that this rationale was too easy to miss, so the revision states it in §3.1 and §5.3 (with the block and position-encoding ablation in Fig. S7b), and App. H opens with a table linking each configuration to where it is used. In App. I, the coverage tables now list only runs that support a claim, with a column naming where each is used. Pilot and exploratory runs, initially included for transparency, are kept in a separate table as each is unused. A single figure now summarizes robustness across configurations (Fig. S7a: 20 architecture and training configurations, 60 runs on D4).
(W2, Q2): The abelianization plateau prediction for Transformers is stated for uniform i.i.d. full-group inputs, as stated in the initial abstract and §1. The exact tracking in App. G happens outside this condition, and in §4 we noted that “restricted alphabets require separate analysis (App. G).”
In fact, the exact tracking in App. G is not a counterexample: it is a counting shortcut. On {r, s}, the k-th r at position p has exactly p − k reflections before it, so the whole D4 state follows from three parities: the counts of r, of s, and of r at even positions (Eq. 12). Our activation-replacement experiment indicates that the successful models read these counts rather than a running product. Lemma 2 in App. O shows that this shortcut exists exactly when the inputs come from at most two of D4’s four abelianization classes ({r, s} uses cosets {r, r³} and {s, r²s}). Adding e uses a third class {e, r²} and breaks the shortcut: in a preregistered test, recipe B reaches exact tracking on {r, s} but stays at 1/2 on {r, s, e} (Table 21). On full-group D4 inputs, counting recovers only the class, which is where the plateau sits. All of this, including the full proof, is in App. O of the initial submission.
So, to answer your Q2, the input condition does not narrow the significanceof the results : the restricted-input setting is a controlled case in which a model gets past the abelianization, and the same algebra predicts when it can.
(W3): We did not assume order-blindness. The initial submission measured it in 28 runs (§4, p. 5) and noted that these tests do not prove it (“…without proving strict order-blindness or explaining training’s choice”). For clarity, the revision now adds a corollary to the main text that turns this measurement into a bound, with the proof in the new App. S.
(Q3): Our initial submission already included Transformer geometry analysis: “we also find internal geometries reflecting learned abelian quotients in Transformers (App. R).” In App. R, we analyzed 40 Transformer endpoints across eleven groups: “We find that their mean hidden vectors usually occupy a small subspace whose geometry agrees with the abelian quotient. Replacing the activation in this subspace transfers the predicted class at the intervention position.” In 38 of the 40 endpoints, “the retained rank and the fitted maps’ trace and determinant patterns agree with a faithful real representation of minimum dimension.”
If this resolves your concerns, we would appreciate it if you would consider updating your assessment. If not, we are happy to clarify anything further.
Recipes and presentation
We thank the reviewer for the review and for the concrete presentation requests.
W1 / Q1 (recipes A and B). In short, recipe A is our main setting, and recipe B is a control that finds out why Liu et al. (2023) reach exact tracking while we do not.
Appendices H and I. Appendix H now opens with a table mapping each configuration to the section that uses it. In Appendix I, every row names the claim it supports and where it is used, and the Status column is removed. The pilot and exploratory runs are now in a separate table with the reason each is unused, such as a single seed, a short run or a missed registered target. We list them for transparency, and no claim relys on them.
Plots. As suggested, a new figure (Fig. S7) replaces the robustness rows of Appendix I. It shows full-state accuracy given a correct quotient for 60 D4 runs across 20 architecture and training variants under full-group inputs, next to the generator-input runs. No variant leaves the plateau.