- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The authors derive an extensive theory of when independently trained neural networks must learn identical internal computations. They define 2 distinct alignment paradigms -- weak alignment (one layer's preactivations are an affine function of the other's), and strong alignment (individual units across networks match one-to-one up to a permutation/rescaling). They prove several theorems to this effect:
- Zippering theorems (Theorems 1 and 2): for ReLU or Softplus networks satisfying minimality and regularity conditions, agreement at a terminal layer forces strong alignment at every earlier layer.
- Weak–strong equivalence theorems (Theorems 3 and 4): weak alignment at the layers immediately before and after a coordinatewise nonlinearity forces every task-used unit to match. The fraction of matched units is lower-bounded by the minimal number of nonlinear units any accurate solution requires, relative to the layer width.
- Transformers: they extend their theory to predict privileged MLP gates and privileged whole attention heads. They define this through the gauge-invariant quantities and .
- Empirical evidence: in AlexNet, ResNets, ViTs and Pythia, they report ordered layerwise alignment, drops in alignment across nonlinearities, learned units and heads matching better than random mixtures, and stronger recurrence of more-used components as predicted by their proposed theories.
Critical strengths
-
The paper tackles a long-standing problem of why convergent learning happens, rather than simply documenting it. They answer this through the lens of identifiability (i.e., what end-to-end function determines about internal computations), which is an excellent way to tackle it.
-
Weak vs. strong alignment is a clean distinction, and clearly motivated. The intuition for weak–strong equivalence is also elegant. This is done through the lens of two different coordinatewise nonlinearities -- a ReLU kink or a Softplus curvature ridge -- both of which determine a network's native axes. Hence, mixing these axes (e.g., via a dense affine map) breaks downstream agreement.
-
The authors handle parameter symmetries in transformers carefully. They compare the heads through the gauge-invariant quantities , rather than individual coordinates defined on . They also explicitly mention that the residual stream has no privileged basis.
-
The empirical controls are well designed:
- Random orthogonal mixtures preserving the linear span of the units
- Sum-preserving head mixtures
- Shuffled-input nulls
- Cycle consistency across three models
- Causal head ablations in Pythia
-
Formalizing the otherwise informal (and philosophical) contravariance principle (harder tasks admit fewer solutions) to establish a lower bound on the fraction of layer- units that must align strongly across networks is a clean formalism.
Critical weaknesses
My main concern is the gap between the proposed theory's conditions and the presented empirical evidence, in particular for transformers. Please bear with me -- this is slightly long, and a sketch rather than a complete proof. I have tried to use the paper's notation in as many places as possible.
Problem exposition. The mechanism by which a coordinatewise nonlinearity selects native axes can be formalized as follows. Consider a layer followed by an affine readout with of full column rank. An orthogonal reparameterization , , (with adjusted) preserves the function for all iff for a constant that is absorbed by the next bias term. Differentiating yields . Right-multiplying by then gives , which is clearly diagonal for every . If is non-constant, choosing with and (), , gives us This is diagonal only if each column is a signed standard basis vector. Moreover, the constant offset then rules out sign flips unless is constant (as for tanh). So, for ReLU, Softplus, and GELU the only orthogonal symmetries of a plain feedforward layer are permutations (for ReLU, the argument applies at almost every ).
(a) Symmetry breaking only holds where the data expose the nonlinearity. The argument presented above requires the realized preactivations to visit points where differs. If a group of units, however, remains in a regime where is approximately linear with a common slope (e.g., always-active ReLU units, or GELU units with , where ), then small rotations among those units become approximate symmetries of the layer. I should caveat this by saying that whether they would remain approximate symmetries of the whole network also depends on the downstream weights. Units in different regimes (e.g., versus , where ) cannot be mixed in this way. Put differently, if the symmetry is only required on the region visited by the data, differentiating the diagonal condition along gives us . Hence, axis is only determined if somewhere in , i.e., if the data reach the curved part of for that unit. Similarly, for nonnegative activations, nonnegativity only forces a permutation when the realized activations span the orthant. In other words, if they lie in a restricted cone, e.g., , any rotation by preserves nonnegativity without being a permutation. The paper's notion of a used axis addresses this for ReLU (the task must cross the kink trace) and Softplus, but there is no corresponding definition for GELU, which lays the ground for my main issue in the next point.
(b) The transformer predictions rely on an unproven extension. The exact theorems cover ReLU and Softplus, but all transformer experiments use GELU. Appendix D.2.4 outlines a proof strategy for a GELU weak–strong theorem by analogy with Softplus, but states that deep zippering for GELU requires a "genuinely new" argument, and that exact GELU is not globally injective (two inverse branches). The predictions of privileged MLP gates in ViTs and Pythia therefore rest on a proposed rather than proven result. This should be stated clearly in the main text, together with which GELU variant (exact or tanh approximation) each model uses, because the appendix notes these must be analyzed separately.
(c) The empirical support for privileged ViT gates is limited and insufficiently quantified. Given (b) and (c), the experiments carry bulk of the argument for transformers. However, in Fig. 4b learned ViT gates match only marginally better than random orthogonal mixtures in most blocks, and the mixtures are higher at block 1. Similarly, random mixtures outperform learned channels at AlexNet conv1 (Fig. 4a), and the gaps for Pythia MLP gates (Fig. 4d) are small. Yet, the text states that ViT gates "match more closely than random orthogonal mixtures" (L437). These comparisons rest on three ViT seeds and are reported without error bars or significance tests, so it is not possible to tell whether the gaps are distinguishable from seed-to-seed variability. The claim should be assessed block by block, or supported by an explicitly defined aggregate statistic. The evidence for privileged heads (Fig. 4c,e) is stronger, but the gap disappears in later blocks.
(d) Are the soft bounds non-vacuous? Exact weak alignment never holds between independently trained networks, so the practically relevant results are the soft theorems. Theorem 4 gives and Theorem 2 depends on and the margin . None of these quantities are estimated for the networks studied. If the error term dominates the first term, I imagine that the bound is vacuous. For this reason, it is unclear to me whether either guarantee says anything about the networks in the experiments. Also, the zippering results are framed around networks that agree at a terminal layer (L59–61), whereas independently trained networks differ in accuracy and final representations across seeds, so I imagine the soft versions are the ones that matter in practice.
(e) Relation to the interpretation of Kapoor et al. (2025) in Remark 6. Remark 6 (Supp. Sec. C.2.4) argues that the weak evidence for privileged axes in ViTs reported by Kapoor et al. (2025) is not evidence against the theory, because their tests were applied to the [CLS] token, i.e., to the residual stream, where the theory makes no native-axis claim. I agree with this part of the argument, that the residual stream has no coordinatewise nonlinearity of its own, so a null result there is expected.
That said, Remark 6 goes further and implies that the theory's prediction of privileged MLP gates holds in ViTs. This is a separate claim that, as far as I can tell, is not established by either work. Kapoor et al. analyze the [CLS] token of GELU-based ViTs but not the MLP-internal GELU units themselves, and the authors' test of them (Fig. 4b) shows learned ViT gates matching only marginally better than random orthogonal mixtures, and worse at block 1. As discussed in the problem exposition and (a), GELU breaks exact rotational symmetry, but units that remain in the same approximately linear regime retain approximate rotational symmetries among themselves. ViT gates may therefore be privileged in principle but only weakly identified in practice. Combined with the fact that the GELU theory is proposed rather than proven (b), I do not think the current results establish that ViT MLP gates are privileged. For this reason, I would encourage the authors to temper the claim in Remark 6 and the main text accordingly.
Clarity and length. In general, I also found the notation and writing very obtuse and non-standard, which makes the paper hard to follow, despite the core ideas being interesting. Most importantly, most of the assumptions behind the theoretical results and their corresponding proofs are in pages of supplementary material. It is not feasible to verify all of these within the review period, which is why my review has focused on what I understand to be the key empirical and theoretical contribution of the work (i.e., the presence of a privileged basis in transformer architectures). That said, I am unable to make a confident judgement about the soundness of the remainder of the theoretical contributions, along with the assumptions that underlie them. I think the main text would benefit from stating the key assumptions explicitly, and perhaps the work may be better suited to a venue where the proofs can be reviewed in full, to do full justice to what is clearly a substantial amount of work.
Questions for authors
- How is a used GELU unit defined? Could the authors report what fraction of ViT MLP preactivations fall in the nonlinear regime of GELU, and whether this predicts the per-block alignment in Fig. 4b?
- Would it be possible to estimate , , , and , or bounds on them? Perhaps just for one model family, and report whether the resulting guarantees in Theorems 2 and 4 are non-vacuous (see weakness (d))?
- If, and how does the dimensions of the affine hull of the task patch affect which axes can be identified in representations?
- Remark 6 argues that the weak axis alignment in ViTs reported by Kapoor et al. reflects tests on the [CLS] token / residual stream, and states that alignment should instead appear at the MLP gates. Could the authors quantify how strongly ViT MLP gates are identified in practice, e.g., by relating per-unit alignment to the fraction of each unit's preactivations in the nonlinear regime of GELU? If weakly nonlinear units align no better than random mixtures, this would suggest that approximate rotational symmetries persist at GELU gates.
6 comments
W5/Q2: (d) Non-vacuous soft bounds. We now provide a proved positive matching guarantee in a ReLU task (Fig. 7b; Appendix C.2, Proposition and Corollary C.2.1) and measure the errors entering our bounds across six architectures (Section 5.5, lines 483-519). These studies support our approximate zippering and WSE predictions (main-text Theorems 2 and 4): our ReLU example provides a fully proved numerical matching guarantee, while our six-architecture study measures the error terms entering our approximate WSE bound.
We first note that finite measurements alone cannot certify every quantity the reviewer requests, as sampling can miss a smaller accurate network or different internal computations with almost identical outputs. Certifying these worst-case quantities therefore requires an additional geometric argument, which our ReLU example supplies (lines 520-525; Appendix B.3.4). Our ReLU networks in Fig. 7b fit functions with one, two or four changes in slope. If a network has no change in slope near one required by the function, its prediction loss cannot fall below a threshold we calculate analytically from our Theorem 2 zippering theorem (Appendix C.2, Proposition C.2.1, Eq. 12). This allows our geometric proof to cover all allowed weight choices: any two networks each achieving mean squared error at most 10-6 must match in at least 12.5%, 25% or 50% of their units, respectively. All 67 pairs meeting the proved accuracy condition satisfy this bound, as we note in lines 500-502 in Section 5.4.
Furthermore, across AlexNet, ResNet-18/34/50, ViT-B/16 and Pythia-160M, we also measure held-out affine-prediction errors and their amplification by outgoing weights—the quantities entering our approximate WSE bound (main-text Theorem 4; GELU extension in Corollary 16). Affine comparisons reduce held-out squared error by 79.9-93.5% relative to comparing coordinates directly (Section 5.5, lines 483-519).
W6: (e) Why ViT gates are privileged. Our new GELU weak-strong equivalence theorem, which was the theoretical result whose absence you identified, establishes when preserving that computation forces individual gates to match (App. Theorem 21; approximate version in Corollary 16). Our new Fig. 6a directly supports this prediction, by testing whether the learned gate coordinates recur more consistently than alternative coordinates carrying the same information. Keeping one model’s learned gates fixed, we find closer individual counterparts among another model’s learned gates than among random orthogonal mixtures of those gates on held-out ImageNet images, with a positive average advantage in every block. Because these mixtures preserve the layer’s information, this advantage supports the learned coordinates themselves being privileged for individual correspondence across models. Fig. 6b further shows a larger matching advantage for gates with stronger measured nonlinear contributions, reinforcing the evidence that individual learned ViT gates are privileged.
W7: Clarity & length. We have substantially rewritten our ten-page main text: we define our comparisons in plain language (lines 46-53), explain the mechanism through a worked example (lines 189-211), and state our main theorems and error bounds in words. We now explicitly separate visibility, minimality, recovery and regularity, distinguishing assumptions from proved recovery results (lines 153-177). We also state the attention-head assumptions, including no-splitting, in the main text (lines 243-247).
Furthermore, the supplement is now organized into figures (A), experimental methods (B) and theory (C), providing direct routes to the evidence and proofs relevant to each claim. To help readers check the theory without searching through the full supplement, our new reader’s guide connects each main claim to its formal statement, assumptions and proof (Appendix C.1), with assumptions collected in C.1.2 and full bounds in C.1.3.
Q1: Used GELU units & nonlinear responses. A used GELU unit makes a distinct nonlinear contribution to the output; duplicate and cancelling contributions are removed before counting. We explain these conditions in the main text (lines 153-158), with the GELU-specific definition in Definition 37 (App. C.8.1).
On held-out ImageNet images, 14.88% of ViT MLP [CLS] token preactivations fall in the GELU’s nonlinear regime (reported in App. B.2.3, line 1485). We note, however, that this fraction does not consistently predict the per-block alignment in Fig. 4b because inputs can occupy only a narrow range where GELU behaves almost linearly. We therefore also measure how poorly a straight line predicts each gate’s responses in Fig. 6b (methods described in Appendix B.2). We find that the gates with the strongest nonlinear contributions have a larger matching advantage over random mixtures than those with the weakest contributions, reinforcing the evidence for privileged learned gates.
Q2: Bound quantities and non-vacuity. As detailed in W5, we provide a proved positive numerical matching guarantee in our ReLU example (Fig. 7b), and empirically validate the affine agreement underlying our approximate WSE theorem across six architectures (Theorem 4; Section 5.5). As we explain in W5 (and in revised Section 5.5 lines 520-525), certifying the remaining quantities in general requires geometric arguments beyond finite measurements, which motivates our new ReLU example in Fig. 7b to begin with, since we can supply the necessary geometric argument in this setting to demonstrate non-vacuity of our bounds.
Q3: Input dimension. A lower-dimensional affine hull can hide nonlinear responses or make axes indistinguishable, but it does not bound the number of identifiable axes because even one input dimension can distinguish many ReLU units by where they change slope, as we explain in App. C.1.4 (lines 2138-2144).
Q4: Do weakly nonlinear gates still outperform mixtures? Yes. We show in Fig. 6b that even among the 20% of gates with the weakest measured nonlinear contributions, learned gates match better than random mixtures on average, with the 95% confidence interval for this advantage entirely above zero (exact averages and confidence intervals reported in App. B.2.3, lines 1500-1503). This supports our prediction of privileged individual MLP gates in trained ViTs, even in the group with the weakest measured nonlinear contributions.
Thank you. In the intervening months since our initial submission, we have established zippering and weak-strong equivalence (WSE) for GELU (Appendix C.8). With our new GELU theory in hand, we also empirically show GELU gates behave nonlinearly on task inputs and match better than random mixtures (Fig. 6a,b). We validate our proved bound on how many ReLU units must match (Fig. 7a), and predict trained models' layer responses from one another across six architectures.
W1: Problem exposition. We agree that mixing features generally changes their nonlinear responses, so preserving the computation can force individual units to match. We now explain this through a worked example in Section 3.2 (lines 189-211) and address your specific concerns (a)-(e) below:
W2: (a) Inputs must expose GELU’s nonlinearity. We now explicitly define counted GELU units as distinct nonlinear contributions with varying inputs and nonzero outgoing weights, after removing duplicate and cancelling contributions (Definition 37; App. C.8.1, lines 7040-7049). Our new analysis in Fig. 6b (and discussed in Section 5.3) directly tests nonlinear exposure in trained ViTs: namely, straight-line fits leave measurable error in GELU responses on held-out ImageNet images, supporting the requirement that the experimental inputs expose nonlinear behavior (methods in App. B.2).
W3: (b) GELU theory established. We now prove GELU unit matching and zippering for exact and approximate agreement, including a new argument ruling out the alternative inverse branch as another network solution (Appendix C.8). Our deep-zippering results cover feedforward networks with the same architecture, independently variable weights and at least two outputs (Appendix C.8), and sequential GELU transformers under the stated prefix and attention conditions (Appendix C.9). We also explicitly identify the activation used in ViT and Pythia as exact GELU, rather than its tanh approximation (main text, lines 251-253; implementation details in Appendix B.1). We have Lean-verified our main exact/soft zippering and weak-strong equivalence (WSE) theorems across GELU, Softplus and ReLU, and have attached the Lean certificates as a zip file under Supplementary Material, with App. C.1.6 pg. 41 listing the verified theorems. In doing so, Lean verification has also allowed us to simplify some of our lemmas and arguments.
W4: (c) Gate evidence and uncertainty. In addition to providing a novel GELU theory in revision for our WSE and zippering theorems (see (b) above), we also add more empirical evidence for transformers:
Small gaps & “learned gates outperform mixtures.” We now describe this advantage layer by layer, explicitly noting the mixture advantage at AlexNet conv1 and early ViT blocks (main text, lines 394-398; Fig. 4). Note that weaker head advantages in later blocks do not contradict our theory, since our attention-head theorem bounds the fraction of heads that must match under its stated assumptions, rather than the average advantage over random mixtures (App. Theorem 12).
Error bars & variation across seeds. We now plot all three ViT model pairs separately in updated Fig. 4b and new Fig. 6a, with 95% confidence intervals computed by resampling the 5,000 held-out ImageNet images (methods in Appendix B.2). The separate curves show variation among the available trained models, and the intervals quantify uncertainty from held-out images with those models fixed.
Block-by-block results & an aggregate statistic. Updated Fig. 4b reports effect sizes and confidence intervals for every block, showing mixture advantages in blocks 1-3 and showing learned-gate advantages in blocks 5-12, with all eight 95% confidence intervals above zero.
Direct evidence for individual gate correspondence. Our new Fig. 6a keeps one model’s learned gates fixed and mixes only the other model’s gates, so both comparisons test counterparts of the same learned gates. We find that learned gates have closer individual counterparts than random mixtures on held-out ImageNet images, supporting the individual-gate correspondence predicted by our GELU weak-strong equivalence theorem (Appendix Theorem 21; approximate version in Corollary 16).