acceptodds
Under review as a conference paper at ICLR 2027

Convergent Evolution of Neural Networks: Zippering and Weak-Strong Equivalence

Abstract

Why and when should independently trained neural networks develop the same internal mechanisms? We develop a theory of this convergent evolution through *identifiability*: what a network's end-to-end function determines about its internal computation. Our *zippering* theorems establish when agreement at the end forces corresponding representations throughout a nonlinear network, up to linear transformations. Our *weak-strong equivalence* theorems then show when this correspondence forces individual channels and gates to recur. Tasks that require a greater fraction of these components force stronger convergence. We prove exact and approximate versions of both results. Most strikingly, when our theory is applied to modern transformer architectures, it predicts *privileged attention heads*: individual head computations that recur across models. We find privileged heads and gates in both vision and language, ordered alignment across unequal depths, and stronger recurrence of more-used components. Task constraints can therefore force a shared computational hierarchy and reproducible mechanisms across models trained on the same tasks. These results thus give mechanistic interpretability stable structures to explain and AI safety research recurring targets for intervention in models spanning different tasks, modalities, and architectures.

Primary area:
interpretability and explainable AI
Keywords:
deep learning theory, neural network identifiability, convergent evolution, representational alignment, mechanistic interpretability, privileged axes, attention heads, NeuroAI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YEDmOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The authors derive an extensive theory of when independently trained neural networks must learn identical internal computations. They define 2 distinct alignment paradigms -- weak alignment (one layer's preactivations are an affine function of the other's), and strong alignment (individual units across networks match one-to-one up to a permutation/rescaling). They prove several theorems to this effect:

  • Zippering theorems (Theorems 1 and 2): for ReLU or Softplus networks satisfying minimality and regularity conditions, agreement at a terminal layer forces strong alignment at every earlier layer.
  • Weak–strong equivalence theorems (Theorems 3 and 4): weak alignment at the layers immediately before and after a coordinatewise nonlinearity forces every task-used unit to match. The fraction of matched units is lower-bounded by the minimal number of nonlinear units any accurate solution requires, relative to the layer width.
  • Transformers: they extend their theory to predict privileged MLP gates and privileged whole attention heads. They define this through the gauge-invariant quantities Γh=WQWK⊤/dh\Gamma_h = W_Q W_K^\top/\sqrt{d_h} and Ch=WVWOC_h = W_V W_O.
  • Empirical evidence: in AlexNet, ResNets, ViTs and Pythia, they report ordered layerwise alignment, drops in alignment across nonlinearities, learned units and heads matching better than random mixtures, and stronger recurrence of more-used components as predicted by their proposed theories.
Critical strengths
  1. The paper tackles a long-standing problem of why convergent learning happens, rather than simply documenting it. They answer this through the lens of identifiability (i.e., what end-to-end function determines about internal computations), which is an excellent way to tackle it.

  2. Weak vs. strong alignment is a clean distinction, and clearly motivated. The intuition for weak–strong equivalence is also elegant. This is done through the lens of two different coordinatewise nonlinearities -- a ReLU kink or a Softplus curvature ridge -- both of which determine a network's native axes. Hence, mixing these axes (e.g., via a dense affine map) breaks downstream agreement.

  3. The authors handle parameter symmetries in transformers carefully. They compare the heads through the gauge-invariant quantities (Γh,Ch)(\Gamma_h, C_h), rather than individual coordinates defined on Q/K/V/OQ/K/V/O. They also explicitly mention that the residual stream has no privileged basis.

  4. The empirical controls are well designed:

    1. Random orthogonal mixtures preserving the linear span of the units
    2. Sum-preserving head mixtures
    3. Shuffled-input nulls
    4. Cycle consistency across three models (A→B→C→A)(A \rightarrow B \rightarrow C \rightarrow A)
    5. Causal head ablations in Pythia
  5. Formalizing the otherwise informal (and philosophical) contravariance principle (harder tasks admit fewer solutions) to establish a lower bound mℓ(ε)/dℓAm_\ell(\varepsilon)/d_\ell^A on the fraction of layer-ℓ\ell units that must align strongly across networks is a clean formalism.

Critical weaknesses

My main concern is the gap between the proposed theory's conditions and the presented empirical evidence, in particular for transformers. Please bear with me -- this is slightly long, and a sketch rather than a complete proof. I have tried to use the paper's notation in as many places as possible.

Problem exposition. The mechanism by which a coordinatewise nonlinearity selects native axes can be formalized as follows. Consider a layer z↦ϕ(z)z \mapsto \phi(z) followed by an affine readout W2h+b2W_2 h + b_2 with W2W_2 of full column rank. An orthogonal reparameterization W↦QWW \mapsto QW, b↦Qbb \mapsto Qb, W2↦W2Q⊤W_2 \mapsto W_2 Q^\top (with b2b_2 adjusted) preserves the function for all zz iff ϕ(Qz)=Qϕ(z)+c\phi(Qz) = Q\phi(z) + c for a constant cc that is absorbed by the next bias term. Differentiating yields diag(ϕ′(Qz)) Q=Q diag(ϕ′(z))\mathrm{diag}(\phi'(Qz))\,Q = Q\,\mathrm{diag}(\phi'(z)). Right-multiplying by Q⊤Q^\top then gives Q diag(ϕ′(z)) Q⊤=diag(ϕ′(Qz))Q\,\mathrm{diag}(\phi'(z))\,Q^\top = \mathrm{diag}(\phi'(Qz)), which is clearly diagonal for every zz. If ϕ′\phi' is non-constant, choosing zz with zk=tz_k = t and zj=sz_j = s (j≠kj \neq k), ϕ′(t)≠ϕ′(s)\phi'(t) \neq \phi'(s), gives us Q diag(ϕ′(z)) Q⊤=ϕ′(s) I+(ϕ′(t)−ϕ′(s)) qkqk⊤.Q\,\mathrm{diag}(\phi'(z))\,Q^\top = \phi'(s)\,I + \big(\phi'(t) - \phi'(s)\big)\, q_k q_k^\top. This is diagonal only if each column qkq_k is a signed standard basis vector. Moreover, the constant offset then rules out sign flips unless ϕ(t)+ϕ(−t)\phi(t) + \phi(-t) is constant (as for tanh). So, for ReLU, Softplus, and GELU the only orthogonal symmetries of a plain feedforward layer are permutations (for ReLU, the argument applies at almost every zz).

(a) Symmetry breaking only holds where the data expose the nonlinearity. The argument presented above requires the realized preactivations to visit points where ϕ′\phi' differs. If a group of units, however, remains in a regime where ϕ\phi is approximately linear with a common slope (e.g., always-active ReLU units, or GELU units with z≫0z \gg 0, where GELU(z)≈z\mathrm{GELU}(z) \approx z), then small rotations among those units become approximate symmetries of the layer. I should caveat this by saying that whether they would remain approximate symmetries of the whole network also depends on the downstream weights. Units in different regimes (e.g., z≫0z \gg 0 versus z≪0z \ll 0, where GELU(z)≈0\mathrm{GELU}(z) \approx 0) cannot be mixed in this way. Put differently, if the symmetry is only required on the region UU visited by the data, differentiating the diagonal condition along eke_k gives us ϕ′′(zk) qkqk⊤\phi''(z_k)\, q_k q_k^\top. Hence, axis kk is only determined if ϕ′′(zk)≠0\phi''(z_k) \neq 0 somewhere in UU, i.e., if the data reach the curved part of ϕ\phi for that unit. Similarly, for nonnegative activations, nonnegativity only forces a permutation when the realized activations span the orthant. In other words, if they lie in a restricted cone, e.g., M={λ(1,1)/2:λ≥0}⊂R2\mathcal{M} = \{\lambda (1,1)/\sqrt{2} : \lambda \ge 0\} \subset \mathbb{R}^2, any rotation by ∣θ∣<π/4|\theta| < \pi/4 preserves nonnegativity without being a permutation. The paper's notion of a used axis addresses this for ReLU (the task must cross the kink trace) and Softplus, but there is no corresponding definition for GELU, which lays the ground for my main issue in the next point.

(b) The transformer predictions rely on an unproven extension. The exact theorems cover ReLU and Softplus, but all transformer experiments use GELU. Appendix D.2.4 outlines a proof strategy for a GELU weak–strong theorem by analogy with Softplus, but states that deep zippering for GELU requires a "genuinely new" argument, and that exact GELU is not globally injective (two inverse branches). The predictions of privileged MLP gates in ViTs and Pythia therefore rest on a proposed rather than proven result. This should be stated clearly in the main text, together with which GELU variant (exact or tanh approximation) each model uses, because the appendix notes these must be analyzed separately.

(c) The empirical support for privileged ViT gates is limited and insufficiently quantified. Given (b) and (c), the experiments carry bulk of the argument for transformers. However, in Fig. 4b learned ViT gates match only marginally better than random orthogonal mixtures in most blocks, and the mixtures are higher at block 1. Similarly, random mixtures outperform learned channels at AlexNet conv1 (Fig. 4a), and the gaps for Pythia MLP gates (Fig. 4d) are small. Yet, the text states that ViT gates "match more closely than random orthogonal mixtures" (L437). These comparisons rest on three ViT seeds and are reported without error bars or significance tests, so it is not possible to tell whether the gaps are distinguishable from seed-to-seed variability. The claim should be assessed block by block, or supported by an explicitly defined aggregate statistic. The evidence for privileged heads (Fig. 4c,e) is stronger, but the gap disappears in later blocks.

(d) Are the soft bounds non-vacuous? Exact weak alignment never holds between independently trained networks, so the practically relevant results are the soft theorems. Theorem 4 gives AxisAlignℓθ(A,B;Eℓ)≥mℓ(ε)dℓA−(∥Wℓ+1B∥op ωℓ+ωℓ+1)2κK(θ)2 dℓA\mathrm{AxisAlign}^\theta_\ell(A,B;E_\ell) \ge \frac{m_\ell(\varepsilon)}{d^A_\ell} - \frac{\left(\Vert W^B_{\ell+1}\Vert_{\mathrm{op}}\, \omega_\ell + \omega_{\ell+1}\right)^2}{\kappa_K(\theta)^2 \, d^A_\ell} and Theorem 2 depends on ∏ℓLℓ2\prod_\ell L_\ell^2 and the margin cc. None of these quantities are estimated for the networks studied. If the error term dominates the first term, I imagine that the bound is vacuous. For this reason, it is unclear to me whether either guarantee says anything about the networks in the experiments. Also, the zippering results are framed around networks that agree at a terminal layer (L59–61), whereas independently trained networks differ in accuracy and final representations across seeds, so I imagine the soft versions are the ones that matter in practice.

(e) Relation to the interpretation of Kapoor et al. (2025) in Remark 6. Remark 6 (Supp. Sec. C.2.4) argues that the weak evidence for privileged axes in ViTs reported by Kapoor et al. (2025) is not evidence against the theory, because their tests were applied to the [CLS] token, i.e., to the residual stream, where the theory makes no native-axis claim. I agree with this part of the argument, that the residual stream has no coordinatewise nonlinearity of its own, so a null result there is expected.

That said, Remark 6 goes further and implies that the theory's prediction of privileged MLP gates holds in ViTs. This is a separate claim that, as far as I can tell, is not established by either work. Kapoor et al. analyze the [CLS] token of GELU-based ViTs but not the MLP-internal GELU units themselves, and the authors' test of them (Fig. 4b) shows learned ViT gates matching only marginally better than random orthogonal mixtures, and worse at block 1. As discussed in the problem exposition and (a), GELU breaks exact rotational symmetry, but units that remain in the same approximately linear regime retain approximate rotational symmetries among themselves. ViT gates may therefore be privileged in principle but only weakly identified in practice. Combined with the fact that the GELU theory is proposed rather than proven (b), I do not think the current results establish that ViT MLP gates are privileged. For this reason, I would encourage the authors to temper the claim in Remark 6 and the main text accordingly.

Clarity and length. In general, I also found the notation and writing very obtuse and non-standard, which makes the paper hard to follow, despite the core ideas being interesting. Most importantly, most of the assumptions behind the theoretical results and their corresponding proofs are in ≈100\approx 100 pages of supplementary material. It is not feasible to verify all of these within the review period, which is why my review has focused on what I understand to be the key empirical and theoretical contribution of the work (i.e., the presence of a privileged basis in transformer architectures). That said, I am unable to make a confident judgement about the soundness of the remainder of the theoretical contributions, along with the assumptions that underlie them. I think the main text would benefit from stating the key assumptions explicitly, and perhaps the work may be better suited to a venue where the proofs can be reviewed in full, to do full justice to what is clearly a substantial amount of work.

Questions for authors
  1. How is a used GELU unit defined? Could the authors report what fraction of ViT MLP preactivations fall in the nonlinear regime of GELU, and whether this predicts the per-block alignment in Fig. 4b?
  2. Would it be possible to estimate ωℓ\omega_\ell, ∥Wℓ+1B∥op\Vert W^B_{\ell+1}\Vert_{\mathrm{op}}, κK(θ)\kappa_K(\theta), LℓL_\ell and mℓ(ε)m_\ell(\varepsilon), or bounds on them? Perhaps just for one model family, and report whether the resulting guarantees in Theorems 2 and 4 are non-vacuous (see weakness (d))?
  3. If, and how does the dimensions of the affine hull of the task patch affect which axes can be identified in representations?
  4. Remark 6 argues that the weak axis alignment in ViTs reported by Kapoor et al. reflects tests on the [CLS] token / residual stream, and states that alignment should instead appear at the MLP gates. Could the authors quantify how strongly ViT MLP gates are identified in practice, e.g., by relating per-unit alignment to the fraction of each unit's preactivations in the nonlinear regime of GELU? If weakly nonlinear units align no better than random mixtures, this would suggest that approximate rotational symmetries persist at GELU gates.
6 comments

Thank you. In the intervening months since our initial submission, we have established zippering and weak-strong equivalence (WSE) for GELU (Appendix C.8). With our new GELU theory in hand, we also empirically show GELU gates behave nonlinearly on task inputs and match better than random mixtures (Fig. 6a,b). We validate our proved bound on how many ReLU units must match (Fig. 7a), and predict trained models' layer responses from one another across six architectures.

W1: Problem exposition. We agree that mixing features generally changes their nonlinear responses, so preserving the computation can force individual units to match. We now explain this through a worked example in Section 3.2 (lines 189-211) and address your specific concerns (a)-(e) below:

W2: (a) Inputs must expose GELU’s nonlinearity. We now explicitly define counted GELU units as distinct nonlinear contributions with varying inputs and nonzero outgoing weights, after removing duplicate and cancelling contributions (Definition 37; App. C.8.1, lines 7040-7049). Our new analysis in Fig. 6b (and discussed in Section 5.3) directly tests nonlinear exposure in trained ViTs: namely, straight-line fits leave measurable error in GELU responses on held-out ImageNet images, supporting the requirement that the experimental inputs expose nonlinear behavior (methods in App. B.2).

W3: (b) GELU theory established. We now prove GELU unit matching and zippering for exact and approximate agreement, including a new argument ruling out the alternative inverse branch as another network solution (Appendix C.8). Our deep-zippering results cover feedforward networks with the same architecture, independently variable weights and at least two outputs (Appendix C.8), and sequential GELU transformers under the stated prefix and attention conditions (Appendix C.9). We also explicitly identify the activation used in ViT and Pythia as exact GELU, rather than its tanh approximation (main text, lines 251-253; implementation details in Appendix B.1). We have Lean-verified our main exact/soft zippering and weak-strong equivalence (WSE) theorems across GELU, Softplus and ReLU, and have attached the Lean certificates as a zip file under Supplementary Material, with App. C.1.6 pg. 41 listing the verified theorems. In doing so, Lean verification has also allowed us to simplify some of our lemmas and arguments.

W4: (c) Gate evidence and uncertainty. In addition to providing a novel GELU theory in revision for our WSE and zippering theorems (see (b) above), we also add more empirical evidence for transformers:

Small gaps & “learned gates outperform mixtures.” We now describe this advantage layer by layer, explicitly noting the mixture advantage at AlexNet conv1 and early ViT blocks (main text, lines 394-398; Fig. 4). Note that weaker head advantages in later blocks do not contradict our theory, since our attention-head theorem bounds the fraction of heads that must match under its stated assumptions, rather than the average advantage over random mixtures (App. Theorem 12).

Error bars & variation across seeds. We now plot all three ViT model pairs separately in updated Fig. 4b and new Fig. 6a, with 95% confidence intervals computed by resampling the 5,000 held-out ImageNet images (methods in Appendix B.2). The separate curves show variation among the available trained models, and the intervals quantify uncertainty from held-out images with those models fixed.

Block-by-block results & an aggregate statistic. Updated Fig. 4b reports effect sizes and confidence intervals for every block, showing mixture advantages in blocks 1-3 and showing learned-gate advantages in blocks 5-12, with all eight 95% confidence intervals above zero.

Direct evidence for individual gate correspondence. Our new Fig. 6a keeps one model’s learned gates fixed and mixes only the other model’s gates, so both comparisons test counterparts of the same learned gates. We find that learned gates have closer individual counterparts than random mixtures on held-out ImageNet images, supporting the individual-gate correspondence predicted by our GELU weak-strong equivalence theorem (Appendix Theorem 21; approximate version in Corollary 16).

W5/Q2: (d) Non-vacuous soft bounds. We now provide a proved positive matching guarantee in a ReLU task (Fig. 7b; Appendix C.2, Proposition and Corollary C.2.1) and measure the errors entering our bounds across six architectures (Section 5.5, lines 483-519). These studies support our approximate zippering and WSE predictions (main-text Theorems 2 and 4): our ReLU example provides a fully proved numerical matching guarantee, while our six-architecture study measures the error terms entering our approximate WSE bound.

We first note that finite measurements alone cannot certify every quantity the reviewer requests, as sampling can miss a smaller accurate network or different internal computations with almost identical outputs. Certifying these worst-case quantities therefore requires an additional geometric argument, which our ReLU example supplies (lines 520-525; Appendix B.3.4). Our ReLU networks in Fig. 7b fit functions with one, two or four changes in slope. If a network has no change in slope near one required by the function, its prediction loss cannot fall below a threshold we calculate analytically from our Theorem 2 zippering theorem (Appendix C.2, Proposition C.2.1, Eq. 12). This allows our geometric proof to cover all allowed weight choices: any two networks each achieving mean squared error at most 10-6 must match in at least 12.5%, 25% or 50% of their units, respectively. All 67 pairs meeting the proved accuracy condition satisfy this bound, as we note in lines 500-502 in Section 5.4.

Furthermore, across AlexNet, ResNet-18/34/50, ViT-B/16 and Pythia-160M, we also measure held-out affine-prediction errors and their amplification by outgoing weights—the quantities entering our approximate WSE bound (main-text Theorem 4; GELU extension in Corollary 16). Affine comparisons reduce held-out squared error by 79.9-93.5% relative to comparing coordinates directly (Section 5.5, lines 483-519).

W6: (e) Why ViT gates are privileged. Our new GELU weak-strong equivalence theorem, which was the theoretical result whose absence you identified, establishes when preserving that computation forces individual gates to match (App. Theorem 21; approximate version in Corollary 16). Our new Fig. 6a directly supports this prediction, by testing whether the learned gate coordinates recur more consistently than alternative coordinates carrying the same information. Keeping one model’s learned gates fixed, we find closer individual counterparts among another model’s learned gates than among random orthogonal mixtures of those gates on held-out ImageNet images, with a positive average advantage in every block. Because these mixtures preserve the layer’s information, this advantage supports the learned coordinates themselves being privileged for individual correspondence across models. Fig. 6b further shows a larger matching advantage for gates with stronger measured nonlinear contributions, reinforcing the evidence that individual learned ViT gates are privileged.

W7: Clarity & length. We have substantially rewritten our ten-page main text: we define our comparisons in plain language (lines 46-53), explain the mechanism through a worked example (lines 189-211), and state our main theorems and error bounds in words. We now explicitly separate visibility, minimality, recovery and regularity, distinguishing assumptions from proved recovery results (lines 153-177). We also state the attention-head assumptions, including no-splitting, in the main text (lines 243-247).

Furthermore, the supplement is now organized into figures (A), experimental methods (B) and theory (C), providing direct routes to the evidence and proofs relevant to each claim. To help readers check the theory without searching through the full supplement, our new reader’s guide connects each main claim to its formal statement, assumptions and proof (Appendix C.1), with assumptions collected in C.1.2 and full bounds in C.1.3.

Q1: Used GELU units & nonlinear responses. A used GELU unit makes a distinct nonlinear contribution to the output; duplicate and cancelling contributions are removed before counting. We explain these conditions in the main text (lines 153-158), with the GELU-specific definition in Definition 37 (App. C.8.1).

On held-out ImageNet images, 14.88% of ViT MLP [CLS] token preactivations fall in the GELU’s nonlinear regime (reported in App. B.2.3, line 1485). We note, however, that this fraction does not consistently predict the per-block alignment in Fig. 4b because inputs can occupy only a narrow range where GELU behaves almost linearly. We therefore also measure how poorly a straight line predicts each gate’s responses in Fig. 6b (methods described in Appendix B.2). We find that the gates with the strongest nonlinear contributions have a larger matching advantage over random mixtures than those with the weakest contributions, reinforcing the evidence for privileged learned gates.

Q2: Bound quantities and non-vacuity. As detailed in W5, we provide a proved positive numerical matching guarantee in our ReLU example (Fig. 7b), and empirically validate the affine agreement underlying our approximate WSE theorem across six architectures (Theorem 4; Section 5.5). As we explain in W5 (and in revised Section 5.5 lines 520-525), certifying the remaining quantities in general requires geometric arguments beyond finite measurements, which motivates our new ReLU example in Fig. 7b to begin with, since we can supply the necessary geometric argument in this setting to demonstrate non-vacuity of our bounds.

Q3: Input dimension. A lower-dimensional affine hull can hide nonlinear responses or make axes indistinguishable, but it does not bound the number of identifiable axes because even one input dimension can distinguish many ReLU units by where they change slope, as we explain in App. C.1.4 (lines 2138-2144).

Q4: Do weakly nonlinear gates still outperform mixtures? Yes. We show in Fig. 6b that even among the 20% of gates with the weakest measured nonlinear contributions, learned gates match better than random mixtures on average, with the 95% confidence interval for this advantage entirely above zero (exact averages and confidence intervals reported in App. B.2.3, lines 1500-1503). This supports our prediction of privileged individual MLP gates in trained ViTs, even in the group with the weakest measured nonlinear contributions.

Reviewer_fbPmOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper uses ideas from identifiability ("weak-strong equivalence") to propose a recursive argument for layerwise and neuron-wise agreement between independently trained networks ("zippering"). They formulate their results in terms of theoretical bounds. Empirically, they show "recurrence" of individual neurons/heads in CNN and Transformers based networks.

Critical strengths

The claim regarding privileged MLP gates and attention heads in Transformers is interesting and useful. They show empirical evidence of this correspondence between networks, albeit not to the extent that validates their theoretical bounds. Empirical comparisons with random mixing are instructive and clean.

Critical weaknesses
  • The presentation of the paper requires substantial improvement. The central claim(s) are obscured by excessive terminology and convoluted notation in many places. Also, the broader claims made in the conclusion section (e.g. task difficulty and inevitability of neuroAI convergence) are too strong and not justified by the evidence.
  • The connection between theory and experiments is weak. For example, fig.2 is somewhat orthogonal to their claims -- it does not demonstrate that nonlinearities lead to neuron-wise correspondence. Their experiments do not directly support their quantitative predictions, such as the proposed lower bounds. The alignment results in fig.3 are well-known and provide little evidence for the quantitative claims of the paper. The claim about task difficulty does not receive convincing empirical justification in the paper. The theory (for both neurons and heads) makes numerous assumptions, however, how well they hold in experiments is not established.
  • The theoretical novelty of "weak-strong equivalence" is unclear beyond the existing literature in identifiability. (If this is inaccurate, I ask the authors to clarify how.)
Questions for authors
  • Can the authors clarify the novelty of their "weak-strong equivalence" (and "zippering") beyond the existing work on identifiability?
  • Is there empirical evidence that the assumptions that go into the theorem (identifiability, minimality, no-splitting) are justified?
  • Can the authors clarify how their experiments test/establish the proposed quantitative relationship between task-difficulty, minimum required nonlinear units and representation alignment?
  • Empirical work on platonic representation hypothesis (especially between two language models) shows that representation similarity (measured by CKA, mKNN etc.) peaks in the middle layers. Can the proposed framework give insight into this non-monotonicity?
5 comments

Thank you for highlighting MLP gates and attention heads. We clarify our identifiability and NeuroAI advances, directly test our quantitative predictions about how many units networks must share (Fig. 7a,b), and empirically validate our theory’s predictions of shared internal computations in trained models across six architectures (Section 5.5).

W1: Clearer presentation and the scope of our conclusions. We have substantially rewritten our ten-page main text in the uploaded revision to explain our comparisons upfront in plain terms (lines 46-53), introduce the main ideas through worked examples, and state our theorems and error bounds in words (lines 189-234). We also summarize our contributions in a short list (lines 58-74). Appendix C.1 provides an overall guide connecting claims to assumptions and proofs, with full formulas collected in C.1.3.

Regarding our task-difficulty claim, it has a precise meaning: at a fixed network size, requiring more nonlinear units to achieve a given accuracy raises the minimum guaranteed fraction of matching units under our stated conditions (lines 206-215).

The relation to NeuroAI is also not loosely inspired; rather, our work is deeply motivated by explaining key empirical results in this field, for which we note that there has been no prior theory. Specifically, our zippering theorems (Theorems 1-2, pg. 4) are a formalization of Cao & Yamins’ (2024) informal Contravariance Principle turns a qualitative idea of challenging tasks converging onto brain-aligned solutions into quantitative predictions of shared units and hierarchies, providing a theoretical mechanism for why task-trained models can predict brain responses. We also advance NeuroAI methodology: As we note in lines 100-106 (and expand in depth in App. C.7), RSA/CKA scores used in model-brain comparisons are sensitive to when features are rescaled or duplicated without changing the computation. Our analysis separates these effects from recording noise and uneven neuron or stimulus sampling, explaining how task-irrelevant differences can produce conflicting similarity scores. This helps interpret the prior empirical discrepancies observed by Soni et al. (2024) and Bo et al. (2024), and guides practical brain-model comparisons of what metrics to choose going forward, e.g. ridge regression and our normalized RSA/CKA measures (RSA: Definition 33, p. 112, lines 5995-6013; CKA: Definition 36, p. 124, lines 6655-6661).

W2: Connection between theory and experiments. In the revised manuscript, we clarify the connection between our theory and our experiments (and run new experiments), specifically to your questions:

Nonlinearities and neuron-wise correspondence. Figure 2 addresses the mechanism behind weak-strong equivalence (WSE) in Thms. 3-4: nonlinearities constrain which mixtures of neurons’ responses can preserve agreement. Individual correspondence is tested directly in Figure 4 and our new Figure 6a, where learned ViT gates match better than random mixtures of those same gates in every block on held-out images. Figure 6b connects this correspondence to nonlinear behavior: the group with the strongest measured nonlinear contribution has a larger matching advantage than the weakest group (lines 415-431 & Appendix B.2).

Quantitative lower bounds. We now directly test zippering’s (Thms. 1-2) central prediction—that agreement at the output can force matching internal units—in an explicit ReLU example with a proved numerical bound (Fig. 7a-b; Appendix C.2). All 1,230 checkpoints from 30 runs satisfy the bound proved for this class (Proposition C.2.1, Eq. 11), and all 20 runs reaching the error target recover every required unit (Fig. 7a). Figure 7b tests the corresponding guarantee between independently trained networks (Corollary C.2.1; Appendix C.2).

Layer alignment in Figure 3. Figure 3’s ordered correspondences across depths and architectures address our shared-hierarchy claim in our zippering theorems. We strengthen the quantitative evidence with a new six-architecture study (AlexNet, ResNet-18, ResNet-34, ResNet-50, ViT-B/16, and Pythia-160M) measuring the prediction errors entering our deep-network bounds from affine transforms. Affine comparisons—weighted sums plus offsets—reduce held-out squared error by 79.9-93.5% relative to comparing coordinates directly. We also measure how outgoing weights amplify those errors, connecting the observed agreement to the quantities in our bounds (Section 5.5, lines 483-525; Appendix B.3.2-B.3.4).

Task difficulty. To directly test our claims about task difficulty, we now vary the minimum number of units needed for accuracy, established in Proposition C.2.1, while keeping network size fixed. The minimum guaranteed matching fraction rises accordingly, and the measured matches satisfy these guarantees. This directly tests our prediction about required computation, rather than relying on an informal judgment of task difficulty (Fig. 7a-b; further numbers in our response in Q3).

Assumptions in trained networks. We test our visibility assumption by measuring whether task inputs expose GELU’s nonlinear behavior (Fig. 6). For attention heads, Figure 10a-b supports our WSE theorem’s no-splitting assumption: adding a second head improves predictions less for learned heads than for random mixtures, supporting individual counterparts rather than combinations of heads (Appendix Theorem 12). Supporting our WSE prediction of corresponding whole-head computations (Appendix Theorem 12), replacing matched heads by their mean outputs produces more similar changes in next-token scores than replacing randomly matched heads (Fig. 10c). Our response in Q2 gives further details of these experimental tests.

W3/Q1: Novelty beyond prior identifiability work. In revised Section 2, we explain how our bounds quantitatively predict how many units must match even when networks have extra units and make somewhat different predictions, without requiring the entire network to be uniquely determined (unlike prior identifiability work). Specifically, our weak-strong equivalence (WSE) theorems in Thms. 3-4 show when affine agreement at two adjacent layers forces individual units to match, and our zippering theorems (Thms. 1-2) show when output agreement can be traced backward through earlier layers.

By comparison, Phuong & Lampert (2020) construct identifiable ReLU networks with one output and layers that never widen. Bona-Pellissier et al. (2023) likewise require non-widening fully connected layers and conditions ensuring that each unit’s change in slope remains detectable in the network’s output. These results establish full-network recovery within their stated settings; however, our quantitative guarantees identify shared features even when other units can differ. Moreover, our ReLU and Softplus theory extends to unequal widths and depths, residual blocks and whole attention-head computations under our recovery conditions (lines 78-93; Appendix C.1.5). We additionally establish exact and approximate GELU matching and zippering for the stated classes of networks with the same architecture (Appendices C.8–C.9).

We have Lean-verified our main exact/soft zippering and weak-strong equivalence (WSE) theorems across GELU, Softplus and ReLU, and have attached the Lean certificates as a zip file under Supplementary Material, with App. C.1.6 pg. 41 listing the verified theorems. In doing so, Lean verification has also allowed us to simplify some of our lemmas and arguments.

Q2: Evidence for our assumptions. As detailed in our W2 response above and with the assumptions explained in Section 3 and Appendix C.1.2:

Visibility: We show that held-out ImageNet images expose GELU’s nonlinear behavior in ViT gates (Fig. 6b; Appendix B.2).

No-splitting: We find that adding a second head improves reconstruction less for learned heads than for random mixtures of those same heads, supporting individual head counterparts (Fig. 10a-b).

Minimality: We prove that every sufficiently accurate solution to our three ReLU tasks needs at least one, two or four units, respectively (Fig. 7a; Proposition C.2.1).

Identifiability: Our new GELU and Softplus proofs establish unit recovery from output agreement for almost every allowed choice of weights under our stated conditions (Appendix Theorems 22 and 10).

Q3: Task difficulty, required units and alignment.

Task difficulty: In a new experiment in Section 5.4 (Fig. 7a-b), we vary the function to be learned from one to two to four changes in slope, holding network size and required accuracy fixed.

Minimum required nonlinear units: We prove that these tasks require at least one, two or four units, respectively (Proposition C.2.1).

Representation alignment: This guarantees that at least 12.5%, 25% or 50% of units match between independently trained networks reaching the required accuracy, and all 67 network pairs meeting the proved accuracy condition satisfy their corresponding bound (Fig. 7b; Corollary C.2.1).

Q4: Why can similarity peak in middle layers? For CKA, our theory identifies a concrete mechanism: namely, that networks can compute the same function while duplicating shared features differently, changing which features dominate the similarity score (Appendix C.7.2.1, Proposition 22). If these differences are smaller in middle layers and larger later, CKA can peak in the middle even though shared computations persist. Our Pythia results in Fig. 9a,b in App. A, which we have added in revision, show that CKA differences can strongly depend on response normalization: normalizing token responses nearly removes the apparent CKA separation between model groups (which we derive in App. B.4.2).

Reviewer_nbY4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors study the identifiability of network parameters from the induced function. In their theoretical section, they study ReLU and Softplus networks, and use the geometric signatures of these nonlinear activation functions to identify underlying network parameters from a function's input/output behavior. They distinguish two kinds of layer alignment--weak alignment by an injective affine map, and strong alignment by a neuron-wise permutation matrix (with certain allowed coordinate-wise transformations). Their "zippering" theorems give layer conditions which allow weak alignment between the last layers of two networks to be extended backwards to deduce the strong alignment of earlier layers. Their "weak–strong equivalence" theorems suppose a commutative square whose horizontal maps are layers from two different networks and whose vertical maps are weak equivalences, and give conditions which force the input-side map to be a strong equivalence. Under the same hypotheses, they quantitatively lower-bound the fraction of strongly aligned axes by the minimum number of task-used nonlinear units required to achieve a specified task loss. They also generalize this paradigm to transformers, describing which aspects of the transformer architecture can serve as identifiable non-linear signatures. In their empirical section, they demonstrate several related alignment effects in real-world networks (convolutional networks, vision transformers, and language models), including hierarchical affine predictability, alignment across nonlinearities, component reoccurrence, and depth correspondence.

Critical strengths
  1. Their zippering and weak-strong equivalence results are strong and elegant, and give a useful general geometric framework for studying layer alignment and parameter identifiability. They make precise how nonlinearities constrain affine changes of representation, and the paper's systematic use of complementary weak and strong equivalences is a good organizing tool.
  2. Their extension to transformers is illuminating, including the observation that certain aspects of a transformer are identifiable, and others aren't. They provide empirical evidence that these identifiable structures in fact reoccur across trained networks.
Critical weaknesses

(Edited for clarity).

  1. Although I appreciate the identifiability results, what confused me throughout the paper, and in particular during the empirical results section, was an apparent conflation between identifying two networks' internal parameters given sufficient functional alignment (parameter identifiability), and explaining why two networks trained to similar performance on the same task should become sufficiently functionally aligned (functional identifiability). Specifically, the zippering and strong-weak equivalence results start from certain weak functional alignments--of terminal representations in the former case, and adjacent representations in the latter--and from that and certain assumptions, deduce strong alignment. However, convergence from training alone would then rest on the additional question, why should two networks trained with similar task performance have this functional alignment, on a sufficiently dense set of input points (beyond the training data)?
    In particular, the empirical section repeatedly describes the paper's theorems as making predictions about trained networks, without clearly explaining how the required hypotheses enter this interpretation. For example, §5.1 says, “Our zippering theorems predict that terminal agreement extends to earlier representations,” but I could not identify where the applicability of this implication to the trained networks is explicitly hypothesized, or where this matter is even discussed. Likewise, several of the motivation and concluding paragraphs imply that the paper’s results explain convergence between two networks trained on the same task, whereas, as far as I can tell, the results explain how functional agreement constrains internal representations under suitable assumptions, but leave the connection between successful training and sufficient functional agreement unresolved.
    This all left me feeling unable to really evaluate the empirical results. It's possible I'm misunderstanding something about all this, in which case I would welcome clarification and will of course reassess, so please let me know.

  2. I found it confusing to trace through the various genericity and minimality assumptions, to understand how much of the identifiability conclusion is established by the results and how much is already built into the hypotheses. For example, the ReLU MFR condition explicitly assumes that the nonlinear signatures determine the underlying scalar arguments, while the general-position condition in Appendix Definition 22 effectively excludes full-dimensional families of non-identifiable realizations. The measure-zero conclusion therefore depends on independently justifying this exclusion. These could definitely be legitimate hypotheses, but they carry substantial identifiability content, making it difficult for me to assess the strength and scope of the resulting theorems. I would appreciate a clearer separation between what is proved from basic architectural properties, what requires additional recovery assumptions, and where those assumptions, including the required rank witnesses for genericity, are independently established.

Questions for authors

(Edited to replace bullets by numbers, and for clarity).

  1. Could the authors more directly position their results against other results in the literature that use geometric non-linearities to identify relu network internal parameters? They have a list of related work, but it doesn't go into any mathematical detail about how their techniques and results are related to those from past work.
  2. Can the authors find any way to make the first half of the paper less notation heavy? One approach may be to make the theorem statements a bit more abstract, though this may not be to everyone's tastes.
  3. Can the authors find any way to shorten the appendix?

These are related to the first "Critical weaknesses" item:

  1. Does “hard tasks force convergent computation” assert anything beyond the conditional used-axis bound? In particular, is there a result constraining similarity across the set of parameter values yielding the same task loss, or the distribution training selects within it, without separately assuming terminal or adjacent agreement?
  2. This control is meant to help me understand why the authors' results would apply to two networks which share only the fact that they are both well-trained on the same task. Imagine taking two network prefixes, of the same shape, with independent random initializations. A priori they would not be alignable, right?. Now add trainable suffixes to each, and train each full network to match a common target, to low loss, with the prefixes frozen. The zippering result would have to fail somehow, since the randomly initialized frozen prefix layers are not alignable. Which recovery assumptions would fail here, and (why) do the authors assume that ordinary trained networks avoid the same failure? (Does this example make sense?)

If my understanding of some of this were improved, and the novelty positioning was done more precisely, I would certainly consider moving to an accept.

3 comments

Thank you for your questions. We clarify functional agreement and our identifiability & NeuroAI advances, remove extra assumptions for GELU, Softplus, and individual ReLU blocks, and empirically validate our bound on how many ReLU units must match. Across the six architectures we consider, we also measure the prediction errors entering our bounds on unseen inputs, using affine transformations (weighted sums plus offsets) of one model’s layer responses to predict another’s.

W1: Why functional alignment beyond training data? Our zippering theorems (Theorems 1-2) assume exact or approximate affine agreement at the final layer, and our weak-strong equivalence (WSE) theorems at two adjacent layers (Theorems 3-4), in both cases on unseen inputs, as we say on lines 43-44 in the revision. Similar classification scores alone do not establish this agreement. We prove in App. C.1.1 (lines 2007-2020) that when both networks' final outputs have small average squared error against the same target on unseen inputs, those outputs must also be close to each other on average. This supplies the affine agreement on the test set assumed by zippering because the affine transformation can simply leave the outputs unchanged. We also test this affine agreement assumption directly in trained networks: across six architectures, our affine fits reduce squared error in predicting one model's layer responses from another's by 79.9-93.5% on held-out inputs, compared with comparing coordinates directly (lines 479-523). Thus, our experiments directly demonstrate that trained networks' responses remain predictable from one another on unseen inputs.

W2: What we prove vs. assume. For GELU and Softplus, we now completely remove the general-position assumption of Definition 22 that you questioned, for both exact and soft zippering in these settings (App. C.4.3 & C.8). We therefore prove that affine agreement between the networks’ outputs forces contributing internal units to match for almost every allowed choice of weights, except for a measure-zero set (Theorem 22 for GELU; Theorem 10 for Softplus). Our measure-zero argument establishes this without needing to classify every alternative way different networks could compute the same function. For ReLU, we explicitly retain MFR—the assumption that each unit’s visible bends identify the computation feeding into it—and Definition 22. We have Lean-verified our main exact/soft zippering and weak-strong equivalence (WSE) theorems across GELU, Softplus and ReLU, and have attached the Lean certificates as a zip file under Supplementary Material, with App. C.1.6 pg. 41 listing the verified theorems. In doing so, Lean verification has also allowed us to simplify some of our lemmas and arguments. We also now separate the remaining assumptions more explicitly upfront (lines 138-184; detailed further in the guide in App. C.1.2): task inputs expose the units’ nonlinear behavior (visibility), counted units cannot be removed without changing the function (minimality), and that small independent changes in the internal computation produce distinguishable changes in the outputs (regularity). Our new guide in App. C.1.2 provides the technical definitions of our assumptions and points to the rank witnesses for genericity in our GELU and Softplus proofs (Lemmas 27 and 15).

We show that these assumptions have concrete empirical relevance in real, trained networks: On held-out ImageNet images, ViT gates show nonlinear GELU responses that a straight line cannot fully predict, supporting the visibility requirement that inputs expose the units’ nonlinear behavior (Fig. 6b & App. B.2). These gates also match better than random mixtures (Fig. 6a), alongside recurring units and heads in trained vision and language networks (Fig. 4a-e), supporting our theory’s predictions, explained in Sections 3.2-3.3 and established in App. Theorems 21 and 12, under the stated conditions above.

Q1: Novelty relative to prior identifiability work. In revised Section 2, we go into more detail about the novelty. Specifically, our bounds quantitatively predict how many units must still match even when networks have extra units and make somewhat different predictions. Furthermore, unlike prior identifiability work, we make these predictions about shared features without requiring the entire network to be uniquely determined. Finally, our ReLU and Softplus results cover different widths and depths, apply to modern architectures like deep CNNs and transformers (and therefore applicable to residual blocks and whole heads); the GELU extension is in Apps. C.8–C.9. By contrast, Phuong & Lampert (2020) construct ReLU networks with only one output and layers that never widen. Bona-Pellissier et al. (2023) likewise require non-widening fully connected layers and conditions ensuring that each ReLU unit’s change in slope remains detectable in the network’s output.

We also note that in addition to advancing prior identifiability work, we advance the (currently nonexistent!) theory of NeuroAI. Specifically, we turn Cao & Yamins' (2024) prior informal “Contravariance Principle” into quantitative predictions of shared units and hierarchies—providing a better theoretical understanding of why, over the past 15 years, task-optimized models have converged onto brain responses. In lines 100-106, and in more depth in App. C.7, we also connect to the methodology of NeuroAI practice, showing how commonly used model-brain representation matching metrics like RSA & CKA can be more sensitive to task-irrelevant geometries than ridge regression, especially under neuron/stimulus sampling bias, providing an explanation for the conflicting similarity scores found in recent prior empirical studies of Soni et al., 2024 & Bo et al., 2024.

Q2: Less notation in the first half. Our ten-page main text now explains our comparisons upfront in plain terms (p. 1, lines 46-53). We introduce the main ideas through worked examples rather than dense notation (lines 189-215), and state the theorems and error bounds in the main text in words (lines 216-231). Finally, we summarize our contributions in a short list (lines 58-74) and provide a reader’s guide connecting our claims to their assumptions and proofs (Appendix C.1, beginning at line 1998, with full bounds collected in Appendix C.1.3 (lines 2092-2126)).

Q3: Appendix length and navigation. We shortened the appendix by consolidating repeated algebra and reusing earlier proofs, keeping complete derivations and explaining each reuse. We also split up the appendix into three major sections: section A collects figures, section B describes the empirical methods, and C concerns the proofs of our theorems (and we also provide a reader’s guide in C.1 to the theory that connects our claims to their assumptions and proofs).

Q4: Similarity without assuming terminal or adjacent agreement. Yes, we show that similarity occurs without assuming terminal or adjacent agreement, for sufficiently small squared prediction loss against a common target. Specifically, we show when both networks predict that common target accurately on unseen inputs, their final outputs must be close (Appendix C.1.1, lines 2007-2024). Substituting this loss-derived agreement into our deep-network zippering bounds constrains earlier-unit similarity under our stated recovery conditions (main-text Theorem 2, lines 173-184; in the revision we extend this guarantee to GELU nonlinearities in App. C.8, Corollary 16). In fact, using this theoretical result, we construct an explicit ReLU example in which low prediction loss forces networks to place units near the same changes in slope, constraining similarity across the entire set of sufficiently accurate parameter choices without separately assuming terminal or adjacent agreement (Fig. 7b & App. C.2, Corollary C.2.1). We then test this example’s predicted number of matching ReLU units from our zippering theorems by evaluating 1,230 checkpoints from 30 training runs: all satisfy the bound in Proposition C.2.1 (Eq. 11), and all 20 runs reaching the error target recover every required unit (Fig. 7a).

Q5: Frozen prefixes and trainable suffixes. Thank you for this great question. The frozen-prefix control you mention does not contradict our zippering theorem, as we need to assume “visible units” (explained in W2), which successful task performance alone does not establish. E.g., for ReLU it requires that each counted unit introduces a new, detectable change in slope beyond those introduced by earlier layers (Definition 12(A)). As an example to illustrate this, we show in App. C.1.1 that two prefixes can change slope at different input locations while their respective suffixes exactly undo those changes. Both networks then return the input unchanged, despite their prefixes not being affinely related, and the later layer ReLU acts at an inherited boundary, thereby violating Definition 12(A) (lines 2026-2036).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes

1. Readability: Responding to all three reviewers’ concerns about readability, we have substantially rewritten (and reuploaded) our ten-page main text with worked examples, theorem statements in words, and clearer explanations of the key assumptions. Furthermore, in the Appendix we have shortened and removed any reused arguments, as well as provided a “reader’s guide” in App. C.1 connecting claims to their full proofs.

2. New GELU theory, Lean verification, & empirical validation in trained transformers: Addressing nbY4’s questions about assumptions and YEDm’s concern about our theoretical results not being proven for GELU, we now establish exact and approximate GELU weak-strong equivalence (WSE) and zippering (Thms. 1-4; Apps. C.8–C.9), and furthermore remove needing to assume Definition 22 (the general-position assumption) for GELU & Softplus in the stated settings (full proofs in Apps. C.4.3 & C.8). We have Lean-verified our main exact/soft zippering and weak-strong equivalence (WSE) theorems across GELU, Softplus and ReLU, and have attached the Lean certificates as a zip file under Supplementary Material, with App. C.1.6 pg. 41 listing the verified theorems. In doing so, Lean verification has also allowed us to simplify some of our lemmas and arguments. In response to fbPm and YEDm, we strengthen our existing quantitative evidence for our theory’s predictions of shared units and privileged ViT gates by adding direct tests of individual ViT gate correspondence (Fig. 6), proved positive matching guarantees for zippering in ReLU networks (Fig. 7), and measurements of the affine agreement underlying our approximate WSE theorem across six architectures (Section 5.5). Following fbPm's follow-up and YEDm's non-vacuity question, new Appendix D now gives positive numerical soft-WSE matching guarantees for trained ViT-B/16 and Pythia-160M comparisons, with certified separation constants: at least 89.73% and 47.23% at the stated matching thresholds (Table D.1). It also transfers the bounds to recorded native units (Table D.3) and verifies GELU noncancellation in all 12 ViT blocks. In addition to Lean verifications of our main theorems, the updated Supplementary Material ZIP includes our certified constants and verification scripts.

3. Identifiability & NeuroAI novelty: We also clarify the novelty requested by nbY4 and fbPm: Unlike prior identifiability work, our bounds predict for networks (transformers & deep CNNs) and nonlinearities (Softplus, ReLU, GELU) used in practice how many internal units must be shared without requiring the entire network to be uniquely determined, allowing other units to differ. Under our stated conditions, our zippering theorems explain and quantitatively predict when agreement between models’ outputs forces shared representations in earlier layers (Theorems 1–2). Our WSE theorems establish when agreement between representations also forces individual units to perform matching computations—connecting representational convergence to mechanistic convergence (Theorems 3–4). Our WSE and zippering theorems also formalize for the first time the prior informal Contravariance Principle of Cao & Yamins (2024) as a mechanism for why representational convergence can occur in NeuroAI (Section 2), and provide insights into improved NeuroAI practice by showing how commonly-used similarity metrics like RSA/CKA can be sensitive to task-irrelevant geometries and disagree about shared computations, especially under subsampling and noise that commonly occurs in practice (Appendix C.7).

We hope these revisions address your feedback and allow you to give a higher rating on our submission. If not, please let us know how we can improve.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.