acceptodds
Under review as a conference paper at ICLR 2027

Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth

Abstract

We study how each component of the Transformer feedforward block preserves rank across depth at initialization. We reinterpret skip connections and normalization as mechanisms for preserving gradient rank as well as magnitude across depth, since the operations that make a network expressive also reduce its rank. Skip connections route the gradient around the residual branch, where rank is lost, rather than along the long gradient paths that encourage the layers to compose. Across a combinatorial sweep of residual architectures that vary the normalization placement, initialization and branch scales, depth, branch shape and activation, we show that the effective rank of the input–output Jacobian is one function of the mean number of branches a gradient path passes through, which is determined by the branch-to-skip ratio. Deep residual networks therefore trade off rank collapse against ensemble-like behavior; preserving rank makes them ensemble-like by necessity. Normalization placement controls the branch-to-skip ratio across depth, which explains why rank collapses for Post-Norm but decays only slowly for Pre-Norm and unifies much of the normalization placement and depth scaling literature. The two-matrix structure uses additional parameters to preserve rank: the second matrix decorrelates a coherent mean spike that would grow across blocks with a single matrix and an uncentered activation, and expanding the width between the two matrices lets enough directions survive the activation to span the original space. In transformer language models, the rank at initialization follows the same function of path length, and most training failures coincide with rank collapse. Taken together, our results recast architecture design for deep networks as navigating an intrinsic tradeoff among rank collapse, ensemble-like behavior, and parameter count.

Primary area:
optimization
Keywords:
rank collapse, Jacobian spectra, residual connections, normalization, signal propagation, random matrix theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_L7qZOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors investigate the effective rank in deep residual networks, discussing how architectural components such as skip connections, normalization and width expansion impacts rank collapse.

Critical strengths
  1. The work proposes an interesting spectral interpretation of different commonly used architectural choices.
  2. The work contains a large number of experimental results to showcase the analysis presented.
Critical weaknesses
  1. My main concern is regarding the implicitly positive treatment of rank collapse in the paper presentation. More precisely, it is not clear (and I am personally not even convinced it's true) that effective rank preservation should be something we want to aim in the first place. In fact, given that linear maps are the only learnable parts of modern networks, rank collapse is the only general mechanism which allows to discart and compress information.
  2. While intuitive, the discussed tradeoff between rank preservation and compositional depth given by residual layers seems to be largely speculative. Moreover, the claimed universal relationship between effective rank and mean path length is established only at initialization, without any further discussion on how this might evolve during training.
  3. The current presentation mainly explains and reinterprets empirically observed effect of certain architectural choices, but it remains unclear what is the conclusion from these observations.
  4. The scale of the experiment is small to moderate, and they do not seem to include multiple seeds per run, which might be important in the proposed type of analysis. Moreover, some experiments (such as the one presented in figure 5), seem to be somehow detached from the main discussion, in particular the main focus seems to be on the trainability-rank collapse relationship, which is in fact not deeply discussed in the current version of the manuscript.
Questions for authors
  1. Is it possible for the authors to justify (even numerically) that mean path length is a useful predictor for compositional ability?
  2. Can the proposed framework lead to novel practically useful architectural recommendations?
Reviewer_cAyTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work provides a detailed analysis of various transformer configurations such as computational depth and norm placement, through the lens of how each configuration affects the effective rank of the "input–output Jacobian" (the Jacobian of the residual state at layer L to the input to the transformer) at initialization. Following prior work, the paper expands the input-output Jacobian into paths (skip vs computational branch paths), and then defines a mean path length k, as the average number of computational branches (MLP or Attention) that a gradient path passes through. The primary empirical finding of this work shows that, across a number of configurations the effective rank of the input-output Jacobian is a decreasing function of this path length k. This is used to argue, among other things, that: One, residual skips preserve rank while computational branches lose rank, forcing deep models to use shorter paths, making them more "ensemble-like". Two, explain the importance of norm placement by its effect on this path length k. And Three, relating rank collapse or expansion to various properties of the network such as the FFN's second matrix and expansion width, as well as Post-LN training failure.

Critical strengths
  1. Proposed metric predicts rank loss reliably. The paper's defined metric of the mean path length "k" (avg. number of residual branches a gradient passes) appears to reliably predict the effective rank of the input–output Jacobian at initialization. This is shown to be reliably true, across several configurations of norm placement, branch scales, depth, among others. This relationship to the effective rank is used to carefully support explanations for why rank collapses quickly with depth with Post-Norm but slowly under Pre-Norm, and why scaling each branch by L^(-1/2) keeps rank from shrinking. The fact that this proposed mean path length "k" metric can therefore be used to explain broad depth-related effects, is quite a strong result in my opinion. There are however, concerns on the uniqueness of this metric compared to prior work.

  2. Strong analysis, with careful design and ablations:. While the paper is quite a tough read, the provided experimental analysis is comprehensive. Authors build their analysis from MLP to Transformer blocks, and cover a large number of architectural configurations and depth effects. The analysis is also well designed and supported, and given the consistency of the rank-based prediction, the primary analysis appears reliable at least at initailization. I however have concerns about claims following from this analysis.

Despite the strong analysis, there are important concerns regarding the evidence for the claims made beyond the analysis, as well as the uniqueness of the proposed mean path length metric relative to metrics from prior work.

Critical weaknesses
  1. Unclear how proposed rank-based explanation differs from prior work. The proposed rank-based explanation appears to be quite similar to known results. Although it isn't entirely clear if the rank explanation in the paper is the general case, or another interpretation of the same measure. For instance, as the paper mentions, mean path length "k" is within 1.5% of the measure that Liu et alia [1] used to explain Post-LN instability, along with the same depth laws reported in this work. Work from Liu et al [1] therefore explains the Post-LN trainability problem equally well. Additionally, prior work [2] has already tied properties of the Jacobian (spread of its singular values), which is infact exactly what the paper's effective rank measures, to a single measure (the size of the residual branches) [2]. This once again is similar to the paper's primary finding that rank is a single function of "k". It is therefore currently unclear if the proposed rank-based view is another interpretation of the same measure, and if predicts nothing beyond prior work. Without clarification on this, it is difficult to evaluate the value of the proposed method.

  2. Key claim appears unsupported: It is important to state that these analytical results in this work appear to be largely at initialization. Authors claim that preventing rank collapse forces deep networks to be "ensemble-like", where they are dominated by short paths, causing them to compose their depth poorly. But this is simply inferred by measuring a small "k" at initialization. And the claimed effect of this short path property on depth utilization is never measured, to my knowledge. Given that training is undestood to increase "k", it is further unclear if this initialization effect would carry over. Since this is also one of the paper's primary claims and is central to its explanation of the structure of Transformers today, this is a significant concern.

  3. Readability: The paper, in its current form, is quite difficult to read and follow. The authors disclose significant AI use in drafting the paper's content; it is evident in the writing: The content is quite dense, and the language used is typically incredibly hard to follow, or more convoluted than is necessary. In my opinion, the poor readability is a significant enough concern to be grounds for rejection if other reviewers share the same concerns.

References:

[1]: Liu, L., Liu, X., Gao, J., Chen, W., & Han, J. (2020, November). Understanding the difficulty of training transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 5747-5763).

[2]: Tarnowski, W., Warchoł, P., Jastrzȩbski, S., Tabor, J., & Nowak, M. (2019, April). Dynamical isometry is achieved in residual networks in a universal way for any activation function. In The 22nd International Conference on Artificial Intelligence and Statistics (pp. 2221-2230). PMLR.

Questions for authors
  • Could the authors clarify what their proposed rank-based view explains or predicts, beyond prior works? (See Weaknesses #1 for example comparisons). Would there be a configuration where the prior measures would predict differently from the paper's proposed measure?

  • The authors are requested to clarify whether there is any evidence that short paths at initialization limit how much depth the trained model uses or hurt its performance. Note that I'm interested in whether the evidence translates to behavior during/after training.

  • Authors are also strongly requested to improve the readability of the text of the paper. It is currently unusually convoluted and hard to follow. This is a significant concern in my opinion, and must be resolved for the work to be readable by others.

Reviewer_9iotOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Core Claim by authors: "the architecture design of the feedforward block navigates an intrinsic tradeoff across depth between rank collapse and ensemble-like behavior" (lines 47-48).

They track the rank, across depth, of 3 objects (lines 114-122): (1) the input-output Jacobian ∂hL/∂h0\partial h_L / \partial h_0 (whose rank also corresponds to the "gradient rank"), (2) the forward residual representation ZℓZ_\ell, and (3) the Jacobian of the branch without its normalization, JfJ_f. Using a decomposition of the Jacobian into paths through the skips and branches (Eq. 4), they define a mean path length ⟨k⟩\langle k \rangle (Eq. 5) and report that, across 132 residual MLP configurations, the effective rank at initialization is "one function of" ⟨k⟩\langle k \rangle (Figure 2, right). They then study normalization placement (Section 4), the two-matrix branch and width expansion (Section 5), and small transformer language models (Section 6).

Critical strengths
  • The set up is clean and well-written.
  • I enjoyed going through some of the math.
  • Figure 2 (right): 132 configurations fall on one curve of effective rank against ⟨k⟩\langle k \rangle (lines 195-198), and the authors are explicit that this is an empirical finding (lines 198-199).
Critical weaknesses

High-level comment: several times things are simply stated without enough context or support for a non-expert to understand (i.e., expert in the exact niche of this paper). Also the paper is lacking in providing intuition, and the significance of certain statements is dubious. Also, a combinatorial explosion of jargon makes it difficult to keep in mind what the core questions being asked are. I believe diagrams describing the information flow would be much more helpful rather than tons of plots.

Detailed comments:

  • It would be helpful to include a diagram or visualization of the architecture and associated information flow being studied. E.g., the structure of ff, what pre-norm vs post norm looks like, etc.
  • In line 119 the authors define the branch Jacobian Jℓ:=∂f(hℓ−1)/∂hℓ−1J_\ell := \partial f(h_{\ell-1}) / \partial h_{\ell-1}. Then in lines 140-142 they also mention the Jacobian of the block, ∂hℓ/∂hℓ−1\partial h_\ell / \partial h_{\ell-1}. This is consistent but the different notions of Jacobian can be confusing.
  • Lines 144-145: you say rℓ2=pβ2α2Mr_\ell^2 = p \beta^2 \alpha^{2M} "in expectation", why not write E[rℓ2]=…\mathbb{E}[r_\ell^2] = \ldots instead?
  • The paragraph in lines 148-155 is written in a very opaque way. Can the authors clarify what they're trying to convey?
  • The notions of "path" (lines 156-161) and "ensemble-like behavior" (lines 78-81) are central to the paper, but how sequentially arranged blocks can behave like an ensemble is never made explicit. I feel like this explanation is important but lacking from the exposition. For example, expanding a two-block network, h2=x+βf1(x)+βf2(x+βf1(x))≈x+βf1(x)+βf2(x)+β2J2f1(x)h_2 = x + \beta f_1(x) + \beta f_2(x + \beta f_1(x)) \approx x + \beta f_1(x) + \beta f_2(x) + \beta^2 J_2 f_1(x), shows that each term in the expansion is a path, and that the term where block 2 acts on block 1 carries an extra factor of β\beta. Adding an example like this would make the path decomposition (Eq. 4), the mean path length, and the ensemble-like view much easier to follow (along with some diagrams).
  • The tradeoff claim (lines 47-48, 21-22) is never tested on the ensemble side: no experiment shows a cost from ensemble-like behavior, and lines 470-473 acknowledge that the experiments measure trainability only.
  • The transformer experiments are small (width 256, at most 48 blocks, one seed; lines 410-412, 470-471), and the mean path length does not determine trainability across placements (lines 427-428).
Questions for authors
  1. In the intro (lines 38-44), when discussing how "the" Transformer feedforward block has evolved since the original Transformer paper, which standard are the authors referring to? Surely the details of pre-norm vs post-norm, activation function, etc. vary widely across architectures and applications? Also, what do they mean when they say "expands the width with one matrix" (line 39)?
  2. What is "ensemble-like behavior"? Architecturally, the blocks are arranged sequentially, so please make explicit how they can behave like an ensemble (e.g., by expanding a two-block network).
  3. Are the weights "tied" depth-wise? I.e., is ff the same function across residual blocks? Eq. 1 writes ff without a block index.
  4. They call ∇h0L\nabla_{h_0} \mathcal{L} the "gradient" (lines 119-120), which is the gradient of the loss w.r.t. the inputs, but usually in this context I would assume "gradient" would be referring to the gradient of the loss w.r.t. the weights. So why do they mention the former but not the latter?
  5. Lines 133-134: "skip connections route the gradient around the residual branch, where rank is lost, through an identity path that is trivially full rank". What do you mean that the rank is "lost"?
  6. Lines 136-137: "... if it all passed through the skips, the network would act like an ensemble of individual layers rather than a deep network whose layers compose." I don't know what is meant by "an ensemble of individual layers". If the gradient all went through the skip connections, the network wouldn't learn any structured representations, i.e., it would not change over time? Or would it?
  7. Line 144: what is "stream scale"?
  8. Paragraph starting at line 156: why do the authors talk about the paths going through "either" the skip or the branch at each block, and not going through both? In what scenario would the path go through just one of the two? I need clarification on the idea of "mean path length" (line 76).
  9. How is "path length" defined?
  10. The authors use the phrase "the effective rank is one function of ⟨k⟩\langle k \rangle" (lines 196-197). What does this mean? What does "one function" mean?
  11. From looking at Figure 2 (right panel), effective rank decreases as mean path length increases? Why? What's the intuition?
  12. Line 200: what is "downscaling the residual branch"? Like, what does "downscaling" mean here?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.