acceptodds
Under review as a conference paper at ICLR 2027

A Conditioning Lens on Multi-Head Attention

Abstract

Transformers have reshaped machine learning in large part through multi-head attention, yet the optimization role of multiple heads remains less understood. In this work, we provide a theoretical and empirical perspective on multi-head attention through the conditioning of attention-layer Jacobians. Under explicit assumptions on the statistical structure of attention-head weights, we show that increasing attention heads can improve the conditioning of the attention Jacobian, suggesting a conditioning-like mechanism for multi-head attention. This analysis motivates the hypothesis that allocating capacity to additional heads can improve trainability more effectively than simply increasing depth. We test this hypothesis across a range of transformer architectures and domains, and find that head-rich configurations consistently exhibit better-conditioned attention Jacobians and improved optimization behaviour. Building on these findings, we redesign standard transformer models to use more heads and fewer layers, achieving reductions of 30–50% in parameter count, TFLOPs, and memory while preserving accuracy. These leaner architectures also converge in fewer training steps than commonly used transformer baselines. Overall, our results identify attention heads as an important architectural factor in transformer conditioning and provide a practical route to more efficient transformer design without degrading downstream performance.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
multi-head attention, conditioning, jacobian

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_g4VCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposes a conditioning-based view on multi-head attention, which analyzes how the number of attention heads affects the conditioning of the attention-layer Jacobian. Thorough attention-head weight statistics, authors show that adding more attention heads improves the conditioning of the attention Jacobian. The finding suggests that adding head capacity help enhancing model performance more effectively than just adding depth. Across diverse Transformer architectures in vision and nlp tasks, head-rich configurations consistently generate better-conditioned Jacobians and better optimization performance. Leveraging this insight, the authors desigend a new Transformer arhcitecture to use more attention head.

Critical strengths
  1. Extensive experiments are demonstrated for different backbones such as Vit and DINO, across vision and tasks. They effectively proves the theory proposed in the paper.
  2. The theory in the paper can provide some guidance on tranformer archetecture design.
Critical weaknesses
  1. In Theorem 3.1, authors assumed that 1) the singular value of the attention metrix A and the projection matrix P are positive 2) the attention matrix columns are independent, mean zero, Gaussian distributed. Why do they hold? Can authors provide theoretical or empirical evidence? If ithey do not hold, it undermines the contribution of the paper. Increasing attention head numbers will improve the preformance is something we already know in the community. I do not think we need to trade depth for heads. I think a more interesting problem is what is the optimal attention head under different neural net architecture, such as depth.
  2. Can we compute the Jacobian condition number for very large models, like more than 100B? If not, the theory has less impact in real-life use.
Questions for authors
  1. I suggest the author to reduce the number of subplots in figure 1. It is better to put different method in one plot when x axis and y axis are the same.
  2. How will the result change if use different optimiser (adamw vs muon), different attnetion kernel ( for example, flash attention)?
3 comments

We thank the reviewer for taking the time to review our paper and all the comments/questions they have raised. Below we answer all the weakness points and questions raised. We have also made explicit changes to the main paper, uploading a new version, in red coloured font so as to make it easy for the reviewer to check and track the changes.

1. In Theorem 3.1, authors assumed that 1) the singular value of the attention matrix A and the projection matrix P are positive 2) the attention matrix columns are independent, mean zero, Gaussian distributed. Why do they hold? Can authors provide theoretical or empirical evidence? If they do not hold, it undermines the contribution of the paper.

Thank you for your comment and question. We have explained the assumptions in remark 3.2 after the statement of theorem 3.1. In that remark, we also write that we empirically verify the assumptions in Appendix A.1.2. In Appendix A.1.2 we run several experiments that clearly verifies all the assumptions about the singular values of the attention matrix A and the projection matrix P (see Figs. 5 and 6). We have done this for both vision transformers and the language one. In all cases, we have empirically shown that our assumptions hold.

2. Increasing attention head numbers will improve the performance is something we already know in the community.

We thank the reader for their comment. However, no where in the paper do we claim increasing attention heads will improve performance. We claim that increasing attention heads will reduce the condition number of the attention Jacobians and this should lead to better convergence. We then show that this is indeed the case, see Fig. 1 and Fig. 3 in the paper, and we then hypothesize that since increasing heads gives the model better conditioning the model does not need so many parameters which we test by reducing the depth and MLP width. In almost all cases increasing attention heads leads to comparable performance which empirically is a by product of the better optimization given the better conditioning.

3. I do not think we need to trade depth for heads. I think a more interesting problem is what is the optimal attention head under different neural net architecture, such as depth.

Unfortunately we are not sure what exactly the reviewer means by this comment. If they can elaborate we will be more than happy to have a discussion. We in fact show that many transformers do not have the optimal head and depth allocation and by considering the conditioning of the attention Jacobian one can get much more optimal architectures that obtain comparable performance to standard baselines where optimal here means efficient i.e. less parameters, memory, training iteration time and TFLOPS.

4. Can we compute the Jacobian condition number for very large models, like more than 100B? If not, the theory has less impact in real-life use.

In principle, Jacobian condition numbers can be computed even for models with more than 100B parameters. The computational cost grows with model size, and training models at that scale and computing their Jacobian condition numbers would exceed our available compute capacity. However, we do have experiments that support the insights from the idealised analysis at the 1B-parameter scale. We use those insights to guide the design of smaller transformers that achieve performance comparable to that of larger baselines. We still believe that this is valuable insight for the community working on transformer pretraining.

5. I suggest the author to reduce the number of subplots in figure 1. It is better to put different method in one plot when x axis and y axis are the same.

We thank the reviewer for this suggestion. Since the plots in figure 1 show the same trend we feel it might in fact be better to move the second and third rows of the plots to the appendix. We will do this towards the end of the author-reviewer discussion period of the rebuttal process after we have included every reviewers comments from the discussion.

6. How will the result change if use different optimiser (adamw vs muon) different attnetion kernel ( for example, flash attention)?

Our proposed changes are architectural. In response to the reviewer, we reran all experiments from Table 1 using the Muon optimizer. Across all eight model variants, the leaner configurations achieve comparable top-1 accuracy with substantially fewer parameters. Memory usage, iteration time (as this is computing the forward + backward pass time), and TFLOPs match those reported in Table 1 of the main paper.

Table 1 experiments rerun with Muon

ModelConfigurationTop-1 (%)Params
ViT-B(12, 12, 4) baseline80.186M
ViT-B(7, 16, 2)80.440M
ViT-L(24, 16, 4) baseline79.3304M
ViT-L(13, 20, 2)79.5124M
ViT-G(40, 16, 48/11) baseline77.21B
ViT-G(21, 24, 2)77.4351M
XCiT-M(24, 8, 4) baseline84.684M
XCiT-M(13, 16, 2)84.747M
XCiT-L(24, 16, 4) baseline85.1189M
XCiT-L(13, 20, 2)85.299M
DaViT-B((1, 1, 9, 1), (4, 8, 16, 32), 4) baseline84.887M
DaViT-B((1, 1, 5, 1), (4, 8, 20, 32), 2)84.845M
DaViT-L((1, 1, 9, 1), (6, 12, 24, 48), 4) baseline84.9197M
DaViT-L((1, 1, 5, 1), (6, 12, 28, 48), 2)84.8100M
DaViT-G((1, 1, 12, 3), (12, 24, 48, 96), 4) baseline82.81.4B
DaViT-G((1, 1, 7, 2), (6, 12, 52, 98), 2)83.3621M

Within the attention module, we vary the number of heads and retain the same per-head attention operation. The resulting configurations can use attention kernel implementations such as FlashAttention. We added a remark clarifying this in the main paper (lines 264–266).

Dear reviewer g4VC, we thank you very much for your review. Before the discussion period ends, we would like to check if our response has addressed your concerns?

Reviewer_ypMbOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates tradeoffs in network design by trading transformer depth for more heads in multi-head attention. Unlike standard ablations over number of heads, where the token embedding dimension is held fixed, the authors scale the hidden dimension as well (effectively increasing the width of the network). The main theoretical results show that as the number of heads is increased, the gradients through multi-head attention with respect to the Q, K, V matrices become better conditioned.

Critical strengths

Strengths:

 - The paper is incredibly well-written, clearly motivated, and has fairly convincing and thorough experiments and is of high impact. The experiments and the underlying methodology are very clean and focused, with extensive experiments across multiple domains. Although the paper does not propose a fundamentally new architecture, it is an extremely well-executed from an experimental standpoint.  - Similar prior work (https://arxiv.org/pdf/2210.00640) shows similar overall conclusions, but this work does so with much improved methodology, clear theoretical insights as to why training speed might improves, and with a more modern protocol looking convergence speed to a given accuracy level rather than just final accuracy.

Critical weaknesses

Weaknesses:

  • On a minor note, the way A/MA are defined could use improvement. Writing A without the subscript suggests a stacked version of all the A_i's, but looking at line 162, you define A(X)A(X) as being just a single attention head, while MA(X) is the projected version of all of the attention heads? I would also define (3) directly in terms of Q/K/V rather than first defining q, k, v. This is not a huge deal since readers should be familiar with attention mechanisms, but some notational changes could make reading a bit easier.

  • Relatedly, I would also suggest defining something like M := [Q, K, V] and then you can write ∂A∂M\frac{\partial A}{\partial M} instead of introducing a "J" operator that looks like a function but is actually the jacobian of its argument.

  • The main theoretical claim is regarding better conditioning of the gradient as a result of using more heads. The authors show in Figure 1 that their proposed changes do in fact improve the conditioning. However, considering a shallower network naturally increases convergence speed, and I would expect the condition number to change naturally over the course of training, irrespective of architecture. As the Figure 1 results average over the entire 300-epoch training run, it is possible Figure 1 conflates these two effects.

I think it is necessary to have a plot where the y axis is the condition number and the x axis is either the epoch or a given train or test loss and multiple lines are shown for different architectural choices (e.g. the parameter-matched pairs shown in Figure 2 perhaps?).

  • There is a mismatch between the theorem statement and the proofs: eq (7) states C_1, C_2 as a bound on the singular value of A_i, while at line 803 you cite C_1, C_2 as bounds on the Jacobian of A_i.

  • In equation (15) the fraction should be reversed to be sigma_max over sigma_min. This appears to be a typo.

  • At the bottom of page 14, the authors write J J^\top. Is this meant to be J^\top J? As currently stated line 749 is incorrect.

Questions for authors

Questions:

 - Do you have experiments showing the conditioning number over the course of training across different numbers of heads?

 - See weaknesses regarding questions about issues found in the proofs. These issues may be fixable or may arise from a misunderstanding, but I did not attempt a fix of the proofs myself.

3 comments

We thank the reviewer for taking the time to review our paper and all the comments/questions they have raised. Below we answer all the weakness points and questions raised. We have also made explicit changes to the main paper, uploading a new version, in red coloured font so as to make it easy for the reviewer to check and track the changes.

1. On a minor note, the way A/MAA/MA are defined could use improvement. Writing A without the subscript suggests a stacked version of all the AiA_i's, but looking at line 162, you define as being just a single attention head, while MA(X)MA(X) is the projected version of all of the attention heads? I would also define (3) directly in terms of Q/K/VQ/K/V rather than first defining q, k, v. This is not a huge deal since readers should be familiar with attention mechanisms, but some notational changes could make reading a bit easier.

Thank you for your comments. Yes A(X)A(X) is a single attention head, we have now clearly stated this in the paper writing "note that when we write A(X)\mathbf{A}(X) without any subscript it is to be interpreted as a single attention head" (see lines 167-169). We have also added an extra equation to (3) making it clear as to how it depends on Q, K and V and X.

2. Relatedly, I would also suggest defining something like M := [Q, K, V] and then you can write instead of introducing a "J" operator that looks like a function but is actually the Jacobian of its argument.

Thank you for the suggestion. We would like to retain J(⋅)J(\cdot) because it gives us a consistent notation for the Jacobians of individual heads and the full multi-head attention block.

3. The main theoretical claim is regarding better conditioning of the gradient as a result of using more heads. The authors show in Figure 1 that their proposed changes do in fact improve the conditioning. However, considering a shallower network naturally increases convergence speed, and I would expect the condition number to change naturally over the course of training, irrespective of architecture. As the Figure 1 results average over the entire 300-epoch training run, it is possible Figure 1 conflates these two effects. / Do you have experiments showing the conditioning number over the course of training across different numbers of heads?

We thank the reviewer for this suggestion. We have included a plot in the appendix that shows the Jacobian condition numbers for a ViT-B, ViT-L and ViT-G throughout training as we increase the number of heads, see Fig. 14 (lines 1335-1346). We clearly see that the condition number can oscillate through training but the general trend is that it decreases as the number of heads increases. We point out that we are reporting the average Jacobian condition number over the number of heads and layers for each model so we get one number for each architecture.

4. There is a mismatch between the theorem statement and the proofs: eq (7) states C1C_1, C2C_2 as a bound on the singular value of AiA_i, while at line 803 you cite C1C_1, C2C_2 as bounds on the Jacobian of AiA_i.

Thank you for flagging this mismatch. Equation (7) bounds the singular values of the head outputs AiA_i, which enter the full Jacobian through the output-projection block. We have revised the step at line 803 to control the query, key, and value derivative block UU separately.

We have included Lemma A.3 (lines 785–797) in Appendix A.1.1, which gives ∣U∣op=O(log⁡h)\\|U\\|_{\mathrm{op}} = O(\log h) (where ∣∣⋅∣∣op||\cdot||_{op} is the operator norm i.e. max singular value) under the stated growth conditions on the input and projected norms. Consequently, ∣U∣op2/(hd)\\|U\\|_{\mathrm{op}}^2/(hd) tends to zero. The revised asymptotic argument uses these additional growth conditions to control the derivative term.

5. In equation (15) the fraction should be reversed to be σmax⁡\sigma_{\max} over σmin⁡\sigma_{\min}. This appears to be a typo.

This was a typo and we really appreciate the reviewer pointing it out. We have fixed this.

6. At the bottom of page 14, the authors write JJ⊤J J^\top. Is this meant to be J⊤JJ^\top J? As currently stated line 749 is incorrect.

Thank you! This was a typo and has been fixed.

Thanks for additional experiments and fixing the typos. Good luck. I will keep my score. This is a well done empirical paper, even if it's not necessarily ground breaking.

Thank you! We have really appreciated all your questions and comments and no doubt they have helped make the paper better.

Reviewer_JkwpOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that adding attention heads improves Jacobian conditioning, so you can reduce depth. The shallower models with more heads reportedly match or beat deeper ones, with fewer parameters and lower training cost, on ViT, XCiT, DaViT, GPT-2, and BERT.

Critical strengths
  • Looking at heads through Jacobian conditioning is an interesting angle
  • The experiments cover several architectures and tasks, and the savings in parameters and training cost look promising
Critical weaknesses
  1. I have some concerns about the theoretical analysis
  • Theorem 3.1 assumes bounds on A_i, but the proof uses bounds on J(A_i). I am not sure how the former justifies the latter.
  • Theorem A.2 uses a minimum singular value bound that does not seem to hold in general for rectangular matrices. Since the theory is one of the main contributions, I think these points need clarification.
  1. The experiments do not clearly isolate the effect of heads
  • Heads, depth, and MLP width all change together, so it is unclear whether increasing heads is what helps.
  • The ViT baselines seem relatively weak. ViT-L gets 79.1%, while previous supervised training from scratch has reported 82.6%. The larger baselines also perform worse than ViT-B.
  • "Wide Attention Is The Way Forward For Transformers?" already studied increasing attention heads while reducing depth. However, this paper does not discuss that work, and I am not sure what the main difference is.
Questions for authors
  1. Could the authors clarify the assumptions and proof of Theorem 3.1 and Theorem A.2, particularly the bounds used for A_i and J(A_i)?

  2. How does ViT-B (7, 12, 2) compare with (7, 16, 2)? Is increasing heads necessary to maintain performance?

  3. Why do the ViT-L / G baselines perform worse than ViT-B? Were the training settings tuned for each model size?

  4. How does this work differ from Wide Attention Is The Way Forward For Transformers?, which also studied trading depth for attention heads?

2 comments

We thank the reviewer for taking the time to review our paper and all the comments/questions they have raised. Below we answer all the weakness points and questions raised. We have also made explicit changes to the main paper, uploading a new version, in red coloured font so as to make it easy for the reviewer to check and track the changes.

1. Theorem 3.1 assumes bounds on AiA_i, but the proof uses bounds on J(Ai)J(A_i)/Could the authors clarify the assumptions and proof of Theorem 3.1

We thank the reviewer for raising this point. In Theorems 3.1 and A.2, the head outputs AiA_i enter through the derivative with respect to the output projection PP. The Gram matrix of this block is determined by AcatAcat⊤=∑i=1hAiAi⊤A_{\mathrm{cat}}A_{\mathrm{cat}}^\top = \sum_{i=1}^h A_i A_i^\top, so bounds on the individual head outputs AiA_i are needed to control its eigenvalues.

The head Jacobians J(Ai)J(A_i) enter separately through a block UU corresponding to the query, key, and value parameters. We apologize if this was hard to read from the statement of the theorem and its proof. We have therefore added Lemma A.3 (line 785) in Appendix A.1.1, which bounds ∣U∣op\\|U\\|_{\mathrm{op}} (the operator norm of U) in terms of the input and the projected query, key, and value norms.

Under the norm-growth conditions stated in the lemma, ∣U∣op2/(hd)\\|U\\|_{\mathrm{op}}^2/(hd) tends to zero. This makes explicit the additional derivative control used in the proof of Theorem 3.1. Under that control, the limiting covariance of the concatenated head outputs determines the conditioning limit. We have also added a paragraph explaining Lemma A.3 (lines 798–804).

2. Theorem A.2 uses a minimum singular value bound that does not seem to hold in general for rectangular matrices. Since the theory is one of the main contributions, I think these points need clarification.

We thank the reviewer for raising this point. We understand the comment to refer to Equation (21), which is used in Theorem A.2. Please let us know if you had a different equation in mind.

For a matrix M∈Rm×nM \in \mathbb{R}^{m \times n}, we use the following convention for the minimum singular value.

σmin⁡(M)=λmin⁡(M⊤M)=min⁡x∈Rn∣x∣2=1∣Mx∣2.\sigma_{\min}(M) = \sqrt{\lambda_{\min}(M^\top M)} = \min_{\substack{x \in \mathbb{R}^{n} \\\\ \\|x\\|_2 = 1}}\\|Mx\\|_2.

This definition includes any zero eigenvalues of M⊤MM^\top M. For any matrices AA and BB for which ABAB is defined, we have

∣ABx∣2≥σmin⁡(A)∣Bx∣2≥σmin⁡(A)σmin⁡(B)∣x∣2\\|ABx\\|_2 \ge \sigma_{\min}(A) \\|Bx\\|_2 \ge \sigma_{\min}(A) \sigma_{\min}(B) \\|x\\|_2

Taking the minimum over unit vectors xx gives the lower bound in Equation (21). We have added this clarification to the Matrix notations paragraph in Section 3.1 (lines 184–188).

3. Heads, depth, and MLP width all change together, so it is unclear whether increasing heads is what helps.

Figure 3 clearly shows the effects of increasing heads with fixed depth versus increasing depth with fixed heads, matching the same number of parameters in each case, for 3 different sized ViTs. In each case the plots show that increasing heads with fixed depth, and keeping the parameter size the same, results in lower condition numbers and faster convergence clearly showing the effect of increasing heads is much more useful than that of increasing depth and isolating this effect when compared against increasing depth.

We than asked the question of whether increasing MLP width can provide an analogous result, see lines 405-410 and show in appendix A.2.1, with detailed ablations, that keeping parameter budgets the same increasing heads yields lower condition number and faster convergence than increasing MLP width. These 2 experiments isolate the effect of increasing each component and show that increasing heads yields the most optimal design.

Finally, because of the above isolated result showing that increasing heads yields better conditioning. We then hypothesize empirically that this could lead to learner configurations where we cut down layers and MLP width. This is explained in lines 412-426. This is the only place where we change heads, depth and MLP width all at the same time and we do so by increasing heads and reducing depth and MLP width based on our previous results. As can be seen in Table 1, in doing so we obtain significantly more efficient parameterizations of transformer architectures in terms of parameters, memory, training iteration time and TFLOPS.

4. The ViT baselines seem relatively weak. ViT-L gets 79.1%, while previous supervised training from scratch has reported 82.6%. The larger baselines also perform worse than ViT-B / Why do the ViT-L and G baselines perform worse than ViT-B? Were the training settings tuned for each model size?

Our ViT baselines follow the training setup of Dosovitskiy et al. (An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale) and use data augmentation from Steiner et al. (How to Train Your ViT? Data, Augmentation, and Regularization in Vision Transformers). With this setup, our baselines achieve slightly higher accuracy than the corresponding results reported by Dosovitskiy et al.

Table 5 of Dosovitskiy et al. shows that ViT-L achieves lower accuracy than ViT-B when trained on ImageNet-1k alone. In the ImageNet-21k row of the same table, ViT-L achieves higher ImageNet-1k accuracy than ViT-B after pretraining on ImageNet-21k.

In response to the reviewer, we pretrained the ViT-B, ViT-L, and ViT-G baselines and their leaner configurations on ImageNet-21k and evaluated them on ImageNet-1k. Under this setup, both ViT-L and ViT-G outperform ViT-B.

ImageNet-1k accuracy after pretraining on ImageNet-21k

Model and configurationAccuracy (%)
ViT-B (12, 12, 4) baseline84.3
ViT-B (7, 16, 2) ours84.5
ViT-L (24, 16, 4) baseline85.3
ViT-L (13, 20, 2) ours85.4
ViT-G (40, 16, 48/11) baseline86.1
ViT-G (21, 24, 2) ours86.1

We selected our baselines based on published results and have been unable to locate a reference for the reported 82.6% accuracy for the original ViT-L architecture trained from scratch on ImageNet-1k with supervised learning. If the reviewer can share the reference, we would be happy to run that baseline and include the result in the paper. We also welcome references to other relevant baselines.

5. How does ViT-B (7, 12, 2) compare with (7, 16, 2)? Is increasing heads necessary to maintain performance?

The ViT-B (7, 12, 2) does not perform well compared to the case of increasing heads to (7, 16, 2). When training on ImageNet-1k ViT-B (7, 12, 2) gets 78.6% Top-1 accuracy while (7, 16, 2) gets 80.6%. The (7, 16, 2) has better conditioned attention Jacobians that empirically leads to better optimization and hence better performance.

"Wide Attention Is The Way Forward For Transformers?" already studied increasing attention heads while reducing depth. However, this paper does not discuss that work, and I am not sure what the main difference is / How does this work differ from Wide Attention Is The Way Forward For Transformers?, which also studied trading depth for attention heads?

We thank the reviewer for pointing us to Brown et al. "Wide Attention Is The Way Forward For Transformers?". In their main NLP experiments, Brown et al. vary depth LL and heads per layer hh together while keeping LhLh fixed. This redistributes a fixed attention-projection parameter budget across layers. Furthermore, they work with models that only seem to go up to 6 layers and only have empirical results.

Our approach is very different. Our controlled experiment in Figure 3 asks how to allocate an increase in total model parameters. We add heads at fixed depth in one set of models and add layers at fixed head count in the other, matching parameter counts across the two sets. This changes the parameters of the product LhLh in contrast to Brown et al. Furthermore, we have a concrete motivation to increase heads namely given by considering the conditioning of the Jacobian.

We then use the conditioning results to guide the lean designs. For example, our lean ViT-B changes from 12 layers with 12 heads per layer to 7 layers with 16 heads per layer, reducing LhLh from 144 to 112 while also reducing MLP width. This tests whether the redesign motivated by the conditioning analysis can match baseline accuracy at lower cost. Furthermore, we have experiments on both language and vision.

We do believe it is important to bring awareness to prior work that considers similar approaches to our work and have therefore cited the Brown et al. paper in the Related work section clearly explaining what their work does and how it differs from ours, see lines 106-111.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.