- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors proposes a conditioning-based view on multi-head attention, which analyzes how the number of attention heads affects the conditioning of the attention-layer Jacobian. Thorough attention-head weight statistics, authors show that adding more attention heads improves the conditioning of the attention Jacobian. The finding suggests that adding head capacity help enhancing model performance more effectively than just adding depth. Across diverse Transformer architectures in vision and nlp tasks, head-rich configurations consistently generate better-conditioned Jacobians and better optimization performance. Leveraging this insight, the authors desigend a new Transformer arhcitecture to use more attention head.
Critical strengths
- Extensive experiments are demonstrated for different backbones such as Vit and DINO, across vision and tasks. They effectively proves the theory proposed in the paper.
- The theory in the paper can provide some guidance on tranformer archetecture design.
Critical weaknesses
- In Theorem 3.1, authors assumed that 1) the singular value of the attention metrix A and the projection matrix P are positive 2) the attention matrix columns are independent, mean zero, Gaussian distributed. Why do they hold? Can authors provide theoretical or empirical evidence? If ithey do not hold, it undermines the contribution of the paper. Increasing attention head numbers will improve the preformance is something we already know in the community. I do not think we need to trade depth for heads. I think a more interesting problem is what is the optimal attention head under different neural net architecture, such as depth.
- Can we compute the Jacobian condition number for very large models, like more than 100B? If not, the theory has less impact in real-life use.
Questions for authors
- I suggest the author to reduce the number of subplots in figure 1. It is better to put different method in one plot when x axis and y axis are the same.
- How will the result change if use different optimiser (adamw vs muon), different attnetion kernel ( for example, flash attention)?
3 comments
6. How will the result change if use different optimiser (adamw vs muon) different attnetion kernel ( for example, flash attention)?
Our proposed changes are architectural. In response to the reviewer, we reran all experiments from Table 1 using the Muon optimizer. Across all eight model variants, the leaner configurations achieve comparable top-1 accuracy with substantially fewer parameters. Memory usage, iteration time (as this is computing the forward + backward pass time), and TFLOPs match those reported in Table 1 of the main paper.
Table 1 experiments rerun with Muon
| Model | Configuration | Top-1 (%) | Params |
|---|---|---|---|
| ViT-B | (12, 12, 4) baseline | 80.1 | 86M |
| ViT-B | (7, 16, 2) | 80.4 | 40M |
| ViT-L | (24, 16, 4) baseline | 79.3 | 304M |
| ViT-L | (13, 20, 2) | 79.5 | 124M |
| ViT-G | (40, 16, 48/11) baseline | 77.2 | 1B |
| ViT-G | (21, 24, 2) | 77.4 | 351M |
| XCiT-M | (24, 8, 4) baseline | 84.6 | 84M |
| XCiT-M | (13, 16, 2) | 84.7 | 47M |
| XCiT-L | (24, 16, 4) baseline | 85.1 | 189M |
| XCiT-L | (13, 20, 2) | 85.2 | 99M |
| DaViT-B | ((1, 1, 9, 1), (4, 8, 16, 32), 4) baseline | 84.8 | 87M |
| DaViT-B | ((1, 1, 5, 1), (4, 8, 20, 32), 2) | 84.8 | 45M |
| DaViT-L | ((1, 1, 9, 1), (6, 12, 24, 48), 4) baseline | 84.9 | 197M |
| DaViT-L | ((1, 1, 5, 1), (6, 12, 28, 48), 2) | 84.8 | 100M |
| DaViT-G | ((1, 1, 12, 3), (12, 24, 48, 96), 4) baseline | 82.8 | 1.4B |
| DaViT-G | ((1, 1, 7, 2), (6, 12, 52, 98), 2) | 83.3 | 621M |
Within the attention module, we vary the number of heads and retain the same per-head attention operation. The resulting configurations can use attention kernel implementations such as FlashAttention. We added a remark clarifying this in the main paper (lines 264–266).
Dear reviewer g4VC, we thank you very much for your review. Before the discussion period ends, we would like to check if our response has addressed your concerns?
We thank the reviewer for taking the time to review our paper and all the comments/questions they have raised. Below we answer all the weakness points and questions raised. We have also made explicit changes to the main paper, uploading a new version, in red coloured font so as to make it easy for the reviewer to check and track the changes.
1. In Theorem 3.1, authors assumed that 1) the singular value of the attention matrix A and the projection matrix P are positive 2) the attention matrix columns are independent, mean zero, Gaussian distributed. Why do they hold? Can authors provide theoretical or empirical evidence? If they do not hold, it undermines the contribution of the paper.
Thank you for your comment and question. We have explained the assumptions in remark 3.2 after the statement of theorem 3.1. In that remark, we also write that we empirically verify the assumptions in Appendix A.1.2. In Appendix A.1.2 we run several experiments that clearly verifies all the assumptions about the singular values of the attention matrix A and the projection matrix P (see Figs. 5 and 6). We have done this for both vision transformers and the language one. In all cases, we have empirically shown that our assumptions hold.
2. Increasing attention head numbers will improve the performance is something we already know in the community.
We thank the reader for their comment. However, no where in the paper do we claim increasing attention heads will improve performance. We claim that increasing attention heads will reduce the condition number of the attention Jacobians and this should lead to better convergence. We then show that this is indeed the case, see Fig. 1 and Fig. 3 in the paper, and we then hypothesize that since increasing heads gives the model better conditioning the model does not need so many parameters which we test by reducing the depth and MLP width. In almost all cases increasing attention heads leads to comparable performance which empirically is a by product of the better optimization given the better conditioning.
3. I do not think we need to trade depth for heads. I think a more interesting problem is what is the optimal attention head under different neural net architecture, such as depth.
Unfortunately we are not sure what exactly the reviewer means by this comment. If they can elaborate we will be more than happy to have a discussion. We in fact show that many transformers do not have the optimal head and depth allocation and by considering the conditioning of the attention Jacobian one can get much more optimal architectures that obtain comparable performance to standard baselines where optimal here means efficient i.e. less parameters, memory, training iteration time and TFLOPS.
4. Can we compute the Jacobian condition number for very large models, like more than 100B? If not, the theory has less impact in real-life use.
In principle, Jacobian condition numbers can be computed even for models with more than 100B parameters. The computational cost grows with model size, and training models at that scale and computing their Jacobian condition numbers would exceed our available compute capacity. However, we do have experiments that support the insights from the idealised analysis at the 1B-parameter scale. We use those insights to guide the design of smaller transformers that achieve performance comparable to that of larger baselines. We still believe that this is valuable insight for the community working on transformer pretraining.
5. I suggest the author to reduce the number of subplots in figure 1. It is better to put different method in one plot when x axis and y axis are the same.
We thank the reviewer for this suggestion. Since the plots in figure 1 show the same trend we feel it might in fact be better to move the second and third rows of the plots to the appendix. We will do this towards the end of the author-reviewer discussion period of the rebuttal process after we have included every reviewers comments from the discussion.