- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies why short causal convolutions help Transformers by separating their effects into channel-wise scaling and token shifting. The authors argue that scaling improves optimization stability, while token shifting changes how positional information is represented and can improve length generalization.
Critical strengths
-
The paper studies a widely used architectural component from a mechanistic perspective rather than only reporting performance improvements.
-
The experiments cover multiple convolution sizes, normalization schemes, and positional-encoding settings, and the paper also includes a theoretical result on the different roles of query, key, and value convolutions.
Critical weaknesses
-
The role of the rank-collapse metric in Figure 2 is unclear. The paper treats lower mean pairwise cosine similarity as improved representational diversity and uses the reduction in this metric as evidence supporting the optimization benefits of convolutions. However, in Fig. 2(a,c), the different kernel sizes have very similar values in the final layers, even though their optimization behavior and performance differ. This shows that the metric is not by itself an indicator of representation quality or model performance. If its intended role is instead to diagnose harmful intermediate-layer optimization dynamics, the paper needs a direct controlled experiment showing that reducing this metric actually alleviates the proposed gradient/optimization problem. Fig. 9 only analyzes the model and therefore does not establish this link.
-
In Fig. 5, the authors compress a trained (k=4) filter by reassigning, merging, or even dropping its coefficients, and argue that this preserves channel-wise scaling while changing token shifting. I do not see why this is guaranteed. In particular, dropping coefficients can clearly change the overall scale of the filter, and the paper does not define what quantity is actually preserved by this intervention. Therefore, the resulting perplexity changes do not seem to isolate token shifting from other changes to the convolution. I also find the interpretation too strong. The paper uses a 15% perplexity-increase threshold to decide the minimum required kernel size, and then concludes that queries “do not utilize” token shifting because the best (k'=1) compression stays below this threshold. However, a nontrivial degradation of up to 15% does not seem sufficient to support such a categorical conclusion, and the choice of the 15% threshold is not justified.
Questions for authors
Is the filter-compression procedure in Fig. 5 (reassigning, merging, and dropping taps at inference time) adapted from prior work, or is it introduced in this paper? If it is introduced here, could you please explain the motivation behind this particular intervention and why it is an appropriate way to isolate token shifting from channel-wise scaling?
4 comments
Official response to reviewer zwte (2/2)
Regarding the filter compression experiment
We introduce this compression procedure in this work. In this experiment, we aimed to preserve the scaling behavior of the retained filter coefficients while changing their token-shifting behavior. One can think of several ways to preserve scaling, because a filter’s scaling behavior varies depending on the input. For instance, the filter () suppresses a constant signal but amplifies an alternating signal. Motivated by the analysis in the RoPE paper [6], we chose a constant signal as our reference. For each channel, the constant-input response is determined by the sum of the filter coefficients. We therefore merge retained coefficients by addition, preserving their combined constant-input response. This is the specific sense in which our intervention preserves scaling; deliberately deleted coefficients are excluded from this guarantee. It does not guarantee unchanged activation magnitudes for arbitrary inputs.
For each target kernel size, we consider all candidate reassignments, mergers, and deletions of the learned coefficients and select the combination with the lowest validation perplexity. Our motivation is to allow the compressed filter to retain task-relevant channel-wise scaling wherever possible while modifying its token-shifting behavior. This avoids basing our conclusions on a single, arbitrary reassignment and measures how well model performance can be preserved within the candidate family.
The 15% threshold was chosen as a permissive tolerance intended to exclude severe performance degradation. It is a heuristic choice, without theoretical significance. Accordingly, the reported kernel size is the smallest tested size satisfying this tolerance, rather than a definitive measure of whether a component utilizes token shifting.
Regarding the claim about the query vectors not utilizing the token shifting behavior, we note that the increase in perplexity in this specific case is actually and in the NoPE and RoPE models, respectively. This value is much smaller than the threshold we picked for this experiment (), and perplexity in the NoPE model only increases from to . However, ultimately the choice of a threshold is arbitrary, and we emphasize that the only purpose this experiment serves is to show that some components are much more sensitive to token shifting than other components. We therefore interpret this experiment as measuring sensitivity to filter compression, rather than as a strict isolation of token shifting from scaling. We have modified the language in the paper accordingly (lines 356-357, lines 360-367).
We again thank Reviewer zwte for their thoughtful review and hope to have addressed their questions.
References
[1] Noci, L., Anagnostidis, S., Biggio, L., Orvieto, A., Singh, S.P. and Lucchi, A., 2022. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse.
[2] Dong, Y., Cordonnier, J.B. and Loukas, A., 2021, July. Attention is not all you need: Pure attention loses rank doubly exponentially with depth.
[3] Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A.P., Caron, M., Geirhos, R., Alabdulmohsin, I. and Jenatton, R., 2023, July. Scaling vision transformers to 22 billion parameters.
[4] Wortsman, M., Liu, P., Xiao, L., Everett, K., Alemi, A., Adlam, B., Co-Reyes, J.D., Gur, I., Kumar, A., Novak, R. and Pennington, J., 2024, May. Small-scale proxies for large-scale transformer training instabilities.
[5] Sun, S., Canziani, A., LeCun, Y. and Zhu, J., 2026. The spike, the sparse and the sink: Anatomy of massive activations and attention sinks.
[6] Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W. and Liu, Y., 2024. Roformer: Enhanced transformer with rotary position embedding.
Thank you for the detailed response and the additional experiments. My concerns are largely addressed by the revised manuscript. I am raising my rating from 2 to 3.
One small suggestion: Next time, please try to submit your rebuttal earlier. Submitting it just 2 hours before the deadline makes it difficult for reviewers to evaluate the response and follow up.
Dear reviewer, thank you very much for your patience and for being so considerate, even in the face of such suboptimal timing from our side. We are happy to have largely addressed your concerns and aim to submit our rebuttal earlier next time.
Official response to reviewer zwte (1/2)
We thank Reviewer zwte for their thoughtful review. We appreciate their recognition of our mechanistic perspective, the range of experimental settings, and the theoretical result on query, key, and value convolutions, and hope to address the remaining questions below.
Regarding the role of rank-collapse and Figure 2
We ackownledge that in the original manuscript, the role of rank-collapse was not made clear. In the new version of the paper, we first provide an explanation of what rank-collapse is, and how it's measured (lines 183-192). In Figure 2, which is entirely dedicated to post-norm Transformers, we show a connection between trainability and a rank-collapse measure. Specifically, we show that if a run fails (the k=0 baselines), it also suffers from rank-collapse. However, all of the convolution runs (k≥1) successfully train up to learning rate 4×10−3. But we do not make any claims about rank-collapse being impacted by filter size, as Section 3 now clearly states that all the attribution goes to channel-wise scaling. This is due to the fact that even a k=1 convolution can mitigate rank-collapse. Furthermore, we also do not make any claims that the kernel size is related to the trainability of post-norm Transformers. We consider any kernel size improvements to be mostly related to the results of Section 4. Futhermore, we note that the connection between rank-collapse and post-norm Transformers' trainability problems is already well established in the literature [1, 2]. Nevertheless, we agree that mean pairwise cosine similarity is not a general measure of representation quality or a predictor of performance across kernel sizes. We now use it specifically as a diagnostic of token alignment in the post-norm experiments.
In Proposition 1, we provide some theoretical motivation for convolutions acting as residual scaling factors. As residual scaling is proven to be an effective way of mitigating rank-collapse [1], this proposition motivated us to further experiment on the magnitude of the convolution filter at initialization. In Figure 3, we show that a post-norm Transformer is in fact very sensitive to initialization. At the presence of a small enough convolution filter, the post-norm Transformer successfully trains, while a large initial convolution filter impedes successful training. As rank-collapse is known to be a depth-related phenomenon, in Appendix C2 we provide experiment results for deeper (up to 96 Transformer blocks, or 192 layers) post-norm Transformers. We observe that convolutions still successfully manage to improve the trainability of post-norm Transformers, even at such high depth. In Appendix C3.1 (Figure 16(b,d)), we compare our results to various rank-collapse related interventions, including LayerScale, frozen k=1 convolutions, and learned residual scales (learnable gains in the legend). We observe similar effect from these baselines, further proving that the observed trainability of post-norm Transformers is rank-collapse related.
We further disentangle the results of pre-norm Transformers in the paper from rank-collapse, investigating the learning rate robustness observed in pre-norm Transformers from another perspective. Specifically, in Figure 5 of the revised version of the paper we show that the k=1 baseline, only applied to the query and the key representations, is enough to stabilize the model for the largest learning rate (1.6×10−2). We compare this behavior to qk-normalization, and show similar effects in terms of training stability. We conjecture that this effect is related to sparsity (Figure 5d) and shrinking massive activations (Figure 4), which have been shown to be effective for learning rate robustness in the related literature [3, 4, 5].