acceptodds
Under review as a conference paper at ICLR 2027

The Many Faces of Short Convolutions in Transformers

10 views

Abstract

Short causal convolutions improve the performance of sequence models, a benefit often attributed to the formation of single-layer induction heads. By disentangling the scaling and shifting operations within a convolution, we investigate their role from a **stability** perspective, showing that their benefits extend beyond expressivity. Starting from a simple channel-wise gain, we discover that convolutions mitigate rank-collapse by functioning as learnable **residual scaling**. However, this is not the only benefit for trainability. By extending the filter size, we show that convolutions alter the **positional information** learned through causal masks (NoPE), improving representational diversity. This enables training an otherwise unstable post-norm Transformers, while broadening the range of viable learning rates for pre-norm Transformers. Together, these findings identify channel-wise scaling and local temporal mixing as complementary contributors to the effectiveness of short convolutions.

Primary area:
optimization
Keywords:
Transformers, Short Convolutions, Rank Collapse, Positional Encoding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zwteOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why short causal convolutions help Transformers by separating their effects into channel-wise scaling and token shifting. The authors argue that scaling improves optimization stability, while token shifting changes how positional information is represented and can improve length generalization.

Critical strengths
  1. The paper studies a widely used architectural component from a mechanistic perspective rather than only reporting performance improvements.

  2. The experiments cover multiple convolution sizes, normalization schemes, and positional-encoding settings, and the paper also includes a theoretical result on the different roles of query, key, and value convolutions.

Critical weaknesses
  1. The role of the rank-collapse metric in Figure 2 is unclear. The paper treats lower mean pairwise cosine similarity as improved representational diversity and uses the reduction in this metric as evidence supporting the optimization benefits of convolutions. However, in Fig. 2(a,c), the different kernel sizes have very similar values in the final layers, even though their optimization behavior and performance differ. This shows that the metric is not by itself an indicator of representation quality or model performance. If its intended role is instead to diagnose harmful intermediate-layer optimization dynamics, the paper needs a direct controlled experiment showing that reducing this metric actually alleviates the proposed gradient/optimization problem. Fig. 9 only analyzes the k=0k=0 model and therefore does not establish this link.

  2. In Fig. 5, the authors compress a trained (k=4) filter by reassigning, merging, or even dropping its coefficients, and argue that this preserves channel-wise scaling while changing token shifting. I do not see why this is guaranteed. In particular, dropping coefficients can clearly change the overall scale of the filter, and the paper does not define what quantity is actually preserved by this intervention. Therefore, the resulting perplexity changes do not seem to isolate token shifting from other changes to the convolution. I also find the interpretation too strong. The paper uses a 15% perplexity-increase threshold to decide the minimum required kernel size, and then concludes that queries “do not utilize” token shifting because the best (k'=1) compression stays below this threshold. However, a nontrivial degradation of up to 15% does not seem sufficient to support such a categorical conclusion, and the choice of the 15% threshold is not justified.

Questions for authors

Is the filter-compression procedure in Fig. 5 (reassigning, merging, and dropping taps at inference time) adapted from prior work, or is it introduced in this paper? If it is introduced here, could you please explain the motivation behind this particular intervention and why it is an appropriate way to isolate token shifting from channel-wise scaling?

4 comments
AuthorsOpenReview ↗

Official response to reviewer zwte (1/2)

We thank Reviewer zwte for their thoughtful review. We appreciate their recognition of our mechanistic perspective, the range of experimental settings, and the theoretical result on query, key, and value convolutions, and hope to address the remaining questions below.

Regarding the role of rank-collapse and Figure 2

We ackownledge that in the original manuscript, the role of rank-collapse was not made clear. In the new version of the paper, we first provide an explanation of what rank-collapse is, and how it's measured (lines 183-192). In Figure 2, which is entirely dedicated to post-norm Transformers, we show a connection between trainability and a rank-collapse measure. Specifically, we show that if a run fails (the k=0k=0 baselines), it also suffers from rank-collapse. However, all of the convolution runs (k≥1k\geq 1) successfully train up to learning rate 4×10−34\times 10^{-3}. But we do not make any claims about rank-collapse being impacted by filter size, as Section 3 now clearly states that all the attribution goes to channel-wise scaling. This is due to the fact that even a k=1k=1 convolution can mitigate rank-collapse. Furthermore, we also do not make any claims that the kernel size is related to the trainability of post-norm Transformers. We consider any kernel size improvements to be mostly related to the results of Section 4. Futhermore, we note that the connection between rank-collapse and post-norm Transformers' trainability problems is already well established in the literature [1, 2]. Nevertheless, we agree that mean pairwise cosine similarity is not a general measure of representation quality or a predictor of performance across kernel sizes. We now use it specifically as a diagnostic of token alignment in the post-norm experiments.

In Proposition 1, we provide some theoretical motivation for convolutions acting as residual scaling factors. As residual scaling is proven to be an effective way of mitigating rank-collapse [1], this proposition motivated us to further experiment on the magnitude of the convolution filter at initialization. In Figure 3, we show that a post-norm Transformer is in fact very sensitive to initialization. At the presence of a small enough convolution filter, the post-norm Transformer successfully trains, while a large initial convolution filter impedes successful training. As rank-collapse is known to be a depth-related phenomenon, in Appendix C2 we provide experiment results for deeper (up to 96 Transformer blocks, or 192 layers) post-norm Transformers. We observe that convolutions still successfully manage to improve the trainability of post-norm Transformers, even at such high depth. In Appendix C3.1 (Figure 16(b,d)), we compare our results to various rank-collapse related interventions, including LayerScale, frozen k=1k=1 convolutions, and learned residual scales (learnable gains in the legend). We observe similar effect from these baselines, further proving that the observed trainability of post-norm Transformers is rank-collapse related.

We further disentangle the results of pre-norm Transformers in the paper from rank-collapse, investigating the learning rate robustness observed in pre-norm Transformers from another perspective. Specifically, in Figure 5 of the revised version of the paper we show that the k=1k=1 baseline, only applied to the query and the key representations, is enough to stabilize the model for the largest learning rate (1.6×10−21.6\times 10^{-2}). We compare this behavior to qk-normalization, and show similar effects in terms of training stability. We conjecture that this effect is related to sparsity (Figure 5d) and shrinking massive activations (Figure 4), which have been shown to be effective for learning rate robustness in the related literature [3, 4, 5].

AuthorsOpenReview ↗

Official response to reviewer zwte (2/2)

Regarding the filter compression experiment

We introduce this compression procedure in this work. In this experiment, we aimed to preserve the scaling behavior of the retained filter coefficients while changing their token-shifting behavior. One can think of several ways to preserve scaling, because a filter’s scaling behavior varies depending on the input. For instance, the filter ((1,−1)(1,-1)) suppresses a constant signal but amplifies an alternating signal. Motivated by the analysis in the RoPE paper [6], we chose a constant signal as our reference. For each channel, the constant-input response is determined by the sum of the filter coefficients. We therefore merge retained coefficients by addition, preserving their combined constant-input response. This is the specific sense in which our intervention preserves scaling; deliberately deleted coefficients are excluded from this guarantee. It does not guarantee unchanged activation magnitudes for arbitrary inputs.

For each target kernel size, we consider all candidate reassignments, mergers, and deletions of the learned coefficients and select the combination with the lowest validation perplexity. Our motivation is to allow the compressed filter to retain task-relevant channel-wise scaling wherever possible while modifying its token-shifting behavior. This avoids basing our conclusions on a single, arbitrary reassignment and measures how well model performance can be preserved within the candidate family.

The 15% threshold was chosen as a permissive tolerance intended to exclude severe performance degradation. It is a heuristic choice, without theoretical significance. Accordingly, the reported kernel size is the smallest tested size satisfying this tolerance, rather than a definitive measure of whether a component utilizes token shifting.

Regarding the claim about the query vectors not utilizing the token shifting behavior, we note that the increase in perplexity in this specific case is actually 4.2%4.2\% and 1.8%1.8\% in the NoPE and RoPE models, respectively. This value is much smaller than the threshold we picked for this experiment (15%15\%), and perplexity in the NoPE model only increases from 15.2315.23 to 15.9915.99. However, ultimately the choice of a threshold is arbitrary, and we emphasize that the only purpose this experiment serves is to show that some components are much more sensitive to token shifting than other components. We therefore interpret this experiment as measuring sensitivity to filter compression, rather than as a strict isolation of token shifting from scaling. We have modified the language in the paper accordingly (lines 356-357, lines 360-367).

We again thank Reviewer zwte for their thoughtful review and hope to have addressed their questions.


References

[1] Noci, L., Anagnostidis, S., Biggio, L., Orvieto, A., Singh, S.P. and Lucchi, A., 2022. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse.

[2] Dong, Y., Cordonnier, J.B. and Loukas, A., 2021, July. Attention is not all you need: Pure attention loses rank doubly exponentially with depth.

[3] Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A.P., Caron, M., Geirhos, R., Alabdulmohsin, I. and Jenatton, R., 2023, July. Scaling vision transformers to 22 billion parameters.

[4] Wortsman, M., Liu, P., Xiao, L., Everett, K., Alemi, A., Adlam, B., Co-Reyes, J.D., Gur, I., Kumar, A., Novak, R. and Pennington, J., 2024, May. Small-scale proxies for large-scale transformer training instabilities.

[5] Sun, S., Canziani, A., LeCun, Y. and Zhu, J., 2026. The spike, the sparse and the sink: Anatomy of massive activations and attention sinks.

[6] Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W. and Liu, Y., 2024. Roformer: Enhanced transformer with rotary position embedding.

Thank you for the detailed response and the additional experiments. My concerns are largely addressed by the revised manuscript. I am raising my rating from 2 to 3.

One small suggestion: Next time, please try to submit your rebuttal earlier. Submitting it just 2 hours before the deadline makes it difficult for reviewers to evaluate the response and follow up.

Dear reviewer, thank you very much for your patience and for being so considerate, even in the face of such suboptimal timing from our side. We are happy to have largely addressed your concerns and aim to submit our rebuttal earlier next time.

Reviewer_yzgPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work the authors investigate how short convolutions improve transformers by separating the effects of channel-wise scaling and temporal mixing. The paper shows that scaling improves optimization stability while temporal mixing improve the length generatlization and alters how positional information is represented.

Critical strengths
  1. This is a very interesting research question and the authors probe to find out why short convolutions improve transformers beyond their known expressivity benefits.
  2. The separation of channel-wise scaling and temporal mixing provides useful insights into the different benefits of convolutions.
  3. Comprehensive experiments: The authors evaluate different kernel sizes, normalization methods, positional encodings, and learning rates. The theoretical analysis provided also has useful insights!
  4. The mechanistic analysis i.e representation collapse, filter compression, and PCA provide interesting perspectives on how convolutions affect transformer behaviour
Critical weaknesses
  1. Some optimization baselines are missing. How does the convolutional scaling compare against very simple alternatives like residual scaling, layerscale or gating ?
  2. The downstream gains are somewhat modest and for some configuraitons, the convs actually degrade recall while improving ppl.
  3. Results are only at 340M and 10B tokens. Not asking the authors to train larger than that as I understand that could be computationally demanding but I wonder if these gains just washup as you scale ? do they not work at 100M scale ? why do you not include those.
  4. It would also be great to know the computation overhead introduced by the convolutional modifications.
Questions for authors
  1. I wonder how K=1 convolutions for example would compare against simpler residual scaling or gating ? Just wondering if the the learned convolution scaling is actually provides additional benefits.

  2. Do the optimization and positional encoding benefits actually persist as we scale e.g training and FLOPS ? I could see these gains not mattering much at scale.

  3. Could the authors provide more insight on why some of the conv configuratiosn degrade recall while improving ppl ?

  4. For long and short convs how do they compare with computation cost vs performance improvements.

2 comments
AuthorsOpenReview ↗

Official response to reviewer yzgP (1/2)

We thank Reviewer yzgP for their thoughtful review. We appreciate their recognition of the insights from separating channel-wise scaling and temporal mixing and the breadth of our experiments, and hope to address the remaining questions below.

W1/Q1: Residual-scaling and gating baselines

We thank the reviewer for the very interesting and relevant question. In our revised manuscript we comapred the following baselines: 1. learnable scalar residual scaling, that scales the output of each residual branch; 2. LayerScale [1], which learns one gain per channel; and 3. output gating, which is applied to the attention output.

We analyse them in the context of learning rate robustess, which is indicative of trainability, especially at scale [2], and evaluate them on various downstream tasks. For the experimental setting, results and discussion, we refer the reviwer to the Appendix C.3 (Convolution-free Baselines). Here, we highlight the most important conlcusions about the comparison of the convolution variants and the mentioned convolution-free baselines.

Regarding optimization/trainability: We note that there are a few of the checkpoints are currently missing in this experiment (Figure 16). However, they do not change our conclusion. We will upload them in the camera-ready version of the paper. For the discussion here we focus on the post-norm Transformer variants, as this is the most challenging setting for optimization, where the model without any residual scaling diverges at every learning rate. For the exact results that support the conclusions below, we refer the reviewer to Figure 16 in the revised manuscript.

  1. Learned rescaling of the residual branches is enough to make the model trainable. A learnable scalar gain and LayerScale both stabilize it, as the k=1 convolution does.
  2. Although the baselines outperform convolutional scaling (k=1k=1), rescaling alone does not give the best perplexity: no scaling baseline matches the convolutions that also mix tokens (k≥2k \geq 2).
  3. No baseline widens the range of trainable learning rates beyond that of the convolutions across the different normalization and positional encodings. LayerScale matches it at best, while the scalar gain and the output gate diverge at learning rates where all convolution models still train. We conclude that rescaling makes the model trainable, token mixing is needed for the best perplexity, and convolutions provide both. Furthermore, in Section 3.2 we investigate a possible mechanism through which this learning rate robustness is achieved.

Regaring the downstream task evaluation and length generalization From the updated Table 1 with added performances of the mentioned baselines we conclude that while convolution-free residual scaling methods are strong baselines, with them outperforming the convolutional scaling, the best preformance on average is obtained with convolutions. The general trend in these results indicate that larger filter sizes improve performance, although the optimal kernel size can vary depending on the task. Furthermore, we note that the best-performing model in terms of average accuracy is a RoPE model with k=4k=4 convolution, which indicates further benefits for how convolutions make post-norm trainable.

W2/Q3: Recall and downstream performance

We thank the reviewer for noticing this issue. Upon inspecting the evaluaiton, we descovered a bug in the convolution implementation which only impacted the short convolution results. The bug is related to mishandling the token masks for short convolutions. Since in the recall tasks, the sequences are attached to each other and the task involves a generative process (no prefill), not passing the masks to the short convolution kernel would result in the history from the previous sequences in the batch leaking into the next sequences that are concatenated to them. As these tasks were the only ones evaluated by concatenation, the recall performances in the Table 1 were the only results in the paper impacted by this bug. Upon fixing the bug, we observe the expected gradual improvement with larger kernel sizes in these tasks as well. The changes are reflected in the updated Table 1.

W3/Q2: Smaller-model results and the scope of scaling evidence

In the additional experiments in Appendix C.4 (Scaling the model size), we assess whether the optimization and positional encoding benefits also apply, both to the smaller (≈115\approx 115M) and bigger (≈1.36\approx 1.36B-parameter) models. The smaller model was trained on 10B tokens and the bigger one on 30B tokens of the FineWeb dataset.

AuthorsOpenReview ↗

Official response to reviewer yzgP (2/2)

Optimization benefits For the smaller model size, we did extensive experiments while also assessing learning rate robustness (shown in Figure 18). The conclusions at the smaller scale are quite consistent with the main 340M models. As the kernel size grows, the validation perplexity and the learning rate robustness are improving. The ranking of the models stays the same compared to the bigger setting. Similarly, in Table 2, we show that the downstream task performance is generally improved with the model variants with convolutions. Additionally, we see that the best performance per each task is most often obtained with the post-norm model with RoPE, crucially, with short-convolutions on. In the setting of the 1B-parameter models, we show that the post-norm model with k=1k=1 can be trained successfully, further establishing the conclusion of Section 3.1 across scales. Following the smaller-scale conclusion, we also see that the performance improves once the convolutional token mixing is added, both in case of the pre-norm and post-norm models on language modeling, which translates into a better downstream task performance.

Positional encoding No matter the scale of the model (115M, 340M, 1.36B), the long convolution filters (k=∞\infty) NoPE models are the only ones not to experience severe performance drop on the lenght generalizaiton task.

W4/Q4: Computation overhead and quality

In Figure 21 of the revised manuscript, we provide an analysis on the convolution implementations used in this paper. As the cost of the convolution operations increases (at most) linearly with sequence length, compared to the quadratic cost of attention, for a long enough sequence the overhead becomes negligible.

The addional paramters of the short convolutions used in this paper are calculated as k×Ck\times C, where kk is the kernel size and CC the number of channels. For the long convolution it is 3×C3 \times C, as the long convolution filter is defined by three learned scalars per channel. Hence, the size of the additional parameters are neglible.

We again thank Reviewer yzgP for their thoughtful review and hope to have addressed their questions.


References

[1] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Herv´e J´egou. Going deeper with image transformers.

[2] Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie E. Everett, Alexander A. Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Small-scale proxies for large-scale transformer training instabilities.

Reviewer_PVMvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a suite of experiments showing that, for a particular dataset and a particular Transformer architecture, adding short causal convolutions (even with k=1, i.e., with just scaling) produces broad benefits.

The paper proposes several plausible mechanistic hypotheses for these phenomena.

Critical strengths
  • The paper presents an interesting set of experiments and an interesting set of mechanistic hypotheses
Critical weaknesses
  • I found the writing quite difficult to follow. The paper mixes experiments with theoretical derivations that are merely suggestively connected to the experiments, sometimes in the same sections

  • If the paper's claim were "short causal convolutions improve Transformer training and we understand why", this would be an important and significant paper, but would need much more empirical and theoretical validation. As it is, the authors show empirical evidence that for a particular architecture, even scaling appears to have broad benefits, and hypothesize but don't prove mechanistic reasons for that. I am not sure about the broad significance

  • The above point could, of course, apply to many MechInterp papers. But I think this paper could benefit from being split up into more in-depth investigation of the separate claims, and particularly from devising experiments that firm up the evidence for the mechanistic hypotheses.

Questions for authors
  • I acknowledge that I may have misunderstood the framing. Can you justify the organization of the paper?

  • I think we'd agree that many of the mechanistic claims are more conjectured rather than proved. Can you point to the mechanistic claims that you think hold up the best?

  • One criticism of the paper is that it was already known that similar techniques improve optimization performance in similar architectures. How would you distinguish your contribution?

5 comments
AuthorsOpenReview ↗

Official response to reviewer PVMv (1/3)

We thank Reviewer PVMv for their thoughtful review. We appreciate their recognition of our experiments and mechanistic hypotheses, and hope to address the remaining questions below.

Regarding the organization and the writing of the paper

We acknowledge that the original submission required improvements with its organization. In the revised version, we have substantially re-organized Section 3, improved the flow of Section 4, and provided further supporting experiments for the claims within the paper.

We dedicate Section 3 entirely to investigating the benefits of channel-wise scaling on optimization. The section considers two separate normalization choices: post-norm and pre-norm. Given that post-norm suffers from instability due to rank-collapse [1], the paper first provides evidence for convolutions making post-norm trainable, and this trainability being accompanied by mitigated rank-collapse. Then, the paper draws a theoretical line between channel-wise scaling through convolutions and residual scaling, the solution to rank-collapse. This relationship is further supported by showing that the observed trainability is highly impacted by initialization (Figure 3). Namely, the trainability benefits for post-norm only persist when the convolutions are initialized small enough. In Appendix C2, the same experiment is performed on Transformers with up to 96 Transformer blocks (192 layers), showing that even at large depth, the trainability of convolution augmented post-norm Transformers persists. In Appendix C3.1 (Figure 16(b,d)), we compare our results to various rank-collapse related interventions, including LayerScale, frozen k=1k=1 convolutions, and learned residual scales (learnable gains in the legend). We observe similar effect from these baselines, further proving that the observed trainability of post-norm Transformers is rank-collapse related.

We further disentangle the results of pre-norm Transformers in the paper from rank-collapse, investigating the learning rate robustness observed in pre-norm Transformers from another perspective. Specifically, in Figure 5 of the revised version of the paper we show that the k=1k=1 baseline, only applied to the query and the key representations, is enough to stabilize the model for the largest learning rate (1.6×10−21.6\times 10^{-2}). We compare this behavior to qk-normalization, and show similar effects in terms of training stability. We conjecture that this effect is related to sparsity (Figure 5d) and shrinking massive activations (Figure 4), which have been shown to be effective for learning rate robustness in the related literature [1, 2, 3].

Section 4, on the other hand is focused on the role of token shifting in convolutions, and we hope the reorganization will improve the readability of this section. We start the section by showing that short convolutions with a filter size of 2 can encode position through either the key or the value in an attention layer. In order to investigate whether the model is actually using these capabilities or not, we do the compression experiment in Figure 6, where we show that in the presence of explicit positional encoding, a trained model becomes much less sensitive to interventions on the token shifting capabilities of the key and value convolutions. We treat these results as evidence supporting the use of convolutions as positional encodings by these components.

Further investigation is done by following the method introduced by [4], wherein a set of position dependent directions involved in positional encoding in a pre-norm NoPE Transformer are extracted and investigated in the models with and without convolutions or RoPE. We observe that in the presence of explicit positional encoding (RoPE) or convolutions, the sensitivity of the model to removing these directions from the representations are reduced, and with larger kernel sizes the behavior of the model becomes gradually more similar to RoPE when it comes to these directions. We conclude the section by noting the positive correlation between filter size, length generalization, and downstream performance (although the optimal kernel size can vary depending on the task).

AuthorsOpenReview ↗

Official response to reviewer PVMv (2/3)

Which mechanisms are most strongly supported

We believe the results of Section 3.1, namely channel-scaling being responsible for the trainability of post-norm Transformers, is the most strongly supported result in the paper. We show that by removing the token-shfiting capabilities from the convolutions (focusing on k=1k=1), showcasing the sensitivity of the network to the magnitude of the initialization of the convolution filter (a behavior simialr to residual-scaling), and providing further results showing that most complications arising from larger depth (the cause of rank-collapse [5]) are overcome by convolutions. We provide theoretical results showing that convolutions can act are residual scaling factors, a well-known solution to rank-collapse. We also provide additional experiment results with different scales (1.36B and 115M parameters), showing that the behavior persists, while also emprically showing that how convolutions behave with respect to post-norm is similar to other rank-collapse related interventions like layer scale and residual scaling (Figure 16). Therefore, we believe that these interventions establish the sufficiency of scaling, with the rank-collapse measurements and existing theory supporting our interpretation through residual scaling.

The second most well-supported mechanism in our opinion is that similar to qk-norm, convolutions, solely due to channel-wise scaling, can provide training stability in pre-norm Transformers. We do so by comparing the behavior to qk-normalization, showing that a similar sparsity inducing effect in the query and key representations emerges in both. As in this experiment the only intervention in the architecture is done on the query and key representations, and since sparsity has been linked to more stable training [3], and since this sparsity effect has been shown to be caused by the weights themselves (Figure 10), we conclude that this sparsity effect could be the main source of stability in the model. We further strengthen this link by looking into how convolutions shrink massive activations (Figure 4), which are also linked to instability during training [3].

Finally, we think the results of Section 4 are next in terms of support and strength of argument. We note that for token shifting, Proposition 2 provides an exact result under its assumptions: a convolution filter of size k=2k=2 on either the key or value suffice to encode position. This establishes a posiblity that the model may use convolutions for positional encoding, which we show evidence for by doing the intervention in Figure 6 and the investigation in Figure 7. Both of these results show that the positional encoding behavior of the model somehow interacts with convolutions and their filter sizes, showing some form of a (at least) correlational relationship between them.

Distinction from existing techniques and broader significance

We acknowledge that the techniques and methods investigated in the paper are in no way novel or new. While the overall impact of convolutions in sequence models is established, as in [6, 7, 8], the mechanisms through which they introduce these benefits is not. Therefore, the paper focuses on explaining how convolutions improve the performance of a sequence model by separating the role of channel-wise scaling from the role of token shifitng. We believe this dichotomy is novel within the literature, and the findings, namely the relationship to residual scaling, qk-normalization, and positional encoding, have not been established as explicitly in the literature previously as they have been in this paper.

Furthermore, in the revision of the paper these conclusions have been further strengthened by showing that they persist across scale, depth, and when compared to baselines. However, we acknowledge that the training experiments remain within one Transformer family and one training corpus.

We again thank Reviewer PVMv for their thoughtful review and hope to have addressed their questions.

AuthorsOpenReview ↗

Official response to reviewer PVMv (3/3)


References

[1] Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A.P., Caron, M., Geirhos, R., Alabdulmohsin, I. and Jenatton, R., 2023, July. Scaling vision transformers to 22 billion parameters.

[2] Wortsman, M., Liu, P., Xiao, L., Everett, K., Alemi, A., Adlam, B., Co-Reyes, J.D., Gur, I., Kumar, A., Novak, R. and Pennington, J., 2024, May. Small-scale proxies for large-scale transformer training instabilities.

[3] Sun, S., Canziani, A., LeCun, Y. and Zhu, J., 2026. The spike, the sparse and the sink: Anatomy of massive activations and attention sinks.

[4] Sun, S., Canziani, A., LeCun, Y. and Zhu, J., 2026. The spike, the sparse and the sink: Anatomy of massive activations and attention sinks.

[5] Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W. and Liu, Y., 2024. Roformer: Enhanced transformer with rotary position embedding.

[6] Zeyuan Allen-Zhu, 2025. Physics of language models: Part 4.1, architecture design and the magic of canon layers

[7] Li, M., Zhang, X., Huang, Y. and Oymak, S., 2025, April. On the power of convolution-augmented transformer. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 17, pp. 18393-18402).

[8] Arora, S., Eyuboglu, S., Timalsina, A., Johnson, I., Poli, M., Zou, J.Y., Rudra, A. and Ré, C., 2024, May. Zoology: Measuring and improving recall in efficient language models. In International conference on learning representations (Vol. 2024, pp. 15664-15730).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global statement by the authors

We thank the reviewers for their thoughtful, constructive, and encouraging reviews. We appreciate your patience as we finalized the additional experiments and revisions during the rebuttal period. Your questions and suggestions have substantially strengthened the paper. Namely, the suggestions for further experimentation strengthened the results and the arguments in the paper. We further improved the presentation and the analysis in the paper as a result of the reviewers' comments. The main points of revision are summarized below. We highlight the changes in the paper in blue.

  1. Experiments across model scales (Reviewer yzgP). We repeated the main experimental sweep (Figures 2 and 5 in the current manuscript) with 115M-parameter models and added experiments with 1.36B-parameter models trained on 30B tokens, extending the original 340M-parameter study. These experiments provide additional evidence that the analysis and the conclusions in the paper hold across scale.

  2. Additional baselines and controls (Reviewer yzgP). We added further baselines to our experiments. Specifically, we experiment with learnable scalar residual gains, LayerScale, attention output gating, and frozen gain-only convolutions. These baselines are introduced to provide a point of reference for the results, while also connecting the analysis in the paper to previous work.

  3. Deeper optimization analysis (Reviewers PVMv and zwte). We added a theoretical analysis of residual scaling at initialization, while further clarifying the role of rank collapse in post-norm trainability. For pre-norm Transformers, we show that convolutions behave similar to QK-normalization, which could be a potential explanation for their learning-rate robustness benefit. These behaviors are related to query/key sparsity and reduced activation outliers.

  4. More detailed evaluation (Reviewer yzgP). We corrected a masking issue affecting short-convolution recall evaluation in the downstream table (Table 1). We also investigated SQuAD-completion’s sensitivity to repeated leading-space tokens, with the goal of further understanding why convolutions perform well on this specific recall task. We also added throughput measurements to quantify the computational overhead of the convolution implementations (both short and long) used in our experiments.

  5. Clearer organization and interpretation (Reviewers PVMv and zwte). We reorganized the presentation for better clarity. The main sections, namely Sections 3 and 4, investigate channel-wise scaling and how it interacts with optimization, followed by token shifting and its relationship to positional information, respectively. We clarified what the filter-compression intervention preserves, while also qualifying the conclusions drawn from the this experiment as it is impacted by our set threshold on the change sin perplexity. We also carefully distinguish the mechanistic evidence from hypotheses, only making claims about identified mechanisms at the presence of enough evidence. The revised paper strengthens our central message: the distinguishable role of channel-wise scaling and token shifting. We show that these two separate behaviors make complementary contributions to the benefits of convolutions in Transformers. With a suitable initialization, scaling improves trainability in post-norm Transformers. In pre-norm Transformers, scaling partly impacts learning rate robustness, with a mechanism that could be similar to QK-norm. On the other hand, token shifting contributes additional benefits, which may partly be related to how convolutions alter the positional encoding capabilities of the model. The expanded results also clarify that the preferred kernel size depends on the task. However, the general trend is that longer convolution filters provide better length extrapolation and downstream performance.

We greatly appreciate all three reviewers’ suggestions and comments, as they directly motivated these additions. We believe these additions have made the paper substantially clearer, more thorough, with the claims better supported.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.