acceptodds
Under review as a conference paper at ICLR 2027

The Convex Corridor: Learning Attention–Laplacian Mixtures Helps Pretraining

Abstract

Attention computes input-dependent weighted averages of token vectors through the softmax probability matrix . Recently proposed Laplacian heads instead use , where is the identity matrix, to compute the deviation of each token vector from its corresponding average. Although using both Laplacian and attention heads in transformers has been shown to improve performance on many tasks, how to best combine the two operators is still unclear. We address this problem by parameterizing each head's operator as , where are learnable parameters. This modification consistently leads to lower validation loss in causal language modelling pretraining across tested model scales. To understand why, we examine the evolution of and throughout training, and find that most heads beyond the first layer learn , an interval we call the convex corridor. When falls within this corridor, , so each head's operator becomes a scaled convex combination of attention and the negative Laplacian , with controlling the weights of the two operators and controlling the size of the head's output. This pattern emerges across different initializations and alternative parameterizations of the operator such as and , which also improve pretraining. We perform a series of controlled experiments to show that being in the convex corridor contributes to the improvements in pretraining: restricting increases loss, while fixing to be some value within the convex corridor already improves upon standard attention even with fixed to 1. Together, these findings suggest that attention should be viewed as one endpoint of the convex corridor, and that exploring the corridor’s interior is a promising direction for future transformer architectures.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Transformers, attention mechanisms, graph Laplacians, language model pretraining

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_mo8iOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes an alternative parameterization of attention heads in LLMs. Given a softmax probability matrix PP corresponding to a particular attention head and input, the proposal is to parameterize the head by δ(γI+P)\delta (\gamma I + P) where γ,δ\gamma, \delta are learnable scalar parameters. They show that this usually learns γ∈(−1,0)\gamma \in (-1, 0) so that the head operator is a scaled convex combination of attention PP and the negative Laplacian P−IP - I. They perform several ablations to show that this helps in LLM pretraining given appropriate engineering on Nanochat-scale models.

Critical strengths
  • The idea seems to work, since the validation loss seems to improve; it also seems to improve by a larger relative amount at larger scales (see Figure 1).
  • The empirical analysis and ablations are very thorough, demonstrating how the new γ\gamma and δ\delta change, defending the particular parameterization, etc.
Critical weaknesses
  • There's not that much intuition about why such a parameterization helps.
  • It may be hard given the authors' resource limits, but a scaling study would be much more solid evidence for this architecture; how does it do at larger scales, what's the scaling law, etc. Two points cannot fit such a scaling law.
Questions for authors
  • Why does convex combination of attention pattern and its Laplacian help? Can there basically be a cheaper implementation (such as with a bias) that obtains the same or equivalent outcome?
Reviewer_dCazOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a novel method to combine vanilla softmax attention PP and the negative Laplacian P−IP - I into an operator δ(γI+P)\delta (\gamma I + P) that uses learnable parameters at each attention head. Doing so, the authors show that, for most heads beyond the first layer, γ\gamma tends to evolve to stay within what they call the ``convex corridor'', i.e. γ∈(−1,0)\gamma \in (-1, 0), since this makes the parameterization a scaled convex interpolation between PP and P−IP - I. In addition, γ∈(−1,0)\gamma \in (-1, 0) improves the loss of decoder-only models when optimizing a language modeling objective. They also compare their parameterization to other equivalent parameterizations and report an improvement on those methods. Finally, the authors apply interventions to the parameters during training and show that:

  1. compared to the unrestricted model, constraining γ∉(−1,0)\gamma \notin (-1, 0) increases loss;
  2. fixing γ\gamma in that range lowers loss compared to standard attention;
  3. learning or keeping δ\delta fixed has little impact when keeping γ\gamma fixed; and
  4. learning δ\delta is slightly better than keeping it fixed when learning γ\gamma, for some initialization values γ0\gamma_0.
Critical strengths

The arguments and narrative of the paper are laid out clearly, just like the explanations of the methods and the experiments. The paper is well structured and the figures are easy to analyze.

The effect of γ\gamma on the BPB metric is interesting. Section 7.2 provides insight that past work by He et al. (2024) had not considered (they only used nonnegative γ\gamma). Here, Figure 7 is especially insightful and summarizes well the effectiveness of the method.

Critical weaknesses

The main problems I found concern the novelty of the parameterization and the fairness and reliability of the BPB comparisons.

The central point of the paper is the parameterization δ(γI+P)\delta (\gamma I + P); however, it is a reparameterization of the learned identity–attention mixture used in Value-SkipInit (He et al., 2023, Eq. 7), Attn⁡(X)=(αI+βA(X))V(X)\operatorname{Attn}(X)=\bigl(\alpha I+\beta A(X)\bigr)V(X), assuming β≠0\beta \neq 0. The authors do show the equivalence between the two parameterizations in Section 5, but they do not cite the related work there. Although these works are already cited in Section 2, in Section 5 it would be appropriate to cite Value-SkipInit, He et al. (2023) beside α0I+α1P\alpha_0 I+\alpha_1P at line 254 and FDAM (Chen et al., 2025) and FAiT (He et al., 2026) beside αP+β(I−P)\alpha P+\beta(I-P) at line 263 since they combine PP and I−PI-P.

As to the comparison between parameterizations of Section 6, using the Laplacian I−PI - P in every head is only one configuration of the prior method of Zhang & Papyan (2026). This means the authors are not comparing their method to the best config of Zhang & Papyan (2026), but to a configuration that is suboptimal in terms of the downstream language-model results of Zhang & Papyan (2026).

Still on this point, in Figure 4(a,b) the initialization matches the one used for Figures 2(a), 3(a), since δ(γI+P)=P+I\delta(\gamma I+P) = P + I for δ=1\delta = 1 and γ=1\gamma = 1. In Figures 2(b), 3(b) γ=0\gamma = 0 for layer 0, but in Figures 4(c,d) γ=−1\gamma = -1 for layer 0. Is this discrepancy intentional? This also matters for Figure 5, because the authors' method uses PP in layer 0 and P−IP-I in later layers, while α0I+α1P\alpha_0 I + \alpha_1 P and αP+β(I−P)\alpha P + \beta (I-P) start at either I+PI+P or I−PI-P in every layer.

As regards Section 6, the authors also report that the learning rates are different for these experiments: Appendix C gives (0.005,0.005)(0.005,0.005) for the alternatives, versus (0.0025,0.02)(0.0025,0.02) for the proposed method. Were the learning rates (0.005,0.005)(0.005,0.005) chosen through hyperparameter tuning?

Figure 5 (Section 6) shows that the authors' configuration is better, but in order to be able to attribute the improvement to the parameterization, the initializations have to match and the learning rates of the alternative parameterizations also need to be chosen through hyperparameter tuning (if they were not already).

The performance differences are also very small, and error bars of 1 standard deviation are much smaller than 95% tt-confidence intervals (which would be x2.48 wider), since there are 2 degrees of freedom with 3 seeds. I think it could be useful to report CIs for the mean BPB differences between methods across matched seeds and see if they exclude 0, since I am not sure how reliable the performance differences are compared to the Laplacian baselines.

Questions for authors

Considerations:

I found the paper interesting, but it would have been nice to see some interpretation as to the following points:

  1. The meaning of scaling and subtracting the tt-th token's value vector, and hypotheses as to why this seems to help improve loss. The way I understand it, when γ\gamma becomes negative in δ(γI+P)\delta(\gamma I+P), the γI\gamma I term introduces this subtraction. Maybe this helps because it creates a contrast between the token’s own value vector and the attention-weighted value vectors of every other token in the context?
  2. Why layer 0 behaves in a completely different way compared to the other ones. On this point, the authors also treat layer 0 differently with ad-hoc choices (e.g., line 323). I can see that the behavior of the layer being different might warrant different choices, but explaining the rationale behind them would help make the approach more transparent.

Note that He et al. (2024) use Pre-LN transformers with skip connections in the I+βPI+\beta P experiment, so paragraph "Combinations of identity and attention" (lines 96-101) is a bit misleading when the authors say "We instead show [...] with skip connections."

On presentation: at lines 78--79, it is not clear how α0I+α1P\alpha_0 I +\alpha_1 P and αP+β(I−P)\alpha P + \beta (I - P) relate to γ\gamma. It only becomes clear later in Section 5. Maybe add a couple of words saying they end up being equivalent?

Questions:

  • Lines 139--140: What is the rationale behind the number of training tokens? The authors seem to be training on 10--11 tokens/parameter, so it does not look like they are following, for example, Chinchilla's 20 tokens/parameter recommendation (Hoffmann et al., 2022, https://arxiv.org/abs/2203.15556).

  • Have the authors tried adding a full learned parameter matrix (with causal masking) to PP instead of just γI\gamma I? Maybe that would be generally more expressive, although it would not show the fact that γ\gamma evolves to stay in (−1,0)(-1, 0) when it only affects the diagonal. I also suppose one drawback of this is that it adds more parameters than the approach of the authors, but I still would be curious to see how this compares to the authors' approach. Maybe it could be done as future work?

Reviewer_yhiVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Previously, a paper previously proposes the Laplacian heads, which instead of using P, it uses I + P for the attention weight matrix. In this paper, the author further improve the laplacian heads, by changing it to δ(γI + P), where δ and γ are learnable. They find it improves the performance, and more importantly the author find that γ usually in the range that is negative.

Critical strengths

The paper have a good amount of experiments to show the soundness of the claim. And it also have good study on alternative methods, so it is informative.

Critical weaknesses

None

Questions for authors
  1. The gain seems quiet marginal, especially considering the Laplacian baseline is weakened, you may want to update the Laplacian baseline such that keep the first layer to be pure attention, and the rest layers to be Laplacian attention (which is what you did at your experiments.

  2. Is there any connection to attention sink? attention sink will heavily influence the P matrix

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.