acceptodds
Under review as a conference paper at ICLR 2027

Gradient Flow Polarizes Softmax Outputs towards Low-Entropy Solutions

Abstract

Sparse attention patterns are prevalent in trained transformers, yet the mechanisms that favor their emergence during training remain poorly understood. In this work, we investigate the role of softmax in shaping these patterns by studying the gradient flow dynamics of an attention head trained to produce a fixed bias vector for all inputs, a task that dense and sparse attention solve equally well. Leveraging symmetries of this task, we show that the dynamics of the head simplify to a *value–softmax* model , where and are a learnable value matrix and attention vector. We prove that gradient flow on this model drives the attention scores toward low entropy: under logistic loss they converge to a one-hot vector, and under square loss we obtain quantitative polarization bounds. Finally, we use experiments in trained transformers to examine how the choice of attention nonlinearity affects the influence of individual tokens and sparse attention patterns.

Primary area:
foundation or frontier models, including LLMs
Keywords:
softmax, low-entropy, sparse attention, implicit bias

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3AQPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims to explain, through a gradient-flow analysis, why attention concentration and attention sinks emerge during training even when concentrated attention is not necessary for solving the task. It considers a task with a shared, input-independent target and shows that, under a particular initialization, training the value map and the merged query–key matrix reduces exactly to learning position-specific logits and value vectors. For the reduced model, it shows that attention converges to a single position under logistic loss, and that attention becomes increasingly polarized under additional conditions for square loss and other losses. The paper further analyzes a setting in which multiple queries concentrate their attention on the same position and discusses its connection to attention sinks. It also explains theoretically how normalization induces competition between positions, and shows empirically that normalization promotes sink formation even with attention functions other than the exponential.

Critical strengths

The paper analyzes how the joint learning of values and attention leads to attention concentration in several settings: concentration on a single position under logistic loss, attention polarization under square loss and other losses, and competition between positions induced by normalization. Together with experiments using attention functions other than the exponential, these results examine the mechanism of concentration from several angles.

Critical weaknesses

The theoretical analysis considers a restricted setting, and its significance for explaining attention sinks in realistic models is not sufficiently clear. The paper uses a shared, input-independent target and a special initialization to reduce training to the direct learning of position-specific logits and value vectors. While it usefully clarifies how interactions between values and attention amplify initial differences, I have reservations about how much new understanding of attention sinks is gained from this analysis in such a restricted setting.

Questions for authors

Why are different initializations used for logistic and square losses? Can the attention-concentration results for each loss also be obtained under a common initialization?

Reviewer_pZGjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates whether learning attention scores and values together pushes attention toward sparse patterns, even when dense attention would solve the task just as well. It studies a task where an attention head must output a constant bias. Under specific initial conditions, this task reduces to a value–softmax model. The paper proves that attention converges to a one-hot vector under logistic loss, and it gives polarization bounds for regression that hold only under extra conditions. Experiments on synthetic tasks and pretrained models demonstrate how the choice of attention function and normalization affects sparsity, attention sinks, and the influence of single tokens.

Critical strengths

The paper identifies and formalizes a clear mechanism for polarization when the task does not require sparsity. The constant-bias task separates two things: whether the model can fit the target, and how it spreads attention to do so. In the reduced model, the analysis finds a feedback loop. Coordinates that get more attention learn their values faster, and the resulting gaps between values pull in even more attention. This explains concretely how training values and scores together can pick a sparse solution over dense ones that work equally well.

Critical weaknesses
  1. Close theoretical prior work is missing or insufficiently differentiated. Several central phenomena presented as contributions have substantial precedents that the paper does not adequately discuss. Tarzanagh et al. [1] establish max-margin token selection and asymptotically sparse attention, including an analysis of joint attention–decoder optimization through regularization paths. More directly, Vasudeva et al. [2] analyze joint training and prove increasing gaps between optimal and nonoptimal token scores and nondecreasing attention on optimal tokens, properties closely related to Theorems 3.2–3.3. Tian et al. [4] characterize the ordered suppression of attention weights during training, starting from uniform attention. Finally, Guo et al. [3] explicitly identify and theoretically analyze mutual reinforcement between attention allocation and value states as a mechanism for attention sinks, although in a different dynamical regime. The paper does not cite [1], [2], or [4], and does not meaningfully compare its proposed reinforcement mechanism with [3]. While its constant-bias reduction, general-loss polarization bounds, and regression analysis appear distinct, the manuscript should clarify which conclusions go beyond these established results and which provide alternative derivations or extensions.

  2. The results do not cover standard attention. The model trains the merged attention matrix rather than separate query and key matrices, which Corollary B.2 shows changes the dynamics. The guarantees off the invariant set hold only over a finite time, and Figure 4c shows that perturbations do not decay back to it.

[1] Tarzanagh et al. Max-Margin Token Selection in Attention Mechanism (NeurIPS 2023)

[2] Vasudeva et al. Implicit Bias and Fast Convergence Rates for Self-attention (TMLR 2025)

[3] Guo et al. Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs (NeurIPS 2024 workshop)

[4] Tian et al. Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer (NeurIPS 2023)

Questions for authors
  1. How do Theorems 3.2–3.3 differ substantively from the joint attention–decoder training results of Vasudeva et al. [2], which also establish increasing token-score gaps and monotonic attention concentration? More broadly, how does the one-hot convergence result compare with the sparse-attention implicit bias established by Tarzanagh et al. [1]? Could the authors clarify which aspects of their analysis are genuinely new, beyond the differences in model and loss assumptions?

  2. Can the authors show, in theory or in experiments, that the one-hot limit persists under logistic loss when the semantic blocks start nonzero and the initial scores are non-uniform?

Reviewer_3519OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why jointly learning attention scores and values can favour concentrated attention, even when dense attention can solve the task equally well. The authors introduce a constant-target task and derive an exact reduction to a value–softmax model on an invariant parameter set. Under logistic loss and specified initial conditions, they prove order preservation, increasing separation between value projections, and convergence of the attention weights to a one-hot vector. For square-loss regression, they establish conditional polarization bounds and relate their strength to the cumulative output gradient. Experiments on synthetic Transformer tasks and existing pretrained models examine attention sinks, attention concentration, and sensitivity to individual tokens.

Critical strengths
  1. The constant-target task isolates a meaningful question about implicit bias. Dense and concentrated attention can both solve this task, so the analysis separates a preference induced by optimization from sparsity required by the task. Proposition 2.1 provides an exact reduction under stated conditions, giving the simplified model a clear connection to an attention head.

  2. The analysis makes the interaction between attention and value learning explicit. Theorems 3.2 and 3.3 show how an initial difference in value projections can be reinforced through joint training, leading to a one-hot limit under logistic loss. The distinction from square-loss regression is also useful: the paper explains how the accumulated gradient controls polarization, rather than suggesting that all losses necessarily produce the same limiting behaviour.

Critical weaknesses
  1. The connection between the asymptotic result and standard Transformer training remains limited. The exact reduction uses population gradients, a merged attention matrix, and initialization on a particular invariant set. The main polarization results also require specific initial conditions. The authors acknowledge these restrictions and provide perturbation analyses, but the finite-horizon bounds do not establish that the asymptotic one-hot behaviour persists away from the invariant set. Moreover, Corollary B.2 shows that separately training the query and key matrices changes the dynamics. These limitations leave uncertainty about how broadly the proved mechanism explains attention concentration in ordinary training.

  2. The Transformer experiments support the predicted pattern more directly than the proposed mechanism. Differences in sink frequency and attention concentration are consistent with the theory, but do not establish that the same feedback between value learning and attention allocation drives those differences. The substantial sink rates for other normalized positive attention functions in Figure 3b also make it important to distinguish the role of normalization from the particular softmax parameterization. Direct measurements of the coupled learning dynamics, or controlled interventions on value learning, would provide stronger evidence for the explanation.

Questions for authors
  1. Can you provide a controlled comparison on the constant-target task between directly trained logits or merged scores and separately trained query and key matrices? In particular, does polarization persist under generic small random initialization, and how does it change with the size of the semantic parameters? This would help assess how representative the reduced dynamics are.

  2. In the Transformer experiments, can you measure whether positions receiving more attention subsequently improve faster in the relevant value projections, and whether those improvements predict later attention increases? A controlled comparison with fixed versus trainable value parameters would also help distinguish the proposed feedback from other causes of attention concentration.

  3. How sensitive is the regression result to the residual sign condition in Corollary 3.6? Reporting how often the condition holds across representative random problems, together with polarization before and after sign changes, would clarify the practical scope of the bound.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.