- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper considers a dynamic model for sparse attention, analyzing the dynamics of token interactions throughout depth. The main phenomenon that the authors try to capture is the clustering of tokens through the dynamics. In the analysis, they consider two models: an explicit model in which the tokens interact with fixed representatives, and an implicit model in which tokens interact with one another under sparsity constraints. The authors prove that, under specific assumptions and constraints, the tokens cluster onto clusters over limiting directions. They show that these results also hold in practice by using particle dynamics simulations in settings that fit their model exactly, and also by analyzing the hidden representation of LLMs at the final layer.
Critical strengths
-
The theoretical analysis is comprehensive and well written, with a high level of mathematical rigor. The authors carefully develop the convergence arguments and state the assumptions under which their guarantees hold.
-
The paper complements its theoretical results with useful numerical simulations and empirical comparisons with actual pretrained models and additional ones that fit their analyzed setting exactly; In both they show the clustering happens in practice. This effort to connect the theory with practical observations is valuable and goes beyond what is commonly included in theoretical work.
Critical weaknesses
-
The paper aims to explain how sparse interactions support separated token groups. However, in the implicit model (which is the practical one in my view), separation relies on the disconnected interactions assumption and the disjoint caps assumption. Accordingly, the theorem establishes the limiting dynamics within already separated groups while simply preserving their separation. This makes the result not so surprising.
-
My main concern is the significance of the contribution beyond existing work. References [1] and [2] study closely related mechanisms and dynamics and attraction to frozen centers. Although these works are cited, the paper should state precisely which assumptions, conclusions, or quantitative bounds are new and explain the importance of these advances. Beyond establishing clustering under the stated assumptions, stronger insights into the meaning or practical consequences of the resulting clusters would strengthen the contribution.
[1] Jingkun Liu, Yisong Yue, Max Welling, and Yue Song. Krause synchronization transformers. ICML, 2026. URL https://arxiv.org/abs/2602.11534.
[2] Nikita Karagodin, Yury Polyanskiy, and Philippe Rigollet. Clustering in causal attention masking. NeurIPS, 2024. URL https://arxiv.org/abs/2411.04990.
Questions for authors
-
Could you provide additional intuition about the explicit model? What do the fixed representatives correspond to in practical architectures, and why is it reasonable to keep them fixed throughout the dynamics?
-
What is the practical interpretation of the resulting clusters? Does this clustering have implications for the expressivity of attention or the information retained in token representations? What practical insights or predictions follow from the analysis?
-
The evaluated LLMs include feed-forward layers and other components absent from the theoretical model. What evidence indicates that the observed clustering is driven by the attention dynamics analyzed here?
-
Could you provide intuition on how Softmax normalization and multiple attention heads might affect the clustering behavior? Which conclusions are expected to remain valid, and which parts of the analysis would require modification?
Clarifications
-
Corollary 2.1 is presented without a connection to a specific theorem. Please identify in the main text the theorem that Corollary 2.1 is built on.
-
Assumption 2.2 is difficult to follow. Please distinguish its concentration, separation, and representative-dominance conditions explicitly and explain the roles of , , , and . Including the definitions of and alongside the assumption would also improve readability.
-
Please state in the caption of Figure 4 which architectural interpretation is associated with each model: explicit, implicit, hybrid, or the dense-attention baseline.
6 comments
Response to Reviewer hZkm (Part 2)
3. Motivation and scope of the model (Q1)
Architectural motivation and intuition. Practical sparse attention architectures based on landmark representations [3, 4] or MLP-compressed representatives [5] summarize token groups and use these summaries to aggregate information or guide retrieval. Such representatives can provide common reference directions towards which tokens evolve. Our model abstracts this operation by constructing representatives from initial token groups and studying how they guide subsequent token evolution.
The representatives jointly induce an attraction potential. Its local maxima determine the attracting directions, which need not coincide exactly with individual representative directions. This provides a mathematical description of how compressed summaries can guide groups of tokens toward distinct representations.
Fixed representatives. Fixed representatives are the central assumption of our model and is essential to our current analysis. It produces a time-independent attraction potential, allowing us to relate its geometry to clustering structure and convergence rates. This is an idealization of the practical architectures, where representatives evolve across layers. Studying attention dynamics with evolving representatives is therefore an important direction for our future work. We have clarified this assumption and the corresponding direction for future work in Appendix A of the revised manuscript.
[3] Mohtashami, A., and Jaggi, M. Landmark attention: Random-access infinite context length for transformers. NeurIPS, 2023.
[4] Leng, J., Hu, X., Wang, J., Li, J., Wu, W., and Lu, Y. Understanding and improving length generalization in hierarchical sparse attention models. ICLR, 2026.
[5] Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., ... and Zeng, W. Native sparse attention: Hardware-aligned and natively trainable sparse attention. ACL, 2025.
4. Semantic and functional structure of token clusters (Q2)
We thank the reviewer for this thoughtful question, which prompted us to examine what the observed final-layer clusters represent. We analyzed the tokens assigned to each cluster in DeepSeek-V4-Flash across 100 samples each from ARC-Easy and MATH.
The clusters exhibit clear semantic and functional coherence, grouping tokens associated with answer choices, mathematical notation, and prompt templates. These findings give the geometric clusters a practical interpretation: they correspond to recognizable aspects of token content and function across tasks. We report this new analysis in Section 3.4 (Cluster Semantics) of the revised manuscript.
Our theoretical results show how within-group alignment can coexist with distinct group-level representations. The semantic analysis further provides empirical evidence that such groups can have interpretable content in trained LLMs! Quantifying the implications for expressivity or information retention remains an important direction for further study.
5. Token clustering before and after the final FFN (Q3)
To examine the contribution of the final FFN, we compute the cosine silhouette scores across 100 random ARC-E samples before and after the final layer FFN in DeepSeek-V4-Flash. The scores are similar: before and after the FFN. These results show that substantial cluster separation is already present before the final FFN and that the scores remain similar in magnitude afterward. We have added the analysis in Appendix G.4.5 and regard a fuller investigation of how individual components shape token dynamics as future work.
Response to Reviewer hZkm (Part 3)
6. Clustering under softmax normalization and multi-head attention (Q4)
Softmax normalization. Introducing softmax normalization affects our two models differently:
- Explicit model. Let . The normalized dynamics are . For fixed representatives, is bounded above and away from zero on the sphere. Dividing the vector field by changes only the speed at which each token moves along its trajectory. The equilibria, attraction basins, and limiting clusters are therefore preserved. Exponential convergence persists, with adjusted rate bounds.
- Implicit model. Under the same fixed-graph and cap assumptions, normalization preserves the interaction support and uniformly positive weights on existing edges. Cap invariance, componentwise consensus, and separation between components therefore persist, preserving the qualitative multiple-cluster conclusion. However, the convergence-rate bound requires further analysis, as the token-dependent normalization breaks symmetry under fixed degree weights.
Multiple attention heads. Considering multiple attention heads also has distinct effects on clustering in our two models:
- Explicit model. Combining the head vector fields with positive weights yields the spherical gradient of the potential where is the number of heads, is the weight of head , and is its inverse temperature. Head has fixed representatives with positive weights satisfying . Combining heads can shift the attractors, but the gradient structure is preserved. If the combined field preserves disjoint caps and is uniformly strictly geodesically concave within each cap, the same proof strategy yields exponential convergence to distinct attracting directions. The dominance conditions and rate bounds must therefore be established for the combined potential in the multi-head settings.
- Implicit model. With positive head weights, the combined interaction graph is the union of the individual head graphs. If all heads respect a common group partition and the union graph is connected within each group, interactions remain confined to groups while allowing information to propagate throughout each one. Under the cap assumptions, componentwise consensus and distinct group limits are preserved. Quantitative rates require analysis of the combined interaction operator.
7. Clarification
Corollary. Thank you for identifying the missing connection. Corollary 2.1 follows the classical Łojasiewicz convergence theorem for analytic gradient flows [6, Section 2]. The potential is real analytic and the sphere is compact, ensuring that each trajectory converges to a critical point. The underlying argument is provided in Appendix F.1.2. We have added an explicit statement before Corollary 2.1 to discuss this connection.
Assumption 2.2. In the revised manuscript, we have reorganized the assumption into three conditions and clarified their roles:
- Concentration: bounds the initial spread of each group, while bounds the angle between its center and representative direction. The condition places each group inside a cap of radius around its representative.
- Separation: is a lower bound on the distance between group centers. The condition ensures that the representative-centered caps are disjoint.
- Representative dominance: ensures that the vector field points into the cap at its boundary, making the cap forward invariant. ensures strict concavity of the potential within the cap, yielding a unique attractor and an explicit exponential convergence rate.
We have placed the definitions of and alongside the assumption to improve readability. Please refer to Section 2.1 of the revised manuscript.
Figure.4. The original manuscript used different colors to indicate the architectural categories. We have now added explicit descriptions in the caption identifying the attention mechanism associated with each model as , , $\textcolor{#5178a1}{\textbf{hyb}}$$\textcolor{#b11f23}{\textbf{rid}}$, or architectures.
[6] Absil, P. A., Mahony, R., and Andrews, B. Convergence of the iterates of descent methods for analytic cost functions. SIAM Journal on Optimization, 16(2), 531-547.
We thank the reviewer again for the constructive feedback. We hope these clarifications and additional analyses address the concerns, and we welcome further questions and discussion.
Response to Reviewer hZkm (Part 1)
We thank Reviewer hZkm for positive assessment of our theoretical analysis and empirical investigation. We believe it would be always helpful to connect theory with pratical experiments nowadays. Below, we clarify the contributions of both theoretical models, their relationship to prior work and practical architectures, and the interpretation of our experiments. We also discuss Softmax normalization and multiple attention heads, and address the requested presentation improvements.
1. Contribution of the implicit model (W1)
We agree that Assumption 2.4 excludes cross-group interactions and that Assumption 2.5 places the groups in disjoint initial caps. Theorem 2.6 establishes cap invariance, ensuring that this separation persists. Its additional contributions concern the dynamics within each sparsely connected component:
We note that group separation alone does not establish either within-group consensus or these quantitative convergence rates. Our result characterizes how sparse connectivity governs alignment within groups while their separation is preserved.
2. Contribution beyond existing work (W2)
Comparison with Krause Attention [1]. Krause Attention is closely related to our implicit model. Its clustering analysis establishes exponential convergence under a small-cap condition. Our analysis allows a wider initial cap and proves cap invariance and exponential consensus on fixed sparse interaction graphs. Our explicit convergence-rate bound quantifies the effect of within-group connectivity through the normalized Laplacian spectral gap.
Comparison with Karagodin et al. [2]. In their frozen-token analysis on the circle, Karagodin et al. [2] study causal self-attention with additional attraction to fixed tokens, which idealize quasi-stationary cluster centers. Our explicit model is motivated by representative-based information compression in practical sparse attention architectures: each initial token group is summarized by a fixed representative, and tokens on Sd−1 interact exclusively with these representatives. These representatives summarize token groups, reflecting compression mechanisms in practical sparse attention. We further go beyond convergence guarantees to derive explicit convergence-rate bounds and quantitative lower bounds on the pairwise separation of the limiting clusters. We have added this discussion in Appendix B (Comparison with Prior Clustering Analyses) of the revised manuscript.
[1] Liu, J., Yue, Y., Welling, M., and Song, Y. Krause synchronization transformers. ICML, 2026.
[2] Karagodin, N., Polyanskiy, Y., and Rigollet, P. Clustering in Causal Attention Masking. NeurIPS, 2024.