acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Attention Lottery: Expressive Power and Topological Complexity via Algebraic Geometry

Abstract

We study algebraic and geometric properties of simplified attention models. For exact computation of all pairwise inner products between two independently varying collections of vectors in , we prove that a single-hidden-layer network with square activations requires at least hidden units and give an explicit construction using units. Thus the width is for fixed , while the optimal dependence on a growing is left open. For a fixed latent representation and explicitly specified admissible query and key maps, allowing a nonlinear key map strictly enlarges the linear span of attention score functions. When and each summand may instead use an unrestricted preceding map, the standard, single-sided, and dual-sided spans coincide. In a bias-free, softmax-free residual attention model, we derive that every output coordinate is a polynomial of odd degree at most after layers; the occurrence of particular lower degrees requires nondegeneracy conditions. We then discuss Milnor–Thom bounds only for the corresponding idealized polynomial decision boundaries, and analyze actual input scaling in softmax attention: small scales produce nearly uniform averaging, whereas large scales concentrate on maximal scores under a positive score gap. The final discussion presents the “lottery of attention” as a conjectural geometric interpretation of parameter sparsity and initialization; it is not a theorem guaranteeing optimization feasibility or homotopic convergence.

Primary area:
learning theory
Keywords:
transformer, attention mechanisms, expressive capacity, polynomial bounds, Milnor-Thom bound

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AQE3OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies algebraic and geometric aspects of simplified attention mechanisms. The results include a comparison of the linear spans of attention scores under different simplifications of the attenations mechanisms, a width lower bound to compute pairwise inner products with feedforward networks with quadratic activation function as well as an analysis of the polynomial degrees in simplified residual attention architectures. Moreover, bounds on the Betti number of decision boundaries using Milnor-Thom are derived.

Critical strengths

The paper provides several sound and concrete mathematical claims about simplified attention networks. I also appreciate that the authors distinguish the results on their simplified models with transformer thereby avoiding potential misinterpretations.

Critical weaknesses

My main concern is that the paper appears to present a collection of rather elementary statements without a clear story and I find it difficult to identify a substantial new contribution.

For instance, the strict inclusions between the spans of different attention simplifications in Proposition 3.1 follow quite directly from the distinction between linear and nonlinear functions. Similarly, the lower bound in Theorem 4.2 essentially follows from the linear independence of the target quadratic forms, together with a simple trace argument, while the upper bound uses a polarization identity.

Moreover, the discussion of the geometry of ReLU networks feels somewhat out of place, since on one hand it is not connected to the remaining paper and it is also just a summary of very well-known facts.

Finally, I could not make a lot of sense of the topological lottery ticket conjecture. The paper says that "we conjecture that a randomly initialized attention model may already contain local portions of token decision surfaces that training can move toward useful class boundaries". It is not clear to me, what is conjectured, based on what evidence and what about this conjecture is topological and specific to transformers.

Questions for authors
  • What do the authors consider the main mathematical contributions and/or message of this paper?
  • Could you clarify what the topological lottery ticket conjecture is saying and based on what evidence you are conjecturing it?
Reviewer_LVn6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies properties of attention, or a simplified model thereof, extended to (a simplified model of) multi-layer transformers. They present three main results: One on the expressivity of self-attention under various query and key-mappings. One on parameter bounds of simulation of attention via a particular MLP-formulation. And one on the topology of decision boundaries.

Critical strengths

None

Critical weaknesses

The main weakness of the paper is that most statements (i.e. those made in sections 3-6), while internally consistent, seriously inflates their importance and relevance: Sections 3 and 4 provide largely elementary results that, as far as I can tell, are not used in the subsequent argument. Section 5 introduces what seems to be the papers main contribution, but it hinges on a cubic model of attention that is then extrapolated to a polynomial model of transformer stacks. Using this construct, they provide topological bounds on logit equality sets, which they themself characterize as incomplete surrogates for the true object of study, prediction boundaries. As such, section 5 seems to be a statement about a surrogate object (logit equality sets) in a simplfied model (stacked polynomial attention) that aims to shed light on the true object of study (prediction boundaries) in a real model (stacked transformers). This connection is very thin.

Questions for authors

How does section 3 and 4 relate to the larger conjecture (section 7)?

How does the analysis in section 6 relate to normalization strategies such as layer norm, or QK-normalization? Would it survive, and what could be said about models using layer normalization, using your models?

What does Corollary 5.3 tell us about prediction boundaries in actual transformers?

Do any (relevant) result survive when we move from unnormalized attention, to attention, and from stacks of unnormalized attention layers, to stacked attention layers with feedforward blocks in between?

Reviewer_PzPzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the expressive power and geometry of simplified attention models. It establishes strict inclusions between the spans of attention score functions when the latent representation is fixed, and shows how these relations change under unrestricted preceding representations. It also proves hidden-width bounds of N2+1≤Hmin⁡(N,d)≤2N2dN^2+1 \leq H_{\min}(N,d) \leq 2N^2d for exact computation of all pairwise inner products using a single hidden layer with square activations. For a residual attention stack without softmax, feed-forward sublayers, or normalization, it derives a polynomial degree bound of 3L3^L and applies classical bounds to pairwise logit-equality sets. A separate analysis describes attention under input scaling. The paper concludes with a conjectural topological interpretation of initialization and training. Validation is theoretical, with proofs provided in the appendices and no empirical experiments.

Critical strengths
  1. The comparison of attention score spaces carefully distinguishes fixed representations from unrestricted preceding representations, and individual heads from linear spans. This makes the conclusions precise and helps explain when nonlinear query or key maps provide additional expressive power.

  2. The width bounds and polynomial filtration give concrete mathematical characterizations of simplified attention operations. The trace argument strengthens the basic dimension lower bound, while the scalar construction in Appendix C establishes that all claimed odd degrees are attainable without cancellation. These results provide a useful foundation for further analysis.

Critical weaknesses
  1. The technical advance appears modest relative to the breadth of the paper. The score-space hierarchy follows from the linear dependence on each input, while the polynomial filtration follows from cubic composition and residual addition. The topology result then applies a classical degree-based bound without establishing how much complexity the attention architecture actually attains. The width theorem is more concrete, but its dependence on feature dimension remains loose. In particular, taking C=INC=I_N in Equation (B.9) gives a target matrix of rank 2Nd2Nd, which implies the additional lower bound Hmin⁡≥2NdH_{\min}\geq 2Nd. Discussing this bound would sharpen the result and clarify the remaining gap.

  2. The connection between the formal results and the proposed topological lottery explanation remains limited. The polynomial analysis excludes internal softmax and normalization, and bounds full pairwise logit-equality sets rather than active prediction boundaries. The separate softmax scaling analysis does not connect these bounds to initialization or training. Section 7 appropriately acknowledges these limitations, but provides neither a theorem nor empirical evidence showing that useful boundary structures exist at initialization or can be reached through training. This leaves the broader explanation of attention and long-context capacity largely unresolved.

Questions for authors
  1. Which result do the authors consider the main substantive advance, and what conclusion about attention follows from it beyond the stated function-space and degree characterizations? Clarifying this would help assess the significance of the contribution.

  2. Can the authors incorporate the rank bound Hmin⁡≥2NdH_{\min}\geq 2Nd, obtained by setting C=INC=I_N in Equation (B.9), and discuss whether the combined bound Hmin⁡≥max⁡{N2+1,2Nd}H_{\min}\geq\max\{N^2+1,2Nd\} changes their interpretation of the feature-dimension dependence? For N=1N=1, it matches the constructive upper bound exactly.

  3. Is there evidence within the present work that connects the attainable polynomial family to active prediction boundaries or useful boundary structures at initialization? If no such evidence is available, how should the topological lottery discussion affect the assessment of the paper’s established contributions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.