- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the expressive limitations of potential-based attention models. The authors show that increasing the complexity of a convex potential cannot always eliminate approximation errors due to structural constraints imposed by gradient representations.
The main contribution is a characterization of the minimum number of independent nonlinear correction directions required for exact representation and arbitrarily accurate approximation, based on the rank of a residual matrix determined by the attention keys and values. The authors also establish quantitative error bounds and show that reducing the correction dimension may require ill-conditioned output transformations.
Numerical experiments, including certified error bounds and finite-model comparisons, support the theoretical results. The experiments also demonstrate that fewer correction directions do not necessarily reduce computational costs.
Critical strengths
-
Rigorous theoretical characterization. The paper provides a mathematical characterization of the expressive limitations of potential-based attention models. In particular, the authors establish quantitative error bounds and identify the minimum number of nonlinear correction directions required for exact representation and arbitrarily accurate approximation. These results clarify the structural limitations that cannot be overcome simply by increasing model capacity.
-
Connection between theoretical results and numerical validation. The paper supports its theoretical findings with numerical experiments, including certified approximation error bounds. The analysis of output conditioning and computational costs further clarifies the practical implications of the theoretical results. This combination of mathematical analysis and numerical validation strengthens the contribution of the paper.
Critical weaknesses
Weakness 1: Restrictive assumptions on the attention model and input distribution.
The theoretical results rely on several restrictive assumptions, including fixed attention keys and values, genericity conditions on the keys, and a uniform probability measure on a compact domain. It is unclear to what extent the results extend to more general attention models or realistic input distributions. These restrictions limit the applicability of the theoretical findings to practical attention architectures.
Weakness 2: Limited experimental validation on practical attention models.
Although the paper provides rigorous theoretical results and certified numerical experiments, the empirical evaluation relies primarily on synthetic examples and a relatively simple regression task. Experiments involving larger attention models or more realistic learning tasks would help establish the practical significance of the proposed framework.
Questions for authors
Question 1: Explicit characterization of the lower-bound constant.
In Theorem 1, the constant c_k=c(K,\eta,D,r-k)>0 plays an important role in quantifying the approximation error. Although the proof provides a construction of this constant, could the authors give a more explicit or computable lower bound in terms of the key geometry, attention bias, and domain D? In particular, it would be helpful to clarify how the constant behaves when the key configuration approaches degeneracy.
Question 2: Sharpness of the conditioning-dependent error bound.
Theorem 2 establishes a lower bound involving the factor \chi^{-2}, and the paper demonstrates its sharpness for a particular rank-two example. Is this dependence optimal for more general attention configurations? In particular, under what additional assumptions can a matching upper bound be established?
5 comments
Thank you for the clarification and for stating the upper bound in Theorem 2. It remains unclear to me whether the dependence on is necessary. Could you clarify whether the upper estimate can be obtained without the factor under the same assumptions?
Thank you for this question. Your feedback has helped us recognize an important blind spot in our thinking and make our contributions more solid. Yes, under the same assumptions, the upper estimate can be obtained without an explicit factor. We have strengthened Theorem 2 to
,
where depends only on the fixed keys. This bound is uniform over all finite attention biases.
The key observation is that the derivative of the attention weights in mean coordinates is a convex combination of barycentric-coordinate derivatives of key simplices. This provides a bound determined entirely by the keys, without separately estimating . The complete proof is now included in Appendix O.2. This improvement allows us to refine the theoretical proof without affecting the rest of the paper.
Response to Reviewer 8g84
We sincerely appreciate your insightful comments and your recognition of the rigorous representation results and their numerical validation. We have added explicit quantitative answers, clarified the assumptions, and evaluated the framework in pretrained models.
1. Fixed banks, genericity, and input distributions (W1)
The revision treats each assumption separately. A nonuniform mean law ν with locally positive continuous density admits an interior-ball uniform law μ0 and ν≥αμ0. Hence
Ek,χ∗,ν≥αck,0(τk2+χ−2σk+1(R)2),τk2=∑j=k+2rσj(R)2.
The upper construction is pointwise and applies to every probability law on the compact domain. For context-dependent keys and values, the lower bound applies conditionally and is then averaged over contexts, including for shared-parameter readouts, when the conditional banks and query laws satisfy the assumptions.
The upper construction requires affine spanning alone. Matching lower bounds use the stronger requirements that every (d+1)-key subset is affinely independent and distinct such subsets have distinct vector sums. Appendix Q.3 tests affine spanning; Appendix O.2–O.3 specifies the assumptions for each extension.
2. Practical model validation (W2)
We evaluate Qwen2.5-0.5B and 1.5B on WikiText-2 and Penn Treebank, replacing prefix attention in interleaved layers and recomputing subsequent network states. Final prediction preservation is:
The learned-bank comparison also separates capacity from direction choice. Entries below are median error reductions relative to an affine readout; all corrected models use the same correction dimension.
Table 2 and Table 4 contain these results. Appendix Q gives direction controls and paired uncertainty; Appendix R reports computation and numerical sensitivity.
3. Explicit constant and behavior near degeneracy (Q1)
For n>d+1, an explicit choice is
c∗=2Cψ∥Hinc∥op2λmin(G)2λmin(S),ck=c∗,ck=c∗2(r−k)r−k−1.
Here G is the Gram matrix of simplex weight functions, S sums outer products of barycentric gradients, Cψ sums test-function gradient energies, and Hinc records simplex vertex replacements. All depend only on keys, bias, domain, and the chosen tests. Appendix O.1 gives their definitions and query-coordinate integrals for evaluating the constant independently of the values.
For compact generic key families with bounded biases and uniformly controlled interior test support, continuity gives a uniform positive bound. The formula identifies geometric factors that can weaken the certificate near degeneracy. Figure 2 holds the residual spectrum fixed while changing key-and-domain geometry and measures the resulting certificate.
4. Optimal conditioning dependence beyond the rank-two example (Q2)
The revision establishes matching bounds for general residual spectra. Under the lower-bound assumptions, for 0≤k≤r−2,
ck(τk2+χ−2σk+1(R)2)≤Ek,χ∗≤2τk2+2ρ2Mp2χ−2σk+1(R)2.
Here ρ is the domain radius and Mp bounds the derivative of attention probabilities in mean coordinates. The spectral-tail and conditioning terms match up to constants fixed by geometry, bias, and domain. In particular,
rank(R)=k+1<r⟹Ek,χ∗=Θ(χ−2).
This proves optimal inverse-square decay throughout this class. The upper construction itself requires only affine spanning. Theorem 2 states the bounds, and Appendix O.2 gives the construction and proof.
The common response also describes the theoretical advances and new experiments. We hope these additions address the central weaknesses you identified and respectfully ask whether you would consider revisiting your rating in light of the revised evidence. Thank you again for your careful review; we welcome further discussion to improve the final manuscript!