- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies precision–expressivity tradeoffs in single-layer Transformers with one attention head and value dimension one, followed by a ReLU MLP, without residual connections, layer normalization, or CoT. Besides, primitive operations are rounded, accumulations are left-associated, and value projection precedes attention aggregation. Its main result establishes a tight -versus- separation for under this special setting.
Critical strengths
The paper presents a clean precision threshold, though under a quite special setting and for a quite special function.
Critical weaknesses
My main concern is thus the limited generality of the contribution, which limits the practical relevance/implication of these theoretical results.
Questions for authors
What is the generality of these techniques? Do they apply to broader setting?
4 comments
Thanks for the responses. I remain concerned about the generality and practical relevance of the results. The analysis focuses on single-layer Transformers without CoT. It also relies on specific architectural/computational assumptions, such as no residual connections, no layer normalization, and applying value projection before attention aggregation. I keep my score.
Thank you for the follow-up. Our objective is to isolate the intrinsic expressivity of a single attention layer under bounded precision. Such isolation is widely studied in the literature, as we discussed in our introduction. We clarify the distinctions between the assumptions mentioned:
-
CoT and related strategies: CoT augments a model with additional inference-time computation; other examples include repeated model application/looping, beam search, etc. These strategies can be applied across different model architectures and are not specific to attention. They address how a model is used during inference, whereas our work isolates what a single attention layer can compute under bounded precision.
-
Residual connections: Residual connections are used empirically to improve gradient flow and stabilize training. This can be simulated within the attention-based framework using an additional attention-head [2].
-
Layer normalization: When applied after attention at the readout, tokenwise normalization is local post-processing and cannot bypass the same communication bottleneck. Preserving the matching upper-bound constructions under normalization is a separate question.
-
Operation order: Applying value projection before attention aggregation follows the original Transformer formulation [1]. Different evaluation orders can produce different results under finite-precision arithmetic, making an explicit computational model necessary for exact one-bit separations. Our results characterize the stated model.
Overall, these decoding and inference-time strategies can be applied across different architectures and are not specific to attention. They address how a model is used and what additional computation it receives. Our work addresses the complementary question of what a single attention layer can intrinsically compute under bounded precision.
[1] Vaswani et al., “Attention Is All You Need,” NeurIPS 2017, Section 3.2.2.
[2] Sanford, Hsu, and Telgarsky, “Representational Strengths and Limitations of Transformers,” NeurIPS 2023, Section 2.1.
Thank you for your responses. However, my main concerns largely remain. I keep my score.
We thank the reviewer for their thoughtful comments regarding the generality, practical relevance, and architectural assumptions of our results. We address these concerns below.
Generality and practical relevance: Understanding the expressivity of a single attention layer is fundamental to identifying which computational capabilities are intrinsic to attention itself and which require deeper architectures. Furthermore, we have now generalized our results to arbitrary value dimension dv and number of attention heads H, establishing tight precision thresholds and substantially broadening the scope of our theoretical results.
Regarding practical relevance, empirical studies have already observed substantial performance degradation under aggressive quantization, including on reasoning tasks [1]. Our work provides a theoretical understanding of how precision-dependent expressivity limitations can arise, rather than claiming to fully characterize these practical settings.
Architectural assumptions and CoT: Chain-of-thought reasoning is a complementary approach that provides additional sequential computation. Our goal, however, is to understand the intrinsic expressive power of transformer architectures under bounded precision, independently of additional computation through CoT. We believe isolating these capabilities is an important step toward understanding how precision, width, and depth interact.
We hope these clarifications and our new generalizations address the reviewer's concerns and better highlight the significance of our theoretical contributions.
[1] Liu et al., "Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study," 2023. https://arxiv.org/abs/2307.08072