- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how head count limits gain-stable threshold decisions in a single attention layer. The main result is a robust capacity law of , together with more detailed bounds under score constraints. I think the main capacity result is technically nontrivial, and I did not find an obvious correctness issue. My main concern is significance: the paper develops a fairly elaborate theory around a very specific notion of localized-source decision capacity, but does not yet show why this quantity matters for attention computation more broadly. The novelty positioning is also incomplete, especially given several closely related attention-only universality papers missing from the discussion.
Critical strengths
-
Thm 3 gives a clean head-count limitation that is independent of width, context length, and output coefficient magnitude. The matching finite construction makes this substantially stronger than a simple upper-bound or parameter-counting result.
-
Thm 2 and Thm 3 together give an interesting separation: two heads already approximate arbitrary smooth scalar responses in this interface, while robust decision capacity still grows with head count. I think this is the clearest conceptual contribution of the paper.
Critical weaknesses
-
Significance/relevance. My main concern is that robust alternating capacity is a fairly specialized quantity. It counts threshold crossings of one source intervention along an affine query path, with the threshold and witness locations also optimized. The paper characterizes this quantity in considerable depth, but I am not yet convinced what important capability or limitation of Transformers this explains. The technical results may be new, but the significance currently feels limited for ICLR.
-
Novelty/positioning. I do not think the main capacity theorem is obviously subsumed by prior work. However, the paper misses several directly relevant attention-only universality papers, including work by Hu et al., Liu et al., and Petrov et al. This is especially problematic for Thm 2 and the discussion in Sec 8, which currently contrasts mainly with universality results that still use feed-forward layers. The paper needs a much clearer comparison of what is actually new in terms of architecture, number of heads, preprocessing, depth, and approximation setting.
- Hu et al., Universal Approximation with Softmax Attention
- Liu et al., Attention Mechanism, Max-Affine Partition, and Universal Approximation
- Petrov et al., Prompting a Pretrained Transformer Can Be a Universal Approximator
- Petrov et al., When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations
- Dana et al., Memorization in Attention-only Transformers
Questions for authors
-
What concrete consequence does robust alternating capacity have for actual Transformer computation, learning, or interpretability? Is there a natural task where this capacity directly determines what the model can or cannot do?
-
How does Thm 2 compare directly with Hu et al., Liu et al., and Petrov et al. on architecture, head count, FFN use, preprocessing, target fn, and approximation topology? What exactly remains new?
-
How essential is the localized-source setup where keys and scores are fixed during the intervention? Can the main capacity result be extended to a less restricted attention setting?