- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This manuscript investigates whether higher-order (cubic) attention, which models interactions between a query and pairs of keys, offers better scaling properties than standard quadratic attention in language models. The authors introduce techniques for stable training, ALiBi3D positional encoding for context-length extrapolation, and optimized Triton kernels for dense and sparse higher-order attention.
Critical strengths
1.) The manuscript discusses an important and well-defined research question.
Critical weaknesses
1.) The experimental context lengths are extremely short: The claimed scaling advantages are most relevant at larger model sizes and longer contexts, but the actual context experiments stop at 2048 tokens and dense cubic training stops at 512. Results at these lengths are insufficient to establish practical advantages for modern long-context LLMs. In the current landscape of large language models, long-context processing typically refers to sequence lengths of 32K tokens or more.
2.) Increased KV-cache memory requirements raise concerns about practical scalability: Higher-order attention doubles the KV-cache memory requirements, raising concerns about practical scalability, especially at longer contexts. Given the ongoing efforts to reduce KV-cache memory consumption [1], this additional overhead is a significant drawback. While the manuscript provides a valuable study of higher-order attention, its practical applicability remains unclear without demonstrating that the performance gains justify the increased memory costs.
3.) Downstream evaluation does not establish new capabilities: The authors evaluate six benchmarks: ARC-Easy, ARC-Challenge, HellaSwag, MathQA, PIQA, and WinoGrande. They show that downstream accuracy correlates with pretraining loss, but this does not demonstrate that higher-order attention is better at tasks requiring three-way token interactions, long-range retrieval, or complex compositional reasoning.
4.) Sparse attention needs stronger validation: This variant is interesting because it maintains global reach. However, lower loss does not necessarily demonstrate preservation of useful long-range information. Long-context retrieval and reasoning experiments are required for this claim to convince the reader.
5.) Limited discussion of related efficient-attention methods (minor): Although the authors discuss several sparse and linear attention approaches in Appendix C, other relevant works, such as CosFormer [2], Reformer [3], RACE Attention [4], YOSO [5], and vAttention [6], are not discussed. A more comprehensive discussion of these methods and recent developments in efficient attention would help better position the proposed approach within the existing literature, particularly regarding its computational and memory-efficiency tradeoffs.
[1] Devoto, A. and Jeblick, M., et al. Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution. arXiv preprint arXiv:2510.00636.
[2] Qin, Z., Sun, W., et al. cosFormer: Rethinking Softmax In Attention. ICLR 2022.
[3] Kitaev, N., Kaiser, L., et al. Reformer: The Efficient Transformer. ICLR 2020.
[4] Joshi, S., Chowdhury, A., et al. RACE Attention: A Strictly Linear-Time Attention Layer for Training on Outrageously Large Contexts. ICLR 2026.
[5] Zeng, Z., Xiong, Y., et al. You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling. ICML 2021.
[6] Desai, A. et al. vAttention: Verified Sparse Attention via Sampling. ICLR 2026.
Questions for authors
1.) Refer to Weaknesses.
2.) Does higher-order attention learn capabilities that standard attention cannot reproduce at similar computational cost?
3.) How do the practical training and inference costs compare? A clearer end-to-end comparison of throughput, GPU memory, and decoding latency would help establish practical applicability.
4 comments
(2/3) Downstream evaluation and capabilities
W3. Downstream evaluation does not establish new capabilities
The authors evaluate six benchmarks: ARC-Easy, ARC-Challenge, HellaSwag, MathQA, PIQA, and WinoGrande. They show that downstream accuracy correlates with pretraining loss, but this does not demonstrate that higher-order attention is better at tasks requiring three-way token interactions, long-range retrieval, or complex compositional reasoning.
We agree that our benchmarks do not isolate specialized capabilities such as three-way reasoning or long-range retrieval, and we do not make such claims.
Section 7 instead asks whether higher-order attention's improvement in pretraining loss translates to downstream accuracy. It does: both architectures follow the same accuracy-versus-cross-entropy curve (Figure 7.1; Appendix H), so the lower loss carries over to accuracy, with no additional task-specific lift on these benchmarks.
Relation to prior work. Specialized capabilities have been explored in complementary settings. Clift et al. (2019) proposed 2-simplicial attention as an inductive bias for logical reasoning in deep reinforcement learning. Roy et al. (2025) studied mixture-of-experts models with 2-simplicial attention in every fourth layer, reporting a steeper active-parameter scaling exponent on GSM8k, MMLU, MMLU-Pro, and MBPP negative log-likelihood at a fixed token budget, but no gains below 2B active parameters. Theoretical work by Sanford et al. (2023), Kozachinskiy et al. (2025), and Chakrabarti et al. (2026) establishes representational separations on token-triple tasks.
Our setting and findings differ. We study dense models with higher-order attention in every layer, up to 2.8B parameters, fitting joint scaling laws in parameters and tokens. Unlike Roy et al., we observe improvements at every evaluated model size. The advantage at our scales is a lower loss, not a steeper scaling exponent: higher-order attention has smaller parameter and data exponents, and our data cannot tell whether its floor is lower (Appendix G). On our downstream benchmarks, these improvements are explained by the reduction in cross-entropy. Our results show a general language-modeling advantage at the scales studied; they do not rule out task-specific advantages at other scales or on tasks that require three-way interactions.
Long-range retrieval. We ran needle-in-a-haystack retrieval on the 2048-token models; the results are in our reply to W4 and in Appendix H.1 (new). Token-triple tasks remain untested. We have clarified in Section 7 that our downstream experiments evaluate the transfer of language-modeling improvements to accuracy, rather than specialized capabilities.
Q2. Capabilities standard attention cannot reproduce at similar compute
Does higher-order attention learn capabilities that standard attention cannot reproduce at similar computational cost?
Theoretically, it has been shown that a single higher-order attention layer can represent functions over token triples that a standard attention layer of constant width cannot (Sanford et al., 2023; Kozachinskiy et al., 2025; Chakrabarti et al., 2026), so there are tasks on which the two operators are separated at the representational level. Empirically, at the scales we study, we have not observed such a difference. Across our six benchmarks the two architectures lie on the same accuracy-versus-cross-entropy curve (Figure 7.1), so the advantage of higher-order attention appears as lower loss rather than as a capability that standard attention lacks. In the retrieval test of our reply to W4, standard, dense higher-order and dilated higher-order attention draw comparable information from distant needles, and exact-match accuracy at this scale varies more between training seeds than between architectures, so retrieval shows no capability that standard attention lacks.
(3/3) Long-context retrieval, measured cost and summary
W4. Sparse variant: long-context retrieval and reasoning validation
This variant is interesting because it maintains global reach. However, lower loss does not necessarily demonstrate preservation of useful long-range information. Long-context retrieval and reasoning experiments are required for this claim to convince the reader.
We agree that lower loss does not show this, so we ran the retrieval test you ask for (Appendix H.1 and Figure H.2, new).
Setup. Two standard needle-in-a-haystack tasks: passkey retrieval (Mohtashami & Jaggi, 2023) and RULER's single-needle task S-NIAH-1 (Hsieh et al., 2024), with prompt text and samples taken from the original task code. A 5- or 7-digit number is placed at a controlled distance before the question in a 2048-token prompt. We evaluate the 2048-token models of Section 6.2 at both sizes (0.21B and 0.64B parameters), and the 22 sparsity-sweep models of Figure 6.1b (0.21B), without fine-tuning. We measure how much the model draws from the needle as the gain in log-likelihood of the answer over a no-needle control, and also report greedy exact match (computed by teacher forcing; this equals greedy decoding). The no-needle control scores 0% exact match for every model.
Gain in answer log-likelihood over the no-needle control (nats), S-NIAH-1, needles under 256 / at least 512 tokens before the question:
| 0.21B | 0.64B | |
|---|---|---|
| Standard attention | 21.8 / 19.5 | 21.1 / 21.2 |
| Higher-order, windowed | 19.6 / 0.0 | 21.6 / 0.0 |
| Higher-order, dilated | 21.9 / 22.1 | 21.4 / 21.3 |
- Dilated attention uses needles anywhere in the context. Its second key axis is dense, so one layer reaches every position, and it draws as much from a needle 2000 tokens back as from one nearby. Passkey agrees (20-21 nats for distant needles).
- Windowed attention uses only what lies within its window. Beyond its 256-token window, the needle changes the answer likelihood by nothing: it equals the no-needle control. Its stacked receptive field spans the context, but the model does not use it to retrieve. On the Figure 6.1b models this holds for every window from 128 to 1024 tokens: the cutoff sits at the window width (at most 0.33 nats beyond it, 14 to 19 nats within it). Every full-reach variant there (dense higher-order attention, dilated attention at six sparsity levels, and windowed attention with one dense key axis) draws 17 to 23.5 nats from needles 512 to 2048 tokens back, with one exception among 16 models that Appendix H.1 reports.
- Exact match is too noisy at this scale to rank architectures. At 0.21B, two training seeds of the same configuration differ by up to 86 points in exact match on distant needles. Every failed greedy answer we inspected continues the prompt template ("The pass key is the pass key") instead of giving a wrong number. We report the accuracies in Appendix H.1 but draw no ranking from them.
So the distinction you draw is real, and it separates the two sparse variants: windowed attention's lower loss comes without access to information beyond its window, dilated attention's does not.
Scope. These results hold within the trained 2048-token context, for single-needle retrieval, with at most two training seeds per configuration. We have not run multi-hop reasoning tasks, and Appendix B now lists this as a limitation.
Q3. End-to-end throughput, GPU memory, decoding latency
How do the practical training and inference costs compare? A clearer end-to-end comparison of throughput, GPU memory, and decoding latency would help establish practical applicability.
We agree; the revision adds all three (Table K.4, new), and all favor standard attention.
- Training: per step, dense higher-order attention is 30$\times$, 105$\times$ and 530$\times$ slower than standard attention at 512, 1024 and 2048 tokens; dilated 11$\times$, 21$\times$ and 48$\times$; windowed 9.4$\times$, 11$\times$ and 16$\times$ (H100 GPUs, matched depth). At 512 tokens the dense slowdown is a 9.7$\times$ FLOP ratio times a 3.1$\times$ utilization gap in our kernel.
- Memory: no kernel stores the score tensor; one dense layer needs 1.85 GiB at every context up to 4,096 tokens (20,480 tokens per call), against 1.42 GiB (dilated: 4.1-5.8 GiB).
- Decoding: one token at 2048 tokens costs 1,567 GFLOP of attention against 0.51 GFLOP, quadratic rather than linear in context, and the cache is twice as large per token. Without an optimized decoding kernel we cannot yet give a fair latency; a compiled PyTorch reference takes 660 ms per token at 16,384 tokens, against 1.6 ms.
Summary
We agree the paper does not show higher-order attention to be practical for long-context deployment today.
It shows lower loss in all 34 matched pairs, transferring to downstream accuracy, at a measured cost that the dilated variant brings from cubic to quadratic in context. Thank you for pushing us to measure and report that cost.
Response R1 to Authors
I thank the authors for their efforts to address my concerns. Overall, I believe that W4 and W5 have been substantially addressed through the additional experiments and expanded discussion of related work. However, my remaining concerns are largely unresolved, as the authors primarily acknowledge the limitations raised in my review without providing sufficient additional empirical evidence to address them.
I appreciate the authors' transparency regarding these limitations, and I do not see a need to reiterate the same arguments. Nevertheless, I believe the manuscript, in its current form, is not yet ready for acceptance at ICLR 2027, primarily for two reasons:
-
Insufficient practical significance: Although higher-order attention demonstrates improvements in language-modeling loss, these gains come at substantial computational and memory costs. The manuscript does not establish whether these improvements justify the overhead or offer meaningful advantages over standard attention under comparable resource budgets.
-
Insufficient empirical support for the broader conclusions: The central scaling predictions remain inadequately validated beyond the limited experimental regime. In particular, the absence of experiments at substantially longer context lengths and larger computational budgets makes it difficult to assess whether the predicted advantages materialize in practice. Consequently, I remain unconvinced that the empirical evidence sufficiently supports the broader conclusions regarding the scalability and benefits of higher-order attention.
While I appreciate the authors' additional analysis, my primary concerns remain unresolved. I will therefore maintain my original rating of (1) Clear Rejection.
(1/3) Context length, KV cache and related work
Thank you for your detailed review, for describing our question as "important and well-defined", and for the pointers to related work. You ask whether higher-order attention is practical at scale; at the scales and contexts we study it is not yet: standard attention is cheaper per FLOP, per cached byte and per GPU-hour, and our experiments stop at 2048 tokens. What it offers is the measured accounting needed to judge that: what higher-order attention costs (now including training throughput, memory and decoding, Appendix K.4) and what it buys.
W1. Context lengths too short (dense 512, max 2048)
We agree: experiments up to 2048 tokens do not show how higher-order attention behaves at 32K+, and Appendix B states this limitation.
Validation at 32K+ is the natural next step, and we will release the kernels and training recipe to make it possible.
W2. KV-cache doubling
We agree: higher-order attention caches two keys and two values per token, twice standard attention, and decoding also costs more computation per token (our reply to Q3).
Within the regime we evaluate, the extra memory is not justified for deployment, and we do not claim it is.
W5. Related work: cosFormer, Reformer, RACE Attention, YOSO, vAttention
Thank you; Appendix C now discusses all five, with Expected Attention [1]. cosFormer, Reformer, YOSO and RACE Attention approximate the pairwise softmax with a linearizable kernel, hashing or sampling, and vAttention and Expected Attention reduce the cost of decoding. They lower the cost of pairwise attention, whereas we raise the order of the interaction and use structured sparsity to bring its cost back to that of dense standard attention. The two are complementary: hashing, sampling or cache compression could act on either key axis of the three-way score and on its doubled KV cache.