acceptodds
Under review as a conference paper at ICLR 2027

Scaling Laws for Higher-Order Attention

Abstract

Language-model scaling relies on ever more training data, and treats per-token computation as a cost to minimize. As high-quality data runs out, we ask whether increasing per-token computation can improve performance instead. We compare scaling laws of standard and higher-order attention, where each query attends jointly over pairs of keys, making attention cubic rather than quadratic in sequence length. To enable this, we release dense and dilated higher-order-attention Triton kernels with ALiBi3D positional encoding, supporting long-context training and length extrapolation. At matched parameters and data, higher-order attention attains lower loss; standard attention needs – more tokens to match it. The gain stems from a lower irreducible loss ( nats), not a faster data exponent, and carries over to downstream tasks. We then ask whether higher-order attention is more efficient. We find that it depends on the resource: in our regime, higher-order attention wins per parameter and per token, while standard attention wins per FLOP and per byte of KV cache. Because higher-order attention has a lower loss floor, our fitted laws do predict it overtakes standard attention per FLOP and per byte of KV cache beyond our training scale, signaling the promise of higher-order attention at large compute budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling laws, cubic attention, higher-order attention, attention mechanisms, language models, compute-efficient architectures, sparse attention, length extrapolation, KV cache, Triton kernels, ALiBi

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_A42mOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript investigates whether higher-order (cubic) attention, which models interactions between a query and pairs of keys, offers better scaling properties than standard quadratic attention in language models. The authors introduce techniques for stable training, ALiBi3D positional encoding for context-length extrapolation, and optimized Triton kernels for dense and sparse higher-order attention.

Critical strengths

1.) The manuscript discusses an important and well-defined research question.

Critical weaknesses

1.) The experimental context lengths are extremely short: The claimed scaling advantages are most relevant at larger model sizes and longer contexts, but the actual context experiments stop at 2048 tokens and dense cubic training stops at 512. Results at these lengths are insufficient to establish practical advantages for modern long-context LLMs. In the current landscape of large language models, long-context processing typically refers to sequence lengths of 32K tokens or more.

2.) Increased KV-cache memory requirements raise concerns about practical scalability: Higher-order attention doubles the KV-cache memory requirements, raising concerns about practical scalability, especially at longer contexts. Given the ongoing efforts to reduce KV-cache memory consumption [1], this additional overhead is a significant drawback. While the manuscript provides a valuable study of higher-order attention, its practical applicability remains unclear without demonstrating that the performance gains justify the increased memory costs.

3.) Downstream evaluation does not establish new capabilities: The authors evaluate six benchmarks: ARC-Easy, ARC-Challenge, HellaSwag, MathQA, PIQA, and WinoGrande. They show that downstream accuracy correlates with pretraining loss, but this does not demonstrate that higher-order attention is better at tasks requiring three-way token interactions, long-range retrieval, or complex compositional reasoning.

4.) Sparse attention needs stronger validation: This variant is interesting because it maintains global reach. However, lower loss does not necessarily demonstrate preservation of useful long-range information. Long-context retrieval and reasoning experiments are required for this claim to convince the reader.

5.) Limited discussion of related efficient-attention methods (minor): Although the authors discuss several sparse and linear attention approaches in Appendix C, other relevant works, such as CosFormer [2], Reformer [3], RACE Attention [4], YOSO [5], and vAttention [6], are not discussed. A more comprehensive discussion of these methods and recent developments in efficient attention would help better position the proposed approach within the existing literature, particularly regarding its computational and memory-efficiency tradeoffs.

[1] Devoto, A. and Jeblick, M., et al. Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution. arXiv preprint arXiv:2510.00636.

[2] Qin, Z., Sun, W., et al. cosFormer: Rethinking Softmax In Attention. ICLR 2022.

[3] Kitaev, N., Kaiser, L., et al. Reformer: The Efficient Transformer. ICLR 2020.

[4] Joshi, S., Chowdhury, A., et al. RACE Attention: A Strictly Linear-Time Attention Layer for Training on Outrageously Large Contexts. ICLR 2026.

[5] Zeng, Z., Xiong, Y., et al. You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling. ICML 2021.

[6] Desai, A. et al. vAttention: Verified Sparse Attention via Sampling. ICLR 2026.

Questions for authors

1.) Refer to Weaknesses.

2.) Does higher-order attention learn capabilities that standard attention cannot reproduce at similar computational cost?

3.) How do the practical training and inference costs compare? A clearer end-to-end comparison of throughput, GPU memory, and decoding latency would help establish practical applicability.

4 comments
AuthorsOpenReview ↗

(1/3) Context length, KV cache and related work

Thank you for your detailed review, for describing our question as "important and well-defined", and for the pointers to related work. You ask whether higher-order attention is practical at scale; at the scales and contexts we study it is not yet: standard attention is cheaper per FLOP, per cached byte and per GPU-hour, and our experiments stop at 2048 tokens. What it offers is the measured accounting needed to judge that: what higher-order attention costs (now including training throughput, memory and decoding, Appendix K.4) and what it buys.

W1. Context lengths too short (dense 512, max 2048)

The claimed scaling advantages are most relevant at larger model sizes and longer contexts, but the actual context experiments stop at 2048 tokens and dense cubic training stops at 512. Results at these lengths are insufficient to establish practical advantages for modern long-context LLMs. In the current landscape of large language models, long-context processing typically refers to sequence lengths of 32K tokens or more.

We agree: experiments up to 2048 tokens do not show how higher-order attention behaves at 32K+, and Appendix B states this limitation.

  • Why 512 and 2048. Dense higher-order attention costs cubically in context: a training step at 2048 tokens is 530$\times$ slower than standard attention (Table K.4, new). Scaling-law studies commonly fit at 1-4K tokens (Kaplan et al., 2020; Hoffmann et al., 2022; Busbridge et al., 2025); we fit at 512 and test context separately up to 2048 (Section 6.2).
  • What the context experiments show. The dilated variant, whose cost per token grows linearly with context, matches dense attention up to 512 tokens and stays ahead of standard attention up to 2048 at both model sizes; the windowed variant falls behind (Figure 6.1a). At 2048 tokens it is 48$\times$ slower than standard attention rather than 530$\times$, and ALiBi3D holds its loss at up to four times the training context (Figure 3.1c).
  • What we extrapolate. The crossover moves about 0.85 decades per doubling of context (Figure 5.2b), assuming the law fitted at 512 tokens transfers with only the cost rescaled, as Section 5.2 and Appendix B state.

Validation at 32K+ is the natural next step, and we will release the kernels and training recipe to make it possible.

W2. KV-cache doubling

Higher-order attention doubles the KV-cache memory requirements, raising concerns about practical scalability, especially at longer contexts. Given the ongoing efforts to reduce KV-cache memory consumption [1], this additional overhead is a significant drawback. While the manuscript provides a valuable study of higher-order attention, its practical applicability remains unclear without demonstrating that the performance gains justify the increased memory costs.

We agree: higher-order attention caches two keys and two values per token, twice standard attention, and decoding also costs more computation per token (our reply to Q3).

  • The paper measures this trade-off directly. At equal KV cache, standard attention reaches lower loss across the range we train (Section 5.2); under our best-fitting law, higher-order attention overtakes only at roughly five times the largest standard-attention cache we train (Figure 5.3a), and the break-even model size falls as the training budget grows (Figure 5.3b). Appendix B lists this as a limitation, and the crossover is a prediction, not a demonstrated gain.
  • Compression. Thank you for pointing us to Expected Attention [1], now discussed in Appendix C. Compression methods could act on higher-order attention's cache too, but we have not evaluated them.

Within the regime we evaluate, the extra memory is not justified for deployment, and we do not claim it is.

W5. Related work: cosFormer, Reformer, RACE Attention, YOSO, vAttention

Thank you; Appendix C now discusses all five, with Expected Attention [1]. cosFormer, Reformer, YOSO and RACE Attention approximate the pairwise softmax with a linearizable kernel, hashing or sampling, and vAttention and Expected Attention reduce the cost of decoding. They lower the cost of pairwise attention, whereas we raise the order of the interaction and use structured sparsity to bring its cost back to that of dense standard attention. The two are complementary: hashing, sampling or cache compression could act on either key axis of the three-way score and on its doubled KV cache.

AuthorsOpenReview ↗

(2/3) Downstream evaluation and capabilities

W3. Downstream evaluation does not establish new capabilities

The authors evaluate six benchmarks: ARC-Easy, ARC-Challenge, HellaSwag, MathQA, PIQA, and WinoGrande. They show that downstream accuracy correlates with pretraining loss, but this does not demonstrate that higher-order attention is better at tasks requiring three-way token interactions, long-range retrieval, or complex compositional reasoning.

We agree that our benchmarks do not isolate specialized capabilities such as three-way reasoning or long-range retrieval, and we do not make such claims.

Section 7 instead asks whether higher-order attention's improvement in pretraining loss translates to downstream accuracy. It does: both architectures follow the same accuracy-versus-cross-entropy curve (Figure 7.1; Appendix H), so the lower loss carries over to accuracy, with no additional task-specific lift on these benchmarks.

Relation to prior work. Specialized capabilities have been explored in complementary settings. Clift et al. (2019) proposed 2-simplicial attention as an inductive bias for logical reasoning in deep reinforcement learning. Roy et al. (2025) studied mixture-of-experts models with 2-simplicial attention in every fourth layer, reporting a steeper active-parameter scaling exponent on GSM8k, MMLU, MMLU-Pro, and MBPP negative log-likelihood at a fixed token budget, but no gains below 2B active parameters. Theoretical work by Sanford et al. (2023), Kozachinskiy et al. (2025), and Chakrabarti et al. (2026) establishes representational separations on token-triple tasks.

Our setting and findings differ. We study dense models with higher-order attention in every layer, up to 2.8B parameters, fitting joint scaling laws in parameters and tokens. Unlike Roy et al., we observe improvements at every evaluated model size. The advantage at our scales is a lower loss, not a steeper scaling exponent: higher-order attention has smaller parameter and data exponents, and our data cannot tell whether its floor is lower (Appendix G). On our downstream benchmarks, these improvements are explained by the reduction in cross-entropy. Our results show a general language-modeling advantage at the scales studied; they do not rule out task-specific advantages at other scales or on tasks that require three-way interactions.

Long-range retrieval. We ran needle-in-a-haystack retrieval on the 2048-token models; the results are in our reply to W4 and in Appendix H.1 (new). Token-triple tasks remain untested. We have clarified in Section 7 that our downstream experiments evaluate the transfer of language-modeling improvements to accuracy, rather than specialized capabilities.

Q2. Capabilities standard attention cannot reproduce at similar compute

Does higher-order attention learn capabilities that standard attention cannot reproduce at similar computational cost?

Theoretically, it has been shown that a single higher-order attention layer can represent functions over token triples that a standard attention layer of constant width cannot (Sanford et al., 2023; Kozachinskiy et al., 2025; Chakrabarti et al., 2026), so there are tasks on which the two operators are separated at the representational level. Empirically, at the scales we study, we have not observed such a difference. Across our six benchmarks the two architectures lie on the same accuracy-versus-cross-entropy curve (Figure 7.1), so the advantage of higher-order attention appears as lower loss rather than as a capability that standard attention lacks. In the retrieval test of our reply to W4, standard, dense higher-order and dilated higher-order attention draw comparable information from distant needles, and exact-match accuracy at this scale varies more between training seeds than between architectures, so retrieval shows no capability that standard attention lacks.

AuthorsOpenReview ↗

(3/3) Long-context retrieval, measured cost and summary

W4. Sparse variant: long-context retrieval and reasoning validation

This variant is interesting because it maintains global reach. However, lower loss does not necessarily demonstrate preservation of useful long-range information. Long-context retrieval and reasoning experiments are required for this claim to convince the reader.

We agree that lower loss does not show this, so we ran the retrieval test you ask for (Appendix H.1 and Figure H.2, new).

Setup. Two standard needle-in-a-haystack tasks: passkey retrieval (Mohtashami & Jaggi, 2023) and RULER's single-needle task S-NIAH-1 (Hsieh et al., 2024), with prompt text and samples taken from the original task code. A 5- or 7-digit number is placed at a controlled distance before the question in a 2048-token prompt. We evaluate the 2048-token models of Section 6.2 at both sizes (0.21B and 0.64B parameters), and the 22 sparsity-sweep models of Figure 6.1b (0.21B), without fine-tuning. We measure how much the model draws from the needle as the gain in log-likelihood of the answer over a no-needle control, and also report greedy exact match (computed by teacher forcing; this equals greedy decoding). The no-needle control scores 0% exact match for every model.

Gain in answer log-likelihood over the no-needle control (nats), S-NIAH-1, needles under 256 / at least 512 tokens before the question:

0.21B0.64B
Standard attention21.8 / 19.521.1 / 21.2
Higher-order, windowed19.6 / 0.021.6 / 0.0
Higher-order, dilated21.9 / 22.121.4 / 21.3
  • Dilated attention uses needles anywhere in the context. Its second key axis is dense, so one layer reaches every position, and it draws as much from a needle 2000 tokens back as from one nearby. Passkey agrees (20-21 nats for distant needles).
  • Windowed attention uses only what lies within its window. Beyond its 256-token window, the needle changes the answer likelihood by nothing: it equals the no-needle control. Its stacked receptive field spans the context, but the model does not use it to retrieve. On the Figure 6.1b models this holds for every window from 128 to 1024 tokens: the cutoff sits at the window width (at most 0.33 nats beyond it, 14 to 19 nats within it). Every full-reach variant there (dense higher-order attention, dilated attention at six sparsity levels, and windowed attention with one dense key axis) draws 17 to 23.5 nats from needles 512 to 2048 tokens back, with one exception among 16 models that Appendix H.1 reports.
  • Exact match is too noisy at this scale to rank architectures. At 0.21B, two training seeds of the same configuration differ by up to 86 points in exact match on distant needles. Every failed greedy answer we inspected continues the prompt template ("The pass key is the pass key") instead of giving a wrong number. We report the accuracies in Appendix H.1 but draw no ranking from them.

So the distinction you draw is real, and it separates the two sparse variants: windowed attention's lower loss comes without access to information beyond its window, dilated attention's does not.

Scope. These results hold within the trained 2048-token context, for single-needle retrieval, with at most two training seeds per configuration. We have not run multi-hop reasoning tasks, and Appendix B now lists this as a limitation.

Q3. End-to-end throughput, GPU memory, decoding latency

How do the practical training and inference costs compare? A clearer end-to-end comparison of throughput, GPU memory, and decoding latency would help establish practical applicability.

We agree; the revision adds all three (Table K.4, new), and all favor standard attention.

  • Training: per step, dense higher-order attention is 30$\times$, 105$\times$ and 530$\times$ slower than standard attention at 512, 1024 and 2048 tokens; dilated 11$\times$, 21$\times$ and 48$\times$; windowed 9.4$\times$, 11$\times$ and 16$\times$ (H100 GPUs, matched depth). At 512 tokens the dense slowdown is a 9.7$\times$ FLOP ratio times a 3.1$\times$ utilization gap in our kernel.
  • Memory: no kernel stores the score tensor; one dense layer needs 1.85 GiB at every context up to 4,096 tokens (20,480 tokens per call), against 1.42 GiB (dilated: 4.1-5.8 GiB).
  • Decoding: one token at 2048 tokens costs 1,567 GFLOP of attention against 0.51 GFLOP, quadratic rather than linear in context, and the cache is twice as large per token. Without an optimized decoding kernel we cannot yet give a fair latency; a compiled PyTorch reference takes 660 ms per token at 16,384 tokens, against 1.6 ms.

Summary

We agree the paper does not show higher-order attention to be practical for long-context deployment today.

It shows lower loss in all 34 matched pairs, transferring to downstream accuracy, at a measured cost that the dilated variant brings from cubic to quadratic in context. Thank you for pushing us to measure and report that cost.

Reviewer_A42mOpenReview ↗

Response R1 to Authors

I thank the authors for their efforts to address my concerns. Overall, I believe that W4 and W5 have been substantially addressed through the additional experiments and expanded discussion of related work. However, my remaining concerns are largely unresolved, as the authors primarily acknowledge the limitations raised in my review without providing sufficient additional empirical evidence to address them.

I appreciate the authors' transparency regarding these limitations, and I do not see a need to reiterate the same arguments. Nevertheless, I believe the manuscript, in its current form, is not yet ready for acceptance at ICLR 2027, primarily for two reasons:

  1. Insufficient practical significance: Although higher-order attention demonstrates improvements in language-modeling loss, these gains come at substantial computational and memory costs. The manuscript does not establish whether these improvements justify the overhead or offer meaningful advantages over standard attention under comparable resource budgets.

  2. Insufficient empirical support for the broader conclusions: The central scaling predictions remain inadequately validated beyond the limited experimental regime. In particular, the absence of experiments at substantially longer context lengths and larger computational budgets makes it difficult to assess whether the predicted advantages materialize in practice. Consequently, I remain unconvinced that the empirical evidence sufficiently supports the broader conclusions regarding the scalability and benefits of higher-order attention.

While I appreciate the authors' additional analysis, my primary concerns remain unresolved. I will therefore maintain my original rating of (1) Clear Rejection.

Reviewer_ePN5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the scaling ability of higher-order attention, where each query attends to pairs of keys. It proposes ALiBi3D positional encoding and Triton kernel designs that are compatible with higher-order attention, along with a stabilized training recipe. Notably, higher-order attention achieves lower loss at matched parameter and data budgets. Its sparse variants retain these benefits.

Critical strengths

Clear motivation and very good writing, with comprehensive advances in the field of higher-order attention.

Critical weaknesses

I respect the authors' efforts in this research direction, but my major concern lies in the topic itself. Higher-order attention (or 2-simplicial attention) seems like a very preliminary direction for improving existing models. It provides very limited insights for the community and remains an attempt in industry. I humbly suppose that this topic should not be published at a conference like ICLR. More technically, the paper presents no material evidence for higher-order attention's promise. It only outperforms standard attention under the same number of parameters and data size. According to my Question 1, this is not a convincing advantage.

Questions for authors
  1. Higher-order attention is motivated by scaling computation on high-quality data, and it indeed requires more computation per token. However, this motivation can be achieved in multiple ways, even without scaling the parameters. For example, training the model with more steps, looping layers, or even training and distilling a larger model. What is the unique advantage of higher-order attention for this computation-scaling purpose? Instead, I suppose that the triplet modeling in higher-order attention inevitably incurs more redundant computation.
  2. As Occam's razor states, "entities must not be multiplied beyond necessity". A more complex attention mechanism is definitely harder to justify than efficient attention. I find higher-order attention less elegant than standard attention. For example, the triplet relations captured in it can be replaced by multiple layers of standard attention, thereby supporting the advanced reasoning capability in existing LLMs. What is the unique expressiveness of higher-order attention beyond the one-layer case?
3 comments
AuthorsOpenReview ↗

(1/2) What the study shows beyond matched parameters and data

Thank you for your review. We're happy you found the motivation clear, the writing very good and the advances in higher-order attention comprehensive. We appreciate you engaging with the direction itself, and answer that concern directly.

W1. Topic is preliminary; evidence limited to matched parameters and data

Higher-order attention (or 2-simplicial attention) seems like a very preliminary direction for improving existing models. [...] It only outperforms standard attention under the same number of parameters and data size.

We agree the direction is early. We do not argue that higher-order attention should replace standard attention; the paper asks when it is worth its cost, and measures that per resource.

  • Why study it. Public high-quality data is finite (Villalobos et al., 2024), which shifts the question from how much data to use toward how much computation to spend per token (Section 1). Higher-order attention spends more computation per token at a fixed parameter count. A single higher-order layer can represent functions that a standard layer of constant width cannot (Sanford et al., 2023; Kozachinskiy et al., 2025), and Roy et al. (2025), with it in every fourth layer of mixture-of-experts models, report a steeper parameter exponent on mathematics, knowledge and coding benchmarks (GSM8K, MMLU, MBPP).
  • The evidence goes beyond matched parameters and data. We compare the architectures under four resources: parameters, tokens, training FLOPs and KV-cache bytes (Section 5), and the revision adds measured GPU time (Appendix K.4, new). Standard attention wins at our scales per FLOP and per cached byte, which the paper states as a main finding (Section 1, finding 3), and per GPU-hour on current kernels (Appendix K.4). Higher-order attention wins per parameter and per token: lower loss in all 34 matched pairs, standard attention needing at least 9% more tokens to match its loss at every model size we train under either fitted law, and an advantage that transfers to downstream accuracy (Section 7).
  • What the paper contributes. Kernels and a training recipe that train dense higher-order attention stably in every layer up to 2.8B parameters (Sections 3-4), and we will release the kernels; ALiBi3D, a positional encoding for three-way scores that extrapolates to four times the training context; and a per-resource accounting of what higher-order attention costs and what it buys.
AuthorsOpenReview ↗

(2/2) Other ways to scale per-token compute; expressiveness

Q1. Why higher-order attention rather than more steps, looped layers, or distillation?

What is the unique advantage of higher-order attention for this computation-scaling purpose? Instead, I suppose that the triplet modeling in higher-order attention inevitably incurs more redundant computation.

We agree that there are several ways to spend more compute per token, and we do not claim that higher-order attention is the best of them. The options differ in where the computation goes:

  • More training steps spend more compute on the dataset rather than on each token; in the data-limited setting that motivates the paper (Section 1), they mean repeating data.
  • Distillation requires the larger teacher to be trained first.
  • Looped layers, like chain-of-thought at inference, add serial computation per token: the same layer applied again, or more generated tokens.
  • Higher-order attention adds parallel computation within one layer, by scoring pairs of keys jointly.

Looping is the closest alternative, and it runs on standard kernels today. Which of the two makes better use of a FLOP is an open question that a matched comparison would settle, and we agree it is worth running.

Our paper supplies the higher-order side of that comparison: a recipe and kernels that train higher-order attention stably in every layer up to 2.8B parameters (Sections 3-4), scaling laws fitted per architecture and compared per resource (Section 5), and a measured cost ledger (Appendix K.4, new). It reaches a given loss on fewer tokens: standard attention needs at least 9% more tokens to match it at every model size we train, under either fitted law, and the advantage transfers to downstream accuracy (Section 7). Per training FLOP, and per GPU-hour on current kernels, standard attention is the better choice at our scales (Section 5.2, Appendix K.4).

On redundancy, you are right that the extra computation is substantial: about 10$\times$ the training FLOPs of standard attention at 512 tokens (Table K.4). If it were redundant, equal parameters and tokens would give equal loss, but higher-order attention has lower loss in all 34 matched pairs.

Q2. Occam's razor; expressiveness beyond stacked standard attention layers

A more complex attention mechanism is definitely harder to justify than efficient attention. [...] What is the unique expressiveness of higher-order attention beyond the one-layer case?

We agree that a more complex mechanism has to earn its place, and that is the question the paper asks. On expressiveness, the separation is proven for one layer and conjectured beyond it:

  • One layer. Sanford et al. (2023) show that a single layer of standard multi-head attention cannot decide, for each token, whether two others sum with it to zero (Match3) unless its number of heads or embedding dimension grows polynomially with sequence length, while a single third-order attention unit with embedding dimension 5 can. Kozachinskiy et al. (2025) prove such limits for one-layer softmax transformers at any precision, for Match3 and for function and relation composition.
  • More layers. Sanford et al. conjecture that standard transformers of any depth need width, depth, embedding dimension or precision growing polynomially with sequence length to solve Match3, and support this with a heuristic information-theoretic argument; it is not proven. As you say, stacked standard layers can capture many three-way relations, and whether they match higher-order attention at equal cost is open.

Our scaling results point the same way. A law in which both architectures share one loss floor fits about as well as separate floors (p=0.097p = 0.097; Section G.5), and under it standard attention reaches the same loss as higher-order attention with more parameters and data. A lower floor for higher-order attention would be the empirical counterpart of a separation at scale, and our data cannot tell whether the floors differ.

Our case for higher-order attention therefore does not rest on expressiveness; it is empirical. At matched parameters and tokens it reaches lower loss in all 34 matched pairs, and the advantage transfers to downstream accuracy (Section 7). We report this beside its costs, per FLOP and per GPU-hour, where standard attention is the better choice at our scales (Section 5.2, Appendix K.4), so that the added complexity can be weighed against what it buys.

Thank you again for engaging with the direction itself - your questions have sharpened how the paper states its case.

The rebuttal has improved the overall quality of this work, but it has not resolved my central concern about whether higher-order attention offers a sufficient advantage over alternative ways of allocating additional computation. Therefore, we would like to keep my rating.

Reviewer_t6CZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an extensive empirical study of cubic attention, which models interactions among three tokens rather than the pairwise interactions used in conventional attention. The paper investigates its scaling behavior in language modeling and examines architectural choices, including positional encoding and sparsity. Its central finding is that cubic attention can achieve a given language-modeling loss with fewer training tokens, while the relationship between loss and downstream performance remains broadly similar to that of conventional attention. The study provides substantial experimental evidence, although the practical efficiency and persistence of the gains at larger scales require further clarification.

Critical strengths
  1. The paper presents a clear and informative empirical narrative.

The distinction between token efficiency and the relationship between loss and downstream performance is valuable. The finding that cubic attention requires fewer tokens to reach a given loss, without substantially changing how that loss translates into downstream performance, provides a coherent account of where its benefits arise. The paper is supported by a substantial collection of experiments and results, making the empirical observations reasonably convincing within the evaluated settings.

  1. The architectural ablations are informative.

The ablations on positional encoding and sparsity provide useful insights into the design of cubic attention. These experiments help readers understand how specific architectural choices affect the results, rather than presenting cubic attention as a single fixed configuration. They are a meaningful contribution to the paper’s empirical analysis.

Critical weaknesses
  1. The practical benefit is insufficiently established once computational cost and context length are considered.

Improved token efficiency does not necessarily imply improved computational efficiency. Given the additional cost of cubic attention, the paper needs to establish whether the reduction in required training tokens compensates for the increased cost of processing them. This concern is particularly relevant to long-context language modeling and generation with long reasoning traces.

Since the paper focuses on language modeling and emphasizes scaling laws, compute-aware comparisons are central to evaluating its contribution. Comparisons in terms of training FLOPs, wall-clock time, peak memory, and inference cost across relevant context lengths would make the practical implications much clearer. These comparisons should also account for the sparse variants studied in the paper.

  1. The baseline comparisons in Figure 3.1(c) are not sufficiently convincing.

The baselines in Figure 3.1(c) appear relatively weak, which limits the strength of the conclusions drawn in this section. Stronger, well-tuned conventional-attention baselines would help establish whether the observed advantage is attributable to cubic attention rather than to limitations of the comparison setup. The relevant architectural choices, tuning procedures, and training budgets should be clearly reported. In particular, matched-token and matched-compute comparisons should be distinguished, as they answer different questions.

  1. The persistence of the advantage at larger scales remains unclear.

The results in 5.1 suggest that the gap between cubic and conventional attention narrows as (N) increases. This raises an important question about whether the observed benefit persists at larger scales or is primarily a finite-scale effect. The paper should explicitly discuss this trend and qualify its broader scaling claims accordingly. A narrowing gap does not establish that the advantage will disappear, but it does require more careful interpretation.

Context length is another important dimension that could substantially alter the comparison. A trend established at one context length need not hold at longer lengths, particularly when the competing architectures have different attention costs. Evaluating context length as a separate scaling axis would strengthen the analysis.

  1. Figure 5.2 should show the empirical observations underlying the fitted curves.

The measured data points should be overlaid on the fitted curves in Figure 5.2. Smooth curves on log–log axes alone are insufficient for assessing the quality and robustness of a scaling-law fit. Without the underlying observations, readers cannot readily determine whether the fitted relationship is broadly supported or sensitive to the selection of points and fitting range.

The paper should clearly identify the observations used for fitting, distinguish interpolation from extrapolation, and report appropriate uncertainty or fit diagnostics. An analysis of how the estimated scaling behavior changes with the fitting range would further strengthen confidence in the conclusions.

Questions for authors
  1. How do the authors interpret the narrowing gap as (N) increases in 5.1? What evidence supports the persistence of the advantage beyond the evaluated scales?

  2. Can Figure 5.2 include the underlying measurements, uncertainty estimates, and fitting ranges? How sensitive are the inferred scaling trends to the points included in the fit?

3 comments
AuthorsOpenReview ↗

(1/2) Benefit per resource; Figure 3.1(c) baselines

Thank you for the detailed and constructive review. We're happy you found the distinction between token efficiency and how loss transfers downstream "a coherent account of where its benefits arise", and the positional-encoding and sparsity ablations "a meaningful contribution". Since submission we have added a 3×10203\times10^{20}-FLOP profile (three times our previous largest budget, up to 2.8B parameters), a profile at a fixed 64B tokens, and two runs off the IsoFLOP profiles, with a refit on 34 matched runs per architecture (Appendix G). Several of our answers to your questions draw on them.

W1. Practical benefit once compute and context are counted

We agree that token efficiency does not imply compute efficiency, and within our budgets it does not: per training FLOP and per byte of KV cache, standard attention is the better choice (Section 5.2).

We report efficiency per resource because the verdict differs across them, and the paper claims the token advantage as one axis among four.

End-to-end training step time at nctx=512n_{\mathrm{ctx}} = 512, relative to standard attention, on identical hardware and parallelism (32 H100 GPUs, 655,360 tokens per step):

  • Dense: 10$\times$ at 64M parameters, 22$\times$ at 206M, 30$\times$ at 870M.
  • Dilated: 4.0-11$\times$.
  • Windowed: 3.4-9.4$\times$.

These exceed the FLOP ratios (about 9-11$\times$ dense, 2.2-2.9$\times$ sparse) because our kernels reach lower hardware utilization than FlashAttention-2. A token saving of at least 9% does not offset a 10-30$\times$ cost per step at these scales. We see the paper as an investigation of whether this extra per-token computation has utility, and the answer at our scales is a modest but consistent advantage per parameter and per token, bought at a much higher cost per FLOP and per GPU-hour. We think that is worth knowing where data, rather than compute, is the limiting resource.

The gap grows with context: at 2048 tokens and 2.8B parameters, dense higher-order attention is 530$\times$ slower per step and the dilated variant 48$\times$. Working memory grows little, since no kernel stores the score tensor: one dense layer needs 1.85 GiB at every context up to 4,096 tokens (20,480 tokens per call), against 1.42 GiB for standard attention (dilated: 4.1-5.8 GiB). Decoding one token at 2048 tokens costs 1,567 GFLOP of attention against 0.51 GFLOP, with twice the KV cache per token. Appendix K.4 (new) collects these numbers in one table and splits the slowdown into the FLOP ratio and a kernel utilization gap.

W2. Baselines in Figure 3.1(c)

All three curves in that panel are higher-order attention; none is a standard-attention baseline.

The panel compares positional encodings for the cubic operator, trained at nctx=256n_{\mathrm{ctx}} = 256 and evaluated at up to 4$\times$ that length:

  • The comparators: RoPE is absent because its relative-position property does not survive the trilinear score (Appendix E.1). The encodings well defined for a three-way score are none, absolute (sinusoidal) and additive biases (ALiBi3D), so the panel compares those.
  • In distribution: the ALiBi3D exponent sweep (Figure 3.1b) shows in-distribution loss is insensitive to the exponent; ALiBi3D's advantage is extrapolation.
  • Setup: 8 layers (0.22B non-embedding parameters), trained at nctx=256n_{\mathrm{ctx}} = 256 on 40 tokens per parameter (8.9B tokens of the same Nemotron-CC data). All arms match tokens and compute and share the architecture's tuned learning rate, which was not re-tuned per encoding. The ALiBi3D curve averages nine seeds; NoPE and sinusoidal are single runs.

Appendix L now gives this setup. We agree the panel would be stronger with learned relative biases such as T5's or FIRE. Both would need new forward and backward kernels for the three-way score, which are out of scope for this investigation, so we concede this point and have added it to our limitations (Appendix B).

The comparison between the two architectures is in Section 5, at matched parameters and tokens (Figure 5.1a) and at matched real compute (dotted curves; Section 5.1). We agree these answer different questions, and keep them separate.

AuthorsOpenReview ↗

(2/2) Why the gap narrows; observations against the fits

W3 and Q1. The narrowing gap

We agree the gap narrows with NN, and the new data let us separate the axes.

  • Measured. On the fixed-64B profile, which varies NN at constant tokens, the advantage at equal parameters falls from 0.074 nats at 93M to 0.020 at 680M, interpolated between neighboring runs, with no fitted law. Holding NN fixed and adding tokens, by contrast, shows no consistent trend in our runs. The narrowing is therefore a real effect of model size, not only of falling tokens per parameter along each profile. The gap stays positive in all 34 matched pairs, including the largest (0.024 nats at 2.4B/2.8B parameters on the 3×10203\times10^{20} profile, 0.036 at 20 tokens per parameter).
  • Why it narrows. The narrowing is the shape power laws produce: as the reducible loss shrinks, so does any difference in it, measured in nats, and the gap approaches the floor difference ΔE\Delta E as NN and DD grow together. Higher-order attention's smaller exponents (α\alpha 0.59 vs 0.67, β\beta 0.55 vs 0.66; a law with shared exponents is rejected, p=0.001p = 0.001) mean its reducible loss also falls more slowly, which narrows the gap further. The form of the law is symmetric in model size and data: on log axes, an IsoFLOP profile trades factors of NN one-for-one against factors of DD, and swapping NN and DD only swaps their coefficients. Which axis moves the gap depends on which term dominates the reducible loss. The fixed-64B profile sits at 65-720 tokens per parameter, where the fitted parameter term dominates (about four times the data term at 300M parameters), so the gap moves with NN and barely with DD, as measured above. At few tokens per parameter the same law predicts the mirror image: the gap narrows along DD and barely along NN.
  • Whether it persists depends on ΔE\Delta E, which our data cannot settle (0.117 [−0.006,0.193]0.117\,[-0.006, 0.193]; a law with a shared floor is not rejected, p=0.097p = 0.097; Section G.5). Under the best-fitting law the advantage stays positive (interior minimum 0.015 nats) and grows along the compute-optimal frontier (Figure 5.1b). Under the shared floor it closes at large scale and turns slightly negative in part of the extrapolated plane (minimum −0.003-0.003).

Context length. Dense higher-order attention costs cubically in context, so we train it only up to 512 tokens; the sparse variants exist for this reason (Section 6.2). Up to 2048 tokens, at 0.21B and 0.64B parameters, the dilated variant stays ahead of standard attention, while the windowed variant falls behind as context grows (Figure 6.1a). Section 5.2 estimates that the crossover moves about 0.85 decades per doubling of context, assuming the law fitted at 512 tokens transfers with only the cost rescaled (Appendix B); a second fitted context length would test this.

W4 and Q2. Observations, fitting ranges, sensitivity

We agree that fitted laws should be shown against the data they come from, and the revision now does this throughout.

  • Observations against the fit. Figure 5.2 shows something different: both panels are derived from the fitted laws, (a) by minimizing the fitted loss over model size at each budget, which no trained run does exactly, and (b) by extrapolating the crossover to context lengths we did not fit, so no measured run lies on them. We therefore overlay the observations wherever the axes are quantities of the trained runs. The new Figure G.1 does this for extrapolation: it fits the four smaller budgets, overlays the held-out runs at the larger budget and at more tokens with 95% predictive bands, and shades NN beyond the largest fitted model. Figure G.2 plots predicted against observed loss with relative errors, Table G.4 lists every fitted run, and Figure 5.1a shows all IsoFLOP observations. Figure 5.2a now shades the fitted budget range.
  • Uncertainty. Paired-bootstrap intervals for every coefficient (Table G.1); the crossover band in Figure 5.2b counts bootstrap draws without a crossover as infinite.
  • Sensitivity to the points included. - The original fit, on the four submitted budgets, predicts the new runs with 0.25-0.73% error along compute but overestimates the loss of every run trained on more tokens, which is why the revised fit includes them; its 95% bands covered 58% of the new runs. - The revised fit, leaving out one group at a time, gives 0.40% error and 75% coverage. - Fitting only runs above a loss threshold overestimates the lower-loss runs by 0.2-1.4% on average (Figure G.2). - ΔE\Delta E turns negative when only the highest-loss half is fitted (Section G.5).

The measured advantage does not depend on which runs are fitted; the floor and the crossovers do, as we now state in Appendix B.

Thank you again for your detailed comments - they have made the paper's scaling analysis more careful.

Thank you for the detailed response and the substantial additional experiments. I appreciate the clearer accounting of computational costs and the expanded scaling analysis. I also accept the clarification that Figure 3.1(c) compares positional encodings within cubic attention, and the additional observations and fit diagnostics improve the transparency of the analysis. However, I remain unconvinced that the response resolves my central concerns.

W1 and W3/Q1: The additional results strengthen my concern that scaling up makes the practical trade-off less favorable.

The fixed-64B-token analysis confirms that the equal-parameter loss advantage decreases as model size increases, falling from 0.074 nats at 93M parameters to approximately 0.020 nats at 680M. Meanwhile, the computational burden becomes particularly severe at longer contexts: at 2,048 tokens and 2.8B parameters, the reported training-step slowdowns are 530× for dense cubic attention and 48× for the dilated variant. Although these runtime penalties include kernel-utilization effects, the response also explicitly acknowledges that standard attention remains preferable in terms of training FLOPs and KV-cache efficiency.

These results suggest an increasingly unfavorable combination: scaling model size reduces the observed benefit, while scaling context length magnifies the computational burden. This is especially concerning for the paper’s application to language modeling, including long-context inputs and long reasoning traces. The sparse variants are relevant, but the reported results do not yet demonstrate that they adequately resolve this trade-off. The data-limited motivation is reasonable; however, the modest token-efficiency gains do not establish a sufficiently compelling benefit given the substantial additional computational cost.

W3/Q1 and W4/Q2: The persistence of the advantage remains dependent on uncertain extrapolation.

I acknowledge that the advantage remains positive across the 34 matched pairs. My concern is whether this supports the broader scaling conclusions. As the response explains, the extrapolated behavior depends on the fitted loss floors, which the available data cannot reliably distinguish. The shared-floor alternative is not rejected and permits the advantage to vanish or slightly reverse in parts of the extrapolated regime. Consequently, the preferred fit’s prediction of a persistent advantage is not yet convincing evidence that this benefit will survive further scaling.

The context-length extrapolation adds another uncertainty, since it assumes that a law fitted at 512 tokens transfers to longer contexts after rescaling the cost. The additional diagnostics make these limitations clearer, but do not resolve them.

Overall, I continue to find the empirical investigation informative, but the additional results reinforce my concerns about the magnitude, scalability, and practical relevance of the benefits. I therefore maintain my original score of 2 (weak rejection).

Reviewer_aS2DOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper represents a scaling-law study of whether increasing per-token computation with high-order attention is worth its added cost. To apply higher-order attention at every layer, the authors introduce separate normalization of value vectors and ALiBi3D positional encoding, and implement a new backward kernel that works for any window size. In experiments, higher-order attention reaches lower loss than standard attention at the same model size and amount of data, which the authors attribute to a lower loss floor. However, in terms of training compute and KV cache memory, standard attention is more efficient within the test range, and the point where higher-order attention overtakes it is given only as a prediction. In addition, the sparse variant keeps performance while greatly reducing cost, and the lower loss also carries over to benchmark accuracy.

Critical strengths

A key strength of this paper is that it does not evaluate the efficiency of higher-order attention by a single criterion, but compares it separately under four cost criteria: parameters, tokens, training compute, and KV cache memory. This shows that higher-order attention is favorable in terms of parameters and tokens, while it is less efficient than standard attention in terms of compute and memory within the tested range. Also, by reporting where the method falls short, the paper gives a more balanced conclusion than prior single-criterion work.

Critical weaknesses

The paper’s main conclusions rely more on extrapolation than on measurement. Within the tested range, standard attention is more efficient in both training compute and KV cache memory, and the claimed compute advantage of higher-order attention appears only at a budget about 1000 times larger than the largest tested. This prediction rests on a lower estimated loss floor, which is the least certain fitted coefficient and whose difference includes zero under independent bootstrapping.

Moreover, the cost accounting itself may understate the true cost. Training cost is reported only in theoretical FLOPs, with no kernel throughput or wall-clock time, and inference cost is compared only by KV cache size, ignoring that higher-order attention needs computation quadratic in context length for each generation token. The crossover point may therefore lie even further away, and a throughput-based comparison should be added.

Questions for authors

Which positional encoding is used for the standard attention baseline? Section 2.4 and Appendix L only describe ALiBi3D for higher-order attention. If the baselines uses ALiBi rather than RoPE, could the authors discuss whether this weakens the baseline, and report results with a RoPE baseline if available?

3 comments
AuthorsOpenReview ↗

(1/2) Extrapolation now conditional on the floor; measured cost added

Thank you for the clear summary and the thoughtful review. We're happy you found that comparing the architectures under four separate cost criteria, and reporting where the method falls short, gives "a more balanced conclusion than prior single-criterion work". We agree with both weaknesses; they led to new runs and measurements.

W1. Conclusions rely on extrapolation; the floor difference includes zero

We agree.

To address this, we trained both architectures at a budget three times our previous largest (3×10203\times10^{20} FLOPs, up to 2.8B parameters), on a profile at a fixed 64B tokens, and on two runs off the IsoFLOP profiles, and refit both laws on 34 matched runs per architecture (Appendix G). We also predicted the new runs from the submitted ones.

  • Measured, with no fit: higher-order attention has lower loss than standard attention at matched depth and training budget in all 34 pairs, and at equal parameters and tokens, interpolated along the fixed-64B profile (0.074 nats at 93M parameters, 0.020 at 680M). This transfers to downstream accuracy (Figure 7.1).
  • The floor: the refit gives ΔE=0.117\Delta E = 0.117 nats, with paired-bootstrap interval [−0.006,0.193][-0.006, 0.193], which now includes zero. A law with one floor for both architectures fits about as well and is not rejected by a calibrated test (p=0.097p = 0.097; Section G.5). We therefore no longer attribute higher-order attention's lower loss at matched NN and DD to the floor: the revised abstract and Section 5.1 say that our data cannot tell whether the floors differ.
  • The crossover: under the best-fitting law it is 1.4×10231.4\times10^{23} FLOPs, about 500$\times$ our largest budget; under the shared floor it moves to about 7×10267\times10^{26}. The revision presents it as conditional on the floor difference, an extrapolation rather than a finding (Section 5.2, Appendix B).
  • The law itself: the original fit predicted too high a loss for runs trained on more tokens (its 95% bands covered 58% of the new runs), which is why the revised fit includes them (Section G.6). Its remaining bias along tokens means extrapolations carry more uncertainty than their bootstrap intervals suggest, as we now state in Appendix B.

W2. Cost accounting understates the true cost

We agree on both counts.

Training. Measured end-to-end step time at nctx=512n_{\mathrm{ctx}} = 512 on identical hardware and parallelism (32 H100 GPUs, 655,360 tokens per step), relative to standard attention:

Depth (standard NN)DenseDilatedWindowed
4 (64M)10$\times$4.0$\times$3.4$\times$
8 (206M)22$\times$7.9$\times$6.8$\times$
15 (870M)30$\times$11$\times$9.4$\times$

Dense higher-order attention costs about 9-11$\times$ the forward FLOPs of standard attention at this context, so the kernel adds a utilization penalty of up to about 3.1$\times$ at this context, growing with size. A wall-clock crossover would therefore lie further out than the per-FLOP one, as you anticipate. At 2048 tokens and 2.8B parameters the slowdown is 530$\times$ (dense) and 48$\times$ (dilated); Appendix K.4 (new) splits each slowdown into the FLOP ratio and the utilization gap.

Inference. You are right that per-token decode costs higher-order attention O(nctx2)O(n_{\mathrm{ctx}}^2) operations against O(nctx)O(n_{\mathrm{ctx}}) for standard attention, on top of the 2$\times$ KV cache. Our KV-cache comparison counts memory only, so it understates inference cost. At nctx=2048n_{\mathrm{ctx}} = 2048 and 22 layers, decoding one token costs 1,567 GFLOP of attention for dense higher-order attention against 0.51 GFLOP for standard attention, and every higher-order variant caches 484 KiB per token against 242 KiB. We have not written an optimized decoding kernel; a compiled PyTorch reference takes 660 ms per token at 16,384 tokens, against 1.6 ms for standard attention. The revision reports these measurements in Appendix K.4 (Table K.4, new), and the KV-cache comparison should be read as a lower bound on inference cost.

AuthorsOpenReview ↗

(2/2) Baseline positional encoding: ALiBi, with a RoPE comparison

Q1. Baseline positional encoding

The standard-attention baseline uses ALiBi in every run (geometric per-head slopes); the revision now states this in Appendix L.

Thank you for raising this, as it lets us make the choice explicit. Our question is what changing the attention mechanism does, not which complete architecture performs best, so we keep everything else fixed, including how position is encoded. Both architectures therefore use additive distance biases, with ALiBi3D as the three-way extension of ALiBi. RoPE could not play this role, because its relative-position property does not survive the trilinear score (Appendix E.1).

We agree that RoPE is the stronger encoding for standard attention, and we measured how much. On the same data, recipe and held-out sample as the paper, at 8 layers and 30 tokens per parameter, with each encoding's learning rate tuned over a bracketing grid, RoPE lowers standard attention's loss by 0.026 nats relative to ALiBi at each encoding's best learning rate, and by 0.031 ±\pm 0.004 nats paired over three seeds at a shared learning rate. Against that tuned RoPE baseline, higher-order attention with ALiBi3D at the same depth and token ratio is still 0.029 nats lower, against 0.055 for the ALiBi baseline; its own learning rate was not re-tuned for this comparison. So a stronger positional encoding for the baseline narrows the gap but does not close it at this scale.

The revision reports this measurement in Appendix L, and states in Appendix B that our comparison holds the positional mechanism fixed rather than giving each architecture its best available encoding.

Thank you again for your thoughtful comments - they made the paper clearer about what it measures and what it predicts.

I appreciate the authors’ responses and the additional experiments, which help clarify several of my concerns. However, the results still leave open questions about the extrapolation, comparisons with stronger baselines, and practical efficiency. Given these remaining concerns, I will keep my original rating of 2.

Reviewer_j2FpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether increasing computation per token can be a useful scaling axis when high-quality training data is limited. Its higher-order—or “cubic”—attention uses one query, two keys, and two values. Thus, each query attends to ordered pairs of previous tokens rather than individual previous tokens.

To make the higher-order attention trainable, the authors proposed to use separate RMS normalization of the two value streams and freeze key normalization gains to avoid cubic gain amplification. They also use da sparse multi-scale construction that keeps one key axis globally accessible and provided customized dense, windowed, and dilated Triton kernels.

Critical strengths
  1. Efficiency analysis on multiple metrics. This paper does not collapse efficiency into one metric. It separately analyzes on different factors, including parameters, training tokens, training FLOPs, KV-cache cost and context length. This produces a much more honest conclusion and the authors explicitly acknowledge that standard attention is better under the two resources that currently matter most: FLOPs and inference cache.

  2. The experimental scale is meaningful. The model family extends to approximately 1.62B non-embedding parameters for standard attention and 1.85B for higher-order attention. That is large enough for the language-modeling result to be considerably more persuasive than small synthetic experiments. Three seeds per IsoFLOP cell also strengthen the primary scaling study.

Critical weaknesses
  1. The irreducible-loss claim is not firmly established. There are six fitted coefficients for each architecture’s scaling law, but only 21 averaged observations. The irreducible loss �0��0� is inherently determined by extrapolating beyond the measured loss range, and the authors identify �1��1� as the least constrained coefficient. However, The reported floor difference is 0.1340.134, with its paired-bootstrap 95% interval [0.007,0.358][0.007,0.358]. Under independent bootstrapping, the interval becomes [−0.040,0.403][-0.040,0.403], which includes no floor advantage.

  2. The predicted compute crossover is extremely far outside the data. The claimed FLOP crossover occurs around 102310^{23} FLOPs, approximately 1,000 times beyond the largest trained budget.

  3. Architectural attribution is incomplete. Higher-order attention differs from standard attention in several ways, such as two additional projection matrices, multiplicative keys and values, etc. Matching total parameters does not isolate which component causes the improvement.

Questions for authors
  1. How well does each scaling law predict a held-out compute profile or held-out model size?

  2. What is the measured end-to-end throughput ratio between standard, dense cubic, and dilated cubic attention at each context length?

2 comments
AuthorsOpenReview ↗

(1/2) New runs; the floor difference and crossover are now stated as unresolved

Thank you for the careful review. We're happy you found the per-resource accounting "a much more honest conclusion". You are right that our data did not establish the floor difference or the crossover. Your W1, W2 and Q1 led to new experiments (Appendix G), and we have softened the paper accordingly. The revised abstract, introduction and conclusion say our data cannot tell whether the floors differ, and that the per-FLOP crossover is near 102310^{23} FLOPs under our best-fitting law but beyond 102610^{26} under a shared floor.

What we added

For both architectures we trained a 3×10203\times10^{20}-FLOP IsoFLOP profile (three times our previous largest budget, up to 2.4B and 2.8B non-embedding parameters), a profile at a fixed 64B tokens, and two runs off the IsoFLOP profiles.

Every run has a twin of the other architecture at the same depth, trained on the same compute budget or the same number of tokens. We refit both laws on all 34 runs per architecture with the same fitting procedure, now with confidence intervals from a paired bootstrap over matched runs (Section G.4). Section 5 and every fitted figure now use this fit, and Table G.4 lists the runs.

W1. The floor difference is not established

We agree, and the new data confirm it.

The refit gives ΔE=0.117\Delta E = 0.117 nats, with paired-bootstrap 95% interval [−0.006,0.193][-0.006, 0.193], which includes zero (independent resampling: [−0.038,0.203][-0.038, 0.203]; Table G.5). A law with one floor for both architectures fits about as well, and a formal test does not reject it (p=0.097p = 0.097; Section G.5). The estimate also depends on which runs are fitted: it turns negative when only the higher-loss half of the runs is fitted. The revised abstract, Section 5.1 and Appendix B now say that our data cannot tell whether the floors differ, and no longer attribute higher-order attention's lower loss at matched NN and DD to the floor.

These results do not depend on the floor:

  • At matched depth and training budget, higher-order attention has lower loss in all 34 pairs (0.008-0.108 nats).
  • At equal parameters and tokens, interpolating standard attention's loss to higher-order attention's NN along the fixed-64B profile (no fitted law), the advantage is 0.074 nats at 93M parameters and 0.020 at 680M.
  • Under either law, at every model size we train, standard attention needs at least 9% more tokens to match the loss.
  • A law with shared exponents is rejected (p=0.001p = 0.001): higher-order attention has smaller parameter and data exponents.

W2. The crossover is far outside the data

We agree.

Under our best-fitting law, the per-FLOP crossover at nctx=512n_{\mathrm{ctx}} = 512 is 1.4×10231.4\times10^{23} FLOPs, about 500$\times$ our largest budget. It appears in 96% of 400 paired bootstrap draws, with 90% interval [5.9×1022,1.5×1026][5.9\times10^{22}, 1.5\times10^{26}]. Under the shared-floor law it moves to about 7×10267\times10^{26}. It does not vanish there, because the attention share of higher-order attention's extra cost shrinks as models widen. The revision presents the crossover as conditional on the floor difference (Sections 5.2, G.5), not as a finding. On current kernels, a wall-clock crossover would lie further out still (Q2).

W3. Attribution

We agree that matching parameters does not fully isolate which component causes the improvement.

It does show that parameter count alone does not explain it. The extra K2K^2 and V2V^2 projections raise the attention parameters per layer from 4d24d^2 to 6d26d^2 (Appendix K.2), all counted in NN, and at equal parameters and tokens higher-order attention is still 0.020-0.074 nats lower (see W1). Separating multiplicative keys from multiplicative values needs controls we have not run. Replacing the trilinear score with an additive one, q⋅kj1+q⋅kk2q\cdot k^1_j + q\cdot k^2_k, factorizes the joint softmax, so the output becomes the elementwise product of two standard attention outputs; summing the two outputs instead also removes the multiplicative values. Both controls keep the parameters and the 2$\times$ KV cache of higher-order attention at quadratic cost, so comparing them with standard and higher-order attention may shed light on which component causes the improvement, though it would still not fully resolve it. We have added this to the limitations (Appendix B).

AuthorsOpenReview ↗

(2/2) Held-out validation and measured throughput

Q1. Held-out prediction

The original fit extrapolated well to larger budgets but not to more tokens, which is why the revision refits on runs that span both.

  • Original fit (21 runs), predicting the 13 new runs: to larger budgets, 0.25% mean relative error for higher-order attention and 0.73% for standard attention; to more tokens, both laws predicted too high a loss, by 0.033 nats (standard) and 0.018 (higher-order) on average. Its 95% bands contained 58% of the new runs.
  • Revised fit (34 runs), leaving out one group at a time: 0.40% mean relative error with no systematic sign, except on the 64B profile, overestimated by about 0.02 nats when held out. Its bands contain 75% of held-out runs.

Every number in the revision comes from the revised fit (Section G.6, Figures G.1-G.2). Its remaining bias along tokens means extrapolations carry more uncertainty than their bootstrap intervals suggest, as we now state in Appendix B.

Q2. Measured throughput

Per training step, dense higher-order attention is 30$\times$ slower than standard attention at 512 tokens, 105$\times$ at 1024 and 530$\times$ at 2048; the dilated variant is 11$\times$, 21$\times$ and 48$\times$ (matched depth, 1B parameters at 512 and 2.8B at 1024 and 2048, H100 GPUs; Table K.4, new).

At 512 tokens the slowdown also grows with model size. Steady-state end-to-end step time relative to standard attention, on identical hardware and parallelism (32 H100 GPUs, 655,360 tokens per step), with FLOP ratios in parentheses:

Depth (standard NN)DenseDilatedWindowed
4 (64M)10$\times$ (9.3$\times$)4.0$\times$ (2.5$\times$)3.4$\times$ (2.2$\times$)
8 (206M)22$\times$ (11$\times$)7.9$\times$ (2.9$\times$)6.8$\times$ (2.5$\times$)
15 (870M)30$\times$ (9.7$\times$)11$\times$ (2.6$\times$)9.4$\times$ (2.3$\times$)

The wall-clock gap exceeds the FLOP gap and grows with size: our kernel's utilization stays flat while standard attention's rises with width. A kernel profile places the gap in instruction overhead rather than memory traffic: per tensor-core FLOP, our kernel issues 5-16$\times$ the instructions of FlashAttention. The gap beyond the FLOP ratio is headroom in our kernel, which is not tuned for throughput: a kernel at standard attention's utilization would sit at the FLOP ratio (9.7$\times$ for dense attention at 512 tokens). Under wall-clock accounting on current kernels, the crossover would therefore lie further out still.

Two results suggest part of this gap can be closed. The dilated variant matches dense higher-order attention's loss with cost that grows linearly rather than quadratically in context (measured slopes 1.00 and 1.95), so at 2048 tokens it is 48$\times$ slower than standard attention rather than 530$\times$. And the profile points to specific overheads a tuned kernel could remove: work off the tensor cores (masking, ALiBi3D distances, index arithmetic) spread over small tiles of 32 queries, and a backward pass that recomputes the scores in 11 matrix products where 6 would suffice. Our kernels are written in Triton; a handwritten CUDA kernel, as FlashAttention-2 and -3 are, is one route to removing this instruction overhead. We will release the kernels, and Appendix K.4 (new) reports every measurement, including layer-level memory, decode cost and the profile.

Summary

We agree that the floor difference and the crossover were not established, as we now state in the abstract, Section 5 and Appendix B.

What the new runs do establish, without relying on the floor: higher-order attention has lower loss in all 34 matched pairs up to 2.8B parameters, and under either fitted law standard attention needs at least 9% more tokens to match it at every size we train. The per-resource accounting you credited now includes measured training and decoding cost (Appendix K.4), and the held-out validation you asked for in Q1 is in Section G.6.

Thank you again for your insightful comments - your W1, W2 and Q1 led directly to the new experiments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

Thank you all for your careful and constructive reviews. We're happy you found the question important and well-defined (A42m), the motivation and writing clear (ePN5), the per-resource accounting honest (j2Fp, aS2D), and the token-efficiency versus downstream distinction valuable (t6CZ).

What the paper asks. Is spending more computation per token worth it, as high-quality data runs out? We answer per resource for higher-order attention: parameters, tokens, training FLOPs, KV-cache bytes and, now, GPU time. Contributions: kernels and a recipe that train dense higher-order attention in every layer up to 2.8B parameters (to be released), ALiBi3D, and matched scaling laws with a measured cost accounting.

What changed. Every change is marked in the PDF and listed, with the review that prompted it, before the appendices.

  • New runs and a matched refit (j2Fp, aS2D, t6CZ): a 3×10203\times10^{20} IsoFLOP profile (3$\times$ our previous largest, up to 2.8B parameters), a fixed-64B-token profile and two off-profile runs, each twinned; 34 runs per architecture, paired bootstrap, held-out validation (Appendix G).
  • Floor and crossover: our data cannot tell whether the floors differ (ΔE=0.117 [−0.006,0.193]\Delta E = 0.117\,[-0.006, 0.193]; shared floor not rejected, p=0.097p = 0.097). The crossover is now conditional: near 102310^{23} FLOPs under our best-fitting law, beyond 102610^{26} under a shared floor.
  • Measured cost (j2Fp, aS2D, t6CZ, A42m): step time, memory, decoding, kernel profile (Appendix K.4).
  • Retrieval (A42m): needle-in-a-haystack for the sparse variants (Appendix H.1).
  • Smaller: baseline positional encoding and RoPE (aS2D), Figure 3.1(c) setup (t6CZ), related work (A42m), limitations (j2Fp, aS2D, t6CZ).
  • Abstract, introduction and conclusion: the floor attribution is removed, the crossover is conditional, and GPU time is added to the per-resource verdict; the change page sets the submitted and revised abstract sentences side by side.

What we find. Higher-order attention reaches lower loss in all 34 matched pairs, up to 2.8B parameters and at three times our previous largest budget. Standard attention needs at least 9% more tokens to match it under either fitted law, and the advantage transfers downstream. The dilated variant stays ahead of standard attention up to 2048 tokens with per-token cost linear in context, the same order as standard attention, and retrieves a single needle from anywhere in that context (Appendix H.1). Standard attention wins per FLOP, per cached byte and per GPU-hour at our scales. In practice, standard attention is the better choice today. What we show is that extra computation per token buys lower loss at fixed parameters and data. Looping, test-time compute and other ways to spend it are worth comparing on the same per-resource accounting.

Thank you again for your insightful comments - they have significantly improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.