acceptodds
Under review as a conference paper at ICLR 2027

Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling

19 views

Abstract

Recurrent neural networks (RNNs) compress the historical context into a memory state of fixed size, thus allowing for constant-time inference. The memory state size is a crucial factor in their performance, as exemplified by the strong performance and resurgence of linear attention, which extends the vector-valued hidden states of ordinary RNNs to matrix-valued hidden states. Crucially, linear attention does so in a parameter-efficient way, in particular by using an outer product of the key and value vectors to write to the matrix-valued hidden state. We generalize this construction and propose triadic linear attention, which writes the triadic outer product of a key, a second key, and a value, into a third-order tensor state, and reads from it by contracting both key axes with two queries. An -dimensional second key thus yields an -fold increase in state size while adding only two projections. Triadic linear attention is compatible with data-dependent forgetting, the delta rule, and chunkwise-parallel training. Applied to Gated DeltaNet and scalar-gated linear attention, triadic linear attention substantially improves long-context language modeling and recall, outperforming alternatives that enlarge the state.

Primary area:
foundation or frontier models, including LLMs
Keywords:
linear attention, fast-weight programming, pretraining, deltanet

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Xh3GOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to improve recall and long-context modeling in linear attention with little additional parameter cost. It proposes triadic linear attention, which expands the recurrent state into a third-order tensor using two keys and a value. The method supports forgetting, the delta rule, and chunkwise-parallel training.

Critical strengths
  1. The paper proposes a way to enlarge recurrent memory with little parameter overhead, and it substantially improves recall. The method also works with gated delta updates and efficient chunkwise training, which makes it useful for modern linear-attention models.
  2. The experiment setup is well controlled. At matched state size and parameter count, the method outperforms the alternatives for enlarging the state, and the gains hold for both GDN and sGLA. The upcycling and hybrid-model results further show that the method also benefits pretrained models and hybrid architectures.
  3. The paper also contributes efficient GPU kernels that make the larger state practical, keeping the training overhead moderate relative to the base model.
Critical weaknesses

The comparisons with alternative state-expansion strategies are limited to the 400M scale. The 1.3B results are reported for a single seed and do not include the competing expansion strategies. Although the results show improvements over the base models, the robustness of some smaller perplexity differences and the advantage over alternative expansion methods at larger scales remain less clear.

Questions for authors

Could the authors report the variance across seeds for the comparison with other ways of enlarging the state? It would also help to include at least the strongest alternative at the larger scale, to show that the advantage still holds as the model scales.

2 comments

We thank the reviewer for their review. We have addressed both points below:

Seed robustness

We report the standard deviations of our 400M GDN runs across three training seeds below. Generally, perplexity evaluations are robust to training seed, while performance on recall-intensive tasks varies more. Even the worst seed for Triadic GDN achieves lower perplexities than the best seed of every state-matched baseline.

Model (400M)State0-shot ↑Wiki. ↓PG19 16k–64k ↓Recall ↑
GDN (base)1×53.1 ± 0.511.25 ± 0.0614.15 ± 0.0126.2 ± 0.2
Larger heads (d=256)2×52.6 ± 0.411.22 ± 0.0414.17 ± 0.0226.6 ± 0.6
Grouped values (2 per key)2×52.9 ± 0.411.18 ± 0.0214.12 ± 0.0427.1 ± 0.5
Wider values (d_v=256)2×53.2 ± 0.211.20 ± 0.0614.14 ± 0.0327.4 ± 0.8
More heads (16)2×53.3 ± 0.211.32 ± 0.0214.28 ± 0.0427.5 ± 0.5
Triadic (E=2)2×53.4 ± 0.511.08 ± 0.0113.95 ± 0.0428.4 ± 0.1
Larger heads (d=512)4×53.0 ± 0.511.25 ± 0.1514.24 ± 0.1228.5 ± 1.8
Grouped values (4 per key)4×52.7 ± 0.311.35 ± 0.0214.36 ± 0.0327.3 ± 0.6
Wider values (d_v=512)4×52.5 ± 0.611.41 ± 0.0314.44 ± 0.0327.5 ± 0.5
Triadic (E=4)4×53.6 ± 0.310.93 ± 0.0513.79 ± 0.0231.1 ± 1.0

We additionally ran GDN and Triadic GDN at the 1.3B scale for a second training seed and found the results to closely match the first seed in perplexity evaluations:

Model (1.3B)Seed0-shot ↑Wiki. ↓PG19 ≤4k ↓PG19 4k–16k ↓PG19 16k–64k ↓Recall ↑
GDN159.68.0810.9110.3810.2036.1
GDN260.98.0910.9110.3810.2037.9
Triadic GDN (E=8)160.97.8710.7410.169.9444.4
Triadic GDN (E=8)261.17.9010.7410.169.9446.3

Additional baseline comparisons at 1.3B scale

We had only included comparisons to state-size matched baselines at the 400M scale in the paper. We have produced additional results for grouped values (2x) and larger heads (4x) (the strongest baselines at the 400M scale) at the 1.3B scale. At this larger scale, Triadic Gated DeltaNet again outperforms the corresponding state-matched baselines across all benchmarks, for both 2x and 4x state-size increase:

Model (1.3B)StateParams0-shot ↑Wiki. ↓PG19 ≤4k ↓PG19 4k–16k ↓PG19 16k–64k ↓Recall ↑
GDN (base)12.61360.259.68.0810.9110.3810.2036.1
Grouped values (2 per key)25.21362.060.78.0610.9210.3910.2039.6
Triadic (E=2)25.21364.261.27.9710.8210.2810.0840.8
Larger heads (d=512)50.31359.160.58.0310.9810.4110.2040.9
Triadic (E=4)50.31368.960.87.9310.7810.219.9943.2

Thanks for the additional experiments. The multi-seed results and the state-matched comparisons at 1.3B help address my main concerns. I appreciate the authors' efforts and will keep my rating at 3.

Reviewer_ryfROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Triadic Linear Attention, which expands a matrix-valued recurrent state into a third-order tensor using an additional key–query pair. The construction increases memory capacity with limited parameter overhead. The authors incorporate forgetting and delta-rule updates, derive chunkwise computation forms, and develop GPU kernels. Experiments at 400M and 1.3B parameters evaluate language modeling and recall, with additional studies on converting pretrained models and integrating triadic states into hybrid architectures.

Critical strengths
  1. The authors compare state-expansion methods at matched state sizes and approximately matched parameter counts. Triadic variants achieve lower WikiText and PG19 perplexity than the tested alternatives, supporting their usefulness under a fixed parameter budget.
  2. The authors exploit the joint-key factorization to reduce intra-chunk inner-product computation. They adapt the existing chunkwise delta-rule formulation to per-slice forgetting and describes value-axis tiling. They provide timing evidence for this implementation under the tested settings.
  3. The authors report improvements when pretrained models are converted to triadic variants during long-context extension. This provides additional evidence that the construction can benefit existing models.
Critical weaknesses

The scope of the efficiency evaluation is limited. Figure 4 reports single-block forward-and-backward timings on H100 GPUs at batch size 4, leaving full-model training throughput, peak training memory, and autoregressive decoding performance unclear.

Questions for authors

Could you provide end-to-end training throughput and peak GPU memory measurements for representative existing configurations, and clarify how the enlarged state affects autoregressive decoding latency?

1 comment

We thank the reviewer for their feedback. We address the reviewer’s concerns regarding efficiency evaluations below.

Further efficiency evaluations

We reported forward and backward timings of a single block in the paper, as these are independent of the particular training parallelization strategy (e.g., data/pipeline/tensor parallelism). With appropriate parallelization, these timings should be indicative of end-to-end training overhead, modulo communication, optimizer overhead, and other model components (like LM head). To complement these results, we measured end-to-end training throughput and peak GPU memory in our purely data-parallel setting at the 1.3B scale.

Importantly, for our single block timings (Figure 4) we use the highly optimized FlashQLA kernels from Qwen for the GDN baseline, which is a stronger baseline than the Triton flash-linear-attention (FLA) kernels benchmarked against in most academic works. For completeness, we additionally report results for the FLA Triton kernels below.

ModelKernelStep timeThroughputPeak GPU memory
GDNFlashQLA726.5 ms1.443M tokens/s48.5 GiB
GDNFLA Triton795.4 ms1.318M tokens/s52.1 GiB
Triadic GDN (E=4)Ours806.6 ms1.300M tokens/s58.6 GiB
Triadic GDN (E=8)Ours874.0 ms1.200M tokens/s66.6 GiB

The end-to-end training overhead compared to GDN with the FlashQLA kernels is 11.0% for Triadic GDN E=4 and 20.3% for Triadic GDN E=8, which is lower than the single block overheads reported in the paper (14-15% for E=4, 28-30% for E=8), since the cost outside of the block is constant for all variants. Compared to GDN with the FLA Triton kernels the overhead of Triadic GDN is 1.4% for E=4 and 9.9% for E=8.

Regarding inference, we expect the increased state size to slow down decoding compared to smaller states, particularly at high batch sizes. This is analogous to transformers, where a larger kv-cache increases inference cost, although our recurrent state remains constant size across sequence length. For triadic linear attention specifically, we have preliminary experiments showing that applying top-k sparsity to the second query incurs little degradation. This allows for only having to read the corresponding slices of the state, which reduces memory movement. Writes could be sparsified similarly, or alternatively, dense writes could be performed lazily (meaning writes to a slice of the state are delayed until the slice is read).

Reviewer_UnrQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses the limited recurrent-state capacity of linear attention models in long-context sequence modeling. The authors propose Triadic Linear Attention, which introduces an additional key and query to extend the conventional matrix-valued state into a third-order tensor state. Specifically, the state is updated using the outer product of two keys and a value, yielding a state of size (d \times E \times d) and increasing its size by a factor of (E) with only two additional projection layers. The authors evaluate the proposed method by applying it to Gated DeltaNet (GDN) and scalar-gated linear attention (sGLA), comparing the resulting models with their baseline counterparts and alternative state-expansion methods. Experiments on associative recall, language modeling, and long-context benchmarks show improvements in recall and perplexity at both 400M and 1.3B parameter scales. The paper also investigates pretrained-model upcycling, hybrid architectures, and computational efficiency to assess the practical benefits and costs of the proposed approach.

Critical strengths
  1. The paper introduces a simple and principled way to expand the recurrent state of linear attention by adding a second key-query axis. This increases the state size by a factor of E while requiring only two additional projections. The MQAR experiments demonstrate that this design can substantially increase associative recall capacity with minimal parameter overhead, addressing an important limitation of existing linear attention architectures.
  2. The paper evaluates the proposed method across two sequence-mixing architectures, multiple model scales, long-context language modeling benchmarks, and recall-intensive tasks. The state-matched comparisons against several alternative expansion strategies provide useful evidence that the gains are not merely due to increasing the state size, but also depend on how the additional capacity is parameterized. The upcycling and hybrid-model experiments further demonstrate the practical potential of the proposed approach.
Critical weaknesses

The central idea of this paper is to extend the matrix-valued recurrent state of linear attention to a third-order tensor by introducing a second key-query axis. While this construction is technically appealing, its conceptual novelty is unclear. HGRN2 has already established parameter-efficient state expansion through outer-product-based representations, and the contemporaneous work RunningTensor appears to formulate a closely related generalization of linear attention to higher-order recurrent tensor states. The authors should more clearly distinguish the proposed formulation from these approaches and identify which aspects represent genuinely new methodological contributions, rather than an extension of the existing state-expansion principle. Although the implementation, integration with modern linear-attention variants, and empirical evaluation provide practical value, it remains unclear whether these contributions are sufficient to establish a substantially distinct methodological advance.

Questions for authors
  1. First, how does the proposed Triadic Linear Attention fundamentally differ from HGRN2 and RunningTensor, particularly in terms of its core state-expansion mechanism? Beyond architectural adaptation and efficient implementation, what constitutes its distinct methodological contribution?
  2. Second, could the authors provide additional controlled experiments to isolate the benefits of the triadic state representation from differences in MLP capacity and parameter allocation? In particular, how much of the observed performance improvement can be attributed specifically to the proposed state parameterization?
  3. Third, the efficiency evaluation in Section 3.5 focuses on block-level forward and backward computation time. Could the authors report end-to-end training throughput and peak GPU memory under comparable settings to better establish the practical efficiency of the proposed method? These clarifications would help assess the method's novelty, the reliability of its empirical claims, and its practical advantages over existing approaches.
1 comment

We thank the reviewer for their feedback and suggestions. We address the raised concerns below:

Difference to HGRN2 and RunningTensor

HGRN2 is a linear attention variant with a two dimensional state and single key/query pair. HGRN2 expands the vector valued state used in HGRN to a matrix state, and thus is a linear attention variant similar to the Gated DeltaNet/Gated Linear Attention/Mamba-2 family. Triadic Linear Attention could in principle be applied to HGRN2 as well.

It should first be noted that RunningTensor is concurrent work that was also submitted to ICLR 2027. Therefore our work should be evaluated independent of the RunningTensor work. Having said that, we provide some similarities and differences. The core idea is similar, and both RunningTensor and Triadic Linear Attention use third-order tensor state construction through the use of two keys/queries. The main distinction is that our work focuses on making this construction practical for large-scale language modeling. This is enabled by efficient GPU kernels that exploit the factorized structure of the keys and queries and tile the state to fit on-chip, as well as architecture-hardware co-design (for example, we deliberately keep the second-key dimension small). Our work focuses on language modeling at scale (1.3B parameter, 65B tokens), and our main findings regarding improvements on long-context and recall-intensive tasks are a consequence of this. Meanwhile, RunningTensor only performs synthetic recall, time-series and speech experiments at a very small scale (2 layers, d_model=128, <10M parameters) and does not provide efficient implementations.

State parametrization vs. MLP capacity

We match parameters in Table 1 because parameter efficiency is the main point of this work, meaning triadic linear attention enables a much larger state without inflating the parameter count. For example, increasing the state size by 8x through 8x more heads would require 8x the parameters in the query, key, value and output projections. Table 1 shows that, at a fixed parameter budget, it is better to use a parameter-efficient parametrization for linear attention and keep the parameters in the MLP. That said, Table 1 already contains this controlled comparison, since the larger-heads baseline adds no parameters and therefore uses the same MLP width as triadic linear attention, and it still clearly underperforms it at both 2x and 4x the state size.

Further efficiency evaluations

We reported forward and backward timings of a single block in the paper, as these are independent of the particular training parallelization strategy (e.g., data/pipeline/tensor parallelism). With appropriate parallelization, these timings should be indicative of end-to-end training overhead, modulo communication, optimizer overhead, and other model components (like LM head). To complement these results, we measured end-to-end training throughput and peak GPU memory in our purely data-parallel setting at the 1.3B scale.

Importantly, for our single block timings (Figure 4) we use the highly optimized FlashQLA kernels from Qwen for the GDN baseline, which is a stronger baseline than the Triton flash-linear-attention (FLA) kernels benchmarked against in most academic works. For completeness, we additionally report results for the FLA Triton kernels below.

ModelKernelStep timeThroughputPeak GPU memory
GDNFlashQLA726.5 ms1.443M tokens/s48.5 GiB
GDNFLA Triton795.4 ms1.318M tokens/s52.1 GiB
Triadic GDN (E=4)Ours806.6 ms1.300M tokens/s58.6 GiB
Triadic GDN (E=8)Ours874.0 ms1.200M tokens/s66.6 GiB

The end-to-end training overhead compared to GDN with the FlashQLA kernels is 11.0% for Triadic GDN E=4 and 20.3% for Triadic GDN E=8, which is lower than the single block overheads reported in the paper (14-15% for E=4, 28-30% for E=8), since the cost outside of the block is constant for all variants. Compared to GDN with the FLA Triton kernels the overhead of Triadic GDN is 1.4% for E=4 and 9.9% for E=8.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.