- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper elevates the matrix-form recurrent state found in linear attention and SSMs to an order- tensor, where the state updates via a rank-1 outer product () and readout is performed by contracting with query vectors (e.g., and when ). It retains both recurrent and parallel formulations and reduces to causal linear attention when and (Proposition 1). The authors augment GLA-style gating with a data-dependent forget gate , a query-skip gate that allows for querying two axes, one axis, or neither, and a "ConcOnlyC" mechanism—replacing Gated DeltaNet's fused linear projection with concatenation followed by a 3-parameter width-wise convolution. The full model incorporates a third-order delta rule (with a coverage gate) and a FusedRMSNormGated output gate, all embedded within a Pre-LN Llama-style block. Experiments include a toy example with 8 slots illustrating the rationale for using two queries, MQAR tasks with (sequence lengths of 1024, 2048, and 4096), component ablation studies, and three real-world signal tasks: UEA classification, long-horizon forecasting, and Google Speech Commands.
Critical strengths
Elevating recurrent memory from a matrix to a higher-order tensor and employing multiple queries for contraction represents a natural and intuitive generalization; the fact that the formulation reduces to linear attention when the order provides a clear conceptual anchor.
Methodologically, the implementation of the query-skip gate via a homogeneous feature map, using a shared additive/delta core after padding with dummy slots, is an elegant approach; furthermore, the third-order delta rule and the chunk-parallel UT/WY implementations demonstrate that the authors have successfully realized the method in practice.
The ablation study in Table 2 is also well-designed: by comparing "tensor states" against "matrix states" under identical parameter counts (175k) and model dimensions (), and by isolating input mixing strategies (ConcOnlyC / ConcOnly / OneLinear), the study pinpoints the shift to tensor states as the primary driver of MQAR's performance gains.
A caveat must be noted immediately, however: while the ablation controls for parameter count, it does not match state size (8192 vs. 512 slots); consequently, the effects of "extra modes" versus "extra slots" remain confounded in this comparison—see Weakness 2 for details.
Critical weaknesses
(1) The claim of being "strictly more expressive" requires a clear basis for comparison. Proposition 1 shows that when , RT degenerates into linear attention; thus, if the per-mode width is fixed and is allowed—increasing the state size from to —the function class of RT indeed encompasses linear attention. The issue is not whether this containment holds, but rather that it conflates "larger state size" with "superior structural design" under the single umbrella of "expressivity." By expanding the readout score into , RT can be reformulated at the algebraic level of state/readout operations as a matrix-state attention mechanism (with dimensionality ) utilizing separable key/query parameterization; whether this falls within the baseline function class adopted by the authors depends on the specific feature maps and parameterizations permitted for that baseline. Consequently, the paper has not yet demonstrated that RT surpasses the function class of general matrix-state attention when state sizes are equated. In Figure 2, the authors themselves state, "With , four keys can in principle separate the four pair-queries," indicating that a matrix state can handle this toy example given the same number of slots. Therefore, the claim needs to be refined: RT trades capacity for more slots and separable addressing while keeping the per-mode width constant; demonstrating the inherent advantages of the tensor structure itself requires a comparison where state sizes are matched. I do not deny that tensor decomposition may offer benefits in terms of parameter efficiency or optimization; however, the current evidence fails to disentangle these factors from the effects of simply increasing the state size.
(2) The MQAR baseline figures require further details regarding configuration, and the comparison protocol does not match state sizes. In Table 1, Mamba-2 scores 0.6/0.1/0.0 and GLA scores 0.1/1.9/0.0 for sequence lengths of 1024/2048/4096, whereas other matrix-state baselines (ΔNet / Gated ΔNet) achieve scores in the ranges of 94.5–100 and 48–78, respectively. The results for GLA and Mamba-2 are near zero; the authors should clearly specify the learning rates, step counts, initialization methods, and the number of key-value pairs (relative to MQAR) used for these baselines so that one can determine whether the performance gap stems from architectural differences or configuration choices—though I do not assert here that the baselines were poorly tuned. Another more fundamental issue, independent of hyperparameter tuning, is the mismatch in state budgets: the attention baseline (934k parameters, 100 F1) and RT (176k parameters) differ in both parameter count and state capacity. Specifically, RT utilizes 8,192 slots for its per-head state, whereas the matrix-based baseline uses only 512 slots (a discrepancy acknowledged in Section 3.4). To substantiate the claim that "extra modes yield increased capacity," a comparison under state-matched (and FLOPs-matched) conditions is essential: one should adjust the linear attention dimension to (or increase the number of heads) to bring the slot count closer to 8,192 before evaluating F1 performance; otherwise, it is impossible to disentangle the effects of tensor factorization from those of having more state slots. Additionally, the "ms/step" values in Table 1 reflect validation forward-pass latency rather than training throughput.
(3) Real-world signal tasks are insufficient to establish robust conclusions regarding competitiveness. In Table 4, the performance of RT/RT/4/RT/4-Lin clusters with that of GLA, Mamba-2, and ΔNet in the ~0.43–0.47 MSE range, showing no clear separation. In Table 3 (UEA dataset), the average score for RT/4-Lin (64.4) is only marginally higher than that of ΔNet (63.1), and this result is based on a single random seed (seed 1). In Table 5 (SC10 dataset), ΔNet achieves 82.0, outperforming RT/4 (77.8), while the full RT model failed to complete training within the shared 16-hour budget. All real-world tasks relied on a single seed (seed 1), and the learning rate was determined via the Heartbeat task and then fixed across all datasets. While these settings provide preliminary evidence, they are insufficient to robustly support the broad claim of being "competitive with linear baselines at long sequence lengths."
(4) The empirical performance benefits of the "linear in T" property are not fully demonstrated, and key performance curves are missing. Let us first acknowledge a point: the paper states that training implements this recurrence using chunk-parallel UT/WY (§2.2), so the "linear in T" claim holds algorithmically (O(dv·dk·dr) per token); I do not dispute the validity of the claim itself. The issue lies in the presentation. The parallel formulation in §2.2—O = ((QK^T) ⊙ (UR^T) ⊙ M) V—is expressed in T×T terms; although the main text notes that actual training employs chunk-parallel UT/WY, it fails to report the chunk size, whether large T×T intermediate tensors are still generated, or the actual training complexity and VRAM usage figures. In Table 1, at T=4096, the latency for RT is 311.9 ms versus 333.5 ms for attention—only about 6% faster—without clearly demonstrating practical gains for long sequences. The reason may lie in the "width" dimension: RT’s per-token tensor state involves dv·dk·dr = 16·16·32 = 8192 operations, whereas the baseline matrix state is 16×32 = 512, making the per-token overhead 16 times higher; thus, "linear in T" does not equate to "faster," as the cubic cost associated with width at moderate T values can outweigh the T² cost of attention. The paper should provide three curves—latency vs. T, latency vs. width, and peak VRAM usage—marking the crossover point where RT outperforms attention. A similar ambiguity exists with ConcOnlyC: it compresses the dense projection of dmodel·dfused into three parameters, conflating the effects of parameter savings with those of architectural superiority; OneLinear performs strongly in short-context scenarios but collapses in long-context ones (97.2/98.2/54.2), yet lacks a matched-parameter dense projection control (such as a dense layer plus an MLP with equivalent parameter count) to isolate this variable.
(5) The positioning regarding higher-order/bilinear associative memory is not addressed. The paper compares RT only with linear attention, SSMs, and the delta rule, without discussing how it differs from the line of research involving higher-order (bilinear) attention, tensorized fast-weights, or two-query associative memory. While I cannot determine from the main text alone whether there is existing work directly corresponding to these methods, a paper that markets itself on "elevating the state to a higher-order tensor" should clearly articulate the distinctions between RT and such approaches; otherwise, the boundaries of its novelty remain ambiguous. This point is intended to prompt the authors to clarify their positioning rather than to assert that they have overlooked specific citations.
Questions for authors
- Can you provide a formal argument demonstrating that RunningTensor is strictly more expressive than matrix-state linear attention when the state size is the same? If not, please revise "strictly more expressive" to "more state slots at a fixed per-mode width."
- Please include state-matched (and FLOPs-matched) MQAR comparisons: scale the , heads, or state dimensions of linear attention/Gated Net to match the number of slots in RT, then compare F1 scores and training times. Do the currently low scores for Mamba-2/GLA stem from training configuration issues? What learning rates, step counts, and initialization methods were used?
- What are the chunk size, actual computational complexity, and peak VRAM usage during training? Do chunk-parallel UT/WY approaches generate large intermediate tensors? Could you plot training time and peak VRAM curves for RT versus attention across different values of and width, marking the crossover points?
- For real-world tasks, could you use at least three random seeds and tune the learning rate individually for each dataset? Also, please report results for the full RT model using higher VRAM limits rather than simply omitting them.
- How does ConcOnlyC compare to a dense projection with matching parameter counts in long-context scenarios? Is the failure of OneLinear at a sequence length of 4096 due to parameter count, optimization issues, or expressivity limitations?
- What are the specific novel contributions of this paper compared to existing work on high-order/bilinear attention and tensorized fast-weight mechanisms?
5 comments
Questions.
We no longer say "strictly more expressive." nor claim a larger function class at equal state size.
Slots are matched in that RunningTensor matrix ablation, and the equal-slot matrix is slower, so the comparison is not FLOPs-matched. We are rerunning Attention and GLA for table 1, with a better head-layout match. All models used the same learning rates, 100 epochs, seeds, and default initialization. Those choices are in the appendix.
Chunk size is now reported in the Appendix. Peak memory and training-step time against length, at fixed width 128, are in the new scaling figure. Full RunningTensor stays above SDPA attention through length 8192. The equal-slot matrix lags behind the factorized state on both time and memory. The ms column in Table 1 remains validation forward time.
The time-series tables use three seeds and one learning rate chosen on Heartbeat. A per-dataset search was beyond this sweep. Full RunningTensor is reported on Speech Commands, at 71.9, with RT/4-Lin second at 77.5 behind ΔNet at 81.1. LIBERO is one seed.
OneLinear is the dense fused map, at 275K parameters against 175K for ConcOnlyC, and its average is 40.8 against 95.0. We do not have a separate dense layer plus an MLP locked to the same parameter count, and the 4096 drop is not isolated into parameter count, optimization, or expressivity.
The novel-contribution boundary is the related-work paragraph above.
Dear Reviewer,
Given that the author-reviewer discussion period will end tomorrow (November 18), we wanted to follow up on our detailed response to your extensive and helpful feedback.
We have made substantial revisions addressing your concerns, including refining our expressivity claims, adding state-matched comparisons, expanding the experimental evaluation with additional seeds, reporting training-time and memory measurements, and strengthening the discussion of related work. We have also clarified the remaining limitations of our experimental evaluation.
We would greatly appreciate it if you could revisit our response and the revised manuscript before the discussion period concludes, and let us know whether the revisions address your main concerns or whether there are any remaining points you would like us to clarify.
Thank you again for your thorough review and constructive suggestions.
Thanks a lot for the positive statements about the strengths of the article.
We have removed the "strictly more expressive" claim. The paragraph after Proposition 1 now states the Kronecker form you pointed out whose keys factor across the two modes. An unrestricted matrix with free keys of that width contains this update. The appendix records the same fact for the delta rule.
The slots-matched comparison is the ConcOnlyC row of the ablation table. The tensor stores 16×16×32=8192 slots per head, uses 175K parameters, and scores 96.5 F1 at length 4096 (314 ms). The matrix with the same 8192 slots, dk x dv =256×32, uses 186K parameters and scores 24.9 F1 at that length (431 ms). Parameter count stays in the same band. The gain at equal slot count is the factorization. We do not claim a larger function class at equal state size.
The appendix now records the shared protocol for every method: learning rates {10e-2, 10e-3, 10e-4}, seeds {0,1,2}, 100 epochs, batch size 16, early stopping on validation loss with patience 20, and the configuration's default initialization. The MQAR generator uses its default of 64 key-value pairs, vocabulary 8192, and power-law exponent 0.1. The cell is the mean validation F1 over finished seeds at the learning rate with the highest mean. The ms column is validation forward time, not training throughput.
The slots-matched rows are an ablation of our own state, a matrix of width 256×32 against the 16×16×32 tensor. They do not rescale Gated ΔNet or Attention up to 8192 slots, only the RunningTensor, and they are not FLOPs-matched: the equal-slot matrix is the slower run. We are rerunning Attention and GLA so the head layout matches the other rows (6 heads, per-head value width 32 for GLA; 6 heads of dimension 16 for attention). Those runs are not yet in the current table.
Longhorn (ICLR 2025, Figure 4) sweeps MQAR at widths {64,128,256,512} and lengths up to 512, and reports that Mamba loses recall in that range. Our lengths are longer, to further stress the models, and that figure is about Mamba, so we treat it as context for the Mamba-2 row.
The time-series tables are now means over finished seeds at 10e-3. That rate was selected on Heartbeat, where 10e-3 beat 10e-4 for attention, Gated ΔNet, and RunningTensor, and it is then used for every architecture and dataset. A separate search on every dataset would have multiplied the sweep by the number of datasets, which we did not have the budget for. We also added Tiny-scale ImageNet and open-loop LIBERO distillation. The LIBERO table is one seed.
On Speech Commands, ΔNet leads at 81.1. RT/4-Lin is second at 77.5. Full RunningTensor is reported, at 71.9, rather than omitted. It is true that on the Speech Commands dataset, our approach only gets the second position, but we thought it is still a fair result compared to remaining linear and non linear attention, and it was also fair to show a non perfect result if that was the output of our experiments.
The new scaling figure times one layer, depth 1, width 128, no feed-forward, no residual, batch 1, 8 heads of dimension 16, for lengths 256 to 8192. One panel is a training step and one is a forward with no gradient. At length 8192, SDPA attention takes 5.8 ms and 76 MB per training step. Full RunningTensor takes 186 ms and 844 MB. The matrix with the same 8×8192 slots takes 204 ms and 1.7 GB, and it is slower and larger at every length in this range. Full RunningTensor does not cross under attention by length 8192 at this width. RT/4 does stay close to the matrix-state models. Chunk size is now reported in the appendix, but the new scaling figure does not sweep width.
The related work now has this paragraph. Higher-order attention grows with length because its score runs over tokens (Alman and Song, ICLR 2024); bilinear attention is still one pairwise score (Kim, Jun, and Zhang, NeurIPS 2018). RunningTensor raises the state order and stays linear: one rank-1 write, scored by a product of two inner products. Tensorized fast weights compress a matrix (Ma et al., NeurIPS 2019); the additive core is linear attention on a two-axis key, already in a free matrix of that width, with each axis queryable. Two-query memory reads each value with two queries (Schlag, Munkhdalai, and Schmidhuber, arXiv 2020; Schlag and Schmidhuber, NeurIPS 2018): one key leaves a matrix, and one query per axis of a third-order state is that outer product, plus a forget gate, a tensor delta rule, and query-skip gates. The matched-slot ablation is the comparison with the free matrix.