acceptodds
Under review as a conference paper at ICLR 2027

RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States

Abstract

Linear attention and state-space models provide linear-time sequence modeling, but their recurrent memory remains a second-order tensor, a matrix, limiting the order of interactions that can be represented in the state. We introduce the RunningTensor, which generalizes this memory to an order- tensor, updated by a rank-1 outer product and read by contracting against vector queries. Order recovers linear attention; we study order as a proof of concept, retaining both recurrent and parallel forms while remaining linear in sequence length and improving working memory capacity from to . On synthetic multi-query associative recall, RunningTensor outperforms linear attention and SSM baselines. It also improves performance on time-series classification and forecasting, suggesting that higher-order recurrent state can provide useful additional memory capacity beyond matrix-valued state.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
linear attention, state space models, time series

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9BRmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper elevates the matrix-form recurrent state found in linear attention and SSMs to an order-oo tensor, where the state updates via a rank-1 outer product (St=St−1+vt⊗kt⊗rtS_t = S_{t-1} + v_t \otimes k_t \otimes r_t) and readout is performed by contracting with o−1o-1 query vectors (e.g., qtq_t and utu_t when o=3o=3). It retains both recurrent and parallel formulations and reduces to causal linear attention when o=2o=2 and dr=1dr=1 (Proposition 1). The authors augment GLA-style gating with a data-dependent forget gate αt\alpha_t, a query-skip gate ξ\xi that allows for querying two axes, one axis, or neither, and a "ConcOnlyC" mechanism—replacing Gated DeltaNet's fused linear projection with concatenation followed by a 3-parameter width-wise convolution. The full model incorporates a third-order delta rule (with a β\beta coverage gate) and a FusedRMSNormGated output gate, all embedded within a Pre-LN Llama-style block. Experiments include a toy example with 8 slots illustrating the rationale for using two queries, MQAR tasks with dmodel=128d_{model}=128 (sequence lengths of 1024, 2048, and 4096), component ablation studies, and three real-world signal tasks: UEA classification, long-horizon forecasting, and Google Speech Commands.

Critical strengths

Elevating recurrent memory from a matrix to a higher-order tensor and employing multiple queries for contraction represents a natural and intuitive generalization; the fact that the formulation reduces to linear attention when the order o=2o=2 provides a clear conceptual anchor.

Methodologically, the implementation of the query-skip gate via a homogeneous feature map, using a shared additive/delta core after padding with dummy slots, is an elegant approach; furthermore, the third-order delta rule and the chunk-parallel UT/WY implementations demonstrate that the authors have successfully realized the method in practice.

The ablation study in Table 2 is also well-designed: by comparing "tensor states" against "matrix states" under identical parameter counts (175k) and model dimensions (dmodeld_{model}), and by isolating input mixing strategies (ConcOnlyC / ConcOnly / OneLinear), the study pinpoints the shift to tensor states as the primary driver of MQAR's performance gains.

A caveat must be noted immediately, however: while the ablation controls for parameter count, it does not match state size (8192 vs. 512 slots); consequently, the effects of "extra modes" versus "extra slots" remain confounded in this comparison—see Weakness 2 for details.

Critical weaknesses

(1) The claim of being "strictly more expressive" requires a clear basis for comparison. Proposition 1 shows that when dr=1dr=1, RT degenerates into linear attention; thus, if the per-mode width is fixed and dr>1dr>1 is allowed—increasing the state size from O(W2)O(W^2) to O(Wo)O(W^o)—the function class of RT indeed encompasses linear attention. The issue is not whether this containment holds, but rather that it conflates "larger state size" with "superior structural design" under the single umbrella of "expressivity." By expanding the readout score into (kiTq)(riTu)=(ki⊗ri)T(q⊗u)(k_i^T q)(r_i^T u) = (k_i \otimes r_i)^T (q \otimes u), RT can be reformulated at the algebraic level of state/readout operations as a matrix-state attention mechanism (with dimensionality dk⋅drdk \cdot dr) utilizing separable key/query parameterization; whether this falls within the baseline function class adopted by the authors depends on the specific feature maps and parameterizations permitted for that baseline. Consequently, the paper has not yet demonstrated that RT surpasses the function class of general matrix-state attention when state sizes are equated. In Figure 2, the authors themselves state, "With dk=4dk=4, four keys can in principle separate the four pair-queries," indicating that a matrix state can handle this toy example given the same number of slots. Therefore, the claim needs to be refined: RT trades capacity for more slots and separable addressing while keeping the per-mode width constant; demonstrating the inherent advantages of the tensor structure itself requires a comparison where state sizes are matched. I do not deny that tensor decomposition may offer benefits in terms of parameter efficiency or optimization; however, the current evidence fails to disentangle these factors from the effects of simply increasing the state size.

(2) The MQAR baseline figures require further details regarding configuration, and the comparison protocol does not match state sizes. In Table 1, Mamba-2 scores 0.6/0.1/0.0 and GLA scores 0.1/1.9/0.0 for sequence lengths of 1024/2048/4096, whereas other matrix-state baselines (ΔNet / Gated ΔNet) achieve scores in the ranges of 94.5–100 and 48–78, respectively. The results for GLA and Mamba-2 are near zero; the authors should clearly specify the learning rates, step counts, initialization methods, and the number of key-value pairs (relative to MQAR) used for these baselines so that one can determine whether the performance gap stems from architectural differences or configuration choices—though I do not assert here that the baselines were poorly tuned. Another more fundamental issue, independent of hyperparameter tuning, is the mismatch in state budgets: the attention baseline (934k parameters, 100 F1) and RT (176k parameters) differ in both parameter count and state capacity. Specifically, RT utilizes 8,192 slots for its per-head state, whereas the matrix-based baseline uses only 512 slots (a discrepancy acknowledged in Section 3.4). To substantiate the claim that "extra modes yield increased capacity," a comparison under state-matched (and FLOPs-matched) conditions is essential: one should adjust the linear attention dimension dkd_k to 16×16=25616 \times 16 = 256 (or increase the number of heads) to bring the slot count closer to 8,192 before evaluating F1 performance; otherwise, it is impossible to disentangle the effects of tensor factorization from those of having more state slots. Additionally, the "ms/step" values ​​in Table 1 reflect validation forward-pass latency rather than training throughput.

(3) Real-world signal tasks are insufficient to establish robust conclusions regarding competitiveness. In Table 4, the performance of RT/RT/4/RT/4-Lin clusters with that of GLA, Mamba-2, and ΔNet in the ~0.43–0.47 MSE range, showing no clear separation. In Table 3 (UEA dataset), the average score for RT/4-Lin (64.4) is only marginally higher than that of ΔNet (63.1), and this result is based on a single random seed (seed 1). In Table 5 (SC10 dataset), ΔNet achieves 82.0, outperforming RT/4 (77.8), while the full RT model failed to complete training within the shared 16-hour budget. All real-world tasks relied on a single seed (seed 1), and the learning rate was determined via the Heartbeat task and then fixed across all datasets. While these settings provide preliminary evidence, they are insufficient to robustly support the broad claim of being "competitive with linear baselines at long sequence lengths."

(4) The empirical performance benefits of the "linear in T" property are not fully demonstrated, and key performance curves are missing. Let us first acknowledge a point: the paper states that training implements this recurrence using chunk-parallel UT/WY (§2.2), so the "linear in T" claim holds algorithmically (O(dv·dk·dr) per token); I do not dispute the validity of the claim itself. The issue lies in the presentation. The parallel formulation in §2.2—O = ((QK^T) ⊙ (UR^T) ⊙ M) V—is expressed in T×T terms; although the main text notes that actual training employs chunk-parallel UT/WY, it fails to report the chunk size, whether large T×T intermediate tensors are still generated, or the actual training complexity and VRAM usage figures. In Table 1, at T=4096, the latency for RT is 311.9 ms versus 333.5 ms for attention—only about 6% faster—without clearly demonstrating practical gains for long sequences. The reason may lie in the "width" dimension: RT’s per-token tensor state involves dv·dk·dr = 16·16·32 = 8192 operations, whereas the baseline matrix state is 16×32 = 512, making the per-token overhead 16 times higher; thus, "linear in T" does not equate to "faster," as the cubic cost associated with width at moderate T values ​​can outweigh the T² cost of attention. The paper should provide three curves—latency vs. T, latency vs. width, and peak VRAM usage—marking the crossover point where RT outperforms attention. A similar ambiguity exists with ConcOnlyC: it compresses the dense projection of dmodel·dfused into three parameters, conflating the effects of parameter savings with those of architectural superiority; OneLinear performs strongly in short-context scenarios but collapses in long-context ones (97.2/98.2/54.2), yet lacks a matched-parameter dense projection control (such as a dense layer plus an MLP with equivalent parameter count) to isolate this variable.

(5) The positioning regarding higher-order/bilinear associative memory is not addressed. The paper compares RT only with linear attention, SSMs, and the delta rule, without discussing how it differs from the line of research involving higher-order (bilinear) attention, tensorized fast-weights, or two-query associative memory. While I cannot determine from the main text alone whether there is existing work directly corresponding to these methods, a paper that markets itself on "elevating the state to a higher-order tensor" should clearly articulate the distinctions between RT and such approaches; otherwise, the boundaries of its novelty remain ambiguous. This point is intended to prompt the authors to clarify their positioning rather than to assert that they have overlooked specific citations.

Questions for authors
  • Can you provide a formal argument demonstrating that RunningTensor is strictly more expressive than matrix-state linear attention when the state size is the same? If not, please revise "strictly more expressive" to "more state slots at a fixed per-mode width."
  • Please include state-matched (and FLOPs-matched) MQAR comparisons: scale the dkd_k, heads, or state dimensions of linear attention/Gated Δ\DeltaNet to match the number of slots in RT, then compare F1 scores and training times. Do the currently low scores for Mamba-2/GLA stem from training configuration issues? What learning rates, step counts, and initialization methods were used?
  • What are the chunk size, actual computational complexity, and peak VRAM usage during training? Do chunk-parallel UT/WY approaches generate large T×TT \times T intermediate tensors? Could you plot training time and peak VRAM curves for RT versus attention across different values ​​of TT and width, marking the crossover points?
  • For real-world tasks, could you use at least three random seeds and tune the learning rate individually for each dataset? Also, please report results for the full RT model using higher VRAM limits rather than simply omitting them.
  • How does ConcOnlyC compare to a dense projection with matching parameter counts in long-context scenarios? Is the failure of OneLinear at a sequence length of 4096 due to parameter count, optimization issues, or expressivity limitations?
  • What are the specific novel contributions of this paper compared to existing work on high-order/bilinear attention and tensorized fast-weight mechanisms?
5 comments

Thanks a lot for the positive statements about the strengths of the article.

Critical weakness (1).

We have removed the "strictly more expressive" claim. The paragraph after Proposition 1 now states the Kronecker form you pointed out whose keys factor across the two modes. An unrestricted matrix with free keys of that width contains this update. The appendix records the same fact for the delta rule.

The slots-matched comparison is the ConcOnlyC row of the ablation table. The tensor stores 16×16×32=8192 slots per head, uses 175K parameters, and scores 96.5 F1 at length 4096 (314 ms). The matrix with the same 8192 slots, dk x dv =256×32, uses 186K parameters and scores 24.9 F1 at that length (431 ms). Parameter count stays in the same band. The gain at equal slot count is the factorization. We do not claim a larger function class at equal state size.

Critical weakness (2).

The appendix now records the shared protocol for every method: learning rates {10e-2, 10e-3, 10e-4}, seeds {0,1,2}, 100 epochs, batch size 16, early stopping on validation loss with patience 20, and the configuration's default initialization. The MQAR generator uses its default of 64 key-value pairs, vocabulary 8192, and power-law exponent 0.1. The cell is the mean validation F1 over finished seeds at the learning rate with the highest mean. The ms column is validation forward time, not training throughput.

The slots-matched rows are an ablation of our own state, a matrix of width 256×32 against the 16×16×32 tensor. They do not rescale Gated ΔNet or Attention up to 8192 slots, only the RunningTensor, and they are not FLOPs-matched: the equal-slot matrix is the slower run. We are rerunning Attention and GLA so the head layout matches the other rows (6 heads, per-head value width 32 for GLA; 6 heads of dimension 16 for attention). Those runs are not yet in the current table.

Longhorn (ICLR 2025, Figure 4) sweeps MQAR at widths {64,128,256,512} and lengths up to 512, and reports that Mamba loses recall in that range. Our lengths are longer, to further stress the models, and that figure is about Mamba, so we treat it as context for the Mamba-2 row.

Critical weakness (3).

The time-series tables are now means over finished seeds at 10e-3. That rate was selected on Heartbeat, where 10e-3 beat 10e-4 for attention, Gated ΔNet, and RunningTensor, and it is then used for every architecture and dataset. A separate search on every dataset would have multiplied the sweep by the number of datasets, which we did not have the budget for. We also added Tiny-scale ImageNet and open-loop LIBERO distillation. The LIBERO table is one seed.

On Speech Commands, ΔNet leads at 81.1. RT/4-Lin is second at 77.5. Full RunningTensor is reported, at 71.9, rather than omitted. It is true that on the Speech Commands dataset, our approach only gets the second position, but we thought it is still a fair result compared to remaining linear and non linear attention, and it was also fair to show a non perfect result if that was the output of our experiments.

Critical weakness (4).

The new scaling figure times one layer, depth 1, width 128, no feed-forward, no residual, batch 1, 8 heads of dimension 16, for lengths 256 to 8192. One panel is a training step and one is a forward with no gradient. At length 8192, SDPA attention takes 5.8 ms and 76 MB per training step. Full RunningTensor takes 186 ms and 844 MB. The matrix with the same 8×8192 slots takes 204 ms and 1.7 GB, and it is slower and larger at every length in this range. Full RunningTensor does not cross under attention by length 8192 at this width. RT/4 does stay close to the matrix-state models. Chunk size is now reported in the appendix, but the new scaling figure does not sweep width.

Critical weakness (5).

The related work now has this paragraph. Higher-order attention grows with length because its score runs over tokens (Alman and Song, ICLR 2024); bilinear attention is still one pairwise score (Kim, Jun, and Zhang, NeurIPS 2018). RunningTensor raises the state order and stays linear: one rank-1 write, scored by a product of two inner products. Tensorized fast weights compress a matrix (Ma et al., NeurIPS 2019); the additive core is linear attention on a two-axis key, already in a free matrix of that width, with each axis queryable. Two-query memory reads each value with two queries (Schlag, Munkhdalai, and Schmidhuber, arXiv 2020; Schlag and Schmidhuber, NeurIPS 2018): one key leaves a matrix, and one query per axis of a third-order state is that outer product, plus a forget gate, a tensor delta rule, and query-skip gates. The matched-slot ablation is the comparison with the free matrix.

Questions.

We no longer say "strictly more expressive." nor claim a larger function class at equal state size.

Slots are matched in that RunningTensor matrix ablation, and the equal-slot matrix is slower, so the comparison is not FLOPs-matched. We are rerunning Attention and GLA for table 1, with a better head-layout match. All models used the same learning rates, 100 epochs, seeds, and default initialization. Those choices are in the appendix.

Chunk size is now reported in the Appendix. Peak memory and training-step time against length, at fixed width 128, are in the new scaling figure. Full RunningTensor stays above SDPA attention through length 8192. The equal-slot matrix lags behind the factorized state on both time and memory. The ms column in Table 1 remains validation forward time.

The time-series tables use three seeds and one learning rate chosen on Heartbeat. A per-dataset search was beyond this sweep. Full RunningTensor is reported on Speech Commands, at 71.9, with RT/4-Lin second at 77.5 behind ΔNet at 81.1. LIBERO is one seed.

OneLinear is the dense fused map, at 275K parameters against 175K for ConcOnlyC, and its average is 40.8 against 95.0. We do not have a separate dense layer plus an MLP locked to the same parameter count, and the 4096 drop is not isolated into parameter count, optimization, or expressivity.

The novel-contribution boundary is the related-work paragraph above.

Dear Reviewer,

Given that the author-reviewer discussion period will end tomorrow (November 18), we wanted to follow up on our detailed response to your extensive and helpful feedback.

We have made substantial revisions addressing your concerns, including refining our expressivity claims, adding state-matched comparisons, expanding the experimental evaluation with additional seeds, reporting training-time and memory measurements, and strengthening the discussion of related work. We have also clarified the remaining limitations of our experimental evaluation.

We would greatly appreciate it if you could revisit our response and the revised manuscript before the discussion period concludes, and let us know whether the revisions address your main concerns or whether there are any remaining points you would like us to clarify.

Thank you again for your thorough review and constructive suggestions.

Reviewer_xns6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RunningTensor, an architecture that generalizes the second-order matrix state of linear attention to a higher-order (specifically instantiated as order-3) tensor state. The framework updates the state via rank-1 outer products and reads it by contracting against multiple query vectors, aiming to increase memory capacity while maintaining linear complexity with respect to sequence length. The model is evaluated on Multi-Query Associative Recall as well as real-world tasks including multivariate time-series classification, long-horizon forecasting, and speech recognition.

Critical strengths

The paper provides thorough ablation studies isolating the independent contributions of the ConcOnlyC input mix, query-skip gates, output gating, and the tensor delta rule. The method is validated on diverse real-world signals (UEA, forecasting, and Speech Commands), demonstrating that the reduced RT/4 variant remains viable and competitive in long-context scenarios.

Critical weaknesses
  1. The proposed additive tensor update St=St−1+vt⊗kt⊗rt\mathcal S_t=\mathcal S_{t-1}+v_t\otimes k_t\otimes r_t and readout ot=⟨St,qt,ut⟩o_t=\langle\mathcal S_t,q_t,u_t\rangle can be exactly rewritten. By defining k~t=kt⊗rt\tilde k_t=k_t\otimes r_t and q~t=qt⊗ut\tilde q_t=q_t\otimes u_t, and flattening the third-order tensor into a matrix MtM_t, the mechanism becomes mathematically identical to standard linear attention on expanded features: Mt=Mt−1+vtk~t⊤M_t=M_{t-1}+v_t\tilde k_t^\top and ot=Mtq~to_t=M_t\tilde q_t. Similarly, the proposed tensor delta rule unrolls into a standard Gated DeltaNet update on these high-dimensional features[cite: 51]. Proposition 1 only proves that standard linear attention is a special case[cite: 50], but it fails to prove that a tensor organization is strictly superior to a matrix model utilizing an identical feature expansion mapping.

  2. In the core ablation study (Table 2), the matrix state is restricted to a budget of 16×32=51216\times32=512 elements per head, whereas the tensor state consumes 16×16×32=8,19216\times16\times32=8,192 elements[cite: 54, 55]. Although both models have a similar network parameter count (around 175k), the online state size directly dictates information storage capacity and computational overhead. The current results only prove that a 16x larger state budget improves recall, completely failing to demonstrate that the tensor organization itself is superior under a strictly matched state budget. Furthermore, the capacity claim of O(W2)→O(Wo)\mathcal{O}(W^2) \to \mathcal{O}(W^o) merely describes dimensional growth, lacking rigorous proof of how many associations can actually be reliably stored.

  3. While the model maintains linear complexity with respect to sequence length TT, the direct cost per read/write step for a third-order tensor spikes to O(dvdkdr)\mathcal{O}(d_vd_kd_r). The full RT model exceeded the time budget and failed to finish the longest Speech Commands task, proving that this immense computational penalty cannot be offset simply by having fewer network parameters. The paper urgently needs comprehensive end-to-end efficiency tests under strictly matched state and computational budgets. This must include training throughput, peak memory consumption, and loss reduction ratios. The high-order scalar key-value erasure and update mechanisms are highly likely to severely drag down practical hardware training efficiency.

Questions for authors

Refer to weakness

3 comments

Thanks a lot for your feedback.

The additive update is linear attention on Kronecker features, and Proposition 1 does not prove that a tensor is strictly superior to a matrix with the same expansion.

We agree, and the introduction now says this instead of claiming that the extra modes are strictly more expressive. The comparison that holds the state budget comparable between tensor state and matrix state is now in the ablation, for the same ConcOnlyC mix. The tensor is 16x16x32 (8192 slots, 175K, 313.7 ms) and averages 95.0. The slots-matched matrix is 256x32 (8192 slots, 186K, 431.4 ms) and averages 71.9. The dk-matched matrix is 16x32 (512 slots, 175K, 282.5 ms) and averages 23.0. The slots-matched matrix holds 96.0 and 94.6 at 1024 and 2048, then drops to 24.9 at 4096, where the tensor is still at 96.5.

The old matrix budget was 512 slots and the tensor budget was 8192, so the result only shows that a larger state helps, and the O(W^3) claim does not prove how many associations can be stored.

This is the same comparison. The 16×32 matrix stores 512 scalars per head and the 16×16×32 tensor stores 8192. The new rows in the ablation study give also the 8192 slots matrix option. Under that match, ConcOnlyC still favors the tensor, at nearly the same parameter count (175K against 186K) and at a lower forward time (313.7 ms against 431.4 ms). The O(W^o) sentence is the slot count and is a first step in a gradual introduction to the idea. The MQAR ablation table shows the empirical measurement of recall under that budget, it is not a theorem about how many associations the state can store.

Full RT is too expensive per step, it did not finish Speech Commands, and the paper needs matched measurements of throughput, memory, and loss.

Full RT has now finished Speech Commands: 71.9 accuracy at 278.5 ms, behind RT/4 at 76.0 and 14.9 ms. The scaling figure times and measures peak memory of one layer, batch 1, with no feed-forward, from length 256 to 8192, with and without gradient computation. At 8192, the matrix with the same 8×8192 slots takes 204 ms and 1.7 GB per training step. Full RT takes 186 ms and 844 MB. RT/4, at 8×2048 slots, takes 11 ms and 258 MB. GLA, Mamba-2, Gated ΔNet stay between 4.8 and 7.7 ms and between 65 and 104 MB.

Dear Reviewer,

Given that the author-reviewer discussion period will end tomorrow (November 18), we wanted to follow up on our response to your insightful comments.

In particular, we have revised the manuscript to clarify the relationship between RunningTensor and matrix-state linear attention, removed the claim of strict expressivity superiority, and added comparisons under matched state sizes. We have also expanded the efficiency analysis to include training time and peak memory measurements.

We would be very grateful if you could take another look at our response and the revised manuscript before the discussion period concludes, and let us know whether these changes address your main concerns.

Thank you again for your careful assessment and valuable suggestions.

Dear reviewer, thanks a lot again for your reply, it has undoubtedly helped to improve the manuscript. Since the deadline is here, we were wondering if you found our replies and the additional material convincing!

Reviewer_4mqROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

They replace the v k outer product of linear attention with a v k r outer product, making the state an order 3 tensor instead of a matrix (and make clear it can be extended beyond order 3). They then contract with additional query vectors (u in addition to q in the order 3 case) to keep the output of the unit the same dimension as before. They claim this to be a new contribution, and they demonstrate it working.

There is some additional complexity, some of which is standard, some of which is new to the order > 2 state case.

They demonstrate good performance on the MQAR benchmark, show ablations, and go on to show how well it performs on real time series benchmarks.

Critical strengths

In the methodology it was refreshingly clear to see what they have done.

MQAR validation appears to be very impressive. Though I am surprised Mamba-2 and GLA do so badly here.

The ablations demonstrate what the authors want to show. Especially showing that moving from a matrix state to a third order state contributes substantially.

Critical weaknesses

On the time series benchmarks I have a few concerns. We are looking at a single seed and single learning rate (not tuned per model per dataset, only selected from a probe on ‘Heartbeat’). Oddly poor results for RT in table 3, with ‘RT/4 - lin’ outperforming.

I would like to see an update to 3.4 with more careful comparisons with several random seeds, tuned learning rate and some confidence that the models were trained for long enough.

An appendix with some fleshed out details and hyperparameters is missing and would be extremely helpful to understand some of the later sections.

Questions for authors

What exactly is the Attention comparison is on table 1 (or the Softmax Attention comparisons in 3.4)? Is it just a standard causal Transformer? Why does it have that parameter count listed? Could you please describe what this comparison is?

What were the state sizes for the recurrent comparisons in table 1? How does a different choice of state size affect the comparison?

Ablations roughly parameter-matched the matrix and 3rd order cases, but what are the results if they are matched in state size?

For table 1 ‘the best validation F1 over 3 seeds and 3 learning rates’ is this an average over the seeds at the best learning rate or the best seed? I think it should be the former and more seeds would be helpful.

6 comments

Thanks for spelling out the strengths, and for the concrete requests.

On the time series benchmarks...

We have added two additional seeds per configuration and two more datasets. The extra seeds do not reverse the real-data ranking. Full RunningTensor leads on MQAR. On UEA, forecasting, and Speech Commands, RT/4-Lin is the strongest RunningTensor variant, and on UEA it is the best row (65.5). The larger RT remains behind it. Two further real tasks are in the paper. On Tiny-scale ImageNet, RT/4-Lin with the fused output gate reaches 76.10 top-1. On LIBERO distillation, each reported row is one seed: small RT/4 is the best student (val 0.1018, Huber 0.0103, MSE 0.0862), and large RT/4 matches the Gated ΔNet plateau at 0.2506.

ImageNet validation, Tiny-scale models. Top-1 is the last epoch. Best is the best top-1 during training.

MethodPar.Top-1Top-5BestLoss ↓
Mamba7.15M75.8992.7875.911.079
Gated ΔNet7.17M75.7992.7275.791.103
Longhorn6.85M75.6492.8475.751.084
RT/4-Lin ∑ g^out @t7.08M76.1092.9676.271.088
RT/4-Lin ∑ g^out7.08M75.3892.6075.511.124
RT/4-Lin7.14M71.1089.9571.101.324
RT/45.05M67.2888.2667.571.508

LIBERO open-loop distillation. One seed per row. Lower is better. The teacher rows are omitted.

MethodSizePar.msValHuberMSEE. stop
Qwen3L19.4M7.90.12630.01360.1189Yes
Qwen3S5.6M5.40.13300.01440.1229Yes
ΔNetL6.9M6.90.16910.02500.2719Yes
ΔNetS1.8M4.70.12530.01300.1206No
Gated ΔNetL8.9M6.90.25060.02500.2126Yes
Gated ΔNetS2.3M4.40.24750.02510.2282Yes
RT/4L7.0M77.40.25060.02500.2126Yes
RT/4S1.9M13.40.10180.01030.0862No
RT/4 (no Δ)L7.1M10.30.25060.02500.2129Yes
RT/4 (no Δ)S1.9M6.50.11480.01170.0976Yes

I would like to see an update...

Now we have 3 seeds per model per dataset, and the results are more clear than the 1 seed results. The small Heartbeat grid search compared 10e-3 and 10e−4 for attention, Gated ΔNet, and RunningTensor onlyr. 10e-3 had the lower validation loss on all three, and every architecture and dataset uses it. A separate search on every dataset would also have multiplied the sweep by the number of datasets, which we did not have the budget for. We also specify in the appendix the exact experimental protocol, number of epochs and early stopping patience included. UEA classification runs for 30 epochs with accuracy patience 10, and early stopping fired on 55 of 69 finished cells. Forecasting runs for 10 epochs with validation-MSE patience 5, and early stopping fired on 148 of 265 finished cells. Speech Commands runs for 10 epochs with patience 10, so the stopper does not fire inside that budget. The appendix records this protocol. As expected, the training setting is held the same for every architecture.

An appendix with some...

Done, as you suggested.

What exactly is the Attention...

You are right for pointing out, we actually used a larger head dimension of 128, instead of the 32 used by the other methods by mistake, and we are currently running those experiments.

What were the state sizes...

Now we specify all relevant hyper parameters in the table: number heads, head dimension, d_k, d_v, d_r, expansion and slots. In the ablation we show that if we increase the matrix state to have the same number of slots as the tensor state, we improve the performace wrt the smaller matrix state, without reaching the performance of the tensor state.

Ablations roughly parameter-matched...

We added this scenario in the ablation, as suggested by you and other reviewers. The slots matched matrix state improves wrt the small matrix state without reaching the tensor state. The Avg ConcOnlyC column at matched-slots matrix state is 71.9, against 23.0 for the matched-d_k matrix state and 95.0 for the tensor state.

For table 1 ‘the best validation F1...

It used to be max lr of max seed, but now is max lr of mean seed. The results are coherent with the previous ones, but this is the correct way, thanks for pointing out.

Dear Reviewer,

Given that the author-reviewer discussion period will end tomorrow (November 18), we wanted to follow up on our response to your thoughtful comments.

We have made several revisions addressing your concerns, including adding experiments with three random seeds, expanding the evaluation to additional datasets, and providing more detailed experimental protocols and hyperparameters in the appendix. We have also clarified the attention baselines and updated the relevant comparisons.

We would greatly appreciate it if you could take another look at our response and revised manuscript before the discussion period concludes, and let us know whether these changes address your concerns.

Thank you again for your time and constructive feedback.

Thank you to the authors for the response.

It is good to see you are using the mean across seeds, and have added more seeds to some results. But I think you should quote the variation among seeds. The differences are small enough in many cases that I'm not sure what to make of the mean across 3 seeds with no variation. Could you also quote the standard deviation?

I still have concerns about learning rate selection, but the matched state size comparison is a welcome addition.

I appreciate the effort that is being made to improve the results section, but I'm not yet confident in the soundness and significance of the results.

I appreciate that there is little time left in the author-reviewer response period. However, since the significance of the work depends heavily on the empirical comparisons, I need greater confidence that those comparisons support the conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.