acceptodds
Under review as a conference paper at ICLR 2027

Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling

Abstract

The quadratic cost of self-attention limits scalability to long sequences from multidimensional data. We introduce Tucker bottleneck attention (TuBA), which exploits low-rank tensor structure for efficient global token mixing. TuBA projects hidden tensors into compact Tucker cores, performs multi-head self-attention and linear projections on the cores, and writes updates back to the ambient space, enabling subquadratic computation. Its autoregressive extension combines bidirectional interactions within cores with causal attention across cores. On video prediction and global weather forecasting, TuBA achieves favorable accuracy–efficiency trade-offs over standard and efficient attention and task-specific models. Compared to standard self-attention, TuBA reduces error and computation by up to and for video prediction and and for autoregressive weather forecasting, with speedups up to . Low-rank Tucker cores and multi-frame generation also outperform full-rank attention and frame-by-frame generation, respectively.

Primary area:
learning on time series and dynamical systems
Keywords:
Efficient attention, Transformers, Low-rank representations, Multidimensional sequence modeling, Tensor methods

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HQTrOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Tucker Bottleneck Attention (TuBA) for multidimensional sequence modeling. Learned mode-wise projections compress token axes into a Tucker core, standard self-attention operates on the core, and separate write projections map the update back to the original token space. The paper provides an interpretation of this operation, analyzes the expressivity of independent read/write factors, and introduces a block-causal extension. Experiments on video prediction and weather forecasting demonstrate favorable accuracy–efficiency trade-offs against the evaluated baselines.

Critical strengths
  1. The motivation is reasonable for data with native multidimensional structure. The observation that hidden representations admit approximately shared Tucker subspaces supports using learned mode-wise projections. The method is clearly formulated and integrates naturally into a transformer residual block.
  2. The experiments demonstrate useful trade-offs on two application domains. Comparisons using a shared backbone help isolate the attention mechanism, and the paper reports both predictive metrics and computational measurements. The benefits are meaningful even though TuBA does not dominate every metric or execution setting.
Critical weaknesses
  1. The contribution relative to a directly related predecessor is insufficiently established. TEAFormers (Kong et al., 2024) already describes Tucker compression of embedded representations, attention on the compressed core, and reconstruction. Its compression includes the sequence-length axis. I could not find a citation, discussion, or comparison with this work in the submission. TuBA has potentially meaningful distinctions, including independent read/write factors and its causal formulation, but the current presentation does not establish which advances go beyond this closely related pipeline or their empirical importance.
  2. The experiments do not isolate the benefit of Tucker structure from that of a generic token bottleneck. Equation (8) expresses TuBA as B\mathrm{MHSA}(A^\top X), with Kronecker-structured read/write matrices. A direct control would replace these with unstructured learned projections while retaining core attention and the same residual architecture. Linformer, Nyström attention, and linear attention do not provide this control. The rank sweep demonstrates benefits of compression within TuBA, but does not establish that the multidimensional factorization itself is responsible. Similarly, the expressivity result for independent read/write factors is not accompanied by a tied-versus-untied performance ablation.

Reference

Kong et al. (2024). TEAFormers: TEnsor-Augmented Transformers for Multi-Dimensional Time Series Forecasting. https://arxiv.org/abs/2410.20439

Questions for authors
  1. How does TuBA differ substantively from TEAFormer’s compression–attention–reconstruction pipeline? Please clarify the methodological distinctions and provide a controlled comparison, or explain any concrete obstacles to implementing one.
  2. Can you compare TuBA with a variant that replaces the Kronecker-structured (A,B) in Equation (8) with unconstrained learnable dense matrices of the same shapes, while retaining core attention and the same residual architecture? Comparisons at the same core size and at comparable resource budgets would help determine whether the gains come from Tucker structure or generic token compression. A tied-versus-untied ablation would further clarify the practical benefit of separate read/write factors.
2 comments
AuthorsOpenReview ↗

Response to Reviewer HQTr

W1/Q1. The contribution relative to a directly related predecessor (TEAFormers ) is insufficiently established.

We thank the reviewer for pointing out TEAFormer (Kong et al., 2024), an arXiv preprint we were not aware of; we cite and discuss it in the revision. Both methods attend over a Tucker core, but TEAFormer obtains its core by running an SVD-based decomposition (using TensorLy’s HOOI, up to 100 iterations) on every input, in every layer, at every training step, and never learns the factors. This increases the computational cost and offsets the efficiency gain of attending over a smaller core. Their experiments use short sequences (at most 720) and 2 encoder layers, while our benchmarks have much longer token sequences (KittiCaltech 3,200, Human3.6M 4,096, ERA5 6,144) and deeper models, where this cost would grow substantially. Their code is also released as a Google Drive folder rather than a maintained repository, and we found issues that prevent a faithful reproduction: the decomposition is applied across the batch dimension, so each sample’s output depends on other samples in the batch, and the gradient path is cut at the decomposition and reconstruction steps. TuBA, in contrast, performs no decomposition at any point: its factors are ordinary parameters learned end-to-end with backpropagation and applied as a cheap linear projection. Moreover, TuBA’s factors are shared across the entire dataset, so each sample’s distinct content is captured in its core, which attention fully processes. In TEAFormer, the factors are recomputed for each batch, so part of the input-specific information resides in the factors and bypasses attention, which operates only on the core. Further differences include TuBA’s untied read/write factors (Theorem 1), its full-resolution residual update, and its block-causal formulation for autoregressive generation, which TEAFormer’s authors report they could not achieve.

W2/Q2. The experiments do not isolate the benefit of Tucker structure from that of a generic token bottleneck.

We thank the reviewer for suggesting this control. We replaced the Kronecker-structured read and write matrices with unconstrained learned dense matrices, keeping core attention and the same residual architecture, at comparable parameters and FLOPs. We also added a tied-factor variant of TuBA (Aₙ = Bₙ). The dense bottleneck is worse on every error metric across all three benchmarks: TuBA reduces MSE by 35% on KittiCaltech and 69% on Human3.6M, and NRMSE by 39% on ERA5. A key shortcoming of dense projections is their parameter cost: each read or write matrix adds N×R parameters per layer, which grows with the number of tokens. TuBA’s Kronecker factorization breaks this into small per-mode factors, so its overhead is negligible. As a result, keeping the dense variant’s parameters close to TuBA’s forces a much smaller core, which limits its accuracy. This shows that TuBA’s gains come from the multidimensional representation, not from generic token compression. The Kronecker structure lets TuBA keep a larger, mode-structured core at the same cost, with per-layer factor parameters that scale with the sum of the mode sizes rather than with the number of tokens. The dense variant can have lower latency, since its read and write are each a single matrix multiplication, but this comes at a substantial cost in accuracy. For tied versus untied factors, untied factors improve accuracy on ERA5 and KittiCaltech and match tied factors on Human3.6M, consistent with Theorem 1. We will add this comparison to the revised paper.

KittiCaltech.

MethodMSE↓MAE↓SSIM↑PSNR↑LPIPS↓ParamsFLOPsFPSFPS*
TuBA (untied)156.071641.70.896327.250.07726.98M29.2G142382
TuBA (tied)158.201632.50.888427.140.09136.97M29.2G143388
Dense241.432062.60.848225.130.12488.60M27.8G319504

Human3.6M.

MethodMSE↓MAE↓SSIM↑PSNR↑LPIPS↓ParamsFLOPsFPSFPS*
TuBA (untied)113.211499.30.966433.790.033911.15M63.5G580739
TuBA (tied)112.991493.30.966433.790.034711.14M63.5G579737
Dense362.652338.50.949229.080.044812.18M64.5G546753

ERA5.

MethodNRMSE↓z500↓t850↓t2m↓u10↓v10↓ParamsFLOPsclips/s
TuBA (untied)0.3990.1410.1620.1260.5360.6278.72M40.8G600
TuBA (tied)0.4260.1600.1750.1330.5720.6718.71M40.8G598
Dense0.6550.3050.2940.2580.7910.92110.87M41.0G700

Thank you for the clarifications and additional experiments. Your response addressed my main concerns, and I’ll raise my score from 2 to 3, good luck!

Reviewer_xih6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Tucker Bottleneck Attention (TuBA). The key idea is to compress the multi dim tensors into a smaller core, using a trainable paraemters along each axis of the input tensors. The model then map the results back to the original dimensions. The read and write matrices are learned separately. The paper also extends this idea to autoregressive prediction. The method is tested on video and weahther prediction.

Critical strengths
  1. The method is relatively simple and easy to understand. Conducting both attention and the Q/K/V projections within a smaller core gives a clear explanation for the computational savings.

  2. The experiments use shared architectures for several attention baselines, which helps compare the attention operations. The paper also includes useful analysis of core size,training memory, and running time.

  3. The improvement over standard attention is substantial on several tasks.

Critical weaknesses
  1. It is unclear how much of the improvement specifically comes from the Tucker structure. The general idea of compressing inputs into a smaller latent space before conducting attention has been studied before, such as in Perceiver. Though the paper compares with several efficient attention methods, these methods also differ in where and how the compression is applied. For example, what if I simply pool the tokens, conduct attention, and then upsample the results? Or use read and write projections without the Tucker factorization? In my opinion, the current results support the overall bottleneck design, but do not sufficiently justify the additional value of the proposed factorization.

  2. The comparison with efficient attention baselines needs more checking. In the video experiments, Linformer uses a projection length of 128 and Nystrom uses 64 landmarks, while TuBA uses 768 or 1287 core tokens. These numbers are not directly equivalent across methods, but the compression settings are quite different. It is unclear whether the poor baseline results partly come from compressing too aggressively. The paper should tune these sizes and compare the accuracy under similar computational budgets, instead of relying on one configuration for each baseline.

  3. The practical advantages are less consistent than the FLOPs reduction suggests. On KittiCaltech, mixed-precision throughput is 382 FPS for TuBA and 392 FPS for ViT. Also, TuBA’s MSE improvement over SimVP is small, from 156.94 to 156.07, while its SSIM and LPIPS are worse. The authors acknowledge the effect of optimized attention kernels, but the results still need a clearer discussion of when TuBA gives an actual speed advantage and when it mainly reduces FLOPs. I also could not find repeated-seed results to check whether the smaller accuracy differences are stable.

  4. Part of the weather forecasting improvement comes from generating multiple frames jointly. Table 4 shows that TuBA’s NRMSE changes from 0.426 with single-frame generation to 0.349 with four-frame generation, compared with 0.442 for FourCastNet. The paper already discusses this point. However, the current comparison does not fully separate the benefit of Tucker attention from the benefit of changing the generation granularity. The authors should clarify and match the generation settings across the autoregressive baselines.

Questions for authors
  1. Could the authors compare TuBA with a pooling-attention-upsampling baseline and a non-factorized read/write bottleneck under comparable budgets? This would help distinguish the contribution of the Tucker structure from simply reducing the number of tokens.

  2. Could the authors vary the projection length and number of landmarks for Linformer and Nyström, and report accuracy against FLOPs and actual running time? Do the main advantages remain after tuning these baselines?

  3. Theorem 1 shows that independent read and write factors can represent mappings that tied factors cannot. Could the authors provide a direct forecasting ablation comparing tied and untied factors at the same core size? How much improvement does this design give in practice?

  4. For autoregressive weather forecasting, does ViT use the same four-frame generation, context, and KV-cache settings as TuBA? Could the main baselines be compared at matched generation granularity?

  5. Could the authors report results over multiple random seeds, particularly for the comparison with SimVP and the core-size ablation? Some differences are small, and it is unclear whether the selected configuration is consistently better.

2 comments
AuthorsOpenReview ↗

Response to Reviewer xih6 (Part 1)

W1/Q1. Isolating the contribution of the Tucker structure from generic token compression: why TuBA over a dense bottleneck or pooling?

We thank the reviewer for suggesting these controls. We compared with both. The dense bottleneck is worse on every error metric across all three benchmarks. This shows that TuBA’s gains come from the multidimensional representation, not from generic token compression. The Kronecker structure lets TuBA keep a larger, mode-structured core at the same cost, with per-layer factor parameters that scale with the sum of the mode sizes rather than with the number of tokens. The dense variant can have lower latency, since its read and write are each a single matrix multiplication, but this comes at a substantial cost in accuracy. Pooling applies fixed, non-trainable average pooling and upsampling. In principle it is a special case of TuBA or of a dense bottleneck. It is consistently worse than TuBA showing that learning the factors matters beyond simply reducing the number of tokens. We add these results to the revised paper.

KittiCaltech.

MethodMSE↓MAE↓SSIM↑PSNR↑LPIPS↓ParamsFLOPsFPSFPS*
TuBA156.071641.70.896327.250.07726.98M29.2G142382
Dense241.432062.60.848225.130.12488.60M27.8G319504
Pool178.561793.70.877226.560.10446.97M54.0G123338

Human3.6M.

MethodMSE↓MAE↓SSIM↑PSNR↑LPIPS↓ParamsFLOPsFPSFPS*
TuBA113.211499.30.966433.790.033911.15M63.5G580739
Dense362.652338.50.949229.080.044812.18M64.5G546753
Pool155.981688.10.960232.300.039811.13M82.2G534792

ERA5.

MethodNRMSE↓z500↓t850↓t2m↓u10↓v10↓ParamsFLOPsclips/s
TuBA0.3990.1410.1620.1260.5360.6278.72M40.8G600
Dense0.6550.3050.2940.2580.7910.92110.87M41.0G700
Pool0.5460.2520.2380.1680.7250.8758.71M72.5G606

W2/Q2. Ablation on configuration of Informer and Nystrom baselines to ensure the main advantage remains. We ran this ablation on KittiCaltech, reducing the baselines’ compression by increasing Linformer’s projection length (128 → 512) and Nyström’s landmarks (64 → 256), and report accuracy, FLOPs, and runtime. Reducing compression does not close the gap. Linformer does not improve with larger k, even with up to 6.6× more parameters and 1.5× more FLOPs than TuBA. Nyström stays roughly flat, even at 3.2× TuBA’s FLOPs. TuBA remains the most accurate on every metric, at comparable or lower FLOPs and comparable or higher mixed-precision throughput. The main advantages therefore hold. We also note that the original settings were not over-compressed. Linformer compresses only the keys and values, while the queries stay at full length. Nyström uses landmarks only to approximate the full N×N attention, while every token still attends and is updated. Their FLOPs were already matched to TuBA.

KittiCaltech.

MethodMSE↓MAE↓SSIM↑PSNR↑LPIPS↓ParamsFLOPsFPSFPS*
TuBA r = (9, 11, 13)156.071641.70.896327.250.07726.98M29.2G142382
Linformer k = 128285.742303.40.833724.290.134616.80M29.6G287381
Linformer k = 256289.162292.20.833824.200.130126.63M34.6G249325
Linformer k = 512291.592263.90.832524.250.124546.29M44.7G191251
Nyström m = 64206.801972.40.867225.870.10476.97M29.0G216185
Nyström m = 128205.101965.20.867025.920.10406.97M39.5G180190
Nyström m = 256210.772027.50.865325.710.11736.97M94.8G103145
AuthorsOpenReview ↗

Response to Reviewer xih6 (Part 2)

W3 (part1). When does TuBA yield actual speedups?

In fp32, the FLOP reduction translates directly into speed: 2.3× on KittiCaltech and 3.8× on Human3.6M over ViT, along with 3.5–5.1× lower training memory and up to 4.3× faster training (Table 6). With FlashAttention and bf16, the gain depends on sequence length. At KittiCaltech’s scale (N = 3,200), the fused kernel already makes ViT’s attention cheap, so TuBA is on par (382 vs. 392 FPS). As N grows, the quadratic cost outpaces what the kernel can hide, and the advantage appears. On ERA5, by using smaller patches with the core fixed, TuBA’s mixed-precision speedup over ViT grows steadily with N (table below). TuBA’s read/write contractions also currently run on generic kernels, so these numbers are a lower bound.

BenchmarkNCore SizeFLOPs vs. ViTMixed-precision speedup vs. ViT
KittiCaltech3,2001,2873.0× fewer0.97×
Human3.6M4,0967682.7× fewer1.06×
ERA5 (8×8 patches)6,1441,2804.9× fewer1.66×
ERA5 (4×8 patches)12,2881,28010.5× fewer2.71×
ERA5 (4×4 patches)24,5761,28022.0× fewer4.84×

W3 (Part2)/Q5. Does seed change the conclusion?

We now report results over three seeds (mean ± half-range) for TuBA and SimVP, and for the core-size ablation on KittiCaltech. The conclusions hold. On KittiCaltech and Human3.6M, TuBA’s and SimVP’s ranges do not overlap, so TuBA’s advantage is consistent across seeds. On ERA5, SimVP remains more accurate, as reported in the paper, while TuBA uses 40% fewer FLOPs and has 3.1× higher throughput. The comparison with SimVP is not only about accuracy: attention already runs faster than convolutional mixers on GPUs but is far less accurate, and TuBA closes that gap while keeping attention’s efficiency.

BenchmarkTuBASimVP
KittiCaltech MSE↓155.15 ± 0.99158.82 ± 1.88
Human3.6M MSE↓112.74 ± 0.47117.33 ± 0.42
ERA5 NRMSE↓0.405 ± 0.0060.373 ± 0.000

In the KittiCaltech core-size ablation, intermediate cores remain the best across seeds.

Core size R7209721188128716203200 (full)
KittiCaltech MSE↓168.30 ± 3.20167.60 ± 4.43157.62 ± 1.88155.15 ± 0.99161.55 ± 5.52172.41 ± 4.86

Q3. On tied versus untied factors.

We compared tied (Aₙ = Bₙ) and untied factors at the same core size. Untied factors improve accuracy on ERA5 and KittiCaltech and match tied factors on Human3.6M, consistent with Theorem 1.

Human3.6M.

MethodMSE↓MAE↓SSIM↑PSNR↑LPIPS↓ParamsFLOPsFPSFPS*
TuBA (untied)113.211499.30.966433.790.033911.15M63.5G580739
TuBA (tied)112.991493.30.966433.790.034711.14M63.5G579737

KittiCaltech.

MethodMSE↓MAE↓SSIM↑PSNR↑LPIPS↓ParamsFLOPsFPSFPS*
TuBA (untied)156.071641.70.896327.250.07726.98M29.2G142382
TuBA (tied)158.201632.50.888427.140.09136.97M29.2G143388

ERA5.

MethodNRMSE↓z500↓t850↓t2m↓u10↓v10↓ParamsFLOPsclips/s
TuBA (untied)0.3990.1410.1620.1260.5360.6278.72M40.8G600
TuBA (tied)0.4260.1600.1750.1330.5720.6718.71M40.8G598

W4/Q4. Matched autoregressive granularity. We now add ViT at four-frame generation (g = 4), with the same context and KV-cache setting as TuBA, so that both models are compared at both granularities. We also add an autoregressive linear-attention baseline at both granularities. Multi-frame generation improves all three models. At g = 1, TuBA is far more accurate than ViT and linear attention (0.426 vs. 0.555 and 0.604) with 5× fewer FLOPs than ViT. At g = 4, ViT is slightly more accurate (0.328 vs. 0.349), but TuBA uses 5.7× fewer FLOPs and runs 2.2× faster, while linear attention at similar cost lags far behind (0.508).

MethodgNRMSE↓z500↓t850↓t2m↓u10↓v10↓ParamsFLOPsclips/s
TuBA10.4260.1440.1610.1180.5470.6397.27M89.3G100
ViT10.5550.2030.2030.1840.6520.7677.26M451.2G62
Linear10.6040.2800.3100.2820.7410.8897.26M87.1G136
TuBA40.3490.1130.1360.1070.4580.5347.67M67.2G265
ViT40.3280.1040.1260.1010.4260.4967.66M382.3G121
Linear40.5080.2210.2150.1550.6810.8287.66M75.7G318
Reviewer_8mpoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the quadratic computational complexity of self-attention for multidimensional data, including videos and weather fields for data types. It introduces Tucker Bottleneck Attention (TuBA), which uses learned factor matrices along each tensor axis to compress hidden representations into a compact Tucker core. Standard multi-head attention is then applied to this core, and separate learned factors project the output back to the original grid. The authors also propose a block-causal variant that enables autoregressive (AR) generation by applying causal attention across Tucker cores. Experiments on Human3.6M, KittiCaltech, and ERA5 show lower prediction errors and reduced FLOP counts compared with ViT and efficient-attention baselines. The results further suggest that intermediate ranks can outperform full-rank attention.

Critical strengths
  • Compressing the hidden tensor directly, rather than approximating the attention matrix, is an elegant approach that naturally exploits multidimensional structure. Extending the method to block-causal attention over Tucker cores is also a logical and potentially useful contribution.

  • The proofs appear correct. By sharing the encoder, decoder, and transformer block across methods, the experimental setup helps isolate the effect of the token mixer. TuBA demonstrates consistent improvements over ViT across all three benchmarks.

  • The paper is clearly written and easy to follow. Implementation details, memory usage, and training time are provided in Appendix. TuBA could replace the standard attention sublayer with negligible additional parameter overhead, making it relatively straightforward to integrate into existing architectures.

Critical weaknesses

Major Weaknesses:

  • Full-rank TuBA already outperforms ViT by approximately 15% on KittiCaltech, reducing MSE from 207.33 to 176.98. From my understanding, this suggests that the learned factorization itself, rather than compression through rank reduction, may account for a substantial portion of the gains. The paper lacks some important ablations: fixed versus learned full-rank factors, an identity core, and tied versus untied factors. In particular, the theoretical result in Theorem 1 is not validated experimentally.

  • The rank sweep is limited to a single dataset, with only one run per configuration. I think having more seeds per experiment is very important for validating TuBA. Differences between neighboring settings reach approximately 9.5 MSE, exceeding the variation among the best-performing ranks. It is also unclear whether the ranks were selected using the test set. Meanwhile, Linformer and Nyström are evaluated at single configurations and no tuning is reported.

  • The evaluated ranks are fixed at approximately 19-40% of N, which corresponds to quadratic scaling with a smaller constant factor if the rank remains proportional to N. No experiments examine how complexity changes with increasing sequence length or resolution. Furthermore, with FlashAttention and mixed precision, TuBA achieves throughput broadly comparable to ViT, as suggested by Tables 1 and 2. I believe further validating this argument can prove very beneficial for TuBA specifically.

  • The choice of autoregressive baselines I think is not sufficientlly justified, and the AR ViT configuration is not specified. Additionally, no autoregressive linear-attention baseline is included.

  • Omranpour et al. (2025) is cited but not evaluated experimentally, leaving the relationship between TuBA and this work unclear.

Minor concerns:

  • The methodology for FLOP reporting is not documented. The authorss should specify the counting tool, which operations are included, and whether Figure 2 reports newly measured values or numbers taken from previous work.

  • TuBA's SSIM for Human3.6M is lower than that of SimVP and ViT, but this result is not discussed. Qualitative prediction examples would also help assess the visual quality of the generated sequences.

  • The learning rate, model depth, and positional encoding for the Tucker cores should be specified. In addition, the use of fixed-size factor matrices restricts changes to input resolution and sequence length. This constraint should be acknowledged explicitly.

  • No AI Statement in the main text (unsure whether it was required).

Questions for authors
  • Were the hyperparameters (e.g. ranks) selected using a validation set or the test set? Could you report mean and standard deviation across multiple (3+) random seeds? A clearly defined validation protocol and stable results would increase my confidence in the soundness of the evaluation.

  • How do fixed identity factors, learned full-rank factors, and tied versus untied factors compare? If the experiments establish that rank reduction itself accounts for most of the improvement, that would change my assessment of the method's central contribution.

  • Does the observed advantage of intermediate ranks over full-rank attention persist in a coarse rank sweep on Human3.6M and ERA5?

  • How do the best-performing ranks change as N increases, for example, with higher spatial resolution or longer prediction horizons? Such results will be very insightful and would help establish whether TuBA delivers the claimed subquadratic scaling in practice.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 8mpo (Part 1)

W1/Q2. The paper lacks some important ablations: fixed versus learned full-rank factors, identity factors, and tied versus untied factors.

We added the requested factor ablation on KittiCaltech and ERA5. We agree with the reviewer that the learned factors account for a substantial portion of the gains. Identity factors at full rank reduce TuBA to standard self-attention and the results are close on KittiCaltech, somewhat better on ERA5 (the small difference arises because, although both models use the same seed, their layers are initialized in a different order, so they start from different weights). Learned full-rank factors improve substantially over ViT on both benchmarks (MSE 207.33 → 176.98; NRMSE 0.492 → 0.379), because they add learned mixing along each mode. Rank reduction then provides compression. It lowers cost on both benchmarks, improves accuracy further on KittiCaltech (156.07), and retains most of the gain on ERA5 (0.399) at a fraction of the cost. The improvement thus comes from the learned Tucker factors, and the efficiency from rank reduction. We also added the requested tied-versus-untied comparison: untied factors improve over tied factors on ERA5 and KittiCaltech and match them on Human3.6M, consistent with Theorem 1.

Factor ablation.

VariantFactorsKittiCaltech core RKittiCaltech MSE↓ERA5 core RERA5 NRMSE↓
ViTnone (standard attention)3200207.3361440.492
TuBA, identity factorsfixed (A = B = I)3200209.8661440.467
TuBA, learned full-ranklearned3200176.9861440.379
TuBAlearned, rank-reduced1287156.0712800.399

KittiCaltech.

MethodMSE↓MAE↓SSIM↑PSNR↑LPIPS↓ParamsFLOPsFPSFPS*
TuBA (untied)156.071641.70.896327.250.07726.98M29.2G142382
TuBA (tied)158.201632.50.888427.140.09136.97M29.2G143388

Human3.6M.

MethodMSE↓MAE↓SSIM↑PSNR↑LPIPS↓ParamsFLOPsFPSFPS*
TuBA (untied)113.211499.30.966433.790.033911.15M63.5G580739
TuBA (tied)112.991493.30.966433.790.034711.14M63.5G579737

ERA5.

MethodNRMSE↓z500↓t850↓t2m↓u10↓v10↓ParamsFLOPsclips/s
TuBA (untied)0.3990.1410.1620.1260.5360.6278.72M40.8G600
TuBA (tied)0.4260.1600.1750.1330.5720.6718.71M40.8G598

Q3. Does the benefit of intermediate ranks over full-rank persist? We do not claim that lower rank is generally better, nor that TuBA’s gains come from rank reduction. Normally, a larger core with more parameters and FLOPs is expected to perform better, and our point is that this does not always hold. On KittiCaltech, our rank ablation, showed that intermediate cores are consistently better than the full-rank core. A possible explanation is that full attention over many tokens can over-mix them. Attending within a compact learned core may limit this effect in some settings. It does not hold universally: on ERA5, accuracy improves with core size, and R = 1280 retains most of the full-rank accuracy at a fraction of the cost. Full rank is therefore not necessarily optimal, and the core size sets an accuracy–efficiency trade-off. We revise the paper to include both rank ablations and discuss them together, and revised the wording to clarify that full rank is not necessarily optimal, rather than implying that lower rank is generally better, while reduced ranks may provide a favorable accuracy–efficiency trade-off.

Rank ablation, ERA5.

Core R320128039206144 (full)
Ranks (t, h, w)(10, 4, 8)(10, 8, 16)(10, 14, 28)(12, 16, 32)
NRMSE↓0.4400.3990.3930.379
AuthorsOpenReview ↗

Response to Reviewer 8mpo (Part 2)

W2/Q1. On rank selection, seeds, and baseline tuning.

We did not optimize the ranks on any dataset. The ranks were chosen by a budget rule, not tuned on test performance. In our main runs, the temporal rank was set close to the number of frames, and the spatial ranks were chosen only to keep TuBA’s parameters and FLOPs comparable to or lower than those of the attention baselines. So ranks were decided before any run conducted. The rank ablation in the paper, run after the main experiments, happened to show that the selected rank performs better than both larger and smaller cores for KittiCaltech. To address the reviewer’s comment, we repeated this ablation on over three seeds (mean ± half the min–max range). The seeds do not change the observation that intermediate cores remain the best for KittiCaltech. We also ran three seeds for TuBA and SimVP, the closest-performing baseline. On KittiCaltech and Human3.6M, TuBA is consistently better, with non-overlapping ranges. On ERA5, SimVP is consistently more accurate, as already reported in the paper, while TuBA uses 40% fewer FLOPs and has 3.1× higher throughput. The gaps to all other baselines are far larger than seed variation. Finally, we varied Linformer’s projection length (128–512) and Nyström’s landmarks (64–256) on KittiCaltech. Neither closes the gap to TuBA, even with up to 6.6× more parameters or 3.2× more FLOPs. Overall, neither the additional seeds nor the baseline tuning changes any of the conclusions reported in the paper.

Rank ablation, KittiCaltech (3 seeds, mean ± half-range).

Core size R7209721188128716203200 (full)
KittiCaltech MSE↓168.30 ± 3.20167.60 ± 4.43157.62 ± 1.88155.15 ± 0.99161.55 ± 5.52172.41 ± 4.86

TuBA vs. SimVP (3 seeds, mean ± half-range).

BenchmarkTuBASimVP
KittiCaltech MSE↓155.15 ± 0.99158.82 ± 1.88
Human3.6M MSE↓112.74 ± 0.47117.33 ± 0.42
ERA5 NRMSE↓0.405 ± 0.0060.373 ± 0.000

Linformer and Nyström sweep, KittiCaltech.

MethodMSE↓MAE↓SSIM↑PSNR↑LPIPS↓ParamsFLOPsFPSFPS*
TuBA r = (9, 11, 13)156.071641.70.896327.250.07726.98M29.2G142382
Linformer k = 128285.742303.40.833724.290.134616.80M29.6G287381
Linformer k = 256289.162292.20.833824.200.130126.63M34.6G249325
Linformer k = 512291.592263.90.832524.250.124546.29M44.7G191251
Nyström m = 64206.801972.40.867225.870.10476.97M29.0G216185
Nyström m = 128205.101965.20.867025.920.10406.97M39.5G180190
Nyström m = 256210.772027.50.865325.710.11736.97M94.8G103145

W3/Q4. How does TuBA scales with N and practical speed.

We increased N on ERA5 with smaller patches while keeping R fixed, and TuBA’s mixed-precision speedup over ViT grows steadily with N, from 1.66× to 4.84×. In fp32, the FLOP reduction already yields 2.3–3.8× speedups and 3.5–5.1× lower training memory (Table 6). The ranks are not a fixed fraction of N: N, R, and R/N vary across our benchmarks, and ERA5 uses a core comparable to KittiCaltech’s despite nearly twice as many tokens. The rank is an intrinsic property of the hidden representations and depends on the benchmark; we do not claim a universal guarantee. TuBA remains effective whenever the rank grows more slowly than the mode dimensions or N.

BenchmarkNRR/NFLOPs vs. ViTMixed-precision speedup vs. ViT
KittiCaltech3,2001,28740.2%3.0× fewer0.97×
Human3.6M4,09676818.8%2.7× fewer1.06×
ERA56,1441,28020.8%4.9× fewer1.66×
ERA512,2881,28010.4%10.5× fewer2.71×
ERA524,5761,2805.2%22.0× fewer4.84×
AuthorsOpenReview ↗

Response to Reviewer 8mpo (Part 3)

W4. Autoregressive baselines: justification, ViT configuration, and missing linear attention.

We have added the requested autoregressive linear-attention baseline. Our autoregressive baselines are ViT and linear attention, which share TuBA’s architecture, 12-frame context, KV cache, and training and differ only in the token mixer, and FourCastNet, an established autoregressive weather-forecasting model. In the submission, ViT generated one frame per step (g = 1); we now report ViT and linear attention at both one-frame (g = 1) and four-frame (g = 4) generation. At g = 1, TuBA is far more accurate than ViT and linear attention (0.426 vs. 0.555 and 0.604) with 5× fewer FLOPs than ViT. At g = 4, ViT is slightly more accurate (0.328 vs. 0.349), but TuBA uses 5.7× fewer FLOPs and runs 2.2× faster, while linear attention lags far behind (0.508). We revise the abstract, contribution, and paper to reflect this new comparison.

MethodgNRMSE↓z500↓t850↓t2m↓u10↓v10↓ParamsFLOPsclips/s
TuBA10.4260.1440.1610.1180.5470.6397.27M89.3G100
ViT10.5550.2030.2030.1840.6520.7677.26M451.2G62
Linear10.6040.2800.3100.2820.7410.8897.26M87.1G136
TuBA40.3490.1130.1360.1070.4580.5347.67M67.2G265
ViT40.3280.1040.1260.1010.4260.4967.66M382.3G121
Linear40.5080.2210.2150.1550.6810.8287.66M75.7G318

W5 Comparison with Omranpour et al. (2025). We have added a controlled comparison with HOT (Higher-Order Transformer; Omranpour et al., 2025), using its Kronecker-factorized attention in place of TuBA within the same model. HOT is not uniformly better or worse. On KittiCaltech, TuBA is more accurate on every metric (MSE 156.07 vs. 201.79) with fewer FLOPs. On Human3.6M, the two are almost on par in accuracy, but HOT needs about 31% more FLOPs and is slower. On ERA5, HOT is more accurate (NRMSE 0.367 vs. 0.399), but at 41% more FLOPs and 1.4× lower throughput. HOT’s gains thus come at a consistently higher cost. The two methods also differ structurally. HOT applies separate attention along each mode, so each attention is bounded by that mode’s full size and its cost is fixed by the input dimensions. TuBA instead reduces the rank of each mode, so its cost can be lowered further by choosing a smaller core. In addition, HOT is formulated and evaluated only as an encoder: its mode-wise attention pools over all other modes, including time, so causal or block-causal generation would require redesigning it. TuBA supports autoregressive generation directly through block-causal attention over its cores.

KittiCaltech.

MethodMSE↓MAE↓SSIM↑PSNR↑LPIPS↓ParamsFLOPsFPSFPS*
TuBA156.071641.70.896327.250.07727.0M29.2G142382
HOT201.791938.10.868426.060.111010.1M35.1G234235

Human3.6M.

MethodMSE↓MAE↓SSIM↑PSNR↑LPIPS↓ParamsFLOPsFPSFPS*
TuBA113.211499.30.966433.790.033911.1M63.5G580739
HOT111.921488.50.966833.800.034014.3M83.6G492573

ERA5.

MethodNRMSE↓z500↓t850↓t2m↓u10↓v10↓ParamsFLOPsclips/s
TuBA0.3990.1410.1620.1260.5360.6278.72M40.8G600
HOT0.3670.1220.1480.1200.4860.56710.82M57.5G417

Comments on the paper

1 comment

Dear Reviewers,

We thank you for your constructive comments. We have addressed all comments, and all changes are highlighted in blue in the revised PDF.

Main text

New baselines: HOT, Dense, Pool, and TuBA with tied factors. Autoregressive ViT and linear attention at one- and four-frame generation. Discussion of TEAFormer. Revised abstract, contributions, analysis, and conclusion.

Appendix

Training hyperparameters and FLOP-counting protocol. Factor, rank, and temporal-rank ablations, including three seeds and ERA5. Three-seed comparison with SimVP, and Linformer/Nyström sweeps. Scaling with sequence length. Qualitative predictions.

We thank you for your time and effort.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.