- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Tucker Bottleneck Attention (TuBA) for multidimensional sequence modeling. Learned mode-wise projections compress token axes into a Tucker core, standard self-attention operates on the core, and separate write projections map the update back to the original token space. The paper provides an interpretation of this operation, analyzes the expressivity of independent read/write factors, and introduces a block-causal extension. Experiments on video prediction and weather forecasting demonstrate favorable accuracy–efficiency trade-offs against the evaluated baselines.
Critical strengths
- The motivation is reasonable for data with native multidimensional structure. The observation that hidden representations admit approximately shared Tucker subspaces supports using learned mode-wise projections. The method is clearly formulated and integrates naturally into a transformer residual block.
- The experiments demonstrate useful trade-offs on two application domains. Comparisons using a shared backbone help isolate the attention mechanism, and the paper reports both predictive metrics and computational measurements. The benefits are meaningful even though TuBA does not dominate every metric or execution setting.
Critical weaknesses
- The contribution relative to a directly related predecessor is insufficiently established. TEAFormers (Kong et al., 2024) already describes Tucker compression of embedded representations, attention on the compressed core, and reconstruction. Its compression includes the sequence-length axis. I could not find a citation, discussion, or comparison with this work in the submission. TuBA has potentially meaningful distinctions, including independent read/write factors and its causal formulation, but the current presentation does not establish which advances go beyond this closely related pipeline or their empirical importance.
- The experiments do not isolate the benefit of Tucker structure from that of a generic token bottleneck. Equation (8) expresses TuBA as B\mathrm{MHSA}(A^\top X), with Kronecker-structured read/write matrices. A direct control would replace these with unstructured learned projections while retaining core attention and the same residual architecture. Linformer, Nyström attention, and linear attention do not provide this control. The rank sweep demonstrates benefits of compression within TuBA, but does not establish that the multidimensional factorization itself is responsible. Similarly, the expressivity result for independent read/write factors is not accompanied by a tied-versus-untied performance ablation.
Reference
Kong et al. (2024). TEAFormers: TEnsor-Augmented Transformers for Multi-Dimensional Time Series Forecasting. https://arxiv.org/abs/2410.20439
Questions for authors
- How does TuBA differ substantively from TEAFormer’s compression–attention–reconstruction pipeline? Please clarify the methodological distinctions and provide a controlled comparison, or explain any concrete obstacles to implementing one.
- Can you compare TuBA with a variant that replaces the Kronecker-structured (A,B) in Equation (8) with unconstrained learnable dense matrices of the same shapes, while retaining core attention and the same residual architecture? Comparisons at the same core size and at comparable resource budgets would help determine whether the gains come from Tucker structure or generic token compression. A tied-versus-untied ablation would further clarify the practical benefit of separate read/write factors.
2 comments
Thank you for the clarifications and additional experiments. Your response addressed my main concerns, and I’ll raise my score from 2 to 3, good luck!
Response to Reviewer HQTr
W1/Q1. The contribution relative to a directly related predecessor (TEAFormers ) is insufficiently established.
We thank the reviewer for pointing out TEAFormer (Kong et al., 2024), an arXiv preprint we were not aware of; we cite and discuss it in the revision. Both methods attend over a Tucker core, but TEAFormer obtains its core by running an SVD-based decomposition (using TensorLy’s HOOI, up to 100 iterations) on every input, in every layer, at every training step, and never learns the factors. This increases the computational cost and offsets the efficiency gain of attending over a smaller core. Their experiments use short sequences (at most 720) and 2 encoder layers, while our benchmarks have much longer token sequences (KittiCaltech 3,200, Human3.6M 4,096, ERA5 6,144) and deeper models, where this cost would grow substantially. Their code is also released as a Google Drive folder rather than a maintained repository, and we found issues that prevent a faithful reproduction: the decomposition is applied across the batch dimension, so each sample’s output depends on other samples in the batch, and the gradient path is cut at the decomposition and reconstruction steps. TuBA, in contrast, performs no decomposition at any point: its factors are ordinary parameters learned end-to-end with backpropagation and applied as a cheap linear projection. Moreover, TuBA’s factors are shared across the entire dataset, so each sample’s distinct content is captured in its core, which attention fully processes. In TEAFormer, the factors are recomputed for each batch, so part of the input-specific information resides in the factors and bypasses attention, which operates only on the core. Further differences include TuBA’s untied read/write factors (Theorem 1), its full-resolution residual update, and its block-causal formulation for autoregressive generation, which TEAFormer’s authors report they could not achieve.
W2/Q2. The experiments do not isolate the benefit of Tucker structure from that of a generic token bottleneck.
We thank the reviewer for suggesting this control. We replaced the Kronecker-structured read and write matrices with unconstrained learned dense matrices, keeping core attention and the same residual architecture, at comparable parameters and FLOPs. We also added a tied-factor variant of TuBA (Aₙ = Bₙ). The dense bottleneck is worse on every error metric across all three benchmarks: TuBA reduces MSE by 35% on KittiCaltech and 69% on Human3.6M, and NRMSE by 39% on ERA5. A key shortcoming of dense projections is their parameter cost: each read or write matrix adds N×R parameters per layer, which grows with the number of tokens. TuBA’s Kronecker factorization breaks this into small per-mode factors, so its overhead is negligible. As a result, keeping the dense variant’s parameters close to TuBA’s forces a much smaller core, which limits its accuracy. This shows that TuBA’s gains come from the multidimensional representation, not from generic token compression. The Kronecker structure lets TuBA keep a larger, mode-structured core at the same cost, with per-layer factor parameters that scale with the sum of the mode sizes rather than with the number of tokens. The dense variant can have lower latency, since its read and write are each a single matrix multiplication, but this comes at a substantial cost in accuracy. For tied versus untied factors, untied factors improve accuracy on ERA5 and KittiCaltech and match tied factors on Human3.6M, consistent with Theorem 1. We will add this comparison to the revised paper.
KittiCaltech.
Human3.6M.
ERA5.