- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper suggests Tucker maps method as a novel method faster than quadratic attention methods. Here tokens are arranged on an H×W grid and mixed with low-rank factors along the height, width and channel modes, summed over several heads. They authors suggest 3 ways this idea can used depending on the use case. There is a Tucker free version, a spectral version using Fourier transforms (DFT or DCT basis, O(N log N), parameter count independent of resolution) and a causal version for next-token prediction. Only the attention layer is swapped and everything else is kept the same. The method is evaluated on a small ViT trained on CIFAR-10, a character-level GPT trained on Tiny Shakespeare, and a nanoGPT model trained with Muon. The authors report reduced training FLOPs with broadly comparable performance, although the nanoGPT results show a modest drop in validation performance.
Critical strengths
- It’s nice that they have multiple versions for different requirements, including a separate causal one for next-token prediction. The non-causal spectral layer is simple and mathematically sound. It is basically a band-limited global convolution with low-rank channel mixing, it really is O(N log N), and its parameter count doesn’t depend on resolution.
- The comparisons are controlled. Only the mixer changes and initialization is shared, so the differences they see can be attributed to the Tucker layer, even though the experiments are too small to support the main claims.
- Mathematical explanations are sound and easily understandable.
Critical weaknesses
- The work isn’t really a replacement for attention as attention decides what to look at based on the input, since queries and keys come from the tokens. Here the mixing weights are fixed after training, so every input gets the same global filter. That makes it cheaper, but it likely loses content-based lookup like copying and recall, which language models depend on, and nothing in the paper tests this.
- The causal version used in all the GPT experiments is defined by applying a dense causal mask to G_H ⊗ G_W, so a naive implementation would be quadratic in sequence length. The paper does not provide enough implementation detail to establish the actual runtime complexity of its implementation. Footnote 4 acknowledges that the FFT-based complexity argument does not automatically extend to the causal case, but offers “no experimentally observed slowing” rather than a complexity analysis. Nevertheless, the abstract and conclusion still claim that the method is provably (quasi)linear. The authors should clarify how the causal operator is implemented and establish its computational complexity.
- At sequence lengths of 64–128 tokens and width 192, the Q/K/V/O projections can account for a substantial share of the computation relative to the quadratic attention products. Reduced projection costs may therefore explain some of the reported FLOP savings, rather than the improved scaling being the primary source of the gains. However, the paper does not provide a detailed cost breakdown or explain how FFT operations are counted. An ablation with parameter- and FLOP-matched baselines, such as attention with rank-32 projections, would help isolate the source of the savings.
- The experiments are too small. It’s a small ViT on CIFAR-10 for 20 epochs with one seed, and character-level Tiny Shakespeare. Nothing tests long inputs, where the method should actually matter, and there are no realistic language benchmarks.
- Closely related work isn’t cited or compared against. FNet, GFNet, AFNO, Hyena, S4 and MLP-Mixer all replace attention with Fourier or convolution-style mixing, so it’s unclear what Tucker adds beyond what’s already known.
- Some technical statements need clarification or correction. The paper says the Tucker layer preserves the rank and dimensions of the input tensor, but it is unclear what notion of rank is intended so the authors should clarify this given the constraints imposed by the low-rank factorization. The discussion of DeltaNet describes its state as N×N, but the recurrent state in Equation (1) has dimensions (d_v×d_k), independent of sequence length. The abstract's claim that performance deterioration occurs “almost never” should also be qualified in light of the nanoGPT results, where the Tucker model has higher validation loss and lower next-token accuracy. Finally, the statement that linear-attention approaches are “never deployed on their own” is too broad; standalone alternatives such as RWKV and RetNet are relevant counterexamples.
- These are smaller nit picks. G_W is written as W×H instead of W×W, and F_H F_W = I_H should be F_H F_H = I_H. Equation numbers 11 and 13 are each used twice, Table 1 headers are garbled with “NR” never defined, and Dosovitskiy et al. is cited twice. Footnote 5 describes a per-layer hyperparameter search that the experiments don’t seem to use.
- As a suggestion, the claims would need baselines matched for FLOPs and parameters (a smaller transformer, attention with rank-32 projections, FNet, GFNet, Hyena). Would also love to see how to method scales as the parameters increases.
Questions for authors
- Is the causal layer implemented with the dense masked matrix or as an FFT convolution? Can you show measured time and memory against sequence length (for example 512 to 32K) compared to attention with FlashAttention? This could change my assessment of the main claim.
- How were FFTs counted in the FLOPs? How much of the savings comes from the attention-score terms versus the removed projection layers?
- Was the hyper parameter search in footnote 5 actually run, and what is the 'uniform convergence criterion'? Is there a test split separate from the one used to pick checkpoints?
- Which factor is zero-initialized? If both α and B_C start at zero, their gradients vanish.