acceptodds
Withdrawn from ICLR 2027

EFFICIENT GENERATIVE AI BY TUCKER NETS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We propose a trainable token mixing transformation based on Tucker maps — which are of (quasi)linear complexity in the input. On experiments on ViT and GPTs we show that replacing attention layers by our Tucker maps caused % reduction in the number of FLOPs used in training — while almost never causing any deterioration of performance. We verify that the advantages of Tucker Nets on transformers, hold for both Adam(W) and Muon being used for training. Our Tucker layers are designed as drop-in replacements of the attention layers in any transformer architecture and thus pave a new way to efficient generative AI.

Primary area:
generative models
Keywords:
transformers, generative-AI, neural networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fPiLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper suggests Tucker maps method as a novel method faster than quadratic attention methods. Here tokens are arranged on an H×W grid and mixed with low-rank factors along the height, width and channel modes, summed over several heads. They authors suggest 3 ways this idea can used depending on the use case. There is a Tucker free version, a spectral version using Fourier transforms (DFT or DCT basis, O(N log N), parameter count independent of resolution) and a causal version for next-token prediction. Only the attention layer is swapped and everything else is kept the same. The method is evaluated on a small ViT trained on CIFAR-10, a character-level GPT trained on Tiny Shakespeare, and a nanoGPT model trained with Muon. The authors report reduced training FLOPs with broadly comparable performance, although the nanoGPT results show a modest drop in validation performance.

Critical strengths
  1. It’s nice that they have multiple versions for different requirements, including a separate causal one for next-token prediction. The non-causal spectral layer is simple and mathematically sound. It is basically a band-limited global convolution with low-rank channel mixing, it really is O(N log N), and its parameter count doesn’t depend on resolution.
  2. The comparisons are controlled. Only the mixer changes and initialization is shared, so the differences they see can be attributed to the Tucker layer, even though the experiments are too small to support the main claims.
  3. Mathematical explanations are sound and easily understandable.
Critical weaknesses
  1. The work isn’t really a replacement for attention as attention decides what to look at based on the input, since queries and keys come from the tokens. Here the mixing weights are fixed after training, so every input gets the same global filter. That makes it cheaper, but it likely loses content-based lookup like copying and recall, which language models depend on, and nothing in the paper tests this.
  2. The causal version used in all the GPT experiments is defined by applying a dense causal mask to G_H ⊗ G_W, so a naive implementation would be quadratic in sequence length. The paper does not provide enough implementation detail to establish the actual runtime complexity of its implementation. Footnote 4 acknowledges that the FFT-based complexity argument does not automatically extend to the causal case, but offers “no experimentally observed slowing” rather than a complexity analysis. Nevertheless, the abstract and conclusion still claim that the method is provably (quasi)linear. The authors should clarify how the causal operator is implemented and establish its computational complexity.
  3. At sequence lengths of 64–128 tokens and width 192, the Q/K/V/O projections can account for a substantial share of the computation relative to the quadratic attention products. Reduced projection costs may therefore explain some of the reported FLOP savings, rather than the improved scaling being the primary source of the gains. However, the paper does not provide a detailed cost breakdown or explain how FFT operations are counted. An ablation with parameter- and FLOP-matched baselines, such as attention with rank-32 projections, would help isolate the source of the savings.
  4. The experiments are too small. It’s a small ViT on CIFAR-10 for 20 epochs with one seed, and character-level Tiny Shakespeare. Nothing tests long inputs, where the method should actually matter, and there are no realistic language benchmarks.
  5. Closely related work isn’t cited or compared against. FNet, GFNet, AFNO, Hyena, S4 and MLP-Mixer all replace attention with Fourier or convolution-style mixing, so it’s unclear what Tucker adds beyond what’s already known.
  6. Some technical statements need clarification or correction. The paper says the Tucker layer preserves the rank and dimensions of the input tensor, but it is unclear what notion of rank is intended so the authors should clarify this given the constraints imposed by the low-rank factorization. The discussion of DeltaNet describes its state as N×N, but the recurrent state in Equation (1) has dimensions (d_v×d_k), independent of sequence length. The abstract's claim that performance deterioration occurs “almost never” should also be qualified in light of the nanoGPT results, where the Tucker model has higher validation loss and lower next-token accuracy. Finally, the statement that linear-attention approaches are “never deployed on their own” is too broad; standalone alternatives such as RWKV and RetNet are relevant counterexamples.
  7. These are smaller nit picks. G_W is written as W×H instead of W×W, and F_H F_W = I_H should be F_H F_H = I_H. Equation numbers 11 and 13 are each used twice, Table 1 headers are garbled with “NR” never defined, and Dosovitskiy et al. is cited twice. Footnote 5 describes a per-layer hyperparameter search that the experiments don’t seem to use.
  8. As a suggestion, the claims would need baselines matched for FLOPs and parameters (a smaller transformer, attention with rank-32 projections, FNet, GFNet, Hyena). Would also love to see how to method scales as the parameters increases.
Questions for authors
  1. Is the causal layer implemented with the dense masked matrix or as an FFT convolution? Can you show measured time and memory against sequence length (for example 512 to 32K) compared to attention with FlashAttention? This could change my assessment of the main claim.
  2. How were FFTs counted in the FLOPs? How much of the savings comes from the attention-score terms versus the removed projection layers?
  3. Was the hyper parameter search in footnote 5 actually run, and what is the 'uniform convergence criterion'? Is there a test split separate from the one used to pick checkpoints?
  4. Which factor is zero-initialized? If both α and B_C start at zero, their gradients vanish.
Reviewer_6rhjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Tucker layers as trainable token-mixing transformations to replace self-attention in Transformer blocks. The free Tucker map factorizes the token-mixing operation, while the proposed spectral Tucker maps parameterize spatial/sequence factors in Fourier or cosine bases, with a causal construction for autoregressive GPT models. The authors argue that the resulting maps have quasi-linear complexity in the number of input tokens and can be resolution-independent. They evaluate matched standard-attention and Tucker variants on CIFAR-10 ViTs and character-level GPT models on Tiny Shakespeare, reporting accuracy or language-modeling quality together with estimated FLOPs, training time, and scaling curves. They also report an experiment using Muon. The central empirical claim is that Tucker replacements substantially reduce computation with little or no predictive-performance loss.

Critical strengths
  1. The architectural idea is technically interesting: spectral Tucker maps provide a global, factorized token mixer and the paper defines a causal variant for autoregressive models. The resolution-independent parameterization and the attempt to replace attention rather than merely approximate its implementation are potentially meaningful contributions.

  2. The authors perform the right basic control for the replacement claim. In the ViT and GPT experiments, they state that the standard-attention and Tucker models share the architecture, data, optimization setup, and non-attention components, and they report multiple seeds together with predictive quality and computational measurements.

Critical weaknesses
  1. The main GPT evidence appears internally inconsistent. The text states that Tucker GPT reaches higher peak validation accuracy than Standard GPT, but Table 2 appears to report lower best validation accuracy and worse perplexity for the causal spectral Tucker model than for nanoGPT. If these values are read correctly, the claim of almost no performance deterioration is not supported; if they refer to different runs or metrics, the paper must explain the discrepancy. This issue directly affects the central quality-versus-efficiency conclusion.

  2. The empirical scope is too limited to support the broad “efficient generative AI” and general Transformer-replacement framing. The experiments use CIFAR-10 ViTs and Tiny Shakespeare character-level GPTs. The paper does not provide comparisons with strong efficient-attention or sequence-mixing alternatives, nor does it demonstrate larger language models, longer contexts, or realistic-scale pretraining. The paper should also distinguish clearly between lower FLOPs or time-to-best-checkpoint and equal predictive quality at a matched compute budget.

Questions for authors
  1. Please reconcile the statement that Tucker GPT has higher peak validation accuracy with the values in Table 2, which appear to show lower validation accuracy and higher perplexity than nanoGPT. Are the table and prose reporting different runs, metrics, or stopping rules?

  2. At equal validation quality, how much training time, memory, and wall-clock inference cost does Tucker save relative to standard attention? Please separate comparisons at equal steps, equal FLOPs, equal wall-clock time, and independently selected best checkpoints.

  3. Why are there no experiments against established efficient-attention or sequence-mixing baselines such as kernelized linear attention, DeltaNet variants, low-rank attention, or MLP/token mixers? Such comparisons are needed to support the claim that Tucker maps are a useful alternative rather than only a replacement for standard attention.

  4. How does the causal Tucker implementation enforce causality without reintroducing quadratic memory or computation through a dense causal mask? Please report the actual implementation complexity and memory use as sequence length grows.

  5. Can the authors expand the evaluation beyond CIFAR-10 and Tiny Shakespeare, particularly to larger language-modeling data and longer contexts? The current benchmarks are useful sanity checks but do not establish scalability to the generative-AI settings emphasized in the paper.

  6. Please revise the introduction to state precisely how Tucker token mixing differs from prior linear-attention methods: which part of attention is removed, what is retained, and what new expressivity or efficiency property follows from the spectral Tucker construction.

Reviewer_dUpTOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Tucker attention (sometimes referred to as Tucker- nets, layers, maps, model), a computationally efficient alternative to self-attention in Transformers. It uses tensor factorization and spectral transformations with quasi-linear complexity and evaluates them on ViT and GPT models. They report reductions in training FLOPs with small performance changes. The authors conclude that Tucker Nets offer a new direction for efficient architectures.

Critical strengths

1. Novel architectural approach: The paper introduces an interesting alternative to traditional self-attention through Tucker-based token mixing, contributing to the important research direction of efficient Transformer architectures.

2. Computational efficiency: The proposed approach demonstrates promising reductions in training FLOPs while maintaining relatively competitive performance on ViT and GPT benchmarks.

3. Ease of integration and potential for optimisation: Tucker Maps offer a relatively straightforward replacement for self-attention in existing Transformers.

Critical weaknesses

Weaknesses

Please take the following as constructive suggestions intended to strengthen the paper and increase its value to the research community.

  1. Insufficient motivation and comparison with existing alternatives: While reducing computational complexity is an important objective, the paper does not clearly establish the advantages of Tucker Maps over existing efficient attention mechanisms such as Linear Attention and DeltaNet. The experimental evaluation primarily compares against standard self-attention (Section 3). Including efficient attention baselines and evaluating computational scaling would better demonstrate when and why Tucker Maps should be preferred.

  2. Insufficient justification and ablation of architectural design choices: The transition from Free Tucker Maps to Spectral Tucker Maps in Section 2.1 introduces considerable complexity without clearly motivating its necessity. The Free Tucker Map already appears to offer a viable alternative to attention, yet its performance and limitations are not established before introducing the spectral formulation. Ablations comparing these variants and explaining the design choices would help identify which components contribute to performance improvements and provide valuable insights for future research.

  3. Clarity of mathematical formulation and presentation: The mathematical formulation, particularly in Section 2.1, is difficult to follow and would benefit from more intuitive notation aligned with existing attention literature (attentions all you need or linear attention). The token-mixing mechanism (Equations 2–5) and causal masking implementation (Section 2.2, Equation 11) would benefit from clearer explanations and illustrations. Consistent terminology across TuckerNets, TuckerMaps, TuckerLayers, Spectral Tucker Attention and the TuckerModel would also improve readability.

Questions for authors

Do the authors plan to publicly release their implementation of Tucker Maps to support reproducibility and further research?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.