acceptodds
Withdrawn from ICLR 2027

OmniTTT: Linear Complexity Multimodal Visual Generation with Test-Time Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Unified multimodal generative models jointly model text, images and videos within a shared context, enabling flexible visual generation but producing increasingly long token sequences dominated by visual tokens. Test-Time Training (TTT) offers a promising alternative to quadratic attention by compressing long contexts into an online-updated inner model with linear complexity. However, directly converting attention to TTT in pretrained generative models leads to substantial performance degradation under lightweight continual training. We attribute this difficulty to a modality asymmetry: language tokens are relatively few and inexpensive to retain with attention, yet difficult to compress without information loss, whereas visual tokens dominate computation and can be compressed effectively with the appropriate architectural design. Based on this observation, we introduce OmniTTT, a simple TTT-based architecture for efficient multimodal generation, which models visual interactions with TTT while retaining minimal attention for language. OmniTTT inherits weights from pretrained full-attention models and can be adapted through lightweight continual training without the need for training from scratch. Extensive experiments on text-to-image, text-to-video, and omnimodal reference-to-video generation demonstrate that OmniTTT substantially improves efficiency, achieving, for example, a 8.75 attention kernel and 2.87 full model speedup on reference-to-video generation with comparable quality.

Primary area:
generative models
Keywords:
Multimodal Visual Generation, Test-Time Training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eXtXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OmniTTT, which combines global visual TTT, local visual attention, and language attention to continually train pretrained multimodal generative models. It maintains generation quality close to full-attention baselines on text-to-image and text-to-video tasks while substantially reducing computational costs, and also demonstrates acceleration in reference-to-video generation.

Critical strengths

By combining global visual TTT, local attention, and language attention, the authors apply the approach to image, video, and reference-conditioned generation models. This provides a valuable practical exploration of reusing pretrained weights to reduce generation costs.

Critical weaknesses
  1. The claim regarding “unified generation models” is not adequately supported, as the three task categories are evaluated using three different backbones. Describing OmniTTT as “a general-purpose acceleration architecture for multimodal generative backbones” would be more appropriate.
  2. The experimental evaluation is insufficient. The comparisons cover relatively few methods, only one backbone is evaluated for each task category, and benchmark coverage is limited. In particular, the reference-to-video generation tasks lack quantitative quality evaluation. The paper appears insufficiently developed, and its core contributions are not clearly articulated.
Questions for authors
  1. The boldfacing of the “All” score in Table 1 appears incorrect: the +SFT baseline achieves a higher score.
  2. Please clarify the claim regarding “unified generation models,” given that the three task categories are evaluated using separate backbones. Would “a general-purpose acceleration architecture for multimodal generative backbones” more accurately describe the contribution?
  3. Please include additional comparison methods, evaluate more backbones for each task, broaden benchmark coverage, and provide quantitative quality evaluation for the reference-to-video generation tasks.
Reviewer_Nsi8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes OmniTTT, an efficient linear-complexity architecture for multimodal visual generation that takes text and other modalities as inputs to generate images and videos. The authors present preliminary observations showing that compressing language tokens with vanilla TTT degrades prompt alignment, while compressing visual tokens makes it difficult to recover the original generation quality under a limited training budget. To address these challenges, OmniTTT preserves softmax attention for language interactions while combining global TTT with tiled local window attention for visual interactions. This design enables the conversion of quadratic-complexity attention in pretrained diffusion Transformers into a more efficient architecture, preserving the original visual generation quality while reducing inference costs through lightweight continual training.

Critical strengths
  • Generation Quality and Efficiency: The proposed method maintains generation quality comparable to full-attention baselines and outperforms several existing efficient attention methods while substantially reducing computational costs. This is a meaningful contribution to efficient multimodal visual generation, where preserving generation quality while improving inference efficiency remains challenging.

  • Efficiency Across Multimodal Tasks: The paper demonstrates substantial inference speedups across text-to-video, multi-image reference-to-video, and video reference-to-video generation. In particular, the evaluation on large-scale pretrained models and long visual contexts provides valuable evidence of the practical applicability of the proposed approach.

Critical weaknesses
  • Limited Novelty Compared to Prior Work: Most of the proposed components build upon existing methods. In particular, T5T^5 [1] already explored combining global TTT with neighborhood attention (NAT) to preserve local visual interactions. Although OmniTTT adopts tiled local window attention, both approaches share the same underlying design principle of combining compressed global context with explicit local attention. Therefore, the proposed method appears to be more of an incremental engineering extension than a fundamentally novel approach.

  • Insufficient Justification of the Observations: The preliminary experiments do not sufficiently disentangle the causes of the observed performance degradation. In Observation 1, degraded prompt alignment is attributed to information loss from language compression, but it remains unclear whether this degradation is caused by TTT compression itself or by the training procedure and data. A full-attention baseline fine-tuned on the same data with the same training budget would help clarify this issue. In Observation 2, the failure of visual TTT is attributed to a limited training budget, but it remains unclear whether longer training would resolve the issue or whether the fixed-size inner model has insufficient capacity to compress the much longer visual sequences. These observations are also derived from CrossDiT, whereas the proposed method primarily targets joint attention in MMDiT architectures. More controlled ablations are needed to disentangle these factors and establish whether the observations generalize to the proposed architecture.

  • Insufficient Evaluation of Generation Quality and Prompt Alignment: The text-to-image evaluation and most ablation studies rely primarily on HPSv3[2], which does not explicitly measure fine-grained prompt following. Given that preserving semantic information is a key motivation, additional evaluations using GenEval[3], T2I-CompBench[4], or DPG-Bench[5] are necessary to assess whether the proposed method preserves objects, attributes, and spatial relationships. Furthermore, the multi-image and video reference-to-video experiments report efficiency results and qualitative examples without quantitative evaluations of generation quality, reference fidelity, or alignment. These missing evaluations limit the evidence supporting the method's effectiveness in complex multimodal generation tasks.

[1] Li, Yining, et al. "Linearizing Vision Transformer with Test-Time Training." arXiv preprint arXiv:2605.02772 (2026).
[2] Ma, Yuhang, et al. "Hpsv3: Towards wide-spectrum human preference score." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.
[3] Ghosh, Dhruba, Hannaneh Hajishirzi, and Ludwig Schmidt. "Geneval: An object-focused framework for evaluating text-to-image alignment." Advances in Neural Information Processing Systems 36 (2023): 52132-52152.
[4] Huang, Kaiyi, et al. "T2i-compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation." IEEE Transactions on Pattern Analysis and Machine Intelligence 47.5 (2025): 3563-3579.
[5] Hu, Xiwei, et al. "Ella: Equip diffusion models with llm for enhanced semantic alignment." arXiv preprint arXiv:2403.05135 (2024).

Questions for authors

[Limited Novelty Compared to Prior Work]

  • Given that T5T^5 already combines global TTT with neighborhood attention (NAT), could the authors clarify the key methodological differences and advantages of OmniTTT beyond prior work?

[Insufficient Justification of the Observations]

  • Could the authors provide a full cross-attention baseline fine-tuned on the same data with the same 2K-step training budget, along with training curves and quantitative prompt alignment evaluations (e.g., VideoAlign or VBench)? This would help disentangle the effects of TTT compression from those of the training procedure and data.
  • Could the authors provide training curves for vanilla visual TTT with a larger training budget (e.g., 5K, 10K, or 20K steps) and different inner-model capacities? This would help determine whether the failure is caused by insufficient training or limited memory capacity when compressing long visual sequences.
  • Could the authors clarify whether the issues identified in the preliminary observations on CrossDiT also arise in MMDiT and provide experimental evidence supporting the generalizability of these observations to MMDiT?

[Insufficient Evaluation of Generation Quality and Prompt Alignment]

  • Could the authors provide additional evaluations of SD3.5-Medium, SD3.5-Medium + SFT, T5T^5, OmniTTT, and OmniTTT-Fast using benchmarks such as GenEval, T2I-CompBench++, or DPG-Bench to assess whether OmniTTT preserves fine-grained prompt alignment, including objects, attributes, and spatial relationships?
  • For multi-image and video reference-to-video generation, could the authors provide quantitative evaluations of generation quality, reference consistency?
Reviewer_DE13OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OmniTTT, a linear-complexity architecture for multimodal visual generation based on Test-Time Training (TTT). The method is motivated by the observation that language representations are sensitive to compression while visual tokens dominate computational costs. To address this asymmetry, OmniTTT retains explicit attention for language conditioning and local visual interactions while using an online-updated TTT inner model to represent global visual context.

The proposed architecture enables the conversion of pretrained full-attention models through lightweight continual training. Experiments across text-to-image, text-to-video, and reference-to-video generation demonstrate substantial reductions in computation and inference latency while maintaining comparable generation quality on the reported evaluations. Additional ablations investigate the roles of TTT, local attention, gating mechanisms, memory capacity, and alternative attention acceleration methods.

Critical strengths

S1. Substantial and practically relevant computational improvements.

OmniTTT achieves significant reductions in FLOPs and inference latency across multiple visual generation tasks, including a 7.30× full-model speedup for 4K text-to-image generation and a 2.87× speedup for video reference-to-video generation. These results provide meaningful evidence of its practical efficiency benefits.

S2. Compatibility with pretrained models and reasonably comprehensive architectural comparisons.

The method supports lightweight conversion of existing pretrained diffusion Transformers rather than requiring training from scratch. The paper also evaluates several architectural components and compares against alternative linear-complexity and sparse-attention methods, providing useful evidence for the effectiveness of its proposed design.

Critical weaknesses

W1. Insufficient quantitative support for the central modality-asymmetry motivation.

The paper argues that language tokens are difficult to compress without losing fine-grained semantic information, whereas visual representations can be compressed more effectively. However, this central observation is primarily supported by qualitative examples in Figure 1. Quantitative evaluations of instruction following and compositional semantic preservation would be necessary to substantiate this claim and distinguish modality-specific limitations from optimization effects.

W2. Limited evaluation of reference fidelity in multimodal generation.

Although OmniTTT demonstrates substantial efficiency improvements on reference-to-video generation, the corresponding quality assessment relies mainly on qualitative examples. Since the method compresses global visual context, it is important to evaluate whether reference-specific details and relationships are preserved. Without quantitative reference-fidelity evaluations, the claim of maintaining comparable generation quality remains insufficiently supported for this application.

W3. Unclear contribution and capacity limitations of the TTT-based global memory.

The ablation studies demonstrate the effectiveness of combining TTT with local attention and show improvements over alternative linear-complexity mechanisms. However, they do not fully explain the specific benefits of online parameter updates or how compressed memory preserves diverse visual information as context length increases. This limits the conclusions that can be drawn about the representational advantages and long-context behavior of the proposed method.

W4. Insufficient separation of architectural and optimization effects.

The preliminary study shows that vanilla visual TTT fails under a limited continual-training budget, while adding local attention enables effective conversion. However, it remains unclear whether this reflects an inherent limitation of vanilla TTT or a difficulty in adapting pretrained attention representations under the chosen training budget. Further analysis would strengthen the interpretation of the proposed architectural design.

Questions for authors

Q1. Is the modality-specific compression hypothesis quantitatively supported?

The paper motivates OmniTTT through the observation that language tokens are more sensitive to compression than visual tokens. However, this claim is primarily supported by qualitative examples in Figure 1. Could the authors provide quantitative evaluations of language compression, particularly for compositional prompts involving objects, attributes, and spatial relationships? Such results would help determine whether the observed degradation reflects a fundamental modality asymmetry or limitations of the particular TTT conversion setting.

Q2. Does OmniTTT preserve reference-conditioned generation fidelity?

OmniTTT demonstrates substantial computational improvements for reference-to-video generation, but the corresponding quality evaluation relies primarily on qualitative comparisons. Since the method compresses global visual context, could the authors provide quantitative evaluations of reference fidelity, such as subject identity, appearance consistency, or multi-reference adherence? This would help establish whether the reported efficiency gains preserve the capabilities most relevant to reference-conditioned generation.

Q3. What is the specific contribution of TTT-based global memory?

Table 6 demonstrates the complementary roles of TTT and local attention, while Table 7 compares TTT against alternative linear-complexity mechanisms. However, the specific benefit of the online-updated inner model remains unclear. Could the authors further clarify or experimentally isolate the contribution of test-time parameter updates relative to alternative fixed-capacity global representations under comparable computational and training budgets? In particular, how does memory capacity affect the preservation of diverse visual information over increasingly long contexts?

Q4. Is the failure of vanilla visual TTT primarily an optimization limitation?

The preliminary study suggests that vanilla MLP-based visual TTT fails to converge under lightweight continual training, whereas adding local attention enables successful adaptation. Is this failure primarily due to limited optimization budget, insufficient representational capacity, or the removal of explicit local interactions? Training curves or comparisons under larger conversion budgets would help clarify whether local attention is essential for visual modeling or mainly facilitates conversion from pretrained full-attention models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.