acceptodds
Under review as a conference paper at ICLR 2027

Learning to Read the Contextual Tokens in Diffusion Transformers

Abstract

Multimodal Diffusion Transformers (MM-DiTs) jointly process visual and textual representations throughout generation. These models repeatedly update the text tokens through multimodal attention, forming dynamic contextual tokens whose function is not well understood. In this work, we introduce a framework for reading this contextual space through natural-language interrogation. We train a lightweight bottleneck network that maps intermediate contextual tokens into the input space of a frozen Large Language Model (LLM), allowing the LLM to answer questions about the emerging image directly from these hidden representations. Our reader reveals that contextual tokens encode a rich, global representation of the emerging scene: generation-specific semantics, including attributes left underspecified by the prompt, are accessible surprisingly early in denoising, while increasingly fine-grained details become readable over time. Remarkably, this information remains decodable even when the MM-DiT receives an empty prompt, showing that contextual tokens accumulate substantial image-specific information from the evolving visual representation itself. We further find that generations with more readable contextual representations tend to receive higher human-preference scores. Building on these observations, we introduce Contextual Alignment, a training technique that explicitly reinforces the visual-semantic information encoded in the contextual tokens, improving generation quality and distributional coverage. Together, our results establish contextual tokens as both an interpretable view into the internal dynamics of MM-DiTs and an effective target for improving generative models.

Primary area:
generative models
Keywords:
Image Generation, Multimodal Diffusion Transformers, Representation Alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JEQYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates contextual tokens in multimodal diffusion models by training a Contextual Reader that interfaces with a frozen LLM. It finds that semantic information about the final generated image can be decoded from contextual tokens early in generation, and uses this finding to improve generative model training.

Critical strengths
  1. The paper finds that early contextual tokens in multimodal diffusion models contain not only information from the prompt but also additional semantic information about the final generated image. The proposed natural-language interface offers a useful tool for better understanding the internal mechanisms of diffusion models.
  2. Building on this finding, the paper introduces CoAl into generative model training. Adding CoAl reduces the FID of REPA, HASTE, and SRA without introducing additional modules at inference time.
Critical weaknesses

The connection between the findings about contextual tokens and the proposed training method could be more firmly established. The paper observes a correlation between semantic readability and generation quality and uses this observation to motivate alignment supervision. However, it remains unclear whether the improvements arise specifically from strengthening contextual representations or from the more general benefit of additional image-semantic supervision.

Questions for authors
  1. Could aligning contextual tokens with the SigLIP representations of ground-truth images reduce generation diversity under the same prompt?
  2. Related to the weakness above, could the authors clarify whether the improvements arise specifically from strengthening contextual representations or from additional image-semantic supervision more generally?
2 comments
AuthorsOpenReview ↗

Response to reviewer JEQY

W1: The connection between the findings about contextual tokens and the proposed training method could be more firmly established.

Our original analysis links VQA readability to human preference, while CoAl supervises contextual tokens using image embeddings. To connect the two, we train a QFormer to predict the final image's SigLIP embedding, using the same reader training and evaluation protocol without an LLM or VQA objective. We train this SigLIP reader on both SD3.5 and FLUX.2 (conditional and unconditional settings). In this setting, better embedding predictions identify generations with higher HPS across both models and conditioning settings. These SigLIP reader scores also correlate with VQA readability (Pearson 0.419 to 0.522). This supports supervising the global image semantics that contextual tokens implicitly encode. See Appendix B.1.4, Figure 19.

Q2: Do the improvements come from strengthening contextual representations or from additional image-semantic supervision more generally?

Our original supplementary material addresses whether the gains depend on introducing another teacher. In Ablation Studies (Appendix B.2.1, Table 2, “Semantic Teacher: DINOv2”), we use the same DINOv2 encoder for both REPA and CoAl, as specified in Appendix A.2.3. Compared with REPA alone (Table 3), adding CoAl still improves FID from 4.95 to 4.67 and recall from 0.338 to 0.392. These gains do not require a different teacher and support the benefit of reinforcing contextual tokens alongside visual representations.

Q1: Could aligning contextual tokens with SigLIP reduce generation diversity under the same prompt?

We evaluate this directly in the added Diversity subsection (Appendix B.2.7, Table 9), computing Vendi scores over eight images per prompt for 500 prompts. CoAl increases same-prompt diversity by 0.5% to 2.1% when fine-tuning SD3, while reducing it by 0.9% to 3.4% when training from scratch. Figure 23 shows that the diversity of the REPA and HASTE baselines also decreases as training improves prompt alignment, suggesting that some reduced variation may reflect better adherence rather than lost meaningful diversity. Moreover, we believe distributional diversity is the more fitting notion when training from scratch, and CoAl improves it: recall on MS-COCO increases for all four baselines (by 10.6% to 20.5%, Table 3), and on Fine-T2I the highest recall is also obtained with CoAl (Table 4), indicating better coverage of the modes of the target distribution.

Thank you to the authors for their detailed responses and additional analyses regarding W1 and Q2. These clarifications and experiments have effectively addressed my concerns and further strengthened my appreciation of this work.

Regarding Q1, I still have a minor concern: the current experiments do not clearly distinguish whether the reduction in generation diversity results from improved instruction following, which reduces generations that fail to satisfy the prompt, or from the suppression of valid variations in aspects not explicitly constrained by the prompt, such as composition, viewpoint, and appearance. Although the improvement in CLIP similarity suggests better semantic alignment, it does not entirely rule out the possibility of reduced meaningful diversity.

Nevertheless, this concern does not affect my overall positive assessment of the paper's contributions and quality. If time permits, I encourage the authors to further investigate whether CoAl still reduces diversity when the generated images equally satisfy the text prompts, which would help clarify the actual impact of semantic alignment on generation diversity.

Thank you again for the authors' thoughtful and thorough responses.

Reviewer_q8ckOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims to understand the text tokens inside MM-DiT models. A lightweight Q-former style bottlenekck network is trained to read these “contextual tokens”, with a frozen LLM. The LLM gets capable of answering questions based on these contextual tokens. The authors find that these tokens encode rich global representations of the scene in generation. Using this readable representation, the authors find that generations which have more readable contextual representations tend to generate samples with better quality (better human preferences). Based on this observation, the authors introduce Contextual Alignment, which aligns the global semantic representation from a pre-trained encoder and a global representation encoded from the contextual tokens during the generation process. This alignment skill improves generation quality with better text-image alignment.

Critical strengths
  • The problem definition is clear, and the approach is reasonably designed, being simple yet effective, with extensive experimental results.
  • The Contextual Reader and Contextual Alignment method seems to have a potential to be adopted in future studies.
Critical weaknesses
  • Although I deeply acknowledge the paper’s study on the text stream of MM-DiT, I must say that there has been several efforts on trying to understand / make use of the text steam of MM-DiTs [1,2,3,4]. I think these work should have been discussed, and comparisons would have been necessary. These papers mostly focus on solving the problems commonly occurring in text-to-image models, such as attribute binding, object omission, object relationships, etc. I think this paper could be considered a more comprehensive approach than these baselines, and worth a comparative study on diverse aspects, rather than a simple generation performance comparison.

[1] Lv et al., Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers, ICCV 2025

[2] Rare Text Semantics Were Always There in Your Diffusion Transformer, NeurIPS 2025

[3] Baek et al., Diagnosing and Correcting Concept Omission in Multimodal Diffusion Transformers, ICML 2026

[4] Li et al. "Unraveling mmdit blocks: Training-free analysis and enhancement of text-conditioned diffusion." arXiv 2601.02211

Questions for authors
  • On from-scratch (full model training) experiments on CoAl, the results are used on top of existing visual alignment methods, like REPA vs REPA + CoAl. Although its standalone effectiveness has been shown in the fine-tuning experiment, I wonder how much it would help when used alone in the from scratch setting too. Existing visual alignment methods are better known to provide accelerated training; this suggestion is to see if CoAl also has such property, how fast it is compared to other approaches, or if it just provides a better optima rather than accelerated training.
2 comments
AuthorsOpenReview ↗

Response to reviewer q8ck

W1: Several related methods that study or manipulate the text stream of MM-DiTs are missing, and comparisons with them are needed.

We thank the reviewer for identifying these relevant works. Their objective differs from ours: TACA, ToRA, OSI, and TexTailor improve prompt adherence post hoc, after the model weights have already been learned, whereas CoAl adapts the model’s generative distribution to a target dataset during training or fine-tuning.

Nevertheless, we now discuss these methods in the revised Related Work section. We further compare against all four methods in the Appendix subsection Comparison with Training-Free Text-Stream Methods (Appendix B.2.6, Tables 7 and 8). On PartiPrompts, where we evaluate post-hoc prompt adherence, these methods, particularly TexTailor, provide stronger improvements than CoAl. On MS-COCO, where the objective is to learn the target data distribution, CoAl provides substantially better distribution quality and coverage. Finally, applying the methods together retains their respective benefits and improves upon the original baselines. These experiments establish that the approaches address different objectives and are complementary rather than competing alternatives.

Q1: How effective is CoAl by itself when training from scratch? Does it accelerate training, or does it only reach a better optimum?

The original supplementary material reports CoAl without any visual alignment method: adding CoAl to Vanilla training improves FID from 6.52 to 6.23 (Section B.2.2, Table 3). In the revision, we added the Training Speedup subsection (Appendix B.2.4, Table 5 and Figure 21).

We measure speedup as the number of steps a baseline needs to reach its best FID divided by the number of steps CoAl needs to reach that same FID. Under this definition, CoAl provides estimated speedups across all four configurations: 1.25× for Vanilla, 1.59× for REPA, 1.04× for HASTE, and 1.40× for SRA. It then continues improving with additional training, ultimately achieving lower FID than each corresponding baseline at the same 150k training steps (Table 3). Thus, CoAl both reaches the baseline performance faster and converges to a better final solution.

I thank the authors for their rebuttal. I have read the authors' response, and the updates made to the paper. I am satisfied and will maintain my rating.

Reviewer_ZY7YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In MMDiTs, text tokens got updated to form contextual tokens. This paper tries to understand the functionality and dynamics of these contextual tokens. The paper first trains a contextual token reader which maps the contextual tokens to tokens in the LLM input token space, and have the LLM answer questions directly from contextual tokens. The paper finds that contextual token can be decoded very early in the denoising phase, and when the text prompt is empty. The paper then finds the more readable contextual tokens lead to better preferred generations by human. Then based on these observation, the paper enforces visual semantics in contextual tokens to allow better training of MM-DiTs

Critical strengths

(+) the probing of contextual token reveals interesting results on them, including early denoising process contextual tokens still reveal rich information, contextual tokens without text prompt as input are still meaningful. And better contextual tokens lead to better images. This reveals fascinating insights that the community can benefit from.

(+) The paper then presents a training method to align contextual token to allow MM-DiTs to train better. This is well motivated

(+) the proposed method is complementary to representation learning like repa, shown in table 1

(+) the paper is relatively easy to follow, and experiments are relatively thorough

Critical weaknesses

(-) Limited architectural generality. The entire paper only applies to MM-DiT where the text token got updated to form contextual tokens

(-) all visual results shown in the paper are not very impressive (yes, the proposed method looks better than baselines, but none of them look like a SotA image generation model can generate) and far behind state-of-the-art image generation models, and are rather toy-ish examples. It is a bit unclear is the gain will transfer to SotA models

Questions for authors

n/a

1 comment
AuthorsOpenReview ↗

Response to reviewer ZY7Y

W1: Limited architectural generality. The paper only applies to MM-DiTs in which text tokens are updated into contextual tokens.

We agree that our framework is specific to MM-DiTs. However, MM-DiTs have become a dominant architecture for modern image and video generation, adopted almost unanimously by modern models.

Despite this broad adoption, the evolving contextual representations created by these architectures remain considerably less understood than their visual representations.

W2: The visual results appear toy-like, and it is unclear whether the gains transfer to state-of-the-art models.

We thank the reviewer for raising this concern. First, we replaced the qualitative figures with more appealing examples: the main qualitative comparison (Figure 7) and the additional comparisons in the appendix (Figures 24 and 25) now show more and more varied prompts, and the Contextual Reader figures (Figures 3 and 4) include additional examples.

Second, to test whether CoAl transfers to state-of-the-art models, we apply it to FLUX.2-klein-base-9B and compare Vanilla, REPA, HASTE, and SRA with and without CoAl in the new Appendix subsection Fine-Tuning FLUX.2 (Appendix B.2.5, Table 6 and Figure 22). CoAl improves FID for all four configurations, and the qualitative examples show corrected object counts, more coherent high-level structure, and fewer visual artifacts, similar to the advantages we show for SD3. We transferred CoAl to FLUX.2 with the same optimization settings and loss weights as in the SD3 experiments, applying it at an intermediate block chosen without tuning. The improvements therefore require no architecture-specific tuning, and dedicated hyperparameter selection may further improve them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you for your feedback. Manuscript revisions are colored blue.

We thank all reviewers for their thoughtful feedback, and we are glad that you found our paper to be a positive contribution. Criticisms are valid and well thought out, and we are confident that addressing them will improve our paper. Below, we address each reviewer's concerns separately. These responses are accompanied by changes in the manuscript, which are colored blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.