- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates contextual tokens in multimodal diffusion models by training a Contextual Reader that interfaces with a frozen LLM. It finds that semantic information about the final generated image can be decoded from contextual tokens early in generation, and uses this finding to improve generative model training.
Critical strengths
- The paper finds that early contextual tokens in multimodal diffusion models contain not only information from the prompt but also additional semantic information about the final generated image. The proposed natural-language interface offers a useful tool for better understanding the internal mechanisms of diffusion models.
- Building on this finding, the paper introduces CoAl into generative model training. Adding CoAl reduces the FID of REPA, HASTE, and SRA without introducing additional modules at inference time.
Critical weaknesses
The connection between the findings about contextual tokens and the proposed training method could be more firmly established. The paper observes a correlation between semantic readability and generation quality and uses this observation to motivate alignment supervision. However, it remains unclear whether the improvements arise specifically from strengthening contextual representations or from the more general benefit of additional image-semantic supervision.
Questions for authors
- Could aligning contextual tokens with the SigLIP representations of ground-truth images reduce generation diversity under the same prompt?
- Related to the weakness above, could the authors clarify whether the improvements arise specifically from strengthening contextual representations or from additional image-semantic supervision more generally?
2 comments
Thank you to the authors for their detailed responses and additional analyses regarding W1 and Q2. These clarifications and experiments have effectively addressed my concerns and further strengthened my appreciation of this work.
Regarding Q1, I still have a minor concern: the current experiments do not clearly distinguish whether the reduction in generation diversity results from improved instruction following, which reduces generations that fail to satisfy the prompt, or from the suppression of valid variations in aspects not explicitly constrained by the prompt, such as composition, viewpoint, and appearance. Although the improvement in CLIP similarity suggests better semantic alignment, it does not entirely rule out the possibility of reduced meaningful diversity.
Nevertheless, this concern does not affect my overall positive assessment of the paper's contributions and quality. If time permits, I encourage the authors to further investigate whether CoAl still reduces diversity when the generated images equally satisfy the text prompts, which would help clarify the actual impact of semantic alignment on generation diversity.
Thank you again for the authors' thoughtful and thorough responses.
Response to reviewer JEQY
Our original analysis links VQA readability to human preference, while CoAl supervises contextual tokens using image embeddings. To connect the two, we train a QFormer to predict the final image's SigLIP embedding, using the same reader training and evaluation protocol without an LLM or VQA objective. We train this SigLIP reader on both SD3.5 and FLUX.2 (conditional and unconditional settings). In this setting, better embedding predictions identify generations with higher HPS across both models and conditioning settings. These SigLIP reader scores also correlate with VQA readability (Pearson 0.419 to 0.522). This supports supervising the global image semantics that contextual tokens implicitly encode. See Appendix B.1.4, Figure 19.
Our original supplementary material addresses whether the gains depend on introducing another teacher. In Ablation Studies (Appendix B.2.1, Table 2, “Semantic Teacher: DINOv2”), we use the same DINOv2 encoder for both REPA and CoAl, as specified in Appendix A.2.3. Compared with REPA alone (Table 3), adding CoAl still improves FID from 4.95 to 4.67 and recall from 0.338 to 0.392. These gains do not require a different teacher and support the benefit of reinforcing contextual tokens alongside visual representations.
We evaluate this directly in the added Diversity subsection (Appendix B.2.7, Table 9), computing Vendi scores over eight images per prompt for 500 prompts. CoAl increases same-prompt diversity by 0.5% to 2.1% when fine-tuning SD3, while reducing it by 0.9% to 3.4% when training from scratch. Figure 23 shows that the diversity of the REPA and HASTE baselines also decreases as training improves prompt alignment, suggesting that some reduced variation may reflect better adherence rather than lost meaningful diversity. Moreover, we believe distributional diversity is the more fitting notion when training from scratch, and CoAl improves it: recall on MS-COCO increases for all four baselines (by 10.6% to 20.5%, Table 3), and on Fine-T2I the highest recall is also obtained with CoAl (Table 4), indicating better coverage of the modes of the target distribution.