- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper studies unified latent diffusion, an architecture in which a single model generates both text and images by continuous latent denoising. Images are represented with frozen SigLIP2 encoder features. In the latent diffusion variant, text is represented with frozen T5 encoder features. The paper presents an empirical study. Three models with are trained from scratch with the same data and a data budget. The vision side is fixed, and the text modeling is varied: autoregressive, masked discrete diffusion, or continuous latent diffusion. The authors report that the continuous variant improves image generation and vision-centric VQA. The autoregressive variant achieves the best generative perplexity and CLIPScore results. Additional analyses study the impact of the number ofdenoising steps and the choice of text latent space.
Critical strengths
-
The paper addresses a relevant question for unified multimodal models: how the choice of language modeling affects joint training across modalities.
-
The authors run an extensive set of experiments. Three models are trained from scratch under a controlled setup (same backbone, data, and token budget) and evaluated on language generation, visual understanding benchmarks.
Critical weaknesses
-
Confounds in the experimental protocol: My main concern is that the comparison does not fully support the central claim. Only the latent diffusion variant uses a frozen pretrained text encoder. Its image generator is therefore conditioned on stronger caption features, which could explain the visual gains independently of the generative formulation. The latent diffusion variant is also substantially weaker at language generation, and its visual gains may result from the shared backbone allocating less capacity to text. The claim that unified latent diffusion improves joint training requires a comparison of the variants at matched language quality, which the paper does not provide.
-
A second concern is the novelty of the approach. Unified diffusion models for text and images already exist. The paper builds on established architectures so its contribution rests on the empirical comparison. The paper defines unification as continuous latent denoising in both modalities, but it does not identify which aspect of this formulation produces the reported gains. The discrete diffusion variant also denoises text, yet it gives the weakest image generation of the three variants.
-
The results may reflect each unimodal modeling approach's strengths rather than multimodal effects. All variants use continuous diffusion for images and differ only in text modeling; the gains may therefore reflect continuous diffusion’s suitability for images and autoregressive modeling strength for text. The paper does not establish whether the differences arise from multimodal training especially that autoregressive approach resulted in the best text modeling performance.
-
Missing related work. The paper cites the main recent unified diffusion models, but it omits several earlier and closely related works. The idea of of making joint diffusion over multiple modalities in discrete or continuous spaces, such as UniD3 [1], CoDi [2], MLD [3], Omni-Diffusion [4] and many more.
[1] Hu, M., Zheng, C., Zheng, H., Cham, T. J., Wang, C., Yang, Z., ... & Suganthan, P. N. (2022). Unified discrete diffusion for simultaneous vision-language generation. arXiv preprint arXiv:2211.14842.
[2] Tang, Z., Yang, Z., Zhu, C., Zeng, M., & Bansal, M. (2023). Any-to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 36, 16083-16099.
[3] Bounoua, M., Franzese, G., & Michiardi, P. (2024). Multi-modal latent diffusion. Entropy, 26(4), 320.
[4] Li, L., Long, Z., Shen, Y., Gao, H., Cao, H., Sun, X., ... & Fu, C. (2026). Omni-diffusion: Unified multimodal understanding and generation with masked discrete diffusion. arXiv preprint arXiv:2603.06577.
[5] Bill, E. T., Simsar, E., Tonioni, A., & Hofmann, T. (2026). FullFlow: Upgrading Text-to-Image Flow Matching Models for Bidirectional Vision--Language Generation. arXiv preprint arXiv:2605.20316.
Questions for authors
-
The appendix distinguishes this work from earlier continuous models by its evaluation settings and by its decoding mechanism. The key contribution of the paper is hard to determine. Can the authors state whether the main contribution is empirical or methodological, and specify which component of the method is new?
-
Related to point 3, can the authors discuss their modeling choices for each modality and their impact on the results? How can the authors explain whether the results are due to the multimodal setting or to the choice of modeling for each modality alone?
-
How do the autoregressive and discrete diffusion variants perform when given the same frozen pretrained text conditioning? Can the authors compare the three variants at matched language quality, for example by varying the language loss weight?
-
Fig. 5 reports the effect of the number of denoising steps on visual understanding only. How does the number of steps affect language generation quality/efficiency?
-
Why does discrete diffusion, which also uses a denoising objective, not yield the same benefits for visual generation?
4 comments
Author Rebuttal (2/3)
W2 & Q1 & Q5. Novelty, and which aspect of the formulation produces the gains.
We address (i) what our contribution is and (ii) which aspect of the formulation the gains can be attributed to.
(i) Contribution (Q1). Our contribution is empirical; we do not claim a new architecture or training mechanism. We adopt existing components (Transfusion-style multimodal training and ELF for language modeling) so that the comparison is not confounded by new design choices. Prior unified diffusion models, including those listed in W4, show that joint diffusion over text and images is feasible. To our knowledge, however, none of them compares language modeling formulations (autoregressive, discrete diffusion, and continuous latent diffusion) under a matched backbone, training data, and token budget. Our study fills this gap and produces findings that, to our knowledge, have not been reported before:
- Unified latent diffusion formulation eases modality competition. Our new gradient analysis (W1 (iii)) shows that unified latent diffusion eases joint optimization between vision and language. Consequently, unified latent diffusion reaches a lower visual diffusion loss at every training budget and exceeds the AR baseline's 30B-token DPG score after 20B tokens.
- Iterative denoising enables test-time scaling of text generation. Increasing the number of denoising steps improves overall visual understanding and language generation. Compute scales without lengthening the answer, unlike autoregressive test-time scaling, which typically relies on generating longer outputs (e.g., reasoning chains).
- Language latent spaces behave differently from visual ones. T5 encoder give little gain over T5 vocabulary embeddings (25.3 vs. 24.8) in visual understanding benchmarks. In visual diffusion, by contrast, semantic latents give large gains.
(ii) Which aspect produces the gains. We agree that the original writing did not make this clear. We isolate it by elimination:
- Pretrained T5 encoder (W1 (i)). Replacing the T5 encoder with vocabulary embeddings retains 89% of the DPG gain over AR at 15B tokens, ruling out this explanation.
- Reduced language capacity (W1 (ii)). This explanation is not supported: ULD keeps its visual advantage at near-matched language quality, and weaker language modeling does not predict stronger visual generation, as discrete diffusion shows with both the weakest language generation and the weakest visual generation.
- Why does discrete diffusion not yield the gain? (Q5) Although both use denoising, discrete diffusion predicts categorical tokens using cross-entropy, like AR, whereas continuous latent diffusion uses flow matching in continuous latent space. This difference is reflected in cross-task affinity: text updates interfere with visual learning under discrete diffusion (−0.30 × 10⁻⁴) but benefit it under ULD (3.48 × 10⁻⁴), consistent with their visual generation performance (see our common comment, "Gradient-based Modality Competition Analysis").
Overall, these results suggest that the gains are associated with formulating text generation as continuous flow matching, the same formulation used for vision. We have incorporated the corresponding analyses and discussion into the revised paper (Sec. 5.1, with supporting details in Appx. D and E.2).
W3 & Q2. Multimodal effects vs. per-modality modeling strengths
We agree that per-modality strengths exist; AR gives the best text-only generation. However, per-modality strengths alone would predict two outcomes that we do not observe:
- Visual generation should be similar across variants. Vision is fixed by design: the formulation, latents, data, and sampler are identical. “Continuous diffusion’s suitability for images” is therefore constant. Yet visual generation differs substantially across variants (Tab. 1, Fig. 3). These differences can only come from the text formulation that shares the backbone, i.e., from multimodal training. W1 (i) rules out the only remaining per-modality path, the caption features.
- The text-only ranking should carry over to image-conditioned text generation. It does not. ULD outperforms both baselines on detailed captioning and outperforms AR on three of the four VQA categories. Even discrete diffusion, the weakest text-only model, beats AR on General, Knowledge, and Vision-Centric (Tab. 1).
Our gradient analysis (W2 (ii), comment “Gradient-based Modality Competition Analysis”) measures this cross-modal effect directly. Text updates help the visual objective under ULD but hurt it under discrete diffusion, consistent with their visual generation results. Per-modality strengths cannot account for this.
Thus, how the unimodal strengths of a formulation manifest in multimodal setting cannot be inferred from unimodal results alone. It depends on how the objectives interact or how the task is changed (text generation vs. image conditioned text generation).
Author Rebuttal (3/3)
Modeling choices for each modality (Q2).
- Vision. Visual generation methods have largely converged on continuous diffusion as the way to scale [c, d], and recent UMMs adopt it as well [e, f, g]. We therefore keep it fixed across all variants.
- Language. Language modeling has not converged in the same way. AR is still the dominant approach, but diffusion language models are being actively studied and scaled [h, i]. Continuous latent diffusion in particular has recently shown promising results [b, j, k], yet it has rarely been studied in multimodal models. We therefore treat language as the variable to study.
Comparing fundamentally different formulations in a perfectly fair way is difficult. We therefore follow the standard recipe for each formulation: next-token prediction for AR, and MDLM [l] for discrete diffusion. We also note that the shared setup, including the backbone architecture (Qwen2.5) and the optimizer settings, follows standard practice for AR language models (Sec. 4.1). Continuous latent diffusion for language does not have a standard recipe yet, so we adopt ELF [k], a recent method that has shown effective generation in high-dimensional latent spaces.
[c] Esser et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. ICML, 2024.
[d] Black Forest Labs et al. FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space. arXiv, 2025.
[e] Zhou et al. Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model. ICLR, 2025.
[f] Deng et al. Emerging Properties in Unified Multimodal Pretraining. arXiv, 2025.
[g] Tong et al. Beyond Language Modeling: An Exploration of Multimodal Pretraining. arXiv, 2026.
[h] Nie et al. Large Language Diffusion Models. NeurIPS, 2025.
[i] Bie et al. LLaDA2.0: Scaling Up Diffusion Language Models to 100B. arXiv, 2025.
[j] Guo et al. Continuous Latent Diffusion Language Model. arXiv, 2026.
[k] Hu et al. ELF: Embedded Language Flows. arXiv, 2026.
[l] Sahoo et al. Simple and Effective Masked Diffusion Language Models. NeurIPS, 2024.
W4. Missing Related Work
We agree that these studies provide important precedents for multimodal diffusion. We have expanded Appendix A of the revised manuscript to include all five references and clarify their methodological distinctions, as summarized below.
- UniD3 [1]:
Unifies discrete diffusion for vision–language generation through a shared transition matrix and cross-modal attention. - CoDi [2]:
Enables any-to-any generation by composing aligned, modality-specific diffusion models, including synchronized multimodal generation. - MLD [3]:
Models concatenated latents from modality-specific autoencoders using multi-time diffusion for joint and conditional generation. - Omni-Diffusion [4]:
Uses unified masked discrete diffusion to model text, image, and speech tokens for any-to-any generation. - FullFlow [5]:
Extends pretrained text-to-image flow models to bidirectional vision–language generation through lightweight adapters combining continuous image flow and discrete text insertion.
These additions clarify that joint multimodal diffusion itself is not our claimed novelty. Rather, our contribution is a controlled empirical comparison of AR, discrete diffusion, and continuous latent diffusion language modeling within a shared multimodal framework, examining their effects on joint training, visual understanding, and visual generation.
Q4. Effect of Denoising Steps on Language Generation
We additionally evaluated language generation quality across different numbers of denoising steps, reporting generation perplexity (PPL) below.
| N (step) | Gen PPL |
|---|---|
| 16 | 51.5 |
| 32 | 35.7 |
| 64 | 24.3 |
| 128 | 18.6 |
| 256 | 15.0 |
The results show that increasing the number of denoising steps consistently improves language generation quality, reducing PPL from 51.5 at 16 steps to 15.0 at 256 steps, approaching the autoregressive baseline (14.68). Importantly, the performance gap at 64 steps largely reflects the limited inference budget rather than an inherent limitation of latent diffusion in language generation.
I thank the authors for their detailed response and the considerable effort put into the additional experiments. The ablation replacing the T5 encoder with vocabulary embeddings addresses my main concern, as most of the visual gains remain without the pretrained encoder. The clarification that the contribution is an empirical study, together with the added related work, also resolves my concern about the novelty and positioning of the paper . Some points remain, such as the comparison at matched language quality and the length of the generated captions in the new captioning evaluation, but I appreciate the authors' efforts and I am raising my score to 3.
Author Rebuttal (1/3)
W1 & Q3. Pretrained Text Encoder and Language Capacity
We thank the reviewer for this important concern.
W1 raises two alternative explanations for the visual gains: (i) stronger caption features from the frozen T5 encoder, and (ii) weaker language modeling freeing backbone capacity for vision. We address each below, and then (iii) provide direct evidence for the central claim of eased joint training.
(i) Pretrained T5 encoder (Q3, first part). We retrained Unified Latent Diffusion (ULD) with vocabulary embeddings in place of the T5 encoder (see our comment “Unified Latent Diffusion without the T5 Encoder”). At 15B tokens, DPG drops only from 64.6 to 63.3. This retains 89% of the gain over AR (53.1), so the visual gains do not stem from T5’s pretrained features.
We did not run experiments adding T5 to the baselines for following reasons
(ii) Language capacity and matched language quality (Q3, second part). We agree that a language-loss-weight sweep would be a direct test. However, it requires retraining several 1.3B models from scratch, which exceeds our compute budget during the rebuttal period. The following observations from existing models address the matched-quality question instead.
(iii) Direct Comparison in Modality competition. Our gradient-based analysis provides additional evidence of eased joint optimization (see our common comment “Gradient-based Modality Competition Analysis”).
Taken together, these results indicate that the visual gains are not explained by the T5 encoder or by reduced language capacity. We will add these results to the revised paper.
[a] Gulrajani et al. Likelihood-Based Diffusion Language Models. NeurIPS, 2023.
[b] Yang et al. Continuous Diffusion Scales Competitively with Discrete Diffusion for Language. NeurIPS, 2026.