acceptodds
Under review as a conference paper at ICLR 2027

Exploring Unified Latent Diffusion for Multimodal Modeling

Abstract

Unified multimodal models commonly combine autoregressive language generation with continuous image diffusion, requiring a shared model to learn two different generative processes. We study Unified Latent Diffusion, which generates both language and images through continuous latent denoising, and ask how this choice affects joint multimodal learning. Through controlled pretraining from scratch, we compare autoregressive, discrete diffusion, and latent diffusion language modeling with matched backbone architectures, training data, and token budgets, while keeping visual modeling fixed. We identify three findings: (i) Unifying generative modeling through latent diffusion eases joint training, surpassing the autoregressive baseline's visual generation (DPG-Bench) with one-third fewer training tokens. (ii) Iterative denoising refines answers and strengthens visual understanding (Vision-Centric: 33.76 vs. 20.84 for AR at 30B training tokens). (iii) Effective language latents for diffusion modeling require different properties from vision. These comparisons examine the capabilities and limitations of unified latent diffusion as a generative formulation for general-purpose UMMs.

Primary area:
generative models
Keywords:
Latent Diffusion Model, Unified Multimodal Model, Continuous Diffusion Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_W7mgOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies unified latent diffusion, an architecture in which a single model generates both text and images by continuous latent denoising. Images are represented with frozen SigLIP2 encoder features. In the latent diffusion variant, text is represented with frozen T5 encoder features. The paper presents an empirical study. Three models with are trained from scratch with the same data and a data budget. The vision side is fixed, and the text modeling is varied: autoregressive, masked discrete diffusion, or continuous latent diffusion. The authors report that the continuous variant improves image generation and vision-centric VQA. The autoregressive variant achieves the best generative perplexity and CLIPScore results. Additional analyses study the impact of the number ofdenoising steps and the choice of text latent space.

Critical strengths
  1. The paper addresses a relevant question for unified multimodal models: how the choice of language modeling affects joint training across modalities.

  2. The authors run an extensive set of experiments. Three models are trained from scratch under a controlled setup (same backbone, data, and token budget) and evaluated on language generation, visual understanding benchmarks.

Critical weaknesses
  1. Confounds in the experimental protocol: My main concern is that the comparison does not fully support the central claim. Only the latent diffusion variant uses a frozen pretrained text encoder. Its image generator is therefore conditioned on stronger caption features, which could explain the visual gains independently of the generative formulation. The latent diffusion variant is also substantially weaker at language generation, and its visual gains may result from the shared backbone allocating less capacity to text. The claim that unified latent diffusion improves joint training requires a comparison of the variants at matched language quality, which the paper does not provide.

  2. A second concern is the novelty of the approach. Unified diffusion models for text and images already exist. The paper builds on established architectures so its contribution rests on the empirical comparison. The paper defines unification as continuous latent denoising in both modalities, but it does not identify which aspect of this formulation produces the reported gains. The discrete diffusion variant also denoises text, yet it gives the weakest image generation of the three variants.

  3. The results may reflect each unimodal modeling approach's strengths rather than multimodal effects. All variants use continuous diffusion for images and differ only in text modeling; the gains may therefore reflect continuous diffusion’s suitability for images and autoregressive modeling strength for text. The paper does not establish whether the differences arise from multimodal training especially that autoregressive approach resulted in the best text modeling performance.

  4. Missing related work. The paper cites the main recent unified diffusion models, but it omits several earlier and closely related works. The idea of of making joint diffusion over multiple modalities in discrete or continuous spaces, such as UniD3 [1], CoDi [2], MLD [3], Omni-Diffusion [4] and many more.

[1] Hu, M., Zheng, C., Zheng, H., Cham, T. J., Wang, C., Yang, Z., ... & Suganthan, P. N. (2022). Unified discrete diffusion for simultaneous vision-language generation. arXiv preprint arXiv:2211.14842.

[2] Tang, Z., Yang, Z., Zhu, C., Zeng, M., & Bansal, M. (2023). Any-to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 36, 16083-16099.

[3] Bounoua, M., Franzese, G., & Michiardi, P. (2024). Multi-modal latent diffusion. Entropy, 26(4), 320.

[4] Li, L., Long, Z., Shen, Y., Gao, H., Cao, H., Sun, X., ... & Fu, C. (2026). Omni-diffusion: Unified multimodal understanding and generation with masked discrete diffusion. arXiv preprint arXiv:2603.06577.

[5] Bill, E. T., Simsar, E., Tonioni, A., & Hofmann, T. (2026). FullFlow: Upgrading Text-to-Image Flow Matching Models for Bidirectional Vision--Language Generation. arXiv preprint arXiv:2605.20316.

Questions for authors
  1. The appendix distinguishes this work from earlier continuous models by its evaluation settings and by its decoding mechanism. The key contribution of the paper is hard to determine. Can the authors state whether the main contribution is empirical or methodological, and specify which component of the method is new?

  2. Related to point 3, can the authors discuss their modeling choices for each modality and their impact on the results? How can the authors explain whether the results are due to the multimodal setting or to the choice of modeling for each modality alone?

  3. How do the autoregressive and discrete diffusion variants perform when given the same frozen pretrained text conditioning? Can the authors compare the three variants at matched language quality, for example by varying the language loss weight?

  4. Fig. 5 reports the effect of the number of denoising steps on visual understanding only. How does the number of steps affect language generation quality/efficiency?

  5. Why does discrete diffusion, which also uses a denoising objective, not yield the same benefits for visual generation?

4 comments
AuthorsOpenReview ↗

Author Rebuttal (1/3)

W1 & Q3. Pretrained Text Encoder and Language Capacity

We thank the reviewer for this important concern.

W1 raises two alternative explanations for the visual gains: (i) stronger caption features from the frozen T5 encoder, and (ii) weaker language modeling freeing backbone capacity for vision. We address each below, and then (iii) provide direct evidence for the central claim of eased joint training.

(i) Pretrained T5 encoder (Q3, first part). We retrained Unified Latent Diffusion (ULD) with vocabulary embeddings in place of the T5 encoder (see our comment “Unified Latent Diffusion without the T5 Encoder”). At 15B tokens, DPG drops only from 64.6 to 63.3. This retains 89% of the gain over AR (53.1), so the visual gains do not stem from T5’s pretrained features.

We did not run experiments adding T5 to the baselines for following reasons

  • Using T5 on target text, its bidirectional features would leak future tokens in AR and masked tokens in discrete diffusion
  • Using T5 only for conditions would give the backbone two text input spaces (T5 features for conditions, token embeddings for targets). This asymmetric design is not standard for the baselines.

(ii) Language capacity and matched language quality (Q3, second part). We agree that a language-loss-weight sweep would be a direct test. However, it requires retraining several 1.3B models from scratch, which exceeds our compute budget during the rebuttal period. The following observations from existing models address the matched-quality question instead.

  • Near-matched language quality. Increasing denoising steps from 64 to 256 lowers ULD’s Gen PPL from 24.3 to 15.0, close to AR’s 14.68 (see our response to Q4). The default PPL gap reflects the sampling budget rather than reduced language capacity. Differences in training efficiency may also contribute: prior studies report that continuous diffusion can require more training compute to reach comparable language likelihoods [a, b].
  • Discrete diffusion as a counterexample. Discrete diffusion has the weakest language generation (Gen PPL 30.63). Yet it also has the weakest visual generation at both 15B and 30B tokens (DPG 48.3 and 59.19, vs. 53.1 and 65.16 for AR). If weaker language modeling freed capacity for vision, discrete diffusion should show the largest visual gain. It shows the opposite.
  • Language generation in multimodal settings. On detailed image captioning, ULD outperforms the baselines (see our common comment “Detailed Image Captioning Evaluation”). This is inconsistent with a backbone that allocates less capacity to text generation objective.

(iii) Direct Comparison in Modality competition. Our gradient-based analysis provides additional evidence of eased joint optimization (see our common comment “Gradient-based Modality Competition Analysis”).

Taken together, these results indicate that the visual gains are not explained by the T5 encoder or by reduced language capacity. We will add these results to the revised paper.

[a] Gulrajani et al. Likelihood-Based Diffusion Language Models. NeurIPS, 2023.

[b] Yang et al. Continuous Diffusion Scales Competitively with Discrete Diffusion for Language. NeurIPS, 2026.

AuthorsOpenReview ↗

Author Rebuttal (2/3)

W2 & Q1 & Q5. Novelty, and which aspect of the formulation produces the gains.

We address (i) what our contribution is and (ii) which aspect of the formulation the gains can be attributed to.

(i) Contribution (Q1). Our contribution is empirical; we do not claim a new architecture or training mechanism. We adopt existing components (Transfusion-style multimodal training and ELF for language modeling) so that the comparison is not confounded by new design choices. Prior unified diffusion models, including those listed in W4, show that joint diffusion over text and images is feasible. To our knowledge, however, none of them compares language modeling formulations (autoregressive, discrete diffusion, and continuous latent diffusion) under a matched backbone, training data, and token budget. Our study fills this gap and produces findings that, to our knowledge, have not been reported before:

  • Unified latent diffusion formulation eases modality competition. Our new gradient analysis (W1 (iii)) shows that unified latent diffusion eases joint optimization between vision and language. Consequently, unified latent diffusion reaches a lower visual diffusion loss at every training budget and exceeds the AR baseline's 30B-token DPG score after 20B tokens.
  • Iterative denoising enables test-time scaling of text generation. Increasing the number of denoising steps improves overall visual understanding and language generation. Compute scales without lengthening the answer, unlike autoregressive test-time scaling, which typically relies on generating longer outputs (e.g., reasoning chains).
  • Language latent spaces behave differently from visual ones. T5 encoder give little gain over T5 vocabulary embeddings (25.3 vs. 24.8) in visual understanding benchmarks. In visual diffusion, by contrast, semantic latents give large gains.

(ii) Which aspect produces the gains. We agree that the original writing did not make this clear. We isolate it by elimination:

  • Pretrained T5 encoder (W1 (i)). Replacing the T5 encoder with vocabulary embeddings retains 89% of the DPG gain over AR at 15B tokens, ruling out this explanation.
  • Reduced language capacity (W1 (ii)). This explanation is not supported: ULD keeps its visual advantage at near-matched language quality, and weaker language modeling does not predict stronger visual generation, as discrete diffusion shows with both the weakest language generation and the weakest visual generation.
  • Why does discrete diffusion not yield the gain? (Q5) Although both use denoising, discrete diffusion predicts categorical tokens using cross-entropy, like AR, whereas continuous latent diffusion uses flow matching in continuous latent space. This difference is reflected in cross-task affinity: text updates interfere with visual learning under discrete diffusion (−0.30 × 10⁻⁴) but benefit it under ULD (3.48 × 10⁻⁴), consistent with their visual generation performance (see our common comment, "Gradient-based Modality Competition Analysis").

Overall, these results suggest that the gains are associated with formulating text generation as continuous flow matching, the same formulation used for vision. We have incorporated the corresponding analyses and discussion into the revised paper (Sec. 5.1, with supporting details in Appx. D and E.2).

W3 & Q2. Multimodal effects vs. per-modality modeling strengths

We agree that per-modality strengths exist; AR gives the best text-only generation. However, per-modality strengths alone would predict two outcomes that we do not observe:

  • Visual generation should be similar across variants. Vision is fixed by design: the formulation, latents, data, and sampler are identical. “Continuous diffusion’s suitability for images” is therefore constant. Yet visual generation differs substantially across variants (Tab. 1, Fig. 3). These differences can only come from the text formulation that shares the backbone, i.e., from multimodal training. W1 (i) rules out the only remaining per-modality path, the caption features.
  • The text-only ranking should carry over to image-conditioned text generation. It does not. ULD outperforms both baselines on detailed captioning and outperforms AR on three of the four VQA categories. Even discrete diffusion, the weakest text-only model, beats AR on General, Knowledge, and Vision-Centric (Tab. 1).

Our gradient analysis (W2 (ii), comment “Gradient-based Modality Competition Analysis”) measures this cross-modal effect directly. Text updates help the visual objective under ULD but hurt it under discrete diffusion, consistent with their visual generation results. Per-modality strengths cannot account for this.

Thus, how the unimodal strengths of a formulation manifest in multimodal setting cannot be inferred from unimodal results alone. It depends on how the objectives interact or how the task is changed (text generation vs. image conditioned text generation).

AuthorsOpenReview ↗

Author Rebuttal (3/3)

Modeling choices for each modality (Q2).

  • Vision. Visual generation methods have largely converged on continuous diffusion as the way to scale [c, d], and recent UMMs adopt it as well [e, f, g]. We therefore keep it fixed across all variants.
  • Language. Language modeling has not converged in the same way. AR is still the dominant approach, but diffusion language models are being actively studied and scaled [h, i]. Continuous latent diffusion in particular has recently shown promising results [b, j, k], yet it has rarely been studied in multimodal models. We therefore treat language as the variable to study.

Comparing fundamentally different formulations in a perfectly fair way is difficult. We therefore follow the standard recipe for each formulation: next-token prediction for AR, and MDLM [l] for discrete diffusion. We also note that the shared setup, including the backbone architecture (Qwen2.5) and the optimizer settings, follows standard practice for AR language models (Sec. 4.1). Continuous latent diffusion for language does not have a standard recipe yet, so we adopt ELF [k], a recent method that has shown effective generation in high-dimensional latent spaces.

[c] Esser et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. ICML, 2024.

[d] Black Forest Labs et al. FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space. arXiv, 2025.

[e] Zhou et al. Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model. ICLR, 2025.

[f] Deng et al. Emerging Properties in Unified Multimodal Pretraining. arXiv, 2025.

[g] Tong et al. Beyond Language Modeling: An Exploration of Multimodal Pretraining. arXiv, 2026.

[h] Nie et al. Large Language Diffusion Models. NeurIPS, 2025.

[i] Bie et al. LLaDA2.0: Scaling Up Diffusion Language Models to 100B. arXiv, 2025.

[j] Guo et al. Continuous Latent Diffusion Language Model. arXiv, 2026.

[k] Hu et al. ELF: Embedded Language Flows. arXiv, 2026.

[l] Sahoo et al. Simple and Effective Masked Diffusion Language Models. NeurIPS, 2024.

W4. Missing Related Work

We agree that these studies provide important precedents for multimodal diffusion. We have expanded Appendix A of the revised manuscript to include all five references and clarify their methodological distinctions, as summarized below.

  • UniD3 [1]: Unifies discrete diffusion for vision–language generation through a shared transition matrix and cross-modal attention.
  • CoDi [2]: Enables any-to-any generation by composing aligned, modality-specific diffusion models, including synchronized multimodal generation.
  • MLD [3]: Models concatenated latents from modality-specific autoencoders using multi-time diffusion for joint and conditional generation.
  • Omni-Diffusion [4]: Uses unified masked discrete diffusion to model text, image, and speech tokens for any-to-any generation.
  • FullFlow [5]: Extends pretrained text-to-image flow models to bidirectional vision–language generation through lightweight adapters combining continuous image flow and discrete text insertion.

These additions clarify that joint multimodal diffusion itself is not our claimed novelty. Rather, our contribution is a controlled empirical comparison of AR, discrete diffusion, and continuous latent diffusion language modeling within a shared multimodal framework, examining their effects on joint training, visual understanding, and visual generation.

Q4. Effect of Denoising Steps on Language Generation

We additionally evaluated language generation quality across different numbers of denoising steps, reporting generation perplexity (PPL) below.

N (step)Gen PPL
1651.5
3235.7
6424.3
12818.6
25615.0

The results show that increasing the number of denoising steps consistently improves language generation quality, reducing PPL from 51.5 at 16 steps to 15.0 at 256 steps, approaching the autoregressive baseline (14.68). Importantly, the performance gap at 64 steps largely reflects the limited inference budget rather than an inherent limitation of latent diffusion in language generation.

I thank the authors for their detailed response and the considerable effort put into the additional experiments. The ablation replacing the T5 encoder with vocabulary embeddings addresses my main concern, as most of the visual gains remain without the pretrained encoder. The clarification that the contribution is an empirical study, together with the added related work, also resolves my concern about the novelty and positioning of the paper . Some points remain, such as the comparison at matched language quality and the length of the generated captions in the new captioning evaluation, but I appreciate the authors' efforts and I am raising my score to 3.

Reviewer_JwtEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates unified latent diffusion for multimodal modeling, where both language and images are generated through continuous latent denoising rather than combining autoregressive language generation with visual diffusion. The authors conduct controlled comparisons among autoregressive, discrete diffusion, and continuous latent diffusion language modeling using matched transformer backbones, training data, and token budgets. Experimental results show that unified latent diffusion improves visual generation and several visual understanding benchmarks, particularly vision-centric tasks, while underperforming autoregressive modeling in language generation and OCR-related tasks. The paper further analyzes iterative denoising and different language latent representations, providing useful insights into the benefits and limitations of unified generative modeling.

Critical strengths
  1. The experimental comparisons are relatively well controlled. The authors compare three language modeling paradigms while keeping the transformer backbone, visual modeling pipeline, training data, and token budgets consistent. This setup helps isolate the influence of language modeling choices on multimodal learning.

  2. The improvements in visual generation and vision-centric understanding are encouraging. Unified latent diffusion achieves a DPG score of 74.00 compared with 65.16 for the autoregressive baseline, and improves the Vision-Centric score from 20.84 to 33.76. These results suggest that continuous latent diffusion could provide meaningful benefits beyond language generation.

  3. The analysis provides insights beyond benchmark comparisons. The investigations of denoising steps, attention refinement, and language latent representations help characterize the behavior of unified latent diffusion. In particular, the observation that contextual language representations do not necessarily outperform vocabulary embeddings is interesting.

Critical weaknesses
  1. The methodological novelty is relatively limited. The proposed framework mainly integrates existing techniques, including Transfusion-style multimodal modeling and ELF-based continuous language diffusion. Although the controlled experimental study is valuable, the paper introduces few new architectural or optimization mechanisms. Its contribution is therefore primarily empirical rather than methodological.

  2. The central claim that unified diffusion alleviates modality competition is not sufficiently substantiated. The authors attribute the improved visual generation performance to reduced interference between language and vision learning. However, the experiments only demonstrate better downstream performance without directly measuring gradient conflicts, optimization interference, or modality-specific learning dynamics. Moreover, changing the language modeling paradigm also changes its representation and training objectives, making it difficult to isolate the effect of objective unification itself.

  3. The performance improvements come with substantial trade-offs that deserve deeper investigation. Compared with the autoregressive baseline, latent diffusion performs considerably worse in language generation (24.30 vs. 14.68 PPL), OCR & Chart understanding (15.71 vs. 27.33), and CLIPScore (15.40 vs. 23.31). These gaps raise concerns about whether the proposed formulation provides a favorable overall trade-off for general-purpose multimodal models, particularly for tasks requiring precise linguistic understanding.

  4. The computational efficiency analysis is incomplete. While the paper emphasizes achieving better visual generation results with fewer training tokens, token budgets do not necessarily reflect actual computational costs. Continuous latent diffusion requires iterative denoising, and the visual understanding experiments use considerably more inference steps than autoregressive decoding. Comparisons of training FLOPs, inference latency, and performance under matched computational budgets would make the efficiency claims more convincing.

Questions for authors
  1. Can the authors provide direct evidence that unified latent diffusion reduces optimization conflicts between language and vision? For example, analyzing gradient cosine similarity, gradient interference, or modality-specific convergence could help validate the proposed explanation for improved joint training.

  2. How much of the improvement comes specifically from unifying the generative objectives, rather than from the latent representations, bidirectional attention, or additional training mechanisms introduced by ELF? Could the authors provide more targeted ablations to disentangle these factors?

2 comments
AuthorsOpenReview ↗

Author Rebuttal (1/2)

W1. Novelty

Our contribution is intentionally empirical. To isolate the effect of the language modeling formulation, we adopted existing formulations and kept the architecture, data, and optimization identical across variants, since new mechanisms would confound the comparison. Given that continuous diffusion LM remains underexplored in multimodal modeling, we believe these controlled findings are a valuable contribution.

W2. Modality Competition Analysis

(Revised paper: Sec. 5.1 and Tab. 2; evaluation details in Appx. D)

We thank the reviewer for suggesting gradient-based analysis. Please see our common comment, “Gradient-based Modality Competition Analysis.” At 30B tokens, Unified Latent Diffusion shows higher gradient alignment and stronger positive cross-task affinity, supporting our hypothesis of reduced modality competition.

W3. Are the Gains Merely Performance Trade-offs?

We acknowledge that different modeling approaches involve inherent trade-offs. However, our additional analyses suggest that the reported gaps in language-related tasks are largely attributable to evaluation settings, rather than fundamental limitations of latent diffusion.

  • Language generation. The reported PPL of 24.30 was obtained with only 64 denoising steps, substantially fewer generation steps than the AR baseline. Increasing the denoising steps to 128 and 256 reduces PPL to 18.6 and 15.0, respectively, approaching the AR baseline of 14.68 (please see our response to Q4 of reviewer W7mg for detailed results). This suggests that the initial gap is largely driven by the inference budget rather than an inherent weakness in language modeling.
  • T2I CLIPScore. (Revised paper: Tab. 1 and Sec. 4.2; caption-length analysis in Appx. E.1) The original COCO30K evaluation uses substantially shorter captions than our i1 training data. To better align the evaluation with the training distribution, we additionally evaluated with longer, recaptioned COCO prompts. Under this setting, latent diffusion outperforms AR in CLIPScore (27.47 vs. 25.31 at 30B tokens), consistent with its advantages in other visual generation metrics. The same trend holds at 15B tokens, with and without the T5 encoder (see the “Visual Generation” section of our comment, “Unified Latent Diffusion without the T5 Encoder”).
  • Detailed image captioning. (Revised paper: Appx. E.3 and Fig. 9) In our comment, “Detailed Image Captioning Evaluation,” we provide an additional visual understanding evaluation, where unified latent diffusion outperforms both baselines on all metrics.

Overall. The language generation gap narrows with more denoising steps. With prompts closer in length to the training captions, latent diffusion also leads on CLIPScore, consistent with other T2I metrics. While trade-offs remain in OCR & Chart, these results support our finding that Unified Latent Diffusion can benefit visual generation and understanding within the studied setting.

AuthorsOpenReview ↗

Author Rebuttal (2/2)

W4. Training and Inference Compute

We agree that matched training-token budgets do not necessarily imply matched computational costs. To address this concern, we conducted additional training FLOPs-matched and inference compute-matched evaluations to examine whether the observed performance trends persist under comparable computational budgets.

  • Training compute-matched comparison. We measured the training FLOPs of each modeling approach across different training token budgets, as shown below.

    Training TokensARDiscrete Diff.Latent Diff.
    5B0.5240.5150.610
    10B1.0481.0301.219
    15B1.5711.5451.828
    20B2.0952.0592.438
    25B2.6192.5743.048
    30B3.1433.0893.657

    While Unified Latent Diffusion requires more FLOPs per training token, our model at 25B tokens has a comparable training FLOPs budget to AR and Discrete Diffusion at 30B tokens. We therefore conducted a FLOPs-matched comparison across all evaluation categories.

    CategoryMetricAR (30B)Discrete Diff. (30B)Latent Diff. (25B)
    LanguagePPL ↓14.6830.6326.73
    Visual UnderstandingGen ↑23.5824.7524.86
    Visual UnderstandingKnow ↑15.9229.4526.75
    Visual UnderstandingOCR ↑27.3322.7314.28
    Visual UnderstandingVision-Centric ↑20.8429.3432.15
    Visual GenerationDPG ↑65.1659.1972.97
    Visual GenerationDiff. Loss ↓0.76280.79470.7218
    Visual GenerationFID ↓56.8365.0651.10
    Visual GenerationCLIPScore ↑23.3117.2215.36

    Under comparable training FLOPs, Unified Latent Diffusion preserves the overall performance trends, achieving the best results on vision-centric understanding and most visual generation metrics, while retaining the previously observed trade-offs in language-related tasks. We also observed no substantial differences in wall-clock training time across the approaches. These results suggest that the advantages of Unified Latent Diffusion are not merely due to differences in training compute. We will add this comparison to the revised paper.

  • Inference compute-matched comparison. (Revised paper: Appx. E.4 and Fig. 10) We additionally conducted compute-matched comparisons in terms of both inference FLOPs and latency. Please refer to our response to W3 of reviewer zVSC for detailed results and discussion.

Q1.

Please see W2 (Sec. 5.1 and Tab. 2; details in Appx. D).

Q2.

Text latent space. (Revised paper: Sec. 5.1 and Tab. 3; setup details in Appx. E.2) We replace contextual T5 features with vocabulary embeddings and find that most visual-generation gains remain. Please see our common comment, “Unified Latent Diffusion without the T5 Encoder.”

Bidirectional attention. Discrete Diffusion and Latent Diffusion use identical bidirectional attention masks for target text, yet Unified Latent Diffusion achieves higher visual-generation scores. Bidirectional attention alone therefore does not explain the difference.

Parameter counts. (Revised paper: Appx. C.1 and Tab. 4) Unified Latent Diffusion uses fewer total and trainable parameters, as reported in our response to W2 of reviewer zVSC. A larger parameter count therefore does not explain its gains.

Training mechanisms. We follow established recipes for each formulation, retaining standard training for AR and Discrete Diffusion, and self-conditioning and training-time CFG for Unified Latent Diffusion (Appx. B). We agree that component-wise ablations are important for attribution; given our limited compute budget, we focus on formulation-level comparisons and leave such ablations for future work.

We thank the reviewer for this constructive discussion, which helps clarify the interpretation of the observed gains. We have incorporated the new analyses into the revised paper (highlighted in blue) and will further clarify these points.

Reviewer_zVSCOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Unified Latent Diffusion, which generates both language and images through continuous latent denoising, and asks how this choice affects joint multimodal learning. Through controlled pretraining from scratch, they compare autoregressive, discrete diffusion, and latent diffusion language modeling with matched backbone architectures, training data, and token budgets, while keeping visual modeling fixed. They identify three findings: unifying generative modeling through latent diffusion eases joint training, iterative denoising refines answers and strengthens visual understanding, and effective language latents for diffusion modeling require different properties from vision. These comparisons examine the capabilities and limitations of unified latent diffusion as a generative formulation for general-purpose unified multimodal models.

Critical strengths
  • The experimental design is the strongest part of the work. Backbone, data, token budget, loss weights, and the whole vision path are held fixed, and only the language modeling formulation changes. This is the comparison the unified multimodal model literature has been missing, and it is executed carefully.

  • The evaluation is broad. Sixteen visual understanding benchmarks grouped into four categories, plus four visual generation metrics and a language metric, give a fair picture of the trade-offs rather than a single favored axis.

  • The analysis goes beyond the tables. The attention map refinement, the denoising step scaling, and the four-way latent space ablation are all useful and clearly presented.

Critical weaknesses
  • Scale is the central limitation. Everything runs at a 1.3B dense backbone and a 30B token budget. Autoregressive and diffusion language models are known to converge at different rates, so a ranking measured at this scale says little about the regime where unified multimodal models are actually built. The abstract claims the method is competitive with autoregressive and discrete diffusion baselines, which overstates what the tables show on the language side.

  • The comparison is not as matched as claimed. The autoregressive and discrete variants use the Qwen2.5 tokenizer and embeddings, while the latent diffusion variant uses a frozen T5-base encoder with extra projections and a decode head. Total parameter count differs, and more importantly, the same text consumes a different number of tokens under the two tokenizers, so a fixed token budget does not correspond to the same amount of text. This confound is never discussed.

  • Inference cost is ignored throughout. The autoregressive baseline decodes 32 tokens greedily while latent diffusion runs a 64-step sampler with classifier-free guidance, and the discrete baseline unmasks one token per step. Every headline gain in visual understanding and visual generation is bought with substantially more test-time compute, and no FLOPs-matched or latency-matched comparison is given.

Questions for authors
  • Report total parameter counts and per-sample inference FLOPs for each variant, and add at least one FLOPs-matched or wall-clock-matched comparison on visual understanding.

  • Add a diagnostic for the modality competition hypothesis, such as the cosine similarity between the vision and language loss gradients over training.

  • Would the OCR and chart regression survive instruction tuning? A light supervised stage would show whether it reflects weaker language priors or simply the absence of tuning.

2 comments
AuthorsOpenReview ↗

Author Rebuttal (1/2)

W1. Scale, Convergence and Overstatement in competitiveness

  • Model and data scale. We agree that our 1.3B / 30B-token setting cannot establish whether the rankings hold at larger scales, and we do not claim they do. Our claim is narrower: under a matched budget, how does the language modeling formulation affect multimodal learning?
  • Different convergence speed. Notably, since continuous diffusion is known to converge more slowly than AR on language [a, b], this budget is unfavorable to our method. Consistent with this, it underperforms on language generation. Yet even in this regime it achieves better DPG and FID, stronger Vision-Centric VQA, and higher detailed-captioning scores (see “Detailed Image Captioning Evaluation”).
  • Competitiveness claim. We would like to clarify that the abstract reports specific DPG-Bench and Vision-Centric gains, rather than claiming competitiveness with baselines. However, we agree that the TL;DR phrase “converges faster and is competitive” overstates the scope of our findings. We will revise it to state the visual gains alongside limitations including language performance.

[a] Gulrajani et al. Likelihood-Based Diffusion Language Models. NeurIPS, 2023.

[b] Yang et al. Continuous Diffusion Scales Competitively with Discrete Diffusion for Language. NeurIPS, 2026.

W2. T5 Encoder Ablation and Comparison Fairness

(Revised paper: Sec. 5.1 and Tab. 3 for the T5 ablation; Appx. C.1 and Tab. 4, 5 for parameter counts and token budget)

We thank the reviewer for pointing out this limitation. Our experiments match the shared backbone architecture, vision pipeline, data sources, and training-token budgets. However, these controls do not imply identical total parameter counts, or raw-text data exposure. We provide the following measurements and additional ablation to clarify these differences.

  • Unified Latent Diffusion without the T5 encoder still shows similar trends. To test whether the gains depend on T5 features, we replace the encoder with the vocab embedding and compare all models at 15B training tokens. As reported in our general response, “Unified Latent Diffusion without the T5 Encoder”, this variant outperforms both baselines in large margin in visual generation. These results show that the T5 encoder alone does not explain the gains.
  • Unified latent diffusion experiment uses fewer parameters. The vocabulary sizes are 151,665 for Qwen2.5 and 32,100 for T5. The larger Qwen vocabulary results in larger embedding and language-model output matrices. Although Unified Latent Diffusion adds a frozen T5 encoder and additional projections and decoding modules, its total and trainable parameter counts remain lower. These counts show that the gains are not explained by a larger trainable parameter budget.
Language ModelingShared backboneQwen vocab embedding & LM headAdditional modulesFrozen text encoder (T5)Frozen vision encoder & decoderTotalTrainable
Autoregressive1,321.6M465.9M0.0M—1,240.5M3.03B1.79B
Discrete Diff.1,321.6M465.9M0.0M—1,240.5M3.03B1.79B
Latent Diff.1,321.6M—41.1M109.6M1,240.5M2.71B1.36B
  • Tokenizer and text budget discussion.

    We measure the token count ratio, i.e., the number of Qwen2.5 tokens divided by the number of T5 tokens on identical text data. All ratios are below 1, indicating that T5 requires more tokens to encode the same text. Consequently, under the same training-token budget, T5 covers less raw text than Qwen2.5. Hence, the improvements of Unified Latent Diffusion are not explained by additional text exposure.

    DCVLMDCLMi1
    token count ratio (Qwen2.5 / T5)0.9350.9270.861

    Ratios are computed on 160K, 100K, and 62.5K randomly sampled examples from DCVLM, DCLM, and i1, respectively.

Overall. The visual gains remain without T5 encoder and are not explained by a larger trainable parameter or greater text coverage per token. These results support our finding that unified latent diffusion benefits multimodal visual learning within the studied multimodal setting.

AuthorsOpenReview ↗

Author Rebuttal (2/2)

W3. Inference Cost

(Revised paper: Appx. E.4 and Fig. 10)

We thank the reviewer for highlighting the importance of inference efficiency. We agree that differences in inference compute should be accounted for when comparing modeling approaches. To address this concern, we conducted additional FLOPs-matched and latency-matched evaluations.

  • FLOPs-matched and latency-matched comparisons. We varied the number of denoising steps for diffusion-based models and evaluated the average performance across four visual understanding task categories. For visual generation, all three approaches use the same image denoising procedure and sampling configuration, keeping the core generation budget matched.

    FLOPs were measured using torch.utils.flop_counter.FlopCounterMode with batch size 1, and latency was measured in BF16 on an NVIDIA RTX A6000 GPU. We used the same 16 randomly selected samples from each benchmark, with three warm-up runs before measurement.

  • Results and manuscript revision. We provide FLOPs–performance and latency–performance Pareto-frontier plots in Appendix E.4 (Figure 10). The results show that latent diffusion maintains its advantage over AR under comparable inference budgets. For a direct comparison, the table below reports visual understanding performance at approximately matched FLOPs and latency.

Matching criterionLanguage modelingDenoisingFLOPs ↓Latency ↓Avg. ↑
FLOPs-matchedAR-1.07T442ms22.59
Discrete Diff.1 token/step1.06T106ms27.53
Latent Diff.N=41.08T355ms25.84
Latency-matchedAR-1.07T442ms22.59
Discrete Diff.1 token/step1.06T106ms27.53
Latent Diff.N=41.08T355ms25.84

These comparisons show that the visual understanding gains persist under comparable inference costs, rather than arising solely from additional denoising compute.

Q1.

Please see W2 for parameter counts (Appx. C.1, Tab. 4).

Q2.

Please see our comment, “Gradient-based Modality Competition Analysis,” for gradient-based study of modality competition (Sec. 5.1 and Tab. 2; details in Appx. D).

Q3.

Thank you for raising this important point. Due to limited compute, we are not certain we can complete this experiment within the rebuttal period, but we will update here as soon as the results are available.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to Reviewers (1/2)

We thank the reviewers for their constructive feedback and valuable suggestions. To address the concerns raised about modality competition, pretrained text representations, and visual understanding capabilities, we conducted additional experiments and incorporated the results into the revised manuscript. Since these analyses address concerns shared across multiple reviews, we present them together here to avoid repetition in our individual responses. Below, we provide three analyses: (1) Gradient-based Modality Competition Analysis, (2) Unified Latent Diffusion without the T5 Encoder, and (3) Detailed Image Captioning Evaluation.

Gradient-based Modality Competition Analysis

(Revised paper: Sec. 5.1 and Tab. 2; evaluation details in Appx. D)

We analyze text–image modality competition using Gradient Conflict and Inter-task Affinity. We evaluate all three models at their 30B-token EMA checkpoints using 32 batches. Each batch contains 256 text-only (DCLM), 256 image-to-text (DCVLM), and 256 text-to-image (i1) samples.

We describe the definitions and computation of the two metrics below.

  • Gradient Conflict

    Following Uni-X [a], we measure gradient conflict between two objectives on the shared Transformer parameters θs\theta_s: LtextL_{\mathrm{text}} (text-only and image-to-text losses) and LimageL_{\mathrm{image}} (text-to-image loss).

    We compute the cosine similarity between their gradients (SinterS_{\mathrm{inter}}) and compare it against a reference similarity (SbaseS_{\mathrm{base}}), obtained from two joint-objective gradients computed on disjoint half-batches (128 samples per source in each half). The conflict score is defined as

    cg=Sbase−Sinter.c_g = S_{\mathrm{base}} - S_{\mathrm{inter}}.

    Lower cgc_g indicates less inter-task gradient conflict relative to the reference. All similarities use flattened, concatenated, unclipped gradients of the shared Transformer parameters, averaged over 32 batches.

  • Inter-task Affinity

    Following TAG [b], we measure how a gradient update for one objective affects the loss of the other. For each objective a∈text,imagea\in{\mathrm{text},\mathrm{image}}, we perform a single virtual SGD step with learning rate η=0.05\eta=0.05. Gradients are computed over all trainable parameters, globally clipped to norm 1, and applied only to the shared Transformer parameters, keeping all other parameters fixed.

    Let θ\theta denote all trainable parameters and θ(a)\theta^{(a)} the parameters after this update. The affinity from objective aa to the other objective bb is

    =1-\frac{L_b(\theta^{(a)})}{L_b(\theta)}, \qquad a\neq b.$$ Positive affinity indicates that the update reduces the receiver's loss, while negative affinity indicates interference. Each virtual update starts independently from the 30B-token EMA checkpoint.

Results. Unified latent diffusion achieves the highest gradient alignment and lowest conflict among all three models. Image→Text and Text→Image affinities are approximately 6× and 7× those of the autoregressive model, respectively. These results provide direct evidence of reduced modality competition in unified latent diffusion model.

Language ModelingGradient Cosine Sim. ↑Gradient Conflict cgc_g ↓Affinity: Image→Text (×10⁻⁴) ↑Affinity: Text→Image (×10⁻⁴) ↑
Autoregressive0.011 ± 0.0010.838 ± 0.0060.579 ± 0.2310.490 ± 0.141
Discrete Diff.0.001 ± 0.0010.4777 ± 0.041−0.183 ± 0.279−0.302 ± 0.143
Continuous Latent Diff. (Ours)0.074 ± 0.0070.305 ± 0.0203.448 ± 1.0083.483 ± 0.407

Each cell reports the mean across 32 batches and its 95% bootstrap confidence interval.

[a] Hao et al. Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Models. ICLR, 2026.

[b] Fifty et al. Efficiently Identifying Task Groupings for Multi-Task Learning. NeurIPS, 2021.

AuthorsOpenReview ↗

General Response to Reviewers (2/2)

Unified Latent Diffusion without the T5 Encoder

(Revised paper: Sec. 5.1 and Tab. 3; setup details in Appx. E.2)

Unified latent diffusion with vocab embedding. Unlike AR and discrete diffusion, Unified Latent Diffusion uses a pretrained T5 encoder. To assess its contribution to visual gains, we replace it with T5 vocabulary embeddings (32.1K tokens, 768 dimensions, 24.7M parameters) and compare all models at 15B training tokens, following Sec. 4.1 for training and Appx. C.3 for evaluation unless otherwise noted.

Results. Without the T5 encoder, Unified Latent Diffusion retains most visual generation gains and comparable visual understanding performance, suggesting that these benefits do not depend solely on pretrained contextual T5 features.

1. Visual Generation (T2I)

Even without the T5 encoder, Unified Latent Diffusion scores remain close to the T5-encoder variant. These results show that the visual-generation gains cannot be explained by T5 features alone. In addition, the T5 encoder yields a higher DPG score, suggesting a benefit for prompt following.

Language ModelingDPGDiff. LossFID* (FID-shortcap)CLIP* (CLIP-shortcap)
Autoregressive53.10.87392.5 (101.6)22.02 (20.5)
Discrete Diff.48.30.89798.30 (126.7)20.12 (14.7)
Latent Diff. w/ T5 Encoder64.60.80767.5 (100.2)23.71 (14.7)
Latent Diff. w/ Vocab Embedding63.30.79257.8 (86.1)23.65 (16.8)
  • denotes evaluation with recaptioned prompts. Parentheses show results with the original short captions.

Evaluation with recaptioned prompts. The original COCO prompts are much shorter than the i1 training captions. We therefore also evaluated FID* and CLIP* using COCO-Recap prompts [a] to reduce potential evaluation bias. For COCO CLIPScore, AR leads with the original short captions, while Unified Latent Diffusion leads with recaptioned prompts that are closer in length to the training captions.

(Revised paper: Appx. E.1)

i1 training captionsCOCOCOCO-Recap
Token length: median [P10–P90]210 [163–267]13 [10–18]73 [39–104]

Token lengths are measured with the T5 tokenizer.

2. Visual Understanding (I2T)

Replacing the T5 encoder with vocabulary embeddings preserves visual understanding performance.

Language ModelingGeneralKnowledgeOCR & ChartVision-Centric
Autoregressive19.7717.9320.7123.79
Discrete Diff.20.9926.6314.2828.26
Latent Diff. w/ T5 Encoder23.4226.0610.3331.01
Latent Diff. w/ Vocab Embedding23.8326.3012.3729.82

3. Language

Removing the T5 encoder increases Gen PPL from 49.53 to 61.69. Contextual T5 features help language generation, even though they are not required for the observed visual gains.

Language ModelingGen PPL
Autoregressive15.9
Discrete Diff.38.5
Latent Diff. w/ T5 Encoder49.53
Latent Diff. w/ Vocab Embedding61.69

[a] Xianhang Li et al. What If We Recaption Billions of Web Images with LLaMA-3? ICML, 2025.

Detailed Image Captioning Evaluation (Visual Understanding)

(Revised paper: Appx. E.3 and Fig. 9)

We add detailed image captioning evaluation to further examine the models’ visual understanding capabilities.

Results. Unified Latent Diffusion outperforms both baselines by a large margin across all three metrics. We hypothesize that these gains arise from iterative parallel refinement of the full latent sequence.

Language ModelingCIDEr ↑\uparrowMETEOR ↑\uparrowCAPTURE ↑\uparrow
Autoregressive2.23 (0.20)6.52 (3.08)28.84 (19.83)
Discrete Diff.3.029.9339.47
Latent Diff.4.6011.9050.05
  • Data: We evaluate 5,000 test samples from DOCCI [a] and DetailCaps-4870 [b], two recent datasets for detailed image captioning. Their reference captions average approximately 142 and 188 tokens, respectively, measured with the CLIP tokenizer.
  • Metrics: CAPTURE [b] compares objects, attributes, and relations in generated and reference captions. We also report CIDEr [c] and METEOR [d], conventional metrics for caption–reference agreement.
  • Prompt: We use the training captioning prompt: “Describe the image concisely.”
  • Output length: All models generate 128 tokens. For AR, we do not sample EOS to reach this length. Results with standard EOS stopping are reported in parentheses.
  • Sampling: For latent diffusion, we use a 32-step Euler solver, CFG 3, and self-conditioning CFG 1.

[a] Yasumasa Onoe et al. DOCCI: Descriptions of Connected and Contrasting Images. ECCV, 2024.

[b] Hongyuan Dong et al. Benchmarking and Improving Detail Image Caption. arXiv:2405.19092, 2024.

[c] Ramakrishna Vedantam et al. CIDEr: Consensus-based Image Description Evaluation. CVPR, 2015.

[d] Satanjeev Banerjee and Alon Lavie. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. ACL Workshop, 2005.

Dear Reviewers, thank you for your time and effort in reviewing this submission. As the discussion period is ongoing, I would appreciate it if you could join the discussion and engage with the authors' rebuttal at your earliest convenience.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.