- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents Dynin-Omni, which extends MMaDA’s discrete diffusion framework to video understanding, speech recognition, and speech synthesis. It also supports joint text–image–speech generation within a single reverse diffusion process. The three-stage training recipe first aligns new modalities, then uses model merging to recover inherited capabilities, and finally introduces harder tasks. Evaluations show competitive performance with modality-specific experts on several benchmarks.
Critical strengths
A single generative model supports text understanding, image understanding, generation and editing, video understanding, and speech understanding and generation. It also enables joint text–image–speech generation and demonstrates strong overall performance across multiple benchmarks.
Critical weaknesses
- The shared backbone, discrete representations, unified masked-token prediction objective, and basic generation mechanism largely follow MMaDA. Speech uses an existing frozen tokenizer, while video reuses image representations for sampled frames. Linear interpolation of shared parameters follows established model-merging approaches. The proposed parameter selection for the original and expanded vocabularies is intuitive, but its methodological novelty is limited.
- Table 4(c) compares enabling and disabling EOS supervision from the same initialization within each stage. These experiments support stage-specific effects. However, they do not directly establish that disabling EOS supervision in Stage 1 and enabling it in Stage 2 outperforms enabling it throughout both stages.
Questions for authors
Could the authors elaborate on the main methodological contributions beyond extending MMaDA to additional modalities, particularly the novel aspects of the proposed three-stage training recipe?
5 comments
Response to Reviewer sb2y (2/3)
| Stage 2 updates | Merging rule | ASR WER ↓ | TTS WER ↓ | Caption BLEU ↑ | Text BLEU ↑ | CLIP-T ↑ | CLIP-I ↑ |
|---|---|---|---|---|---|---|---|
| 0 | Shared | 289.84 | 98.37 | 4.41 | 10.74 | 0.3152 | 0.6527 |
| 0 | Stage 1-only | 131.94 | 97.98 | 4.41 | 10.06 | 0.3064 | 0.6411 |
| 0 | modality disent. (ours) | 108.12 | 88.24 | 6.43 | 11.15 | 0.3297 | 0.6693 |
| 8,192 | Shared | 28.69 | 7.74 | 8.20 | 14.97 | 0.5164 | 0.8489 |
| 8,192 | Stage 1-only | 29.25 | 7.79 | 5.52 | 23.04 | 0.5127 | 0.8364 |
| 8,192 | modality disent. (ours) | 22.11 | 6.43 | 10.25 | 31.16 | 0.5372 | 0.8502 |
At the same update count, our rule has the best point estimates on all six reported metrics. The post-training comparison therefore tests the vocabulary-row choice beyond its immediate initialization effect. At full scale, the complementary merge-versus-no-merge comparison after 10K Stage 2 updates improves ASR/TTS WER from 10.50/20.00 to 6.46/10.22% and GSM8K/POPE/MME-P from 18/50/820 to 38/76/1,150 (Table 4(b) of the revised manuscript).
2. Native joint text–image–speech generation. MMaDA [4] establishes text–image masked diffusion, while Omni-Diffusion [5] supports text–speech co-generation and image generation. Our contribution brings all three generated modalities into one interleaved output and a shared denoising trajectory. Text, image, and speech are target tokens in the same masked sequence, so predictions can use the intermediate tokens of other generated modalities before the output is finalized. This enables joint generation without first completing and freezing an anchor modality or handing generation to external modality-specific generators.
We train on interleaved three-modality output sequences and evaluate the resulting capability through joint-generation benchmarks and same-checkpoint decoding comparisons. At the same 16 NFE budget, Joint achieves 7.71 Overall versus 7.57 for MA-Text (Table 30). Blocking cross-modal attention lowers the mean text and image scores and increases mean speech WER relative to full joint generation (Table 6). These comparisons support the value of cross-modal communication within the shared generation process, alongside the demonstrated ability to produce all three modalities natively.
Beyond stories, the same native model generates all three modalities on cooking, DIY installation/repair, and sports/body-movement instructions without external orchestration. It achieves the highest text BERTScore among the four compared systems in all three domains (0.8378/0.8622/0.8599). Its auxiliary Overall scores of 5.696/5.569/5.549 exceed Omni-Diffusion’s 4.808/4.784/5.002 and MMaDA + Parler’s 0.929/3.998/4.618, demonstrating competitive cross-domain generation. Qwen + Kontext obtains higher auxiliary Overall scores of 6.796/6.889/6.511. Auxiliary Overall averages 10×BERTScore, 10×image CLIP, text judge, image judge, and speech consistency; Table 22 in Appendix I.10 reports the component metrics for all systems.
Together, these results connect the new joint-output capability to a tested adaptation recipe: native three-modality generation is demonstrated in one model, while the merging and EOS controls explain how the chosen training design recovers inherited capabilities and improves termination behavior.
W2. Off→On improves ASR and TTS WER over AlwaysOn across the full path
We test the effect of deferring EOS supervision during initial alignment in a reduced-data control with the same 8B backbone. We compare Off→On with AlwaysOn and an Omni-Diffusion attenuation→On condition (γ = 0.6) using the same 8B MMaDA backbone, speech vocabulary, 256 speech–transcript adaptation pairs, and multimodal task data described above. All three first undergo 4,096 ASR/TTS alignment updates corresponding to Stage 1, then continue without model merging into multimodal integration corresponding to Stage 2. Each receives 2,048 multimodal integration updates, reaching 6,144 total updates. Mixed AlwaysOn mixes all modalities from the start and trains for 6,144 updates with EOS supervision enabled throughout.
| Training condition | ASR WER % ↓ | TTS WER % ↓ |
|---|---|---|
| Staged AlwaysOn | 71.29 | 16.41 |
| Staged attenuation→On (γ = 0.6) | 65.16 | 17.08 |
| Staged Off→On (ours) | 64.27 | 15.57 |
| Mixed AlwaysOn | 80.19 | 14.78 |
At the common endpoint, Off→On improves ASR/TTS WER by 7.02/0.84 percentage points over AlwaysOn and 0.89/1.51 points over attenuation→On. Off→On achieves the lowest ASR WER among all four conditions and the lowest ASR and TTS WER among the three staged schedules. This whole-path experiment complements the stage-specific EOS controls in the proposed training recipe.
Response to Reviewer sb2y (3/3)
References
[1] Chen et al., Model Composition for Multimodal Large Language Models, ACL 2024. https://aclanthology.org/2024.acl-long.606/
[2] Li et al., Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling, ACL 2025. https://aclanthology.org/2025.acl-long.1491/
[3] Wei et al., OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging, ICLR 2026. https://arxiv.org/abs/2505.19892
[4] Yang et al., MMaDA: Multimodal Large Diffusion Language Models, arXiv 2025. https://arxiv.org/abs/2505.15809
[5] Li et al., Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion, ICML 2026. https://proceedings.mlr.press/v306/li26dv.html
Response to Reviewer sb2y (1/3)
Thank you for recognizing that a single generative model supports text and image understanding and generation, video understanding, speech understanding and generation, and joint text–image–speech generation, with strong performance across benchmarks. The revised manuscript clarifies both contributions in Sec. 1: (1) an effective capability-preserving staged adaptation recipe combining scheduled EOS supervision with modality-disentangled merging, and (2) the first masked-diffusion model to natively generate interleaved text, image, and speech within a single backbone and reverse process. To support the training recipe, we added whole-path EOS-schedule comparisons in Sec. 5.1 and Table 5, and matched post-merging comparisons in Appendix I.7, Table 19.
W1 + Q1. Contributions beyond MMaDA: effective capability-preserving adaptation and first native three-modality joint generation
Our contributions beyond MMaDA are (1) a staged adaptation recipe that combines scheduled EOS supervision with modality-disentangled vocabulary merging to preserve inherited capabilities, and (2) native generation of interleaved text, image, and speech within a single masked-diffusion backbone and reverse process. The recipe combines scheduled EOS supervision, which defers termination learning until after initial content alignment, with modality-disentangled merging, which separately handles inherited and newly introduced token representations during vocabulary expansion. The following sections present the training recipe and the new native joint-generation capability, with controlled comparisons supporting the recipe’s two components.
1. An effective staged adaptation recipe that preserves inherited capabilities. The recipe first aligns speech and video with text, then merges the adapted model with the original backbone and integrates the modalities, and finally scales the training mixture and task difficulty. The recipe combines two complementary components:
(a) Scheduled EOS supervision. During initial alignment, EOS supervision is deferred so that the model learns content before termination; it is restored during integration. Omni-Diffusion’s attenuation-based masking [5] reduces padding-token masking through an attenuation coefficient γ, leaving its strength as an additional hyperparameter for heuristic selection or search. Our stage-based schedule controls when termination targets enter training, without requiring a search over a continuous attenuation coefficient. In the matched 6,144-update reduced-data whole-path comparison, Off→On achieves ASR/TTS WER of 64.27/15.57%, lower than AlwaysOn at 71.29/16.41% and Omni-Diffusion attenuation→On at 65.16/17.08%. The full comparison is presented in W2 below and Table 5 of the revised manuscript.
(b) Modality-disentangled merging. Prior multimodal merging work largely integrates understanding capabilities by reusing modality-specific encoders and merging language-model or expert parameters: NaiveMC [1] averages language-model weights, while MMER [2] and OptMerge [3] develop retention and expert-merging mechanisms. NaiveMC in particular illustrates that a multimodal composition contribution can lie in which modality-specific components are preserved and how their capabilities are combined, even when shared parameters are linearly averaged. Expanding the vocabulary of native token-based generators such as MMaDA [4] and Omni-Diffusion [5] introduces a distinct interface issue: newly introduced speech tokens add rows to both the input embedding and output head. Our rule interpolates the shared Transformer, preserves the original model’s existing-token rows, and imports the aligned expert’s new speech-token rows. The distinction is the origin-specific treatment of both input and output token representations: inherited rows retain the original token representations, while new rows retain those learned during speech alignment.
Full-scale adaptation required 63,425 H100 GPU-hours. To compare vocabulary-merging rules within the rebuttal period, we use a reduced-data control with the full 8B backbone and matched post-merging training across rules. The control retains the 8B backbone and uses the same speech-aligned checkpoint, 5:5 interpolation of shared Transformer parameters, multimodal training mixture, and 8,192 Stage 2 updates. Speech adaptation uses 256 speech–transcript pairs from the EMOVA Alignment 7M LibriSpeech–AISHELL-2 training subset; integration includes Flickr8k image-caption examples and text-only language training examples. Text and image metrics use 32 examples per task (Appendix I.7, Table 19).