acceptodds
Under review as a conference paper at ICLR 2027

Dynin-Omni: Omnimodal Unified Large Diffusion Language Model

Abstract

Omnimodal unified models aim to understand text, images, video, and speech and generate coordinated text, image, and speech outputs. The challenge is to extend a single backbone to new modalities efficiently while preserving established capabilities and enabling coordinated cross-modal generation. We present Dynin-Omni, an open-source 8B-scale native omnimodal masked-diffusion model that unifies these capabilities within a single token-level backbone under a shared masked-token prediction objective. We propose a three-stage training recipe for effective omnimodal adaptation that first aligns new modalities without replaying the full task mixture, then applies modality-disentangled merging to recover inherited capabilities. Scheduled padding supervision delays termination learning until semantic grounding is established, enabling flexible-length generation. Moreover, Dynin-Omni is the first masked-diffusion model to natively generate interleaved text, image, and speech within a single reverse diffusion process, enabling mutual conditioning and joint refinement without modality-wise orchestration. Across 20 benchmark settings covering text, image, video, speech, and joint omnimodal generation, Dynin-Omni outperforms prior unified systems on a broad range of directly comparable benchmarks while remaining competitive with modality-specific experts. Stage-wise and controlled analyses show capability recovery and improved length control, while native joint decoding approaches the quality of the best evaluated two-stage generation configuration with a lower sampling budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Omnimodal Model, Diffusion Language Model, Any-to-Any Model, Unified Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sb2yOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents Dynin-Omni, which extends MMaDA’s discrete diffusion framework to video understanding, speech recognition, and speech synthesis. It also supports joint text–image–speech generation within a single reverse diffusion process. The three-stage training recipe first aligns new modalities, then uses model merging to recover inherited capabilities, and finally introduces harder tasks. Evaluations show competitive performance with modality-specific experts on several benchmarks.

Critical strengths

A single generative model supports text understanding, image understanding, generation and editing, video understanding, and speech understanding and generation. It also enables joint text–image–speech generation and demonstrates strong overall performance across multiple benchmarks.

Critical weaknesses
  1. The shared backbone, discrete representations, unified masked-token prediction objective, and basic generation mechanism largely follow MMaDA. Speech uses an existing frozen tokenizer, while video reuses image representations for sampled frames. Linear interpolation of shared parameters follows established model-merging approaches. The proposed parameter selection for the original and expanded vocabularies is intuitive, but its methodological novelty is limited.
  2. Table 4(c) compares enabling and disabling EOS supervision from the same initialization within each stage. These experiments support stage-specific effects. However, they do not directly establish that disabling EOS supervision in Stage 1 and enabling it in Stage 2 outperforms enabling it throughout both stages.
Questions for authors

Could the authors elaborate on the main methodological contributions beyond extending MMaDA to additional modalities, particularly the novel aspects of the proposed three-stage training recipe?

5 comments
AuthorsOpenReview ↗

Response to Reviewer sb2y (1/3)

Thank you for recognizing that a single generative model supports text and image understanding and generation, video understanding, speech understanding and generation, and joint text–image–speech generation, with strong performance across benchmarks. The revised manuscript clarifies both contributions in Sec. 1: (1) an effective capability-preserving staged adaptation recipe combining scheduled EOS supervision with modality-disentangled merging, and (2) the first masked-diffusion model to natively generate interleaved text, image, and speech within a single backbone and reverse process. To support the training recipe, we added whole-path EOS-schedule comparisons in Sec. 5.1 and Table 5, and matched post-merging comparisons in Appendix I.7, Table 19.

W1 + Q1. Contributions beyond MMaDA: effective capability-preserving adaptation and first native three-modality joint generation

Our contributions beyond MMaDA are (1) a staged adaptation recipe that combines scheduled EOS supervision with modality-disentangled vocabulary merging to preserve inherited capabilities, and (2) native generation of interleaved text, image, and speech within a single masked-diffusion backbone and reverse process. The recipe combines scheduled EOS supervision, which defers termination learning until after initial content alignment, with modality-disentangled merging, which separately handles inherited and newly introduced token representations during vocabulary expansion. The following sections present the training recipe and the new native joint-generation capability, with controlled comparisons supporting the recipe’s two components.

1. An effective staged adaptation recipe that preserves inherited capabilities. The recipe first aligns speech and video with text, then merges the adapted model with the original backbone and integrates the modalities, and finally scales the training mixture and task difficulty. The recipe combines two complementary components:

(a) Scheduled EOS supervision. During initial alignment, EOS supervision is deferred so that the model learns content before termination; it is restored during integration. Omni-Diffusion’s attenuation-based masking [5] reduces padding-token masking through an attenuation coefficient γ, leaving its strength as an additional hyperparameter for heuristic selection or search. Our stage-based schedule controls when termination targets enter training, without requiring a search over a continuous attenuation coefficient. In the matched 6,144-update reduced-data whole-path comparison, Off→On achieves ASR/TTS WER of 64.27/15.57%, lower than AlwaysOn at 71.29/16.41% and Omni-Diffusion attenuation→On at 65.16/17.08%. The full comparison is presented in W2 below and Table 5 of the revised manuscript.

(b) Modality-disentangled merging. Prior multimodal merging work largely integrates understanding capabilities by reusing modality-specific encoders and merging language-model or expert parameters: NaiveMC [1] averages language-model weights, while MMER [2] and OptMerge [3] develop retention and expert-merging mechanisms. NaiveMC in particular illustrates that a multimodal composition contribution can lie in which modality-specific components are preserved and how their capabilities are combined, even when shared parameters are linearly averaged. Expanding the vocabulary of native token-based generators such as MMaDA [4] and Omni-Diffusion [5] introduces a distinct interface issue: newly introduced speech tokens add rows to both the input embedding and output head. Our rule interpolates the shared Transformer, preserves the original model’s existing-token rows, and imports the aligned expert’s new speech-token rows. The distinction is the origin-specific treatment of both input and output token representations: inherited rows retain the original token representations, while new rows retain those learned during speech alignment.

Full-scale adaptation required 63,425 H100 GPU-hours. To compare vocabulary-merging rules within the rebuttal period, we use a reduced-data control with the full 8B backbone and matched post-merging training across rules. The control retains the 8B backbone and uses the same speech-aligned checkpoint, 5:5 interpolation of shared Transformer parameters, multimodal training mixture, and 8,192 Stage 2 updates. Speech adaptation uses 256 speech–transcript pairs from the EMOVA Alignment 7M LibriSpeech–AISHELL-2 training subset; integration includes Flickr8k image-caption examples and text-only language training examples. Text and image metrics use 32 examples per task (Appendix I.7, Table 19).

AuthorsOpenReview ↗

Response to Reviewer sb2y (2/3)

Stage 2 updatesMerging ruleASR WER ↓TTS WER ↓Caption BLEU ↑Text BLEU ↑CLIP-T ↑CLIP-I ↑
0Shared289.8498.374.4110.740.31520.6527
0Stage 1-only131.9497.984.4110.060.30640.6411
0modality disent. (ours)108.1288.246.4311.150.32970.6693
8,192Shared28.697.748.2014.970.51640.8489
8,192Stage 1-only29.257.795.5223.040.51270.8364
8,192modality disent. (ours)22.116.4310.2531.160.53720.8502

At the same update count, our rule has the best point estimates on all six reported metrics. The post-training comparison therefore tests the vocabulary-row choice beyond its immediate initialization effect. At full scale, the complementary merge-versus-no-merge comparison after 10K Stage 2 updates improves ASR/TTS WER from 10.50/20.00 to 6.46/10.22% and GSM8K/POPE/MME-P from 18/50/820 to 38/76/1,150 (Table 4(b) of the revised manuscript).

2. Native joint text–image–speech generation. MMaDA [4] establishes text–image masked diffusion, while Omni-Diffusion [5] supports text–speech co-generation and image generation. Our contribution brings all three generated modalities into one interleaved output and a shared denoising trajectory. Text, image, and speech are target tokens in the same masked sequence, so predictions can use the intermediate tokens of other generated modalities before the output is finalized. This enables joint generation without first completing and freezing an anchor modality or handing generation to external modality-specific generators.

We train on interleaved three-modality output sequences and evaluate the resulting capability through joint-generation benchmarks and same-checkpoint decoding comparisons. At the same 16 NFE budget, Joint achieves 7.71 Overall versus 7.57 for MA-Text (Table 30). Blocking cross-modal attention lowers the mean text and image scores and increases mean speech WER relative to full joint generation (Table 6). These comparisons support the value of cross-modal communication within the shared generation process, alongside the demonstrated ability to produce all three modalities natively.

Beyond stories, the same native model generates all three modalities on cooking, DIY installation/repair, and sports/body-movement instructions without external orchestration. It achieves the highest text BERTScore among the four compared systems in all three domains (0.8378/0.8622/0.8599). Its auxiliary Overall scores of 5.696/5.569/5.549 exceed Omni-Diffusion’s 4.808/4.784/5.002 and MMaDA + Parler’s 0.929/3.998/4.618, demonstrating competitive cross-domain generation. Qwen + Kontext obtains higher auxiliary Overall scores of 6.796/6.889/6.511. Auxiliary Overall averages 10×BERTScore, 10×image CLIP, text judge, image judge, and speech consistency; Table 22 in Appendix I.10 reports the component metrics for all systems.

Together, these results connect the new joint-output capability to a tested adaptation recipe: native three-modality generation is demonstrated in one model, while the merging and EOS controls explain how the chosen training design recovers inherited capabilities and improves termination behavior.

W2. Off→On improves ASR and TTS WER over AlwaysOn across the full path

We test the effect of deferring EOS supervision during initial alignment in a reduced-data control with the same 8B backbone. We compare Off→On with AlwaysOn and an Omni-Diffusion attenuation→On condition (γ = 0.6) using the same 8B MMaDA backbone, speech vocabulary, 256 speech–transcript adaptation pairs, and multimodal task data described above. All three first undergo 4,096 ASR/TTS alignment updates corresponding to Stage 1, then continue without model merging into multimodal integration corresponding to Stage 2. Each receives 2,048 multimodal integration updates, reaching 6,144 total updates. Mixed AlwaysOn mixes all modalities from the start and trains for 6,144 updates with EOS supervision enabled throughout.

Training conditionASR WER % ↓TTS WER % ↓
Staged AlwaysOn71.2916.41
Staged attenuation→On (γ = 0.6)65.1617.08
Staged Off→On (ours)64.2715.57
Mixed AlwaysOn80.1914.78

At the common endpoint, Off→On improves ASR/TTS WER by 7.02/0.84 percentage points over AlwaysOn and 0.89/1.51 points over attenuation→On. Off→On achieves the lowest ASR WER among all four conditions and the lowest ASR and TTS WER among the three staged schedules. This whole-path experiment complements the stage-specific EOS controls in the proposed training recipe.

AuthorsOpenReview ↗

Response to Reviewer sb2y (3/3)

References

[1] Chen et al., Model Composition for Multimodal Large Language Models, ACL 2024. https://aclanthology.org/2024.acl-long.606/

[2] Li et al., Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling, ACL 2025. https://aclanthology.org/2025.acl-long.1491/

[3] Wei et al., OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging, ICLR 2026. https://arxiv.org/abs/2505.19892

[4] Yang et al., MMaDA: Multimodal Large Diffusion Language Models, arXiv 2025. https://arxiv.org/abs/2505.15809

[5] Li et al., Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion, ICML 2026. https://proceedings.mlr.press/v306/li26dv.html

Reviewer_NJyBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Dynin-Omni extends MMaDA into an 8B-scale masked-diffusion model for text, image, video, and speech understanding, with text, image, and speech generation. Its three-stage training recipe combines modality-disentangled merging with delayed EOS supervision to preserve inherited capabilities and learn output termination. The model also generates interleaved text-image-speech outputs jointly. Evaluation across individual tasks and Omni-StoryBench, including overlap filtering and decoding comparisons, demonstrates broad capabilities and competitive joint-generation quality.

Critical strengths
  1. The strongest contribution is the demonstrated breadth of native capability within a shared backbone and objective. Tables 1-3 show competitive reasoning, vision, and speech results together with joint text-image-speech generation, rather than attaching external generators for each output. The Omni-StoryBench comparison remains favorable after filtering potential training overlap: Table 14 reports 7.55 Overall on the same 712 clean examples, compared with 7.42 for the strongest listed orchestration baseline. This makes the system valuable even though specialists and some narrower unified models remain stronger on individual tasks.

  2. The training analysis provides concrete evidence about capability recovery and termination behavior. Table 4 and Appendix I.1 report matched-update controls for merging versus no merging and for EOS supervision. At 10K Stage 2 updates, merging improves both inherited-task scores and ASR/TTS WER. The EOS controls distinguish premature stopping from repetition and decoding-cap failures, supporting the proposed sequence of semantic grounding followed by termination learning. These targeted controls and the documented adaptation cost make the training recipe more informative than a collection of final benchmark scores.

Critical weaknesses
  1. The incremental methodological contribution over the closest prior work is incompletely established. The paper acknowledges Omni-Diffusion, which already uses a shared masked-token objective across text, image, and speech, progressive training, and a padding-related intervention. However, it is absent from the quantitative comparisons, and scheduled EOS supervision is not directly compared with its attenuated padding approach. Likewise, Figure 5 compares alternative vocabulary-merging rules at initialization, while Table 9's 10K-update rows do not include the Shared and Stage 1-only alternatives. Thus, the controls establish benefits over no merging and fixed EOS choices more clearly than benefits over the most relevant alternatives. This limits the strength of the methodological novelty claim without negating the system contribution.

  2. The joint-decoding efficiency and uncertainty evidence should be interpreted cautiously. In Table 22, the 128-NFE Joint versus MA-Text difference is only 7.93 versus 7.91 Overall and 6.97 versus 6.93 Omni, without paired uncertainty estimates. Equal NFE does not necessarily imply equal computation when passes use different active modalities or sequence lengths. The shaded region in Figure 6 is cross-category dispersion rather than statistical uncertainty, and Table 13 uses only five bootstrap resamples for reasoning intervals. These issues do not invalidate the demonstrated capabilities, but weaken stronger claims about reliable quality advantages and computational efficiency.

Questions for authors
  1. How does Dynin-Omni compare with Omni-Diffusion on shared tasks under documented comparable evaluation settings? Can the EOS schedule be compared with attenuated padding supervision under the same initialization, data, and update budget? This would clarify which benefits are specific to the proposed recipe.

  2. Do the alternative vocabulary-merging rules remain inferior after the same 10K Stage 2 updates used in Table 9? The initialization sweep is useful, but a matched post-training comparison would more directly support the claimed advantage of modality-disentangled merging.

  3. For joint versus anchored generation, can you report paired uncertainty for the quality differences and actual latency or computation at comparable output lengths? Please also recompute the reasoning bootstrap intervals with enough resamples for a stable estimate and keep Figure 6's cross-category spread distinct from uncertainty. These clarifications would determine whether I interpret the result as comparable quality at a lower nominal sampling budget or a stronger efficiency advantage.

4 comments
AuthorsOpenReview ↗

Response to Reviewer NJyB (1/2)

Thank you for recognizing the breadth of native capabilities under one backbone, the favorable 712-example overlap-filtered joint-generation comparison, and the matched-update evidence for capability recovery and termination behavior. We added a quantitative Omni-Diffusion comparison, matched EOS and merging controls, measured decoding costs, and expanded uncertainty analysis. In the revised manuscript, the complete-path EOS comparison appears in Sec. 5.1 and Table 5; the matched vocabulary comparison, actual costs, and Omni-Diffusion comparison appear in Appendices I.7, I.8, and I.11, Tables 19, 20, and 23, respectively. Appendix G.3 explains Figure 6's shading, and Table 15 reports updated reasoning uncertainty.

W1 + Q1. Omni-Diffusion on shared tasks and scheduled EOS vs. attenuated padding

Shared benchmarks and reasoning. We compare Dynin-Omni with Omni-Diffusion [1] on shared understanding, speech, and reasoning benchmarks. The first three rows report the results from the two papers; the last two use the same fixed inputs and grading in our new evaluation, with each model’s native prompting and 1,024 decoding steps.

BenchmarkEvaluationDynin-Omni (ours)Omni-Diffusion [1]
MME-Perception ↑Reported1672.91176.1
MMMU-val % ↑Reported52.831.1
LibriSpeech test-clean ASR WER % ↓Reported2.16.69
GSM8K, 256 examples, accuracy % ↑Same-input80.863.91
MATH, 128 examples, accuracy % ↑Same-input35.162.34

Across the five comparisons, Dynin-Omni achieves higher understanding and reasoning scores and lower ASR WER than Omni-Diffusion. The common-input reasoning evaluation also favors Dynin-Omni on both GSM8K and MATH.

EOS schedule. Full-scale adaptation required 63,425 H100 GPU-hours. To compare EOS schedules within the rebuttal period, we use a reduced-data control with the full 8B backbone. We compare Off→On, AlwaysOn, and the Omni-Diffusion attenuation rule (gamma = 0.6) under the same 8B MMaDA [2] initialization, new speech vocabulary, data, optimizer, training batches, decoding, and 6,144 updates. Adaptation uses 256 speech–transcript pairs from the EMOVA Alignment 7M LibriSpeech–AISHELL-2 training subset; integration includes Flickr8k image-caption examples for captioning and text-to-image, and language replay examples. In the staged conditions, 4,096 ASR/TTS alignment updates correspond to Stage 1, followed by 2,048 multimodal integration updates corresponding to Stage 2, totaling 6,144 updates. The mixed condition mixes all modalities from the start and trains for 6,144 updates. EOS supervision is enabled throughout Mixed AlwaysOn and during integration in all staged conditions.

Training conditionASR WER % ↓TTS WER % ↓
Staged, AlwaysOn71.2916.41
Staged, attenuation→On65.1617.08
Staged, Off→On (ours)64.2715.57
Mixed AlwaysOn80.1914.78

Off→On obtains the lowest ASR WER among the four conditions and the lowest ASR and TTS WER among the three staged EOS schedules. These results support deferring EOS supervision during initial speech alignment and enabling it during integration. Unlike attenuation-based masking, this stage-based switch does not require a heuristic search for an attenuation coefficient γ.

W1 + Q2. Do alternative vocabulary-merging rules remain inferior after matched post-training?

We compare the merging rules using the same backbone, speech vocabulary, and data as the reduced-data control in W1+Q1. Starting from the same speech-aligned checkpoint, we interpolate the shared Transformer parameters at 5:5 and compare the Shared, Stage 1-only, and modality-disentangled vocabulary rules. Each initialization then receives the same multimodal Stage 2 training for 8,192 updates. We selected 8,192 updates as a practical Stage 2 budget, balancing the extent of post-merging adaptation against the computational cost of the reduced-data ablation. Text and image metrics use 32 examples per task.

Stage 2 updatesMerging ruleASR WER ↓TTS WER ↓Caption BLEU ↑Text BLEU ↑CLIP-T ↑CLIP-I ↑
0Shared289.8498.374.4110.740.31520.6527
0Stage 1-only131.9497.984.4110.060.30640.6411
0modality disent. (ours)108.1288.246.4311.150.32970.6693
8,192Shared28.697.748.2014.970.51640.8489
8,192Stage 1-only29.257.795.5223.040.51270.8364
8,192modality disent. (ours)22.116.4310.2531.160.53720.8502
AuthorsOpenReview ↗

Response to Reviewer NJyB (2/2)

After the same 8,192 Stage 2 updates, modality-disentangled merging leads both alternative vocabulary-merging rules on all six reported metrics, supporting its advantage in this controlled adaptation setting. In Table 4(b) of the revised manuscript, the full-scale comparison shows that merging also improves over no merging after 10K updates, with ASR/TTS WER 6.46/10.22% versus 10.50/20.00%.

W2 + Q3. Joint vs. anchored: paired uncertainty, actual cost, reasoning bootstrap, and Figure 6 bands

Our stated contribution is a quality–budget tradeoff: joint denoising updates modalities together, allowing a useful joint output before an anchored two-stage process has spent a comparable total budget. Appendix I, Table 30 of the revised manuscript reports Joint versus MA-Text Overall of 7.71/7.57 at 16 NFE, 7.76/7.62 at 32, 7.86/7.80 at 64, and 7.93/7.91 at 128. At 256 NFE MA-Text reaches 7.95. Overall averages text BERTScore, text judge, image CLIP, image judge, and speech judge scores.

The added paired analysis quantifies the low-budget gains highlighted by Reviewer oTTR. On a separate 32-input, two-repeat panel, Joint−MA-Text Overall differences are +0.1098 [0.0550, 0.2198] at 16 NFE and +0.0598 [−0.1050, 0.2198] at 128 NFE (95% paired bootstrap CIs). This panel is separate from the Table 30 evaluation; its settings are described in Appendix I.8.

We measure actual forward calls, Transformer FLOPs, latency and GPU memory with fixed token lengths allocated to text, image and speech. Each method pair uses the same image refinement setting; latency is measured on two inputs on one B200. Budget denotes total denoising steps across passes, as in the paper.

Budget (NFE)MethodForward callsTransformer TFLOPs ↓Latency, s ↓Peak GiB ↓Paper Overall ↑
128Joint (ours)1595642.3210.54317.487.93
128MA-Text1605677.8110.61217.487.91
256MA-Text28310042.6218.39817.487.95

Paper Overall is from Table 30 of the revised manuscript; costs use the fixed token lengths above on two inputs.

Joint-128 reduces measured latency by 42.7% and Transformer FLOPs by 43.8% relative to MA-Text-256. The reported Overall scores are 7.93 and 7.95, respectively. Joint-128 achieves a similar reported Overall score to MA-Text-256 at lower measured cost, using half the denoising-step budget. At equal budget, measured costs are nearly identical, and our separate paired analysis further confirms a clear quality gain at 16 NFE (see Appendix I.8 for full results).

Figure 6 and reasoning uncertainty. Appendix G.3 identifies the shaded bands as ±0.2 standard deviations across categories, rather than confidence intervals. For reasoning, we recomputed the intervals from 20,000 bootstrap resamples of evaluation examples: GSM8K 207/256 = 80.86%, 95% CI [75.78, 85.55]; MATH 45/128 = 35.16%, 95% CI [27.34, 43.75]. These fixed-subset evaluations are separate from the full-benchmark results in the main tables.

References

[1] Li et al., Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion, ICML 2026. https://proceedings.mlr.press/v306/li26dv.html

[2] Yang et al., MMaDA: Multimodal Large Diffusion Language Models, arXiv 2025. https://arxiv.org/abs/2505.15809

Thank you for the detailed response. The additional baseline comparisons, EOS and merging controls, and uncertainty and cost analyses substantially address my main concerns. Although the reduced-data controls have limited generality, the new evidence strengthens the contribution. I am raising my recommendation to 3, with no further substantive questions.

Thank you for your follow-up and positive assessment. We are glad that the additional baseline comparisons, EOS and merging controls, and uncertainty and measured-cost analyses substantially address your main concerns. Your constructive feedback helped us strengthen the evidence for both the training recipe and the quality–budget trade-off of joint generation.

Reviewer_oTTROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Dynin-Omni, an 8B-scale masked-diffusion model supporting text and image understanding and generation, video understanding, and speech understanding and generation. It demonstrates text, image, and speech generation within a single masked-diffusion model. The experiments support the proposed training methods, and joint decoding shows improvements at lower sampling budgets. However, the practical efficiency advantage remains unclear, and the claimed mutual refinement between modalities is not directly verified. The evaluation of joint generation is also limited to a relatively narrow setting. Despite these concerns, the unified architecture and experimental results support a borderline-positive assessment.

Critical strengths
  • The paper unifies text, image, and speech generation within a single masked-diffusion backbone. It supports interleaved multimodal generation without relying on separate image or speech generators.
  • The three-stage training strategy helps recover capabilities lost when introducing new modalities. Table 4(a) shows substantial improvements in text and image tasks after Stage 1, while maintaining the newly learned speech and video capabilities.
  • Tables 4(b) and 4(c) show that model merging improves performance on existing and new tasks, while scheduled EOS supervision reduces premature termination and repetition.
  • Figure 6 compares joint diffusion with modality-anchored decoding using the same checkpoint and matched function-evaluation budgets. The results show that joint decoding performs better at lower sampling budgets.
Critical weaknesses
  • The efficiency advantage of joint decoding is not fully established. Figure 6 compares decoding methods using the number of function evaluations. However, the computational cost of each evaluation may differ depending on sequence length and modality composition. A comparison of actual inference time, FLOPs, and GPU memory would better support the efficiency claim.
  • The claimed mutual refinement between modalities is not directly demonstrated. The paper argues that text, image, and speech can refine one another during diffusion. However, the experiments mainly compare final generation results. It remains unclear whether updating one modality actually improves the others. A cross-modal attention or token-freezing ablation would help verify this claim.
  • The complete training strategy lacks a matched baseline. Table 4 evaluates the individual effects of model merging and EOS supervision, but the three-stage approach is not compared with mixed-modality training from the beginning under the same compute budget. Therefore, the efficiency benefit of the overall training strategy remains unclear.
  • The evaluation of joint generation is limited. The main results rely on Omni-StoryBench and automatic metrics, including LLM-based judgments. These metrics may not fully capture narrative coherence, consistency between images and speech, or perceptual quality. Human evaluation would strengthen the conclusions.
  • Generalization beyond the joint-generation training data is unclear. The paper uses approximately 10K joint-generation examples, including synthetic procedural data. The experiments do not sufficiently show how well the model performs on different types of multimodal tasks outside the story-oriented setting.
Questions for authors
  • Does joint decoding maintain its advantage over modality-anchored decoding when compared using actual inference time, FLOPs, and GPU memory rather than function evaluations?
  • How does the three-stage training strategy compare with mixed-modality training from the beginning under the same data and compute budget?
  • How well does the model generalize to joint-generation tasks outside the story-oriented and procedural training distribution?
4 comments
AuthorsOpenReview ↗

Response to Reviewer oTTR (1/3)

Thank you for recognizing native interleaved generation within one backbone, the recovery of inherited capabilities during three-stage training, the merging and EOS controls, and the low-budget gains of joint decoding. We performed new measurements of actual decoding cost, cross-modal attention interventions, staged versus mixed training, and joint generation outside stories. The revised manuscript adds the attention intervention to Sec. 5.2 and Table 6 and summarizes nonstory results in Sec. 5.2. The EOS/order, measured-cost, cross-modal attention, and nonstory comparisons are in Appendices I.7–I.10, Tables 18, 20, 21, and 22, respectively.

W1 + Q1. Does the joint-decoding advantage hold for actual latency, FLOPs, and memory?

Appendix I, Table 30 of the revised manuscript shows the low-budget quality pattern. We measure forward calls, Transformer FLOPs, end-to-end latency, and peak memory. Budget denotes total denoising steps across passes.

Budget (NFE)MethodForward callsTransformer TFLOPs ↓Latency, s ↓Peak GiB ↓Paper Overall ↑
128Joint (ours)1595642.3210.54317.487.93
128MA-Text1605677.8110.61217.487.91
256MA-Text28310042.6218.39817.487.95

Overall scores are from Table 30 of the revised manuscript. Costs are measured on two inputs on one B200, with fixed token lengths allocated to text, image and speech.

Joint-128 reduces measured latency by 42.7% and Transformer FLOPs by 43.8% relative to MA-Text-256, while their reported Overall scores are 7.93 and 7.95. At equal 128-step budgets, the methods have nearly equal measured costs. The separate quality-evaluation results and additional settings are in Appendix I.8.

W2. Does cross-modal attention directly support mutual refinement?

We use the same checkpoint, 64 inputs, seeds, and refinement budgets, and block attention connections in both directions between one generated modality and the other two while retaining input context. All conditions use a 128-step budget.

ConditionText BERT ↑Text judge ↑Image GT CLIP ↑Image judge ↑Speech WER % ↓
Full joint (ours)0.93427.6290.82886.49631.32
Block text connections0.91087.1600.80576.01667.09
Block image connections0.92227.2230.79945.88942.04
Block speech connections0.91857.0840.81856.15262.64

Full joint obtains the best mean on every displayed metric. Disconnecting image lowers its CLIP from 0.8288 to 0.7994 and judge score from 6.496 to 5.889; disconnecting text or speech increases speech WER from 31.32 to 67.09 or 62.64. Speech WER compares the utterance extracted from each condition’s generated text with the ASR transcript of its generated audio. These attention-blocking interventions show that cross-modal communication during denoising contributes substantially to final joint-generation quality, directly supporting the importance of cross-modal conditioning.

Comparison with a frozen anchor. The Modality Anchoring (MA) comparison already reported in the manuscript provides a token-freezing perspective: MA generates an anchor modality first, keeps its tokens fixed, and generates the remaining modalities in a second pass. With the same checkpoint and total NFE budget across passes, Joint outperforms MA-Text in Overall at 16, 32, 64, and 128 NFE: 7.71/7.57, 7.76/7.62, 7.86/7.80, and 7.93/7.91, respectively (revised manuscript Table 30). Together with the attention intervention, this comparison supports the benefit of generating and refining modalities jointly.

W3 + Q2. How does staged training compare with mixed-modality training under the same budget?

This new comparison complements the capability-recovery controls highlighted as a strength by Reviewer NJyB. Full-scale adaptation required 63,425 H100 GPU-hours. To compare staged and mixed training within the rebuttal period, we use a reduced-data control with the full 8B backbone. Both use the same MMaDA [1] initialization and new speech vocabulary, with matched data, training batches, and 6,144 total updates. Staged uses 4,096 ASR/TTS alignment updates corresponding to Stage 1, followed by 2,048 multimodal integration updates corresponding to Stage 2 without model merging, totaling 6,144 updates. The mixed condition mixes all modalities from the start and trains for 6,144 updates. The control uses 256 speech–transcript pairs from the EMOVA Alignment 7M LibriSpeech–AISHELL-2 training subset, alongside Flickr8k image-caption examples for captioning and text-to-image, and language replay examples.

AuthorsOpenReview ↗

Response to Reviewer oTTR (2/3)

ConditionASR WER % ↓TTS WER % ↓Caption BLEU ↑Text BLEU ↑Image CLIP-T ↑Image CLIP-I ↑
Mixed AlwaysOn80.1914.7811.9113.770.31600.6981
Staged AlwaysOn71.2916.419.1330.750.33820.7431
Staged Off→On (ours)64.2715.5710.7727.590.33980.7435

For captioning, text-only generation, and image generation, all conditions are evaluated on the same 32 inputs per task; text BLEU measures GSM8K solutions. Holding EOS supervision fixed to AlwaysOn isolates the effect of staging. Staged training reduces ASR WER by 8.90 percentage points and more than doubles text BLEU (13.77→30.75), while improving both image CLIP scores. These gains span newly introduced speech recognition and inherited text/image tasks, supporting the alignment-to-integration order for balancing modality acquisition with existing capabilities. Adding the proposed Off→On schedule further reduces ASR WER to 64.27% and TTS WER to 15.57%. Together, these results support our staged design: aligning the new modality first, then consolidating inherited and newly acquired capabilities during integration, with EOS supervision introduced after content grounding.

W4. Do automatic metrics capture coherence, cross-modal consistency, and perceptual quality?

Omni-StoryBench [2] evaluates the coherence and alignment of generated text, image, and speech through its Omni judge score. Joint decoding achieves higher mean Omni scores than MA-Text at matched 16/32 NFE: 6.32/6.51 versus 6.13/6.18 (Table 30), supporting improved cross-modal consistency at low budgets.

For speech perceptual quality, our TTS UTMOS is 4.049 on the 1–5 predicted-MOS scale (95% CI [3.984, 4.114], four decoding seeds; Table 15). This complements our independent-ASR spoken-content consistency check with an automatic perceptual-quality measure; missing outputs remain in the evaluation.

The benchmark’s human validation covers 24 annotators, 32 systems, and 687 scored evaluations, with an integrated system-ranking Spearman correlation of 0.884 (Appendix F.5).

W5 + Q3. How well does joint generation generalize beyond stories and procedural training content?

We evaluate the unchanged model on RecipeQA cooking [3] and VisualHow [4] DIY installation/repair and sports/body movement, with 32 inputs per domain; neither dataset was included in our joint-generation training data. Dynin-Omni natively generates text, image, and speech jointly in a single masked-diffusion process, without handing off generation to external modality-specific models. We compare it with MMaDA + Parler [5], Qwen2.5-Omni + FLUX.1 Kontext [6, 7], and Omni-Diffusion [8], which also uses a unified diffusion backbone. Given the current step, each model generates the next step’s text, image, and speech. For VisualHow, the next action title, target sentence, and image are withheld.

DomainModelText BERTScore ↑Image GT CLIP ↑Text judge ↑Image judge ↑Speech consistency ↑ /10Auxiliary overall ↑
RecipeDynin-Omni (ours)0.83780.62734.0863.9925.7505.696
RecipeMMaDA + Parler0.13750.07840.8280.1561.5000.929
RecipeQwen + Kontext0.81990.67123.7586.1889.1256.796
RecipeOmni-Diffusion0.73660.51181.8911.5088.1564.808
DIYDynin-Omni (ours)0.86220.68381.3835.1565.8445.569
DIYMMaDA + Parler0.67770.33861.4610.8987.4693.998
DIYQwen + Kontext0.82320.69303.9536.0789.2506.889
DIYOmni-Diffusion0.85180.49971.3671.0707.9694.784
MovementDynin-Omni (ours)0.85990.69272.1334.6485.4385.549
MovementMMaDA + Parler0.78560.38961.1720.9459.2194.618
MovementQwen + Kontext0.70610.68793.6805.1889.7506.511
MovementOmni-Diffusion0.84810.49101.6251.3058.6885.002
AuthorsOpenReview ↗

Response to Reviewer oTTR (3/3)

Auxiliary overall is the mean of 10×BERTScore, 10×image CLIP, text judge, image judge, and speech semantic consistency; it is not the official Omni-StoryBench Overall because these domains lack reference labels for speech emotion, speaking rate, pitch and gender. Speech consistency compares the utterance extracted from generated text with an independent ASR transcript of the generated audio; missing audio or unextractable utterance text scores zero.

Dynin-Omni achieves the highest text BERTScore in all three domains. Its auxiliary overall is 5.696/5.569/5.549 on Recipe/DIY/Movement, above Omni-Diffusion's 4.808/4.784/5.002 and MMaDA + Parler's 0.929/3.998/4.618; Qwen + Kontext scores 6.796/6.889/6.511. These results extend our native joint-generation evidence beyond stories: a single backbone produces text, images, and speech together without external generator orchestration, with higher auxiliary Overall than Omni-Diffusion and MMaDA + Parler in all three evaluated domains. The results demonstrate competitive cross-domain transfer while retaining the advantages of native multimodal generation.

References

[1] Yang et al., MMaDA: Multimodal Large Diffusion Language Models, arXiv 2025. https://arxiv.org/abs/2505.15809

[2] Hyeon et al., What Comes Next? Omni-StoryBench for Evaluating Story-Grounded Omnimodal Generation, arXiv 2026. https://arxiv.org/abs/2609.37317

[3] Yagcioglu et al., RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes, EMNLP 2018. https://aclanthology.org/D18-1166/

[4] Yang et al., VisualHow: Multimodal Problem Solving, CVPR 2022. https://www-users.cse.umn.edu/~qzhao/visualhow.html

[5] Lyth and King, Natural Language Guidance of High-Fidelity Text-to-Speech with Synthetic Annotations, arXiv 2024. https://arxiv.org/abs/2402.01912

[6] Xu et al., Qwen2.5-Omni Technical Report, arXiv 2025. https://arxiv.org/abs/2503.20215

[7] Black Forest Labs et al., FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space, arXiv 2025. https://arxiv.org/abs/2506.15742

[8] Li et al., Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion, ICML 2026. https://proceedings.mlr.press/v306/li26dv.html

AuthorsOpenReview ↗

Follow-up on additional experiments and revised manuscript

Dear Reviewer oTTR,

Thank you again for your thoughtful review. We would like to follow up on our response and provide a brief guide to the additional evidence addressing your questions.

  1. Actual decoding efficiency (W1 + Q1; Response 1/3). We added measurements of forward calls, Transformer FLOPs, latency, and peak GPU memory in Appendix I.8, Table 20, complementing the quality–budget comparison in Table 30.

  2. Cross-modal interaction (W2; Response 1/3). We added attention-blocking interventions in Sec. 5.2, Table 6, directly testing the contribution of cross-modal communication to joint-generation quality. Additional directional analyses appear in Appendix I.9, Table 21. Our response also explains how the existing Joint–MA comparison provides a complementary frozen-anchor comparison.

  3. Staged versus mixed training (W3 + Q2; Responses 1/3 and 2/3). We added a matched training comparison covering speech, text, and image metrics in Appendix I.7, Table 18.

  4. Evaluation quality (W4; Response 2/3). We provided additional evidence on cross-modal consistency and speech perceptual quality, including Table 15, and clarified the benchmark’s human-validation results.

  5. Generalization beyond stories (W5 + Q3; Responses 2/3 and 3/3). We added evaluations on cooking, DIY, and movement instructions, summarized in Sec. 5.2, with full comparisons in Appendix I.10, Table 22.

We hope these additions help address your concerns and make the contributions of our training recipe and native joint-generation system clearer. We would greatly appreciate hearing whether the new evidence resolves your concerns. If any questions remain, we would be happy to clarify them during the discussion.

Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Findings and Manuscript Revisions

Dear Area Chair and Reviewers,

We sincerely thank you for your thoughtful reviews and constructive feedback. We particularly appreciate your recognition of the following strengths:

  • Broad unified capabilities: A single model supports text, image, video, and speech tasks with competitive performance alongside joint generation (Reviewers sb2y and NJyB).
  • Native interleaved generation: Text, image, and speech are generated within one masked-diffusion backbone without separate output generators (Reviewers oTTR and NJyB).
  • Capability recovery and termination control: Staged training restores inherited capabilities, while merging and EOS controls support capability preservation and improved termination (Reviewers oTTR and NJyB).
  • Low-budget joint decoding: Same-checkpoint comparisons show the benefits of joint generation at lower sampling budgets (Reviewer oTTR).
  • Overlap-filtered evaluation: The joint-generation comparison remains favorable after filtering potential training overlap (Reviewer NJyB).

Following our responses, Reviewer NJyB confirmed that the additional comparisons, controls, and analyses substantially addressed their main concerns, with no further substantive questions. Reviewer sb2y likewise confirmed that the whole-path EOS and matched post-merging controls addressed their main experimental concerns. We appreciate that both reviewers raised their recommendations.

We addressed the reviewers’ main concerns with new experiments and revisions to the manuscript:

  1. Training recipe and vocabulary merging. We added whole-path comparisons against AlwaysOn and Omni-Diffusion’s attenuation rule. Off→On achieves the lowest ASR and TTS WER among the three staged schedules. We also compared three vocabulary-merging rules before and after matched post-training; modality-disentangled merging achieves the best point estimates on all six metrics. These results strengthen the evidence for our adaptation recipe (Reviewer sb2y, W1 + Q1 and W2; Reviewer NJyB, W1 + Q1–Q2). The EOS comparison appears in Sec. 5.1, Table 5, with additional controls in Appendix I.7, Tables 18–19.
  2. Staged versus mixed training. We added a matched comparison covering speech, text, and image tasks. Under the shared AlwaysOn policy, staged training improves ASR, text generation, and image generation over mixed training, supporting the alignment-to-integration order (Reviewer oTTR, W3 + Q2). The comparison appears in Appendix I.7, Table 18.
  3. Cross-modal interaction during joint generation. We added attention-blocking interventions using the same checkpoint, inputs, and generation budgets. Full joint generation achieves the best mean across all reported text, image, and speech metrics, supporting cross-modal communication during denoising. We also explained the complementary frozen-anchor perspective of the Joint–MA comparison (Reviewer oTTR, W2). These analyses appear in Sec. 5.2, Table 6, and Appendix I.9, with decoding comparisons in Table 30.
  4. Actual efficiency and uncertainty. We measured forward calls, Transformer FLOPs, latency, and peak GPU memory. Joint-128 achieves similar reported quality to MA-Text-256 with lower measured latency and FLOPs, while paired analysis supports a quality gain at 16 NFE. We also recomputed the reasoning bootstrap intervals and clarified the Figure 6 shaded bands (Reviewer oTTR, W1 + Q1; Reviewer NJyB, W2 + Q3). These additions appear in Appendix I.8, Table 20, Table 15, and Appendix G.3.
  5. Evaluation quality, generalization, and baselines. We added evidence on cross-modal consistency and speech perceptual quality and clarified the benchmark’s human validation (Reviewer oTTR, W4). On cooking, DIY, and movement instructions, the unchanged Dynin-Omni achieves the highest text BERTScore among four systems and higher auxiliary Overall than Omni-Diffusion and MMaDA + Parler across all three domains (Reviewer oTTR, W5 + Q3). We also added Omni-Diffusion comparisons on shared understanding, speech, and reasoning tasks (Reviewer NJyB, W1 + Q1). These results appear in Sec. 5.2, Table 15, and Appendices I.10–I.11, Tables 22–23.

We also revised Sec. 1 to clarify our two complementary contributions: a capability-preserving adaptation recipe combining scheduled EOS supervision with modality-disentangled merging, and native interleaved text–image–speech generation within a single masked-diffusion backbone and reverse process (Reviewer sb2y, W1 + Q1).

Together, these revisions strengthen the evidence for broad unified capabilities, the adaptation recipe, cross-modal communication, and efficient joint generation across domains. We believe they make a strong case for acceptance and hope the reviewers and Area Chair will consider the strengthened evidence and overall contribution in their final deliberations.

Thank you again for helping us improve the work. We would be happy to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.