acceptodds
Under review as a conference paper at ICLR 2027

Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Models

Abstract

Unified multimodal diffusion large language models (dLLMs) offer a single architecture for both image generation and multimodal understanding, but their iterative decoding requires tens to hundreds of forward passes. Existing few-step distillation methods focus solely on either image generation or text generation, making it unclear how to compress a fully discrete multimodal dLLM into a single efficient student while preserving both generation and understanding. We introduce **Omni-Diffusion-Distill**, a unified two-stage distillation framework that retains strong generation and understanding capabilities while substantially reducing the inference cost of a unified multimodal dLLM. Omni-Diffusion-Distill aligns the distillation of both generation and understanding, for both images and text, in the discrete token space. In the first stage, the student is trained to skip decoding steps by replaying cached teacher trajectories, and in the second stage the student is refined on intermediate states along its own rollouts. We further remedy two sources of degradation in unified distillation with a pairwise collision penalty that reduces repetition under parallel text decoding, and entropy-matched guidance that prevents entropy collapse caused by fitting the sharpened teacher distribution in image generation. Omni-Diffusion-Distill achieves state-of-the-art trade-offs between decoding efficiency and generation and understanding performance for multimodal dLLMs, reducing image generation from to decoding steps and multimodal understanding from to , giving and wall-clock speedups. Under these budgets, it scores **0.828** on GenEval and **83.0** on DPG-Bench for text-to-image generation, while reaching GPT judge scores of **20.0** on MM-Vet and **57.2** on COCO captioning (twice the teacher's 28.4 at the same steps) for multimodal understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Large Language Models, Unified Multimodal Model, Few-Step Distillation, Generation and Understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_TyYvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Omni-Diffusion-Distill. It mainly focuses on the few-step distillation of unified multimodal dLLM. As most of the work only focuses single target, like generation or understanding, this paper can distill both the generation and understand ability to the students. It also proposes a two stage distillation (off/on policy, respectively) to better learn from teacher as well as decreasing the rollout distribution mismatch.

Critical strengths
  1. During the second stage, the authors deploy different distillation target, e.g. DMD-like for image and on-policy knowledge distillation for text, according to their feature and gain a good performance in benchmarks.

  2. This paper proposes Entropy-Matched Guidance & Pairwise Collision Penalty, which relieve the entropy collapse & repeat words in image & text branch, respectively.

Critical weaknesses
  1. In Table 19, the authors compare stage 1-only, stage 2-only, and the full two-stage pipeline. But the authors doesn't clarify whether these settings are compared under the same training compute budget. According to Table 7, teacher trajectory collection and stage 1 introduce substantial additional computational costs. Could the authors compare the full method against stage 2-only training with the same budget to show the necessity of stage 1?

  2. Each training stages have many loss terms and many of them are adapted from existing distillation methods. This make the whole system complex and the necessity of each part is not well justified. And authors didn't analyse those loss coefficients' sensitivity. Could you provide more loss ablations results?

Questions for authors

Please see my weakness.

2 comments

We sincerely thank Reviewer TyYv for the time and effort in reviewing our paper. We also thank the reviewer for recognizing that our modality-specific targets, entropy-matched guidance and collision penalty effectively address entropy collapse and repetition.

W1: Stage 1 under the same training budget & necessity of Stage 1

We would like to make three points.

  • More Stage-2 training would not close the gap. Each stage has its own schedule, and Stage-2-only training has already converged under its schedule, so giving it the Stage-1 budget would not close the gap to the full method.
  • Stage 1 provides what on-policy training cannot. Stage 2 learns from the student's own few-step rollouts. Started from the teacher, these rollouts are poor, so on-policy refinement starts from bad states. Stage 1 gives Stage 2 a good starting point by replaying the teacher's own trajectories. Table 19 runs the same Stage-2 schedule with and without this initialization: adding Stage 1 improves every benchmark.
  • Most of the extra cost is one-time. The teacher trajectories are collected once and reused by every Stage-1 run, and Stage 1 itself costs less than Stage 2 because it keeps only the student in memory, while Stage 2 keeps four models.

W2: Necessity of each loss term and coefficient sensitivity

For Stage 1, we removed each term in turn, with the same number of updates:

BenchmarkFullw/o Ltraj\mathcal L_{\rm traj}w/o Lsc\mathcal L_{\rm sc}w/o Lpath\mathcal L_{\rm path}w/o Lgt\mathcal L_{\rm gt}
Image generation
GenEval0.7400.7150.7340.7400.750
ImageReward0.830.780.730.780.85
DPG-Bench68.1967.2167.3767.3567.68
UniGenBench0.5850.5790.5750.5780.580
OneIG0.2820.2840.2850.2840.283
I2I benchmark2.102.102.062.082.10
Understanding
MM-Vet17.018.017.218.619.0
Vibe-Eval7.347.345.956.516.23
LLaVA-Bench15.016.314.715.014.8
COCO36.1542.5734.1336.6533.63
DeCapBench12.5211.7711.5812.2413.09
Benchmark score ranked top-285157

Each term contributes to Stage 1. The four terms supervise different signals (the teacher's predictive distributions, consistency across skipped steps, intermediate states and real data), so they are complementary: removing any one of them lowers 5 to 9 of the 11 benchmarks, and each removal hurts a different capability.

In Stage 2, every component we introduce is already ablated in the paper: removing the collision penalty lowers COCO from 57.2 to 49.8 and raises Loop@5 from 7.7% to 21.6% (Tables 18 and 20); replacing the sequential target with the one-step marginal lowers COCO from 49.8 to 47.3 (Table 20); and removing entropy-matched guidance lowers the I2I benchmark from 2.31 to 2.04, while a fixed temperature reaches at most 2.03 (Table 21).

Coefficient sensitivity. Our method does not rely on finely tuned coefficients.

  • First, a single set of weights (Table 6) is used for all four tasks and both modalities. We also transfer it unchanged to MMaDA-8B, a different teacher with a different image tokenizer, where the student still outperforms the same-budget teacher on all benchmarks.
  • Second, the weights have a simple structure. Apart from λcol\lambda_{\rm col}, the main teacher-matching term of each stage has the largest weight (λtraj=1.0\lambda_{\rm traj} = 1.0, λmatch=0.5\lambda_{\rm match} = 0.5), the auxiliary terms have smaller weights (0.2–0.5), and the ground-truth anchor has 0.1 in both stages.

Thanks for the clarification and the new analyses. However, I think my original score is appropriate given the overall novelty and limitations of the study.

Reviewer_Hp6jOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Omni-Diffusion-Distill, a unified two-stage distillation framework that retains strong generation and understanding capabilities while substantially reducing the inference cost of a unified multimodal dLLM. The authors align the distillation of both generation and understanding, for both images and text, in the discrete token space. In the first stage, the student is trained to skip decoding steps by replaying cached teacher trajectories, and in the second stage the student is refined on intermediate states along its own rollouts. They further remedy two sources of degradation in unified distillation with a pairwise collision penalty that reduces repetition under parallel text decoding, and entropy-matched guidance that prevents entropy collapse caused by fitting the sharpened teacher distribution in image generation. The student also adopts block-causal attention over text outputs, so the KV cache of finished blocks can be reused exactly at inference.

Critical strengths
  • The problem is well chosen and clearly motivated. Compressing a fully discrete unified dLLM has to shorten image decoding and text decoding at once, and the two failure modes the authors identify- repetition under parallel text decoding and entropy collapse from the sharpened guidance target- are real and specific to this setting.

  • Both remedies are validated with direct diagnostics rather than benchmark scores alone. Loop rate, Rep-1, Rep-4 and Distinct-1 on the text side and predictive entropy on the image side show that each component moves the quantity it was designed to move.

  • The evaluation is broad and honest. Seventeen benchmarks across four tasks, three decoding budgets per task, a fair NFE accounting that counts the teacher's extra guidance passes, and a clear statement that editing still trails the few-step teacher.

  • The baselines are set up carefully. All distilled methods share the teacher, the initialization, the training mixture, the number of optimizer updates, and the sampler, and a single checkpoint is evaluated at every budget with no budget-specific retraining.

  • The ablations go beyond component toggling. Varying the scored state, the teacher prediction form, and the repetition term isolates why the sequential conditional target and the pairwise penalty matter, and the fixed-temperature comparison shows the gain comes from per-position entropy matching rather than from softening the target.

Critical weaknesses
  • Novelty is mostly recombination. Off-policy trajectory replay, on-policy refinement on student rollouts, a fake-score auxiliary network in the style of distribution matching, and a preference term borrowed from T3D all have clear antecedents. The genuinely new content is the collision penalty and entropy-matched guidance. The manuscript should state this boundary more plainly instead of framing the whole framework as a unified contribution.

  • The Stage-1 objective has four weighted terms, but only the presence of the stage is ablated. There is no per-term ablation of the trajectory, self-consistency, path and ground-truth anchor terms, so it is unclear which of them carry the cold start and whether four terms are needed.

  • Cross-task interference is never measured. The paper asks whether image and text distillation can be unified, but there is no comparison of the joint student against per-task distilled students. Nothing rules out the possibility that one joint student is worse than two specialists, which would undercut the central motivation.

Questions for authors
  • How much of the understanding speedup comes from distillation and how much from the block-causal mask and exact KV reuse alone? A block-causal student trained without Stage 2 and a bidirectional student trained with it would separate the two.

  • What happens with a second fully discrete unified teacher, or with a smaller variant of the same one?

  • Does the joint student beat two per-task distilled students at equal training cost?

2 comments
AuthorsOpenReview ↗

Response (1/2)

We sincerely thank Reviewer Hp6j for the detailed review. We also thank the reviewer for recognizing that the problem is well motivated, that both remedies are validated with direct diagnostics, and that the baselines and ablations are carefully designed.

W1: Novelty is mostly recombination

We thank the reviewer for recognizing the pairwise collision penalty and entropy-matched guidance as genuinely new. More broadly, our contribution is to make few-step distillation of a fully discrete unified dLLM work, whereas naively porting or combining existing objectives fails.

  • First, the problem is new, and existing components do not solve it when combined directly. To our knowledge, no prior work distills a fully discrete unified dLLM into one few-step student for generation, editing, controllable generation and understanding. In our matched setting, each existing objective fails outside its native modality (Tables 1 and 3). The reviewer also notes that the two failure modes we address are real and specific to this setting.
  • Second, making the components work together required new design. Both modalities share one on-policy loop, with a modality-specific scored state, teacher prediction and matching term.
  • Third, the combination is not additive. Neither stage works well alone: Stage 1 + Stage 2 improves over Stage 1 alone or Stage 2 alone on every benchmark in Table 19. A well-designed combination that solves a new problem is itself a contribution. Following the reviewer's suggestion, we will mark in Section 3 which components are adopted and which are new, so that the boundary of our contribution is clear.

W2: Which Stage-1 terms carry the cold start, and are all four needed?

We added a Stage-1 ablation. Each variant removes one term and is trained for the same number of Stage-1 updates:

BenchmarkFullw/o Ltraj\mathcal L_{\rm traj}w/o Lsc\mathcal L_{\rm sc}w/o Lpath\mathcal L_{\rm path}w/o Lgt\mathcal L_{\rm gt}
Image generation
GenEval0.7400.7150.7340.7400.750
ImageReward0.830.780.730.780.85
DPG-Bench68.1967.2167.3767.3567.68
UniGenBench0.5850.5790.5750.5780.580
OneIG0.2820.2840.2850.2840.283
I2I benchmark2.102.102.062.082.10
Understanding
MM-Vet17.018.017.218.619.0
Vibe-Eval7.347.345.956.516.23
LLaVA-Bench15.016.314.715.014.8
COCO36.1542.5734.1336.6533.63
DeCapBench12.5211.7711.5812.2413.09
Benchmark score ranked top-285157

The four terms are complementary rather than redundant: each supervises a different signal, and removing any one of them lowers 5 to 9 of the 11 benchmarks. Ltraj\mathcal L_{\rm traj}, the only term that matches the teacher's full distributions, mainly drives image generation; Lsc\mathcal L_{\rm sc} keeps the student consistent when it skips ahead and matters for both modalities; Lpath\mathcal L_{\rm path} supports image quality and open-ended answers; and Lgt\mathcal L_{\rm gt} mainly supports captioning. Since individual benchmark scores fluctuate across variants, we also count how often each variant ranks in the top two. The full objective does so on 8 of the 11 benchmarks, more than any ablated variant, and it is the only one that does so on most benchmarks of both modalities. As Stage 1 initializes Stage 2 for both modalities, we keep all four terms.

AuthorsOpenReview ↗

Response (2/2)

W3 & Q3: Cross-task interference and equal training cost

We trained an image-only and a text-only student from the same Stage-1 checkpoint with the same Stage-2 recipe. The comparison is at matched total training cost, so the joint student has no cost advantage:

BenchmarkJoint (ours)Image-onlyText-only
GenEval0.8280.8110.754
ImageReward1.251.230.87
DPG-Bench83.081.569.2
UniGenBench0.7100.6870.598
OneIG0.3810.3470.286
I2I benchmark2.312.252.11
MM-Vet20.014.516.9
Vibe-Eval8.186.417.43
LLaVA-Bench16.913.914.2
COCO57.226.245.2
DeCapBench19.610.114.5

Under this matched-total-cost setting, joint refinement improves both modalities: the joint student is better than the corresponding specialist on all 11 benchmarks. Adding understanding data does not hurt image generation: the joint student is higher than the image-only student on all six image benchmarks. Adding image data does not hurt understanding either.

Q1: Speedup from distillation vs. block-causal mask and KV reuse

Our results already separate the two. Without the KV cache, the student is 6.2× faster than the teacher, which comes from fewer decoding steps. Enabling exact KV reuse on the same student, with the weights, attention mask and step count fixed, reduces the latency from 13.12 s to 3.81 s, a further 3.44×, for 21.2× in total (Table 8).

Q2: A second teacher or a smaller variant

We distill a second fully discrete unified teacher, MMaDA-8B, with the same hyperparameters and schedule. Steps are image / text decoding steps; MMaDA has no editing task:

BenchmarkTeacher (16 / 256 steps)Teacher (4 / 64 steps)Ours (4 / 64 steps)
Image generation
GenEval0.5300.3110.553
ImageReward0.42−0.690.57
Aesthetic3.663.443.58
DPG-Bench70.353.768.7
UniGenBench0.4700.3760.514
OneIG0.2970.2590.289
Understanding
MM-Vet17.417.018.1
Vibe-Eval11.7110.6912.64
LLaVA-Bench19.016.020.0
COCO42.455.557.4
DeCapBench9.616.423.5

The distilled MMaDA student outperforms the teacher at the same budget on all benchmarks, and even the teacher with 4× more steps on 8 of them. This shows that our method is not limited to a single teacher model.

Reviewer_xiP4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of efficient inference for unified multimodal discrete diffusion language models (dLLMs), which employ a single architecture for both image generation and multimodal understanding. Unlike existing few-step distillation approaches that primarily focus on either image diffusion models or text diffusion models, the paper proposes Omni-Diffusion-Distill, a two-stage distillation framework that substantially reduces inference costs for both image generation and multimodal understanding while maintaining competitive task performance. Built upon Lumina-DiMOO, an 8B unified dLLM, the proposed framework combines off-policy trajectory replay with on-policy refinement, incorporating modality-specific objectives to address the different characteristics of image and text generation. Experiments across 17 benchmarks covering text-to-image generation, image-to-image generation, and multimodal understanding demonstrate substantial inference speedups with relatively small performance degradation.

Critical strengths
  1. The paper addresses an important and timely problem in the emerging field of diffusion language models.

  2. The proposed two-stage distillation strategy is reasonable and addresses two complementary challenges. Stage 1 uses off-policy trajectory replay to transfer the teacher's multi-step denoising behavior to the student through cached teacher trajectories. Stage 2 performs on-policy refinement using student-generated rollouts, addressing the distribution mismatch between cached teacher trajectories and the states encountered by the student during inference.

  3. The evaluation is relatively comprehensive, covering 17 benchmarks across text-to-image generation, image-to-image generation, and multimodal understanding. The reported inference speedups are impressive.

Critical weaknesses
  1. The two-stage framework is reasonable but many of its individual components are closely related to existing methods. This weakens the novelty of this paper.

  2. The proposed pairwise collision penalty (Eq. 3) penalizes the overlap between the categorical probability distributions predicted at two different token positions. It seems to me that, distributional overlap does not necessarily imply that the decoded tokens will actually be identical.

  3. The paper claims to introduce a unified distillation framework for multimodal dLLMs, but all experiments are conducted on a single backbone, Lumina-DiMOO-8B.

  4. The paper demonstrates that a single student model can support both image generation and multimodal understanding, and that the proposed distillation framework can accelerate both types of tasks. However, it does not sufficiently investigate whether joint multi-task distillation provides benefits beyond enabling a single model to handle multiple tasks.

Questions for authors

Please see the weaknesses.

3 comments
AuthorsOpenReview ↗

Response (1/2)

We sincerely thank Reviewer xiP4 for the time and effort in reviewing our paper. We also thank the reviewer for recognizing that our problem is important and timely, that the two-stage strategy addresses two complementary challenges, and that the speedups are impressive.

W1: Novelty of the components

Our contribution is to make few-step distillation of a fully discrete unified dLLM work, whereas naively porting or combining existing objectives fails. Applied to the other modality, the text method CDLM reaches only 0.521 on GenEval and the image method Di[M]O only 33.4 on COCO. Two design choices make it work.

  • First, as the reviewer notes, the two stages address two complementary challenges: Stage 1 teaches the student to skip decoding steps, and Stage 2 supervises the states that the few-step student actually visits. GenEval is 0.740 with Stage 1 only, 0.771 with Stage 2 only and 0.828 with both (Table 19).
  • Second, we introduce a dedicated fix for each modality: the pairwise collision penalty cuts text Loop@5 from 21.6% to 7.7%, and entropy-matched guidance raises the final image-token entropy from 1.55 to 3.32 nats (Table 18). Other reviewers also find the method technically sound (ghsc) and the two failure modes real and specific to this setting (Hp6j).

W2: Distributional overlap vs. identical decoded tokens

We would like to clarify that the penalty acts on the probability of repetition. Since tokens committed in the same step are sampled independently, ⟨qθi,qθj⟩\langle q^i_\theta, q^j_\theta \rangle is exactly the probability that positions ii and jj decode to the same token, and −log⁡(1−⟨qθi,qθj⟩)-\log(1 - \langle q^i_\theta, q^j_\theta \rangle) is the negative log-likelihood that they differ. Like SFT cross-entropy, it supervises probabilities rather than decoded tokens, because sampling is not differentiable. Under the confidence-based (top-1) committing that our sampler uses, a small overlap also prevents two positions from both being confident about the same token, since min⁡(qθi(v),qθj(v))≤⟨qθi,qθj⟩\min(q^i_\theta(v), q^j_\theta(v)) \le \sqrt{\langle q^i_\theta, q^j_\theta \rangle} for any token vv. Empirically, the penalty reduces repetition in the decoded text: removing it raises Loop@5 from 7.7% to 21.6% and Rep-4 from 0.020 to 0.122 (Table 18).

AuthorsOpenReview ↗

Response (2/2)

W3: A unified framework validated on a single backbone

Our framework is designed for fully discrete unified dLLMs, and it uses nothing specific to Lumina-DiMOO. It only needs a shared discrete vocabulary, masked decoding and CFG for images. To support the claim beyond one backbone, we apply the framework unchanged, including all hyperparameters, to MMaDA-8B, which uses a different image tokenizer. Steps are image / text decoding steps; MMaDA has no editing task:

BenchmarkTeacher (16 / 256 steps)Teacher (4 / 64 steps)Ours (4 / 64 steps)
Image generation
GenEval0.5300.3110.553
ImageReward0.42−0.690.57
Aesthetic3.663.443.58
DPG-Bench70.353.768.7
UniGenBench0.4700.3760.514
OneIG0.2970.2590.289
Understanding
MM-Vet17.417.018.1
Vibe-Eval11.7110.6912.64
LLaVA-Bench19.016.020.0
COCO42.455.557.4
DeCapBench9.616.423.5

The framework transfers without any change and improves both modalities: at the same budget, the student is better than the teacher on all six image benchmarks and all five understanding benchmarks, and even better than the teacher with 4× more steps on 8 of them. The framework is therefore not tied to a single backbone.

W4: Benefits of joint distillation beyond serving all tasks with one model

To test this, we trained an image-only and a text-only student from the same Stage-1 checkpoint with the same recipe, and compared them with the joint student at matched total training cost:

BenchmarkJoint (ours)Image-onlyText-only
GenEval0.8280.8110.754
ImageReward1.251.230.87
DPG-Bench83.081.569.2
UniGenBench0.7100.6870.598
OneIG0.3810.3470.286
I2I benchmark2.312.252.11
MM-Vet20.014.516.9
Vibe-Eval8.186.417.43
LLaVA-Bench16.913.914.2
COCO57.226.245.2
DeCapBench19.610.114.5

Beyond serving all tasks with one model, joint refinement brings two benefits.

  • First, quality: at matched total training cost, the joint student is better than the corresponding specialist on all 11 benchmarks. This holds even for image generation, where the image-only student spends all of its updates on images: the joint student is still higher on every image benchmark. Under this setting, joint refinement therefore improves quality, not only convenience.
  • Second, practicality: one checkpoint serves all four tasks, and training needs one Stage-2 run instead of two.

Thanks for the clarification and the new results. In my initial review, this paper addresses an important and timely problem in diffusion language models. The response further addressed some of my concerns. I maintain my positive rating of 3 for weak acceptance.

Reviewer_ghscOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies few-step distillation for fully discrete unified multimodal diffusion language models. The proposed method has two stages. Stage 1 uses cached teacher trajectories to train the student to skip decoding steps. Stage 2 trains on student rollouts to reduce the train-test state mismatch. The image branch uses discrete distribution matching and entropy-matched classifier-free guidance. The text branch uses sequential teacher targets and a pairwise collision loss to reduce repetition during parallel decoding. The method distills an 8B Lumina-DiMOO model into one student for text-to-image generation, image editing, controllable generation, and multimodal understanding. The student uses 8 steps for image generation and 64 steps for understanding. The reported wall-clock speedups are 18.2x and 21.2x, respectively.

Critical strengths
  1. Technically sound and easy to follow. The proposed method is technically sound and the research problem is crucial for unified multimodal models.

  2. Strong results under matched training conditions. All baselines use the same teacher initialization, data mixture, number of updates, sampler, and decoding budgets. At 8 image steps, the proposed method leads the distilled baselines on all five T2I benchmarks in Table 1. At 64 text steps, it also leads them on all discriminative and open-ended MMU benchmarks in Tables 3 and 4. The advantage generally remains at smaller budgets in Tables 9-17.

  3. The main design choices are supported by targeted analysis. Table 18 directly measures the two failure modes. Removing the collision loss raises Loop@5 from 0.077 to 0.216. Removing entropy-matched guidance lowers the final image-token entropy from 3.32 to 1.55. Table 20 compares marginal and sequential teacher targets as well as two repetition losses.

  4. The efficiency evaluation is comprehensive. The results report both NFE and measured latency. Table 8 shows that KV caching is important for the MMU speedup. The authors also disclose the training cost, model states kept in memory, and key optimization settings. This makes the system-level claims easier to assess.

Critical weaknesses
  1. The independence of the training and evaluation data is unclear. Appendix C.1 lists Graph-200K as a training source. Table 2 and Appendix C.4 also use Graph-200K for evaluation. The paper does not specify the split or report deduplication. It is therefore unclear whether the reported 0.789 DINOv2 similarity is measured on held-out examples. The full 51K training mixture also draws from several public instruction datasets and prior Lumina-DiMOO data. A contamination check against the 17 evaluation sets is needed.

  2. Generality beyond Lumina-DiMOO is not established. All experiments use the same 8B teacher. The framework assumes a shared discrete vocabulary, masked iterative decoding, and image CFG. It is unclear how sensitive the method is to the architecture, tokenizer, or model scale. In particular, rho=0.7 and the collision-loss weight may be model-specific. The paper also does not compare joint distillation with separate image and text students. Such a comparison would reveal whether unified training causes positive transfer or interference.

  3. The state-of-the-art claim should be stated more narrowly. DiMO, CDLM, and T3D were designed for different single-modality settings and are extended to new modalities in this paper. The matched setup is useful, but the paper does not describe modality-specific tuning of these ports. Unified acceleration methods such as UniCM and Hyper-Bagel are discussed but not empirically contextualized. The experiments establish a strong result on Lumina-DiMOO. They do not yet establish a general frontier across unified multimodal models.

Questions for authors
  1. How does the unified student compare with separate image and text students trained under matched total compute? This would help quantify cross-modal transfer and the benefit of sharing one model.

  2. How were rho=0.7 and lambda_col=8 selected? Do these settings transfer to another model scale or another fully discrete unified dLLM?

3 comments
AuthorsOpenReview ↗

Response (1/2)

We sincerely thank Reviewer ghsc for the time and effort in reviewing our paper. We also thank the reviewer for recognizing that our method is technically sound, our comparisons are matched, and our efficiency evaluation is comprehensive.

W1: Independence of training and evaluation data

Thank you for pointing this out. There is no data leakage. Our training data and most of our benchmarks are the same as those used by Lumina-DiMOO (Appendix C.1 and Section 4.1), so we keep exactly the teacher's separation between training and evaluation. For Graph-200K, we use its official split: the train split for distillation and the held-out test split for evaluation. No evaluation sample is used in training.

W2 & Q2: Sensitivity to the architecture, tokenizer and hyperparameters

Our method does not depend on a specific architecture, tokenizer or model size. All losses only use the per-token categorical predictions of the teacher and the student, so a different tokenizer only changes the vocabulary, not the method. To test whether the method and its hyperparameters transfer, we distill MMaDA-8B, a second fully discrete unified dLLM with a different image tokenizer (MAGVIT-v2), reusing ρ=0.7\rho = 0.7 and λcol=8\lambda_{\rm col} = 8. Steps are image / text decoding steps; MMaDA has no editing task:

BenchmarkTeacher (16 / 256 steps)Teacher (4 / 64 steps)Ours (4 / 64 steps)
Image generation
GenEval0.5300.3110.553
ImageReward0.42−0.690.57
Aesthetic3.663.443.58
DPG-Bench70.353.768.7
UniGenBench0.4700.3760.514
OneIG0.2970.2590.289
Understanding
MM-Vet17.417.018.1
Vibe-Eval11.7110.6912.64
LLaVA-Bench19.016.020.0
COCO42.455.557.4
DeCapBench9.616.423.5

Without re-tuning any hyperparameters, our student is better than the same-budget teacher on all benchmarks and better than the teacher with 4× more steps on 8 of them. Hence, neither the tokenizer nor the two hyperparameters are specific to Lumina-DiMOO.

The two hyperparameters the reviewer mentioned (ρ\rho and λcol\lambda_{\rm col}) are defined independently of the model architecture. ρ\rho is a fraction of each model's own conditional entropy, so it does not depend on the logit scale of a particular model, and the collision penalty depends only on probabilities. Both values were set empirically in our preliminary experiments, and the results are robust around them. ρ=0.7\rho = 0.7 is the best on seven of the eight benchmarks in Table 21, and the results vary only slightly when λcol\lambda_{\rm col} changes from 8 to 5:

λcol\lambda_{\rm col}MM-VetVibe-EvalLLaVA-BenchCOCODeCapBench
520.48.0916.457.919.0
820.08.1816.957.219.6
AuthorsOpenReview ↗

Response (2/2)

W2 & Q1: Joint vs. separate students under matched compute

We trained an image-only and a text-only student from the same Stage-1 checkpoint with the same Stage-2 recipe, and compared them with the joint student at matched total training cost:

BenchmarkJoint (ours)Image-onlyText-only
GenEval0.8280.8110.754
ImageReward1.251.230.87
DPG-Bench83.081.569.2
UniGenBench0.7100.6870.598
OneIG0.3810.3470.286
I2I benchmark2.312.252.11
MM-Vet20.014.516.9
Vibe-Eval8.186.417.43
LLaVA-Bench16.913.914.2
COCO57.226.245.2
DeCapBench19.610.114.5

At matched total training cost, the joint student is better than the corresponding specialist on every benchmark, which shows positive cross-modal transfer. The clearest case is image generation: although the image-only student spends all of its updates on images, it is lower than the joint student on all six image benchmarks. Each specialist also loses much of the other ability.

W3: The state-of-the-art claim

  • More precise claim. We will state the claim more precisely as "state-of-the-art efficiency–quality trade-offs among few-step distillation methods for fully discrete unified multimodal dLLMs." We will also specify the teacher, the baselines and the decoding budgets wherever we state the main result.
  • How the baselines are applied. Each baseline keeps its published loss unchanged and is applied to the answer tokens of every task. Di[M]O's token-level distribution matching on re-masked outputs is also applied to text answers. CDLM is trained with its three-term loss on cached teacher trajectories of both modalities, with block-causal attention on text as in the original CDLM. T3D's trajectory self-distillation with its DDO loss is applied to both image and text trajectories. We do not tune any method separately for each modality.
  • UniCMs and Hyper-Bagel. They target different model types, so neither applies to a fully discrete model in which both modalities share one masked-diffusion denoiser. We will make this scope explicit in Section 2.
Method (base model)Accelerates gen. & und.Text: masked diffusionImage: discrete tokensMasked diffusion for both modalities
UniCMs (Show-o)✅❌ autoregressive (Jacobi decoding)✅❌
Hyper-Bagel (BAGEL)✅❌ autoregressive (speculative decoding)❌ continuous latents (rectified flow)❌
Ours (Lumina-DiMOO, MMaDA-8B)✅✅✅✅
Reviewer_ghscOpenReview ↗

Official Comment by Reviewer ghsc

Thanks for the authors' efforts in addressing my concerns. In my initial review, the proposed method is technically convincing and effective to me, and the detailed rebuttal addressed my further concenrs with more details and clarifications. Thus I maintain my positive rating as weak accept. Good luck!

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response (1/2)

We sincerely thank all reviewers (ghsc, cNnv, xiP4, Hp6j, TyYv) for their thoughtful and constructive feedback. We are encouraged that the reviewers recognized the value of our work in several key aspects:

  • Important problem and well-targeted method. The problem is "crucial for unified multimodal models" (ghsc), "important and timely" (xiP4), and "well chosen and clearly motivated" (Hp6j). The method is "technically sound" (ghsc), its two stages address "two complementary challenges" (xiP4), and the two failure modes it targets are "real and specific to this setting" (Hp6j); TyYv notes that our two remedies relieve them.
  • Careful and comprehensive evaluation. Reviewers highlighted "strong results under matched training conditions" (ghsc), baselines that "are set up carefully" (Hp6j), "broad validation of one deployable checkpoint" (cNnv), an evaluation that is "broad and honest" (Hp6j), and "impressive" speedups (xiP4).
  • Convincing analysis. The design choices are "supported by targeted analysis" (ghsc), both remedies are "validated with direct diagnostics rather than benchmark scores alone" (Hp6j), and the ablations go "beyond component toggling" (Hp6j).

In response to the main concerns, we provide the following new results and clarifications.

1. Generality beyond Lumina-DiMOO (ghsc W2&Q2; xiP4 W3; Hp6j Q2)

Our method does not depend on a specific architecture. It only requires a fully discrete unified dLLM, i.e., a shared discrete vocabulary, masked decoding and CFG for images. We chose Lumina-DiMOO because it is currently the best-performing model of this family and covers all four tasks. New experiment: we apply our method unchanged to MMaDA-8B, which uses a different image tokenizer. Without re-tuning any hyperparameters, the few-step student outperforms the teacher at the same budget on all benchmarks, and even the teacher with 4× more steps on 8 of them (steps are image / text decoding steps; MMaDA has no editing task):

BenchmarkTeacher (16 / 256 steps)Teacher (4 / 64 steps)Ours (4 / 64 steps)
Image generation
GenEval0.5300.3110.553
ImageReward0.42−0.690.57
Aesthetic3.663.443.58
DPG-Bench70.353.768.7
UniGenBench0.4700.3760.514
OneIG0.2970.2590.289
Understanding
MM-Vet17.417.018.1
Vibe-Eval11.7110.6912.64
LLaVA-Bench19.016.020.0
COCO42.455.557.4
DeCapBench9.616.423.5
AuthorsOpenReview ↗

General Response (2/2)

2. Benefit of joint distillation (ghsc W2&Q1; xiP4 W4; Hp6j W3&Q3)

New experiment: starting from the same Stage-1 checkpoint, we trained an image-only and a text-only student. At matched total training cost, the joint student outperforms the corresponding specialist on all benchmarks, so, under this setting, joint refinement improves both modalities, beyond serving all tasks with one model:

BenchmarkJoint (ours)Image-onlyText-only
GenEval0.8280.8110.754
ImageReward1.251.230.87
DPG-Bench83.081.569.2
UniGenBench0.7100.6870.598
OneIG0.3810.3470.286
I2I benchmark2.312.252.11
MM-Vet20.014.516.9
Vibe-Eval8.186.417.43
LLaVA-Bench16.913.914.2
COCO57.226.245.2
DeCapBench19.610.114.5

3. Novelty (xiP4 W1; Hp6j W1)

Our framework is not a mere recombination. (i) It addresses a new problem: distilling a fully discrete unified dLLM into one few-step student for both image generation and multimodal understanding. (ii) It introduces one component for each modality-specific challenge: a pairwise collision penalty for repetitive text generation under parallel decoding, and entropy-matched guidance against the entropy collapse that reduces image diversity. (iii) Its two stages are complementary: the full pipeline improves over either stage alone on every benchmark in Table 19.

For other questions, we refer to our individual responses:

  • Necessity of each Stage-1 loss term: [Hp6j W2; TyYv W2]
  • Training budget of the two stages and the role of Stage 1: [TyYv W1]
  • Matched training budgets and teacher compute: [cNnv W1&Q1]
  • Scope of the state-of-the-art claim: [ghsc W3]
  • No overlap between training and evaluation data: [ghsc W1]
  • Assumptions behind the image update: [cNnv Q2]
  • Why distributional overlap measures token repetition: [xiP4 W2]
  • Hyperparameter selection, robustness and run-to-run variance: [ghsc W2&Q2; cNnv W2&Q3&Q4; TyYv W2]
  • COCO judging protocol and timing scope: [cNnv W2&Q3&Q4]
  • Speedup from distillation vs. exact KV reuse: [Hp6j Q1]

We believe these results and clarifications show that Omni-Diffusion-Distill is not a recombination of existing parts, but a unified and effective framework for few-step distillation of fully discrete unified dLLMs that generalizes beyond a single teacher. All new results and clarifications will be added to the revision. Thank you very much for your time and effort.

AuthorsOpenReview ↗

Response (1/2)

We sincerely thank Reviewer cNnv for the detailed review. We also thank the reviewer for recognizing that our results substantiate a practical acceleration contribution and that our ablations directly address the failure modes.

W1 & Q1: Attribution of the gains and training budgets

We would like to clarify two points.

First, the training budgets are matched wherever they can be matched.

  • All methods use the same backbone, data, batch size and number of updates, so they see exactly the same training examples. They are also evaluated at the same inference budget.
  • The teacher compute per example cannot be made identical, because each method uses the teacher differently. CDLM trains only on offline teacher trajectories. T3D also uses offline trajectories, plus online samples from a periodically refreshed copy of the student for its DDO term. Di[M]O queries the teacher on every re-masked student sample, and our text branch lets the teacher complete the block from the student's own rollout. Forcing the same number of teacher calls would mean changing these methods.
  • The gains from our two new components come without additional teacher forward passes: the collision penalty raises COCO from 49.8 to 57.2 (Table 20), and entropy-matched guidance raises the I2I benchmark from 2.04 to 2.31 (Table 21).

Second, regarding a shared-backbone combination of modality-appropriate objectives, all baselines already use the same teacher, data, sampler and decoding budgets as ours, and each is a strong objective designed for its own modality. Taking the best baseline separately for each benchmark is therefore a strong proxy for such a combination, and our student is still better on all T2I and MMU benchmarks (representative ones shown below):

GenEvalDPGOneIGTIIFMME-PMMBenchMMMUMM-VetCOCODeCapBench
Best baseline per benchmark0.81080.80.3270.705145878.450.718.550.316.9
Ours0.82883.00.3810.768153789.456.320.057.219.6

Q2: Assumptions behind the image update

We follow the distribution-matching approach that is widely used for few-step distillation of image generators [1, 2, 3]. DMD [1, 2] updates the student with the difference between a "fake" score (an auxiliary network trained on student samples) and the teacher's "real" score on a noised student sample. Di[M]O [3] extends this to masked models: it replaces noising with re-masking and applies this difference to the logits that generated the sample, assuming it gives a valid update direction for them. We make the same assumption: ss is the re-masked student sample, rψr_\psi is the fake-score network, and sg[rψ−pϕ]\mathrm{sg}[r_\psi - p_\phi] updates the logits at xtx_t that produced ss. Computing both terms on xtx_t instead would reduce the update to ordinary per-position KD (Appendix B).

[1] Yin, Tianwei, et al. One-step Diffusion with Distribution Matching Distillation. CVPR 2024.

[2] Yin, Tianwei, et al. Improved Distribution Matching Distillation for Fast Image Synthesis. NeurIPS 2024.

[3] Zhu, Yuanzhi, et al. Di[M]O: Distilling Masked Diffusion Models into One-step Generator. ICCV 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.