- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces Omni-Diffusion-Distill. It mainly focuses on the few-step distillation of unified multimodal dLLM. As most of the work only focuses single target, like generation or understanding, this paper can distill both the generation and understand ability to the students. It also proposes a two stage distillation (off/on policy, respectively) to better learn from teacher as well as decreasing the rollout distribution mismatch.
Critical strengths
-
During the second stage, the authors deploy different distillation target, e.g. DMD-like for image and on-policy knowledge distillation for text, according to their feature and gain a good performance in benchmarks.
-
This paper proposes Entropy-Matched Guidance & Pairwise Collision Penalty, which relieve the entropy collapse & repeat words in image & text branch, respectively.
Critical weaknesses
-
In Table 19, the authors compare stage 1-only, stage 2-only, and the full two-stage pipeline. But the authors doesn't clarify whether these settings are compared under the same training compute budget. According to Table 7, teacher trajectory collection and stage 1 introduce substantial additional computational costs. Could the authors compare the full method against stage 2-only training with the same budget to show the necessity of stage 1?
-
Each training stages have many loss terms and many of them are adapted from existing distillation methods. This make the whole system complex and the necessity of each part is not well justified. And authors didn't analyse those loss coefficients' sensitivity. Could you provide more loss ablations results?
Questions for authors
Please see my weakness.
2 comments
Thanks for the clarification and the new analyses. However, I think my original score is appropriate given the overall novelty and limitations of the study.
Response
We sincerely thank Reviewer TyYv for the time and effort in reviewing our paper. We also thank the reviewer for recognizing that our modality-specific targets, entropy-matched guidance and collision penalty effectively address entropy collapse and repetition.
We would like to make three points.
For Stage 1, we removed each term in turn, with the same number of updates:
Each term contributes to Stage 1. The four terms supervise different signals (the teacher's predictive distributions, consistency across skipped steps, intermediate states and real data), so they are complementary: removing any one of them lowers 5 to 9 of the 11 benchmarks, and each removal hurts a different capability.
In Stage 2, every component we introduce is already ablated in the paper: removing the collision penalty lowers COCO from 57.2 to 49.8 and raises Loop@5 from 7.7% to 21.6% (Tables 18 and 20); replacing the sequential target with the one-step marginal lowers COCO from 49.8 to 47.3 (Table 20); and removing entropy-matched guidance lowers the I2I benchmark from 2.31 to 2.04, while a fixed temperature reaches at most 2.03 (Table 21).
Coefficient sensitivity. Our method does not rely on finely tuned coefficients.