- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper explores how computation should be organized inside the single forward pass of a one-step distilled diffusion model and proposes DeMoE. It replaces the dense FFNs of base model (FLUX.2-4B) with one always-active shared slice and seven routed slices. It uses a router conditioned on the pre-FFN hidden state and the text embedding to select the top-4 routed slices per token. It designs two further components, an optional fast-weight depth-recurrent state that feeds the router, and an auxiliary loss that predicts the executed routed response from a stop-gradient target. Training with the DMD2 objective, the paper's 7E4A1S configuration reaches similar GenEval compared with the 50-step dense teacher.
Critical strengths
- The paper focuses on how a one-step student’s computation is organized, which is a relatively under-explored angle.
- The proposed DeMoE 7E4A1S configuration reaches similar GenEval compared with the 50-step dense teacher.
Critical weaknesses
- The core structure of the DeMoE is a standard shared-expert MoE conversion. Nothing in it depends on the number of sampling steps, and it could be applied unchanged to a multi-step model. There seems no connection between the MoE design and one-step distillation demonstrated in this paper.
- There is no comparison with previous methods (e.g. other one-step methods or other diffusion MoE methods like Dense2MoE, cited in the paper). All comparisons are against the authors’ own dense variants.
- The proposed 7E4A1S is slower and uses more memory compared with the paper's efficiency claim.
- There are questionable results reported in this paper. 3E1A1S and 7E3A1S both report 3.45B active parameters, while their active FFN shares are 0.500 and 0.740.
- Presentation of this paper needs improvement. Figure 1 takes up about half of page 2 but is never referenced in the text. Except the one-step DMD, the five paradigms it shows are neither used nor compared against. The introduction describes “Figure 2(a)” as illustrating three paradigms, but the actual Figure 2(a) shows a standard DiT block. Appendix Figs. 7–13 are confusing.
Questions for authors
N/A. See the critical weakness.