- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies sparse capacity scaling for pixel-space diffusion. It combines LLM-style token-choice MoE, shared experts, and bias-based load balancing with hierarchical flow matching and stage-wise endpoint prediction. Experiments on ImageNet at 256 and 512 resolutions show improved FID over controlled dense baselines and benefits from increasing the expert pool at a fixed active-parameter budget.
Critical strengths
-
The controlled sparse-scaling experiments are a strength. The dense baselines share the hierarchical flow setup, prediction head, optimizer, and training recipe. Table 1 shows consistent improvements across three scales, including 2.95 to 1.78 FID at the Large scale with comparable active parameters. Table 2a further shows improvements when the expert pool grows while active parameters remain fixed. These results provide useful evidence that the proposed recipe benefits from sparse capacity.
-
The method achieves competitive image quality without a learned tokenizer, representation alignment, or perceptual supervision, reporting FID 1.49 at 256 resolution and 1.56 at 512 resolution. This is a useful result for self-contained pixel-space generation.
Critical weaknesses
-
The incremental contribution over the cited prior work remains limited and insufficiently distinguished. JiTMoE (Liu et al., 2025) already applies LLM-style shared and routed experts with token-choice routing and bias-based balancing to pixel-space diffusion; multi-resolution flow paths and scale transitions are established in Pyramidal Flow Matching and PixelFlow. Although the present configuration produces substantially stronger results, its comparison with JiTMoE also changes patch size, backbone and expert configuration, prediction head, optimizer, and training duration. The controlled dense comparisons establish the value of sparse capacity within this recipe, but do not clearly identify the new design insight responsible for the advance over earlier recipes. The additional hierarchical-flow improvement in Table 9b is small (2.54 to 2.46). I find the methodological and conceptual advance less compelling than the empirical performance.
-
The practical resource trade-off is insufficiently quantified. The paper reports active parameters and training epochs, but not FLOPs, wall-clock cost, throughput, or memory. In particular, the 2.8x convergence comparison in Figure 5 measures epochs. The MoE model stores substantially more parameters and incurs routing overhead, so these proxies do not establish the end-to-end cost advantage. A representative comparison at matched generation quality would help determine the practical significance of this otherwise incremental design.
Questions for authors
-
What is the principal new design insight beyond the cited JiTMoE recipe, and which controlled evidence establishes its contribution? In particular, how should we distinguish the effect of the MoE configuration from changes to token granularity, the backbone, and the training recipe?
-
For a representative scale such as Large, what are the training time and peak memory required to reach a matched FID, and the sampling throughput, relative to the controlled dense baseline? Please specify the hardware and the final CFG, guidance interval, and EMA settings.