- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes REACT, an MoE Upcycling method for DiTs. The method constructs timestep-centered token clusters from frozen denoising trajectories and uses cluster-specific neuron responses to assemble routed experts. The corresponding cluster centroids initialize the router, while global co-activation statistics determine the shared expert. The converted model is subsequently trained through layer-wise reconstruction and end-to-end diffusion adaptation. On ImageNet-256, REACT achieves generation quality comparable to the dense DiT while reducing active FFN computation and improving sampling efficiency.
Critical strengths
- The proposed method offers an intuitive approach to dense-to-sparse DiT conversion by jointly considering expert construction and router initialization.
- REACT demonstrates practical inference efficiency, achieving faster sampling while largely preserving the generation quality of the pretrained dense model. It also consistently outperforms random initialization across the evaluated recovery checkpoints.
Critical weaknesses
- The experiments primarily compare REACT against random initialization variants, despite existing dense-to-MoE conversion methods for diffusion models. In particular, although the authors cite Dense2MoE (ICCV 2025), they do not include it as an experimental baseline. While Dense2MoE is evaluated on a different architecture and generation task, its Taylor-importance-based expert construction provides a relevant alternative. A comparison under the same DiT backbone and recovery setting would better establish the effectiveness of REACT beyond random initialization.
- The analysis of expert specialization is limited. Figure 4 reports routing coherence and expert utilization only for REACT, without comparisons against other initialization baselines. It is therefore unclear whether the observed patterns are specific to the proposed method. More generally, the paper lacks comparative representation-level analysis that could explain how response-based expert assembly influences learned representations or functional expert specialization, beyond improvements in generation quality and inference efficiency.
- It remains unclear whether REACT's advantage persists with sufficient recovery training. Figure 3 shows that the FID gap between REACT and random initialization generally decreases as training progresses. For the DiT-M source, the gap decreases from 3.02 at 100K steps to 1.66 at 500K steps, while both methods continue to improve. Longer recovery experiments would help clarify whether REACT provides a sustained advantage or primarily accelerates early-stage optimization. Furthermore, given the substantial additional training required for upcycling, a comparison with continued training of the dense DiT under a comparable compute budget would help contextualize the marginal generation quality improvements.
- The motivation for the neuron replication and allocation limits is insufficiently justified. REACT limits each pretrained neuron to at most two copies and imposes additional per-expert allocation constraints, resulting in at least 56.25% of routed expert slots being initialized as zero-output padding in REACT-M32. Although the paper explains how these padded neurons are initialized and trained, it remains unclear why introducing substantial newly initialized capacity is preferable to reusing more pretrained neurons. Further justification and sensitivity analysis of these design choices would strengthen the proposed method.
Questions for authors
- Could the authors compare REACT against existing diffusion upcycling approaches, particularly Dense2MoE, under a comparable backbone and recovery setting?
- How do the routing coherence and expert utilization results in Figure 4 compare with those of the random initialization baselines?
- Is there additional representation-level evidence that REACT promotes meaningful functional expert specialization?
- Does REACT's performance advantage persist with longer recovery training? How much of the generation quality improvement could be achieved by continued training of the dense model under a comparable compute budget?
- What motivates the replication limit of two and the per-expert allocation constraints? Have the authors examined how greater pretrained neuron reuse affects reconstruction quality or downstream generation performance?
Dense2MoE and REACT use different parameter budgets and pursue different design goals. Dense2MoE partitions the original FFN neurons into non-overlapping experts without expanding the FFN's parameter capacity. Each neuron belongs to only one expert, reducing inference cost through sparse activation. REACT expands the total capacity and allows the same pretrained neuron to be copied into multiple experts. This gives the model more capacity to improve generation quality while keeping inference sparse. Given these differences in parameter budgets and design goals, Dense2MoE and REACT are not directly comparable in their original formulations.
Figure 4 examines whether the recovered REACT model uses its experts and groups similar inputs together. All 32 experts receive traffic, with usage ranging from 1.477% to 4.333% and a coefficient of variation of 0.161. Among the eight analyzed experts, mean within-expert similarity is 0.571, compared with 0.460 across experts. The figure therefore documents coherent routing and broad expert use in the analyzed checkpoint, while Tables 3–4 provide the generation-quality comparisons with random initialization. (Section 4.4, Figure 4)
REACT gives each expert a concrete starting point: pretrained neurons selected for their responses to a particular group of trajectory inputs. The router is initialized from that same group. We evaluate this design through both generation results and routing behavior. The full method outperforms the tested initialization controls, including the no-centering, random-router variant with the same experts, and Figure 4 shows within-expert input coherence alongside broad expert use. Together, these results support the practical value of constructing experts and initializing their routing from the same contexts. (Section 3.3, Tables 3–4, Figure 4)
Faster recovery matters when converting a pretrained model under a limited training budget. REACT's advantage remains throughout the reported 100K–500K interval: at 500K steps, FID is 8.60 versus 10.26 for the M source and 2.54 versus 2.78 for the XL source. For the M source, REACT at 300K steps already reaches 9.61 FID, better than the random baseline's 10.26 at 500K. The benefit therefore extends beyond the first stages of recovery and gives better quality at the evaluated training budgets. (Table 3)
Our main goal is to show that REACT can achieve generation quality close to the dense model with substantially fewer active parameters. REACT-M32 achieves an FID of 2.21, compared with the published dense reference of 2.27, while activating only 37.5% of the dense FFN parameters and sampling 1.47× faster in our measurements. The main result is this combination of near-dense quality and lower inference cost, rather than the small FID difference. (Tables 1–2)
The two-copy limit allows a pretrained neuron to serve more than one context while capping repeated reuse across experts. The allocation limits also bound how many inherited neurons each expert receives. The remaining slots are trainable units with zero output at initialization: they start without changing the expert's output and can learn during recovery. Thus, the model combines selected pretrained computations with additional capacity for adaptation. The reported padding fraction describes the initialization, not permanently inactive capacity. The full parameter counts and recovery results include these trainable units. (Sections 3.1 and 3.3, Appendix A.1–A.4)