acceptodds
Under review as a conference paper at ICLR 2027

REACT: Response-Based Expert Assembly From Contextual Trajectories

Abstract

Dense feed-forward networks are unaffected by how their neurons are grouped: every neuron is evaluated. Sparse execution makes this grouping consequential, because each input can access only a few experts. An arbitrary partition can therefore retain pretrained weights while separating computations that an input needs together. We argue that upcycling should organize neurons according to their responses to similar inputs. To establish this correspondence, we introduce a context-guided upcycling approach REACT, Response-based Expert Assembly from Contextual Trajectories, that uses frozen denoising trajectories to connect input contexts with pretrained neuron responses. We first identify contexts after timestep-wise centering to account for distribution shifts across noise levels, then use their associated responses to allocate intact neurons to routed experts. The same contexts initialize the corresponding router entries, aligning expert contents with their intended inputs, while global activation statistics select an always-active shared expert. Layer-wise reconstruction and end-to-end adaptation refine this initialization for sparse execution. On ImageNet-1K, our approach achieves better generation quality while activating only 37.5% of the original FFN parameters and reducing total per-token activated parameters by 27.6%.

Primary area:
generative models
Keywords:
Diffusion Transformers, Mixture of Experts, Dense-to-MoE Conversion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gsjeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes REACT, an MoE Upcycling method for DiTs. The method constructs timestep-centered token clusters from frozen denoising trajectories and uses cluster-specific neuron responses to assemble routed experts. The corresponding cluster centroids initialize the router, while global co-activation statistics determine the shared expert. The converted model is subsequently trained through layer-wise reconstruction and end-to-end diffusion adaptation. On ImageNet-256, REACT achieves generation quality comparable to the dense DiT while reducing active FFN computation and improving sampling efficiency.

Critical strengths
  • The proposed method offers an intuitive approach to dense-to-sparse DiT conversion by jointly considering expert construction and router initialization.
  • REACT demonstrates practical inference efficiency, achieving faster sampling while largely preserving the generation quality of the pretrained dense model. It also consistently outperforms random initialization across the evaluated recovery checkpoints.
Critical weaknesses
  • The experiments primarily compare REACT against random initialization variants, despite existing dense-to-MoE conversion methods for diffusion models. In particular, although the authors cite Dense2MoE (ICCV 2025), they do not include it as an experimental baseline. While Dense2MoE is evaluated on a different architecture and generation task, its Taylor-importance-based expert construction provides a relevant alternative. A comparison under the same DiT backbone and recovery setting would better establish the effectiveness of REACT beyond random initialization.
  • The analysis of expert specialization is limited. Figure 4 reports routing coherence and expert utilization only for REACT, without comparisons against other initialization baselines. It is therefore unclear whether the observed patterns are specific to the proposed method. More generally, the paper lacks comparative representation-level analysis that could explain how response-based expert assembly influences learned representations or functional expert specialization, beyond improvements in generation quality and inference efficiency.
  • It remains unclear whether REACT's advantage persists with sufficient recovery training. Figure 3 shows that the FID gap between REACT and random initialization generally decreases as training progresses. For the DiT-M source, the gap decreases from 3.02 at 100K steps to 1.66 at 500K steps, while both methods continue to improve. Longer recovery experiments would help clarify whether REACT provides a sustained advantage or primarily accelerates early-stage optimization. Furthermore, given the substantial additional training required for upcycling, a comparison with continued training of the dense DiT under a comparable compute budget would help contextualize the marginal generation quality improvements.
  • The motivation for the neuron replication and allocation limits is insufficiently justified. REACT limits each pretrained neuron to at most two copies and imposes additional per-expert allocation constraints, resulting in at least 56.25% of routed expert slots being initialized as zero-output padding in REACT-M32. Although the paper explains how these padded neurons are initialized and trained, it remains unclear why introducing substantial newly initialized capacity is preferable to reusing more pretrained neurons. Further justification and sensitivity analysis of these design choices would strengthen the proposed method.
Questions for authors
  • Could the authors compare REACT against existing diffusion upcycling approaches, particularly Dense2MoE, under a comparable backbone and recovery setting?
  • How do the routing coherence and expert utilization results in Figure 4 compare with those of the random initialization baselines?
  • Is there additional representation-level evidence that REACT promotes meaningful functional expert specialization?
  • Does REACT's performance advantage persist with longer recovery training? How much of the generation quality improvement could be achieved by continued training of the dense model under a comparable compute budget?
  • What motivates the replication limit of two and the per-expert allocation constraints? Have the authors examined how greater pretrained neuron reuse affects reconstruction quality or downstream generation performance?
1 comment
  1. Dense2MoE and REACT use different parameter budgets and pursue different design goals. Dense2MoE partitions the original FFN neurons into non-overlapping experts without expanding the FFN's parameter capacity. Each neuron belongs to only one expert, reducing inference cost through sparse activation. REACT expands the total capacity and allows the same pretrained neuron to be copied into multiple experts. This gives the model more capacity to improve generation quality while keeping inference sparse. Given these differences in parameter budgets and design goals, Dense2MoE and REACT are not directly comparable in their original formulations.

  2. Figure 4 examines whether the recovered REACT model uses its experts and groups similar inputs together. All 32 experts receive traffic, with usage ranging from 1.477% to 4.333% and a coefficient of variation of 0.161. Among the eight analyzed experts, mean within-expert similarity is 0.571, compared with 0.460 across experts. The figure therefore documents coherent routing and broad expert use in the analyzed checkpoint, while Tables 3–4 provide the generation-quality comparisons with random initialization. (Section 4.4, Figure 4)

  3. REACT gives each expert a concrete starting point: pretrained neurons selected for their responses to a particular group of trajectory inputs. The router is initialized from that same group. We evaluate this design through both generation results and routing behavior. The full method outperforms the tested initialization controls, including the no-centering, random-router variant with the same experts, and Figure 4 shows within-expert input coherence alongside broad expert use. Together, these results support the practical value of constructing experts and initializing their routing from the same contexts. (Section 3.3, Tables 3–4, Figure 4)

  4. Faster recovery matters when converting a pretrained model under a limited training budget. REACT's advantage remains throughout the reported 100K–500K interval: at 500K steps, FID is 8.60 versus 10.26 for the M source and 2.54 versus 2.78 for the XL source. For the M source, REACT at 300K steps already reaches 9.61 FID, better than the random baseline's 10.26 at 500K. The benefit therefore extends beyond the first stages of recovery and gives better quality at the evaluated training budgets. (Table 3)

  5. Our main goal is to show that REACT can achieve generation quality close to the dense model with substantially fewer active parameters. REACT-M32 achieves an FID of 2.21, compared with the published dense reference of 2.27, while activating only 37.5% of the dense FFN parameters and sampling 1.47× faster in our measurements. The main result is this combination of near-dense quality and lower inference cost, rather than the small FID difference. (Tables 1–2)

  6. The two-copy limit allows a pretrained neuron to serve more than one context while capping repeated reuse across experts. The allocation limits also bound how many inherited neurons each expert receives. The remaining slots are trainable units with zero output at initialization: they start without changing the expert's output and can learn during recovery. Thus, the model combines selected pretrained computations with additional capacity for adaptation. The reported padding fraction describes the initialization, not permanently inactive capacity. The full parameter counts and recovery results include these trainable units. (Sections 3.1 and 3.3, Appendix A.1–A.4)

Reviewer_w7YWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes REACT, a method for converting a pretrained dense Diffusion Transformer (DiT) into a mixture-of-experts (MoE) model. It centers inputs from denoising trajectories within timestep buckets, clusters them, and assigns pretrained neurons based on cluster-specific responses. The corresponding cluster centroids are used to initialize the router. Generation quality is then recovered through layerwise reconstruction of teacher outputs and end-to-end training. On class-conditional ImageNet-1K generation at 256 × 256 resolution, REACT-M32 reports an FID of 2.21 and 489M active non-router parameters. It achieves a 1.47× sampling speedup under the authors’ measurement setup.

Critical strengths
  1. Expert construction and router initialization use the same partition of input contexts. The design has a clear rationale and accounts for changes in input distributions across diffusion timesteps.
  2. Across both model scales, the full method outperforms the bundled random baseline at every reported recovery checkpoint. The control that fixes the experts and randomizes only the router further supports the value of centroid initialization. The paper also reports actual sampling latency and the cost of increased total model capacity, helping readers assess the benefits and costs.
Critical weaknesses
  1. Lack of a matched comparison with the closest prior method. Although the paper discusses Dense2MoE, it does not provide a quantitative comparison under matched conditions. The current experiments mainly show that REACT outperforms the tested random or partially random initializations. They do not yet establish its advantage over existing DiT conversion methods.
  2. Some mechanisms have not been evaluated in isolation. The paper includes a control that fixes the experts and randomizes the router. However, the centering experiments only vary the number of timestep buckets and do not isolate the effect of centering itself. There is also no experiment that directly tests the contribution of expert–centroid matching by permuting only their correspondence.
Questions for authors
  1. Could the authors compare REACT with Dense2MoE using the same source DiT and data, with matched active-parameter and recovery training budgets? If task differences complicate adaptation, please clarify which components can be compared fairly.
  2. Could the authors compare no centering, global centering, and timestep centering? Could they also keep the expert and centroid parameter sets unchanged and permute only their correspondence?
1 comment
  1. Dense2MoE and REACT use different parameter budgets and pursue different design goals. Dense2MoE partitions the original FFN neurons into non-overlapping experts without expanding the FFN's parameter capacity. Each neuron belongs to only one expert, reducing inference cost through sparse activation. REACT expands the total capacity and allows the same pretrained neuron to be copied into multiple experts. This gives the model more capacity to improve generation quality while keeping inference sparse. Given these differences in parameter budgets and design goals, Dense2MoE and REACT are not directly comparable in their original formulations.

  2. We do include a no-centering experiment: “Experts + random router” in Figure 3(b) and Tables 3–4. This variant retains the same experts but disables timestep centering and uses a randomly initialized router. At 500K steps, REACT achieves an FID of 8.60, compared with 8.90 for this variant. Thus, our evaluation includes a no-centering configuration, not just changes to the number of timestep buckets.

  3. The “Experts + random router” control retains the same experts but disables timestep centering and randomly initializes the router. REACT achieves lower FID at all five checkpoints, reaching 8.60 versus 8.90 at 500K steps. This supports our combined centering and centroid-based router initialization with expert contents held fixed. The expert–centroid pairing follows the construction itself: the same context defines both the neurons assigned to an expert and its initial router row. (Figure 3(b), Table 3)

Reviewer_6rRXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To improve performance while keeping the inference efficiency, REACT converts a dense DiT model into sparse MoE with scaling the total parameters and shrinking the activated ones. By analyzing the activations with clustering, REACT constructs a shared expert and routed experts. Overall, the downstream performance is close to the dense baseline.

Critical strengths
  1. The research question is clean and intuitive. The natural question is how to partition pretrained neurons during upcycling so that an input can still reach the computations it needs. REACT organizes units by similar trajectory contexts, which follows directly from this framing.
  2. The method is clearly presented, with a nice pipeline figure and reasonably complete appendix (conversion procedure, accounting, centering-state checks, optimizer settings).
Critical weaknesses
  1. Dense-to-MoE baselines are cited but not compared: MoEfication, LLaMA-MoE, Drop-Upcycling, Sparse Upcycling, and especially Dense2MoE for DiT conversion.
  2. The headline quality comparison is confounded. REACT gets ~700K extra training steps (Stage I + Stage II) on top of the dense checkpoint, while 2.27 is a published number with no matched continued-training baseline. Or do I miss something?
Questions for authors

see weaknesses above

1 comment
  1. Dense2MoE and REACT use different parameter budgets and pursue different design goals. Dense2MoE partitions the original FFN neurons into non-overlapping experts without expanding the FFN's parameter capacity. Each neuron belongs to only one expert, reducing inference cost through sparse activation. REACT expands the total capacity and allows the same pretrained neuron to be copied into multiple experts. This gives the model more capacity to improve generation quality while keeping inference sparse. Given these differences in parameter budgets and design goals, Dense2MoE and REACT are not directly comparable in their original formulations.

  2. Our main goal is to show that REACT can achieve generation quality close to the dense model with substantially fewer active parameters. REACT-M32 achieves an FID of 2.21, compared with the published dense reference of 2.27, while activating only 37.5% of the dense FFN parameters and sampling 1.47× faster in our measurements. The main result is this combination of near-dense quality and lower inference cost, rather than the small FID difference. (Tables 1–2)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.