acceptodds
Withdrawn from ICLR 2027

Functional Cross-layer Experts Routing for One-step Diffusion Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

One-step distillation compresses diffusion trajectories into a single forward evaluation, yet how internal capacity should be structured within that remaining pass remains unexplored. In this setting, identical parameters must simultaneously preserve general generative priors and resolve prompt-specific details without iterative refinement. We formalize this challenge as functional allocation. Our analysis reveals that decoupling representations is beneficial only when expert interference outweighs shared transfer, favoring an always-active shared backbone augmented with conditionally routed residuals over standard isolated MoE architectures. We propose , which pairs a persistent shared expert with conditionally routed expert slices across image FFNs, trained under a one-step distribution-matching objective with executed-route supervision. On one-step FLUX.2-4B generation, DeMoE achieves a GenEval score of 0.8220 with only 80.5% active image-FFN capacity—substantially outperforming dense one-step baselines and matching the 50-step teacher while surpassing it on complex composition and counting. Across COCO-10K, DeMoE establishes state-of-the-art fidelity and diversity among single-step models, demonstrating that structured internal allocation provides a superior compression paradigm for diffusion models.

Primary area:
generative models
Keywords:
Diffusion Model, Distribution Distillation, Mixture-of-Experts

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pjvbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper explores how computation should be organized inside the single forward pass of a one-step distilled diffusion model and proposes DeMoE. It replaces the dense FFNs of base model (FLUX.2-4B) with one always-active shared slice and seven routed slices. It uses a router conditioned on the pre-FFN hidden state and the text embedding to select the top-4 routed slices per token. It designs two further components, an optional fast-weight depth-recurrent state that feeds the router, and an auxiliary loss that predicts the executed routed response from a stop-gradient target. Training with the DMD2 objective, the paper's 7E4A1S configuration reaches similar GenEval compared with the 50-step dense teacher.

Critical strengths
  1. The paper focuses on how a one-step student’s computation is organized, which is a relatively under-explored angle.
  2. The proposed DeMoE 7E4A1S configuration reaches similar GenEval compared with the 50-step dense teacher.
Critical weaknesses
  1. The core structure of the DeMoE is a standard shared-expert MoE conversion. Nothing in it depends on the number of sampling steps, and it could be applied unchanged to a multi-step model. There seems no connection between the MoE design and one-step distillation demonstrated in this paper.
  2. There is no comparison with previous methods (e.g. other one-step methods or other diffusion MoE methods like Dense2MoE, cited in the paper). All comparisons are against the authors’ own dense variants.
  3. The proposed 7E4A1S is slower and uses more memory compared with the paper's efficiency claim.
  4. There are questionable results reported in this paper. 3E1A1S and 7E3A1S both report 3.45B active parameters, while their active FFN shares are 0.500 and 0.740.
  5. Presentation of this paper needs improvement. Figure 1 takes up about half of page 2 but is never referenced in the text. Except the one-step DMD, the five paradigms it shows are neither used nor compared against. The introduction describes “Figure 2(a)” as illustrating three paradigms, but the actual Figure 2(a) shows a standard DiT block. Appendix Figs. 7–13 are confusing.
Questions for authors

N/A. See the critical weakness.

Reviewer_1UaqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Diffusion models generate images iteratively from noise over multiple steps by first generating high-level details, then introducing finer details. In practice, diffusion models are distilled down to a single step. In this case, the diffusion model needs to do multiple tasks in a single forward pass such as parsing the prompt, composing objects, and rendering fine details, which can be challenging. This paper considers a distribution matching distilled model (in practice, they consider Flux 2.4B), and partitions a subset of the dense FFN layers in the backbone into slices. One of these slices is a frozen shared slice S that is always active, and seven routed slices R1:7, of which four are selected per token. This is trained with DMD objective with executed route supervision. In addition, this paper also proposes test-time training (TTT) which gives the network a temporary “working memory” during sampling by updating a tiny set of weights on-the-fly for each image.

Critical strengths
  1. This paper considers an interesting and relevant problem of structuring and smartly routing information within one-step distilled generators where temporal refinement over multiple steps is not an option.
  2. DeMOE improves metrics over the original DMD as shown in Table 2 on different GenEval Components.
Critical weaknesses
  1. The shared + touted FFN partition seems to help 1 step generation quality as reflected in metrics but it comes with an additional overhead in terms of wall clock time (21% increase), and memory requirements (~16.68GB vs 39-40 GB) which is >2X increase as shown in Table 1.
  2. The proposed test-time training (TTT) has only marginal gains as shown in Figure 4(c). In addition, it might be a bit redundant as by default transformer architectures have a residual stream which already accumulates prior layer outputs across depth.
  3. Table 7 (lines 1074–1075) reports slice widths of q=1/5.4q = 1/5.4 and q=1/6.2q = 1/6.2 and Total FFN Width Ratios of 1.48×1.48\times and 1.29×1.29\times which seems incorrect unless additional trainable parameters are introduced in FFN layers.
Questions for authors
  1. As DeMOE is applied to only around 13 layers, js TTT which is meant for much deeper/longer sequences effective? (Minor) In extended related work, “sparse experts and diffusion routing” has been repeated twice.
  2. Does the auxiliary loss from Figure 2 correspond to Equation 9 from the appendix? Is this loss optimized in parallel to self supervised inner loss from Figure 2?
  3. In Table 7, why is total FFN Width ratio grater than 1 for 7E3A1S and 7E4A1S? This would imply that there are additional slices that were not part of the original architecture.
  4. Why does minimizing prediction error over an already-chosen discrete top-k action steer the router towards higher quality images?
Reviewer_e6erOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses a critical bottleneck in one-step diffusion models: the degradation of compositional generation and fine-grained control when iterative denoising steps are compressed into a single forward pass. The authors hypothesize that this performance drop is due to parameter conflicts caused by forcing a single dense network to simultaneously handle diverse and conflicting generative demands. To solve this, the authors propose DeMoE, an architecture that replaces standard dense FFNs with a combination of a frozen shared expert and dynamically routed experts. Additionally, they introduce a Depth-Recurrent TTT State to allow the router to condition its choices on the execution history of previous layers, and an "executed-response supervision" loss to stably train the router. Evaluated on FLUX.2-4B using DMD2 distillation, DeMoE achieves state-of-the-art 1-NFE performance. It attains a GenEval score of 0.8220 and a COCO-10K FID of 14.85, significantly outperforming dense 1-NFE baselines and matching or exceeding the 50-step teacher model, particularly in complex compositional tasks.

Critical strengths
  1. Large, consistent gains within one controlled pipeline. On the same backbone and evaluation stack, 7E4A1S improves over dense 1-NFE DMD2 on every GenEval category and every COCO-10K metric. If the gain can be attributed to the architecture, it matters, because one-step students typically lag teachers on compositional prompts (counting, position, attribute binding).
  2. A reasonable, under-explored question, with candid reporting of null results. The paper openly reports results that do not support it: the history-vs-current difference overlaps zero, the shuffled and random route controls are unresolved, the hard-subset gain against a continued dense model is not significant, and buggy legacy runs are excluded. This candor helps calibrate the claims.
Critical weaknesses
  1. The headline gain is not attributed to the proposed mechanism. Training budget and missing ablations confound it.
    • Budget. 7E4A1S is a 300-update continuation of 7E3A1S, which was itself trained further from the DMD2 parent. It has therefore received more generator updates than the dense parent. The only budget-controlled comparison (App. B.3) gives +3.13 pp over a continued dense model, with a 95% CI of [−3.13, 10.42], i.e. not resolved. The authors themselves write that additional training accounts for much of the gain, yet Sec. 4.5 states that DeMoE "retains a gain." The budget of the "dense progressive" baseline relative to DeMoE is not specified.
    • Ablations. There is no ablation on the deployed model isolating the shared slice, the executed-response loss, the TTT history state, or random/fixed routing at the same k. It is unclear whether the deployed 7E4A1S uses the history state at all; the atlas figures are labeled AR_CURRENT.
    • Mechanism evidence. The experiments in Sec. 4.3–4.4 use different frozen banks (a four-band bank, F156d, F148, a three-expert Top-1 bank) and show very small effects (pseudo-loss differences of ~1e-3 to 1e-5, ranking win probability 0.569). The authors state these should not be read as measurements of 7E4A1S.
  2. The capacity and efficiency accounting is internally inconsistent, and the provenance of several numbers is unclear.
    • Slice widths. App. A.2 describes a fresh partition of the parent FFN into one shared and seven routed slices of width 1/8, applied in 13 of 25 layers. This is consistent with the active shares in Table 1 ((12 + 13·5/8)/25 = 0.805; (12 + 13·4/8)/25 = 0.740). Table 7, however, describes the same models with per-expert widths q = 1/6.2 and 1/5.4 and total FFN widths of 1.29× and 1.48× the parent. This contradicts both the fresh-partition description (total width 1×) and the statement that 7E4A1S is a 300-update continuation of 7E3A1S, which could not change q. Moreover, under the A.2 accounting, 3E1A1S (q = 1/4) in the same 13 layers would have an active share of 0.74, not the 0.500 in Table 1, unless it modifies a different set of layers, which the paper does not state.
    • Other table conflicts. Table 1 gives the same active parameter count (3.45B) for 3E1A1S (active share 0.5) and 7E3A1S (0.74). Table 7 reports a 7E3A1S with GenEval 0.746 / FID 21.15, which differs from Table 1 (0.7812 / 16.92).
    • Efficiency framing. Latency is 21% higher and peak memory 2.4× higher than dense. Appendix D states that "we make no latency, memory, energy, or total-FLOP claim," which conflicts with the abstract's "activating only 80.5%" and "superior compression paradigm" framing.
  3. Minor:
    • The heavy internal jargon ("ownership," "writers," "route work," "registered," F148/F156d) makes Sec. 4 hard to follow.
    • Figures 8–13 all carry the same caption.
    • Appendix D duplicates a paragraph.
    • App. B.2 says the teacher "frequently suffers severe semantic degradation," which is inconsistent with its 0.8245 GenEval.
    • There is no related-work section in the main text.
Questions for authors
  1. Table 7 lists q = 1/6.2 and 1/5.4 with total widths of 1.29× and 1.48×, whereas App. A.2 describes a fresh 1/8-width partition in 13 layers. Which is correct, and what is the total FFN parameter count of 7E4A1S relative to the dense parent? How can a 300-update continuation of 7E3A1S have a different q? Which layers does 3E1A1S modify, and why do 3E1A1S and 7E3A1S have identical active parameter counts?
  2. Please give the exact number of generator updates for the dense DMD2 parent, the dense progressive continuation, 7E3A1S, and 7E4A1S. Can you report a budget-matched dense continuation on full GenEval and COCO-10K?
  3. Does 7E4A1S use the TTT history state? Please report 7E4A1S without the executed-response loss, without history, with an unfrozen shared slice, and with random or fixed top-4 routing.
  4. How were the MJHQ-30K numbers in Table 3 and the SDXL results in Table 4 obtained? What does "pipeline reconstructed from logs" mean concretely: were the checkpoints in Table 1 trained with the code in the supplementary material? Why is peak memory 2.4× higher than dense, and higher for 3E1A1S than for 7E4A1S?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.