acceptodds
Under review as a conference paper at ICLR 2027

CROWN: Conversion to Routed Experts with Output-Aware Width Narrowing for Audio-Video Diffusion

Abstract

Feed-forward networks (FFNs) account for a substantial fraction of the param eters and per-step computation in video diffusion transformers, yet their intra expert redundancy remains underexplored in multimodal token-routing settings. We start from DaVinci-MagiHuman (DaVinci), a pretrained single-stream audio video diffusion transformer, and sparse-upcycle the shared MLPs in its 32 mid dle blocks into token-level Top-1 Sparse-MoE layers, while leaving the modality specific early and late blocks unchanged. This conversion restructures the shared FFNs into routed expert subspaces without reducing the active intermediate width. Building on TENP, we propose Elastic Expert Neuron Pruning (EENP), a modality-aware method for structured intra-expert pruning in routed multimodal DiTs. EENP estimates intermediate-neuron importance separately from video, audio, and text tokens, normalizes the aggregated statistics within each modality, and fuses them into a modality-aware score with weights 3:1:1 for video, audio, and text. By aggregating statistics within each modality before fusion, the scoring procedure prevents modalities with more tokens from dominating the pruning decision. Low-importance neurons are then structurally removed, with the retained neurons stored as a nested Fast prefix of a wider Quality width. Under a 60% nominal pruning ratio,EENP retains 92.4% of the unpruned SparseMoE VBench- 2.0 score and reduces DiT denoising latency by 22.0% relative to the unpruned SparseMoE checkpoint.

Primary area:
generative models
Keywords:
Audio-Video Generation, Diffusion Transformers, Sparse Mixture-of-Experts, Structured Pruning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xVF2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies structured compression and efficient inference for single-stream audio-video diffusion transformers, with DaVinci-MagiHuman as the target model. The proposed method, CROWN, operates in two stages. It first converts the 32 shared FFNs in DaVinci-MagiHuman into three-expert Top-1 SparseMoE layers. It then applies Elastic Expert Neuron Pruning (EENP), which estimates neuron contributions separately for video, audio, and text tokens and uses these estimates to construct nested Quality and Fast widths. However, dense-to-MoE conversion and sparse MoE have all been explored in prior work. The main contribution of CROWN is therefore modality-wise normalization and fusion with fixed 3:1:1 weights, which seems like an engineering trick.

Critical strengths
  1. Measured reduction in denoising latency: The experiments indicate that CROWN reduces DiT denoising latency from 26.83s to 20.92s at a 60% pruning ratio.

  2. Large parameter reduction at comparable CLIPScore: In the generalization experiment on MAGI-2 Preview, the method reduces the parameter count from 114B to 48.8B while retaining comparable CLIPScore.

Critical weaknesses

Major weaknesses

W1. Incremental novelty: Dense-to-MoE conversion and sparse MoE have all been explored in prior work. The main contribution of CROWN is modality-wise normalization and fusion with fixed 3:1:1 weights, which seems like an engineering trick.

W2. The pruning evaluation is not comprehensive: The evaluation of the proposed pruning focuses mainly on video. It reports no audio results and no video-audio alignment results for the pruned model, both of which are also important in video-audio joint generation.

W3. No user study: The submission contains no user study.

W4. No playable video samples: No generated videos are available in the supplementary material. Without playable samples, the quality of the video and therefore the performance of the proposed method are hard to evaluate independently of the reported scores.

Minor weaknesses

W5. Limited improvement over Wanda: Table 7 in the supplementary material indicates that the proposed EENP has limited improvement over Wanda (0.4791 vs. 0.4737).

W6. Choice of the 3:1:1 setting: It remains unclear why the fusion weights are set to 3:1:1 rather than, for example, 2:1:1 or 2:2:1. Absent a comparison against such intermediate settings, the choice reads as an engineering trick.

W7. Figures not cited in the main text: Several figures are not cited in the main text, including Figures 1, 4, and 6.

Questions for authors

See Weaknesses.

Reviewer_DCxrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors focus on a single-stream multimodal transformer for joint audio and video generation. They propose a two-stage pipeline: (a) Sparse MoE upcycling (building on existing dense-to-MoE conversion work, although the paper lists it as a contribution), and (b) EENP (Elastic Expert Neuron Pruning), which prunes neurons within each expert while keeping the router frozen.

EENP builds upon TENP, which scores intermediate neurons based on their contribution to the full expert output. However, instead of computing a single score, the authors compute scores separately for video, audio, and text tokens, standardize them within each modality, and then fuse them using fixed weights. The motivation is that the most frequent modality should not dominate the pruning decision. The scores are estimated using calibration clips.

The authors instantiate two retention ratios: one targeting fast inference (more aggressive pruning) and one targeting quality preservation (more conservative pruning). These are then combined into a final retention policy that varies across denoising steps.

Critical strengths
  • The problem is very relevant as FFN layers account for a substantial fraction of both parameters and computation in video DiTs, and joint audio-video generation is particularly expensive. The multimodal setting is also significantly less explored than text-only MoE pruning.
  • Practical training-free pruning technique. The method is training-free at the pruning stage and requires only 32 calibration clips. It produces a single checkpoint with two operating modes (Quality and Fast), and the physical narrowing of the experts yields actual dense-GEMM speedups rather than merely masking weights.
Critical weaknesses
  • Headline results might be overclaiming. The reported 22% latency reduction and 92.4% VBench retention are measured relative to the converted SparseMoE model, which is itself slower than the original dense model (26.83s vs. 23.99s) despite having the same active parameter count. When compared against the dense DaVinci model (Table 4), EENP-60% is approximately 12.8% faster while retaining roughly 95% of the VBench score, but it also stores more parameters (21.7B vs. 15.3B). As a result, the practical improvement over the original model appears substantially smaller than suggested by the headline numbers.
  • The multimodal motivation is under-supported. The paper argues that modality imbalance motivates modality-aware scoring, but the main text does not provide token-count statistics or an ablation against pooled scoring. Moreover, the experts do not appear to be modality-specialized, so the role of modality-specific scoring remains somewhat unclear.
Questions for authors
  • Can EENP be combined with orthogonal acceleration techniques such as token merging, quantization, step distillation, or feature caching, which are discussed in the related work but not compared against? If so, does it retain a similar relative gain, i.e., are the reported gains complementary?
  • Are the calibration clips and evaluation clips fully disjoint? Could the authors evaluate on a held-out prompt set and report audio-quality metrics after pruning?
  • At 20% and 40% pruning, latency (24.70 s, 22.91 s) is not clearly better than the original dense model (23.99 s). What is the break-even point, and what is the overhead of turning the model into MoE?
  • Since the experts start as near-copies of the dense FFN and the router is trained only by the load-balancing loss, how does EENP applied directly to the dense DaVinci FFNs (no upcycling) compare in quality and latency? This would show whether the SparseMoE stage is needed at all.
  • On MAGI-2 Preview, the evaluation uses a uniform width without the fast/quality schedule, what is the reason for such a choice. Morover there are no reported results on VBench and overall latency, so it is not very helpful.
  • At 60% pruning, the paired fidelity on the 32-clip diagnostic subset degrades substantially (FVD from 64.3 to 121.0, CLIP-Sim from 0.919 to 0.823; Table 4). Why such a big gap exists compared to the original model? And could the authors report FVD with more clips or seeds?
Reviewer_kJgeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CROWN compresses an audio-video diffusion transformer through sparse upcycling followed by structured expert pruning. It first converts shared dense FFNs into top-1 routed experts and adapts the resulting model. Elastic Expert Neuron Pruning (EENP) then aggregates output-contribution statistics separately for video, audio, and text tokens, normalizes and combines them to rank neurons, and constructs nested Quality and Fast widths for different denoising steps. The paper evaluates generation quality, paired fidelity, parameter counts, and denoising latency, and additionally applies the scoring rule to MAGI-2 Preview.

Critical strengths

1.EENP addresses a practical compression problem in multimodal diffusion transformers. Aggregating and normalizing neuron-contribution statistics within each modality helps account for differences in modality token counts. Physically narrowing the expert projections also converts pruning into smaller dense matrix operations, providing an implementable route to reducing computation.

2.The evaluation includes measured denoising latency and distinguishes stored parameters from active parameters. Table 1 demonstrates a quality–latency trade-off across pruning ratios, while the MAGI-2 Preview experiment provides additional evidence that the scoring rule can be applied to another backbone. These results support the practical usefulness of structured expert pruning.

Critical weaknesses

1.The main experiments do not establish whether dense-to-MoE conversion improves the final compression trade-off over directly pruning the dense backbone. Conversion increases stored capacity without reducing active FFN width, but an equivalent scoring and Quality/Fast scheduling procedure applied directly to the adapted dense model is not shown. At the 60% setting in Table 4, EENP reduces denoising latency from 23.99 to 20.92 seconds, but increases stored parameters from 15.300B to 21.673B and lowers VBench from 0.504 to 0.479. This is a valid speed–quality–storage trade-off; however, without the direct dense-pruning control, the contribution of the conversion stage to compression remains uncertain.

2.Table 6 compares pruning methods at matched nominal sparsity rather than matched computation. At 60% nominal pruning, EENP retains 65% width for eight steps and 40% for twenty-four steps, giving a step-averaged retention of 46.25%, whereas the constant-width structured baselines retain 40%. The paper explicitly acknowledges this difference, but the comparison therefore combines the effects of neuron scoring, width scheduling, and additional computation. Matched-budget comparisons are needed to determine whether EENP offers a better quality–efficiency trade-off, and a fixed-schedule ablation would help isolate the modality-aware scoring contribution.

Questions for authors

1.What results are obtained when the same modality-aware scoring, calibration protocol, and Quality/Fast scheduling are applied directly to the adapted dense FFNs? A comparison of quality, latency, and stored parameters would clarify the contribution of dense-to-MoE conversion.

2.Does EENP retain its quality advantage over fixed-width TENP and the other structured baselines at matched average active width or measured denoising latency? Please also clarify which scoring components differ between the TENP and EENP implementations, since the paper states that TENP reuses the EENP projection scores.

3.Since the router is optimized solely through the load-balancing auxiliary loss, is there evidence that its learned assignments improve the final compression trade-off over a fixed or random balanced routing control? Near-uniform expert utilization alone does not establish this benefit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.