- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies expert pruning in video diffusion transformers with mixture-of-experts layers (DiT-MoEs). It first caches expert outputs and routing states through an all-expert calibration pass, then replays individual expert deletions and the resulting grouped rerouting at fixed layer inputs to construct deletion-response signatures. An Overall Diversity Loss (ODL), based on nearest-neighbor cosine distances, selects retained experts within each layer, while a regression surrogate guides the allocation of retention budgets across layers. On LingBot-Video 30B-A3B, the method removes 50% of the experts without fine-tuning and reduces the checkpoint size from 57 GB to 30 GB.
The work has practical deployment value. However, I have substantial concerns about the extent of its methodological contribution, generalization across models, and the evidence supporting its quality improvements.
Critical strengths
-
Practical relevance. Sparse activation does not directly reduce the storage required for expert parameters. Physically removing experts and producing a deployable, compact checkpoint is a useful engineering contribution.
-
Deletion-aware characterization. Frozen-state replay captures grouped rerouting and gate renormalization, providing a more direct characterization of local deletion effects than routing frequency alone. Appendix O also verifies agreement between replayed outputs and actual execution.
-
Useful experimental controls. The experiments include several baseline families, comparisons under uniform layer budgets, and a reconstruction objective evaluated with the same cache and solver. Resolution transfer, calibration-size analyses, and paired statistical tests help characterize the method's strengths and limitations.
Critical weaknesses
1. Evaluation on a single model does not establish general conclusions about video DiT-MoEs.
All experiments in the main paper and appendices use LingBot-Video 30B-A3B. Varying retention ratios and resolutions, or analyzing different layers of the same model, does not substitute for evaluation across models. Section 5 also acknowledges that cross-architecture generalization remains unresolved.
Several central findings—including the near-orthogonality of deletion signatures, the importance of the sparse high-similarity tail, and the advantage of directional coverage over reconstruction—may depend on this model's expert granularity, grouped routing, and training procedure. The current evidence does not distinguish broadly applicable properties from model-specific observations.
I recommend evaluating at least one additional model with a different expert configuration or routing mechanism. If model availability or computational cost makes this impractical, the authors should explain the constraint and narrow the scope of their conclusions accordingly.
2. The distinction from existing MoE expert pruning and the contribution to video diffusion need clearer articulation.
Most existing MoE expert-pruning research focuses on LLMs, so studying video diffusion models is valuable. Section 2.2 identifies challenges involving multi-step denoising and classifier-free guidance (CFG), while Sections 4.2–4.3 show that lower local reconstruction error does not necessarily yield better generation quality. These observations provide meaningful motivation for the method.
The paper could nevertheless explain more clearly which key difficulties arise in video diffusion relative to existing reconstruction-based pruning and expert-redundancy modeling, how deletion-response representations and directional coverage address those difficulties, and what the experiments reveal about expert functionality in video diffusion. Connecting these points through the existing results, and distinguishing established ideas from new designs and findings, would clarify the contribution beyond extending pruning to another application domain.
3. Independent evaluation and comparisons with strong baselines do not yet establish consistent quality improvements.
The main results use 284 prompts, of which 142 also participate in the layer-budget search (Appendix E). The aggregate scores therefore combine search-set and test-set performance. The authors do report results on the other 142 held-out prompts: across 138 completed pairs, the improvement over the unpruned model is +0.0142, with a 95% confidence interval of [−0.0008, 0.0298]. This supports a positive trend but does not establish a quality improvement on independent test data.
The advantage over strong baselines also requires qualification. Under the full evaluation protocol, the paired improvement over HTS is +0.0180, with a 95% confidence interval of [−0.0013, 0.0377] (Appendix R). The advantage over D-optimal selection is likewise not statistically significant (Appendix K). Although DIET significantly outperforms several other baselines, these results do not establish a consistent advantage over the most competitive alternatives. Conversely, a lack of statistical significance does not imply that the method is ineffective.
Evaluation is also limited to this prompt collection and VBench, without an independent prompt distribution or human preference assessment. Appendix N includes a native-480p run over the full 284-prompt protocol, but this primarily compares the same pruned model across resolutions. The paired comparison against the unpruned model at 480p covers only 71 cases and does not include strong pruning baselines. Resolution-transfer results therefore do not replace a complete comparison of methods at native resolution.
I recommend making prompts excluded from search the primary test set, comparing DIET with the unpruned model and strong baselines under matched conditions, and reporting paired confidence intervals. Multiple generation seeds and an independent prompt distribution would further test the consistency of the gains.
4. The link between deletion-response similarity and functional substitutability under joint pruning remains insufficiently established.
Each signature is measured with all other experts present and the layer input fixed. After substantial pruning, substitution relationships among surviving experts, downstream activations, and subsequent denoising trajectories can all change. Although ODL does not linearly sum deletion responses, it still selects the entire retained set using single-deletion signatures measured against the original expert bank.
Showing that single-deletion responses do not combine linearly does not establish nearest-neighbor directional coverage as a reliable surrogate for joint pruning. Similar deletion-response directions do not, by themselves, imply that retaining one expert compensates for removing another.
Appendix K, Table A4 further shows that only 135 of the 3,072 deleted experts have a cosine similarity above 0.3 to their nearest retained expert, with a mean nearest-neighbor cosine similarity of approximately 0.166. The interpretation that retained experts cover the functions of deleted experts therefore warrants more direct validation.
I recommend examining the relationship between ODL, actual joint-deletion damage, and independently evaluated end-to-end quality, including how this relationship changes with pruning ratio and network depth. The authors should also clarify the information lost through averaging across tokens and the treatment of small-norm or zero-norm signatures.
5. Fairness of the full-pipeline comparison and its computational cost require further clarification.
Comparisons under uniform layer budgets help isolate the selection criterion. Under searched budgets, however, competing methods inherit the allocation found for DIET; the paper does not report results after independently tuning each baseline with the same search budget. Section 4.2 appropriately acknowledges this limitation, but it still limits conclusions about the superiority of the full pipeline.
For example, at 50% retention, DIET scores 0.7996 under uniform budgets, compared with 0.7921 for HTS, and reaches 0.8115 after layer-budget search. Improvements from the selection criterion and from the additional budget search should therefore be assessed separately.
Moreover, “zero additional forward passes” applies to local deletion replay after caching. Appendices C and D report approximately 375 GB of cached state, 24 GPU-hours for calibration, and 400 GPU-hours for layer-budget search involving 87 end-to-end evaluations. Replay reduces local evaluation costs, but the complete procedure still incurs substantial overhead.
I recommend comparing quality under equal total search-compute budgets and reporting the actual time, hardware, and storage needed for a single target retention ratio. Deployment benefits should also be reported separately from offline compression costs.
Questions for authors
-
Could the authors provide results on a second model, or explain more specifically why only LingBot-Video was selected? Which findings are expected to depend on its grouped routing structure?
-
Could DIET, HTS, D-optimal selection, and the reconstruction baseline be compared on a fully independent held-out set under matched conditions, with confidence intervals and equal budgets for method-specific layer-allocation search?
-
Section 3.3 describes a regularized linear model with an unpenalized intercept, but the stated formula uses an uncentered design matrix while centering only the targets . In general, this differs from the standard ridge-regression solution with a free intercept, which requires centering the design matrix as well. Is this a presentation issue or the actual implementation, and does it affect the budget-search results?
-
Appendix J primarily assesses stability through mask overlap and changes in ODL. Could the authors report actual generation-quality variation across calibration samples and solver seeds to establish whether surrogate stability translates into stable end-to-end performance?