acceptodds
Withdrawn from ICLR 2027

DIET: Deletion-response Expert Trimming for Video Diffusion Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video diffusion transformers (DiTs) increasingly rely on mixture-of-experts (MoE) architectures, where only a sparse subset of a large expert bank is activated per token. While dynamic sparsity reduces active compute, it leaves the full expert storage footprint intact. Furthermore, conventional one-shot pruning criteria rely on static activation or routing statistics, failing to capture the layer-level re-routing behavior triggered after expert deletion. To make this post-deletion behavior observable without prohibitive cost, we record all expert outputs and router states across matched conditional and unconditional tokens in a single all-expert calibration pass. Consequently, replaying any single-expert deletion and its grouped re-routing reduces to tensor arithmetic on cached states, requiring zero additional model forward passes. Using these deletion responses, we formulate retained-expert selection as an Overall Diversity Loss (ODL): each pruned expert is matched with its nearest retained expert and the summed nearest-neighbor cosine distances define a set-level coverage loss under which deleted experts closer to retained ones cost less, directly prioritizing functional coverage. We solve this optimization in two stages: an intra-layer local search combining greedy initialization, single-swap refinement, and simulated annealing to select retained set under a fixed retention ratio, and an inter-layer regression-guided search that optimizes expert-count allocation across layers. Evaluated on LingBot-Video 30B-A3B, pruning 50% of the experts (6,144 → 3,072) without fine-tuning reduces the checkpoint footprint from 57 GB to 30 GB, enabling single-card deployment on a 48 GB GPU while generation quality remains essentially intact: we observe the official VBench Total rise from 0.7941 to 0.8115 across a fixed 284-case protocol. Across all other tested retention budgets, DIET consistently outperforms competitive baselines adapted from large language models.

Primary area:
generative models
Keywords:
Video DiT-MoE Pruning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GouAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies expert pruning in video diffusion transformers with mixture-of-experts layers (DiT-MoEs). It first caches expert outputs and routing states through an all-expert calibration pass, then replays individual expert deletions and the resulting grouped rerouting at fixed layer inputs to construct deletion-response signatures. An Overall Diversity Loss (ODL), based on nearest-neighbor cosine distances, selects retained experts within each layer, while a regression surrogate guides the allocation of retention budgets across layers. On LingBot-Video 30B-A3B, the method removes 50% of the experts without fine-tuning and reduces the checkpoint size from 57 GB to 30 GB.

The work has practical deployment value. However, I have substantial concerns about the extent of its methodological contribution, generalization across models, and the evidence supporting its quality improvements.

Critical strengths
  1. Practical relevance. Sparse activation does not directly reduce the storage required for expert parameters. Physically removing experts and producing a deployable, compact checkpoint is a useful engineering contribution.

  2. Deletion-aware characterization. Frozen-state replay captures grouped rerouting and gate renormalization, providing a more direct characterization of local deletion effects than routing frequency alone. Appendix O also verifies agreement between replayed outputs and actual execution.

  3. Useful experimental controls. The experiments include several baseline families, comparisons under uniform layer budgets, and a reconstruction objective evaluated with the same cache and solver. Resolution transfer, calibration-size analyses, and paired statistical tests help characterize the method's strengths and limitations.

Critical weaknesses

1. Evaluation on a single model does not establish general conclusions about video DiT-MoEs.

All experiments in the main paper and appendices use LingBot-Video 30B-A3B. Varying retention ratios and resolutions, or analyzing different layers of the same model, does not substitute for evaluation across models. Section 5 also acknowledges that cross-architecture generalization remains unresolved.

Several central findings—including the near-orthogonality of deletion signatures, the importance of the sparse high-similarity tail, and the advantage of directional coverage over reconstruction—may depend on this model's expert granularity, grouped routing, and training procedure. The current evidence does not distinguish broadly applicable properties from model-specific observations.

I recommend evaluating at least one additional model with a different expert configuration or routing mechanism. If model availability or computational cost makes this impractical, the authors should explain the constraint and narrow the scope of their conclusions accordingly.

2. The distinction from existing MoE expert pruning and the contribution to video diffusion need clearer articulation.

Most existing MoE expert-pruning research focuses on LLMs, so studying video diffusion models is valuable. Section 2.2 identifies challenges involving multi-step denoising and classifier-free guidance (CFG), while Sections 4.2–4.3 show that lower local reconstruction error does not necessarily yield better generation quality. These observations provide meaningful motivation for the method.

The paper could nevertheless explain more clearly which key difficulties arise in video diffusion relative to existing reconstruction-based pruning and expert-redundancy modeling, how deletion-response representations and directional coverage address those difficulties, and what the experiments reveal about expert functionality in video diffusion. Connecting these points through the existing results, and distinguishing established ideas from new designs and findings, would clarify the contribution beyond extending pruning to another application domain.

3. Independent evaluation and comparisons with strong baselines do not yet establish consistent quality improvements.

The main results use 284 prompts, of which 142 also participate in the layer-budget search (Appendix E). The aggregate scores therefore combine search-set and test-set performance. The authors do report results on the other 142 held-out prompts: across 138 completed pairs, the improvement over the unpruned model is +0.0142, with a 95% confidence interval of [−0.0008, 0.0298]. This supports a positive trend but does not establish a quality improvement on independent test data.

The advantage over strong baselines also requires qualification. Under the full evaluation protocol, the paired improvement over HTS is +0.0180, with a 95% confidence interval of [−0.0013, 0.0377] (Appendix R). The advantage over D-optimal selection is likewise not statistically significant (Appendix K). Although DIET significantly outperforms several other baselines, these results do not establish a consistent advantage over the most competitive alternatives. Conversely, a lack of statistical significance does not imply that the method is ineffective.

Evaluation is also limited to this prompt collection and VBench, without an independent prompt distribution or human preference assessment. Appendix N includes a native-480p run over the full 284-prompt protocol, but this primarily compares the same pruned model across resolutions. The paired comparison against the unpruned model at 480p covers only 71 cases and does not include strong pruning baselines. Resolution-transfer results therefore do not replace a complete comparison of methods at native resolution.

I recommend making prompts excluded from search the primary test set, comparing DIET with the unpruned model and strong baselines under matched conditions, and reporting paired confidence intervals. Multiple generation seeds and an independent prompt distribution would further test the consistency of the gains.

4. The link between deletion-response similarity and functional substitutability under joint pruning remains insufficiently established.

Each signature is measured with all other experts present and the layer input fixed. After substantial pruning, substitution relationships among surviving experts, downstream activations, and subsequent denoising trajectories can all change. Although ODL does not linearly sum deletion responses, it still selects the entire retained set using single-deletion signatures measured against the original expert bank.

Showing that single-deletion responses do not combine linearly does not establish nearest-neighbor directional coverage as a reliable surrogate for joint pruning. Similar deletion-response directions do not, by themselves, imply that retaining one expert compensates for removing another.

Appendix K, Table A4 further shows that only 135 of the 3,072 deleted experts have a cosine similarity above 0.3 to their nearest retained expert, with a mean nearest-neighbor cosine similarity of approximately 0.166. The interpretation that retained experts cover the functions of deleted experts therefore warrants more direct validation.

I recommend examining the relationship between ODL, actual joint-deletion damage, and independently evaluated end-to-end quality, including how this relationship changes with pruning ratio and network depth. The authors should also clarify the information lost through averaging across tokens and the treatment of small-norm or zero-norm signatures.

5. Fairness of the full-pipeline comparison and its computational cost require further clarification.

Comparisons under uniform layer budgets help isolate the selection criterion. Under searched budgets, however, competing methods inherit the allocation found for DIET; the paper does not report results after independently tuning each baseline with the same search budget. Section 4.2 appropriately acknowledges this limitation, but it still limits conclusions about the superiority of the full pipeline.

For example, at 50% retention, DIET scores 0.7996 under uniform budgets, compared with 0.7921 for HTS, and reaches 0.8115 after layer-budget search. Improvements from the selection criterion and from the additional budget search should therefore be assessed separately.

Moreover, “zero additional forward passes” applies to local deletion replay after caching. Appendices C and D report approximately 375 GB of cached state, 24 GPU-hours for calibration, and 400 GPU-hours for layer-budget search involving 87 end-to-end evaluations. Replay reduces local evaluation costs, but the complete procedure still incurs substantial overhead.

I recommend comparing quality under equal total search-compute budgets and reporting the actual time, hardware, and storage needed for a single target retention ratio. Deployment benefits should also be reported separately from offline compression costs.

Questions for authors
  1. Could the authors provide results on a second model, or explain more specifically why only LingBot-Video was selected? Which findings are expected to depend on its grouped routing structure?

  2. Could DIET, HTS, D-optimal selection, and the reconstruction baseline be compared on a fully independent held-out set under matched conditions, with confidence intervals and equal budgets for method-specific layer-allocation search?

  3. Section 3.3 describes a regularized linear model with an unpenalized intercept, but the stated formula uses an uncentered design matrix K\mathbf{K} while centering only the targets y\mathbf{y}. In general, this differs from the standard ridge-regression solution with a free intercept, which requires centering the design matrix as well. Is this a presentation issue or the actual implementation, and does it affect the budget-search results?

  4. Appendix J primarily assesses stability through mask overlap and changes in ODL. Could the authors report actual generation-quality variation across calibration samples and solver seeds to establish whether surrogate stability translates into stable end-to-end performance?

Reviewer_ArNqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies expert pruning for video diffusion transformers with an MoE backbone. While MoE architectures reduce active computation, the full expert bank must still be stored, resulting in substantial memory overhead. The authors propose DIET, a training-free method that addresses two questions: which experts to retain and how many experts to keep per layer. The method relies on a calibration set to record all-expert forward passes. It stores output and router states for matched conditional and unconditional tokens, which only need to be computed once and can subsequently be used to simulate the removal of individual experts through tensor arithmetic. The pruning problem is then formulated as a combinatorial optimization problem that optimizes an Overall Diversity Loss, focusing on function coverage rather than minimizing output residuals. Finally, the authors estimate a non-uniform layer-wise allocation through a regression-based procedure, resulting in the final pruning strategy.

Critical strengths
  • The problem is relevant, particularly given the growing body of work on video MoEs, where storage requirements remain an important and relatively underexplored challenge.
  • Simulating leave-one-out pruning using cached states makes an otherwise prohibitively expensive analysis computationally feasible.
Critical weaknesses
  • The paper makes an interesting point regarding the differences between video and language MoEs, but it is not entirely clear how these differences are reflected in the proposed method or what the specifically video-oriented contributions are. As far as I can tell, classifier-free guidance (i.e., matching conditional and unconditional tokens during calibration) is the main diffusion-specific component, yet its impact is not ablated. As this is one of the core contributions it should be further discussed and explored.
  • As acknowledged by the authors, the evaluation is limited to a single backbone. This is a particularly important limitation for a training-free and primarily empirical method, as its generality remains unclear. While the limitation is discussed, broader validation would significantly strengthen the work.
Questions for authors
  • What are the video-specific design choices within DIET? What prevents it to be applied to an LLM? A more detailed discussion would be helpful.
  • Where does the calibration set come from? Does it overlap with the VBench prompts? How sensitive are the results to the choice and size of the calibration set?
  • Could the authors provide a more detailed breakdown of the metrics, especially the motion-related metrics (e.g., dynamic degree, motion smoothness, and temporal flickering)? Looking at the videos in the supplementary material this is the most notable loss but it is not discussed in the paper.
  • It would be interesting to include a user study. Based on the examples provided in the supplementary material, the visual differences between methods appear noticeable and may not be fully captured by the reported metrics.
  • Can we compare with other baselines, e.g. quantization, token pruning? A broad level discussion would be useful to frame the problem anyway.
  • Why is the main evaluation performed at 240×416 rather than the native 480p resolution? Would it not be preferable to perform inference at the default resolution and downsample the generated videos afterward?
Reviewer_vXSaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents DIET, an expert pruning method for video diffusion transformers. It caches expert outputs and router scores for all experts across all layers and uses them to evaluate the change in output resulting from the deletion of each expert in a layer across sampled calibration tokens. Each expert then has a deletion-response signature vector, and the retained set of experts is determined by minimizing the sum of cosine distances between each pruned expert and its most similar retained expert. A regression-guided step then allocates the expert budget across different layers. The authors present VBench scores (video quality results) on the LingBot-Video model, showing higher total scores than the evaluated pruning baselines.

Critical strengths

1/Though I have concerns about the non-uniform pruning results for reasons delineated in the weaknesses below, the results for uniform pruning in Table 2 seem promising, as DIET achieves the highest total score across all 3 retention levels. 2/ I appreciate the methodological development involved in selecting retained experts starting from multiple initializations followed by local swaps and simulated annealing, as well as the ablation showing that the authors’ selection method achieves better downstream performance compared to solely using backward selection. I also find the regression-guided budget allocation an interesting approach.

Critical weaknesses

1/ The results are limited to one video generation model. Thus, there is not sufficient evidence that method works more broadly rather than specifically due to this architecture. The authors acknowledge this limitation in Section 5.

2/ Evaluation results for the regression-guided non-uniform pruning method in Table 2 use a pool of prompts that is not disjoint from the pool used for the budget allocation step described in 3.3. In Appendix E, they provide results on a held-out evaluation dataset, but a CI for the difference with HTS is not provided, so we cannot evaluate if the performance improvement is statistically significant.

3/ For the regression-guided results, the budget allocations for all baselines is done using the budget allocations inherited from the DIET method rather than independently optimized allocations. The authors acknowledge this limitation in Section 4.2. A more fair comparison with HTS, for example, would be to form retained expert subsets for different allocations k1,...,kLk_1,...,k_L using HTS (same number of tested allocation combinations as for DIET), regress each model's validation VBench score on the budgets to fit HTS-specific regression coefficients ww, and use the fitted model to optimize HTS's budget per layer. Alternatively, the authors can compare with non-uniform pruning approaches like DiEP and EvoESAP.

4/ Comparison of runtimes is missing. How much more does DIET cost compared to the strongest baselines? This matters especially for cases where the VBench gains are small or uncertain, e.g. as seen in appendix R the gains vs HTS at 50% retention on the combined evaluation dataset (which contains prompts used for budget allocations) are not statistically significant at the 95% CI.

Questions for authors

Please address weaknesses above.

Please report how beneficial each component of the retained expert selection step ends up being in terms of downstream VBench performance. The authors show gains of their full selection procedure compared to backward-greedy, but do not break down the contributions coming from multiple initializations, local swaps, and simulated annealing. It would be good to know the additional runtime involved for each of those steps, to assess whether a cheaper, simpler variant of the procedure is going to lead to comparable gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.