acceptodds
Withdrawn from ICLR 2027

VideoEditor-MoE: Scaling Complex Video Editing with Mixture of Experts

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Complex video editing requires understanding complex instructions while jointly fulfilling multiple editing requests. Meeting these demands through dense model scaling substantially increases computational cost. We introduce VideoEditor-MoE, the first open-source mixture-of-experts framework for video editing. Beyond sparse MLP-MoE layers, MoE-Attention expands attention capacity through routed key–value experts while preserving global interactions among video tokens. Initialized from a 5B video editing model, VideoEditor-MoE scales to 14B and 56B parameters, with approximately 8B parameters activated per token in both variants. Comparisons with existing methods and two complementary baselines, Explicit-MoE and MLP-MoE, assess its benefits over explicit expert guidance and MLP-only expert architectures. Further analyses demonstrate that all experts remain active and reveal that learned routing responds more strongly to sample content and diffusion noise levels than to human-defined editing categories. Spatial analysis further indicates sensitivity to object-level semantic features. We also identify decoupled weight decay as a strategy for stabilizing expert scaling.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Complex Video Editing, Mixture of Expert

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_91LXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents VideoEditor-MoE, a mixture-of-experts (MoE) approach to video editing. Motivated by the difficulty current video editing models have with complex editing instructions, the authors propose an MoE-based scaling solution that expands a 5B base model to up to 56B total parameters, with 8B activated per token. Rather than applying experts only to the MLP layers, as is common, they extend the MoE design to the attention layers as well. Their analysis shows that all experts remain active during inference. Empirically, the method outperforms several existing video editing methods under VLM-as-a-judge evaluation, and the authors support their architectural choices with MoE ablations.

Critical strengths
  1. Under the authors' VLM-as-a-judge evaluation protocol, the method shows meaningful quantitative gains over both the base model and the architectural ablations (Tab. 1).

  2. The expert analysis is a nice addition, offering some insight into the internal behavior of the experts and the routing mechanism.

Critical weaknesses
  1. Motivation -- The authors do not provide sufficient evidence that current state-of-the-art video editing models or frameworks perform poorly on more complex editing instructions.

  2. Evaluation -- In my view, this is the most critical weakness of the paper. The main comparative evaluation (Tab. 1) compares the proposed approach against only three existing models, one of which is the base model being scaled, and relies exclusively on a custom VLM-as-a-judge evaluation pipeline. Moreover, the method is fundamentally a model-scaling approach, reaching up to 56B parameters, so comparing it against a much smaller base architecture is not entirely fair. The latency, FLOPs, and memory requirements of the approach are also not reported. Specifically, the paper lacks:

  • A more comprehensive comparison to existing video editing models and frameworks, including both supervised and training-free methods.
  • A latency, FLOPs, and memory comparison against these and existing baselines.
  • Evaluation on standard video editing metrics, such as CLIP scores for text alignment, frame-wise CLIP/DINO similarity for temporal consistency, and FVD for video quality.
  • An iso-parameter, iso-memory, or iso-FLOPs comparison against alternative scaling strategies, such as dense expansion of the base model or added depth or width. This is needed to justify that the proposed approach is the right way to scale up video editing models.
  • Evaluation on established video editing benchmarks, such as TGVE, EditVerseBench, and OpenVE-Bench.
Questions for authors

Please address the motivation and evaluation concerns detailed above. In addition:

The expert analysis shows that routing is more sensitive to content and noise level than to editing categories. This seems to be in tension with the motivation stated in the Introduction: "complex editing also requires the exchange of semantic information across regions and frames to coordinate multiple modifications. This motivates extending expert capacity to attention, allowing MoE semantic representations to participate in global token interactions and support coherent editing across space and time." If the experts do not specialize by editing semantics, what evidence supports the claim that attention-level experts help coordinate complex edits? Could you clarify how the routing findings are consistent with this motivation?

Reviewer_2kHQOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VideoEditor-MoE, a mixture of experts model for video editing, which can handle complex edits like multiple changes applied to multiple (or single) objects or both global and local changes applied to the video. The proposed architecture extends MoE-ML with MoE-Attention. Experiments show that the proposed models outperforms MoE baselines as well as recent video editing models. Empirical analyses show that expert utilization is high and routing is more sensitive to samples and noise vs edit category or instruction when compared to MoE-MLP. Code and models (14B-A8B and 56B-A8B) will be open-sourced.

Critical strengths
  1. The best thing about this paper is that it provides substantial analyses that helps to gain a deeper understanding into the proposed model. It empirically shows that all experts are actively utilized and routing sensitivity is lower compared to MoE-MLP. Also, it provides interesting observations such as that routing sensitivity is higher across samples compared to edit category and content-sensitive expert follows object motion. It also provides insights about different optimizers (Adam vs AdamW) when scaling the MoE architecture. Though the proposed architecture itself is a simple idea, it seems effective, and coupled with the deep analyses, the paper was insightful to me (especially sections 4.3 and 4.4). The video results provided in the supplementary material also look good.

  2. The authors promise to open-source the models and code, which will be very helpful for the community. In addition, a lot of implementation details are provided throughout the paper, which is already very helpful for reproducibility.

Critical weaknesses
  1. One thing to note is that the paper doesn't compare to strong commercial models (e.g., Seedance 2.5, Gemini Omni, Aleph 2, etc). Though it's not absolutely necessary to compare to commercial models, having their results as reference (even if not directly compared), would be useful to the readers. If the authors want to restrict to open-source models, Bernini-v2 is another good open-source model.

  2. The method itself (MoE-Attention with MoE-MLP in MoE DiT block) may seem incremental. However, as I mentioned above, with all the analyses, I did find the paper insightful.

[Minor]

  • L031: a real requests -> a real request or real requests
  • MoE-MLP and MLP-MoE are used interchangeably. It would be better to make this consistent throughout the paper
Questions for authors

N/A

Reviewer_MUh5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper scales a 5B video editing DiT into a Mixture-of-Experts model for complex instructions that contain multiple editing requests. Besides a standard MoE-MLP, the authors propose MoE-Attention, in which queries are shared, keys and values come from k routed expert projections, and every query attends to the kT key-value pairs from all tokens. They train 14B-A8B and 56B-A8B variants and compare them with two self-built baselines (Explicit-MoE and MLP-MoE) and existing methods. The paper also analyzes expert utilization and routing sensitivity, and reports that replacing Adam with coupled L2 regularization by AdamW stabilizes large-scale expert training.

Critical strengths

Complex instructions with multiple edits remain a real open problem in video editing, and using MoE to expand capacity without a proportional increase in activated parameters is a reasonable direction.

Critical weaknesses
  • The paper does not evaluate on any public benchmark. Video editing already has established evaluation protocols such as OpenVE-Bench, yet the paper builds its own test set and does not even state how many cases it contains. More seriously, Appendix D states explicitly that the cases were selected based on outcomes: “This outcome-dependent selection favors VideoEditor-MoE-56B-A8B and characterizes favorable examples rather than overall benchmark performance.” In other words, Tab. 1 is computed on examples that favor the proposed model. In addition, the VLM judge only sees frames 1, 25 and 49, and the authors themselves acknowledge that this protocol “does not measure all intermediate-frame dynamics.” For video editing, this leaves temporal consistency largely unevaluated, which does not match standard practice. Under this setup, I cannot draw any conclusion from Tab. 1.
  • It is unclear where the improvement comes from. The paper has no dense baseline fine-tuned on the same data for the same number of steps, so I cannot tell whether the gain comes from MoE or from new data plus continued training. The training data itself is never described anywhere in the paper, and the number of training steps for the 56B model is not reported.
  • The readability, layout and overall rigor of the writing are poor. The novelty of MoE-Attention is also limited, and the related work is incomplete. A line of prior work has already applied MoE to attention, such as MoA (Zhang et al., 2022), which selects attention heads per token, and SwitchHead (Csordás et al., 2024), which places experts on the value and output projections. The paper neither cites nor discusses them. Most puzzling of all, Tab. 1 compares against CoinVE-Edit, yet the paper never cites it.

Overall, the paper is far from complete, and its evaluation cannot support its main claims. I therefore lean toward rejection.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.