acceptodds
Under review as a conference paper at ICLR 2027

DUAL: Closing the Plan–Execution Gap between MLLMs and DiTs for Video Editing

31 views

Abstract

Strong multimodal understanding and high-quality video generation do not guarantee faithful video editing: a model may identify the requested change yet edit the wrong object, assign the wrong attribute, or violate the relation that identifies the target. We call this mismatch the plan–execution gap and trace it to the interface between a multimodal large language model (MLLM) and a diffusion transformer (DiT). We propose DUAL, which routes target-state information and relational constraints through different paths. Its Target-State Path converts MLLM features from multiple layers into tokens that join source and noisy target tokens in DiT self-attention, while its Relational-Constraint Path combines final-layer MLLM features with native text embeddings for cross-attention guidance. DUAL reaches the highest Overall score among the compared open methods on both IVEBench (0.67) and OpenVEBench (4.26), while running up to 15.2 times faster than the slowest compared baseline in average inference latency. These results support separating target-state information, which should participate in denoising, from relational constraints, which should guide it.

Primary area:
generative models
Keywords:
video editing, instruction-based video editing, multimodal large language models, diffusion transformers

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9VYNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DUAL addresses instruction-only video editing by changing how an MLLM's representations condition a diffusion transformer. A Target-State Path aggregates mid-to-late MLLM features and introduces plan tokens into DiT self-attention. A Relational-Constraint Path combines final-layer queries with native text tokens through a gated residual refiner for cross-attention. The model reaches 4.26 on OpenVEBench and ties the best displayed IVEBench overall score at 0.67. Path-removal and layer-selection ablations support the design, while the reported 5B renderer has comparable latency to Kiwi-Edit.

Critical strengths
  1. The distinction between visual target-state information and relational guidance leads to an implementable change in the renderer, rather than only a conceptual description. OpenVEBench drops from 4.26 to 3.44 without the Target-State Path and to 3.40 without the Relational-Constraint Path; using only native text yields 3.23. These sizeable differences make the complementary value of the two paths the strongest argument for acceptance.

  2. The system requires only a source video and instruction, avoiding manual spatial controls. Relative to the 5B Kiwi-Edit baseline, OpenVEBench improves from 3.88 to 4.26, including Local Add from 3.31 to 4.43. Reported latency is 157 versus 168 seconds. This supports a meaningful quality improvement at a comparable renderer scale and inference cost.

Critical weaknesses
  1. The path-removal controls change both the conditioning information and its processing capacity. They do not fully isolate whether the gain comes from routing distinct semantic roles, using richer multilayer features, or adding connector capacity. A matched single-path connector consuming the same features would strengthen the central interpretation. Nevertheless, the existing ablations are sufficient to establish that the proposed interface is useful.

  2. IVEBench instruction compliance is 0.37, below Ditto's 0.49, and the 0.67 overall score is a tie. The results therefore support strong overall editing performance more clearly than a general resolution of object/attribute-binding errors. This limits the breadth of the claim, but does not outweigh the OpenVEBench and controlled-ablation evidence.

Questions for authors
  1. Under identical data, training, feature layers, and approximately matched connector capacity, how does a single-path MLLM connector compare with DUAL?

  2. Do you have a breakdown of wrong-object and attribute-binding errors, or a targeted analysis explaining why IVEBench instruction compliance remains below stronger baselines despite the overall-score tie?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9VYN

We sincerely thank the reviewer for their constructive comments and insightful suggestions, which have been invaluable in improving our manuscript. We are also grateful for their recognition of the strengths of our work. The revisions have been incorporated in the revised manuscript marked in blue. Please kindly find point-to-point responses below.

About W1 & Q1: Feature- and Capacity-Matched Single-Path Ablation

Thank you for this suggestion. We have added the matched Single Cross ablation to Table 3, clarified its routing configuration in Table 4, and discussed the comparison in Section 5.3.

  • Matched single-path comparison:

    Single Cross uses the same MLLM layers (20/24/28/32 and final), 350.60M connector parameters, training data, training procedure, and evaluation settings as DUAL. The difference lies in the conditioning design: DUAL separates detail and semantic conditioning through two specialized paths, whereas Single Cross combines both into a unified cross-attention pathway. As shown in Table R1, DUAL improves Overall by 0.71 and scores higher in all eight editing categories, isolating the value of distinct attention-routing paths under matched features and connector capacity.

Table R1. Single Cross Ablation Results

MethodMLLM featuresConnector paramsOverall ↑Global Style ↑Background ↑Local Change ↑Local Remove ↑Local Add ↑Subtitle ↑Creative ↑Camera ↑
Full DUAL20/24/28/32 + Final350.60M4.263.903.944.444.504.434.963.683.84
Single Cross20/24/28/32 + Final350.60M3.552.903.743.863.323.733.962.993.62

About W2 & Q2: IVEBench Instruction Compliance and Object/Attribute-Binding Claims

We appreciate this observation and agree with the reviewer’s assessment. We have narrowed the claim in Section 6 and added a targeted analysis of IVEBench Instruction Compliance in Appendix A.3.

  • Targeted analysis of IVEBench Instruction Compliance:

    Yes. We provide a targeted analysis of why IVEBench instruction compliance remains below stronger baselines despite the overall-score tie.

    • Spatiotemporal Challenges: IVEBench includes more complex spatiotemporal edits, including subject motion and viewpoint changes. [1] These tasks involve substantial changes to spatiotemporal structure while requiring preservation of source content and temporal consistency, creating a trade-off between edit magnitude and consistency.

    • IC Scoring: IVEBench’s Instruction Compliance (IC) combines semantic consistency, instruction satisfaction, and quantity accuracy; Instruction Satisfaction carries a higher weight and therefore has a greater influence on the IC score. [1]

    • Performance and Future Work: On IVEBench, DUAL achieves higher Video Quality (0.84) and Video Fidelity (0.79) than Ditto [2] and Bernini [3], while maintaining competitive overall editing performance. Future work will explore training data targeting complex spatiotemporal changes to further improve instruction following on these tasks.


Reference:

[1] IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessment. ICLR, 2026.

[2] Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset. CVPR, 2026.

[3] Bernini: Latent Semantic Planning for Video Diffusion. arXiv:2605.22344, 2026.

Dear Reviewer 9VYN,

Thank you again for your thoughtful review and valuable suggestions. We have carefully addressed your concerns and revised the manuscript accordingly, with changes highlighted in blue. As the discussion deadline approaches, we would sincerely appreciate it if you could revisit our responses and let us know whether any concerns remain. We would be happy to provide further clarification.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_NXKZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DUAL addresses the plan–execution gap in instruction-only video editing. Its Target-State Path sends mid-to-late MLLM query features to DiT self-attention. Its Relational-Constraint Path combines final-layer queries with native text tokens for cross-attention guidance. On IVEBench it ties the best open-method Overall score, and on OpenVEBench it leads the listed open methods. Path-removal ablations support the usefulness of both paths.

Critical strengths
  1. Routing Target-State and Relational-Constraint information through self-attention and cross-attention is a clear, plausible interface design that preserves the native text condition.
  2. On OpenVEBench, Overall rises from 3.23 without DUAL to 4.26 with both paths. Removing either path also lowers the score.
  3. DUAL is competitive on both benchmarks without user-provided masks or references, and its 5B renderer has similar measured latency to Kiwi-Edit.
Critical weaknesses
  1. IVEBench Instruction Compliance is 0.37, below Ditto and Bernini, despite the tied Overall score. The submitted materials include no playable videos or user study to assess edit correctness and temporal quality beyond automated scores and selected frames.
  2. The headline 15.2× speedup is measured against 14B Ditto. Compared with 5B Kiwi-Edit, DUAL takes 157 rather than 168 seconds. Reported parameter counts cover only the renderer and exclude the planner and text encoder.
Questions for authors
  1. Can the authors provide matched-prompt playable videos, failure cases, and a blinded user study of edit correctness, source preservation, and motion?
  2. What are the total model footprint and latency contributions of the planner, text encoder, and two paths on the same hardware?
2 comments
AuthorsOpenReview ↗

Response to Reviewer NXKZ

We sincerely thank the reviewer for their constructive comments and insightful suggestions, which have been invaluable in improving our manuscript. We are also grateful for their recognition of the strengths of our work. The revisions have been incorporated in the revised manuscript marked in blue. Please kindly find point-to-point responses below.

About W1 & Q1: IVEBench Instruction Compliance, Matched-Prompt Videos, Failure Cases, and Blinded User Study

Thank you for the suggestion. We have added the blinded user study to Section 5.2 and Appendix A.3 (Table 9), and included matched-prompt playable videos and representative DUAL failure cases in the Supplementary Material.

  • IVEBench Instruction Compliance:

    IVEBench includes more complex spatiotemporal edits, including subject motion and viewpoint changes. [1] These edits require substantial structural changes while preserving source content and temporal consistency, creating a trade-off. Instruction Satisfaction has a higher weight in Instruction Compliance, making it particularly influential in the IC score. [1] Together, these factors contribute to DUAL's lower Instruction Compliance compared with Ditto [2] and Bernini [3], despite the tied Overall score.

  • Blinded User Study:

    We conducted a blinded pairwise user study on 100 cases per baseline, comparing DUAL with each baseline on Edit Correctness, Source Preservation, and Motion Quality using A/B/Tie judgments. The percentages in Table R1 report DUAL's win rates against each listed baseline.

Table R1. Pairwise User Study: DUAL Win Rates vs. Baselines

Compared baselineDUAL Win Rate: Edit Correctness ↑DUAL Win Rate: Source Preservation ↑DUAL Win Rate: Motion Quality ↑
Kiwi65%61%51%
Ditto55%64%45%
OmniWeaving56%69%55%
Lance57%71%59%
Bernini53%66%53%

About W2 & Q2: 15.2× Speedup, Total Model Footprint, and Latency Contributions

We have clarified the comparison baseline in the Abstract and Introduction as up to 15.2 times faster than Ditto in average inference latency [2]. We have clarified the end-to-end latency measurement in the caption of Table 5 and added the full model footprint (15.52B parameters, 5.38B trainable) and component-level latency breakdown in Appendix A.3 (Table 10).

  • Total model footprint and latency contributions:

    The breakdown in Table R2 was measured on NVIDIA H800 GPUs with 50 denoising steps. Image and video query parameters are included in the total model footprint, while their latency is accounted for within the MLLM planner.

Table R2. Model Size and Inference Latency

ModuleParametersTrainable paramsMean (s)Std (s)
MLLM Planner (Qwen + LoRA)3.78B29.49M2.01450.2570
T5 Encoding (UMT5-XXL forward)5.68B00.04200.0328
T5 CPU↔GPU Offloading——11.54940.3822
Video Preprocessing (CPU decode/resize)——5.34090.2913
RCP (SemanticRefiner)327.26M327.26M0.00980.0110
TSP (DetailProjector)23.34M23.34M0.00050.0006
VAE Encoding0.70B03.59800.1314
DiT (50 steps)5.00B5.00B122.67064.3817
VAE Decoding0.70B012.11890.3605
End-to-End15.52B5.38B157.34464.4322

Reference:

[1] IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessment. ICLR, 2026.

[2] Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset. CVPR, 2026.

[3] Bernini: Latent Semantic Planning for Video Diffusion. arXiv:2605.22344, 2026.

Dear Reviewer NXKZ,

Thank you again for your thoughtful review and valuable suggestions. We have carefully addressed your concerns and revised the manuscript accordingly, with changes highlighted in blue. As the discussion deadline approaches, we would sincerely appreciate it if you could revisit our responses and let us know whether any concerns remain. We would be happy to provide further clarification.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_wGtWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The authors formulate instruction-only video editing as a task of translating an edit plan into faithful execution and propose DUAL to route target-state information and relational constraints through separate paths. The Target-State Path injects features from multiple MLLM layers into DiT self-attention, while the Relational-Constraint Path combines final-layer features with text embeddings to guide which object should change and how through cross-attention. Experiments on IVEBench and OpenVEBench showed the effectiveness of DUAL.

Critical strengths
  1. The authors analyze how a single conditioning representation may contribute to the gap between planning and execution in video editing, motivating separate treatment of target-state information and relational constraints.

  2. DUAL allows MLLM features to interact with video tokens while preserving native text guidance, enabling instruction-only video editing without masks or reference images. It achieves the highest overall score among the compared open methods on OpenVEBench and ties for the highest score on IVEBench.

Critical weaknesses
  1. Figure 2 and Section 4 make the data and gradient flow difficult to understand, while Table 6 does not explicitly include the gate, projector, and refiner in its list of trainable parameters. In particular, Equation (6) specifies α(0) = 0 without clearly stating whether α is trainable or how its value changes during training.

  2. The ablations lack a single-representation baseline retaining the same MLLM information and similar capacity, leaving the proposed explanation of the gap between planning and execution insufficiently validated (Section 5.3, Tables 3 and 4). Missing training dataset details and an incomplete description of the inference hardware configuration also limit reproducibility and interpretation of the performance and efficiency comparisons (Sections 4.3 and 5.4; Tables 6–8).

Questions for authors
  1. How are the source and target latents converted into tokens and combined with plan tokens, and which parameters in the projector, refiner, DiT, MLLM queries, and LoRA modules are updated by the flow loss?

  2. Is α in Equation (6) a trainable parameter initialized to zero and subsequently optimized through the flow loss? If it is not trainable, what determines its value after initialization?

  3. Does DUAL retain its advantage over a single-representation baseline using the same MLLM features, similar model capacity, and the same training data and procedure.

6 comments
AuthorsOpenReview ↗

Response to Reviewer wGtW

We sincerely thank the reviewer for their constructive comments and insightful suggestions, which have been invaluable in improving our manuscript. We are also grateful for their recognition of the strengths of our work. The revisions have been incorporated in the revised manuscript marked in blue. Please kindly find point-to-point responses below.

About W1 & Q1: Data Flow, Gradient Flow, and Trainable Parameters

Thank you for pointing this out. We have clarified latent tokenization and attention routing in Section 4.1, gradient flow in Section 4.3 and Appendix A.1, and stage-wise trainable modules in Appendix Table 6.

  • Latent-to-token conversion and combination with plan tokens:

    • Latent Tokenization: The frozen VAE encodes source and target videos into latents. DiT's 3D patch embedding (kernel/stride 1×2×21\times2\times2) converts them into source and noisy target tokens.

    • Target-State Path (TSP): Averages MLLM query states from layers 20/24/28/32 and projects them into 512 plan tokens, concatenated as [Plan; Source; Noisy Target] for DiT self-attention.

    • Relational-Constraint Path (RCP): Concatenates projected final-layer MLLM queries with native T5 tokens as [Final-layer Queries; T5], which are processed by SemanticRefiner to produce the refined text condition for DiT cross-attention.

  • Gradient flow and trainable parameters:

    The flow loss supervises only target-token predictions. Gradients propagate through DiT self-attention to the TSP and through cross-attention to the RCP. Even when the DiT is frozen in Stage 1, gradients still reach both paths. Trainable modules by stage are listed in Table R1.

    Table R1. Stage-wise Trainable Modules

    StageTrainable modules
    Stage 1Image queries; MLLM rank-64 Q/K/V/O LoRA; DetailProjector; SemanticRefiner; α\alpha
    Stages 2–3Image/video queries; DiT; MLLM rank-64 Q/K/V/O LoRA; DetailProjector; SemanticRefiner; α\alpha

    The MLLM backbone, T5 encoder, and VAE remain frozen throughout.

About Q2: Trainable Gate α and Flow-Loss Optimization

Yes. α\alpha is a trainable scalar parameter (nn.Parameter) initialized to zero and updated through the target-only flow loss. We have revised Section 4.2 after Equation 6 to clarify that α\alpha is trainable and updated Appendix Table 6 accordingly.

About W2 & Q3: Matched Single-Representation Baseline, Training Data, and Inference Hardware

Thank you for the suggestion. We have added the Single Cross results to Table 3, its routing configuration to Table 4, and the matched comparison to Section 5.3. Training hardware and datasets are specified in Appendix Tables 6 and 7, respectively.

  • Matched single-representation baseline:

    We conducted a Single Cross ablation using the same MLLM features, connector capacity, training data, procedure, and evaluation settings. Unlike DUAL's two paths, Single Cross combines detail and semantic conditioning through cross-attention. DUAL outperforms Single Cross overall and in all eight categories (Table R2), supporting the benefit of distinct routing.

Table R2. Single Cross Ablation Results

MethodMLLM featuresConnector paramsOverall ↑Global Style ↑Background ↑Local Change ↑Local Remove ↑Local Add ↑Subtitle ↑Creative ↑Camera ↑
Full DUAL20/24/28/32 + Final350.60M4.263.903.944.444.504.434.963.683.84
Single Cross20/24/28/32 + Final350.60M3.552.903.743.863.323.733.962.993.62
  • Training datasets and inference hardware:

    Training and inference use NVIDIA H800 GPUs; data sources are listed in Table R3.

    Table R3. Training Data Sources

    Data TypeDatasets
    I2IGPT-Image-Edit-1.5M [1]
    V2VRefVIE [2], Sparkle [3], ReCo-Data [4], OpenVE-3M (Subtitles Edit) [5], GOKU-2M [6], internal V2V data
    T2VOpenVidHD-0.4M [7]

Reference:

[1] GPT-IMAGE-EDIT-1.5M: A Million-Scale, GPT-Generated Image Dataset. arXiv:2507.21033, 2025.

[2] Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance. arXiv:2603.02175, 2026.

[3] Sparkle: Realizing Lively Instruction-Guided Video Background Replacement via Decoupled Guidance. arXiv:2605.06535, 2026.

[4] In-Context Generation with Regional Constraints for Instructional Video Editing. ICML, 2026.

[5] OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing. arXiv:2512.07826, 2025.

[6] Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing. arXiv:2606.30599, 2026.

[7] OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation. ICLR, 2025.

Dear Reviewer wGtW,

Thank you again for your thoughtful review and valuable suggestions. We have carefully addressed your concerns and revised the manuscript accordingly, with changes highlighted in blue. As the discussion deadline approaches, we would sincerely appreciate it if you could revisit our responses and let us know whether any concerns remain. We would be happy to provide further clarification.

Thank you for your time and consideration.

Best regards,
The Authors

  1. again the figure 2 is hard to follow currently, for example the encoder(blue block) output latent feature(blue) and also orange one, so what is the meaning for this, and why the VAE outputs feature is assigned to the red block, which seems to be a motivation of method. I think the architecture figure is not readable although the authors give detail description in the following section.
  2. I appreciate the comparison between dual and single, I guess the Single Cross routes MLLM features through cross-attention, as in DUAL’s second path, could the authors provide more detail about the difference between these two method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.