acceptodds
Withdrawn from ICLR 2027

Bayesian Reasoning over Latent Editing Programs for Instruction-Based Video Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Instruction-based video editing requires a model to determine what to change, where to apply the change across space and time, and what to preserve. Existing Diffusion Transformer (DiT) editors learn these decisions only implicitly, providing the same conditioning to every block and supervising attention solely through the generation objective. We propose **ReVEDiT** (**Re**asoning-based **V**ideo **E**diting with **Di**ffusion **T**ransformers), which formulates video editing as hierarchical Bayesian inference over a latent editing program. First, a routing posterior determines the depth at which native textual and visual tokens enter the DiT, so that shallow blocks form the editing intent from MLLM-derived primitives and deeper blocks ground it in the source video. Second, the ideal attention of the edit is treated as a latent variable, for which a reference branch conditioned on the ground-truth edit provides privileged evidence during training; the resulting posterior is distilled into the editing branch at the grounded layers. At inference, the split layer is fixed and the reference branch is discarded, so editing requires a single forward pass. On OpenVE-Bench and EditVerseBench, ReVEDiT achieves the best overall performance among open-source methods under two MLLM judges, with particularly strong results on localized edits.

Primary area:
generative models
Keywords:
Video Editing, Semantic Alignment, Bayesian Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rsVPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes ReVEDiT, which formulates instruction-based video editing as hierarchical Bayesian inference over a latent editing program. The method has two components. The first is Bayesian Granularity Routing, which treats the depth at which native textual and visual tokens enter the DiT as a latent variable, so that shallow blocks use only the MLLM-derived editing tokens to form the editing intent and deeper blocks additionally receive the native tokens for grounding. At inference, the split layer is fixed to the mode of the aggregate posterior over the training set. The second is Bayesian Latent Attention Reasoning. During training, a reference branch that shares parameters with the editing branch and is conditioned on the ground-truth edited video is introduced. The attention of the two branches is treated as noisy observations of the same ideal attention, and the resulting posterior is distilled into the editing branch in attention-score space. The reference branch is discarded at inference. The model is built on Wan2.2-TI2V-5B and Qwen2.5-VL-3B and evaluated on OpenVE-Bench and EditVerseBench with two MLLM judges. The paper reports the best overall results among open-source methods, together with ablations, attention analysis, and a user study.

Critical strengths

The theoretical framework is complete. It seems that the authors have solid mathematical skills.

Critical weaknesses
  • What confuses me most is that the authors build such a complex theoretical framework, yet the final improvement is very small. Appendix Table 4, Table 5, and F.6 strongly suggest that ReVEDiT is further trained from Kiwi-Edit stage-2, and the two share the same backbone, query tokens, and training data sources, but the paper never clearly states where the model is initialized from. Compared with Kiwi-Edit, ReVEDiT is only 0.7 to 1.5 points higher on a 100-point scale. Table 2 confuses me even more: with both modules removed, the "vanilla DiT editor" scores only 71.71, which is more than 4 points lower than the original baseline Kiwi-Edit (75.89). In addition, "w/o granularity routing" sets Ls = 34. By Eq. 9 and Eq. 14, this removes the native tokens from the conditioning of every layer and also makes the set of distillation layers empty, so it does not test the case criticized in the Intro, where every block receives the same conditioning. Why does such a complex theoretical framework bring only such a small gain? I do not see a convincing explanation.
  • The Bayesian framework looks intimidating, but it largely collapses in the end. The authors spend a lot of space deriving the ELBO, then set the coefficients of all prior-matching terms to 0, take the modes of the editing primitives and the routing, and fix the split layer to a constant at inference. What is actually optimized is only Eq. 15, namely a plain flow matching loss plus a KL loss on the attention. The so-called input-adaptive routing is just a hyperparameter at inference, and Table 2(b) is simply a sweep over it. After reading the paper, I still cannot tell what actually works, or what this theoretical derivation contributes to the actual method.
  • There are many inconsistencies throughout the paper. Sec. 5 states that the routing split is Ls = 17, while Table 2 and Appendix F.1 both state 15. Sec. 5 also gives "λ_align = 0.75 with InfoNCE temperature τ = 0.07", but the method section contains no such loss. "RAAA" and "GRTC", which appear in Fig. 3 and Table 4, are never defined in the main text. More seriously, both the abstract and Sec. 5.1 claim that ReVEDiT achieves the best overall performance on both benchmarks under both judges, but in the OpenVE-Bench / Gemini-3.1-Pro block of Table 1, Lucy-Edit (64.77) is clearly higher than ReVEDiT (62.48), and yet 62.48 is marked in bold. These issues make it hard for me to tell which version the authors actually trained and evaluated, and overall this paper feels totally undone to me.
  • The benchmark setting is inconsistent with both the original paper and Kiwi-Edit. OpenVE-Bench originally uses a 1-to-5 scale with 8 categories, and Kiwi-Edit also reports its results on the 1-to-5 scale. This paper instead switches to a 1-to-100 scale, uses two new judges, and reports only 4 categories. More importantly, the authors claim that their rubric follows the OpenVE-3M protocol, but the original OpenVE paper specifies that Instruction Compliance upper-bounds the other two axes, while this paper requires the three axes to be scored independently, which directly changes the meaning of the Overall score. As a result, the ranking also changes: in the results that Kiwi-Edit reports under the original protocol, Kiwi-Edit (2.98) is clearly ahead of Lucy-Edit (2.22), whereas in this paper Lucy-Edit even surpasses both Kiwi-Edit and ReVEDiT. In addition, the paper does not compare with strong baselines such as UniVideo and ReCo, which were compared in the Kiwi-Edit paper.
Questions for authors

see the Weaknesses.

Reviewer_5didOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReVEDiT, an instruction-based video editing framework based on a Diffusion Transformer. Shallow DiT layers use only MLLM-derived editing primitives EE to establish high-level intent, while deeper layers additionally receive native text and video tokens for spatial grounding and source preservation. The paper also introduces a training-time reference branch that uses the ground-truth edited video to guide attention. This branch is removed at inference, and the model uses a fixed routing split in a single forward pass. Experiments on OpenVE-Bench and EditVerseBench report strong performance, particularly on localized edits.

Critical strengths

(1) The coarse-to-fine conditioning idea is intuitive and interesting. Separating semantic intent formation from later visual grounding may prevent dense source information from interfering with the requested edit while preserving appearance and motion in deeper layers.

(2) The reference-guided attention distillation is also useful because it exploits privileged training information without adding inference-time cost. The reported quantitative results and ablations suggest that both routing and attention distillation contribute to performance.

Critical weaknesses

(1) The paper omits discussion and comparison with VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization, which is closely related because it also uses VLM-derived conditioning for instruction-based video editing.

(2) The video samples provided at https://anonymous.4open.science/r/RVEDiT-F8A4/readme.md reveal noticeable artifacts in some generated results. Although the method successfully performs the requested edits, it introduces undesirable side effects in other parts of the scene. For example, in “Local Change - Replace the Woman”, the phone appears distorted, while in “Local Change - Replace the Shirt,” the microphone is deformed.

(3) The paper appears to overstate the Bayesian nature of the method. Although Section 4.2 defines a Gaussian prior, observation model, and precision-weighted posterior, the implemented objective sets the prior-matching coefficients to zero. At inference, the model uses a deterministic fixed split layer without posterior construction or sampling. In practice, the method seems closer to coarse-to-fine conditioning with uncertainty-weighted attention distillation than a complete Bayesian inference procedure.

(4) Some implementation details are inconsistent: the main text reports �0��0�, while the tables and appendix use �1��1�. Equation 15 excludes the routing KL term, but Algorithm 1 includes it. These discrepancies affect reproducibility.

Questions for authors

Can the authors provide an ablation without the Gaussian latent-variable construction to isolate its contribution?

  • Full method: Gaussian latent variable �0��0�, learned prior and uncertainties, precision-weighted posterior, and posterior sampling.
  • Simpler baseline: directly match the reference and editing attention maps, for example:$ \mathrm{KL}\big(\mathrm{softmax}(U_l^r),|,\mathrm{softmax}(U_l^e)\big)$ without Gaussian distributions or uncertainty estimation.

Everything else: routing, architecture, data, and training, should remain unchanged.

Reviewer_TdraOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReVEDiT addresses instruction-based video editing. MLLM-derived editing tokens condition shallow DiT blocks, while native textual and visual tokens enter deeper blocks. During training, a shared-weight reference branch receives the target video and guides the editing branch through attention posterior distillation. At inference, the reference branch is removed and the split layer is fixed.

Critical strengths

The target video provides a concrete realization of a successful edit and therefore offers useful attention supervision beyond the generation loss. The reference branch can be removed at inference, so users do not need to supply a target video. This separation between training supervision and deployment inputs is practically useful.

ReVEDiT achieves the highest Overall score in three of the four benchmark–judge settings in Table 1. The user study also reports a higher mean than Kiwi-Edit, 4.43 versus 3.85. These results are encouraging.

Critical weaknesses

Hierarchical Bayesian inference is presented as the main contribution, but the training specification is inconsistent. The objective discussion sets all three prior-matching weights to zero and uses hard argmax routing without explaining how the task loss trains the router. Algorithm 1 nevertheless retains a routing KL term.

Table 2 defines the no-routing variant as Ls=L=34. Under Eq. (9), every block then receives only editing tokens; under Eq. (14), the distillation set S becomes empty. This changes both native conditioning and distillation supervision, so it does not isolate routing. Its distinction from removing both components also requires explanation.

Questions for authors

1.Which losses produced Table 1? Was the routing KL weight zero? What gradient estimator was used for hard argmax? Was the final split 15 or 17, and how was it obtained?

2.Where is distillation applied when Ls=34? Could native conditioning be exposed at every layer while keeping the distillation layer set unchanged? Under the same split, how does direct attention KL compare with posterior distillation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.