acceptodds
Under review as a conference paper at ICLR 2027

ReasonAVEdit: Explicit Multimodal Reasoning for Instruction-Guided Audio-Visual Editing

Abstract

Existing audio-visual editing methods face a key trade-off between precise control and a unified instruction interface. Explicitly conditioned methods provide precise control, but they rely on task-specific priors such as spatial masks and target audio. Instruction-guided methods need no extra control, but they lack an explicit multimodal reasoning representation and therefore localize edit targets imprecisely. To resolve this tension, we propose ReasonAVEdit, a Perceive–Reason–Edit framework. ReasonAVEdit requires an audio-visual diffusion model to predict multimodal Reasoning Tokens while generating the target audio-visual tokens. The Reasoning Tokens represent the visual edit region, the acoustic target, and the edit time span shared by both modalities. The model thus forms a complete editing decision on its own, without user-provided masks or target audio. The Temporal Query Bridge (TQB) passes the edit time span from the acoustic Reasoning Tokens to the video Temporal Queries, so the edited lips follow only the target's own speech. The updated queries and a single-frame visual region jointly guide video editing, so video-side reasoning no longer grows multiplicatively with video length and resolution. We build ReasonAVEdit-Data with 200K samples and ReasonAVEdit-Bench with 1,200 samples. Across 33 metrics on our benchmark and two public benchmarks, ReasonAVEdit achieves the best result on 32. Relative to the strongest baseline on each metric, it improves audio-visual alignment by 57.4% and reduces the audio-visual temporal offset by 44.5%. Single-frame visual reasoning also reduces inference latency by 51.4% compared with a full-length mask sequence.

Primary area:
generative models
Keywords:
Instruction-Guided Video Edit, Video Edit, Audio-Visual Edit

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xUT8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ReasonAVEdit, an instruction-guided joint audio-visual editing framework that predicts explicit visual and acoustic intermediate representations alongside the edited outputs. A single-frame visual anchor localizes the target, while a target-only acoustic trace represents the sound source and its temporal extent. The Temporal Query Bridge (TQB) transfers acoustic temporal information to video editing. The paper additionally introduces a 200K-sample training dataset and a 1,200-sample benchmark, with evaluation on two external benchmarks.

Critical strengths

Well-motivated problem. Jointly determining where, which source, and when to edit addresses a practical difficulty in instruction-guided audio-visual editing. The proposed representations have clear roles and support editing without user-provided masks or target audio.

Substantial experimental support. The evaluation covers editing fidelity, preservation, synchronization, localization, and efficiency across three benchmarks. Importantly, Table 3 controls the backbone, training data, and optimization, providing useful evidence beyond comparisons with independently trained systems.

Convincing intervention analysis. Appendix D.5 goes beyond conventional component ablations by replacing, shuffling, and modifying the predicted intermediate representations. The corresponding changes in the outputs strengthen the claim that these representations actively influence editing. The spatial, source, and temporal interventions are particularly informative.

Useful representation and efficiency tradeoff. The single-frame visual representation substantially reduces inference cost relative to the dense mask-sequence variant while maintaining competitive output quality. The additional localization diagnostics and discussion of small-object limitations help contextualize this tradeoff.

Critical weaknesses

Positioning relative to prior work. The relationship to VideoCoF should be discussed much more explicitly. VideoCoF already predicts gray-highlighted edit-region representations and jointly denoises reasoning and edited video latents, including compact spatial reasoning configurations. Although Yang et al. (2025) is cited, the manuscript does not adequately explain this close methodological relationship. The additional contributions should therefore be positioned primarily around acoustic reasoning, cross-modal temporal coordination, and their empirical benefits. The reported latency reduction should also remain clearly identified as a comparison against the dense mask-sequence variant.

Ambiguity in temporal evaluation. Appendix C.3 extracts the video edit interval using frame differences within the target subject region. However, Section 3.3 allows appearance changes throughout the clip while restricting synchronized motion changes to the acoustic edit interval. For combined appearance and speech editing, it is unclear how this metric distinguishes intended full-clip appearance changes from unwanted motion outside the speech interval. The applicable task subsets and thresholds should be specified. A direct measurement of unwanted target lip motion during another speaker’s speech would further support the central TQB claim.

Incomplete provenance of the data-generation model. Appendix F states that the mask-based editing model is trained on approximately 120K mask-conditioned editing pairs, but the construction of these initial pairs is insufficiently explained. Since this model subsequently generates the proposed dataset’s edited targets, its initial supervision is an important part of the reproducibility chain. It is also unclear whether the source-video and identity separation checks cover this model’s training data and the external evaluation benchmarks.

Questions for authors

How are the approximately 120K editing pairs used to train the data-generation model obtained, and which overlap checks are applied between this data and the evaluation sets?

How is V-T IoU computed for examples combining full-clip appearance replacement with temporally localized speech editing?

Why is JAVEdit omitted from Table 2 despite being discussed in the related work and using an LTX-2.3 backbone? Please clarify the comparison choice and checkpoint availability at submission time. This is a request for context rather than a standalone reason for rejection.

Reviewer_9P2QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReasonAVEdit for editing video and audio from natural-language instructions without user-provided masks or target audio. Alongside the edited outputs, the model predicts “Reasoning Tokens” that identify the visual region and the original sound segments to modify. (1) The Temporal Query Bridge (TQB) uses timing information from the audio Reasoning Tokens to guide video editing. (2) The model uses a single frame to identify the visual target and an acoustic representation covering the full timeline to identify the relevant sound segments. (3) The authors also construct a training dataset of 200,000 samples and a benchmark of 1,200 samples, with annotations of the visual regions, acoustic targets, and edit intervals. Experiments on this benchmark and two public benchmarks report improvements over the evaluated baselines in editing quality and audio-visual synchronization.

Critical strengths
  • The paper is clearly organized, with the problem formulation, model design, and experiments.
  • The paper’s main contribution is TQB, which uses timing information from the target audio representation to guide video editing. This is a reasonable design for aligning lip movements with the intended speaker’s speech.
  • The authors develop a dataset construction pipeline for training models to edit video and audio from natural-language instructions. The resulting dataset and benchmark could support further research on joint audio-visual editing.
Critical weaknesses
  • While the paper presents a useful integration, several of its central ideas have precedents: reasoning-guided audio generation and editing [1], mask-based routing of audio information to visual regions [2, 3], and compact visual reasoning within a single latent frame [4].
  • Although the reported results are encouraging, the external baselines are not retrained on the proposed dataset. This makes it difficult to distinguish methodological gains from the benefits of the training data. The controlled ablations partly address this concern, but mainly demonstrate the value of intermediate representations and cross-modal conditioning.
  • It remains unclear which information transferred through TQB accounts for the improvements. The acoustic reasoning representation contains both timing and sound content, so the current ablations do not isolate their respective effects.

[1] ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing. NeurIPS 2025
[2] SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editing. arXiv
[3] Bind-Your-Avatar: Multi-Talking-Character Video Generation with Dynamic 3D-mask-based Embedding Router. CVPR 2026
[4] VideoCoF: Unified Video Editing with Temporal Reasoner. CVPR 2026

Questions for authors
  1. Could the authors clarify whether additional baselines, particularly SpongeBob, were considered for evaluation?
  2. The related work associates the absence of explicit intermediate representations with imprecise target localization. This motivation applies to a broad range of methods and does not clearly distinguish the proposed contribution. Could the authors state the specific unresolved limitation addressed by their method?
Reviewer_nL9JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReasonAVEdit for instruction-guided audio-visual editing without user-provided masks or target audio. The model predicts visual and acoustic Reasoning Tokens to represent the edit region, target sound, and temporal interval. A Temporal Query Bridge transfers target-specific acoustic timing to the video branch. The paper also introduces a training dataset and benchmark, and evaluates the approach on three benchmarks through comparisons, component ablations, inference-time interventions, and efficiency measurements.

Critical strengths

1.The explicit representation of spatial, acoustic, and temporal editing decisions addresses a concrete difficulty in instruction-guided audio-visual editing. Table 3 shows complementary benefits from visual and acoustic Reasoning Tokens under the same backbone, training data, and optimization, with further improvements from TQB. This provides useful evidence for the contribution of the proposed components beyond the overall system comparison.

2.The intervention experiments strengthen the evidence that the intermediate representations influence the generated edits. In Appendix D.5, changing the visual anchor, acoustic source, or temporal interval produces corresponding changes in the outputs while keeping the trained model and edit noise fixed. The single-frame representation also offers a practical efficiency benefit: Table 5 reports a latency reduction from 379.4 to 184.2 seconds relative to full-length visual reasoning, while Table D2 documents the associated small-object localization trade-off.

Critical weaknesses

The independence of the benchmark from the upstream data-generation model needs clarification. Appendix E.3 describes source- and identity-disjoint splits for ReasonAVEdit, but Appendix F introduces a separate mask-based editor trained on approximately 120K pairs from OpenHumanVid and Koala-36M. It is unclear whether the benchmark source videos and identities were also excluded from this upstream model’s training data. This matters because the model generates evaluation targets as well as training pairs. The external benchmark results provide useful supporting evidence, but do not resolve this specific question about the newly constructed benchmark.

Questions for authors

1.Were the source videos, near-duplicate clips, and identities in ReasonAVEdit-Bench excluded from the approximately 120K pairs used to train the mask-based AVEdit model in Appendix F? Please clarify whether the disjointness checks in Appendix E.3 also cover this upstream training stage. 2. For the three single-factor interventions in Table D5, please report the number of eligible samples and confidence intervals for the follow rates. Since each intervention applies to a different subset, this would help assess the reliability of the evidence that the corresponding editing decisions control the outputs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.