- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents CrossEdit, a unified model for instruction-based image, audio, and video editing. The main idea is to address the lack of high-quality paired data for complex audiovisual editing through cross-modal training.
Specifically, the authors leverage the additive nature of audio signals to construct synthetic audio editing data with compositional instructions. Together with audiovisual masked reconstruction, this allows the model to perform complex audiovisual scene edits without training on corresponding AV scene editing pairs.
The paper also introduces CrossEditBench, a benchmark covering different types of audiovisual editing instructions, and AV-FES, a metric that combines instruction following and consistency. The experimental results show promising editing quality and improvements over several existing baselines. Ablation studies further demonstrate the benefits of synthetic audio editing data and audiovisual masked reconstruction.
Critical strengths
-
Reasonable audiovisual editing quality. The paper shows good qualitative results on several audiovisual editing tasks. In particular, the model can make relatively complex modifications while preserving much of the original scene. The joint audio-video editing examples are also interesting, although it is difficult to fully judge temporal consistency from the qualitative results alone.
-
A useful approach to constructing cross-modal editing data. The idea of using synthetic audio editing data to improve audiovisual editing is reasonable. Since audio signals can be mixed directly, it is easier to construct editing pairs involving multiple operations, such as adding or removing sound sources. The authors use these data together with AV masked reconstruction to train the model without requiring paired data for the target AV scene editing tasks. The improvements in Table 3 provide some evidence that this training strategy is effective.
-
Evaluation across different editing tasks. The paper includes experiments on audiovisual scene editing, lip-synced speech editing, image editing, video editing, and audio editing. CrossEditBench also considers different types of instructions, including edits that require synchronization between audio and video. The comparisons and ablation studies help assess the contribution of the proposed training components, although some aspects of the evaluation could be more thorough.
Critical weaknesses
-
Limited video duration and lack of evaluation on longer sequences. Although the editing results look promising, I noticed that the lip-synced speech editing benchmark only uses 2-second clips, while the paper mentions training and evaluating videos of up to 10 seconds. I think 2 seconds is quite short for video editing, and some important issues may not become apparent at this length, such as identity drift, temporal inconsistency, or degraded audiovisual synchronization.
-
The evidence for cross-modal transfer could be more convincing. One of the central claims of the paper is that compositional instruction-following capabilities learned from audio editing can transfer to visual editing. Table 3 shows that adding audio editing data improves visual editing performance, which is encouraging. However, I am not fully convinced that this improvement necessarily comes from cross-modal transfer rather than simply having more training data or more diverse instructions.
For example, would adding a similar amount of regular audio generation data lead to comparable improvements? What if the model were trained with simpler audio editing instructions instead of compositional ones? I think these controlled comparisons would help better support the main claim of the paper.
- Some concerns about the AV-FES metric. I understand the motivation behind AV-FES, which tries to balance instruction following and consistency. However, I have some concerns about how it handles single-modality editing tasks.
For example, if an instruction only asks to modify the audio, the ideal video output should remain unchanged. However, according to Equation (2), an unchanged video would generally receive zero instruction gain, resulting in zero visual FES even if the video is perfectly preserved. Although the authors discuss the special zero-headroom case in the appendix and provide additional results focusing on edited modalities, I think the behavior of the main metric in these cases still needs clarification.
Questions for authors
-
Video duration: Why is the lip-synced speech editing evaluation limited to 2-second clips? Is this mainly due to the dataset, or does editing quality degrade on longer sequences? Could the authors provide results on 8–10 second videos, particularly regarding identity consistency and audiovisual synchronization?
-
Cross-modal transfer: Table 3 shows that adding audio editing data improves visual editing performance. Have the authors tried using a similar amount of regular audio generation data, or simpler audio editing instructions, as a control? This would help clarify whether the improvement comes from compositional cross-modal transfer or additional training data and instruction diversity.
-
AV-FES: How does the proposed metric handle instructions that only require editing one modality? For example, when the video should remain completely unchanged, its instruction gain may be zero according to Equation (2). Would the main model rankings remain consistent if preservation of the unedited modality were evaluated separately?
-
Zero-shot setting: The model has already been trained on V2V editing, lip-synced AV2AV editing, and audiovisual masked reconstruction. Could the authors further clarify how much of the zero-shot audiovisual scene editing capability comes from compositional instruction transfer, rather than combining editing capabilities already learned during pretraining and fine-tuning?
2 comments
Thank you for your detailed explanation. My main concerns have been resolved. I will maintain my score (3: Weak acceptance).
Rebuttal
We only evaluate on such clips since the model is only trained to perform AV-to-AV tasks on short utterances of a few seconds, and since the evaluation data only contains short clips. The former is due to the heavy computational costs (sequence length) of the extra modalities in both input and output - a single 2 second generation takes several minutes on 64 A100 GPUs. (The sentence in the limitations section is unclear about the max sequence length: the 10 second limit is for video-to-video, not AV-to-AV. We have updated the limitations accordingly.)
CrossEdit-Base’s training data already includes a large amount of audio generation data, so we are confident that the improvement comes from our addition of audio editing data. We also note that model is also trained with simple audio editing instructions, since we sample 1 to N sound sources to do the editing on (if a value of 1 is sampled, its a simple audio edit)
We also want to emphasize that our motivation comes from the combination of cross-modal transfer and increased instruction diversity, not from one only factor. If there was no cross-modal transfer, the visual editing would not be improved despite increased instruction diversity in audio. Simple audio instructions alone would likely introduce minimal benefit, since they are very similar to those already in the visual modalities.
Nevertheless, we agree that having a control with simpler audio editing instructions would be a valuable addition. While we do not have the compute to run the experiments within the rebuttal period, we hope to include it in future manuscripts if resources permit.
We thank the reviewer for pointing this out - this was an edge case we missed when constructing the metric. We operated under the assumption that a judge would assign an instruction following score of 5 for correctly performing a pass-through, which is covered by setting the gain to 1 and setting to 0 in all other cases. However, this is not always the case in practice due to judge noise. Human annotators usually (but not always) assigned a score of 5, but the LLM judges never did. As such, the overall reported scores were lower than intended since the model was being punished when it was correct.
To resolve this issue, we slightly reformulated the averaging of the metric to be at the clip-level instead of clip+modality level. This makes sure that preservation of the unedited modality is evaluated properly within the AV-FES metric itself. Please refer to the revised Section 4.2 and Appendix C.3 for more details.
This change uniformly increases the scores for all models, but with little change in rankings within judges/human ratings (spearmann 0.972 between old and new rankings) while increasing the correlation between humans and LLM judges (0.97 Pearson for both judge models). For example, CrossEdit-Base improved from 0.26 AV-FES → 0.34 and CrossEdit from 0.43 → 0.52 on human scores. We have updated all results in the manuscript to reflect the new methodology.
The zero-shot AV scene editing capabilities are derived from two key aspects:
If the model was only learning to combine capabilities learned during training, only (2) would be necessary. The improvements from only adding in the audio editing data in Table 3 and Figure 5 already suggest that is not the case. Otherwise, we would see little changes in the visual results when adding in audio editing data only. Table 8 in the appendix also supports this: visual consistency in AV voice cloning (a trained task) improves after adding in audio editing. Qualitative evidence is shown in Figure 9 - the model has learned better general input consistency thanks to the audio.
We are also certain that the capabilities do not arise from combining the three video tasks (V2V, lip sync editing, and AV reconstruction) since we do not have any such compositional instructions in any of these categories (V2V contains similar instructions to those from the EditVerse editing pipeline - object removal / addition / change, style transfer, and camera change).