acceptodds
Under review as a conference paper at ICLR 2027

CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing

24 views

Abstract

Multimedia editing requires generative models to interpret complex, compositional instructions and modify only the desired elements across modalities, a capability largely beyond existing systems. Current approaches are typically limited to simple, single-attribute edits within one modality, since diverse, high-quality editing pairs are hard to obtain, especially for cross-modal tasks where aligned audiovisual (AV) data is scarce. We present CrossEdit, a unified omni-modal editing model for images, audio, and video that performs AV movie scene edits zero-shot through cross-modal transfer. Our key observation is that complex instruction-following, once learned in any modality, generalizes to others. We exploit the additive nature of acoustic signals to procedurally generate large-scale audio editing pairs with compositional instructions, and show that fine-tuning on this synthetic data and self-supervised AV masked reconstruction, alongside a targeted set of curated cross-modal tasks, induces instruction-following that transfers zero-shot to unseen modality and instruction combinations. To evaluate this capability, we release CrossEditBench, a human-annotated benchmark of AV edits on movie scenes, and propose AV-FES, a metric that jointly scores instruction following and consistency. We show that our proposed techniques improve performance on zero-shot AV movie scene editing, while maintaining or improving performance on lip-synced speech editing and standard image, video, and audio editing benchmarks. Demos are available at https://crossedit.github.io.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
diffusion, multi-modal, audio-visual, speech

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Je5iOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CrossEdit, a unified model for instruction-based image, audio, and video editing. The main idea is to address the lack of high-quality paired data for complex audiovisual editing through cross-modal training.

Specifically, the authors leverage the additive nature of audio signals to construct synthetic audio editing data with compositional instructions. Together with audiovisual masked reconstruction, this allows the model to perform complex audiovisual scene edits without training on corresponding AV scene editing pairs.

The paper also introduces CrossEditBench, a benchmark covering different types of audiovisual editing instructions, and AV-FES, a metric that combines instruction following and consistency. The experimental results show promising editing quality and improvements over several existing baselines. Ablation studies further demonstrate the benefits of synthetic audio editing data and audiovisual masked reconstruction.

Critical strengths
  1. Reasonable audiovisual editing quality. The paper shows good qualitative results on several audiovisual editing tasks. In particular, the model can make relatively complex modifications while preserving much of the original scene. The joint audio-video editing examples are also interesting, although it is difficult to fully judge temporal consistency from the qualitative results alone.

  2. A useful approach to constructing cross-modal editing data. The idea of using synthetic audio editing data to improve audiovisual editing is reasonable. Since audio signals can be mixed directly, it is easier to construct editing pairs involving multiple operations, such as adding or removing sound sources. The authors use these data together with AV masked reconstruction to train the model without requiring paired data for the target AV scene editing tasks. The improvements in Table 3 provide some evidence that this training strategy is effective.

  3. Evaluation across different editing tasks. The paper includes experiments on audiovisual scene editing, lip-synced speech editing, image editing, video editing, and audio editing. CrossEditBench also considers different types of instructions, including edits that require synchronization between audio and video. The comparisons and ablation studies help assess the contribution of the proposed training components, although some aspects of the evaluation could be more thorough.

Critical weaknesses
  1. Limited video duration and lack of evaluation on longer sequences. Although the editing results look promising, I noticed that the lip-synced speech editing benchmark only uses 2-second clips, while the paper mentions training and evaluating videos of up to 10 seconds. I think 2 seconds is quite short for video editing, and some important issues may not become apparent at this length, such as identity drift, temporal inconsistency, or degraded audiovisual synchronization.

  2. The evidence for cross-modal transfer could be more convincing. One of the central claims of the paper is that compositional instruction-following capabilities learned from audio editing can transfer to visual editing. Table 3 shows that adding audio editing data improves visual editing performance, which is encouraging. However, I am not fully convinced that this improvement necessarily comes from cross-modal transfer rather than simply having more training data or more diverse instructions.

For example, would adding a similar amount of regular audio generation data lead to comparable improvements? What if the model were trained with simpler audio editing instructions instead of compositional ones? I think these controlled comparisons would help better support the main claim of the paper.

  1. Some concerns about the AV-FES metric. I understand the motivation behind AV-FES, which tries to balance instruction following and consistency. However, I have some concerns about how it handles single-modality editing tasks.

For example, if an instruction only asks to modify the audio, the ideal video output should remain unchanged. However, according to Equation (2), an unchanged video would generally receive zero instruction gain, resulting in zero visual FES even if the video is perfectly preserved. Although the authors discuss the special zero-headroom case in the appendix and provide additional results focusing on edited modalities, I think the behavior of the main metric in these cases still needs clarification.

Questions for authors
  • Video duration: Why is the lip-synced speech editing evaluation limited to 2-second clips? Is this mainly due to the dataset, or does editing quality degrade on longer sequences? Could the authors provide results on 8–10 second videos, particularly regarding identity consistency and audiovisual synchronization?

  • Cross-modal transfer: Table 3 shows that adding audio editing data improves visual editing performance. Have the authors tried using a similar amount of regular audio generation data, or simpler audio editing instructions, as a control? This would help clarify whether the improvement comes from compositional cross-modal transfer or additional training data and instruction diversity.

  • AV-FES: How does the proposed metric handle instructions that only require editing one modality? For example, when the video should remain completely unchanged, its instruction gain may be zero according to Equation (2). Would the main model rankings remain consistent if preservation of the unedited modality were evaluated separately?

  • Zero-shot setting: The model has already been trained on V2V editing, lip-synced AV2AV editing, and audiovisual masked reconstruction. Could the authors further clarify how much of the zero-shot audiovisual scene editing capability comes from compositional instruction transfer, rather than combining editing capabilities already learned during pretraining and fine-tuning?

2 comments

Video duration

We only evaluate on such clips since the model is only trained to perform AV-to-AV tasks on short utterances of a few seconds, and since the evaluation data only contains short clips. The former is due to the heavy computational costs (sequence length) of the extra modalities in both input and output - a single 2 second generation takes several minutes on 64 A100 GPUs. (The sentence in the limitations section is unclear about the max sequence length: the 10 second limit is for video-to-video, not AV-to-AV. We have updated the limitations accordingly.)

Cross-modal transfer

CrossEdit-Base’s training data already includes a large amount of audio generation data, so we are confident that the improvement comes from our addition of audio editing data. We also note that model is also trained with simple audio editing instructions, since we sample 1 to N sound sources to do the editing on (if a value of 1 is sampled, its a simple audio edit)

We also want to emphasize that our motivation comes from the combination of cross-modal transfer and increased instruction diversity, not from one only factor. If there was no cross-modal transfer, the visual editing would not be improved despite increased instruction diversity in audio. Simple audio instructions alone would likely introduce minimal benefit, since they are very similar to those already in the visual modalities.

Nevertheless, we agree that having a control with simpler audio editing instructions would be a valuable addition. While we do not have the compute to run the experiments within the rebuttal period, we hope to include it in future manuscripts if resources permit.

AV-FES

We thank the reviewer for pointing this out - this was an edge case we missed when constructing the metric. We operated under the assumption that a judge would assign an instruction following score of 5 for correctly performing a pass-through, which is covered by setting the gain to 1 and setting to 0 in all other cases. However, this is not always the case in practice due to judge noise. Human annotators usually (but not always) assigned a score of 5, but the LLM judges never did. As such, the overall reported scores were lower than intended since the model was being punished when it was correct.

To resolve this issue, we slightly reformulated the averaging of the metric to be at the clip-level instead of clip+modality level. This makes sure that preservation of the unedited modality is evaluated properly within the AV-FES metric itself. Please refer to the revised Section 4.2 and Appendix C.3 for more details.

This change uniformly increases the scores for all models, but with little change in rankings within judges/human ratings (spearmann 0.972 between old and new rankings) while increasing the correlation between humans and LLM judges (0.97 Pearson for both judge models). For example, CrossEdit-Base improved from 0.26 AV-FES →\rightarrow 0.34 and CrossEdit from 0.43 →\rightarrow 0.52 on human scores. We have updated all results in the manuscript to reflect the new methodology.

Zero-shot setting:

The zero-shot AV scene editing capabilities are derived from two key aspects:

  1. Enhanced instruction following via compositional instruction transfer
  2. Learning to better understand and generate AV via masked reconstruction

If the model was only learning to combine capabilities learned during training, only (2) would be necessary. The improvements from only adding in the audio editing data in Table 3 and Figure 5 already suggest that is not the case. Otherwise, we would see little changes in the visual results when adding in audio editing data only. Table 8 in the appendix also supports this: visual consistency in AV voice cloning (a trained task) improves after adding in audio editing. Qualitative evidence is shown in Figure 9 - the model has learned better general input consistency thanks to the audio.

We are also certain that the capabilities do not arise from combining the three video tasks (V2V, lip sync editing, and AV reconstruction) since we do not have any such compositional instructions in any of these categories (V2V contains similar instructions to those from the EditVerse editing pipeline - object removal / addition / change, style transfer, and camera change).

Thank you for your detailed explanation. My main concerns have been resolved. I will maintain my score (3: Weak acceptance).

Reviewer_GJSXOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CrossEdit studies whether instruction-following skills learned in individual modalities can transfer to joint audio-visual scene editing. Starting from an internal omni-modal editing backbone, it adds synthetic compositional audio-editing data and audio-visual masked reconstruction. The paper evaluates task-level zero-shot transfer to movie-scene editing through CrossEditBench and the proposed AV-FES metric, alongside speech, image, video, and audio editing evaluations. AV-FES combines instruction-following gains and content consistency separately for audio and video.

Critical strengths
  1. Synthetic audio-editing supervision improves visual and joint editing, offering a useful direction for reducing dependence on paired AV editing data.
  2. CrossEditBench tests realistic instructions involving single- and joint-modality changes, preservation, and synchronization.
  3. The study includes varied editing baselines, human ratings, a second automatic judge, and capability-retention experiments.
Critical weaknesses
  1. Novelty and attribution need sharper positioning. The paper already compares several relevant joint editors, but JAVEdit is not discussed. Please clarify what is new about the transfer capability and scope the benchmark claim against JAVEditBench and AVE-Compass, whose settings may differ from movie-scene editing. The internal CrossEdit-Base initialization is identified, but limited visibility into its pre-training makes attribution harder. Clearer data-scope information and matched-budget controls would help separate cross-modal transfer from additional instruction diversity and training volume.
  2. Temporal synchronization is insufficiently evaluated for scene editing. Table 2 uses ImageBind, which measures semantic correspondence rather than event timing. The human study does not report a dedicated synchronization axis, while Table 4's lip-sync results concern speech editing. Consequently, neither establishes synchronization for the main general scene-editing task. Please report a suitable temporal metric, such as DeSync where applicable, and/or targeted human judgments on synchronized edit categories.
  3. The semantic-encoder choice needs justification. Qwen3-VL provides semantic embeddings for text, images, and video, while audio is supplied through VAE representations. This gives the modalities different access to high-level semantic conditioning. Please explain why Qwen3-VL was preferred over an encoder such as Qwen3-Omni that accepts both audio and visual inputs, including any compatibility, cost, or performance considerations.
Questions for authors

please see weaknesses

3 comments

...JAVEdit is not discussed.. clarify what is new about transfer capability and scope the benchmark claim against JAVEditBench and AVE-Compass...

We thank the reviewers for pointing out this relevant work, which were missed due to the recency of their release. We have updated our related work with references to these works and how they differ from ours. How they differ from CrossEdit:

JAVEdit: fine-tunes a pre-trained AV generation model for the target AV editing task directly. They train on synthetic editing triplets (instruction + input + output) generated from a pipeline of per-modality generative models (ie. image editing + animation + TTS). They do not make any claims or present results about transfer capabilities, since their method is targeted towards in-domain performance. In contrast, the goal of CrossEdit is zero-shot cross-modal transfer without directly generating such synthetic editing triplets.They also focus primarily on human-centric editing, whereas Cross-Edit targets broader AV editing.

JAVEditBench: is the benchmark created to evaluate JAVEdit. Again, the benchmark focuses on human-centric editing (ie. instructions are always about either editing the human subject or what is around them), while CrossEditBench supports a broad range of AV edit subjects (like animals, spaceships, robots, cars, etc). Finally, edit instructions in JAVEditBench are long, specific and targeted (“Change the man's clothing to Spanish carnival attire, wearing a white sports shirt and a strange hat, while keeping the person's facial features, expression, camera angle, and background unchanged.”) while CrossEditBench uses open-ended abstract instructions that are a stronger test of cross-modal understanding (“Make the video and music more happy”).

AVE-Compass: is more similar to CrossEditBench in that it targets general AV editing. However, the instructions in the benchmark are also highly specific in telling the model what exactly to generate, while CrossEditBench uses general open instructions that require deep language and cross-modal understanding. It is also released only under non-commercial use due to heavy use of copyrighted content, while CrossEditBench is commercially usable since it only includes public domain data.

The internal CrossEdit-Base initialization is identified, but limited visibility into its pre-training makes attribution harder. Clearer data-scope information and matched-budget controls would help separate cross-modal transfer from additional instruction diversity and training volume.

We request the reviewer clarify what additional experiments/information are desired, as it is unclear from the text.

If the concern is about the additional fine-tuning steps, we note that the base model was already well converged prior to our fine-tuning (400K+ steps of training), and showed little variance across checkpoints at this stage, which makes the impact of the extra steps minimal.

If the concern is about the additional instruction diversity from audio editing, we would like to emphasize that such diversity is the primary goal of our method - we want to transfer the better instruction following from audio to other modalities.

Temporal synchronization

While we did not the the resources to design and conduct a human study within the rebuttal period, we evaluated five automatic AV-synchronization metrics(Gemini / QwenOmni judges, Synchformer DeSync and its in-sync probability, and Diff-Foley). On CrossEditBench, none yield valid results.

We test this by shifting the audio of the unedited source clips by known offset as a control. Synchformer recovers the shift only 43% of the time (29% by chance) and Diff-Foley prefers the unshifted version only 51% of the time (50% is chance). In contrast, Synchformer recovers 94% of shifts on Music-AVQA, 90% on our IMDB speech dataset, and 100% on LRS3. This indicates that the metric is not robust to the movie content of CrossEditBench, where there is background music, off-screen dialogue, and SFX. The LLM judges consistently rate the unedited sources as out of sync. On a 1-5 scale, Gemini and Qwen give the input a score of 2.81 and 3.00. These results suggest that there is significant research necessary to even develop understanding/scoring models for synchrony in complex AV scenes. Full results are in Appendix E in the updated manuscript. While we do not believe they are meaningful on CrossEditBench since no metric passes the control test, we also include the system scores in Table 15 for transparency.

Semantic-encoder

We apologize for the confusion. We only use Qwen-VL embeddings for the text instructions. The diagram was based off an early version of the model that also used Qwen-VL as visual embeddings, but we found that there was no practical difference in performance despite the extra cost. In this sense, there is little difference between using QwenOmni and Qwen-VL. We have updated the draft with the correct information

The draft has been updated with the corrected details of the semantic encoder (Section 3.1, Figure 2, Appendix A), which was accidentally left out in the prior update. As the rebuttal phase is about to end, we would appreciate if the reviewer could let us know if the concerns have been addressed and/or if there are any additional questions we can answer.

Thanks for the response, but I'm still not convinced about the applicability and scalability of this paper. The authors claim to perform AV movie scene edits, while the provided demos are more likely toy results. Especially in the comparison against Minimax-H3, the visual quality of the edited video and the consistency of the non-edited parts are very poor. The proposed training method is nothing special, and the diversity and cinematic quality of CrossEditBench (AV->AV) are also inferior to AVE-Compass. Therefore, I changed my rating to 1.

Reviewer_MQ51OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper presents CrossEdit, a single model that edits images, audio and video. The aim is to edit movie scenes from a free-form instruction, changing both the audio and the video, even though there's no paired training data for this task. The authors start from an internal base checkpoint that can already edit images, video and lip-synced speech, and then finetune the model with synthetic audio-editing data and video inpaint data.
  • The authors' main argument is that following instructions doesn't depend much on the modality, so what the model learns from audio carries over to joint audio-video editing without any direct training.
  • A 100-clip benchmark is released.
  • Demo page is quite promising.
  • On the technical side alone I'd lean weak accept. The problem is interesting, the benchmark and metric are useful, and the results beat a wide range of baselines. What pushes me to weak reject for now is the unclear copyright mentioned in the Details Of Ethics Concerns part.
Critical strengths
  • Audio-video scene editing has no paired data, and getting around that with cheap synthetic audio data is well motivated.
  • The comparison is broad including cascades, sampling-based, agentic and end-to-end systems. The authors check results with two judges and a human study with confidence intervals.
  • The benchmark is a useful contribution to the community.
Critical weaknesses
  • The base model can already do video and lip-sync AV editing, and masked inpainting is itself a kind of AV-to-AV editing. Two controls would help a lot including masked reconstruction alone, and the base model trained for 10K more steps on its original data. Adding the audio data alone also hurts the synced audio and video categories. So I'm not fully convinced by the zero-shot claim.
  • Another concern is that the model doesn't preserve the video well. Visual consistency is 2.81, against 3.97 for the input and 3.38 for MiniMax.
  • The human study leaves out the two strongest baselines, MiniMax H3 and InstructAV2AV.
  • The paper says clips go up to 10 seconds, but the longest demo is about 4 seconds for video.
Questions for authors
  • Please check my comments in Critical Weaknesses section as well as Details Of Ethics Concerns.
2 comments

The base model can already do video and lip-sync AV editing, and masked inpainting is itself a kind of AV-to-AV editing. Two controls would help a lot including masked reconstruction alone, and the base model trained for 10K more steps on its original data.

We agree that such experiments would indeed be useful. While we do not have sufficient compute to perform these experiments within the rebuttal period, we hope to include these results in the future if resources permit. We also emphasize that the base model was already well converged prior to our fine-tuning (400K+ steps of training), and showed little variance across checkpoints at this stage.

Adding the audio data alone also hurts the synced audio and video categories. So I'm not fully convinced by the zero-shot claim.

We emphasize that synced audio and synced video are only 2 of the 6 categories - adding audio editing data yields clear improvements in the 4 other categories. ¾ of these categories observe an absolute increase of 0.13 or more FES$^1$, with a max gain of 0.24 (synced audio and synced video dropped by 0.13). We believe this is sufficient evidence for our claim of zero-shot transfer.

1^1 Numbers from the original draft. Reviewer Je5i pointed out an issue with the original AV-FES formula, which we found to incorrectly punish models in certain cases. After a small tweak to fix it, all scores have generally increased, and the zero-shot gains from our training method have increased as well. Improvements on 4 categories are now 0.19+ (up from 0.13). On the two synced categories, our method increases scores by 0.05 FES on audio and 0.14 on video.

Another concern is that the model doesn't preserve the video well. Visual consistency is 2.81, against 3.97 for the input and 3.38 for MiniMax.

First, we want to highlight that these are LLM judge scores. If the concern is in the absolute value of the scores, we note that human ratings yield an average judgement of 3.21 for CrossEdit.

High visual consistency is expected when scoring the input against itself. MiniMax does indeed achieve higher consistency, but this is due to it often not performing any editing at all, as indicated by the lower instruction following scores. We also emphasize that models should primarily be judged by the FES score, as it is difficult to intuitively balance gains in consistency with losses in instruction following.

The human study leaves out the two strongest baselines, MiniMax H3 and InstructAV2AV.

Unfortunately the human study was performed before we obtained access to these models, so we were not able to include them. We hope to also evaluate them in future iterations of the manuscript, if resources permit. However, we believe our results are robust to judge choice (human, Qwen, Gemini) due to the strong inter-judge correlation of the rankings.

The paper says clips go up to 10 seconds, but the longest demo is about 4 seconds for video.

We apologize for the confusion. The 10 second limit refers to the max length for video-to-video. For AV-to-AV tasks, our training data was limited to 2-4 seconds, due to the increased computational costs of more modalities. We will correct this in future versions.

The lip-sync evaluation uses an internal set of 600 IMDB celebrity interview clips. Could the authors clarify the source and license of these videos, whether use for generative editing is permitted, and whether showing edited outputs of real celebrities on the demo page is consistent with likeness/consent rights? A public or consented alternative test set would also improve reproducibility.

All of these interview clips were drawn from publicly available sources with appropriate licensing (including additional details risks violating double blind review). We use data from well known public figures to minimize privacy/consent issues of private individuals, which is also done in concurrent submissions to ICLR (https://openreview.net/forum?id=D6wlQBNg0B) and prior work (https://francis-rings.github.io/StableAvatar/). To minimize potential misuse of our demo, the target utterance the edit was performed with (aka X in “Change the speech to X”) uses a real ground truth sentence already spoken by the person within the same interview. We will also update the demo page with appropriate disclaimers and licensing prohibiting redistribution/reuse for outputs, and add our contact information in case individuals would like any samples deleted.

We agree that an alternative public test set would improve reproducibility. We hope to update our rebuttal with these results in the coming days.

Thanks for the answer. I'll raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.