acceptodds
Under review as a conference paper at ICLR 2027

Edit in Sync: Unified End-to-End Audio-Visual Editing for Human-Centric Videos

17 views

Abstract

Human-centric audio-visual editing requires precise alignment of lip movements, gestures, and expressions with speech. Existing methods process inputs through independent video and audio pipelines, preventing cross-modal interaction between visual editing and audio dubbing. Many of these pipelines further rely on mask-based inpainting. As a result, they suffer from poor lip-audio synchronization, cumulative pipeline errors, and an inherent mask-size dilemma. We present AVE, the first unified end-to-end framework that jointly performs reference-guided person swapping, insertion, and deletion across both audio and video modalities without requiring spatial masks. To overcome data scarcity, we design a synthesis pipeline and introduce AVE-Dataset, the first audio-visual editing dataset to provide 100K pairs with appearance and voice references. By combining Synthetic-to-Real reconstruction with preliminary audio warm-up, our framework transcends synthetic pipeline limits and mitigates audio-video token imbalance. AVE outperforms modality-separate audio-visual editing pipelines in editing realism, temporal consistency, and audio-visual alignment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Editing, Audio Editing, Multimodal Generation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ycmzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AVE, a unified model for human-centric audio-visual editing. It handles person swap, insertion and deletion, edits video and audio together, and needs no spatial masks. The method adds Modality-Aware Sync-RoPE, bidirectional supervision and a three-stage curriculum. On a self-built benchmark of 192 cases, AVE outperforms cascaded pipelines and InstructAV2AV in visual quality, audio quality and lip sync.

Critical strengths
  • The task is useful. Joint audio-visual editing of people with appearance and voice references is closer to real needs than text-only editing.
  • The dataset is a solid contribution. The cross-pair reference construction is careful and helps avoid copy-and-paste shortcuts. If released, it can benefit the community.
  • The backward supervision is reasonable. It lets the model learn from real targets, especially for insertion.
Critical weaknesses
  • The novelty is incremental. The data pipeline combines existing tools. Sync-RoPE extends the design of DreamID-Omni. Reversing removal data for insertion is a common practice.
  • Comparison with the closest work is missing. The text-only InstructAV2AV is a weak baseline for a reference-based task. Other end-to-end methods such as JAVEdit are also not compared.
  • Public benchmarks such as IDBench-Omni are mentioned but not used. VBench scores are near saturation, and there is no user study.
  • The paper states that the method is not tied to a particular backbone, but it is only tested on Ovi. Ovi is a relatively early model. Newer open-source models, such as LTX-2.3 or MiniMax-H3, may give clearly better results. I would like the authors to explain this choice, and ideally show that the framework transfers to a stronger base model.
  • Some demos in the supplementary material show noticeable lip-sync misalignment, and all generated clips are short.
Questions for authors
  • Can the model generate videos longer than 10 seconds? This matters, as it largely determines the method's practical use. I suggest the authors report results on longer videos, or explain how the method could be extended to them.
5 comments

Response 1. Longer-Video Editing (Q1)

Our current AVE checkpoint was fine-tuned on 5.04-second clips, and we have no results for edits beyond 10 seconds. Ovi 1.1 offers a separate 10-second audio-video generation checkpoint. Adapting it would be a first step toward longer editing. To go beyond that duration, we would explore a Self Forcing-style adaptation that generates aligned audio and video chunks autoregressively, conditioning on previously generated chunks as well as the source clip. Ovi 1.1 · Self Forcing

Backbone choice Ovi provides the paired audio and video latent streams and cross-modal attention used by AVE. Our design can be adapted to backbones that expose these interfaces. We have evaluated the editor only on Ovi, and transfer to LTX-2.3 or MiniMax-H3 remains untested.

Novelty and evaluation. AVE extends synchronized RoPE with role-specific coordinates that keep source and target timelines aligned while distinguishing source, target, and reference tokens. This supports Swap, Insert, and Delete within one joint editor. The contribution of the data pipeline is to construct paired audio-visual supervision for these edits, including reuse of removal pairs for insertion. The JAVEditBench and IDBench-Omni results are in our general response, “Comparisons with DreamID-Omni and JAVEdit”. InstructAV2AV is a text-conditioned system comparison, so its scores do not isolate the effect of reference conditioning. Our reply to Reviewer Z5VX (Response 2) reports the human preference evaluation on AVE-Dataset-Bench.

Thank you for clarifying the current limitations. The backbone choice is understandable, although transfer to other backbones remains unverified. Could you clarify whether the added results include a direct comparison with JAVEdit, rather than only evaluation on JAVEditBench? I would also appreciate more details on the human study, particularly whether it evaluates lip sync. The lip-sync issues observed in the supplementary demos remain a concern. Finally, could you explain more concretely what the role-specific coordinates add over DreamID-Omni and which ablations support this change?

Thank you for the follow-up.

1. JAVEdit comparison. As noted in our official comment, the JAVEdit checkpoint was unavailable to us. We ran AVE on the original JAVEditBench inputs and compared our results with the scores reported in the JAVEdit paper. JAVEdit, Appendix D

2. Human evaluation of lip sync. Yes, the human study explicitly evaluated how well mouth movements matched the speech. We report the results in the AV sync column of Response 2 to Reviewer Z5VX. AV sync Indicates lip sync.

11 raters completed a blind Good/Same/Bad (GSB) evaluation on 32 matched pairs per task from AVE-Dataset-Bench (96 total). We selected the best-performing baselines based on the relevant metrics in Table 1 (using VBench scores to guide the selection for Instruction Compliance). The table reports AVE wins / ties / baseline wins (%).

TaskInstruction ComplianceVideo FidelityAV syncAudio QualityBG Fidelity
Swap53 / 12 / 3529 / 54 / 1737 / 45 / 1843 / 37 / 2037 / 35 / 28
Insert87 / 13 / 052 / 36 / 1245 / 28 / 2741 / 36 / 2341 / 36 / 23
Delete52 / 15 / 3342 / 45 / 1368 / 32 / 040 / 48 / 1246 / 25 / 29

3. Role-specific coordinates. DreamID-Omni’s Syn-RoPE assigns a shared temporal slot to the reference image and voice of each identity, and its architecture adds structural conditions to the target latents. AVE encodes source audio as a separate token sequence. In the corrected Eq. (4), the source, target, and reference audio coordinates are (t,H,W)(t,H,W), (t,0,0)(t,0,0), and (M+t,0,0)(M+t,0,0), respectively. The shared time index aligns source and target audio. The (H,W)(H,W) offset also separates source audio from reference audio along the two auxiliary RoPE axes, helping attention distinguish the original audio context from the reference timbre. DreamID-Omni, Sec. 3.2 · AVE, Eq. (4)

4. RoPE ablation. We compared DreamID-Omni–style Syn-RoPE with our Modality-Aware Sync-RoPE within AVE. Time constraints limited this comparison to Swap.

TaskRoPE variant in AVESubj. Cons. ↑BG Cons. ↑Temp. Flick. ↑Motion Sm. ↑Imaging Q. ↑PQ ↑Sync-C ↑Sync-D ↓
SwapDreamID-Omni–style Syn-RoPE96.8895.3197.2498.5866.496.1504.039.52
SwapModality-Aware Sync-RoPE (ours)97.3595.6997.6798.5469.746.7095.398.53
Reviewer_fjHuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AVE, a unified framework for audio-visual person swapping, insertion, and deletion without requiring spatial masks at inference time. Built on Ovi, the method introduces Modality-Aware Sync-RoPE, bidirectional supervision, and a progressive training strategy. The authors also construct a 100K-pair dataset and evaluate the model on a benchmark of 192 samples. The results show improvements in visual quality, reference fidelity, and audio-visual synchronization over several existing editing pipelines.

Critical strengths
  • Unifying person swapping, insertion, and deletion within a single audio-visual framework is a useful contribution. The ability to jointly edit appearance and voice without requiring spatial masks at inference time is particularly appealing for practical applications.
  • The data construction pipeline is carefully designed, including cross-pair reference generation and multiple filtering stages. The progressive training strategy is also reasonable, and the ablations in Table 2 and Appendix C provide evidence that audio warm-up and Modality-Aware Sync-RoPE improve acoustic quality and audio-visual synchronization.
Critical weaknesses
  • The technical novelty over closely related work is not entirely clear. DreamID-Omni (Guo et al., 2026) already explores synchronized RoPE for multimodal identity conditioning, while InstructAV2AV (Zheng et al., 2026) and SpongeBob (Liang et al., 2026) address joint audio-visual editing. AVE extends these ideas with role-specific positional offsets and a unified formulation for three editing tasks. However, Appendix C only compares the proposed RoPE design against conventional sequential RoPE. A comparison with more closely related positional encoding strategies would help establish what is gained from the proposed design.

  • The baseline evaluation leaves some important comparisons unresolved. DreamID-Omni is discussed as a closely related approach but is not evaluated quantitatively, even on the overlapping person-swapping task. For InstructAV2AV, appearance and voice references are converted into textual descriptions because the model does not support direct reference conditioning. This makes it difficult to separate the benefits of AVE's architecture from those of its conditioning setup. Also, the improvements are not uniform across metrics. For example, VACE+PA achieves better background PSNR and LPIPS than AVE on swapping, while AVE has lower background PSNR than both baselines on insertion (Table 1).

  • The audio evaluation does not fully capture editing quality. AudioBox PQ, CAM++, and SyncNet cover acoustic quality, speaker similarity, and synchronization, but they provide limited information about whether speech content and non-target audio are preserved. Specifically, the paper does not report WER/CER, speaker leakage, or background audio preservation. These are important for evaluating multi-speaker scenes, where editing one person's voice should not affect the others. A subjective evaluation of speech naturalness and prosody would also strengthen the results.

  • Bidirectional supervision is presented as an important component, but its individual contribution is not isolated in the ablations. The training mixture also includes 20K real removal samples from VOR (Appendix B), making it difficult to determine how much of the deletion performance comes from the proposed training strategy versus additional real supervision. The benchmark contains only 64 samples per task, and although the test clips are disjoint from training, it is unclear whether identities are also disjoint. Evaluation on longer clips would be useful as well, since training uses fixed 5.04-second videos.

Questions for authors
  • Could the authors provide a quantitative comparison with DreamID-Omni on overlapping person-swapping cases? An Ovi-based editing baseline without the proposed modifications would also help clarify their individual contributions.
  • Could the audio evaluation include speech intelligibility (WER/CER), speaker leakage, and background audio preservation? Results for multi-speaker scenes and subjective listening tests would be particularly helpful.
  • Could the authors provide separate ablations for backward supervision and the real removal data? It would also be helpful to clarify whether training and test identities are disjoint and how the method performs on longer videos.
4 comments

Response 1. DreamID-Omni Comparison and Ovi Controls (Q1)

The 25 whole-person Swap results and the scope of the published DreamID-Omni scores are in our general response, “Comparisons with DreamID-Omni and JAVEdit”.

AVE builds on synchronized RoPE with role-specific positional offsets for source, target, and reference tokens in a shared Swap, Insert, and Delete formulation. Appendix C compares our design with sequential RoPE. A matched Ovi editing control and a comparison against DreamID-Omni's positional encoding remain open. The InstructAV2AV comparison evaluates the systems with their supported input formats, so it measures the combined effects of conditioning and architecture. We will clarify the background fidelity trade-offs in Table 1.

Response 2. Speech Intelligibility and Background Audio Preservation (Q2)

We evaluated speech content and background audio on AVE-Dataset-Bench. AVE has lower English WER and Chinese CER and higher background audio similarity (SIM-bg) than the corresponding baseline on both Swap and Insert. Our pipeline assumes a single active speaker, even when multiple people are visible. BS-RoFormer separates speech from background audio at the clip level.

TaskMetricBaselineAVEBaseline score
SwapEnglish WER ↓VACE+PA6.77%7.15%
SwapChinese CER ↓VACE+PA10.81%16.15%
InsertEnglish WER ↓InstructAV2AV8.54%9.62%
InsertChinese CER ↓InstructAV2AV11.55%12.69%
SwapSIM-bg ↑VACE+PA0.70220.5959
InsertSIM-bg ↑InstructAV2AV0.72480.6779

Response 3. Training Ablations, Identity Splits, and Clip Duration (Q3)

To assess the contribution of the 20K camera-captured VOR pairs, we will retrain Stages 1 and 2 of the joint editor from the original Stage 0 checkpoint. Each VOR draw will be replaced with a synthetic Delete pair from the corresponding stage's pool, using the original quality-filtered pool in Stage 2. We will match the number of Delete training draws, the task mixture, and the optimization schedule. Both variants will be evaluated on the same 64 Delete clips using the metrics in Table 2.

Task / VariantSubj. Cons. ↑BG Cons. ↑Temp. Flick. ↑Motion Sm. ↑Imaging Q. ↑PQ ↑Sync-C ↑Sync-D ↓
Delete / Ours98.4996.6498.1298.4367.156.810——
Delete / w/o VOR95.3794.1096.7696.1463.856.790——

The Ours row reproduces the Delete results in Table 2. The five VBench metrics are reported in percent.

Synthetic removal pairs supervise Delete in the forward direction and Insert in reverse. A separate Swap ablation would compare forward-only and bidirectional supervision using the same underlying pairs, with VOR and the rest of the training mixture held fixed. The AVE-Dataset-Bench test split and AVE's training data are disjoint at both the clip and identity levels.

AuthorsOpenReview ↗

Ethics and release

Ethics and release. Thank you for raising the risks of impersonation and unauthorized voice cloning. The data curation appendix describes our use of publicly available recordings. Access to those recordings alone does not establish consent to edit an individual's likeness or voice, and the submission does not document consent or permissions for every source. We will clarify this in the revised ethics statement. Before release, we will review the permissions for the benchmark materials and limit redistribution to assets with verified permission. For the model weights, we will assess controlled access and use terms prohibiting impersonation and identity editing without consent. We will label released edited examples as synthetic.

I thank the authors for the detailed rebuttal and additional experiments. Most of my concerns have been addressed especially regarding the baseline comparisons and audio evaluation. My main remaining concern is the lack of ablations that clearly isolate the contributions of the proposed RoPE design and bidirectional supervision. But the rebuttal has improved my assessment of the paper, so I will raise my score.

We are glad that most of your concerns have been addressed. Thank you again for your careful reading, constructive feedback, and for raising your score!

Regarding your remaining concern:

  • RoPE design: Our revised manuscript includes controlled RoPE ablations in Appendix C.1–C.2. On Swap (Appendix C.2), Table 5 compares source- vs. target-aligned references, where target alignment improves all three reference similarity metrics, and Table 6 compares our RoPE design with a DreamID-Omni-style layout within AVE, raising Sync-C from 4.03 to 5.39 and lowering Sync-D from 9.52 to 8.53. Appendix C.1 further compares our design against sequential RoPE across all three tasks.
  • Bidirectional supervision: Section 4.1 now specifies the training directions for each task, and Appendix C.3 isolates the contribution of camera-captured VOR pairs on Delete. We agree that a controlled comparison between forward-only and bidirectional training on the exact same Swap pairs (holding all other settings fixed) will make this analysis even cleaner, and we will add this ablation in the final version.
Reviewer_pJwYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AVE, an end-to-end framework for human-centric joint audio-visual editing that supports three tasks (Delete, Insert, and Swap) within a unified architecture. The method introduces Modality-Aware Sync-RoPE, which assigns role-specific positional coordinates to source, target, and reference tokens to preserve temporal correspondence while distinguishing their roles. The paper also constructs AVE-Dataset, a 100K-pair audio-visual editing dataset covering the three tasks with appearance and voice references. Experiments show that AVE generally outperforms modality-separate audio-visual editing pipelines in editing realism, temporal consistency, and audio-visual alignment.

Critical strengths
  1. The paper addresses human-centric audio-visual editing, which is a timely and important problem given recent progress in joint audio-video generation. The motivation is clear, and the proposed unified formulation could be useful for the community. The constructed dataset is also a valuable contribution.
  2. The model design and progressive training strategy are well motivated. The experimental results and supplementary examples suggest that the proposed approach is effective, especially in terms of audio-visual synchronization and reference-conditioned editing quality.
Critical weaknesses
  1. Some important design choices are not fully justified, especially the coordinate assignment for reference tokens in Modality-Aware Sync-RoPE and the exact use of bidirectional supervision across the three tasks. These details are central to the proposed method, so clarifying them would improve the technical soundness of the paper.
  2. Even though the generated videos are good, Table 1 shows that AVE does not consistently outperform baselines across the adopted metrics. More detailed discussion would be desired.
Questions for authors

I would appreciate it if the authors could answer the following questions. The main reason for my current rating of 2 is that the following points are unclear. If these points are clarified satisfactorily, I would consider revising my rating upward.

  1. Questions about the design of Modality-Aware Sync-RoPE. The use of TT, HH, and WW for the target and source video/audio tokens seems reasonable to me. However, I have questions about the coordinate assignment for the reference image/audio tokens.

    • [1-a] Why do the image reference tokens use the same spatial coordinates as the target video tokens, i.e., [0,H)×[0,W)[0, H) \times [0, W), whereas the audio reference tokens use the same auxiliary coordinates as the source audio tokens, i.e., H×W{H} \times {W}?
    • [1-b] More generally, the reference tokens could be aligned with either the target tokens or the source tokens along the spatial/auxiliary coordinate axes. Could the authors explain why they chose the current alignment for each modality? If this design choice is based on intuition or empirical findings, it would be helpful to clarify that.
  2. Regarding Section 3.3.1, I understand the motivation for using real videos as targets, since this can help the model generate high-quality video and audio with fewer synthetic artifacts. However, I am a little confused about how bidirectional supervision is applied across the three tasks. For Swap, the forward/backward setup seems clear. In contrast, for Delete/Insert, the original real clip and the synthetic person-removed clip form an inverse pair. Real-to-removed corresponds to Delete, while removed-to-real seems to correspond to Insert. Could the authors clarify whether Section 3.3.1 discusses all three tasks or mainly person-swapping pairs? More specifically, how are person-swapped pairs and person-removed pairs used in the forward and backward directions?

  3. Table 1 shows that AVE is not uniformly better across all metrics. For Swap and Insert, AVE improves audio quality and AV synchronization, but is not always the best on some temporal/background fidelity metrics such as Temp. Flick., PSNR, or LPIPS. For Delete, AVE achieves strong VBench and background fidelity scores, while its PQ score is roughly tied with or slightly below some post-processed baselines. Could the authors discuss the possible trade-offs behind these trends?

Minor comment

This is a minor comment that I did not consider in deciding my rating. So the authors do not need to respond to the following items.

  • L.200: "It should faithfully preserves" -> "It should faithfully preserve" ("s" is unnecessary)
  • L.273: "the target, source, and reference tokens are concatenated". It may be helpful to clarify that they are concatenated along the token/sequence dimension, rather than the channel dimension, because some models concatenate different modalities in the channel dimension. I know Figure 4 illustrates that the three types of tokens are concatenated along the sequence dimension and that Modality-Aware Sync-RoPE is designed based on such a concatenation. However, I suppose this clarification would help readers read the manuscript more smoothly.
9 comments

Response 1a. Reference Image and Audio Coordinates (Q1a)

Eq. (4) in the submission contains a typo. The implementation assigns the reference audio tokens auxiliary coordinates (0,0)(0,0), as it does for target audio. The correct reference audio coordinates are (t,0,0)(t,0,0) for tt in [M,M+Ta)[M,M+T_a). The reference image assignment in Eq. (3) is correct.

Response 1b. Rationale for Reference Alignment (Q1b)

The temporal margin MM separates the reference tokens from the aligned source and target timelines. This lets both references follow the target coordinate convention on the other two axes. The image uses the unshifted spatial grid [0,H)×[0,W)[0,H) × [0,W), and audio uses zero auxiliary offsets because it has no spatial grid. Appendix C evaluates the full Sync-RoPE design.

Response 2. Training Directions for Swap, Insert, and Delete (Q2)

For Swap, we use both directions, from the real clip to the swapped clip and from the swapped clip back to the real clip. For the synthetic removal pairs, the real-to-removed direction supervises Delete, and the removed-to-real direction supervises Insert. Delete also uses the 20K real-captured VOR pairs described in Appendix B. We use real targets for Insert because the available synthetic insertion outputs were insufficient in quality for supervision. We will clarify these task-specific directions in Sec. 3.3.1.

Response 3. Audio-Visual Quality and Background Trade-offs (Q3)

For Swap and Insert, AVE leads on AudioBox PQ and both SyncNet metrics. Its background PSNR is lower than the best baseline's (34.62 vs. 35.37 dB for Swap and 27.70 vs. 29.38 dB for Insert), and its temporal flickering score trails by 0.30 and 0.05 points, respectively. Joint regeneration can improve speech-motion coherence while introducing changes outside the edited region. For Delete, AVE leads on all five VBench metrics and both background fidelity metrics. Its PQ is 6.810 vs. 6.813 for the cascaded baselines, which share a dedicated audio post-processing pipeline. We will revise the discussion to acknowledge these trade-offs explicitly.

I thank the authors for their responses. I appreciate the clarification of the Eq. (4) typo (Response 1a) and the clear explanation of the task-specific training directions for Swap, Insert, and Delete (Response 2). These address my earlier questions well.

However, two of my questions remain only partially addressed, and I would appreciate further clarification.

Regarding Response 1b (reference token alignment)

My original question was specifically about the choice between aligning the reference tokens with the source tokens or with the target tokens along the spatial/auxiliary axes. The current response explains that both references "follow the target coordinate convention", but this restates the design decision rather than justifying it. Could the authors explain why the target convention was preferred over the source convention for each modality? If this choice stems from intuition or empirical observation (e.g., an ablation comparing source-aligned vs. target-aligned references), it would strengthen the paper considerably to state this explicitly.

Regarding Response 3 (Table 1 trade-offs)

My question asked for a discussion of the mechanism behind the trade-off patterns, rather than a restatement of the numerical gaps. Specifically, I am still unclear on why the pattern is asymmetric across tasks: Swap and Insert improve audio quality and AV synchronization but trail on temporal/background fidelity (Temp. Flick., PSNR, LPIPS), whereas Delete leads on VBench and background fidelity, while its PQ is only comparable to the cascaded baselines. I would find it helpful if the authors could explain this in terms of the underlying mechanism. For example, whether the key factor is the presence or absence of reference-based generation and the resulting extent of regeneration outside the edited region. A mechanistic account along these lines would be more informative than the current task-by-task description.

General observation

I have also read the other reviewers' comments and the authors' responses. I found the other reviews insightful, and I encourage the authors to continue engaging with the points raised there, as doing so would further strengthen the submission.

Thank you for the follow-up. We address both points below.

Response 1b. Why align the reference tokens with the target?

In Swap, the source tells the model what to preserve: the scene, motion, and speech timing. The references tell it whose appearance and voice to generate. We chose the coordinates to reflect that distinction in attention.

For the reference image, using the target grid removes the extra spatial offset between matching grid positions. The temporal margin keeps the image separate from the video frames, so it can guide appearance while the source guides the subject’s position and motion.

For reference audio, the auxiliary axes distinguish token roles. The reference and target share (0,0), while the source uses (H,W). Under source alignment, the original voice and the desired voice would share the same auxiliary coordinates. Keeping them separate while aligning the source and target in time gives the model a positional cue to preserve when the person speaks and use the reference for how they sound.

Given time and resource constraints, we compared the two layouts on Swap. Target alignment improved all three reference similarity metrics.

TaskReference token alignmentSubj. Cons. ↑BG Cons. ↑Temp. Flick. ↑Motion Sm. ↑Imaging Q. ↑PQ ↑Sync-C ↑Sync-D ↓CAM++ ↑Arc ↑CLIP ↑PSNR ↑LPIPS ↓
SwapSource-aligned references96.3495.5897.5196.0268.306.5285.028.970.6020.6110.67332.090.0996
SwapTarget-aligned references (ours)97.3595.6997.6798.5469.746.7095.398.530.6820.6470.71234.620.0549

Response 3. What explains the trade-offs in Table 1?

Swap and Insert generate a reference-conditioned subject and speech together, so the model can coordinate lip motion with speech directly. Without a fixed edit mask, it can adapt the subject’s silhouette to the reference. This flexibility can also change nearby pixels across frames, which can affect temporal flicker and background fidelity. Synthetic edits used to construct the training pairs may contribute to these changes. The background scores can also penalize valid edits: if SAM 2 misses part of a wider replacement silhouette, those pixels count as background in PSNR and LPIPS.

Delete removes the subject and fills the exposed region. The camera-captured VOR pairs provide direct supervision for that operation, and our Delete ablation shows improvements. On the audio side, the cascaded baselines can use BS-RoFormer to remove the single active speaker’s voice and retain the separated background track. This helps explain their slightly higher PQ.

Reviewer_Z5VXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AVE, a unified end-to-end framework for human-centric audio-visual editing that supports person swapping, insertion, and deletion without spatial masks. The authors introduce Modality-Aware Sync-RoPE and a multi-stage training strategy to improve audio-visual consistency. They also construct the AVE-Dataset with 100K editing pairs. Experiments show competitive results against existing methods across multiple editing tasks.

Critical strengths
  1. Person-level editing with appearance and voice references, covering swap, insertion and deletion, is directly useful for film post-production and dubbing. Figure 2 clearly contrasts separate pipelines with joint editing.
  2. The paper tackles a meaningful problem by jointly editing audio and video within a single framework, rather than relying on separate pipelines.
  3. The proposed 100K-pair dataset is a useful contribution. The experiments cover three editing tasks, and the ablation studies support the effectiveness of the main components.
Critical weaknesses
  1. The paper identifies DreamID-Omni as the closest related work but does not compare against it for person swapping. JAVEdit is also discussed but excluded from the experiments. Moreover, all evaluations are conducted on the authors' own benchmark, making generalization unclear.
  2. Each task contains only 64 test cases, with no user study, standard deviations, or confidence intervals. Several improvements are quite small (e.g., Sync-C: 5.39 vs. 5.05 for Swap,, PSNR: 31.74 vs. 31.27 for Delete), making it difficult to judge their significance.
  3. The audio pipeline uses BS-RoFormer to separate speech, but it is not designed for target-speaker separation. In multi-speaker videos, this could accidentally remove or replace other speakers' voices. The paper does not clearly explain how speaker attribution is handled during data construction.
  4. The abstract and conclusion suggest consistent improvements, but Table 1 shows mixed results. AVE performs worse on background preservation for Swap and Insert, while some temporal consistency metrics are also below existing baselines. These limitations should be acknowledged.
Questions for authors
  1. Why are DreamID-Omni and JAVEdit not included as baselines? Can the authors provide comparisons on independent benchmarks?
  2. Can the authors provide human evaluations or statistical significance tests to verify the relatively small performance gains?
  3. How does the data construction pipeline isolate the target speaker's voice when multiple people are speaking? Does it use any speaker diarization or target-speaker extraction?
  4. How do the authors justify the claims of consistent superiority when several metrics in Table 1 are worse than those of existing methods?
3 comments

Response 1. Comparisons with DreamID-Omni and JAVEdit (Q1)

DreamID-Omni's public checkpoint supports R2AV only. We lacked access to its RV2AV checkpoint and the JAVEdit checkpoint. The two external benchmark results are in our general response, “Comparisons with DreamID-Omni and JAVEdit”.

Response 2. Human GSB Results and Paired Bootstrap (Q2)

11 raters completed a blind Good/Same/Bad (GSB) evaluation on 32 matched pairs per task from AVE-Dataset-Bench (96 total). We selected the best-performing baselines based on the relevant metrics in Table 1 (using VBench scores to guide the selection for Instruction Compliance). The table reports AVE wins / ties / baseline wins (%).

TaskInstruction ComplianceVideo FidelityAV syncAudio QualityBG Fidelity
Swap53 / 12 / 3529 / 54 / 1737 / 45 / 1843 / 37 / 2037 / 35 / 28
Insert87 / 13 / 052 / 36 / 1245 / 28 / 2741 / 36 / 2341 / 36 / 23
Delete52 / 15 / 3342 / 45 / 1368 / 32 / 040 / 48 / 1246 / 25 / 29

Paired bootstrap. We used the original Table 1 test samples and evaluation protocol to estimate 95% confidence intervals for the two comparisons highlighted in the review. Both intervals exclude zero.

TaskMetricBaselineMean difference (AVE − baseline)95% bootstrap CI
SwapSync-CMoCha+PA+0.34[+0.12, +0.56]
DeleteBackground PSNREffectErase+PA+0.47 dB[+0.02, +0.91] dB

Response 3. Speech Separation and the Single-Speaker Setting (Q3)

We use BS-RoFormer to separate speech from background audio, including sound effects. It treats speech as a single source. To simplify the task, each clip has one active speaker, regardless of how many people are visible.

Response 4. Performance Trade-offs in Table 1 (Q4)

AVE remains close to the best baselines where it trails in Table 1. Background PSNR gaps are 0.75 dB for Swap and 1.68 dB for Insert, and temporal flickering gaps are 0.30 and 0.05 percentage points, respectively. The blind GSB evaluation in Response 2 favored AVE overall against the selected baselines. We will revise the abstract and conclusion to qualify our claims and report these trade-offs by task and metric.

Dear reviewer: We would appreciate it if you could take a look at our responses and let us know if you have any remaining questions or concerns.

Thank you for your time and consideration.

Dear reviewer,

We have prepared a revised manuscript and detailed responses addressing comments from all reviewers. We would be grateful if you could review these materials and inform us of any outstanding questions or concerns you may have.

Thank you for your time and consideration.

Comments on the paper

3 comments

We thank all reviewers for their thoughtful feedback. We address two concerns raised across the reviews.

1. Comparisons with DreamID-Omni and JAVEdit. Z5VX asked why these baselines were omitted, fjHu requested a comparison on overlapping person-swapping cases, and ycmz asked for comparisons with the closest prior work. The DreamID-Omni RV2AV and JAVEdit checkpoints were unavailable, so we report our results alongside their published scores.

IDBench-Omni · Whole-Person Swap. We evaluated AVE on 25 whole-person cases from the RV2AV split. DreamID-Omni, Table 3

MethodAES ↑ViCLIP ↑ID-Sim. ↑WER ↓T-Sim. ↑Sync-C ↑
DreamID-Omni0.58414.8320.6350.0650.5136.241
AVE0.62815.6370.6910.0510.5796.396

JAVEditBench · Subject Removal. AVE scores higher on five of the six metrics and slightly lower on VTSS. JAVEdit, Appendix D

MethodInstruction Compliance ↑Video Fidelity ↑AV Quality ↑UTMOSv2 ↑VTSS ↑SyncNet ↑
JAVEdit (reported)3.964.303.822.240.04470.0146
AVE4.084.444.122.560.04390.1557

2. Table 1 trade-offs and human preference (Z5VX, pJwY). We agree that our claim of consistent superiority was too broad. In Table 1, AVE leads on AudioBox PQ and both SyncNet measures for Swap and Insert. Some temporal VBench scores cluster near the ceiling, around 98%. AVE trails the best baseline by 0.30 and 0.05 percentage points on temporal flickering for Swap and Insert, respectively, and by 0.19 percentage points on Insert motion smoothness. For Delete, AVE leads on VBench and background fidelity. Its AudioBox PQ is just 0.003 below the cascaded baselines (6.810 vs. 6.813), which use a dedicated audio separation model (BS-Roformer) for post-processing.

Following Z5VX's suggestion, we ran a blind human evaluation on AVE-Dataset-Bench. The ratings showed a clear overall preference for AVE over the selected baselines. Moreover, the paired bootstrap 95% confidence intervals for the modest gains in Swap Sync-C and Delete background PSNR both lie above zero (see Response 2 to Z5VX). We will revise the abstract, conclusion, and discussion to state the gains and trade-offs by task and metric.

Area_Chair_tqpfOpenReview ↗

Please read the response from authors and discuss

Dear reviewer fellows,

The author rebuttal has been posted. The rebuttal period is a critical part of our review process, allowing authors to clarify misunderstandings and address specific concerns.

Please take a moment to read the author response, engage in the discussion by replying to the authors' responses, and update your review if the rebuttal changes your assessment.

Your active participation is vital to ensuring a fair and thorough evaluation of the submissions. A constructive discussion helps us make the most informed decisions.

Thank you for your time and valuable contributions to the review process.

Best regards,

Your AC

AuthorsOpenReview ↗

Summary of Discussion and Revisions

Dear Area Chair and Reviewers, Thank you for the constructive discussion. We summarize the value of AVE, the progress made during rebuttal, and the additions to the revised manuscript.

Paper overview and strengths. AVE is the first mask-free joint audio–visual editor to unify Swap, Insert, and Delete, with external appearance and voice control for Swap and Insert. To make this setting tractable, we co-design a tailored, fully automated data-construction pipeline that yields a 100K-pair dataset and a benchmark, together with a unified editing framework and a multi-stage training curriculum. The experiments demonstrate substantial improvements in editing quality, audio–visual alignment, and reference fidelity. Ablations establish the benefits of role-specific coordinates, audio warm-up, quality refinement, and real removal supervision.

Reviewers' assessment. Reviewers found the problem setting meaningful and practically relevant (pJwY) and the unified mask-free formulation a useful contribution (fjHu, ycmz), while acknowledging the value of the jointly released dataset, benchmark, and training recipe (Z5VX).

Two reviewers have raised their ratings to 3 (Weak Acceptance). fjHu stated that most concerns had been addressed, particularly the baseline comparisons and audio evaluation. After checking the revised manuscript, pJwY confirmed that the added explanations and ablations addressed their remaining concerns.

ConcernRevisions and supporting evidenceStatus
Method rationale, training directions, and task trade-offs (pJwY)Explained the reference-coordinate design, added a controlled source-versus-target alignment ablation, specified the training directions for each task, and explained the performance trade-offs in Table 1. Sections 3.2.2, 4.1–4.2; Appendix C.2.✔ Resolved — pJwY raised the rating to 3.
Baseline comparisons and audio evaluation (fjHu)Added public-benchmark comparisons with published DreamID-Omni and JAVEdit results, together with WER, CER, and background-audio similarity measurements supporting speech and background preservation. Appendix D.1–D.2.✔ Resolved — fjHu acknowledged these improvements and raised the rating to 3.
Benchmark protocol, evaluation scope, and data release (Z5VX, fjHu, ycmz)Confirmed that training and test data are disjoint at both the clip and identity levels, clarified the single-active-speaker setting, and specified permission checks and safeguards for release. Section 4.1; Conclusion and Future Work; Ethics Statement.✔ Resolved — fjHu confirmed the clarified protocol and scope and raised the rating to 3; the same revision addresses the related points from Z5VX and ycmz.
Contribution beyond prior designs (fjHu, ycmz)Clarified the task and input differences from the closest methods. Added a DreamID-Omni–style RoPE control within AVE, improving Swap Sync-C from 4.03 to 5.39, and a matched VOR ablation showing a Delete Imaging Quality gain from 63.85 to 67.15. Related Work; Appendix C.2–C.3.✔ Resolved — fjHu acknowledged the clarified contribution and raised the rating to 3; the same revision covers ycmz's overlapping concern.
Human preference evaluation (Z5VX, ycmz)Added an 11-rater blind study covering five criteria, including lip sync. AVE receives more wins than the selected baselines overall on all three tasks. Figure 6; Appendix D.3.Addressed — added human evidence and the evaluation protocol to the revised paper.
Statistical reliability of the reported gains (Z5VX)Paired bootstrap analysis confirms statistically significant gains in Swap Sync-C and Delete background PSNR, supporting the improvements reported in Table 1. Appendix D.4.Addressed — added statistical validation to the revised paper.

The revised manuscript is available on OpenReview, with changes marked in blue. The discussion has strengthened both the experimental evidence and the explanation of the method. We believe AVE's unified task setting, dataset, and model and training contributions make it a substantial contribution to audio–visual editing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.