- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AVE, a unified model for human-centric audio-visual editing. It handles person swap, insertion and deletion, edits video and audio together, and needs no spatial masks. The method adds Modality-Aware Sync-RoPE, bidirectional supervision and a three-stage curriculum. On a self-built benchmark of 192 cases, AVE outperforms cascaded pipelines and InstructAV2AV in visual quality, audio quality and lip sync.
Critical strengths
- The task is useful. Joint audio-visual editing of people with appearance and voice references is closer to real needs than text-only editing.
- The dataset is a solid contribution. The cross-pair reference construction is careful and helps avoid copy-and-paste shortcuts. If released, it can benefit the community.
- The backward supervision is reasonable. It lets the model learn from real targets, especially for insertion.
Critical weaknesses
- The novelty is incremental. The data pipeline combines existing tools. Sync-RoPE extends the design of DreamID-Omni. Reversing removal data for insertion is a common practice.
- Comparison with the closest work is missing. The text-only InstructAV2AV is a weak baseline for a reference-based task. Other end-to-end methods such as JAVEdit are also not compared.
- Public benchmarks such as IDBench-Omni are mentioned but not used. VBench scores are near saturation, and there is no user study.
- The paper states that the method is not tied to a particular backbone, but it is only tested on Ovi. Ovi is a relatively early model. Newer open-source models, such as LTX-2.3 or MiniMax-H3, may give clearly better results. I would like the authors to explain this choice, and ideally show that the framework transfers to a stronger base model.
- Some demos in the supplementary material show noticeable lip-sync misalignment, and all generated clips are short.
Questions for authors
- Can the model generate videos longer than 10 seconds? This matters, as it largely determines the method's practical use. I suggest the authors report results on longer videos, or explain how the method could be extended to them.
5 comments
Thank you for clarifying the current limitations. The backbone choice is understandable, although transfer to other backbones remains unverified. Could you clarify whether the added results include a direct comparison with JAVEdit, rather than only evaluation on JAVEditBench? I would also appreciate more details on the human study, particularly whether it evaluates lip sync. The lip-sync issues observed in the supplementary demos remain a concern. Finally, could you explain more concretely what the role-specific coordinates add over DreamID-Omni and which ablations support this change?
Thank you for the follow-up.
1. JAVEdit comparison. As noted in our official comment, the JAVEdit checkpoint was unavailable to us. We ran AVE on the original JAVEditBench inputs and compared our results with the scores reported in the JAVEdit paper. JAVEdit, Appendix D
2. Human evaluation of lip sync. Yes, the human study explicitly evaluated how well mouth movements matched the speech. We report the results in the AV sync column of Response 2 to Reviewer Z5VX. AV sync Indicates lip sync.
11 raters completed a blind Good/Same/Bad (GSB) evaluation on 32 matched pairs per task from AVE-Dataset-Bench (96 total). We selected the best-performing baselines based on the relevant metrics in Table 1 (using VBench scores to guide the selection for Instruction Compliance). The table reports AVE wins / ties / baseline wins (%).
| Task | Instruction Compliance | Video Fidelity | AV sync | Audio Quality | BG Fidelity |
|---|---|---|---|---|---|
| Swap | 53 / 12 / 35 | 29 / 54 / 17 | 37 / 45 / 18 | 43 / 37 / 20 | 37 / 35 / 28 |
| Insert | 87 / 13 / 0 | 52 / 36 / 12 | 45 / 28 / 27 | 41 / 36 / 23 | 41 / 36 / 23 |
| Delete | 52 / 15 / 33 | 42 / 45 / 13 | 68 / 32 / 0 | 40 / 48 / 12 | 46 / 25 / 29 |
3. Role-specific coordinates. DreamID-Omni’s Syn-RoPE assigns a shared temporal slot to the reference image and voice of each identity, and its architecture adds structural conditions to the target latents. AVE encodes source audio as a separate token sequence. In the corrected Eq. (4), the source, target, and reference audio coordinates are , , and , respectively. The shared time index aligns source and target audio. The offset also separates source audio from reference audio along the two auxiliary RoPE axes, helping attention distinguish the original audio context from the reference timbre. DreamID-Omni, Sec. 3.2 · AVE, Eq. (4)
4. RoPE ablation. We compared DreamID-Omni–style Syn-RoPE with our Modality-Aware Sync-RoPE within AVE. Time constraints limited this comparison to Swap.
| Task | RoPE variant in AVE | Subj. Cons. ↑ | BG Cons. ↑ | Temp. Flick. ↑ | Motion Sm. ↑ | Imaging Q. ↑ | PQ ↑ | Sync-C ↑ | Sync-D ↓ |
|---|---|---|---|---|---|---|---|---|---|
| Swap | DreamID-Omni–style Syn-RoPE | 96.88 | 95.31 | 97.24 | 98.58 | 66.49 | 6.150 | 4.03 | 9.52 |
| Swap | Modality-Aware Sync-RoPE (ours) | 97.35 | 95.69 | 97.67 | 98.54 | 69.74 | 6.709 | 5.39 | 8.53 |
Response 1. Longer-Video Editing (Q1)
Our current AVE checkpoint was fine-tuned on 5.04-second clips, and we have no results for edits beyond 10 seconds. Ovi 1.1 offers a separate 10-second audio-video generation checkpoint. Adapting it would be a first step toward longer editing. To go beyond that duration, we would explore a Self Forcing-style adaptation that generates aligned audio and video chunks autoregressively, conditioning on previously generated chunks as well as the source clip. Ovi 1.1 · Self Forcing
Backbone choice Ovi provides the paired audio and video latent streams and cross-modal attention used by AVE. Our design can be adapted to backbones that expose these interfaces. We have evaluated the editor only on Ovi, and transfer to LTX-2.3 or MiniMax-H3 remains untested.
Novelty and evaluation. AVE extends synchronized RoPE with role-specific coordinates that keep source and target timelines aligned while distinguishing source, target, and reference tokens. This supports Swap, Insert, and Delete within one joint editor. The contribution of the data pipeline is to construct paired audio-visual supervision for these edits, including reuse of removal pairs for insertion. The JAVEditBench and IDBench-Omni results are in our general response, “Comparisons with DreamID-Omni and JAVEdit”. InstructAV2AV is a text-conditioned system comparison, so its scores do not isolate the effect of reference conditioning. Our reply to Reviewer Z5VX (Response 2) reports the human preference evaluation on AVE-Dataset-Bench.