- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes to evaluate the joint audio-video editing capabilities of current methods, where real-world editing requires coordinated modifications across visual and auditory modalities while preserving unmodified content. The authors present two core designs: AVE-Compass, a comprehensive diagnostic benchmark for free-form audio-video editing, and AVE-Agent, a planning-based editing framework designed to improve cross-modal consistency and content fidelity.
Critical strengths
-
Unlike prior work that evaluates modalities in isolation or focuses only on object-level audio-visual correspondence, this paper assesses global scene-level cross-modal dependencies, speech editing, and non-target fidelity across diverse editing scenarios. The three-axis difficulty stratification enables fine-grained diagnosis of model limitations rather than just aggregate ranking.
-
The decoupled four-dimensional evaluation framework provides actionable insights into distinct model failure modes. The tripartite-call judging mechanism minimizes cross-modal bias, and the absolute affirmative paradigm ensures strict, consistent scoring criteria.
-
The paper provides extensive analysis beyond main results, including difficulty-stratified performance, single-modality editing performance, error category counts, and qualitative case studies. These analyses clearly identify distinct failure patterns across models.
-
AVE-Agent introduces a principled planning-and-reflection paradigm for multi-modal editing, moving beyond black-box end-to-end models. Explicit modeling of cross-modal dependencies via subtask DAGs makes editing decisions interpretable.
Critical weaknesses
-
With 145 videos and 196 instructions, AVE-Compass is modest in scale compared to mature video-only benchmarks. Short clip lengths mean the benchmark does not evaluate long-form editing, multi-scene narrative consistency, or sustained cross-modal alignment over extended sequences. The limited sample size also reduces statistical power for rare editing types.
-
Both the MLLM evaluator and AVE-Agent’s backend tools are closed-source APIs. This creates reproducibility risks as API versions and pricing change, and the high evaluation cost creates barriers for independent researchers. No open-source reference implementation of the evaluation pipeline is provided.
-
Spearman correlations between automated metrics and MLLM-as-Judge scores are consistently very low (near zero for most pairs). While the authors position this as complementarity, it also indicates a lack of consensus on what constitutes good audio-video editing quality, and raises questions about how to weight or combine these very different signals in practice.
-
AVE-Agent roughly doubles latency (12.0 min vs. 7.7 min per clip) and increases cost by ~13% compared to direct editing with the same Wan 2.7 backend. While quality improves, this overhead may be impractical for production use. Additionally, the agent is only evaluated on in-distribution AVE-Compass tasks; its generalization to out-of-domain editing instructions or video domains is not demonstrated.
-
The paper acknowledges ethical risks of deceptive media and non-consensual voice manipulation, but proposed mitigations (watermarking, content credentials) are only mentioned in passing without technical evaluation. The benchmark also does not include any safety or misuse-oriented evaluation dimensions.
Questions for authors
Refer to Weaknesses part.