acceptodds
Withdrawn from ICLR 2027

AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While instruction-based video editing has advanced rapidly, real-world videos contain tightly coupled audio and visual signals, and editing one modality often requires coordinated changes in the other. Existing benchmarks primarily evaluate visual transformations on silent clips or isolated audio editing, leaving complex audio-visual editing and cross-modal consistency underexplored. We introduce **AVE-Compass**, a comprehensive benchmark with 145 curated source videos, 196 human-verified editing instructions, and 2,688 fine-grained checklist items. It evaluates Instruction Following, Fidelity Preservation, Realism, and Editing Intent through checklist-based MLLM judging and a dedicated realism rubric, complemented by automated cross-modal, video, and audio metrics. Evaluation of six systems shows persistent failures in cross-modal execution and non-target preservation. We further propose **AVE-Agent**, which decomposes instructions into dependent subtasks and improves edits through an evaluator-guided loop. AVE-Agent improves instruction execution, Fidelity Preservation, and audio-visual alignment in joint editing while maintaining competitive perceptual quality.

Primary area:
datasets and benchmarks
Keywords:
video editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vARKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to evaluate the joint audio-video editing capabilities of current methods, where real-world editing requires coordinated modifications across visual and auditory modalities while preserving unmodified content. The authors present two core designs: AVE-Compass, a comprehensive diagnostic benchmark for free-form audio-video editing, and AVE-Agent, a planning-based editing framework designed to improve cross-modal consistency and content fidelity.

Critical strengths
  1. Unlike prior work that evaluates modalities in isolation or focuses only on object-level audio-visual correspondence, this paper assesses global scene-level cross-modal dependencies, speech editing, and non-target fidelity across diverse editing scenarios. The three-axis difficulty stratification enables fine-grained diagnosis of model limitations rather than just aggregate ranking.

  2. The decoupled four-dimensional evaluation framework provides actionable insights into distinct model failure modes. The tripartite-call judging mechanism minimizes cross-modal bias, and the absolute affirmative paradigm ensures strict, consistent scoring criteria.

  3. The paper provides extensive analysis beyond main results, including difficulty-stratified performance, single-modality editing performance, error category counts, and qualitative case studies. These analyses clearly identify distinct failure patterns across models.

  4. AVE-Agent introduces a principled planning-and-reflection paradigm for multi-modal editing, moving beyond black-box end-to-end models. Explicit modeling of cross-modal dependencies via subtask DAGs makes editing decisions interpretable.

Critical weaknesses
  1. With 145 videos and 196 instructions, AVE-Compass is modest in scale compared to mature video-only benchmarks. Short clip lengths mean the benchmark does not evaluate long-form editing, multi-scene narrative consistency, or sustained cross-modal alignment over extended sequences. The limited sample size also reduces statistical power for rare editing types.

  2. Both the MLLM evaluator and AVE-Agent’s backend tools are closed-source APIs. This creates reproducibility risks as API versions and pricing change, and the high evaluation cost creates barriers for independent researchers. No open-source reference implementation of the evaluation pipeline is provided.

  3. Spearman correlations between automated metrics and MLLM-as-Judge scores are consistently very low (near zero for most pairs). While the authors position this as complementarity, it also indicates a lack of consensus on what constitutes good audio-video editing quality, and raises questions about how to weight or combine these very different signals in practice.

  4. AVE-Agent roughly doubles latency (12.0 min vs. 7.7 min per clip) and increases cost by ~13% compared to direct editing with the same Wan 2.7 backend. While quality improves, this overhead may be impractical for production use. Additionally, the agent is only evaluated on in-distribution AVE-Compass tasks; its generalization to out-of-domain editing instructions or video domains is not demonstrated.

  5. The paper acknowledges ethical risks of deceptive media and non-consensual voice manipulation, but proposed mitigations (watermarking, content credentials) are only mentioned in passing without technical evaluation. The benchmark also does not include any safety or misuse-oriented evaluation dimensions.

Questions for authors

Refer to Weaknesses part.

Reviewer_N24FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AVE-Compass, a benchmark for instruction-guided audio-video editing, comprising 145 source videos, 196 editing instructions, and 2,688 fine-grained checklist items across 28 editing operation types. It evaluates Instruction Following (IF), Fidelity Preservation (FP), Realism (REAL), and Editing Intent (EI), complemented by automated metrics for audio-visual synchronization, audio quality, and video quality. In addition to the benchmark, the paper proposes AVE-Agent, a multi-tool editing framework that uses DAG-based planning to model cross-modal dependencies and an evaluator-driven reflection loop to iteratively refine editing results. Experiments show that AVE-Agent improves Editing Intent from 42.4 to 59.8 over Wan2.7, with particularly notable gains in audio Fidelity Preservation.

Critical strengths
  1. Existing editing benchmarks primarily focus on visual modifications, with relatively limited attention to audio preservation, audio-visual coordination, and non-target content fidelity. AVE-Compass integrates these aspects into a unified evaluation framework, addressing an important practical need.

  2. The experiments cover multiple editing models and provide analyses across difficulty levels, single-modality editing tasks, and error categories, along with ablation studies of AVE-Agent. The appendix also includes detailed checklist prompts, evaluation formulas, and tool configurations, contributing to the overall completeness and reproducibility of the work.

Critical weaknesses
  1. The paper emphasizes holistic audio-video editing and cross-modal dependency; however, its evaluation primarily focuses on modality-specific editing correctness, fidelity preservation, temporal synchronization, and perceptual plausibility, without systematically assessing whether audio and video changes are consistent with each other. For example, when a drum strike is edited to appear more forceful, the corresponding sound intensity should also increase appropriately. While the existing checklists may partially capture such relationships, the paper lacks an explicit evaluation of edit-conditioned cross-modal change consistency and targeted experiments validating its effectiveness. More fine-grained assessments of cross-modal editing dependencies would better demonstrate the unique value of AVE-Compass for joint audio-video editing.

  2. Although AVE-Compass covers 28 editing operation types, the benchmark contains only 145 source videos and 196 editing instructions, averaging approximately seven instructions per operation type. Considering that these instructions also include video-only, audio-only, and speech editing tasks, the actual number of samples dedicated to evaluating complex joint audio-video editing may be even more limited.

  3. The paper employs Gemini 3.1 Pro as both the checklist evaluator and Realism evaluator in AVE-Compass, while the internal separation, generation, post-branch, and mixed evaluators of AVE-Agent also rely on Gemini 3.1 Pro. This means that the agent's iterative refinement process is guided by evaluation preferences that are closely aligned with those of the final benchmark evaluator. Evaluator-model alignment bias remains a concern, as AVE-Agent shows larger gains on MLLM-as-Judge metrics than on automated metrics.

Questions for authors

See weaknesses above

Reviewer_8CeqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AVE-Compass, a benchmark for instruction-based audio-video editing, consisting of 145 source videos, 196 editing instructions, and 2,688 fine-grained evaluation checklists. The benchmark evaluates instruction following, fidelity preservation, realism, and editing intent, with particular emphasis on cross-modal consistency and synchronization.

The paper further proposes AVE-Agent, a planning-based editing framework that decomposes complex instructions into subtasks and employs an evaluator-driven reflection loop. Experiments across six systems demonstrate limitations in existing audio-video editing models, while AVE-Agent achieves improvements in editing accuracy, content preservation, and audio-video alignment.

Critical strengths
  1. Timely and meaningful task. Audio-video editing is an important emerging research direction. I appreciate the paper's emphasis on coordinated audio-visual editing, particularly the need to preserve non-target content and maintain cross-modal consistency.

  2. Comprehensive evaluation framework. The benchmark covers diverse editing scenarios and evaluates multiple complementary dimensions. The fine-grained checklist-based evaluation provides more interpretable feedback than conventional similarity-based metrics.

  3. Interesting agent-based framework. AVE-Agent introduces a reasonable approach to decomposing complex editing instructions and improving results through evaluator-driven feedback. The experimental results and ablation studies support the effectiveness of the proposed framework.

  4. Thorough experimental analysis and visualization. The evaluation includes representative editing systems, difficulty-based analysis, and qualitative examples that clearly illustrate the limitations of existing approaches.

Critical weaknesses
  1. Limited benchmark scale. Although the benchmark covers 28 editing categories, it contains only 196 instructions. This raises questions about whether the dataset is sufficiently representative, particularly for complex cross-modal editing scenarios. A discussion of category-level sample sizes and statistical reliability would be helpful.

  2. Reliability of MLLM-based evaluation. The benchmark relies heavily on Gemini 3.1 Pro for checklist-based evaluation. Although the paper reports approximately 90% agreement with human judgments, further analysis of disagreement cases and particularly challenging cross-modal evaluations would strengthen the evaluation's credibility.

  3. Novelty of the agent-based framework. AVE-Agent primarily combines existing editing tools with task decomposition and evaluator-driven refinement. It would be helpful to clarify the technical novelty beyond existing planning and self-refinement approaches, and to provide more detailed comparisons with simpler agent-based baselines.

  4. Trade-offs in editing quality and efficiency. While AVE-Agent improves instruction following and fidelity preservation, its speech-quality score decreases compared with the Wan2.7 backbone (0.368 vs. 0.428). Additionally, the iterative framework introduces additional computational cost and latency. Further discussion of these trade-offs would be valuable.

Questions for authors

See the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.