acceptodds
Under review as a conference paper at ICLR 2027

Beyond Coherence: Benchmarking Professional Editing-Technique Execution in Multi-Shot Audio-Video Generation

Abstract

Recent multi-shot audio-video generators can produce increasingly coherent and cinematic outputs, but coherence does not imply the ability to execute editing techniques. Professional editing depends on shot structure, transition grammar, audio-video cut relations, and montage, yet existing benchmarks largely rely on proxies such as content quality, synchronization, or physical plausibility, systematically missing whether such editing instructions are actually executed. We introduce CutCraft, the first benchmark for editing-technique execution in multi-shot audio-video generation. CutCraft extends structured multi-shot prompts with explicit editing specifications and is paired with a hierarchical hybrid evaluation framework that combines shot-structure alignment, expert-model metrics, tool-grounded multimodal judgment, and rubric-based question answering. Beyond evaluation, we design an agentic editing baseline that decomposes generation into planning, shot-level synthesis, and post-hoc composition, explicitly realizing editing semantics such as J-cuts, L-cuts, and transition timing. Across 13 state-of-the-art closed- and open-source models, CutCraft reveals a consistent gap between coherence and editing-technique execution: current systems often produce plausible multi-shot videos yet fail to execute editorial instructions reliably. We find unstable shot structures, weak control of transition execution, and sharp degradation on higher-order montage, while aesthetic quality is only weakly correlated with editing-technique compliance. We will release the benchmark and metrics, and the editing agent baseline.

Primary area:
datasets and benchmarks
Keywords:
Multi-shot Audio-Video Generation, Montage and Transition Evaluation, Cinematic Video Generation, Editing Techniques Benchmark

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_euDDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CutCraft, a benchmark for evaluating professional editing-technique execution in multi-shot audio-video generation. The benchmark contains 295 structured prompts, covering 1,228 shots, 933 transitions, and 13 montage subtypes. To assess editing performance, the authors propose a hierarchical evaluation framework that combines shot alignment, specialized expert models, and multimodal language model judgments across 16 metrics. They also introduce an agentic editing baseline that decomposes video generation into planning, shot-level synthesis, composition, and iterative repair.

Experiments on 13 closed- and open-source video generation models reveal substantial gaps between visual coherence and editing-technique execution, particularly in complex montage structures and audio-video transitions. The agentic baseline improves several editing-related metrics, suggesting that explicit editing operations can complement existing generative models.

Critical strengths
  1. An interesting and practically relevant evaluation problem. The paper highlights an important distinction between generating coherent videos and executing professional editing instructions. Evaluating shot organization, montage, and audio-video transitions is relevant to practical video production and deserves further investigation.

  2. Broad coverage of professional editing techniques. CutCraft incorporates diverse editing concepts, including narrative structure, montage, cinematographic transitions, optical effects, and audio-video relationships. The structured prompts and expert-assisted annotations provide a useful foundation for fine-grained analysis.

  3. Extensive evaluation of contemporary video generation models. The experiments cover 13 representative models and provide detailed quantitative results and qualitative failure cases. The analysis of continuous versus discontinuous montage and the degradation associated with increasing shot complexity offers useful insights into current model limitations.

  4. An initial attempt to validate the evaluation framework. The authors conduct human-alignment experiments, shot-alignment ablations, and metric sensitivity analyses. These experiments are valuable for investigating the reliability of the proposed evaluation framework.

Critical weaknesses
  1. Insufficient differentiation from existing benchmarks. Although CutCraft claims to be the first benchmark for editing-technique execution, its distinction from related benchmarks, particularly CineTechBench, is not sufficiently established. The paper lacks a systematic comparison demonstrating which evaluation capabilities are unique to CutCraft and what additional insights it provides.

  2. Missing comparisons with simpler MLLM-based evaluation baselines. The proposed hierarchical evaluation framework combines expert models, heuristic rules, and VLM judgments, but lacks a direct comparison with a simple MLLM-as-a-Judge baseline using the same evaluation rubrics. It remains unclear whether this complex framework offers substantial improvements over direct multimodal evaluation.

  3. Insufficient ablation of the agentic editing baseline. The reported improvements in shot duration, optical transitions, and audio-video relations may primarily result from deterministic post-processing rather than agentic planning or reasoning. Without comparisons against simple rule-based composition and variants without iterative repair, the actual contribution of the agentic components remains unclear.

Questions for authors
  1. Could the authors provide a systematic comparison with CineTechBench and other closely related benchmarks to clarify CutCraft's unique evaluation capabilities and additional value?

  2. How does the proposed hierarchical evaluation framework compare with a straightforward MLLM-as-a-Judge baseline using the same prompts and evaluation rubrics, particularly in terms of human alignment and evaluation cost?

  3. Could the authors provide controlled ablations comparing the full agentic pipeline against simple shot-level generation with rule-based composition and variants without iterative repair, to isolate the benefits of agentic planning and reasoning?

3 comments
AuthorsOpenReview ↗

Response to Reviewer euDD

[Q1]Could the authors provide a comparison with CineTechBench and other closely related benchmarks?

[A1] CutCraft focuses on whether video generation models can execute explicitly specified editing instructions in synthesized multi-shot audio-video outputs. This differs from benchmarks primarily concerned with recognizing or interpreting cinematographic techniques in existing footage. Our emphasis is on editing-instruction execution rather than general audiovisual quality alone. The comparison with general-purpose video generation benchmarks is already discussed in Section 2.

Specifically, CutCraft explicitly evaluates montage types, cross-shot editing logic, and transition execution. It also incorporates event-chain-aware shot alignment and grouping to handle missing, fragmented, or merged shots and support structurally consistent evaluation. In addition, we include within-shot cinematographic attributes to complement the cross-shot editing dimensions. CineTechBench you mentioned evaluates cinematographic understanding using real film images and clips. Its focus is on within-shot cinematographic techniques, such as shot scale, camera angle, and camera movement, rather than higher-level montage structures or the execution of transitions between shots. Our CutCraft evaluates multiple levels, from shots and transitions to editing logic and montage. Built on a dedicated event-chain-based shot-alignment framework and complemented by selected general-purpose metrics, it provides a broad and detailed assessment of editing execution.

[Q2]How does the proposed hierarchical evaluation framework compare with a straightforward MLLM-as-a-Judge baseline using the same prompts and evaluation rubrics?

[A2] Our hierarchical hybrid evaluation framework was motivated by the limited reliability of direct MLLM-based evaluation in our initial experiments. To examine the contribution of expert evidence, we removed all expert-model signals from Pathway II and asked Qwen3.5-Omni to answer the corresponding evaluation questions directly, using the same scoring rules. The table below reports the correlations between these direct evaluations and human judgments across the relevant dimensions.

MetricB3C1D1D3E1
Pearson0.70720.51810.49310.63390.3706
Spearman0.66670.79040.28570.59520.5714

The results show that removing all expert-model signals substantially reduces correlation with human judgments, suggesting that the model struggles to resolve complex editing semantics from visual input alone. We add the results in Table 8 in the main text.

[Q3]Could the authors provide controlled ablations comparing the full agentic pipeline against simple shot-level generation with rule-based composition and variants without iterative repair?

[A3] Iterative repair and retry are integral components of our agent architecture. To isolate their contribution, we disabled all repair and retry mechanisms and evaluated a single-pass workflow consisting of shot-level generation and composition. The results are shown below;

ModelA1A2A3A4B1B2B3C1D1D2D3E1E2E3F1F2Overall
Happyhorse1.1*0.8940.6790.6410.5560.8700.6030.5700.4330.3760.6950.5210.5580.5370.8620.5190.5790.618
Wan2.7*0.9200.7310.6730.5730.8980.5400.5410.4830.3030.6470.4800.5440.5440.8970.2190.5880.597
LTX2.3*0.8140.5480.5620.5940.6940.5170.5380.4310.2470.4470.3650.4920.4900.8570.4700.5350.535

Removing iterative repair reduces the overall scores relative to the full agentic pipeline, supporting the contribution of the feedback-and-repair loop. Meanwhile, the single-pass workflow still outperforms the corresponding end-to-end generation baseline overall, suggesting that shot-level generation and explicit composition also contribute to the gains. Together, these comparisons support the complementary roles of shot-level generation/composition and iterative repair, which form the two core components of our agent architecture. We add the results in Appendix C.

Thank the authors for their response. Most of my concerns have been addressed, so I will keep my score.

Thank you once again for the time and effort you have dedicated to reviewing our paper.

Best regards!

Cutcraft Authors

Reviewer_ED4SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents a new benchmark, CutCraft, designed to evaluate the ability of multi-shot AV (video with audio) generation models to adhere to specific editing instructions included in the generation prompt. They describe the process through which they generated and filtered the prompts and the coverage of edit styles and techniques that they cover, based on an expert taxonomy from the filmmaking literature. They propose and describe an evaluation framework based on a large set of automated metrics that are computed from the output videos to measure the model's adherence to the editing instructions. They evaluate 13 multi-shot video generation models on the benchmark. Additionally, they propose an agentic editing baseline which generates the shots one by one as opposed to a single multi-shot generation, and then edits them together to increase the adherence of the output to the editing instructions. The results show that the models are successful along some dimensions, but consistently weak along others, such as adhering to montage and transition editing instructions. The proposed agentic editing baseline scores more favorably on adherence to transitions and is close to the top-performing multi-shot models on montage. However, it is consistently outperformed by the top performing multi-shot models on mostly all other metrics.

Critical strengths

The paper proposes a benchmark specifically targeting adherence to editing instructions for multi-shot video generation models. This is not covered by prior benchmarks and thus provides insight into the model's ability to follow instructions, specifically as they pertain to editing across and between shots. They propose a methodology designed to produce prompts with specific editing instructions that cover a broad range of editing styles and techniques from the filmmaking literature.

Critical weaknesses

The evaluation framework. Critical to the contribution is the ability to automatically evaluate the output of video models against this benchmark. For this, the paper proposes an automated framework based on the computation of multiple automated metrics that try to capture different aspects of the edit instruction. While the framework has merit, it also suffers from some significant limitations and challenges:

  • Not all of the metrics are validated against human experts to confirm they actually measure the desired trait. It is not clear to me why not all metrics were validated, and this raises concerns that not all metrics map to human experts reliably.
  • I took a closer look at audio metrics F1 and F2 and have serious concerns about their implementation's validity. For AV synchronization, they rely on aligning visual and audio onsets, which is very error-prone, instead of using capable models that are readily available to measure synchronization, like SyncFormer. Similarly, to measure audio quality, they rely on relatively low-level DSP audio features, as opposed to using existing and more powerful methods like Meta's AudioBox Aesthetics. I didn't dig into the implementation details of every other metric, but finding these significant limitations raises my concern that there could be similar implementation details in the rest of the metrics that may not be sufficiently reliable or justified.
  • While the goal of the framework is to focus on adherence to editing instructions, I think it's a problem to entirely ignore the more classic measures related to coherence and continuity. A model could score strongly on the proposed benchmark using the evaluation framework while still producing content which is pragmatically unusable due to poor consistency or lack of continuity. More on this in the following section of the review about the agentic editing baseline.

The agentic editing baseline. The baseline generates shots one at a time, as opposed to a single multi-shot generation, and then edits them together. The hypothesis is that in this way, it can adhere much better to the editing instructions in the prompt. In practice, this hits two limitations:

  • From the main results table 1, we see that the agentic approach underperforms on almost every measure compared to the top-performing multi-shot model (H3).
  • I actually watched and listened to some of the video outputs of the agentic baseline and contrasted them to the outputs of H3 and seedance. This reveals a pretty different story to the one told in the paper - pretty much every result from the agentic baseline I inspected was of poor quality and realistically unusable. The key failure point was consistency - both visual and audio, but where it was really stark was in the audio inconsistency: The changes and transitions in audio between shots were often jarring, and audio was inconsistent between shots in terms of content (e.g. objects making different sounds, ambiences changing abruptly, etc.). This is a well-known challenge when working with shots that are generated separately. Conversely, this is something that the multi-shot video generation models did pretty well - spanning ambiences across shots and ensuring objects make consistent sounds in different shots.
  • It is plausible that the multi-shot video generation models suffer from an instruction-grounding gap, i.e. maybe they were just not trained with editing instructions, and are evaluated to zero-shot adhere to them in this benchmark. So it could be the case that the observed limitations of the multi-shot models along some metrics are not inherent, and could be addressed by incorporating editing instructions into the training captions.
Questions for authors
  • Why aren't all of the proposed metrics in the evaluation framework validated against human experts?
  • Did you measure whether any of the multi-shot generation models sometimes apply editing techniques or styles (like specific video transitions or J/L cuts), even when they are not prompted to? The framework only measures adherence when specific edit instructions are included in the prompt, but we don't have a baseline understanding of whether the multi-shot models are capable of leveraging these filmmaking techniques without being prompted explicitly for them .
  • Why are the audio metrics built from the ground up using relatively basic DSP features, as opposed to leveraging models like syncformer and the tools in AudioBox Aesthetics. Are any of the other metrics similarly "handcrafted" when they could instead leverage more powerful models/tools from the literature?
  • Audio continuity and consistency across separately generated shots can be a huge challenge. In practice, this really limits the usability of the agentic editing baseline. Is there a way to overcome this limitation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ED4S (Part1)

[Q1]Why not all of the metrics are validated against human experts?

[A1] We have now expanded the human alignment study to cover previously unvalidated dimensions. Their correlations with expert judgments are presented below:

MetricA4B2B3D2E1F1F2
Pearson0.820.830.960.870.900.770.72
Spearman0.810.810.830.710.830.810.63

Our initial study focused on seven core editing-related dimensions, while several supporting dimensions were based on established predictors or concepts from prior benchmarks.

E2 uses the aesthetic-quality predictor adopted by VBench, preserving its core prediction and aggregation procedure. B1 measures compliance with explicit shot-duration requirements and is therefore amenable to objective timing validation, subject to accurate boundary detection and shot alignment. Appendix B.2.3 also provides controlled positive and negative examples for D2 and D3.

The additional results indicate good agreement with expert judgments across these dimensions. We add the results in Table 7 in the main text.

[Q2]Are the metrics sufficiently reliable or justified?

[A2] Our metric design draws on prior literature, with the references and computation procedures detailed in Appendix B.1. Some metrics reuse established approaches adopted and validated in existing benchmarks. For example, E2 adopts VBench’s aesthetic-quality predictor as a supporting measure that complements our editing-focused evaluation. The main text reports human-alignment experiments for the core metrics, and our response to your Q1 provides additional human-alignment results for the remaining dimensions.

Regarding the audio-related dimensions, CutCraft primarily evaluates the execution of professional editing instructions. General audio-quality and synchronization measures serve as supporting indicators rather than a comprehensive replacement for specialized audio evaluation. Our samples contain speech, music, sound effects, and ambient audio, so evaluators must be validated on this mixed-audio setting. We therefore did not rely on a speech-specific model as a universal evaluator. The current implementation uses signal features related to energy, spectral structure, onset activity, and temporal correspondence.

We supplemented the audio-related dimensions with human-correlation validation and additionally compared DNSMOS and Meta Audiobox Aesthetics PQ for audio quality. The results are shown below:

MetricOriginal F2DNSMOSAudiobox
Pearson0.7161-0.58280.7323
Spearman0.6347-0.65870.6826

In the audio-quality comparison, the original metric achieves Pearson and Spearman correlations comparable to those of Audiobox Aesthetics. These findings provide empirical support for its use as a lightweight supporting indicator in this evaluation. And we will consider using Audiobox as the evaluator for F2 in future work. DNSMOS exhibits negative correlation with human judgments in this evaluation, suggesting limited suitability for our mixed-audio setting.

AuthorsOpenReview ↗

Response to Reviewer ED4S (Part2)

[Q3]Problem to ignore the more classic measures related to coherence and continuity.

[A3] CutCraft focuses on the execution of professional editing instructions, while also incorporating general quality dimensions. Specifically, §3.2.1 and Appendix B.1 include causal coherence (A3), image aesthetic quality (E2), cross-shot physical consistency (E3), audio-visual synchronization (F1), and audio quality (F2). E3 explicitly considers abrupt changes in object attributes, such as color and shape, across shots. Furthermore, Table 14 in Appendix B.2.5 separately aggregates editing-execution and supporting metrics to distinguish these capabilities.

Comprehensive evaluation of audiovisual continuity is beyond the primary scope of CutCraft, whose central goal is to measure whether explicitly requested editing techniques are executed. We include selected general-quality measures as supporting indicators and intend CutCraft to complement established quality and consistency evaluations, rather than replace other general benchmarks or diminish their value.

[Q4]Why the agentic approach underperforms compared to the top-performing multi-shot model (H3)?

[A4] Comparisons across different base models conflate differences in model capabilities with differences in generation workflows, and therefore cannot isolate the contribution of the agent architecture. A more controlled comparison evaluates end-to-end and agentic generation using the same base model. Accordingly, we apply our agent to Happyhorse1.1, Wan2.7, and LTX2.3 and report changes relative to their respective end-to-end counterparts. These comparisons show that shot-level planning, generation, and explicit composition can improve the execution of certain editing instructions.

The lower within-shot visual quality you observed relative to leading models may also partly reflect differences in base-model capabilities. Under the same base-model settings, we observe modest improvements in some general-quality metrics with agentic generation, suggesting that generating individual shots can benefit certain aspects of quality.

We provide paired examples of agentic and end-to-end generation using the same base model in this anonymous repository https://huggingface.co/datasets/PaperED4S/review_ed4s_videos/tree/main. In these examples, the agent achieves better editing execution, including correctly rendered transition effects and audio-video cut relations. However, it does not consistently improve base-model-dependent qualities such as visual fidelity. This is expected, as the agent changes the generation and editing workflow without modifying the base model’s underlying capabilities.

Furthermore, independently generating and composing shots can introduce continuity problems, which remain a limitation of the current agent implementation. However, our agent’s contribution is to demonstrate the potential of an explicit editing workflow. It serves as a baseline showing that a workflow inspired by human editing practices can improve performance relative to the same base model’s end-to-end generation. The current system leaves substantial room for improvement and we did not present it as a state-of-the-art generation solution.

[Q5]Is it plausible that the multi-shot video generation models suffer from an instruction-grounding gap? Maybe they were just not trained with editing instructions, and are evaluated to zero-shot adhere to them in this benchmark.

[A5] Training base models to acquire editing capabilities is beyond the scope of this work; our objective is to evaluate their existing capabilities. Some models’ failure to execute editing instructions may stem from a lack of explicit editing descriptions in their training data, rather than insurmountable limitations of their architectures. Insufficient training coverage and an observed capability gap are compatible explanations: identifying editing operations that models have not reliably mastered, and providing concrete targets for future training, is one of CutCraft’s intended contributions.

Even if incorporating editing instructions into training improves performance, a benchmark is still needed to measure those gains and assess whether they generalize across scenes, shot structures, and combinations of editing techniques.

AuthorsOpenReview ↗

Response to Reviewer ED4S (Part3)

[Q6]Did you measure whether any of the multi-shot generation models sometimes apply editing techniques or styles (like specific video transitions or J/L cuts), even when they are not prompted to?

[A6] Our qualitative observations suggest that, without explicit editing instructions, most models favor visual hard cuts and synchronous audio-visual cuts, rarely producing optical transitions or clear J/L-cut relations spontaneously. Some weaker models struggle to execute these operations reliably even when explicitly instructed, instead reverting to hard cuts and synchronous audio-visual cuts. These observations highlight two distinct phenomena: a default preference for simple concatenation and a failure to execute more complex editing operations on request. The former reflects a model’s default generation tendency and does not itself indicate insufficient capability; the latter reveals an instruction-execution gap. CutCraft primarily evaluates the latter: whether models can realize the requested editing operations when users explicitly specify them. We provide examples at the anonymous repository https://huggingface.co/datasets/PaperED4S/review_ed4s_videos/tree/main comparing four leading models' (Seedance2.5, Minimax H3, Kling V3, Happyhorse1.1) outputs generated from prompts with the same content, with and without explicit editing instructions. In these examples, outputs generated without editing instructions exhibit neither transition effects nor explicitly asynchronous audio-video relations at shot boundaries. With editing instructions, the model can produce some of the requested effects, but does not execute them all correctly, indicating a remaining gap in editing-instruction execution.

[Q7]Audio continuity and consistency across separately generated shots can be a huge challenge. In practice, this really limits the usability of the agentic editing baseline. Is there a way to overcome this limitation?

[A7] Simple loudness adjustments or cross-fades can smooth the joins but cannot ensure continuity of audio content. A promising direction is to partially decouple visual shot segmentation from audio generation: visuals would still be generated shot by shot, together with speech and sounds from key on-screen objects, while background audio would be generated jointly afterward. Shared audio references could then be maintained for recurring characters, objects, and scenes, allowing ambience and background music to extend naturally across shot boundaries. J/L-cuts would operate on sounds suitable for anticipation or continuation, rather than simply shifting an entire shot’s audio track. Furthermore, cross-shot audio generation or local repair could be applied after the initial video composition to jointly refine sounds around transitions while preserving existing correct synchronization, supported by cross-shot audio-consistency checks.

Reviewer_vuFcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CutCraft, a benchmark for evaluating professional editing-instruction execution in multi-shot audio-video generation.

It contains 295 structured prompts covering 13 montage subtypes, uses a hierarchical evaluation framework including shot alignment, specialized models, and multimodal judgment.

The results reveal limitations in montage and transition execution, while explicit planning and post-production composition improve some aspects of editing control.

Critical strengths
  • The paper distinguishes editing-instruction execution from visual quality and coherence, targeting capabilities such as montage organization, transitions, and J/L-cuts that most video metrics overlook.
  • Experiments on 13 models, expert assessment of 640 videos, shot-alignment ablations, and sensitivity analyses provide great empirical support for the benchmark.
Critical weaknesses
  • No analysis of generation cost, especially for the agentic baseline. The agentic pipeline requires an evaluate–attribute–repair loop with up to two re-generation rounds, which is more expensive than one-shot end-to-end generation. However, the paper reports no generation time, API call count, or monetary cost for any setting.
Questions for authors
  • How are the evaluator’s training, validation, and test sets constructed? Could the authors include a Qwen3-VL-8B baseline to quantify the gains from training?
  • In Table 1, D3 improves substantially for all three agentic settings, yet F1 (within-shot audio-video synchronization) drops sharply in every case. Could the authors explain this consistent degradation?
2 comments
AuthorsOpenReview ↗

Response to Reviewer vuFc

[Q1]No analysis of generation cost, especially for the agentic baseline.

[A1] Generation time varies across models. In our experiments, closed-source APIs typically took approximately 2-5 minutes per video. Among open-source models, LTX was relatively fast, requiring approximately 1-2 minutes per sample, whereas MOVA took more than 30 minutes per sample. The agentic pipeline incurs additional latency because it makes multiple model calls to generate individual shots and, when needed, regenerate them during repair. The average end-to-end generation times for our agentic settings were 715s for HappyHorse, 940s for Wan, and 568s for LTX. We add the analysis in Appendix C.

[Q2]How are the evaluator’s training, validation, and test sets constructed? Could the authors include a Qwen3-VL-8B baseline to quantify the gains from training?

[A2] We used the 640 human-annotated video samples described in the paper, allocating 70% to training, 15% to validation, and 15% to testing. Videos generated from the same prompt are kept in the same split to prevent data leakage. Our preliminary experiments on a small subset showed poor performance when using the open-source model directly as an evaluator. To examine this more systematically, we evaluated the unmodified Qwen3-VL-8B model on the test set. The table below reports the correlations between its evaluations and human judgments.

MetricC1D1D3
Pearson0.57670.51210.6934
Spearman0.57490.54760.5952

These results suggest that editing-related judgments require higher-level video understanding that the small model struggles to perform without task-specific training. We add the results in Table 9 in the main text.

[Q3]Could the authors explain the F1 degradation in agent results?

[A3] To explicitly realize the requested transitions, particularly J- and L-cuts, our agent currently introduces some additional margin in the audio-video offsets, which may disrupt the correspondence between local sound events and visual actions at shot junctions. Meanwhile, independently generated adjacent shots are not guaranteed to maintain continuity in loudness, sound content, or onset characteristics, further affecting audio onsets near shot boundaries. The current repair pipeline prioritizes shot duration, transition effects, and cross-shot audio-video relations without explicitly constraining F1 degradation. These results therefore reveal a trade-off in the current implementation between editing-instruction execution and synchronization preservation. We would also like to clarify that our agent is intended as a baseline illustrating how models can approximate the shot-by-shot workflow of human editors, and substantial room for improvement remains. We will further distinguish these effects by analyzing transition regions and shot interiors separately, and explore separating background sounds that can lead or extend across cuts from action-linked sounds that need to remain synchronized.

Dear Reviewer,

We hope that our detailed responses and new experiments have effectively addressed your comments. Since the discussion window will be closing soon, we look forward to your feedback and remain available to address any follow-up questions. We would greatly appreciate it if our clarifications meet your expectations.

Many thanks once again for your time and thoughtful review of our manuscript.

Sincerely,

The CutCraft Authors

Reviewer_HskAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed a benchmark for recent audio-video generators that evaluates their adherence to instructions for various editing techniques. The dataset is constructed by alternating LLM automation and expert filtering multiple times. The scoring system comprises multiple evaluation methods, and it reports 16 (sub)metrics plus their average. The benchmark results reveal the current state-of-the-art generators' weaknesses at executing editing techniques.

Critical strengths

Prior benchmarks mostly focused on semantic adherence and coherence of the (audio-)video generators. The authors proposed a benchmark that allows in-depth analysis of those models' editing-technique execution capabilities, which could benefit the relevant community.

Critical weaknesses

I find that the design choices for the proposed metrics, only available in the appendix, are sometimes arbitrary. To list a few:

  • Why does B3_mood need to be match_ratio - 0.5 * conflict ratio? How was this determined?
  • How can the audio quality (F2) be determined based on "signal-to-noise ratio"? What is the signal, and what is the noise here? How were the target source and the residual separated? Is there any reference that supports this design choice? Similarly, does a low dynamic range necessarily indicate bad audio for a given video?

Some evaluation scores are validated with correlation analysis w.r.t. the expert scores, but the test included only 7 of the proposed 16 metrics; hence, it is unclear how valid and reliable those scores and their average "overall score" are.

It seems that the agentic baselines use the same evaluation metrics to improve themselves, raising the concerns of overfitting.

Overall, I find this submission borderline. I think the manuscript can be improved with an extra page during the rebuttal process. Specifically, could the authors improve the justification regarding the metrics, scoring pipeline, and fairness of the baseline agentic system?

I believe that the presentation could also be much improved. Throughout the paper, both the main text and appendices, there are too many in-line enumerations without a clear layout, which adds an extra burden to comprehend the content.

Questions for authors

I included relevant questions in the weakness section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer HskA (Part1)

[Q1]Are the design choices for the proposed metrics in the appendix arbitrary?

[A1] Our metrics combine evidence relevant to each evaluation dimension, and some combinations involve heuristic parameters. For B3, the default coefficient λ=0.5\lambda=0.5 assigns conflicting evidence a smaller penalty than the reward for matching evidence. We evaluated λ∈0,0.25,0.5,0.75,1\lambda \in {0, 0.25, 0.5, 0.75, 1} while keeping all other components unchanged. The resulting B3 scores and rankings are shown below.

Modelλ = 0λ = 0.25λ = 0.5λ = 0.75λ = 1
Minimax H30.725 (3)0.691 (3)0.677 (3)0.665 (3)0.655 (3)
Seedance2.50.765 (1)0.711 (1)0.686 (2)0.672 (2)0.661 (2)
Seedance2.00.702 (4)0.668 (4)0.653 (4)0.640 (4)0.631 (4)
Happyhorse1.10.745 (2)0.708 (2)0.688 (1)0.675 (1)0.664 (1)
Kling V30.671 (5)0.646 (5)0.633 (5)0.623 (5)0.614 (5)
Veo3.10.640 (7)0.622 (6)0.608 (6)0.598 (6)0.589 (6)
Wan2.70.572 (10)0.555 (9)0.543 (9)0.536 (9)0.530 (8)
Vidu Q30.613 (8)0.581 (8)0.557 (8)0.540 (8)0.529 (9)
LTX2.30.591 (9)0.553 (10)0.530 (10)0.516 (10)0.507 (10)
MOVA0.538 (11)0.509 (11)0.487 (11)0.468 (11)0.452 (11)
Ovi0.470 (13)0.417 (13)0.386 (13)0.367 (13)0.352 (13)
Davinci0.643 (6)0.611 (7)0.584 (7)0.562 (7)0.546 (7)
JavisDiT++0.473 (12)0.459 (12)0.451 (12)0.445 (12)0.440 (12)

As expected, increasing the conflict penalty monotonically decreases the B3 scores of all models. More importantly, the relative rankings remain largely stable. We add the results in Appendix B.2.4.

The term “signal-to-noise ratio” in F2 was imprecise: the implementation measures RMS-based energy contrast without separating signal and noise. We have corrected this terminology in the revised manuscript, thank you for pointing out this issue.

F2 currently combines features extracted using librosa, including dynamic range, energy contrast, spectral structure. F2 was therefore designed as a supporting signal-level indicator for the editing benchmark. To address this concern, we additionally evaluated the generated audio using pretrained DNSMOS and Meta Audiobox Aesthetics PQ. The results are reported below:

ModelOriginal F2 (Rank)DNSMOS Normalized (Rank)Audiobox PQ Normalized (Rank)
Minimax H30.526 (8)0.108 (9)0.708 (3)
Seedance2.50.538 (4)0.040 (12)0.829 (2)
Seedance2.00.555 (2)0.041 (11)0.685 (4)
Happyhorse1.10.533 (7)0.273 (7)1.000 (1)
Kling V30.560 (1)0.000 (13)0.592 (6)
Veo3.10.546 (3)0.293 (6)0.610 (5)
Wan2.70.512 (11)0.593 (4)0.052 (12)
Vidu Q30.537 (5)0.095 (10)0.581 (7)
LTX2.30.516 (10)0.176 (8)0.399 (8)
MOVA0.494 (12)0.352 (5)0.202 (11)
Ovi0.517 (9)0.608 (3)0.208 (10)
Davinci0.534 (6)1.000 (1)0.318 (9)
JavisDiT++0.364 (13)0.654 (2)0.000 (13)

We also compared the original F2 and these pretrained predictors against human quality annotations:

MetricOriginal F2DNSMOSAudiobox
Pearson0.7161-0.58280.7323
Spearman0.6347-0.65870.6826

The results suggest that DNSMOS, which primarily targets perceptual speech quality, is less aligned with human judgments on our dataset, where many samples are not speech-dominant. This limits its suitability as the evaluator for this dimension. Meanwhile, our method achieves human correlation comparable to Audiobox. We will consider using Audiobox as the evaluator for F2 in future work. Audiobox metrics can also be used as rewards in post-training to improve audio quality. Top models may have adopted similar approaches.

[Q2] Why only 7 of the proposed 16 metrics are validated with human experts?

[A2] We have expanded human-alignment validation to additional dimensions—A4, B2, B3, D2, E1, F1, and F2. The additional dimensions achieve model-level Pearson correlations of 0.72–0.96 and Spearman correlations of 0.63–0.83 with expert judgments. The results are added in Table 7 in the revised manuscript.

MetricA4B2B3D2E1F1F2
Pearson0.820.830.960.870.900.770.72
Spearman0.810.810.830.710.830.810.63

Specifically, E2 adopts the aesthetic-quality predictor used in VBench, retaining its core prediction and aggregation procedure. B1 measures adherence to explicitly specified shot durations and can therefore be assessed through objective timing measurements rather than repeated subjective ratings, provided that boundary detection and alignment are accurate.

AuthorsOpenReview ↗

Response to Reviewer HskA (Part2)

[Q3]Do the agentic baselines use the same evaluation metrics to improve themselves? raising the concerns of overfitting.

[A3] Importantly, during inference, our agent uses only B1, D2, and D3 to trigger retries, localize failures, guide repairs, and select candidate outputs. Evaluation scores are not used to update the generation model’s parameters and it does not use the full set of benchmark metrics. As shown in Table 1 of the main text, the agent also improves scores on metrics beyond these three relative to the same base model, providing evidence that its gains are not confined to the metrics used for feedback. To further assess the effectiveness of our shot-by-shot editing workflow, we conducted the following experiments. We disabled all intermediate evaluation, feedback-based repair, and retries, retaining only planning, shot-level generation, and composition. Results on the same evaluation samples are shown below:

ModelA1A2A3A4B1B2B3C1D1D2D3E1E2E3F1F2Overall
Happyhorse1.1*0.8940.6790.6410.5560.8700.6030.5700.4330.3760.6950.5210.5580.5370.8620.5190.5790.618
Wan2.7*0.9200.7310.6730.5730.8980.5400.5410.4830.3030.6470.4800.5440.5440.8970.2190.5880.597
LTX2.3*0.8140.5480.5620.5940.6940.5170.5380.4310.2470.4470.3650.4920.4900.8570.4700.5350.535

The single-pass workflow still outperforms the corresponding end-to-end generation baselines overall. This indicates that at least part of the improvement comes from shot-level generation and explicit composition, rather than requiring repeated repair guided by benchmark feedback.

Furthermore, to assess the effectiveness of our full pipeline under different intermediate evaluators, we replaced the intermediate checks with direct judgments of the rendered videos by Qwen3.5-Omni, without providing CutCraft’s expert-model signals or feedback computed using its original scoring formulas. Results on the same evaluation samples and repair budget are shown below:

ModelA1A2A3A4B1B2B3C1D1D2D3E1E2E3F1F2Overall
Happyhorse1.1*0.8740.6590.6980.6540.7990.5930.6020.5100.3790.5130.6090.5470.5520.8900.5310.5760.625
Wan2.7*0.9020.7240.6810.6480.8120.5840.4760.4830.3540.5210.4760.6030.5050.8670.3700.5990.601
LTX2.3*0.8570.5130.5520.6650.5530.5010.6640.3670.3410.4320.4940.5130.5050.7590.4550.5530.545

Improvements remain observable after replacing the feedback mechanism. This suggests that feedback-based repair does not depend exclusively on the original expert signals and scoring rules: actionable diagnostic feedback can also support the repair process. We provide the results above in Appendix C in the revised version.

We provide representative agent-generated samples from the two settings described above in this anonymous repository https://huggingface.co/datasets/PaperRe/HsKA/tree/main.

[Q4]There are too many in-line enumerations without a clear layout, which adds an extra burden to comprehend the content.

[A4] We have reorganized the descriptions of the evaluation dimensions and scoring pathways into clearly structured lists, and broken up dense passages in both the main text and appendices. Thank you for your suggestion.

Dear Reviewer,

We hope our response has addressed your questions. As the discussion phase is coming to a close, we are looking forward to your feedback and would like to know if you have any remaining concerns we can address. We are grateful if you find our revisions satisfactory and consider raising your score for our paper.

Thank you once again for the time and effort you have dedicated to reviewing our paper.

Best regards!

CutCraft Authors

Dear Reviewer,

As the discussion window will close in less than one day, we would like to kindly follow up on our previous response. We have carefully revised the paper according to your suggestions, supplemented the requested explanations, and conducted the additional experiments which are provided in our anonymous repository.

We sincerely hope that our revisions have adequately addressed your concerns. We would be very grateful if you could share your feedback and consider raising the score for our paper.

Thank you once again for your valuable time and effort in reviewing our work.

Best regards!

CutCraft Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to all Reviewers

We thank all reviewers for their constructive feedback. We are encouraged that the reviewers recognized the value of distinguishing editing-instruction execution from visual quality and coherence (HskA, vuFc, ED4S, euDD), highlighted the broad coverage of professional editing techniques enabled by our structured prompts and expert-assisted annotations (ED4S, euDD), and acknowledged the empirical support provided by evaluations across 13 models, expert assessment, shot-alignment ablations, and sensitivity analyses (vuFc, euDD).

Below, we summarize our responses to shared concerns and the main revisions to the manuscript.

Shared concerns

  • Metric design and human validation (HskA, ED4S). We extend expert validation to additional dimensions, with Pearson correlations of 0.72–0.96 and Spearman correlations of 0.63–0.83 for the newly validated dimensions. We clarify the rationale and prior work underlying the metrics, distinguish established predictors from task-specific adaptations, and add sensitivity analysis of the B3 conflict-penalty coefficient. We also clarify F2’s signal-level interpretation and compare its human alignment with DNSMOS and Meta Audiobox Aesthetics PQ.

  • Evaluator training and direct MLLM baselines (vuFc, euDD). We add the unmodified Qwen3-VL-8B baseline to quantify the gains from evaluator training. Separately, we evaluate direct Qwen3.5-Omni judgments without expert-model evidence, retaining the same evaluation questions and scoring rules. These comparisons examine the contributions of task-specific evaluator training and expert evidence to human alignment.

  • Agentic baseline contribution, fairness, and limitations (HskA, euDD). We clarify that the agent uses only B1, D2, and D3 for inference-time feedback without updating the generation model’s parameters. We evaluate variants without intermediate feedback and repair and with an alternative MLLM checker; both retain overall gains over the corresponding end-to-end baselines. The agent is presented as a preliminary baseline demonstrating the potential of explicit editing workflows, with substantial room for improvement.

Main revisions in the updated manuscript

  • Section 4.4: Expanded human-alignment validation, direct MLLM-as-a-Judge comparisons, and an unmodified Qwen3-VL-8B baseline for evaluator validation (HskA, ED4S, euDD, vuFc).
  • Section 3.2 and Appendix B.1: Reorganized descriptions of the evaluation dimensions and scoring pathways into clearly structured lists(HskA).
  • Appendix B.2.4: Additional sensitivity analysis of the B3 conflict-penalty coefficient, complementing the analyses of A2 and A3 (HskA).
  • Appendix C: Agent ablations without feedback-based repair and with an alternative MLLM checker, together with generation latency and discussion of the agent’s computational trade-offs (HskA, euDD, vuFc).
  • Limitations: Expanded discussion of cross-shot audio continuity and the preliminary nature of the agentic baseline (ED4S, vuFc).

We also provide the newly generated results in two anonymous repositories https://huggingface.co/datasets/PaperRe/HsKA/tree/main (HskA) and https://huggingface.co/datasets/PaperED4S/review_ed4s_videos/tree/main (ED4S).

Changes to the manuscript are highlighted in blue.

Additional quantitative results and explanations are provided in our detailed response to each reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.