acceptodds
Under review as a conference paper at ICLR 2027

MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation

Abstract

Video generation is rapidly evolving from single-shot synthesis to complex multi-shot audio-video (MSAV) narratives to meet real-world demands. However, evaluating such frontier models remains a fundamental challenge. Existing benchmarks are limited in scope and data diversity, and rely on rigid evaluation pipelines, preventing systematic and reliable assessment of modern MSAV models. To bridge these gaps, we introduce , the first comprehensive benchmark and adaptive hybrid evaluation framework for multi-shot audio-video generation. Our benchmark spans four key dimensions, video, audio, shot, and reference, covering diverse task settings, varying shot counts of up to 15, and challenging non-realistic scenarios. Our evaluation framework improves robustness through an adaptive self-correction mechanism for shot segmentation, instance-wise rubrics for subjective metrics, and tool-grounded evidence extraction for complex judgments. Furthermore, MSAVBench achieves high alignment with human judgments, reaching a Spearman rank correlation of 91.5%. Our systematic evaluation of 20 state-of-the-art closed- and open-source models shows that the evaluated systems still struggle with director-level control and fine-grained audio-visual synchronization, while modular or agentic generation pipelines offer a promising path toward narrowing the gap between open- and closed-source models. The benchmark data and evaluation code will be made publicly available.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Audio-Video Generation, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KtUqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MSAVBench evaluates multi-shot audio-video generation with 286 expert-curated prompts, 2,198 shots, and 20 metrics spanning video, audio, shot, and reference conditions. Its adaptive hybrid evaluation framework combines shot self-correction, instance-wise rubrics, expert models, and tool-grounded scoring. Across 20 systems, it identifies weaknesses in cinematic control and audio-visual alignment. Its overall score correlates with expert system rankings at Spearman ρs=0.915\rho_s=0.915.

Critical strengths
  1. The curated prompts cover up to 15 shots, non-realistic scenarios, multilingual audio, cinematic controls, and reference conditioning, addressing a real gap in existing benchmarks.
  2. Shot self-correction improves exact shot-count accuracy from 55% to 78% on 200 annotated videos. Instance-wise rubrics and tool-grounded scoring also improve human agreement on complex metrics.
  3. The paper reports 1,200 blinded expert pairwise judgments for overall ranking and targeted validation of audio-visual metrics, alongside results from 20 systems.
Critical weaknesses
  1. The treatment of absent speech and failed speaker or subject detection in the reported 11-dimension aggregate is unclear. The shot-completion coefficient is capped at one, so excess shots do not incur a count penalty.
  2. The reference assets are unavailable in the submitted materials, which also contain no playable model outputs.
Questions for authors
  1. How are absent dialogue, failed detections, and extra or missing shots scored?
  2. Can the reference assets and representative playable outputs be shared for review, with per-video evaluation time and judge cost?
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We sincerely thank you for recognizing that our benchmark addresses a "real gap in existing benchmarks", the effectiveness of our "shot self-correction", "instance-wise rubrics", and "tool-grounded scoring", as well as our validation with "1,200 blinded expert pairwise judgments" and "results from 20 systems". Below, we address each of your questions and hope to clarify any concerns.


[W1 & Q1] Treatment of absent speech, undetected speakers/subjects, and shot-count mismatches.

We sincerely thank the reviewer for the insightful comments. We address your questions in three parts below:

  1. Scoring for missing content and undetected speakers/subjects.

We clarify that, in cases where required speech, speakers, or subjects are not detected, our evaluation system assigns either 0 or None, depending on whether the metric remains applicable. A score of 0 is included in the metric mean and overall score, whereas None is excluded from aggregation. Specifically:

1) Audio-related metrics. When a prompt explicitly requires speech, timbre consistency is assigned 0 if no audio speaker activity is detected. Lip-sync and sound attribution are assigned 0 if no speech, usable visual speaker, or valid audio-visual speaker match is found. If speech is detected but no valid cross-shot pair of the same speaker is available, timbre consistency is assigned None.

2) Subject-related metrics. For cross-shot subject consistency, only shots with detected subjects are used for feature extraction and comparison. If fewer than two valid shots remain, subject consistency is assigned None.

  1. Scoring for missing and extra shots.

When the generated shot count differs from that specified in the script, the evaluator first checks for incorrect or missed cuts and adjusts the shot boundaries accordingly. It then handles missing and extra shots as follows.

1) Missing shots. If fewer valid shots are generated than required, the overall score is multiplied by a completion factor of min(1, valid shots / required shots). For example, three valid shots for a five-shot script yield a factor of 0.6.

2) Extra shots. If more shots are generated than required, the evaluator sequentially matches each script shot to the generated video shots, and skips unmatched extra shots during scoring. This design focuses on the quality of the requested shots, because users can easily keep or trim extra shots as needed. Therefore, the completion factor does not impose additional penalty for extra shots. For example, if 6 shots are generated for a 5-shot script and 5 match the script, the completion factor remains 1.0.

  1. Handling evaluation-call failures.

We further note that the evaluator includes a fault-tolerance mechanism that automatically retries requests affected by transient failures, such as timeouts or interrupted connections.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

[W2 & Q2] Shared reference assets, playable outputs, and evaluation time/cost.

We thank the reviewer for these constructive suggestions. Below, we clarify the availability of review materials and report the evaluation time and cost.

  1. Reference assets and validation.

Since the licenses of the original source datasets restrict direct redistribution, we provide a replacement set of reference assets consisting of 138 unique generated images and 83 unique generated audio clips corresponding to the original 96 reference prompts. This replacement set preserves the original task coverage and distribution, including categories, styles, languages, and reference combinations.

These new assets are generated solely from the text descriptions. Specifically, the images are generated with qwen-image-2.0-pro-2026-06-22, gpt-image-2.6-sunburst, and gemini-2.5-flash-image. The audio clips are synthesized using qwen-audio-3.1-tts-flash, cosyvoice-v3.5-flash, and gemini-3.8-flash-tts. These reference assets are included in the updated supplementary materials under references/.

To verify consistency, we rerun Wan2.7, HappyHorse-1.0, and DreamID-Omni using these new references under the exact same evaluation setup. As shown below, the relative system rankings on all three metrics remain unchanged from the original Table 5. For example, Wan2.7 and HappyHorse-1.0 both continue to outperform DreamID-Omni on subject-image and face fidelity, and the pairwise ordering among the systems also remains unchanged. These results confirm that replacing the reference assets still preserve our benchmark conclusions.

MethodImg-DINO ↑Img-Face ↑Voice ↑
Wan2.7 R2V0.2130.4160.698
HappyHorse-1.0 R2V0.2540.259-
DreamID-Omni0.1150.0500.639
  1. Representative playable outputs.

We also provide playable audio-video outputs through our anonymous video gallery. These samples cover the typical failure cases and audio-visual failure cases analyzed in the paper, as well as side-by-side outputs from 17 evaluated systems across 10 prompts spanning 8 genres.

  1. Evaluation time and judge cost.

We measure the end-to-end evaluation time for 15-second, 720p multi-shot videos on 4×A800 (80G) GPUs, and report the evaluation time across different shot-count ranges as well as the overall average in the table below. As expected, the evaluation time increases with the number of shots, with an overall average of 139 seconds (2.31 minutes) per video.

Regarding judge cost, since Qwen3.5-397B-A17B and all expert tools are open-source and support local deployment, the entire pipeline can be run locally with zero API fees. If commercial APIs are used instead, the average judge cost is US$0.073 per video, based on measured token usage and public pricing.

Shot-Count RangeBenchmark PromptsMean Evaluation Time (s)
1–3 shots2064.5
4–6 shots8595.9
7–9 shots100142.8
10–12 shots65191.9
13–15 shots16219.5
Overall286138.8

Finally, we sincerely thank you again for your time and constructive feedback, which strengthens the clarity and reproducibility of our evaluation. In the revised manuscript, we clarify the scoring and shot-completion rules in Appendix Section C.1.6, describe the reference assets in Appendix Section B.4, and report evaluation time and API cost in Appendix Section D.2 (Table 12). All new and revised content is highlighted in blue. We hope our responses can address your concerns, and we would be glad to address any further questions.

Reviewer_UwydOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark for comprehensively evaluating the quality of multi-shot audio-video generation. To cover diverse content, the benchmark data are constructed through a four-stage pipeline. In Stage 1, domain experts define a taxonomy of video genres and themes. In Stage 2, an LLM generates initial scripts, which are then rewritten into more detailed ones. In Stage 3, domain experts screen and refine the scripts. In Stage 4, reference audio and video are collected. For evaluation, a VLM first inspects the shot boundaries detected by TransNet V2 and corrects them by merging or splitting segments for up to two iterations. The framework then applies one of three scoring paradigms depending on the nature of each metric. The first one uses existing specialized expert models. The second is instance-wise rubric-based scoring, in which a VLM answers prompt-specific multiple-choice questions. The third is tool-grounded agentic scoring, in which a VLM makes the final judgment based on evidence obtained from perception tools such as object detectors. Metrics at three hierarchical levels (global, cross-shot, and intra-shot, plus a reference level when reference data are available) include narrative coherence, lip synchronization, sound attribution, visual consistency, and camera parameter adherence. These metrics are aggregated into 11 dimensions to obtain the overall score. In the experiments, both closed-source and open-source models are evaluated, and their results are reported.

Critical strengths
  • Several joint audio-video generation models have emerged recently. In this context, a benchmark that comprehensively evaluates the quality of multi-shot audio-video generation is a valuable contribution to the community.
  • The construction pipeline is carefully designed to produce reliable benchmark data. I particularly appreciate that domain experts define the video content categories and filter the scripts.
Critical weaknesses
  • The findings obtained through the experiments are interesting, but the evidence supporting them is limited.
    • Finding 1: Based on the results of LTX-2.3 (TI2AV) and ViMax + LTX-2.3, the paper claims that modular or agentic open-source pipelines can narrow the gap with closed-source models. However, both configurations may rely on closed-source models.
      • The images used in the TI2AV setting are generated by Wan2.7-Image, which is a closed-source model.
      • The example configurations in the official ViMax implementation use Nano Banana and Gemini. However, Appendix F.5 does not specify which LLM was used for planning or which image generation model was used.
    • Finding 2: The paper claims that open-source models show “markedly lower” scores in layout alignment and camera control (C-Layout, I-Layout, and Cam.). However, of the four highlighted top cells in the Cam. column of Table 2, three belong to open-source pipelines (Wan2.2, LongLive, and Helios). In addition, for I-Layout, LTX-2.3 (TI2AV) (0.765) outperforms four of the six closed-source models. The claim therefore does not appear to be sufficiently supported.
    • Finding 4: According to the results in Appendix F.6, LongLive performs better when combined with MMAudio than with HunyuanFoley, but this comparison is not included in the main results. Moreover, both HunyuanFoley and MMAudio are specialized in sound effect generation and cannot generate speech that follows the given dialogue. These results alone are therefore not sufficient to conclude that the limitations of the “video-first, post-hoc dubbing” paradigm demonstrate the need for unified audio-video architectures.
    • In Section 4.3, the paper attributes the performance decline at larger shot counts due to “the inherent difficulty of long-horizon generation.” This claim is unclear in the following two respects.
      • The total video duration appears to be fixed at 15 seconds. Increasing the shot count therefore does not result in long-horizon generation.
      • When the shot count is large, many shots in the benchmark are 2 seconds or shorter. In such cases, the scores of the following metrics may decrease for other reasons.
        • Since frames are extracted at 1 fps, only one or two frames are available for judging camera motion in shots of 2 seconds or shorter.
        • Speaker timbre consistency excludes utterances shorter than 1 second, and the score is set to 0 when no valid utterances remain.
        • Music consistency estimates BPM and beats from music segments of 2 seconds or shorter, which is likely to make the estimates unstable.
  • I have concerns regarding the licensing of the reference data and the evaluation of reference-conditioned generation.
    • Please see Details Of Ethics Concerns for details.
    • The main results of this paper would not be affected. However, if these concerns cannot be resolved, I believe the results in Table 5 should either be removed from the paper or be replaced with results on a reduced subset that excludes the problematic data.
Questions for authors
  • Experimental settings
    • What is the basis for describing LTX-2.3 (TI2AV) and ViMax + LTX-2.3 as “modular or agentic open-source pipelines”?
    • What is the basis for the claim that open-source models show “markedly lower” scores in layout alignment and camera control (C-Layout, I-Layout, and Cam.) in Table 2?
    • Is there any evidence that each metric works properly even when the shot count is large?
  • Is the source code submitted as supplementary material identical to the code used in the experiments? While examining the implementation, I found the following issues. How were they addressed?
    • The dialogue, music, and OCR metrics are never computed. executor.py decides which metrics to apply, but the category names used for this decision do not match those stored as metadata in the jsonl files (e.g., “multi_dialogue” vs. “Multi-person Dialogue (Voice Attribution, Lip Sync, Social Interaction)”).
    • The overall score computation described in the paper does not appear to be implemented. The implementation simply averages the per-agent averages, without merging metrics into Visual Quality and Multi-Speaker Dialogue Audio. In addition, lip-sync and sound attribution are counted in both the story agent and the shot agent.
    • Lines 379-380 and 505-506 of rubrics.py raise a TypeError, which prevents the metrics of the story agent and the shot agent from being computed.
    • Text rendering accuracy does not compare the recognized text with the target text. As a result, it does not evaluate whether the specified characters are rendered correctly (CER is not computed).
    • Line 908 of layout/server.py contains a bug (“cross-shot” is used instead of “cross_shot”), which causes the cross-shot metric to be computed in the intra-shot mode.
    • The evaluation is biased toward the beginning of the video. The story-level VLM evaluations (narrative coherence and MCQ) use only the first 8 frames sampled at 1 fps, and DeSync analyzes only the first 8 seconds (line 32 of desync/server.py). According to the template in Appendix B.2, the default video length is 15 seconds, so the later shots are likely not evaluated.
14 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/5)

We sincerely thank you for recognizing MSAVBench as "a valuable contribution to the community", and for appreciating our "carefully designed" construction pipeline that produces "reliable benchmark data" with "domain experts" defining categories and filtering scripts. Below, we address each of your questions and hope to clarify any concerns.


[W1 & Q1] Experimental evidence for the main findings and metric reliability.

We sincerely thank the reviewer for the invaluable feedback. We address your concerns point by point below:

  1. More evidence for Finding 1.

We first clarify that in the original paper, ViMax uses Qwen3.5-397B-A17B for planning, Wan2.7-Image for storyboard generation, and LTX-2.3 for video generation.

To strictly verify Finding 1 using purely open-source components, we evaluate two new variants across all 286 prompts by replacing all underlying modules with open-source models: both adopt SenseNova-U1.5-8B-MoT [1] for image generation and LTX-2.3 for video generation. Specifically, the modular pipeline (LTX TI2AV) generates a storyboard image per shot as the first-frame condition and concatenates the generated clips in script order; the agentic pipeline (ViMax) employs the open-source Qwen3.8-Flash-Next to plan keyframes, calls SenseNova-U1.5-8B-MoT to synthesize images, and adaptively selects first-frame or first-and-last-frame conditioning based on shot content.

As shown below, compared to direct text-to-audio-video generation (LTX T2AV), the fully open-source LTX (TI2AV) and ViMax pipelines improve the Overall Score by +3.46 and +6.27 points, respectively (with ViMax outperforming TI2AV by +2.81 points), confirming that modular and agentic pipelines built entirely from open-source components effectively improve performance.

ConfigurationPlanningImage GenerationVideo GenerationOverall95% confidence interval
LTX (T2AV)——LTX-2.3 (open-source)64.40[63.23, 65.59]
LTX (TI2AV)—SenseNova-U1.5-8B-MoT (open-source)LTX-2.3 (open-source)67.86[66.66, 69.08]
ViMaxQwen3.8-Flash-Next (open-source)SenseNova-U1.5-8B-MoT (open-source)LTX-2.3 (open-source)70.67[69.52, 71.85]
  1. More evidence for Finding 2.

We clarify that the lower performance of open-source models in Finding 2 refers to group-average performance rather than to individual top outliers. Following the grouping in Table 2, we compute the group means across the 6 closed-source and 11 open-source systems for C-Layout, I-Layout, and Cam.. As shown in the table below, the closed-source group achieves higher averages on all three metrics, with particularly pronounced gaps in C-Layout (+24.53 points) and I-Layout (+25.00 points). We have revised the wording of Finding 2 in the paper to explicitly state this group-average comparison.

MetricClosed-Source Mean (6 systems)Open-Source Mean (11 systems)Difference (Points)
C-Layout72.8848.35+24.53
I-Layout69.7044.70+25.00
Cam.78.8871.98+6.90
AuthorsOpenReview ↗

Rebuttal by Authors (2/5)

  1. More evidence for Finding 4.

To evaluate post-hoc dubbing capable of following scripted dialogue, we integrate VoiceCraft-Dub [2], which synthesizes speech conditioned on both script text and video, and compare it against HunyuanFoley across four video backbones (LongLive, Wan2.2, ShotStream, and Helios) on 53 applicable English prompts, regenerating the audio while keeping the generated videos fixed.

As shown below, while VoiceCraft-Dub improves timbre consistency, sound attribution, and transcription accuracy, it reduces temporal alignment metrics such as Lip-Sync and AV-Sync, resulting in no clear improvement in the overall score. This suggests that, even with script-conditioned dubbing, achieving precise audio-visual synchronization in post-hoc pipelines remains challenging, further reinforcing the need for unified audio-video architectures in Finding 4.

Video BackboneDubbing ModelTimbre Consist.Sound Attr.Trans. Acc.Lip-SyncAV-SyncAudio Qual.Overall
LongLiveHunyuanFoley19.662.100.004.9687.1260.2556.52
LongLiveVoiceCraft-Dub48.3714.7237.313.4059.1668.4356.65
Wan2.2HunyuanFoley13.3939.111.369.0090.0059.0359.85
Wan2.2VoiceCraft-Dub59.9642.0337.325.5057.9768.2760.29
ShotStreamHunyuanFoley39.2442.911.0427.8589.1662.9556.70
ShotStreamVoiceCraft-Dub41.0135.7139.3810.7266.4867.8755.89
HeliosHunyuanFoley0.006.731.160.7576.9460.0648.31
HeliosVoiceCraft-Dub22.3417.7020.851.9063.4068.6249.33
4-Backbone MeanHunyuanFoley18.0722.710.8910.6485.8160.5755.35
4-Backbone MeanVoiceCraft-Dub42.9227.5433.725.3861.7568.3055.54
AuthorsOpenReview ↗

Rebuttal by Authors (3/5)

  1. Wording in Section 4.3 and reliability of short-shot metrics.

First, following your suggestion, we have replaced "the inherent difficulty of long-horizon generation" in Section 4.3 with "increased structural and cross-shot consistency demands as more shots are required" to describe the setting more accurately. Second, we validate the reliability of the three metrics on short shots as follows:

1) Camera motion (Cam.). To ensure that short shots contain enough frames for camera-motion evaluation, we re-evaluate all 16 systems by uniformly sampling up to 8 frames per shot instead of using 1 fps, while keeping all other settings fixed.

As shown below, changing the sampling strategy from 1 fps to 8 frames per shot yields a Cam. rank correlation of ρ=0.9912\rho = 0.9912 and an Overall rank correlation of ρ=0.9882\rho = 0.9882. These results indicate that system-level comparisons remain stable across the two sampling settings, supporting the reliability of 1-fps camera evaluation for short shots.

SystemCam. Score (1 fps → 8 frames)Cam. Rank (1 fps → 8 frames)
Seedance-2.080.10 → 78.866 → 6
LTX-2.3 TI2AV81.40 → 80.035 → 5
Wan2.7-T2V61.70 → 59.5712 → 12
Kling-V3-T2V84.60 → 90.284 → 4
HappyHorse-1.073.20 → 70.459 → 9
Sora-278.40 → 76.487 → 7
MoVA TI2AV68.90 → 68.6010 → 10
daVinci-MagiHuman TI2AV56.30 → 54.2114 → 14
LTX-2.3 T2AV78.10 → 75.068 → 8
Wan2.2 + HunyuanFoley95.70 → 96.341 → 3
daVinci-MagiHuman T2AV47.20 → 42.5216 → 16
ShotStream + HunyuanFoley58.10 → 56.5813 → 13
LongLive + HunyuanFoley95.60 → 98.622 → 1
JavisDiT++53.70 → 52.2215 → 15
Helios + HunyuanFoley94.40 → 96.503 → 2
JavisGPT62.40 → 62.5011 → 11

2) Speaker timbre consistency. We clarify two implementation details that prevent short utterances from being unfairly penalized. (i) The 1-second threshold is applied to the cumulative speech duration of the same speaker within a shot. When a speaker has multiple short utterances of less than 1 second within the same shot, these utterances are first concatenated before checking the duration threshold and performing evaluation. Thus, multiple short utterances may still contribute to timbre comparisons. (ii) In the actual evaluation, missing utterances are handled in two different ways: 0 and None. Timbre consistency is assigned 0 when the prompt requires speech but no speech is detected anywhere in the video, and this score contributes to the Overall score. If speech is detected but no valid cross-shot comparison of the same speaker is possible, timbre consistency is assigned None and is excluded from both the metric mean and the Overall aggregation.

3) Music consistency. To evaluate whether short audio segments lead to unstable beat/BPM estimation, we extend the audio of shots lasting $\leq$2 seconds by concatenating the complete audio of adjacent shots in the same video until the combined duration reaches at least 4 seconds. We use this extended context for beat detection, retaining only the beats within the original shot boundaries to compute cross-shot rhythm consistency. Shots longer than 2 seconds remain unchanged.

As shown below, across all 16 systems, extending the audio context to at least 4 seconds yields a music-ranking correlation of ρ=0.9934\rho = 0.9934 relative to the original setting. The Overall rankings remain unchanged (ρ=1.0000\rho = 1.0000). These results demonstrate the stability of Music Consistency evaluation on short shots.

SystemMusic Score (Original → ≥4s)Music Rank (Original → ≥4s)Overall Rank (Original → ≥4s)
Seedance-2.084.90 → 85.516 → 71 → 1
LTX-2.3 TI2AV76.70 → 76.7815 → 152 → 2
Wan2.7-T2V88.00 → 89.744 → 43 → 3
Kling-V3-T2V89.20 → 90.403 → 34 → 4
HappyHorse-1.083.30 → 82.788 → 95 → 5
Sora-283.40 → 86.187 → 66 → 6
MoVA TI2AV80.10 → 80.3112 → 127 → 7
daVinci-MagiHuman TI2AV95.70 → 95.702 → 28 → 8
LTX-2.3 T2AV77.00 → 77.1314 → 149 → 9
Wan2.2 + HunyuanFoley81.40 → 81.7410 → 1010 → 10
daVinci-MagiHuman T2AV95.80 → 95.801 → 111 → 11
ShotStream + HunyuanFoley86.20 → 86.205 → 512 → 12
LongLive + HunyuanFoley83.00 → 82.969 → 813 → 13
JavisDiT++81.40 → 81.4910 → 1114 → 14
Helios + HunyuanFoley47.50 → 49.0416 → 1615 → 15
JavisGPT79.20 → 79.1413 → 1316 → 16
Reviewer_WWhpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MSAVBench evaluates multi-shot audio-video generation using 286 scripts spanning 2,198 shots, with up to 15 shots per script. It includes multilingual dialogue, camera instructions and reference conditioning. Its evaluator combines expert models, rubric-based judgments and perception tools. A VLM corrects shot boundaries before scoring, and the aggregate score penalises incomplete shot coverage.

The evaluation of 20 systems finds persistent weaknesses in cinematic control and audio-visual coordination. Storyboard and agentic pipelines help open-source models, while post-hoc dubbing leaves substantial dialogue errors. The overall score has a reported Spearman correlation of 0.915 with human rankings.

Critical strengths

The benchmark fills a useful gap between single-shot audio-video evaluation and multi-shot visual evaluation. Comparing native AV, storyboard and dubbing pipelines reveals practical failure modes, particularly in dialogue and cross-shot control. The evaluator is also reasonably well supported: corruption tests check sensitivity to mismatches, human studies compare scoring designs, and another VLM backbone is tested. The improvement in shot-count accuracy from 55% to 78% gives concrete evidence for the preprocessing step.

Critical weaknesses

I would be cautious about interpreting the 0.915 correlation as reliable discrimination between nearby systems. It is computed over 16 system rankings, not individual videos, and the metric-specific studies involve even fewer models. Without confidence intervals or annotator agreement, the small differences between aggregation schemes and adjacent leaderboard entries are difficult to assess. It is also unclear whether the aggregation and completion penalty were selected against the same human ranking used for validation. If so, a held-out check would be needed to estimate alignment on new systems.

The remaining concern is stability and reproducibility. Proprietary judges can change over time, and several judges come from vendors represented among the generators. That does not establish bias, but an audio-judge swap would be reassuring, particularly for the conclusions about Veo. With 286 prompts spread over languages, shot counts and reference settings, subgroup sample counts also matter. Releasing cached outputs would help others reproduce the results without relying on changing APIs. Concurrent CutCraft uses a similar hybrid framework; its editing focus differs, so I would not regard the overlap alone as grounds for rejection, but the relationship and the "first benchmark" claim should be clarified.

Questions for authors

Can you report confidence intervals for scores and human correlations, together with annotator agreement? Was aggregation selected using the validation ranking, and is there a held-out check? These details would most affect my confidence in the leaderboard.

Do the conclusions remain stable under an audio judge from another family? Please give sample counts for the harder subgroups and clarify which generated outputs, judge responses and cached tool results will be released.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

We sincerely thank you for recognizing that MSAVBench "fills a useful gap between single-shot audio-video evaluation and multi-shot visual evaluation" and that comparing generation pipelines "reveals practical failure modes". We also appreciate your recognition that the evaluator is "reasonably well supported" and that the improvement in shot-count accuracy "gives concrete evidence for the preprocessing step". Below, we address each of your questions and hope to clarify any concerns.


[W1 & Q1] Statistical reliability of the evaluation.

We sincerely thank the reviewer for the invaluable feedback. We address this concern in four parts below:

  1. Confidence intervals for overall scores.

Following prior work on uncertainty in model evaluation, including HELM [1] and Chatbot Arena [2], we compute 95% confidence intervals (CIs) using paired bootstrap resampling. Specifically, we conduct 5,000 bootstrap trials based on the evaluation results of the generated videos. In each trial, we sample 286 prompts with replacement from the full benchmark set of 286 prompts. To ensure a fair comparison, all methods are evaluated on the same resampled prompt set in each trial. We then retrieve the corresponding video-level scores for each method and recompute the Overall Score using the original aggregation protocol.

We report the 2.5th and 97.5th percentiles as the 95% CIs. The results, shown in the table below, indicate that our benchmark rankings are statistically robust and consistent with our main findings.

SystemOverall Score95% CI
Seedance-2.076.06[75.00, 77.13]
LTX-2.3 TI2AV72.64[71.36, 73.93]
Wan2.7-T2V72.41[71.24, 73.51]
Kling-V3-T2V72.37[71.19, 73.55]
HappyHorse-1.072.00[70.82, 73.20]
Sora-271.26[70.14, 72.42]
MoVA TI2AV70.24[68.90, 73.61]
daVinci-MagiHuman TI2AV65.00[63.61, 66.45]
LTX-2.3 T2AV64.43[63.27, 65.61]
Wan2.2 + HunyuanFoley63.42[61.57, 65.12]
daVinci-MagiHuman T2AV60.64[59.46, 61.79]
ShotStream + HunyuanFoley58.86[57.60, 60.16]
LongLive + HunyuanFoley58.59[56.32, 61.28]
JavisDiT++58.12[57.12, 59.13]
Helios + HunyuanFoley54.11[51.02, 56.81]
JavisGPT53.95[52.69, 55.17]
  1. Confidence interval for the agreement between automatic and human rankings.

We use the same 5,000 bootstrap samples to assess the stability of the ranking agreement. In each trial, we rank the 16 systems according to their recomputed Overall Scores and compute Spearman's ρ\rho with respect to the fixed human ranking. We then take the 2.5th and 97.5th percentiles of the resulting 5,000 correlation coefficients to obtain a 95% confidence interval of [0.8882, 0.9794]. This indicates that our automatic evaluation maintains a good correlation with human ranking under prompt resampling.

  1. Inter-annotator agreement.

We further reanalyze the original 1,200 judgments collected from 30 annotators, each of whom evaluates 40 video pairs. Each pair is independently assessed by two annotators. For each judgment, annotators select one of three options: “A is better,” “B is better,” or “tie.” We count two judgments as agreeing when the two annotators select the same option.

Across all judgments, annotators achieve an overall agreement of 92.67%. We also report several model pairs in the table below, for which pair-level agreement ranges from 80.00% to 100.00%. These results indicate the high reliability of our human evaluation.

Model pairAnnotator agreement
Seedance–Wan2.7100.00%
Seedance–Kling87.50%
Seedance–HappyHorse1.085.71%
Wan2.7–Kling88.89%
Wan2.7–HappyHorse1.080.00%
Kling–HappyHorse1.0100.00%
AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

  1. Clarification on aggregation selection and held-out check.

We first clarify that the Overall aggregation rule and completion penalty are selected using the same 16-system human ranking used for validation in the paper.

To estimate alignment on new systems, we further conduct a held-out check by fixing these rules and evaluating five additional configurations: Veo-3.1-Quality, OVI TI2AV, OVI T2AV, and two new open-source variants (LTX-2.3 TI2AV and ViMax). Compared to the original versions of LTX-2.3 TI2AV and ViMax in the paper, these two variants replace all underlying modules with open-source models: both adopt SenseNova-U1.5-8B-MoT [3] for image generation and LTX-2.3 for video generation, with ViMax additionally using Qwen3.8-Flash-Next as the central agent.

As shown below, across all five held-out configurations, the automatic and human rankings are fully consistent (Spearman’s ρ=1.00\rho = 1.00), confirming strong alignment with human judgments on new systems.

ConfigurationAutomatic ScoreAutomatic RankHuman Rank
Veo-3.1-Quality72.7811
ViMax (open-source components)70.6822
LTX-2.3 TI2AV (open-source components)67.8933
OVI TI2AV39.0644
OVI T2AV33.9155

[W2 & Q2] Evaluation stability, subgroup coverage, and reproducibility.

We thank the reviewer for these constructive suggestions and address each point below.

  1. Clarification on audio judges and cross-family swap experiments.

We first clarify that the audio metrics in the main table are computed by expert models from different sources. Specifically, speaker timbre consistency is evaluated with w2v-BERT, ASR transcription with FireRed / Whisper-large-v3, and music consistency with MuQ. Gemini is used to classify reference assets during dataset construction and is not involved in scoring.

To further assess stability across different judge models, we replace the timbre, ASR, and music judges with ECAPA [4], Qwen3-ASR-1.7B [5], and MERT [6], respectively, and rescore six representative configurations, including Seedance, Veo, Wan2.7, Sora, LTX (T2AV), and LTX (TI2AV), while keeping the generated videos, shot boundaries, and scoring rules fixed.

As shown below, the ranking for timbre consistency remains unchanged (ρ=1.000\rho = 1.000). For ASR transcription, LTX (TI2AV) and LTX (T2AV) exchange the 2nd and 3rd positions (ρ=0.943\rho = 0.943). For music consistency, only Veo and Seedance exchange the 2nd and 3rd positions (ρ=0.943\rho = 0.943). These high correlations (0.943–1.0000.943\text{--}1.000) indicate that our conclusions, including those regarding Veo, remain stable under cross-family audio judges.

MetricOriginal →\rightarrow Swapped JudgeSystem Ranking (Highest to Lowest)Spearman’s ρ\rho
Timbre Consistencyw2v-BERT →\rightarrow ECAPAUnchanged: Wan2.7 > Seedance > LTX (T2AV) > LTX (TI2AV) > Sora > Veo1.000
ASR TranscriptionFireRed / Whisper →\rightarrow Qwen3-ASR-1.7BBefore: Wan2.7 > LTX (TI2AV) > LTX (T2AV) > Seedance > Sora > Veo<br>After: Wan2.7 > LTX (T2AV) > LTX (TI2AV) > Seedance > Sora > Veo0.943
Music ConsistencyMuQ →\rightarrow MERTBefore: Wan2.7 > Veo > Seedance > Sora > LTX (T2AV) > LTX (TI2AV)<br>After: Wan2.7 > Seedance > Veo > Sora > LTX (T2AV) > LTX (TI2AV)0.943
AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

  1. Subgroup sample counts.

Building on the distribution analysis in Figure 2 of the paper, we further provide the exact prompt counts for languages, shot counts, and reference settings below. Across the 286 prompts, our benchmark covers six languages, scripts with 2 to 15 shots, and five distinct reference combinations, ensuring comprehensive coverage across diverse and challenging evaluation subgroups.

DimensionSubgroups and Exact Prompt Counts
LanguageChinese (165), English (64), Japanese (15), Korean (15), Spanish (14), French (13)
Shot Count2–4 shots (38), 5–10 shots (202), 11–15 shots (46)
Reference Setting (96 prompts total)Subject image + audio (57), Scene only (26), Subject image + audio + scene (6), Subject image only (5), Audio only (2)
  1. Release of generated outputs and evaluation caches.

Representative generated audio-video outputs are available in our anonymous video gallery. Upon acceptance, we commit to fully releasing: (i) all generated audio-video outputs across evaluated systems, (ii) complete judge responses (including both raw scores and rationales), and (iii) cached tool results from all expert evaluators, alongside the benchmark prompts and evaluation scripts.

  1. Clarification on the "first benchmark" claim and relation to CutCraft.

We have revised the "first benchmark" claim in the paper and added a dedicated discussion comparing our MSAVBench with the concurrent work CutCraft [7].

While both adopt hybrid evaluation frameworks, they differ fundamentally in three aspects: (i) Task scope: CutCraft targets the specialized task of executing professional video editing, whereas MSAVBench evaluates general-purpose multi-shot audio-video generation; (ii) Evaluation dimensions: CutCraft centers on editing-specific metrics (e.g., editing rhythm, montage, and transitions), while MSAVBench systematically assesses audiovisual quality, audio-visual synchronization, cross-shot consistency, and reference fidelity; and (iii) Data coverage and difficulty: MSAVBench is constructed along four dimensions (video, audio, shot, and reference), spans six languages, and features complex multi-subject layouts as well as a mix of realistic and stylized scenes. Notably, our scripts contain more shots (2–15 shots, averaging 7.7 vs. CutCraft's 2–6 shots, averaging ~4.2) and uniquely include 96 image/audio reference-conditioned prompts to evaluate subject and voice fidelity, an external-reference evaluation setting that is absent in CutCraft.


Finally, we sincerely thank you again for your time and constructive feedback, which have strengthened the clarity, reliability, and stability of our benchmark. In the revised manuscript (with all updates highlighted in blue), we have incorporated the statistical analyses and audio-judge swap experiments into Section 4.4 and Appendix Sections F.8–F.9 (Tables 9–10 and 20–23), detailed the subgroup sample counts in Appendix Section B.3.1 (Table 11), added the discussion on CutCraft in Section 2 and Appendix A. We hope our responses can address your concerns, and we would be glad to address any further questions.


References

[1] Percy Liang et al. Holistic Evaluation of Language Models. TMLR, 2023.

[2] Wei-Lin Chiang et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. ICML, 2024.

[3] Haiwen Diao et al. SenseNova-U1.5: Towards Native Unified Visual Intelligence. arXiv:2609.11929, 2026.

[4] Brecht Desplanques et al. ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. Interspeech, 2020.

[5] Xian Shi et al. Qwen3-ASR Technical Report. arXiv:2601.21337, 2026.

[6] Yizhi Li et al. MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training. ICLR, 2024.

[7] Tianyi Zeng et al. Beyond Coherence: Benchmarking Professional Editing-Technique Execution in Multi-Shot Audio-Video Generation. arXiv:2609.08275, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the Area Chair and all three reviewers for their time and constructive feedback. We are greatly encouraged that all reviewers recognize MSAVBench as addressing a "timely and practical setting" (KtUq), "filling a useful gap" in existing evaluation (WWhp, KtUq), and serving as "a valuable contribution to the community" (Uwyd). The reviewers also appreciate our "carefully designed" data construction with "domain experts" (Uwyd) and "broad coverage" (KtUq), our "thoughtful design choices" supported by "evaluator validation and human preference alignment" (KtUq, WWhp), and our comprehensive evaluation of "17 systems across 4 paradigms" (KtUq) that reveals "clear, practically relevant failure modes" (WWhp).

To thoroughly address all reviewer questions and concerns, we have conducted seven sets of new experiments and analyses during the rebuttal:

  1. Statistical reliability (WWhp). We compute 95% CIs via 5,000 paired bootstrap resamples for all Overall scores and human-automatic correlations, confirm a 92.67% inter-annotator agreement across the 1,200 human judgments, and conduct a held-out check on five new configurations that achieves perfect human-automatic rank agreement (ρ=1.0000\rho = 1.0000), demonstrating the strong statistical reliability of our evaluation system.

  2. Evaluator robustness (WWhp). We replace our timbre, ASR, and music judges with new judge models across six systems, which yields identical rankings for timbre (ρ=1.000\rho = 1.000) and near-identical rankings for ASR and music (ρ=0.943\rho = 0.943, one adjacent swap each), confirming our evaluator's robustness to specific tool choices.

  3. Further evidence for Finding 1 (Uwyd). We re-evaluate LTX-2.3 (TI2AV) and ViMax across all 286 prompts using purely open-source components, improving Overall scores over LTX-2.3 (T2AV) by +3.45 and +6.24 points and further validating the potential of modular and agentic open-source pipelines in Finding 1.

  4. Further evidence for Finding 4 (Uwyd). We evaluate four video backbones with a new dubbing model that supports scripted dialogue generation, finding that it improves speech-related metrics but degrades audio-visual synchronization. This highlights the synchronization bottleneck in post-hoc dubbing pipelines and reinforces the conclusion in Finding 4.

  5. Short-shot metric stability (Uwyd). We re-evaluate all 16 systems by sampling up to 8 frames per shot instead of 1 fps, which yields high rank correlations for Cam. (ρ=0.9912\rho = 0.9912) and Overall (ρ=0.9882\rho = 0.9882). We also extend short-shot audio ($\leq$2s) to at least 4 seconds for beat detection, which yields a music-ranking correlation of ρ=0.9934\rho = 0.9934 and unchanged Overall rankings (ρ=1.0000\rho = 1.0000). These results confirm that our metrics remain reliable on short shots.

  6. Resolving ethics concerns via synthetic references (Uwyd). We resolve all copyright and identity/voice concerns by replacing the original references with 138 images and 83 audio clips synthesized strictly from text descriptions across all 96 reference prompts, and re-evaluate the models to confirm that the original conclusions are preserved.

  7. Reporting evaluation time and cost (KtUq). We measure and report the average per-video evaluation runtime across shot-count groups as well as the commercial API judge cost, while clarifying that all evaluators support local deployment with zero API fees.

Clarifications and Supplementary Materials. We also clarify the Overall aggregation rules, the implementation details of speaker timbre consistency, the scoring rules for missing content and inapplicable metrics, and the handling of missing and extra shots. The updated supplementary material provides the complete evaluation codebase and synthetic reference set (references/). Playable outputs are available in our anonymous video gallery. We commit to releasing all generated videos, judge responses, and tool caches upon acceptance.

Manuscript Revisions. We incorporate the new experiments into Sections 4.2 and 4.4, clarify and refine the wording of Findings 1, 2, and 4, and update the shot-count discussion in Section 4.3. We also revise the "first benchmark" claim and the discussion of CutCraft, update subgroup statistics and reference asset descriptions in Appendix B, clarify scoring rules and implementation details in Appendices C–D, and revise the licensing statement in Appendix G. All revisions are highlighted in blue.

Finally, we sincerely thank the Area Chair and reviewers again for their time and consideration. We hope these additional experiments and clarifications address the raised concerns, and we welcome any further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.