- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MSAVBench evaluates multi-shot audio-video generation with 286 expert-curated prompts, 2,198 shots, and 20 metrics spanning video, audio, shot, and reference conditions. Its adaptive hybrid evaluation framework combines shot self-correction, instance-wise rubrics, expert models, and tool-grounded scoring. Across 20 systems, it identifies weaknesses in cinematic control and audio-visual alignment. Its overall score correlates with expert system rankings at Spearman .
Critical strengths
- The curated prompts cover up to 15 shots, non-realistic scenarios, multilingual audio, cinematic controls, and reference conditioning, addressing a real gap in existing benchmarks.
- Shot self-correction improves exact shot-count accuracy from 55% to 78% on 200 annotated videos. Instance-wise rubrics and tool-grounded scoring also improve human agreement on complex metrics.
- The paper reports 1,200 blinded expert pairwise judgments for overall ranking and targeted validation of audio-visual metrics, alongside results from 20 systems.
Critical weaknesses
- The treatment of absent speech and failed speaker or subject detection in the reported 11-dimension aggregate is unclear. The shot-completion coefficient is capped at one, so excess shots do not incur a count penalty.
- The reference assets are unavailable in the submitted materials, which also contain no playable model outputs.
Questions for authors
- How are absent dialogue, failed detections, and extra or missing shots scored?
- Can the reference assets and representative playable outputs be shared for review, with per-video evaluation time and judge cost?
2 comments
Rebuttal by Authors (2/2)
[W2 & Q2] Shared reference assets, playable outputs, and evaluation time/cost.
We thank the reviewer for these constructive suggestions. Below, we clarify the availability of review materials and report the evaluation time and cost.
- Reference assets and validation.
Since the licenses of the original source datasets restrict direct redistribution, we provide a replacement set of reference assets consisting of 138 unique generated images and 83 unique generated audio clips corresponding to the original 96 reference prompts. This replacement set preserves the original task coverage and distribution, including categories, styles, languages, and reference combinations.
These new assets are generated solely from the text descriptions. Specifically, the images are generated with qwen-image-2.0-pro-2026-06-22, gpt-image-2.6-sunburst, and gemini-2.5-flash-image. The audio clips are synthesized using qwen-audio-3.1-tts-flash, cosyvoice-v3.5-flash, and gemini-3.8-flash-tts. These reference assets are included in the updated supplementary materials under references/.
To verify consistency, we rerun Wan2.7, HappyHorse-1.0, and DreamID-Omni using these new references under the exact same evaluation setup. As shown below, the relative system rankings on all three metrics remain unchanged from the original Table 5. For example, Wan2.7 and HappyHorse-1.0 both continue to outperform DreamID-Omni on subject-image and face fidelity, and the pairwise ordering among the systems also remains unchanged. These results confirm that replacing the reference assets still preserve our benchmark conclusions.
| Method | Img-DINO ↑ | Img-Face ↑ | Voice ↑ |
|---|---|---|---|
| Wan2.7 R2V | 0.213 | 0.416 | 0.698 |
| HappyHorse-1.0 R2V | 0.254 | 0.259 | - |
| DreamID-Omni | 0.115 | 0.050 | 0.639 |
- Representative playable outputs.
We also provide playable audio-video outputs through our anonymous video gallery. These samples cover the typical failure cases and audio-visual failure cases analyzed in the paper, as well as side-by-side outputs from 17 evaluated systems across 10 prompts spanning 8 genres.
- Evaluation time and judge cost.
We measure the end-to-end evaluation time for 15-second, 720p multi-shot videos on 4×A800 (80G) GPUs, and report the evaluation time across different shot-count ranges as well as the overall average in the table below. As expected, the evaluation time increases with the number of shots, with an overall average of 139 seconds (2.31 minutes) per video.
Regarding judge cost, since Qwen3.5-397B-A17B and all expert tools are open-source and support local deployment, the entire pipeline can be run locally with zero API fees. If commercial APIs are used instead, the average judge cost is US$0.073 per video, based on measured token usage and public pricing.
| Shot-Count Range | Benchmark Prompts | Mean Evaluation Time (s) |
|---|---|---|
| 1–3 shots | 20 | 64.5 |
| 4–6 shots | 85 | 95.9 |
| 7–9 shots | 100 | 142.8 |
| 10–12 shots | 65 | 191.9 |
| 13–15 shots | 16 | 219.5 |
| Overall | 286 | 138.8 |
Finally, we sincerely thank you again for your time and constructive feedback, which strengthens the clarity and reproducibility of our evaluation. In the revised manuscript, we clarify the scoring and shot-completion rules in Appendix Section C.1.6, describe the reference assets in Appendix Section B.4, and report evaluation time and API cost in Appendix Section D.2 (Table 12). All new and revised content is highlighted in blue. We hope our responses can address your concerns, and we would be glad to address any further questions.
Rebuttal by Authors (1/2)
We sincerely thank you for recognizing that our benchmark addresses a "real gap in existing benchmarks", the effectiveness of our "shot self-correction", "instance-wise rubrics", and "tool-grounded scoring", as well as our validation with "1,200 blinded expert pairwise judgments" and "results from 20 systems". Below, we address each of your questions and hope to clarify any concerns.
[W1 & Q1] Treatment of absent speech, undetected speakers/subjects, and shot-count mismatches.
We sincerely thank the reviewer for the insightful comments. We address your questions in three parts below:
We clarify that, in cases where required speech, speakers, or subjects are not detected, our evaluation system assigns either
0orNone, depending on whether the metric remains applicable. A score of0is included in the metric mean and overall score, whereasNoneis excluded from aggregation. Specifically:1) Audio-related metrics. When a prompt explicitly requires speech, timbre consistency is assigned
0if no audio speaker activity is detected. Lip-sync and sound attribution are assigned0if no speech, usable visual speaker, or valid audio-visual speaker match is found. If speech is detected but no valid cross-shot pair of the same speaker is available, timbre consistency is assignedNone.2) Subject-related metrics. For cross-shot subject consistency, only shots with detected subjects are used for feature extraction and comparison. If fewer than two valid shots remain, subject consistency is assigned
None.When the generated shot count differs from that specified in the script, the evaluator first checks for incorrect or missed cuts and adjusts the shot boundaries accordingly. It then handles missing and extra shots as follows.
1) Missing shots. If fewer valid shots are generated than required, the overall score is multiplied by a completion factor of
min(1, valid shots / required shots). For example, three valid shots for a five-shot script yield a factor of 0.6.2) Extra shots. If more shots are generated than required, the evaluator sequentially matches each script shot to the generated video shots, and skips unmatched extra shots during scoring. This design focuses on the quality of the requested shots, because users can easily keep or trim extra shots as needed. Therefore, the completion factor does not impose additional penalty for extra shots. For example, if 6 shots are generated for a 5-shot script and 5 match the script, the completion factor remains 1.0.
We further note that the evaluator includes a fault-tolerance mechanism that automatically retries requests affected by transient failures, such as timeouts or interrupted connections.