- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
StreamAV-Bench evaluates long-horizon and interactive streaming audio-video generation. It contains various expert-verified scenarios across progressive and interactive tracks. Progressive cases use a single global prompt for a 180-second rollout; interactive cases use an initial prompt followed by five updates at 30-second content-time intervals. The framework comprises metrics covering quality, alignment, continuity, efficiency, instruction adherence, temporal drift, update response, and state retention and reuse. The paper evaluates 13 native joint or cascaded systems and reports complementary strengths, long-horizon degradation, and substantial difficulties with successful updates and history reuse. The appendices add failure-adjusted response latencies, source-coverage-adjusted history scores, and controls for changing update composition.
Critical strengths
1、This paper curate high-quality evaluation cases through strict expert verification and design a unified framework with 32 dimensions for systematic assessment. It is useful for community in audio-video generation
2、The benchmark targets important capabilities that completed-clip quality metrics do not capture. Separating progressive generation from runtime updates gives a useful structure for studying instruction drift, update achievement, and preservation of unchanged state.
3、The scenario scripts, modality-specific checklists, and independent expert verification provide a more systematic foundation than unstructured long-video prompts. The evaluation also includes thoughtful safeguards and a substantial initial validation.
Critical weaknesses
1、The paper lacks illustrative examples, particularly audiovisual samples. Since its main contribution is a new benchmark, such examples are important for understanding the benchmark’s tasks and assessing its value. It is also unclear whether the benchmark and associated resources will be publicly released.
2、All ten cascaded systems use the same video-to-audio (V2A) model, HunyuanVideo-Foley, and differ only in their video generators. Consequently, differences in their generated audio primarily reflect differences in the input videos. Under this setup, the interpretation and added value of the audio evaluation metrics are unclear. Please clarify what these metrics reveal beyond the differences already attributable to video generation.
Questions for authors
Could the use of incrementally updated prompts inherently disadvantage for most of the video generation baselines? In Table A11, only the initial prompt at 0 seconds provides a complete scene description. The five subsequent prompts are short, incremental instructions that specify what should change without restating the scene. The evaluated open-source streaming T2V models, including Self-Forcing, Rolling-Forcing, Deep Forcing, and Causal-Forcing, appear to operate as conditional generators that map a text description to a video segment, using only the current text condition at each time point. They do not appear to have an explicit mechanism for interpreting each new instruction as a modification of the preceding description. This raises concerns about whether the evaluation setup is compatible with these models’ conditioning mechanisms and whether it supports a fair comparison.
Response to Reviewer Kpwh
Thank you for the constructive comments and suggestions. We address each of your concerns below.
W1.1. Audiovisual examples.
A1.1: We agree that audiovisual examples make the benchmark tasks easier to understand. We have added four full-length 180-second synchronized audio-video examples to the revised supplementary material, covering both the progressive and interactive tracks.
W1.2. Public release.
A1.2: We confirm that StreamAV-Bench will be publicly released upon acceptance, including the complete benchmark dataset, evaluation code, and associated resources.
W2. Diagnostic value of audio metrics under a shared V2A backend.
A2: Following recent audio-video benchmarks such as MSAVBench [1], we use the same V2A backend for all cascaded systems to control for the audio renderer. StreamAV-Bench evaluates complete streaming audio-video pipelines rather than video models in isolation. Under this controlled setup, the audio metrics evaluate whether each generated video provides the semantic, motion, and timing cues needed to produce instruction-aligned and synchronized audio.
Because the V2A backend is fixed, differences in the resulting audio and cross-modal performance offer a controlled assessment of how well the generated videos support downstream audio generation. They reveal whether a video supplies audio-relevant information that visual metrics do not directly assess, such as the semantics of sound-producing events, motion cues indicating when sounds should occur, and temporal structure needed for synchronization and update response. Thus, the added value of the audio metrics is that they quantify how these video-side properties affect audio instruction following, update response, alignment, and synchronization.
Q1. Fairness of incremental update prompts.
A3: The incremental update setup is compatible with these streaming models and supports a fair comparison for two reasons:
Therefore, all systems receive the same runtime updates while continuing from their own generated visual histories, matching the intended interactive setting.
[1] Wei et al., "MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation", 2026.