acceptodds
Under review as a conference paper at ICLR 2027

StreamAV-Bench: A Comprehensive Benchmark for Streaming Audio-Video Generation

Abstract

Recent advancements in generative models are pushing video generation toward unbounded streaming audio-video generation for real-time interactive worlds. However, existing benchmarks primarily evaluate completed sequences and struggle to capture streaming properties. To bridge this gap, we introduce StreamAV-Bench, the first comprehensive benchmark tailored for streaming audio-video generation. StreamAV-Bench establishes a unified evaluation framework, including the progressive track for instruction adherence and long-horizon stability, and the interactive track for interactive response and state retention and reuse. With expert-verified evaluation cases across 32 fine-grained dimensions, we conduct an extensive evaluation of 13 representative systems. Our analysis reveals that current models suffer from temporal drift in progressive generation and responsiveness bottlenecks during interactive control. Based on a comprehensive failure analysis, we share insights to advance the development of native joint audio-video streaming models.

Primary area:
datasets and benchmarks
Keywords:
streaming audio-video generation, long-horizon generation, interactive generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KpwhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StreamAV-Bench evaluates long-horizon and interactive streaming audio-video generation. It contains various expert-verified scenarios across progressive and interactive tracks. Progressive cases use a single global prompt for a 180-second rollout; interactive cases use an initial prompt followed by five updates at 30-second content-time intervals. The framework comprises metrics covering quality, alignment, continuity, efficiency, instruction adherence, temporal drift, update response, and state retention and reuse. The paper evaluates 13 native joint or cascaded systems and reports complementary strengths, long-horizon degradation, and substantial difficulties with successful updates and history reuse. The appendices add failure-adjusted response latencies, source-coverage-adjusted history scores, and controls for changing update composition.

Critical strengths

1、This paper curate high-quality evaluation cases through strict expert verification and design a unified framework with 32 dimensions for systematic assessment. It is useful for community in audio-video generation

2、The benchmark targets important capabilities that completed-clip quality metrics do not capture. Separating progressive generation from runtime updates gives a useful structure for studying instruction drift, update achievement, and preservation of unchanged state.

3、The scenario scripts, modality-specific checklists, and independent expert verification provide a more systematic foundation than unstructured long-video prompts. The evaluation also includes thoughtful safeguards and a substantial initial validation.

Critical weaknesses

1、The paper lacks illustrative examples, particularly audiovisual samples. Since its main contribution is a new benchmark, such examples are important for understanding the benchmark’s tasks and assessing its value. It is also unclear whether the benchmark and associated resources will be publicly released.

2、All ten cascaded systems use the same video-to-audio (V2A) model, HunyuanVideo-Foley, and differ only in their video generators. Consequently, differences in their generated audio primarily reflect differences in the input videos. Under this setup, the interpretation and added value of the audio evaluation metrics are unclear. Please clarify what these metrics reveal beyond the differences already attributable to video generation.

Questions for authors

Could the use of incrementally updated prompts inherently disadvantage for most of the video generation baselines? In Table A11, only the initial prompt at 0 seconds provides a complete scene description. The five subsequent prompts are short, incremental instructions that specify what should change without restating the scene. The evaluated open-source streaming T2V models, including Self-Forcing, Rolling-Forcing, Deep Forcing, and Causal-Forcing, appear to operate as conditional generators that map a text description to a video segment, using only the current text condition at each time point. They do not appear to have an explicit mechanism for interpreting each new instruction as a modification of the preceding description. This raises concerns about whether the evaluation setup is compatible with these models’ conditioning mechanisms and whether it supports a fair comparison.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Kpwh

Thank you for the constructive comments and suggestions. We address each of your concerns below.

W1.1. Audiovisual examples.

A1.1: We agree that audiovisual examples make the benchmark tasks easier to understand. We have added four full-length 180-second synchronized audio-video examples to the revised supplementary material, covering both the progressive and interactive tracks.

W1.2. Public release.

A1.2: We confirm that StreamAV-Bench will be publicly released upon acceptance, including the complete benchmark dataset, evaluation code, and associated resources.

W2. Diagnostic value of audio metrics under a shared V2A backend.

A2: Following recent audio-video benchmarks such as MSAVBench [1], we use the same V2A backend for all cascaded systems to control for the audio renderer. StreamAV-Bench evaluates complete streaming audio-video pipelines rather than video models in isolation. Under this controlled setup, the audio metrics evaluate whether each generated video provides the semantic, motion, and timing cues needed to produce instruction-aligned and synchronized audio.

Because the V2A backend is fixed, differences in the resulting audio and cross-modal performance offer a controlled assessment of how well the generated videos support downstream audio generation. They reveal whether a video supplies audio-relevant information that visual metrics do not directly assess, such as the semantics of sound-producing events, motion cues indicating when sounds should occur, and temporal structure needed for synchronization and update response. Thus, the added value of the audio metrics is that they quantify how these video-side properties affect audio instruction following, update response, alignment, and synchronization.

Q1. Fairness of incremental update prompts.

A3: The incremental update setup is compatible with these streaming models and supports a fair comparison for two reasons:

  1. Conditioning on visual history: These models do not generate each 30-second segment from the short update prompt alone. During continuous rollout, previously generated frames are preserved across each boundary and generation continues from the existing visual history when the text condition switches to the active update. Because the visual history already encodes the scene, each segment is conditioned on both the preceding visual context and the new instruction.
  2. Realistic interactive evaluation: In interactive streaming, users give concise runtime updates rather than repeating the full scene description at every step. Restating the full scene at each boundary would re-supply unchanged states in text, turning the task into repeated scene generation and preventing VSR, ASR, and HDF from measuring state retention and history reuse.

Therefore, all systems receive the same runtime updates while continuing from their own generated visual histories, matching the intended interactive setting.

[1] Wei et al., "MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation", 2026.

Reviewer_9JZHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StreamAV-Bench contains progressive and interactive tracks with 160 scenarios each. Every scenario produces a 180-second rollout; the interactive track introduces five updates at 30-second intervals after the initial prompt. Cases are constructed from human-authored scripts, LLM-generated prompts and checklists, two independent expert reviews, and third-expert adjudication. The benchmark uses 32 metrics to assess quality, instruction adherence, temporal stability, interactive response, state retention, and streaming efficiency.

Critical strengths

The benchmark asks the right questions for an emerging setting. Streaming AV systems are appearing quickly, but existing benchmarks either ignore audio or only score finished clips. The paper splits evaluation into two parts: "keep going without drifting" (progressive) and "respond to runtime updates while keeping what should stay" (interactive). That is a clean decomposition. The update taxonomy (interaction type × modality × temporal dependency) also shows which kind of update fails. Scoring the same rollout at 60/120/180 s is a cheap but useful touch. This is the main reason I lean positive: I can see people actually using this benchmark to diagnose streaming models, which existing suites don't allow.

Critical weaknesses

Reliability validation remains insufficient for fine-grained comparisons. Each metric’s human study uses 120 output pairs and ultimately tests aggregate rankings of 13 systems, without instance-level agreement, inter-annotator agreement, or confidence intervals. Strong model-level correlation therefore does not establish accurate target-achievement times at 0.5-second resolution, source-establishment decisions, or subtle state-retention judgments. Most semantic metrics share one Gemini judge, without repeated-scoring or alternative-judge checks, and small differences in Table 2 lack uncertainty estimates. Calibration is also needed for AVSync on music/ambience without a visible sound-producing event and for the ranking sensitivity of PVC’s 0.70/0.30 weighting. These directly affect the credibility of a benchmark’s conclusions.

Questions for authors

Q1: What are the audio segment lengths, available video context, and cross-segment processing in the cascaded systems? Do FPS/TTFC include audio generation? Can you report elapsed end-to-end latency from a user update to target realization in synchronized audio-video, and clarify its relationship to content time?

Q2: Could you report: (a) sample-level agreement between Gemini and the human majority label, e.g., pairwise accuracy or Kendall's τ per metric; (b) inter-annotator agreement and tie rates; (c) the judge's decoding settings and test–retest variance on a subset; (d) for PVRL/PARL, how far Gemini's predicted achievement times are from human-annotated times?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 9JZH (1 / 3)

Thank you for the insightful comments and constructive suggestions. We address each of your concerns below.

W1.1 / Q2. Human-alignment and evaluator reliability.

A1.1: On the same human-alignment subset, we report the requested fine-grained reliability analyses together with alternative-judge checks.

  • (a) Sample-level human agreement: Across all 17 metrics, Gemini 3.1 Pro achieves a mean pairwise accuracy of 81.2% against the human-majority labels. Per-metric results are reported below, including VSR, ASR, and HDF for state-retention judgments and history reuse. Separately, for the binary source-establishment decisions used within HDF, Gemini agrees with the human-majority labels on 86.7% of the instances, with an F1 score of 0.88.
  • (b) Inter-annotator agreement and tie rates: Each pair is independently labeled by three annotators. Majority vote determines the final label, and a three-way split is counted as a tie. The annotations achieve a mean Fleiss' κ\kappa of 0.70 and an overall tie rate of 11.1%.
MetricPairwise Acc. (%) ↑\uparrowFleiss' κ\kappa ↑\uparrowTie Rate (%)Test–retest SD ↓\downarrow
VQ84.20.688.30.014
AQ81.70.7310.80.019
VIF85.00.719.20.016
AIF79.20.6912.50.021
VID76.70.6714.20.017
AID80.00.6411.70.020
VUF83.30.7210.00.018
AUF85.00.708.30.016
PVC86.70.767.50.008
PAC80.80.7410.00.023
PVUAR84.20.7715.00.006
PAUAR81.70.7513.30.007
PVRL74.20.659.20.091
PARL76.70.6210.80.082
VSR82.50.6911.70.017
ASR80.00.7112.50.022
HDF78.30.6613.30.024
Mean81.20.7011.1--
  • (c) Decoding settings and test–retest variance: We query Gemini 3.1 Pro with temperature = 0 and a strict JSON output schema. Three repeated runs yield test–retest standard deviations of 0.006–0.024 across score and rate metrics and 0.082–0.091 s on latency metrics, averaged across the 13 system-level scores. The mean run-to-run ranking correlation is ρ=0.98\rho = 0.98.
  • (d) Timestamp error for PVRL and PARL: PVRL and PARL are calculated from the first timestamp at which the requested visual and audio targets are achieved after an update, respectively. The mean absolute errors between Gemini's predicted achievement times and human-annotated times are 0.85 s for PVRL and 0.75 s for PARL, providing a direct measure of target-achievement-time localization accuracy.
  • Alternative-judge checks: We re-evaluated all 13 systems on the human-alignment subset using Qwen3.5-Omni-Plus for all 17 MLLM-based metrics and GPT-5.6 Sol for the visual MLLM-based metrics. Their system rankings strongly correlate with those of Gemini 3.1 Pro, with mean Spearman correlations of ρ=0.89\rho = 0.89 and 0.920.92, respectively.

We have added these analyses to Appendix E.1 and E.2 of the revised supplementary material.

W1.2. Uncertainty estimates for Table 2.

A1.2: To quantify the uncertainty of the results in Table 2, we computed 95% confidence intervals using 1,000 scenario-level bootstrap resamples for all 32 metrics. The table below reports the mean 95% confidence-interval half-width (±\pm) across systems for each metric, providing metric-specific uncertainty estimates for interpreting inter-system differences:

MetricVAVQPQAQAVAlignAVSyncNBCFPSTTFC
Mean 95% CI (±\pm)0.0070.0260.0560.0380.0080.0350.7990.0390.057
MetricVIFAIFVIDAIDVA-DVQ-DPQ-DAQ-DSCBCAVAlign-DAVSync-D
Mean 95% CI (±\pm)0.0550.0610.0410.0540.0050.0310.0320.0380.0030.0020.0080.021
MetricVUFAUFPVCPACPVUARPAUARPVRLPARLVSRASRHDF
Mean 95% CI (±\pm)0.0550.0670.0620.0710.0340.0480.4120.3120.0520.0770.078

We have added this analysis to Appendix E.3 of the revised supplementary material.

AuthorsOpenReview ↗

Response to Reviewer 9JZH (2 / 3)

W2.1. AVSync calibration for music and ambience.

A2.1: Synchformer [1] is used as a temporal synchronization metric in recent audio-video generation benchmarks, including T2AV-Compass [2] and VABench [3]. To directly evaluate its applicability to StreamAV-Bench, we conducted a human study on 120 outputs: 60 from scenarios with visible sound-producing events and 60 from music or ambience scenarios without visible sound producers. Direction-adjusted AVSync scores correlate strongly with human synchronization judgments in both groups, with Spearman correlations of ρ=0.83\rho = 0.83 and 0.760.76, respectively. These results support its use for both types of scenarios.

W2.2. PVC weighting sensitivity.

A2.2: Following LongAV-Compass [4], PVC combines deterministic artifact detection (SalgS_{\mathrm{alg}}) with MLLM-based continuity judgment (SMLLMS_{\mathrm{MLLM}}). Changing (walg,wMLLM)(w_{\mathrm{alg}}, w_{\mathrm{MLLM}}) from the default (0.70,0.30)(0.70, 0.30) to (0.50,0.50)(0.50, 0.50), (0.60,0.40)(0.60, 0.40), and (0.80,0.20)(0.80, 0.20) yields 13-system ranking correlations of ρ=0.98\rho = 0.98, 0.990.99, and 0.970.97 with the default ranking, respectively. These results show that the system ranking is robust to the choice of weights.

We have added these analyses to Appendix E.4 of the revised supplementary material.

[1] Iashin et al., "Synchformer: Efficient Synchronization from Sparse Cues", ICASSP 2024.

[2] Cao et al., "T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation", ICML 2026.

[3] Hua et al., "VABench: A Comprehensive Benchmark for Audio-Video Generation", CVPR 2026.

[4] Liu et al., "LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AV", 2026.

AuthorsOpenReview ↗

Response to Reviewer 9JZH (3 / 3)

Q1.1. Cascaded-system audio protocol.

A3.1: Motivated by prior long-form V2A studies, including LoVA [5] and LVAS [6], StreamAV-Bench adopts a controlled window-aligned protocol for consistent comparison across all cascaded systems:

  1. Audio segment length: We use consecutive, non-overlapping 10-second windows. This produces three audio segments for each 30-second interactive interval and aligns the audio windows with the update boundaries.
  2. Video context: Each audio segment is conditioned only on its corresponding video window and the prompt active during that window, without access to subsequent video windows.
  3. Cross-segment processing: Following the official inference pipeline, each audio segment is generated from its corresponding video window and active prompt, and the segments are then concatenated in chronological order. No additional cross-segment post-processing is applied.

Q1.2. Scope of FPS and TTFC.

A3.2: In StreamAV-Bench, FPS and TTFC evaluate the intrinsic efficiency of each model's native streaming generation process. For native joint models, they include joint audio-video generation. For cascaded pipelines, they measure the streaming video generator and exclude downstream V2A synthesis, since including the shared V2A backend would introduce external overhead and obscure the efficiency of the streaming model itself. The complete pipeline cost, including V2A synthesis, is captured by the end-to-end latency reported in Q1.3. We have clarified this scope in Appendix C.4 of the revised supplementary material.

Q1.3. End-to-end response latency.

A3.3: PVRL and PARL measure the delay on the generated-content timeline from an update boundary to the first realization of the requested targets. They therefore characterize content responsiveness independently of generation speed. To complement them, we report end-to-end visual and audio response latency (E2E-VRL and E2E-ARL) for 10 systems. These metrics measure the wall-clock time from receiving an update to delivering synchronized audio-video output in which the requested target is first realized. The measurement covers the complete generation pipeline, including downstream V2A synthesis for cascaded systems. The three web-only systems are excluded because their interfaces do not provide reproducible local timing.

SystemPVRL (content time) ↓\downarrowPARL (content time) ↓\downarrowE2E-VRL (wall-clock) ↓\downarrowE2E-ARL (wall-clock) ↓\downarrow
OmniForcing13.2957.92526.16116.136
Self-Forcing14.16412.36930.28830.280
LongLive11.79213.07838.09642.230
Rolling-Forcing13.23412.82038.84541.335
Deep Forcing13.43312.65037.31239.159
MemFlow13.69813.15438.51540.531
Causal-Forcing14.03212.68030.37130.046
Helios14.04312.20755.42957.606
SWIFT10.34012.09430.50834.193
IAMFlow11.68313.00155.08463.496

The two types of metrics are complementary. PVRL and PARL measure how quickly the generated content responds, whereas E2E-VRL and E2E-ARL measure how long the user waits. For example, IAMFlow ranks second in PVRL at 11.683 s but requires 55.084 s in E2E-VRL because of its lower generation speed. We have added these results to Appendix D.3 of the revised supplementary material.

[5] Cheng et al., "LoVA: Long-form Video-to-Audio Generation", ICASSP 2025.

[6] Zhang et al., "Orchestrating Audio: Multi-Agent Framework for Long-Video Audio Synthesis", EMNLP 2025.

Reviewer_9QGpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StreamAV-Bench evaluates 13 streaming audio-video systems, showing that current models can generate about three continuous minutes but still drift over long horizons and respond to interactive updates slowly and unreliably.

Critical strengths

Coverage is broad. StreamAV-Bench spans progressive and interactive streaming, with 320 expert-verified scenarios across 8 scene domains, 5 audio domains, 5 subject categories, and 4 visual styles, and it tests 13 systems from both native joint models and cascaded pipelines. Evaluation is broad. It scores 32 dimensions, from quality and stability to interaction and efficiency, at up to 180 seconds.

Critical weaknesses

The evaluation is mainly based on Gemini 3.1 Pro to provide the metrics. Why select such a model? Is the ranking stable across multiple models, such as GPT or Qwen (Max)? All 10 cascaded systems use HunyuanVideo-Foley. How much of the audio and synchronization gap is caused by this shared audio backend rather than by the streaming video model?

Questions for authors

See the Weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer 9QGp

Thank you for the valuable comments and constructive feedback. We address each of your suggestions below.

W1.1. Choice of the primary MLLM evaluator.

A1.1: StreamAV-Bench adopts a hybrid evaluation framework that combines non-MLLM metrics with MLLM-based metrics. We select Gemini 3.1 Pro as the MLLM evaluator for three main reasons:

  1. Strong audiovisual understanding: Our evaluation requires understanding both audio and video streams and verifying fine-grained checklists. Gemini 3.1 Pro provides the strong audiovisual reasoning required for these tasks.
  2. Established use in related benchmarks: Recent audio-video generation benchmarks, including LongAV-Compass [1] and T2AV-Compass [2], also use Gemini-family models as multimodal evaluators.
  3. Strong alignment with human judgments: Across all 17 MLLM-based metrics, Gemini 3.1 Pro achieves strong Spearman correlations with human judgments (ρ=0.77\rho = 0.77–0.940.94). On the same human-alignment subset, it also achieves a higher mean correlation than Qwen3.5-Omni-Plus (ρ=0.87\rho = 0.87 vs. 0.810.81), providing direct empirical support for our choice of evaluator.

W1.2. Cross-evaluator ranking stability.

A1.2: We re-evaluated all 13 systems on the human-alignment subset using the same checklist-guided protocol and evaluation rubrics. For the Qwen comparison, we use Qwen3.5-Omni-Plus across all 17 MLLM-based metrics because the evaluation requires native audio-video input, which Qwen-Max does not currently support. We also use GPT-5.6 Sol to evaluate the visual MLLM-based metrics, as it does not support audio input. Both evaluators produce system rankings that strongly correlate with those of Gemini 3.1 Pro:

Judge PairEvaluated MetricsMean Spearman ρ\rho ↑\uparrow
Gemini 3.1 Pro vs. Qwen3.5-Omni-PlusAll 17 MLLM-based metrics0.89
Gemini 3.1 Pro vs. GPT-5.6 SolVisual MLLM-based metrics0.92

These results show that the system rankings are stable across evaluators. We have added this analysis to Appendix E.2 of the revised supplementary material.

W2. Effect of the shared V2A backend.

A2: Following established practice in recent audio-video benchmarks such as MSAVBench [3], StreamAV-Bench pairs all 10 streaming video generators with the same V2A backend, HunyuanVideo-Foley. This controls for the audio renderer and enables a fair comparison across streaming video models.

To separate the effects of the audio renderer and the upstream video model, we conducted a counterfactual re-dubbing experiment on the three native joint models across all 320 benchmark scenarios. We kept each generated video fixed, removed its original audio, and synthesized new audio using HunyuanVideo-Foley under the same cascaded protocol.

MethodAudio SettingPQ ↑\uparrowAQ ↑\uparrowAVAlign ↑\uparrowAVSync ↓\downarrow
PixVerse R1Original6.3482.8100.2340.855
Re-dubbed6.5292.6250.2920.646
HappyOysterOriginal6.8173.1570.2061.044
Re-dubbed6.6022.7160.3000.749
OmniForcingOriginal6.3512.5280.1191.423
Re-dubbed6.3212.4640.2111.261
MeanOriginal6.5052.8320.1861.107
Re-dubbed6.4842.6020.2680.885

Re-dubbing leaves the mean PQ nearly unchanged (6.505 to 6.484) while reducing the mean AQ from 2.832 to 2.602. The re-dubbed AQ is close to the cascaded-system average of 2.651, indicating that the shared V2A backend accounts for most of the observed AQ gap between the native and cascaded systems. Re-dubbing also raises AVAlign from 0.186 to 0.268 and reduces AVSync error from 1.107 to 0.885, confirming that the backend contributes to the alignment and synchronization gaps as well. Notably, the three re-dubbed systems still obtain different AVAlign and AVSync scores under the same backend. Because the V2A model conditions on the generated video, this remaining variation shows that these gaps are not determined by the backend alone but are also influenced by the input videos. These metrics can therefore reveal audio-relevant aspects of the generated videos, such as the semantics of sound-producing events, motion cues indicating when sounds should occur, and temporal structure needed for synchronization.

We have added the protocol and full paired results to Appendix D.8 of the revised supplementary material.

[1] Liu et al., "LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AV", 2026.

[2] Cao et al., "T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation", ICML 2026.

[3] Wei et al., "MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation", 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response

We sincerely thank all reviewers for their constructive feedback and for acknowledging the important capabilities (Kpwh) targeted in this emerging benchmark setting (9JZH); our broad coverage and evaluation (9QGp); high-quality evaluation cases (Kpwh) with expert verification (9QGp/Kpwh); clean progressive–interactive decomposition (9JZH); unified and systematic 32-dimensional framework (Kpwh); useful update taxonomy and multi-horizon evaluation (9JZH); thoughtful safeguards and substantial initial validation (Kpwh); and practical diagnostic value (9JZH).

Next, we provide responses to each reviewer in detail.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.