acceptodds
Under review as a conference paper at ICLR 2027

V2SA-Bench: A Source-Aware Benchmark for Video-to-Spatial Audio Generation

Abstract

Rapid growth in augmented and virtual reality applications has increased the focus on video-to-spatial audio generation, which conveys immersive cues about objects and their positions in space. However, evaluation efforts remain mono-centric and operate at the global-scene level rather than on individual sources. We introduce V2SA-Bench, a source-aware benchmark for video-to-spatial audio models, built on a human-annotated dataset of 1,274 videos across perspective and 360° videos, focusing on stereo and binaural audio and extending to first-order ambisonics. We define five evaluation dimensions, i.e., semantics, temporal, spatial, acoustics, and perceptual quality, with 12 fine-grained, reference-free, human-aligned metrics. Across 13 baselines, we find that models are strongly center-biased, that spatial alignment is the largest bottleneck, and that performance drops from single to multiple sources. We further find that verifying source presence is essential and that higher perceptual quality does not imply better audio-visual alignment. We hope V2SA-Bench and its interpretable, source-aware metrics offer actionable directions for improving spatial audio generation. We will release our code and benchmark to facilitate future research. The demo webpage is included in the supplementary material.

Primary area:
datasets and benchmarks
Keywords:
Spatial audio benchmarking, video to spatial audio generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Ex52OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce V2SA-Bench, a source-aware benchmark for video-to-spatial audio models, built on a human-annotated dataset of 1,274 videos across perspective and 360° videos, focusing on stereo and binaural audio and extending to first-order ambisonics.

Critical strengths
  1. The figures are clear.
  2. The paper is well organized.
  3. The benchmark is import and the problem is indeed important.
Critical weaknesses
  1. The writing needs further improvement. For example, Lines 110 and 112 convey nearly identical information about the dataset and benchmark. Since the paper does not involve training, I find this distinction unnecessary.

  2. The paper lacks a comparison table with existing benchmarks in terms of scale, characteristics, and evaluation capabilities. I believe such a comparison is essential for a benchmark-focused paper.

  3. I find the reported insight that “current models are strongly center-biased with spatial alignment as the key bottleneck” rather unsurprising. This seems to be a well-known limitation rather than a novel finding.

  4. I am confused by the “high qual” and “low qual” labels in Figure 2. Their meanings and distinctions should be better explained.

  5. The paper repeatedly mentions Qwen3-Omni without specifying the exact model variant or size. This information is essential for reproducibility.

  6. The paper lacks a user study examining the alignment between V2SA-Bench and human preferences. Reporting correlation coefficients between benchmark scores and human judgments would be important for validating the reliability of the proposed benchmark.

Questions for authors

See above.

Reviewer_ZMghOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper introduces V2SA-Bench, a source-level evaluation benchmark for spatial audio generated from video. It comprises 1,274 video clips and 12 metrics spanning five dimensions. The evaluation process first verifies the presence of target sound sources, then assesses the alignment of their temporal, spatial, and acoustic properties with the corresponding video content.

Critical strengths

The problem holds practical value. Source-level evaluation can reveal semantic and spatiotemporal misalignments that are often overlooked by global audio metrics—a capability particularly useful in multi-source scenarios.

The task coverage is comprehensive. The benchmark encompasses both standard-view and 360° video, as well as stereo, binaural, and FOA (First-Order Ambisonics) audio, while offering a variety of fine-grained task categorizations.

The experiments provide diagnostic insights. Semantic perturbation experiments underscore the necessity of verifying the presence of sound sources, while baseline comparisons highlight the distinctions between audio quality and audiovisual alignment. (Figure 4A; Table 3)

Critical weaknesses

The technical innovation is relatively limited. The framework primarily integrates existing tools for semantic recognition, source separation, synchronization, and localization, with several metrics formulated as combinations of semantic gating and established evaluation measures. The current contribution focuses largely on dataset construction and the integration of evaluation methods; it fails to clearly articulate the novelty of the proposed metrics—relative to existing source-level or spatial audio evaluation approaches—or the distinct value provided by these specific designs.

The reliability of the metrics is insufficiently validated. The analysis of human alignment for each dimension is based on aggregated data points from only three models, which is inadequate to substantiate broad claims regarding human alignment. Furthermore, errors stemming from source separation and other evaluators have not been systematically isolated; in particular, there is a lack of verification concerning the fidelity of spatial cues before and after the separation process. (Figure 5; Appendix A.1)

Some conclusions are constrained by the definitions of the metrics. The "multi-source spatial score" requires both sound sources to pass a semantic check; consequently, a decline in this score cannot be attributed solely to a reduction in localization capability. Moreover, the classification of left/right regions and the use of relative ranking are overly coarse, failing to fully capture the precision of spatial layout. It is recommended to include results conditioned on semantic accuracy, as well as continuous localization error metrics. (Equations 10–14)

[The author's response addressed my questions to some extent.]

Questions for authors

N/A

Reviewer_i696OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes V2SA-Bench, a video to spatial audio generation benchmarks that resolves several issues of existing benchmarks by proposing a high quality datasets that's manually verified, reference-free metrics in fine-grained categories. The evaluation show that current models are strongly center-biased.

Critical strengths
  1. Evaluating individual sources is useful. A scene-level score can miss a missing instrument or a sound placed on the wrong side. This benchmark makes these errors easier to identify.

  2. The source-presence check is well motivated. The controlled experiments show that a wrong sound can still receive good timing and spatial scores. Checking whether the correct sound exists addresses this problem.

Critical weaknesses
  1. This paper is mainly an ensemble to existing models and datasets, therefore the novelty is harder to spot.
  2. The reliability of the evaluation pipeline is not established well enough. Varies models are used for for separation, recognition, synchronization, and localization and their errors can affect the final scores. For example, ground-truth audio scores only 77.7 on multi-source semantics and 64.2 on transient synchronization, which is a low ceiling and indicate issues in the pipeline.
  3. The human study is quite limited to support the claim of "human alignment". It covers only three models, with 5–10 participant sessions per dimension. Agreement among a few models does not support reliable judgments on individual clips or models with similar quality.
Questions for authors

No

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.