- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes interactive spatial audio generation. The authors tackle three challenging: data is scarce - the use panoramic video with FoA recordings and gaming data; latency requirement for realtime use - they use AR flow matching with consistency distillation; prediction quality - semantic, temporal and spetial consistency is ensured by a contrastive pretraining approach to learn good AV representations, and GRPO to further ensure consistency.
Critical strengths
The paper is generally complete and sound in that it construct the dataset, proposes modeling approach including repr learning and generation models, and have comprehensive eval.
Critical weaknesses
- the description of ViewSpatial is missing. Section 3 and Appendix B describe the rendering and filtering procedure, but do not mention the underlying recording sources, their actual contributions to the dataset. We need more explanation on this to assess reproducibility and generalization.
- the contribution of the contrastive learning approach is not well established. CSAVP incorporates existing semantic and spatial representations, including CLAP, DSpAST, and an audio-derived azimuth token. But, the retrieval experiment compares only the full and causal CSAVP variants against chance, without mentioning how much improvement comes from the proposed trajectory-contrastive training over the pretrained representations. Similarly, removing causal CSAVP changes external S-CLAP only from 0.702 to 0.693 in table
- Several evaluation metrics share learned components with the conditioning or reward mechanisms. The generator uses causal CSAVP features, its spatial reward uses the CSAVP audio direction branch, and C-sim evaluates outputs using the bidirectional CSAVP representation. Similarly, PE-A-Frame and audiobox-aesthetics contribute to both reward optimization and reported evaluation metrics. A dedicated human localization study would better prove the claimed spatial improvements.
- the evaluation dataset is in-domain for their model but out of domain for other models they are comparing. Matched-data fine-tuning and comparisons under equivalent conditioning would make the methodological conclusions more convincing.
Questions for authors
No questions
4 comments
Response to Reviewer tQqG [2/2]
W3: Shared evaluation components and human spatial assessment
Conditioning, reward, and evaluation clarification. We appreciate this insightful observation. Our system shares modular components to maintain architectural consistency: generation leverages causal CSAVP visual features with causal temporal attention for streaming, while C-sim employs the frozen bidirectional model with full-clip video context for offline evaluation (Section 4.2; Appendix C.2). The spatial reward computes similarities via the audio direction branch, "rather than the full audiovisual C-sim score" (Appendix C.1.1), and external metrics (Spatial-CLAP, MS-CLAP) serve as independent judges. As stated in Appendix C.2, Spatial-CLAP, distributional metrics, and MOS-Q/C do not contribute to the rewards, and MS-CLAP supplies a semantic encoder distinct from the LAION-CLAP reward model.
To further evaluate localization robustly, we trained an event-conditioned localizer on real InteractSpatial audio, achieving 93.9% L/R accuracy on held-out reference audio.
Reward trade-offs. As discussed in our paper (Section 5.5; Table 5(c)–(e)), joint reward optimization balances multiple targets:
| Configuration | CLAP ↑ | PQ ↑ | S-CLAP ↑ |
|---|---|---|---|
| Aesthetic only | 0.519 | 7.02 | 0.737 |
| Full GRPO | 0.534 | 6.89 | 0.773 |
While aesthetic-only optimization maximizes PQ, Full GRPO achieves superior semantic and spatial alignment. Combining rewards prevents reliance on a single proxy metric.
Dedicated human spatial study (MOS-S). To directly address potential concerns regarding metric coupling, we conducted a dedicated human spatial study focusing specifically on source-position and motion consistency (MOS-S) on InteractSpatial. Evaluated by 10 expert listeners, we compare our Full GRPO model against strong baselines:
| Model | NFE | MOS-S ↑ |
|---|---|---|
| Reference audio | — | 4.025 ± 0.175 |
| AudioX | 250 | 3.045 ± 0.265 |
| ThinkSound | 24 | 3.030 ± 0.250 |
| PrismAudio | 24 | 3.615 ± 0.230 |
| ISAudio Full GRPO | 4 | 3.805 ± 0.215 |
Our model achieves the highest human MOS-S among all evaluated generators while requiring only 4 steps, confirming genuine spatial improvements as perceived by human listeners.
W4: Matched training data and conditioning
For a fair comparison, we fine-tuned AudioX, ThinkSound, PrismAudio, and ViSAudio on the exact same training data. Furthermore, we evaluate an ablation of ISAudio under a video-only setting (removing listener and actor inputs). Results on InteractSpatial:
| Model | Inputs | NFE | IoU ↑ (%) | FPO ↓ (%) | S-CLAP ↑ | L/R of detected gunshots ↑ (%) |
|---|---|---|---|---|---|---|
| AudioX, fine-tuned | Video | 250 | 7.01 | 4.92 | 0.678 | 57.0 |
| ThinkSound, fine-tuned | Video | 24 | 9.20 | 12.65 | 0.719 | 56.0 |
| PrismAudio, fine-tuned | Video | 24 | 9.23 | 14.76 | 0.721 | 58.4 |
| ViSAudio, fine-tuned | Video | 25 | 8.46 | 11.34 | 0.717 | 53.7 |
| ISAudio Full GRPO | Video only | 4 | 8.72 | 18.76 | 0.737 | 54.4 |
| ISAudio Full GRPO | Video + interaction | 4 | 9.85 | 12.57 | 0.773 | 61.9 |
All baselines observe full video clips (Appendix C.1.3), whereas ISAudio operates under causal streaming constraints (using only the visual prefix available at each chunk start). Even with video-only input, no future frames, and only 4 steps, ISAudio still exceeds all fine-tuned baselines in S-CLAP (0.737 vs. at most 0.721). For reference, Video-guided ISAudio (NFE = 25, observations through the current chunk) reaches 10.36% IoU and 0.830 S-CLAP (Table 4), exceeding all fine-tuned baselines on both. Compared with baselines fine-tuned on our data, interactive Full GRPO achieves the highest event IoU, S-CLAP, and gunshot L/R accuracy, demonstrating the power of our action-conditioned causal architecture. Moreover, ISAudio accomplishes this in streaming mode with only 4 steps.
On ViewSpatial, against AudioX, ThinkSound, PrismAudio, and ViSAudio fine-tuned on our data, Video-guided ISAudio achieves the highest C-sim and S-CLAP, and Interactive ISAudio remains competitive without future frames.
We thank you again for the constructive feedback. We hope these new results address your concerns, and we would be grateful if you would take them into account in your final assessment.
Thank you again for your thoughtful review and the time you have devoted to our paper. Your comments have helped us substantially improve the manuscript. We have uploaded a revised version, with all changes highlighted in blue. The revisions related to your concerns are as follows.
W1 (ViewSpatial recording sources). Section 3 now states that ViewSpatial is built from 5,036 YouTube 360° recordings with FOA audio, and a new "Recording sources" paragraph in Appendix B.1 describes how they were collected. Table 1 now cites the compared datasets, and the Reproducibility Statement commits to releasing source IDs, timestamps, trajectories, split manifests, and the rendering, filtering, and annotation code.
W2 (Contribution of trajectory-contrastive learning). Table 2 adds a "w/o traj." variant trained without the trajectory objectives: chunk/clip retrieval accuracy drops from 32.66%/34.91% to 14.45%/16.48%, close to the 12.50% chance level (Section 5.2). Section 5.5 and Table C.9 (Appendix C.3.7) split the visual ablation by dataset: on ViewSpatial, removing causal CSAVP lowers S-CLAP from 0.582 to 0.562 and C-sim from 0.170 to 0.144.
W3 (Shared evaluation components). Table 4 adds an L/R column from an external event-conditioned localizer that uses no CSAVP or reward weights (Section 5.1; "External localizer" in Appendix C.2). Appendix C.3.6 (Table C.8) adds the blinded human spatial study (MOS-S), where ISAudio scores 3.805 versus 3.615 for the best baseline.
W4 (Matched data and equivalent conditioning). Tables 3 and 4 now include ViSAudio and all binaural baselines fine-tuned on our training data (Section 5.1; Appendix C.1.3). On ViewSpatial, Video-guided ISAudio achieves the highest C-sim (0.2488) and S-CLAP (0.621). On InteractSpatial, four-step ISAudio outperforms all fine-tuned baselines in event IoU (9.85% vs. 9.23%), S-CLAP (0.773 vs. 0.721), and L/R accuracy (61.9% vs. 58.4%) (Section 5.4). For equivalent conditioning, Table 5(b) adds the video-only four-step model, whose S-CLAP (0.737) still exceeds every fine-tuned full-clip baseline in Table 4.
We hope these revisions resolve your concerns, and we would sincerely appreciate it if you could take them into account in your final assessment.
Dear Reviewer tQqG,
Thank you again for your time and constructive feedback on our paper.
As the discussion period is approaching its end, we would like to kindly follow up on our earlier message. We have provided a detailed response to your concerns, supported by additional experiments and corresponding revisions to the paper.
We would sincerely appreciate it if you could take a moment to review our response and share any remaining concerns before the discussion period closes. We would be more than happy to provide further clarifications if needed.
Thank you very much for your time and consideration.
Best regards,
The Authors
Response to Reviewer tQqG [1/2]
We sincerely thank you for the thoughtful, detailed, and constructive review. We are greatly encouraged that the reviewer recognizes the completeness, sound modeling approach, and comprehensive evaluations of our work. Below, we address each concern and present several newly conducted experiments, ablation studies, and cross-model fine-tuning results.
W1: ViewSpatial recording sources
Section 3 describes the synchronized panoramic-video and FOA inputs, and Appendix B details their processing. To further clarify for reproducibility and generalizability, ViewSpatial consists of 5,036 YouTube 360° recordings with FOA audio, collected via our customized crawler. This collection is similar in modality to Sphere360 [1]. we will release the complete source IDs, timestamps, trajectories, split manifests, and data-processing code.
W2: Contribution of trajectory-contrastive learning
Trajectory-contrastive training ablation. To isolate the specific contribution of our trajectory-contrastive learning (Eqs. 3–4, Section 4.2), we train a CSAVP variant by removing the within-scene view-angle and temporal trajectory objectives, while keeping the pretrained initialization, data distribution, and training budget identical. Following the retrieval protocol of Section 5.2, introducing full trajectory objectives significantly boosts chunk Top-1 accuracy from 14.45% to 32.66%, clearly demonstrating the efficacy of our contrastive design beyond raw pretrained representations.
Dataset-specific S-CLAP and C-sim analysis. The pooled S-CLAP in the initial submission averaged across datasets (Table 5 caption; Appendix C.3.5). To provide a granular view, we report the ablation separately for each dataset:
As shown, removing causal CSAVP noticeably degrades spatial scores on ViewSpatial (where explicit 3D geometry is absent from video alone). On InteractSpatial, explicit listener and actor states are available, making the performance robust; thus, the pooled metrics previously masked CSAVP's strong contribution in video-driven spatial setups.
References
[1] Liu, Huadai, et al. "Omniaudio: Generating spatial audio from 360-degree video." arXiv preprint arXiv:2504.14906 (2025).