acceptodds
Under review as a conference paper at ICLR 2027

ISAudio: Interactive Spatial Audio Generation for World Models

11 views

Abstract

Spatial audio is essential for world models to simulate immersive interactions with action-responsive auditory feedback. Unlike video-to-audio synthesis from fully observed clips, this interaction loop requires streaming spatial audio that responds to listener actions and predicts upcoming sounds before the corresponding visual observations arrive. In this work, we introduce ISAudio, an action-conditioned framework for streaming binaural audio prediction. To address data scarcity, we construct spatial audiovisual data from panoramic recordings with controlled listener trajectories and first-person gameplay with aligned action signals. For efficient streaming, ISAudio combines autoregressive flow matching in a spatial latent space with causal consistency distillation, enabling few-step prediction conditioned only on observed visual prefixes, audio history, and current actions. To preserve semantic, temporal, and spatial consistency, we propose Contrastive Spatial Audio-Video Pretraining (CSAVP), which learns representations that capture changes in listener viewpoint through same-scene trajectory contrasts. We further refine the distilled generator with dual-granularity Group Relative Policy Optimization (GRPO), combining clip-level perceptual rewards with chunk-local event and spatial feedback. Experiments show that ISAudio achieves an S-CLAP score of 0.787, compared with 0.664 for the strongest baseline, and enables real-time streaming inference with a real-time factor of 0.53. Together, these results highlight interactive spatial audio generation as an important step toward richer multimodal world models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial Audio Generation, Interactive World Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tQqGOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes interactive spatial audio generation. The authors tackle three challenging: data is scarce - the use panoramic video with FoA recordings and gaming data; latency requirement for realtime use - they use AR flow matching with consistency distillation; prediction quality - semantic, temporal and spetial consistency is ensured by a contrastive pretraining approach to learn good AV representations, and GRPO to further ensure consistency.

Critical strengths

The paper is generally complete and sound in that it construct the dataset, proposes modeling approach including repr learning and generation models, and have comprehensive eval.

Critical weaknesses
  1. the description of ViewSpatial is missing. Section 3 and Appendix B describe the rendering and filtering procedure, but do not mention the underlying recording sources, their actual contributions to the dataset. We need more explanation on this to assess reproducibility and generalization.
  2. the contribution of the contrastive learning approach is not well established. CSAVP incorporates existing semantic and spatial representations, including CLAP, DSpAST, and an audio-derived azimuth token. But, the retrieval experiment compares only the full and causal CSAVP variants against chance, without mentioning how much improvement comes from the proposed trajectory-contrastive training over the pretrained representations. Similarly, removing causal CSAVP changes external S-CLAP only from 0.702 to 0.693 in table
  3. Several evaluation metrics share learned components with the conditioning or reward mechanisms. The generator uses causal CSAVP features, its spatial reward uses the CSAVP audio direction branch, and C-sim evaluates outputs using the bidirectional CSAVP representation. Similarly, PE-A-Frame and audiobox-aesthetics contribute to both reward optimization and reported evaluation metrics. A dedicated human localization study would better prove the claimed spatial improvements.
  4. the evaluation dataset is in-domain for their model but out of domain for other models they are comparing. Matched-data fine-tuning and comparisons under equivalent conditioning would make the methodological conclusions more convincing.
Questions for authors

No questions

4 comments
AuthorsOpenReview ↗

Response to Reviewer tQqG [1/2]

We sincerely thank you for the thoughtful, detailed, and constructive review. We are greatly encouraged that the reviewer recognizes the completeness, sound modeling approach, and comprehensive evaluations of our work. Below, we address each concern and present several newly conducted experiments, ablation studies, and cross-model fine-tuning results.


W1: ViewSpatial recording sources

Section 3 describes the synchronized panoramic-video and FOA inputs, and Appendix B details their processing. To further clarify for reproducibility and generalizability, ViewSpatial consists of 5,036 YouTube 360° recordings with FOA audio, collected via our customized crawler. This collection is similar in modality to Sphere360 [1]. we will release the complete source IDs, timestamps, trajectories, split manifests, and data-processing code.


W2: Contribution of trajectory-contrastive learning

Trajectory-contrastive training ablation. To isolate the specific contribution of our trajectory-contrastive learning (Eqs. 3–4, Section 4.2), we train a CSAVP variant by removing the within-scene view-angle and temporal trajectory objectives, while keeping the pretrained initialization, data distribution, and training budget identical. Following the retrieval protocol of Section 5.2, introducing full trajectory objectives significantly boosts chunk Top-1 accuracy from 14.45% to 32.66%, clearly demonstrating the efficacy of our contrastive design beyond raw pretrained representations.

VariantChunk Top-1 ↑ (%)Clip Top-1 ↑ (%)
Chance12.5012.50
No trajectory objectives14.4516.48
Full trajectory objectives32.6634.91

Dataset-specific S-CLAP and C-sim analysis. The pooled S-CLAP in the initial submission averaged across datasets (Table 5 caption; Appendix C.3.5). To provide a granular view, we report the ablation separately for each dataset:

DatasetS-CLAP: full / w/o causal CSAVP ↑Local C-sim: full / w/o causal CSAVP ↑
ViewSpatial0.582 / 0.5620.170 / 0.144
InteractSpatial0.787 / 0.7870.185 / 0.186

As shown, removing causal CSAVP noticeably degrades spatial scores on ViewSpatial (where explicit 3D geometry is absent from video alone). On InteractSpatial, explicit listener and actor states are available, making the performance robust; thus, the pooled metrics previously masked CSAVP's strong contribution in video-driven spatial setups.

References

[1] Liu, Huadai, et al. "Omniaudio: Generating spatial audio from 360-degree video." arXiv preprint arXiv:2504.14906 (2025).

AuthorsOpenReview ↗

Response to Reviewer tQqG [2/2]

W3: Shared evaluation components and human spatial assessment

Conditioning, reward, and evaluation clarification. We appreciate this insightful observation. Our system shares modular components to maintain architectural consistency: generation leverages causal CSAVP visual features with causal temporal attention for streaming, while C-sim employs the frozen bidirectional model with full-clip video context for offline evaluation (Section 4.2; Appendix C.2). The spatial reward computes similarities via the audio direction branch, "rather than the full audiovisual C-sim score" (Appendix C.1.1), and external metrics (Spatial-CLAP, MS-CLAP) serve as independent judges. As stated in Appendix C.2, Spatial-CLAP, distributional metrics, and MOS-Q/C do not contribute to the rewards, and MS-CLAP supplies a semantic encoder distinct from the LAION-CLAP reward model.

To further evaluate localization robustly, we trained an event-conditioned localizer on real InteractSpatial audio, achieving 93.9% L/R accuracy on held-out reference audio.

Reward trade-offs. As discussed in our paper (Section 5.5; Table 5(c)–(e)), joint reward optimization balances multiple targets:

ConfigurationCLAP ↑PQ ↑S-CLAP ↑
Aesthetic only0.5197.020.737
Full GRPO0.5346.890.773

While aesthetic-only optimization maximizes PQ, Full GRPO achieves superior semantic and spatial alignment. Combining rewards prevents reliance on a single proxy metric.

Dedicated human spatial study (MOS-S). To directly address potential concerns regarding metric coupling, we conducted a dedicated human spatial study focusing specifically on source-position and motion consistency (MOS-S) on InteractSpatial. Evaluated by 10 expert listeners, we compare our Full GRPO model against strong baselines:

ModelNFEMOS-S ↑
Reference audio—4.025 ± 0.175
AudioX2503.045 ± 0.265
ThinkSound243.030 ± 0.250
PrismAudio243.615 ± 0.230
ISAudio Full GRPO43.805 ± 0.215

Our model achieves the highest human MOS-S among all evaluated generators while requiring only 4 steps, confirming genuine spatial improvements as perceived by human listeners.


W4: Matched training data and conditioning

For a fair comparison, we fine-tuned AudioX, ThinkSound, PrismAudio, and ViSAudio on the exact same training data. Furthermore, we evaluate an ablation of ISAudio under a video-only setting (removing listener and actor inputs). Results on InteractSpatial:

ModelInputsNFEIoU ↑ (%)FPO ↓ (%)S-CLAP ↑L/R of detected gunshots ↑ (%)
AudioX, fine-tunedVideo2507.014.920.67857.0
ThinkSound, fine-tunedVideo249.2012.650.71956.0
PrismAudio, fine-tunedVideo249.2314.760.72158.4
ViSAudio, fine-tunedVideo258.4611.340.71753.7
ISAudio Full GRPOVideo only48.7218.760.73754.4
ISAudio Full GRPOVideo + interaction49.8512.570.77361.9

All baselines observe full video clips (Appendix C.1.3), whereas ISAudio operates under causal streaming constraints (using only the visual prefix available at each chunk start). Even with video-only input, no future frames, and only 4 steps, ISAudio still exceeds all fine-tuned baselines in S-CLAP (0.737 vs. at most 0.721). For reference, Video-guided ISAudio (NFE = 25, observations through the current chunk) reaches 10.36% IoU and 0.830 S-CLAP (Table 4), exceeding all fine-tuned baselines on both. Compared with baselines fine-tuned on our data, interactive Full GRPO achieves the highest event IoU, S-CLAP, and gunshot L/R accuracy, demonstrating the power of our action-conditioned causal architecture. Moreover, ISAudio accomplishes this in streaming mode with only 4 steps.

On ViewSpatial, against AudioX, ThinkSound, PrismAudio, and ViSAudio fine-tuned on our data, Video-guided ISAudio achieves the highest C-sim and S-CLAP, and Interactive ISAudio remains competitive without future frames.


We thank you again for the constructive feedback. We hope these new results address your concerns, and we would be grateful if you would take them into account in your final assessment.

Thank you again for your thoughtful review and the time you have devoted to our paper. Your comments have helped us substantially improve the manuscript. We have uploaded a revised version, with all changes highlighted in blue. The revisions related to your concerns are as follows.

W1 (ViewSpatial recording sources). Section 3 now states that ViewSpatial is built from 5,036 YouTube 360° recordings with FOA audio, and a new "Recording sources" paragraph in Appendix B.1 describes how they were collected. Table 1 now cites the compared datasets, and the Reproducibility Statement commits to releasing source IDs, timestamps, trajectories, split manifests, and the rendering, filtering, and annotation code.

W2 (Contribution of trajectory-contrastive learning). Table 2 adds a "w/o traj." variant trained without the trajectory objectives: chunk/clip retrieval accuracy drops from 32.66%/34.91% to 14.45%/16.48%, close to the 12.50% chance level (Section 5.2). Section 5.5 and Table C.9 (Appendix C.3.7) split the visual ablation by dataset: on ViewSpatial, removing causal CSAVP lowers S-CLAP from 0.582 to 0.562 and C-sim from 0.170 to 0.144.

W3 (Shared evaluation components). Table 4 adds an L/R column from an external event-conditioned localizer that uses no CSAVP or reward weights (Section 5.1; "External localizer" in Appendix C.2). Appendix C.3.6 (Table C.8) adds the blinded human spatial study (MOS-S), where ISAudio scores 3.805 versus 3.615 for the best baseline.

W4 (Matched data and equivalent conditioning). Tables 3 and 4 now include ViSAudio and all binaural baselines fine-tuned on our training data (Section 5.1; Appendix C.1.3). On ViewSpatial, Video-guided ISAudio achieves the highest C-sim (0.2488) and S-CLAP (0.621). On InteractSpatial, four-step ISAudio outperforms all fine-tuned baselines in event IoU (9.85% vs. 9.23%), S-CLAP (0.773 vs. 0.721), and L/R accuracy (61.9% vs. 58.4%) (Section 5.4). For equivalent conditioning, Table 5(b) adds the video-only four-step model, whose S-CLAP (0.737) still exceeds every fine-tuned full-clip baseline in Table 4.

We hope these revisions resolve your concerns, and we would sincerely appreciate it if you could take them into account in your final assessment.

Dear Reviewer tQqG,

Thank you again for your time and constructive feedback on our paper.

As the discussion period is approaching its end, we would like to kindly follow up on our earlier message. We have provided a detailed response to your concerns, supported by additional experiments and corresponding revisions to the paper.

We would sincerely appreciate it if you could take a moment to review our response and share any remaining concerns before the discussion period closes. We would be more than happy to provide further clarifications if needed.

Thank you very much for your time and consideration.

Best regards,

The Authors

Reviewer_AeUUOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ISAUDIO, a framework for action-conditioned, chunk-causal, streaming generation of binaural spatial audio for world models. The approach combines an autoregressive flow-matching generator in a binaural latent space, a contrastive spatial audio–video pretraining scheme (CSAVP) built on same-scene trajectory contrasts for viewpoint-dependent conditioning, causal consistency distillation for few-step inference, and a dual-granularity GRPO post-training that couples clip-level perceptual rewards with chunk-local event and spatial feedback. The authors also curate two datasets (ViewSpatial from FOA/panoramic captures and InteractSpatial from Counter-Strike 2 replays), and report strong improvements on spatial metrics (e.g., S-CLAP 0.787 vs. 0.664 baseline) and real-time streaming efficiency (RTF 0.53 with NFE=4).

Critical strengths

1, Technical novelty

a, Introduces a coherent pipeline for interactive spatial audio: chunk-causal flow matching with a causal conditioning mask, a causal visual encoder, and few-step causal consistency distillation adapted to audio

b, CSAVP is an interesting idea: within-scene trajectory contrasts explicitly force viewpoint-dependent AV representations, which is well aligned with spatial audio needs

c, Dual-granularity GRPO blends clip-level perceptual signals with chunk-level event/spatial rewards, a sensible design for interactive, streaming settings where local accuracy matters

2, Experimental validation

a, Comprehensive metrics across semantic, temporal, aesthetic, spatial (Local C-sim, S-CLAP), distribution, and human MOS; includes IoU/FPO for events in the interactive setting.

b, Reports both video-guided (non-causal) and interactive (causal-prefix) regimes, isolating the impact of causality and streaming.

c, Efficiency results (RTF, TTFA, service RTF) and few-step distillation are well characterized.

Critical weaknesses

1, Technical limitations

a, CSAVP’s audio encoder is used both as a conditioner and as part of evaluation/reward (Local C-sim and chunk-level spatial rewards), creating potential metric coupling and bias toward the proposed representation.

b, GRPO increases FPO in some settings; the trade-off between higher IoU and hallucinated events is not fully resolved.

2, Experimental gaps

a, Missing comparisons with the closest spatial V2A method ViSAudio (CFM-based binaural generation) and with recent interactive AV world models that already model spatial stereo (e.g., HelixWorld, AV CDiT), even if domain gaps exist. This weakens claims of state of the art.

b, According to Tables 4, 5, and C.6, distillation improves semantic and certain quality metrics, but it degrades two spatial metrics and worsens the FD score. While the GRPO-based model recovers some of the performance lost during distillation, it also leads to an increase in erroneous event activity. Furthermore, the spatial scores of the final model still fail to surpass those achieved by direct four-step SFT.

c, There is also insufficient evidence for the dual-granularity reward mechanism:

  • In Table 5, the S-CLAP score for the "Spatial-only" setting is 0.772, compared to 0.773 for the full GRPO model.

  • The IoU for the "Chunk-level Event-only" setting is 9.86, versus 9.85 for the full GRPO model.

  • The difference in IoU between the "Chunk-level Event-only" and "Clip-level Event-only" settings is marginal (9.86 vs. 9.79)—and falls within the margin of error.

  • A comprehensive controlled experiment (keeping all rewards constant while varying only the credit assignment) is missing.

3, Clarity

The abstract conflates results from different configurations. S-CLAP 0.787 is derived from a 25-step Interactive SFT, with an RTF of 1.31, while RTF 0.53 corresponds to a 4-step model; the final GRPO-based S-CLAP is 0.773. While both figures are valid in their respective contexts, presenting them side-by-side may lead readers to mistakenly believe they were achieved by a single, final model.

4, Missing related works

Empirical comparison to ViSAudio (video→binaural via CFM) and FoleySpace (trajectory-conditioned binauralization) is absent.

Questions for authors

1, How sensitive are your spatial and semantic metrics to the choice of HRTF used in FOA→binaural decoding? Have you tested cross-HRTF generalization or listener variability?

2, Can you provide results for an additional, external spatial metric that does not rely on CSAVP features (e.g., angular error vs. InteractSpatial’s directional labels), to mitigate potential evaluation coupling?

3, What is the effect of chunk length (8 latent frames ≈0.74 s) on TTFA and spatial continuity? Could smaller chunks reduce TTFA without harming spatial stability?

4, Why was ViSAudio omitted from empirical comparisons, and can you report a “video-guided” (non-causal) ISAUDIO vs. ViSAudio comparison on a shared subset?

5, How robust is the model to action/state noise or partial observability (e.g., missing off-screen actor states)? Please quantify degradation and the benefits of actor-state conditioning.

5 comments
AuthorsOpenReview ↗

Response to Reviewer AeUU [1/3]

We sincerely thank Reviewer AeUU for the detailed, constructive review and for appreciating our technical novelty and comprehensive experimental validation. Below, we address your specific concerns and questions.


W1(a) and Q2: Metric coupling and external spatial evaluation

Generation relies exclusively on causal CSAVP visual features available at the current chunk boundary. While our spatial reward and offline C-sim share learned components (the frozen bidirectional audiovisual representation), offline evaluation strictly preserves the causal constraint by never exposing future observations to generation (Section 4.2, "Causal adaptation"). Following V2A practices like PrismAudio [1], we complement these with external metrics and human evaluations: as described in Appendix C.2, Spatial-CLAP, FD/KL, and MOS do not contribute to the rewards, and MS-CLAP uses a semantic encoder distinct from the LAION-CLAP reward model.

To evaluate spatial direction without CSAVP-tuned weights, we train an event-conditioned localizer using public DSpAST weights on real InteractSpatial audio (excluding all four test matches). It receives no CSAVP or reward weights—only audio, event class, and query time. We also report human MOS-S for source-position and motion consistency (10 blinded expert listeners):

ModelL/R direction accuracy of detected gunshots ↑ (%)MOS-S ↑
Reference audio93.94.025 ± 0.175
AudioX53.73.045 ± 0.265
ThinkSound55.13.030 ± 0.250
PrismAudio52.93.615 ± 0.230
ISAudio (Full GRPO, 4 steps)61.93.805 ± 0.215

These external localizer and human evaluation results confirm that our model accurately tracks sound sources and delivers superior perceptual spatial quality beyond C-sim.


W1(b): GRPO and erroneous event activity

To assess the impact of the absent-event penalty, we compare event-only configurations with and without it (Table 5(d)):

Event-only variantIoU ↑ (%)FPO ↓ (%)
Without penalty9.9714.09
With penalty9.8611.35

The penalty reduces FPO by 2.74 percentage points with a minimal 0.11-point change in IoU, consistent with Section 5.5. Full GRPO balances multiple competing objectives, resulting in an FPO of 12.57% (compared to 9.23% for pure distillation). We will explicitly discuss this event-coverage vs. FPO trade-off in the revision. Note that FPO measures predicted activity of each event class in windows where that class is absent (Appendix C.2, Eq. C.7). Our penalty effectively curbs such phantom events.


W2(a): HelixWorld and AV-CDiT

AV-CDiT and HelixWorld focus on joint visual and auditory generation. As discussed in Section 2, AV-CDiT autoregressively predicts future visual frames, audio, and task rewards from audiovisual history and navigation actions. In contrast, ISAUDIO operates on an externally supplied visual stream, conditioning binaural generation on observed visual prefixes, listener controls, and actor states. We will clarify this task distinction and discuss HelixWorld's joint multimodal setup in the related work. A direct comparison with the video-to-binaural baseline ViSAudio is provided below.

References

[1] Liu, Huadai, et al. "Prismaudio: Decomposed chain-of-thought and multi-dimensional rewards for video-to-audio generation." International Conference on Learning Representations. Vol. 2026. 2026

AuthorsOpenReview ↗

Response to Reviewer AeUU [2/3]

W2(b): Distillation and GRPO trade-offs

To compare training stages at the identical inference cost, we report 4-step results on InteractSpatial (Tables 4 and C.6):

ModelCLAP ↑PQ ↑S-CLAP ↑FD ↓
Direct SFT0.4676.550.78317.36
Distilled0.5156.830.73124.90
Full GRPO0.5346.890.77323.73

Distillation improves semantic alignment and perceptual quality, and GRPO further strengthens these gains while partially recovering S-CLAP and FD. At the same four-step setting, Full GRPO achieves higher CLAP (0.534 vs. 0.467), event IoU (9.85% vs. 8.39%), MOS-Q (3.96 vs. 3.84), and MOS-C (3.55 vs. 3.47) than direct SFT (Table 4). On ViewSpatial, distillation also improves spatial correspondence over direct four-step SFT (C-sim 0.0934 → 0.1133, S-CLAP 0.520 → 0.539; Table C.3). These results demonstrate improvements in semantic alignment, event accuracy, and human-rated quality, accompanied by a trade-off in S-CLAP and C-sim on InteractSpatial. We will clarify this balance in the revised paper.


W2(c): Dual-granularity rewards and credit assignment

To validate the dual-granularity reward mechanism, we perform a controlled ablation keeping initialization, data, and compute budgets fixed, varying only the event credit assignment strategy:

Event credit assignmentIoU ↑ (%)FPO ↓ (%)S-CLAP ↑
Clip broadcast9.81112.2410.7681
Local (our design)9.84912.5740.7733

Chunk-local credit yields higher IoU and S-CLAP than clip broadcasting. More importantly, the dual-granularity design targets balance across objectives rather than any single metric: Full GRPO achieves the highest CLAP (0.534) with near-best PQ (6.89), S-CLAP (0.773), and IoU (9.85), whereas single-reward variants excel on one axis at the cost of others (e.g., Aesthetic-only: PQ 7.02 but S-CLAP 0.737; Spatial-only: S-CLAP 0.772 but PQ 6.82; Table 5(c)–(e)).


W3: Configurations in the abstract

We thank the reviewer for pointing out this potential confusion. We will refine the abstract to explicitly pair each performance figure with its corresponding operating regime, consistent with Table 4 and Table C.5: the 25-step Interactive SFT achieves an S-CLAP of 0.787 with an RTF of 1.31, whereas the 4-step GRPO-refined model achieves an S-CLAP of 0.773 with an optimized RTF of 0.53.


W4 and Q4: ViSAudio and FoleySpace

ViSAudio was omitted initially due to the lack of a runnable official implementation during our experiment preparation; Section 2 already discusses it as particularly close to our output space. We have now evaluated its released model alongside Video-guided ISAudio on identical ViewSpatial and InteractSpatial test windows (both at NFE = 25), using the matched 8-second evaluation protocol of Appendix C.1.3. Note that ViSAudio observes the full clip, whereas Video-guided ISAudio only observes through the current chunk.

DatasetModelCLAP ↑S-CLAP ↑
ViewSpatialViSAudio0.4850.536
ViewSpatialISAudio (Video-guided)0.4870.621
InteractSpatialViSAudio0.3270.581
InteractSpatialISAudio (Video-guided)0.5910.830

Video-guided ISAudio outperforms the released ViSAudio in both CLAP and S-CLAP on both datasets. We further fine-tune ViSAudio on our training data. Video-guided ISAudio still achieves higher C-sim (0.2488 vs. 0.2028) and S-CLAP (0.621 vs. 0.620) on ViewSpatial, and four-step Interactive ISAudio exceeds it on InteractSpatial in S-CLAP (0.773 vs. 0.717) and event IoU (9.85% vs. 8.46%). These results are included in Tables 3 and 4 of the revised paper. Regarding FoleySpace, due to resource constraints and the absence of official code/checkpoints, we will instead thoroughly discuss its trajectory-conditioned binauralization approach in the related work.

AuthorsOpenReview ↗

Response to Reviewer AeUU [3/3]

Q1: HRTF sensitivity and listener variability

As described in Appendix B.1, ViewSpatial rotates the FOA sound field along the listener trajectory before SOFA-HRTF binaural decoding. To test whether our findings depend heavily on the specific HRTF used during this conversion, we keep our Interactive SFT outputs fixed and re-render all ViewSpatial references using two alternative HRTFs:

Reference HRTFMS-CLAP ↑S-CLAP ↑
H0: SADIE-II D1 / KU100 (Default)0.4640.582
H1: SADIE-II H30.4680.597
H2: CIPIC subject 0030.4800.578

The maximum variation is only 0.016 in MS-CLAP and 0.019 in S-CLAP, demonstrating strong robustness against HRTF choices. Combined with our high human MOS-S scores, this confirms that our spatial metrics reliably reflect genuine auditory spatial perception rather than artifacts of a specific HRTF.


Q3: Chunk length, TTFA, and spatial continuity

To analyze latency and quality tradeoffs, we evaluate 4-step Full GRPO across different chunk lengths (tested on 4, 8, and 16 latents) under real-time observation streaming on an A100 GPU (latency metrics, following Appendix C.1.4), alongside paired ViewSpatial windows under 25-step Interactive SFT (quality metrics). Since training samples chunk boundaries at random latent positions (Appendix A.1), the same model can be evaluated with different inference chunk lengths:

Chunk size (latents)TTFA ↓ (s)Smooth-playback delay ↓ (s)Service RTF ↓S-CLAP ↑Excess ILD jump ↓ (dB)
41.501.550.960.577+0.054
81.181.630.650.582+0.035
160.351.910.510.606−0.021

Smaller chunks (e.g., 4 latents) do not reduce TTFA because the neural audio decoder requires a minimum lookahead context of 12 additional latents (1.11 s). Conversely, 16-latent chunks yield lower TTFA by batching more initial content, but require longer playback buffering and rely on older visual observations. Meanwhile, S-CLAP and spatial continuity improve moderately with larger chunks. Overall, our default 8-latent chunk size strikes an optimal balance between low latency, spatial stability, and generation quality.


Q5: Actor-state benefits and partial observability

To isolate the contribution of actor states, we conduct intervention experiments on fixed 4-step Full GRPO using InteractSpatial windows:

Actor conditioning settingIoU ↑ (%)FPO ↓ (%)S-CLAP ↑Localizer L/R accuracy ↑ (%)
Full (Default)9.8512.570.77361.9
All actor slots removed8.7719.300.73953.8
Off-screen actor slots masked9.479.810.72357.4

Removing all actor slots drops S-CLAP by 0.034 and localizer accuracy by 8.1%, proving that multi-actor states provide vital off-screen sound cues that visual frames alone cannot capture. This is consistent with Table 5(b), where masking action/state conditioning lowers S-CLAP from 0.787 to 0.740 and raises FPO from 11.26% to 15.35%.

To test robustness against missingness and temporal misalignment:

InterventionΔIoU (points)ΔS-CLAP
Randomly remove 50% of other actors−0.26−0.045
Delay actor state by 250 ms+0.01−0.018
Delay actor state by 500 ms+0.02−0.016

Partial missingness hurts more than removing all slots (as does off-screen masking above) because Stage2 dropout removes all actor slots jointly, so partial removal falls outside the training distribution; per-slot dropout is a simple remedy. Temporal delays up to 500 ms cause negligible degradation, demonstrating robust tolerance to minor state synchronization lags.


We thank you again for the constructive feedback. We will incorporate all of the above results and clarifications into the revised manuscript.

Reviewer_864EOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

As world models become popular, this paper explores how spatial audio generation can be integrated into interactive world models. The authors introduce two datasets, including real-world recordings and interactive gameplay. They then propose Contrastive Spatial Audio-Video Pretraining (CSAVP), which learns how spatial audio changes with the listener’s viewpoint. They also apply distillation to reduce the number of sampling steps and use GRPO to recover some of the performance lost during distillation. The proposed method achieves strong performance on the benchmarks.

Critical strengths
  1. The paper introduces two datasets with a considerable number of samples.
  2. The motivation is interesting, and the proposed method has potential for practical applications, as the performance is quite promising.
Critical weaknesses
  1. All experiments are conducted on the two newly introduced datasets, not on previous standard benchmarks such as VGGSound.
  2. Although GRPO helps recover from performance degradation caused by distillation, the improvements are not consistent across all metrics and are sometimes not very helpful. For example, C-sim barely improves, while FPO becomes worse.
Questions for authors
  1. Will the two datasets be publicly released?
  2. The following work [1] may also be relevant, as it explores online video-to-audio generation. [1] SOUNDREACTOR: Frame-Level Online Video-to-Audio Generation (arXiv, 2025)
  3. The proposed method relies on three different vision encoders (CLIP, DINO, and CSAVP). However, the ablation results in Table 5 show relatively small performance differences when individual encoders are removed. Also, for real-time streaming inference, it might be difficult to have sufficient memory. Therefore, it would be helpful to break down the computational cost and memory for each component.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 864E [1/2]

We sincerely thank you for the thorough review and for recognizing the novelty, motivation, and practical potential of our work. Below, we address your concerns regarding dataset generalization, the multi-objective trade-offs in GRPO, and the computational and memory overhead of our pipeline.


W1: Evaluation beyond our two datasets

Interactive spatial audio requires aligned binaural audio, listener trajectories, and interaction signals. Conventional benchmarks such as VGGSound do not provide this multimodal combination, making them unsuitable for evaluating the spatial correspondence and interaction-conditioned generation studied here. As summarized in Table 1 of our paper, existing spatial audiovisual datasets provide either FOA audio without actions (Sphere360) or rotation-only control (BiAudio), whereas InteractSpatial pairs binaural audio with player controls and listener + actor states. Therefore, ViewSpatial (real-world recordings with controlled viewpoint changes) and InteractSpatial (first-person gameplay with aligned listener controls and multi-actor states) are uniquely tailored and indispensable for covering the complementary requirements of this task (Section 3).

Furthermore, generator training and evaluation strictly adhere to source-disjoint splits: ViewSpatial's training and test source videos are completely disjoint, with all trajectories from a given source kept in the same split; InteractSpatial is evaluated on held-out matches. As stated in Appendix B.2, "generation evaluation sources are excluded from both CSAVP representation training and generator training."


W2: Non-uniform GRPO gains, C-sim, and FPO

GRPO is designed to balance semantic alignment, event coverage, perceptual quality, and spatial correspondence. Combining multiple rewards inherently prevents over-optimizing a single proxy metric at the expense of others. Consequently, Full GRPO does not need to maximize every individual metric in isolation. For example, while the aesthetic-only configuration achieves a higher PQ, Full GRPO attains superior CLAP and S-CLAP scores.

We compare the reward configurations on InteractSpatial (Table 5(c)–(e)):

ConfigurationCLAP ↑PQ ↑C-sim ↑S-CLAP ↑IoU ↑ (%)FPO ↓ (%)
Distilled, no RL0.5156.830.17030.7319.489.23
Semantic only0.5326.880.18460.7639.7811.78
Aesthetic only0.5197.020.14750.7379.339.45
Chunk-level event only0.5226.780.16660.7609.8611.35
Full GRPO0.5346.890.17070.7739.8512.57

As noted in Section 5.5, relative to the distilled baseline, Full GRPO improves CLAP, PQ, S-CLAP, and event IoU, offering a more balanced combination across perceptual and spatial dimensions than single-reward variants. Crucially, in human evaluations, Full GRPO achieves a higher MOS-S score of 3.805 compared to 3.745 for distillation. This indicates that the slight increase in FPO does not degrade perceived realism; instead, broader event coverage yields a better overall listening experience.

To isolate the effect of the absent-event penalty, we compare event-only GRPO with and without it (Table 5(d)):

ConfigurationIoU ↑ (%)FPO ↓ (%)
Without penalty9.9714.09
With penalty9.8611.35

As defined in Appendix C.2 (Eq. C.7), FPO measures predicted event activity during intervals where a specific event is absent, rather than general background-noise energy. Our absent-event penalty successfully reduces FPO from 14.09% to 11.35% while keeping IoU nearly unchanged (9.97% → 9.86%). In Full GRPO, optimizing jointly for the remaining perceptual objectives results in an FPO of 12.57% (vs. 9.23% for distillation), reflecting a deliberate trade-off between comprehensive event coverage and false-positive control.

AuthorsOpenReview ↗

Response to Reviewer 864E [2/2]

Q1: Dataset release

We will publicly release both ViewSpatial and InteractSpatial along with our complete data-processing pipeline, including source identifiers, train/test splits, trajectory and interaction annotations, as well as acquisition, rendering, and filtering code. The CS2 demo-to-dataset tool used to build InteractSpatial is already publicly available (Section 3, footnote 1).


Q2: SoundReactor

Thank you for pointing out SoundReactor [1]. We will cite its causal, frame-level online stereo generation and few-step acceleration. In contrast, ISAUDIO focuses on interactive binaural generation conditioned on listener controls, observed visual prefixes, and multi-actor states, leveraging trajectory-aware spatial representations. We will clarify this connection and distinction in the related work section (Section 2).


Q3: Computational cost and memory of the visual encoders

Thank you for this valuable suggestion. The three vision encoders represent a deliberate quality–cost trade-off: CLIP supplies per-frame semantic features, DINO supplies spatial grids, and causal CSAVP supplies spatial-temporal features (Section 4.1; Appendix A.1). The visual-branch ablation in Table 5(a) quantifies their individual contributions:

ConditioningCLAP ↑C-sim ↑S-CLAP ↑
Full SFT0.5140.17870.702
Without DINO0.5070.18070.699
Without CLIP0.5070.17710.702
Without causal CSAVP0.5060.16880.693

While Table 5(a) pools both datasets, on ViewSpatial specifically, removing causal CSAVP drops C-sim from 0.170 to 0.144 and S-CLAP from 0.582 to 0.562, underscoring its critical role in viewpoint-dependent spatial grounding.

To quantify inference efficiency, we profile the four-step Full GRPO streaming pipeline on a single NVIDIA A100 (80GB) using batch size 1, covering the same stages as the timing protocol in Appendix C.1.4:

ComponentTime per 8-s input (s)Resident weights (GiB)
Video decoding (CPU)0.631—
CLIP (DFN5B ViT-H/14, 8 fps)0.9431.84
DINOv3 ViT-L/16 (4 fps)1.0091.13
Causal CSAVP (image embedding / temporal prefix)0.366 / 0.0632.27
Condition assembly and transfer0.042—
Generator sampling (4 steps × 11 chunks)1.2291.67
SAME-L decoding and CPU delivery0.8943.22
Full pipeline5.2210.1

These profiling results highlight two findings:

  1. Efficient spatial conditioning: Causal CSAVP takes 0.429 s per eight-second input, less than either CLIP or DINOv3, while its ablation demonstrates its contribution to spatial alignment.
  2. Quality–efficiency trade-off: The profile gives a compute RTF of 0.65 for incremental delivery; the paper's complete-segment RTF of 0.53 (Table C.5) uses a separate 32-input timing protocol. CLIP and DINOv3 together account for 1.95 s of computation and 2.97 GiB of resident model memory. Their individual ablations clarify each branch's quality contribution and inform choices under different resource budgets.

We thank you again for the constructive feedback. We will incorporate all of the above profiling results, detailed ablation analyses, and clarifications into the revised manuscript to further strengthen the paper.

References

[1] Saito, Koichi, et al. "Soundreactor: Frame-level online video-to-audio generation." arXiv preprint arXiv:2510.02110 (2025).

Thank you for the detailed responses and additional experiments! The additional results with baselines fine-tuned on the same datasets in your response to Reviewer tQqG's W4 partially address my concern regarding dataset generalization. Overall, I appreciate the novelty of the work, the dataset contributions, and the competitive performance under a more restrictive causal setting. Therefore, I will maintain my original rating.

Thank you very much for your follow-up and for your positive assessment. We are glad that the matched-data comparisons help address your concern about dataset generalization. We will further strengthen the discussion of dataset generalization in the revised version. Thank you again for your constructive feedback and support.

Comments on the paper

1 comment

Dear AC and Reviewers,

We sincerely thank you for your time and constructive feedback. We briefly summarize the discussion and how each reviewer's concerns were addressed.

Overall. All three reviewers rated the presentation as clear. After the discussion, Reviewer AeUU confirmed that all of my concerns have now been resolved and raised the score to 4 (clear acceptance). Reviewer 864E maintained a positive score of 3, noting that they appreciate the novelty of the work, the dataset contributions, and the competitive performance under a more restrictive causal setting.

Reviewer AeUU (3 → 4). The concerns covered metric coupling, missing baselines (ViSAudio), distillation and GRPO trade-offs, and abstract clarity. We added an external event-conditioned localizer and a blinded human spatial study (MOS-S), both independent of CSAVP and the rewards; compared against ViSAudio and other baselines, both released and fine-tuned on our data; added a controlled credit-assignment comparison; and revised the abstract to report a single operating regime. We also provided HRTF sensitivity, chunk-length, and actor-state analyses.

Reviewer 864E (3, maintained). The concerns covered evaluation beyond our two datasets, GRPO trade-offs, and computational cost. We explained why existing benchmarks lack the required binaural and interaction signals and added comparisons with baselines fine-tuned on the same data, which the reviewer noted partially address the generalization concern. We also showed that the absent-event penalty reduces FPO from 14.09% to 11.35% and that GRPO improves MOS-S over the distilled model, and we provided a per-component latency and memory breakdown. We will further strengthen the discussion of dataset generalization in the revised version.

Reviewer tQqG (2, no response during the discussion). We regret that Reviewer tQqG was not able to join the discussion. We addressed all four concerns with new experiments and revisions:

W1 (ViewSpatial sources). We documented the 5,036 YouTube 360° FOA recordings and the source-disjoint splits, and committed to releasing both datasets.

W2 (Contribution of CSAVP). Removing the trajectory objectives drops same-scene retrieval accuracy from 32.66% to 14.45%. A per-dataset ablation further shows that the pooled score masked CSAVP's effect on ViewSpatial, where S-CLAP drops from 0.582 to 0.562.

W3 (Shared evaluation components). We added an external localizer and MOS-S that share no weights with CSAVP or the rewards. ISAudio achieves the highest L/R accuracy (61.9%) and MOS-S (3.805) among all generators.

W4 (Matched-data comparisons). Against all binaural baselines fine-tuned on our data, four-step ISAudio still achieves the highest event IoU (9.85% vs. 9.23%), S-CLAP (0.773 vs. 0.721), and L/R accuracy (61.9% vs. 58.4%), while observing only the visual prefix.

Notably, Reviewer AeUU raised closely related concerns about metric coupling and additional baselines, and after reading our responses, including those to Reviewer tQqG, confirmed that they were resolved. Reviewer 864E likewise noted that our response to tQqG's W4 helped address their own concern. We hope these responses and revisions are helpful in assessing Reviewer tQqG's concerns.

We thank you again for handling our submission and for your consideration.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.