- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a new task: Video-to-4DAVScene, and proposes a framework named SonicVista that generates a freely explorable 4D audio-visual scene from a monocular video. On the visual side, it combines multiple foundation models to decompose the scene into 360° static background Gaussians and dynamic foreground Gaussians with world-space trajectories, followed by foreground-background alignment. On the audio side, the core contribution is dual-space diffusion synchronization. During flow-matching sampling, forward acoustic rendering, together with SAM-Audio separation and approximate inverse rendering, repeatedly couples the video-conditioned camera-space audio mixture with per-object audio tracks. As a result, the source tracks stay aligned with the video while remaining re-renderable as ambisonic/binaural audio at novel listener poses. The authors also collect SonicVista360, a real-world FOA dataset of 54 clips with the microphone placed away from the camera. On this dataset, SonicVista outperforms baselines including ViSAGE, OmniAudio, SonoWorld, and Sonic4D on both spatial and semantic metrics, as well as in a 50-participant user study.
Critical strengths
-
This paper proposes an interesting and practical task, Video-to-4DAVScene, and is the first to extend 4D scene generation to spatial audio with moving sound sources. The task is decomposed into explicit geometry and trajectories, heterogeneous sound sources, and consistency across camera, object, and world spaces. Both the motivation and the solution follow a clear logic, and I believe the work offers clear value for follow-up research in the world-model field.
-
SonicVista360 is highly practical. It uses real FOA recordings, with the microphone placed at a position different from the camera. This genuinely tests spatial audio at novel listener poses rather than merely reproducing the input-view audio, and it makes a concrete contribution toward establishing objective evaluation for this task in the community.
-
The core mechanism is well designed and strongly supported by experiments. The SonicVista approach makes a lot of sense, and it incorporates most of the currently available visual techniques to ensure visual stability and consistency. The full method consistently improves over the four baselines and the T2A/V2A variants. The multi-track V2A experiments on FAIR-Play further show that the mechanism has a degree of generality.
Critical weaknesses
-
The detailed operation of R and Q is unclear. Sec. 4 only briefly introduces the concepts of R and Q, but since they constitute the core contribution of the paper, a more detailed description of the procedure is needed. In addition, the use of input audio appears contradictory. In the task definition in Sec. 3, the input consists only of RGB frames, and in Sec. 4.3, a^cam is generated by the flow-matching of HY-VideoFoley rather than using the observed audio track. I am therefore unsure where the input audio is actually used.
-
The values of the synchronization hyperparameters are not reported. Although Appendix A.1 states that the hyperparameters from the authors' own 4D setting are used, the parameters of the 4D setting itself are not reported either. It is also not specified whether synchronization is performed at every flow step or how many steps are used in total.
-
The authors attribute the large improvement over Sonic4D to object-space audio generation and dual-space synchronization. However, the Sonic4D baseline retains its own visual backbone, models only a single dominant sound source, and undergoes a binaural-to-FOA adaptation, so the comparison confounds a large number of factors. The only segment that isolates the effect of synchronization is Ours (V2A) to Ours. Synchronization brings about one third of the spatial metric gains, but the gains on semantic metrics are small, and elevation remains completely unchanged. This may indicate that synchronization mainly improves the temporal/energy alignment between source signals and visual events rather than localization itself, yet the main task provides no corresponding metric to measure temporal alignment.
Questions for authors
Please see the Critical Weaknesses section. I hope the authors can address my concerns.
5 comments
2. The values of the synchronization hyperparameters are not reported. Although Appendix A.1 states that the hyperparameters from the authors' own 4D setting are used, the parameters of the 4D setting itself are not reported either. It is also not specified whether synchronization is performed at every flow step or how many steps are used in total.
We apologize for this omission. The synchronization hyperparameters are:
- Flow steps: We use 100 flow-matching steps total. Synchronization (the audio-space sync cycle in Eq. 7-10) is performed at every flow step after .
- Synchronization weights: and .
- Synchronization start time: We observe that synchronization is most effective starting from flow time , as the audio latents already begin to encode meaningful content at this point. Before , the latent states are dominated by noise and synchronization provides minimal benefit.
These values are used consistently across all experiments. We have added a complete hyperparameter paragraph to the revised appendix K.2.
3. The authors attribute the large improvement over Sonic4D to object-space audio generation and dual-space synchronization. However, the Sonic4D baseline retains its own visual backbone, models only a single dominant sound source, and undergoes a binaural-to-FOA adaptation, so the comparison confounds a large number of factors. The only segment that isolates the effect of synchronization is Ours (V2A) to Ours. Synchronization brings about one third of the spatial metric gains, but the gains on semantic metrics are small, and elevation remains completely unchanged. This may indicate that synchronization mainly improves the temporal/energy alignment between source signals and visual events rather than localization itself, yet the main task provides no corresponding metric to measure temporal alignment.
We appreciate this careful analysis and provide additional evidence to disentangle the contributing factors.
FOA adaptation. We clarify that our binaural-to-FOA adaptation of Sonic4D replaces its two binaural microphones with seven virtual microphones (one omnidirectional plus three orthogonal figure-eight pairs) to capture the W, X, Y, Z components of first-order ambisonics. This is a physically exact transformation and introduces no additional error theoretically.
Comprehensive factor-isolation ablation. To disentangle the contributions of each component, we conduct a full set of ablations including naive separation/generation compositions and a visual-backbone-controlled Sonic4D variant. All results are means over 54 scenes:
| Method | CC ↑ | AUC ↑ | Angular | Azimuth | Elevation | CLAP-A | CLAP-T |
|---|---|---|---|---|---|---|---|
| V2A | 0.591 | 0.801 | 46.1 | 43.3 | 16.9 | 0.527 | 0.262 |
| V2A → SAM Audio | 0.383 | 0.696 | 61.2 | 60.1 | 18.6 | 0.395 | 0.187 |
| SAM3 → V2A | 0.550 | 0.780 | 49.1 | 46.7 | 16.8 | 0.416 | 0.247 |
| T2A | 0.566 | 0.787 | 48.4 | 45.7 | 16.9 | 0.349 | 0.185 |
| Sonic4D + Ours Visual | 0.599 | 0.804 | 44.9 | 42.0 | 16.7 | 0.424 | 0.218 |
| Ours | 0.647 | 0.835 | 40.7 | 36.6 | 16.9 | 0.560 | 0.274 |
To isolate the effect of the visual backbone, Sonic4D + Ours Visual pairs our exact geometry with Sonic4D's audio pipeline and extends it to multi-source by per-source V2A generation. We note that spatial metrics depend on two critical factors: (a) source localization (via visual tracking), and (b) energy distribution (plausible loudness and spectral clarity). This comprehensive ablation precisely identifies where alternative formulations fail:
-
Background Contamination (V2A / T2A): These variants may produce foreground tracks containing background sounds, distorting the foreground energy pattern. Attempting post-hoc background removal (V2A -> SAM Audio) introduces severe separation artifacts that further destroy spatial and temporal structure (CC: 0.591 -> 0.383), proving that sequential filtering is suboptimal.
-
Limitations of SAM3 → V2A: SAM3 masking can introduce artifacts and does not guarantee background-free audio generation. These issues may contribute to its lower performance compared with unmasked V2A (CC: 0.550 vs. 0.591), suggesting that masking alone is insufficient to obtain clean object-level audio.
-
Geometric Assumptions in Sonic4D: Sonic4D + Ours Visual demonstrates that Sonic4D's audio pipeline models the scene as a fixed 200×200 box. This strong geometric assumption introduces spatialization artifacts even when using our same visual branch (Angular error: 44.9° vs. Ours 40.7°), and its audio backbone concurrently degrades semantic quality (CLAP-A: 0.424 vs. Ours 0.560).
Overall, this experiment confirms the attribution of our performance gains. Our visual backbone accounts for the initial geometry (Sonic4D CC 0.446 –> 0.599), while our dual-space diffusion synchronization and proposed audio generation account for the remaining energy correlation.
Temporal alignment metric. We agree that the original spatial metrics do not explicitly isolate the temporal and energy alignment between source signals and visual events. To address this, we added an envelope-correlation metric (Env-r), computed as the zero-lag Pearson correlation between the RMS envelopes of the generated and recorded FOA W channels. In addition, assessing audio-visual alignment in scene-level generation is non-trivial since the alignment is highly viewpoint-dependent, requiring coupled view selection during rendering.
| Method | OmniAudio | SonoWorld | Sonic4D | Ours |
|---|---|---|---|---|
| Env-r ↑ | -0.0060 ± 0.059 | 0.0121 ± 0.049 | 0.207 ± 0.041 | 0.271 ± 0.042 |
Ours achieves the highest Env-r (0.271 vs. Sonic4D's 0.207), supporting improved temporal alignment. We have added this metric and analysis to the revised manuscript.
We sincerely thank the reviewer for the detailed and constructive feedback.
We thank the reviewer for identifying this confusion and provide clarification on both points. Input audio. We confirm that our method does not use the input video's audio track. The input consists solely of RGB frames, as defined in Sec. 3. In Sec. 4.3, acam is generated by flow-matching of HY-VideoFoley conditioned on the input video frames, not from any observed audio. We have corrected the ambiguous language in the revised manuscript. In addition, we note that our framework can optionally incorporate input audio when available, and we newly demonstrate this capability on the updated anonymous project webpage.
Operations R and Q. We provide a detailed description, now also expanded in the revised Appendix (Sec. J). The forward renderer Ri maps an object-space signal to its camera-space contribution along the observed camera trajectory. We adopt a direct-path propagation model accounting for Doppler shifts, distance attenuation, and air absorption: Ri(aiobj)(t)=gi(t)⋅aiobj(φi(t)), where φi(t) is the emission-time mapping determined by the source and camera trajectories (its local rate captures the Doppler shift), and gi(t) is the multiplicative gain from distance attenuation and air absorption (Eq. 19 in Appendix). The inverse renderer Qi is the exact analytical inverse of Ri under this mode. It first reverses the time mapping and then divides out the propagation gain: Qi(bi)(u)=bi(φi−1(u))/gi(φi−1(u)).
In discrete time, resampling introduces minor numerical errors, and source-separation errors from SAM-Audio further affect the recovered object-space estimate. Occluded intervals or intervals with large obj-cam distances are excluded from inverse-rendering feedback via a validity mask (Eq. 22–24), so synchronization only operates where reliable camera-space observations are available. We have added an interactive visualization of the R and Q operators on the project webpage, showing the forward and inverse rendering process step-by-step with audio examples.