acceptodds
Under review as a conference paper at ICLR 2027

SonicVista: Generating a 4D Audio-Visual Scene from One Video via Diffusion Synchronization

Abstract

Recent progress in visual scene generation has begun to move beyond static 3D scenes toward the more challenging problem of generating dynamic 4D scenes from minimal input. However, sound, despite being intrinsically coupled with dynamic objects and events, remains largely unmodeled in prior work. We present V2AVS, a new task that turns a video into an exploratable, time-varying 4D audio-visual scene, and SonicVista, the first framework for this task. From an input monocular video, our method generates a 360-degree static background, reconstructs explicit motion trajectories of dynamic foreground sounding objects, and synthesizes a physically-grounded, multi-source consistent sound field through dual-space audio diffusion synchronization. The resulting 4D scene supports both real-time novel-view visual rendering and spatial audio rendering. Evaluations on real-world datasets show that SonicVista achieves the state-of-the-art spatial audio rendering quality and geometry-grounded audio-visual consistency in generated 4D audio-visual scenes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-Visual Learning, 4D Scene Generation, Spatial Audio, Audio Generation, First-Order Ambisonics (FOA)

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yDCNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a new task: Video-to-4DAVScene, and proposes a framework named SonicVista that generates a freely explorable 4D audio-visual scene from a monocular video. On the visual side, it combines multiple foundation models to decompose the scene into 360° static background Gaussians and dynamic foreground Gaussians with world-space trajectories, followed by foreground-background alignment. On the audio side, the core contribution is dual-space diffusion synchronization. During flow-matching sampling, forward acoustic rendering, together with SAM-Audio separation and approximate inverse rendering, repeatedly couples the video-conditioned camera-space audio mixture with per-object audio tracks. As a result, the source tracks stay aligned with the video while remaining re-renderable as ambisonic/binaural audio at novel listener poses. The authors also collect SonicVista360, a real-world FOA dataset of 54 clips with the microphone placed away from the camera. On this dataset, SonicVista outperforms baselines including ViSAGE, OmniAudio, SonoWorld, and Sonic4D on both spatial and semantic metrics, as well as in a 50-participant user study.

Critical strengths
  1. This paper proposes an interesting and practical task, Video-to-4DAVScene, and is the first to extend 4D scene generation to spatial audio with moving sound sources. The task is decomposed into explicit geometry and trajectories, heterogeneous sound sources, and consistency across camera, object, and world spaces. Both the motivation and the solution follow a clear logic, and I believe the work offers clear value for follow-up research in the world-model field.

  2. SonicVista360 is highly practical. It uses real FOA recordings, with the microphone placed at a position different from the camera. This genuinely tests spatial audio at novel listener poses rather than merely reproducing the input-view audio, and it makes a concrete contribution toward establishing objective evaluation for this task in the community.

  3. The core mechanism is well designed and strongly supported by experiments. The SonicVista approach makes a lot of sense, and it incorporates most of the currently available visual techniques to ensure visual stability and consistency. The full method consistently improves over the four baselines and the T2A/V2A variants. The multi-track V2A experiments on FAIR-Play further show that the mechanism has a degree of generality.

Critical weaknesses
  1. The detailed operation of R and Q is unclear. Sec. 4 only briefly introduces the concepts of R and Q, but since they constitute the core contribution of the paper, a more detailed description of the procedure is needed. In addition, the use of input audio appears contradictory. In the task definition in Sec. 3, the input consists only of RGB frames, and in Sec. 4.3, a^cam is generated by the flow-matching of HY-VideoFoley rather than using the observed audio track. I am therefore unsure where the input audio is actually used.

  2. The values of the synchronization hyperparameters are not reported. Although Appendix A.1 states that the hyperparameters from the authors' own 4D setting are used, the parameters of the 4D setting itself are not reported either. It is also not specified whether synchronization is performed at every flow step or how many steps are used in total.

  3. The authors attribute the large improvement over Sonic4D to object-space audio generation and dual-space synchronization. However, the Sonic4D baseline retains its own visual backbone, models only a single dominant sound source, and undergoes a binaural-to-FOA adaptation, so the comparison confounds a large number of factors. The only segment that isolates the effect of synchronization is Ours (V2A) to Ours. Synchronization brings about one third of the spatial metric gains, but the gains on semantic metrics are small, and elevation remains completely unchanged. This may indicate that synchronization mainly improves the temporal/energy alignment between source signals and visual events rather than localization itself, yet the main task provides no corresponding metric to measure temporal alignment.

Questions for authors

Please see the Critical Weaknesses section. I hope the authors can address my concerns.

5 comments

We sincerely thank the reviewer for the detailed and constructive feedback.

1. The detailed operation of R and Q is unclear. Sec. 4 only briefly introduces the concepts of R and Q, but since they constitute the core contribution of the paper, a more detailed description of the procedure is needed. In addition, the use of input audio appears contradictory. In the task definition in Sec. 3, the input consists only of RGB frames, and in Sec. 4.3, acama^{cam} is generated by the flow-matching of HY-VideoFoley rather than using the observed audio track. I am therefore unsure where the input audio is actually used.

We thank the reviewer for identifying this confusion and provide clarification on both points. Input audio. We confirm that our method does not use the input video's audio track. The input consists solely of RGB frames, as defined in Sec. 3. In Sec. 4.3, acama^{cam} is generated by flow-matching of HY-VideoFoley conditioned on the input video frames, not from any observed audio. We have corrected the ambiguous language in the revised manuscript. In addition, we note that our framework can optionally incorporate input audio when available, and we newly demonstrate this capability on the updated anonymous project webpage.

Operations R and Q. We provide a detailed description, now also expanded in the revised Appendix (Sec. J). The forward renderer RiR_i maps an object-space signal to its camera-space contribution along the observed camera trajectory. We adopt a direct-path propagation model accounting for Doppler shifts, distance attenuation, and air absorption: Ri(aiobj)(t)=gi(t)⋅aiobj(φi(t)),R_i(a^{obj}_i)(t) = g_i(t) · a^{obj}_i(φ_i(t)), where φi(t)φ_i(t) is the emission-time mapping determined by the source and camera trajectories (its local rate captures the Doppler shift), and gi(t)g_i(t) is the multiplicative gain from distance attenuation and air absorption (Eq. 19 in Appendix). The inverse renderer QiQ_i is the exact analytical inverse of RiR_i under this mode. It first reverses the time mapping and then divides out the propagation gain: Qi(bi)(u)=bi(φi−1(u))/gi(φi−1(u)).Q_i(b_i)(u) = b_i(φ_i^{-1}(u)) / g_i(φ_i^{-1}(u)).

In discrete time, resampling introduces minor numerical errors, and source-separation errors from SAM-Audio further affect the recovered object-space estimate. Occluded intervals or intervals with large obj-cam distances are excluded from inverse-rendering feedback via a validity mask (Eq. 22–24), so synchronization only operates where reliable camera-space observations are available. We have added an interactive visualization of the R and Q operators on the project webpage, showing the forward and inverse rendering process step-by-step with audio examples.

2. The values of the synchronization hyperparameters are not reported. Although Appendix A.1 states that the hyperparameters from the authors' own 4D setting are used, the parameters of the 4D setting itself are not reported either. It is also not specified whether synchronization is performed at every flow step or how many steps are used in total.

We apologize for this omission. The synchronization hyperparameters are:

  • Flow steps: We use 100 flow-matching steps total. Synchronization (the audio-space sync cycle in Eq. 7-10) is performed at every flow step after τ=0.2\tau=0.2.
  • Synchronization weights: ηcam=0.8\eta_{cam} = 0.8 and ηobj=0.5\eta_{obj} = 0.5.
  • Synchronization start time: We observe that synchronization is most effective starting from flow time τ=0.2\tau = 0.2, as the audio latents already begin to encode meaningful content at this point. Before τ=0.2\tau = 0.2, the latent states are dominated by noise and synchronization provides minimal benefit.

These values are used consistently across all experiments. We have added a complete hyperparameter paragraph to the revised appendix K.2.

3. The authors attribute the large improvement over Sonic4D to object-space audio generation and dual-space synchronization. However, the Sonic4D baseline retains its own visual backbone, models only a single dominant sound source, and undergoes a binaural-to-FOA adaptation, so the comparison confounds a large number of factors. The only segment that isolates the effect of synchronization is Ours (V2A) to Ours. Synchronization brings about one third of the spatial metric gains, but the gains on semantic metrics are small, and elevation remains completely unchanged. This may indicate that synchronization mainly improves the temporal/energy alignment between source signals and visual events rather than localization itself, yet the main task provides no corresponding metric to measure temporal alignment.

We appreciate this careful analysis and provide additional evidence to disentangle the contributing factors.

FOA adaptation. We clarify that our binaural-to-FOA adaptation of Sonic4D replaces its two binaural microphones with seven virtual microphones (one omnidirectional plus three orthogonal figure-eight pairs) to capture the W, X, Y, Z components of first-order ambisonics. This is a physically exact transformation and introduces no additional error theoretically.

Comprehensive factor-isolation ablation. To disentangle the contributions of each component, we conduct a full set of ablations including naive separation/generation compositions and a visual-backbone-controlled Sonic4D variant. All results are means over 54 scenes:

MethodCC ↑AUC ↑AngularAzimuthElevationCLAP-ACLAP-T
V2A0.5910.80146.143.316.90.5270.262
V2A → SAM Audio0.3830.69661.260.118.60.3950.187
SAM3 → V2A0.5500.78049.146.716.80.4160.247
T2A0.5660.78748.445.716.90.3490.185
Sonic4D + Ours Visual0.5990.80444.942.016.70.4240.218
Ours0.6470.83540.736.616.90.5600.274

To isolate the effect of the visual backbone, Sonic4D + Ours Visual pairs our exact geometry with Sonic4D's audio pipeline and extends it to multi-source by per-source V2A generation. We note that spatial metrics depend on two critical factors: (a) source localization (via visual tracking), and (b) energy distribution (plausible loudness and spectral clarity). This comprehensive ablation precisely identifies where alternative formulations fail:

  1. Background Contamination (V2A / T2A): These variants may produce foreground tracks containing background sounds, distorting the foreground energy pattern. Attempting post-hoc background removal (V2A -> SAM Audio) introduces severe separation artifacts that further destroy spatial and temporal structure (CC: 0.591 -> 0.383), proving that sequential filtering is suboptimal.

  2. Limitations of SAM3 → V2A: SAM3 masking can introduce artifacts and does not guarantee background-free audio generation. These issues may contribute to its lower performance compared with unmasked V2A (CC: 0.550 vs. 0.591), suggesting that masking alone is insufficient to obtain clean object-level audio.

  3. Geometric Assumptions in Sonic4D: Sonic4D + Ours Visual demonstrates that Sonic4D's audio pipeline models the scene as a fixed 200×200 box. This strong geometric assumption introduces spatialization artifacts even when using our same visual branch (Angular error: 44.9° vs. Ours 40.7°), and its audio backbone concurrently degrades semantic quality (CLAP-A: 0.424 vs. Ours 0.560).

Overall, this experiment confirms the attribution of our performance gains. Our visual backbone accounts for the initial geometry (Sonic4D CC 0.446 –> 0.599), while our dual-space diffusion synchronization and proposed audio generation account for the remaining energy correlation.

Temporal alignment metric. We agree that the original spatial metrics do not explicitly isolate the temporal and energy alignment between source signals and visual events. To address this, we added an envelope-correlation metric (Env-r), computed as the zero-lag Pearson correlation between the RMS envelopes of the generated and recorded FOA W channels. In addition, assessing audio-visual alignment in scene-level generation is non-trivial since the alignment is highly viewpoint-dependent, requiring coupled view selection during rendering.

MethodOmniAudioSonoWorldSonic4DOurs
Env-r ↑-0.0060 ± 0.0590.0121 ± 0.0490.207 ± 0.0410.271 ± 0.042

Ours achieves the highest Env-r (0.271 vs. Sonic4D's 0.207), supporting improved temporal alignment. We have added this metric and analysis to the revised manuscript.

Reviewer_kG7ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Video-to-4DAVScene, a new task: turning a single monocular video into an explorable 4D audio-visual scene with novel-view rendering and listener-dependent spatial audio. The proposed method, SonicVista, is training-free. It builds a 4D visual scene (panoramic background plus dynamic foreground Gaussians), then generates a spatial sound field through dual-space diffusion synchronization between camera-space and object-space audio. The authors collect a new dataset, SonicVista360, with ground-truth first-order ambisonic audio recorded at a novel listening pose. SonicVista outperforms prior baselines on spatial and semantic metrics and is preferred in a user study.

Critical strengths

A new, well-motivated task with a principled formulation. Prior 4D scene generation ignores sound, and prior spatial audio work is limited to static scenes or constrained camera setups. The paper defines the problem formally, with an explorable scene and listener-dependent audio. It also provides a dataset with ground-truth FOA recorded at a novel listener pose,.

Video and appendix. The supplementary video make the generated 4D audio-visual scenes easier to understand and judge than the paper alone. The appendix adds FAIR-Play results and ablation studies (T2A and V2A variants).

Critical weaknesses

Cascaded, accumulated error is not analyzed. The pipeline is long and training-free: geometry and pose estimation, segmentation, VLM captioning, panorama outpainting, panorama-to-3D, foreground Gaussians, foreground-background alignment, source separation, and finally audio synthesis. Each stage can fail (e.g., poor depth or poses, wrong masks, hallucinated outpainted background, misaligned trajectories), and errors can propagate downstream into the sound field. Without end-to-end training to compensate, this is a real robustness concern. The paper offers no quantitative or qualitative analysis of how upstream errors affect final audio quality, such as per-stage ablations, failure cases, or perturbation studies.

The audio rendering and conditioning pipeline is under-specified. Several key questions are left unanswered:

  • How the input video’s audio is used and embedded.
  • How SAM-Audio separates different sources, and how the separation prompts are chosen.
  • How the sound-object labels are generated by the VLM, and how it is ensured that they generalize enough to cover all sounding objects in the scene.
  • How the final mixture is shown to be grounded in time and 3D position rather than being largely the product of a hallucinating generative audio model.

Because the claimed contribution is physically grounded spatial audio, these gaps undermine the central claim.

Experimental evidence is limited. Although the task is new, results come mainly from a small self-collected set (54 clips of 8 seconds) plus a limited FAIR-Play evaluation. This is not enough to demonstrate the robustness and generality of a complex training-free pipeline across diverse scenes, source types, and acoustic conditions. Broader quantitative evaluation is needed.

Input Audio is missing in the supplementary video. In the supplementary video, the segments labeled “input” contain no audio. This makes it challenging to judge whether the generated audio is consistent with the original audio in the input video, which is a core claim of the method.

Questions for authors

Please refer to the weakness section.

7 comments

We thank the reviewer for the thorough evaluation.

Cascaded, accumulated error is not analyzed. The pipeline is long and training-free: geometry and pose estimation, segmentation, VLM captioning, pano outpainting, pano-to-3D, FG Gaussians, FG-BG alignment, source separation, and finally audio synthesis. Each stage can fail and errors can propagate downstream into the sound field.

We address this concern with the systematic perturbation study reported in Table below. The key insight is that, unlike pipelines composed of small task-specific models or heuristics, our pipeline is built on large foundation models (VGGT-Ω for geometry/poses, SAM3 for segmentation, Kimi K3 for VLM captioning), which are inherently robust to noisy inputs.

AblationCCAUCazimuthelevationangularCLAP-ACLAP-T
Ours0.6470.83536.616.940.70.5600.274
Depth 50%0.6350.83037.117.541.40.5420.269
Mask Boundary ±\pm 30px0.6370.83037.416.941.50.5580.266
Camera Pose 25 deg0.6230.82536.719.842.90.5560.273
Mask drop 50%0.6470.83536.616.940.70.5590.274
Mask first 50%0.6430.83237.217.041.10.5470.271
FG-BG scale-only0.6460.83536.716.940.80.5580.273
FG-BG drop 10% contact frame0.6470.83536.616.940.70.5600.274
Track σ 1.00.6390.83136.918.041.40.5560.270
VLM caption drop 25%0.6150.81740.417.043.40.5490.264

Specifically:

  • Geometric noise causes only modest degradation. Even 50% depth noise or ±25° camera pose perturbation degrades audio metrics only modestly (CC drops by 0.012-0.024).
  • VLM label robustness. Dropping 25% of VLM labels produces the largest semantic degradation, but in practice, modern VLMs (e.g., Kimi K3) reliably identify sounding objects with very low failure rates. Furthermore, hallucinated labels are rejected by SAM3, where we did not observe false-positive segmentation cases.
  • Mask dropping and FG-BG alignment are highly robust. Dropping 50% of mask frames or removing local displacement correction (scale-only alignment) has negligible impact on audio metrics. Human scenes are largely unaffected by mask perturbations, since GEM recovers SMPL-X motion without requiring segmentation masks. For rigid-body scenes, these results show that the pipeline gracefully handles incomplete or imperfect visual inputs through interpolation and mask-aware extrapolation.
  • Trajectory noise has moderate impact. We test adding Gaussian noise (σ=1.0 in world unit) to intermediate object trajectories causes a noticeable but bounded degradation (CC: 0.647 to 0.639, angular: 40.7 to 41.4 degree), confirming that the audio rendering is sensitive to object position but not catastrophically so. Combining with bounded geometric perturbation results, this result doubly confirms that upstream foundation models absorb geometric perturbations.
  • Source separation and generation are coupled. We clarify that in our framework, separation (via SAM-Audio) and generation (via HY-VideoFoley) are jointly applied within each flow-matching step during diffusion synchronization (see Sec. 4.3, Eq. 7-9). Therefore, separation error cannot be measured independently; instead, we compare our synchronized pipeline against the T2A and V2A ablations, which isolate the effect of removing synchronization.

Without end-to-end training to compensate, this is a real robustness concern.

We also note that end-to-end training does not inherently guarantee robustness. For instance, OmniAudio, which is trained end-to-end, generalizes poorly to out-of-distribution inputs in our evaluation, while our modular, foundation-model-based pipeline demonstrates consistent performance across scenes -- given these reasons we stated above.

Failure cases?

We have observed two typical failure cases:

  1. Visual reconstruction failures. Lift4D, our current instantiation for objects with general deformable motion, may fail when objects are heavily occluded or move outside the camera’s field of view. Additionally, in our experiments, thin, moving structures can be incompletely reconstructed or appear semi-transparent. In these cases, the audio pipeline continues to operate using the recovered source trajectories, but the visual quality of the reconstructed scene is degraded.
  2. Ambiguity between acoustically similar sources. When multiple sources of the same audio class appear in a video, such as two violins, SAM-Audio may fail to distinguish them reliably. We note that even sometimes humans may find it challenging to distinguish multiple audio sources of the same class at the same location. A promising direction for future work is to further exploit motion cues to improve instance-level separation in Sam-Audio.

2. The audio rendering and conditioning pipeline is under-specified. Several key questions are left unanswered:

We appreciate this feedback and provide the requested clarifications:

2.1. How the input video’s audio is used and embedded?

We clarify that our method does not use the input video's audio track. The input consists only of RGB frames, as stated in the task definition (Sec. 3). All audio is generated from visual and textual conditioning. We have corrected any ambiguous language in the revised manuscript.

In addition, we note that our framework can optionally incorporate input audio when available, and we demonstrate this capability on the updated project webpage (https://sonicvista.github.io/).

2.2. How SAM-Audio separates different sources, and how the separation prompts are chosen?

SAM-Audio is a query-conditioned audio separation model. The separation prompts are derived from the VLM's object-level descriptions: for each detected sounding object, the VLM produces a sound description (e.g., "car engine and tire sound on asphalt"), which is used as the text query for SAM-Audio. We have added the full VLM prompt template to the revised appendix K.

2.3. How the sound-object labels are generated by the VLM, and how it is ensured that they generalize enough to cover all sounding objects in the scene.

We use Kimi K3 to analyze the input video and identify all potential sounding objects, producing both a scene-level acoustic description and per-object sound labels. We acknowledge an intrinsic ambiguity in sound-source identification: some visible objects may be silent (e.g., a parked car), while some invisible sources may produce sound (e.g., birds in trees). Our VLM prompt is designed to capture the major sources; if users wish to add, remove, or modify sound sources, they can simply edit the text prompts at essentially no cost. For evaluation, we provide major source labels as input, which is fair since all text-conditioned baselines also receive this information. We have added our VLM prompt in the updated Appendix (Sec. K.1).

2.4. How the final mixture is shown to be grounded in time and 3D position rather than being largely the product of a hallucinating generative audio model?

To clarify the intended scope of our contribution, our objective is to physically ground the generative process using acoustic priors, rather than to perform deterministic physical simulation (i.e., strict physical accuracy). Specifically:

  1. Object positions and velocities are recovered from RGB video through explicit 3D reconstruction and tracking.
  2. The acoustic renderer R maps object-space sounds to camera space accounting for source position, distance attenuation, and occlusion (Eq. 4, 11).
  3. The diffusion synchronization mechanism (Sec. 4.3) enforces bidirectional consistency between object-space and camera-space audio at every flow step, ensuring that the global sound's motion effects are constrained by the underlying propagation model rather than purely hallucinated by the diffusion model.

This is fundamentally different from purely object-space generation (SonoWorld & Ours T2A) or purely camera-space generation (Sonic4D & Ours V2A). Figure 4 in our paper shows a qualitative case where camera-space models produce duplicate motion effects at novel views, while text-based methods lack timbre fidelity, while our method correctly produces the motion effect, loudness pattern, and timbre at the anticipated time. Quantitatively, our temporal alignment metric (Env-r) further confirms this grounding.

MethodOmniAudioSonoWorldSonic4DOurs
Env-r ↑\uparrow-0.0060 ± 0.0590.0121 ± 0.0490.207 ± 0.0410.271 ± 0.042

3. Experimental evidence is limited. Although the task is new, results come mainly from a small self-collected set (54 clips of 8 seconds) plus a limited FAIR-Play evaluation.

While prior audio-visual spatial audio rendering methods typically evaluate on only two [Chen et al. 2024] or a few [Jin et al. 2025] real scenes, as collecting scene-level real-world spatial audio with calibrated microphone poses is non-trivial, SonicVista360 surpasses earlier works to offer a benchmark set of 54 clips from 8 captured setups with 25 vehicle scenes and 29 human scenes, each with 1–2 primary dynamic sources and natural background noise, paired with synchronized ground-truth FOA audio recorded at a novel listener pose. We additionally report uncertainty via 95% confidence intervals computed as 1.96 × SEM, indicating stable gains consistent with our user study. We have updated table 1 in the paper to include these confidence intervals, and the numbers are as below:

MethodAzimuth (°) ↓Elevation (°) ↓Angular (°) ↓CC ↑AUC ↑CLAP-A ↑CLAP-T ↑
ViSAGe74.1 ± 7.81024.5 ± 2.82875.9 ± 6.3520.189 ± 0.0660.615 ± 0.0360.078 ± 0.030-0.059 ± 0.022
OmniAudio80.9 ± 7.47430.0 ± 3.84783.1 ± 6.6410.124 ± 0.0780.579 ± 0.0410.234 ± 0.0570.057 ± 0.044
SonoWorld65.7 ± 6.61024.3 ± 3.65266.6 ± 5.4390.333 ± 0.0730.672 ± 0.0380.365 ± 0.0500.154 ± 0.026
Sonic4D54.3 ± 6.73317.2 ± 1.87255.5 ± 5.7790.446 ± 0.0750.728 ± 0.0400.223 ± 0.0390.103 ± 0.027
Ours (T2A)45.7 ± 6.23716.8 ± 1.91348.4 ± 5.3610.566 ± 0.0670.787 ± 0.0370.349 ± 0.0440.185 ± 0.025
Ours (V2A)43.2 ± 5.81716.9 ± 1.87746.1 ± 5.0410.591 ± 0.0620.800 ± 0.0340.527 ± 0.0410.261 ± 0.025
Ours36.6 ± 5.14016.9 ± 1.81540.7 ± 4.2980.647 ± 0.0540.835 ± 0.0290.560 ± 0.0440.274 ± 0.025

The confidence intervals, as indicated in this table above, confirm that our improvements are statistically significant across metrics, with non-overlapping intervals between our method and all baselines on the key metrics (e.g. CC, AUC, Angular, CLAP-A and CLAP-T).

This is not enough to demonstrate the robustness and generality of a complex training-free pipeline across diverse scenes, source types, and acoustic conditions. Broader quantitative evaluation is needed.

Qualitative breadth. Beyond our quantitative benchmark, we demonstrate qualitative results across diverse scene types, layouts, and source types on the anonymous project webpage, covering vehicles, humans, animals, aircraft, and musical instruments.

User study. In addition to objective metrics, our user study with 50 participants across 9 scenes further corroborates the quantitative findings, with strong preference rates (94.0% over OmniAudio, 66.4% over Sonic4D).

Reviewer_a7q7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Video-to-4DAVScene, a new task that aims to convert a monocular video into an explorable, time-varying 4D audio-visual scene, and proposes SonicVista, a training-free framework for this task. The method first constructs a decomposed 4D visual representation consisting of a generated panoramic background and dynamic foreground objects with explicit world-space trajectories. It then generates a dynamic spatial sound field containing ambient audio, static background sources, and object-centric dynamic source tracks.

Critical strengths
  1. Important and novel problem formulation. The paper goes beyond conventional video-to-audio generation and static 3D audio-visual scenes by considering dynamic geometry, moving sound sources, and arbitrary listener trajectories in a unified setting. This is a timely and potentially impactful direction for immersive scene generation.

  2. Well-motivated scene and audio representations. The separation among world, camera, and object spaces is conceptually clean. In particular, representing dynamic source signals in object space is appropriate for persistent source identity and listener-dependent re-rendering.

  3. Interesting diffusion synchronization mechanism. The bidirectional coupling between the camera-space mixture and object-space source tracks is intuitive and technically interesting. The forward rendering constraint encourages the object tracks to explain the global mixture, while the reverse path transfers video-aligned semantic and temporal information back to individual sources.

Critical weaknesses
  1. Visual integration and immersion. The inserted foreground objects sometimes appear visually detached from the surrounding environment, particularly because their lighting, shadows, and overall appearance are inconsistent with the background. Could the authors discuss potential solutions for improving foreground-background harmonization and making the generated scene more immersive.

  2. Scalability to many sound sources. How does the method scale when a scene contains many dynamic sound-producing objects? What is the maximum number of object-level audio tracks that can be generated and synchronized simultaneously under the current implementation? It would be helpful to report the computational and memory costs as the number of sound sources increases, as well as whether audio quality or source separation degrades in crowded scenes.

  3. Sensitivity to visual reconstruction errors. Since spatial audio rendering depends on the reconstructed geometry and object trajectories, how sensitive are the audio results to errors in depth, camera poses, object tracking, and foreground-background alignment? An oracle-geometry or perturbed-trajectory experiment would help separate audio-generation errors from visual-reconstruction errors.

Questions for authors
  1. How long does it take to complete the entire offline processing pipeline?
8 comments

1. Visual integration and immersion. The inserted foreground objects sometimes appear visually detached from the surrounding environment, particularly because their lighting, shadows, and overall appearance are inconsistent with the background. Could the authors discuss potential solutions for improving foreground-background harmonization and making the generated scene more immersive.

We thank the reviewer for this suggestion and agree that inconsistencies in lighting, shadows, and appearance between foreground objects and the background can reduce visual immersion. Following this suggestion, we additionally explored DiffusionRenderer (Cosmos version)[Liang et al. 2025] as an offline relighting step to improve foreground–background harmonization. Specifically, the inverse renderer first predicts per-pixel G-buffers (base color, normal, depth, roughness, metallic) of the composited clip. The forward renderer then synthesizes the relit video conditioned on these G-buffers and an environment map obtained by rendering a background-only panorama from our reconstructed 3D Gaussians at each camera position. Qualitative examples are provided in Appendix Fig. 7 and on the project webpage.

Our current demonstrations focus on short clips, and this additional relighting step is not real-time. Recent work such as HorizonRelight [Yang et al. 2026] improves temporal consistency for longer videos through cross-chunk latent propagation, offering a promising direction for extending this approach. However, its implementation is not publicly available at the time of this rebuttal.

Achieving reliable foreground–background harmonization across time and viewpoints remains challenging for general scenes reconstructed from in-the-wild RGB videos. We view incorporating such an efficient harmonization stage that benefits from the most recent advances while preserving temporal and multi-view consistency as an important direction for future work.

2. Scalability to many sound sources. How does the method scale when a scene contains many dynamic sound-producing objects? What is the maximum number of object-level audio tracks that can be generated and synchronized simultaneously under the current implementation? It would be helpful to report the computational and memory costs as the number of sound sources increases, as well as whether audio quality or source separation degrades in crowded scenes.

We additionally demonstrate a three-source musical ensemble scene, featuring three musicians playing three instruments, on the updated anonymous project webpage (https://sonicvista.github.io/). We note that, in practice, it is challenging to find in-the-wild videos with many distinct, mostly visible foreground objects; heavily occluded or out-of-frame objects remain an open problem for existing visual methods. This example uses a simplified visual representation: Lift4D [Litman et al. 2026], our current instantiation for objects with general deformable motion, struggles to faithfully reconstruct fine-grained human–object interactions (HOI), particularly hand and finger movements during instrument playing. We therefore use TeHOR [Nam et al. 2026] to reconstruct a static 3D Gaussian representation of each musician–instrument pair from a single video frame, keeping their poses and positions fixed throughout the sequence. This example serves as a proof of concept for three-source audio generation and synchronization under a static visual approximation.

Reliable reconstruction of dynamic HOI from monocular video remains an active research direction. This limitation primarily concerns visual reconstruction and is largely orthogonal to our core contribution of dual-space audio diffusion synchronization. Very recent work such as HORIA [Nam et al. 2026] demonstrates progress in reconstructing temporally coherent human–object Gaussians, although fine-grained hand articulation remains challenging. Advances in these visual backbones could enable more faithful reconstruction of such scenes within our framework.


To characterize computational scaling, we conduct a new case study that inserts 1-10 repeated sources into the 8-second Porsche scene and measure runtime and memory (Figure 8a in updated Appendix):

  • Time scales linearly: Preprocessing, foreground generation, FG-BG alignment, and audio generation/synchronization are executed per-object, while background generation remains a fixed scene-level cost.
  • Memory remains approximately constant: We offload latent representations to the CPU between source-specific generation passes to strictly bound peak VRAM usage.

To systematically evaluate audio quality degradation in crowded environments, we perform a new analysis on an 8-second real-world YouTube video (https://youtu.be/qlJ8frL63Es 3:30-3:38) featuring a 5-source jazz performance (piano, saxophone, drums, double bass, and guitar). As illustrated in Figure 8b, we progressively increase the number of active object branches N and measure semantic alignment (CLAP-T). We observe two key phenomena:

  • General Degradation from Source Interference: The average semantic quality gradually drops as the scene becomes more crowded. We suspect this is primarily bounded by SAM-Audio's separation capacity in the wild and HunyuanVideo-Foleys’s capability of generating multi-source audio, which naturally degrades when trying to isolate multiple stems from a highly overlapped acoustic mixture.
  • Track-Specific Robustness: We observe this degradation is highly non-uniform. The most acoustically distinct tracks, such as the drums (transient) and double bass (low-frequency), exhibit strong robustness, maintaining essentially the same high semantic scores (CLAP-T ~0.45) even at maximum capacity (N=5). The overall performance drop is instead dominated by instruments with heavily overlapping harmonic structures (e.g., piano and saxophone) that suffer from cross-track leakage during separation.

3. Sensitivity to visual reconstruction errors. Since spatial audio rendering depends on the reconstructed geometry and object trajectories, how sensitive are the audio results to errors in depth, camera poses, object tracking, and foreground-background alignment? An oracle-geometry or perturbed-trajectory experiment would help separate audio-generation errors from visual-reconstruction errors.

We conduct system perturbation experiments to evaluate how upstream visual errors affect final audio quality. Results are summarized below:

AblationCCAUCazimuthelevationangularCLAP-ACLAP-T
Ours0.6470.83536.616.940.70.5600.274
Depth 50%0.6350.83037.117.541.40.5420.269
Mask Boundary ±\pm 30px0.6370.83037.416.941.50.5580.266
Camera Pose 25 deg0.6230.82536.719.842.90.5560.273
Mask drop 50%0.6470.83536.616.940.70.5590.274
Mask first 50%0.6430.83237.217.041.10.5470.271
FG-BG scale-only0.6460.83536.716.940.80.5580.273
FG-BG drop 10% contact frame0.6470.83536.616.940.70.5600.274
Track σ 1.00.6390.83136.918.041.40.5560.270
VLM caption drop 25%0.6150.81740.417.043.40.5490.264

And we find that:

  1. Geometric perturbations (depth, masks, camera poses) cause only modest degradation. Adding 50% noise to depth maps or ±\pm 30px boundary perturbation to masks degrades CC by only 0.010-0.012 and angular error by less than 1 degree. Even severe camera pose perturbations of +-25 deg increase angular error by only 2.2 degree. We observe that this is because the upstream foundation models in our pipeline (e.g. FoundationPose) absorbs upstream noise rather than amplifying it.
  2. Mask dropping and FG-BG alignment are highly robust. Dropping 50% of mask frames or removing local displacement correction (scale-only alignment) has negligible impact on audio metrics. Human scenes are largely unaffected by mask perturbations, since GEM recovers SMPL-X motion without requiring segmentation masks. For rigid-body scenes, these results show that the pipeline gracefully handles incomplete or imperfect visual inputs through interpolation and mask-aware extrapolation.
  3. Trajectory noise has moderate impact. We test adding Gaussian noise (σ=1.0 in world unit) to intermediate object trajectories causes a noticeable but bounded degradation (CC: 0.647 to 0.639, angular: 40.7 to 41.4 degree), confirming that the audio rendering is sensitive to object position but not catastrophically so. Combining with geometric perturbation results, this result double confirms that upstream foundation models absorb geometric (spatial) perturbations.
  4. VLM caption quality matters most for semantic metrics. Randomly dropping 25% of VLM labels causes the largest degradation in CLAP scores (0.560 to 0.549), reflecting the importance of accurate semantic descriptions for audio generation quality. However, we note that hallucinated VLM labels are rejected by SAM3 during segmentation, we did not observe false-positive cases in practice.

Overall, these results demonstrate that our training-free pipeline built on foundation models is robust to cascaded errors. Unlike small, task-specific models, the large foundation models we employ (for geometry, segmentation, and VLM captioning) are inherently tolerant of noisy inputs, and errors are attenuated rather than amplified through the pipeline. We will add these new analyses into the final version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revision

We thank all reviewers for their valuable comments and constructive feedback. We address each concern in our responses to the individual reviewers and summarize the key revisions in the global response here, along with the revision marked in blue in our newly uploaded manuscript:

  1. New robustness analysis (a7q7-W3, kG7R-W1). We conduct systematic perturbation experiments injecting noise into depth, camera poses, segmentation masks, foreground–background alignment, object trajectories, and VLM captions. Results (Table R1) show that our pipeline, built on foundation models, is robust to upstream errors, where performance degrades gracefully rather than catastrophically.
  2. New scalability experiments (a7q7-W2). We add three complementary experiments: (i) a three-source musical instrument scene; (ii) a computational scaling study with 1-10 sources, showing approximately linear runtime scaling and constant peak GPU memory usage; and (iii) an audio quality study with 1-5 active sources in a jazz performance, measuring how semantic alignment (CLAP-T) changes as the number of sources increases.
  3. New foreground-background harmonization extension (a7q7-W1). We explore DiffusionRenderer (Cosmos) as an offline relighting step and provide qualitative examples.
  4. New temporal alignment metric (yDCN-W4). We report an envelope-correlation metric (Env-r) that directly measures temporal/energy alignment, where our method achieves 0.271 vs. 0.207 for Sonic4D with >90% confidence.
  5. Confidence intervals (kG7R-W3). We report 95% confidence intervals for all methods, confirming statistically significant improvements.
  6. Pipeline timing (a7q7-Q1). We report end-to-end offline processing times for three representative scenes.
  7. Updated supplementary materials (kG7R-W4, yDCN-W1). We add input audio to supplementary videos and provide interactive visualizations of the R and Q operators on the anonymous website: https://sonicvista.github.io/

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.