acceptodds
Under review as a conference paper at ICLR 2027

Enabling immersive experience from any video

Abstract

Most videos capture only a narrow field of view and provide no spatial audio, limiting the sense of immersion they can provide. Recent video generation models can expand perspective videos into panoramic ones, but do not provide the corresponding spatial soundscape. Without spatially consistent audio, these expanded visual worlds remain incomplete. This paper presents Omnidream, a training-free framework that transforms a silent monocular video into an immersive audiovisual experience, where viewers can freely look around while sounds remain spatially aligned with the visual scene. At the core of Omnidream is an object-centric audio representation that disentangles each sound source's intrinsic audio content from its scene-dependent acoustic effects, enabling independent audio generation, physics-based simulation of propagation effects, and flexible spatial audio rendering. Experiments show improved audio-visual alignment, spatial correctness, and perceptual immersiveness over baselines. Examples are available on https://huggingface.co/spaces/CuriousAlien000/spatial-audio-360-demo.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Immersive experience, audio generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cvRWOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a pipeline that leverage VLMs, depth estimation models, semantic segmentation models, V2A models, acoustic rendering models to turn a silent video in to panoramic video with spatial video.

Critical strengths
  1. The direction is interesting and promising, and this paper is the first attempt on this problem. The examples in the huggingface link is impressive
  2. The object-centric approach is intuitive and sensible. It also provide meaningful control.
Critical weaknesses
  1. The proposed approach is a pipeline, which degrades the novelty and excitement
  2. Due to the nature of the approach being a pipeline, the comparison with other approach is not fair, for example, MMAudio is a component of the proposed pipeline, and therefore a fair comparison would be an ablation study that gradually strip down other added components and see which techniques proposed by this paper bring benefits
  3. the evaluation is only done on a narrower scope. ImserseSet contains only a smaller amount of cliips and explicitly excludes sever camera motion, jitter, etc. Not sure how this approach would work in in-the-wild video. Object centric approach is also posit a challenge as in the wild video might have too many moving piece for this approach to work well
Questions for authors

In each stage of the pipeline, a model is selected, for example, gemini for VLM, SAM2 for segmentation, AcoustiX for acoustic effect simulation, have you tried alternative models for different stages and how did they perform?

Reviewer_deVrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents OmniDream, a training-free framework that converts monocular videos into immersive 360° videos with spatial audio. It combines panoramic video generation, object-centric audio synthesis, and physics-based acoustic rendering. Experiments demonstrate improved spatial alignment and perceptual immersiveness.

Critical strengths
  1. Practical and well-motivated framework. The paper addresses an interesting application by integrating panoramic video generation with object-centric spatial audio. The explicit separation of sound sources and acoustic effects enables flexible and controllable rendering.

  2. Promising experimental results. The method achieves improved spatial correspondence and perceptual immersiveness compared with existing baselines. The authors also provide component ablations, robustness analysis, and evaluation against measured impulse responses.

Critical weaknesses
  1. Limited methodological novelty. The proposed framework shares substantial similarities with existing methods. In particular, Sonic4D (AAAI 2026) already combines 3D sound-source localization, tracking, and physics-based spatial audio rendering in a training-free framework. SonoWorld (CVPR 2026) generates panoramic 3D scenes with spatial ambisonic audio, while SEE-2-SOUND also integrates object-level sound generation with acoustic simulation. Although OmniDream extends these ideas to temporally aligned 360° video and object-specific audio tracks, its main contribution appears to be the integration of existing techniques rather than a fundamentally new algorithm. Moreover, direct comparisons with Sonic4D and SonoWorld are missing, making it difficult to assess the improvements over the most closely related approaches.

  2. Limited evidence for physics-grounded rendering. Table 4 shows only marginal improvements from the full acoustic renderer over simpler spatialization methods (overall CC: 0.50 vs. 0.49; AUC: 0.72 vs. 0.71). This suggests that much of the improvement may come from object-centric audio decomposition rather than physics-based simulation. Furthermore, the acoustic accuracy evaluation is limited to a single room, with a relatively high zero-shot RT60 error of 31.9%. The ImmerseSet spatial metrics also rely on optical-flow-based references rather than ground-truth spatial audio, potentially limiting their reliability for stationary sound sources.

Questions for authors
  1. Could the authors provide stronger evidence that physics-based acoustic rendering significantly improves perceptual realism over simpler spatialization methods?

  2. How reliable are optical-flow-based spatial metrics for stationary sound sources?

  3. Section 3.3 uses (1−α) for reflection weighting, while Appendix F defines α as reflectivity. Could the authors clarify this inconsistency?

  4. Demo Website Issue. The project demo appears to have an audio playback issue, as no sound can be heard when playing the videos on the website.

Reviewer_xfKnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents OmniDream, a training-free system that converts a silent perspective video into a panoramic video with spatial audio. It combines panoramic video generation, depth estimation, sound-source detection and tracking, and per-object audio generation. The generated tracks are then spatialized through acoustic simulation using estimated scene geometry and material properties, producing first-order ambisonic audio that responds to changes in viewing direction. The results show improvements in audiovisual alignment, spatial correspondence, and perceived immersion over the evaluated baselines.

Critical strengths
  • The per-object design fits the motivation well. Separate audio tracks and explicit source locations allow the system to place multiple sounds in scene and render them consistently as the viewer changes orientation, where replacing per-object processing with a global track lowers overall spatial CC.

  • The YT360 evaluation provides large scale results. OmniDream improves overall CC/AUC over OmniAudio with a larger CC improvement The abnlations and user study also supports the usefulness of the complete system.

Critical weaknesses
  • Related to prior work. Sonic4D already combines monocular-video-based scene generation, 3D source tracking, and spatial audio synthesis using time-varying simulated room impulse responses, while Table 1 marks Sonic4D as lacking physics-aware rendering. SonoWorld also combines panoramic expansion, object-level sound generation. Extending these ideas to synchronized panoramic video with multiple sources and visually estimated acoustic materials is useful, but it needs a clearer comparison.

  • The full acoustic renderer is notconvincing. In Table 4, per-object spatialization with distance attenuation and propagation delay already reaches 0.49 CC. The short-window gains are larger but the user study does not include this matched spatialization baseline.

  • The ImmerseSet spatial reference is derived from optical-flow magnitude and does not directly validate the simulated acoustic field. How much of the perceptual improvement comes from recovering scene acoustics?

Questions for authors
  • Renderer: Can you compare the full renderer with the strongest per-object spatialization ablation on YT360, keeping the generated source tracks, trajectories, visual outputs fixed? It is not sure whether the scene-dependent acoustic simulation is the improvement in immersion.

  • MMAudio baseline: Which MMAudio variant was used in Figure 8? Under the paper’s stated encoding, nonzero X, Y, and Z channels introduce directional components, so this is not an omnidirectional mono control.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.