acceptodds
Under review as a conference paper at ICLR 2027

WorldSonus: Bringing Sound to Worlds

Abstract

Recent advances in world models have enabled increasingly realistic visual synthesis. However, these generated environments remain largely silent. Bringing sound to world models poses three core challenges: **real-time generation** to keep pace with interactive video streams, **interactive control** to respond to mid-stream sound instructions, and **spatially aligned stereo** to reflect scene geometry and camera motion. To address these demands, we introduce WorldSonus, an interactive video-to-audio framework designed for real-time spatial sound synthesis in world models. For real-time generation, WorldSonus employs a streaming causal autoregressive diffusion architecture that synthesizes audio chunks at a low real-time factor (RTF) of . For interactive control, we incorporate an audio-centric captioning pipeline with chunk-indexed prompt scheduling, enabling dynamic manipulation of sound events during generation. For spatial alignment, we leverage high-quality stereo supervision curated from diverse stereo and ambisonic data. Extensive experiments demonstrate that while tailored for world models, WorldSonus generalizes effectively to open-domain video-to-audio benchmarks, matching or outperforming state-of-the-art bidirectional models in both acoustic quality and spatial alignment.

Primary area:
generative models
Keywords:
Audio Generation, Video to Audio Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nShfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes WorldSonus, a video-to-audio system that supports causal streaming generation, dynamic text instructions, and stereo output. The method combines a bounded-context autoregressive Transformer, a rectified-flow generation head, dual-timescale visual conditioning, and a ShiftNCE temporal synchronization objective during the training phase, while establishing spatial supervision through natural stereo filtering and panoramic ambisonic data conversion. Experiments demonstrate high audio quality, strong left-right channel matching performance, and the text-switching effect under fixed visual input.

Critical strengths

1.The research problem addressed is of practical significance. Currently, streaming video-to-audio generation lacks applications that effectively combine dynamic instructions with stereo output. 2.The methodology is clearly articulated, logically sound, and well-written. 3. The experiments are clearly presented, and the results provide strong and convincing support for the proposed approach.

Critical weaknesses

See Questions.

Questions for authors
  1. How effective is the method's dynamic control capability? The control via dynamic text instructions should not be limited to improvements in semantics and CLAP similarity. For example: 1.a. Can you provide additional examples demonstrating separate dynamic control for on-screen and off-screen sounds? 1.b. Is it possible to simultaneously apply different controls to on-screen and off-screen sounds? 1.c. What is the time interval (or resolution) for dynamic control? Can the model handle frequent semantic changes within a short period?
  2. I would appreciate some discussion on the interplay and control capabilities between dynamic instructions and visual information. For example: 2.a. What happens when there is a conflict between the dynamic instructions and the visual information? 2.b. If no dynamic instruction is provided and the video frame is simply flipped (e.g., horizontally), will the stereo output flip accordingly?
  3. Data quality significantly impacts V2A generation quality. Although a detailed data construction pipeline is provided, is there any analysis and evaluation of the data and annotation quality? Since these attributes directly form the core supervision, a small-scale independent human validation is needed, and it should be explicitly clarified whether the evaluation prompts were generated using reference audio.
  4. Why does the removal of the flow head and DINO components (as in the "w/o flow-head, DINO" ablation) have such a substantial impact on quality?
  5. (Minor) Could you compare your method with MMAudio and HunyuanVideo-Foley? I am curious why these two well-known V2A methods were overlooked. If time is limited, it is acceptable to omit this comparison.
3 comments

Q1. Dynamic control of on-screen and off-screen sounds

Yes. Our supplementary examples demonstrate separate control and simultaneous control of on-screen and off-screen sounds.

Prompts can be updated every 100 ms. Our quantitative Switch/Hold evaluation switches at 5 s, while the additional example switches prompts every 0.5 s.

Q2. Interaction between text instructions and visual information

The model has no fixed priority when text and video conflict. Depending on the case, it follows the visible event, follows the instruction, or produces a mixture. The matched and conflicting prompt examples illustrate these behaviors.

In the horizontal flip examples, we use no text instructions, hold the seed fixed, and flip only the video. The generated left/right sound placement changes with the flipped visual input. These examples demonstrate spatial responsiveness without implying exact waveform symmetry.

Q3. Human validation and evaluation prompts

We audited 100 samples from VGGSound and AudioSet, equally divided between retained and rejected samples. Excluding 27 samples rated as medium quality, filtering decisions agreed with human judgments of high or low quality in 84.9% of cases (62/73).

Among captions for the 50 retained samples, 82% correctly described the main audible events. The remaining 18% were partially correct, and none were entirely incorrect. Errors in descriptions of timing and visible source positions occurred in 4% and 2%, respectively, with 78% passing all three checks. These checks assess errors in descriptions provided, rather than requiring every caption to describe timing and position. We will report these audit results in the revision.

Yes, the original evaluation prompts used reference audio alongside video. We therefore added evaluation without text prompts on all five splits, achieving the lowest FAD on four of the five splits among the original comparison models. We will explicitly describe in the main paper how the evaluation prompts were generated using both video and reference audio, and clearly distinguish the two evaluation settings.

Q4. Why removing the direct DINO path has a large effect

This ablation removes only the direct DINO conditioning path into the flow head while the flow head and DINO conditioning through the AR backbone remain.

The AR summary captures historical and semantic context, while the direct path supplies visual details aligned with individual frames for local audio generation. Removing it forces all visual information through the summary for each chunk, potentially losing fine temporal and event details. We attribute the FAD increase from 2.42 to 15.82 in the Interactive 10 s ablation using checkpoints from pretraining to this information bottleneck.

Q5. Comparisons with MMAudio and HunyuanVideo-Foley

WorldSonus supports streaming stereo generation. Our main comparison therefore focused on methods that support at least one of these capabilities. MMAudio and HunyuanVideo-Foley generate mono audio offline, so we did not include them in the original comparison. Following your suggestion, we have added comparisons on all five test splits using the same text prompts as in the main table. The results are shown below and in the supplementary comparison tables. WorldSonus achieves the lowest FAD in all five splits. IB and CLAP are multiplied by 100, following the main paper.

DatasetModelFAD↓FD-PaSST↓FD-OpenL3↓KL↓IB↑CLAP↑DeSync↓
VGGSound 5 sMMAudio2.5989.6527.141.5533.3445.890.500
VGGSound 5 sHunyuanVideo-Foley2.66120.0244.921.7730.6036.860.599
VGGSound 5 sWorldSonus1.73159.7439.031.4827.0638.350.687
VGGSound 10 sMMAudio2.80103.6628.931.3135.5941.490.415
VGGSound 10 sHunyuanVideo-Foley3.60123.8033.621.4734.0336.680.541
VGGSound 10 sWorldSonus1.79167.5241.901.3228.8236.750.687
Interactive 5 sMMAudio5.32133.2449.751.4330.3446.460.669
Interactive 5 sHunyuanVideo-Foley2.87139.0659.601.2934.2735.440.617
Interactive 5 sWorldSonus2.68163.7440.991.2926.4835.900.831
Interactive 10 sMMAudio4.96202.4056.811.3332.4442.320.602
Interactive 10 sHunyuanVideo-Foley4.38189.2558.941.1336.7337.000.566
Interactive 10 sWorldSonus2.62262.2542.911.3028.6334.220.820
Interactive 30 sMMAudio4.36231.7155.621.5432.8643.660.772
Interactive 30 sHunyuanVideo-Foley3.46217.2562.421.3336.4338.310.776
Interactive 30 sWorldSonus2.03253.2126.671.2222.9334.210.867

Thank you for the response and the additional results. The control examples, human data audit, video-only evaluation, and comparisons with MMAudio and HunyuanVideo-Foley address my main questions. The clarification about the direct DINO-to-flow path also resolves my concern about the ablation. I maintain my Weak Accept rating. Optional additional checks. If feasible, prompts generated from video alone would help isolate the benefit of access to reference audio. A small controlled test that changes only the off-screen sound instruction and checks whether the on-screen sound is preserved would also strengthen the evidence for independent control.

Reviewer_4uRbOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents an interactive video-to-audio framework, WorldSonus. WorldSonus employs a streaming AR-diffusion architecture, emitting 100 ms audio chunks. The model is trained on time-varying prompt schedules, combined with chunk-level cross-attention, so the model is capable of dynamically revising text instructions for sounds. Also, the model is trained using curated stereo and ambisonic data, which enables it to generate high-quality stereo audio. Experimental results show that WorldSonus performs even better than offline bidirectional baselines overall and supports robust interactive control.

Critical strengths
  1. The proposed model performs very well. Quantitative results show that it performs even better than state-of-the-art models such as AudioX and PrismAudio. Qualitative examples (included in the supplementary material) also demonstrate its great performance.
  2. The model supports many aspects/dimensions of audio: spatial audio, long-term generation, and interactive textual control.
Critical weaknesses
  1. The paper is technically strong and well engineered, but its core novelty appears to be mainly in integrating existing ingredients into a streaming V2A system rather than introducing a broadly generalizable modeling principle. ShiftNCE and two-timescale visual conditioning are useful for this setting, but they read more as task-specific engineering solutions than as conceptual advances. I find the contribution level somewhat below the bar despite the strong empirical results.
  2. The paper frames the system as real-time, and the reported RTF is indeed below 1. However, the effective conditioning/control granularity is 100 ms: each audio chunk corresponds to three video frames, and newly visible events can only affect the generated audio at chunk boundaries. This may be limiting for interactive scenarios requiring frame-level responsiveness or very tight audiovisual synchronization.
Questions for authors
  1. S-FD$_O$ based on OpenL3 is considered to be a metric that assesses both audio quality and spatial intensity in the literature. For instance, the Stable Audio series papers utilize it for evaluating the quality of generated audio. I am not convinced that the differences in S-FD$_O$ are derived only from spatial alignment. Indeed, the user-study result is not correlated with S-FD$_O$. I would appreciate it if the authors could provide their thoughts on this.
  2. Related to the previous question, Appendix D.5 gives a qualitative diagnostic argument, but I would appreciate quantitative details of this diagnostic experiment, e.g., the ITD-MSE values for mono duplication and model outputs.
  3. The explanation of BiasSkill (Appendix E) is not clear to me. I would appreciate it if the authors could provide a more detailed explanation. For example, Wi,sW_{i,s} and y^i,w\hat{y}_{i,w} are not defined. Also, ⊭\nvDash is used, but its meaning (indicator function?) is only implied. (One minor comment related to this is that, since I could not find BiasSkill in prior work, it appears to be originally designed by the authors. If this is correct, the paper should explicitly state this in the main body, e.g., with "we introduce" or "we design".)
  4. Regarding how to use Synchformer features, the paper says, "possibly reflecting a mismatch between its 640 ms feature window and our 100 ms streaming chunks" in Line 430. It would be appreciated if the authors could provide an intuition for why feeding features into the model suffers from that mismatch whereas ShiftNCE does not, although both use the same features.
5 comments

W1. Methodological contributions

Our contribution lies in how persistent context, local rendering, and temporal supervision are separated under streaming constraints. The AR backbone maintains compact context at the chunk level, while a separate visual path aligned with individual frames supplies the flow head with local evidence without compressing it into the chunk summary. ShiftNCE trains the recurrent state to distinguish aligned from temporally shifted teacher targets without requiring the synchronization encoder at inference.

The controlled ablations demonstrate why these choices matter. Removing the direct visual path increases FAD from 2.42 to 15.82. Removing ShiftNCE increases DeSync from 0.831 to 1.067. Directly injecting synchronization features gives FAD 2.81 and DeSync 1.043, compared with 2.42 and 0.831 for our design. These comparisons use checkpoints from pretraining and the same Interactive 10 s evaluation set.

Furthermore, paired Switch/Hold interventions keep visual inputs and seeds fixed and show positive gains from prompt switching on both datasets. They test whether instructions actively steer subsequent audio rather than merely accompany visual transitions.

W2. Control granularity and audiovisual synchronization

However, the effective conditioning/control granularity is 100 ms: each audio chunk corresponds to three video frames, and newly visible events can only affect the generated audio at chunk boundaries.

The 100 ms chunk size defines the interval for generation and prompt updates, not the audiovisual alignment resolution. For each chunk, we first receive the corresponding 100 ms of video. The flow head then uses all three visual tokens aligned with individual frames to generate the audio, allowing sound onsets to align with visual events within that interval, rather than only at chunk boundaries.

The boundary restriction applies to incorporating new prompts and releasing generated audio chunks. It does not restrict where sound events can occur within a chunk. Thus, collecting and processing a chunk introduces response latency, but does not impose a 100 ms lower bound on audiovisual synchronization error.

Q1. What S-FDₒ measures

We agree that S-FDₒ reflects both audio quality and stereo characteristics, so its differences cannot be attributed solely to spatial alignment. Its ranking therefore need not match human judgments of spatial alignment.

This is why we also report BiasSkill, which directly measures agreement in left/right energy dominance with reference audio rather than using general audio embeddings. Our audiovisual user study provides additional evidence of perceived spatial alignment. We will distinguish these roles explicitly.

Q2. Quantitative ITD-MSE results

We evaluate all 4,096 paired clips from Interactive 5 s, with audio resampled to 16 kHz. ITD estimation follows StereoCRW, using a 100 ms stride, and ITD-MSE aggregation follows Both Ears Wide Open (BEWO). The dual-mono control duplicates the reference mid-channel, L′=R′=(L+R)/2L'=R'=(L+R)/2.

Model / controlITD-MSE ↓
GT compared with itself0.00
GT mid-channel duplicated to both channels30.71
V-AURA30.71
AudioX31.66
WorldSonus, 150k checkpoint35.50
ThinkSound40.75
PrismAudio57.65

Monophonic V-AURA scores best among the generated outputs, almost identically to the dual-mono control derived from GT. Both outperform all evaluated stereo generators under ITD-MSE despite having no stereo separation.

These results support the concern discussed in Appendix D.5: in the production stereo used in our data, channel differences do not reliably reflect physical delays in sound arrival. Meaningful stereo separation therefore need not correspond to a nonzero ITD. Thus, a low ITD-MSE does not necessarily indicate successful stereo generation in this setting. We therefore omit it from our spatial evaluation and instead report S-FDₒ and BiasSkill as complementary measures of stereo distribution and left/right channel agreement.

Q3. BiasSkill definitions

Yes, we introduce BiasSkill in this work, and we will state this explicitly in the main text. We will also clarify the notation and evaluation procedure in Appendix E.

For each window, channel dominance is measured by:

d=ER−ELER+EL+10−12.d = \frac{E_R-E_L}{E_R+E_L+10^{-12}}.

Eligibility is determined solely by the reference audio:

∣dGT∣≥0.3,RMSGT≥10−4.|d_{\mathrm{GT}}| \geq 0.3, \qquad \mathrm{RMS}_{\mathrm{GT}} \geq 10^{-4}.

  • Wi,sW_{i,s} denotes the eligible windows in clip ii whose reference audio is dominant on side ss, where ss is L or R.
  • CsC_s denotes the clips containing at least one such window. A clip may belong to both side-specific sets.
  • y^i,w\hat{y}_{i,w} denotes the generated label: L, R, or invalid. It is invalid if the generated RMS is below 0.001 or the absolute channel dominance is below 0.1.
  • 1[⋅]\mathbf{1}[\cdot] is the indicator function: 1 when the condition holds and 0 otherwise.

Outputs that are nearly mono, insufficiently active, or biased toward the opposite side count as incorrect and remain in the denominator.

For each reference side, we first average correctness over eligible windows within each clip, then equally across clips containing such windows:

As=1∣Cs∣∑i∈Cs(1∣Wi,s∣∑w∈Wi,s1[y^i,w=s]).A_s = \frac{1}{|C_s|} \sum_{i \in C_s} \left( \frac{1}{|W_{i,s}|} \sum_{w \in W_{i,s}} \mathbf{1}[\hat{y}_{i,w}=s] \right).

We then give the two sides equal weight:

Abal=AL+AR2.A_{\mathrm{bal}} = \frac{A_L+A_R}{2}.

We estimate a model-specific null score μ\mu using 10,000 permutations of whole generated clips, with no clip paired with its original reference and with window positions preserved. Models evaluated on the same set share the permutations but obtain their own null score. We define:

BiasSkill=100×Abal−μ1−μ.\mathrm{BiasSkill} = 100 \times \frac{A_{\mathrm{bal}}-\mu}{1-\mu}.

A score of 0% matches the model's shuffled baseline, 100% indicates perfect agreement on eligible windows, and negative values indicate performance below that baseline.

Q4. Why ShiftNCE differs from direct feature injection

The difference is the learning task, not the 640 ms window alone. With direct injection, the generator must learn through the audio generation objective which information in the representation of the longer window is relevant to the current 100 ms chunk. Injection itself does not explicitly supervise this temporal correspondence.

ShiftNCE provides a direct objective for temporal discrimination. Teacher features are extracted at 100 ms strides, and the AR state is trained to distinguish the aligned window from temporally shifted alternatives, including neighboring windows. Their shared context makes the contrast focus on cues that distinguish temporal positions rather than common scene content. Highly similar targets are filtered to reduce ambiguous negatives.

The ablation supports this design, although it does not establish window mismatch alone as the cause of direct injection’s poorer performance.

Thank you for the detailed response. I have also read the other reviewers’ comments and the authors’ responses to them. Overall, I found the rebuttal helpful and constructive.

Since manuscript revisions are possible during the rebuttal/discussion period, I would like to see the key clarifications and additional results from the responses incorporated into the paper. If this is done clearly, I would be willing to revise my score from 2 to 3. Of course, if the authors have concerns, constraints, or a different view on any of the requested revisions below, I would also be happy to hear them.

In particular, I would appreciate the following updates.


1. Effective temporal granularity and latency

I understand from the response that the 100 ms chunk size defines the interval for generation and prompt updates, while the flow head can still place audio events within the chunk using frame-aligned visual tokens. This distinction is helpful and should be made explicit somewhere in the paper.

I also think some high-level statements could be slightly misread as implying frame-by-frame real-time generation. For example, the abstract says "real-time generation to keep pace with interactive video streams", and the introduction says "audio that seamlessly tracks an evolving visual stream". The contribution phrase "real-time causal generation" may also benefit from a short clarification. I do not think these statements are wrong, but I suggest revising or qualifying them so that readers do not interpret the system as responding at every video frame.

I would also appreciate adding a limitation noting that shorter chunks such as 33 ms may be applicable to scenarios requiring finer temporal granularity, but the current results show a quality trade-off compared with the 100 ms setting.

2. BiasSkill and ITD-MSE clarifications

I think the paper would be easier to understand if it explicitly states that BiasSkill is introduced in this work, defines the notation carefully, and gives an intuitive explanation of why simple left/right accuracy is insufficient. My understanding is that BiasSkill measures agreement with the reference in left/right energy dominance after subtracting a model-specific channel-bias baseline estimated by permutation. This intuition would be useful for readers.

The quantitative ITD-MSE diagnostic is also useful and should be included. I agree that the dual-mono and V-AURA results suggest that ITD-MSE is poorly aligned with meaningful stereo separation in this benchmark. Given that the user study also supports the spatial alignment quality of WorldSonus, I think this diagnostic provides a reasonable justification for not using ITD-MSE as a primary spatial metric here. I would recommend making this intuition explicit in the paper.

3. Explanation of ShiftNCE versus direct synchronization-feature injection

I found the response to Q4 convincing. My understanding is that the main difference is the learning objective rather than the feature window size alone. Direct injection may provide useful synchronization features, but it does not explicitly encourage discrimination between the current 100 ms chunk and nearby shifted chunks within a largely overlapping 640 ms context. In contrast, ShiftNCE uses a contrastive objective to emphasize temporally discriminative cues and does not require the synchronization encoder at inference time.

This explanation was clearer to me than the current manuscript text. I would appreciate revising the relevant paragraph to make this distinction more explicit.

4. Additional clarifications from responses to the other reviewers

I also think it would be helpful to incorporate useful clarifications from the broader discussion with the other reviewers. For example:

  • clearly distinguish the original evaluation with reference-audio-informed prompts from the new video-only evaluation
  • include the end-to-end latency breakdown and the concurrent deployment clarification
  • include the explanation of why removing the direct DINO path has such a large impact, namely that it removes frame-aligned local visual evidence to the flow head and forces visual information through the chunk-level AR summary

Overall, the rebuttal addressed most of my technical questions. My main remaining concern is still the overall contribution level, but the system is practically important and empirically strong. If the above clarifications and additional results are reflected in the revised manuscript, I am inclined to update my recommendation to weak accept. I would also welcome any response from the authors if they disagree with some of these suggested revisions or think there is a better way to address the underlying concerns.

Reviewer_HtQtOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces WorldSonus, a causal streaming video-to-audio generation framework. WorldSonus can produce temporally synchronized and spatially consistent stereo audio for interactive visual environments. The pipeline combines an autoregressive diffusion model with two-timescale visual conditioning and dynamic text conditioning to support real-time audio generation and mid-stream prompt updates, trained under a contrastive temporal alignment objective. This model can generate 100 ms audio chunks with low inference latency while maintaining temporal context. Experiments on VGGSound and interactive-video benchmarks demonstrate strong audio quality, competitive temporal and spatial alignment, and effective text-based controllability compared to existing video-to-audio generation methods.

Critical strengths
  1. Clear motivation and the system design are practical: I appreciate that the paper addresses the challenging setting of causal, real-time audio generation for interactive world models, which is not adequately handled by existing offline V2A methods. The proposed architecture can generate 100 ms audio chunks with ~40 ms computation time, providing promising evidence of its practical feasibility.
  2. Strong empirical results with informative ablations: The improvements in audio quality are impressive, particularly considering that WorldSonus operates causally while several competing methods have access to future video frames. The evaluation covers multiple benchmarks, and the ablation studies provide convincing evidence for the effectiveness of the proposed visual conditioning and temporal alignment components.
Critical weaknesses
  1. I am concerned about the use of audio-centric captions for evaluation. The paper (Section 4) states that the paired training data captions are generated using both video and audio, but it is unclear whether the evaluation prompts also incorporate information from the ground-truth audio. If so, this would provide privileged acoustic information unavailable in the intended application, potentially overstating the model's video-to-audio generation capabilities. I would like to see a clarification of the evaluation protocol and results using video-only conditioning. Currently, this is my biggest concern.
  2. Some important metrics about end-to-end latency are not reported: Does the 41.2 ms latency include DINOv3 feature extraction, autoregressive computation, all flow-sampling steps, and waveform decoding? What is the end-to-end delay from receiving a live visual event to playing the corresponding audio? Can the system run concurrently with a real-time visual world model on shared hardware? Any actual deployment results?
  3. The paper is still using the causal VAE from Soundreactor, which is a significant bottleneck for audio quality. And in the demo video, WorldSonus has a hollow, shrill sound similar to SoundReactor. Why still use this VAE? Did the authors explore other solutions/archituctures for streaming audio VAE?
Questions for authors

Please see Critical Weaknesses: W1 as Q1, W2 as Q2, W3 as Q3.

4 comments

W1. Prompts informed by reference audio and evaluation without text

Yes, the evaluation prompts in the original setting with both video and text were generated using both the video and its reference audio. These captions describe acoustic details, including off-screen events, that may not be recoverable from video alone. Therefore, the original results evaluate generation conditioned on both video and text rather than video alone.

To directly address this concern, we conducted additional evaluations using video without text prompts. The complete results for all five test splits are provided below and in the supplementary results. We will include these tables in the main paper. IB is multiplied by 100, following the main paper.

VGGSound 5 s

ModelFAD ↓FD-P ↓FD-O ↓KL ↓IB ↑DeSync ↓
V-AURA4.06285.6752.601.9626.481.287
ThinkSound2.65131.7634.812.0623.380.478
AudioX3.79144.1155.402.2524.071.043
PrismAudio7.94228.8564.872.8517.970.524
WorldSonus1.82149.0241.392.0326.410.651

VGGSound 10 s

ModelFAD ↓FD-P ↓FD-O ↓KL ↓IB ↑DeSync ↓
V-AURA4.35318.1359.361.9526.731.228
ThinkSound1.64134.0133.531.8426.340.507
AudioX3.30167.9152.502.0526.700.977
PrismAudio6.35225.2559.652.6021.420.481
WorldSonus1.94185.5644.391.9827.690.634

Interactive 5 s

ModelFAD ↓FD-P ↓FD-O ↓KL ↓IB ↑DeSync ↓
V-AURA8.18438.5085.511.9226.841.344
ThinkSound7.88296.6750.492.2418.300.724
AudioX9.30361.46109.852.5819.221.085
PrismAudio10.27458.1580.362.3921.220.729
WorldSonus3.15229.5439.491.7026.320.827

Interactive 10 s

ModelFAD ↓FD-P ↓FD-O ↓KL ↓IB ↑DeSync ↓
V-AURA9.75532.7488.041.9627.601.270
ThinkSound6.85341.4449.522.1520.200.740
AudioX8.56433.84110.432.4621.691.033
PrismAudio8.29509.5175.492.0023.740.692
WorldSonus3.09309.4140.731.7627.840.807

Interactive 30 s

ModelFAD ↓FD-P ↓FD-O ↓KL ↓IB ↑DeSync ↓
V-AURA13.33628.86111.152.7119.931.311
ThinkSound6.51415.4564.702.3513.770.968
AudioX12.62524.11161.773.0421.890.987
PrismAudio9.80543.8898.482.4522.210.829
WorldSonus1.69223.6329.401.3723.170.860

These results show that WorldSonus performs well without text prompts derived from reference audio. We will explicitly describe in the main paper how the evaluation prompts were generated using both video and reference audio, and report results with and without text prompts separately.

W2. End-to-end latency and concurrent deployment

Yes. The 41.21 ms includes image transfer and resizing, DINOv3 feature extraction, visual conditioning, AR computation, all 15 flow sampling steps, causal VAE decoding, and audio transfer back to the CPU.

The detailed latency breakdown is provided below. We will include this table in the main paper.

Processing stageP50 latency
Image transfer and resizing0.77 ms
DINO backbone2.51 ms
DINO feature downsampling0.26 ms
Video delta adapter0.08 ms
Video projection0.06 ms
Spatial aggregation0.22 ms
Temporal encoder1.01 ms
AR Transformer7.99 ms
Flow head: 15 denoising steps20.57 ms
Causal VAE decoding7.56 ms
Audio transfer to CPU0.06 ms
Steady-state whole-chunk wall time (P50)41.21 ms

These are independently measured P50 values and are not additive. The measurement for the whole chunk gives the processing latency in steady state, corresponding to RTF ≈ 0.412. The delay from a visual event to audio playback additionally includes frame accumulation and playback buffering.

Yes. WorldSonus runs concurrently with a real-time visual world model on the same GPU. A video of our deployment test is available on the supplementary webpage.

W3. Choice of the SoundReactor VAE and alternative solutions

We agree that the current VAE limits audio quality. We used SoundReactor because it was the only open-source solution we found that supported stateful causal stereo decoding without accessing future chunks or decoding past audio again.

We tested EAR2 and LTX-2.3 and observed better perceptual quality, but their tested decoding pipelines did not meet these streaming requirements. In particular, LTX’s final vocoder is non-causal even though its VAE includes causal components. We therefore retained SoundReactor to support streaming.

I appreciate the authors' efforts in providing the additional results. In interactive settings, WorldSonus perform the best in most metrics using video as the solo condition. And the latency breakdown is also helpful, I suggest the authors add it to the Appendix. The above address my concerns about audio-centric captions and the streaming latency.

The causal VAE quality improvement remains a underexplored topic for now.

We sincerely appreciate your positive feedback and constructive suggestions. In the revised manuscript, we have incorporated the video-only results into Table 1 and added the latency breakdown to Appendix D.3 (Table 13). These additions are highlighted in blue for your convenience.

Furthermore, we fully agree that improving causal audio VAE quality remains a critical open challenge. Closing this quality gap while preserving stateful causal decoding is a primary goal for our future work, and we have now explicitly acknowledged this limitation in Section 7.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.