- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes WorldSonus, a video-to-audio system that supports causal streaming generation, dynamic text instructions, and stereo output. The method combines a bounded-context autoregressive Transformer, a rectified-flow generation head, dual-timescale visual conditioning, and a ShiftNCE temporal synchronization objective during the training phase, while establishing spatial supervision through natural stereo filtering and panoramic ambisonic data conversion. Experiments demonstrate high audio quality, strong left-right channel matching performance, and the text-switching effect under fixed visual input.
Critical strengths
1.The research problem addressed is of practical significance. Currently, streaming video-to-audio generation lacks applications that effectively combine dynamic instructions with stereo output. 2.The methodology is clearly articulated, logically sound, and well-written. 3. The experiments are clearly presented, and the results provide strong and convincing support for the proposed approach.
Critical weaknesses
See Questions.
Questions for authors
- How effective is the method's dynamic control capability? The control via dynamic text instructions should not be limited to improvements in semantics and CLAP similarity. For example: 1.a. Can you provide additional examples demonstrating separate dynamic control for on-screen and off-screen sounds? 1.b. Is it possible to simultaneously apply different controls to on-screen and off-screen sounds? 1.c. What is the time interval (or resolution) for dynamic control? Can the model handle frequent semantic changes within a short period?
- I would appreciate some discussion on the interplay and control capabilities between dynamic instructions and visual information. For example: 2.a. What happens when there is a conflict between the dynamic instructions and the visual information? 2.b. If no dynamic instruction is provided and the video frame is simply flipped (e.g., horizontally), will the stereo output flip accordingly?
- Data quality significantly impacts V2A generation quality. Although a detailed data construction pipeline is provided, is there any analysis and evaluation of the data and annotation quality? Since these attributes directly form the core supervision, a small-scale independent human validation is needed, and it should be explicitly clarified whether the evaluation prompts were generated using reference audio.
- Why does the removal of the flow head and DINO components (as in the "w/o flow-head, DINO" ablation) have such a substantial impact on quality?
- (Minor) Could you compare your method with MMAudio and HunyuanVideo-Foley? I am curious why these two well-known V2A methods were overlooked. If time is limited, it is acceptable to omit this comparison.
3 comments
Q5. Comparisons with MMAudio and HunyuanVideo-Foley
WorldSonus supports streaming stereo generation. Our main comparison therefore focused on methods that support at least one of these capabilities. MMAudio and HunyuanVideo-Foley generate mono audio offline, so we did not include them in the original comparison. Following your suggestion, we have added comparisons on all five test splits using the same text prompts as in the main table. The results are shown below and in the supplementary comparison tables. WorldSonus achieves the lowest FAD in all five splits. IB and CLAP are multiplied by 100, following the main paper.
| Dataset | Model | FAD↓ | FD-PaSST↓ | FD-OpenL3↓ | KL↓ | IB↑ | CLAP↑ | DeSync↓ |
|---|---|---|---|---|---|---|---|---|
| VGGSound 5 s | MMAudio | 2.59 | 89.65 | 27.14 | 1.55 | 33.34 | 45.89 | 0.500 |
| VGGSound 5 s | HunyuanVideo-Foley | 2.66 | 120.02 | 44.92 | 1.77 | 30.60 | 36.86 | 0.599 |
| VGGSound 5 s | WorldSonus | 1.73 | 159.74 | 39.03 | 1.48 | 27.06 | 38.35 | 0.687 |
| VGGSound 10 s | MMAudio | 2.80 | 103.66 | 28.93 | 1.31 | 35.59 | 41.49 | 0.415 |
| VGGSound 10 s | HunyuanVideo-Foley | 3.60 | 123.80 | 33.62 | 1.47 | 34.03 | 36.68 | 0.541 |
| VGGSound 10 s | WorldSonus | 1.79 | 167.52 | 41.90 | 1.32 | 28.82 | 36.75 | 0.687 |
| Interactive 5 s | MMAudio | 5.32 | 133.24 | 49.75 | 1.43 | 30.34 | 46.46 | 0.669 |
| Interactive 5 s | HunyuanVideo-Foley | 2.87 | 139.06 | 59.60 | 1.29 | 34.27 | 35.44 | 0.617 |
| Interactive 5 s | WorldSonus | 2.68 | 163.74 | 40.99 | 1.29 | 26.48 | 35.90 | 0.831 |
| Interactive 10 s | MMAudio | 4.96 | 202.40 | 56.81 | 1.33 | 32.44 | 42.32 | 0.602 |
| Interactive 10 s | HunyuanVideo-Foley | 4.38 | 189.25 | 58.94 | 1.13 | 36.73 | 37.00 | 0.566 |
| Interactive 10 s | WorldSonus | 2.62 | 262.25 | 42.91 | 1.30 | 28.63 | 34.22 | 0.820 |
| Interactive 30 s | MMAudio | 4.36 | 231.71 | 55.62 | 1.54 | 32.86 | 43.66 | 0.772 |
| Interactive 30 s | HunyuanVideo-Foley | 3.46 | 217.25 | 62.42 | 1.33 | 36.43 | 38.31 | 0.776 |
| Interactive 30 s | WorldSonus | 2.03 | 253.21 | 26.67 | 1.22 | 22.93 | 34.21 | 0.867 |
Thank you for the response and the additional results. The control examples, human data audit, video-only evaluation, and comparisons with MMAudio and HunyuanVideo-Foley address my main questions. The clarification about the direct DINO-to-flow path also resolves my concern about the ablation. I maintain my Weak Accept rating. Optional additional checks. If feasible, prompts generated from video alone would help isolate the benefit of access to reference audio. A small controlled test that changes only the off-screen sound instruction and checks whether the on-screen sound is preserved would also strengthen the evidence for independent control.
Q1. Dynamic control of on-screen and off-screen sounds
Yes. Our supplementary examples demonstrate separate control and simultaneous control of on-screen and off-screen sounds.
Prompts can be updated every 100 ms. Our quantitative Switch/Hold evaluation switches at 5 s, while the additional example switches prompts every 0.5 s.
Q2. Interaction between text instructions and visual information
The model has no fixed priority when text and video conflict. Depending on the case, it follows the visible event, follows the instruction, or produces a mixture. The matched and conflicting prompt examples illustrate these behaviors.
In the horizontal flip examples, we use no text instructions, hold the seed fixed, and flip only the video. The generated left/right sound placement changes with the flipped visual input. These examples demonstrate spatial responsiveness without implying exact waveform symmetry.
Q3. Human validation and evaluation prompts
We audited 100 samples from VGGSound and AudioSet, equally divided between retained and rejected samples. Excluding 27 samples rated as medium quality, filtering decisions agreed with human judgments of high or low quality in 84.9% of cases (62/73).
Among captions for the 50 retained samples, 82% correctly described the main audible events. The remaining 18% were partially correct, and none were entirely incorrect. Errors in descriptions of timing and visible source positions occurred in 4% and 2%, respectively, with 78% passing all three checks. These checks assess errors in descriptions provided, rather than requiring every caption to describe timing and position. We will report these audit results in the revision.
Yes, the original evaluation prompts used reference audio alongside video. We therefore added evaluation without text prompts on all five splits, achieving the lowest FAD on four of the five splits among the original comparison models. We will explicitly describe in the main paper how the evaluation prompts were generated using both video and reference audio, and clearly distinguish the two evaluation settings.
Q4. Why removing the direct DINO path has a large effect
This ablation removes only the direct DINO conditioning path into the flow head while the flow head and DINO conditioning through the AR backbone remain.
The AR summary captures historical and semantic context, while the direct path supplies visual details aligned with individual frames for local audio generation. Removing it forces all visual information through the summary for each chunk, potentially losing fine temporal and event details. We attribute the FAD increase from 2.42 to 15.82 in the Interactive 10 s ablation using checkpoints from pretraining to this information bottleneck.