- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents HelixWorld, a real-time interactive audio-visual world model that generates video and camera-aligned stereo audio under camera and action control. Built on LTX-2.3 and trained on approximately 3,000 hours of data, it uses PRoPE for camera conditioning and AdaLN-Zero for action conditioning. Teacher distillation enables causal streaming at 24 FPS on a single H800. The authors also introduce HelixBench, comprising 1,015 cases for evaluating spatial audio consistency. HelixWorld delivers competitive but not superior visual quality on WBench and outperforms EchoWM and cascaded V2A baselines in audio quality and spatial consistency.
Critical strengths
- The paper is among the first to combine interactive world modeling with real-time video and spatial stereo audio, with applications in gaming, VR, and embodied simulation.
- Well-designed data pipeline. Stereo validation, causal audio filtering, metric pose recovery, and decoupled captions address data quality and spatial leakage. The dataset could be a valuable resource if released.
- HelixBench evaluates synchronization, semantics, audio fidelity, and spatial consistency, using visual-only spatial targets to avoid circular evaluation.
- Strong audio results. The method outperforms EchoWM and cascaded V2A baselines in spatial consistency and most audio metrics, with consistent preference in the audio-enabled user study.
Critical weaknesses
-
The term camera-grounded spatial stereo sound, repeatedly emphasized in the abstract and introduction, corresponds to the Spatial metric. However, the computation described in Appendix D.4 involves no camera poses or actions at all. This metric therefore appears to be visual-grounded rather than camera-grounded as claimed. Judging from the paper as a whole, this does not seem to be a wording error, and it poses a risk of overclaiming.
-
The ablation in Table 7 is incompletely described. Appendix B.4 explains only the DMD-only long-horizon comparison and the L_traj comparison before long-horizon tuning. The configuration of the L_traj + long-horizon row, reported as the "best overall score" (79.1), is never specified. Moreover, the final model in Table 3 (79.9, Physical 70.9) matches no row in Table 7, and its Physical score exceeds every ablated configuration (max 68.9), so the ablation does not explain the final performance. Adding long-horizon tuning on top of L_traj also lowers Consistency and Physical, contradicting the claimed "orthogonal gains."
-
Sec. 6.2 claims HelixWorld surpasses EchoWM and cascaded baselines across all dimensions including temporal sync, but Table 4 does not support this. The student's DeSync (0.5867) is worse than ThinkSound (0.4236) and PrismAudio (0.4582), which use the same video, and worse than its own teacher (0.4988). This suggests distillation degrades synchronization and undermines the claim that joint generation outperforms cascades, yet the paper does not discuss it.
-
Minor but important
(1) DreamX-World in Table 3 is cited as DreamX-Phi, which is a robotic-manipulation model.
(2) The Related Work section states that distribution matching suffers from multimodal mode collapse, causing color degradation and acoustic dropouts, and Sec. 4.2 further mentions acoustic fading over long rollouts. However, the paper provides no quantitative or qualitative audio-side evidence for these issues.
Questions for authors
Please refer to the "Weaknesses" section.
6 comments
W2: Ablation Details and Analysis in Table 7
"The configuration of the + long-horizon row, reported as the 'best overall score' (79.1), is never specified."
In Table 7, the + long-horizon row initializes long-horizon tuning from the self-forcing checkpoint trained with and its paired fake-score model. We have clarified the initialization and training setup for these ablations in Appendix B.4 (Lines 1019–1022) of the revised paper.
"Moreover, the final model in Table 3 (79.9, Physical 70.9) matches no row in Table 7, and its Physical score exceeds every ablated configuration (max 68.9), so the ablation does not explain the final performance."
The four rows in Table 7 are controlled ablations trained under an equal reduced budget (16 NVIDIA H200 GPUs, batch size 16, 1,000 training steps). The final causal model in Table 3 uses the exact same configuration as the final row of Table 7 ( + long-horizon), but was trained at full scale (64 NVIDIA H200 GPUs, batch size 64, over 2,000 training steps), which accounts for its higher scores, including Physical. We have clarified these training details in Appendix B.4 (Lines 1016–1019) of the revised paper.
"Adding long-horizon tuning on top of also lowers Consistency and Physical, contradicting the claimed 'orthogonal gains.'"
In Table 7, improves Consistency and Physical, while long-horizon tuning improves Quality, Setting, and Interaction, and using both reaches the highest overall score. Adding long-horizon tuning on top of lowers Consistency and Physical because several WBench sub-metrics inherently conflict with Dynamic Degree. For Consistency, Photometric, Geometric, Background, and Subject consistency compare neighboring frames and score higher when they look more alike, whereas Dynamic Degree rewards larger camera and scene motion; higher motion therefore makes adjacent frames differ more and lowers these similarity scores, even though the shot itself remains continuous . For Physical, the score averages Causal Fidelity and Visual Plausibility. Table R9 shows that with , Causal Fidelity remains intact after long-horizon tuning, confirming that physical reasoning does not degrade; instead, the lower Physical average comes mainly from Visual Plausibility, which is penalized by the larger frame-to-frame changes under higher Dynamic Degree. We have added this breakdown in Appendix B.4 (Lines 1022–1038, Table R9) of the revised paper.
Table R9: Sub-metric breakdown of Consistency and Physical for Table 7. Dynamic Degree is reported alongside four Consistency sub-metrics and both Physical sub-metrics on WBench.
| Trajectory Loss (Self-Forcing) | Long-Horizon Tuning | Dynamic Degree ↑ | Segment Continuity ↑ | Geometric Consistency ↑ | Photometric Consistency ↑ | Perspective Consistency ↑ | Causal Fidelity ↑ | Visual Plausibility ↑ |
|---|---|---|---|---|---|---|---|---|
| ✗ | ✗ | 72.78 | 100.00 | 93.76 | 77.79 | 72.13 | 75.74 | 58.13 |
| ✓ | ✗ | 76.58 | 100.00 | 94.74 | 81.56 | 73.14 | 77.63 | 60.10 |
| ✗ | ✓ | 88.61 | 100.00 | 91.78 | 74.86 | 69.50 | 73.13 | 57.04 |
| ✓ | ✓ | 90.51 | 100.00 | 92.61 | 75.58 | 73.32 | 77.51 | 57.93 |
W3: DeSync Comparison and Impact of Distillation
"Sec. 6.2 claims HelixWorld surpasses EchoWM and cascaded baselines across all dimensions including temporal sync, but Table 4 does not support this. The student's DeSync (0.5867) is worse than ThinkSound (0.4236) and PrismAudio (0.4582), which use the same video..."
Thank you for pointing this out. We wrote that sentence in Sec. 6.2 to summarize the overall results in Table 4 rather than every single metric, and we have revised the wording in Sec. 6.2 (Lines 358–359) of the revised paper to make this clear.
"...and worse than its own teacher (0.4988). This suggests distillation degrades synchronization and undermines the claim that joint generation outperforms cascades."
To examine the synchronization gap between the bidirectional teacher and the causal student in Table 4, we add a diagnostic comparison across the bidirectional teacher, the intermediate block-causal teacher-forced (TF) initialization model, and the distilled causal student on HelixBench (Table R5). Both the bidirectional teacher and the block-causal TF model are evaluated under matched multi-step sampling before step distillation to separate causal conversion from distillation. Table R5 shows that converting bidirectional attention to block-causal TF degrades synchronization, audio fidelity, and spatial alignment, whereas subsequent few-step distillation with recovers synchronization and surpasses the teacher in audio fidelity, semantic alignment, and spatial consistency. We hold that the slight remaining DeSync gap relative to offline V2A models and the bidirectional teacher comes from the lack of future visual context in causal generation; for example, when synthesizing audio for the current chunk, the causal model cannot observe sound-producing visual events that occur in the next chunk. Overall, few-step distillation improves rather than degrades audio-visual performance over causal initialization, while the minor synchronization offset reflects the causal streaming constraint; we have added this discussion and comparison in Sec. 6.3 (Lines 467–472 and 505–509, Table R5) and Appendix B.1 (Lines 945–950) of the revised paper.
Table R5: Audio-visual comparison across causal conversion and distillation stages. Both the bidirectional teacher and the block-causal teacher-forced (TF) model use 50 sampling steps on HelixBench, whereas the distilled causal student uses 4 steps.
| Model | KL ↓ | FAD ↓ | DeSync (s) ↓ | IB ↑ | CLAP ↑ | Spatial ↑ |
|---|---|---|---|---|---|---|
| HelixWorld (bidirectional teacher) | 1.5414 | 3.0775 | 0.4988 | 0.2700 | 0.2676 | 33.2428 |
| HelixWorld (block-causal TF) | 1.5703 | 3.5122 | 0.7836 | 0.2580 | 0.2678 | 18.9356 |
| HelixWorld (causal student) | 1.3934 | 2.3872 | 0.5867 | 0.2987 | 0.3016 | 41.7583 |
We are grateful to Reviewer Ynaa for thoroughly reviewing our paper and acknowledging our data pipeline, HelixBench design, and audio performance. We value your detailed feedback on our camera-grounded claims, ablation setups, causal synchronization, and audio drift, and respond to each concern below. Tables and figures added during the rebuttal are numbered globally as Table R1–R9 and Figure R1, in the order of their first appearance in this response, and are not restarted for each reviewer.
W1: Isolating Camera-Dependent Behavior from Source Motion
Thank you for raising this valuable point. Indeed, our original Spatial evaluation in HelixBench measured overall on-screen alignment without decoupling camera motion from sound-source motion. To evaluate the two factors separately, we have further constructed a decoupled spatial evaluation comparing HelixWorld against EchoWM, V2A baselines, and SonoWorld [1] across two controlled motion settings (Table R2). Under matched visual rollouts, Source Motion Only fixes the camera pose while sound sources move in the scene, whereas Camera Motion Only keeps the 3D sound source stationary in the world so on-screen shifts arise purely from prescribed camera motion. When the sound source is stationary and only the camera moves, V2A models and SonoWorld fail to follow viewpoint rotation, whereas HelixWorld outperforms EchoWM and other baselines. These results confirm that prescribed camera motion directly steers stereo panning in HelixWorld alongside source motion; we have added both decoupled subsets and results in Sec. 5 (Lines 331–336) and Sec. 6.2 (Lines 362–363 and 401–412, Table R2), clarified the visual-grounded scope of the general Spatial metric in Appendix D.4 (Lines 1348–1349) of the revised paper, and will release both decoupled subsets in HelixBench.
Table R2: Spatial alignment under decoupled source and camera motion. We report Spatial scores across 40 test clips per setting, separating scenes where a sound source moves before a static camera (Source Motion Only) from scenes where the camera moves around a stationary sound source (Camera Motion Only).
References:
[1] Jin et al., SonoWorld: From One Image to a 3D Audio-Visual Scene, CVPR 2026.