acceptodds
Under review as a conference paper at ICLR 2027

HelixWorld: A Real-time Interactive Audio-Visual World Model

Abstract

World simulation is inherently multisensory, demanding synchronized visual and acoustic dynamics in real time. Yet prevailing interactive world models remain strictly silent, focusing exclusively on visual rendering and control while overlooking the acoustic dimension. We present HelixWorld, a real-time interactive audio-visual world model where visual scenes and camera-grounded spatial stereo sound co-evolve natively under user interaction. We curate a high-fidelity spatial audio-visual dataset with true stereo acoustics and metric camera poses, upon which we pre-train a bidirectional teacher conditioned on 6-DoF camera trajectories and user actions. To enable low-latency causal interaction, we distill the teacher into a few-step streaming student via an online trajectory distillation loss, sustaining drift-free joint audio-visual rollouts at 24 FPS on a single GPU. Furthermore, we formalize spatial-acoustic consistency and introduce HelixBench to evaluate whether synthesized sound fields faithfully track dynamic viewpoint motion. Extensive experiments demonstrate that HelixWorld matches state-of-the-art silent world models in visual fidelity and responsiveness, while significantly surpassing existing baselines in camera-aligned spatial-acoustic immersion.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
world model, audio-video joint generation, casual

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YnaaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HelixWorld, a real-time interactive audio-visual world model that generates video and camera-aligned stereo audio under camera and action control. Built on LTX-2.3 and trained on approximately 3,000 hours of data, it uses PRoPE for camera conditioning and AdaLN-Zero for action conditioning. Teacher distillation enables causal streaming at 24 FPS on a single H800. The authors also introduce HelixBench, comprising 1,015 cases for evaluating spatial audio consistency. HelixWorld delivers competitive but not superior visual quality on WBench and outperforms EchoWM and cascaded V2A baselines in audio quality and spatial consistency.

Critical strengths
  1. The paper is among the first to combine interactive world modeling with real-time video and spatial stereo audio, with applications in gaming, VR, and embodied simulation.
  2. Well-designed data pipeline. Stereo validation, causal audio filtering, metric pose recovery, and decoupled captions address data quality and spatial leakage. The dataset could be a valuable resource if released.
  3. HelixBench evaluates synchronization, semantics, audio fidelity, and spatial consistency, using visual-only spatial targets to avoid circular evaluation.
  4. Strong audio results. The method outperforms EchoWM and cascaded V2A baselines in spatial consistency and most audio metrics, with consistent preference in the audio-enabled user study.
Critical weaknesses
  1. The term camera-grounded spatial stereo sound, repeatedly emphasized in the abstract and introduction, corresponds to the Spatial metric. However, the computation described in Appendix D.4 involves no camera poses or actions at all. This metric therefore appears to be visual-grounded rather than camera-grounded as claimed. Judging from the paper as a whole, this does not seem to be a wording error, and it poses a risk of overclaiming.

  2. The ablation in Table 7 is incompletely described. Appendix B.4 explains only the DMD-only long-horizon comparison and the L_traj comparison before long-horizon tuning. The configuration of the L_traj + long-horizon row, reported as the "best overall score" (79.1), is never specified. Moreover, the final model in Table 3 (79.9, Physical 70.9) matches no row in Table 7, and its Physical score exceeds every ablated configuration (max 68.9), so the ablation does not explain the final performance. Adding long-horizon tuning on top of L_traj also lowers Consistency and Physical, contradicting the claimed "orthogonal gains."

  3. Sec. 6.2 claims HelixWorld surpasses EchoWM and cascaded baselines across all dimensions including temporal sync, but Table 4 does not support this. The student's DeSync (0.5867) is worse than ThinkSound (0.4236) and PrismAudio (0.4582), which use the same video, and worse than its own teacher (0.4988). This suggests distillation degrades synchronization and undermines the claim that joint generation outperforms cascades, yet the paper does not discuss it.

  4. Minor but important

(1) DreamX-World in Table 3 is cited as DreamX-Phi, which is a robotic-manipulation model.

(2) The Related Work section states that distribution matching suffers from multimodal mode collapse, causing color degradation and acoustic dropouts, and Sec. 4.2 further mentions acoustic fading over long rollouts. However, the paper provides no quantitative or qualitative audio-side evidence for these issues.

Questions for authors

Please refer to the "Weaknesses" section.

6 comments

We are grateful to Reviewer Ynaa for thoroughly reviewing our paper and acknowledging our data pipeline, HelixBench design, and audio performance. We value your detailed feedback on our camera-grounded claims, ablation setups, causal synchronization, and audio drift, and respond to each concern below. Tables and figures added during the rebuttal are numbered globally as Table R1–R9 and Figure R1, in the order of their first appearance in this response, and are not restarted for each reviewer.

 

W1: Isolating Camera-Dependent Behavior from Source Motion

"The term camera-grounded spatial stereo sound, repeatedly emphasized in the abstract and introduction, corresponds to the Spatial metric. However, the computation described in Appendix D.4 involves no camera poses or actions at all. This metric therefore appears to be visual-grounded rather than camera-grounded as claimed."

 

Thank you for raising this valuable point. Indeed, our original Spatial evaluation in HelixBench measured overall on-screen alignment without decoupling camera motion from sound-source motion. To evaluate the two factors separately, we have further constructed a decoupled spatial evaluation comparing HelixWorld against EchoWM, V2A baselines, and SonoWorld [1] across two controlled motion settings (Table R2). Under matched visual rollouts, Source Motion Only fixes the camera pose while sound sources move in the scene, whereas Camera Motion Only keeps the 3D sound source stationary in the world so on-screen shifts arise purely from prescribed camera motion. When the sound source is stationary and only the camera moves, V2A models and SonoWorld fail to follow viewpoint rotation, whereas HelixWorld outperforms EchoWM and other baselines. These results confirm that prescribed camera motion directly steers stereo panning in HelixWorld alongside source motion; we have added both decoupled subsets and results in Sec. 5 (Lines 331–336) and Sec. 6.2 (Lines 362–363 and 401–412, Table R2), clarified the visual-grounded scope of the general Spatial metric in Appendix D.4 (Lines 1348–1349) of the revised paper, and will release both decoupled subsets in HelixBench.

 

Table R2: Spatial alignment under decoupled source and camera motion. We report Spatial scores across 40 test clips per setting, separating scenes where a sound source moves before a static camera (Source Motion Only) from scenes where the camera moves around a stationary sound source (Camera Motion Only).

MethodSource Motion Only ↑Camera Motion Only ↑
AudioX-25.870.17
ThinkSound7.58-2.56
PrismAudio-0.92-14.37
SonoWorld [1]-2.953.79
EchoWM18.0039.40
HelixWorld50.2554.77

 

References:
[1] Jin et al., SonoWorld: From One Image to a 3D Audio-Visual Scene, CVPR 2026.

W2: Ablation Details and Analysis in Table 7

"The configuration of the Ltraj\mathcal{L}_{\mathrm{traj}} + long-horizon row, reported as the 'best overall score' (79.1), is never specified."

 

In Table 7, the Ltraj\mathcal{L}_{\mathrm{traj}} + long-horizon row initializes long-horizon tuning from the self-forcing checkpoint trained with Ltraj\mathcal{L}_{\mathrm{traj}} and its paired fake-score model. We have clarified the initialization and training setup for these ablations in Appendix B.4 (Lines 1019–1022) of the revised paper.

 

"Moreover, the final model in Table 3 (79.9, Physical 70.9) matches no row in Table 7, and its Physical score exceeds every ablated configuration (max 68.9), so the ablation does not explain the final performance."

 

The four rows in Table 7 are controlled ablations trained under an equal reduced budget (16 NVIDIA H200 GPUs, batch size 16, 1,000 training steps). The final causal model in Table 3 uses the exact same configuration as the final row of Table 7 (Ltraj\mathcal{L}_{\text{traj}} + long-horizon), but was trained at full scale (64 NVIDIA H200 GPUs, batch size 64, over 2,000 training steps), which accounts for its higher scores, including Physical. We have clarified these training details in Appendix B.4 (Lines 1016–1019) of the revised paper.

 

"Adding long-horizon tuning on top of Ltraj\mathcal{L}_{\mathrm{traj}} also lowers Consistency and Physical, contradicting the claimed 'orthogonal gains.'"

 

In Table 7, Ltraj\mathcal{L}_{\mathrm{traj}} improves Consistency and Physical, while long-horizon tuning improves Quality, Setting, and Interaction, and using both reaches the highest overall score. Adding long-horizon tuning on top of Ltraj\mathcal{L}_{\mathrm{traj}} lowers Consistency and Physical because several WBench sub-metrics inherently conflict with Dynamic Degree. For Consistency, Photometric, Geometric, Background, and Subject consistency compare neighboring frames and score higher when they look more alike, whereas Dynamic Degree rewards larger camera and scene motion; higher motion therefore makes adjacent frames differ more and lowers these similarity scores, even though the shot itself remains continuous . For Physical, the score averages Causal Fidelity and Visual Plausibility. Table R9 shows that with Ltraj\mathcal{L}_{\mathrm{traj}}, Causal Fidelity remains intact after long-horizon tuning, confirming that physical reasoning does not degrade; instead, the lower Physical average comes mainly from Visual Plausibility, which is penalized by the larger frame-to-frame changes under higher Dynamic Degree. We have added this breakdown in Appendix B.4 (Lines 1022–1038, Table R9) of the revised paper.

 

Table R9: Sub-metric breakdown of Consistency and Physical for Table 7. Dynamic Degree is reported alongside four Consistency sub-metrics and both Physical sub-metrics on WBench.

Trajectory Loss (Self-Forcing)Long-Horizon TuningDynamic Degree ↑Segment Continuity ↑Geometric Consistency ↑Photometric Consistency ↑Perspective Consistency ↑Causal Fidelity ↑Visual Plausibility ↑
✗✗72.78100.0093.7677.7972.1375.7458.13
✓✗76.58100.0094.7481.5673.1477.6360.10
✗✓88.61100.0091.7874.8669.5073.1357.04
✓✓90.51100.0092.6175.5873.3277.5157.93

W3: DeSync Comparison and Impact of Distillation

"Sec. 6.2 claims HelixWorld surpasses EchoWM and cascaded baselines across all dimensions including temporal sync, but Table 4 does not support this. The student's DeSync (0.5867) is worse than ThinkSound (0.4236) and PrismAudio (0.4582), which use the same video..."

 

Thank you for pointing this out. We wrote that sentence in Sec. 6.2 to summarize the overall results in Table 4 rather than every single metric, and we have revised the wording in Sec. 6.2 (Lines 358–359) of the revised paper to make this clear.

 

"...and worse than its own teacher (0.4988). This suggests distillation degrades synchronization and undermines the claim that joint generation outperforms cascades."

 

To examine the synchronization gap between the bidirectional teacher and the causal student in Table 4, we add a diagnostic comparison across the bidirectional teacher, the intermediate block-causal teacher-forced (TF) initialization model, and the distilled causal student on HelixBench (Table R5). Both the bidirectional teacher and the block-causal TF model are evaluated under matched multi-step sampling before step distillation to separate causal conversion from distillation. Table R5 shows that converting bidirectional attention to block-causal TF degrades synchronization, audio fidelity, and spatial alignment, whereas subsequent few-step distillation with Ltraj\mathcal{L}_{\mathrm{traj}} recovers synchronization and surpasses the teacher in audio fidelity, semantic alignment, and spatial consistency. We hold that the slight remaining DeSync gap relative to offline V2A models and the bidirectional teacher comes from the lack of future visual context in causal generation; for example, when synthesizing audio for the current chunk, the causal model cannot observe sound-producing visual events that occur in the next chunk. Overall, few-step distillation improves rather than degrades audio-visual performance over causal initialization, while the minor synchronization offset reflects the causal streaming constraint; we have added this discussion and comparison in Sec. 6.3 (Lines 467–472 and 505–509, Table R5) and Appendix B.1 (Lines 945–950) of the revised paper.

 

Table R5: Audio-visual comparison across causal conversion and distillation stages. Both the bidirectional teacher and the block-causal teacher-forced (TF) model use 50 sampling steps on HelixBench, whereas the distilled causal student uses 4 steps.

ModelKL ↓FAD ↓DeSync (s) ↓IB ↑CLAP ↑Spatial ↑
HelixWorld (bidirectional teacher)1.54143.07750.49880.27000.267633.2428
HelixWorld (block-causal TF)1.57033.51220.78360.25800.267818.9356
HelixWorld (causal student)1.39342.38720.58670.29870.301641.7583
Reviewer_S7kUOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

HelixWorld is an interactive audio-visual world model. Users steer a scene with keyboard actions and camera trajectories, and the model generates synchronized video and camera-grounded stereo audio, which prior silent world models lack. A bidirectional teacher is trained with progressive action and camera conditioning, then distilled into a few-step causal student for real-time streaming on a single GPU. The paper also contributes a curated spatial audio-visual dataset and HelixBench, a benchmark that measures whether stereo audio tracks on-screen sound sources as the camera moves.

Critical strengths

Well-motivated. World models are almost entirely silent, even though sound carries cues about materials, contact, and off-screen events. The paper identifies this gap and proposes concrete ways to close it. This matters because audio-visual interactivity is a natural next step for world models, and the paper frames spatial-acoustic consistency as a criterion in its own right.

Promising results. The system generates interactive, synchronized audio and video in real time. It stays competitive on visual quality while clearly improving spatial-acoustic alignment over existing baselines. Showing that both modalities can respond to user control at interactive speed is a meaningful demonstration.

Complete work. The paper does more than propose a model. It also contributes a curated dataset, a method improvement, and a new benchmark, with ablations and a user study. The data and benchmark are likely to be reusable for follow-up work.

Critical weaknesses

Limited methodological novelty. The recipe is two familiar stages: action-condition injection into a bidirectional teacher, then causal distillation with DMD and self-forcing. Both stages follow existing practice, and the trajectory distillation loss is an incremental addition. The contribution is therefore mostly engineering and system integration rather than a new technique, which weakens the case for acceptance.

Interaction is limited to exploration. Users can only navigate and move the camera. There is no interaction between a subject and the environment, such as manipulating objects or triggering events. This limits how far the model qualifies as a world model, and the sounds that would come from such interactions are never tested.

Long-term consistency is not addressed. A world model needs persistent content and object identity over long horizons, for example when the viewpoint returns to a previously seen location. The evaluation covers only about 30-second rollouts and focuses on drift. It does not test memory or scene and object consistency, which is central to making an audio-video generator a true world model.

Questions for authors

Please refer to the weakness section.

4 comments

We appreciate Reviewer S7kU's thoughtful assessment, particularly for highlighting our motivation on spatial-acoustic consistency and the completeness of our work. Thank you for your insightful comments regarding methodological novelty, broader interaction, and long-term consistency, which we address below. Tables and figures added during the rebuttal are numbered globally as Table R1–R9 and Figure R1, in the order of their first appearance in this response, and are not restarted for each reviewer.

 

W1: Methodological Novelty

"The recipe is two familiar stages: action-condition injection into a bidirectional teacher, then causal distillation with DMD and self-forcing... The contribution is therefore mostly engineering and system integration rather than a new technique."

 

HelixWorld is the first real-time interactive world model that jointly generates video and spatial stereo audio at 24 FPS on a single GPU, solving specific challenges in interactive audio-visual generation rather than simple system integration. First, injecting camera and action controls directly into a pretrained audio-visual backbone disrupts cross-modal alignment and visual quality, so we propose three-stage progressive condition injection (Sec. 4.1), which reduces camera rotation and translation errors over full fine-tuning (Table 5) and enables camera motion to steer stereo panning (Table R2). Second, to counter error accumulation and mode collapse in streaming autoregressive rollouts, we propose an online trajectory distillation loss Ltraj\mathcal{L}_{\mathrm{traj}} that supervises intermediate student rollout states with teacher velocity fields (Sec. 4.2), improving WBench Consistency and Physical scores (Table 7), boosting visual diversity (Table 8), and curbing visual and acoustic drift over 30–60 s rollouts (Fig. 6, Table R4, and Fig. R1). Third, we argue that interactive audio-visual world models should generate spatial audio rather than mono sound, curate a 3.0k-hour spatial corpus with three-track captions that improve cross-modal alignment (Sec. 3, Table 6), and introduce HelixBench as the first benchmark for interactive audio-visual world models (Sec. 5). Together, these designs enable HelixWorld to match top silent world models in visual quality on WBench (Table 3) while clearly outperforming EchoWM and V2A baselines across audio quality, spatial alignment, and human preference (Table 4, Fig. 5).

 

W2: Interaction Beyond Exploration

"Users can only navigate and move the camera. There is no interaction between a subject and the environment, such as manipulating objects or triggering events... and the sounds that would come from such interactions are never tested."

 

To evaluate subject-environment interaction and event triggering beyond camera navigation, we benchmark HelixWorld on the Event Edit and Subject Action splits of WBench (Table R6), complementing the visual comparisons presented in Appendix G (Lines 1620–1711, Figs. 9 and 10). Given shared initial frames and text prompts specifying environmental events or subject actions, all models generate rollouts that are scored by WBench's official VLM judge for event execution and action adherence. In Table R6, HelixWorld ranks first on Subject Action and third on Event Edit among all evaluated world models while also generating synchronized spatial stereo audio. These results show that HelixWorld supports prompt-driven event triggering and subject-environment interaction beyond passive camera exploration; we have added these results in Sec. 6.2 (Lines 363–364 and 401–412, Table R6) of the revised paper.

 

Table R6: Prompt-driven interaction evaluation on WBench. Models are evaluated across 65 Event Edit cases and 76 Subject Action cases using the official Qwen3.8-27B VLM judge; Audio indicates native audio synthesis.

MethodAudioEvent Edit ↑Subject Action ↑
Alaya-EVOKE-Turbo✗32.9724.49
EchoWM✓33.5229.52
Zing-0.5✗27.0026.20
LingBot-World v2✗29.6030.30
HY-World 1.5✗16.8914.92
Lyra 2.0✗21.8118.34
SANA-WM✗25.0427.13
DreamX-World✗31.3223.23
Matrix-Game 3✗16.1414.24
HelixWorld✓31.6131.71

W3: Long-Term Consistency

"The evaluation covers only about 30-second rollouts and focuses on drift. It does not test memory or scene and object consistency, which is central to making an audio-video generator a true world model."

 

To evaluate long-term scene and object consistency up to 60 seconds, we compare HelixWorld against world-model baselines across six WBench consistency metrics covering background, perspective, segment continuity, 3D geometry, photometry, and subject identity (Tables R7 and R8). All methods are evaluated under shared initial frames, text prompts, and continuous control trajectories that include viewpoint reversals returning to previously seen locations (as also illustrated in Fig. 4, where scene landmarks and objects remain consistent when the camera pans back). By anchoring the sliding KV cache with initial sink frames (Sec. 4.2), HelixWorld maintains strong segment continuity, 3D geometric structure, background stability, and subject identity over 30- and 60-second horizons while streaming synchronized stereo audio. We have added these results in Sec. 6.4 (Lines 479–480) and Appendix H (Lines 1859–1889, Tables R7 and R8) of the revised paper.

 

Table R7: Long-horizon visual consistency on 30-second WBench rollouts. All methods are evaluated across 45 test cases under shared initial frames, text prompts, and control trajectories.

MethodAverage ↑Background Consistency ↑Perspective Consistency ↑Segment Continuity ↑Geometric Consistency ↑Photometric Consistency ↑Subject Consistency ↑
Alaya-EVOKE-Turbo87.1489.9762.34100.0097.5184.9988.05
EchoWM82.3188.7855.6186.6796.6482.8283.33
Zing-0.579.2886.9731.9793.3395.2285.0283.18
LingBot-World v280.3087.7545.9297.7890.6576.4583.26
HY-World 1.585.8387.1259.41100.0096.7883.2288.45
SANA-WM79.0189.1535.5497.7892.3677.1582.09
DreamX-World83.0088.6150.80100.0092.8379.6186.13
Matrix-Game 370.7483.2510.1873.3391.0684.4082.19
HelixWorld84.6189.7454.86100.0095.4382.5085.11

 

Table R8: Long-horizon visual consistency on 60-second WBench rollouts. All methods are evaluated across the same 45 test cases extended to minute-long control trajectories.

MethodAverage ↑Background Consistency ↑Perspective Consistency ↑Segment Continuity ↑Geometric Consistency ↑Photometric Consistency ↑Subject Consistency ↑
Alaya-EVOKE-Turbo83.9488.6651.88100.0096.2080.7286.19
EchoWM79.3287.7345.2788.8994.4178.4081.24
Zing-0.576.9985.0024.9193.3392.6684.0382.00
LingBot-World v275.0485.9938.1388.8983.7072.0581.49
HY-World 1.576.9283.9539.9882.2294.7076.0784.61
SANA-WM72.9587.2721.2186.6789.2573.1880.10
DreamX-World79.4587.9535.48100.0091.1777.5684.53
Matrix-Game 367.4883.0410.2564.4482.8082.1582.18
HelixWorld80.0887.8141.2297.7892.0478.4983.16
Reviewer_S7kUOpenReview ↗

Official Comment by reviewer

Thank you for the detailed rebuttal and the additional experimental results. I appreciate the authors’ effort in addressing my concerns thoroughly.

Regarding W2, I am pleased to see that HelixWorld supports prompt-driven interaction beyond passive camera exploration. The results on the Subject Action and Event Edit splits of WBench demonstrate that the action space extends meaningfully beyond navigation. I encourage the authors to include real generated video demonstrations of these interaction capabilities in the final version to better showcase this feature to the community.

For W3, the extended evaluation to 60-second rollouts is a good supplemenatary experiment. The consistency metrics in Tables R7 and R8 show promising results.

Overall, the authors have addressed my main concerns. I am happy to maintain my current positive rating for this paper.

Thank you very much for your positive feedback and support of our work! As suggested, we will incorporat the additional experimental results into the revised paper. We truly appreciate your valuable comments that helped improve our paper.

Reviewer_SdR4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HelixWorld is an interactive audiovisual world model that jointly generates video and stereo audio conditioned on continuous 6-DoF camera trajectories and discrete navigation actions. Built on LTX-2.3, it combines progressive control conditioning, causal initialization, online trajectory distillation, and long-horizon streaming fine-tuning. The paper also introduces a ~3,000-hour curated training corpus and HelixBench, a 1,015-case human-verified evaluation suite. The main contribution is unifying interactive control, spatial stereo audio, and streaming generation in a single system.

Critical strengths

S1. The paper addresses a practical problem: interactive world models should produce coherent visual and auditory responses to user control. Incorporating spatial audio substantially increases relevance to immersive applications.

S2.The system balances visual quality and speed well, achieving a WBench navigation score of 79.9 and a steady-state RTF of 0.77 on a single H800 at 768×512, 24 fps, including audiovisual decoding.

S3. Table 4 reports the best FAD and Spatial scores among evaluated methods. The matched-video comparisons usefully isolate audio performance, and the user study provides supporting evidence of improved audiovisual experience.

Critical weaknesses
  1. The spatial comparisons lack a simple spatialization baseline. The paper argues that V2A models cannot respond to ego-motion, but input videos still contain apparent motion and screen positions of visible sources. Under the matched-video protocol, augmenting a V2A baseline with source tracking and stereo panning would help determine how much of the measured Spatial improvement can be obtained through simple post-hoc spatialization, particularly because the spatial subset is curated around a single dominant visible source. Scene-level methods such as Sonic4D [1] and SonoWorld [2] also deserve discussion as related work; where feasible, restricted comparisons (e.g., an explicitly adapted spatialization component from Sonic4D or moving-camera evaluation in static scenes for SonoWorld) would further strengthen the analysis.

  2. The Spatial metric supports narrower conclusions than the paper’s claims suggest. Appendix D.4 measures time-sampled lateral agreement between a tracked source’s horizontal image position and the generated audio’s left–right energy balance. Although the paper includes threshold-sensitivity analysis and qualitative stereo examples, this score does not isolate camera-induced changes from source motion or assess consistency across viewpoints. Claims about sound-field rotation under ego-motion should therefore be narrowed or supported by controlled camera-motion experiments.

  3. HelixBench’s contribution should be distinguished more clearly from prior work. SAVGBench [3] already evaluates spatial correspondence between generated video and stereo audio, primarily using short, fixed-viewpoint clips. HelixBench introduces camera-conditioned evaluation and broader scene coverage, but the paper should explicitly distinguish its protocol and spatial metric from SAVGBench, which is neither cited nor discussed.

  4. The claim of "drift-free" joint audiovisual rollouts needs stronger quantitative support. Visual comparisons span 30 seconds, but audio quality, synchronization, and spatial alignment are not reported as functions of rollout duration. KL and FAD use only the first five seconds; DeSync averages the first and last windows without reporting them separately. Time-resolved measurements would better substantiate the long-horizon claims. Table 4 also shows lower FAD but higher DeSync for the causal student than for the bidirectional teacher; this observed quality–synchronization tradeoff deserves discussion.

[1] Xie et al, "Sonic4D: Spatial Audio Generation for Immersive 4D Scene Exploration", AAAI 2026

[2] Jin et al, "SonoWorld: From One Image to a 3D Audio-Visual Scene", CVPR 2026.

[3] Shimada et al, "SAVGBench: Benchmarking Spatially Aligned Audio-Video Generation"IEEE ICASSP 2026

Questions for authors
  1. Under the matched-video protocol, how does HelixWorld compare with a V2A baseline augmented with source tracking and stereo panning in Spatial, audio quality, synchronization, and computational cost? Please also clarify how HelixWorld relates to prior audiovisual scene-generation methods such as Sonic4D and SonoWorld.

  2. What are HelixBench’s specific contributions beyond SAVGBench in control inputs, evaluation protocol, and spatial consistency assessment? Can controlled tests with stationary sources and prescribed camera motion isolate camera-dependent behavior?

  3. How do audio quality, synchronization, and spatial alignment evolve across fixed-length windows at increasing rollout times (e.g., 5, 15, and 30 s, and 60 s where feasible), using appropriate reference data where required by the metric? Can the existing ablation checkpoints clarify how online trajectory distillation and long-horizon tuning affect these trends?

7 comments

We sincerely thank Reviewer SdR4 for the careful review and for recognizing the practical value of HelixWorld and its balance between visual quality and speed. Below, we address your constructive suggestions on spatialization baselines, camera-motion and benchmark comparisons, and multi-horizon audio evaluation. Tables and figures added during the rebuttal are numbered globally as Table R1–R9 and Figure R1, in the order of their first appearance in this response, and are not restarted for each reviewer.

 

W1.1 & Q1.1: Post-hoc Spatialization Baselines and Tracker Ablation

"Under the matched-video protocol, augmenting a V2A baseline with source tracking and stereo panning would help determine how much of the measured Spatial improvement can be obtained through simple post-hoc spatialization... where feasible, restricted comparisons (e.g., an explicitly adapted spatialization component from Sonic4D or moving-camera evaluation in static scenes for SonoWorld) would further strengthen the analysis."

 

Thank you for this suggestion, which helps us add important baselines to our spatial evaluation. On the spatial subset of HelixBench, we evaluat V2A baselines and downmixed HelixWorld audio with 2D tracking-and-panning across four trackers (SAM 3, SAM 2-Tiny, EdgeTAM, and LiteTrack) and with the depth-based spatializer from Sonic4D [1], as well as SonoWorld [2] on both the full subset and its static-scene moving-camera clips (Table R1). In Table R1, native HelixWorld outperforms unaugmented V2A baselines and SonoWorld. When augmented with 2D tracking-and-panning or Sonic4D, both V2A models and HelixWorld reach high Spatial scores. This is expected on the single-source spatial subset because explicit 2D tracking-and-panning directly mirrors the evaluator's 2D bounding-box panning rule. However, such post-hoc spatialization leaves underlying audio quality and synchronization unchanged, cannot generalize to off-screen sound sources or multi-source environments, and requires pre-generated video segments, which introduces 1.15–13.61 s (2D tracking) to 235.04 s (Sonic4D) of offline latency per clip. In contrast, HelixWorld natively synthesizes camera-aligned stereo sound in a single causal stream with zero post-processing overhead, enabling real-time interactive rollouts. We have updated these results in Appendix D.5 (Lines 1360–1391, Table R1) of the revised paper.

 

Table R1: Comparison with post-hoc spatialization baselines and SonoWorld. All methods are evaluated on the spatial subset of HelixBench under the matched-video protocol, where Runtime measures per-clip spatial post-processing latency; † restricts evaluation to clips within this subset depicting static scenes under camera motion.

MethodPost-processingSpatial ↑Runtime (s) ↓
AudioXNone-5.740
AudioXTracking + panning (SAM 3)65.7713.59
AudioXTracking + panning (SAM 2-Tiny)63.993.33
AudioXTracking + panning (EdgeTAM)63.414.13
AudioXTracking + panning (LiteTrack)65.511.15
AudioXSonic4D [1]60.76235.04
ThinkSoundNone14.850
ThinkSoundTracking + panning (SAM 3)65.7513.61
ThinkSoundTracking + panning (SAM 2-Tiny)64.063.33
ThinkSoundTracking + panning (EdgeTAM)63.434.13
ThinkSoundTracking + panning (LiteTrack)65.531.15
ThinkSoundSonic4D [1]60.86235.04
PrismAudioNone-9.510
PrismAudioTracking + panning (SAM 3)65.8013.61
PrismAudioTracking + panning (SAM 2-Tiny)64.043.33
PrismAudioTracking + panning (EdgeTAM)63.464.12
PrismAudioTracking + panning (LiteTrack)65.511.15
PrismAudioSonic4D [1]60.66235.04
SonoWorld [2]None-6.120
SonoWorld† [2]None13.520
HelixWorldNone41.760
HelixWorldTracking + panning (SAM 3)65.7213.61
HelixWorldTracking + panning (SAM 2-Tiny)63.943.33
HelixWorldTracking + panning (EdgeTAM)63.324.13
HelixWorldTracking + panning (LiteTrack)65.511.15
HelixWorldSonic4D [1]60.57235.04

W1.2 & Q1.2: Relation to Sonic4D [1] and SonoWorld [2]

"Please also clarify how HelixWorld relates to prior audiovisual scene-generation methods such as Sonic4D and SonoWorld."

 

Sonic4D [1] and SonoWorld [2] are training-free, multi-stage pipelines that rely on external video and mono-audio generators and treat spatialization as an offline post-processing step. Specifically, Sonic4D uses off-the-shelf models to synthesize novel-view video and mono audio, then tracks a visible sound source in 3D and runs physics-based room acoustic simulation offline, which requires the full video sequence in advance. SonoWorld reconstructs a static 3D scene from a single image and pairs it with text-generated mono audio, so its sound cannot respond to moving objects or visual events. In contrast, HelixWorld is an end-to-end audio-visual world model that generates video and spatial stereo audio jointly in a single causal stream without post-processing or 3D reconstruction while supporting real-time interaction; we have added this discussion in Sec. 2 (Lines 107–108) and added Sonic4D [1] and SonoWorld [2] to the reference list (Lines 813–815 and 675–676) of the revised paper.

 

W2 & Q2.2: Isolating Camera-Dependent Behavior from Source Motion

"Can controlled tests with stationary sources and prescribed camera motion isolate camera-dependent behavior?"

 

To separate camera-driven spatialization from visual-driven source motion, we construct a decoupled spatial evaluation comparing HelixWorld against EchoWM, V2A baselines, and SonoWorld [2] across two controlled motion settings (Table R2). Under matched visual rollouts, Source Motion Only fixes the camera pose while sound sources move in the scene, whereas Camera Motion Only keeps the 3D sound source stationary in the world so on-screen shifts arise purely from prescribed camera motion. When the sound source is stationary and only the camera moves, V2A models and SonoWorld fail to follow viewpoint rotation. These results confirm that prescribed camera motion directly steers stereo panning in HelixWorld alongside source motion; we have added both decoupled subsets and results in Sec. 5 (Lines 331–336) and Sec. 6.2 (Lines 362–363 and 401–412, Table R2), clarified the visual-grounded scope of the general Spatial metric in Appendix D.4 (Lines 1348–1349) of the revised paper, and will release both decoupled subsets in HelixBench.

 

Table R2: Spatial alignment under decoupled source and camera motion. We report Spatial scores across 40 test clips per setting, separating scenes where a sound source moves before a static camera (Source Motion Only) from scenes where the camera moves around a stationary sound source (Camera Motion Only); the latter extends the static-scene clips in HelixBench with newly curated test cases.

MethodSource Motion Only ↑Camera Motion Only ↑
AudioX-25.870.17
ThinkSound7.58-2.56
PrismAudio-0.92-14.37
SonoWorld [2]-2.953.79
EchoWM18.0039.40
HelixWorld50.2554.77

 

W3 & Q2.1: Distinction from SAVGBench [3]

"What are HelixBench's specific contributions beyond SAVGBench in control inputs, evaluation protocol, and spatial consistency assessment?"

 

First, HelixBench is the first benchmark for interactive audio-visual world models and conditions streaming rollouts on an initial frame, detailed captions, and continuous camera trajectories and actions across spatial panning, event triggering, and out-of-view entry scenarios, whereas SAVGBench [3] targets unconditional short-clip generation under fixed viewpoints using indoor clips with only single-source onscreen speech and musical instruments. Second, SAVGBench evaluates each method on its own unconditionally generated videos, whereas HelixBench uses a matched-video protocol so V2A and spatialization baselines are compared on identical visual rollouts independent of video-generation quality, supplemented by controlled splits that separate source motion from camera motion (Table R2). Third, SAVGBench's Spatial AV-Align relies on YOLOX person detection and a supervised SELD model trained only on indoor speech and instruments, whereas HelixBench uses open-vocabulary tracking and stereo channel-energy balance, generalizing to open-world scenes and general sound-making objects without retraining class-specific detectors. We have added SAVGBench [3] to the reference list (Lines 766–768) and included this discussion in Sec. 2 (Lines 125–129) of the revised paper.

W4.1, Q3: Multi-Horizon Audio Evaluation — Baseline Comparison

"How do audio quality, synchronization, and spatial alignment evolve across fixed-length windows at increasing rollout times (e.g., 5, 15, and 30 s, and 60 s where feasible), using appropriate reference data where required by the metric?"

 

To evaluate how audio quality, synchronization, and spatial alignment change with rollout duration, we add multi-horizon comparisons against the baselines (Table R3). Because each V2A baseline has a fixed maximum generation length, we use a sliding-window approach so they can generate audio for long videos, and we report first-window, last-window, and mean DeSync separately alongside KL, FAD, ImageBind, CLAP, and Spatial at each duration. As shown in Table R3, HelixWorld maintains solid audio generation performance across rollout lengths. We have added these results in Appendix D.8 (Lines 1475–1502, Table R3) of the revised paper.

 

Table R3: Multi-horizon audio-visual comparison against baselines. Models are evaluated across 60 held-out rollouts at 5 s and 15 s and 40 rollouts at 30 s and 60 s; cascaded V2A baselines re-dub HelixWorld videos via sliding-window concatenation.

DurationModelKL ↓FAD ↓DeSync First Window (s) ↓DeSync Last Window (s) ↓DeSync Mean (s) ↓IB ↑CLAP ↑Spatial ↑
5 sHelixWorld + AudioX3.07205.35271.21671.27671.24670.22390.3011-6.4416
HelixWorld + ThinkSound2.92525.47320.42000.62330.52170.22820.32073.8411
HelixWorld + PrismAudio3.14856.23460.38000.45000.41500.24490.34621.3168
EchoWM4.174010.18930.70330.98330.84330.14820.25492.6442
HelixWorld2.41385.44670.62670.50330.56500.24730.269011.4937
15 sHelixWorld + AudioX2.95428.50151.22221.37781.30000.25600.30311.0892
HelixWorld + ThinkSound2.72946.59540.28330.56110.42220.25050.3358-2.9373
HelixWorld + PrismAudio2.858810.73620.27220.38330.32780.25450.3253-3.8010
EchoWM3.698712.44720.80560.95000.87780.17150.23704.7606
HelixWorld2.32059.46610.46110.38890.42500.26790.283614.4704
30 sHelixWorld + AudioX2.238311.78711.30771.29231.30000.25840.3208-6.6479
HelixWorld + ThinkSound2.25458.81290.57690.63850.60770.26080.3404-5.6473
HelixWorld + PrismAudio2.362916.61100.47690.72310.60000.24440.3091-2.2567
EchoWM3.719816.32720.72310.95380.83850.17920.25505.0132
HelixWorld2.095014.56890.63850.83080.73460.23320.282715.0006
60 sHelixWorld + AudioX2.55709.40711.04001.08001.06000.22160.30453.0488
HelixWorld + ThinkSound2.64058.97380.42860.70860.56860.20670.3164-0.1450
HelixWorld + PrismAudio2.73648.86630.37710.55430.46570.22050.30322.8553
EchoWM4.053213.53490.62860.85140.74000.14860.23837.5956
HelixWorld2.714013.00030.51430.60570.56000.17930.257017.3244

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.