acceptodds
Under review as a conference paper at ICLR 2027

Scaling Interactive Omnimodal World Models from Heterogeneous Supervision

Abstract

Interactive omnimodal world models should respond to continuous control while generating the synchronized video, environmental sound, speech, and music that make a world feel alive. Existing approaches typically use source-specific action spaces or focus on visual prediction, making it difficult to combine heterogeneous interaction and audiovisual supervision. We present EchoWM, an interactive omnimodal world model that canonicalizes interaction as a dataset-calibrated relative 6-DoF camera trajectory. The same camera-intent interface supports first-person observer motion and third-person camera-subject-world evolution, with viewpoint semantics learned from data. To reconcile complementary supervision, we progressively train an audiovisual prior, trajectory conditioning, and their joint realization on AV-rich, control-clean, and balanced data mixtures. We then convert the model for causal few-step deployment through audiovisual teacher forcing, self-generated distribution matching, and long-horizon rollouts, exposing it to the errors that accumulate during interactive generation. On WBench Navigation and SANA-WM-Bench, \model achieves leading interactive and visual-quality performance, while its four-step causal variant largely preserves these capabilities over long-horizon rollout.

Primary area:
generative models
Keywords:
Interactive World Models, Audio-Visual Generation, Long-Horizon Generation, Multimodal Generative Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_p3CpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes an interactive joint audio-video world model named EchoWM. Given reference images, structured text, user control, or/and preceeding audio-video sequences, EchoWM can generate synchronized audio-video results, and support multi-turn long audio-video generation. Specifically, to inject user control, EchoWM first converts control signals to 6DoF trajectories to align the input formats. The paper also proposes a full-stack pipeline covering data, model architecture design, and model training, together with quantitative and qualitative comparisons to some representative world model works on two public benchmarks and one self-constructed benchmark.

Critical strengths

(a) The paper proposes a full-stack recipe, covering dataset construction, a unified and calibrated user control interface design, a multi-stage training pipeline, and distillation for few-step generation. It looks like a well-engineered and industry-level solution. For the world model area, which usually needs large-scale data and various training strategies, I think such a solution is well aligned with the paradigm, without introducing over-designed modules and loss functions.

(b) The quantitative, qualitative, and ablation results are sufficiently provided. The paper compares EchoWM with many representative world model works, making the results and improvements convincing.

Critical weaknesses

(a) The paper's title contains "Omnimodal World Models". However, it seems to be a bit of over statement. EchoWM mainly takes reference images, structured text, user control, or/and preceeding audio-video sequences as inputs, and generates synchronized audio-video sequences. Although they contain richer modalities than most video world models that only produce videos, they still lack some typical modalities, such as 3D input and output. I understand that the paper chooses "Omnimodal World Models" probably because it is more universal and easier to spread than "Audio-Video World Models". However, it is helpful to at least state what modalities are actually contained in "omnimodal", and discuss how some other representative modalities could be incorporated in the future.

Questions for authors

(a) In Figure 2, it seems that the Unified Camera Positional Encoding (UCPE) is only injected into the video stream. I am curious whether EchoWM can generate audio that is aligned with user control signals. For example, in a third-person scenario, users control an entity to approach a playing radio. Does the audio volume of the radio gradually increase as the entity approaches?

(b) The paper also proposes EchoWM-Flash, which is a turbo version of EchoWM, supporting 4-step causal streaming inference. I am curious about the gap between the current inference latency and the real-time interaction, as the real-time interaction is getting more attention in the current world model community.

1 comment
AuthorsOpenReview ↗

Response to Reviewer p3Cp

Thank you for the positive assessment and the thoughtful questions.

W(a): "Omnimodal" is an overstatement.

"The paper's title contains 'Omnimodal World Models'. However, it seems to be a bit of over statement. [...] it is helpful to at least state what modalities are actually contained in 'omnimodal', and discuss how some other representative modalities could be incorporated in the future."

We agree. We replaced "omnimodal" with "audio-visual" in the Abstract, Introduction, and Conclusion, and will revise the title to "Scaling Interactive Audio-Visual World Models from Heterogeneous Supervision". To clarify the scope: inputs are a reference image or audio-visual prefix, structured text, and 6-DoF camera intent. Outputs are synchronized video and audio (environmental sound, speech, music). There is no explicit 3D, depth, or haptic output, and the 3D reconstructions in Fig. 7 are qualitative diagnostics. As you suggested, the Conclusion now names further modalities as future work. EchoWM already couples modality token streams through joint attention, so:

  • depth or point maps can be decoded as an extra output stream;
  • binaural or ambisonic audio can be conditioned on explicit source and listener poses;
  • proprioceptive, haptic, or agent-action signals can enter as conditioning streams alongside the camera trajectory.

Q(a): Is audio aligned with user control, e.g., does the radio get louder on approach?

"In Figure 2, it seems that the Unified Camera Positional Encoding (UCPE) is only injected into the video stream. I am curious whether EchoWM can generate audio that is aligned with user control signals. For example, in a third-person scenario, users control an entity to approach a playing radio. Does the audio volume of the radio gradually increase as the entity approaches?"

The audio stream has no direct trajectory input, but it is coupled to control indirectly. UCPE conditions the video tokens, and the bidirectional audio-video cross-attention in every block lets audio respond to the controlled visual evolution. We made this choice deliberately. In third-person scenes the "listener" is ambiguous: games attach it to either the camera or the character. Feeding camera poses to audio would conflate camera orbiting with source-listener distance.

Evidence:

  1. On HelixBench [1], the Spatial (stereo-panning) score improves from 0.98 (LTX-2.3) to 12.65 (App. B.5), so stereo follows the on-screen source position.
  2. Following your radio example, we added Fig. 16 in App. B.5, which shows frames, the mel spectrogram, and loudness for a third-person rollout in which a robot walks toward a playing radio. The radio becomes audible as the robot approaches it, and the background level rises by about 20 dB once the robot is near.

Because this coupling is visually mediated, it does not guarantee physically accurate attenuation. Conditioning audio on explicit source/listener poses is future work (Conclusion).

Q(b): Gap to real-time interaction.

"[...] I am curious about the gap between the current inference latency and the real-time interaction, as the real-time interaction is getting more attention in the current world model community."

On a single H200 at 720p, 24 fps, BF16, with VAE and audio decoding included, four-step EchoWM-Flash runs at RTF 1.23. Throughput is therefore about 19% short of real-time playback (App. B.6). This measurement uses none of FP8 quantization, compiled static KV caches, or decoding pipelined with generation. We expect these optimizations to close the remaining throughput gap, but have not yet verified this.

[1] Ke, Lei, et al. "HelixWorld: A Real-time Interactive Audio-Visual World Model." arXiv preprint arXiv:2609.38123 (2026).

Reviewer_2SELOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method for integrating data exhibiting heterogeneous interactions, along with a progressive training scheme that enables simultaneous generation across various modalities. Furthermore, it introduces a real-time generative world model by combining camera control capabilities with audio and visual generation models, demonstrating strong performance on world model benchmarks.

Critical strengths
  • The paper resolves data discrepancies and addresses the first-person and third-person perspective issues using an intuitive approach.
  • It adopts a progressive training scheme to build an audio-generating world model, thereby preventing data conflicts.
  • Furthermore, the ablation studies for each component are well-organized.
Critical weaknesses
  • The contribution appears to be primarily engineering-driven, effectively combining existing published methods rather than offering significant methodological novelty.
  • As shown in Tables 1 and 2, the proposed model underperforms compared to baseline models in core metrics such as Quality, Physical, AQ, and PQ.
  • The released supplementary videos lack direct visual comparisons with other baseline models.
Questions for authors
  • What are the specific reasons for the cases where other models show higher performance in Tables 1 and 2? (Are these differences attributed to the choice of backbones, training datasets, or other factors?)

  • Please specify the pretrained audio-visual backbone utilized in the proposed model. Furthermore, I recommend providing a direct comparison between this backbone's baseline performance and the final model to clearly isolate the improvements attributable to the proposed method.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2SEL (1/2)

Thank you for the review.

Q2 (answered first): Which backbone, and how much does the method add?

"Please specify the pretrained audio-visual backbone utilized in the proposed model. Furthermore, I recommend providing a direct comparison between this backbone's baseline performance and the final model to clearly isolate the improvements attributable to the proposed method."

EchoWM is initialized from the pretrained LTX-2.3 audio-visual model (now stated in Sec. 3.1). We added it as a reference row wherever it can be evaluated:

Same backbone (LTX-2.3 → EchoWM)Change
WBench Avg. (Table 1)74.2 → 81.6 (+7.4)
WBench Interaction / Consistency66.4 → 86.6 (+20.2) / 77.2 → 89.8 (+12.6)
WBench Quality / Physical77.1 → 81.5 (+4.4) / 64.9 → 70.6 (+5.7)
WBench Setting85.2 → 79.4 (−5.8)
AIF / AVAlign / AVSync↓ (Table 2)2.94 → 3.19 / 0.34 → 0.36 / 0.74 → 0.68 s
PQ / AQ (Table 2)7.08 → 6.96 / 3.31 → 3.27
HelixBench Spatial [1] (App. B.5)0.98 → 12.65

We also added the concurrent HelixWorld [1] (same LTX-2.3 backbone, four-step AR, with audio) to Table 1:

WBenchAvg.QualitySettingInteractionConsistencyPhysical
HelixWorld (LTX-2.3, 4-step AR) [1]79.979.276.586.486.470.9
EchoWM-Flash (LTX-2.3, 4-step AR)81.081.177.587.988.370.1

With backbone and step count matched, our design improves 5 of 6 scores. Table 4 additionally isolates each component under a fixed backbone and fixed data.

W2 & Q1: Why do some baselines score higher on individual metrics?

"As shown in Tables 1 and 2, the proposed model underperforms compared to baseline models in core metrics such as Quality, Physical, AQ, and PQ."

"What are the specific reasons for the cases where other models show higher performance in Tables 1 and 2? (Are these differences attributed to the choice of backbones, training datasets, or other factors?)"

EchoWM has the highest Average on WBench, EchoWM-Flash ranks #1 in Interaction, and EchoWM ranks #1 in VBench on both SANA-WM 241-frame splits. We address the main per-metric gaps below.

  • Table 1, Quality / Physical / Consistency. EchoWM is within 1.3 / 2.7 / 0.1 points of the best entries (Alaya-EVOKE, HiDream-O1-World, LingBot-World). Most of the Quality and Physical gap is already present at initialization: LTX-2.3 starts 5.7 / 8.4 points below these best entries, and our recipe closes 4.4 / 5.7 points of it (backbone).
  • Table 1, Setting. On WBench, Setting and Interaction tend to trade off: general-purpose video generators lead in Setting but trail in Interaction (Kling 3.0: 91.0 / 69.4, Cosmos3-Super: 91.6 / 58.0, and our LTX-2.3 backbone: 85.2 / 66.4). Among all entries with Interaction ≥ 84, EchoWM has the highest Setting score (79.4).
  • Table 2, PQ / AQ. These scores are 1.7% / 1.2% below the LTX-2.3 backbone, while AIF, AVAlign, and AVSync all improve. We attribute this small residual gap to adapting the backbone to a fine-tuning distribution that differs from its pretraining corpus, e.g., gameplay and simulation footage (training data), rather than to control supervision itself: in Fig. 6, PQ increases from 6.64 to 6.96 as more trajectory-conditioned data is added. Our curriculum keeps this cost small. In Table 4(b), naive joint training is 0.32 lower in PQ, 0.36 lower in AQ, and 0.22 s worse in AVSync than our full schedule.

[1] Ke, Lei, et al. "HelixWorld: A Real-time Interactive Audio-Visual World Model." arXiv preprint arXiv:2609.38123 (2026).

AuthorsOpenReview ↗

Response to Reviewer 2SEL (2/2)

W1: Engineering vs. methodological novelty.

"The contribution appears to be primarily engineering-driven, effectively combining existing published methods rather than offering significant methodological novelty."

We agree that the paper is system-level and builds on published components (UCPE, SGF, DMD). However, our core claim is that combining them naively fails in interactive audio-visual world modeling because trajectory scale, viewpoint geometry, and acoustic dynamics interact non-trivially across heterogeneous data sources. Our methodological contribution lies in diagnosing these four coupled failure modes and designing a principled remedy for each—every one validated against its naive counterpart under a fixed backbone and fixed data:

  • Cross-source metric scale ambiguity →\rightarrow Global Q0.9Q_{0.9} calibration. Per-clip normalization destroys physical speed consistency across heterogeneous game, real-world, and simulation corpora (making a subtle step indistinguishable from a fast sprint). Anchoring relative trajectories via global Q0.9Q_{0.9} calibration restores a shared metric scale, raising displacement correlation from 0.56 →\rightarrow 0.92 and cutting scale MAE from 0.218 →\rightarrow 0.064 (Table 4c).
  • First/third-person geometric ambiguity →\rightarrow Explicit viewpoint semantics. Identical 6-DoF camera deltas correspond to ego-motion in first-person views but subject-orbiting or following in third-person views. Conditioning the shared camera interface on explicit viewpoint semantics disambiguates these regimes without separate character-action channels, yielding +2.4 Interaction and +3.1 Consistency (Table 4a).
  • Control supervision degrading native audio →\rightarrow Decoupled-to-joint curriculum. Because trajectory-clean gameplay/simulation datasets exhibit narrower acoustic distributions than general media corpora, naive joint optimization induces severe cross-modal interference. Decoupling audio-visual prior adaptation (AV-CPT) from narrative-free trajectory learning (Control-SFT) prior to Joint-FT prevents audio collapse and achieves +4.0 Interaction, +0.36 AQ, and 0.22 s lower AVSync error over naive joint training (Table 4b; see also monotonic audio scaling in Fig. 6).
  • Coupled audio-video drift in streaming →\rightarrow Long-horizon audio-visual SGF/DMD. Extending self-gradient forcing and distribution matching distillation from video-only rollouts to jointly denoised audio-visual macro-chunks under a bounded sink-plus-FIFO cache prevents error accumulation across modalities over 60-second horizons, improving Interaction (83.8 →\rightarrow 87.9) and audio-state retention (2.92 →\rightarrow 3.48) over teacher forcing (Table 4d).

Together, these four designs form a unified, reproducible methodology for scaling interactive audio-visual world models from heterogeneous supervision rather than an ad-hoc engineering pipeline.

W3: Visual comparisons in supplementary videos.

"The released supplementary videos lack direct visual comparisons with other baseline models."

To make temporal trajectory adherence, viewpoint preservation, and drift easy to inspect directly, the updated supplementary page adds a dedicated Compare section featuring 20 synchronized side-by-side videos (12 first-person and 8 third-person cases) against four representative world-model baselines from Tables 1 and 3: DreamX-World, SANA-WM, LingBot-World-v2, and Alaya-EVOKE. In every comparison, all five methods start from the exact same initial frame, text prompt, and 10-second 6-DoF control sequence (mapped to each baseline's native control interface), arranged in a time-aligned 5-panel layout. Across these rollouts, baselines frequently exhibit under-actuated motion, viewpoint drift in first-person navigation, or subject loss during third-person tracking, whereas EchoWM maintains accurate trajectory following and subject framing. The comparison clips are video-only because all four baselines are silent, and all panels were downscaled and re-encoded identically to meet the archive size limit so that compression affects every method equally.

Dear Reviewer,

Thank you again for your careful review and thoughtful feedback. We have made a sincere effort to address your concerns through our responses, revisions, and additional experiments.

If you have time, we would greatly appreciate hearing which concerns remain unresolved or where our explanations are still insufficient. Your perspective would help us understand what we may have missed, and we would welcome the opportunity to clarify these points.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_deDvOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents EchoWM, an interactive omnimodal world model that combines navigation with native generation of synchronized video and audio. By heterogeneous mapping controls into a shared relative 6-DoF camera trajectory, the model can interact in first and third perspective. A progressive training strategy learns from complementary audiovisual and trajectory supervision, followed by causal post-training to enable few-step, long-horizon generation. Evaluations on WBench Navigation and SANA-WM-Bench, together with audiovisual metrics and human studies, show strong overall performance in navigation control, visual quality, and audiovisual generation.

Critical strengths
  1. I think the biggest contribution of the paper is that system combination of omnimodal in World Model area. EchoWM combines continuous navigation, native audio-visual generation, and multi-turn continuation, extending the capabilities of interactive world models.
  2. Model design and training recipe are also helpful in the area.
  3. Quantitative and qualitative results are both satisfying. The model achieves leading WBench Navigation results and competitive SANA-WM-Bench performance against strong world-model baselines, supported by human evaluations. And videos in supplementary materials are really convincing.
  4. All ablation studies are important and clear.

I agree EchoWM should be one of the most important paper in recent World Model area.

Critical weaknesses
  1. Evaluations about audio, especially spatial audio should be raised. Existing evaluations are mostly visual-related, adding audio-related benchmark would complement the existing audio-visual evaluation.
  2. More details are preferable for better presentation. Such as deployment efficiency report, more detailed training recipe.
Questions for authors

I agree this is a good paper, yet the authors should address my concerns in weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer deDv

Thank you for the strong support. We address both suggestions below.

W1: Audio and spatial-audio evaluation.

"Evaluations about audio, especially spatial audio should be raised. Existing evaluations are mostly visual-related, adding audio-related benchmark would complement the existing audio-visual evaluation."

  • Existing audio metrics. Beyond the visual benchmarks, audio is evaluated in Table 2 (PQ, AQ, AIF, AVAlign, AVSync), Table 4(b) (audio quality and synchronization under each training schedule), and Table 4(d) (long-horizon audio drift and audio-state retention).
  • New spatial-audio benchmark (Sec. 4.1, App. B.5). Following your suggestion, we evaluated on the Perspective subset of the public HelixBench benchmark [1]. EchoWM reaches a Spatial (stereo-panning) score of 12.65 versus 0.98 for the unadapted LTX-2.3 backbone. App. B.5 reports all six metrics, and the result is a trade-off. The backbone remains better on KL, FAD, ImageBind, CLAP, and DeSync (0.40 vs. 0.68 s), so we do not claim better audio quality or synchronization on HelixBench. On our own 120-case benchmark, by contrast, EchoWM has lower AVSync error than the backbone (0.68 vs. 0.74 s). The two benchmarks differ in prompts, content, and synchronization metric, and we report both.
  • Scope. This spatial behavior is visually mediated: the trajectory conditions video tokens, and audio responds through audio-video cross-attention. Physically accurate distance attenuation is therefore not guaranteed. Conditioning audio on explicit source/listener poses is listed as future work in the Conclusion.

W2: Deployment efficiency and training recipe.

"More details are preferable for better presentation. Such as deployment efficiency report, more detailed training recipe."

  • Efficiency (Sec. 4.1, App. B.6). On a single NVIDIA H200 at 720p, 24 fps, BF16, including VAE and audio decoding, four-step EchoWM-Flash runs at RTF 1.23.
  • Training recipe (App. B.1). All variants start from LTX-2.3. App. B.1 now has a stage-wise table listing the data mixture, trainable parameters, steps, global batch size, and precision of each stage: AV-CPT (15k steps), Control-SFT (10k), and Joint-FT (5k) with global batch 128, followed by causal post-training (4k steps) with global batch 64.

[1] Ke, Lei, et al. "HelixWorld: A Real-time Interactive Audio-Visual World Model." arXiv preprint arXiv:2609.38123 (2026).

Reviewer_deDvOpenReview ↗

Satisfying rebuttal

Dear authors,

The rebuttal material fully addressed my concerns. Hence I keep my initial attitude towards this paper, and agree this should be a clear accept.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. Reviewers found the unified interface intuitive (2SEL, p3Cp), the progressive training recipe useful (deDv, 2SEL, p3Cp), and the ablations well-organized and convincing (all). We have uploaded a revised PDF; all changes are marked in red. Summary:

  1. Scope and title (p3Cp). We will replace "omnimodal" with "audio-visual" in the text, will revise the title to "Scaling Interactive Audio-Visual World Models from Heterogeneous Supervision", and name depth, spatial audio, and haptic/action signals as future modalities in the Conclusion.
  2. Same-backbone comparison (2SEL). EchoWM is initialized from LTX-2.3. We now report LTX-2.3 as a reference row in Table 1 (WBench) and Table 2 (audio-visual). On WBench, EchoWM improves over its own backbone by +7.4 Average, including +20.2 Interaction, +12.6 Consistency, +5.7 Physical, and +4.4 Quality (Table 1). We also added the concurrent HelixWorld, a four-step autoregressive audio-visual model on the same LTX-2.3 backbone. Our equally four-step EchoWM-Flash outperforms it on 5 of 6 WBench scores (Average 81.0 vs. 79.9).
  3. Spatial-audio evaluation (deDv, p3Cp). On the public HelixBench benchmark, EchoWM's stereo-panning Spatial score is 12.65 vs. 0.98 for LTX-2.3. The backbone remains better on the other five HelixBench metrics, and we report all six. App. B.5 also adds a radio-approach example with its spectrogram and loudness curve.
  4. Efficiency and recipe (deDv, p3Cp). EchoWM-Flash streams at RTF 1.23 on a single H200 (720p, 24 fps, BF16, decoding included) (Sec. 4.1, App. B.6). App. B.1 adds a stage-wise training recipe.
  5. Side-by-side videos (2SEL). The updated supplement adds a Compare section with 20 synchronized side-by-side videos (12 first-person, 8 third-person) against four world-model baselines: DreamX-World, SANA-WM, LingBot-World-v2, and Alaya-EVOKE.
  6. Correction. While preparing the revision we found that EchoWM-Flash's Setting and Consistency entries in Table 1 were swapped. This does not change its Average or any conclusion, and is reflected in the revised Table 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.