- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes an interactive joint audio-video world model named EchoWM. Given reference images, structured text, user control, or/and preceeding audio-video sequences, EchoWM can generate synchronized audio-video results, and support multi-turn long audio-video generation. Specifically, to inject user control, EchoWM first converts control signals to 6DoF trajectories to align the input formats. The paper also proposes a full-stack pipeline covering data, model architecture design, and model training, together with quantitative and qualitative comparisons to some representative world model works on two public benchmarks and one self-constructed benchmark.
Critical strengths
(a) The paper proposes a full-stack recipe, covering dataset construction, a unified and calibrated user control interface design, a multi-stage training pipeline, and distillation for few-step generation. It looks like a well-engineered and industry-level solution. For the world model area, which usually needs large-scale data and various training strategies, I think such a solution is well aligned with the paradigm, without introducing over-designed modules and loss functions.
(b) The quantitative, qualitative, and ablation results are sufficiently provided. The paper compares EchoWM with many representative world model works, making the results and improvements convincing.
Critical weaknesses
(a) The paper's title contains "Omnimodal World Models". However, it seems to be a bit of over statement. EchoWM mainly takes reference images, structured text, user control, or/and preceeding audio-video sequences as inputs, and generates synchronized audio-video sequences. Although they contain richer modalities than most video world models that only produce videos, they still lack some typical modalities, such as 3D input and output. I understand that the paper chooses "Omnimodal World Models" probably because it is more universal and easier to spread than "Audio-Video World Models". However, it is helpful to at least state what modalities are actually contained in "omnimodal", and discuss how some other representative modalities could be incorporated in the future.
Questions for authors
(a) In Figure 2, it seems that the Unified Camera Positional Encoding (UCPE) is only injected into the video stream. I am curious whether EchoWM can generate audio that is aligned with user control signals. For example, in a third-person scenario, users control an entity to approach a playing radio. Does the audio volume of the radio gradually increase as the entity approaches?
(b) The paper also proposes EchoWM-Flash, which is a turbo version of EchoWM, supporting 4-step causal streaming inference. I am curious about the gap between the current inference latency and the real-time interaction, as the real-time interaction is getting more attention in the current world model community.
Response to Reviewer p3Cp
Thank you for the positive assessment and the thoughtful questions.
W(a): "Omnimodal" is an overstatement.
We agree. We replaced "omnimodal" with "audio-visual" in the Abstract, Introduction, and Conclusion, and will revise the title to "Scaling Interactive Audio-Visual World Models from Heterogeneous Supervision". To clarify the scope: inputs are a reference image or audio-visual prefix, structured text, and 6-DoF camera intent. Outputs are synchronized video and audio (environmental sound, speech, music). There is no explicit 3D, depth, or haptic output, and the 3D reconstructions in Fig. 7 are qualitative diagnostics. As you suggested, the Conclusion now names further modalities as future work. EchoWM already couples modality token streams through joint attention, so:
Q(a): Is audio aligned with user control, e.g., does the radio get louder on approach?
The audio stream has no direct trajectory input, but it is coupled to control indirectly. UCPE conditions the video tokens, and the bidirectional audio-video cross-attention in every block lets audio respond to the controlled visual evolution. We made this choice deliberately. In third-person scenes the "listener" is ambiguous: games attach it to either the camera or the character. Feeding camera poses to audio would conflate camera orbiting with source-listener distance.
Evidence:
Because this coupling is visually mediated, it does not guarantee physically accurate attenuation. Conditioning audio on explicit source/listener poses is future work (Conclusion).
Q(b): Gap to real-time interaction.
On a single H200 at 720p, 24 fps, BF16, with VAE and audio decoding included, four-step EchoWM-Flash runs at RTF 1.23. Throughput is therefore about 19% short of real-time playback (App. B.6). This measurement uses none of FP8 quantization, compiled static KV caches, or decoding pipelined with generation. We expect these optimizations to close the remaining throughput gap, but have not yet verified this.
[1] Ke, Lei, et al. "HelixWorld: A Real-time Interactive Audio-Visual World Model." arXiv preprint arXiv:2609.38123 (2026).