- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Aero Realtime, a streaming VLM that processes video, audio, and text on a single causal timeline. The main contribution is extending the audio-text recurrence of Voxtral Realtime to video-conditioned response generation and experimentally demonstrating the applicability of this slot-aligned interface to an existing VLM. Specifically, the paper introduces the model pipeline, training dataset, and training infrastructure for duplex multi-modal modeling. Experiments on OVOBench show that the proposed method retains video-understanding performance comparable to several baselines, though it trails the strongest ones. It is also demonstrated that the proposed method achieves low processing lag over the first 20 minutes of a continuous audiovisual stream.
Critical strengths
This paper presents a comprehensive framework for duplex multimodal modeling, including the model architecture, training dataset, training infrastructure, and incremental inference pipeline.
Critical weaknesses
- The motivation for using a single causal timeline with a slot-aligned interface is not sufficiently supported. A natural alternative for human-LLM interaction is to interrupt or update generation in response to a relevant visual event or a human interruption. The proposed method instead uses a fixed 80ms interface with a silence token; it can be considered as updating the conditioning context at regular intervals. However, the paper does not sufficiently explain or demonstrate when and why this design is preferable to event-driven interruption and updating for duplex multimodal interaction.
- Experiments do not directly test the advantage of continuous observation during generation. The main results are on OVOBench, and autonomous response timing in realtime interaction, which is a main motivation for the proposed pipeline, is not directly evaluated. Also, latency is measured as processing lag relative to the source timeline, but the response time to an interruption is not measured or compared with online baseline methods. While the method is designed to handle realtime input on an approximately 80-ms slot grid, it is unclear how quickly the proposed system can respond to users’ needs in practice.
Questions for authors
- Could the authors clarify the benefits and trade-offs of the proposed approach relative to an event-driven method using the same backbone?
- Could the authors evaluate whether incorporating new observations during generation improves response quality or speed, compared with keeping observations fixed during each response?