acceptodds
Under review as a conference paper at ICLR 2027

Aero Realtime: Duplex Video Interaction within a Unified Causal Stream

Abstract

The world continues to change while a video-language model generates a response. Continuous interaction requires new observations to enter during generation, yet turn-formatted execution defers them to a later input turn, while separate perception and response paths introduce additional coordination requirements. Can a video-language model support continuous input and output while preserving the training and incremental execution primitives of causal language models? We introduce Aero Realtime, a 4B model that places video, audio, silence, and lexical output on one causal timeline. At each approximately 80-ms audio slot, the model incorporates newly available visual states and predicts one lexical token or silence. Our contributions are threefold. 1) An append-only causal-stream architecture removes the native turn boundary, enabling continuous observation and generation within a single causal stream while preserving causal-language-model training and incremental execution. 2) We assemble 1.5M QA samples over 713K unique videos and address dense silence supervision with silence-label masking to adapt a pretrained VLM to the interface. 3) We provide parallel training and incremental serving as a distributed training and deployment implementation. Aero Realtime achieves 48.31 on OVOBench and records 84-ms median and 173-ms P95 processing lag over the first 20 minutes of a continuous stream on four NVIDIA A6000 GPUs. These results establish the learning and execution feasibility of duplex video interaction within a unified causal stream, with longer-run latency and autonomous response timing remaining limitations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ekvhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Aero Realtime, a streaming VLM that processes video, audio, and text on a single causal timeline. The main contribution is extending the audio-text recurrence of Voxtral Realtime to video-conditioned response generation and experimentally demonstrating the applicability of this slot-aligned interface to an existing VLM. Specifically, the paper introduces the model pipeline, training dataset, and training infrastructure for duplex multi-modal modeling. Experiments on OVOBench show that the proposed method retains video-understanding performance comparable to several baselines, though it trails the strongest ones. It is also demonstrated that the proposed method achieves low processing lag over the first 20 minutes of a continuous audiovisual stream.

Critical strengths

This paper presents a comprehensive framework for duplex multimodal modeling, including the model architecture, training dataset, training infrastructure, and incremental inference pipeline.

Critical weaknesses
  • The motivation for using a single causal timeline with a slot-aligned interface is not sufficiently supported. A natural alternative for human-LLM interaction is to interrupt or update generation in response to a relevant visual event or a human interruption. The proposed method instead uses a fixed 80ms interface with a silence token; it can be considered as updating the conditioning context at regular intervals. However, the paper does not sufficiently explain or demonstrate when and why this design is preferable to event-driven interruption and updating for duplex multimodal interaction.
  • Experiments do not directly test the advantage of continuous observation during generation. The main results are on OVOBench, and autonomous response timing in realtime interaction, which is a main motivation for the proposed pipeline, is not directly evaluated. Also, latency is measured as processing lag relative to the source timeline, but the response time to an interruption is not measured or compared with online baseline methods. While the method is designed to handle realtime input on an approximately 80-ms slot grid, it is unclear how quickly the proposed system can respond to users’ needs in practice.
Questions for authors
  • Could the authors clarify the benefits and trade-offs of the proposed approach relative to an event-driven method using the same backbone?
  • Could the authors evaluate whether incorporating new observations during generation improves response quality or speed, compared with keeping observations fixed during each response?
Reviewer_6hkSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Aero Realtime, which adapts Qwen3-VL-4B-Instruct to a unified causal stream that temporally aligns audio-video observations with text or silence outputs, allowing continuous observation during textual generation. To support this interface, the authors assemble approximately 1.5M QA samples over 713K videos, use slot-aligned supervision and silence-label masking for training, and provide modality-aware parallel training and incremental serving implementations. The method shows effectiveness on OVO-Bench. Yet, the findings primarily establish the learning and execution feasibility of the interface; the quality of autonomous response timing is not evaluated, and spoken interruption handling remains outside the scope of the paper.

Critical strengths
  1. Unified causal-stream formulation for real-time interaction.

The paper proposes a unified causal timeline for video, audio, silence, and lexical output, removing conventional turn boundaries so that new observations can be incorporated while the model is generating a response. This is a meaningful architectural direction for streaming video-language models, as it directly addresses the mismatch between continuously changing environments and turn-based inference. Preserving causal-language-model training and incremental execution also makes the approach potentially practical.

  1. Promising empirical validation and systems implementation.

The paper reports a score of 48.31 on OVO-Bench and median/P95 processing lags of 84/173 ms over the first 20 minutes of continuous streaming on four NVIDIA A6000 GPUs. It also describes a dataset of 1.5 million QA samples from 713,000 unique videos, alongside silence-label masking and a parallel training and incremental-serving implementation. These results provide initial evidence of both task performance and real-time execution feasibility.

Critical weaknesses
  1. The benefit of incorporating new evidence during generation is not directly evaluated. The paper emphasizes that an ongoing response can incorporate newly arriving evidence, but the experiments do not directly test this benefit. OVOBench accuracy and processing-lag measurements do not show whether observations arriving after a response begins affect its remaining content, and the appendix examples do not clearly identify when key evidence first appears or how it maps to the generated output. As a result, it remains unclear what improvement continuous observation provides over fixing the context during a response.

  2. Capability retention after adaptation lacks sufficient comparison and validation. Aero is adapted from Qwen3-VL-4B-Instruct, but the paper does not report the original model’s performance under comparable visual input and scoring protocols. The reported results therefore do not quantify how video-understanding capability changes after adaptation. Video understanding is also evaluated only on OVOBench, without validation on other data distributions or task settings, leaving the broader impact of the streaming adaptation on the backbone’s video-understanding capability unclear.

Questions for authors

Questions for clarification

  1. Appendix F defines processing lag, but its measurement boundary remains unclear. Please specify whether it includes video decoding and preprocessing, visual and audio encoding, buffering, queuing, and device transfer, and whether precomputed features are used. Please also provide the inference configuration, including batch size, concurrency, initial context length, and cache initialization.

  2. Please specify the source of each baseline result in Table 1, the model configuration, and the evaluation version or protocol. If results from different reports are combined across tracks, please explain whether their configurations and evaluation protocols are compatible.

  3. The paper discusses MOSS-Video-Preview and Interaction Models, but please clarify how Aero differs from them in incorporating new observations during generation. The related work does not discuss MiniCPM-o 4.5’s Omni-Flow framework, which appears to organize temporally grouped multimodal inputs and outputs into a unified sequence. What are the key differences between Aero and these methods, and what constitutes Aero's main additional contribution?

Suggested experiments

  1. Compare continuous observation against freezing subsequent observations during a response, using cases where the correct continuation depends on newly arriving evidence. Evaluate whether access to that evidence improves the remaining response.

  2. Report the original Qwen3-VL-4B-Instruct backbone’s performance under matched visible-video coverage and scoring protocols, and compare it with Aero Realtime to quantify changes after adaptation. Please also evaluate both models on an additional video-understanding benchmark with a different data distribution or task setting, to assess whether capability retention extends beyond OVOBench.

Reviewer_TTQHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduced a framework splitting the input video/audio into 80ms slots and dynamically processes them to generate silence and lexical output supporting continuous interaction. Their contribution is mainly introducing the append-only structure to add the 80ms chunk to the causal stream and the cache instead of the traditional turn-based interaction structure. They also built a large supervised dataset with a silence label to adapt a pretrained model to that 80-ms slow interface. Within the 80-ms framework, they also designed a load-balancing strategy to reduce OOM and increase throughput. Eventually, the whole process shows a middle-level score on video understanding tests with just a 4B model and shows very low latency during processing.

Critical strengths

The paper brings in a novel append-only stream architecture, which eliminates the traditional turn boundaries and enables duplex interaction on video understanding models. With proving this architecture can balance all factors like compute resources, latency and benchmark scores, it shows that such a framework can be used in the future to provide a more fluent duplex video understanding experience.

Critical weaknesses
  1. Although I understand the authors are using a slightly smaller model compared with the baseline, the Aero Realtime framework scores a little behind other streaming baselines like 55.62->48.31. The evaluation makes it hard to tell whether the gap is from the model size or from the inherent ability brought by the 80-ms framework.
  2. The value of such a low-latency system usually depends on how it can improve the experience of end-users with conversational interruptions. Unfortunately, the paper also admits that this is a current limitation.
Questions for authors
  1. Can we try to evaluate the models with an aligned size, like evaluating on a baseline with 4B or training our 80-ms framework on an 8B model?
  2. How does the model behave when communicating with a human or with sudden interruptions during an active conversation in terms of latency and performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.