- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces AURA, a unified streaming VideoLLM framework built on Qwen3-VL-8B. AURA continuously processes video chunks, decides between silence and response, and supports Real-Time QA, Proactive QA, and Multi-Response QA. Its main components are a dual sliding window over recent video and textual QA history, a synthetic streaming-data pipeline, a Silent-Speech Balanced Loss, and a vLLM-based inference system with batched context truncation and KV-cache reuse. The authors report 73.1% on StreamingBench, 65.3% on OVO-Bench, and 25.4% on OmniMMI, plus offline-benchmark preservation and a component-wise latency analysis.
Critical strengths
The paper addresses an important and well-motivated gap between offline VideoLLMs and live, asynchronous visual assistance. Coordinating response timing, silence, memory, data construction, and serving is substantially more useful than treating streaming as ordinary long-video inference, and the unified design gives the work a coherent systems contribution.
Critical weaknesses
The paper’s “long-horizon” and “always-on understanding” claims are not adequately established by the proposed memory mechanism or experiments. The default system retains only the most recent 30 seconds of visual content; older visual evidence disappears unless a previous response already verbalized it. There is no sensitivity analysis over the video and QA window sizes, no accuracy analysis conditioned on evidence age, and no test of questions about events observed but never previously queried. Multi-Response QA, a central claimed capability, is also illustrated but not evaluated with a separate accuracy, timing, or consistency metric. As written, the evidence supports stable operation over an unbounded stream, but not necessarily long-horizon visual understanding.
The experimental protocol does not yet support the strength of the state-of-the-art and real-time claims. The paper mixes official baseline results with newly evaluated baselines, and explicitly disables MiniCPM-o-4.5’s native full-duplex streaming mode while using it as a headline comparison. The 115K streaming training samples are generated and verified entirely by unspecified MLLMs, with no human error-rate estimate or train/evaluation video-overlap audit.
Questions for authors
-
Please report accuracy as a function of evidence age and ablations over the 30-second video window and QA-history length. Can AURA answer questions about an event from several minutes ago if that event was not previously queried?
-
Can you provide a quantitative evaluation specifically for Multi-Response QA, including response timing, duplicate responses, missed updates, and answer consistency over time?
-
Why was MiniCPM-o-4.5’s full-duplex mode disabled, and what are its results under its native streaming mode?
4 comments
Questions for Authors
Q1. Please report accuracy as a function of evidence age and ablations over the 30-second video window and QA-history length. Can AURA answer questions about an event from several minutes ago if that event was not previously queried?
We use the original AURA checkpoint trained with selective supervision and class balancing. Both tests are diagnostics built from OmniMMI Multi-turn Dependency Reasoning annotations rather than official OmniMMI scores. The N/M ablation replays the original question sequence for the same 94 completed episodes while changing the recent-video window and retained QA history. GPT-6.1-Sol scores all open-ended answers with one fixed semantic-match prompt.
| Video window N (s) | QA history M | Episode-macro accuracy |
|---|---|---|
| 15 | 10 | 44.5% |
| 30 | 10 | 47.7% |
| 60 | 10 | 48.8% |
| 30 | 0 | 47.0% |
The N/M ablation measures general sensitivity across all questions in the completed episodes. Accuracy changes only slightly when N varies from 15 to 60 seconds or when M changes from 0 to 10. The evidence-age test below isolates questions about a specific event after its visual evidence becomes old.
To isolate old-event recall, we construct 166 matched pairs from the same OmniMMI source. The target question is delayed by 15 to 180 seconds after its evidence ends. In one condition, the event is not queried earlier. In the other, AURA first answers the question while the evidence is recent, and this generated QA is retained until the delayed query. The reference answer is never added to the context.
| Time since the event | No earlier question | Earlier AURA QA retained | Accuracy gain | Pairs |
|---|---|---|---|---|
| 15 s | 22.0% | 26.0% | +4.0 points | 50 |
| 30 s | 23.9% | 43.5% | +19.6 points | 46 |
| 60 s | 31.3% | 53.1% | +21.8 points | 32 |
| 120 s | 7.7% | 42.3% | +34.6 points | 26 |
| 180 s | 0.0% | 41.7% | +41.7 points | 12 |
| Overall | 20.5% | 39.8% | +19.3 points [95% CI: 12.1, 26.5] | 166 |
Retaining AURA's earlier QA improves accuracy at every tested age, and the gain increases with delay. Without earlier QA, questions about old events are much harder once their frames leave the 30-second window. The retained QA therefore helps preserve relevant information beyond that video window.
Q2. Can you provide a quantitative evaluation specifically for Multi-Response QA, including response timing, duplicate responses, missed updates, and answer consistency over time?
We construct a targeted evaluation from OmniMMI Dynamic State Grounding with 290 complete episodes and 658 required answer updates that can be scored. Each question is asked once, and each annotated state change defines a valid response window. GPT-6.1-Sol judges semantic matches with one fixed prompt. Correct updates are pooled counts, while recall and consistency are averaged per episode. An update is consistent when it receives at least one response and every response in its window matches the target.
| Correct updates (pooled) | Recall (episode avg.) | Consistency (episode avg.) | Missed updates | Duplicates | Latency p50 / p90 |
|---|---|---|---|---|---|
| 93 / 658 | 14.8% | 14.5% | 565 | 4 | 1 s / 1 s |
Correctly matched updates arrive within one second at both p50 and p90, and duplicate matches are rare.
Q3. Why was MiniCPM-o-4.5’s full-duplex mode disabled, and what are its results under its native streaming mode?
AURA targets streaming video QA and does not claim native audio-video full-duplex capability. The QA tasks in our main comparison insert text questions at fixed video timestamps. MiniCPM-o-4.5's native full-duplex mode instead requires spoken input and produces spoken output, so it cannot be enabled without changing the benchmark timeline.
Converting the questions with TTS is not valid because a spoken question may last longer than the interval before the next scheduled question. A spoken answer also takes time to complete, while the benchmark may not leave enough uninterrupted video after the question for the model to finish its answer. MiniCPM-o-4.5 often remained silent or returned unrelated answers on streams longer than two minutes, so we did not use this mode in the main comparison. We therefore do not report a native-mode score on these QA benchmarks because it would not be a like-for-like result.
Dear Reviewer W75L, as the author-reviewer discussion period is coming to an end, we would like to kindly remind you that we have posted our responses and an updated manuscript. We would be very grateful if you could take a look at your convenience, and we remain happy to discuss any point further. We look forward to hearing from you. Thank you for your time and consideration.
Dear Reviewer W75L, as the author-reviewer discussion period is coming to an end, we would like to kindly remind you that we have posted our responses and an updated manuscript. We would be very grateful if you could take a look at your convenience, and we remain happy to discuss any point further. We look forward to hearing from you. Thank you for your time and consideration.
Critical Weaknesses
We add four evaluations for this concern: a two-hour continuous run, a video-window and QA-history test, an evidence-age test, and Multi-Response QA. In a 7,200-second Inf-Streams-Eval replay through one TCP connection, AURA crosses 446 context evictions without a post-start reset or disconnect. Changing the video window has only a small effect on the completed subset. Keeping earlier QA helps answer delayed questions after the relevant frames leave the recent-video window. The Multi-Response test adds explicit timing, duplicate, recall, and consistency metrics. Q1 and Q2 give the remaining settings and results.
Sec. 6.2 distinguishes published results from our runs. Qwen3-VL-8B-Instruct and MiniCPM-o-4.5 are evaluated under the official benchmark protocol. MiniCPM-o-4.5's native full-duplex mode is a different interface, so it is not mixed into the main benchmark table. Q3 below explains this protocol mismatch. We measure wall-clock service latency separately with 20 recorded spoken queries across 10 videos. ASR and TTS share one H800 GPU, while AURA runs on a second H800 GPU. All 20 queries complete the full pipeline. From complete query-audio delivery to receipt of the first complete output PCM frame, latency is 1.45 s at p50, 1.63 s at p90, and 2.36 s at p99. Thus, 90% of the queries return the first complete PCM frame within 1.63 seconds.
Sec. 4 and Appendix A describe the data engine, which uses models from the Gemini and Doubao-Seed series for generation and verification. The 115K figure denotes training samples, not unique videos. Three annotators reviewed 1,000 samples each and judged about 95% of the answers and 92% of the response windows accurate. We also sampled train and evaluation videos at 1 FPS and matched pHash values at Hamming distance 5. We found no exact duplicate videos or reused segments; 0.2% of evaluation videos were near-duplicates. The manual audit provides direct quality estimates, while the overlap check finds no exact train-evaluation reuse.