- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
UniVIA unifies offline video understanding and duplex streaming interaction in a shared vision-language model. Its fork-and-commit mechanism enables continuous perception during answer generation, supported by training that models delayed answer visibility and balances silence/response supervision. Experiments report improved proactive interaction performance, competitive offline accuracy, and reduced perception delay.
Critical strengths
-
The framework offers a coherent design for continuous perception and complete-answer generation within a shared model.
-
The training formulation explicitly accounts for delayed answer visibility and balances silence/response supervision.
-
Experiments cover multiple benchmarks and backbones, with promising proactive-response results and reduced perception delay under the tested settings.
Critical weaknesses
-
Incremental novelty and insufficient validation of the key distinction. Several ingredients have clear precedents: AURA uses response-specific training samples and silence reweighting [1], Speak While Watching studies parallel visual prefill and text decoding [2], and PagedAttention supports shared-prefix branching [3]. UniVIA's clearest distinction is the combination of fixed-snapshot generation, delayed textual commits, and corresponding training. However, the paper lacks a direct comparison between immediate and delayed answer visibility, leaving the benefit of this distinctive design insufficiently established.
-
The training gains are not fully attributed. In Table 4, SBL and the adapted 1/N baseline both balance silence/response supervision but assign different total weights to control decisions relative to answer tokens. Without tuning the baseline's control-loss coefficient, the results do not isolate the benefit of the specific SBL formulation. Moreover, Table 3 omits PA/PO results for the variant without offline QA, while its reported offline scores are slightly higher than those of the full recipe. The claimed complementarity of offline and online training therefore remains insufficiently supported.
-
End-to-end asynchronous execution is insufficiently characterized. The evaluation description does not clearly specify whether Table 1 runs Algorithm 1 with a continuously advancing video clock, or how response timestamps are measured. Figure 3 also leaves unclear whether the reported latency includes answer commit/prefill and sustained overlapping responses. These omissions limit the generality of the claim that added perception latency is independent of answer length.
-
Key comparisons do not fully isolate method gains. The streaming comparison with AURA uses different evaluation protocols, as the authors acknowledge, which limits attribution of the reported PA/PO improvements. The Qwen3.5-9B variant also lacks a corresponding unmodified-backbone baseline under the same protocol, making it difficult to separate the contribution of UniVIA from backbone strength.
-
Important methodological details are incomplete. Appendix A.2 retains placeholders for data sources, models, benchmarks, and the QVA pilot results. The total training-data composition, key training settings, and simulated-delay parameters are also insufficiently specified. These omissions affect both reproducibility and the assessment of the proposed training recipe.
[1] Lu et al. AURA: Always-On Understanding and Real-Time Assistance via Video Streams, 2026. See Sections 4.4 and 5.1.
[3] Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023.
Questions for authors
-
Can you compare immediate versus delayed answer visibility using the same data, training budget, and asynchronous inference setup? Does randomizing the simulated delay improve robustness to different decoding speeds?
-
Can you report the missing PA/PO results for the variant without offline QA and compare SBL against the 1/N baseline with a tuned control-loss coefficient?
-
Does Table 1 execute Algorithm 1 with a real-time video clock, and are response timestamps measured at triggering, first-token generation, or answer completion? Does Figure 3 include commit/prefill costs, and how do perception and answer latency change with overlapping responses?
-
Can you provide matched-protocol PA/PO comparisons with AURA and corresponding results for the original Qwen3.5-9B backbone?
-
Can you complete the QVA pilot description and specify the training-data composition, main training hyperparameters, and the values/distributions used for the simulated generation delay?
4 comments
Response to Reviewer Hvfc (2/2)
W3 & Q3: Wall-Clock Timestamps, Commit Cost, and Sustained Overlapping Streams (Table R3)
Table R3: Memory and parent-step latency scaling with M concurrent child streams (128-token answers, 5-min history on A800).
| Metric (128-token answer per child) | M = 1 | M = 2 | M = 4 | M = 8 |
|---|---|---|---|---|
| Extra KV Memory (Shared Snapshot, UniVIA) | 20 MiB | 41 MiB | 81 MiB | 162 MiB |
| Extra KV Memory (Copying a 5-min History) | 4.9 GiB | 9.9 GiB | 19.8 GiB | 39.5 GiB |
| Parent Step Latency (104 ms with M = 0) | 127 ms | 130 ms | 190 ms | 241 ms |
- Timestamp definition and wall-clock equivalence in Table 1. Under Algorithm 1 (1 FPS), the response timestamp is recorded as the video timestamp of the frame step where the parent emits the response token (the exact instant the forked child starts streaming tokens to the user), while answer content is scored on the completed response. Because every parent step finishes in 104–241 ms (< 1 s frame interval, Table R3), wall-clock execution produces the exact same trigger timestamps.
- Commit cost in Figure 3 (+29 ms to +35 ms) and 10-minute stress test. Figure 3 isolates parent step latency during child decoding. When a child finishes, committing its answer text appends those tokens to the next parent step's prefill, adding a one-off +29 ms (30 tokens) to +35 ms (128 tokens), so the commit step finishes in 133–139 ms total. In a 10-minute continuous stress test where a new 128-token response is triggered every 2 seconds (maintaining up to M = 6 simultaneously active child streams), the median parent step latency is 149 ms with zero frame backlog.
W4 & Q4: Matched-Protocol Comparison with AURA (Table R4) & Role of Qwen3.5-9B
In both our protocol and AURA's released script, the model decides every second whether to stay silent or respond; our protocol feeds one continuous full-history stream with the model's own committed outputs, whereas AURA's script rebuilds the prompt every second from the latest 30 one-second clips with template-filled silent turns. We evaluated both models under both protocols on OmniMMI PA:
Table R4: Cross-protocol evaluation between AURA and UniVIA on OmniMMI PA (Qwen3-VL-8B backbone, %).
| Model (Qwen3-VL-8B backbone) | UniVIA Continuous Protocol (PA ↑) | AURA Released Protocol (PA ↑) |
|---|---|---|
| AURA | 36.0 | 37.5 (reported) |
| UniVIA (Ours) | 51.5 (+15.5) | 57.5 (+20.0) |
- Consistent +15.5 to +20.0 PA gain & same-backbone attribution. UniVIA outperforms AURA under both protocols (+15.5 PA and +20.0 PA). Furthermore, all method claims in Table 1 are established on the same Qwen3-VL-8B backbone (improving over Qwen3-VL-8B-Instruct by +8.8 RTVU, +20.2 OSU, +21.2 MCU, +30.0 ACU, +19.2 RTVP, +22.0 SQA, +19.4 BT, +21.2 FAR, and +11.6 PO). UniVIA (Qwen3.5-9B) (58.0 PA / 70.0 PO; 73.0 on VideoMME w/o sub.) is included only to show scaling to a newer backbone (clarified in Section 4.1).
W5 & Q5: Complete Methodological Details in Appendix A.2
We have filled in all specifications in Appendix A.2: (1) Data mixture (by input tokens): 35% Streaming (refined JoyAI 8.9%, in-house streaming QA 8.8%, streaming narration 16.3%, TimeLens2 1.1%) + 36% Offline VQA (images 22.5%, videos 13.6%) + 29% Offline QVA (question-first). (2) QVA pilot study (Qwen3-VL-8B-Instruct): On VideoMME w/o subtitles (2,700 QA pairs), placing the question before video frames drops accuracy from 71.5 to 59.2 (-12.3 pts) across short (79.4 to 71.2), medium (72.1 to 57.8), and long (63.0 to 48.7) videos. (3) Simulated delay: δⱼ = min(15 s, (3 s + cⱼ / 40) × uⱼ) with uⱼ ~ Uniform[0.7, 1.3], committed at the first frame step ≥ τⱼ + δⱼ. (4) Hyperparameters: Full fine-tuning for 8,817 steps, batch size 64 × 32,768 tokens, peak LR 1e-5 (200 warm-up steps, cosine decay), BF16, 4,096–131,072 pixels/frame. We would be grateful if you would consider raising your score.
Further Discussion or Feedback
Dear Reviewer Hvfc,
As the discussion period is approaching its end, we would greatly appreciate any further feedback on our responses, especially regarding whether our clarifications have addressed your main concerns.
Thank you again for your time and consideration!
Thank you for the detailed response and additional experiments. The new ablations and cross-protocol comparisons address several of my initial concerns and strengthen the empirical evidence.
However, the novelty remains relatively incremental, and the contribution of delayed answer visibility is still not directly validated through a controlled ablation. Additionally, the interpretation of wall-clock response latency could be clarified further.
Overall, I appreciate the additional efforts, but the remaining concerns regarding novelty and the validation of the central training design lead me to maintain my Weak Reject recommendation.
Response to Reviewer Hvfc (1/2)
Thank you for the clear analysis of how our work relates to prior streaming literature and for suggesting targeted experimental controls. Below, we address W1/Q1 and W2/Q2 (with our new Streaming-only ablation in Table R1 and 1/N control-loss sweep in Table R2) in Part 1/2, and W3/Q3, W4/Q4, and W5/Q5 (with multi-stream scaling in Table R3 and bi-directional AURA evaluation in Table R4) in Part 2/2.
W1 & Q1: Multi-Stream Full-Duplex Novelty & Why Delayed Answer Visibility is Needed
W2 & Q2: Streaming-Only Ablation (Table R1) & Tuned 1/N Control Weight (Table R2)
Row (d) in Paper Table 3 had a label typo in the submitted manuscript: it is actually the Offline-only specialist (trained without streaming data, so PA/PO are n/a). Below, we relabel row (d) and add the true Streaming-only variant (w/o offline QA) as row (e) under a strictly matched 1K-step token budget (1K steps × 64 sequences × 32,768 tokens ≈ 2.1B tokens):
Table R1: Controlled training-data ablation under matched 1K-step token budgets (extends Paper Table 3; Qwen3-VL-8B, %).
We also sweep the control-loss multiplier λ ∈ {1, 4, 8, 16} on the 1/N baseline (where λ = 8 gives total control weight 2λ = 16 ≈ mean ∣Wₖ∣, matching SBL's average segment length):
Table R2: Sweeping the control-loss coefficient λ on the 1/N baseline vs. SBL (extends Paper Table 4; 1K steps, %).