acceptodds
Under review as a conference paper at ICLR 2027

UniVIA: A Unified Video Intelligence Architecture for Offline Understanding and Duplex Streaming Interaction

Abstract

Streaming video interaction requires a model to keep perceiving an evolving video while deciding when to respond and generating answers. Existing chunk-based formulations couple response timing with answer generation: fine chunks split each answer across observation updates, departing from the complete-answer supervision of offline video-language training, while coarse chunks sacrifice the temporal granularity of response decisions. We present UniVIA, a unified video intelligence architecture that serves offline video understanding and full-duplex streaming interaction with a single shared vision-language model. At inference, an asynchronous fork-and-commit mechanism lets a persistent parent stream keep perceiving the video while each response decision forks a child stream that generates the complete answer from a fixed context snapshot and commits it back as memory. For training, we split each streaming interaction into response-ended samples that reproduce the parent and child states of inference, and propose a Segment-Balanced Loss that supervises the silence and response decisions of each query with balanced class weights. UniVIA achieves the best results among open-source models on seven of ten tasks across StreamingBench, OVO-Bench, and OmniMMI, with the largest gains on proactive tasks, keeps the perception delay added by answering at 63–94 ms at 1 FPS regardless of answer length, and achieves the best results on nine of eleven offline video metrics among all evaluated models; with the Qwen3-VL-8B backbone, it improves over the base model on eight of these metrics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
streaming vision-language models, full-duplex interaction, continual video understanding, causal multimodal modeling, real-time multimodal assistants

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HvfcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UniVIA unifies offline video understanding and duplex streaming interaction in a shared vision-language model. Its fork-and-commit mechanism enables continuous perception during answer generation, supported by training that models delayed answer visibility and balances silence/response supervision. Experiments report improved proactive interaction performance, competitive offline accuracy, and reduced perception delay.

Critical strengths
  1. The framework offers a coherent design for continuous perception and complete-answer generation within a shared model.

  2. The training formulation explicitly accounts for delayed answer visibility and balances silence/response supervision.

  3. Experiments cover multiple benchmarks and backbones, with promising proactive-response results and reduced perception delay under the tested settings.

Critical weaknesses
  1. Incremental novelty and insufficient validation of the key distinction. Several ingredients have clear precedents: AURA uses response-specific training samples and silence reweighting [1], Speak While Watching studies parallel visual prefill and text decoding [2], and PagedAttention supports shared-prefix branching [3]. UniVIA's clearest distinction is the combination of fixed-snapshot generation, delayed textual commits, and corresponding training. However, the paper lacks a direct comparison between immediate and delayed answer visibility, leaving the benefit of this distinctive design insufficiently established.

  2. The training gains are not fully attributed. In Table 4, SBL and the adapted 1/N baseline both balance silence/response supervision but assign different total weights to control decisions relative to answer tokens. Without tuning the baseline's control-loss coefficient, the results do not isolate the benefit of the specific SBL formulation. Moreover, Table 3 omits PA/PO results for the variant without offline QA, while its reported offline scores are slightly higher than those of the full recipe. The claimed complementarity of offline and online training therefore remains insufficiently supported.

  3. End-to-end asynchronous execution is insufficiently characterized. The evaluation description does not clearly specify whether Table 1 runs Algorithm 1 with a continuously advancing video clock, or how response timestamps are measured. Figure 3 also leaves unclear whether the reported latency includes answer commit/prefill and sustained overlapping responses. These omissions limit the generality of the claim that added perception latency is independent of answer length.

  4. Key comparisons do not fully isolate method gains. The streaming comparison with AURA uses different evaluation protocols, as the authors acknowledge, which limits attribution of the reported PA/PO improvements. The Qwen3.5-9B variant also lacks a corresponding unmodified-backbone baseline under the same protocol, making it difficult to separate the contribution of UniVIA from backbone strength.

  5. Important methodological details are incomplete. Appendix A.2 retains placeholders for data sources, models, benchmarks, and the QVA pilot results. The total training-data composition, key training settings, and simulated-delay parameters are also insufficiently specified. These omissions affect both reproducibility and the assessment of the proposed training recipe.

[1] Lu et al. AURA: Always-On Understanding and Real-Time Assistance via Video Streams, 2026. See Sections 4.4 and 5.1.

[2] Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models, 2026.

[3] Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023.

Questions for authors
  1. Can you compare immediate versus delayed answer visibility using the same data, training budget, and asynchronous inference setup? Does randomizing the simulated delay improve robustness to different decoding speeds?

  2. Can you report the missing PA/PO results for the variant without offline QA and compare SBL against the 1/N baseline with a tuned control-loss coefficient?

  3. Does Table 1 execute Algorithm 1 with a real-time video clock, and are response timestamps measured at triggering, first-token generation, or answer completion? Does Figure 3 include commit/prefill costs, and how do perception and answer latency change with overlapping responses?

  4. Can you provide matched-protocol PA/PO comparisons with AURA and corresponding results for the original Qwen3.5-9B backbone?

  5. Can you complete the QVA pilot description and specify the training-data composition, main training hyperparameters, and the values/distributions used for the simulated generation delay?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Hvfc (1/2)

Thank you for the clear analysis of how our work relates to prior streaming literature and for suggesting targeted experimental controls. Below, we address W1/Q1 and W2/Q2 (with our new Streaming-only ablation in Table R1 and 1/N control-loss sweep in Table R2) in Part 1/2, and W3/Q3, W4/Q4, and W5/Q5 (with multi-stream scaling in Table R3 and bi-directional AURA evaluation in Table R4) in Part 2/2.

W1 & Q1: Multi-Stream Full-Duplex Novelty & Why Delayed Answer Visibility is Needed

  1. From single-stream precedents to multi-stream full-duplex (1 Parent + M Children). PagedAttention provides the low-level KV-sharing primitive we build on, AURA introduced response-ended training with 1/N silence weighting, and Speak While Watching interleaves visual prefill with chunked text decoding. However, both AURA and Speak While Watching remain single-output-stream models: because decoding shares the perception timeline, they must either pause perception, chop answers into frame-step chunks, or drop overlapping events. UniVIA decouples them into 1 persistent parent stream + M concurrent child streams, allowing the parent to fork new child streams on the fly while earlier children are still decoding, and keeping every child's sequence identical to offline QA (enabling +6.2 PA / +3.1 PO transfer from QVA offline data in Table R1 below).
  2. Why delayed visibility (τⱼ + δⱼ) is required over immediate visibility (τⱼ). If training placed the full answer Aⱼ immediately after the response token at τⱼ, the parent stream would never see a state where a response was triggered at τⱼ without Aⱼ already in memory; during real-time inference, while child j is still decoding across [τⱼ, τⱼ + δⱼ), the parent would repeatedly fire duplicate triggers across consecutive frames. Keeping the response token at τⱼ and inserting Aⱼ only at τⱼ + δⱼ teaches the parent that an emitted response token already represents an active in-flight child stream.
  3. Robustness to hardware decoding speed. Our delay formula δⱼ = min(15 s, (3 s + cⱼ / 40) × uⱼ) samples a random speed factor uⱼ ~ Uniform[0.7, 1.3] for every response (3 s prefill/scheduling offset + 40 chars/s nominal decoding), training the parent across a wide range of decoding speeds.

W2 & Q2: Streaming-Only Ablation (Table R1) & Tuned 1/N Control Weight (Table R2)

Row (d) in Paper Table 3 had a label typo in the submitted manuscript: it is actually the Offline-only specialist (trained without streaming data, so PA/PO are n/a). Below, we relabel row (d) and add the true Streaming-only variant (w/o offline QA) as row (e) under a strictly matched 1K-step token budget (1K steps × 64 sequences × 32,768 tokens ≈ 2.1B tokens):

Table R1: Controlled training-data ablation under matched 1K-step token budgets (extends Paper Table 3; Qwen3-VL-8B, %).

Training Variant (Matched 1K-Step Budget)PA (OMMI) ↑PO (SB) ↑VideoMME w/o ↑VideoMME w/ ↑LVBench ↑
(a) UniVIA Full Recipe (Ours)42.521.270.375.752.1
(c) Replace QVA with Video-First VQA Order37.018.870.177.053.1
(d) Offline Only (Relabelled)n/an/a71.176.052.8
(e) Streaming Only (w/o Offline QA)36.318.166.872.448.5

We also sweep the control-loss multiplier λ ∈ {1, 4, 8, 16} on the 1/N baseline (where λ = 8 gives total control weight 2λ = 16 ≈ mean ∣Wₖ∣, matching SBL's average segment length):

Table R2: Sweeping the control-loss coefficient λ on the 1/N baseline vs. SBL (extends Paper Table 4; 1K steps, %).

Loss Setting (1K Steps, Qwen3-VL-8B)Total Control Weight vs. LₖPA (OMMI) ↑PO (SB) ↑
1/N silence weighting, λ = 1 (Paper Table 4)215.58.4
1/N silence weighting, λ = 4819.610.3
1/N silence weighting, λ = 8 (matches avg. segment length)16 (≈ mean ∣Wₖ∣)21.811.3
1/N silence weighting, λ = 163219.49.8
w/o last-segment loss (Paper Table 4)Prefix length τₖ31.713.6
SBL (Ours, Paper Table 4)Adaptive ∣Wₖ∣42.521.2
  1. Offline QA to Streaming (+6.2 PA / +3.1 PO). Adding QVA offline data (row a vs. row e in Table R1) improves streaming from 36.3 to 42.5 PA (+6.2) and 18.1 to 21.2 PO (+3.1) while keeping offline accuracy within 0.3–0.8 pts of the offline-only specialist (row d).
  2. SBL vs. tuned 1/N (+20.7 PA / +9.9 PO). In Table R2, tuning λ on 1/N saturates at 21.8 PA / 11.3 PO (λ = 8) and drops at λ = 16 (19.4 PA / 9.8 PO), whereas SBL's adaptive ∣Wₖ∣ scaling and active-segment localization reach 42.5 PA / 21.2 PO.
AuthorsOpenReview ↗

Response to Reviewer Hvfc (2/2)

W3 & Q3: Wall-Clock Timestamps, Commit Cost, and Sustained Overlapping Streams (Table R3)

Table R3: Memory and parent-step latency scaling with M concurrent child streams (128-token answers, 5-min history on A800).

Metric (128-token answer per child)M = 1M = 2M = 4M = 8
Extra KV Memory (Shared Snapshot, UniVIA)20 MiB41 MiB81 MiB162 MiB
Extra KV Memory (Copying a 5-min History)4.9 GiB9.9 GiB19.8 GiB39.5 GiB
Parent Step Latency (104 ms with M = 0)127 ms130 ms190 ms241 ms
  1. Timestamp definition and wall-clock equivalence in Table 1. Under Algorithm 1 (1 FPS), the response timestamp is recorded as the video timestamp of the frame step where the parent emits the response token (the exact instant the forked child starts streaming tokens to the user), while answer content is scored on the completed response. Because every parent step finishes in 104–241 ms (< 1 s frame interval, Table R3), wall-clock execution produces the exact same trigger timestamps.
  2. Commit cost in Figure 3 (+29 ms to +35 ms) and 10-minute stress test. Figure 3 isolates parent step latency during child decoding. When a child finishes, committing its answer text appends those tokens to the next parent step's prefill, adding a one-off +29 ms (30 tokens) to +35 ms (128 tokens), so the commit step finishes in 133–139 ms total. In a 10-minute continuous stress test where a new 128-token response is triggered every 2 seconds (maintaining up to M = 6 simultaneously active child streams), the median parent step latency is 149 ms with zero frame backlog.

W4 & Q4: Matched-Protocol Comparison with AURA (Table R4) & Role of Qwen3.5-9B

In both our protocol and AURA's released script, the model decides every second whether to stay silent or respond; our protocol feeds one continuous full-history stream with the model's own committed outputs, whereas AURA's script rebuilds the prompt every second from the latest 30 one-second clips with template-filled silent turns. We evaluated both models under both protocols on OmniMMI PA:

Table R4: Cross-protocol evaluation between AURA and UniVIA on OmniMMI PA (Qwen3-VL-8B backbone, %).

Model (Qwen3-VL-8B backbone)UniVIA Continuous Protocol (PA ↑)AURA Released Protocol (PA ↑)
AURA36.037.5 (reported)
UniVIA (Ours)51.5 (+15.5)57.5 (+20.0)
  1. Consistent +15.5 to +20.0 PA gain & same-backbone attribution. UniVIA outperforms AURA under both protocols (+15.5 PA and +20.0 PA). Furthermore, all method claims in Table 1 are established on the same Qwen3-VL-8B backbone (improving over Qwen3-VL-8B-Instruct by +8.8 RTVU, +20.2 OSU, +21.2 MCU, +30.0 ACU, +19.2 RTVP, +22.0 SQA, +19.4 BT, +21.2 FAR, and +11.6 PO). UniVIA (Qwen3.5-9B) (58.0 PA / 70.0 PO; 73.0 on VideoMME w/o sub.) is included only to show scaling to a newer backbone (clarified in Section 4.1).

W5 & Q5: Complete Methodological Details in Appendix A.2

We have filled in all specifications in Appendix A.2: (1) Data mixture (by input tokens): 35% Streaming (refined JoyAI 8.9%, in-house streaming QA 8.8%, streaming narration 16.3%, TimeLens2 1.1%) + 36% Offline VQA (images 22.5%, videos 13.6%) + 29% Offline QVA (question-first). (2) QVA pilot study (Qwen3-VL-8B-Instruct): On VideoMME w/o subtitles (2,700 QA pairs), placing the question before video frames drops accuracy from 71.5 to 59.2 (-12.3 pts) across short (79.4 to 71.2), medium (72.1 to 57.8), and long (63.0 to 48.7) videos. (3) Simulated delay: δⱼ = min(15 s, (3 s + cⱼ / 40) × uⱼ) with uⱼ ~ Uniform[0.7, 1.3], committed at the first frame step ≥ τⱼ + δⱼ. (4) Hyperparameters: Full fine-tuning for 8,817 steps, batch size 64 × 32,768 tokens, peak LR 1e-5 (200 warm-up steps, cosine decay), BF16, 4,096–131,072 pixels/frame. We would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Further Discussion or Feedback

Dear Reviewer Hvfc,

As the discussion period is approaching its end, we would greatly appreciate any further feedback on our responses, especially regarding whether our clarifications have addressed your main concerns.

Thank you again for your time and consideration!

Thank you for the detailed response and additional experiments. The new ablations and cross-protocol comparisons address several of my initial concerns and strengthen the empirical evidence.

However, the novelty remains relatively incremental, and the contribution of delayed answer visibility is still not directly validated through a controlled ablation. Additionally, the interpretation of wall-clock response latency could be clarified further.

Overall, I appreciate the additional efforts, but the remaining concerns regarding novelty and the validation of the central training design lead me to maintain my Weak Reject recommendation.

Reviewer_1uySOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UniVIA supports offline video understanding and duplex streaming interaction with shared model parameters. A persistent parent stream observes video and decides when to respond; a child stream generates a complete answer from a fixed KV-cache snapshot and commits the completed text to the parent's memory. The training recipe combines response-ended streaming samples, simulated delayed commits, question-first offline QA serialization, and Segment-Balanced Loss (SBL). Experiments cover streaming interaction, offline video understanding, and perception latency during answer generation. The primary contribution is a systems design and an associated training formulation for maintaining perception while producing complete answers.

Critical strengths
  1. A concrete design for concurrent perception and complete-answer generation. Sections 3.2–3.3 explicitly define the observations available to an answer and the point at which it becomes part of the parent's memory. This state separation, together with corresponding training samples, is a useful contribution to deployable streaming video assistants. Figure 3 provides relevant evidence that it reduces perception blocking on the measured shared-accelerator workload.

  2. Broad evaluation with visible trade-offs. The paper evaluates several streaming benchmarks and offline tasks, rather than optimizing only one setting. It also reports offline regressions in Table 2. This supports the practical value of studying a single model across the two task types, even though the source of the gains needs stronger attribution.

Critical weaknesses
  1. The key training claims are not causally established. Table 3 omits PA/PO for the streaming-only variant, the most direct test of whether offline QA benefits streaming interaction. That variant's three reported offline scores exceed the full recipe: 71.1/76.0/52.8 versus 70.3/75.7/52.1. Thus, simultaneous support for both modes is demonstrated more clearly than mutual benefit from joint training. In addition, Section 4.4 states that the SBL comparison changes the total control-to-answer loss weight, approximately from 2 to the segment length. Table 4 therefore does not isolate the proposed segmentation/class balancing from a stronger control objective. These gaps affect the central explanation for the gains.
  2. The principal superiority claims rely on incompletely controlled and specified comparisons. Sections 4.1–4.2 acknowledge that AURA's streaming results use a different protocol, yet these support several prominent margins. History, image budgets, and deployment settings also differ across configurations. Appendix A.2 still contains unresolved placeholders such as [sources] and [models], while important data and delay-simulation details remain unspecified. Disclosing protocol differences is welcome, but same-protocol results and a reproducible recipe are needed to attribute the improvements to UniVIA.
Questions for authors
  1. Clarification and decisive missing result: Can you provide PA/PO for the streaming-only variant in Table 3, clarify the training-token budgets, and identify which controlled results establish positive transfer in each direction?
  2. Decisive control: How much of SBL's advantage remains when the comparator has a tuned control-to-answer loss coefficient? Can the key result be checked closer to the final training scale?
  3. Comparability and reproducibility: Do the principal PA/PO gains persist under matched AURA/UniVIA history, image, decoding, and scoring settings? Please also resolve the Appendix A.2 placeholders and specify the data mixture and delay-simulation parameters needed to interpret the comparison.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 1uyS (1/3)

Thank you for the sharp and constructive review. Every control experiment you suggested was spot-on, and we have executed all of them. Below, we present the Streaming-only ablation under matched token budgets (Table R1) in Part 1/3, the control-loss coefficient sweep on 1/N vs. SBL (Table R2) in Part 2/3, and the bi-directional AURA evaluation (Table R3) plus complete Appendix A.2 specifications in Part 3/3.

W1 & Q1: Streaming-Only Ablation and Token Budget (Table R1)

Good catch on Table 3. Row (d) had a label typo in the submitted manuscript: it is actually the offline-only specialist (trained without streaming data, so it never emits silence tokens and PA/PO are n/a). We have relabelled row (d) as Offline only (w/o streaming data) and added the true Streaming-only variant (trained exclusively on streaming data without offline QA) as row (e) below (alongside row (b) w/ JoyAI data: 21.0 PA / 10.8 PO / 69.1 / 73.2 / 50.1 from Paper Table 3). All rows use the exact same token budget (1K steps × 64 packed sequences × 32,768 tokens ≈ 2.1B tokens); when a data subset is removed, the remaining sources proportionally fill the same 1K-step budget.

Table R1: Controlled training-data ablation under matched 1K-step token budgets (extends Paper Table 3; Qwen3-VL-8B, %).

Training Variant (Matched 1K-Step Budget)PA (OMMI) ↑PO (SB) ↑VideoMME w/o ↑VideoMME w/ ↑LVBench ↑
(a) UniVIA Full Recipe (Ours)42.521.270.375.752.1
(c) Replace QVA with Video-First VQA Order37.018.870.177.053.1
(d) Offline Only (Relabelled)n/an/a71.176.052.8
(e) Streaming Only (w/o Offline QA)36.318.166.872.448.5
  1. Offline QA to Streaming (+6.2 PA / +3.1 PO from QVA). Comparing row (a) against row (e) shows that adding offline QA in Question-First (QVA) order boosts proactive streaming from 36.3 to 42.5 PA (+6.2) and 18.1 to 21.2 PO (+3.1). By contrast, adding standard video-first VQA (row c vs. row e) restores offline accuracy (66.8 to 70.1 on VideoMME w/o) but barely moves streaming (+0.7 PA / +0.7 PO). Placing the question before the video frames (QVA, row a vs. row c) drives +5.5 PA / +2.4 PO because it matches the exact causal ordering seen by a forked child stream when visual evidence arrives.
  2. Streaming to Offline (retaining >99% of specialist accuracy). In the reverse direction, streaming data is not meant to beat a 100%-offline specialist on static videos, but our unified recipe eliminates the sharp offline drop of streaming-only training (row e: 66.8 / 72.4 / 48.5) and stays within 0.3–0.8 points of the offline-only specialist (row d) across all three benchmarks. We have updated Sections 1 and 4.4 to state this directional relationship precisely.
AuthorsOpenReview ↗

Response to Reviewer 1uyS (2/3)

W1 & Q2: Segment-Balanced Loss (SBL) vs. Tuned Control-Loss Coefficient on 1/N (Table R2)

As noted in Section 4.4, both SBL and the 1/N silence-weighting baseline (Lu et al., 2026) balance silence against response tokens inside the active segment Wₖ, but they differ in how much the control steps weigh relative to the Lₖ answer tokens:

  • Standard 1/N (λ = 1): Assigns total weight 1 to all silence steps and 1 to all response steps, giving a fixed total control weight of 2 against Lₖ answer tokens.
  • SBL (Eq. 4): Assigns per-token weight ∣Wₖ∣ / (2 n(c)), giving ∣Wₖ∣ / 2 to each control class (an adaptive total control weight of ∣Wₖ∣ against Lₖ answer tokens) and restricting control supervision strictly to the active segment Wₖ.

To test whether SBL's advantage comes simply from a larger control weight, we scale the control loss of the 1/N baseline by a multiplier λ ∈ {1, 4, 8, 16} (giving total control weight 2λ against Lₖ answer tokens, where λ = 8 yields 2λ = 16 ≈ mean ∣Wₖ∣, matching SBL's average control weight):

Table R2: Sweeping the control-loss coefficient λ on the 1/N baseline vs. SBL (extends Paper Table 4; 1K steps, %).

Loss Setting (1K Steps, Qwen3-VL-8B)Total Control Weight vs. LₖPA (OMMI) ↑PO (SB) ↑
1/N silence weighting, λ = 1 (Paper Table 4)215.58.4
1/N silence weighting, λ = 4819.610.3
1/N silence weighting, λ = 8 (matches avg. segment length)16 (≈ mean ∣Wₖ∣)21.811.3
1/N silence weighting, λ = 163219.49.8
w/o last-segment loss (Paper Table 4)Prefix length τₖ31.713.6
SBL (Ours, Paper Table 4)Adaptive ∣Wₖ∣42.521.2
  1. A global multiplier λ saturates at 21.8 PA / 11.3 PO. Raising λ from 1 to 8 lifts 1/N from 15.5 to 21.8 PA and 8.4 to 11.3 PO, confirming that default 1/N (λ = 1) under-weights control tokens against Lₖ. However, performance plateaus at λ = 8 and drops at λ = 16 (19.4 PA / 9.8 PO): a fixed global weight over-penalizes short segments (causing early false triggers) while still under-weighting long segments and diluting answer supervision Lₖ.
  2. Adaptive ∣Wₖ∣ scaling + last-segment localization drives +20.7 PA / +9.9 PO. Scaling control weight proportionally to each sample's active segment length ∣Wₖ∣ and supervising only the active segment Wₖ (+10.8 PA / +7.6 PO over w/o last-segment loss) allows SBL to outperform the best-tuned 1/N setting by +20.7 PA and +9.9 PO.
AuthorsOpenReview ↗

Response to Reviewer 1uyS (3/3)

W2 & Q3: Bi-Directional Matched-Protocol Comparison with AURA & Complete Appendix A.2

In both our protocol and AURA's released evaluation script, the model decides at every 1 s step whether to stay silent or respond, and the first response is scored against the ground-truth window. The difference lies in how history is fed: our protocol runs one continuous stream with the full video history and the model's own committed outputs, whereas AURA's script rebuilds the prompt every second from the latest 30 one-second clips with template-filled silent turns. Using AURA's released weights and evaluation code, we evaluated both models under both protocols on OmniMMI PA:

Table R3: Cross-protocol evaluation between AURA and UniVIA on OmniMMI PA (Qwen3-VL-8B backbone, %).

Model (Qwen3-VL-8B backbone)UniVIA Continuous Protocol (PA ↑)AURA Released Protocol (PA ↑)
AURA36.037.5 (reported)
UniVIA (Ours)51.5 (+15.5)57.5 (+20.0)
  1. Consistent +15.5 to +20.0 PA gain across both protocols. UniVIA outperforms AURA by +15.5 PA under our continuous full-history protocol and by +20.0 PA under AURA's windowed protocol, confirming that the gain comes from our architecture and training recipe rather than evaluation protocol differences.
  2. Complete specifications for Appendix A.2. We have filled in all placeholder numbers and formulas in Appendix A.2 of the revised manuscript:
    • Training data mixture (by input tokens): 35% Streaming (refined JoyAI event grounding 8.9%, in-house streaming QA 8.8%, streaming narration 16.3%, TimeLens2 1.1%) + 36% Offline VQA (images 22.5%, videos 13.6%, Sec. 4.1) + 29% Offline QVA (question-first).
    • QVA pilot study (Qwen3-VL-8B-Instruct): On VideoMME w/o subtitles (2,700 QA pairs), placing the question before video frames (QVA) drops accuracy from 71.5 to 59.2 (-12.3 pts) across short (79.4 to 71.2), medium (72.1 to 57.8), and long (63.0 to 48.7) videos, directly motivating explicit QVA training.
    • Simulated delay δⱼ (Sec. 3.3 & App. A.2): The conceptual delay δⱼ = Lⱼ / r + εⱼ is instantiated for an answer of cⱼ characters as δⱼ = min(15 s, (3 s + cⱼ / 40) × uⱼ) with uⱼ ~ Uniform[0.7, 1.3] (3 s prefill/scheduling offset + 40 chars/s nominal decoding), committed at the first frame step ≥ τⱼ + δⱼ.
    • Hyperparameters: Full fine-tuning (LLM, vision tower, projector) for 8,817 steps; global batch size 64 packed sequences × 32,768 tokens; peak LR 1e-5 (200 warm-up steps, cosine decay); BF16; 4,096–131,072 pixels/frame.

We hope these controlled experiments and complete specifications address all of your questions, and we would appreciate it if you would consider raising your rating.

Reviewer_spFdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes, UniVIA, a unified video intelligence architecture that jointly trains a shared vision-language model with offline and online supervision. The core idea is an asynchronous fork-and-commit procedure: when the model decides to respond, it forks a child stream that generates a complete answer from a fixed context snapshot, while the parent stream continues perceiving incoming video frames.

Critical strengths

The proposed method addresses a core problem of the previous streaming video models, that is, how to keep watching the video while also generating an answer. Earlier methods either paused the video stream to finish answer, or broke the answer into chunks, or used two separate models. The proposed method, UniVIA, runs the answer generation in a separate "child" stream that shares the same model weights, so the main stream can continue the perception. The experiments confirm this works well in practice, with good results on both streaming and offline benchmarks.

Critical weaknesses

The fork-and-commit idea is basically an egineering solution, rather than a novel contribution to a learning problem. Moreover, the paper only measures the latency reduction of the system, but does not evaluate how much extra GPU memory is needed, especially when multiple child streams are running at the same time. As the explicit design choice allows many child streams, it is unclear how much additional memory or complexity may be added, which is an important factor for assessing the real-world feasibility of the approach.

Questions for authors
  1. Does the child stream run the full parameter set as the parent stream, or does it use only a subset of the model?
  2. Is there a separate classification head for the silence/response decision, or is it handled by the standard language modeling head? When there are multiple queries, how does the model differentiate which control decision belongs to which query?
  3. Is each step exactly one video frame?
  4. It is unclear on what is the role of the dashed line 'Now' in Figure 2. There is no interaction between the parent and child until the child stream finishes the answer.
  5. How many queries are often sequenced in the training samples? If one query can receive multiple responses, how are these responses arranged in the training sample, and how is each response associated with its originating query?
  6. In Equation (4), the loss terms don't use ground-truth, so it seems not like a loss function. Is the ⟨response⟩ decision always emitted only at step touktou_k? Can the subsequent steps (such as touk+1tou_k+1) also emit ⟨response⟩ for the same query, since it also has the sufficient information to anwer the query? Additionally, should logpk,jlogp_{k,j} condition on the ⟨response⟩ token, given that a ⟨silence⟩ decision would never trigger a child stream?
  7. The metrics in Table 1 are not defined in the caption or surrounding text. It would be good to have a brief explanation.
6 comments
AuthorsOpenReview ↗

Response to Reviewer spFd (1/3)

Thank you for the thorough review and for highlighting the importance of maintaining continuous video perception while generating complete answers. Below, we address W1 and W2 (with new multi-stream memory and latency measurements in Table R1) in Part 1/3, Q1–Q5 in Part 2/3, and Q6–Q7 in Part 3/3.

W1: Why Fork-and-Commit is a New Multi-Stream Full-Duplex Paradigm

We agree that low-level KV-prefix sharing (PagedAttention) is an existing serving primitive, and we cite it strictly as infrastructure. Our core contribution is architectural and algorithmic: turning streaming video modeling from a single-output-stream pipeline into a multi-stream full-duplex paradigm (1 Parent + M Concurrent Children).

  1. Overlapping responses without blocking perception. Prior streaming VLLMs (AURA, Speak While Watching) operate on a single output timeline: while generating an answer, they must either block perception, chop answers into frame-step fragments, or drop overlapping triggers. In UniVIA, the parent stream only prefills frames and predicts one control token per step; it never blocks on text decoding and can fork a second or third child stream on the fly while earlier children are still decoding.
  2. Exact structural match with offline QA. Each forked child decodes a full, unfragmented answer from a frozen KV snapshot (parent prefix Hₜ followed by the response token). Every streaming response therefore shares the exact same sequence layout as offline QA, which lets Question-First (QVA) offline data boost proactive streaming from 36.3 to 42.5 PA (+6.2 PA) and 18.1 to 21.2 PO (+3.1 PO) under a matched 1K-step token budget (Table 3).
  3. Delayed-commit training (τⱼ + δⱼ). During the in-flight window [τⱼ, τⱼ + δⱼ), the parent continues receiving frames while the child is still decoding. Keeping the response token at τⱼ and inserting the completed answer text only at τⱼ + δⱼ trains the parent to recognize that an in-flight response is already underway and avoid duplicate triggers.

W2: GPU Memory and Runtime Latency with Concurrent Child Streams (Table R1)

A forked child stream adds zero model parameters and never copies the video KV history. It references the physical KV pages of the parent's snapshot and allocates new KV blocks only for the answer tokens it generates. For Qwen3-VL-8B (36 layers, 8 KV heads, 128 head dim, BF16), each token of KV cache is 0.14 MiB. Table R1 reports real measurements on an 80 GB A800 GPU at 1 FPS:

Table R1: Memory and parent-step latency scaling with M concurrent child streams (128-token answers, 5-min history on A800).

Metric (128-token answer per child)M = 1M = 2M = 4M = 8
Extra KV Memory (Shared Snapshot, UniVIA)20 MiB41 MiB81 MiB162 MiB
Extra KV Memory (Copying a 5-min History)4.9 GiB9.9 GiB19.8 GiB39.5 GiB
Parent Step Latency (104 ms with M = 0)127 ms130 ms190 ms241 ms
  1. 249× smaller KV memory footprint. Even with 8 concurrent child streams decoding at once, UniVIA adds only 162 MiB of KV cache (0.2% of an 80 GB GPU), compared to 39.5 GiB if duplicating a 5-minute video history.
  2. Real-time parent perception (< 0.25 s per 1 FPS step). Across M = 1–8 active children, each parent step finishes in 127–241 ms (well below the 1 s frame interval) using a standard continuous-batching scheduler without custom CUDA kernels.
AuthorsOpenReview ↗

Response to Reviewer spFd (2/3)

Q1–Q5: Architectural and Execution Clarifications

  1. Q1 (Does each child stream use the full parameter set?): Yes. The parent and all child streams share the exact same model weights θ with no separate adapters, sub-networks, or auxiliary heads. At each trigger step, the child decodes answer tokens from the frozen KV snapshot while the parent decodes one control token per frame step.
  2. Q2 (Is there a separate classification head, and how are multiple queries associated with responses?): There is no separate classification head. The silence and response control tokens are vocabulary tokens predicted directly by the standard LM head. A control token is not hard-coded to a synthetic query ID: emitting the response token simply indicates that at least one pending query has enough visual evidence to be answered now. Because the snapshot Sₜ contains all pending queries (with timestamps) and previously committed answers, the child stream naturally answers the satisfied query in text (clarified in Section 3.2).
  3. Q3 (Does each step process one video frame?): Yes. At our standard 1 FPS rate, each step t processes one new video frame (plus any user query or committed answer M that arrived since step t - 1) and outputs one silence or response control token. At higher frame rates, a step can pack multiple frames.
  4. Q4 (Role of the dashed line "Now" in Figure 2): Your reading is spot-on: the parent and child do not exchange tokens until the child finishes decoding and commits its answer. The dashed line "Now" in Figure 2 simply marks a single wall-clock instant during asynchronous execution, showing that while the child forked at step t is still mid-decoding, the parent has already moved on to steps t + 1 and t + 2. We have redrawn Figure 2 on a single wall-clock axis with an explicit commit arrow so this is immediately clear.
  5. Q5 (How many queries are in each training sample, and how are multiple responses ordered?): A multi-turn or multi-trigger stream with K responses is unpacked into K response-ended training samples X₁, ..., Xₖ, one ending at each response k ∈ {1, ..., K}. In sample Xₖ, the prefix H(τₖ) contains all frames and queries up to step τₖ, where every earlier response j < k keeps its response control token at its trigger step τⱼ while its completed answer text Aⱼ is inserted as committed memory Mⱼ at the first frame step at or after τⱼ + δⱼ. Queries and responses are linked through natural dialogue context and timestamps rather than synthetic IDs (illustrated in the revised Figure 5).
AuthorsOpenReview ↗

Response to Reviewer spFd (3/3)

Q6–Q7: Eq. (4) Notation and Benchmark Metrics in Table 1

  1. Q6.1 (Explicit ground-truth targets in Eq. 4): Eq. (4) is standard token-level cross-entropy. To avoid any ambiguity between random variables and ground-truth labels, we denote the ground-truth control target at step t as d*(t) ∈ {silence, response} and the j-th ground-truth answer token of response k as a*(k, j). In Eq. (4), pₜ = p(d*(t) given Hₜ) is the probability assigned to the ground-truth control token at step t, and p(k, j) = p(a*(k, j) given S(τₖ), a*(k, 1:j-1)) is the probability assigned to the j-th ground-truth answer token conditioned on the snapshot prefix S(τₖ) (parent prefix H(τₖ) followed by the response token). Thus, -log pₜ and -log p(k, j) are standard cross-entropy losses on the ground-truth tokens.
  2. Q6.2 (Why response is supervised at τₖ and silence at τₖ + 1): For a given response event, the ground-truth response label is placed strictly at τₖ (the earliest frame step with sufficient visual evidence, Eq. 5), while subsequent steps τₖ + 1, ... in the next segment are supervised with silence. Because the parent's context at τₖ + 1 already contains the response token emitted at τₖ, supervising post-trigger steps with silence is exactly how the parent learns not to fire duplicate triggers on the same query while the child stream is answering it. If a query genuinely requires periodic updates, each new response k forms a new segment Wₖ = [τₖ₋₁ + 1, τₖ] (Eq. 3).
  3. Q6.3 (Conditioning of log p(k, j)): Yes. As noted above and right below Eq. (4) in the paper, p(k, j) is conditioned on the snapshot prefix H(τₖ) followed by the response token.
  4. Q7 (Definitions of benchmark abbreviations in Table 1): All metrics in Table 1 are accuracies (%), and we have added the full glossary to the Table 1 caption and Section 4.1:
    • StreamingBench: RTVU (Real-Time Visual Understanding), OSU (Omni-Source Understanding), MCU (Misleading-Context Understanding), ACU (Anomalous-Context Understanding), SQA (Sequential QA), and PO (Proactive Output, scored as correct if triggered within ≤ 2 s of the ground-truth timestamp with the expected content).
    • OVO-Bench: RTVP (Real-Time Visual Perception), BT (Backward Tracing), and FAR (Forward Active Responding, the macro-average of REC, SSR, and CRR).
    • OmniMMI: PA (Proactive Alerting, scored as correct if the first triggered response falls inside the annotated ground-truth time window).

We hope these clarifications and system measurements fully address your questions, and we would be grateful if you would consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.