acceptodds
Under review as a conference paper at ICLR 2027

SyncPro: Synchronous Reasoning and Responding for Proactive Streaming Video Understanding

Abstract

Proactive streaming video understanding requires assistants to track unfolding events and respond when a user's requested condition is met. To make these decisions, assistants must accumulate relevant evidence and reassess response conditions as new observations arrive. Existing work offers complementary capabilities for this process: proactive video assistants learn when to respond, while streaming video reasoning methods generate intermediate text to organize observations and support subsequent answers. Together, these capabilities provide a foundation for proactive reasoning, but how to coordinate ongoing reasoning with timely responses remains underexplored. In particular, common reasoning formulations schedule updates within a single generation stream, so updates may be redundant or lag behind relevant state changes. The shared generation path also has to accommodate the distinct objectives of internal monitoring and user-facing task execution, while responses placed after reasoning must await its completion. To address these challenges, we introduce SyncPro, a synchronous dual-stream framework that separates internal monitoring from user-facing responses. Its Inner and Response branches share causal history and activate independently, allowing reasoning to follow relevant evidence and responses to proceed concurrently. We construct 189K dual-stream examples to train this framework. Experiments across three proactive benchmarks show improved response timing and task performance in event monitoring, repetition counting, sequential step tracking, and anomaly-triggered responses.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, MLLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YV3XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SyncPro, a dual-stream framework for proactive streaming video understanding. It separates continuous internal reasoning from user-facing responses through two specialized branches that share causal history while activating independently and decoding concurrently. The authors construct 189K dual-stream trajectories and employ supervised fine-tuning followed by trajectory-level reinforcement learning. Experiments demonstrate improvements in response timing and accuracy across several benchmarks. However, concerns remain regarding the methodological novelty, generalizability of Inner reasoning, insufficient validation of key components, and computational overhead.

Critical strengths

S1. Well-Motivated Problem Formulation.

Decoupling continuous internal reasoning from user-facing responses into two streams is a reasonable and promising direction for streaming video understanding. The proposed framework allows both streams to share causal history while activating independently and decoding concurrently.

S2. Dataset Construction.

The paper constructs a substantial dataset of 189K trajectories, providing supervision for proactive reasoning and responding. The data construction pipeline incorporates response reconstruction to improve the quality of the generated trajectories.

Critical weaknesses

W1. Unclear Research Gap and Novelty.

The paper does not clearly distinguish its contributions from existing multi-stream reasoning frameworks, such as [1] [2]. While several challenges are identified, the connection between these challenges and the proposed design could be better articulated. As a result, the novelty and necessity of the synchronous dual-stream framework remain unclear.

W2. Limited Evidence of Generalizable Inner Reasoning.

The Inner stream mainly focuses on query-specific observations, event counts, and intermediate states. It is unclear whether the model develops generalizable continuous reasoning capabilities or primarily learns to generate structured state summaries for predefined tasks. In addition, the accuracy, complexity, and long-term consistency of Inner reasoning receive limited evaluation, particularly for complex or unseen monitoring tasks.

W3. Insufficient Validation of Dual-Stream Coordination and Training Objectives.

The framework involves several architectural components and training objectives, yet their individual contributions are not fully examined. Notably, only 8.6K out of 13.52M training micro-turns (0.064%) involve simultaneous activation of both streams, raising questions about how effectively concurrent reasoning and responding are learned and evaluated. Moreover, the contribution of Action Temporal Ranking (ATR) is not directly assessed through ablation experiments.

W4. Limited Analysis of Computational Overhead.

While the paper demonstrates speedups from concurrent decoding, the additional computational cost of Inner reasoning relative to single-stream approaches is not fully explored. A more detailed analysis of the trade-off between reasoning benefits, computational overhead, and real-time efficiency would strengthen the evaluation.

Questions for authors

Q1. Quality of Synthetic Training Data.

Given that most training trajectories are synthetically generated, how do the authors ensure the factual correctness and diversity of Inner reasoning annotations? Beyond response reconstruction, is there any independent evaluation of the quality of the generated intermediate reasoning states?

Q2. Generalization and Performance Limitations.

Why does SyncPro underperform existing methods on certain ProactiveVideoQA subsets, particularly EGO, while also exhibiting degradation in general video understanding? Could these limitations result from task-specific specialization or information loss during state updates? Additionally, would similar performance trends be observed with other backbone models?

[1]Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs.

[2]Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models. CVPR 2026.

11 comments
AuthorsOpenReview ↗

Response to Reviewer YV3X (1)

Response to W1

  1. Research gap: proactive assistants and streaming reasoning methods are complementary, but how to coordinate them remains underexplored. Proactive assistants learn when to respond, and streaming reasoning methods generate intermediate text as frames arrive. The closest combination, ThinkStream, reasons at every input step on a fixed schedule, with reasoning and response arranged serially in one sequence. This leaves three requirements unmet:
  • Adaptive reasoning: updates should follow query-relevant evidence, since fixed schedules cause redundant or delayed updates (Figure 4).
  • Specialized roles: reasoning should maintain query-relevant state, while responses only fulfill the user's request.
  • Decoupled activation: responses supported by available evidence should not wait for ongoing reasoning.
  1. Method: each requirement maps to one design element, and each is supported by a dedicated experiment.
  • Independent per-micro-turn activation (⟨silent⟩/⟨inner⟩ decided from the shared history and current observation, trained with ATR) makes Inner updates selective, their timing matches the annotations well beyond random (Figure 5).
  • Separate LoRA routes specialize Inner for evidence tracking and Response for user-facing output, swapping them raises NLL on both branches (Table 5).
  • A block-causal mask within each micro-turn lets the branches decode concurrently, coupled only through completed history. The first response token arrives 199.52 ms earlier (Table 6), and Joint@2s rises from 17.2% (Interleaved) to 30.8% (Table 3).
  1. Comparison: SyncPro differs from Multi-Stream LLMs and Think-as-You-See (TaYS) in what is parallelized and how it is scheduled.
  • Multi-Stream LLMs align streams by format: all streams advance together at every forward pass, and "when to act" is not an optimization target. In SyncPro, Inner and Response each decide independently whether to activate on a frame-level time base, and neither waits for or gates the other.
  • TaYS parallelizes visual input with reasoning generation: it generates reasoning incrementally as frames arrive, using a parallel dual KV cache. SyncPro parallelizes two roles, internal evidence tracking and user-facing response, and treats response timing as the prediction target. TaYS reports no results on streaming proactive benchmarks. We compare against VST and ThinkStream, which also schedule reasoning at predetermined intervals, under their official protocols (Tables 1 and 2). Interleaved (Table 3) gives a controlled comparison of the one-sequence formulation under the same backbone, data and decoding budget.
  1. Data: SyncPro-189K is, to our knowledge, the first dataset with per-step supervision for both internal updates and user-facing responses in proactive streaming video. It has 189,403 trajectories from 12 sources, with 409K Inner and 377K Response annotations (Appendix E).

Response to W2

Inner maintains query-relevant state that supports later decisions. Measurements show it is factual, integrates multi-frame evidence, holds up on long streams, and transfers to a new benchmark without extra training.

  1. Inner serves as persistent reasoning: it maintains state rather than producing long reasoning text. Event counting, condition tracking and cross-time evidence integration are continuous reasoning even when expressed as concise state updates. Results on held-out benchmarks support generalization to new videos and questions.
  2. To address your concern, we added three measurements. | Aspect | Evaluation | SFT | RL | | ------------------------- | ------------------------------------------------------- | -------- | -------- | | Accuracy | Factual correctness of Inner (50 OVO-Bench questions) | 74.58% | 78.60% | | Multi-frame integration | Joint@2s, 75 multi-frame PO questions | 26.67% | 30.67% | | Long streams | F1@2s, 7 OVO-REC questions over 180 s | 33.48% | 36.65% |
  3. Inner transfers to a new benchmark without additional training or tuning. We evaluated the existing model on the full ProReady-QA, a recent benchmark that scores content correctness (Acc) and timing (ARS), and it performs competitively in both. An audit of 50 questions covering the five task types gives a verifiable factual correctness of 88.24% for the generated Inner. | Method | SSR Acc | SSR ARS | CRR Acc | CRR ARS | REC Acc | REC ARS | GSD Acc | GSD ARS | CTD Acc | CTD ARS | | ----------------- | -------------- | ------------- | -------------- | ------------- | ------------- | ------------- | ------------- | --------- | -------------- | ------------- | | VideoLLM-online | 53.20 | 0.30 | 51.1 | 0.38 | 18.3 | 0.12 | 30.5 | 0.34 | 24.90 | 0.44 | | Dispider | 47.20 | 0.59 | 52.3 | 0.55 | 20.3 | 0.20 | 34.2 | 0.33 | 27.20 | 0.41 | | StreamForest | 65.80 | 0.41 | 57.4 | 0.51 | 24.4 | 0.24 | 49.2 | 0.29 | 32.3 | 0.40 | | SyncPro | 60.51 | 0.57 | 54.58 | 0.50 | 33.10 | 0.21 | 46.3 | 0.49 | 27.14 | 0.43 |
AuthorsOpenReview ↗

Response to Reviewer YV3X (2)

Response to W3

Thank you for the attention to the architecture design. Each component is supported by a single-factor control, concurrent activation is learned from full-trajectory supervision rather than from joint-activation turns alone, and removing ATR lowers both StreamingBench-PO and OVO-FAR.

  1. Every component has a single-factor control that favors the full design. All controls use the same backbone and source trajectories.
ComponentControlControl vs. SyncPro
Internal-update targetResponse-onlyPO Joint@2s 10.4 vs. 30.8
Independent branch decisionsInterleaved (one shared activation sequence)PO Joint@2s 17.2 vs. 30.8, Exact Content Acc.@Response 8.98 vs. 45.24
Role-specialized routesSwap the Inner and Response LoRA routesReference-token NLL rises by 0.706 (Inner) and 1.578 (Response)
State-merge weightsLearned per branch vs. equal (γ = 0.5)OVO-FAR mean F1 30.71 vs. 32.37
ATRλ_ATR = 0PO Acc@2s 36.40 vs. 38.80, OVO-FAR F1 (CRR/SSR/REC) 25.66/10.22/34.08 vs. 28.06/21.36/47.69
Trajectory-level RLSFT onlyPO Joint@2s 30.8 vs. 33.6
Execution scheduleInner-first serial decodingReal-time Joint@2s 21.2 vs. 24.0, 615 vs. 498 ms/frame
  1. Concurrent activation is learned from action supervision on every micro-turn and is effective at the level of action timing, Inner content and final responses. Both branches receive action supervision (⟨silent⟩ or active) at all 13.52M micro-turns and learn content generation separately, over 409K Inner entries and 377K Response outputs, so learning does not depend on the joint-activation turns alone. On action timing, the mean per-case action-event F1 on 50 rollouts is 0.71 for Inner and 0.61 for Response. On Inner content, an audit of 50 OVO-Bench questions gives a direct query-relevance rate of 90.24% (SFT) and 91.41% (RL). On final responses, SyncPro-SFT reaches 30.8% Joint@2s versus 17.2% for Interleaved.

  2. To address your concern, we ablated ATR, and it helps the model learn trigger timing. ATR resolves a trade-off in silent weighting: raising the silent weight from 0.1 to 0.2 lowers the false-trigger rate on repeated-silent steps from 17.28% to 9.76%, but also lowers recall to 12%, since suppressing repeated speech also suppresses the positions where the model should speak. Adding ATR improves every metric:

Setting (SyncPro-SFT)PO Acc@2sOVO F1 CRROVO F1 SSROVO F1 REC
w/ ATR (2.5/0.2/3)38.8028.0621.3647.69
w/o ATR (2.5/0.2/3)36.4025.6610.2234.08
Δ+2.40+2.40+11.14+13.61

Response to W4

Thank you for the suggestion. Inner adds about 20% per-frame latency (414.2 →\rightarrow 517.4 ms), and in return it improves content accuracy by 7.6 points, response rate by 8.4 points and Joint@2s by 7.2 points (all statistically significant), while 91.5% of frames still complete within 1 second.

  1. The cost of Inner is small relative to its gains. To address your concern, we ablated Inner directly. Disabling it saves 103.3 ms per frame and raises the share of frames completed within 1 s from 91.5% to 99.9%, but lowers content accuracy by 7.6 points, response rate by 8.4 points and input-frame-anchored Joint@2s by 7.2 points. Because Inner is activated adaptively rather than at every step, this cost is paid only when the evidence changes.

  2. Concurrent execution offsets part of the cost. Compared with running Inner first and then Response, running the two branches concurrently finishes 114.74 ms earlier (1.37× faster) and delivers the first response token 199.52 ms earlier (2.22× faster) (Table 6). The two branches produce identical outputs on all 1,870 held-out examples, so the speedup does not change the results.

  3. Memory stays stable on long streams, and latency grows gradually with retained context. On a 480-s stream at 1 fps, persistent GPU allocation changes by only 1.1 MiB, and the per-frame peak stays at 11.75 GiB while retained KV grows from 2,126 to 11,220 tokens (Figure 6). Dual-branch decoding latency grows by 15.6 ms per 1k retained KV tokens when only Response is active and 32.4 ms when both branches are active, and 74.9% of frames meet the 1-s budget over the full stream. Evicting stale context is a straightforward way to flatten this growth.

Revision: Appendix I reports the quality and runtime cost of Inner updates (Table 13).

AuthorsOpenReview ↗

Response to Reviewer YV3X (3)

Response to Q1

Thank you for the concern. Inner annotations go through quality control beyond response reconstruction, and an independent audit of 100 trajectories directly measures their factuality (88.34% correct among verifiable facts) and cross-turn consistency (94.68%).

  1. Inner annotations are generated without future information and then revised in several rounds. Each Inner entry is generated from the video truncated to the current moment. Multi-round revision then reviews content, timing and event coverage, removes redundant entries and adds missed state changes. Response reconstruction further checks that the retained internal states suffice to support later answers.

  2. SyncPro-189K covers diverse scenarios and task types rather than a single state-summary template. It combines proactive query synthesis with adaptation of existing temporal annotations, spanning 12 video sources, 3 interaction functions and 8 task subtypes. The supervision includes factual QA, counting, state monitoring and continuous assistance.

  3. Human review and an independent audit confirm quality. After three rounds of revision, we manually checked 5,000 synthesized trajectories, and human judgments agree with the LLM quality judgments in about 89.7% of cases. To address your concern about intermediate states, we audited 100 randomly sampled trajectories:

Metric (100 trajectories, 803 atomic facts)Result
Atomic facts supported by the visuals84.31%
Correct among verifiable facts88.34%
Cross-turn consistency94.68%

Response to Q2

SyncPro stays within 2.0 points of its base model on general video understanding, and the lower subsets on ProactiveVideoQA trace to the focus of our supervision, with a clear path for improvement.

  1. On WEB, SyncPro ranks second (48.28), and EGO has clear room for improvement. To address your concern, we audited 50 low-scoring response windows on each subset. The main WEB issue is incomplete key points (38%), and the main EGO issue is missed or late triggering (76%). Our annotations emphasize concise answers after an explicit trigger, while WEB's multi-point descriptions and EGO's long-horizon step tracking require broader content coverage and continuous updating. These are the directions for improvement.

  2. General video understanding is largely preserved without mixing in general VQA data. Averaged over MVBench, EgoSchema and Video-MME, SyncPro-RL is 1.68 points below the base model, and only 0.06 points below SFT.

  3. Other streaming adaptation work on different backbones also reports small drops on general benchmarks. StreamBridge reports a 2.6-point drop on MVBench (Qwen2-VL-7B) and a 2.0-point drop on LongVideoBench (LLaVA-OneVision-7B) [1], and MMDuet2 shows a small drop on LongVideoBench with Qwen2.5-VL-3B [2]. Our goal is continuous state updating with timely, correct responses: under the same backbone and data, SyncPro-SFT raises Joint@2s from 17.2% (Interleaved) to 30.8%, while keeping general ability close to the base model.

[1] StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant.

[2] MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning.

Reviewer_UrXKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SYNCPRO for proactive streaming video understanding, where the model needs to keep track of an ongoing video and respond when the requested condition is met. The main idea is to separate internal reasoning/evidence tracking from response generation using two branches that share the same causal history but can operate independently. This avoids relying on fixed reasoning intervals and allows the model to respond without waiting for an ongoing reasoning step to finish. The authors train the model on 189K dual-stream examples using supervised fine-tuning and trajectory-level reinforcement learning. Results on three proactive video benchmarks show gains in both response timing and answer accuracy.

Critical strengths
  • The proposed 189K trajectories with per-step supervision for internal updates and user-facing responses will be useful for the community.

  • The proposed solution with shared backbone is simplistic without any complicated design.

Critical weaknesses
  • It is not clear how this two stream idea is different from prior works which use multi-stream such as Multi-Stream LLMs, Asynchronous Reasoning, and Mind-Paced Speaking. Also, this problem can be solved using proactive approaches as well, such as Streambridge, Streamready, Streammind, etc., it is not clear what is the need of such two steams in this problem setup. Ideally an agent based approach could be more generalized solution for this. These relevant works should be discussed in more detail to differentiate the novelty.

  • It seems all the scores from prior works are from a different backbone and the authors have used a more recent variant. Considering this, it is hard to evaluate the performance improvements shown in the paper. Also, the margins are small and no variance is shown.

  • There might be some overlap between the training set used and the testing videos. Most of those datasets used to prepare the training set have overlapping videos and some filtering might be required. It is not clear if any such curation was done.

Questions for authors
  • It will be good to discuss more on why performance is not good on WEB and ego-centric videos.

  • The authors should either train prior works on selected backbone or show results using consistent backbone for a fair comparison. Also, please train some of the baselines with similar data and budget for a fair comparison.

  • The authors should also show performance on Proready-QA which evaluates response timing in addition to correctness.

  • The figures are so hard to read, I was not able to understand most of the figures.

11 comments
AuthorsOpenReview ↗

Response to Reviewer UrXK (1)

Response to W1

SyncPro's novelty is a formulation of proactive streaming reasoning built on three requirements (adaptive reasoning, specialized roles, decoupled activation), a dual-stream design that meets all three, and, to our knowledge, the first dual-stream supervision data for this setting. Existing multi-stream, gating-based and agent methods address parts of these requirements, but not all three together.

  1. Motivation: proactive streaming imposes three requirements on how a model should reason and respond.
  • Adaptive reasoning. Reasoning should follow query-relevant evidence, since fixed schedules cause redundant or delayed updates.

  • Specialized roles. Reasoning should maintain query-relevant state, while responses should only fulfill the user's request.

  • Decoupled activation. Responses supported by available evidence should proceed without waiting for ongoing reasoning.

  1. Method: each requirement maps to one design element.
  • Independent activation of two branches lets each decide at every micro-turn whether to generate or stay silent, so internal updates follow the evidence rather than a fixed schedule (as in ThinkStream and Mind-Paced Speaking).

  • Separate low-rank adapters specialize Inner for evidence tracking and Response for user-facing output (Table 5), whereas prior methods generate both behaviors through a single path, or gate responses without an explicit reasoning trace (StreamBridge, StreamMind, StreamReady).

  • Concurrent decoding on a shared history lets responses proceed without waiting for reasoning (first response token 199.52 ms earlier, Table 6), whereas Asynchronous Reasoning and Multi-Stream LLMs tie response timing to reasoning progress.

  1. Comparison: SyncPro combines the three elements in one model, and each is supported by controlled results. Under the same backbone, data and decoding budget, Joint@2s is 10.4% for Response-only, 17.2% for Interleaved and 30.8% for SyncPro-SFT (33.6% after RL) (Table 3). Compared with agent pipelines (e.g., Dispider, StreamAgent), SyncPro makes trigger decisions inside one model from the accumulated reasoning state, with no interface between modules to synchronize.

  2. Data: SyncPro-189K is, to our knowledge, the first dataset with per-step supervision for both internal updates and user-facing responses in proactive streaming video. It has 189,403 trajectories from 12 sources, with 409K Inner and 377K Response annotations. It provides the supervision that both SFT and trajectory-level RL need for the two behaviors (Appendix E).

Response to W3

SyncPro-189K is built only from the official training splits of its source datasets, and we filtered by video ID so that no training video overlaps with the evaluation videos. We never use the validation or test splits of any source, and StreamingBench, OVO-Bench and ProactiveVideoQA are reserved for evaluation. For UCF-Crime, which is a source of both our training data and ProactiveVideoQA, we kept only training videos whose IDs are disjoint from the evaluation videos. The same ID check against StreamingBench and OVO-Bench found no overlap.

Response to Q1

SyncPro transfers well to WEB and EGO even though neither was a design target: it ranks second on WEB (48.28) among compared methods and clearly outperforms the Interleaved and Response-only controls on both subsets. EGO is where the most headroom remains, and our visual analysis traces it to a specific, addressable cause.

  1. The dual-stream design brings consistent gains on WEB and EGO without dedicated tuning. Our supervision centers on concise, condition-triggered responses, while WEB and EGO stress content coverage and continuous updating. Under the same backbone and data, SyncPro-SFT still improves over Interleaved on WEB (29.89 →\rightarrow 47.83) and EGO (25.00 →\rightarrow 33.89), and SyncPro-RL further reaches 48.28 and 33.74. The gains also extend to VAD (32.43 vs. 25.91 for Interleaved).

  2. Our visual case analysis of 50 low-scoring response windows per subset identifies one main factor on each, and each is tied to the task style. On WEB, it is key-point coverage (incomplete in 38% of windows), because multi-point descriptions need broader content coverage than our concise-answer annotations target. On EGO, it is trigger timing (missed or late in 76% of windows), because long-horizon, fine-grained step tracking needs continuous response updating, while our annotations focus on responses after an explicit trigger.

  3. These factors point to clear improvements on the data and supervision side. Adding multi-point answer coverage for WEB and continuous-event updating for EGO would target the main factors directly.

Revision: We add the WEB/EGO error analysis to Section 4.2, with the detailed audit in Appendix G (Table 11).

AuthorsOpenReview ↗

Response to Reviewer UrXK (2)

Response to Q2 and W2

SyncPro's gains are attributable to the method itself: we provide a same-backbone, same-data control in which SyncPro-SFT leads Interleaved by 13.6 points of Joint@2s, and the cross-method comparison follows the standard protocol for this field.

  1. The controlled comparison isolates the contribution of our design. SyncPro-SFT and Interleaved both use Qwen3.5-4B, the same source trajectories and the same decoding budget. On PO, containment-matched Joint@2s is 30.8% vs. 17.2% (+13.6 points), and Acc@2s is 38.8 vs. 34.0 (Tables 1 and 3). SyncPro-SFT also leads on OVO-FAR (Avg. F1 32.37 vs. 10.87). This shows the value of the dual-branch design under identical conditions.

  2. Comparing methods that choose their own backbone and training scheme is the common protocol in proactive video understanding. The field aims to improve response timing accuracy while keeping answers correct, and each work adopts its own backbone and training scheme to reach this goal. For example, VideoLLM-online uses Llama-3-8B [1], MMDuet uses LLaVA-OneVision-7B [2], and Em-Garde uses Qwen2.5-VL-7B [3]. Their streaming training data also differ in scale, including 134K streaming dialogues [1], 109K training samples [2], and 668 queries from 92 videos [3]. We follow this protocol and compare final performance on shared benchmarks.

  3. The backbone and data scale do not explain SyncPro's advantage. SyncPro uses a 4B model, smaller than the 7B–8B baselines (VST, Em-Garde, VideoLLM-online) and close to ThinkStream (3B). None of the base models has native streaming ability, so the proactive behavior comes from the formulation and data. On general benchmarks, SyncPro stays within 2.0 points of Qwen3.5-4B (Table 4), so the gains do not come at the cost of general ability.

[1] VideoLLM-online: Online Video Large Language Model for Streaming Video.

[2] VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format.

[3] Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding.

Revision: Sections 4.1 and 4.3 specify the external-baseline protocols and the shared backbone, source data, input rate, and decoding budget used for the controlled comparisons.

Response to Q3

Thank you for pointing to the joint evaluation of response timing and answer correctness. On ProReady-QA, SyncPro obtains the highest mean ARS (0.44) and the second-highest mean Acc (44.33) among the compared methods, and ProactiveVideoQA in our main table is also a benchmark that scores timing and content jointly.

  1. Our existing evaluation is already a joint one. PAUC (ω = 0.5) combines answer quality and response timing in one score. On ProactiveVideoQA, SyncPro ranks second on WEB (48.28) and first on VAD (32.43).

  2. The ProReady-QA results show strong overall balance between content and timing. To address your concern, we evaluated SyncPro on ProReady-QA, where Acc measures content correctness and ARS measures response timing. SyncPro has the best mean ARS and the best results on REC (Acc) and GSD (ARS), and its mean Acc is close to the best.

MethodAvg AccAvg ARSREC AccREC ARSGSD AccGSD ARSSSR AccSSR ARSCRR AccCRR ARSCTD AccCTD ARS
VideoLLM-online35.600.3218.30.1230.50.3453.200.3051.10.3824.900.44
Dispider36.240.4220.30.2034.20.3347.200.5952.30.5527.200.41
StreamForest45.820.3724.40.2449.20.2965.800.4157.40.5132.30.40
SyncPro44.330.4433.100.2146.30.4960.510.5754.580.5027.140.43
  1. SyncPro's gains are largest where answering depends on tracking and judging evidence. It leads by a clear margin on REC Acc (33.10 vs. 24.4 for the next best) and on GSD ARS (0.49 vs. 0.34). ProReady-QA asks the model to answer when the supporting evidence appears, so early and late answers are both penalized, and SyncPro's timing is consistently competitive across tasks. Our training targets concise answers after explicit condition triggers, so tasks with different answer styles (e.g., CTD Acc, CRR ARS) leave room for further improvement.

Revision: We add the ProReady-QA evaluation to Appendix H (Table 12).

AuthorsOpenReview ↗

Response to Reviewer UrXK (3)

Response to Q4

To help reading, we summarize what each figure shows below, and we have revised the figures and captions accordingly.

  1. Figures 1–3 present the problem, the model and the data. Figure 1 shows three requirements of proactive streaming reasoning: adaptive reasoning, specialized roles, and decoupled activation. Figure 2 shows the dual-stream model, where Inner and Response select actions independently from the same prefix and only Response is user-visible. Figure 3 shows how SyncPro-189K is constructed and what it covers.

  2. Figures 4–5 and 7 show model behavior. Figure 4 is a rollout with scheduled Inner activation, which gives repeated and redundant updates. Figure 5 shows SFT learning dynamics: validation content-token accuracy and action-event F1 over training. Figure 7 gives case studies of Inner updates and Response outputs.

  3. Figure 6 shows long-stream cost. It reports memory and latency of dual-branch inference on a 480 s stream.

Revisions: We unify the Inner/Response notation in Figures 1 and 3, clarify the error markers in Figure 4, and expand the captions of Figures 2, 4, and 5 to explain their components, colors, and symbols. We are happy to provide further clarification if needed.

Reviewer_u2H6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies proactive streaming video understanding, where a model observes frames at 1 fps and must respond at the moment a user-specified condition becomes visible, and it identifies three limitations of current streaming reasoning methods: reasoning on a fixed schedule, a single generation path shared by state tracking and user-facing output, and responses that must wait for reasoning to finish. SyncPro attaches two LoRA routes (rank 64, MLP projections only) to a frozen Qwen3.5-4B, an Inner branch that records query-relevant state and a Response branch that answers; at each micro-turn both branches choose between a silent token and content, decode from the same prefix under a block-causal mask without seeing each other's current output, and are then merged into the history by KV concatenation in full-attention layers and by averaging the two branch states in Gated DeltaNet layers. Training data (SyncPro-189K, 189,403 trajectories from 12 sources) are produced by LLM-based synthesis on truncated prefixes and by adaptation of temporal annotations, and synthesized trajectories are kept only if their Inner entries allow every later response to be reconstructed once earlier frames are removed. The model is trained with action and content cross-entropy plus a local ranking loss at response boundaries (ATR), then with GRPO on 800 trajectories using hard-window F1 as the only reward. It reports 41.2% Acc@2s on StreamingBench Proactive Output, 34.19 mean F1 on OVO-Bench Forward Active Responding and 32.43 PAUC on the VAD subset of ProactiveVideoQA, and the controlled comparison in Table 3 shows that separate Inner supervision and separate routes raise Joint@2s from 10.4% (Response-only) and 17.2% (Interleaved) to 30.8%.

Critical strengths
  1. This work proposes to separate evidence tracking from responding at the level of the model rather than the prompt. Inner and Response have their own action spaces and adapters, share the encoded prefix, and write into one history, and the paper specifies how this is realized in a hybrid decoder, including the KV layout for full-attention layers and the state update for linear-attention layers. The adapter swap in Table 5 (reference NLL rises by 0.706 for Inner and 1.578 for Response) confirms that the two routes learn different distributions.
  2. This work contributes a dual-stream training corpus whose filtering criterion is tied to the purpose of the internal state. Requiring that each later response be reconstructable from the accumulated Inner entries and only the most recent visual window is a sensible test of whether the written state is sufficient, and it is stricter than filtering on response correctness alone. The corpus is large (13.52M micro-turns, 409K Inner entries, 377K responses) and is built only from official training splits.
  3. This work isolates the formulation from the backbone and data in Table 3. Under the same backbone, source trajectories and decoding budget, removing Inner targets lowers Joint@2s from 30.8% to 10.4%, and merging both roles into one route lowers exact content accuracy on responded samples from 45.24% to 8.98%. The timing analysis in Appendix D adds that learned Inner updates match annotated update times far more often than random times with the same count (56 versus 14.3 exact matches on 79 reference updates).
Critical weaknesses
  1. Lack of evidence that concurrent decoding contributes anything beyond latency, and lack of an evaluation in which latency is measured at all. Section 4.4 reports that Inner-first serial execution produces identical outputs on all 1,870 held-out examples, so the accuracy numbers in Tables 1 to 3 would be unchanged under serial decoding, and the only difference is about 115 ms to finish both branches and about 200 ms to the first response token (Table 6). All benchmarks are frame-indexed, so this saving never enters a score. The scenario that motivates Figure 1(c) is also nearly absent from training: joint activations are 8.6K of 13.52M micro-turns, and the synthesis route places the final Inner update and the Response in different micro-turns, which builds in a one-step (one-second) delay between the evidence and the answer that the 2 s tolerance then absorbs. Two experiments are missing: a wall-clock evaluation in which frames arrive in real time and late answers are penalized, and a variant in which Response may attend to the current-step Inner output, which would measure the cost of the independence assumption.
  2. Lack of matched comparisons with external methods. SyncPro uses Qwen3.5-4B at 1 fps, while most baseline entries are taken from the Em-Garde compilation and use older 7B to 8B backbones, other frame rates and other training data; no baseline is retrained on SyncPro-189K, so the 3.2-point PO margin and the 3.20-point OVO-FAR margin mix backbone, data and formulation. The internal Interleaved baseline is the only matched alternative, but its gap between containment accuracy (59.28%) and exact accuracy (8.98%) on responded samples suggests an output-format failure, which deserves an explanation before it is used as evidence against interleaving. The gains are also uneven: SyncPro trails on ProactiveVideoQA EGO (33.74 versus 52.30 for Em-Garde) and WEB (48.28 versus 53.30 for MMDuet2), and on the StreamingBench real-time subset (70.84 versus 79.50 for VST and 75.00 for ThinkStream).
  3. Lack of analysis of the linear-attention state merge. With γ = 0.5, St = (S^I + S^R)/2, so in the frequent case where one branch emits only a silent token, the branch that wrote content has its state change halved at every micro-turn. Over hundreds of frames in the Gated DeltaNet layers, which make up most of the decoder, this could attenuate the retained evidence, and it is not obvious that LoRA routes restricted to MLP projections can compensate for an altered recurrent state. The only alternative studied (learned per-branch weights, Table 7) does not address this case. A comparison with sequential composition, additive merging of the two deltas, and a merge that depends on which branch is active, along with accuracy as a function of stream length, is needed to justify the rule.
  4. Lack of statistical support for the RL stage and for the size of the reported gains, and lack of component ablations for SFT. RL uses 800 trajectories, 50 updates, four rollouts per query and zero KL, is run once, and the last checkpoint is reported. All StreamingBench-PO percentages are multiples of 0.4, which implies about 250 items, so the 2.4-point RL gain corresponds to about six items and the margin over Em-Garde to about eight. RL also lowers CRR F1 (28.06 to 26.91), and since the GRPO advantage is applied to all tokens of both branches while the reward depends only on Response timing, Inner content may drift without being measured after RL. The ATR loss and the action-token reweighting (w = 2.5 and 3.0, 0.2 for later silent tokens) are never ablated. A supplementary experiment with several seeds for SFT and RL, bootstrap intervals on PO and OVO-FAR, an evaluation of Inner quality after RL, and ablations of ATR and reweighting is needed.
  5. Lack of transparency about the quality of synthetic supervision and its overlap with evaluation data. Queries, trajectories, revisions and the reconstruction judge are all produced by Gemini 3.1 Pro, Qwen-VL-Max and GPT-5.4, and no human audit or judge agreement rate is given, so stylistic preferences of these models (trigger timing, response length) may be learned and then rewarded. Video-level disjointness is documented only for UCF-Crime against ProactiveVideoQA. The three OVO-Bench Forward Active Responding tasks draw their videos from Perception Test and THUMOS (REC), COIN (SSR) and MovieNet (CRR) (Niu et al., 2025), and Perception Test, COIN and MovieNet are all training sources of SyncPro-189K. OVO-Bench samples come from the validation and test splits, so this is probably not leakage, but SyncPro is trained in-distribution for these tasks while most baselines are not, which should be taken into account when reading the OVO-FAR margin. The conditional content metrics in Table 3 are computed only on samples with a response, which favors models that respond less often and should be reported together with response rates. Finally, the long-stream profile shows that only 74.9% of frames meet the 1 s budget over 480 s, with retained KV growing from 2,126 to 11,220 tokens, which limits the claims for longer streams.
Questions for authors
  1. Given that concurrent and Inner-first decoding produce identical outputs, what is the evidence that concurrency improves the system beyond latency? Can you report a real-time evaluation in which latency affects the score?
  2. How does performance change if Response may attend to the Inner output of the same micro-turn? How many errors in your evaluation are caused by the one-step gap between an Inner update and the corresponding Response?
  3. What is the effect of the averaging rule when only one branch writes content, and have you compared it with sequential or additive composition on long streams?
  4. Can you provide seeds or confidence intervals for SFT and RL, ablations of ATR and of the action-token weights, and an assessment of Inner content quality after RL?
  5. Have you checked video-level overlap between SyncPro-189K and the test videos of StreamingBench and OVO-Bench, and how much of the OVO-FAR gain remains if Perception Test, COIN and MovieNet are removed from training?
  6. What fraction of the synthesized trajectories was inspected by humans, and how often did the human judgment agree with the LLM judge?
  7. Why does the Interleaved baseline have such a large gap between containment and exact content accuracy?
4 comments
AuthorsOpenReview ↗

Response to Reviewer u2H6 (1)

Response to Q1 and W1

Thank you for this question. Concurrent decoding keeps the outputs of serial decoding unchanged and adds earlier answer delivery, which we quantify with a new real-time evaluation that counts queueing and generation time.

  1. Dual-branch modeling is the source of the accuracy gain, and concurrent execution adds timeliness on top. The dual-branch design reduces the mismatch between the learning targets of reasoning and responding. Under the same backbone and data, SyncPro-SFT reaches 30.8% containment-matched Joint@2s, versus 17.2% for Interleaved. Concurrent decoding then delivers these same outputs earlier.

  2. Concurrent decoding delivers the same outputs earlier. Concurrent and Inner-first decoding agree on all 1,870 held-out examples, and the offline profile shows the first response token arriving 199.52 ms earlier (2.22×) (Table 6).

  3. Under real-time scoring, concurrent decoding gives the highest Joint@2s and the earliest delivery. We evaluate StreamingBench-PO with the same checkpoint, feeding video at a true 1 FPS and counting queueing and generation time in the delivery time of each complete answer.

Execution orderReal-time Joint@2s
Concurrent24.0%
Inner-first21.2%
Response-first24.0%

On the 83 samples answered correctly by both schedules, concurrent decoding delivers the answer 254 ms earlier than Inner-first on average.

Response to W2

Thank you for the question. SyncPro's gains hold under a same-backbone, same-data control and against baselines of equal or larger scale. The exact-match gap for Interleaved reflects instruction following, and a case analysis of WEB and EGO points to clear directions for further gains.

Part 1: The gains come from the method rather than the backbone or data scale.

  1. Under the same backbone, data and input rate, SyncPro-SFT outperforms both alternative formulations on every benchmark. With Qwen3.5-4B, SyncPro-189K data and 1 fps, PO Acc@2s is 38.8 vs. 34.0 (Interleaved) / 22.8 (Response-only), PO Joint@2s is 30.8 vs. 17.2 / 10.4, and OVO-FAR Avg. F1 is 32.37 vs. 10.87 / 14.66 (Tables 1–3). None of the base models has native streaming understanding, so the proactive ability comes from the formulation and data.

  2. SyncPro leads external baselines of equal or larger scale, and baselines with their own backbone and training scheme follow the common protocol of this field. SyncPro uses 4B parameters, versus 7B–8B for VST, Em-Garde and VideoLLM-online (ThinkStream: 3B), and we evaluate VST and ThinkStream at 2 fps, twice our input rate. Each of the compared works selects its own backbone and training data (e.g., VideoLLM-online uses Llama-3-8B, MMDuet uses LLaVA-OneVision-7B, Em-Garde uses Qwen2.5-VL-7B), so we compare final performance on shared benchmarks.

Part 2: The exact-match gap for Interleaved reflects instruction following, and SyncPro's advantage also holds on metrics that do not use exact matching.

  1. SyncPro-SFT keeps both content metrics high, whereas Interleaved's exact score drops sharply. Containment checks that the target appears in the response, while exact matching also requires following the query's output instruction (e.g., output "41"). SyncPro-SFT scores 79.37% (containment) and 45.24% (exact), while Interleaved scores 59.28% and 8.98%. Interleaved carries both roles in one activation sequence, so its instruction following degrades.

  2. The advantage extends to metrics without exact matching. SyncPro-SFT outperforms Interleaved on PO timing (Acc@2s 38.8 vs. 34.0), OVO-FAR timing-based F1 (32.37 vs. 10.87), and judge-scored PVQA PAUC (WEB 47.83 vs. 29.89, EGO 33.89 vs. 25.00).

Part 3: SyncPro transfers well to WEB (second among compared methods, 48.28), and a case analysis points to clear directions for further gains on EGO.

  1. The dual-stream design brings consistent gains on WEB and EGO without dedicated tuning. Our supervision targets concise answers after explicit triggers, while WEB stresses multi-point coverage and EGO long-horizon step tracking. SyncPro-SFT still improves over Interleaved on WEB (29.89 →\rightarrow 47.83) and EGO (25.00 →\rightarrow 33.89).

  2. An independent case analysis of 50 low-scoring response windows per subset identifies one main factor on each. On WEB, it is key-point coverage (incomplete in 38% of windows). On EGO, it is trigger timing (missed or late in 76% of windows). Both relate to the focus of our annotations, which points to multi-point answer coverage for WEB and continuous response updating for EGO as clear directions for improvement

Revision: We add the WEB/EGO error analysis to Section 4.2, with the detailed audit in Appendix G (Table 11).

AuthorsOpenReview ↗

Response to Reviewer u2H6 (2)

Response to Q2

Thank you for the question. The one-step gap between Inner and Response is not a source of errors in SyncPro: Response reads the same prior state and current frame as Inner, and our check of the full PO and OVO-Bench trajectories found no such error.

  1. Both branches read the same state, so Response does not wait for Inner. Accumulated state and the current frame are visible to both branches in the same micro-turn, and only the newly generated Inner content is shared at the end of the turn. For example, if two events were recorded earlier and the third appears in the current frame, Response can combine the existing count with the current observation and answer directly, without waiting for Inner to produce a "three" conclusion in another step.

  2. The design maintains intermediate state and learns state updating and answering as separate targets. This lets Response answer as soon as the evidence is sufficient. Under the same backbone and data, SyncPro-SFT reaches 30.8% Joint@2s, versus 17.2% for Interleaved and 10.4% for Response-only (Table 3).

  3. To address your concern, we added a trajectory analysis of the original SyncPro outputs on all 250 PO questions and all 172 OVO-Bench questions, and it found no one-step-gap errors. On PO, none of the 165 Joint errors has Inner giving the correct conclusion in the current turn and Response answering correctly only in the next turn, past the 2 s window. The 6 samples whose first correct answer came exactly at GT+3 s all had a silent Inner in the preceding turn. On OVO-Bench, none of the 147 unmatched events shows an active Inner followed by a next-turn reply just outside the window.

This matches the independent-activation design: Response decides from existing state and the current observation, so cross-turn sharing does not imply a fixed one-second reply delay.

Response to Q3 and W3

Inner and Response have complementary roles and both matter, so SyncPro merges their states with symmetric equal weights (γ = 0.5). Performance falls as the merge approaches a single dominant branch, and the rule remains effective on streams up to 540 s.

  1. The alternative merge rules are different views of one delta-form update. Both branches start from the same shared state, so the merge can be written as S_t = S_t^0 + 0.5(ΔS_t^I + ΔS_t^R). The shared initial state is kept once, and a silent branch still runs its own state update. Summing the two increments is this same form with a different scale. Training and inference use the same rule, so the model learns to accumulate and use evidence under this state transition. Our core contribution is reducing the mismatch between internal reasoning and external response targets, and realizing synchronous reasoning and responding with two branches. State merging is the mechanism that supports this design, not a separate algorithmic contribution.

  2. Equal weights give the best result on every metric, and a near single-branch merge (0.1 + 0.9) degrades clearly. Both branches contribute, so letting one branch dominate lowers performance.

α + βSB Acc@2sOVO Avg. F1@2sCRRRECSSR
0.5 + 0.5 (SyncPro)41.234.1926.9150.0025.67
0.4 + 0.638.431.923.848.922.9
0.6 + 0.437.631.822.249.124.2
0.1 + 0.933.630.422.244.524.5
0.9 + 0.134.428.323.041.919.9
  1. To address your concern about long streams, we added an evaluation bucketed by continuous input duration, covering streams up to 540 s. SyncPro-RL retains the training gains on long streams: on the >180 s bucket it still improves over SFT on both REC and SSR.
Input duration≤60 s(60, 180] s>180 s
SyncPro-RL, REC F1@2s53.39%47.24%36.65%
RL gain over SFT, REC––+3.17
RL gain over SFT, SSR––+6.94

The current merge rule therefore supports sustained task execution on long streams.

AuthorsOpenReview ↗

Response to Reviewer u2H6 (3)

Response to Q4 and W4

Thank you for these suggestions. SyncPro's results are stable under paired bootstrap, ATR and the action-token weights are supported by ablations, and RL improves the quality of Inner content.

  1. Paired bootstrap shows low variance, and RL improves over SFT on both benchmarks. We ran a 10,000-resample paired bootstrap over source videos.
MetricRL (mean, sd, 95% CI)SFT (mean, sd, 95% CI)RL − SFT (mean, sd, 95% CI)
OVO Avg. F1@2s34.19 (33.28, 1.87, [29.6, 37.0])32.37 (31.79, 1.80, [28.3, 35.4])+1.49, 1.42, [−0.96, 4.44]
StreamingBench Acc@2s41.2 (41.18, 4.09, [33.2, 49.2])38.8 (38.19, 3.78, [30.0, 44.4])+2.99, 1.51, [1.20, 7.20]
  1. ATR resolves a trade-off between false triggers and missed responses, and it is effective in the ablation. With silent-weight 0.1, the false-trigger rate on repeated-silent steps is 17.28%. Raising the weight to 0.2 lowers it to 9.76%, but recall also drops to 12%, since suppressing repeated speaking also suppresses the positions where the model should speak. ATR addresses this trade-off.
Setting (action-token weights)StreamingBench-PO Acc@2sOVO F1 (CRR / SSR / REC)
SyncPro-SFT, w/ ATR (2.5/0.2/3)38.8028.06 / 21.36 / 47.69
w/o ATR (2.5/0.2/3)36.4025.66 / 10.22 / 34.08
w/ ATR (2/0.2/2)35.2018.37 / 10.17 / 35.86
  1. To address your concern about Inner quality, we manually reviewed the Inner content on a fixed set of 50 OVO questions, and RL improves it over SFT.
MetricSFTRL
Directly query-relevant90.24%91.41%
Verifiable factual correctness74.58%78.60%
Evidence coverage23.72%26.51%

Response to Q5

SyncPro-189K is built entirely from the official training splits of widely used public datasets, has no video overlap with the evaluation benchmarks, and the benefit of our method is shown under identical training data.

  1. All training data come from official train splits, and the video IDs do not overlap with the test videos. As described in Appendix A, every example in SyncPro-189K is constructed from the official training splits of its source datasets, while StreamingBench, OVO-Bench and ProactiveVideoQA are reserved for evaluation. We checked the video IDs and found no overlap with the test videos.

  2. COIN, Perception Test and MovieNet are standard public training sources in this field. Em-Garde [1] and VideoChat-Online [2] use COIN, and StreamForest [3] uses COIN and Perception Test. Using them follows established practice, so they are not a special source of advantage.

  3. The benefit of the dual-branch design is measured under the same backbone and the same training data. On OVO-FAR, SyncPro-SFT reaches an Avg. F1 of 32.37, versus 10.87 for Interleaved (Table 2, Section 4.3). The gain therefore comes from the method itself, and does not rely on differences in training data between external methods.

[1] Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding

[2] Online Video Understanding: OVBench and VideoChat-Online

[3] StreamForest: Efficient Online Video Understanding with Persistent Event Memory

Response to Q6 and W5

Quality control of SyncPro-189K combines automatic revision, reconstruction checks and human review, with 5,000 manually checked trajectories (about 3.6% of the fully synthesized part) and an 89.7% human–LLM agreement rate.

  1. Every trajectory passes automatic quality checks before any human review. The LLM judge examines each response for factual correctness and trigger timing, removes content-free repetitions, finds missed response opportunities, and regenerates the full trajectory when a correction is needed (Appendix E). We keep a trajectory only if the inner-state context can reconstruct every response after earlier visual evidence is removed.

  2. After three rounds of refinement, we manually checked 5,000 synthesized trajectories, about 3.6% of the fully synthesized part. Annotators used the same two criteria as the LLM judge: timing accuracy and content correctness. Human judgments agree with the LLM quality judgments in about 89.7% of cases.

  3. We separately checked 2,000 candidate trajectories manually in the RL data selection stage, which were used to select the final RL training data.

AuthorsOpenReview ↗

Response to Reviewer u2H6 (4)

Response to Q7 and W2

The gap reflects an instruction-following problem that arises when reasoning and responding share one generation stream, and the dual-branch design addresses it: exact-match Content Acc.@Response rises from 8.98% (Interleaved) to 45.24% (SyncPro-SFT).

  1. Exact match measures instruction following, so it drops most when the two roles are not decoupled. Containment only checks that the correct answer appears in the response, while exact match also requires the response to follow the output format specified by the user. The model can identify the correct answer yet describe the observation instead of giving the requested response. For example, in an early single-stream experiment, the user asked: "when the number 20 appears in the center of the frame, output only '20'". The model replied "The number 20 appears in the middle of the sun." This passes containment matching but fails exact matching.

  2. This is a motivation for our decoupled design. Internal reasoning needs to describe and accumulate state, while external responses need to follow user instructions, and the two have different generation targets. Sharing one generation stream lets the two output modes interfere with each other. SyncPro learns the two behaviors separately with independent branches and specialized parameters.

  3. Role decoupling improves instruction following. SyncPro-SFT raises exact-match Content Acc.@Response from 8.98% to 45.24% over Interleaved, while containment-based Content Acc.@Response is 59.28% vs. 79.37% (Table 3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.