acceptodds
Under review as a conference paper at ICLR 2027

Stream-of-Thought: Always-On Reasoning for Reactive and Proactive Assistance

Abstract

Live multimodal assistants, the systems that continuously stream a user's egocentric viewpoint, make persistent perception of the user's environment practical. Yet their interaction paradigm remains *reactive*: reasoning begins only once a user speaks, and the incoming stream goes unexamined until then. We propose **Stream-of-Thought** (SoT), which enables an online vision-language model to simultaneously and adaptively emit thought tokens as it perceives a streaming input, and to decide on its own when to turn those thoughts into speech. Thoughts cached across a session serve both modes of assistance: (1) *reactive*, where complex questions are answered faster and long-horizon tasks are tracked because much of the required reasoning is already done, and (2) *proactive*, where the same cached thoughts ground the help that the user never asked for. Since unconstrained “daydreaming” only lengthens context, we train thought to be *utility-directed*, optimal under both objectives: a two-stage SFT over a curated SoT-SFT dataset, followed by **Think–Speak Group Policy Optimization** (TSGPO), a novel RL algorithm that credits thinking and speaking separately. We further introduce **UnpromptedBench**, a manual collection of unprompted, open-ended assistance in egocentric streams that tests what is said and when. Through extensive evaluations, we show that a single SoT model combines strong reactive *and* proactive capabilities under a tight latency budget, outperforming prior streaming systems at comparable scale and rivaling far larger single-regime specialists.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming video understanding, Multimodal LLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XjvmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Stream-of-Thought, a streaming vision-language model that accumulates reasoning while observing video and separately decides when to speak. Staged supervised fine-tuning and Think–Speak Group Policy Optimization train these behaviors. The authors evaluate reactive and proactive assistance and introduce UnpromptedBench for assessing unsolicited interventions.

Critical strengths
  1. Separating thinking from speaking allows the assistant to retain information for later questions without turning every state update into an interruption. The evaluation across both types of task makes this a useful contribution to streaming assistance.

  2. The ablations show that the hidden-quiz reward improves recall of earlier video content and decision-local credit reduces repeated responses.

Critical weaknesses
  1. The caption of Table 10 says the unregularized run achieves the highest average, although it scores 57.03 versus 64.37 with entropy regularization. The SFT OVO score is 59.60 in Tables 5 and 10 but 59.93 in Table 7. Reasoning rates also differ. Are these editorial errors, or were different checkpoints or measurement definitions used?

  2. UnpromptedBench results use the development split, with SoT covering five of 237 reference opportunities. Precision also credits help outside the catalog, but the stratified expert audit rejected eight of twenty sampled responses that the automatic response-quality evaluator had passed. I would like to see whether the ranking holds on the held-out split or in a blinded human comparison.

Questions for authors
  1. Could authors reconcile the scores, reasoning rates, and caption in Tables 5, 7, and 10, specifying the checkpoints and measurement definitions? Please also check the Qwen3-VL-4B aggregate in Table 1: its reported component scores average to approximately 62.62, whereas the final column gives 62.12.

  2. Are held-out results or a blinded human comparison available for UnpromptedBench? Paired video-level uncertainty estimates for the main comparison would also be helpful.

4 comments
AuthorsOpenReview ↗

Response to reviewer Xjvm

We thank the reviewer for the valuable suggestions and comments. To answer the questions/comments:

  • Consistency across tables. We checked the reported results against our training logs. The entries in Table 5 are correct. We corrected the Qwen3-VL-4B aggregate in Table 1 and the entries in Appendix Tables 7 and 10 that had been inadvertently mixed across checkpoint versions. We also corrected the caption of Table 10. We appreciate the reviewer bringing these reporting errors to our attention.

    The reasoning rates differ because they are measured at different checkpoints in RL training, not by different definitions. Table 5 studies how each SFT schedule affects RL, so its RL column reports the rate at the same middle checkpoint for every schedule. Table 10 reports the rate before RL and at the checkpoint evaluated in Tables 1 and 2. We now clarify this in the manuscript. For the reviewer's reference, the final reasoning rates of Table 5's runs are:

    ScheduleFinal think %
    Qwen3-VL-4B w. SoT prompt82.98
    Standard SFT55.44
    SFTKey0.00
    Ours57.82

    These final rates lead to the same conclusion as indicated in Section 5.2.

  • Held-out results and uncertainty.

    • Held-out results. We scored all works on the held-out split with 157 candidates, of which 81 opportunities are accepted by a strict majority of three complete ratings. We used the same settings as on the development split, and the evaluator is unchanged from Table 2.

      ModelPrecision (%)Coverage (%)F0.5F_{0.5} (%)
      VST9.30.000.00
      Em-Garde8.50.000.00
      Qwen3-VL-4B0.60.000.00
      MMDuet20.00.000.00
      StreamOPDN/A0.000.00
      ThinkStreamno replies0.000.00
      SoT45.62.4710.15

      The ranking holds as on the development split. SoT is the only one that covers any held-out opportunity. It ranks first in coverage and F0.5F_{0.5}, and its precision (45.6%) exceeds that of every prior work except StreamOPD, whose 3 replies are too few to compare. Appendix I.4 gives further details.

    • Paired video-level uncertainty. For the main comparison, we resample whole videos 10,000 times, keeping the number of videos from each source fixed, and score SoT and every prior work on the same resampled videos. The paired difference, SoT minus each prior work in Table 2, ranges from +7.1 to +8.9 in F0.5F_{0.5}, with every 95% interval above zero. Against every prior work, SoT has the higher F0.5F_{0.5} in 99.6% of the resamples. The smallest difference, against Em-Garde, is +7.12 [+2.12, +12.31]. In precision, the paired difference ranges from +17 to +46. Appendix I.1 (Tables 15 and 16) gives the detailed intervals and paired differences for every model and metric, and Appendix I.4 (Table 19) those for the held-out split.

    • The expert audit and credit outside the catalog. We added three checks to further verify the ranking.

      • Who the rejected passes came from. In the frozen 100-item audit, all 8 rejected automatic passes came from prior works, while none of SoT's sampled passes was rejected. The samples per model are small, so this is indicative only.
      • No credit outside the catalog. Counting only replies that match a human-approved opportunity, SoT scores 1.10% [0.30, 2.03] on the development split, against at most 0.10% for any prior work, and 1.47% (2 of 136) on the held-out split, against 0 for every prior work.
      • Help the annotators rejected. SoT's valid replies match a human-rejected candidate in 2.1% of development cases and in none on the held-out split. Prior works have 0–1 such matches.

    We have added the held-out results, the paired intervals (now in Appendix I) and these checks to the revision.

AuthorsOpenReview ↗

Follow-up on Author Response

Dear Reviewer Xjvm,

Thank you again for your time and valuable comments. As the discussion period nears its end, we would like to follow up on our author response and revised manuscript. We would appreciate any further feedback and would be happy to clarify any remaining concerns.

Best regards,

The authors

Thank authors for the held-out results and paired uncertainty estimates. These strengthen the evidence that the ranking holds. My remaining concern is that held-out coverage corresponds to only 2 of 81 opportunities, while the bootstrap intervals quantify video-sampling uncertainty conditional on the unchanged automatic evaluator. A brief clarification of whether these two matches were independently verified by a human would be helpful.

AuthorsOpenReview ↗

Additional response to reviewer Xjvm

Thank you for the follow-up. We are glad our response was helpful in addressing your earlier concerns.

Following your comment, we asked the expert reviewer to check whether each of the two held-out replies delivers the same help as its matched opportunity. The reviewer labeled these pairs mixed with eight unmatched pairs from the same split, without seeing the evaluator's decisions or rationales, and confirmed both semantic matches. We have added this check to Appendix I.4 of the revised paper.

We hope this addresses your remaining concern, and we would be glad to clarify anything further.

Reviewer_GV11OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Stream-of-Thought (SoT), an streaming VLM that incrementally generates reasoning traces while observing video and decides whether to respond to users. Unlike conventional reactive systems, SoT aims to support both faster responses to later queries and proactive assistance. The method combines a two-stage SFT with Think–Speak Group Policy Optimization, which introduces the reward for useful historical reasoning, decision-local advantages for response timing, and entropy regularization for adaptive thinking. The authors additionally introduce UnpromptedBench, a human-annotated evaluation of unsolicited assistance. Experiments report strong performance across reactive and proactive streaming benchmarks.

Critical strengths
  1. The unified formulation is compelling: a single 4B model jointly learns when to reason and when to speak, achieving strong results on reactive and proactive benchmarks. This provides a practical direction for always-on assistants.
  2. The hidden-quiz reward and decision-local advantage directly address two streaming-specific challenges: preserving query-relevant information before a question arrives and discouraging redundant interruptions. Ablations, particularly the substantial backward-tracking improvement, provide meaningful evidence that these design choices matter.
Critical weaknesses
  1. Despite outperforming baselines, SoT achieves only 2.11% coverage on UnpromptedBench, corresponding to approximately five of 237 annotated assistance opportunities. This extremely low absolute recall raises doubts about whether the system delivers useful proactive assistance in practice.
  2. The novelty is insufficiently distinguished from closely related works. ThinkStream [1] and VST [2] already explore reasoning during video streaming, while MMDuet2 [3] and Vinci2 [4] address proactive assistance. Although TSGPO introduces specialized optimization mechanisms, the paper lacks sufficiently controlled comparisons to establish what adaptive thinking and the proposed training strategy uniquely contribute beyond existing approaches.

[1] Thinking in Streaming Video

[2] Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously

[3] MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning

[4] Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos

Questions for authors
  1. Can the authors explain the very low UnpromptedBench coverage?
  2. What are the key methodological differences between SoT and ThinkStream, VST, and MMDuet2, beyond combining their existing capabilities?
  3. Can controlled comparisons isolate the contributions of adaptive thinking and TSGPO relative to existing streaming reasoning and proactive assistance methods?
3 comments
AuthorsOpenReview ↗

Response to reviewer GV11 (1/2)

We thank the reviewer for the thoughtful feedback and suggestions. We carefully address your concerns and comments as follows:

  • Low coverage in UnpromptedBench. First, we would like to emphasize that coverage is low for every work, and SoT has the highest. It covers 2.11% of the opportunities, against at most 0.42% for any prior work, and it leads every prior work in F0.5F_{0.5}. Three properties of the benchmark contribute to the low absolute number.

    • Acceptable help vs. required help. Annotators approved 53 opportunities per video-hour, about one every 68 seconds, each a moment at which some help would be acceptable. No assistant is expected to deliver most of them, and F0.5F_{0.5} deliberately weights precision over coverage.

    • Strict rubrics. This benchmark is deliberately strict. An opportunity is covered only by a reply that is valid and non-redundant (grounded, correct, timely, useful, safe, and not a repeat of earlier speech), falls within a small window of the adjudicated moment, and realizes the same useful intent as the human reference. Mentioning the same objects or actions is not enough. Matching is one-to-one, and an "unsure" verdict counts as a miss. Loosening the match (a wider window, counting "unsure" verdicts, or matching only the category of help) raises SoT's covered opportunities by 80% to 360%, and SoT's lead over every prior work holds under each variant.

    • Diverse intents. The catalog spans five categories of help. SoT's covered opportunities are procedural, safety and recall ones; it covers none of the 74 social and recommendation opportunities, 31% of the catalog. Our training data are general-purpose video corpora and, for proactive RL, egocentric and live-commentary videos, none of which targets social or recommendation help. Curating such data could close part of this gap.

    The table below follows each system's replies through the evaluator's checks. SoT produces more replies that pass the automatic validity and non-redundancy checks than any prior work: 208. These checks ask whether a reply is acceptable on its own. Coverage further requires it to be the specific help the annotators approved for that moment.

    ModelRepliesValidValid and non-redundantCovered (of 237)
    SoT4542912085
    Em-Garde957168901
    Qwen3-VL-4B1,303140151
    VST23198180
    StreamOPD14740
    MMDuet27,43016610

    Further, precision matters most for unsolicited help, since every reply interrupts the user. Our evaluator judges 45.81% of SoT's unprompted replies valid and non-redundant, the highest of all works compared, against 9.40% for Em-Garde, the strongest prior work by F0.5F_{0.5}. We have added this analysis to Appendix H.2 (Table 14) and Appendix I, referenced from Section 5.1.

    We agree that coverage stays low even after accounting for the three properties above, that some of SoT's replies still fail the evaluator's checks, and that automatic acceptance does not show that users would find the accepted ones useful. Our claim is relative: on a deliberately strict benchmark that remains far from solved for every system, SoT makes the most progress and covers the most opportunities.

AuthorsOpenReview ↗

Response to reviewer GV11 (2/2)

  • Comparison with other works.

    • Formulation. VST and ThinkStream generate incremental reasoning at video clip or streaming-step boundaries. MMDuet2 trains question-conditioned proactive dialogue without a separate persistent thought channel. SoT additionally learns whether to generate a non-empty thought at each step, independently of whether to speak. Vinci2's formulation is orthogonal to the works above. It develops a memory agent system rather than training a shared reasoning and interaction policy within one VLM.

    • Optimization. Compared to prior works directly rewarding the trajectory-level timing, format, or correctness of the final answer (e.g., ThinkStream, VST), or optimizing for the PAUC metric (e.g., MMDuet2), SoT introduces mechanisms targeting historical thought utility and the separate thinking and speaking decisions:

      • Staged SFT: learn speaking structure and answers with reasoning content masked from the loss, then jointly supervise the full reasoning and speaking.
      • Hidden-quiz reward: assess cached thought utility by answering a later query with visual contents removed.
      • Decision-local advantage: add marginal-reference coverage credit at emitted response gates, with trajectory centering, a moving normalization scale, and response-probability-based sign scaling.
      • Entropy regularization: encourage exploration of whether to reason.
    • Results and controlled comparison. Tables 1 and 2 of the paper compare SoT with VST, ThinkStream, and MMDuet2 in both reactive and proactive regimes. Each is designed for one regime: VST and ThinkStream for reactive streaming reasoning, and MMDuet2 for proactive interaction. SoT outperforms each of them in its own regime, and by a wide margin in the other. Across all works compared, a single SoT model achieves the highest overall reactive score in Table 1 and the strongest overall proactive results in Table 2. We would like to also emphasize that even comparing to Vinci2 with an agentic system, our proposed hidden-quiz reward is capable of bringing the backward subtask of OVO-Bench to 68.57 compared to 49.60 for Vinci2.

      We additionally compare different methods using the same Qwen3-VL-4B as base model and training corpus per the reviewer's request. We see that SoT achieves the highest score on both reactive and proactive benchmarks. The paper’s component ablations separately examine the contributions of different components of SoT.

      MethodOVO-Bench ↑StreamingBench ↑ProactiveVideoQA WEB (PAUC ↑ / repetition ↓)ProactiveVideoQA EGO (PAUC ↑ / repetition ↓)
      ThinkStream59.3477.7132.2/11.428.1/26.1
      VST57.4678.9044.3/40.238.5/59.3
      MMDuet255.6777.0952.2/4.841.9/10.2
      SoT64.3781.2754.3/4.146.5/8.7
AuthorsOpenReview ↗

Follow-up on Author Response

Dear Reviewer GV11,

Thank you again for your time and valuable comments. As the discussion period nears its end, we would like to follow up on our author response and revised manuscript. We would appreciate any further feedback and would be happy to clarify any remaining concerns.

Best regards,

The authors

Reviewer_FXPZOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Stream-of-Thought (SoT), a unified streaming video framework that enables adaptive continuous reasoning and both reactive and proactive assistance through staged SFT and a novel Think–Speak Group Policy Optimization (TSGPO) algorithm.

Critical strengths

The paper presents a well-motivated training framework with extensive experiments demonstrating strong performance across both reactive and proactive streaming video benchmarks.

Critical weaknesses

Although the paper demonstrates reduced response latency, it does not analyze the per-chunk inference latency introduced by continuous thought generation. Generating reasoning tokens may cause processing delays that accumulate across incoming video chunks, potentially preventing the model from maintaining real-time interaction. The authors should evaluate whether SoT can consistently process streaming inputs within their arrival intervals to demonstrate its practicality in real-world applications.

Questions for authors

Work such as JoyAI-VL-Interaction[1] delegates complex reasoning tasks to a background model while maintaining real-time interaction. In contrast, SoT integrates continuous reasoning and interaction within a single model. How do the authors view the trade-offs between these two architectures, particularly regarding latency, computational efficiency, and reasoning capability? Under what circumstances would each approach be preferable?

[1] Yao D, Zhou J, Yang C, et al. JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence[J]. arXiv preprint arXiv:2606.14777, 2026.

4 comments
AuthorsOpenReview ↗

Response to reviewer FXPZ

We thank the reviewer for the thoughtful review and suggestions to improve the quality of the paper. To answer the questions/comments:

  • Latency for continuous streaming. We measure how long SoT takes to process each incoming chunk, i.e., to encode its frames, decide whether to reason, write the reasoning step if any, and decide whether to respond, and compare it with the interval at which chunks arrive. The measurements cover streams from 30 seconds to 53 minutes, all at 1s chunks of two frames, the shortest and most demanding chunk interval in our evaluation protocol. All runs use a single H100 with batch size 1.

    Stream lengthTime per chunk (median / max)Late chunks
    0.5–1 min0.23 / 0.59 s0 / 720
    2–15 min0.43 / 0.69 s0 / 13,921
    16–26 min0.51 / 0.74 s0 / 9,717
    31–53 min0.54 / 0.79 s0 / 19,660

    We see that every chunk is processed within its arrival interval. Since no chunk takes longer than its arrival interval, each chunk is finished before the next one arrives. No chunk ever waits, so delays cannot build up. We add this analysis to Section 5.1.2, with more details included in Appendix L.

  • Comparison to JoyAI-VL-Interaction. We have included JoyAI-VL-Interaction in Section 6. Its 8B foreground model chooses whether to remain silent, respond, or delegate a demanding task to a larger background model or agent. SoT instead learns a shared reasoning stream and both thinking and speaking decisions within one 4B policy. Its thoughts accumulate as the video plays even before any question is asked.

    • Latency. With delegation, the final answer waits for the background model, which starts reasoning only after the hand-off. Reasoning after a question is slow even on short streams, as shown in Section 5.1.2. JoyAI-VL-Interaction also compresses its memory with a separate model, which adds latency. SoT instead spends a small, steady amount of time on every chunk (see Appendix L), so most of its reasoning is already done when a question or an event arrives.

    • Computational efficiency. We measure compute FLOPS and memory footprints of both systems. The evaluated JoyAI-VL-Interaction release uses an 8B foreground model, a 4B memory model that summarizes video every 100 seconds, and an optional reasoning backend. SoT uses one 4B model for both reasoning and interaction. For 0.5–26-minute streams, the reported computational rates are 5.6–11.2 TFLOP/s for SoT and 6.0–13.1 TFLOP/s for JoyAI-VL-Interaction, even excluding JoyAI-VL-Interaction's reasoning backend. SoT also uses less GPU memory at every measured duration (at most 18 GiB vs. 34 GiB).

    • Reasoning capability. In JoyAI-VL-Interaction, the ceiling of delegation is that of the larger background model or an agent with tools, so delegation is potentially stronger on hard questions that do not depend on the stream. A background model consulted afterwards sees the past only through what it is handed, which in JoyAI-VL-Interaction is the question and the most recent frames. SoT's reasoning is bounded by a 4B model, but it concerns the stream itself, i.e., it is written while the evidence is still in view, and it precedes each decision to speak. On OVO-Bench, SoT scores 64.37, even higher than JoyAI-VL-Interaction's 59.20.

    • When each is preferable. SoT is attractive for constrained deployments and for streams where incremental reasoning pays off frequently. Delegation is attractive for occasional demanding tasks alongside continuous interaction. A future system could combine SoT’s learned policy with optional delegation.

AuthorsOpenReview ↗

Follow-up on Author Response

Dear Reviewer FXPZ,

Thank you again for your time and valuable comments. As the discussion period nears its end, we would like to follow up on our author response and revised manuscript. We would appreciate any further feedback and would be happy to clarify any remaining concerns.

Best regards,

The authors

Thank you for the authors' detailed response. I have raised my score accordingly.

Thank you very much for taking the time to go through our rebuttal and for raising your score. We sincerely appreciate your constructive feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.