acceptodds
Under review as a conference paper at ICLR 2027

StreamAct: Streaming or Just Answering? Diagnosing Streaming Intelligence Illusions in Video-Language Models

15 views

Abstract

Are video-language models truly capable of streaming video understanding, or do they merely answer questions from a given causal context? Many streaming-like evaluations provide a causal prefix or recent window and score the resulting answer at an evaluator-chosen moment. This can make models appear streaming-capable without testing whether they autonomously maintain relevant state, judge evidence sufficiency, or choose appropriate actions as the stream unfolds. We call this mismatch the Streaming Intelligence Illusion. To diagnose it, we introduce StreamAct, which evaluates streaming video understanding as a strict-causal online trajectory over state, readiness, action, and answer. StreamAct-Bench evaluates 5,436 replayable trajectories under a strict-causal protocol. Target-time answerability alone does not establish this behavior: Qwen3-Omni-30B-A3B achieves 45.02% target-time answer accuracy, yet only 6.68% of its first self-timed responses are timely and correct. To test whether this missing evidence-to-action conversion is learnable, StreamAct-SFT provides explicit supervision over state, readiness, and action, selectively improving these trajectory-level behaviors beyond answer-only and format-only training. These results establish the distinction between streaming and just answering: Streaming means maintaining state and choosing the right action as evidence unfolds.

Primary area:
datasets and benchmarks
Keywords:
Streaming Video Understanding, Online Decision-Making, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_Sb6sOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that streaming video LLM look good on OVO-Bench/StreamingBench only because the evaluator decides when to ask questions. When models have to decide themselves when to answer, then their performance drop. The authors propose StreamAct-Bench and formalize a per-chunk policy with state, readiness, action and an answer. The authors also propose StreamScore (mean of State, Ready, Action, VFR). They evaluate around 11 models under a strict-causal replay interface and fine-tune Qwen3-Omni on 60K trajectories. Oracle experiments suggest that timing is more critical than perception in terms of performance.

Critical strengths
  • This topic is very timely and also interesting for several real-world applications. Most of video based applications will face streaming setup and whether models are appropriate for streaming scenario or not is important to study.

  • Paired evaluator vs self-timed analysis on same items is useful as it isolates timing form perception. Per-chunk decision labels in addition to performance on final answer are also useful for analyzing models behavior.

Critical weaknesses
  • The contributions of this work are very weak. The first contribution has been observed in several prior works such as StreamReady etc., where response timing is explored. The fact that explicit supervision is not super helpful, does not add any value to the list of contributions. Also, why just one backbone was tested?

  • The evaluation setup for replay is not well generalized. Ideally, memory based streaming models such as MMDuet, HiearQ, VideoLLM-online, ReKV, etc. are ideal test bed for this setup. Whether models can answer late or not will depend a lot on how good the memory is.

  • The figures are so hard to read and understand. The presentation needs a lot of work in this submission. I was not able to read most of the important figures.

Questions for authors
  • The authors should evaluate the streaming models in their own native mode, memory/cache/etc., instead of using a specialized replay setup.

  • Can you share details on self-timed prompt? Was it a generalized prompt for all?

  • In the annotation, was there any analysis on overlap between llm vs human and was any inter-annotator agreement computed?

  • Can we get evaluations on some closed-set models too?

  • Why is Full SFT VFR 11.11 in Table 3 but 8.84 in Table 4? Any reason why these are not on same shared trajectories?

  • The writing of this manuscript is very convoluted, very hard to understand. This is a very interesting topic, but the way it has been written, its hard to understand what the authors want to convey.

Some missing references:

  • Chen et al., VideoLLM-online: Online Video Large Language Model for Streaming Video, CVPR 2024.

  • Wang et al., VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format (MMDuet), Findings of EMNLP 2025.

  • Wang et al., Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning, ICLR 2026.

  • Qian et al., Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction, CVPR 2025.

  • S. Azad, V. Vineet, Y. S. Rawat. HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understanding. CVPR 2025.

  • Wang et al., StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant, NeurIPS 2025.

  • Li et al., LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistant, CVPR 2025.

8 comments

We thank the reviewer for taking the time to read the paper and provide detailed feedback. We take seriously your concerns about the scope of the contribution, the generality of the replay setup, presentation quality, and reproducibility. We will address each in the revision and respond to the main weaknesses and questions below.


Weakness 1. Contribution beyond prior work, benefits of explicit supervision, and backbone coverage

We agree that StreamReady and related work have identified the problem of when to respond. StreamAct addresses a broader set of streaming behaviors. Our first contribution is to identify what accounts for the gap in current evaluations. Most video benchmarks measure answer correctness at a time selected by the evaluator, leaving the joint behavior of state maintenance, evidence readiness, action selection, and conditional emission across the full timeline unmeasured. Our second contribution is a trajectory-level evaluation that formalizes how state, readiness, action, and answers evolve chunk by chunk. We construct 5,436 replayable trajectories and 91,839 decision nodes and use strict causal replay to locate the largest gap in the transition from evidence to action. StreamReady focuses on when evidence supports an answer. StreamAct additionally audits state revision and the decisions to wait, remain silent, trigger, or answer. Our contribution is to identify and attribute this measurement gap and provide a complete, auditable trajectory evaluation that also supports training.

We would like to clarify the role of explicit supervision. As the main text and conclusion state, StreamAct-SFT is the supervised intervention in a framework that connects diagnosis, evaluation, and training. It tests whether the identified state, readiness, and action behaviors can be learned. On Qwen3-Omni, Full SFT changes Action by +17.35, Miss by −24.36, and StreamScore by +5.74, showing that the process signals are learnable and that missed responses decrease substantially. VFR improves by only +4.43, and Early increases by +13.97, so timing calibration remains unresolved. This trade-off between conservative waiting and proactive responding is a finding of the paper. Explicit process supervision improves action selection and reduces missed responses, while aligning readiness with response emission still requires work on the architecture, training objectives, or response controller. The diagnostic framework makes these remaining gaps measurable and auditable. We have also added experiments on Qwen3-VL-30B-A3B, InternVL3.5-30B-A3B, and InternVL3.5-20B-A4B. Full SFT improves Action, StreamScore, and VFR and reduces Miss on all three backbones. The direction of the change in Early depends on the backbone's initial action policy. The results show that the supervision signal transfers beyond the Qwen family. They also support reporting timing calibration by backbone and treating timing control as a remaining challenge for architecture and training design.

BackboneTrainingState↑Ready↑Action↑VFR↑Early↓Miss↓StreamScore↑ΔVFR vs Base
Qwen3-Omni-30B-A3BBase39.5959.2836.586.6820.6572.9435.53-
Qwen3-Omni-30B-A3BFull SFT39.4960.5553.9311.1134.6248.5841.27+4.43
Qwen3-VL-30B-A3BBase39.7159.8636.615.9215.0878.8435.53-
Qwen3-VL-30B-A3BFull SFT39.6460.7254.079.8526.9463.2141.07+3.93
InternVL3.5-30B-A3BBase36.8355.1928.734.7732.5062.7331.18-
InternVL3.5-30B-A3BFull SFT37.0255.5844.368.7028.9162.3936.42+3.93
InternVL3.5-20B-A4BBase32.2549.8328.464.5927.8367.5828.78-
InternVL3.5-20B-A4BFull SFT32.0653.7542.107.1528.5664.2933.77+2.56

Weakness 2. Replay generality and native memory in MMDuet, HierarQ, VideoLLM-online, and ReKV

Thank you for this important suggestion. The StreamAct-Bench main leaderboard already includes recent streaming models such as ThinkStream-3B and MOSS-VL-Realtime. In response to your suggestion about native memory mechanisms, we added MMDuet and VideoLLM-online, two earlier streaming models that represent different capabilities. Together, the four models cover streaming reasoning and online state, real-time streaming control, proactive interaction and action selection, and online causal memory in the native-mode audit. Each model is evaluated on the same dataset, at the same decision nodes, and with the same scorer under both the unified explicit-state interface and the native interface that retains its KV cache, recurrent state, or memory bank. A lightweight adapter only serializes the structured fields. Neither mode exposes reference states, action labels, response windows, or future observations. With native memory, StreamScore increases modestly (+1.27 to +4.57), VFR remains generally low, and Early decreases slightly. The ranking is identical to that under the explicit interface: ThinkStream-3B > MOSS-VL-Realtime > VideoLLM-online > MMDuet, with 0 changes in the top 3. The native interface yields limited gains, leaves the rankings unchanged, and leaves timing calibration unresolved.

We agree that memory is essential to answering later in a stream. Models also need to learn when to respond proactively: they should trigger a response when evidence is sufficient and wait or remain silent when it is insufficient. A model can retain more history and still be uncertain about when to respond. StreamAct evaluates state and memory separately from readiness and action. In our results, the transition from evidence to action poses a greater bottleneck than memory capacity. The main leaderboard uses a unified explicit-state interface to diagnose proactive response under a common memory interface. The native-mode audit further shows that the rankings remain unchanged and timing calibration remains a problem when models retain native memory. These results support the central conclusion that answerability alone does not determine online behavior. Streaming intelligence requires state, readiness, action, and answers to work together across the trajectory. We will clarify the relationship between memory and proactive response in the revision and scope the leaderboard findings to controlled diagnosis under the unified explicit-state interface.

ModelInterfaceState↑Ready↑Action↑VFR↑Early↓Miss↓StreamScore↑ΔStream.
ThinkStream-3BExplicit28.3652.2822.692.1935.2360.5026.38-
Native29.7454.8633.925.2834.9129.1430.95+4.57
MOSS-VL-RealtimeExplicit21.6448.0627.852.2133.8458.8324.94-
Native23.4151.3736.125.8233.4157.3629.18+4.24
VideoLLM-onlineExplicit27.0047.0021.501.3030.5765.0324.20-
Native27.1247.9127.762.4530.2164.3826.31+2.11
MMDuetExplicit24.3746.8520.942.6338.2169.7423.70-
Native25.0647.7324.122.9737.8468.9224.97+1.27

Weakness 3. Figure readability and clarity of writing

We take your criticism in Weakness 3 seriously. Several figures and tables are too dense, and some labels and legends need to be clearer. Parts of the text are also compressed, making it harder for readers to follow the method and interpret the conclusions. We appreciate you pointing this out. In the revision, we will systematically review the figures, tables, and organization of the text to present the key information more clearly and make it easier to understand.


Q1. Can streaming models be evaluated in their native modes?

Thank you for the suggestion. As explained in our response to Weakness 2, the revision adds a native-mode audit for ThinkStream-3B, MOSS-VL-Realtime, VideoLLM-online, and MMDuet. We run each model under the unified explicit-state interface and under its native interface, retaining the KV cache, recurrent state, or memory bank as appropriate. A lightweight adapter only serializes the structured fields, and future observations are withheld. Under the native interface, StreamScore increases modestly (+1.27 to +4.57), VFR remains generally low, Early decreases slightly, and the ranking is identical to that under the explicit interface. Native memory yields limited gains and leaves timing calibration unresolved. We therefore scope the leaderboard conclusions to controlled diagnosis of online behavior under the unified explicit-state interface. The revision will report both protocols' results and paired confidence intervals.

Q2. What is the self-timed prompt, and do all models use the same prompt?

Thank you for asking about the prompt. The paper uses two related settings with different output templates. Both follow the same strict causality constraints and use the same style of general instructions.

First, the paired evidence-boundary diagnostic in Section 2.2 uses self-timed replay. The model receives only the current video chunk and the question, chooses when to make its first response, and outputs its answer. This setting does not require structured state, readiness, or action fields. The full prompt follows.

You are observing a video stream incrementally. At each step, you receive only the current video chunk and the question. You do not have access to future frames, reference answers, or any annotation about when to respond.

Question: {question}
Options: {options}

Decide on your own whether the evidence observed so far is sufficient to answer the question.
- If the evidence is sufficient, output the answer immediately using exactly this format:
  ANSWER: {option_label}
- If the evidence is not yet sufficient, output exactly:
  WAIT

Do not explain your reasoning. Do not output anything else. Once you emit ANSWER, your response is final and will be scored as your first response.

Second, StreamAct-Bench's strict causal replay at each decision node requires the model to output state_phase, state, readiness, action, and answer. The state field has a hard budget of 128 tokens. The full prompt follows.

You are observing a video stream incrementally under strict causality. At each replay node, you receive only the current media chunk, the task objective, the question and options, and your own previously emitted state. You do not have access to future observations, reference answers, action labels, response windows, full media prefixes, or native hidden caches.

Objective: {objective}
Question: {question}
Options: {options}
Previous carried state: {previous_state}

Observe only the current media chunk. Update the carried state if needed, judge whether the evidence is sufficient, choose an action, and answer only when warranted. Return one compact JSON object with exactly these fields:

{
  "state_phase": "before" | "changing" | "after" | "na",
  "state": "concise task-relevant summary, at most 128 tokens",
  "readiness": <number between 0 and 1>,
  "action": "wait" | "silence" | "trigger" | "answer",
  "answer": "<option label A-D>" | null
}

Rules:
- Use "wait" when evidence is not yet sufficient but may become sufficient later.
- Use "silence" when no response is warranted under the task condition.
- Use "trigger" when the instructed condition occurs and a proactive response is required.
- Use "answer" only when the evidence supports a specific option; otherwise set answer to null.
- Do not use future observations, reference answers, action labels, or response windows.
- Return only the JSON object, with no additional text.

All models within each setting use the same general prompt. Model-specific wrappers only serialize the media input and output schema. They leave the instructions identical across models.

Reviewer_kWXxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the gap between causal-prefix answerability and autonomous online behavior in video-language models. The authors introduce StreamAct, which formulates streaming video understanding as a strict-causal decision trajectory over carried state, evidence readiness, action selection, and answer emission. StreamAct-Bench contains 5,436 replayable trajectories across 91,839 decision nodes, and StreamAct-SFT provides 60,000 training trajectories from a source pool disjoint from the evaluation media.

The results show a substantial gap between target-time answerability and self-timed response validity. Existing benchmark rankings do not reliably transfer to trajectory-level streaming performance, although target-time answer accuracy remains positively associated with StreamScore. Process supervision improves action macro-F1 and reduces missed response opportunities, but premature emissions remain a significant practical limitation.

Critical strengths

This paper identifies an important evaluation gap by distinguishing causal answerability from autonomous streaming behavior. Its trajectory-level formulation makes state maintenance, evidence sufficiency, action selection, and response timing separately observable rather than evaluating only answers at evaluator-selected moments.

Critical weaknesses

The shared explicit-state interface constrains each replay step to the current media chunk, task information, and a previously emitted textual state of at most 20 words, while withholding native hidden caches and full prefixes. This facilitates controlled comparison and auditability, but it may affect models differently depending on their native memory mechanisms and interface adaptation. The paper acknowledges this distinction, but the extent to which the results transfer to native online deployments remains unclear.

Questions for authors

How robust are the model rankings and response-validity results when architecture-specific KV caches, recurrent state, or native memory mechanisms are preserved? Can the authors provide a native-memory comparison where feasible, or further characterize the scope of the shared explicit-state interface?

2 comments

We thank the reviewer for the careful assessment. We agree that the unified explicit-state interface may affect models with different native memory mechanisms unevenly. We would like to clarify the “at most 20 words” limit you highlighted. Section E of the submitted version reads “state (at most 20 words).” The implementation enforces a hard budget of 128 tokens for the carried textual state. “20 words” was a brevity instruction in an earlier prompt. We will specify the 128-token state budget consistently in Sections A.4 and E and in the Table 2 footnote. The main leaderboard evaluates every entry with the same explicit textual state channel, which excludes native KV caches, recurrent states, memory banks, full prefixes, and proprietary controllers.

To test the robustness of rankings and response validity to the memory interface, the revision adds the native-mode audit described in our response to Reviewer mhTs. It covers four streaming models with different mechanisms: ThinkStream-3B (streaming reasoning and online state), MOSS-VL-Realtime (real-time streaming control), VideoLLM-online (online causal memory), and MMDuet (proactive interaction and action selection). We use the same dataset, decision nodes, and scorer for both modes for each model. The native interface preserves the model's KV cache, recurrent state, or memory bank. Both modes exclude reference states, action labels, response windows, and future observations. The results are below.

ModelInterfaceState↑Ready↑Action↑VFR↑Early↓Miss↓StreamScore↑ΔStream.
ThinkStream-3BExplicit28.3652.2822.692.1935.2360.5026.38-
Native29.7454.8633.925.2834.9129.1430.95+4.57
MOSS-VL-RealtimeExplicit21.6448.0627.852.2133.8458.8324.94-
Native23.4151.3736.125.8233.4157.3629.18+4.24
VideoLLM-onlineExplicit27.0047.0021.501.3030.5765.0324.20-
Native27.1247.9127.762.4530.2164.3826.31+2.11
MMDuetExplicit24.3746.8520.942.6338.2169.7423.70-
Native25.0647.7324.122.9737.8468.9224.97+1.27

With native memory enabled, StreamScore increases modestly (+1.27 to +4.57, about +3.05 on average), VFR remains generally low (gains of +0.34 to +3.61), and Early decreases slightly. The ranking is identical to that under the explicit interface. The explicit textual state already carries task-relevant information across chunks, and native caches primarily supplement it with low-level visual continuity and past tokens. The marginal gain for state maintenance and evidence assessment is limited. The main bottleneck appears to be temporal calibration from evidence to action, and native caching does not directly improve the alignment of readiness with response emission. Strict causal replay and a unified scorer restrict access to future information, so the ranking appears to reflect underlying comprehension and action policies more strongly than the memory interface.

The modest gains and unchanged ranking indicate that the main leaderboard ranking shows some robustness to the memory interface. We will clarify in the revision that the leaderboard supports a controlled diagnosis under the unified explicit-state interface. Its purpose is to audit state maintenance, evidence readiness, action selection, and answer generation within a common information boundary. The audit supports the paper's central conclusion that answerability alone does not determine online behavior and that the largest gap remains in converting evidence into action. Thank you for prompting us to clarify the scope of this evaluation.

Thank you for the additional experiments and clarification. These resolve my concern, I will keep my rating.

Comments on the paper

3 comments

Response to Reviewer mhTs

We sincerely thank Reviewer mhTs for recognizing the value of our problem formulation, evaluation framework, and controlled comparison of supervision objectives. We also appreciate your identification of two critical limitations. StreamAct-SFT yields limited gains in VFR and substantially increases Early, and the current supervision experiments use a single Qwen3-Omni backbone. We agree with these assessments. The revision will include additional experiments and state the scope of our contributions more clearly. We address the main weakness and the two questions below.


Weakness 1. Limited VFR gains, increased Early, and evaluation on a single backbone

We agree that the current results support a more specific conclusion about online streaming behavior, as we state in the paper. Process supervision shows that state, readiness, and action signals are learnable and substantially reduces missed responses. Calibrating when the model should respond autonomously remains an open problem.

On the Qwen3-Omni backbone, Full SFT increases Action from 36.58 to 53.93 (+17.35), reduces Miss from 72.94 to 48.58 (−24.36), and raises VFR from 6.68 to 11.11 (+4.43). Early also rises from 20.65 to 34.62 (+13.97). Explicit process supervision shifts the model from conservative waiting toward more proactive responding. It responds on more trajectories, and some of those responses occur before the valid window. VFR rises only modestly because the first response must satisfy both the timing and semantic criteria. We will make this trade-off explicit in the main text. Supervision improves Action, Miss, and StreamScore, while the limited VFR gain and higher Early show that timing calibration remains unresolved.

We used a single backbone to isolate the supervision objective while holding the data pool, optimization budget, and evaluation protocol fixed. The conclusions from that controlled experiment are specific to the tested backbone. We agree that cross-backbone validation is needed. The revision will add Appendix G, “Multi-Backbone Generalization Study.” It will retain the original Qwen3-Omni results, add Qwen3-VL-30B-A3B as a within-family control for modality and architecture, and include the InternVL3.5 series for validation outside the Qwen family.

Generalization of StreamAct-SFT across backbones

BackboneTrainingState↑Ready↑Action↑VFR↑Early↓Miss↓StreamScore↑ΔVFR vs Base
Qwen3-Omni-30B-A3BBase39.5959.2836.586.6820.6572.9435.53-
Qwen3-Omni-30B-A3BFull SFT39.4960.5553.9311.1134.6248.5841.27+4.43
Qwen3-VL-30B-A3BBase39.7159.8636.615.9215.0878.8435.53-
Qwen3-VL-30B-A3BFull SFT39.6460.7254.079.8526.9463.2141.07+3.93
InternVL3.5-30B-A3BBase36.8355.1928.734.7732.5062.7331.18-
InternVL3.5-30B-A3BFull SFT37.0255.5844.368.7028.9162.3936.42+3.93
InternVL3.5-20B-A4BBase32.2549.8328.464.5927.8367.5828.78-
InternVL3.5-20B-A4BFull SFT32.0653.7542.107.1528.5664.2933.77+2.56

Q1. How sensitive are the rankings to the explicit-state interface and native KV cache or memory?

This is an important limitation that we partly acknowledge in the paper. The main StreamAct-Bench leaderboard uses a unified explicit-state interface. At each replay node, the model receives only the current media chunk, the task information, and the textual state it emitted at the previous step. The protocol excludes native KV caches, recurrent states, memory banks, full prefixes, and proprietary response controllers from the inputs. The leaderboard is a controlled diagnostic with a common information boundary, auditable state carryover, and a shared output contract. This setup lets us compare state maintenance, evidence readiness, action selection, and answer generation under the same conditions and control for architecture-specific caching, memory implementations, and interface adaptation. Sections A.4, A.5, and E explain that the protocol is designed for controlled diagnosis and leaves the full memory capabilities of native deployments outside its scope.

We agree that this design may underestimate models that rely on native KV caches or recurrent states. To address the robustness question, the revision will add an evaluation of streaming models in their native modes. We selected four representative models covering different native memory and online control mechanisms. ThinkStream and MOSS-VL-Realtime are already on the main leaderboard, and VideoLLM-online and MMDuet were suggested by other reviewers. These four models cover complementary mechanisms: streaming reasoning and online state, real-time streaming control, online causal memory, and proactive interaction and action selection. For each model, we use the same dataset, decision nodes, and scorer under both the unified explicit-state interface and the native interface, which preserves its KV cache, recurrent state, or memory bank. A lightweight adapter only serializes the structured fields. Both modes exclude reference states, action labels, response windows, and future observations. We report metrics including StreamScore, VFR, Early, Miss, and Action.

With native KV caches, recurrent states, or memory banks enabled, StreamScore increases modestly (+1.27 to +4.57), and the ranking remains identical to the explicit-interface ranking: ThinkStream-3B > MOSS-VL-Realtime > VideoLLM-online > MMDuet. We see three possible reasons. First, the explicit textual state already carries task-relevant information across chunks. Native caches primarily add low-level visual continuity and access to past tokens, with limited marginal gains for state maintenance and evidence assessment. Second, the main bottleneck in this evaluation appears to be the timing of the transition from evidence to action. Native caching alone leaves the alignment between readiness and response emission largely unchanged, which is consistent with the limited changes in VFR, Early, and Miss. Third, strict causal replay and a shared scorer restrict access to future information. Under these conditions, the rankings appear to reflect underlying comprehension and action policies more strongly than the memory interface. The modest gains and unchanged ranking indicate some robustness to the memory interface among the tested models. We continue to interpret the main leaderboard as a controlled diagnostic of online behavior under the unified explicit-state interface. The additional evaluation supports the paper's central conclusion that answerability alone does not determine online behavior and that the largest gap remains in converting evidence into action.

Table R1. Native streaming evaluation vs. the explicit-state interface

ModelInterfaceState↑Ready↑Action↑VFR↑Early↓Miss↓StreamScore↑ΔStream.
ThinkStream-3BExplicit28.3652.2822.692.1935.2360.5026.38-
Native29.7454.8633.925.2834.9129.1430.95+4.57
MOSS-VL-RealtimeExplicit21.6448.0627.852.2133.8458.8324.94-
Native23.4151.3736.125.8233.4157.3629.18+4.24
VideoLLM-onlineExplicit27.0047.0021.501.3030.5765.0324.20-
Native27.1247.9127.762.4530.2164.3826.31+2.11
MMDuetExplicit24.3746.8520.942.6338.2169.7423.70-
Native25.0647.7324.122.9737.8468.9224.97+1.27

Q2. How does the Early–Miss trade-off vary across backbones?

Thank you for raising this question. Answer supervision rewards final answer correctness alone and provides no direct signal for when to respond. The baseline therefore tends to wait conservatively, with high Miss and low Early. Process supervision introduces explicit state, readiness, and action targets. In effect, it lowers the emission threshold from “wait when uncertain” to “respond when evidence appears.” Readiness calibration remains weak, so the model is more likely to respond prematurely near the evidence boundary. Early rises as a result, and VFR improves only modestly. The direction of the change in Early varies with the backbone's initial tendency to emit. The Qwen baselines are more conservative, and Early rises markedly as the threshold falls. The InternVL3.5-30B baseline is already relatively proactive (Early 32.50, Miss 62.73). SFT primarily improves Action and VFR, and Early decreases by 3.59. The 20B-A4B model lies between these cases and exhibits smaller changes. Table R1 shows that preserving native memory changes StreamScore only modestly, leaves the ranking unchanged, and slightly reduces Early. This suggests that the phenomenon primarily concerns the alignment of readiness with response emission. The revision will report the threshold shift by backbone and use existing logs to analyze first-response offsets and Early∩Miss.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.