acceptodds
Under review as a conference paper at ICLR 2027

Have I Seen Enough? Frozen Video-Language Models Encode Evidence Readiness

Abstract

Streaming video-language models must decide not only what to answer, but whether the evidence needed for the current question has arrived. Existing systems learn that decision as a separate trigger; we ask whether an unmodified model already computes it. We show that frozen VideoLLMs carry a linearly readable evidence-readiness signal, labelled from timestamped evidence rather than from model output. It decodes in all seven models of a shared byte-identical evaluation (AUROC – under the strictest not-ready sampling, where a fitted clock is near chance), and a probe fitted without any of a benchmark family's footage still reads that family. It is question-conditioned: on byte-identical windows, changing only the question reverses the readout on of pairs, while every question-blind control is at chance by construction. The model can answer incorrectly and still encode readiness: AUROC remains among wrong answers. Readiness also beats uncertainty estimators and their supervised combination on latency-matched answer selection, and tracks independent human judgments more closely than confidence. Released streaming triggers are also linear readouts, but their learned directions are approximately orthogonal to readiness and decode it less accurately than dedicated readiness probes. We turn the readout into Readiness Gating, an answer-timing policy that improves accuracy by up to pp at matched latency with negligible computational overhead. How much it gains varies with the accuracy headroom the task makes available: across configurations the gain tracks that headroom, and an intervention that moves it over identical pixels moves the gain with it.

Primary area:
interpretability and explainable AI
Keywords:
Video-language models, streaming video understanding, evidence readiness, multimodal reasoning, mechanistic interpretability, linear probing, representation analysis, activation steering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3K9aOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether frozen Video Language Models (VideoLLMs) already encode information about evidence readiness, i.e., whether sufficient visual evidence to answer a given question has been observed. The authors train linear probes on hidden representations of frozen VideoLLMs to predict evidence readiness, and evaluate the resulting signals across multiple models and video benchmarks. They further introduce Readiness Gating, a simple inference-time strategy that uses the predicted readiness scores to determine when to answer or which answer to select. Experimental results suggest that evidence readiness can be decoded from frozen representations and that readiness-based gating can improve answer accuracy in several evaluated settings.

Critical strengths

Evaluation Across Multiple Models and Datasets. The paper evaluates its proposed approach across multiple VideoLLMs and video datasets, providing a relatively broad experimental assessment.

Critical weaknesses
  1. Excessive AI-assisted writing and poor readability. The manuscript exhibits extensive signs of AI-assisted writing, which significantly undermine its readability. Moreover, many experiments appear to be defensive analyses designed to anticipate potential reviewer concerns, rather than hypothesis-driven investigations. The writing and overall presentation raise concerns about the extent of substantive human revision and critical oversight.

  2. Limited methodological novelty. The paper does not introduce a fundamentally new algorithm, optimization mechanism, or model architecture. Instead, it primarily relies on existing techniques, including linear probing of frozen representations and threshold-based gating. The methodological contribution is therefore limited.

  3. Insufficient differentiation from prior work. What is the unique contribution of this paper compared with StreamReady [1], which has already investigated evidence readiness and answer timing in streaming video understanding? The authors should more clearly articulate the novelty and significance of their work beyond existing readiness-aware approaches.

  4. Insufficient evaluation in realistic streaming scenarios. It remains unclear whether the proposed method can reliably handle continuous video streams in which the correct answer changes over time. For instance, RTV-Bench [2] introduces Multi-Timestamp Question Answering (MTQA), where answers evolve as the visual scene changes. Evaluating the proposed method under such dynamic conditions would provide stronger evidence of its robustness and practical applicability.

Reference [1] Azad, S., Vineet, V., & Rawat, Y. S. (2026). StreamReady: Learning What to Answer and When in Long Streaming Videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) [2] Xun, S., Tao, S., Li, J., et al. (2025). RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video. Advances in Neural Information Processing Systems (NeurIPS)

Questions for authors
  1. Novelty and differentiation from prior work. What is the key contribution of this paper compared with StreamReady [1]? Given that the proposed approach mainly relies on standard linear probing and threshold-based gating, what distinguishes it from existing readiness-aware methods?
  2. Generalization to realistic streaming scenarios. Can the proposed method reliably handle continuous video streams where answers change over time? For example, RTV-Bench [2] includes Multi-Timestamp Question Answering (MTQA) tasks that require tracking evolving visual information. How well would the proposed method generalize to such scenarios?

Reference [1] Azad, S., Vineet, V., & Rawat, Y. S. (2026). StreamReady: Learning What to Answer and When in Long Streaming Videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) [2] Xun, S., Tao, S., Li, J., et al. (2025). RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video. Advances in Neural Information Processing Systems (NeurIPS)

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3K9a (1/3)

Dear Reviewer 3K9a,

Thank you for your review and for acknowledging the breadth of our evaluation. We especially thank you for pointing us to RTV-Bench, which led us to extend our selection rule to streams whose answer changes; the extension is now part of the method. In short: on SPOT-Bench, a linear readout of a frozen model's state, used as the response trigger, beats the own trigger of each of five systems by +3.5 to +6.7 T-F1@5 with one readout for both tasks (+1.3 to +5.0 with one readout per task); StreamReady trains a module to decide when to answer, whereas we show that frozen models already encode the state such a module needs; and the readout transfers to RTV-Bench without retraining and follows the evidence as the answer changes.

W1. Writing and presentation.

Thank you for these comments on the writing and on how the analyses are presented; the revised version addresses each of them. We improved the readability of the entire paper by hand, with shorter sentences, one idea per sentence, one name for each concept, and one or two numbers per claim in the main text, moving robustness details to the appendix. We also made the logic of the analyses explicit. The Introduction now states four hypotheses: readiness is linearly readable in a frozen model, it depends on the question, it is distinct from confidence and correctness, and it helps decide when to answer. Each subsection of Section 4 now opens with its hypothesis, prediction, test and result, so that every control appears as the test its hypothesis requires, since a probing result can only be interpreted next to controls for what the probe itself can learn (Hewitt & Liang, 2019). We designed the experiments and verified every claim and number ourselves.

W2. Methodological novelty.

The paper's main contribution is a result about what video-language models represent: an unmodified model already encodes whether the evidence for the current question has arrived, separately from its confidence and correctness. The measurement design makes this result credible, and a training-free policy puts it to use. To choose which answer to return, the policy needs no threshold: it returns the answer from the window with the highest readiness score, and only deciding when to act uses a cutoff. We deliberately add no new architecture or training method: a linear probe can only read information the model already represents, so a positive result is evidence about the model. It also works inside complete systems: used as the response trigger on SPOT-Bench, with the same labelled videos as each system's own trigger, it raises timing-weighted F1 by +3.5 to +6.7 points with one readout for both tasks (+1.3 to +5.0 with one readout per task) on five systems, including MMDuet, Dispider and the best published entry, at the cost of one dot product (table in our reply to W4). The Introduction now states this contribution type and lists the SPOT-Bench result among the contributions.

AuthorsOpenReview ↗

Response to Reviewer 3K9a (2/3)

W3 / Q1. Difference from StreamReady.

The two papers ask different questions, which Section 2 now makes explicit:

StreamReadyThis paper
QuestionHow to train a system that knows when to answerWhether an unmodified model already represents that the evidence has arrived
What is trainedA readiness token and head in an added memory-and-retrieval moduleOne linear readout; the model stays frozen
Readiness labelsPseudo evidence timestampsBenchmark evidence timestamps, checked against human raters
Added inference costThe added moduleOne dot product

StreamReady shows that a trained readiness mechanism helps. We show that the state such a mechanism needs is already present in frozen models: it is linearly readable in seven models, depends on the question on identical frames, and is distinct from confidence and correctness.

We could not run StreamReady, because its weights are not released and its benchmark releases annotations only. Both papers report OVO-Bench, so we place them side by side on its forward group (Appendix R): three tasks, of which step recognition and clue reveal ask when to respond and the third asks for a count, where only the backbone acts. StreamReady scores 58.8 there, using no OVO-Bench labels. Under OVO-Bench's own protocol, a frozen Qwen3-VL-8B with our linear readout scores 58.6 with a readout trained on our other benchmarks and only its cutoff set on 8 labelled OVO videos, and 63.4 with the readout fitted on OVO labels from other videos. The plain backbones start level (49.9, against 48.7 for OVO's published Qwen2-VL-7B, the backbone StreamReady builds on). A linear readout of a frozen model therefore reaches the level of a trained readiness module on these tasks.

This is not a controlled comparison, since the backbones and the supervision differ. The controlled comparisons are with the released triggers we could run: MMDuet and Dispider in Section 4.5 and on SPOT-Bench, and Ctrl-SPOT, the best published SPOT-Bench system, in our reply to W4. We also compare the forward group only. On the backward group, and so on the overall average, StreamReady's memory module puts it ahead: those tasks need memory of earlier video, which a readout does not provide.

AuthorsOpenReview ↗

Response to Reviewer 3K9a (3/3)

W4 / Q2. Realistic streams, including answers that change.

Following your suggestion, we evaluated the readout on RTV-Bench: 552 streams of up to 3.1 hours, in which 1,052 question groups ask the same question several times with different correct answers.

Generalisation. A Qwen3-VL-8B probe fitted only on our other benchmarks reads RTV-Bench at 0.629 [0.618, 0.642] AUROC with its controls near chance (raw visual features 0.51, shuffled labels 0.48, elapsed time 0.55, confidence 0.57), retaining 92% of a probe refitted on RTV-Bench (0.681).

Tracking. Across 487 answer changes, the score rises while the new evidence is in view (+0.084 [+0.065, +0.103]) and returns to its earlier level afterwards (−0.009 [−0.028, +0.009]).

Selection. Our rule returns the answer from the highest-readiness window so far, which assumes a fixed answer, so we restrict it to windows ending in the last 32 seconds (a bound chosen on development videos; on test videos, overall accuracy is flat for any bound from 24 to 64 seconds). RTV-Bench asks each question at the moment its answer's evidence appears, so it never tests what happens once that moment has passed, which is when choosing a window matters. We therefore built a protocol from its annotations that asks each question again at later times. Asked 16 seconds or more after a change, on test videos:

Rule (accuracy)Qwen3-VL-8BInternVL3.5-8B
Current 16-second window0.4670.453
Highest readiness so far (unbounded rule)0.4350.404
Highest readiness in the last 32 seconds0.4970.493

That is +6.1 [+3.6, +8.7] and +8.9 [+6.5, +11.6] points over the unbounded rule, and +3.0 [+1.4, +4.6] and +4.0 [+2.5, +5.5] over the current window. We added the bounded rule to Section 3.3, the results to Sections 4.3 and 4.4, and Appendix S.

Complete streaming systems. Beyond RTV-Bench, the revision adds two standard streaming benchmarks, each scored by its own code. On SPOT-Bench (NeurIPS 2026 Datasets and Benchmarks), a system must decide every second whether to respond. The system's own trigger and our readout come from the same forward pass and are tuned on the same labelled videos, cross-fitted by video because SPOT-Bench has no training split; the readout also fits a linear direction on those labels, while the own trigger has only its decision rule tuned. T-F1@5 (timing-weighted F1), action boundaries / point of no return:

SystemOwn trigger, tuned on SPOT labelsReadout, one fit for both tasksGain, both tasks
Qwen3-VL-8B30.29 / 16.2436.36 / 22.05+5.93 [4.62, 7.24]
Qwen2.5-VL-7B27.86 / 14.3833.58 / 20.76+6.05 [4.86, 7.23]
Ctrl-SPOT (best published entry)32.21 / 21.5239.98 / 27.10+6.67 [5.31, 8.05]
MMDuet (released code)26.27 / 19.2931.65 / 20.93+3.51 [2.33, 4.71]
Dispider (released code)22.62 / 13.8728.61 / 17.16+4.64 [3.41, 5.95]

When Ctrl-SPOT acts on the readout live, on all 608 videos, the gain is +5.06 [3.78, 6.38]. On OVO-Bench, the readout decides when to respond on the forward tasks (see our reply to W3). Both are in Section 4.5 and Appendix R.

Thank you again for your review. All new results are included in the revised version. We hope they address your concerns, would be grateful if you took them into account in your final assessment, and would be happy to answer any follow-up questions.

Dear reviewer 3K9a, We would like to thank you again for your time and effort.

We hope that our rebuttal has addressed your concerns. With the discussion period nearing its end, we would appreciate hearing your thoughts.

We would be happy to address any remaining concerns or provide further clarification. If our responses have resolved your concerns, we would greatly appreciate it if you could reconsider your score.

Reviewer_pkCXOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether frozen video-language models already encode evidence readiness in their internal representations: whether the frames received so far contain the visual evidence needed to answer the current question. The authors derive labels from evidence timestamps and train linear probes on frozen hidden states. They examine the signal and its generalization through question changes on identical frames, controls for confidence, analyses of incorrect answers, and evaluations across models and data sources. They also introduce Readiness Gating, which uses probe scores to determine when to answer or which cached answer to select, improving accuracy on some tasks at matched video viewing duration.

Critical strengths
  1. Studying evidence readiness in streaming video understanding through internal representations is a novel and well-motivated perspective. The paper distinguishes evidence availability from model confidence and answer correctness, and shows that the signal can improve answer timing or selection on some tasks, giving the representational finding practical value.

  2. The model coverage is substantial, and the validation procedure is careful. The main seven-model evaluation covers four language-backbone families and model sizes from 2B to 32B using identical video inputs. Nested cross-validation grouped by video separates configuration selection from final testing, strengthening confidence in the core finding.

Critical weaknesses
  1. The paper uses inconsistent definitions of ready. Growing prefixes are labeled ready as soon as some annotated evidence appears, whereas partially covered examples in fixed windows are excluded from probe training and testing and labeled not ready in answer-timing evaluations. Why does seeing only part of the evidence receive different labels in the two settings? Although the authors report results under alternative labeling rules, they do not explain the basis for this distinction.

  2. The comparative experiments provide limited support for the method's advantages. First, according to the supplementary code, the direct yes/no baseline receives only the question stem, without the original answer options included in the probe's input. For broad questions such as “What happens in this video?”, omitting the options may change what the model is being asked to judge. Together with the different supervision conditions, this makes the source of the probe's advantage unclear. Second, the main comparisons with MMDuet and Dispider focus on window-selection tasks adapted by the authors. Although OVO-Bench is used, it mainly serves to test probe transfer. Without direct comparisons of the complete answer-response methods on standard streaming tasks, the practical gains and relative advantages of Readiness Gating on such tasks remain unclear.

Questions for authors
  1. Why is partial evidence coverage labeled ready for growing prefixes, while the same coverage state is excluded or labeled not ready for fixed windows? What justifies this distinction? Do the main conclusions hold if both input formats consistently treat partial coverage as positive, or consistently treat it as negative?

  2. According to the supplementary code, the direct yes/no prompt does not include the original answer options. Please confirm whether this is the setting used for the reported results and explain why the options were omitted. Does the probe's advantage remain when the baseline receives the same question and options? Can the existing evidence distinguish the advantage of reading internal representations from the effects of differences in inputs, prompting, and supervision?

  3. Why were Readiness Gating and existing response-triggering methods not compared in terms of their complete answer-response performance on standard streaming tasks such as those in OVO-Bench? How might differences between the current window-selection procedure and the original response procedures of MMDuet and Dispider affect the comparison?

6 comments
AuthorsOpenReview ↗

Response to Reviewer pkCX (1/3)

Dear Reviewer pkCX,

Thank you for the detailed review and for recognising the novelty of our perspective and the care of our validation. We especially thank you for reading the supplementary code, which led us to a stronger yes/no baseline. In short: the conclusions hold under each single, consistent labelling rule; with the question and the options, the probe still ranks readiness better than asking the model, pooled over our seven models (+0.084 AUROC; ahead on six of them); and on SPOT-Bench, a linear readout used as the response trigger beats the own trigger of each of five systems, MMDuet and Dispider included, with the same labelled videos, by +3.5 to +6.7 T-F1@5 with one readout for both tasks (+1.3 to +5.0 with one readout per task).

W1 / Q1. Partial coverage on prefixes and windows.

The label always asks whether the delivered frames contain the question's annotated evidence; the two conventions follow from what each format was built to test. Fixed windows, our primary format, test whether the model has received the evidence it needs independently of elapsed time. Evidence enters and leaves a window, so a window is ready only when it holds all of it; a partial window is an intermediate state, excluded from binary probing and counted as not ready in the policy evaluation, since a policy must act at every moment. Growing prefixes were built to test the earliest arrival of evidence: a prefix only gains evidence, so it is ready from onset, and its partial states are studied through the graded fraction of segments seen (Appendix E.5). Requiring full coverage on a prefix would mostly measure elapsed time: a prefix holds all of the evidence only once it has played past it, so "ready" would largely mean "late in the video", and a probe could score well by tracking how far the video has progressed rather than the evidence. Elapsed time is one of our controls for this reason; when full coverage is required on prefixes, the probe's lead over it on NExT-GQA and ST-Evidence narrows to +0.025 to +0.048, from up to +0.100 under the other rules.

We agree that the paper should have set out the two conventions and their reasons side by side. Following your suggestion, we re-ran all analyses under one rule at a time: partial coverage counted as ready, as not ready, or excluded, everywhere.

Probe. Across all 35 combinations of rule, benchmark and definition of "partial" (three prefix pools and four window ladders; Appendix O), the conclusions hold under every rule. For example, the probe reads 0.802–0.845 AUROC on the Oops! windows and 0.831–0.935 on the ST-Evidence-Gen prefixes across rules, and changing the rule moves its AUROC by at most 0.062 on windows and 0.104 on prefixes. In all 35 it exceeds the model's confidence and stays above every control; its margin over elapsed time is +0.074 to +0.33 on windows and +0.025 to +0.207 on prefixes, significantly above zero even under the strictest rule.

Answer timing. The rule changes no verdict. On the Oops! ladder, gated stopping beats the same rule on confidence alone at matched video seconds by +6.1 [+3.7, +8.6] points with partial windows counted as not ready (Section 4.4's +6.03, recomputed at a tighter solver tolerance) and +6.2 [+3.8, +8.8] with partial windows counted as ready.

We added a "Labelling rule" box to Section 3.1, the full results in Appendix O, and a table listing the input and decision rule for every benchmark, with the reason for each (Appendix N).

AuthorsOpenReview ↗

Response to Reviewer pkCX (2/3)

W2 (first part) / Q2. The yes/no baseline without the answer options.

We confirm that the reported baseline received the question but not the options, and thank you for checking this. The prompt template came from benchmarks whose questions are specific; on the Oops! and ActivityNet ladders the question is one fixed sentence, so without the options the model is not told which events are in question. The published numbers are correct for that prompt (a re-run reproduces them exactly). We re-ran the baseline on all seven of our models with the options in the multiple-choice prompt's own order and letters, using two wordings and one variant:

  • A: "Do you have enough information from the video so far to answer this question? {question} {options} Answer Yes or No."
  • B: "{question} {options} Can the correct option be determined from the video so far? Answer Yes or No."
  • B′: B without its closing "Answer:" line, added because small models otherwise reply with an option letter.

Each model's wording is fixed by one rule that uses no labels: A, unless the model rarely answers Yes or No to it, and then the wording it answers Yes or No to most (B′ on Qwen3-VL-2B, Idefics3-8B and Pixtral-12B). Asking improves substantially (Qwen3-VL-8B: 0.712 → 0.796 on Oops!, 0.503 → 0.723 on ActivityNet), and the probe, refitted in the same run (0.831 and 0.756), still ranks better on both: +0.035 [+0.017, +0.053] on Oops! and +0.034 [+0.019, +0.048] on ActivityNet. Pooled over the seven models, the probe ranks readiness better, and it is ahead on six of them; on the largest, Qwen3-VL-32B, asking matches it (Oops! ladder, AUROC):

ModelPromptAskingProbeProbe − asking
Qwen3-VL-8BA0.7960.831+0.035 [+0.017, +0.053]
Qwen3-VL-2BB′0.6770.818+0.141 [+0.117, +0.164]
Idefics3-8BB′0.6560.725+0.069 [+0.045, +0.094]
GLM-4.1V-9BA0.6340.737+0.103 [+0.080, +0.127]
Pixtral-12BB′0.5950.698+0.103 [+0.079, +0.127]
LLaVA-OneVision-1.5-8BA0.6640.800+0.136 [+0.113, +0.159]
Qwen3-VL-32BA0.8450.8450.000 [−0.016, +0.016]
Pooled over seven models+0.084 [+0.072, +0.096]

Within Qwen3-VL, asking improves with size much faster than the probe, so the margin is smallest on the models that are best at reporting readiness; the next paragraph shows that even there the model's internal state carries more than its answer says.

Representation versus input, prompt and supervision. The probe and asking differ in input, prompt and supervision, so we removed these differences one at a time.

  1. Same input and prompt. We read both from one forward pass of the yes/no prompt: asking is the model's spoken probability of Yes, and the probe reads the final-layer state from which that Yes/No is computed. The probe still ranks readiness better on all seven models (Oops! ladder): +0.049 [+0.034, +0.065] on Qwen3-VL-8B and +0.021 to +0.152 on the other six, every interval above zero.
  2. Not a reweighted answer. Removing the model's own Yes-minus-No direction from the state makes the spoken answer constant and moves the probe by at most 0.001 AUROC, on all seven models.
  3. Same supervision. We gave the answer's own Yes/No scores the same labels and fitting as the probe. On Qwen3-VL-32B, the hardest case because asking is strongest there (0.845), this gains the answer 0.001 AUROC, and the probe still leads by +0.020 [+0.008, +0.031].

The model therefore represents readiness information that its answer does not use. This same-pass probe is a diagnostic: it needs the yes/no pass, and 32 (Oops!) to 64 (ActivityNet) labelled videos to beat asking in at least 93% of random draws on Qwen3-VL-8B. The probe in the table reads the answering pass and adds one dot product.

As the trigger of our stopping rule, asking also stays behind readiness at matched video duration (+3.7 and +4.2 pp on Oops! and HERBench; +1.6 [−0.0, +3.2] on ActivityNet; Appendix Q).

We added these results in Appendix Q and two sentences on the same-pass result to Section 4.2.

Q3 (second question). The rivals' own procedures in Table 3.

Applied to their own scores on the Table 3 ladders, MMDuet's own firing rule fires before the evidence on 95.7% (HERBench) and 97.8% (ActivityNet) of items, and Dispider's first trigger on 86.4% of HERBench items. With a standard multiple-choice prompt, MMDuet answering at its own firing moment is at chance (HERBench 0.227 [0.202, 0.253], chance 0.25). Table 3 let each trigger pick its best window after the fact, so it favoured the released triggers.

AuthorsOpenReview ↗

Response to Reviewer pkCX (3/3)

W2 (second part) / Q3. Complete systems on standard streaming benchmarks.

Table 3 tests whether a released trigger reads evidence readiness; it does not compare complete systems. We therefore compared complete systems, both rivals you name included, on two standard streaming benchmarks, each scored by its own code: SPOT-Bench (NeurIPS 2026 Datasets and Benchmarks) and OVO-Bench.

SPOT-Bench. We ran MMDuet, Dispider and SPOT's best published system, Ctrl-SPOT, from their released code (our Ctrl-SPOT run is within 1.3 points of its published score per task), and two plain models through SPOT's own prompt wrapper (the rivals take SPOT's raw questions). In each row, the system's own trigger and our readout come from the same forward pass and are tuned on the same labelled videos, cross-fitted by video because SPOT-Bench has no training split; the readout also fits a linear direction on those labels, while the own trigger has only its decision rule tuned. T-F1@5 (timing-weighted F1), action boundaries / point of no return:

SystemLeaderboard setting: own rule, no labelsOwn trigger, tuned on SPOT labelsReadout, one fit for both tasksGain, both tasks
Qwen3-VL-8B11.59 / 9.6330.29 / 16.2436.36 / 22.05+5.93 [4.62, 7.24]
Qwen2.5-VL-7B10.26 / 10.2027.86 / 14.3833.58 / 20.76+6.05 [4.86, 7.23]
Ctrl-SPOT21.65 / 18.2732.21 / 21.5239.98 / 27.10+6.67 [5.31, 8.05]
MMDuet10.41 / 5.4926.27 / 19.2931.65 / 20.93+3.51 [2.33, 4.71]
Dispidern/a22.62 / 13.8728.61 / 17.16+4.64 [3.41, 5.95]

Dispider's replies never contain SPOT's response words ("start", "end", "now"), so it has no leaderboard-setting score. Tuning lifts every system above its leaderboard setting, so the tuned columns are not leaderboard entries; our claim is the last column, the readout against the same system's tuned trigger. With Ctrl-SPOT acting live on each trigger, its answers fed back into its prompt, the readout gains +5.06 [3.78, 6.38] on all 608 videos.

MMDuet's gain comes from action boundaries (point of no return +1.64 [−0.01, 3.43]). Dispider's tuned trigger is no better than a readout fitted on labels shuffled within each video, which our readout beats by +4.14 [3.12, 5.13]. With the same labels and regularisation search, a readout of each rival's state matches or beats its own refitted head (one fit for both tasks: MMDuet +1.72 [0.84, 2.57], Dispider +1.29 [0.44, 2.11]; level per task), and our Qwen3-VL-8B system beats both refitted rivals, per task and for both tasks: MMDuet by +3.09 [1.89, 4.34] (level on point of no return alone) and +4.55 [3.35, 5.75], Dispider by +5.85 [4.58, 7.16] and +6.72 [5.50, 7.91]. The readout also chooses its layer; a refitted head keeps its position.

OVO-Bench. We ran all twelve tasks under OVO's own protocol and scorer. A trigger decides the answer only on the two forward tasks that ask when to respond, step recognition and clue reveal; on the other ten only the backbones differ, so we compare systems on these two:

SystemStep recognitionClue reveal
Dispider, published by OVO-Bench (our re-run)37.36 (36.88)48.75 (50.00)
MMDuet, own stream, released rule, no OVO labels56.7675.83
Qwen3-VL-8B, plain reply, OVO's input63.1242.50
Qwen3-VL-8B, 16 s window, own P(Yes), fitted on OVO labels72.3459.17
Same, our readout decides72.8173.33

Every row answers each query point with no memory of earlier answers; MMDuet answers Yes if it responds on the last frame up to that point. We lead it on step recognition (+16.06 [9.76, 22.93]) and are level on clue reveal (10 films; −2.50 [−10.00, 3.91]), also when both systems hold their answer once given (+8.74 [3.30, 14.40] and −1.67 [−7.04, 3.16]). On step recognition the readout only matches the model's own P(Yes); its matched claim rests on SPOT-Bench. Over all twelve tasks we score 63.89, Dispider 41.78 (42.02 in our re-run) and the plain model 51.47; most of the gain comes from answering from the current window (59.39).

On SPOT-Bench the rivals, like the plain models, see a fresh 4 s clip at each decision moment, not their continuous stream, which may understate their triggers; on OVO-Bench MMDuet runs on its own stream, but OVO's published Dispider row calls Dispider once per pre-cut clip, bypassing its own timing.

We added these results to Section 4.5 and Appendices R and I, with a pointer in the Table 3 caption.

Thank you again for the careful review. All new results and clarifications are in the revised version. We hope they resolve your concerns, would be grateful if you took them into account in your final rating, and would be happy to answer any follow-up questions.

Reviewer_QUDaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether frozen video models encode readiness: whether the currently observed video contains the evidence needed for a question. The authors measure this with a linear probing using labels from evidence timestamps in videos with readiness gating. The experiments include in-depth analyses on how a series of models behave across diverse scenarios with this signal.

Critical strengths

I believe this paper explores very interesting directions and show meaningful findings. Distinguishing evidence availability from answer confidence and correctness is meaningful for streaming setting. The finding that readiness remains decodable even when the model answers incorrectly is particularly interesting. Controlled analyses, such as Identical-frame question swaps and crossed-question designs, provide substantial support that the probe captures question-dependent evidence information rather than merely elapsed time or visual salience.

Critical weaknesses

First, the central readiness interpretation is not fully distinguished from evidence relevance. The supervision primarily measures coverage of annotated evidence rather than whether the observed information is sufficient to answer. Growing-prefix labels mark evidence onset, and partial-coverage windows are excluded from the main binary probing evaluation. Although the human judgments and graded analyses provide supporting evidence, distinguishing relevant-but-insufficient observations remains less established. This is consequential because the paper’s central interpretation concerns knowing whether enough evidence has arrived, which requires more than recognizing question-related content.

Secondly, the practical advantage over existing response triggers is not established under comparable supervision. The proposed probe receives supervision from the target readiness labels, whereas released triggers are evaluated zero-shot. Using MMDuet’s own activations removes the representation difference but preserves this training imbalance. Combined with the substantial threshold-calibration failure under distribution shift, the results leave uncertainty about whether the proposed signal provides a more general response-control mechanism or primarily benefits from task-specific supervision. A comparison under matched supervision would materially strengthen this claim.

Questions for authors

See the above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer QUDa (1/2)

Dear Reviewer QUDa,

Thank you for the careful and constructive review. We are glad you found the separation of readiness from confidence meaningful for streaming. Your distinction between relevance and sufficiency, and your call for a comparison under matched supervision, led to the new experiments below, which are included in the revised paper. In short: on Qwen3-VL-8B, among screens that already show relevant evidence, the probe predicts human "enough" judgments at 0.925 AUROC; and on SPOT-Bench, with the same labelled videos, the readout beats the own response trigger of each of five systems by +3.5 to +6.7 T-F1@5 with one readout for both tasks (+1.3 to +5.0 with one readout per task).

W1. Readiness versus relevance.

We agree that the label measures delivered coverage of annotated evidence, and we tested directly whether the score goes beyond relevance. Three tests show that it carries information about sufficiency beyond coverage and beyond relevance.

Human judgments on relevant screens. Among the 420 rated screens that already show some annotated evidence, the probe (Qwen3-VL-8B) separates those that independent raters judged "enough" from the rest at 0.925 [0.893, 0.950] within the same benchmark and coverage stage, against 0.665 for the model's confidence. It adds +0.31 [+0.24, +0.36] AUROC to the coverage fraction and holds inside single items (0.776 [0.672, 0.871]).

Relevant but insufficient by construction. On 520 HD-EPIC colour composites, the asked-about object is shown in its rendered colour, greyed with its shape kept, or absent. A probe trained only on natural video scores these 0.918 / 0.580 / 0.162 and separates readable from greyed at 0.913 [0.894, 0.931], whereas a probe trained on the composites to detect the object places the greyed case 81–97% of the way to readable. A raw-visual-feature probe trained the same way, on natural video only, is at chance. The ordering holds on Qwen3-VL-2B (readable vs greyed 0.873; greyed vs absent 0.915) and, more weakly, on Idefics3-8B (0.609 and 0.808).

Relevant versus needed, on identical frames. We then built 260 instances on 31 of the same HD-EPIC videos by pasting two objects, A and B, into their frames, and asked different questions about the same frames. On Qwen3-VL-8B, adding B raises the probe's log-odds by +2.5 to +2.7 on average per instance when the question needs both objects (their colours, or whether they match), by +0.28 when it names B without needing it ("What color is the A? Ignore the B."), and by +0.04 when it does not mention B. Showing A twice, with as much object footage as A and B, does not raise it. A alone is relevant but not sufficient when the question needs both, and it earns partial credit: on the colour-pair question, the mean score is 0.033 with no object, 0.585 with A alone and 0.899 with both. On these questions, the probe ranks the complete input above the three inputs with only one of the two objects (A only, B only and A twice) at AUROC 0.925 and 0.966 on Qwen3-VL-8B, against 0.853 and 0.709 for the model's confidence, and at 0.897 and 0.867 on Qwen3-VL-2B, where confidence reads 0.680 and 0.294. A probe on Qwen3-VL-8B's raw visual input does not separate "A and B" from "A twice".

Partial windows. Our main probing leaves out windows that show only part of the evidence, so we also scored them. A probe trained only on windows with all or none of the evidence, which never saw a partial window, places partial windows between the two on all seven models. Inside the same item it ranks the complete window above the partial one at 0.732 [0.706, 0.755] on average over the seven models (Oops! ladder), above the raw-visual-feature control on every model (smallest margin +0.047).

We now state in Sections 1 and 3.1 that the label measures delivered coverage, report the three tests in Section 4.1, and give the full results in Appendix P.

AuthorsOpenReview ↗

Response to Reviewer QUDa (2/2)

W2. Comparison under matched supervision.

Thank you for proposing this comparison. We ran it in two settings where both sides receive the same labelled videos, and in one where neither side receives labels from the target benchmark.

Matched supervision: MMDuet's heads given our labels. Refitted from their trained weights on our probe's training folds, MMDuet's released heads gain 0.10–0.21 AUROC and reach a probe at the same layer and token (0.820 vs 0.823 on Oops!). The readiness state is therefore linearly present in MMDuet's activations; its streaming objective did not select it. With the same labels, our system still reads the two Table 3 ladders well above either refitted head (0.93 vs 0.77 on HERBench; 0.75 vs 0.60–0.61 on ActivityNet). On SPOT-Bench (NeurIPS 2026 Datasets and Benchmarks), a recent streaming benchmark with its own scorer and leaderboard, with the same labels and the same regularisation search on both sides, our Qwen3-VL-8B readout beats MMDuet's refitted heads by +3.09 [1.89, 4.34] per task (level on point of no return alone) and +4.55 [3.35, 5.75] with one fit for both tasks; our readout also chooses its layer, while a refitted head keeps its position.

Against each system's own trigger. On SPOT-Bench, the readout and a system's own response probability come from the same forward pass and are tuned on the same labelled videos, cross-fitted by video because SPOT-Bench has no training split for detection; the readout also fits a linear direction on those labels, while the own trigger has only its decision rule tuned. T-F1@5 (timing-weighted F1), action boundaries / point of no return:

SystemOwn trigger, tuned on SPOT labelsReadout, one fit for both tasksGain, both tasks
Qwen3-VL-8B30.29 / 16.2436.36 / 22.05+5.93 [4.62, 7.24]
Qwen2.5-VL-7B27.86 / 14.3833.58 / 20.76+6.05 [4.86, 7.23]
Ctrl-SPOT (best published entry, released code)32.21 / 21.5239.98 / 27.10+6.67 [5.31, 8.05]
MMDuet (released code)26.27 / 19.2931.65 / 20.93+3.51 [2.33, 4.71]
Dispider (released code)22.62 / 13.8728.61 / 17.16+4.64 [3.41, 5.95]

The table scores each trigger on hidden states recorded once. When Ctrl-SPOT acts on the readout live, on all 608 videos, the gain is +5.06 [3.78, 6.38]. The readout is one fit for both tasks' labels; with one readout per task the gains are +1.3 to +5.0 (Appendix R).

No target labels on either side. A probe fitted only on our other benchmarks and applied unchanged to the two ladders of Table 3 selects better answers than every released head in Table 3 on both (selection accuracy 0.539 on HERBench and 0.573 on ActivityNet).

Threshold. Gated selection needs no threshold. Where one is needed, refitting only the intercept of this transferred probe on 2 labelled target videos raises decision accuracy from 0.584 to 0.869 on HERBench (best possible 0.877) and from 0.472 to 0.737 on ActivityNet.

We added a paragraph "Under matched supervision" to Section 4.5, a subsection to Appendix I, and Appendix R on SPOT-Bench and OVO-Bench.

Thank you again for the careful review and your support. All new results are included in the revised version, and we would be happy to answer any follow-up questions.

Dear reviewer QUDa,

Thank you again for your time and effort.

We hope that our rebuttal has addressed your concerns. With the discussion period nearing its end, we would appreciate hearing your thoughts.

We would be happy to address any remaining concerns or provide further clarification. If our responses have addressed your remaining concerns, we would greatly appreciate it if you could consider strengthening your support for the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

Thank you for your time and for the careful and constructive reviews. We are glad the reviewers found the separation of readiness from confidence meaningful for streaming, and that they recognised the novelty of our perspective, the care of our validation and the breadth of our evaluation. All new results are included in the revised version, with every change marked in blue.

We would like to highlight the main points:

  1. Readiness versus relevance (QUDa). Following QUDa's comment, we now state in Sections 1 and 3.1 that the label measures delivered coverage of annotated evidence, and we tested directly whether the score goes beyond relevance. Three tests support this. On screens that already show relevant evidence, the probe predicts human "enough" judgments (0.925 AUROC on Qwen3-VL-8B, against 0.665 for the model's confidence). A probe trained only on natural video separates a readable object from a greyed one whose colour cannot be read. On identical frames, adding a second object raises the score far more when the question needs it than when it only names it (Section 4.1, Appendix P).

  2. Labelling partial coverage (pkCX, QUDa). Section 3.1 now explains why fixed windows and growing prefixes treat partial coverage differently. Evidence enters and leaves a window, so a window is ready only when it holds all of it; a prefix only gains evidence and holds all of it only after playing past it, so requiring all of it there would make "ready" largely mean "late in the video". Following pkCX's suggestion, we re-ran the analyses with partial coverage counted as ready, as not ready, or excluded, and the conclusions hold under every rule (Appendix O).

  3. Matched supervision and complete systems (QUDa, pkCX, 3K9a). With the same labels, our Qwen3-VL-8B probe still reads both Table 3 benchmarks above MMDuet's released heads refitted on those labels, and with no target labels on either side, a probe fitted only on our other benchmarks selects better answers than every released head. We also compared complete systems on SPOT-Bench and OVO-Bench. On SPOT-Bench, used as the response trigger with the same labelled videos as each system's own trigger, a linear readout raises timing-weighted F1 by +3.5 to +6.7 points on five systems, MMDuet and Dispider included, with one readout for both tasks (+1.3 to +5.0 with one readout per task). The readout also fits a linear direction on those labels, while each system's trigger has only its decision rule tuned (Section 4.5, Appendices I and R).

  4. The yes/no baseline with the answer options (pkCX). We re-ran the baseline with the options on all seven models. It improves substantially on Qwen3-VL-8B, and pooled over the seven models the probe still ranks readiness better (+0.084 AUROC on Oops!), ahead on six and level on the largest. Read from one forward pass of the same prompt, the probe ranks better than the model's Yes/No answer on all seven models, and giving that answer the same labels does not close the gap on the largest. The model therefore represents readiness information that its answer does not use (Appendix Q).

  5. Realistic streams, including answers that change (3K9a). Following 3K9a's suggestion, we evaluated the probe on RTV-Bench. It transfers without retraining and follows the evidence as the answer changes. Since our selection rule assumed a fixed answer, we extended it to search only recent windows, which after an answer change is more accurate than both the original rule and the current window. The extension is now part of the method (Section 3.3, Appendix S).

  6. Contribution and difference from StreamReady (3K9a). The paper's main contribution is a result about what video-language models represent: an unmodified model already encodes whether the evidence for the current question has arrived, separately from its confidence and correctness. Since a linear probe can only read information the model already represents, a positive result is evidence about the model. StreamReady trains a module to decide when to answer, whereas we show that frozen models already encode the state such a module needs. The Introduction and Section 2 now make this explicit.

We also revised the writing throughout. The Introduction now states four hypotheses, and each subsection of Section 4 opens with its hypothesis, prediction, test and result, so that every control appears as the test its hypothesis requires.

We hope these experiments and clarifications address the reviewers' concerns. Thank you again for your time; we would be happy to answer any follow-up questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.