acceptodds
Under review as a conference paper at ICLR 2027

Supervising the Visible State: Separating Representation from View Selection in Streaming Video

15 views

Abstract

An action seen once and the same action seen three times receive the same action-recognition label, so that supervision does not require a model to count. Answers shared across histories that differ in event order, accumulation, or relations leave those distinctions unconstrained; we call this state under-specification. Visible-State Supervision (ViSS) addresses it by training on source-grounded probes of event identity, order, accumulation, relation change, and answer sufficiency; under the same recent-frame interface it improves OVO-Bench over a strict-matched generic temporal supervision baseline while remaining comparable on StreamingBench, with the largest gain in hallucination detection. Test-Time Causal View Search (TVS) then adapts evidence access while keeping the learned model fixed, expanding from recent frames to sparse event and compact history views on demand. The two interventions separate representation from view selection: supervision particularly benefits hallucination detection and causal reasoning, access particularly benefits event recall and forward reasoning. Together they reach the best reported accuracy among the published systems compared here on StreamingBench and OVO-Bench at a fraction of the processed frames of exhaustive selection, and strict replay shows that most of the remaining gap to a three-view oracle is a correct answer that was later replaced. Streaming accuracy depends on what supervision asks the model to keep, what evidence inference exposes, and whether the model keeps a correct answer once it has one.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming video understanding, Temporal representation learning, Vision-language models, Temporal reasoning, Adaptive evidence access

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cDh5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses state underspecification​ in streaming video understanding, where standard answer supervision often does not force models to distinguish event order, counts, or changing relationships. To tackle this, it proposes Visible-State Supervision (ViSS), which fine-tunes Qwen3-VL-8B using source-grounded probes covering event identity, order, accumulation, relation changes, and evidence sufficiency. At test time, Test-Time Causal View Search (TVS)​ adaptively selects among recent, event, and history views without modifying the trained model. Experiments on StreamingBench and OVO-Bench include a carefully matched temporal-supervision baseline, three paired training seeds, fixed-checkpoint view comparisons, and replay diagnostics. Results show that supervision and evidence access provide complementary, task-dependent benefits. The combined system achieves 86.33% accuracy on StreamingBench​ and 69.77% macro accuracy on OVO-Bench, while processing fewer frames than an exhaustive three-view evaluation. Replay analysis further reveals that replacing previously correct answers is the main factor behind the remaining gap to the three-view oracle.

Critical strengths
  1. The controlled comparisons are carefully designed. The generic temporal supervision baseline matches training instances, visual inputs, weights, ordering, optimization, and initialization. Access comparisons freeze the checkpoint. These controls help distinguish additional training from changes in evidence access.

  2. The supervision improvement has statistical support. Across three paired seeds, the OVO-Bench improvement over generic supervision is consistently positive, averaging 1.72 percentage points. A source-video paired bootstrap yields a 95% confidence interval of [0.80, 2.60]. The complete system improves over the recent-four-frame baseline by 5.81 and 10.80 points on the two benchmarks.

  3. The error analysis is concrete and appropriately scoped. The paper distinguishes proposal availability, acquisition, acceptance, and replacement using query-level counts. It explicitly conditions the dominance of replacement errors on coverage by the current three-view oracle, rather than labeling all-view failures as representation errors.

Critical weaknesses
  1. Limited methodological novelty. Supervision for counting, ordering, and state changes, as well as on-demand access to historical evidence, has precedents. The additional methodological contributions of ViSS and TVS over related work are limited.

  2. Insufficient detail on data construction and efficiency. Data-generation configurations and Generic–ViSS pairing rules remain underspecified. Efficiency comparisons mainly report model calls and processed frames, without measurements of actual latency or peak GPU memory.

  3. No additional gain on StreamingBench. Compared with strictly matched generic temporal SFT, ViSS shows a mean change of −0.17 percentage points across three seeds, providing no evidence of an advantage on this benchmark.

  4. Unclear necessity of the supervision mixture. Existing ablations do not sufficiently establish the value of the five supervision categories and their proportions. A controlled comparison with generic temporal QA plus insufficient-evidence supervision is missing, leaving the full mixture’s additional contribution to hallucination detection unclear.

Questions for authors

see above

4 comments
AuthorsOpenReview ↗

A kind reminder of the upcoming end of the discussion period

Dear Reviewer cDh5,

We would like to follow up on the supplementary experiments addressing your questions about supervision composition and hardware efficiency. Q4 reports three Generic+Sufficiency runs and explains their relationship to the original matched comparison. Q5 provides measured latency and peak GPU memory across the access policies. We also clarify the conclusions supported by these experiments and the remaining limits on mixture-level attribution.

We would appreciate your feedback on these additions and how they affect your assessment. Please let us know whether any further explanation would help resolve your remaining concerns.

Thank you for your careful review and consideration.

Best regards,
The Authors

AuthorsOpenReview ↗

How much of the OVO supervision gain comes from hallucination detection

Dear Reviewer cDh5,

Your proposed Generic QA + Sufficiency comparison raises a related question that we can answer from the reported scores: how much of the overall ViSS advantage comes from hallucination detection (HLD) itself? Under OVO’s official scoring, HLD is one of three backward tasks, and backward tracing is one of three equally weighted groups. HLD therefore has weight 1/9 in the overall score.

ComparisonOverall OVO changeHLD contribution to that changeAll other tasks combined
ViSS vs. strictly matched Generic, mean over three paired seeds+1.72+0.86+0.86
Generic + Sufficiency vs. its Generic reference, supplemental seed 59+3.02+3.64−0.62

The first row uses the three-seed HLD improvement of 7.71 points: 7.71/9 ≈ 0.86. HLD thus accounts for about half of the matched ViSS gain; the other tasks together contribute the remaining half. In the supplemental comparison, the 32.80-point HLD improvement contributes 32.80/9 ≈ 3.64 points to the overall score, with the other tasks offsetting 0.62 points.

These are descriptive decompositions of the official score. Each row uses its own Generic reference, and the supplemental experiment uses a different data version. The result sharpens our claim in two ways: generic QA with sufficiency examples already supports strong HLD performance, and the strictly matched ViSS comparison shows a positive aggregate contribution from tasks beyond HLD. The incremental value of each supervision family remains open. We appreciate your suggestion, which helped us distinguish these conclusions.

Thank you for the responses and additional experiments. The analysis is valuable, but I still find the methodological novelty limited and the gains over strong baselines modest. I will keep my original rating unchanged.

Dear Reviewer cDh5,

Thank you for considering the additional experiments. Your suggestion to test Generic QA + Sufficiency helped us sharpen the paper’s conclusion: targeted sufficiency supervision already supports strong hallucination detection, while roughly half of the OVO gain in the strictly matched ViSS comparison comes from other tasks in aggregate.

We understand your reservations about methodological novelty and the size of the gains. We would ask you to weigh one further point in your overall recommendation: the study identifies a concrete failure that aggregate accuracy hides. Among the cases where one of the three views yielded a correct answer but TVS ultimately answered incorrectly, the policy had already retained the correct answer and then replaced it in 67% of cases on StreamingBench and 72% on OVO. This gives researchers a specific question to test when designing streaming systems: when additional history changes a prediction, how reliably does the model preserve an answer it already got right?

Would you be willing to reconsider your overall recommendation in light of that empirical contribution? We appreciate the care you have taken with the paper.

Reviewer_UDceOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates two complementary factors in streaming video understanding: temporal state supervision and test-time evidence selection. The authors introduce Visible-State Supervision (ViSS), which fine-tunes a VLM using 7,677 temporally structured examples covering event identity, temporal ordering, accumulation, relation changes, and answer sufficiency. They further propose Test-Time Causal View Search (TVS), which adaptively expands the available visual context from recent frames to event-level and historical views using confidence-based acquisition and answer selection.

Critical strengths
  • The paper provides an intuitive distinction between temporal state supervision and visual evidence selection, with a well-organized presentation of the two interventions.
  • The improvements in hallucination detection and the analysis of incorrect answer replacement provide useful observations about streaming video reasoning.
Critical weaknesses
  • The proposed ViSS largely reformulates existing temporal instruction-tuning objectives into structured question-answer supervision covering counting, ordering, event recognition, and answer sufficiency. Similarly, TVS combines predefined frame-selection policies with confidence-based inference and answer replacement. These components build upon established temporal post-training and adaptive evidence-selection techniques. Although their integration is reasonable, the paper does not establish a sufficiently distinctive algorithmic contribution beyond the combined formulation and controlled evaluation.
  • Table 3(a) shows that ViSS achieves 84.77% on StreamingBench, slightly below the strict-matched generic supervision baseline at 84.97%. On OVO-Bench, the improvement is only 1.73 percentage points (63.48% to 65.21%). Although the OVO improvement is statistically supported, the limited gain raises questions about whether the proposed visible-state formulation offers a substantial advantage over conventional temporal fine-tuning.
  • Several reported results exceed the published human baselines, including OCR on OVO-Bench and selected perception and reasoning categories on StreamingBench. While exceeding human performance on individual tasks is not inherently problematic, the paper does not sufficiently clarify whether human and model scores were obtained under directly comparable observation conditions.
  • The paper interprets improvements from ViSS as evidence that temporally structured supervision produces more useful visible-state representations. However, the experiments primarily evaluate downstream answer accuracy rather than directly measuring the representation of event order, accumulation, or state transitions.
Questions for authors
  • Could the authors clarify the differences from existing temporal instruction-tuning, adaptive frame sampling, and evidence-selection methods?
  • Why does ViSS slightly underperform strict-matched generic supervision on StreamingBench while improving OVO-Bench by only 1.73 points?
  • Since the fixed full-history view achieves nearly identical accuracy on both benchmarks, what is the specific advantage of TVS beyond computational savings?
1 comment
AuthorsOpenReview ↗

A kind reminder of the upcoming end of the discussion period

Dear Reviewer UDce,

Thank you again for your follow-up questions, which prompted the additional analyses in our November 13 response. We reported confidence intervals for the OVO accuracy gain and the StreamingBench median-latency difference, together with the higher mean and tail latency and the costs of sequential inference.

We would appreciate hearing whether these results clarify the accuracy–latency trade-off you raised. Your feedback on any remaining concern would help us focus the discussion, and we would welcome your updated assessment in light of this additional evidence.

Thank you for your continued engagement.

Best regards,
The Authors

Reviewer_4FTkOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces two methods: VisibleState Supervision (ViSS) and Test-Time Causal View Search (TVS). ViSS trains a model with curated temporal QA samples covering defined probes. TVS freeze the model and retains the answer based on confidence-margin constraints across different views. Both methods show improvements over state-of-the-art video models across OVO-Bench and StreamingBench.

Critical strengths

First, this kinda careful, rigorous data curation and trajectory (view in the paper) perturbations are promising directions for streaming video understanding, which requires correct decisions based on given evidence available at a certain (queried) time, unlike offline video understanding. Second, rather than simply relying on recent observations (i.e., SimpleStream), TVS is way a flexible method to explore reasoning paths for reliable prediction.

Critical weaknesses

I feel very questionable and lukewarm about the originality of this paper. SimpleStream has demonstrated that observing recent frames already provides strong visual evidence for answering questions, and referring to historical information would naturally help when recent frames alone are insufficient. The proposed method defines different view constructors; however, these simply provide different observation combinations and restrict the search to three predefined views. It remains unclear what this design contributes beyond simpler historical sampling strategies. In OVO-Bench, the methods changes Real-Time performance from 82.2 to 81.2 (no gain), while improving Backward and Forward performance. I acknowledge that the method improves StreamingBench, but I'm still unclear about the new insight here.

Questions for authors

See the weaknesses

4 comments
AuthorsOpenReview ↗

A kind reminder of the upcoming end of the discussion period

Dear Reviewer 4FTk,

We would like to follow up on your question about the insight beyond recent-frame baselines and historical sampling. Our response discusses the matched supervision comparisons, direct historical-sampling controls, and the replay finding that correct answers can be lost during subsequent revision. We also clarify the task-dependent benefits and trade-offs of TVS.

We would appreciate your feedback on whether these findings clarify the contribution, and would welcome your updated assessment. Please let us know which remaining concern would benefit most from further discussion.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_4FTkOpenReview ↗

Response by Reviewer 4FTk

Thank you for the rebuttal.

I still feel the novelty of TVS is marginal. Given its marginal improvement over simple historical sampling, I remain uncertain whether the observed gains truly come fromTVS itself or largely from access to historical frames.

Dear Reviewer 4FTk,

Thank you for following up. Table 12 provides a direct test of your question: when both policies have access to the same historical views, does the choice of policy still matter? On OVO-Bench, exhaustive max-confidence selection evaluates all three views and scores 68.43. TVS draws from those same views with the same frozen model and scores 69.77. The 1.34-point difference shows that how the system acquires views and chooses among their answers matters even when the available historical frames are held fixed. Against a single fixed-history view, TVS also gains 0.93 points (95% CI: [+0.09, +1.77]) at nearly identical processed-frame counts.

The replay analysis gives this result a concrete interpretation. In 194 of 270 OVO cases where a tested view produced the correct answer but TVS ultimately answered incorrectly, TVS had already retained the correct answer and later replaced it. This identifies a practical challenge for streaming inference: deciding when an earlier answer should stand as more evidence arrives.

Together with the ViSS comparison, which shows a 1.72-point mean OVO gain across three paired seeds under identical visual inputs, these findings separate what improves through supervision from what improves through historical access and answer selection. Given this evidence, we would appreciate your reconsidering the overall rating in light of the paper’s empirical contribution.

AuthorsOpenReview ↗

What happens when a broader view changes an answer

Dear Reviewer 4FTk,

Your question prompted us to look more closely at what happens on an individual query when additional history becomes available. One result follows directly from the transition counts in Appendix D: on 19 StreamingBench queries and 53 OVO-Bench queries, the recent view gave a wrong answer, the event view corrected it, and the compact-history view then replaced that correct answer with a wrong one.

These counts can be checked from the reported totals. Of the 85 and 194 replacement errors, respectively, 66 and 141 began with a correct recent answer. The remaining 19 and 53 therefore began with a wrong answer. With only three ordered views, the event view is the only stage at which the policy could have acquired and retained the correct answer before losing it.

This trajectory illustrates what the fixed-history comparison alone cannot tell us: a model can find the answer it needs and then discard it as it sees more of the video. That makes the decision to revise an answer a distinct research question. Table 12 measures the practical consequence on OVO: using the same frozen model and candidate views, TVS scores 69.77 versus 68.43 for exhaustive max-confidence selection. Against the single fixed-history view, the increment is 0.93 points (95% CI: +0.09 to +1.77).

We believe this connection between the controlled comparison and an identifiable failure on individual queries is the strongest empirical contribution of TVS. We would be grateful if you would take it into account in your overall assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.