acceptodds
Under review as a conference paper at ICLR 2027

CommitStream: Can Models Keep Their Answers Correct in Streaming Video?

Abstract

Advances in embodied intelligence and world modeling place increasing demands on models’ video understanding. For video assistants, a key challenge is that an answer that is correct when issued can become wrong as the scene changes. If an assistant neither updates nor withdraws it, users may continue acting on outdated information. Reliable assistance therefore requires maintaining an answer that is supported by the available evidence throughout an ongoing task. We introduce CommitStream, a benchmark that jointly evaluates when models should wait for evidence, provide an answer, and revise it as events unfold. To support this evaluation, a reusable pipeline generates and refines annotations of when answers become supported and when they must change, followed by a final human review. The resulting benchmark contains 1,006 tasks from 307 video episodes across six domains. Evaluation follows causal real-time playback, using streaming inference for models with a streaming input interface and asynchronous sliding-window inference for those without one. Under the official ±1 s tolerance, the best model with streaming inference remains correct throughout 19.88% of tasks; the best with asynchronous sliding-window inference reaches 29.03%. Even the latter leaves stale or wrong answers in effect for 13.94% of scored time. These results identify sustained answer reliability as an unresolved challenge and establish CommitStream as a testbed for studying when video assistants should answer, wait, and update. Code and data are available at https://anonymous.4open.science/r/commitstream.

Primary area:
datasets and benchmarks
Keywords:
Streaming Video Understanding, Answer Maintenance, Multimodal Large Language Models, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_EBdWOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CommitStream, a benchmark for evaluating sustained answer reliability in live video understanding, focusing on when models should defer, commit, or update answers as visual evidence evolves. The benchmark comprises 1,006 tasks from 307 video episodes across six domains and five temporal modes, with reference trajectories constructed through automated annotation and human review. The authors evaluate nine streaming models and five asynchronous sliding-window models using metrics such as ETR, role-specific scores, and error exposure. Results show that the best models achieve ETR scores of only 19.88% and 29.03%, respectively, highlighting the challenges of maintaining reliable answers in dynamic video environments.

Critical strengths
  • Meaningful Benchmark Design. The paper extends conventional video question answering from pointwise correctness to sustained answer reliability. By explicitly evaluating deferring, committing, and updating, it addresses an important limitation of existing benchmarks.

  • Comprehensive Evaluation and Informative Analysis. The benchmark evaluates 14 models across multiple domains and temporal tasks using two inference protocols. Its metrics capture persistent errors and correction behavior, revealing important differences in models' ability to maintain reliable answers over time.

Critical weaknesses
  • Temporal Change Recognition. The current evaluation measures answer correctness based on response completion time, which conflates temporal change recognition with inference and scheduling latency. Although Table 3 investigates sensitivity to timing tolerance, it does not distinguish whether delayed updates result from the model failing to recognize relevant visual changes or from the time required to generate responses. Consequently, the evaluation provides limited insight into the models' actual temporal change recognition capabilities.
  • The notation a(S) in Equation (1) should be explicitly defined in the text for clarity.
Questions for authors
  1. Could the authors evaluate models' sensitivity to state changes under the current evaluation protocol? For example, recording the timestamp of the latest video frame used for each response, along with its completion time, could help assess how promptly models recognize state changes and quantify their response latency.

  2. Although the paper states that the model should defer until the target pitcher can be identified, could the authors further clarify why the available visual evidence is insufficient to support an earlier off_table answer in the pitcher example?

4 comments

1. Distinguishing Answer Correctness After State Changes from Response Latency

In streaming video, practical utility depends on both whether an answer is correct and whether it is available in time. Because the video continues to play while the model processes frames and generates text, the user receives a new answer only when generation finishes. We therefore update the public state at response completion time, so evaluation across defer, commit, and update reflects the actual answer available to the user.

To separate recognition of new visual evidence from response latency, we analyze answer accuracy on the first request that processes frames after a reference state change and the mean response time of that request for the five models below:

ModelAccuracy on First Post-Change RequestMean Response Time of That Request
Qwen3.8-Flash-Next64.96%0.60s
GLM-5.3-Flash75.81%1.82s
dots3-note-prev60.93%0.98s
MiMo-V2.6-Flash-RL34.41%0.74s
InternVideo3-8B-Instruct27.06%0.53s

These results show the trade-off between post-change recognition accuracy and response speed. GLM-5.3-Flash achieves the highest accuracy on the first post-change request (75.81%) but has a longer mean response time (1.82s), whereas InternVideo3-8B-Instruct responds faster (0.53s) but has low accuracy after receiving the post-change frames (27.06%). Even when a request already includes frames after a state change, models often fail to output the updated answer. Thus, delayed updates cannot be explained by generation latency alone.

This analysis complements D-Score, C-Score, U-Score, and Mean Role Score (MRS). The role scores measure the correctness of the public state across defer, commit, and update segments, while the table above separates answer correctness upon receiving new visual evidence from the time needed to return that response to the user. We will add these results to the revised paper and explicitly state the newest input-frame timestamp, request submission time, and response completion time for each response.

2. Clarifying the Definition of a(S)a(S) in Equation (1)

We will add the following explicit definition immediately before Equation (1):

a(S)a(S) denotes the percentage of scored seconds in reference segment SS at which the public state is accepted as correct, using the same timing tolerance rule as all other metrics.

D-Score, C-Score, and U-Score then average a(S)a(S) across all reference segments of the corresponding role (defer, commit, and update), and the Mean Role Score (MRS) is the arithmetic mean of these three role scores.

3. Why Figure 1 Cannot Answer off_table Earlier

The key distinction is between "the target object is not in the visible area of the tabletop" and "the target object is not on this table." If the entire target tabletop were clearly visible and we could rule out the pitcher resting anywhere on it, a model could answer off_table even before seeing the pitcher directly.

In Figure 1, when the query is issued at 6 s, the camera covers only part of the wooden table, and the left and right edges of the tabletop extend beyond the frame. Although the current frame does not show the target pitcher, it cannot rule out that the pitcher is resting on an unviewed part of the same table. At this point, both on_table and off_table lack sufficient visual evidence, so the reference trajectory is defer.

At 15 s, the target pitcher with the orange lid and handle is held and brought into the foreground, providing direct visual evidence that it is off the table and supporting a commitment to off_table. At 21 s, its base rests on the tabletop, and the answer updates to on_table. The reason for waiting before 15 s is that the current field of view is insufficient to determine the support relation, not that the model must predict future changes. We will add this clarification to the caption of Figure 1.

Thank you again for your time and thoughtful feedback. With less than two days remaining in the discussion period, we would like to follow up on our responses to your comments.

If you have any remaining questions or concerns, please let us know—we would be happy to provide further clarification before the deadline.

We appreciate your consideration and look forward to hearing from you.

Thanks for the additional experiments and clarifications provided by the authors. I will keep my original score.

Thank you for your response and for taking the time to consider our clarifications. We appreciate your thoughtful feedback and careful consideration of our work.

Reviewer_31gaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors address the task of video models performing videoQA in a streaming setting, where ambiguity of the answer can change as the video progress. To assess this new task, the authors propose CommitStream, which measures three key behaviors for the model: (1) to wait - when the answer isn't clear yet, (2) to answer - when there is sufficient evidence to answer something, and (3) to update - when the answer changes depending on new visual evidence. The benchmark composes videos from several sources (Wikimedia Commons, CaptainCook4D, Perception Test, DSEC, MEVA, Breakfast Actions, the Leeds Complex and Long Activities Dataset, NOAA Fisheries, and DVIDS) with 1,006 tasks from 307 episodes. The authors propose some metrics (D/C/U per-role, exact trajectory rate, etc) for the benchmark, and evaluate several standard VLMs. Overall - preliminary rating is borderline (marking as WR pending rebuttal).

Critical strengths

The task formulation seems well-motivated, clear, and useful. Compared to several related work, it does seem to meaningfully add to the landscape of videoQA benchmarks. Code and example data was also released in the supplement. The authors also do run evaluations over 14 representative models, which provides good coverage over standard methods.

Critical weaknesses

Using GPT-6 Astra to generate the annotations can lead to systemic biases which don't seem to be well characterized yet. While two human authors do review each question, this just leads to an accept/reject decision rather than a refinement. It would be helpful to better understand how well the exact timestamps and annotations align with human annotator judgement (as calibrated against inter-annotator agreement, which is also not reported). There is a ~98% pipeline accuracy statement but that's on some undisclosed test set of ~49. This is all especially relevant since the official ETR metric has a +/- tolerance of 1 second, so if the annotations themselves have tolerance beyond that the metric may not be appropriate and/or need a different tolerance window (or later models can be penalized unfairly). Finally, it would be helpful to have more vanilla reference baselines to help calibrate (e.g., an always silent baseline or oracle baseline) - this can also help identify potential biases.

Questions for authors

Please address the points raised in weaknesses - thanks!

2 comments

Thank you for your constructive suggestions on annotation reliability, timestamp precision, and reference baselines.

1. Human Verification and Inter-Annotator Agreement

To evaluate the reliability of the reference annotations across the dataset, we randomly sampled a fixed subset of 100 tasks (5,506 scored seconds) from all 1,006 tasks for human verification. Two annotators independently performed blind annotation. Afterward, we flagged tasks with disagreements so each annotator could re-check their own work and fix operational annotation errors (such as misselected labels or missing entries), while retaining genuine disagreements on video content or temporal boundaries. After correcting operational errors, per-second agreement between the two annotators is 99.10%. Agreement between the final reviewed human annotations and the original Gold reference trajectory is shown below:

ComparisonAgreement
Between two annotators (after reviewing and fixing operational errors)99.10%
Final human annotation vs. Gold (strict per-second match)98.04%
Final human annotation vs. Gold (±1 s tolerance)98.60%

Strict per-second comparison requires identical labels at the exact same second, while the ±1 s setting uses the official timing tolerance rule from the paper. This verification extends our original task-level accept/reject review to individual scored seconds and their visual evidence.

Within this verified subset, final human judgments match the Gold labels on the vast majority of scored seconds. Even without timing tolerance, agreement reaches 98.04%; applying the ±1 s tolerance adds only 0.56 percentage points, showing that high agreement does not mainly come from relaxing temporal boundaries.

2. Temporal Boundary Precision and ±1 s Tolerance

To check whether timestamp annotations match human judgment, we directly compared the timing of the same answer state transitions between human annotations and the Gold reference trajectory, rather than only counting per-second agreement across entire videos.

Temporal Boundary VerificationResult
Clearly matched state changes with time difference ≤ 1 s98.02%

Among clearly matched state changes, 98.02% have a human–Gold time difference of at most 1 second, providing direct human verification for our official ±1 s tolerance.

The official tolerance accepts adjacent reference states around transitions to reduce the impact of minor boundary differences on scoring. In addition, Table 3 in the paper reports results at ±2 s and ±3 s to show how sensitive model performance is to wider timing tolerances. Together, human boundary verification and multi-tolerance evaluation support both our timestamp precision and scoring stability.

3. Automated Annotation Bias and Origin of the 97.96% Figure

The 97.96% figure in the paper comes from an ablation study of the annotation pipeline. We randomly sampled 49 questions and used blind human annotations as the reference to compare different pipeline configurations on the same tasks. A task was counted as correct only if both its answer content and timestamps matched the human reference across the entire task within the specified tolerance. At ±1 s tolerance, the full pipeline passed 48 of the 49 questions (48/49 = 97.96%).

Thus, the original 97.96% result measures full-task annotation agreement to compare pipeline stages, whereas our new human verification provides per-second agreement and transition boundary precision.

4. Simple Reference Baselines and Metric Calibration

Following your suggestion, we used the official evaluator at the official ±1 s tolerance to evaluate an Always silent baseline and a Reference trajectory Oracle across all 1,006 tasks.

BaselineD-ScoreC-ScoreU-ScoreMRSETR
Always silent100.00%2.82%0.04%34.28%0.20%
Reference trajectory Oracle100.00%100.00%100.00%100.00%100.00%

Always silent keeps the initial defer public state throughout each task, achieving a 100.00% D-Score, but only 2.82% C-Score and 0.04% U-Score. Its C-Score and U-Score are slightly above zero only because the official timing tolerance accepts the defer state near a small number of transition boundaries, not because this strategy ever makes valid commitments or updates. Because MRS averages the three roles equally, Always silent reaches 34.28% MRS; therefore, MRS should be interpreted alongside the individual role scores rather than as overall per-second accuracy.

The Reference trajectory Oracle outputs the corresponding reference answer immediately at each reference state transition, achieving 100.00% on all metrics and providing an upper bound to calibrate the metric scale.

Thank you again for your time and thoughtful feedback. With less than two days remaining in the discussion period, we would like to follow up on our responses to your comments.

If you have any remaining questions or concerns, please let us know—we would be happy to provide further clarification before the deadline.

We appreciate your consideration and look forward to hearing from you.

Comments on the paper

3 comments

Thank you for your constructive feedback on answer withdrawal coverage, response timing, earliest answerable time, and historical memory.

1. Coverage of Explicit Defer

CommitStream evaluates whether models can keep their answers reliable as a streaming video evolves, and explicit defer provides the necessary action to withdraw a committed answer. It differs from silence: silence preserves the current public state, whereas explicit defer withdraws an existing answer when it loses visual support and no replacement answer is yet certain.

The segment coverage in the current benchmark is as follows:

Behavior SegmentCount
Defer: waiting for sufficient evidence, including a few post-commitment withdrawals142
Commit: forming an evidence-backed answer1,011
Update: updating a previous answer1,125

In total, 693 tasks involve answer updates. Thus, our results mainly compare model performance on waiting (defer), answering (commit), and updating (update), while the 5 withdrawal tasks provide an initial evaluation of post-commitment withdrawal. We will clarify this distinction in the revision so that conclusions about withdrawal match the current sample size. In addition, our reusable annotation pipeline supports building targeted withdrawal subsets under the same task definitions and scoring rules.

2. Timing Sensitivity and Failure Sources

Your analysis of BF_20f7303fbe89/C1 is correct: the state changes at second 9, and the next request is sent at second 12, so the system continues displaying the previous public state at seconds 10 and 11. To account for such scheduling effects, Table 3 in the paper reports results on the same predicted trajectories under ±1 s, ±2 s, and ±3 s tolerances. For the example you noted, the ±3 s tolerance accepts the pre-transition state at seconds 10 and 11, so the trajectory does not fail solely due to those two seconds (though it may still fail if subsequent responses are further delayed or incorrect).

In streaming video, both accuracy and timeliness matter. For tasks where sufficient evidence is already available at the start of the query, users need a timely response because the video continues playing while they wait, and the relevant state may change again. Therefore, Exact Trajectory Rate (ETR) measures the continuous correctness of the user-visible public state from the start of the query, jointly evaluating first-response delay, subsequent updates, and waiting time rather than isolated video understanding ability. To separate these failure sources, we analyzed the raw inference logs of five sliding-window models across all 1,006 tasks.

First, we examined whether models answer correctly on the first request containing post-change frames, and how long they take to update to the correct answer after an initial error:

ModelMean First-Request LatencyFirst-Response Error RateSuccessful Update Rate After First ErrorMean Extra Wait for Successful Update
Qwen3.8-Flash-Next0.60s35.04%23.02%9.75s
GLM-5.3-Flash1.82s24.19%35.19%7.82s
dots3-note-prev0.98s39.07%21.10%11.27s
MiMo-V2.6-Flash-RL0.74s65.59%9.15%13.14s
InternVideo3-8B0.53s72.94%14.13%9.30s

Here, "first request" refers to the first request after a state change that includes post-change frames. Response correctness is judged by content rather than latency, and silence carries forward the public state from the request. A successful update requires the correct answer to take effect before the state changes again or the evaluation ends. Extra wait is measured from when the initial incorrect response is returned, averaged over cases that later update successfully with non-negative wait time.

Thus, untimely updates stem not only from request intervals and response latency, but also from incorrect answers and slow corrections.

Second, alongside the ±1 s and ±3 s ETR results in Table 3, we computed the cumulative error duration per task to check whether failures are mostly brief 1–2 second slips:

ModelETR (±1 s)ETR (±3 s)Tasks with Cumulative Error >5 sTasks with Cumulative Error >10 s
Qwen3.8-Flash-Next29.03%42.05%45.03%33.60%
GLM-5.3-Flash8.35%13.32%55.77%37.87%
dots3-note-prev24.95%34.39%48.11%37.48%
MiMo-V2.6-Flash-RL14.71%17.79%71.57%60.34%
InternVideo3-8B9.05%11.53%81.31%71.37%

The last two columns are calculated under the official ±1 s tolerance. For example, Qwen's ETR increases from 29.03% to 42.05%, but more than half of the tasks still fail to stay correct across the full trajectory. Meanwhile, under the official ±1 s setting, 33.60%–71.37% of tasks across models accumulate more than 10 seconds of error. This indicates that low ETR is not generally caused by brief 1–2 second boundary errors.

3. Earliest Answerable Time and Future Information

The earliest answerable time is defined by whether the frames observed up to that second are sufficient to support an answer under the task definition. This differs from the physical state onset determined in hindsight after watching the entire video. Future frames may reveal what physically happened earlier, but they cannot be used to claim that a model already had sufficient visual evidence at that earlier moment.

Our original annotation pipeline used local segments and global observation to generate and verify candidate timelines, recording physical state onset and evidence sufficiency time separately. To independently validate the prefix-restricted condition, we conducted an additional chronological human annotation study. The interface unlocked video frames second by second; annotators could not access future frames before submitting their decision for the current second, and the Gold reference trajectory was hidden. Annotators judged whether to defer or provide a specific answer based solely on the unlocked prefix.

Comparison Between Prefix-Restricted Human Annotation and GoldAgreement Rate
Exact second-by-second agreement98.04%
Agreement under official ±1 s tolerance98.60%

4. Historical Information and Explicit Memory in Sliding Windows

The 16-frame window limits the visual content directly read in each request, but information outside the window does not immediately disappear from the system state. Under our original protocol, each request is given the previously applied public state; the model can either update the answer based on new frames or remain silent to keep the existing answer. Therefore, once relevant evidence leaves the 16-frame window, a previously confirmed answer can still be retained.

For CC4D_12_16/C5, it is necessary to distinguish between key visual evidence having left the current window and whether the model already used that evidence to form an answer while it was visible. The original protocol preserves the latter through public state feedback. If subsequent judgments require historical event details beyond the answer label itself, explicit memory can be introduced.

Following your suggestion, we evaluated two explicit memory mechanisms across all 1,006 tasks on five models while keeping the original 16-frame visual window and 4-second request interval:

  • Original Protocol (A): Feeds back only the latest applied public state.
  • Updated Memory (B1): Maintains and updates a structured historical summary provided to subsequent requests.
  • Accumulated Memory (B2): Accumulates previously generated historical records provided to subsequent requests.

To observe how historical memory affects state updates, we compared accuracy on the first request that sees post-change frames (using identical request opportunities across all three settings):

ModelOriginal Protocol (A)Updated Memory (B1)Accumulated Memory (B2)
Qwen3.8-Flash-Next64.96%71.51%69.27%
GLM-5.3-Flash75.81%76.08%76.16%
dots3-note-prev60.93%50.99%52.33%
MiMo-V2.6-Flash-RL34.41%19.71%19.98%
InternVideo3-8B-Instruct27.06%17.83%12.81%

With explicit memory, Qwen improves under both B1 and B2, GLM remains stable, and the other models decline. This shows that explicit historical memory can help some models update their answers, but the gain depends on how well a model records and uses historical information; storing more history does not automatically improve performance. We will clarify the state feedback mechanism of the original protocol and include these memory experiments in the revised paper.

Thank you again for your time and thoughtful feedback. With less than two days remaining in the discussion period, we would like to follow up on our responses to your comments.

If you have any remaining questions or concerns, please let us know—we would be happy to provide further clarification before the deadline.

We appreciate your consideration and look forward to hearing from you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.