- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CommitStream, a benchmark for evaluating sustained answer reliability in live video understanding, focusing on when models should defer, commit, or update answers as visual evidence evolves. The benchmark comprises 1,006 tasks from 307 video episodes across six domains and five temporal modes, with reference trajectories constructed through automated annotation and human review. The authors evaluate nine streaming models and five asynchronous sliding-window models using metrics such as ETR, role-specific scores, and error exposure. Results show that the best models achieve ETR scores of only 19.88% and 29.03%, respectively, highlighting the challenges of maintaining reliable answers in dynamic video environments.
Critical strengths
-
Meaningful Benchmark Design. The paper extends conventional video question answering from pointwise correctness to sustained answer reliability. By explicitly evaluating deferring, committing, and updating, it addresses an important limitation of existing benchmarks.
-
Comprehensive Evaluation and Informative Analysis. The benchmark evaluates 14 models across multiple domains and temporal tasks using two inference protocols. Its metrics capture persistent errors and correction behavior, revealing important differences in models' ability to maintain reliable answers over time.
Critical weaknesses
- Temporal Change Recognition. The current evaluation measures answer correctness based on response completion time, which conflates temporal change recognition with inference and scheduling latency. Although Table 3 investigates sensitivity to timing tolerance, it does not distinguish whether delayed updates result from the model failing to recognize relevant visual changes or from the time required to generate responses. Consequently, the evaluation provides limited insight into the models' actual temporal change recognition capabilities.
- The notation a(S) in Equation (1) should be explicitly defined in the text for clarity.
Questions for authors
-
Could the authors evaluate models' sensitivity to state changes under the current evaluation protocol? For example, recording the timestamp of the latest video frame used for each response, along with its completion time, could help assess how promptly models recognize state changes and quantify their response latency.
-
Although the paper states that the model should defer until the target pitcher can be identified, could the authors further clarify why the available visual evidence is insufficient to support an earlier off_table answer in the pitcher example?
4 comments
Thank you again for your time and thoughtful feedback. With less than two days remaining in the discussion period, we would like to follow up on our responses to your comments.
If you have any remaining questions or concerns, please let us know—we would be happy to provide further clarification before the deadline.
We appreciate your consideration and look forward to hearing from you.
Thanks for the additional experiments and clarifications provided by the authors. I will keep my original score.
Thank you for your response and for taking the time to consider our clarifications. We appreciate your thoughtful feedback and careful consideration of our work.
1. Distinguishing Answer Correctness After State Changes from Response Latency
In streaming video, practical utility depends on both whether an answer is correct and whether it is available in time. Because the video continues to play while the model processes frames and generates text, the user receives a new answer only when generation finishes. We therefore update the public state at response completion time, so evaluation across defer, commit, and update reflects the actual answer available to the user.
To separate recognition of new visual evidence from response latency, we analyze answer accuracy on the first request that processes frames after a reference state change and the mean response time of that request for the five models below:
These results show the trade-off between post-change recognition accuracy and response speed. GLM-5.3-Flash achieves the highest accuracy on the first post-change request (75.81%) but has a longer mean response time (1.82s), whereas InternVideo3-8B-Instruct responds faster (0.53s) but has low accuracy after receiving the post-change frames (27.06%). Even when a request already includes frames after a state change, models often fail to output the updated answer. Thus, delayed updates cannot be explained by generation latency alone.
This analysis complements D-Score, C-Score, U-Score, and Mean Role Score (MRS). The role scores measure the correctness of the public state across defer, commit, and update segments, while the table above separates answer correctness upon receiving new visual evidence from the time needed to return that response to the user. We will add these results to the revised paper and explicitly state the newest input-frame timestamp, request submission time, and response completion time for each response.
2. Clarifying the Definition of a(S) in Equation (1)
We will add the following explicit definition immediately before Equation (1):
D-Score, C-Score, and U-Score then average a(S) across all reference segments of the corresponding role (defer, commit, and update), and the Mean Role Score (MRS) is the arithmetic mean of these three role scores.
3. Why Figure 1 Cannot Answer off_table Earlier
The key distinction is between "the target object is not in the visible area of the tabletop" and "the target object is not on this table." If the entire target tabletop were clearly visible and we could rule out the pitcher resting anywhere on it, a model could answer off_table even before seeing the pitcher directly.
In Figure 1, when the query is issued at 6 s, the camera covers only part of the wooden table, and the left and right edges of the tabletop extend beyond the frame. Although the current frame does not show the target pitcher, it cannot rule out that the pitcher is resting on an unviewed part of the same table. At this point, both on_table and off_table lack sufficient visual evidence, so the reference trajectory is defer.
At 15 s, the target pitcher with the orange lid and handle is held and brought into the foreground, providing direct visual evidence that it is off the table and supporting a commitment to off_table. At 21 s, its base rests on the tabletop, and the answer updates to on_table. The reason for waiting before 15 s is that the current field of view is insufficient to determine the support relation, not that the model must predict future changes. We will add this clarification to the caption of Figure 1.