acceptodds
Withdrawn from ICLR 2027

StreamPro: Timing-Aware Evaluation and Effective Training for Proactive Streaming Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Proactive streaming video understanding requires models to continuously monitor video streams and autonomously determine whether, when, and what to respond based on evolving user needs and visual context. Despite recent progress, both the evaluation and training of proactive models remain challenging. Existing benchmarks primarily focus on event-triggered responses, leaving anticipatory assistance underexplored, while their evaluation protocols often fail to jointly account for task-specific timing requirements, semantic correctness, and response frequency. Training also remains challenging: SFT suffers from severe imbalance between silence and response signals, whereas RL faces sparse response-level rewards and the challenge of optimizing interdependent response sequences. To address these challenges, we introduce StreamPro, a unified benchmark-and-training framework for proactive streaming video understanding. We first develop StreamPro-Bench, which evaluates three complementary dimensions of proactive capability: Perceptual Understanding, Temporal Reasoning, and Anticipatory Assistance. Its StreamPro-F1 metric jointly evaluates semantic correctness and response timing, while accounting for both excessive and missed responses. We further propose a two-stage training framework that combines class-balanced supervised fine-tuning with reinforcement learning. Specifically, CB-Stream Loss alleviates supervision imbalance during SFT, while GRPO with a refined Response-Matching (RM) reward and a rubric-based Sequence-Quality (SQ) reward provides denser feedback and encourages coherent response sequences. We construct StreamPro-SFT-63K and StreamPro-RL-3K to support the two training stages. Experiments show that StreamPro substantially improves proactive streaming video understanding: it achieves an average StreamPro-F1 of 32.8 on StreamPro-Bench, compared with 9.2 for the strongest existing baseline, while maintaining strong performance on real-time streaming video understanding, reaching 79.8 on StreamingBench-RTVU.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Proactive Streaming Video

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2KqrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper works towards enhancing the quality of response in proactive streaming video understanding, where the answer quality from a model depends on the whether, when and what is said, sometimes even before an event takes place. Understanding the gaps in the current benchmarks and the training methodologies to align with the requirement, the paper proposes a benchmark StreamPro-Bench covering a range of 7 tasks grouped based on their characteristics, a metric to better score the responses based on the quality requirement of the task and a two stage training framework for fine tuning the model to adapt to the silent-ness in the benchmarks & GRPO framework to enhance the overall response quality. The proposed model fine tuned with the proposed framework on the proposed dataset has shown clear model score gains on their proposed metric while staying competitive with the other benchmark methods.

Critical strengths

• The paper adds anticipatory assistance to the benchmark, a relevant task group addressing the clear requirement for proactive video understand while making the benchmark more clear and comprehensive. The Risk forecasting task is especially carefully designed with direct consultation with the visually impaired users while ensuring accuracy with clear human verification.

• The proposed metric removes the bias in the fields and ensures that the model acts as a well aligned optimization function focusing on correctness, timing and response frequency altogether.

• The proposed training methods has been clearly ablated and the design choices are mostly clear and it shows clear improvements in the proposed metric, while maintaining performance on offline benchmarks.

Critical weaknesses

• The performance boost seems to point fitting the model to the benchmark. It could be seen that the training and the test data comes from the same pipeline, and with the proposed model fine tuned on the provided dataset, and further reinforced on almost the same metric as the metric used for evaluation, this seems to show a clear bias in the reported results. This is further seen in the clear gain in performance on the benchmark compared to OVO FAR.

• The task group of anticipatory assistance being a core motivation of the paper, the performance improvement seems to attain a much smaller gain in comparison with a noticeable timing score drop after RL. This needs to be clearly analyzed and discussed.

• CB-stream loss is mostly an existing method (class-balanced loss of Cui et al., 2019) that was applied to the silent/response tokens, and the setting choice of /beta = 0.9999 makes the function unclear as to how much re-weighting actually happens (Especially with the missing on ablation on this hyper parameter). A clearer explanation on this loss, the reasoning and distinction would help the readers better appreciate the contribution.

Questions for authors

• How can the authors explain/ensure the performance gains reflect the proactive ability of the trained model, rather than fitting of the model to their own data/metric?

• Given the main motivation of paper is on anticipatory assistance, why does the performance gain seem sub-optimal with a clear timing performance drop after RL?

• How is the proposed CB-Stream loss different from the pre-existing class balancing loss and how does it’s sensitivity to the hyper-parameter affect the overall performance of the model during training?

Reviewer_vZrGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes StreamPro, a benchmark and training framework for proactive streaming video understanding. First, the authors construct StreamPro-Bench, which covers three dimensions: Perceptual Understanding, Temporal Reasoning, and Anticipatory Assistance, comprising a total of seven task categories. This paper also introduces the StreamPro-F1 metric to jointly evaluate the semantic correctness, temporal accuracy, and response frequency of responses. Second, in terms of model training, this paper adopts a two-stage training strategy. In the SFT stage, CB-Stream Loss is introduced to alleviate the class imbalance between silence and response signals. In the GRPO stage, Response-Matching (RM) Reward and Sequence-Quality (SQ) Reward are combined to optimize the accuracy of individual responses and the quality of the overall response sequence, respectively. Experimental results show that the proposed method achieves good performance on StreamPro-Bench.

Critical strengths
  1. Combining reinforcement learning with proactive streaming video understanding is a promising research direction.
Critical weaknesses
  1. The English writing is very poor, and the narrative logic is quite confusing. For example, you have not even clearly explained the relationships among StreamPro, StreamPro-Bench, and StreamPro-F1. There is also a StreamPro Training component in Figure 1.

  2. In (1) "Limited coverage of proactive behaviors" in the Introduction, the statement "a model may also need to reason about the temporal evolution of an event" is actually already a requirement of proactive responses. You claim that existing benchmarks are all event-triggered, but isn't reasoning about the temporal evolution of an event still based on event triggers? In Line 75, what does "anticipate upcoming assistance needs" mean in the statement "or anticipate upcoming assistance needs from the evolving visual context and respond before the relevant situation fully unfolds"? You have not even provided any textual or visual examples to explain this concept, making it very difficult to understand.

  3. In (2) "Incomplete evaluation of response behavior" in the Introduction, Line 79 states that "Existing metrics do not always capture these requirements jointly". However, the PO metric in StreamingBench and Online Evaluation in OmniStar have already taken these requirements into consideration. Moreover, although the statement "different proactive tasks may tolerate different response delays" in Line 82 is correct, it is unnecessary. Different delay tolerances across tasks are only relevant in specific real-world application scenarios and do not need to be considered when designing a benchmark. For proactive responses across different tasks, lower response latency is certainly always better. [1] StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding [2] LiveStar: Live Streaming Assistant for Real-World Online Video Understanding

  4. Based on the examples in Figure 2, what is the difference between your proposed "Anticipatory Assistance" and proactive output? Isn't it simply requiring the model to respond after observing the relevant information? Even for predictive responses, the PR evaluation metric in StreamingBench already serves a similar function.

  5. You have not compared the performance of the original backbone on any of the benchmarks. Instead, you only present the results after SFT and GRPO. How can you demonstrate that your method is effective, rather than the original backbone already having good performance?

  6. In Table 12 of the Appendix, you evaluate the performance on StreamingBench, but you omit the PO evaluation metric. Moreover, the PR metric is specifically designed to evaluate Temporal Reasoning capabilities, yet you mark the Temporal Reasoning capability of StreamingBench as missing in Table 1. Aren't you exaggerating your novelty by completely denying the contributions of existing work?

Questions for authors
  1. Novelty issues. The motivations (1) and (2) mentioned in your Introduction are both unconvincing, and you are also exaggerating your novelty by denying the contributions of other existing works.

  2. What are the differences between Anticipatory Assistance, proactive output, and the PR evaluation metric in StreamingBench?

  3. You have not compared the performance of the original backbone on any of the benchmarks, but only present the results after SFT and GRPO. How can you prove that your method is effective, rather than the original backbone already having good performance?

  4. Why do you only compare some of the evaluation metrics in your experiments on existing benchmarks? For example, you omit the PO metric on StreamingBench, even though this is also one of the core innovations of your proposed dataset.

Reviewer_8SowOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing benchmarks primarily focus on event-triggered responses, leaving anticipatory assistance underexplored, while their evaluation protocols often fail to jointly account for task-specific timing requirements, semantic correctness, and response frequency. Based on this, this paper proposed StreamPro, a unified benchmark-and-training framework for proactive streaming video understanding. Its StreamPro-F1 metric jointly evaluates semantic correctness and response timing, while accounting for both excessive and missed responses. This paper construct StreamPro-SFT-63K and StreamPro-RL-3K to support the two training stages. Experiments show that StreamPro substantially improves proactive streaming video understanding.

Critical strengths
  1. The paper is well written and the figures are nicely drawn. It is easy to follow.
Critical weaknesses
  1. On Risk Forecasting, the F1 of all models is close to 0, and even StreamPro-4B only reaches 6.3, which shows that anticipation is still far from solved.
  2. Most baselines (VideoLLM-Online, MMDuet) have never seen task formats and instructions like TG, PG, and RF, so it is expected that their F1 is close to 0.
  3. StreamPro-F1 uses a product-style joint score, which is very sensitive to noise in the LLM judge.
  4. The uniform τ=8 in the RM reward is inconsistent with the evaluation.
Questions for authors

Please see Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.