- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper works towards enhancing the quality of response in proactive streaming video understanding, where the answer quality from a model depends on the whether, when and what is said, sometimes even before an event takes place. Understanding the gaps in the current benchmarks and the training methodologies to align with the requirement, the paper proposes a benchmark StreamPro-Bench covering a range of 7 tasks grouped based on their characteristics, a metric to better score the responses based on the quality requirement of the task and a two stage training framework for fine tuning the model to adapt to the silent-ness in the benchmarks & GRPO framework to enhance the overall response quality. The proposed model fine tuned with the proposed framework on the proposed dataset has shown clear model score gains on their proposed metric while staying competitive with the other benchmark methods.
Critical strengths
• The paper adds anticipatory assistance to the benchmark, a relevant task group addressing the clear requirement for proactive video understand while making the benchmark more clear and comprehensive. The Risk forecasting task is especially carefully designed with direct consultation with the visually impaired users while ensuring accuracy with clear human verification.
• The proposed metric removes the bias in the fields and ensures that the model acts as a well aligned optimization function focusing on correctness, timing and response frequency altogether.
• The proposed training methods has been clearly ablated and the design choices are mostly clear and it shows clear improvements in the proposed metric, while maintaining performance on offline benchmarks.
Critical weaknesses
• The performance boost seems to point fitting the model to the benchmark. It could be seen that the training and the test data comes from the same pipeline, and with the proposed model fine tuned on the provided dataset, and further reinforced on almost the same metric as the metric used for evaluation, this seems to show a clear bias in the reported results. This is further seen in the clear gain in performance on the benchmark compared to OVO FAR.
• The task group of anticipatory assistance being a core motivation of the paper, the performance improvement seems to attain a much smaller gain in comparison with a noticeable timing score drop after RL. This needs to be clearly analyzed and discussed.
• CB-stream loss is mostly an existing method (class-balanced loss of Cui et al., 2019) that was applied to the silent/response tokens, and the setting choice of /beta = 0.9999 makes the function unclear as to how much re-weighting actually happens (Especially with the missing on ablation on this hyper parameter). A clearer explanation on this loss, the reasoning and distinction would help the readers better appreciate the contribution.
Questions for authors
• How can the authors explain/ensure the performance gains reflect the proactive ability of the trained model, rather than fitting of the model to their own data/metric?
• Given the main motivation of paper is on anticipatory assistance, why does the performance gain seem sub-optimal with a clear timing performance drop after RL?
• How is the proposed CB-Stream loss different from the pre-existing class balancing loss and how does it’s sensitivity to the hyper-parameter affect the overall performance of the model during training?