- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces StreamCast, a benchmark for future prediction in interactive livestreams. It pairs each prediction question with support questions covering perception localization, interaction dynamics, historical understanding, and causal attribution. The dataset contains 1421 questions from 470 livestreams across six categories. The paper evaluates 15 omni-modal models under Local and Full context settings and conducts modality comparisons, language analysis, human assessment, and capability-transfer experiments.
Critical strengths
- The anchor–support design is an interesting idea. Constructing support questions around the same prediction connects prediction performance to the model’s understanding of relevant evidence. This provides a more detailed basis for examining prediction errors and comparing model capabilities.
- The dataset organizes 470 livestreams across six categories into a practical benchmark for future prediction involving speech, visual events, and audience interactions.
Critical weaknesses
- The prediction answers need stronger validation. The solvable test accepts a question when only one of three text models answers correctly, while human prediction accuracy is just 40%. Quality ratings are collected after viewing the full video, leaving unclear whether the prefix sufficiently supports the intended answer. Please review human disagreements using only the prefix and identify questions with ambiguous options or multiple plausible outcomes.
- The Local–Full comparison also reflects sampling, compression, context limits, and output failures, so it does not isolate evidence-localization ability. Please revise this claim and report output failure rates.
Questions for authors
- Novelty is limited. No methodological contributions.
- The annotation details of the benchmarks, and the paper also lacks comprehensive human studies.
7 comments
Response to Reviewer 1EFJ [2/2]
For the Questions For Authors, we respond as follows:
- Novelty is limited. No methodological contributions.
We respectfully disagree with the assessment that our work lacks methodological contributions. While introducing a new evaluation task and benchmark is indeed a central contribution of our work, we believe that our additional methodological contributions may not have been fully recognized, which we would like to emphasize as follows:
First, we introduce a novel task of interaction-driven near-future prediction in livestreams, extending conventional retrospective video understanding to prospective reasoning over evolving multimodal events and streamer–audience interactions.
Secondly, we propose an structured anchor–support framework that evaluates future prediction alongside four prerequisite capabilities (PL, HU, ID, and CA), enabling fine-grained diagnosis of prediction failures rather than merely reporting overall accuracy.
Additionally, we also develop a scalable construction pipeline featuring hierarchical multimodal representations, interaction-aware highlight selection, anchor-first task generation, and iterative quality verification through blind and solvable Tests.
Together, these contributions go beyond the mere introduction of a new dataset. We believe they represent substantive methodological advances in benchmark construction and diagnostic evaluation, establishing a novel and systematic framework for assessing interaction-driven near-future prediction in livestreams.
- The annotation details of the benchmarks, and the paper also lacks comprehensive human studies.
We provide detailed annotation protocols in Section 3 and the Appendix, including the specific data sources, annotation pipeline and algorithms, annotator agents, and their corresponding configurations and parameters. Furthermore, Section 5.1 and Appendix E present a comprehensive human evaluation of the generated data quality, reporting both quality ratings and inter-annotator consistency.
We would also like to emphasize that the purpose of our human evaluation is not to regenerate the answers for the entire dataset, but rather to independently assess the quality of the automatically constructed questions. This provides a complementary quality assessment that is independent of the automated construction pipeline. Nevertheless, due to budget constraints, our human evaluation was limited to a relatively small sampled subset. We acknowledge this limitation and will consider expanding the scale of human validation in future work, as also discussed in our Limitations section.
Comments after reading the rebuttal letter
Thank you for your detailed response, which partially addresses my concerns. I have two brief follow-up questions.
-
Could you share how many of the 20 prediction questions the annotators disagreed on prior to reviewing the reference evidence? Additionally, providing one or two examples illustrating which cues in the video prefix favor the intended answer over the most plausible alternative would be very helpful.
-
While I appreciate the design of the anchor–support framework and the data construction pipeline, these are primarily contributions to benchmark design. My original concern was the lack of a dedicated modeling approach. Beyond evaluating existing MLLMs and the auxiliary-task fine-tuning in Sec. 5.2, did you test an MLLM prediction pipeline specifically designed for StreamCast?
If these remaining concerns can be addressed, I would be happy to raise my score.
Response to Reviewer 1EFJ [1/2]
Thank you for your valuable feedback and constructive comments, which help us further clarify and strengthen our work. For your Critical Weaknesses, we respond as follows:
1.Regarding the relatively low human prediction accuracy, we would like to highlight two aspects. First, human prediction was conducted as an auxiliary analysis of task difficulty, rather than as a procedure for generating or determining gold labels. Therefore, annotators were not required to repeatedly review the videos until they ensure a ground truth. Second, the evidence support criterion in Section 5.1 provides a complementary assessment of question validity by evaluating the reference explanations.
Furthermore, we would like to clarify that the quality ratings were not collected after viewing the full video (including L>t). Instead, human annotators first watched the same L<t video input as the evaluated models and independently answered the questions. Only after completing the prediction task did they proceed to assess the question quality. Therefore, the findings from the human evaluation, including evidence support and unambiguity, were not affected by information leakage from future video content. We have fixed the expression in Section 5.1 that might lead to misunderstanding in the revision.
Regarding human disagreements, we acknowledge that some annotators reported difficulty distinguishing between certain options during the initial prediction task. However, after reviewing the reference evidence, they generally agreed that the evidence adequately supported the gold answers and that the questions had a single correct answer, as reflected in the high ratings for evidence support and unambiguity. This suggests that the difficulty encountered during prediction does not necessarily indicate ambiguity or insufficient evidence in the questions.
2.Thank you for your suggestion. For models with strong long-context processing capabilities that do not require additional frame sampling, the Local-to-Full comparison can indeed help isolate their evidence-localization ability. However, for smaller models with limited context windows, the comparison also reflects their overall ability to preserve, process, and utilize relevant evidence under long-context input constraints.
We acknowledge that using the term isolates to characterize this comparison across all models was overly absolute. We have refined the wording in the revised manuscript to better reflect these distinctions and avoid potential misunderstandings (with changes colored in blue).
We additionally report the invalid output rates under the Local and Full settings in the following table. These rates represent the proportion of evaluation instances that produced error or unparseable outputs. We report models exhibiting non-negligible invalid output rates (above 1% in at least one setting). During evaluation, invalid outputs were generally retried up to three times to mitigate occasional inference failures.
As shown above, the invalid output rates are substantially higher in the Full setting than in the Local setting. This suggests that these models have limited capabilities in handling long inputs, which may also negatively affect their instruction-following performance.