acceptodds
Under review as a conference paper at ICLR 2027

StreamCast: Benchmarking MLLMs for Future Prediction in Interactive Livestreams

Abstract

Livestreaming is a real-time and interactive form of video, where events evolve continuously through visual, audio, and audience signals. Understanding livestreams requires MLLMs to move beyond retrospective comprehension and anticipate near-future outcomes from evolving context, a capability underexplored by existing benchmarks. In this paper, we introduce **StreamCast**, a benchmark for evaluating near-future prediction in livestreams. Each video contains an anchor question targeting at the prediction of the immediate future, with support questions probing capabilities relevant to the prediction: *perception localization*, *interaction dynamics*, *historical understanding*, and *causal attribution*. Constructed through hierarchical multimodal representation, agentic task formation, and automatic verification, StreamCast comprises 470 real livestreams across six categories and 1,421 questions. We further assess question quality and evidence support through human evaluation of a sampled subset. We evaluate 15 open-source and proprietary MLLMs and analyze performance across diagnostic dimensions, modalities, and languages. The strongest model reaches 63.19% accuracy on future prediction in the Full-context setting while achieving 85.07-90.67% across the four diagnostic dimensions, highlighting the difficulty of anticipating future developments even when prerequisite capabilities are relatively strong. We release our code and dataset at https://anonymous.4open.science/r/StreamCast.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Video Understanding, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_1EFJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces StreamCast, a benchmark for future prediction in interactive livestreams. It pairs each prediction question with support questions covering perception localization, interaction dynamics, historical understanding, and causal attribution. The dataset contains 1421 questions from 470 livestreams across six categories. The paper evaluates 15 omni-modal models under Local and Full context settings and conducts modality comparisons, language analysis, human assessment, and capability-transfer experiments.

Critical strengths
  1. The anchor–support design is an interesting idea. Constructing support questions around the same prediction connects prediction performance to the model’s understanding of relevant evidence. This provides a more detailed basis for examining prediction errors and comparing model capabilities.
  2. The dataset organizes 470 livestreams across six categories into a practical benchmark for future prediction involving speech, visual events, and audience interactions.
Critical weaknesses
  1. The prediction answers need stronger validation. The solvable test accepts a question when only one of three text models answers correctly, while human prediction accuracy is just 40%. Quality ratings are collected after viewing the full video, leaving unclear whether the prefix sufficiently supports the intended answer. Please review human disagreements using only the prefix and identify questions with ambiguous options or multiple plausible outcomes.
  2. The Local–Full comparison also reflects sampling, compression, context limits, and output failures, so it does not isolate evidence-localization ability. Please revise this claim and report output failure rates.
Questions for authors
  1. Novelty is limited. No methodological contributions.
  2. The annotation details of the benchmarks, and the paper also lacks comprehensive human studies.
7 comments
AuthorsOpenReview ↗

Response to Reviewer 1EFJ [1/2]

Thank you for your valuable feedback and constructive comments, which help us further clarify and strengthen our work. For your Critical Weaknesses, we respond as follows:

  1. The prediction answers need stronger validation. The solvable test accepts a question when only one of three text models answers correctly, while human prediction accuracy is just 40%. Quality ratings are collected after viewing the full video, leaving unclear whether the prefix sufficiently supports the intended answer. Please review human disagreements using only the prefix and identify questions with ambiguous options or multiple plausible outcomes.

1.Regarding the relatively low human prediction accuracy, we would like to highlight two aspects. First, human prediction was conducted as an auxiliary analysis of task difficulty, rather than as a procedure for generating or determining gold labels. Therefore, annotators were not required to repeatedly review the videos until they ensure a ground truth. Second, the evidence support criterion in Section 5.1 provides a complementary assessment of question validity by evaluating the reference explanations.

Furthermore, we would like to clarify that the quality ratings were not collected after viewing the full video (including L>t). Instead, human annotators first watched the same L<t video input as the evaluated models and independently answered the questions. Only after completing the prediction task did they proceed to assess the question quality. Therefore, the findings from the human evaluation, including evidence support and unambiguity, were not affected by information leakage from future video content. We have fixed the expression in Section 5.1 that might lead to misunderstanding in the revision.

Regarding human disagreements, we acknowledge that some annotators reported difficulty distinguishing between certain options during the initial prediction task. However, after reviewing the reference evidence, they generally agreed that the evidence adequately supported the gold answers and that the questions had a single correct answer, as reflected in the high ratings for evidence support and unambiguity. This suggests that the difficulty encountered during prediction does not necessarily indicate ambiguity or insufficient evidence in the questions.

  1. The Local–Full comparison also reflects sampling, compression, context limits, and output failures, so it does not isolate evidence-localization ability. Please revise this claim and report output failure rates.

2.Thank you for your suggestion. For models with strong long-context processing capabilities that do not require additional frame sampling, the Local-to-Full comparison can indeed help isolate their evidence-localization ability. However, for smaller models with limited context windows, the comparison also reflects their overall ability to preserve, process, and utilize relevant evidence under long-context input constraints.

We acknowledge that using the term isolates to characterize this comparison across all models was overly absolute. We have refined the wording in the revised manuscript to better reflect these distinctions and avoid potential misunderstandings (with changes colored in blue).

We additionally report the invalid output rates under the Local and Full settings in the following table. These rates represent the proportion of evaluation instances that produced error or unparseable outputs. We report models exhibiting non-negligible invalid output rates (above 1% in at least one setting). During evaluation, invalid outputs were generally retried up to three times to mitigate occasional inference failures.

ModelLocal invalid rateFull invalid rate
Qwen3.5-Omni-Flash13/1421(0.91%)946/1421(66.57%)
MiniCPM-o-4.536/1421(2.53%)108/1421(7.60%)
VITA-1.5583/1421(41.03%)1415/1421(99.58%)

As shown above, the invalid output rates are substantially higher in the Full setting than in the Local setting. This suggests that these models have limited capabilities in handling long inputs, which may also negatively affect their instruction-following performance.

AuthorsOpenReview ↗

Response to Reviewer 1EFJ [2/2]

For the Questions For Authors, we respond as follows:

  1. Novelty is limited. No methodological contributions.

We respectfully disagree with the assessment that our work lacks methodological contributions. While introducing a new evaluation task and benchmark is indeed a central contribution of our work, we believe that our additional methodological contributions may not have been fully recognized, which we would like to emphasize as follows:

First, we introduce a novel task of interaction-driven near-future prediction in livestreams, extending conventional retrospective video understanding to prospective reasoning over evolving multimodal events and streamer–audience interactions.

Secondly, we propose an structured anchor–support framework that evaluates future prediction alongside four prerequisite capabilities (PL, HU, ID, and CA), enabling fine-grained diagnosis of prediction failures rather than merely reporting overall accuracy.

Additionally, we also develop a scalable construction pipeline featuring hierarchical multimodal representations, interaction-aware highlight selection, anchor-first task generation, and iterative quality verification through blind and solvable Tests.

Together, these contributions go beyond the mere introduction of a new dataset. We believe they represent substantive methodological advances in benchmark construction and diagnostic evaluation, establishing a novel and systematic framework for assessing interaction-driven near-future prediction in livestreams.

  1. The annotation details of the benchmarks, and the paper also lacks comprehensive human studies.

We provide detailed annotation protocols in Section 3 and the Appendix, including the specific data sources, annotation pipeline and algorithms, annotator agents, and their corresponding configurations and parameters. Furthermore, Section 5.1 and Appendix E present a comprehensive human evaluation of the generated data quality, reporting both quality ratings and inter-annotator consistency.

We would also like to emphasize that the purpose of our human evaluation is not to regenerate the answers for the entire dataset, but rather to independently assess the quality of the automatically constructed questions. This provides a complementary quality assessment that is independent of the automated construction pipeline. Nevertheless, due to budget constraints, our human evaluation was limited to a relatively small sampled subset. We acknowledge this limitation and will consider expanding the scale of human validation in future work, as also discussed in our Limitations section.

Reviewer_1EFJOpenReview ↗

Comments after reading the rebuttal letter

Thank you for your detailed response, which partially addresses my concerns. I have two brief follow-up questions.

  1. Could you share how many of the 20 prediction questions the annotators disagreed on prior to reviewing the reference evidence? Additionally, providing one or two examples illustrating which cues in the video prefix favor the intended answer over the most plausible alternative would be very helpful.

  2. While I appreciate the design of the anchor–support framework and the data construction pipeline, these are primarily contributions to benchmark design. My original concern was the lack of a dedicated modeling approach. Beyond evaluating existing MLLMs and the auxiliary-task fine-tuning in Sec. 5.2, did you test an MLLM prediction pipeline specifically designed for StreamCast?

If these remaining concerns can be addressed, I would be happy to raise my score.

Reviewer_Wv2SOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StreamCast, a benchmark for evaluating multimodal large language models (MLLMs) on near-future prediction in interactive live-streaming scenarios. Unlike traditional video understanding tasks, which focus on recognizing and reasoning about events that have already occurred, this work evaluates whether a model can use the visual, audio, and audience-interaction information available during a live stream to predict what may happen next. Specifically, StreamCast adopts an Anchor–Support structure. Anchor Questions evaluate the model's near-future prediction ability, while four types of Support Questions—perception localization, interaction dynamics understanding, historical understanding, and causal attribution—probe the underlying abilities related to prediction. The dataset contains 470 real live-stream videos from six categories, with 1,421 multiple-choice questions in total. The authors evaluate 15 open-source and closed-source multimodal models, and conduct modality ablation, language alignment, capability transfer, and human quality evaluation experiments.

Critical strengths

1.The research problem is practically meaningful, and the evaluation is fairly comprehensive.

This paper focuses on near-future event prediction in interactive live-streaming scenarios. It extends video understanding from recognizing past events to judging how events may develop, which has practical application value. In addition, the paper evaluates a range of mainstream multimodal models and analyzes their performance under different experimental settings, so the evaluation covers a broad scope.

2.The Anchor–Support design provides finer-grained diagnostic insights.

Through the Anchor–Support structure, the paper links near-future prediction with four underlying abilities: perception localization, interaction dynamics understanding, historical understanding, and causal attribution. Compared with reporting only final prediction accuracy, this design analyzes model performance from multiple dimensions. It also provides finer-grained diagnostic information for analyzing potential sources of prediction errors.

Critical weaknesses

1.The robustness of the candidate answer set still needs further analysis.

This paper evaluates near-future prediction with multiple-choice questions that use fixed candidate answers. In Section 3.4, the authors already require candidate answers to belong to the same decision space, and they use Blind Test and Solvable Test to verify that questions are context-dependent and answerable. These quality controls are commendable. However, the sensitivity of the current results to how candidate answers are constructed is still not analyzed. If the video, prediction question, and correct answer remain unchanged, but the distractors are replaced with comparably plausible alternatives that pass the same quality controls, it is unclear whether model accuracy or model rankings would show substantial variations. If such variations do occur, especially if rankings change substantially, then there is reason to question the stability of this benchmark as a tool for comparing models. Therefore, I suggest adding a candidate-set robustness analysis.

2.The performance difference between Local and Full-context cannot be directly attributed to evidence-localization ability.

This paper uses Local and Full-context settings to separate two abilities: predicting after relevant evidence is given, and retrieving and using relevant evidence in long videos. In Section 4.1, the authors make a strong claim that the Local-to-Full difference isolates a model's evidence-localization ability. However, when switching from Local to Full-context, more than the input context range changes. Video frame sampling density, audio-visual compression level, and model input processing may also change (Appendix A also notes that different models adjust frame counts and compression based on input capacity). In addition, the authors report in Section 4.2 that some models show clear output failures in the Full-context setting. For example, the future prediction accuracy of Qwen3.5-Omni-Flash drops from 45.32% to 14.04%, and this change is related to frequent empty outputs when processing long videos. Although the authors acknowledge these limitations in Appendix G, the current Local-to-Full comparison does not sufficiently isolate these confounding factors. Therefore, based only on the accuracy difference between Local and Full-context, it is difficult to attribute the performance difference directly to the model's long-video evidence-localization ability.

Questions for authors

Question 1: On the robustness of the candidate answer set.

With the same video, prediction question, and correct answer, can the authors replace the distractors with comparably plausible ones that pass the same quality controls? They could then compare the change in accuracy and the consistency of model rankings across candidate sets.

Question 2: On the attribution of the Local and Full-context results.

Can the authors report valid output rates and inference failure rates for the Local and Full-context settings? Can they also test, on a subset of representative models, how evidence-preserving sampling affects Full-context performance under the same frame budget?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Wv2S [1/2]

Thank you for your thoughtful review and valuable suggestions. For your Critical Weaknesses, we respond as follows:

1.The robustness of the candidate answer set still needs further analysis. This paper evaluates near-future prediction with multiple-choice questions that use fixed candidate answers. In Section 3.4, the authors already require candidate answers to belong to the same decision space, and they use Blind Test and Solvable Test to verify that questions are context-dependent and answerable. These quality controls are commendable. However, the sensitivity of the current results to how candidate answers are constructed is still not analyzed. If the video, prediction question, and correct answer remain unchanged, but the distractors are replaced with comparably plausible alternatives that pass the same quality controls, it is unclear whether model accuracy or model rankings would show substantial variations. If such variations do occur, especially if rankings change substantially, then there is reason to question the stability of this benchmark as a tool for comparing models. Therefore, I suggest adding a candidate-set robustness analysis.

1.Thank you for your suggestion. Regarding the concern of robustness of the candidate answer set, we respond as follows:

We would like to first clarify the scope of our work. The primary contribution of StreamCast is to construct and validate a reliable benchmark for near-future prediction in interactive livestreams, rather than to develop an evaluation methodology that remains stable under arbitrary perturbations of candidate answer sets.

We argue that candidate-set robustness does not necessarily determine the reliability or quality of our benchmark, and is not the primary focus of this work. Nevertheless, we acknowledge that a systematic investigation of candidate-set robustness would be a meaningful direction for future research.

2.The performance difference between Local and Full-context cannot be directly attributed to evidence-localization ability. This paper uses Local and Full-context settings to separate two abilities: predicting after relevant evidence is given, and retrieving and using relevant evidence in long videos. In Section 4.1, the authors make a strong claim that the Local-to-Full difference isolates a model's evidence-localization ability. However, when switching from Local to Full-context, more than the input context range changes. Video frame sampling density, audio-visual compression level, and model input processing may also change (Appendix A also notes that different models adjust frame counts and compression based on input capacity). In addition, the authors report in Section 4.2 that some models show clear output failures in the Full-context setting. For example, the future prediction accuracy of Qwen3.5-Omni-Flash drops from 45.32% to 14.04%, and this change is related to frequent empty outputs when processing long videos. Although the authors acknowledge these limitations in Appendix G, the current Local-to-Full comparison does not sufficiently isolate these confounding factors. Therefore, based only on the accuracy difference between Local and Full-context, it is difficult to attribute the performance difference directly to the model's long-video evidence-localization ability.

2.Regarding the interpretation of evidence localization in the Local and Full settings:

For models with strong long-context processing capabilities that do not require additional frame sampling, the Local-to-Full comparison can indeed help isolate their evidence-localization ability. However, for smaller models with limited context windows, the comparison also reflects their overall ability to preserve, process, and utilize relevant evidence under long-context input constraints. We acknowledge that using the term "isolates" to characterize this comparison across all models was overly absolute. We have refined the wording in the revised manuscript to better reflect these distinctions and avoid potential misunderstandings (changes colored in blue).

AuthorsOpenReview ↗

Response to Reviewer Wv2S [2/2]

For the Questions For Authors, we respond as follows:

Question 1: On the robustness of the candidate answer set. With the same video, prediction question, and correct answer, can the authors replace the distractors with comparably plausible ones that pass the same quality controls? They could then compare the change in accuracy and the consistency of model rankings across candidate sets.

RQ1. Regarding the robustness of the candidate answer set, as discussed above, candidate-set robustness is not the primary focus of this work. However, we would also like to clarify that the candidate answer set is an integral part of each evaluation instance, rather than an independent variable. Replacing distractors, even when the alternatives satisfy the same quality-control criteria, may change the question's semantic ambiguity and reasoning difficulty. In particular, the blind and solvable tests establish consistent quality standards for generated questions by assessing their context dependency and answerability, rather than enforcing identical difficulty across different candidate sets. Therefore, even if performance variations were observed in such an experiment, they would not necessarily indicate that the benchmark itself is unreliable or lacks robustness.

Question 2: On the attribution of the Local and Full-context results. Can the authors report valid output rates and inference failure rates for the Local and Full-context settings? Can they also test, on a subset of representative models, how evidence-preserving sampling affects Full-context performance under the same frame budget?

RQ2. (1) Regarding the attribution of the Local and Full results, we report the invalid output rates under the Local and Full settings in the following table. These rates represent the proportion of evaluation instances that produced error or unparseable outputs. We report models exhibiting non-negligible invalid output rates (above 1% in at least one setting). During evaluation, invalid outputs were generally retried up to three times to mitigate occasional inference failures.

ModelLocal invalid rateFull invalid rate
Qwen3.5-Omni-Flash13/1421(0.91%)946/1421(66.57%)
MiniCPM-o-4.536/1421(2.53%)108/1421(7.60%)
VITA-1.5583/1421(41.03%)1415/1421(99.58%)
As shown above, the invalid output rates are substantially higher in the Full setting than in the Local setting. This suggests that these models have limited capabilities in handling long inputs, which may also negatively affect their instruction-following performance.

(2) To further investigate the impact of frame sampling, we conducted an additional experiment with two representative local models under the same frame budget as the Full setting, providing a more controlled comparison. The following table presents the results on a randomly sampled subset of 200 questions. The original Local setting uses a sampling rate of 1 fps, resulting in an average of 59.1 frames per video.

ModelFull Acc.Original Local Acc.Local Acc. under same fps budgetAvg. Local input under same fps budget (frame/video )
Qwen2.5-Omni-7B46.0%(92/200)55.5%(111/200)52.5%(105/200)16.24
Phi-4-Multimodal39.0%(78/200)42.5%(85/200)47.0%(94/200)4.96

As shown above, under the same frame budget on the 200 sampled questions, the Local accuracy of Qwen2.5-Omni-7B decreases by 3 percentage points, while that of Phi-4-Multimodal increases by 4.5 percentage points.

We hypothesize that sampling more frames introduces not only additional information but also more irrelevant or distracting content. Since different models may vary in their robustness to such noise, the impact of frame sampling can differ across models. This may also result in behaviors such as generating lengthy, unparseable responses or irrelevant video descriptions.

Thank you for your rebuttal. You addressed my concerns and I keep my rating as postive.

Dear Reviewer Wv2S,

Thank you very much for your kind words and positive feedback. We sincerely appreciate your time in reviewing our responses and your continued support of our work. Your constructive suggestions have been valuable in helping us improve the paper.

Best,

Authors

Reviewer_gjt6OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StreamCast is a benchmark for near-future outcome selection in Chinese interactive livestreams from Bilibili. It contains 470 streams and 1,421 four-option questions. Each stream has one anchor question asking what happens after a prediction point t, and 2 to 4 support questions on perception localization, interaction dynamics, historical understanding and causal attribution. Questions are written by an agent over time-aligned textual descriptions of video, speech and audience actions, and filtered by a text-only blind test and a text-evidence solvable test. The paper evaluates 15 omni-modal models under a Local window and a Full-prefix setting, adds vision-only and audio-only models and Chinese versus English prompts, reports a 20-stream human assessment, and fine-tunes Qwen2.5-Omni-7B on public proxy datasets to study transfer from support capabilities to anchor accuracy.

Critical strengths
  1. The anchor-support organization gives a usable handle on why a model fails a prediction. Sections 3.1 and 3.4 attach every anchor to questions about the evidence behind it, and Appendix D (Table 6) shows anchor accuracy rising monotonically from 31.1% when no matched support is answered correctly to 45.2% when all are. This is the paper's most reusable idea, because it turns a single forecasting score into something a developer can act on. The paper correctly stops short of a causal reading of that table.

  2. The reporting is candid enough that readers can locate the weak points themselves. Failures are shown rather than filtered (VITA-1.5 unparsable outputs and Qwen3.5-Omni-Flash empty outputs, Section 4.2), per-model frame caps are listed in Appendix A, and Appendix G states plainly that the label is the outcome that actually occurred and that prediction points were chosen after seeing the full stream. This matters for the assessment because my main concerns below are ones the paper's own appendices make visible.

Critical weaknesses
  1. The claim that the prefix supports a unique answer is asserted by the construction pipeline, but the available evidence does not establish it and partly points the other way. Sections 3.1 and 3.4 require that L<t "support a unique target answer" and say the solvable test guarantees this. Section 3.4 describes the solvable test as giving text-only models the evidence note e_anchor, which is supplied by the question-writing agent that already knows which outcome occurred, whereas Appendix B describes the input as the subtitles, audience actions and omni-modal descriptions within the question's time range (the two descriptions should be reconciled). Appendix B states that a question passes if at least one of the three models answers correctly (Section 3.4 does not give the pass criterion). Because the models may answer "not enough information" in this test, a model that cannot tell the options apart is not forced to guess, so I do not read the pass rate as chance level. The concern is the aggregation: under an "at least one of three" rule an item passes as soon as any single model commits to the gold option, including when the text supports two or more continuations and the models split between them (if two options were equally supported and each model picked one at random, the item would still pass with probability 0.875). How often this happens depends on how often models abstain on ambiguous items, which the paper does not report. The rule therefore shows that the gold option is reachable from the text, not that the alternatives are ruled out, which is what Sections 3.1 and 3.4 say it guarantees. The blind test retains a question unless at least two of the three models answer it correctly without context, so a question that each model answers blind with 60% accuracy is still retained about 35% of the time (0.4³ + 3·0.6·0.4²); this lets through questions with substantial options-only signal, and neither test's accuracy on the final anchors is reported. The human assessment in Section 5.1 (in the paragraph following Table 4) raises a related question. Annotators, who answer "based on the livestream video (L<t), matching the information accessible to evaluated models", are correct on 40.0% of anchors against 82.7% of supports. The paper reads this as "consistent with the anchor–support gap observed in model evaluations", but a 40.0% human accuracy on anchors is also hard to reconcile with the claim that the prefix supports a unique answer. The figure comes from 20 sampled anchors and the paper does not report model accuracy on the same items, so it is unclear whether models exceed humans there. If they do (several models are above 40.0 on the full anchor set, e.g., Gemini-3.1-Pro-Preview at 62.77 in Table 1, Future Prediction, Loc. column), that would point either to anchors that the prefix does not determine, or to models relying on cues other than the stream, such as how the realized option and its distractors are written (Appendix B describes distractors as interpretations "present in the context but incomplete or temporally mismatched"). Either case deserves discussion. The quality ratings in Table 4 do not settle this, since they are given after the annotators saw the full stream and intermediate annotations, by the authors according to the Ethics Statement. This determines whether the headline 63.19% is forecasting skill. Three checks use existing data: report options-only accuracy (no video, no audio) of the strongest evaluated models on the final anchor and support questions, report the evaluated models' accuracy on the 20 audited anchors, and adjudicate those anchors item by item, stating for each human error whether the chosen option is defensible from the prefix alone.

  2. The diagnostic readings of the Local, Full and modality comparisons are stronger than the controls allow. Section 4.1 states that the Local-to-Full difference "isolates a model's evidence-localization ability", but Appendix A shows the input changes in other ways as the context grows: MiniCPM-o-4.5 drops from 128 to 77 frames in Full, Baichuan-Omni-1.5 is capped at 8 frames, and HumanOmni compresses the entire waveform into a 30-second window. Qwen3.5-Omni-Flash falls to 14.04 on prediction in Full (Table 1, Future Prediction, Full column), below the 25% chance level of a four-option question, because of empty outputs. The required evidence can therefore become less observable independently of retrieval difficulty. Section 4.3 draws conclusions about modality contribution (for example that audio-only GPT-Audio beats vision-only GPT-5, and that Qwen2.5-Omni beats both Qwen2.5-VL and Qwen2-Audio) by comparing different checkpoints, and the vision-only model receives no speech at all in streams where the content is carried by speech. These are informative system comparisons but not isolation of a factor. Two models already in the roster, Qwen2.5-Omni-7B and MiniCPM-o-4.5, accept both streams: running the Local setting with audio muted or video removed, and a Full setting that keeps the Local evidence frames and audio at native fidelity inside the longer context, on a modest subset would test these readings directly. Otherwise the text should describe the results as performance under the stated input pipelines.

Questions for authors
  1. What is the accuracy of the three blind-test models, and of the strongest evaluated models run without any video or audio, on the final anchor and support questions? If options-only accuracy on anchors is well above 25%, how should the 63.19% be read? In the solvable test, how often did the models answer "not enough information", and among the retained anchors, how many were solved by exactly one, two or all three models?
  2. For the 20 audited anchors, in how many of the human errors is the chosen option defensible from the prefix alone? Please also state whether the five annotators in Section 5.1 are the authors (the Ethics Statement says all human evaluation was done by the authors), and give the prediction horizon and decisive pre-cutoff cues for the Figure 4 gaming example, where the streamer says "goodnight" yet the gold answer is to continue the stream. Did the annotators see the Local window or the full prefix, and how were the five annotators' answers combined into the 40.0% and 82.7% figures? Could the authors also report the accuracy of the evaluated models on the same 20 anchors used in the human study, so that human and model performance can be compared on identical items?
  3. On a subset, how much of the Local-to-Full drop remains when the evidence frames and audio fidelity are preserved in the Full input, and how much of the audio-versus-vision gap remains when one omni checkpoint is run with a modality removed?
7 comments
AuthorsOpenReview ↗

Response to Reviewer gjt6 [1/5]

Thank you for your thorough and constructive review, as well as your insightful comments and suggestions. For your Critical Weaknesses, we respond as follows:

  1. (1) The claim that the prefix supports a unique answer is asserted by the construction pipeline, but the available evidence does not establish it and partly points the other way. Sections 3.1 and 3.4 require that L<t "support a unique target answer" and say the solvable test guarantees this. Section 3.4 describes the solvable test as giving text-only models the evidence note e_anchor, which is supplied by the question-writing agent that already knows which outcome occurred, whereas Appendix B describes the input as the subtitles, audience actions and omni-modal descriptions within the question's time range (the two descriptions should be reconciled). Appendix B states that a question passes if at least one of the three models answers correctly (Section 3.4 does not give the pass criterion). Because the models may answer "not enough information" in this test, a model that cannot tell the options apart is not forced to guess, so I do not read the pass rate as chance level. The concern is the aggregation: under an "at least one of three" rule an item passes as soon as any single model commits to the gold option, including when the text supports two or more continuations and the models split between them (if two options were equally supported and each model picked one at random, the item would still pass with probability 0.875). How often this happens depends on how often models abstain on ambiguous items, which the paper does not report. The rule therefore shows that the gold option is reachable from the text, not that the alternatives are ruled out, which is what Sections 3.1 and 3.4 say it guarantees. The blind test retains a question unless at least two of the three models answer it correctly without context, so a question that each model answers blind with 60% accuracy is still retained about 35% of the time (0.4³ + 3·0.6·0.4²); this lets through questions with substantial options-only signal, and neither test's accuracy on the final anchors is reported.

1.(1) Regarding the concern about the uniqueness of the correct answer, we acknowledge that the solvable test primarily ensures that identifying the target answer depends on the context in L<t, while the blind test aims to verify that the answer cannot be easily inferred through commonsense reasoning alone.

However, in our task construction pipeline, the annotator agent is explicitly instructed to construct questions, correct answers, and distractors based on the given context (Line 813-814). This design makes the distractors plausible while ensuring that the correct answer is most consistent with the full context and its temporal progression. Blind test reduces the likelihood that it can be selected without contextual understanding, and the subsequent solvable test then provide complementary checks to assess the dependency of the target answer on the context in L<t, thereby narrowing down the plausible options as much as possible. We aim to ensure each question a single best-supported answer, rather than guarantee that all alternative continuations are logically impossible.

Still, we agree that our original use of the term unique was somewhat absolute. Our intention was to indicate that each question has only one designated correct answer among the four options. We have revised this wording in the updated manuscript to avoid overstating this property (changes colored in blue).

Regarding the inconsistency in the descriptions of the solvable test inputs, we would like to clarify that eanchore_{anchor}, provided by the annotator agent to the test-group models in Section 3.4 (Line 256), refers to the "subtitles, audience actions, and omni-modal descriptions within the question's time range" described in Appendix B (Line 822). We will reconcile the descriptions in these two sections and Algorithm 1 to avoid confusion.

Regarding the pass rate of the blind test, please refer to our response to Questions for Authors below.

AuthorsOpenReview ↗

Response to Reviewer gjt6 [2/5]

  1. (2) The human assessment in Section 5.1 (in the paragraph following Table 4) raises a related question. Annotators, who answer "based on the livestream video (L<t), matching the information accessible to evaluated models", are correct on 40.0% of anchors against 82.7% of supports. The paper reads this as "consistent with the anchor–support gap observed in model evaluations", but a 40.0% human accuracy on anchors is also hard to reconcile with the claim that the prefix supports a unique answer. The figure comes from 20 sampled anchors and the paper does not report model accuracy on the same items, so it is unclear whether models exceed humans there. If they do (several models are above 40.0 on the full anchor set, e.g., Gemini-3.1-Pro-Preview at 62.77 in Table 1, Future Prediction, Loc. column), that would point either to anchors that the prefix does not determine, or to models relying on cues other than the stream, such as how the realized option and its distractors are written (Appendix B describes distractors as interpretations "present in the context but incomplete or temporally mismatched"). Either case deserves discussion. The quality ratings in Table 4 do not settle this, since they are given after the annotators saw the full stream and intermediate annotations, by the authors according to the Ethics Statement. This determines whether the headline 63.19% is forecasting skill. Three checks use existing data: report options-only accuracy (no video, no audio) of the strongest evaluated models on the final anchor and support questions, report the evaluated models' accuracy on the 20 audited anchors, and adjudicate those anchors item by item, stating for each human error whether the chosen option is defensible from the prefix alone.

1.(2) We would like to first clarify the human evaluation procedure, as the concern appears to stem partly from a misunderstanding. Human annotators did not have access to the intermediate annotations when answering the questions. Instead, they first watched the same video input (L<tL_{<t}) as the evaluated models under Full setting and then independently answered the questions (Line 415-417). Only after completing the prediction task were they allowed to examine the intermediate annotations and assess the question quality.

Regarding the relatively low human accuracy on anchor questions (40.0%), we would like to highlight two aspects. First, human prediction was conducted as an auxiliary analysis of task difficulty, rather than as a procedure for generating or determining gold labels. Therefore, annotators were not required to repeatedly review the videos until they ensure a ground truth. On the other hand, the evidence support criterion in Section 5.1 provides a complementary assessment of question validity by evaluating the reference explanations. Moreover, the high ratings for Unambiguity indicate that human evaluators generally agreed that the vast majority of questions had a single correct answer (84.1% were rated 3 and 15.9% rated 2, both indicating agreement on single correct answer).

We acknowledge that these details of the human evaluation procedure were not sufficiently described in the original manuscript, which may have led to misunderstandings. We sincerely appreciate the reviewer's careful attention to this issue and have revised the clarifications in the revised version (changes colored in blue).

  1. (1)The diagnostic readings of the Local, Full and modality comparisons are stronger than the controls allow. Section 4.1 states that the Local-to-Full difference "isolates a model's evidence-localization ability", but Appendix A shows the input changes in other ways as the context grows: MiniCPM-o-4.5 drops from 128 to 77 frames in Full, Baichuan-Omni-1.5 is capped at 8 frames, and HumanOmni compresses the entire waveform into a 30-second window. Qwen3.5-Omni-Flash falls to 14.04 on prediction in Full (Table 1, Future Prediction, Full column), below the 25% chance level of a four-option question, because of empty outputs. The required evidence can therefore become less observable independently of retrieval difficulty.

2.(1) Regarding the interpretation of evidence localization in the Local and Full settings: For models with strong long-context processing capabilities, the Local-to-Full comparison can demonstrate their evidence-localization ability. However, for smaller models with limited context windows, the comparison also reflects their overall ability to preserve, process, and utilize relevant evidence under long-context input constraints.

We acknowledge that the term isolate was somehow absolute. We have refined the wording in the revised manuscript to better reflect these distinctions and avoid potential misunderstandings (changes colored in blue).

AuthorsOpenReview ↗

Response to Reviewer gjt6 [3/5]

  1. (2) Section 4.3 draws conclusions about modality contribution (for example that audio-only GPT-Audio beats vision-only GPT-5, and that Qwen2.5-Omni beats both Qwen2.5-VL and Qwen2-Audio) by comparing different checkpoints, and the vision-only model receives no speech at all in streams where the content is carried by speech. These are informative system comparisons but not isolation of a factor. Two models already in the roster, Qwen2.5-Omni-7B and MiniCPM-o-4.5, accept both streams: running the Local setting with audio muted or video removed, and a Full setting that keeps the Local evidence frames and audio at native fidelity inside the longer context, on a modest subset would test these readings directly. Otherwise the text should describe the results as performance under the stated input pipelines.

2.(2) Regarding the modality comparisons, our intention in comparing strong modality-specific models was to assess the contribution of each modality to livestream understanding, rather than analyzing the contributions of different modalities within the same model. We thus investigate the performance current strong vision-only and audio-only models can achieve on StreamCast, instead of experimenting on the same models.

Furthermore, we argue that conducting single-modality ablations on the same omni-modal model may not necessarily yield a more balanced comparison, as omni-modal models themselves can exhibit uneven capabilities across different modalities [1].

[1] Wang, X., Liu, J., Huang, C., Yu, X., Wang, Z., Sun, X., ... & Liu, Z. (2026, April). Xmodbench: Benchmarking cross-modal capabilities and consistency in omni-language models. In International Conference on Learning Representations (Vol. 2026, pp. 19508-19521).

For your Questions For Authors, we respond as follows:

Q1. What is the accuracy of the three blind-test models, and of the strongest evaluated models run without any video or audio, on the final anchor and support questions? If options-only accuracy on anchors is well above 25%, how should the 63.19% be read? In the solvable test, how often did the models answer "not enough information", and among the retained anchors, how many were solved by exactly one, two or all three models?

RQ1. The results of the three test-group models in the blind and solvable tests are presented below, as well as the "E. not enough information" count. In the solvable test, 159/1421 (11.19%) questions were solved by exactly one model, 187/1421 (13.16%) by exactly two models, and 1075/1421 (75.65%) by all three models.

ModelBlind Acc.Solvable Acc.E count
Gemini 3.5 Flash748/1421 = 52.64%1331/1421 = 93.66%19/1421 = 1.33%
Qwen3-235B284/1421 = 19.99%1186/1421 = 83.46%17/1421 = 1.19%
GPT-5 mini263/1421 = 18.51%1215/1421 = 85.50%31/1421 = 2.18%
We additionally conducted a blind test with Gemini-3.1-Pro-Preview, using only textual inputs without any video or audio information. The results are presented below.
Question TypeAcc.
Causal Attribution72/118 = 61.02%
Interaction Dynamics123/221 = 55.66%
Historical Understanding165/300 = 55.00%
Perception Localization146/312 = 46.79%
Future Prediction (Anchor)236/470 = 50.21%

As shown above, both Gemini-3.1-Pro-Preview, the strongest evaluated omni-modal model, and Gemini 3.5 Flash achieve approximately 50% accuracy on anchor questions in the blind test.

We argue that these results reflect the strong reasoning capabilities of these models, which enable them to infer likely future developments from the information provided in the MCQ question text. Importantly, the blind test does not require models to guess randomly, rather, they can still perform reasoning based on the questions and answer options. Therefore, accuracy above the 25% chance level does not necessarily imply answer leakage, but may also reflect strong reasoning capabilities.

In the main experiments, models must additionally process video and audio information beyond the question text. Although these multimodal inputs provide additional reference evidence, they also introduce irrelevant information that may interfere with prediction. We therefore interpret the gap between 63.19% and approximately 50% as reflecting the model's ability to integrate omni-modal information for prediction.

Nevertheless, we acknowledge that this is an interesting issue worth further investigation. The relatively high blind-test accuracy of Gemini models may indicate certain biases in question formulation, which we consider an important direction for future work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewers,

We sincerely appreciate your time and effort in reviewing our work and providing valuable feedback. In response to your constructive suggestions, we have provided detailed clarifications, conducted additional analyses, and revised our manuscript to address your concerns.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our responses and updated manuscript. We hope our revisions have adequately addressed your concerns.

Thank you again for your thoughtful review and valuable suggestions.

Best regards,

Authors

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their insightful comments, constructive suggestions, and careful consideration of our rebuttal. During the rebuttal period, we provided detailed clarifications, additional analyses, and new experiments to address the reviewers' concerns. We are grateful that all reviewers acknowledged our responses and maintained or raised their ratings to positive scores. We summarize the main updates as follows:

  1. To address concerns about question validity and answerability, particularly whether the video prefix sufficiently supports the intended answers, we clarified our definition of a single best-supported answer, provided additional blind- and solvable-test statistics, and reconciled the descriptions of the validation procedure throughout the paper.

  2. We further investigated the role of options-only cues in model performance by conducting an additional options-only evaluation and reporting model results on the same subset used in the human study. These findings, together with a discussion of their implications for interpreting model performance, has been incorporated into the revised manuscript.

  3. The concerns about human evaluation and potential answer ambiguity prompted us to clarify the complete evaluation protocol and examine cases of disagreement among annotators. We also added a case study illustrating how evidence from the video prefix supports the intended answer over plausible alternatives.

  4. For the Local-to-Full comparison, we revised our claims to avoid attributing performance differences solely to evidence-localization ability. To better account for potential confounding factors, we also reported invalid-output rates and conducted an additional frame-budget experiment to examine the effects of long-context input processing.

We sincerely appreciate the reviewers' recognition of the practical value of StreamCast, particularly the usefulness of the Anchor-Support framework for diagnosing prediction failures. We are especially grateful to Reviewer gjt6 for the exceptionally detailed engagement throughout the discussion and for the constructive feedback that helped us substantially improve this work. We thank Reviewer Wv2S for consistently maintaining a positive and supportive view of our paper, as well as for recognizing the value of our contributions. We also thank Reviewer 1EFJ for the active and constructive discussion of both our work and our responses, which helped us further clarify the validation and positioning of StreamCast.

The reviewers' feedback has greatly helped us improve our work. We sincerely appreciate their time and consideration.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.