acceptodds
Under review as a conference paper at ICLR 2027

Guess What You Ask: Anticipating Moment-level Real Viewer Information Needs in Videos

Abstract

A proactive video agent must decide what information to prepare before a viewer asks. We formulate a prerequisite task of proactive video agent as Moment-level Information Need Anticipation (MINA): given the video context observable up to a video timestamp, predict the questions real viewers are likely to ask. We construct MINA-Data from 12,510 timestamped questions naturally posted by viewers from 872 videos, and evaluate whether an answer to a predicted question would satisfy the observed need. Direct Forecasting is difficult (15.53% at best on Hit@5), even though most generated questions are semantically plausible. This reveals that the challenge is to select which of many plausible needs viewers actually express. We therefore propose MINA-GABT, which predicts a gap-formation state comprising an observed cue, the process by which it evokes an information gap, and the gap itself before verbalizing that gap as a question. A taxonomy induced from training-set viewer behavior guides state prediction. MINA-GABT reaches 21.20% Hit@5 and 27.12% Hit@10, with gains also on a held-out content category. Our analyses show that a video moment can support many plausible questions, but predicting which needs viewers will actually express remains difficult; event-specific gap formation and information beyond the video contribute to this uncertainty.

Primary area:
datasets and benchmarks
Keywords:
information need anticipation, proactive video agents, viewer behavior modeling, question prediction, multimodal learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_h8LROpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies Moment-level Information Need Anticipation (MINA): predicting viewers' actual information needs from preceding video context, conditioned on known timestamps of real viewer questions. The authors construct MINA-Data from 872 Bilibili videos and 12,510 real viewer questions and evaluate predictions using an asymmetric information-coverage criterion. To address the limited performance of direct question generation, they propose MINA-GABT, which first predicts a structured intermediate state comprising an observed cue, a gap-formation process, and the resulting information gap, then realizes that state as a natural-language question. A taxonomy induced from training-set viewer behavior guides prediction. The method achieves 21.20% Hit@5 on test_ID, compared with 15.53% for the best Direct Forecasting baseline, and retains its performance advantage on a held-out content category.

Critical strengths
  1. The dataset is constructed from real viewer behavior. The authors collect 12,510 timestamped questions from Bilibili viewers to build MINA-Data, which evaluates prediction of actual viewer questions given a known question timestamp and the preceding video context.

  2. The authors propose and validate an effective prediction method. MINA-GABT combines structured information-gap prediction with a taxonomy induced from viewer behavior to guide question generation. It outperforms multiple Direct Forecasting baselines on the proposed benchmark, and ablations show that both main components improve performance.

Critical weaknesses
  1. The research value of the benchmark has not been sufficiently justified. The paper is motivated by proactive video assistants that anticipate viewers' information needs and prepare answers in advance, yet MINA evaluation assumes that the timestamps of real viewer questions are known. Such timestamps are not available in advance in practice, and the paper does not validate how the current task can be applied to real proactive-assistance scenarios. In addition, MINA uses a single real viewer question as the prediction target, although different viewers may have entirely different information needs given the same video context. The paper does not analyze the distribution or agreement of real viewer questions, making it difficult to determine whether low Hit@K primarily reflects limitations in model prediction or substantial uncertainty in the target itself. The experiments therefore do not adequately establish a clear research value and practical significance for the capability measured by MINA.

  2. The distinctive contributions of the proposed method have not been sufficiently validated. MINA-GABT guides question prediction with a structured gap-formation state and a taxonomy induced from viewer behavior. However, the experiments do not rule out the possibility that the gains mainly come from more detailed reasoning guidance rather than the specific proposed gap-formation mechanism. Although the authors compare Free-form Need and Intermediate S, they do not examine whether a general free-form method with sufficiently strong reasoning guidance can achieve similar results. The paper also does not compare the behavior-derived taxonomy with generic or manually constructed question patterns. The results establish that MINA-GABT outperforms the selected baselines, but do not adequately isolate the distinctive contributions of its two core designs relative to general prompt-based reasoning strategies.

Questions for authors
  1. The paper is motivated by proactive video assistants that anticipate viewers' information needs and prepare answers in advance, but MINA evaluation depends on known timestamps of real viewer questions. How would a practical system use MINA without knowing these timestamps? If historical viewer questions are used to determine when to predict, would it be more reasonable to use those questions directly? If the system predicts continuously or uses an additional timing detector, what are the resulting prediction performance, computational cost, and practical benefit? Could the authors explain and provide evidence for how the current conditional prediction task effectively supports the claimed application?

  2. MINA uses a single real viewer question as the prediction target, although different viewers may have entirely different information needs given the same context. Have the authors analyzed the agreement or distribution of real viewer questions at the same or nearby video timestamps? If these questions are highly diverse, could low Hit@K mainly reflect uncertainty in individual information needs rather than a lack of behavioral prediction capability? Could the authors provide statistics to clarify the learnability and practical value of the predictive capability measured by MINA?

  3. The current experiments do not rule out the possibility that the gains mainly arise because structured prompts encourage more thorough reasoning rather than because of the specific gap-formation mechanism. Could the authors provide evidence that distinguishes these explanations? For example, under a comparable reasoning budget, could Free-form Need Forecasting with more explicit reasoning guidance achieve similar performance? Does the taxonomy induced from real viewer behavior provide an additional advantage over generic question patterns?

  4. In the human data audit in Appendix A, did reviewers see only the same 10-second video window available to the prediction models? Being able to interpret a known question from video context does not establish that the cues that elicited it fall within that window. Have the authors verified the relationship between real viewer questions and the observation window? If a question depends on earlier video content, could the benchmark attribute failures caused by missing context to the difficulty of predicting viewer behavior?

  5. The asymmetric information-coverage criterion allows a broader predicted question to cover a viewer's specific information need, but such predictions may require preparing more additional information. Appendix A combines specific questions and legitimately broader questions in the reported 95.5%. Could the authors report their proportions separately and analyze the effect of broad predictions on Hit@K? In addition, the reported agreement between the LLM judge and human judgments is 98.2%, but misses may account for most samples. Could the authors report precision and recall for hits to validate the reliability of the core evaluation results?

9 comments
AuthorsOpenReview ↗

Response to Reviewer h8LR (1/7)

We would like to thank the reviewer for the detailed and constructive feedback. We hope the response below addresses your concerns.

Response to Q1 (Prediction without Timestamps)

We thank you for raising this insightful question. We agree that the reliance on video timestamps is an important factor affecting the practical applicability of MINA-GABT.

Anticipation without Oracle Timestamps

To investigate this issue, we conducted additional experiments under two types of temporal perturbations. For N∈3,5,10N\in{3,5,10}, we either randomly shifted the input video window by an offset uniformly sampled from [−N,+N][-N,+N] seconds (Jitter), or shifted the window backward such that its ending timestamp was rounded down to the nearest multiple of NN seconds (Floor). The latter setting evaluates the feasibility of anticipating information needs at fixed temporal intervals without access to the exact timestamps of viewer questions.

Jitter Hit@5Jitter MRR@5Floor Hit@5Floor MRR@5
origin21.2011.7421.2011.74
N=320.3710.9520.5011.28
N=520.2611.1520.0110.97
N=1019.2210.1419.8810.82

The results demonstrate that MINA-GABT is robust to moderate variations in the input timestamp. We hypothesize two possible reasons. First, video semantics often exhibit temporal continuity and redundancy, properties that have been extensively exploited in video understanding and representation learning [1,2,3]. Second, the recorded timestamps may themselves be imprecise and may not always align exactly with the video moments that give rise to the information needs [4]. We leave the systematic investigation of these mechanisms to future work.

More importantly, MINA-GABT also exhibits limited performance degradation under simulated fixed-interval anticipation. Even with a 10-second interval, Hit@5 decreases by only 1.32 percentage points. This supports a straightforward deployment strategy in which information needs are anticipated periodically rather than at known question timestamps. For prerecorded videos, a 10-second interval requires only six predictions per minute of video. Crucially, predictions can be precomputed once and reused across subsequent viewing sessions, rather than requiring separate inference for each viewer.

[Response to Q1 continued on next comment]


References

[1] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge J. Belongie, Yin Cui: Spatiotemporal Contrastive Video Representation Learning. CVPR 2021: 6964-6974

[2] Zhan Tong, Yibing Song, Jue Wang, Limin Wang: VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. NeurIPS 2022

[3] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming He: SlowFast Networks for Video Recognition. ICCV 2019: 6201-6210

[4] Bo Yang, Oskar Lindwall, Christian Licoppe, Coherence through sedimentation: Temporal organization and emergent interaction in Danmaku, Discourse, Context & Media, Volume 72, 2026.

AuthorsOpenReview ↗

Response to Reviewer h8LR (2/7)

[Response to Q1 continued]

Cost and Practical Benefit of MINA-GABT

We further estimated the inference cost of MINA-GABT using the official API pricing of Gemini and OpenAI, without applying cached-input discounts (i.e., all input and output tokens are charged at standard non-cached rates). The average cost is approximately \0.012607perprediction,correspondingtoapproximately 0.012607 per prediction, corresponding to approximately \\4.50 per video hour under a 10-second prediction interval. Importantly, such anticipation is precomputed once for the video and reused for all viewers.

To put this cost into perspective, we conducted an illustrative platform-scale deployment analysis using publicly available YouTube statistics. YouTube has reported an upload rate of approximately 720,000 hours of video per day [5], and one billion hours of daily viewing on television devices alone [6]. Besides, video popularity typically follows a long-tailed distribution, with a small fraction of videos accounting for most viewing activity. According to [7], the most-watched 1% of YouTube videos account for 91% of total viewing time. Consider an illustrative scenario in which popular videos have approximately the same average duration as the overall video population. Preprocessing only the most-watched 1% would then cost approximately \11.83millionannually,covering9111.83 million annually, covering 91% of viewing time. Under a long-run, steady-state model that amortizes the one-time preprocessing cost of each daily upload cohort over its lifetime viewing activity, this corresponds to **approximately \\0.59 per million covered viewing minutes**. For comparison, YouTube's advertising and subscription revenue exceeded \60billionin2025[8],makingthesimulatedannualpreprocessingcost∗∗lessthan0.0260 billion in 2025 [8], making the simulated annual preprocessing cost **less than 0.02% of its annual revenue**. Even if the selected popular videos were, on average, ten times longer than the overall video population, the corresponding cost would still be only approximately \\5.9 per million covered viewing minutes, or 0.2% of annual revenue.

We emphasize that this is an illustrative deployment scenario rather than a measurement of actual YouTube deployment costs. Nevertheless, this analysis illustrates the substantial potential for reducing preprocessing costs through selective deployment while retaining broad viewing-time coverage.

Finally, the above estimates are based on a simple uniform temporal sampling strategy and a heuristic popularity-based video selection strategy, without deployment-specific optimizations. There are several additional opportunities to reduce computational costs, including leveraging cached tokens for repeated inputs, deploying self-hosted models instead of relying on commercial APIs, and distilling the prediction capabilities into smaller, more efficient models. Furthermore, prior research on proactive video agents has investigated when assistance should be provided [9,10,11]. Future work could explore more efficient context-aware anticipation scheduling, rather than relying on uniform temporal sampling, to further improve the trade-off between practical benefits and computational cost.


References

[5] UK Parliament. Oral evidence: Children's TV and Video Content, HC 1338. March 10, 2026. https://committees.parliament.uk/oralevidence/17307/html

[6] Neal Mohan. From the YouTube CEO: Our Big Bets for 2025. YouTube Official Blog, 2025. https://blog.youtube/inside-youtube/our-big-bets-for-2025

[7] Omdia. YouTube Reaches 29 Billion Videos as Music and Shorts Dominate Viewing. January 28, 2026. https://omdia.tech.informa.com/pr/2026/jan/youtube-reaches-29-billion-videos-as-music-and-shorts-dominate-viewing

[8] Alphabet Inc. 2025 Q4 Earnings Call. February 4, 2026. https://abc.xyz/investor/events/event-details/2026/2025-Q4-Earnings-Call-2026-Dr_C033hS6/default.aspx

[9] Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, Jiaqi Wang: Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction. CVPR 2025: 24045-24055

[10] Yueqian Wang, Songxiang Liu, Disong Wang, Nuo Xu, Guanglu Wan, Huishuai Zhang, Dongyan Zhao: MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning. ICLR 2026

[11] Gong et al. Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos. ECCV, 2026.

AuthorsOpenReview ↗

Response to Reviewer h8LR (3/7)

Response to Q2 (Using a single question; Distribution of questions)

We will address your sub-questions one by one.

Distribution and agreement of real viewer questions.

Using a 5-second temporal neighborhood, we find that 47.81% of questions have at least one other real viewer question nearby. Among multi-question windows, 89.32% contain at least two distinct information needs, indicating substantial diversity in real viewer questions under the same or similar video contexts. Meanwhile, 15.35% of nearby question pairs express the same information need, and 40.10% of windows with at least three questions exhibit repeated information needs. These results reveal recurring structures in viewer questions and show that different viewers may express the same information need under the same or similar contexts. Furthermore, 31.30% of windows with at least three questions exhibit both repeated and distinct information needs, indicating that diversity and recurring structure frequently coexist.

Regarding the possibility that there may be different information needs given the same video context.

The diversity of information needs exists indeed. As 47.81% of questions have other real questions within a 5-second neighborhood, and 89.32% of multi-question windows contain at least two distinct information needs, it confirms that question diversity within a certain video context is an important and common characteristic of the MINA task. However, this does not imply that using a single observed viewer question as the ground truth is an inappropriate task design.

  • A real viewer question event can be viewed as a sample from the distribution of viewer information needs. Our dataset may not be able to recover the complete question distribution under a specific video context, as only limited questions are sampled under the context. Nevertheless, by collecting a large number of real events across diverse video contexts, the dataset provides samples from the overall distribution of real viewer questions. Metrics such as Hit@K can therefore estimate a method's expected coverage under this distribution and evaluate the relative predictive capabilities of different methods.
  • Similar principles have been widely adopted in research and practical applications. For example, in session-based recommendation or sequential recommendation, datasets typically record only the next item actually selected by a user, without observing the other items that the user might also have selected under the same state [1,2,3,6]. Moreover, metrics such as Hit Rate and MRR are well suited to settings where only a single behavioral event is observed, even though multiple alternative behaviors may be possible under the same context. These metrics are also widely used in sequential recommendation.

[Response to Q2 continued on next comment]


References

[1] Wang-Cheng Kang, Julian McAuley (2018). Self-Attentive Sequential Recommendation. In Proceedings of IEEE International Conference on Data Mining (ICDM'18)

[2] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, Domonkos Tikk: Session-based Recommendations with Recurrent Neural Networks. ICLR (Poster) 2016

[3] Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, Peng Jiang: BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. CIKM 2019: 1441-1450

[6] Malte Ludewig, Noemi Mauro, Sara Latifi, Dietmar Jannach: Empirical analysis of session-based recommendation algorithms. User Model. User Adapt. Interact. 31(1): 149-181 (2021)

Reviewer_pL3pOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

A proactive video agent must decide what information to prepare before a viewer asks. The authors propose a prerequisite task of proactive video agent as Moment-level Information Need Anticipation (MINA): given the video context observable up to a video timestamp, predict the questions real viewers are likely to ask. To improve the performance of question forecasting, the authors further propose MINA-GABT, which predicts a gap-formation state comprising an observed cue, the process by which it evokes an information gap, and the gap itself before verbalizing that gap as a question. Comprehensive experiments demonstrate the effectiveness of proposed MINA-GABT.

Critical strengths
  1. The paper is well written. It is easy to understand and follow the logical flow.

  2. The paper presents a novel and interesting idea with extensive experimental evaluations. The proposed MINA task and MINA-GABT method are interesting. The authors conduct human validation of the collected data and evaluate their method using different models. They also provide detailed analyses of the remaining uncertainty in anticipating viewer information needs.

Critical weaknesses
  1. Insufficient justification of the task motivation and practical utility. Although anticipating viewer information needs may help reduce response latency, the paper does not sufficiently demonstrate the practical benefits of this approach. As the authors acknowledge, a single video moment can trigger many different information needs, making it difficult to predict what a particular viewer actually wants to know. Furthermore, the benchmark assumes that the timestamp at which a viewer asks a question is already known. However, a real proactive agent must also determine when assistance is needed. The authors should clarify how the proposed task can be applied to real-world proactive VideoQA systems and whether it offers meaningful advantages over responding to explicit user questions.

  2. Potential selection bias and insufficient validation of the data filtering process. The dataset construction relies on LLM-based filtering to identify genuine information-seeking questions. Although Appendix A reports human validation of retained samples, this evaluation only assesses the validity of selected questions. The authors do not evaluate whether the filtering process incorrectly discards valid viewer questions. This may introduce selection bias and distort the distribution of real viewer information needs, which is central to the proposed task. The authors should provide a more comprehensive validation of the filtering process, including an analysis of potentially valid questions among discarded comments.

Questions for authors

Please refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer pL3p (1/3)

Thanks for the constructive review. We hope the response below addresses your concerns.

Response to W1 (Prediction without Timestamps)

We address your concerns from two perspectives.

Anticipating Information Needs Without Known Question Timestamps

We agree that the reliance on video timestamps is an important factor affecting the practical applicability of MINA-GABT. To investigate this issue, we conducted additional experiments under two types of temporal perturbations. For N∈3,5,10N\in{3,5,10}, we either randomly shifted the input video window by an offset uniformly sampled from [−N,+N][-N,+N] seconds (Jitter), or shifted the window backward such that its ending timestamp was rounded down to the nearest multiple of NN seconds (Floor). The latter setting evaluates the feasibility of anticipating information needs at fixed temporal intervals without access to the exact timestamps of viewer questions.

Jitter Hit@5Jitter MRR@5Floor Hit@5Floor MRR@5
origin21.2011.7421.2011.74
N=320.3710.9520.5011.28
N=520.2611.1520.0110.97
N=1019.2210.1419.8810.82

The results demonstrate that MINA-GABT is robust to moderate variations in the input timestamp. We hypothesize two possible reasons. First, video semantics often exhibit temporal continuity and redundancy, properties that have been extensively exploited in video understanding and representation learning [1,2,3]. Second, the recorded timestamps may themselves be imprecise and may not always align exactly with the video moments that give rise to the information needs [4]. We leave the systematic investigation of these mechanisms to future work.

More importantly, MINA-GABT also exhibits limited performance degradation under simulated fixed-interval anticipation. Even with a 10-second interval, Hit@5 decreases by only 1.32 percentage points. This supports a straightforward deployment strategy in which information needs are anticipated periodically rather than at known question timestamps. For prerecorded videos, a 10-second interval requires only six predictions per minute of video. Crucially, predictions can be precomputed once and reused across subsequent viewing sessions, rather than requiring separate inference for each viewer.

[Response to W1 continued on next comment]


References

[1] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge J. Belongie, Yin Cui: Spatiotemporal Contrastive Video Representation Learning. CVPR 2021: 6964-6974

[2] Zhan Tong, Yibing Song, Jue Wang, Limin Wang: VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. NeurIPS 2022

[3] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming He: SlowFast Networks for Video Recognition. ICCV 2019: 6201-6210

[4] Bo Yang, Oskar Lindwall, Christian Licoppe, Coherence through sedimentation: Temporal organization and emergent interaction in Danmaku, Discourse, Context & Media, Volume 72, 2026.

AuthorsOpenReview ↗

Response to Reviewer pL3p (2/3)

[Response to W1 continued]

Practical Benefits of MINA-Based Assistance Over Reactive VideoQA

To clarify the practical benefits of MINA, we propose two human–computer interaction models, representing a conventional reactive VideoQA system and a MINA-based assistance system, respectively.

  • In a reactive VideoQA system, the interaction proceeds as follows: a viewer develops a question while watching a video, formulates and enters the question into the system, waits for the system to generate an answer, and finally receives the response.
  • In contrast, a MINA-based assistance system consists of two phases: offline preparation and online interaction. During offline preparation, MINA anticipates potential viewer questions, and a VideoQA module generates and caches their corresponding answers. This phase is precomputed. During online interaction, when a viewer develops a question while watching a video, they first examine the predicted questions associated with the current video timestamp. If one of the predicted questions matches their information need, the viewer can directly access its precomputed answer. Otherwise, the system falls back to the conventional reactive VideoQA workflow: the viewer enters their question, waits for answer generation, and receives the response.

Such paradigm is consistent with prior human-computer interaction (HCI) researchs [1,2,5]. Following established cost–benefit models of proactive interaction [3,4], we formulate the expected benefit of the MINA-based system over the reactive system as: ΔT=h⋅TR−Tscan.\Delta T = h \cdot T_R - T_{\mathrm{scan}}. where TRT_R is the time and interaction cost of formulating and entering a question and waiting for generating an answer; TscanT_{\mathrm{scan}} denote that of examining the predicted questions; and hh denote the probability that predicted questions matches the actual information need. Therefore, the two primary optimization objectives of a MINA-based system are maximizing the question hit rate and minimizing the cost of examining predicted questions. The former is precisely the value and optimization objective of the MINA task. The latter can be improved through HCI designs, such as simplifying question formulations while preserving their semantics, or highlighting keywords to help viewers identify relevant questions more quickly.

More importantly, MINA-based assistance is not intended to anticipate every possible viewer question or replace reactive VideoQA. Instead, it introduces a low-cost interaction pathway while retaining reactive VideoQA as a fallback. Therefore, despite the diversity of viewer information needs, MINA has the potential to reduce user burden and interaction costs while preserving all the capabilities of a reactive system. We clarify that this interaction model is more accurately characterized as a concrete user-initiated interaction paradigm enabled by proactive information preparation, rather than a fully autonomous proactive agent. We will refine the corresponding descriptions in the paper to make this distinction explicit.


References

[1] Valerie Chen, Alan Zhu, Sebastian Zhao, Hussein Mozannar, David A. Sontag, Ameet Talwalkar: Need Help? Designing Proactive AI Assistants for Programming. CHI 2025: 881:1-881:18

[2] Peggy Chi, Senpo Hu, Lei Shi, Tanya Kraljic, Justin Secor, Tao Dong, Irfan Essa, Mike Cleron: WatchWithMe: LLM-Based Interactive Guided Watching of Review Videos. CUI 2025: 59:1-59:15

[3] Leif Azzopardi, Guido Zuccon: An Analysis of the Cost and Benefit of Search Interactions. ICTIR 2016: 59-68

[4] Philip Quinn, Shumin Zhai: A Cost-Benefit Study of Text Entry Suggestion Interaction. CHI 2016: 83-88

[5] Saleema Amershi, Daniel S. Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi T. Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, Eric Horvitz: Guidelines for Human-AI Interaction. CHI 2019: 3

AuthorsOpenReview ↗

Response to Reviewer pL3p (3/3)

Response to W2 (Potential Selection Bias of Filtering)

To balance the cost and effectiveness of data filtering, we designed a multi-stage filtering pipeline with progressively higher per-sample processing costs and increasingly stringent filtering criteria. The earlier stages adopt relatively lenient criteria to retain as many potential questions as possible. Since the vast majority of comments discarded in the earlier stages are clearly non-questions, random sampling for human validation would be relatively inefficient. In contrast, nearly all candidates entering the final stage exhibit question-like characteristics and are subject to more stringent filtering criteria. We therefore focused our additional human validation on comments discarded in the final stage to investigate potential false negatives.

We randomly sampled 100 discarded comments for human validation and classified them into the following five categories.

CategoryDefinitionCount
Non-information-seekingQuestion-like expressions of humor, rhetorical remarks, or emotional reactions10
Ambiguous intentComments that can plausibly be interpreted as both information-seeking questions and non-information-seeking expressions32
Underspecified meaningComments whose specific meaning cannot be reliably determined even with the video context11
Restatement of presented informationQuestion-like restatements of information already explicitly presented in the video, without clear information-seeking intent38
Valid information-seeking questionsQuestions expressing clear and identifiable information needs that were incorrectly discarded9

The exclusion of the first 4 categories is consistent with our filtering principles. As described in the paper, excluding Non-information-seeking and Underspecified meaning comments is an explicit design choice. For Ambiguous intent, the inherent ambiguity prevents us from reliably determining whether the comments express genuine information-seeking intent. For Restatement of presented information, we consider such comments to primarily express skepticism, surprise, or disbelief rather than clear information-seeking intent. We therefore adopt a conservative strategy to exclude these samples from the dataset.

We acknowledge that this conservative filtering strategy inevitably entails missing some genuine information needs. Nevertheless, our human validation shows that only 9% of the samples discarded in the final stage express clear and identifiable information needs. Although this may introduce some selection bias, there is an inherent trade-off between maximizing the coverage of genuine information needs and ensuring the reliability of the dataset, and we prioritize the latter. As reported in Appendix A, human validation indicates that MINA-Data contains very few samples that fail to meet our filtering criteria. We will incorporate these additional validation results into the revised paper.

Reviewer_3TSmOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Moment-level Information Need Anticipation (MINA), where an agent prepares information that a viewer will likely ask. They construct a benchmark to evaluate such criteria for each video content with metrics, measuring the sufficiency relation. Overall experiments and analyses are comprehensive and well conducted.

Critical strengths

This paper pursues a compelling research direction by moving beyond proactive response timing to anticipating viewers’ information needs. Proactivity alone does not ensure useful assistance; a system must also anticipate what information users actually seek. Grounding this task in naturally occurring viewer questions makes the setting practically relevant to real-world video assistance.

Critical weaknesses

Despite the compelling problem formulation, several aspects of the evaluation warrant further clarification.

  • First, the asymmetric information-coverage criterion measures whether answering a predicted question could satisfy the observed need, which does not necessarily establish that the model precisely anticipated that need. The authors’ human validation and specificity audit are valuable, but a complementary evaluation of answer-equivalent predictions would help distinguish precise anticipation from broader information coverage.
  • Although the gap between semantic plausibility and expressed viewer needs is informative, its existence is conceptually expected, and the source of the remaining difficulty is less clearly established. A single observed question represents one realization of potentially diverse viewer preferences; low Hit rates may therefore reflect both prediction limitations and uncertainty arising from unobserved viewer goals, knowledge, or characteristics. Analyses that distinguish these factors would strengthen the interpretation of behavioral anticipation performance.
  • For me, It remains unclear how much prediction requires modeling an event-specific information gap beyond identifying salient recent objects or actions. A matched-budget baseline that generates questions from recent salient cues and common question patterns would help isolate this contribution. The reported degradation with longer observation windows makes this comparison particularly relevant, although that degradation alone cannot establish that longer-term context is unnecessary, since it may also reflect limitations in the models’ use of additional context.
Questions for authors

See the Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3TSm (1/3)

We thank Reviewer pL3p for the insightful review and constructive suggestions. We hope the results below address your concerns.

Response to W1 (Results on Symmetric Criterion)

We designed a symmetric evaluation criterion to determine whether a predicted question and the ground-truth question express the same information need. Using this criterion, we re-evaluated a subset of the methods reported in Table 3 of the original paper.

MethodHit@5 (Origin)MRR@5 (Origin)Hit@5 (Symmetric)MRR@5 (Symmetric)
Direct12.357.378.225.27
Free-form need only16.679.827.674.80
Intermediate S only18.6610.2710.345.92
Taxonomy only15.179.029.916.15
MINA-GABT21.2011.7411.946.69

The absolute scores generally decrease, while the relative ranking of methods remains largely unchanged, with the exception of Free-form need only. Under this stricter criterion, MINA-GABT still outperforms Direct Forecasting by 3.72 pp, demonstrating that its performance gains are not solely attributable to broader predictions. We provide several examples that are judged as hits by the original evaluator but as misses by the symmetric evaluator:

Ground-truth questionPredicted question
Was this filmed in Chengdu?Which city was this filmed in?
Is his name Clark?What is the full name of Gable mentioned in the video?
Were these boiled or candied?How exactly were these cherry tomatoes prepared?

The original evaluator considers these broader questions as hits because their answers would also satisfy the ground-truth information needs. We believe this asymmetric evaluation criterion better aligns with MINA's intended application scenario, where the goal is to anticipate and prepare information sufficient to satisfy viewers' needs.

AuthorsOpenReview ↗

Response to Reviewer 3TSm (2/3)

Response to W2 (Prediction Limitations and Uncertainty)

We agree that the low absolute Hit rates may reflect both limitations in prediction capability and irreducible uncertainty arising from unobserved viewer characteristics. We investigate these two factors separately.

  • Uncertainty in viewer information needs. We conducted an additional analysis of real viewer questions within 5-second temporal neighborhoods. Among neighborhoods containing multiple questions, 89.32% contain at least two distinct information needs. Meanwhile, 15.35% of neighboring question pairs express the same information need, and 40.10% of neighborhoods with at least three questions exhibit recurring needs. These observations reveal the coexistence of shared behavioral patterns and substantial variability in viewer information needs, even under highly similar video contexts.
  • Limitations in prediction capability. Under identical observable video contexts, MINA-GABT improves Hit@5 from 12.35% to 21.20% over Direct Forecasting, demonstrating that a substantial portion of prediction errors can be reduced through better modeling. Furthermore, our oracle analysis in Section 6 helps localize the prediction bottleneck to event-specific gap formation.

Systematically disentangling their contribution is beyond the scope of this paper. We will clarify their distinction and the corresponding limitations in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 3TSm (3/3)

Response to W3 (Results of Salient Cues + Question Patterns; Extended Ablations)

Following your suggestion, we designed a corresponding baseline. With the same induction procedure with MINA-GABT, we derived a set of 43 common question patterns from the same real viewer behavior data. The "Common question patterns + (c)" setting in the table below corresponds directly to your suggested baseline based on "salient cues and common question patterns." Additionally, we conducted more fine-grained and comprehensive ablations of the structured gap-formation state S=(c,p,g) to investigate how it goes beyond merely identifying salient cues (c).

Patterns/TaxonomyIntermediateHit@5MRR@5
Nonec12.277.01
g16.579.13
c+g16.919.56
S18.6610.27
Common question patternsc16.168.62
g18.4910.33
c+g17.9810.25
S19.0710.74
GABT taxonomyc14.778.00
g18.649.74
c+g18.649.75
S (MINA-GABT)21.2011.74
  • The results show that identifying salient cues with common question patterns is less effective than replacing salient cues with the complete structured gap-formation state (S) (16.16% vs. 19.07% Hit@5), while the full MINA-GABT achieves further improvements (21.20% Hit@5).
  • The results also validate the contribution of the GABT taxonomy. With the complete intermediate representation S=(c,p,g), common question patterns improve Hit@5 by 0.41 pp over using no patterns or taxonomy. However, the GABT taxonomy yields substantially larger gains (+2.54 pp over None and +2.13 pp over common question patterns).
    • As described in our paper, the GABT taxonomy abstracts recurring structures in the gap-formation process (p), whereas common question patterns primarily abstract the information gap (g) itself. For example, consider a video emphasizing that "ice water must be used instead of warm water," prompting a viewer to ask, "Why must ice water be used?" A corresponding common question pattern might be "Under <condition>, why choose <option>?" In contrast, the abstraction of (p) would focus on the underlying formation mechanism: "A specific instruction is given without an accompanying explanation, giving rise to a need to understand its rationale." This illustrates a fundamental distinction: common question patterns primarily capture characteristics of the resulting information needs, whereas gap-formation patterns additionally characterize how a particular video cue gives rise to a specific information gap.
    • Interestingly, the relative effectiveness of the common question patterns and GABT taxonomy depends on the intermediate representation. When only (c) is used, common question patterns perform best (16.16% Hit@5), possibly because their empirical prior complements the limited information provided by the cue representation. With (g) or (c+g), the differences between common question patterns and the GABT taxonomy are small (+0.15 pp and +0.66 pp in favor of GABT, respectively). However, with the complete state S=(c,p,g), the GABT taxonomy achieves a substantially larger advantage (+2.13 pp). This suggests that the empirical prior provided by the GABT taxonomy is better aligned with the complete structured representation, allowing the model to more effectively exploit the cue-to-gap formation structure encoded in (S).
  • The experiments also validate the contribution of the structured representation (S).
    • First, we examine the contribution of the gap-formation process (p). Adding (p) to (c+g) consistently improves performance across all three settings (+2.56 pp for GABT taxonomy, +1.09 pp for common question patterns, and +1.75 pp for None). The particularly pronounced improvement under the GABT taxonomy suggests that the behavior-derived taxonomy is better aligned with, and may more effectively exploit, the explicit representation of cue-to-gap formation processes.
    • Interestingly, adding (c) to (g) does not consistently improve performance, whereas introducing (p) into (c+g) consistently yields further gains. This suggests that the benefit of the complete state may lie not merely in representing cues and information gaps separately, but in explicitly modeling the relationship through which a particular cue gives rise to a particular information gap. Such a relationship naturally requires both (c) and (g) as its endpoints.
    • Since (p) is defined as the formation relationship between (c) and (g) c→pgc \xrightarrow{p} g, we focus on the controlled comparison between (c+g) and (c+p+g), rather than treating (p) as an independent information component and evaluating combinations such as (p), (c+p), or (p+g).

Comments on the paper

1 comment

Dear Reviewers,

Thanks for your constructive reviews! We have posted the response to your questions, and would gladly engage in further discussion.

Best Regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.