acceptodds
Under review as a conference paper at ICLR 2027

DreamSeek: Imagination-Guided Evidence Acquisition for Long-Video Question Answering

Abstract

Finding decisive evidence in a long video can require processing many irrelevant frames. Frame selection reduces the answerer's input, while relevance-guided search uses sparse observations to locate promising regions. However, these observations also provide temporal context for predicting unread visual features, offering guidance beyond measured relevance. We introduce DreamSeek, an imagination-guided approach whose question-agnostic temporal world model predicts unread visual features, enabling relevance assessment before real acquisition. New observations refine the predicted timeline and provide relevance corrections that persist across search rounds. This process enriches search guidance through low-cost latent computation, reducing expensive decoding and encoding. With a shared answerer, DreamSeek achieves the highest mean accuracy on three of five long-video QA benchmarks and up to 5.1 evidence-acquisition speedups over strong frame-selection baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long Video Understanding, VLMs, World Model, Latent Features, Efficient AI

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_c5YHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DreamSeek for efficient evidence aquisition in long-video QA. The highlevel idea is to use predicted visual features to guide frame selection, and iteratively refine the feature prediction and frame selection with the feedback from ground-truth visual features under a VLM input budget. To predict the visual features, DreamSeek trains a lightweight temporal world model in a self-supervised way. The experiments show that DreamSeek achieves comparable or slightly better QA accuracy with lower evidence acquisition latency than existing baselines.

Critical strengths
  • Working on an important problem, i.e., efficient evidence acquisition for long-video QA
  • Interesting technical approach of using predicted visual feature to guide frame selection
  • Fairly comprehensive evaluation with promising latency reduction in evidence acquisition
Critical weaknesses
  • DreamSeek doesn't fully address the three challenges mentioned, i.e., predictive ambiguity, evidence sufficient, and computational cost. DreamSeek uses a heuristic way to stop searching based on the search concentration, which is not necessarily equivalent to evidence sufficiency. Also, except latency, there is no comprehensive characterization of computational cost, and it remains unclear to me how much computational cost is actually reduced by replacing video decoding and encoding with iterative prediction and refinement in DreamSeek.
  • Methodology needs more justification
    • Not sure how the trained TWM in DreamSeek can be generalized to different video types and lengths.
    • If I understand it correctly, every iteration DreamSeek will select a subset of observed visual features and use them to predict the unseen features. Why don't use all the observed visual features for prediction, given a possible intuition that more observed visual features can make the prediction more accurate?
    • The stability of the iterative frame selection. There is no gurantee that an evident frame found in an eariler iteration will be selected as the input frame, assuming sufficient VLM input capacity.
  • Evaluation is insufficient and needs some explanation
    • The paper only evaluates with K=32 and 64. This is not enough to should the accuracy-efficiency trade-off of DreamSeek.
    • Given that DreamSeek is a VLM-agnosic approach, comparing DreamSeek with the training-free baselines using VLMs with different reasoning capacities would be nice.
    • In ablation (Table 7), why would more search make QA accuracy worse? f the relevance ranking is reliable, one might expect additional observations to improve evidence localization without displacing previously identified evidence.
    • Ablation does not sufficiently isolate the contributions of prediction-guided search and individual components of the iterative search algorithm, such as persistent relevance correction and playoff search. It remains unclear how much each component contributes to the overall performance and efficiency of DreamSeek.
Questions for authors

See my detailed comments in Critical Weakness

Reviewer_cC5JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

DreamSeek is using a temporal feature predictor to guide evidence acquisition for long-video VQA. From low number of observed frames, a question agnostic model predicts unread visual features through anchor reuse and learned regression. Question relevance scores over these predictions guides further real-frame acquisition, while persistent observational corrections update search priorities. The final answerer receives only real frames, combining targeted evidence with evenly spaced context.

Critical strengths
  • Predicting where to look before decoding and encoding the complete video is an interesting approach to reducing acquisition cost. The distinction between predicted search guidance and observed answering evidence is important. Real-frame verification, persistent corrections, and context sampling provide sensible safeguards against inaccurate predictions.

  • The representation ablations provide meaningful support for learned completion. At matched budgets of 32 and 64 search reads, tight evidence capture improves from 65.57% and 68.68% with nearest-anchor copying to 76.19% and 83.95% with learned completion. This shows that the predictor contributes useful localization information beyond reusing observed features. The reported acquisition speedups and competitive QA results further support its practical value.

Critical weaknesses
  • The verification mechanism addresses incorrectly predicted relevant events more directly than missed events. A high-scoring prediction can be inspected and corrected, but an unexpected, answer-critical event assigned low predicted relevance may remain unread. Predictive ambiguity is a recognized issue in visual feature anticipation; Vondrick et al. explicitly consider multiple possible futures [1]. DreamSeek produces a single proposal feature per position without explicitly representing competing possibilities. The paper therefore leaves unclear how reliably search discovers unusual events or subtle changes that are poorly predicted from the anchors. Evenly spaced context mitigates this risk but does not establish reliable discovery.

  • The stopping criterion measures belief concentration rather than verified evidence sufficiency. Because stopping is checked before further verification, concentrated but incorrect predicted relevance can terminate search. The aggregate stopping ablation demonstrates useful efficiency, but does not establish when concentration is trustworthy. The reported cases where evidence is never found or is found and later lost make this distinction consequential, although they do not by themselves identify prediction or stopping as the cause.

[1] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. “Anticipating Visual Representations from Unlabeled Video.” CVPR, 2016. Paper.

Questions for authors
  • How does the stopping criterion behave on confidently incorrect predictions, brief unexpected events, or near-identical frames containing different answer-critical details? Would requiring real verification of the dominant region before stopping change the accuracy–acquisition tradeoff?

  • How does the stopping criterion behave on confidently incorrect predictions, brief unexpected events, or near-identical frames containing different answer-critical details? Would requiring real verification of the dominant region before stopping change the accuracy–acquisition tradeoff?

Reviewer_GV4yOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

DreamSeek is a framework for answering questions about long videos without wasting time on irrelevant frames. Instead of processing every frame, it looks at only a few and uses a lightweight “temporal world model” to imagine what the unseen parts of the video look like. This model works independently of the question being asked. DreamSeek then compares its predictions with real frames and uses that feedback to update its predictions and decide where to look next. It requires no fine-tuning of the vision-language model (VLM), yet it achieves the highest average accuracy on three of five major benchmarks and finds the necessary evidence up to 5.1× faster than existing frame-selection methods.

Critical strengths

Making Unseen Frames Searchable: Existing methods must first decode and encode candidate frames before they can judge how relevant those frames are. DreamSeek reverses this order. It first imagines the features of unobserved segments, scores them based on those predictions, and decodes only the segments that actually need to be checked.

Decoupling the World Model from the QA Model: The proposed Temporal World Model is question-agnostic and is trained entirely through masked self-supervision, without using any QA answer labels. Because of this, there is no need to fine-tune a large VLM for each task. A single lightweight world model with about 26.9 million parameters can be reused across many different questions and videos.

Correcting Predictions with Real Observations: DreamSeek does not blindly trust its own predictions but continuously corrects them with real observations. It imagines what the video contains, searches for the most promising segments, observes them directly, corrects its predictions based on what it sees, and then imagines again. This cycle is the core mechanism of DreamSeek. Rather than relying only on its initial predictions, it verifies the predicted peak segments by actually looking at them, accumulates this feedback in a log-correction field, and uses it to update its predictions and search beliefs in each subsequent round.

Critical weaknesses

Limited Evidence Completeness and Fine-Grained Spatiotemporal Resolution: DreamSeek is currently strong at relevance localization, which means finding the specific moments relevant to a question. It is weaker at evidence completeness, which requires gathering all the evidence spread across an entire video. As a result, it struggles with counting questions, which require tallying events that occur at multiple points in time, and with summarization questions, which require understanding the overall storyline. Compared with uniform sampling, its accuracy drops by 10.76 percentage points on counting in the long split of Video-MME and by 2.83 percentage points on summarization in LVBench.

In addition, in the final stage DreamSeek extracts only one frame from the midpoint of each 8-second window and passes it to the answering model at a low resolution of 256p. This limits its ability to capture fine visual details, such as small object manipulations that last only about four seconds or spatial relationships against cluttered backgrounds. Handling such cases requires deciding not only where to look but also how densely and in how much detail to look.

Losing Priority on Evidence During Iterative Search (Found, Then Lost): Although DreamSeek accumulates observation feedback, its search is not fully stable. When a new observation updates the predicted features (proposals) across the whole timeline, a key evidence segment found earlier can be pushed down by competing segments and lose its high rank. An analysis on the LVH benchmark shows that in 7.8% of all cases, the search found the correct evidence but lost its priority in a later round. In other words, searching for longer can sometimes hurt localization performance (Figures 2 and 3).

Length-Dependent Bias in the Adaptive Stopping Criterion: The search concentration score, which determines when to stop searching, is computed as the share of belief within a fixed window of 17 segments around the highest peak, relative to the total belief across all T segments of the timeline. Because of how this formula is structured, even a completely flat belief, in which no evidence has been identified at all, yields a concentration score of 17/T. For short videos, where T is small, this baseline score is already high. It can therefore exceed the stopping threshold of 0.05 even when no clear cue has been found, causing the search to stop prematurely. This makes the stopping criterion inherently dependent on video length.

Questions for authors
  1. Mismatch Between Search and Delivery: During search, DreamSeek operates at a 2-second segment resolution using 2-second segments, yet in the final delivery stage it groups them into 8-second windows and passes only the middle frame of each window. What is the rationale for this design? Instead of delivering the high-scoring real frames already acquired during search already decoded during search, DreamSeek decodes a new frame at the center of each 8-second window. Could this cause it to miss short actions lasting less than four seconds, such as the tile-part manipulation shown in Figure 9?

  2. Counting Performance and Multi-Event Coverage: DreamSeek underperforms uniform sampling by 10.76 percentage points on counting questions in the long split of Video-MME. The paper suggests that counting may require coverage of multiple distinct occurrences rather than localization of a single relevant moment. However, several components of the search procedure may contribute to this limitation. In particular, the stopping criterion κ(r)\kappa^{(r)} measures belief concentration only around the strongest peak c∗c_*, which may favor termination once one occurrence is confidently localized. In addition, candidate peaks are greedily selected with a minimum separation of 8 segments, potentially suppressing nearby repeated events. The final delivery policy may also provide insufficient coverage of multiple occurrences. Could the authors clarify which of these factors contributes most to the counting degradation? Ablations that independently vary the stopping criterion, peak-separation distance, and delivery coverage on counting questions would be particularly informative.

  3. Cause of the “Found, Then Lost” Cases: On LVH, 7.8% of cases fall into the “found, then lost” category. What is the main cause of this phenomenon? Possible explanations include the following: (1) adding new anchors substantially changes the world model’s imagined features z~\tilde{z}; (2) during verification or playoff, additional observations of neighboring segments yield low similarity, so the correction factor (γ<1\gamma < 1) suppresses the reference evidence region; or (3) false-positive peaks, which are not actually correct but happen to have high cosine similarity with the SigLIP text embedding, overtake the correct segment. A breakdown of these cases by cause would be informative.

  4. Why Can Longer Search Degrade Performance? Figure 2 shows that evidence localization improves over the early search rounds but slightly deteriorates when search continues to the end, while Table 7 indicates that disabling adaptive stopping generally does not improve QA accuracy and reduces it on several benchmarks. What mechanism causes this degradation with continued search? Because each refreshed completion replaces the previous proposal rather than accumulating predictions directly, this behavior does not appear to be simple iterative error accumulation. Is the degradation instead caused by instability in the completed feature timeline as the anchor set changes, by accumulated relevance corrections, or by competition from newly introduced peaks that demote previously identified evidence? Further analysis of how evidence ranks, predicted features, and correction terms evolve during later rounds would help explain why additional observations can sometimes hurt rather than help.

  5. Premature Stopping on Short Videos: As acknowledged in Appendix C, even a flat belief yields κ=17/T\kappa = 17/T. As a result, for videos with T≤340T \le 340 segments (11 minutes 20 seconds or shorter with 2-second segments), the baseline concentration reaches or exceeds the threshold τ=0.05\tau = 0.05 even without any informative evidence. Since LVH includes videos of around six minutes, how often does DreamSeek terminate immediately after the initial completion on these shorter videos? Does the length dependence of κ\kappa lead to premature stopping in practice, and how does this affect accuracy?

Reviewer_9u95OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DreamSeek, an imagination-guided evidence acquisition framework for efficient long-video question answering. The key idea is to predict the visual representations of unobserved video segments using a lightweight, question-agnostic temporal world model, enabling question-relevant evidence localization without exhaustively decoding and encoding the video. Specifically, the model combines learned feature prediction with observed-feature reuse, while an iterative search mechanism incorporates persistent relevance corrections from newly acquired frames. The framework further employs adaptive stopping and a hybrid frame-delivery strategy that balances targeted evidence with temporal coverage. Importantly, only real video frames are provided to the downstream answerer, without task-specific VLM fine-tuning.

The authors evaluate DreamSeek on five long-video QA benchmarks using a shared frozen answerer. Results show that it achieves the highest mean accuracy on three of five benchmarks at both 32-frame and 64-frame budgets, while providing 1.8–5.1× evidence-acquisition speedups over evaluated frame-selection baselines. Additional ablation studies examine the effectiveness of learned temporal completion, adaptive stopping, and frame-delivery strategies.

Critical strengths
  1. Novel and efficient evidence acquisition framework. The paper introduces a compelling approach that leverages temporal feature prediction to estimate the relevance of unobserved video segments before actual acquisition. The combination of a lightweight, question-agnostic world model and persistent observation-based corrections enables efficient evidence search without task-specific VLM fine-tuning. This provides a meaningful alternative to conventional relevance-based frame selection.

  2. Comprehensive empirical validation with clear efficiency gains. The paper evaluates the proposed method across five long-video QA benchmarks under a shared-answerer setting, achieving the best mean accuracy on three benchmarks and 1.8–5.1× evidence-acquisition speedups over frame-selection baselines. The ablation studies further substantiate the contributions of temporal completion, adaptive stopping, and frame delivery, strengthening the empirical support for the proposed design.

Critical weaknesses
  1. Insufficient ablation of the key methodological components. Although the paper demonstrates the benefits of learned temporal completion, it does not adequately isolate the contributions of the pointer-mixture predictor and persistent observation-based corrections, which are central to the proposed framework. In particular, the impact of retaining relevance corrections across search rounds remains unclear. Without these comparisons, it is difficult to determine which components are essential to the observed improvements and whether the additional design complexity is justified.

  2. Unclear end-to-end efficiency advantages. The reported 1.8–5.1× speedups only cover evidence acquisition, excluding final frame delivery and VLM answering. Since the proposed method introduces additional temporal-model inference and iterative verification, its practical efficiency advantage over existing methods remains uncertain. Moreover, some agent-based methods report lower acquisition latency under different evaluation protocols. A controlled end-to-end comparison is needed to substantiate the overall efficiency claims.

Questions for authors
  1. Contribution of persistent corrections: How much of the performance gain can be attributed specifically to persistent observation-based corrections? Could the authors compare the current method against variants that (i) remove relevance corrections and (ii) reset corrections after each completion round? Reporting both QA accuracy and acquisition cost would help establish whether this mechanism is essential.

  2. End-to-end efficiency: Could the authors report end-to-end inference latency, including evidence acquisition, final frame delivery, and VLM answering, under identical hardware and answerer settings? In particular, how much of the acquisition-stage speedup translates into actual end-to-end acceleration?

  3. Limitations on questions requiring distributed evidence: Table 11 shows that DreamSeek underperforms uniform sampling substantially on counting questions (−10.76 points) and also struggles with summarization. Are these failures primarily caused by incomplete evidence acquisition, the concentration-based stopping criterion, or limitations of the downstream answerer? An analysis distinguishing these factors would help clarify the applicability of the proposed search strategy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.