- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DreamSeek for efficient evidence aquisition in long-video QA. The highlevel idea is to use predicted visual features to guide frame selection, and iteratively refine the feature prediction and frame selection with the feedback from ground-truth visual features under a VLM input budget. To predict the visual features, DreamSeek trains a lightweight temporal world model in a self-supervised way. The experiments show that DreamSeek achieves comparable or slightly better QA accuracy with lower evidence acquisition latency than existing baselines.
Critical strengths
- Working on an important problem, i.e., efficient evidence acquisition for long-video QA
- Interesting technical approach of using predicted visual feature to guide frame selection
- Fairly comprehensive evaluation with promising latency reduction in evidence acquisition
Critical weaknesses
- DreamSeek doesn't fully address the three challenges mentioned, i.e., predictive ambiguity, evidence sufficient, and computational cost. DreamSeek uses a heuristic way to stop searching based on the search concentration, which is not necessarily equivalent to evidence sufficiency. Also, except latency, there is no comprehensive characterization of computational cost, and it remains unclear to me how much computational cost is actually reduced by replacing video decoding and encoding with iterative prediction and refinement in DreamSeek.
- Methodology needs more justification
- Not sure how the trained TWM in DreamSeek can be generalized to different video types and lengths.
- If I understand it correctly, every iteration DreamSeek will select a subset of observed visual features and use them to predict the unseen features. Why don't use all the observed visual features for prediction, given a possible intuition that more observed visual features can make the prediction more accurate?
- The stability of the iterative frame selection. There is no gurantee that an evident frame found in an eariler iteration will be selected as the input frame, assuming sufficient VLM input capacity.
- Evaluation is insufficient and needs some explanation
- The paper only evaluates with K=32 and 64. This is not enough to should the accuracy-efficiency trade-off of DreamSeek.
- Given that DreamSeek is a VLM-agnosic approach, comparing DreamSeek with the training-free baselines using VLMs with different reasoning capacities would be nice.
- In ablation (Table 7), why would more search make QA accuracy worse? f the relevance ranking is reliable, one might expect additional observations to improve evidence localization without displacing previously identified evidence.
- Ablation does not sufficiently isolate the contributions of prediction-guided search and individual components of the iterative search algorithm, such as persistent relevance correction and playoff search. It remains unclear how much each component contributes to the overall performance and efficiency of DreamSeek.
Questions for authors
See my detailed comments in Critical Weakness