- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HieraSeek treats long-video reasoning as repeated access to a hierarchy built inside the visual encoder. Early ViT blocks progressively select representative frames and merge adjacent temporal groups, while cached intermediate states let the model revisit regions without encoding them again from scratch. A policy trained through SFT and reinforcement learning chooses whether to narrow its focus, relocate, or answer. Evaluation covers six benchmarks, with ablations on representation structure, rewards, and state reuse.
Critical strengths
-
Reusing intermediate ViT states connects hierarchy construction with evidence search and reduces repeated encoding. The reuse ablation lowers latency from roughly 33 to 30 seconds per sample, at an increased memory cost (Table 4).
-
Results are consistent across six benchmarks. Hierarchical variants outperform flat variants under both SFT and RL, and the reward ablation supports the usefulness of evidence-seeking supervision (Tables 2–3).
Critical weaknesses
-
The hierarchy’s benefit is not isolated from better frame selection. Appendix E.3 allows up to 4,096 candidate frames despite the 128-frame language-model budget, while Table 2 changes selection alongside structure. A matched comparison with VideoZoomer plus TSPO, or flat search using the same frame selector, is needed. This missing control is my main concern.
-
The reward measures requested-interval overlap, not whether retrieved visual states retain the evidence. Node mapping and pruning can still omit brief decisive cues (Appendix C). The accuracy ablation and successful examples do not establish how often this occurs; evidence retention after retrieval needs evaluation on held-out videos.
Questions for authors
-
What initial candidate-frame pools do VideoZoomer, VideoTree, and the flat variants access? In Table 2, which aspects besides hierarchical organization change, including frame selection and search supervision?
-
How does HieraSeek compare with VideoZoomer plus TSPO under matched candidate sampling, visual context, and end-to-end compute accounting? A flat search variant using the same representative-frame selection would also help isolate the hierarchy’s contribution. This comparison could materially change my recommendation.
-
On held-out videos with temporal evidence annotations, how often do the frames represented by the returned nodes cover the supporting evidence? Does successful interval localization remain successful after node mapping and pruning, particularly for brief events?
I am open to increasing my score if the authors can provide convincing responses to the above questions and sufficiently address my concerns.