acceptodds
Withdrawn from ICLR 2027

HieraSeek: Learning to Search Visual Hierarchies for Long-Video Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable progress in video understanding, yet long-video reasoning remains challenging because task-relevant evidence is sparse and easily obscured by redundant visual content. Existing methods typically compress long videos into fixed representations or iteratively search over flat temporal sequences, which can limit effective evidence discovery. We propose **HieraSeek**, a hierarchical framework that constructs a reusable multi-granularity visual representation directly within the ViT encoder through progressive frame pruning and temporal merging, and exposes this representation as the search space for multi-round reasoning. This shared latent structure tightly couples representation construction with evidence acquisition, avoiding a separate retrieval or memory pipeline during reasoning. A learned policy iteratively refines or relocates its temporal focus, while reusing intermediate visual states to reduce repeated computation. We further optimize the search policy with an evidence-seeking reinforcement learning objective. Across six long-video understanding and reasoning benchmarks, HieraSeek achieves the best performance on all reported metrics, outperforming strong multi-round baselines under the same visual budget while maintaining competitive inference efficiency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Long-Video Understanding, Long-Video Reasoning, Visual Search, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vC48OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HieraSeek treats long-video reasoning as repeated access to a hierarchy built inside the visual encoder. Early ViT blocks progressively select representative frames and merge adjacent temporal groups, while cached intermediate states let the model revisit regions without encoding them again from scratch. A policy trained through SFT and reinforcement learning chooses whether to narrow its focus, relocate, or answer. Evaluation covers six benchmarks, with ablations on representation structure, rewards, and state reuse.

Critical strengths
  1. Reusing intermediate ViT states connects hierarchy construction with evidence search and reduces repeated encoding. The reuse ablation lowers latency from roughly 33 to 30 seconds per sample, at an increased memory cost (Table 4).

  2. Results are consistent across six benchmarks. Hierarchical variants outperform flat variants under both SFT and RL, and the reward ablation supports the usefulness of evidence-seeking supervision (Tables 2–3).

Critical weaknesses
  1. The hierarchy’s benefit is not isolated from better frame selection. Appendix E.3 allows up to 4,096 candidate frames despite the 128-frame language-model budget, while Table 2 changes selection alongside structure. A matched comparison with VideoZoomer plus TSPO, or flat search using the same frame selector, is needed. This missing control is my main concern.

  2. The reward measures requested-interval overlap, not whether retrieved visual states retain the evidence. Node mapping and pruning can still omit brief decisive cues (Appendix C). The accuracy ablation and successful examples do not establish how often this occurs; evidence retention after retrieval needs evaluation on held-out videos.

Questions for authors
  1. What initial candidate-frame pools do VideoZoomer, VideoTree, and the flat variants access? In Table 2, which aspects besides hierarchical organization change, including frame selection and search supervision?

  2. How does HieraSeek compare with VideoZoomer plus TSPO under matched candidate sampling, visual context, and end-to-end compute accounting? A flat search variant using the same representative-frame selection would also help isolate the hierarchy’s contribution. This comparison could materially change my recommendation.

  3. On held-out videos with temporal evidence annotations, how often do the frames represented by the returned nodes cover the supporting evidence? Does successful interval localization remain successful after node mapping and pruning, particularly for brief events?

I am open to increasing my score if the authors can provide convincing responses to the above questions and sufficiently address my concerns.

Reviewer_du5pOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

HieraSeek builds a temporal hierarchy inside the early ViT blocks of Qwen2.5-VL-7B and uses it for iterative long-video question answering. The agent can zoom into or shift between intervals, reusing cached visual states rather than encoding the video again. It is trained with teacher-generated SFT trajectories followed by Dr. GRPO with accuracy and evidence-seeking rewards. The paper reports best or tied-best results among budget-matched methods on six benchmarks under a cumulative 128-frame budget, with representation and reward ablations and an inference-cost comparison.

Critical strengths
  1. Building the search hierarchy within the visual encoder is a useful idea. It connects temporal search with state reuse instead of introducing a separate retrieval representation. Table 2 gives direct support for the hierarchy: under SFT it improves over Flat on all six benchmarks, by 1.3–2.9 points. This is the most convincing evidence for the proposed representation.

  2. The evaluation covers six benchmarks and includes comparisons of representation, training objective and inference cost. The reported reduction in output tokens and latency relative to VideoZoomer is encouraging (207.26 vs. 412.57 tokens; 29.69 vs. 35.74 seconds, Table 4). The accompanying SFT, RL and evaluation code is also valuable, even though the configurations below need to be reconciled with the results.

Critical weaknesses
  1. The training setup behind the results is unclear.

The documented RL launcher defaults to accuracy-only training. Enabling the navigation reward does not recover Eqs. 8–9: the code omits c(T), uses discount 1.0 rather than 0.8, adds a terminal score and weights the reward by 1.0 rather than 0.5. It also gives no navigation reward to incorrect answers. No submitted configuration implements the full objective, leaving the full versus "w/o c(T)" comparison in Table 3 unexplained. The launcher defaults and wrapper also do not identify the SFT-initialised CG-Bench+LongVideoReason run described in App. E.4.

The SFT recipe has a related gap. Four model-control generators include the gold answer in every round's prompt, while the base generator uses answer-hinted recovery of wrong-answer trajectories. App. E.1 describes evidence-interval hints, but the submitted materials do not say how much of the SFT set came from these answer-conditioned paths. The hierarchy defaults also differ: 8-slice groups keeping 4, with compression in blocks 0–3 at the maximum input, rather than n = 32 and six blocks. These differences leave both the reward ablation and the learned search behaviour difficult to interpret.

  1. The equal-budget comparison needs verification.

The paper limits total visual input across all rounds to 128 pre-merge frames, or 64 LLM-side temporal slices (Sec. 4.1 L362–366). The code instead caps each visual slot at 128 slices by default and includes the global slot alongside the local slot after a tool call. A single prompt can therefore theoretically contain 256 slices, and neither agent path maintains a cumulative visual budget. Actual per-sample counts are not reported. This is important for Table 1: in Fig. 4, increasing the backbone's budget to 256 frames brings its LVBench accuracy to approximately 41.5, close to HieraSeek's 42.1 at the stated 128-frame budget.

LVBench use also needs clarification. The archived launcher names LVBench training and validation files and enables best-checkpoint handling, but no reported checkpoint is linked to that configuration. Whether these files were used for training or selection remains unknown. Finally, three margins over VideoZoomer are only 0.0–0.2 points, with no variance estimates (Table 1). The reported scores are promising, but the current materials do not give enough confidence in the claimed comparison.

Questions for authors
  1. Which exact training commands, data and checkpoints produced Tables 1–3? How were the full and "w/o c(T)" rewards implemented, and which hierarchy settings were used?
  2. Where is the cumulative visual budget enforced? What are the mean and maximum total LLM-side slices for HieraSeek, the budget-matched baselines and the Flat variant?
  3. Were any LVBench examples used to train or select a reported checkpoint? What do the archived launcher's training and validation files contain?
  4. Which generators produced the reported SFT set, and how many retained trajectories used gold-answer prompts or answer-hinted recovery?
  5. How many independent runs were evaluated, and is there uncertainty or paired prediction data for the small Table 1 margins?
Reviewer_6U4QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

HieraSeek builds a temporal hierarchy inside the vision encoder by progressively selecting representative frames and merging adjacent groups. A policy uses zoom_in and shift_to actions to retrieve cached intermediate states at suitable temporal scales before answering. Training combines SFT with an evidence-seeking RL objective. The paper evaluates a Qwen2.5-VL-7B model on six video benchmarks and studies the hierarchy, policy training, reward, and reuse of visual states.

Critical strengths
  1. The hierarchy is constructed from the same visual states used for reasoning, so it can support repeated evidence access without a separate captioning or retrieval pipeline. Table 2 compares flat and hierarchical representations under both SFT and SFT+RL. This is a useful attempt to separate the representation and policy contributions.

  2. The paper reports both accuracy and computational costs. HieraSeek improves over VideoZoomer by 1.7 points on LongVideoBench and 1.2 points on Video-MME overall (Table 1). Within HieraSeek, state reuse reduces latency from 32.89 to 29.69 seconds while increasing peak memory from 40.39 to 45.51 GB (Table 4). Reporting this memory cost makes the practical trade-off easier to assess.

Critical weaknesses
  1. Equal language-model context does not establish equal access to the source video. Appendix E.3 explains that the cumulative limit of 128 frame equivalents applies to visual context passed to the language model, while hierarchy construction may start from 4,096 raw frames. The appendix discloses this distinction and includes encoder costs in the efficiency measurements. The remaining issue is whether the baselines and the Flat ablation have comparable raw-frame coverage and resolution. Section 4.3 describes Flat as uniformly sampled frames, without enough detail to establish this. A flat selection/cache baseline using the same initial candidate frames is needed to distinguish the benefit of hierarchical search from the benefit of denser initial observations.

  2. The supplied RL code differs materially from the reward used to explain the results. Equations 8-9, with the settings in Section 4.1, specify a discount of 0.8, the trajectory-length factor c(T), and an evidence-seeking weight of 0.5; Appendix E.4 excludes the final answer from the search reward. In the supplied custom_rewards/vl_agent.py, however, _compute_cgbench_time_reward sums round scores with discount=1.0 and adds a terminal-answer score. The calling compute_score function gates this term on final-answer correctness and adds it with weight 1.0. The README describes the same behavior. The LongVideoReason launcher also defaults USE_TIME_REWARD to False. These may be archived configurations, so this does not show that the reported experiments used them. It does prevent the attachment from establishing which objective produced Table 3, especially the ablation of c(T).

Questions for authors
  1. For Table 1 and the Flat/Hierarchy comparison in Table 2, how many raw frames are available to each method, at what resolution? Does the hierarchy retain its advantage over flat selection from the same candidate pool, with matched training and cumulative language-model visual budgets?

  2. Which code revision and launch configuration produced each row of Table 3? Please identify the implementation of the discount, c(T), and search-reward weight, and explain the differences from the supplied training branch. A corrected implementation linked to the reported runs would resolve this concern.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.