acceptodds
Under review as a conference paper at ICLR 2027

Active Visual Perception and Reasoning for Long Video Understanding

Abstract

Long-video understanding requires identifying sparse, query-relevant evidence from a large amount of visually redundant content, making exhaustive video processing both inefficient and unnecessary. We present QSTAR, a Query-dependent Spatial-Temporal Active Reasoning framework for efficient long-video understanding without retraining. QSTAR casts video understanding as an iterative process of active perception, in which the model progressively concentrates its limited visual budget on moments that are most informative for answering the query. Starting from sparsely sampled frames, QSTAR exploits the VLM's internal visual reasoning signals to estimate query-dependent frame saliency and propagates these observations over the video timeline to refine a temporal sampling distribution. Successive rounds adaptively update the sampling distribution until the change between consecutive distributions falls below a threshold, after which a small set of informative frames is used for final multimodal reasoning. We further explore complementary sampling strategies and diversity-aware model fusion to improve robustness when different perceptual cues provide complementary evidence. We provide finite-round guarantees on reconstruction and induced sampling-distribution error. Extensive experiments on five video-understanding benchmarks, against 17 representative open- and closed-source VLMs and 10 methods specialized for long-video understanding, show that QSTAR consistently improves reasoning accuracy at substantially lower end-to-end latency. These results demonstrate that actively deciding where to look, conditioned on the query and the model's evolving reasoning state, provides an effective alternative to exhaustive long-video processing. Code is available at https://anonymous.4open.science/r/QSTAR.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video understanding, multimodal, visual perception

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Uvh9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Authors propose a training-free query-dependent frame selection framework for long video understanding. They first feed uniformly sampled input frames to a pre-trained VLM and select good frames based on predicted answer alignment against image patch embeddings. This is calculated with normalization against model context prior to predicted answer tokens. Each following round adjusts the frame selection based on the parts of the video that result being most informative in the previous round.

This process is applied iteratively, and stopped when KL divergence between successive distributions is below a threshold. A final VLM call answers the question using the selected frames. Authors evaluate on five video QA benchmarks and demonstrate consistent improvements and strong results.

Critical strengths
  • Interesting approach for frame selection without explicit key-frame supervision.
  • Extensive evaluations across benchmarks to establish SOTA performance.
  • Thorough ablations and comparisons against alternate variants of method.
Critical weaknesses
  • Compare against highly similar prior work ReQuest: Rethinking-based Question-Aware Frame Selection for Long-Form Video QA from ECCV 2026. They build a similar query-dependent frame selection approach using model outputs requiring no key-frame supervision. Clearly discuss novelty of "activation-based frame scoring" contribution over such prior work.

  • Applicability in domains where base model performance is weak remains unclear. Consider discussing significance of base model accuracy for this method to work. Particularly, how important is base model accuracy for iterative refinement to provide meaningful improvements?

  • Generality of interpolation for videos with significant scene changes could be separately evaluated to further establish methods boundaries of operation.

  • Presentation could be much improved. Figures are too complex and very hard to read/understand.

  • Minor: Please fix citations to use \citep when appropriate.

Questions for authors

Interesting paper with clear contributions and strong results. Please clarify novelty against ReQuest.

Reviewer_TLc7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents QSTAR, a training-free framework for query-adaptive frame selection in long-video understanding. Instead of processing a video densely or relying only on an external retrieval model, QSTAR iteratively allocates a limited visual budget to frames that appear useful for answering the current query. Starting from uniformly sampled frames, it generates a provisional answer and estimates frame saliency by projecting final-layer visual-token hidden states onto the LM-head directions of meaningful answer tokens. A context-residualization step is used to suppress activation that can be explained by the prompt and previously generated tokens. The resulting sparse frame scores are propagated over the video timeline using a one-dimensional Gaussian process, and the temporal sampling distribution is updated until the KL divergence between consecutive distributions falls below a threshold. Overall, QSTAR provides a practically useful framework for allocating test-time computation. However, two issues remain insufficiently validated: whether it can recover isolated sparse evidence missed during initial sampling, and whether its gains over the backbone persist under matched total-frame and compute budgets.

Critical strengths
  1. The use of internal VLM signals for frame selection is technically interesting. The answer-conditioned activation score is more closely connected to the backbone model's current reasoning state than external image-text similarity. Projecting visual-token representations onto the actual LM-head directions of generated answer tokens provides a principled representation-alignment signal, while the residualization step is a thoughtful attempt to remove activation attributable mainly to the textual context.
  2. QSTAR is training-free and can be practically deployed on compatible open-source VLMs. It does not require fine-tuning the backbone, training a separate selector, or collecting additional supervision, and can be applied as a test-time wrapper around models that expose final-layer hidden states and the LM head.
  3. The empirical evaluation covers five benchmarks and multiple backbone VLMs. The paper also includes useful component-level comparisons of activation-, attention-, and CLIP-based scoring; GPR and several interpolation alternatives; different frame budgets and stopping conditions; CLIP encoders; fusion strategies; and structured long-video baselines. These experiments help clarify the behavior of individual design components, even though the headline backbone comparison is not fully compute-matched.
  4. The appendix is transparent about failure modes. It explicitly examines failures caused by extremely short evidence, weak temporal continuity, backbone recognition errors, duplicated frames, and answer-conditioned error reinforcement. The answer-transition analysis quantifies both wrong-to-correct and correct-to-wrong changes rather than reporting only aggregate improvements.
Critical weaknesses
  1. The main backbone comparison does not match total test-time computation. Table 1 fixes only the per-call frame budget, while the direct InternVL3-78B baseline invokes the model once and QSTAR invokes it approximately 3.00 times on NeXT-QA and 3.73 times on MLVU. Consequently, the baseline processes only 16/32 VLM frames per question, whereas QSTAR processes approximately 48/119.36 cumulative VLM frames, in addition to performing multiple provisional generations. The reported improvement therefore conflates adaptive frame selection with additional test-time computation. A one-shot backbone using the same cumulative number of uniformly sampled frames would be an important missing baseline, but equal frame count alone is not sufficient because one long VLM call and multiple short calls have different attention and decoding costs. The paper should additionally compare against a multi-call uniform or stratified sampler with the same number of VLM calls and the same final aggregation procedure, as well as equal-FLOPs and equal-latency operating points. Reporting cumulative frames, unique frames, visual tokens, and total VLM calls would make the source of the gain substantially clearer. Figure 5 acknowledges that QSTAR uses more PFLOPs than direct inference at the same per-call K, while Table 3 compares different default operating points rather than strictly matched budgets; neither experiment fully resolves this concern.

  2. Although QSTAR is not permanently restricted to the initial K frames and can select new frames from the full timeline in later rounds, only sampled frames receive actual activation-based observations. Scores at unseen locations are inferred from the GPR posterior mean. Consequently, recovery from an initial miss relies heavily on temporal smoothness: if the decisive event is extremely short and isolated, and neighboring sampled frames contain no predictive cues, the method has no reliable signal indicating where to search next. This issue is exacerbated by the fact that QSTAR does not use GPR posterior uncertainty, employs deterministic inverse-transform sampling without an explicit cross-round novelty constraint, and uses a pre-normalization score floor of 10^-12, which may assign only negligible probability mass to low-scoring or unexplored regions. The sampler may therefore repeatedly exploit an incorrectly estimated high-saliency region rather than explore unobserved portions of the video. Moreover, even when a critical frame is observed once, the method does not explicitly guarantee that this evidence will be retained in the final frame set. The paper's own 8-minute-14-second failure case, in which the decisive evidence lasts only about 1.5 seconds and a uniform 32-frame sample has only about a 9% chance of covering the target frame, directly illustrates this limitation. The theoretical evidence-hit result is conditional on the estimated distribution already being sufficiently close to the latent utility distribution and on the evidence block having mass greater than 1/K; it does not establish recovery after missing an isolated low-mass event.

Questions for authors
  1. How often does QSTAR miss the decisive evidence in the first round but successfully recover it in later rounds? Table 13 reports wrong-to-correct answer transitions, but it does not isolate recovery from an initial evidence miss. Among examples where the decisive evidence is not covered in the first round, what fraction successfully recover that evidence in later rounds?
  2. Can the authors report the number of unique frames and the cross-round revisit rate, rather than only cumulative VLM frames? These statistics would clarify whether QSTAR continues to explore new temporal regions or repeatedly exploits the same high-saliency frames, and how much genuinely new visual information is obtained from the additional test-time computation.
  3. Since GPR already provides posterior variance, why does QSTAR not use it? Could uncertainty-driven exploration help recover extremely short, isolated evidence missed by the initial sample?
  4. Does QSTAR guarantee that high-saliency frames discovered in intermediate rounds are retained in the final frame set? If not, how often are such frames subsequently discarded, particularly when they contain decisive evidence?
  5. How does QSTAR’s performance change as the evidence duration decreases, particularly for isolated events with no informative neighboring context? Could the authors conduct a controlled needle-in-a-haystack stress test to evaluate it?
  6. Please compare QSTAR against one-shot InternVL3-78B using the same cumulative number of VLM frames: approximately 48 frames on NeXT-QA and 119/120 frames on MLVU under the reported average stopping behavior.
  7. Can the authors provide a multi-call uniform or stratified sampling baseline with the same number of VLM calls, the same per-call K, and the same final-answer and aggregation procedure as QSTAR? This would better isolate the contribution of adaptive sampling from generic test-time scaling.
  8. Please report results at matched total FLOPs or matched end-to-end latency, together with cumulative and unique frames, visual tokens, and VLM calls. How much of QSTAR’s gain remains under these matched-compute settings?
  9. Please clarify the exact source and evaluation protocol of the unstarred InternVL3-78B results in Table 1, including the checkpoint, uniform-sampling implementation, prompt, decoding configuration, subtitle usage, and answer parsing. It would also be helpful to report the corresponding official or leaderboard results alongside the controlled reproduction.
Reviewer_zqXaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper describes QSTAR, a multi-turn frame sampling refinement method for improving training-free long video understanding for vision language models (VLMs). QSTAR consists of three phases, with two being in focus in the main paper while a multi-method fusion strategy is explored more in the appendix. The first two phases involve using the intrinsic, query specific estimate by the VLM of frame importance for a sampling a small number of frames and a Gaussian Process Regressor to iteratively refine this importance estimate towards sampling an informative subset of frames. Some heuristics are used to avoid degenerate cases of frame importance measurement. The third phase allows for fusing query importance estimates from multiple methods. QSTAR is extensively evaluated across 5 video reasoning benchmarks using InternVL3-78B as the base model and outperforms most baselines. In addition, most components of the sampling algorithm are ablated towards conclusively stable parameter choices. Further some latency analysis shows that this multi-turn strategy does well in comparison to typical baseline settings.

I lean positive on this paper since it is well-written and has reasonably detailed analysis. I have some questions/concerns which I point below around some analysis and generalization. Assuming these concerns are addressed, I will update my score accordingly, though I have current marked it as leaning reject.

Update: After extensive discussion with the authors, my major concern has been assuaged, so I increased my score and lean towards acceptance.

Critical strengths
  1. QSTAR uses a query-specific intrinsic evaluation which allows it to be applicable across different models.
  2. The intermediate steps for iterative processing are lightweight, reducing the overall latency even at high accuracy.
Critical weaknesses
  1. Missing analysis: The appendix provides extensive ablations for parameter settings, but is missing analysis for convergence across iterations. Further, while latency and total frames/helper frames are analyzed, it is unclear how the latency is splits across the initial iterations, GPR estimator, and final response.
  2. Some terms are not defined. Eg. ITS without replacement, w(am)w(a_m) in lines 257-258. These may be standard but not clearly defining them breaks the linearity of reading.
Questions for authors

Q1: How much do successive iterations contribute to refinement, and how many iterations does it take across tasks to get to the optimal level?

Minor: Sections in the appendix are labeled A-G but the numbering in the introduction (lines 704-716) shows 1-7.

Comments on the paper

2 comments

We thank the reviewer for the constructive follow-up suggestions.

Regarding the definitions, which was commented by Reviewer TLc7 (rating of 3: weak accept), we have responded to that reviewer and also have uploaded a revised manuscript clarifying the duplicate-frame handling in Lines 215–216 and explicitly defining rmr_m and w(am)w(a_m) in Line 257.

Regarding the generalization of KL-based τ\tau stopping, we would like to make two important notes about QSTAR configuration:

First, KL divergence measures sampling-distribution stability rather than directly guaranteeing answer accuracy. Below we provide additional round-wise ablations on the three benchmarks, following the request by this reviewer zqXa:

Benchmarkr=1r=2r=3r=4r=5KL ruleAvg. rounds
EgoSchema84.084.885.084.285.885.22.51
LongVideoBench63.865.465.466.266.265.43.67
Video-MME (w/o subtitles)68.069.071.373.672.372.23.25
Video-MME (w/ subtitles)77.279.779.380.379.880.03.21

The results for these three benchmarks are consistent with the two other benchmarks provided in our initial rebuttal. All five benchmark results consistently demonstrate that this KL τ\tau stability criterion provides an effective early-stopping signal, achieving accuracy close to the best fixed-round settings with fewer rounds on average.

Another reason that accuracy does not necessarily increase monotonically with additional rounds, as observed by Reviewer zqXa, is due to the fact that newly sampled frames may change the model's prediction outcome, which is expected. Hence, the use of τ\tau KL divergence to measure and control the sampling-distribution stability is an important convergence optimization.

Secondly, it is also important to note that QSTAR sets the same τ=10−2\tau=10^{-2} across all five benchmarks without dataset-specific tuning. The threshold was selected to balance refinement quality and computational cost. As shown in Table 1 (page 8), even with this uniform KL τ\tau setting, QSTAR consistently improves backbone accuracy by 3.3 ~ 9.8 points, and Table 4 further demonstrates improvements across different VLM backbones using the same τ\tau threshold.

We sincerely hope that the these additional experimental results address the reviewer's concern about τ\tau-KL generalization.

We once again thank Reviewer zqXa for the constructive comments.

The authors

Reviewer_zqXaOpenReview ↗

Response to Rebuttal Response Comment

Thank you to the authors for their response to my question. Yes, it was clear that KL divergence measures stability, not increase. My comment was more about why stability ensured convergence to a high accuracy. In the provided results, a fixed r=4r=4 seems to outperform the KL rule in 3 out of 4 cases. The KL does reduce the latency by keeping the average number of rounds lower, but any distribution distance would probably perform similarly.

The stability of the KL-threshold across the datasets is more convincing evidence. Given this, I have decided to increase my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.