- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Selective Temporal Reacquisition (STR), a test-time method for long-video understanding with frozen MLLMs. Starting from uniformly sampled frames, STR uses predictive entropy and attention-derived temporal guidance to decide whether and where to reacquire visual evidence under a fixed per-observation frame budget. STR-Refine extends this procedure to multiple observations, while STR-Lite uses a learned router to prune tokens from reacquired views. The paper evaluates these methods on five video-understanding benchmarks, reports transfer across four backbones, and provides additional streaming, evidence-localization, and runtime analyses.
Critical strengths
-
A simple and practically appealing core method. STR reuses the frozen answerer’s uncertainty and attention without requiring backbone fine-tuning or a separate temporal-selection model. Converting an existing answer attempt into a proposal for further observation is an economical design that could be useful for deploying existing MLLMs.
-
Broad empirical validation with a useful evidence-localization control. The experiments cover five benchmarks, four backbones, and an additional streaming application. The temporal-F1 comparison against uniform resampling under a shared uncertainty-based stopping rule provides evidence that targeted reacquisition improves evidence localization. These results support the practical effectiveness of the overall framework.
Critical weaknesses
-
The conceptual advance is limited, and the distinctive design choices are insufficiently validated. Adaptive evidence acquisition has substantial precedent. VideoAgent [1] iteratively gathers information to answer video questions; Temporal Chain of Thought [2] uses the VLM itself to iteratively select relevant frames; and A.I.R. [3] provides training-free adaptive and iterative frame selection. Relevance-based inverse-transform sampling was also explored in BOLT [4]. STR’s direct reuse of the answerer’s attention, joint uncertainty–deformation gate, and fixed per-observation budget are useful distinctions. However, the current evidence does not sufficiently establish the independent contribution of the proposed “repairable uncertainty” criterion. In particular, the paper does not compare the joint gate with uncertainty-only routing at a matched reacquisition budget. It also specifies attention readout layers, an entropy threshold of 0.15 nats, a deformation threshold of 0.025, and a uniform-mixture weight of 0.1, without explaining their selection procedure or reporting sensitivity. Cross-backbone results provide encouraging transfer evidence, but leave unclear how robust the policy is to these choices. These gaps limit confidence that the gains reflect a generally reliable repairability criterion rather than a particular configuration of attention-guided repeated observation.
-
The efficiency evaluation does not establish the accuracy–latency trade-off specifically for long videos or for the strongest variants. Table 1 reports long-video accuracy, but the runtime analysis aggregates a 30-question panel containing ten questions from each duration group. Figure 4 combines this panel’s mean latency with full-benchmark accuracy, without separately reporting latency and reacquisition rates for the long-video subset. Since decoding costs and the need for further observations may vary with duration, the aggregate results do not establish whether the reported efficiency advantage holds for long videos. Furthermore, the largest accuracy improvements come from STR-Refine, whereas the reported latency applies to STR/STR-Lite. Appendix C explicitly states that its panels do not establish matched online stopping-policy latency for Refine. The actual computational cost of the strongest results therefore remains insufficiently characterized. This matters because efficient long-video understanding is a central motivation, and a fixed per-observation frame budget does not imply a matched total computation budget.
References
[1] Xiaohan Wang et al. VideoAgent: Long-form Video Understanding with Large Language Model as Agent. ECCV, 2024.
[2] Anurag Arnab et al. Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames. NeurIPS, 2025.
[3] Yuanhao Zou et al. A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering. ICLR, 2026.
[4] Shuming Liu et al. BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding. CVPR, 2025.
Questions for authors
-
Clarification and targeted validation of parameter choices: How were the attention readout layers, , , and selected? Please specify the data and objective used for selection and whether these settings were fixed before benchmark evaluation. Could a limited sensitivity analysis show whether nearby layers and parameter values preserve the accuracy gains and reacquisition rates?
-
Targeted validation of the repairability criterion: At a matched average reacquisition budget, how does the joint uncertainty–deformation gate compare with uncertainty-only routing? Reporting accuracy and the numbers of corrected and regressed answers would help establish whether deformation provides useful information beyond predictive uncertainty.
-
Clarification and targeted validation of long-video efficiency: Could the authors disaggregate the existing runtime panel by video duration, reporting latency and average observation count alongside the corresponding accuracy results? Please also clarify the measured end-to-end cost of STR-Refine and STR-Refine-Lite under their actual stopping policies. This would help establish whether the efficiency claims hold for long videos and for the variants yielding the strongest accuracy improvements.
-
Clarification of the evidence-localization evaluation: For Figure 5(a), what are the absolute precision, recall, and F1 scores underlying the reported relative improvements? Please also specify the number of evaluated questions and videos and the temporal matching tolerance. These details are needed to interpret the practical magnitude of the reported 61.8% relative F1 improvement.
-
Targeted validation of STR-Lite: At the deployed 60% token-retention ratio, how does the proposed router compare with simple coverage-preserving token selection or representative existing visual-token pruning methods under the same token budget? Could the authors also isolate the contribution of the second-stage temporal adaptation at this retention ratio? The current 10%-retention stress test is informative, but does not clarify the benefits of the specialized router and its adaptation stage under the default deployment setting.
6 comments
Author Response (2/4)
Q1
Clarification and targeted validation of parameter choices
None of , , , , or the attention readout layer were tuned on the evaluation benchmarks. We selected these hyperparameters on a separate internal development dataset, consisting of 120 video-question pairs with human-annotated answers, with no overlap with the evaluation benchmarks. We cannot disclose its identifying information at this stage without compromising double-blind anonymity. We will provide these details in the camera-ready version upon acceptance.
Hyperparameter robustness is a strength of STR. The hyperparameters of STR are based on the properties of MLLMs and are therefore intuitive to set and straightforward to tune. For example, deeper MLLM layers are widely recognized as providing better semantic localization, so any sufficiently deep layer can serve as the attention readout [1]. Our readout-layer study demonstrates this behavior. The same goes for other hyperparameters; additional sensitivity experiments below further demonstrate this robustness, showing nearly flat accuracy curves across broad parameter ranges.
We used the same hyperparameter values across five datasets covering approximately 10,339 video–question pairs, spanning diverse video understanding tasks, from egocentric reasoning to needle-in-a-haystack retrieval.
1. Attention Readout Layer
| Layer | LongVideoBench | VideoMME |
|---|---|---|
| 24 (Paper) | 66.2 | 69.4 |
| 25 | 66.6 | 69.7 |
| 26 | 65.2 | 69.3 |
| 27 | 65.0 | 69.8 |
| 28 | 65.8 | 69.2 |
| 29 | 65.7 | 69.5 |
2. Predictive Uncertainty Threshold ()
We use a shared numerical uncertainty threshold, setting nats in the default configuration. The superscript ''exit'' distinguishes its role in STR-Refine’s stopping rule; it is not independently tuned.
| (nats) | LongVideoBench acc. | VideoMME acc. | VideoMME calls/q |
|---|---|---|---|
| 0.05 | 66.4 | 69.7 | 1.42 |
| 0.075 | 66.2 | 69.6 | 1.40 |
| 0.10 | 66.3 | 69.4 | 1.39 |
| 0.15 (Paper) | 66.2 | 69.4 | 1.33 |
| 0.20 | 66.2 | 69.2 | 1.31 |
| 0.30 | 66.1 | 68.9 | 1.27 |
| 0.50 | 65.7 | 68.5 | 1.21 |
| 1.00 | 64.8 | 67.8 | 1.06 |
3. Attention Deformation Threshold ()
For context, the attention-deformation measure ranges from approximately 0 to 0.15 in our experiments, reflecting the behavior of the underlying MLLM.
| LongVideoBench acc. | VideoMME acc. | VideoMME calls/q | |
|---|---|---|---|
| 0.01 | 66.2 | 69.5 | 1.44 |
| 0.02 | 66.0 | 69.5 | 1.40 |
| 0.025 (Paper) | 66.2 | 69.4 | 1.33 |
| 0.03 | 66.5 | 69.1 | 1.31 |
| 0.04 | 66.1 | 68.6 | 1.23 |
| 0.05 | 65.4 | 68.2 | 1.15 |
| 0.10 | 63.7 | 67.6 | 1.01 |
4. Resampling Parameter ()
| LongVideoBench | VideoMME | |
|---|---|---|
| 0 | 66.7 | 69.4 |
| 0.05 | 66.3 | 68.7 |
| 0.1 (Paper) | 66.2 | 69.4 |
| 0.2 | 66.3 | 69.1 |
| 0.3 | 66.6 | 68.8 |
These hyperparameters primarily control the trade-off between accuracy gains and computational cost. The uncertainty and deformation thresholds determine when the repairability gate triggers additional evidence acquisition, while controls the resampling distribution. At sufficiently stringent gating thresholds, no reacquisition occurs and STR returns the initial uniform-sampling answer. Uniform sampling is therefore a limiting case of STR. More permissive gating allows additional opportunities to address repairable uncertainty, while more stringent gating reduces the number of correction calls and improves efficiency.
[1] Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs, ICLR 2026
Author Response (3/4)
Q2
Targeted validation of the repairability criterion: At a matched average reacquisition budget, how does the joint uncertainty–deformation gate compare with uncertainty-only routing?
Adding attention deformation improves efficiency by reducing reacquisition calls without any loss in accuracy. Attention deformation helps determine whether an uncertain prediction is likely to benefit from additional observations. When predictive uncertainty is high, high deformation indicates that attention is concentrated on specific temporal regions, suggesting that the model has identified partial evidence that can guide targeted resampling. In contrast, low deformation provides little direction for targeted resampling, so the joint gate avoids further calls. We evaluate this by comparing joint uncertainty–deformation gate with, uncertainty-only routing, and budget-matched uncertainty-only routing on VideoMME. Budget-matched uncertainty-only is obtained by increasing the uncertainty threshold to reduce the number of calls so that Total calls/q matches joint uncertainty–deformation gate.
| Variant | Total calls/q ↓ | Accuracy (%) ↑ |
|---|---|---|
| Uncertainty-only | 1.49 | 69.4 |
| Uncertainty-only (budget-matched) | 1.33 | 68.3 |
| Uncertainty + Deformation | 1.33 | 69.4 |
The repairable uncertainty gate (uncertainty + deformation) reduces reacquisition calls by 32.7% compared with uncertainty-only routing during the STR reacquisition call.
At a matched budget of 1.33 calls per question, the joint uncertainty–deformation gate outperforms uncertainty-only routing by 1.1 percentage points (69.4% vs. 68.3%). This demonstrates that attention deformation provides useful routing information beyond predictive uncertainty.
Q3
Efficiency of STR-Refine and STR-Refine-Lite for Long Videos.
STR-Refine’s and STR-Refine-Lite are computationally efficient for long videos. We report the average observation count and latency under each variant’s stopping policy on all 2,700 Video-MME questions:
| Method | Passes per Question | Seconds per Question | × One Uniform Pass |
|---|---|---|---|
| Uniform, one pass | 1.0 | 3.0 | 1.00× |
| STR | 1.3 | 3.6 | 1.2× |
| STR-Refine | 1.9 | 4.5 | 1.5× |
| STR-Refine-Lite | 1.9 | 4.1 | 1.4× |
The duration-wise breakdown below shows STR-Refine’s accuracy, observation count, and latency. The average observation count increases from 1.6 for short videos to 2.2 for long videos. For long videos, the latency is 5.1 seconds, corresponding to 1.7× the uniform-pass reference.
| Duration | Questions | Accuracy (%) | Passes per Question | Latency (s) | × One Uniform Pass |
|---|---|---|---|---|---|
| Short | 900 | 79.9 | 1.6 | 3.7 | 1.2× |
| Medium | 900 | 71.1 | 2.0 | 4.7 | 1.6× |
| Long | 900 | 59.3 | 2.2 | 5.1 | 1.7× |
| All | 2,700 | 70.1 | 1.9 | 4.5 | 1.5× |
Q4
Clarification of the evidence-localization evaluation: For Figure 5(a), what are the absolute precision, recall, and F1 scores underlying the reported relative improvements? Please also specify the number of evaluated questions and videos and the temporal matching tolerance. These details are needed to interpret the practical magnitude of the reported 61.8% relative F1 improvement.
We follow T*/LV-Haystack’s released evaluation protocol for filtering LongVideoBench’s videos and excluding subtitle-dependent questions. After filtering based on T*/LV-Haystack’s evaluation protocol the evaluation contains 342 questions across 114 videos. Temporal matching uses a ±5-second tolerance (again standard T*/LV-Haystack’s protocol) around each annotated keyframe. The absolute precision, recall, and F1 scores underlying Figure 5(a) are:
| Cap | Arm | Precision (%) | Recall (%) | F1 (%) | Relative F1 improvement (%) |
|---|---|---|---|---|---|
| 1 | both | 1.4 | 50.5 | 2.7 | 0.0 |
| 2 | uniform | 1.3 | 47.6 | 2.6 | — |
| 2 | STR-Refine | 1.7 | 54.2 | 3.3 | +26.3 |
| 3 | uniform | 1.4 | 49.9 | 2.7 | — |
| 3 | STR-Refine | 2.1 | 54.7 | 4.0 | +46.1 |
| 4 | uniform | 1.4 | 51.5 | 2.7 | — |
| 4 | STR-Refine | 2.4 | 55.0 | 4.5 | +61.8 |
The low absolute precision is a consequence of the sparse-keyframe evaluation protocol: precision is calculated over all 64 sampled frames, despite having only a one/two annotated evidence keyframes per question. For example, matching one annotated keyframe with one sampled frame yields only 1.56% precision. We therefore use relative F1 improvements.
STR-Refine improves both precision and recall, demonstrating that attention-guided reacquisition increases evidence concentration without sacrificing coverage. The increasing gains demonstrate progressively more effective evidence localization.
Author Response (1/4)
W1
STR is the first method to identify and use repairable uncertainty: a frozen MLLM can remain uncertain about its answer while its first-pass temporal attention provides useful guidance on where to reacquire missing evidence. STR turns this distinction into a self-correction policy: the initial answer pass itself supplies the signals for deciding whether another observation is warranted and where that observation should concentrate. This is the conceptual advance, with the answerer's attention, joint uncertainty–deformation gate, and fixed per-observation budget as method design.
The two categories raised in the comment are already addressed in our related-work paragraphs, “Adaptive evidence acquisition” (L110–125) and “Query-conditioned frame selection” (L127–140), where VideoAgent, Temporal Chain of Thought, and BOLT are explicitly discussed. VideoAgent, Temporal Chain of Thought, and A.I.R. are prompt-based, agentic evidence-acquisition methods that rely on explicit VLM reasoning and frame-selection steps to identify relevant video evidence, making them extremely expensive. STR instead obtains both the reacquisition decision and the temporal sampling guidance directly from the initial answer pass, without any additional prompt-based frame-selection step. BOLT is not only discussed in our related work but also evaluated as a main baseline in Table 1. Our evaluation demonstrates substantial improvements in both accuracy and efficiency over BOLT on all five benchmark-level accuracies (Table 1, Figure 4).