acceptodds
Under review as a conference paper at ICLR 2027

Repairable Uncertainty: Selective Temporal Reacquisition for Long-Video Understanding with MLLMs

Abstract

Multimodal large language models (MLLMs) can process only a limited number of frames from long videos due to their large context requirements. Existing methods typically rely on uniform sampling or a separate frame selector. Uniform sampling can miss brief but decisive events, while selector-based pipelines often rely on a weaker selector than the base MLLM, potentially discarding useful evidence before it reaches the answerer. We observe that under partial evidence, an MLLM can remain uncertain about the answer while its temporal attention is concentrated on a small set of query-relevant regions. We call this repairable uncertainty: the answer remains uncertain, but focused attention provides a signal for where the missing evidence may lie. We introduce Selective Temporal Reacquisition (STR), a lightweight test-time method that uses a frozen MLLM’s answer uncertainty and temporal relevance to decide whether and where to reacquire video evidence. Starting from a uniform grid of \(F\) frames, STR uses the MLLM’s temporal attention to construct a new sampling density, triggering a budget-matched second pass only when the answer is uncertain and attention is sufficiently focused. We further introduce STR-Refine, which gates repeated observations using predictive uncertainty and accumulates relevance across them, and STR-Lite, which reduces the visual-token cost of reacquisition. Across five benchmarks (LongVideoBench, VideoMME, MLVU, VideoVista, and Video-MME-v2) and four MLLMs (Qwen3-VL-8B, Qwen3-VL-2B, Qwen3-VL-32B, and InternVL3-8B), STR consistently improves long-video understanding and achieves state-of-the-art performance. These results establish STR as a test-time framework for selectively reacquiring missing evidence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal LLMs, Vision-Language Models, Video Understanding, Long Video

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_uMfDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Selective Temporal Reacquisition (STR), a test-time method for long-video understanding with frozen MLLMs. Starting from uniformly sampled frames, STR uses predictive entropy and attention-derived temporal guidance to decide whether and where to reacquire visual evidence under a fixed per-observation frame budget. STR-Refine extends this procedure to multiple observations, while STR-Lite uses a learned router to prune tokens from reacquired views. The paper evaluates these methods on five video-understanding benchmarks, reports transfer across four backbones, and provides additional streaming, evidence-localization, and runtime analyses.

Critical strengths
  1. A simple and practically appealing core method. STR reuses the frozen answerer’s uncertainty and attention without requiring backbone fine-tuning or a separate temporal-selection model. Converting an existing answer attempt into a proposal for further observation is an economical design that could be useful for deploying existing MLLMs.

  2. Broad empirical validation with a useful evidence-localization control. The experiments cover five benchmarks, four backbones, and an additional streaming application. The temporal-F1 comparison against uniform resampling under a shared uncertainty-based stopping rule provides evidence that targeted reacquisition improves evidence localization. These results support the practical effectiveness of the overall framework.

Critical weaknesses
  1. The conceptual advance is limited, and the distinctive design choices are insufficiently validated. Adaptive evidence acquisition has substantial precedent. VideoAgent [1] iteratively gathers information to answer video questions; Temporal Chain of Thought [2] uses the VLM itself to iteratively select relevant frames; and A.I.R. [3] provides training-free adaptive and iterative frame selection. Relevance-based inverse-transform sampling was also explored in BOLT [4]. STR’s direct reuse of the answerer’s attention, joint uncertainty–deformation gate, and fixed per-observation budget are useful distinctions. However, the current evidence does not sufficiently establish the independent contribution of the proposed “repairable uncertainty” criterion. In particular, the paper does not compare the joint gate with uncertainty-only routing at a matched reacquisition budget. It also specifies attention readout layers, an entropy threshold of 0.15 nats, a deformation threshold of 0.025, and a uniform-mixture weight of 0.1, without explaining their selection procedure or reporting sensitivity. Cross-backbone results provide encouraging transfer evidence, but leave unclear how robust the policy is to these choices. These gaps limit confidence that the gains reflect a generally reliable repairability criterion rather than a particular configuration of attention-guided repeated observation.

  2. The efficiency evaluation does not establish the accuracy–latency trade-off specifically for long videos or for the strongest variants. Table 1 reports long-video accuracy, but the runtime analysis aggregates a 30-question panel containing ten questions from each duration group. Figure 4 combines this panel’s mean latency with full-benchmark accuracy, without separately reporting latency and reacquisition rates for the long-video subset. Since decoding costs and the need for further observations may vary with duration, the aggregate results do not establish whether the reported efficiency advantage holds for long videos. Furthermore, the largest accuracy improvements come from STR-Refine, whereas the reported 1.17–1.21×1.17–1.21\times latency applies to STR/STR-Lite. Appendix C explicitly states that its panels do not establish matched online stopping-policy latency for Refine. The actual computational cost of the strongest results therefore remains insufficiently characterized. This matters because efficient long-video understanding is a central motivation, and a fixed per-observation frame budget does not imply a matched total computation budget.

References

[1] Xiaohan Wang et al. VideoAgent: Long-form Video Understanding with Large Language Model as Agent. ECCV, 2024.

[2] Anurag Arnab et al. Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames. NeurIPS, 2025.

[3] Yuanhao Zou et al. A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering. ICLR, 2026.

[4] Shuming Liu et al. BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding. CVPR, 2025.

Questions for authors
  1. Clarification and targeted validation of parameter choices: How were the attention readout layers, τU=0.15\tau_U=0.15, τD=0.025\tau_D=0.025, and η=0.1\eta=0.1selected? Please specify the data and objective used for selection and whether these settings were fixed before benchmark evaluation. Could a limited sensitivity analysis show whether nearby layers and parameter values preserve the accuracy gains and reacquisition rates?

  2. Targeted validation of the repairability criterion: At a matched average reacquisition budget, how does the joint uncertainty–deformation gate compare with uncertainty-only routing? Reporting accuracy and the numbers of corrected and regressed answers would help establish whether deformation provides useful information beyond predictive uncertainty.

  3. Clarification and targeted validation of long-video efficiency: Could the authors disaggregate the existing runtime panel by video duration, reporting latency and average observation count alongside the corresponding accuracy results? Please also clarify the measured end-to-end cost of STR-Refine and STR-Refine-Lite under their actual stopping policies. This would help establish whether the efficiency claims hold for long videos and for the variants yielding the strongest accuracy improvements.

  4. Clarification of the evidence-localization evaluation: For Figure 5(a), what are the absolute precision, recall, and F1 scores underlying the reported relative improvements? Please also specify the number of evaluated questions and videos and the temporal matching tolerance. These details are needed to interpret the practical magnitude of the reported 61.8% relative F1 improvement.

  5. Targeted validation of STR-Lite: At the deployed 60% token-retention ratio, how does the proposed router compare with simple coverage-preserving token selection or representative existing visual-token pruning methods under the same token budget? Could the authors also isolate the contribution of the second-stage temporal adaptation at this retention ratio? The current 10%-retention stress test is informative, but does not clarify the benefits of the specialized router and its adaptation stage under the default deployment setting.

6 comments
AuthorsOpenReview ↗

Author Response (1/4)

W1

Is STR Novel? VideoAgent [1] iteratively gathers information to answer video questions; Temporal Chain of Thought [2] uses the VLM itself to iteratively select relevant frames; and A.I.R. [3] provides training-free adaptive and iterative frame selection. Relevance-based inverse-transform sampling was also explored in BOLT [4].

STR is the first method to identify and use repairable uncertainty: a frozen MLLM can remain uncertain about its answer while its first-pass temporal attention provides useful guidance on where to reacquire missing evidence. STR turns this distinction into a self-correction policy: the initial answer pass itself supplies the signals for deciding whether another observation is warranted and where that observation should concentrate. This is the conceptual advance, with the answerer's attention, joint uncertainty–deformation gate, and fixed per-observation budget as method design.

The two categories raised in the comment are already addressed in our related-work paragraphs, “Adaptive evidence acquisition” (L110–125) and “Query-conditioned frame selection” (L127–140), where VideoAgent, Temporal Chain of Thought, and BOLT are explicitly discussed. VideoAgent, Temporal Chain of Thought, and A.I.R. are prompt-based, agentic evidence-acquisition methods that rely on explicit VLM reasoning and frame-selection steps to identify relevant video evidence, making them extremely expensive. STR instead obtains both the reacquisition decision and the temporal sampling guidance directly from the initial answer pass, without any additional prompt-based frame-selection step. BOLT is not only discussed in our related work but also evaluated as a main baseline in Table 1. Our evaluation demonstrates substantial improvements in both accuracy and efficiency over BOLT on all five benchmark-level accuracies (Table 1, Figure 4).

AuthorsOpenReview ↗

Author Response (2/4)

Q1

Clarification and targeted validation of parameter choices

None of τU\tau_U, τUexit\tau_U^{\mathrm{exit}}, τD\tau_D, η\eta, or the attention readout layer were tuned on the evaluation benchmarks. We selected these hyperparameters on a separate internal development dataset, consisting of 120 video-question pairs with human-annotated answers, with no overlap with the evaluation benchmarks. We cannot disclose its identifying information at this stage without compromising double-blind anonymity. We will provide these details in the camera-ready version upon acceptance.

Hyperparameter robustness is a strength of STR. The hyperparameters of STR are based on the properties of MLLMs and are therefore intuitive to set and straightforward to tune. For example, deeper MLLM layers are widely recognized as providing better semantic localization, so any sufficiently deep layer can serve as the attention readout [1]. Our readout-layer study demonstrates this behavior. The same goes for other hyperparameters; additional sensitivity experiments below further demonstrate this robustness, showing nearly flat accuracy curves across broad parameter ranges.

We used the same hyperparameter values across five datasets covering approximately 10,339 video–question pairs, spanning diverse video understanding tasks, from egocentric reasoning to needle-in-a-haystack retrieval.

1. Attention Readout Layer

LayerLongVideoBenchVideoMME
24 (Paper)66.269.4
2566.669.7
2665.269.3
2765.069.8
2865.869.2
2965.769.5

2. Predictive Uncertainty Threshold (τU\tau_U)

We use a shared numerical uncertainty threshold, setting τUexit=τU=0.15\tau_U^{\mathrm{exit}} = \tau_U = 0.15 nats in the default configuration. The superscript ''exit'' distinguishes its role in STR-Refine’s stopping rule; it is not independently tuned.

τU\tau_U (nats)LongVideoBench acc.VideoMME acc.VideoMME calls/q
0.0566.469.71.42
0.07566.269.61.40
0.1066.369.41.39
0.15 (Paper)66.269.41.33
0.2066.269.21.31
0.3066.168.91.27
0.5065.768.51.21
1.0064.867.81.06

3. Attention Deformation Threshold (τD\tau_D)

For context, the attention-deformation measure ranges from approximately 0 to 0.15 in our experiments, reflecting the behavior of the underlying MLLM.

τD\tau_DLongVideoBench acc.VideoMME acc.VideoMME calls/q
0.0166.269.51.44
0.0266.069.51.40
0.025 (Paper)66.269.41.33
0.0366.569.11.31
0.0466.168.61.23
0.0565.468.21.15
0.1063.767.61.01

4. Resampling Parameter (η\eta)

η\etaLongVideoBenchVideoMME
066.769.4
0.0566.368.7
0.1 (Paper)66.269.4
0.266.369.1
0.366.668.8

These hyperparameters primarily control the trade-off between accuracy gains and computational cost. The uncertainty and deformation thresholds determine when the repairability gate triggers additional evidence acquisition, while η\eta controls the resampling distribution. At sufficiently stringent gating thresholds, no reacquisition occurs and STR returns the initial uniform-sampling answer. Uniform sampling is therefore a limiting case of STR. More permissive gating allows additional opportunities to address repairable uncertainty, while more stringent gating reduces the number of correction calls and improves efficiency.

[1] Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs, ICLR 2026

AuthorsOpenReview ↗

Author Response (3/4)

Q2

Targeted validation of the repairability criterion: At a matched average reacquisition budget, how does the joint uncertainty–deformation gate compare with uncertainty-only routing?

Adding attention deformation improves efficiency by reducing reacquisition calls without any loss in accuracy. Attention deformation helps determine whether an uncertain prediction is likely to benefit from additional observations. When predictive uncertainty is high, high deformation indicates that attention is concentrated on specific temporal regions, suggesting that the model has identified partial evidence that can guide targeted resampling. In contrast, low deformation provides little direction for targeted resampling, so the joint gate avoids further calls. We evaluate this by comparing joint uncertainty–deformation gate with, uncertainty-only routing, and budget-matched uncertainty-only routing on VideoMME. Budget-matched uncertainty-only is obtained by increasing the uncertainty threshold to reduce the number of calls so that Total calls/q matches joint uncertainty–deformation gate.

VariantTotal calls/q ↓Accuracy (%) ↑
Uncertainty-only1.4969.4
Uncertainty-only (budget-matched)1.3368.3
Uncertainty + Deformation1.3369.4

The repairable uncertainty gate (uncertainty + deformation) reduces reacquisition calls by 32.7% compared with uncertainty-only routing during the STR reacquisition call.

At a matched budget of 1.33 calls per question, the joint uncertainty–deformation gate outperforms uncertainty-only routing by 1.1 percentage points (69.4% vs. 68.3%). This demonstrates that attention deformation provides useful routing information beyond predictive uncertainty.

Q3

Efficiency of STR-Refine and STR-Refine-Lite for Long Videos.

STR-Refine’s and STR-Refine-Lite are computationally efficient for long videos. We report the average observation count and latency under each variant’s stopping policy on all 2,700 Video-MME questions:

MethodPasses per QuestionSeconds per Question× One Uniform Pass
Uniform, one pass1.03.01.00×
STR1.33.61.2×
STR-Refine1.94.51.5×
STR-Refine-Lite1.94.11.4×

The duration-wise breakdown below shows STR-Refine’s accuracy, observation count, and latency. The average observation count increases from 1.6 for short videos to 2.2 for long videos. For long videos, the latency is 5.1 seconds, corresponding to 1.7× the uniform-pass reference.

DurationQuestionsAccuracy (%)Passes per QuestionLatency (s)× One Uniform Pass
Short90079.91.63.71.2×
Medium90071.12.04.71.6×
Long90059.32.25.11.7×
All2,70070.11.94.51.5×

Q4

Clarification of the evidence-localization evaluation: For Figure 5(a), what are the absolute precision, recall, and F1 scores underlying the reported relative improvements? Please also specify the number of evaluated questions and videos and the temporal matching tolerance. These details are needed to interpret the practical magnitude of the reported 61.8% relative F1 improvement.

We follow T*/LV-Haystack’s released evaluation protocol for filtering LongVideoBench’s videos and excluding subtitle-dependent questions. After filtering based on T*/LV-Haystack’s evaluation protocol the evaluation contains 342 questions across 114 videos. Temporal matching uses a ±5-second tolerance (again standard T*/LV-Haystack’s protocol) around each annotated keyframe. The absolute precision, recall, and F1 scores underlying Figure 5(a) are:

CapArmPrecision (%)Recall (%)F1 (%)Relative F1 improvement (%)
1both1.450.52.70.0
2uniform1.347.62.6—
2STR-Refine1.754.23.3+26.3
3uniform1.449.92.7—
3STR-Refine2.154.74.0+46.1
4uniform1.451.52.7—
4STR-Refine2.455.04.5+61.8

The low absolute precision is a consequence of the sparse-keyframe evaluation protocol: precision is calculated over all 64 sampled frames, despite having only a one/two annotated evidence keyframes per question. For example, matching one annotated keyframe with one sampled frame yields only 1.56% precision. We therefore use relative F1 improvements.

STR-Refine improves both precision and recall, demonstrating that attention-guided reacquisition increases evidence concentration without sacrificing coverage. The increasing gains demonstrate progressively more effective evidence localization.

Reviewer_yzXdOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses frame selection in long video understanding through the guidance of model's temporal attention. Their key insight is that, despite the model output remains uncertain, its temporal attention pattern provides useful guidance towards finding the missing frames. The authors leverage this pattern to reacquire the input frames through a series of methods: STR that establishes the basic method, STR-Refine that boosts the performance via multiple times of reacquisition, and STR-Lite that prunes the reacquired views to improve efficiency. The main experiment results are reported on five benchmarks with comprehensive comparison against baseline methods. Further analysis shows the latency-accuracy trade-off of the proposed method and each selected keyframe retrieval methods, demonstrating that STR-series are more preferable in real-world applications.

Critical strengths
  1. The paper proposes a novel approach towards selecting keyframes in long videos, different from the widely-adopted CLIP-based retrieval. The inverse-CDF resampling method, despite not being unique in this field, fits well with the authors’ intention to shift focus to temporal regions with higher model attention.
  2. The analytical experiments in the paper are comprehensive and convincing, especially the results in latency-accuracy trade-off section. With decent accuracy improvement and little computation overhead, the proposed method could be useful in real-world applications.
Critical weaknesses

None

Questions for authors
  1. How well are the VLLMs (for example the three models used in table 2) calibrated? Given that STR-Refine terminates based on model uncertainty, it would be helpful to know whether the models are prone to over-confidence.
3 comments
AuthorsOpenReview ↗

Author Response

How well are the VLLMs (for example, the three models used in table 2) calibrated? Given that STR-Refine terminates based on model uncertainty, it would be helpful to know whether the models are prone to over-confidence.

We observe that model confidence does reliably identify accurate answers, supporting its use in STR-Refine’s stopping criterion. For Qwen3-VL-8B on Video-MME, we rank questions by increasing predictive entropy (higher rows in the following table are more confident) and evaluate accuracy on progressively larger, cumulative subsets:

Subset with lowest predictive entropyAccuracy
20%98.5%
40%93.5%
50%89.1%
60%84.3%
80%74.6%
100%67.4%

Accuracy increases consistently as evaluation is restricted to more confident predictions. This separation demonstrates that predictive entropy effectively distinguishes more reliable answers from less reliable ones. This behavior also holds for the larger backbone: Qwen3-VL-32B achieves 94.7% accuracy on predictions with top 40%.

These observations are consistent with prior works [1] [2] that language and vision-language models can provide well-calibrated confidence estimates, and calibration generally improves with model size [1].

[1] Kadavath et al., Language Models (Mostly) Know What They Know (2022)

[2] Srinivasan et al., Selective “Selective Prediction”: Reducing Unnecessary Abstention in Vision-Language Reasoning (ACL Findings, 2024)

Reviewer_yzXdOpenReview ↗

Thanks For the Response

I thank the authors for their response. The additional experiment addresses my concern and strengthens the paper. Therefore, I maintain my recommendation for acceptance.

AuthorsOpenReview ↗

Thank you!

Thank you for your constructive feedback. We are glad that the additional experiment addressed your concern.

Reviewer_pmgaOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses frame selection for long-video understanding with MLLMs under a fixed frame budget. It observes that when a model answers from a sparse uniform view, its answer can remain uncertain while its temporal attention already concentrates on query-relevant regions ("repairable uncertainty"). Building on this, STR is a training-free test-time policy: after a first pass on 64 uniformly sampled frames, it computes the answer's predictive entropy and an attention-derived deformation score, and only when the answer is uncertain and the attention is sufficiently focused, it resamples 64 frames from an attention-based density mixed with a uniform floor and answers again with the same frozen model. STR-Refine extends this to up to four observations with accumulated relevance and an entropy-based stopping rule, and STR-Lite trains a small router to prune visual tokens of the reacquired views. On LongVideoBench, VideoMME, MLVU, VideoVista and Video-MME-v2 with Qwen3-VL-8B, STR-Refine improves over uniform sampling (e.g., +6.0 on LongVideoBench, +9.0 on its long-video subset), with gains transferring to Qwen3-VL-2B/32B and InternVL3-8B on LongVideoBench. The paper also reports a gating analysis (about 53% of second passes avoided with under 0.3 points accuracy loss), a temporal-F1 evidence localization analysis, and latency measurements (1.17–1.21× one-pass latency).

Critical strengths
  1. Simple, training-free and easy to adopt. STR uses only signals the frozen answerer already produces in its first pass (answer entropy and temporal attention) to decide whether and where to look again, without a separate frame-selection model or scoring a large candidate pool. The design is simple, plug-and-play across backbones (Table 2), and the gains concentrate on medium and long videos, where uniform sampling is expected to miss evidence. This makes the method practically useful.
  2. Evidence beyond end-task accuracy. The temporal-F1 analysis on annotated evidence (Fig. 5a), the gating analysis showing that most of the benefit of reacquisition is preserved while avoiding about half of the second passes (Fig. 5c), and the accuracy–compute comparison against larger uniform budgets (Fig. 5b) support that the improvement comes from reacquiring relevant frames rather than only from extra computation.
Critical weaknesses
  1. It is unclear how the thresholds and readout settings were chosen. The entropy thresholds (τ_U = τ_U^exit = 0.15 nats), the deformation threshold (τ_D = 0.025), the uniform floor (η = 0.1) and the readout layer (layer 24; layer 48 for the 32B model) are only given as values in Appendix A.2 and B, with no statement of the data used to select them. If they were tuned on the evaluation benchmarks, the gains in Table 1 would be overstated. A sensitivity analysis or a selection protocol on held-out data is needed.
  2. The frame budget in Table 1 is not matched against the frame-selection baselines. The table uses a 64-frame per-view budget, and STR-Refine uses up to four views (up to 256 frames), whereas AKS, BOLT, Q-Frame, OneClip-RAG, AdaQ and CADER select 64 frames in a single pass. Fig. 5b provides a frame-equivalent comparison against uniform sampling, but there is no matched-total-budget comparison against these selection methods.
  3. Several baselines fall below uniform sampling, which raises questions about their configuration. On LongVideoBench, Q-Frame and CADER both reach 61.9 versus 63.0 for uniform sampling, and on VideoMME Q-Frame (67.1), CADER (67.0) and OneClip-RAG (66.8) are also below uniform (67.4). It is not stated clearly whether all baselines were re-run by the authors with Qwen3-VL-8B and how their settings were adapted; Appendix C describes the setup mainly for latency.

Why these matter for my assessment. All three concern how fairly the gains are measured rather than the idea itself. The idea is simple and the analyses in Section 4.4 are convincing, so I lean toward acceptance; my recommendation depends on the authors clarifying the threshold selection and the baseline setup, and on a budget-matched comparison against the selection methods.

Questions for authors

Questions that could change my assessment

  1. On which data were τ_U, τ_U^exit, τ_D, η and the readout layer selected? Were any of them tuned on the evaluation benchmarks? How sensitive are the results in Table 1 to these values?
  2. Can you compare STR-Refine with the frame-selection baselines (e.g., AKS, BOLT, AdaQ) at a matched total frame budget, for example by giving the baselines the average number of frames STR-Refine actually uses?
  3. Were all baselines in Table 1 re-run with Qwen3-VL-8B using their released code? Which settings were changed from the original papers, and how do your numbers compare with those reported by the original authors?
2 comments
AuthorsOpenReview ↗

Author Response (1/2)

Q1, W1

On which data were τU\tau_U, τUexit\tau_U^{\mathrm{exit}}, τD\tau_D, η\eta, and the readout layer selected? Were any of them tuned on the evaluation benchmarks? How sensitive are the results in Table 1 to these values?

None of τU\tau_U, τUexit\tau_U^{\mathrm{exit}}, τD\tau_D, η\eta, or the attention readout layer were tuned on the evaluation benchmarks. We selected these hyperparameters on a separate internal development dataset, consisting of 120 video-question pairs with human-annotated answers, with no overlap with the evaluation benchmarks. We cannot disclose its identifying information at this stage without compromising double-blind anonymity. We will provide these details in the camera-ready version upon acceptance.

Hyperparameter robustness is a strength of STR. The hyperparameters of STR are based on the properties of MLLMs and are therefore intuitive to set and straightforward to tune. For example, deeper MLLM layers are widely recognized as providing better semantic localization, so any sufficiently deep layer can serve as the attention readout [4]. Our readout-layer study demonstrates this behavior. The same goes for other hyperparameters; additional sensitivity experiments below further demonstrate this robustness, showing nearly flat accuracy curves across broad parameter ranges.

We used the same hyperparameter values across five datasets covering approximately 10,339 video–question pairs, spanning diverse video understanding tasks, from egocentric reasoning to needle-in-a-haystack retrieval.

1. Attention Readout Layer

LayerLongVideoBenchVideoMME
24 (Paper)66.269.4
2566.669.7
2665.269.3
2765.069.8
2865.869.2
2965.769.5

2. Predictive Uncertainty Threshold (τU\tau_U)

We use a shared numerical uncertainty threshold, setting τUexit=τU=0.15\tau_U^{\mathrm{exit}} = \tau_U = 0.15 nats in the default configuration. The superscript ''exit'' distinguishes its role in STR-Refine’s stopping rule; it is not independently tuned.

τU\tau_U (nats)LongVideoBench acc.VideoMME acc.VideoMME calls/q
0.0566.469.71.42
0.07566.269.61.40
0.1066.369.41.39
0.15 (Paper)66.269.41.33
0.2066.269.21.31
0.3066.168.91.27
0.5065.768.51.21
1.0064.867.81.06

3. Attention Deformation Threshold (τD\tau_D)

For context, the attention-deformation measure ranges from approximately 0 to 0.15 in our experiments, reflecting the behavior of the underlying MLLM.

τD\tau_DLongVideoBench acc.VideoMME acc.VideoMME calls/q
0.0166.269.51.44
0.0266.069.51.40
0.025 (Paper)66.269.41.33
0.0366.569.11.31
0.0466.168.61.23
0.0565.468.21.15
0.1063.767.61.01

4. Resampling Parameter (η\eta)

η\etaLongVideoBenchVideoMME
066.769.4
0.0566.368.7
0.1 (Paper)66.269.4
0.266.369.1
0.366.668.8

These hyperparameters primarily control the trade-off between accuracy gains and computational cost. The uncertainty and deformation thresholds determine when the repairability gate triggers additional evidence acquisition, while η\eta controls the resampling distribution. At sufficiently stringent gating thresholds, no reacquisition occurs and STR returns the initial uniform-sampling answer. Uniform sampling is therefore a limiting case of STR. More permissive gating allows additional opportunities to address repairable uncertainty, while more stringent gating reduces the number of correction calls and improves efficiency.

Q2, W2

Can you compare STR-Refine with the frame-selection baselines (e.g., AKS, BOLT, AdaQ) at a matched total frame budget?

We conducted a matched total-frame-budget comparison for each video–question pair. If STR-Refine processes (x) frames across its adaptive observations, we allocate exactly (x) answerer frames to each baseline, AKS, BOLT, and AdaQ, for the same input. Each baseline receives the same total number of answerer-frame inputs in a single pass that STR-Refine processes across its adaptive observations.

Under this comparison, STR-Refine achieves higher accuracy on both benchmarks while requiring substantially less end-to-end inference time (measurements performed under the same protocol and hardware configuration):

MethodVideoMMELongVideoBenchLatency (s) (× one uniform pass)
Uniform, 64 frames67.463.03.0 s
AKS69.765.739.3 s (13.3×)
BOLT68.964.411.8 s (4.0×)
AdaQ69.365.211.5 s (4.0×)
STR-Refine70.169.04.5 s (1.5×)
AuthorsOpenReview ↗

Author Response (2/2)

Q3, W3

Why do some frame-selection baselines underperform uniform sampling, and how were these baselines implemented and evaluated to ensure a fair comparison?

All baselines were evaluated with Qwen3-VL-8B as the common frozen answerer, using the same evaluation harness as STR. We used the officially released code for every baseline except CADER. For methods such as Q-Frame, whose original evaluations do not include Qwen3-VL-8B, we adapted the released code to this backbone within our shared evaluation harness. For CADER, no official implementation is available, so we implemented the method as described in the paper. We used their original configuration: Qwen3-VL-8B as the answerer, Qwen3-VL-4B-Instruct as the verifier, up to two reflection rounds, and a verification threshold of 0.6. The original protocol uses the backbone’s native frame/token budget and a 512-frame target for localized sampling. We first reproduced their reported results under their original settings. On VideoMME-Long, we obtained 60.0% accuracy, compared with their reported 60.1%. For the Table 1 comparison, the only change to CADER’s algorithmic configuration was to impose our 64-frame per-view cap.

We observe that some frame-selection baselines slightly underperform uniform sampling because concentrating a fixed frame budget on selected regions sacrifices temporal coverage. Frame selection is most beneficial when it recovers query-relevant evidence missed by uniform sampling. When the uniform view already contains sufficient evidence, further concentration may offer limited additional benefit while reducing the temporal coverage needed for broader understanding. This trade-off is consistent with DIG [1], which finds uniform sampling sufficient for global queries and targeted selection more beneficial for localized queries. AKS’s authors likewise describe uniform sampling as a safe choice for many Video-MME questions requiring high-level comprehension [3].

Independent evaluations also report frame-selection baselines underperforming uniform sampling:

  • DIG [1], Qwen2.5-VL-7B, 64 frames: Q-Frame achieves 57.52 versus 58.94 for uniform sampling on LongVideoBench, and 63.43 versus 63.61 on MLVU (Table 1).
  • FOCUS [2], Qwen2-VL-7B, 32 frames: Q-Frame achieves 56.5 versus 57.4 for uniform sampling on Video-MME (Appendix E, Table 5).
  • DIG [1], Qwen3-VL-8B: Using the same answerer backbone as our experiments, AKS underperforms uniform sampling on LongVideoBench at both 128 frames (53.0 versus 54.4) and 256 frames (55.6 versus 57.6; Appendix F, Table 3).

These results demonstrate that frame selection does not universally outperform uniform sampling, including with the Qwen3-VL-8B backbone.

[1] Jialuo Li et al. Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding. CVPR, 2026.

[2] Zirui Zhu et al. FOCUS: Efficient Keyframe Selection for Long Video Understanding. ICLR, 2026.

[3] Adaptive Keyframe Sampling for Long Video Understanding. arXiv:2502.21271.

[4] Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs, ICLR 2026

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You and Concluding Remarks

We are grateful to the reviewers for their constructive feedback and active engagement throughout the discussion. All reviewers have acknowledged that we have addressed their concerns, and all recommend acceptance.

Strengths highlighted by the reviewers:

  • Novel approach to keyframe selection (yzXd)
  • Simple and practically useful method (uMfD, pmga, yzXd)
  • Strong empirical results and analysis (uMfD, yzXd, pmga)

Our additional experiments empirically show:

  • Hyperparameter and layer robustness. STR is robust across a wide range of hyperparameter settings and deep attention layers.
  • Matched total frame budgets. STR-Refine outperforms AKS, BOLT, and AdaQ even when the baselines receive additional frames to match the total frame budget.
  • Validation of repairable uncertainty. The repairable uncertainty gate improves accuracy over uncertainty-only routing at a matched average call budget, showing the value of attention deformation.
  • Router and temporal adaptation. STR-Lite’s router improves accuracy over coverage-preserving sampling, with further gains from temporal adaptation.
  • Confidence and uncertainty-based stopping. Predictive entropy identifies more reliable answers, supporting uncertainty-based stopping.

We have incorporated the rebuttal experiments into the main paper and appendix, with changes highlighted in blue.

Also thankful to AC for encouraging and facilitating discussion. Upon acceptance, we will open-source the entire codebase and weights.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.