acceptodds
Withdrawn from ICLR 2027

Look Again: Attention-Guided Temporal Resampling for VideoLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Uniform frame sampling can miss the brief events needed to answer questions about long videos. We investigate whether a VideoLLM’s internal attention can guide a second look at relevant moments, even when the initial view does not support a correct answer. We introduce Temporal Attention Resampling (TAR), a training-free method that contrasts question-conditioned attention with a generic reference, retrieves a contiguous temporal interval, and combines densely resampled local frames with global context. On our 502-question MLVU evaluation set, TAR improves Qwen3-VL-8B from 44.42% to 50.20%, outperforming matched 48-frame uniform sampling by 4.38 percentage points and random-window re-sampling by 4.65 points. Accuracy improves across eight benchmarks and two Qwen-family model scales. On the annotated LVBench subset, questions whose retrieved windows overlap evidence gain 6.43 points, compared with 1.19 points for those without overlap. This association is consistent with improved temporal evidence acquisition. TAR requires no parameter updates, temporal supervision, or external retriever.

Primary area:
generative models
Keywords:
training free, videollms, attention

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_HFq1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes TAR, a training-free method for long-video QA. It contrasts question-conditioned attention with a generic reference, retrieves a contiguous temporal interval, and combines densely resampled local frames with global context. Experiments conducted across eight benchmarks and two model scales demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The idea is straightforward. Normalising question attention by a generic-prompt reference beats the raw and difference maps, and the idea may transfer to token or frame selection.
  2. TAR is easy to reproduce as it needs no parameter updates, temporal annotations, or an external retriever.
Critical weaknesses
  1. Limited novelty and a heuristic method design. TAR is essentially the temporal analogue of attention-guided spatial re-looking Vi. Query-guided frame selection and coarse-to-fine temporal search are already well explored (e.g., AKS, MDP 3, VideoTree, A.I.R.). The design itself is ad hoc, and the motivation is not well-justified.

  2. Missing baselines and narrow scope. There is no comparison with the cited training-free frame-selection methods (AKS, MDP3, T*) at matched budgets. Only Qwen models are tested, which is insufficient to validate the generalizability of the proposed method.

Questions for authors
  1. What are the end-to-end latency, memory, and FLOPs compared with Uniform-64/96/128, and what is the accuracy at matched total compute?
  2. How to choose the attention layers? Are there any empirical or experimental results for that?
Reviewer_gzmCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TAR, a training free method that uses question to generic attention ratios to select a video interval for denser sampling. It combines local observations with global frames and reports consistent accuracy gains across eight benchmarks and two Qwen family models.

Critical strengths

TAR outperforms both uniform sampling and random windows under the same final frame budget. And accuracy improves across all 8 benchmarks, and TAR also outperforms the CLIP retriever on Video-MME. These results support the usefulness of the model’s internal attention for temporal resampling without additional training.

Critical weaknesses
  1. TAR adds two attention-extraction passes over the coarse video, but the paper reports neither end-to-end latency nor peak memory. Although the authors distinguish final frame count from total cost, the practical benefit remains unclear without an accuracy–cost comparison against uniform sampling and CLIP retrieval.
  2. Comparisons are limited to uniform sampling, random windows, and CLIP retrieval, leaving TAR’s advantage over existing adaptive frame selection methods unclear. A comparison with a representative method discussed in the related work, using the same answering model and accounting for inference cost, would better establish its contribution.
Questions for authors
  1. What are TAR’s end to end latency and peak memory, including both retrieval passes, and how does its accuracy compare with uniform sampling and CLIP retrieval at matched inference cost?
  2. How does TAR compare with a representative adaptive frame selection method using the same answering model, in terms of both accuracy and inference cost?
Reviewer_fS46OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines how to capture brief events that are easy to miss when uniformly sampling a limited number of frames from a long video. The authors propose Temporal Attention Resampling (TAR), a training-free method that uses the video language model’s own attention to decide where to look again. From 64 uniformly sampled frames, TAR computes the ratio of question-conditioned to generic-prompt attention, AQ/(AG+ϵ)A_Q/(A_G+\epsilon), and sums it over spatial locations. It selects the highest-scoring contiguous window, spanning about one eighth of the coarse sequence. The model then answers using 32 global frames and 16 frames sampled densely from that window.

Across eight video QA benchmarks, TAR improves on 64-frame uniform sampling in all 16 settings tested with Qwen3-VL-8B and Qwen3.6-35B-A3B. Gains range from 2.33 to 5.78 percentage points, with MLVU accuracy rising from 44.42% to 50.20%. TAR also outperforms frame-matched uniform and random-window controls on MLVU and a CLIP-based retriever on Video-MME. On LVBench, gains are larger when the selected window overlaps annotated evidence, although localization remains weak overall.

Critical strengths

1. A focused research question and appropriately scoped claims. The paper starts from a straightforward observation. A model can answer a question about a long video incorrectly simply because its sampled frames missed the decisive event (Sec. 1). This motivates a specific, testable question: can the answering model use its own attention to decide where to look again?

I appreciate how explicitly the authors delimit their conclusions. They describe the comparisons with adaptive search methods as limited in scope (Sec. 2.1), clarify that the controls match the final input to the answering model while total computation differs (Sec. 4.1), and treat the localization analysis as observational (Sec. 4.7). They also acknowledge the higher end-to-end cost and the fact that the evaluation is limited to Qwen models and multiple-choice tasks (Sec. 5). These distinctions make it easier to judge which conclusions the experiments support.

2. A straightforward, training-free procedure. TAR requires no parameter updates, temporal supervision, or external retriever. The authors withhold the answer candidates during window selection so the choices cannot influence where the model looks. Once the window is selected, TAR samples frames directly from the source video, giving the model a chance to inspect details missed in the coarse pass. The procedure is easy to follow and gives future work a clear starting point.

Critical weaknesses

1. Limited novelty and incomplete comparison with prior work. TAR’s central retrieval signal, AQ/(AG+ϵ)A_Q/(A_G+\epsilon), is the relative-attention ratio introduced by Khayatkhoei et al. (ICLR 2025). That work also extracts attention from a single selected layer and combines global and zoomed-in views for answering. Section 2.2 cites it as general inspiration, while Sec. 3.2 presents the ratio as a design choice supported by the paper’s own ablation. I would like the authors to make this connection explicit and clarify what TAR adds. As I understand it, the main additions are to aggregate spatial attention into a temporal score, select a fixed-width contiguous window, and sample frames more densely within it. These are sensible adaptations to video, but the methodological advance appears incremental.

The paper also leaves out several closely related training-free methods that use the video model’s own attention or confidence to select frames or temporal intervals, including FlexSelect (arXiv:2506.00993), DyToK (arXiv:2512.06866), and Temporal Search / Iterative Zoom-In (arXiv:2507.02946). AKS is cited but is absent from the experimental comparisons. Discussing and comparing these methods would help clarify where TAR advances the existing work.

2. The experiments leave open how large the gains are, whether they justify the added cost, and what drives them. In addition to the 48-frame answering pass, TAR requires two attention-extraction passes over the 64-frame coarse view (Secs. 3.4 and 5). On MLVU, uniform sampling with 128 frames already reaches 49.80%, close to TAR’s 50.20% in the 64-frame setting (Table 4). Moreover, TAR’s margin over uniform sampling at the corresponding frame budget narrows from 5.78 percentage points at 64 frames to 2.99 points at 128 frames. A comparison at matched end-to-end cost would help establish whether the additional passes are worthwhile. The baseline scores also fall substantially below those in the Qwen3-VL technical report. For Qwen3-VL-8B, the paper reports 58.39% versus 71.4% on Video-MME and 34.60% versus 58.0% on LVBench. Explaining the protocol differences behind these gaps would put the reported gains in context.

The attention score, layer, and frame budget appear to have been selected on the same MLVU set used for the headline result. The controls that help isolate the effect of interval selection, Uniform-48 and random-window resampling, are also limited to that set. The random-window result uses three seeds, and neither comparison includes a paired significance test. I would like to see whether the gains over these controls hold on a benchmark held out from configuration selection.

Finally, the LVBench localization results are modest, with Recall@1 of 23.34% and mean temporal IoU of 3.20%. As the paper acknowledges in Sec. 4.7, a matched random-window baseline is needed to establish how much localization improves over chance. Since the selected window spans about one eighth of the video, some overlap with annotated evidence is expected even from random selection. The larger gains for questions with evidence overlap in Fig. 4 could therefore reflect useful retrieval, differences in question difficulty, or both. Repeating the same analysis with random windows of equal length would help distinguish these explanations.

Questions for authors

Clarification requests

  1. Implementation details. Which “retrieval-text tokens” are used to extract AQA_Q and AGA_G, and how is attention aggregated across those tokens? What normalization is applied after clipping at the 99.5th percentile to obtain R~\tilde R (Sec. 3.2)? Please clarify how the TT temporal bins correspond to the 64 input frames and how the selected bins map to timestamps in the source video (Sec. 3.3). For Eq. 5, please explain how the global and local frames are ordered and how timestamps and any separators are handled. Please also report the frame resolution and decoding settings, and indicate whether the code will be released.

  2. Hyperparameter selection. Were the attention ratio, layer indices (22 for Qwen3-VL-8B and 28 for Qwen3.6), window length (T/8T/8), and 64-frame coarse view fixed before evaluation, or were they chosen based on results from the 502-question MLVU set? In particular, how was layer 28 selected for Qwen3.6?

  3. Oracle and benchmark subsets. Which benchmark and model were used for the 200-question diagnostic experiment in Sec. 4.2, what frame counts were compared, and how were the questions sampled? Please also identify the MLVU version and split used for the 502-question evaluation and report the evaluation splits and sample sizes for NExT-QA, MotionBench, and TOMATO.

Analyses that could change my assessment

  1. Comparison at matched end-to-end cost. Section 4.1 clarifies that the controls match the input to the final answering pass, and Sec. 5 proposes a comparison at matched total latency or computation. Could the authors report that comparison against uniform sampling, accounting for both attention-extraction passes and the final answering pass? End-to-end latency, FLOPs, or peak memory would help quantify the trade-off. Does the accuracy gain persist when uniform sampling follows the Qwen3-VL technical report’s setting of 2 fps with up to 2,048 frames?

  2. Statistical support for the matched controls. Using the counts of fixes and regressions described in Appendix A, please report exact McNemar pp-values for TAR versus Uniform-48, TAR versus each random-window seed, and each model–benchmark comparison in Table 1. Do the gains over Uniform-48 and the random-window control also hold on Video-MME and LVBench?

  3. Localization relative to chance. Section 4.7 identifies a matched random-window baseline as necessary for assessing above-chance localization. Could the authors report the localization metrics in Table 7 and repeat the Fig. 4 analysis using random windows of the same length? Reporting accuracy gains separately for questions with and without evidence overlap would help show whether the pattern in Fig. 4 is specific to attention-guided selection.

  4. Relation to prior work. Khayatkhoei et al. (2025, “MLLMs Know Where to Look”) already normalize question-conditioned attention by attention under a generic description prompt, extract attention from a single selected layer, and combine the original and zoomed-in views for answering. What does TAR add methodologically beyond adapting these ideas to temporal selection? With the same allocation of 32 global and 16 local frames at answering time, how does TAR compare with training-free selection methods such as FlexSelect (arXiv:2506.00993), DyToK (arXiv:2512.06866), Temporal Search (arXiv:2507.02946), or AKS (Tang et al., 2025)?

Reviewer_voS1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a method, called TAR, to boost accuracy for visual Q&A with vision-language models (VLMs). The method is simple (not a bad thing) and intuitive. The method first processes the video with uniformly sampled frames in two different ways: (1) with the question, and (2) with a generic prompt. Then, a question-relevance map is computed by taking the ratio between the question's attention to the video and a generic prompt's attention to the video. Finally, a temporally low-res version of the video (for global context) and a selected temporally higher-res version of the video (for local, question-relevant context) is fed to the VLM to answer the question. Selection is based on the question-relevance map. The method is training-free.

Gains are consistent across 8 benchmark datasets (Table 1).

Critical strengths

None

Critical weaknesses
  • The method requires 3 separate forward passes using the VLM, which is expensive and is not measured. It is very possible that a compute-matched uniformly-sampled baseline would outperform TAR (this method). I see no evidence that TAR is compute efficient in FLOPs or time. The paper cannot be accepted without addressing this concern.
  • Novelty. The method is very similar to "MLLMs Know Where to Look", which is cited but not credited enough. As far as I understand, (1) taking the ratio between a generic prompt and the question's attention to the video, and (2) feeding the VLM with the low-res full video and the high-res selected/cropped video, were used in "MLLMs Know Where to Look". Additionally, the reference to this paper is missing its first author Jiarui Zhang.
  • Figures 1 and 2 are low quality
Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.