acceptodds
Under review as a conference paper at ICLR 2027

MLLMs Know When Before Speaking: Revealing and Recovering Temporal Grounding via Attention Cues

Abstract

Video temporal grounding (VTG), which localizes the start and end times of a queried event in an untrimmed video, is a key test of whether multimodal large language models (MLLMs) understand not only what happens but also when it happens. Although modern MLLMs describe video content fluently, their timestamp predictions remain unreliable, while existing remedies either require costly post-training on temporal annotations or rely on coarse training-free heuristics. In this work, we probe the cross-modal attention of MLLMs and uncover a perception-generation gap. Our key finding is that MLLMs often know the target interval during prefill, but lose this signal when generating the final answer. In the prefill stage, a sparse set of attention heads, which we call Temporal Grounding Heads (TG-Heads), concentrates query-to-video attention on the ground-truth interval. During autoregressive decoding, however, the answer tokens shift attention away from this interval toward visually salient but query-irrelevant segments. This observation motivates an inference-time read-then-regenerate framework. We first convert TG-Head prefill attention into a debiased frame-level relevance signal and extract the high-attention interval it highlights. We then re-invoke the MLLM with visual context restricted to this interval using video cropping. Without parameter updates or architectural changes, our framework consistently improves MiMo-VL, Qwen3-VL, and Molmo2 on three benchmarks, with gains of up to +3.5 mIoU.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLM, temporal grounding, video understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2gvvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper investigates temporal grounding in MLLMs, showing that query-relevant temporal cues captured by specialized attention heads during prefilling often become misaligned during decoding.
  • To address this, the authors propose a training-free inference framework that extracts and debiases prefill attention signals to identify candidate intervals, selectively regenerating predictions from the cropped video segments.
  • Evaluations across three TimeLens-Bench datasets using MiMo-VL, Qwen3-VL, and Molmo2 demonstrate consistent gains without parameter updates.
Critical strengths
  • Well-motivated problem formulation and clear presentation. The paper compellingly frames the discrepancy between prefill attention and decoded timestamps, seamlessly bridging this empirical observation with the proposed inference framework through concise, well-structured exposition.

  • Broad transferability across model families and scales. Without parameter updates, the framework consistently improves MiMo-VL, Qwen3-VL, and Molmo2 across three benchmarks and scales effectively across Qwen3-VL 4B, 8B, and 32B variants.

  • Clear formulation and strong reproducibility. The paper thoroughly specifies the attention extraction, debiasing, and confidence-gating procedures (alongside prompts and head indices in the appendices), while showing that just 200 held-out calibration samples suffice to recover the optimal heads and downstream performance.

Critical weaknesses
  • Incremental technical novelty over prior attention-based grounding and intervention methods. Prior work has already established training-free grounding via sparse localization heads (A), attention debiasing and guided cropping to recover accurate internal perception (B), and causal interventions on key cross-modal heads in video MLLMs (C).

  • Unestablished causal link between decoding attention drift and localization errors. Comparing query-token and numeric-answer-token attention conflates inference stage with token semantics; because visual cues are often aggregated into question tokens before routing to answer tokens (D), weaker direct answer-to-video attention does not necessarily imply lost visual evidence.

  • Unfavorable accuracy-latency trade-off on stronger grounding models. Mandatory zero-video debiasing passes and two-stage regeneration increase average inference latency by 2.1–2.4× (Table 14), yet yield only marginal gains (0.2–0.4 mIoU, Table 13) on the VTG-specialized TimeLens-8B model.

  • Evaluate transfer to reasoning-intensive temporal grounding. Using the TG-Heads and thresholds calibrated on standard grounding benchmarks, evaluate the framework on queries that require multi-step temporal reasoning to test whether prefill attention remains effective beyond direct event perception.

  • Incompatibility with FlashAttention and unprofiled extraction overhead. Materializing full attention weights appears fundamentally incompatible with efficient FlashAttention inference, yet the paper lacks a thorough analysis of this limitation. Because the authors neither isolate the latency and peak-memory overhead of attention extraction nor clarify whether the baseline in Table 14 uses fused attention kernels, the true deployment cost remains unclear.

[A] Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding, CVPR2025

[B] MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs, ICLR2025

[C] Understanding Temporal Logic Consistency in Video-Language Models through Cross-Modal Attention Discriminability, CVPR2026

[D] Cross-modal Information Flow in Multimodal Large Language Models, CVPR2025

Questions for authors

None

Reviewer_MazUOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies temporal video grounding in general-purpose MLLMs. It hypothesizes that prefill-stage attention contains useful temporal cues that are not always reflected in generated timestamps. The method identifies Temporal Grounding Heads (TG-Heads), extracts a candidate interval from their prefill attention, and uses a confidence gate to decide whether to retain the original prediction or perform a second inference on a cropped video. Experiments on three MLLMs and three benchmarks show consistent improvements without updating model parameters.

Critical strengths
  1. The paper investigates an interesting discrepancy between prefill attention and generated temporal predictions, and turns this observation into a parameter-free inference framework.

  2. Experiments across three MLLMs and three benchmarks show consistent gains. Ground-truth coverage analysis and decode-only attention steering provide evidence beyond qualitative visualizations.

Critical weaknesses
  1. The perception–generation gap claim is not fully established. High ground-truth coverage is measured on intervals after heuristic post-processing, showing that the extracted intervals often retain the target event, but not that raw prefill attention precisely identifies its boundaries. Decode-only steering improves performance, yet does not conclusively show that the model first identifies the correct interval and then forgets it.

  2. The benefit of the proposed attention guidance is not fully isolated from the benefit of a second inference stage. The paper does not sufficiently establish that TG-Heads-derived intervals outperform simpler alternatives, such as random or other attention-based intervals, under comparable inference budgets.

Questions for authors
  1. Could the authors report localization metrics and interval duration before and after each post-processing step, and analyze how candidate-interval width relates to Stage 2 gains?

  2. How does decode-only steering toward the prefill-derived interval compare with steering toward random or alternative candidate intervals under otherwise identical settings?

  3. Could the authors provide an accuracy–latency comparison against simpler inference-time alternatives and report how Stage 2 invocation rates and gains vary across models and benchmarks?

Reviewer_kvnUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies video temporal grounding in multimodal large language models. It identifies Temporal Grounding Heads by measuring the localization degradation caused by individually blocking text-to-video attention on a small labeled calibration set. At inference, entropy-weighted aggregation and contrastive debiasing against a blank video turn prefill attention into a candidate temporal interval. Low-confidence examples are then processed again using a cropped video. The method updates no model parameters, but uses annotations for one-time head selection and threshold calibration. Across three TimeLens-Bench datasets, it improves MiMo-VL, Qwen3-VL, and Molmo2 by 1.3 to 3.5 mIoU points.

Critical strengths
  1. Consistent practical gains on existing models. All nine combinations of three general-purpose models and three datasets in Table 1 improve by 1.3 to 3.5 mIoU points, and Table 2 extends these gains to different model sizes. The selected heads and confidence thresholds are calibrated on data disjoint from evaluation and then reused across datasets. These results support the method as an inference enhancement for frozen models.

  2. Substantial validation of the method's design. The paper ablates head selection, token aggregation, debiasing, and calibration size. On Qwen3-VL-8B and QVHighlights-TimeLens, adjusting only decoding attention raises mIoU from 59.4 to 60.9, while cropping with approximately fixed visual-token density per frame still reaches 61.4. These results support the practical value of the attention cues and show that increased resolution cannot fully explain the improvement.

Critical weaknesses
  1. Missing refinement baseline using the initial prediction. In Table 5, generated intervals achieve 59.4 mIoU, versus 51.7 for attention-derived intervals. Although the latter prioritize recall, the initial prediction remains a reasonable proposal. UniTime [1] already evaluates refinement within the initially predicted interval, and ReVisionLLM [2] uses recursive temporal refinement with specialized training. A baseline that expands the initial prediction and re-localizes with the same model is therefore needed to establish the additional benefit of attention guidance.

  2. Applicability and intervention choice on task-specialized models are unclear. Appendix D.2 evaluates only one VTG-tuned model, TimeLens-8B, and obtains gains of 0.2 to 0.4 mIoU points by replacing Crop with Mask; results for Crop on this model are not reported. The explanation that cropping and timestamp resetting disrupt learned temporal representations is not directly tested. It is therefore unclear whether this is a TimeLens-specific adaptation issue or a broader limitation, which affects my assessment of transferability across models.


References

[1] Li Z, Di S, Zhai Z, et al. Universal video temporal grounding with generative multi-modal large language models[J]. Advances in Neural Information Processing Systems, 2026, 38: 64426-64455.

[2] Hannan T, Islam M M, Gu J, et al. Revisionllm: Recursive vision-language model for temporal grounding in hour-long videos[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 19012-19022.

Questions for authors
  1. What new mechanistic insight or methodological value does the paper provide beyond prior methods? In relation to localization-head and attention-guided cropping studies [3, 4], please clarify what new properties of temporal grounding are revealed by TG-Heads and the prefill–decoding gap, and identify direct evidence supporting these distinctions.

  2. Does attention guidance outperform refinement based on the initial prediction? To establish the methodological benefit, please compare cropping around the expanded initial prediction with cropping the TG-Head interval, using the same model, examples for re-inference, and visual budget, with crop lengths matched as closely as possible. Please report mIoU overall and on the low-confidence subset, along with ground-truth event coverage for both proposal types on that subset. These findings would directly affect my assessment of the paper's distinct contribution.

  3. Does the method remain effective on other task-specialized grounding models? Please report baseline and post-intervention mIoU for representative VTG-tuned models beyond TimeLens, using the same evaluation protocol, and specify whether Crop or Mask is used for each model. For TimeLens-8B, how do Crop and Mask compare with the same guidance intervals and Stage 2 visual budget, and does preserving the original timestamps after cropping improve Crop? These comparisons would help establish whether the gains and intervention choices generalize across task-specialized models.


References

[3] Kang S, Kim J, Kim J, et al. Your large vision-language model only needs a few attention heads for visual grounding[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 9339-9350.

[4] Khayatkhoei M, Chhikara P, Ilievski F. Mllms know where to look: Training-free perception of small visual details with multimodal llms[C]//International Conference on Learning Representations. 2025, 2025: 68194-68213.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.