acceptodds
Withdrawn from ICLR 2027

THMAP : Temporal HeatMap for text-to-video Intelligence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A typical video contains more visual evidence than vision language models (VLMs) can process effectively, making query-relevant events easy to miss. Existing approaches either expand the visual context of VLMs at high training and inference cost, or train an external model to predict query-conditioned temporal regions as a preprocessing step, often requiring task-specific supervision on large training datasets. We study whether relevance is already encoded by the VLM that interprets the selected frames, and introduce THMAP. THMAP extracts a query-conditioned temporal relevance heatmap from text-to-vision token interactions that occur within a small set of attention heads grounding the user query in visual context. We decompose the outputs of these heads in a mechanistically interpretable manner to measure token-level relevance, and aggregate these signals into a temporal relevance heatmap at the desired granularity. To sharpen this heatmap, we learn a lightweight low-rank adaptation using a differentiable ranking objective and a modest amount of labeled data, requiring one to two orders of magnitude less supervision than typical existing approaches. The same heatmap scores clips for highlight detection and selects visual context for temporal grounding and video question answering. On highlight detection, THMAP improves over the previous state of the art by relative gains of in mAP and in Hit@. For temporal grounding, it improves mIoU by upto over a strong trained selector under an eight-frame budget, while remaining competitive across two VLMs for video question answering.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-language models, Video understanding, Temporal Relevance Heat Maps, Query-conditioned frame selection, VLM internals.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xwcsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the visual context limitations faced by vision-language models (VLMs) in video understanding tasks and proposes THMAP, a method that generates heatmaps of query-conditioned temporal relevance by leveraging the internal text-to-visual token interactions of pre-trained VLMs. The approach quantifies the contribution of visual tokens to the query representation via value projection, selects query-relevant attention heads using a relative contextualization metric, and aggregates the contributions of these selected heads into video-segment-level temporal relevance scores. These scores are then optimized under supervision using lightweight LoRA and a Smooth-AP-based ranking loss, enabling the generated heatmaps to be applied to video highlight detection, temporal localization, and video question answering. Experiments spanning multiple datasets, VLMs, and frame budget settings demonstrate significant performance gains in highlight detection and certain temporal localization tasks. However, the paper lacks sufficient validation regarding the independent contributions of core components; the primary performance gains rely heavily on supervised LoRA training; advantages in cross-task performance and computational efficiency remain insufficiently demonstrated; and key ablation studies are missing.

Critical strengths
  1. This paper attempts to extract temporal relevance directly using the cross-modal attention information within pre-trained VLMs, rather than designing separate networks for video retrieval or localization. This approach of deriving query-relevant visual evidence from within the model itself holds significant research value.
  2. The paper unifies video highlight detection, temporal localization, and video question answering under a temporal relevance heatmap framework, optimizing internal correlation signals via lightweight LoRA. This method offers flexibility by allowing the same feature extraction and scoring mechanisms to be reused across different downstream tasks.
  3. The paper achieves strong highlight detection results on the QVHighlights dataset and significantly improves performance on certain temporal localization tasks under low frame budgets. Experiments analyzing factors such as training data scale, different VLMs, the number of attention heads, and the use of frozen models further substantiate the method's effectiveness.
Critical weaknesses
  1. The "Value Projection" and "Relative Contextualization" components in this paper draw upon Attr-Kit and RCStat, respectively, while the subsequent "Head Selection," temporal aggregation, and LoRA ranking optimization are primarily based on existing techniques. The paper lacks sufficient ablation studies to demonstrate the necessity of each component—specifically, direct comparisons with raw attention weights, random or fixed head selection, and simple temporal scoring methods are missing—making the method's independent innovation somewhat unclear.
  2. The frame selection strategies vary across the different baselines. For instance, in the Temporal Grounding experiments, THMAP employs CDF sampling based on relevance distributions, whereas VideoITG uses Top-K frame selection. Consequently, differences in final localization performance may stem from both the quality of relevance scores and the sampling strategies. Furthermore, the methods compared in the highlight detection experiments utilize visual backbones of varying scales and types, further complicating the assessment of the method's intrinsic advantages.
  3. The efficiency analysis focuses primarily on heatmap generation latency with low-resolution video inputs, failing to adequately account for the additional inference costs associated with downstream "Grounder" or "Answerer" modules. THMAP utilizes an input resolution of 192×96 and a limited number of video frames, whereas some baselines employ higher resolutions and different candidate frame counts. Thus, the current comparison of computational efficiency does not fully isolate the algorithmic efficiency gains of the proposed method, nor does it provide sufficient validation regarding long videos or fine-grained visual evidence.
  4. THMAP achieves an mAP of only 26.9 on QVHighlights without LoRA training, whereas the full model reaches 47.33 mAP. This indicates that the final performance relies heavily on additional supervised training rather than stemming directly from the internal relevance signals of the pre-trained VLM. The paper lacks a comparison with simple relevance prediction methods under identical backbone, training data, and LoRA budget conditions; therefore, it cannot conclusively demonstrate that the performance improvement is primarily attributable to the proposed internal attribution mechanism.
Questions for authors
  1. Could the authors provide additional ablation studies isolating the Value Projection, RCScore Head Selection, and temporal aggregation methods, and compare them against baselines using raw attention weights or random/fixed head selection to demonstrate the necessity of each component?
  2. Since the performance of THMAP without LoRA training is significantly lower than that of the full model, could the authors compare it against a simple supervised temporal correlation prediction method—using the same VLM, training data, and LoRA parameter budget—to verify the independent contribution of the internal attribution mechanism?
  3. Could the authors re-evaluate Temporal Grounding using consistent frame sampling strategies across methods—specifically comparing THMAP and VideoITG under Top-K and CDF sampling—to rule out performance differences caused by the sampling strategy itself?
  4. There are significant differences in the visual backbones used by the various methods in Table 1; notably, this paper employs Qwen3-VL-8B, whereas most baselines rely on CLIP+SF or InternVideo2. Although Appendix F.2 presents experiments with different VLM backbones, a direct comparison using the same backbone and training budget is missing. Consequently, the observed SOTA performance gains might stem partly from superior vision-language representations rather than the proposed temporal correlation extraction mechanism. It is recommended to include corresponding ablation experiments. I will consider whether to revise my score based on the author's response and the results of the ablation study.
Reviewer_bUo6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

THMAP reads a query-conditioned temporal relevance curve from a VLM. After one prefill, the contribution of vision tokens to the query tokens is measured by value projection following Attr-Kit, a few heads are selected per example with the RCStat criterion, and their contributions are averaged per chunk. A LoRA adapter on Qwen3-VL-8B is trained with a ranking loss on about 4K QVHighlights and Charades pairs. The curve is used directly for highlight detection, where it reports the best result on QVHighlights, and as a frame sampler for temporal grounding and video QA.

Critical strengths

One relevance curve from a single prefill serves three tasks without a task-specific head, and in each sampler comparison all methods feed the same downstream model with the same prompt.

Critical weaknesses
  1. The main hypothesis is not supported by the paper's own results. The method applies Attr-Kit's value projection and RCStat's head selection to vision tokens, and that a few heads in a frozen LVLM ground the text query was already shown for images by Kang et al. at CVPR 2025, which is not discussed. Without LoRA, the readout reaches 26.9 mAP on highlight detection in Table 8, below every baseline, and is no better than uniform sampling for grounding on Charades. The results therefore come from supervised fine-tuning of an 8B VLM, not from relevance already encoded in it. Value projection, per-example head selection and the coverage loss are never ablated, so it is unclear whether the readout matters compared with a simple scoring head on the same fine-tuned model.
  2. The comparisons are not like-for-like. For highlight detection, a LoRA-tuned 8B VLM is compared with small models on frozen features, and its supervision, less than half of the QVHighlights training split plus some Charades, is not one to two orders of magnitude less than theirs. For grounding, THMAP is trained on QVHighlights and Charades spans while the other samplers are training-free or trained for VQA, and the gains shrink on ActivityNet, which is outside the training data. For video QA, THMAP is below VideoITG in most settings with Qwen3-VL and barely differs from uniform sampling on VideoMME-Long at 16 frames, although long videos motivate the work.
Questions for authors
  1. How does a simple scoring head on the same LoRA-tuned model compare with the THMAP readout, and what do raw attention weights or fixed heads give?
  2. Given that the frozen readout is below BLIP-ITM, in what sense is relevance already encoded in the VLM?
  3. Since the selector already reads every frame at 1 fps, how does the pipeline compare with giving the grounder or answerer those frames at the same low resolution?
Reviewer_B7EbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes THMAP, a unified query‑conditioned temporal relevance heatmap extraction framework for three video understanding tasks: highlight detection, temporal grounding and video QA. Instead of building an external scoring model or expanding VLM long‑context capacity, THMAP extracts relevance signals directly from internal text‑to‑vision value‑projection interactions inside a frozen vision‑language model. It selects query‑specific grounding attention heads via RCScore, aggregates token‑level contributions to produce temporal relevance curves, and sharpens noisy intrinsic signals with a lightweight LoRA adapter trained on merely ~5k query‑video pairs, two orders of magnitude fewer annotations than competing methods. The heatmap can directly rank clips for highlight detection or perform CDF‑based inverse‑transform frame sampling to feed compact visual context to downstream VLMs. Experiments on QVHighlights, TimeLens benchmarks, Video‑MME and LongVideoBench show THMAP establishes new state‑of‑the‑art for highlight detection, yields large mIoU gains for temporal grounding under tight frame budgets, and remains competitive on video QA across multiple VLM backbones.

Critical strengths

The work demonstrates usable query‑aware temporal relevance already exists within pretrained VLM cross‑modal attention, rather than relying on separate external similarity networks or heavy task‑specific prediction heads. This is conceptually meaningful: it shifts frame selection from training an independent scorer towards reading and calibrating intrinsic model signals. It unifies three traditionally separate video tasks under one shared heatmap mechanism without task‑specific output heads, which improves system modularity for video‑VLM pipelines. The work achieves clear SOTA improvements on highlight detection and substantial mIoU improvements for temporal grounding under scarce‑frame budgets, exactly the practical pain point where uniform sampling fails. Abolations confirm each core component (value‑projection attribution, RCScore head selection, LoRA ranking loss + coverage loss) contributes meaningfully.

Critical weaknesses
  1. The method extracts relevance signals by selecting example‑specific grounding attention heads inside Qwen3‑VL‑8B. Although the paper provides one cross‑backbone test on Nemotron‑Nano‑12B, the evaluation is limited only to highlight‑detection task. How consistent are these query‑conditioned grounding‑head patterns across diverse VLMs with very different multimodal attention architectures (e.g. interleaved vs separate vision‑text encoder, different attention masking schemes)? If the useful cross‑modal value‑projection signal is only available for a narrow set of VLMs, the general applicability of THMAP would be substantially limited. Could the authors provide more systematic cross‑VLM results on temporal grounding and video QA to verify that the internal‑signal‑driven pipeline does not overfit to Qwen‑style multimodal attention implementation?

  2. The paper explicitly acknowledges that THMAP operates purely over visual tokens and cannot leverage audio, speech or subtitle information. However, all the main benchmark evaluations are conducted on datasets where subtitles/audio are either ignored or not provided to any competing samplers. Real‑world long‑form video frequently carries critical evidence inside audio/subtitles for text‑query relevance. What performance degradation should practitioners expect when query‑relevant evidence exists only in audio/subtitle rather than visual frames? Have the authors designed any controlled ablation setting to quantify how large the performance gap becomes when visual cues are absent? Without such quantification, it is hard to judge the real‑world practical value for general‑purpose long‑video understanding.

  3. In temporal grounding and video QA evaluation, THMAP uses CDF inverse‑transform sampling over its heatmap under fixed small frame budgets. However, all competing samplers share exactly the same frame‑budget constraint. One concern is that THMAP’s advantages manifest primarily under very low‑frame‑budget regimes; as the frame budget grows large (e.g. 64‑128 frames, approaching dense uniform sampling), does THMAP retain meaningful advantages over simple uniform sampling? The paper only reports k=8,16,32 settings. Also, the CDF sampling is tied to the heatmap output; is there any risk that good downstream performance comes purely from this particular sampling algorithm, rather than quality of underlying temporal relevance heatmap itself? Could authors decouple heatmap quality from sampling strategy, e.g. by using top‑k ranking sampling instead of CDF sampling for ablation?

Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.