- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the visual context limitations faced by vision-language models (VLMs) in video understanding tasks and proposes THMAP, a method that generates heatmaps of query-conditioned temporal relevance by leveraging the internal text-to-visual token interactions of pre-trained VLMs. The approach quantifies the contribution of visual tokens to the query representation via value projection, selects query-relevant attention heads using a relative contextualization metric, and aggregates the contributions of these selected heads into video-segment-level temporal relevance scores. These scores are then optimized under supervision using lightweight LoRA and a Smooth-AP-based ranking loss, enabling the generated heatmaps to be applied to video highlight detection, temporal localization, and video question answering. Experiments spanning multiple datasets, VLMs, and frame budget settings demonstrate significant performance gains in highlight detection and certain temporal localization tasks. However, the paper lacks sufficient validation regarding the independent contributions of core components; the primary performance gains rely heavily on supervised LoRA training; advantages in cross-task performance and computational efficiency remain insufficiently demonstrated; and key ablation studies are missing.
Critical strengths
- This paper attempts to extract temporal relevance directly using the cross-modal attention information within pre-trained VLMs, rather than designing separate networks for video retrieval or localization. This approach of deriving query-relevant visual evidence from within the model itself holds significant research value.
- The paper unifies video highlight detection, temporal localization, and video question answering under a temporal relevance heatmap framework, optimizing internal correlation signals via lightweight LoRA. This method offers flexibility by allowing the same feature extraction and scoring mechanisms to be reused across different downstream tasks.
- The paper achieves strong highlight detection results on the QVHighlights dataset and significantly improves performance on certain temporal localization tasks under low frame budgets. Experiments analyzing factors such as training data scale, different VLMs, the number of attention heads, and the use of frozen models further substantiate the method's effectiveness.
Critical weaknesses
- The "Value Projection" and "Relative Contextualization" components in this paper draw upon Attr-Kit and RCStat, respectively, while the subsequent "Head Selection," temporal aggregation, and LoRA ranking optimization are primarily based on existing techniques. The paper lacks sufficient ablation studies to demonstrate the necessity of each component—specifically, direct comparisons with raw attention weights, random or fixed head selection, and simple temporal scoring methods are missing—making the method's independent innovation somewhat unclear.
- The frame selection strategies vary across the different baselines. For instance, in the Temporal Grounding experiments, THMAP employs CDF sampling based on relevance distributions, whereas VideoITG uses Top-K frame selection. Consequently, differences in final localization performance may stem from both the quality of relevance scores and the sampling strategies. Furthermore, the methods compared in the highlight detection experiments utilize visual backbones of varying scales and types, further complicating the assessment of the method's intrinsic advantages.
- The efficiency analysis focuses primarily on heatmap generation latency with low-resolution video inputs, failing to adequately account for the additional inference costs associated with downstream "Grounder" or "Answerer" modules. THMAP utilizes an input resolution of 192×96 and a limited number of video frames, whereas some baselines employ higher resolutions and different candidate frame counts. Thus, the current comparison of computational efficiency does not fully isolate the algorithmic efficiency gains of the proposed method, nor does it provide sufficient validation regarding long videos or fine-grained visual evidence.
- THMAP achieves an mAP of only 26.9 on QVHighlights without LoRA training, whereas the full model reaches 47.33 mAP. This indicates that the final performance relies heavily on additional supervised training rather than stemming directly from the internal relevance signals of the pre-trained VLM. The paper lacks a comparison with simple relevance prediction methods under identical backbone, training data, and LoRA budget conditions; therefore, it cannot conclusively demonstrate that the performance improvement is primarily attributable to the proposed internal attribution mechanism.
Questions for authors
- Could the authors provide additional ablation studies isolating the Value Projection, RCScore Head Selection, and temporal aggregation methods, and compare them against baselines using raw attention weights or random/fixed head selection to demonstrate the necessity of each component?
- Since the performance of THMAP without LoRA training is significantly lower than that of the full model, could the authors compare it against a simple supervised temporal correlation prediction method—using the same VLM, training data, and LoRA parameter budget—to verify the independent contribution of the internal attribution mechanism?
- Could the authors re-evaluate Temporal Grounding using consistent frame sampling strategies across methods—specifically comparing THMAP and VideoITG under Top-K and CDF sampling—to rule out performance differences caused by the sampling strategy itself?
- There are significant differences in the visual backbones used by the various methods in Table 1; notably, this paper employs Qwen3-VL-8B, whereas most baselines rely on CLIP+SF or InternVideo2. Although Appendix F.2 presents experiments with different VLM backbones, a direct comparison using the same backbone and training budget is missing. Consequently, the observed SOTA performance gains might stem partly from superior vision-language representations rather than the proposed temporal correlation extraction mechanism. It is recommended to include corresponding ablation experiments. I will consider whether to revise my score based on the author's response and the results of the ablation study.