- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The existing methods rely on a single temporal representation and clip–query relevance scores that vary smoothly near target moment boundaries, providing ambiguous and insufficient cues for precise start and end boundary estimation. The authors in this manuscript propose SkimVTG, a coarse-to-fine progressive boundary refinement framework. SkimVTG builds a temporal feature hierarchy and refines boundary representations from a coarse temporal scale to the original video resolution. The core framework consists of two main architectural modules: 1. Boundary Cross-Attention (BCA): At each temporal resolution, BCA extracts boundary-sensitive cues by combining clip–query relevance scores with their temporal differences, capturing relevance transitions into (positive transition) and out of (negative transition) target moments. 2. Temporal Bridging Network (TBN): TBN connects adjacent temporal resolutions via Temporal Distance Cross-Attention (TDCA). It incorporates a temporal-distance penalty into the attention logits to preserve local temporal alignment while propagating refined coarse-scale guidance to finer temporal scales.
The proposed method is evaluated on three benchmark datasets—QVHighlights, Charades-STA, and TACoS. It achieves superior or competitive performance compared to existing SOTAs. The authors demonstrate through ablation studies and dedicated boundary metrics that progressive refinement specifically sharpens boundary precision (e.g., improving [email protected] by +3.21 on QVHighlights in Table 1, and reducing boundary localization error from 4.52s to 2.69s on QVHighlights val split in Table 5).
Critical strengths
- This manuscript is well-written and well-organized.
- The proposed SkimVTG achieves SOTA performance especially under R1.
- The experiment results are extensive. The readers are able to get sufficient insights from the analysis experiment.
- The number of existing VTG methods that take the segement boundary into consideration is limitted, which shows the significance of the proposed method.
Critical weaknesses
- Table 4 shows that performance drops when the number of resolutions is increased from 3 to 4. Similarly, Table 6 shows performance degradation when the temporal subsampling stride is increased from 2 to 3 or 4. As acknowledged by the authors in Section 5 and Section E, fixed temporal hierarchies can over-compress short moments or subtle visual transitions. The authors could discuss or experiment with adaptive subsampling strategies (e.g., dynamic pooling based on video duration) or provide a breakdown of performance categorized by ground-truth moment duration to show how short versus long moments behave under different values of .
- The boundary detection task can be easily handelled by light-weight scene detector like autoshot [1] in real-world scenario. Is it really necessary to refine the moment boundary in VTG model. It would be great to compare the proposed method with the solution with the help of autoshot to detect the moment boundaries.
[1] Zhu, Wentao, et al. "Autoshot: A short video dataset and state-of-the-art shot boundary detection." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, 2023.
Questions for authors
Please see the weakness part.
2 comments
Dear Reviewer 5NaY,
Thank you for your thoughtful review. We have posted our responses to your comments and questions, together with a revised PDF, and would be grateful if you could take a look when convenient.
Should anything remain unclear, we would be glad to provide further clarification and continue the discussion.
Thank you again for your time.
Thank you for the positive assessment and helpful suggestions. We address both concerns through additional experiments and will incorporate the results and corresponding revisions into the main text and appendix.
W1. Duration-dependent effects of hierarchy depth and stride
We break down the auxiliary K/stride experiment by ground-truth moment duration: short (0,10] s, middle (10,30] s, and long >30 s. Here K denotes the number of temporal resolutions.
We conduct an additional experiment to analyze how hierarchy depth K and subsampling stride affect performance across different ground-truth moment durations.
Relative to K=3/stride=2, short-moment mAP drops by 6.36 points with K=4 and by 6.29/6.66 points with stride 3/4, highlighting the importance of preserving fine temporal details. For long moments, mAP partially recovers from 48.95 to 50.41 as the stride increases from 3 to 4, although stride 2 remains best. This differing sensitivity across moment durations suggests that adaptively selecting K and the subsampling stride according to temporal characteristics may help balance global context and fine-grained boundary cues. We will investigate adaptive temporal hierarchies as a future research direction.
W2. Comparison with AutoShot boundary snapping
Thank you for suggesting AutoShot as a lightweight source of boundary cues. Shot boundaries indicate transitions between shots, whereas VTG boundaries depend on the query: an action may begin or end within a continuous shot, and a relevant moment may span multiple shots.
We evaluate AutoShot through post-processing: each predicted start/end boundary is moved to the nearest AutoShot boundary within a temporal radius r, and remains unchanged if no detected boundary lies within ±r seconds. Thus, the snapping radius specifies the maximum permitted temporal displacement.
All snapping experiments are evaluated on the QVHighlights validation split using the official evaluation protocol and the same SkimVTG checkpoint. The default snapping radius is 2 seconds, with an AutoShot detection threshold of [actual default threshold]. The threshold determines which detected shot boundaries are retained as snapping candidates. The radius sensitivity experiments change only the radius, while keeping the other settings fixed.
AutoShot snapping reduces mAP Avg. by 4.10 points and short-moment mAP by 4.36 points. Although a 1-second radius slightly improves [email protected] (+0.07), none of the tested settings improves overall or short-moment mAP. These results suggest that directly moving query-dependent predictions to shot boundaries can introduce misalignment.
Thank you for this suggestion. While direct snapping did not improve mAP in our experiments, we will investigate other ways to incorporate shot boundary information in future work.