acceptodds
Under review as a conference paper at ICLR 2027

SkimVTG: Video Temporal Grounding via Progressive Boundary Refinement

Abstract

Video Temporal Grounding (VTG) aims to localize the temporal segment in an untrimmed video that corresponds to a natural language query. Despite recent progress, precise boundary localization remains challenging because clip-query relevance scores often vary smoothly around the target moment, providing limited cues for identifying exact start and end boundaries. To address this, we propose SkimVTG, a progressive boundary refinement framework that first localizes the target moment from a temporally compressed representation and then refines its boundaries toward the original temporal resolution. At each temporal resolution, Boundary Cross-Attention (BCA) combines clip–query relevance with its temporal difference to extract boundary-sensitive cues, while the Temporal Bridging Network (TBN) propagates refined coarse-level representations to finer temporal resolutions through Temporal Distance Cross-Attention (TDCA). By alternating BCA and TBN, SkimVTG progressively sharpens coarse localization into precise temporal boundaries. Experiments on QVHighlights, Charades-STA, and TACoS demonstrate strong performance in Moment Retrieval and Highlight Detection. The code will be released upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Temporal Grounding, Coarse-to-Fine Boundary Refinement, Multi-Scale Temporal Modeling, Video-Language Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5NaYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The existing methods rely on a single temporal representation and clip–query relevance scores that vary smoothly near target moment boundaries, providing ambiguous and insufficient cues for precise start and end boundary estimation. The authors in this manuscript propose SkimVTG, a coarse-to-fine progressive boundary refinement framework. SkimVTG builds a temporal feature hierarchy and refines boundary representations from a coarse temporal scale to the original video resolution. The core framework consists of two main architectural modules: 1. Boundary Cross-Attention (BCA): At each temporal resolution, BCA extracts boundary-sensitive cues by combining clip–query relevance scores with their temporal differences, capturing relevance transitions into (positive transition) and out of (negative transition) target moments. 2. Temporal Bridging Network (TBN): TBN connects adjacent temporal resolutions via Temporal Distance Cross-Attention (TDCA). It incorporates a temporal-distance penalty into the attention logits to preserve local temporal alignment while propagating refined coarse-scale guidance to finer temporal scales.

The proposed method is evaluated on three benchmark datasets—QVHighlights, Charades-STA, and TACoS. It achieves superior or competitive performance compared to existing SOTAs. The authors demonstrate through ablation studies and dedicated boundary metrics that progressive refinement specifically sharpens boundary precision (e.g., improving [email protected] by +3.21 on QVHighlights in Table 1, and reducing boundary localization error from 4.52s to 2.69s on QVHighlights val split in Table 5).

Critical strengths
  1. This manuscript is well-written and well-organized.
  2. The proposed SkimVTG achieves SOTA performance especially under R1.
  3. The experiment results are extensive. The readers are able to get sufficient insights from the analysis experiment.
  4. The number of existing VTG methods that take the segement boundary into consideration is limitted, which shows the significance of the proposed method.
Critical weaknesses
  1. Table 4 shows that performance drops when the number of resolutions KK is increased from 3 to 4. Similarly, Table 6 shows performance degradation when the temporal subsampling stride is increased from 2 to 3 or 4. As acknowledged by the authors in Section 5 and Section E, fixed temporal hierarchies can over-compress short moments or subtle visual transitions. The authors could discuss or experiment with adaptive subsampling strategies (e.g., dynamic pooling based on video duration) or provide a breakdown of performance categorized by ground-truth moment duration to show how short versus long moments behave under different values of KK.
  2. The boundary detection task can be easily handelled by light-weight scene detector like autoshot [1] in real-world scenario. Is it really necessary to refine the moment boundary in VTG model. It would be great to compare the proposed method with the solution with the help of autoshot to detect the moment boundaries.

[1] Zhu, Wentao, et al. "Autoshot: A short video dataset and state-of-the-art shot boundary detection." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, 2023.

Questions for authors

Please see the weakness part.

2 comments

Thank you for the positive assessment and helpful suggestions. We address both concerns through additional experiments and will incorporate the results and corresponding revisions into the main text and appendix.

W1. Duration-dependent effects of hierarchy depth and stride

We break down the auxiliary K/stride experiment by ground-truth moment duration: short (0,10] s, middle (10,30] s, and long >30 s. Here K denotes the number of temporal resolutions.

We conduct an additional experiment to analyze how hierarchy depth K and subsampling stride affect performance across different ground-truth moment durations.

KStrideOverall mAP ↑Short ↑Middle ↑Long ↑
3247.4416.3446.4454.28
4242.379.9844.0246.21
3342.1510.0544.6048.95
3441.629.6842.3350.41

Relative to K=3/stride=2, short-moment mAP drops by 6.36 points with K=4 and by 6.29/6.66 points with stride 3/4, highlighting the importance of preserving fine temporal details. For long moments, mAP partially recovers from 48.95 to 50.41 as the stride increases from 3 to 4, although stride 2 remains best. This differing sensitivity across moment durations suggests that adaptively selecting K and the subsampling stride according to temporal characteristics may help balance global context and fine-grained boundary cues. We will investigate adaptive temporal hierarchies as a future research direction.

W2. Comparison with AutoShot boundary snapping

Thank you for suggesting AutoShot as a lightweight source of boundary cues. Shot boundaries indicate transitions between shots, whereas VTG boundaries depend on the query: an action may begin or end within a continuous shot, and a relevant moment may span multiple shots.

We evaluate AutoShot through post-processing: each predicted start/end boundary is moved to the nearest AutoShot boundary within a temporal radius rr, and remains unchanged if no detected boundary lies within ±rr seconds. Thus, the snapping radius specifies the maximum permitted temporal displacement.

All snapping experiments are evaluated on the QVHighlights validation split using the official evaluation protocol and the same SkimVTG checkpoint. The default snapping radius is 2 seconds, with an AutoShot detection threshold of [actual default threshold]. The threshold determines which detected shot boundaries are retained as snapping candidates. The radius sensitivity experiments change only the radius, while keeping the other settings fixed.

Variant[email protected] ↑mAP Avg. ↑Short mAP ↑
Ours49.0348.8016.34
+ AutoShot snapping48.0044.7011.98
Snapping radius 1 s49.1046.3013.95
Snapping radius 4 s47.5544.3611.95
Threshold 0.5, radius 2 s48.0044.6912.08

AutoShot snapping reduces mAP Avg. by 4.10 points and short-moment mAP by 4.36 points. Although a 1-second radius slightly improves [email protected] (+0.07), none of the tested settings improves overall or short-moment mAP. These results suggest that directly moving query-dependent predictions to shot boundaries can introduce misalignment.

Thank you for this suggestion. While direct snapping did not improve mAP in our experiments, we will investigate other ways to incorporate shot boundary information in future work.

Dear Reviewer 5NaY,

Thank you for your thoughtful review. We have posted our responses to your comments and questions, together with a revised PDF, and would be grateful if you could take a look when convenient.

Should anything remain unclear, we would be glad to provide further clarification and continue the discussion.

Thank you again for your time.

Reviewer_FerMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SkimVTG, a progressive boundary refinement framework for Video Temporal Grounding (VTG), aiming to improve the precision of temporal boundary localization. The authors argue that smoothly varying clip-query relevance scores make it difficult to identify accurate start and end boundaries. To address this issue, SkimVTG adopts a coarse-to-fine strategy that progressively refines video representations across multiple temporal resolutions. Specifically, the proposed Boundary Cross-Attention (BCA) incorporates temporal differences in clip-query relevance scores to capture boundary-sensitive cues, while the Temporal Bridging Network (TBN) employs Temporal Distance Cross-Attention (TDCA) to propagate information between adjacent temporal resolutions while preserving temporal locality.

Critical strengths

1. Competitive performance across multiple VTG benchmarks.
The proposed method demonstrates competitive performance on QVHighlights, Charades-STA, and TACoS, showing improvements over several evaluated baselines. The results suggest that the proposed framework is effective across different video grounding settings.


2. Boundary-focused experimental analysis.
The paper provides quantitative boundary localization analysis beyond conventional IoU-based evaluation metrics. In particular, the boundary error measurements and component-wise ablations offer useful evidence that the proposed modules contribute to more precise temporal boundary localization.


3. Clear and intuitive framework design.
The overall architecture is clearly presented, with well-defined roles for multi-resolution temporal representations, Boundary Cross-Attention, and the Temporal Bridging Network. The progressive refinement process is easy to follow, and the ablation studies help illustrate the contributions of individual components.

Critical weaknesses

1. Insufficient quantitative validation of the main motivation.

The paper argues that smoothly varying clip-query relevance scores are a key reason for inaccurate temporal boundary localization. However, this phenomenon is primarily illustrated through qualitative visualizations, without a systematic quantitative analysis of relevance smoothness or its relationship with boundary errors. Furthermore, if relevance scores change only weakly around the true boundaries, it remains unclear whether their temporal differences provide sufficiently discriminative boundary cues. Although the ablation studies demonstrate performance improvements from using temporal differences, they do not fully explain when or why these cues are effective.


2. Incomplete comparison with recent state-of-the-art VTG methods.

Several relevant recent VTG approaches, including FlashVTG (WACV 2025), DualGround (NeurIPS 2025), and CVA (CVPR 2026), are missing from the experimental comparisons. This is particularly concerning because the paper emphasizes precise boundary localization, yet some of these methods report stronger performance on strict localization metrics. For instance, CVA achieves 55.32 [email protected] on QVHighlights, compared to SkimVTG's 50.81. Although the methods should be compared under consistent feature and evaluation settings, the omission of these baselines makes it difficult to assess the proposed method's competitiveness.


3. Limited methodological novelty.

The proposed architecture appears to be primarily a combination of existing techniques rather than a substantial methodological advancement. The coarse-to-fine refinement strategy is a well-established paradigm, while BCA introduces a simple first-order temporal difference of clip-query relevance scores, and TBN employs cross-resolution attention with an explicit temporal distance penalty. These components are conceptually similar to existing multi-scale feature refinement, temporal derivative-based boundary cues, and position-aware attention mechanisms.

Although the proposed combination achieves competitive performance, it remains unclear what fundamentally new methodological insight is introduced beyond integrating these established techniques. Furthermore, the ablation studies mainly compare against simplified variants rather than strong existing architectural alternatives, making it difficult to assess whether the proposed design offers distinctive advantages over established approaches.

Questions for authors
  1. Quantitative analysis of relevance smoothness and temporal differences.
    The paper argues that smoothly varying clip-query relevance scores make precise boundary localization difficult. Could the authors provide a quantitative analysis of this phenomenon, particularly its relationship with boundary localization errors? Furthermore, beyond the performance improvements shown in the ablation studies, it would be valuable to investigate how the temporal-difference cue actually affects the model's predictions. For instance, does it consistently sharpen attention around ground-truth boundaries, and under what conditions does it provide the largest improvements?

  1. Comparisons with recent VTG methods.
    Several relevant recent approaches, including FlashVTG, DualGround, and CVA, are missing from the experimental comparisons. Could the authors provide comparisons with these methods under consistent evaluation protocols and feature settings to better establish the competitiveness of SkimVTG?

  1. Generalization to stronger video representations.
    The current experiments primarily use SlowFast and CLIP features. Does the proposed framework remain effective when using stronger video representations, such as InternVideo2? In particular, does the temporal-difference cue still provide meaningful improvements when the underlying video features offer stronger video-language alignment?

  1. Performance under strict temporal localization metrics.
    Since precise boundary localization is the primary objective of SkimVTG, could the authors further explain why the method does not consistently outperform existing approaches at higher IoU thresholds, such as [email protected]? A more detailed analysis of boundary errors and failure cases relative to stronger recent baselines would help clarify when the proposed refinement is effective and where it remains limited.

  1. Comparison with established architectural alternatives.
    The proposed architecture combines coarse-to-fine refinement, temporal relevance differences, and distance-aware cross-attention. Could the authors clarify the distinctive methodological contributions of these components compared with established approaches? In particular, comparisons with conventional multi-scale feature fusion and cross-attention equipped with temporal positional encodings or relative position biases would help determine whether the improvements arise from the proposed mechanisms or from more general architectural choices.
3 comments

Thank you for the detailed suggestions. We provide further analyses of boundary cues, video representations, and architectural alternatives, which we will include alongside the relevant revisions in the main text and appendix.

W1, Q1. Quantitative motivation and temporal differences

We quantitatively evaluate whether relevance changes provide discriminative boundary cues:

AnalysisMetricValue
Relevance-based boundary detectionBoundary AP (%)31.29
Difference-based boundary detectionBoundary AP (%)60.07
Largest-change locationHit rate within ±2 s (%)60.06
Uniform-position baselineHit rate within ±2 s (%)11.31

Temporal differences improve boundary AP by 28.78 points, while the largest-change location achieves a boundary hit rate 5.31 times the uniform-position baseline.

We further group moments by relevance-change strength near their ground-truth boundaries:

Change strengthFull MAE (s)No-Δα MAE (s)Reduction (s)
Weak21.6423.161.52
Middle15.7617.581.82
Strong12.9515.242.29

Weak relevance changes are associated with higher boundary errors. Incorporating Δα reduces MAE across all groups, including weak transitions, with larger gains for stronger transitions. These results support complementing relevance magnitude with temporal variation for precise boundary localization.

Q3. Generalization to InternVideo2 features

We additionally evaluate the temporal-difference cue using InternVideo2 features:

VariantmAP Avg. ↑[email protected] ↑
No temporal difference37.9636.00
Full40.0238.39

Incorporating Δα improves mAP by 2.06 points and [email protected] by 2.39 points, supporting its effectiveness across different video representations.

W2, Q2, Q4. Recent baselines and strict localization

We will add comparisons with FlashVTG, DualGround, and CVA to the revised main paper and update the SOTA-related statements accordingly. We additionally compare boundary localization with FlashVTG:

MethodBoundary MAE (s) ↓Both boundaries within ±5 s (%) ↑
SkimVTG13.2945.71
FlashVTG14.0944.65

SkimVTG reduces boundary MAE by 0.80 s (5.7%) and improves both-boundary accuracy by 1.06 percentage points. This demonstrates more accurate boundary localization than FlashVTG in this evaluation, although overall retrieval scores are not consistently highest.

Our controlled ablations further show that incorporating Δα improves [email protected] from 49.12 to 50.81 and [email protected] from 41.59 to 44.70. The change-strength analysis identifies weak-transition moments as more challenging while showing improvements across all groups. Together, these results substantiate the contribution of the proposed refinement to boundary precision.

W3, Q5. Methodological contribution and architectural alternatives

Our contribution is to explicitly form query-conditioned boundary-transition cues and preserve them throughout cross-resolution refinement. BCA adds a learned projection of relevance and its signed temporal difference to query-aligned features. TBN first retrieves fine temporal details into coarse representations, then propagates the refined guidance to the finer resolution. TDCA aligns both interactions through temporal distances measured on the original-resolution timeline.

We compare these mechanisms with established architectural alternatives:

TDCA comparison[email protected] ↑mAP ↑
Full46.3243.49
Positional encoding45.6142.61
Learned relative-position bias44.8442.51
TBN comparison[email protected] ↑mAP ↑
Full46.3243.49
One-way cross-attention43.8142.26
Interpolation + concat/projection43.4240.58

TDCA improves mAP by 0.88 and 0.98 points over positional encoding and learned relative-position bias, respectively. TBN improves mAP by 1.23 points over one-way cross-attention and 2.91 points over interpolation-based fusion. Both tables use a shared Full-model reference selected from separate runs. These comparisons support the effectiveness of explicit temporal-distance weighting and two-step cross-resolution propagation beyond component-removal ablations.

Dear Reviewer FerM,

Thank you for your thoughtful review. We have posted our responses to your comments and questions, together with a revised PDF, and would be grateful if you could take a look when convenient.

Should anything remain unclear, we would be glad to provide further clarification and continue the discussion.

Thank you again for your time.

Thank you for the additional analyses and experiments.

They provide useful evidence for the effectiveness of the proposed components. However, W2 remains partially addressed, as the rebuttal provides a boundary-error comparison with FlashVTG while deferring comparisons with DualGround and CVA. In addition, InternVideo2 ablation shows improvements from the temporal-difference cue, but the reported performance is substantially below published results from existing state-of-the-art architectures using InternVideo2 features. The insufficiently specified evaluation settings further limit interpretation of this gap. Regarding W3, the architectural comparisons support the proposed design choices but do not sufficiently establish a distinctive methodological advantage over existing boundary-aware and multi-scale refinement approaches. My concerns about competitiveness and methodological novelty therefore remain. Therefore, I retain my original rating.

Reviewer_vSsaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SkimVTG, a coarse-to-fine framework for video temporal grounding. It first identifies candidate regions at a coarse temporal resolution and progressively refines their boundaries at finer resolutions. Its main components use query-relevance changes and distance-biased cross-resolution attention to introduce and propagate boundary cues across temporal scales. The paper evaluates the method on QVHighlights, Charades-STA, and TACoS, and reports comparisons and component ablations. The results are competitive on some metrics, but the current evidence does not fully establish the advantages of the proposed cross-resolution mechanisms over existing multi-scale approaches.

Critical strengths

The paper presents a coherent coarse-to-fine temporal grounding pipeline, and the use of cross-resolution attention to propagate boundary cues is an intuitive design for refining temporal boundaries. The evaluation spans three commonly used benchmarks and includes component ablations. On TACoS, the reported R@1 at IoU 0.7 is higher than EAMAT, suggesting that the approach can be effective in at least some high-IoU settings.

Critical weaknesses
  1. SkimVTG lacks sufficient methodological novelty. SkimVTG first identifies candidate regions at a coarse temporal resolution and then refines their boundaries at finer resolutions. Coarse-to-fine grounding and boundary refinement have already been explored in prior work [1][2][3]. The paper does not clearly explain what technical advance query-relevance changes and distance-biased cross-resolution attention introduce beyond existing multi-scale fusion and cross-scale attention mechanisms.
  2. The experimental evaluation is insufficient. The comparisons on Charades-STA and TACoS cover limited and partly dated baselines. On QVHighlights, SkimVTG’s R@1 at IoU 0.7 is below CausalVTG, while higher mAP at a different threshold does not demonstrate better top-ranked localization. The reported ablations also do not establish whether the proposed mechanisms outperform standard multi-scale fusion, proposal refinement, or cross-scale attention. Recent competitive baselines and controlled comparisons are needed to substantiate the claimed advantages.
  3. The framework is computationally expensive, but its performance advantages are limited. SkimVTG uses a coarse-to-fine pipeline with repeated refinement across temporal resolutions. The authors also acknowledge substantial resource consumption and report using eight H200 GPUs. However, the reported results do not demonstrate sufficiently strong gains over competitive baselines to justify this computational cost.

[1] Qi S, Yang L, Li C, et al. Coarse-to-fine spatial-temporal relationship inference for temporal sentence grounding[J]. IEEE Access, 2021, 9: 97430-97443. [2] Hao J, Sun H, Ren P, et al. Fine-grained text-to-video temporal grounding from coarse boundary[J]. ACM Transactions on Multimedia Computing, Communications and Applications, 2023, 19(5): 1-21. [3] Dong J, Yin Z. Boundary-aware temporal sentence grounding with adaptive proposal refinement[C]//Asian Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 641-657.

Questions for authors
  1. Can the authors compare SkimVTG with recent, relevant temporal-grounding methods on Charades-STA, TACoS, and QVHighlights under matched feature, supervision, and evaluation protocols? Can they also compare the proposed cross-resolution attention against standard multi-scale fusion or cross-scale attention, so that the gains attributable to the proposed design can be isolated?
  2. Can the authors report parameter count, FLOPs, inference latency, peak memory, and clarify whether feature extraction and training are included in the cost? How does SkimVTG's accuracy-efficiency trade-off compare with recent specialized and MLLM-based methods under comparable protocols?
3 comments

Thank you for the constructive feedback. Our responses below present additional architectural comparisons, boundary evaluations, and efficiency measurements. We will update the main text and appendix to reflect these findings and the accompanying clarifications.

W1, W2, Q1. Technical contribution and architectural comparisons

Our contribution is the explicit formation and cross-resolution propagation of query-conditioned boundary-transition cues. BCA combines relevance magnitude with its signed temporal difference to represent semantic support and transitions into and out of relevant moments. TBN first retrieves fine temporal details into coarse representations, then propagates the refined guidance to the finer resolution. TDCA incorporates temporal distances measured on the original-resolution timeline into both interactions.

We additionally compare these mechanisms with established architectural alternatives:

TDCA comparison[email protected]mAP
Full46.3243.49
Positional encoding45.6142.61
Learned relative-position bias44.8442.51
TBN comparison[email protected]mAP
Full46.3243.49
One-way cross-attention43.8142.26
Interpolation + concat/projection43.4240.58

TDCA improves [email protected]/mAP by 0.71/0.88 points over positional encoding and 1.48/0.98 points over learned relative-position bias. TBN improves these metrics by 2.51/1.23 points over one-way cross-attention and 2.90/2.91 points over interpolation-based fusion. Both tables use a shared Full-model reference selected from separate runs. These results support the effectiveness of explicit temporal-distance weighting and two-step cross-resolution propagation.

W2, Q1. Recent baselines and strict localization

We will expand the baseline comparisons on Charades-STA, TACoS, and QVHighlights in the revised main paper, specifying the feature, supervision, and evaluation settings, and update the SOTA-related statements accordingly.

On QVHighlights, SkimVTG achieves [email protected] of 50.81 compared with CausalVTG’s 51.42, while obtaining higher mAP Avg. (48.80 versus 47.40). To directly assess boundary precision beyond retrieval scores, we additionally compare SkimVTG with FlashVTG:

MethodBoundary MAE (s) ↓Both boundaries within ±5 s (%) ↑
SkimVTG13.2945.71
FlashVTG14.0944.65

SkimVTG reduces boundary MAE by 0.80 s (5.7%) and improves both-boundary accuracy by 1.06 percentage points. These results demonstrate more accurate temporal boundary localization than FlashVTG in this evaluation.

W3, Q2. Computational cost and efficiency

We compare SkimVTG and EaTR under the same profiling environment and evaluation settings:

ModelParametersFLOPsMedian / p95 latencyPeak allocation
SkimVTG6.27 M0.515 G5.496 / 5.744 ms68.18 MiB
EaTR9.10 M0.860 G6.647 / 9.053 ms77.45 MiB

SkimVTG uses 31.1% fewer parameters and 40.1% fewer FLOPs, while reducing median inference latency by 17.3% and peak memory allocation by 12.0%. These results demonstrate greater computational efficiency than EaTR under the same conditions. The eight H200 GPUs reported in the manuscript refer to the training setup.

We will include a broader comparison of grounding accuracy and computational efficiency in the revised manuscript.

Dear Reviewer vSsa,

Thank you for your thoughtful review. We have posted our responses to your comments and questions, together with a revised PDF, and would be grateful if you could take a look when convenient.

Should anything remain unclear, we would be glad to provide further clarification and continue the discussion.

Thank you again for your time.

Reviewer_vSsaOpenReview ↗

Official Comment by Reviewer vSsa

Thank you for your response and clarifications. However, I still have concerns about the limited methodological novelty and insufficient experimental validation, particularly the evaluation on a limited number of datasets. I also agree with Reviewer FerM's concern that the InternVideo2 results remain substantially below those of existing state-of-the-art methods using the same features. Therefore, I will maintain my original rating.

Comments on the paper

1 comment
Area_Chair_4YzjOpenReview ↗

Invitation to Discuss the Authors’ Response

Dear Reviewers,

The authors have now provided their response. I encourage all reviewers to read it and participate in the discussion with the authors to clarify any remaining questions or concerns.

Thank you to both the authors and reviewers for your time and effort throughout the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.