acceptodds
Under review as a conference paper at ICLR 2027

CoverVTG: Coverage-Constrained Sparse Observation for Universal Video Temporal Grounding

Abstract

Universal Video Temporal Grounding (VTG) aims to localize diverse natural-language queries in videos spanning different domains, viewpoints, and durations. Recent approaches largely rely on multimodal large language models (MLLMs), whose large parameter scale and dense visual encoding cost make them expensive for long videos and difficult to deploy in resource-constrained scenarios. In this work, we explore a lightweight backbone-centric alternative and propose CoverVTG, a coverage-constrained framework that directly reduces the number of visual tokens requiring expensive backbone encoding. Since shallow patch embeddings lack reliable high-level semantics, CoverVTG formulates pre-backbone sparsification as a structured sparse observation problem rather than semantic token selection. Within each local spatio-temporal tube, a sparse set of tokens is selected under explicit spatial and temporal coverage constraints, so that discarded observations remain locally observable from complementary retained tokens. We formalize this property through a coverage–observability–recoverability connection and establish a bound on local recovery error under spatio-temporal smoothness. Lightweight tube-local recovery adapters inserted throughout a frozen visual backbone progressively exploit such complementary observations as semantic representations emerge. Extensive experiments show that CoverVTG achieves competitive grounding accuracy with substantially fewer parameters and lower inference latency than MLLM-based approaches.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Video Temporal Grounding, Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mQoWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CoverVTG, a lightweight framework for Universal Video Temporal Grounding (VTG) that reduces the computational cost of visual encoding by performing token sparsification before the visual backbone encodes. The proposed Coverage-Constrained Token Assignment (CCTA) selects sparse visual tokens under spatial and temporal coverage constraints, while lightweight Recovery Adapters aggregate information from the preserved observations during visual encoding. Furthermore, based on spatiotemporal smoothness assumptions, the paper provides a theoretical analysis of local recoverability and adopts a coarse-to-fine inference strategy for temporal grounding. Experiments are conducted on four datasets — Ego4D-NLQ, TaCoS, Charades-STA, and ActivityNet-Captions — evaluating the model's zero-shot generalization, fine-tuned performance, inference efficiency, and the effectiveness of each module in the framework.

Critical strengths
  1. The paper notes that the high computational cost brought by dense visual backbone encoding is an important bottleneck in long-video temporal grounding. The proposed framework is organized around coverage-constrained sparse observation, feature recovery, and temporal localization, resulting in a clear overall structure.

  2. The paper conducts experiments on four benchmark datasets under both zero-shot and fine-tuned settings, and provides a fairly comprehensive evaluation of the proposed method through ablation studies, sampling strategy comparisons, and inference efficiency analysis.

Critical weaknesses
  1. CCTA adopts a maximum-weight assignment strategy with spatial and temporal coverage constraints, while the Recovery Adapter employs a fairly conventional bottleneck attention design. The paper fails to adequately clarify the technical distinction between CCTA and existing pre-encoding token reduction approaches, such as MeToM (CVPR 2026), nor does it sufficiently validate the unique advantage of its coverage-constrained formulation.

  2. The experiments do not include UniversalVTG (2026), which also targets lightweight universal VTG. Given that both methods share the same task setting and similar efficiency optimization goals, the absence of this comparison leaves CoverVTG's performance and efficiency advantages over recent related methods insufficiently verified.

  3. The compared methods employ different backbone architectures and pretraining configurations, making it difficult to accurately disentangle CoverVTG's own contributions from the effects of model configuration differences on performance and efficiency gains. Evaluation under more strictly controlled comparisons is needed to further clarify the actual benefits brought by the method design.

  4. CCTA relies only on shallow visual cues for token selection without considering query information; therefore, even when spatial and temporal coverage constraints are satisfied, it may miss key events that lack salient visual features or have short durations. The existing experiments fail to quantify the limitations of this strategy across different event characteristics and sparsity levels. Furthermore, Table 7 is based on only 10 videos and computes feature similarity on the retained token positions, without directly assessing the extent to which information in the discarded observations is preserved.

Questions for authors
  1. What are the technical differences and advantages of CCTA over existing methods? What key technical differences exist between CCTA and existing pre-encoding token reduction methods, especially MeToM (CVPR 2026)? Compared with other coverage-preserving sampling strategies, what specific advantages can the maximum-weight assignment bring?

  2. Why is the comparison with UniversalVTG missing? UniversalVTG (2026) is also a lightweight universal VTG method — why was it not included in the experimental evaluation? How does CoverVTG compare with this method in terms of grounding accuracy, model size, and inference efficiency?

  3. How robust is the query-independent token selection strategy? Under different sparsity levels, is the query-independent token selection strategy sufficiently robust for events with unobtrusive visual features or short durations? Additionally, could the authors directly evaluate feature recovery at the discarded token positions, rather than only computing feature similarity on the retained token positions as in Table 7?

  4. Can the actual contribution of the method be further verified through controlled experiments? Could the authors conduct controlled comparisons under matched backbone architectures and pretraining configurations, so as to more accurately disentangle the actual contribution of CoverVTG's method design to grounding performance and inference efficiency?

7 comments
AuthorsOpenReview ↗

Response to Reviewer mQoW (1)

We sincerely thank Reviewer mQoW for reviewing this paper.

Q1. Technical differences from MeToM and the benefit of maximum-weight assignment.

Comparison with MeToM. MeToM is a relevant pre-encoding reduction approach: its Residual-Guided Patch Merging (RPM) uses codec metadata to merge low-information regions before visual encoding. CoverVTG instead combines coverage-constrained observation before semantic encoding and information exchange among the retained observations during encoding. Thus, our distinction is the coverage-constrained observation-and-recovery formulation, rather than pre-encoding reduction alone.

There is also a concrete difference in the structure of encoder inputs. MeToM explicitly states in Section 3.2.2 that frame-level token sequences are padded to the maximum sequence length within the batch, with padding masks applied in self-attention and feed-forward layers, which, to some extent, diminishes the efficiency gains derived from sparsification.. CCTA regularizes frame-level token counts within the same sparsification configuration.

MeToM's Figure 3 and Section 4.3 also distinguish visual-encoder savings from LLM-prefill savings:

Component in MeToM's reported profilingFull TokensMeToM
Vision Tower525 ms450 ms
LLM backbone prefill1329 ms215 ms
Other50 ms53 ms
Total TTFT1904 ms718 ms

Its Vision Tower acceleration is approximately 525/450=1.17×525/450=1.17\times, a 14.3% latency reduction, while its overall TTFT acceleration is approximately 2.65×2.65\times. The larger overall gain primarily comes from reducing LLM prefilling time. This clarifies the different computational emphasis: CoverVTG focuses on reducing visual-backbone observations for lightweight temporal grounding.

Maximum-weight assignment and complementary recovery.

For a local tube with K2K^2 candidates, where K=L2K=L^2, CCTA retains KK observations by solving

\qquad \mathcal P_K=\{M\in\{0,1\}^{K\times K}:M\mathbf1=\mathbf1,\ M^\top\mathbf1=\mathbf1\}.$$ The constraints ensure coverage, while the objective selects the highest total shallow informativeness score among coverage-valid assignments. Table 5 compares CCTA with Phase Shift, a fixed coverage-preserving pattern: CCTA achieves **10.57/5.60**, versus **9.43/5.27**, improving R@1 at IoU 0.3/0.5 by **1.14/0.33 percentage points**. Our additional score-only ablation without the coverage constraint achieves **6.95/3.57**, compared with **10.57/5.60** for CoverVTG-B. These comparisons address content-adaptive assignment and the benefit of coverage, respectively. Table 4 further shows that, with CCTA and without coarse-to-fine inference, adding the adapter improves **5.27/2.38 to 6.42/4.07**, with latency increasing from **2.14 to 2.33 seconds**. This supports the practical benefit of information exchange under sparse observation while keeping the novelty claim focused on the overall formulation. **Q2. Comparison with UniversalVTG.** We agree that UniversalVTG[1] is a relevant lightweight baseline and should have been included. We initially omitted UniversalVTG because it uses an LLM-based Query Unifier to preprocess queries, whereas CoverVTG operates on the original queries. Although query unification improves grounding performance, this difference raised concerns about the comparability of the evaluation protocols. We acknowledge, however, that such protocol differences should be explicitly documented rather than used as a reason to exclude a relevant baseline. We provide the following comparison. UniversalVTG accuracy is taken from the Unifier-off row of its Table S1; CoverVTG-L accuracy is computed from our benchmark-specific fine-tuned results. Avg. R@1 averages IoU thresholds 0.3/0.5 for Ego4D-NLQ and TaCoS and 0.5/0.7 for Charades-STA and ActivityNet-Captions. | Method | Parameters | NLQ Avg. R@1 / latency | TaCoS Avg. R@1 / latency | Charades-STA Avg. R@1 / latency | ActivityNet-Captions Avg. R@1 / latency | |---|---:|---:|---:|---:|---:| | UniversalVTG, Unifier off | 735M | 21.05 / 6.32 s | 55.55 / 4.00 s | 63.03 / 1.59 s | 37.95 / 1.82 s | | CoverVTG-L | 754M | 22.99 / 4.74 s | 56.18 / 2.31 s | 63.75 / 1.84 s | 41.30 / 2.01 s | The table shows higher accuracy for CoverVTG-L than this Unifier-off configuration, with faster inference on the two long-video benchmarks but slightly slower inference on the two short-video benchmarks. Due to time constraints, we have not yet completed retraining UniversalVTG using original queries throughout training. Once retraining is complete, we will include the results in the revised manuscript. [1] An, Joungbin, Agrim Jain, and Kristen Grauman. "UniversalVTG: A Universal and Lightweight Foundation Model for Video Temporal Grounding." arXiv preprint arXiv:2604.08522 (2026).
AuthorsOpenReview ↗

Response to Reviewer mQoW (2)

Q3. Robustness of query-independent selection strategy, and retained-token recovery.

Robustness of query-independent selection strategy. CCTA uses query-independent shallow visual scores because reliable high-level query–patch alignment is not available in our pre-backbone representations. Coverage limits concentration in a few regions or frames, which can reduce the risk of leaving other local positions unobserved.

Short-event analysis. We evaluated the fine-tuned CoverVTG-B model on short-duration events in Ego4D-NLQ, using nested subsets of events lasting less than 1, 2, and 4 seconds, and report the results stratified by total video duration:

Event durationVideo duration (s)VideosHieraMamba R@[email protected]CoverVTG-B R@[email protected]
<1<1 s(128,512](128,512]1238.6610.19
<1<1 s(512,1152](512,1152]10.000.00
<1<1 s>1152>1152122.383.12
<2<2 s(128,512](128,512]22818.2823.75
<2<2 s(512,1152](512,1152]10.000.00
<2<2 s>1152>1152153.664.76
<4<4 s(128,512](128,512]26620.3427.11
<4<4 s(512,1152](512,1152]10.000.00
<4<4 s>1152>1152157.599.23

Under the stated adaptive budget, the video-duration ranges correspond to nominal retention fractions of 1/41/4, 1/91/9, and 1/161/16, respectively, within the supported single-chunk regime. CoverVTG outperforms HieraMamba in the six rows with nonzero recall.

In the shorter-video group, recall increases as the cumulative event threshold is relaxed, while the longest-video group has lower recall. These observations are consistent with the difficulty of locating very short events in larger temporal search spaces. And at 2 FPS, sub-second events can also be affected by temporal sampling before patch selection.

What is recovered, and the expanded feature analysis. We clarify that “recovery” refers to compensating for sparse-encoding degradation of the retained-token representations, not explicitly reconstructing features at discarded token positions.

A retained token can still lose useful context when other observations are removed, causing its encoded representation to differ from dense encoding. CCTA retains complementary spatial and temporal observations, and the adapter exchanges information among them to mitigate this degradation.

We expand the original 10-video diagnostic to the full evaluated Ego4D-NLQ validation set:

MethodMean cosine similarity at retained positions
Sparse encoding without Recovery Adapter0.61
Sparse encoding with Recovery Adapter0.81

For retained positions Rv\mathcal R_v in video vv, the diagnostic compares sparse features with dense-reference features at the same locations. The increase from 0.61 to 0.81 supports improved dense-reference consistency at retained positions on the expanded set. This directly evaluates the practical function of the adapter; it is not presented as reconstruction accuracy at missing positions or proof that all discarded semantic information is preserved.

We appreciate the reviewer’s suggestion to evaluate feature recovery at discarded positions. Our adapter is designed to improve the representations of retained tokens by exchanging information among complementary observations, and does not explicitly produce features at discarded positions. Consequently, we cannot directly report reconstruction quality at those positions with the current architecture. Such an evaluation would require an additional reconstruction module. We will clarify this distinction and the scope of our retained-position evaluation in the revised manuscript.

Q4. Controlled comparisons under matched backbones and pretraining.

As reported in our response to Q2, we provide an initial comparison between CoverVTG-L and UniversalVTG, both of which use PE-Core-L as the backbone and HieraMamba as the grounding head. The UniversalVTG results used in this comparison are reported as Avg. R@1, which does not provide a breakdown at individual IoU thresholds. Moreover, matching the backbone and grounding head alone does not eliminate differences in pretraining. Once our full reproduction of UniversalVTG is complete, we will include more detailed controlled comparisons under matched settings in the revised manuscript.

Reviewer_mQoWOpenReview ↗

Further Comment by Reviewer mQoW

I appreciate the authors' detailed response, but I still have several concerns:

  1. The ablation studies support the effectiveness of CCTA as a whole. However, my concern is specifically about the necessity of maximum-weight assignment. The current experiments do not establish its advantages over alternative assignment strategies under the same coverage constraints.

  2. The additional comparison with UniversalVTG under the Unifier-off setting is useful. However, since UniversalVTG originally employs an LLM-based Query Unifier, it would be helpful to evaluate CoverVTG with the same query preprocessing strategy and compare it against UniversalVTG with the Unifier enabled. This would provide a more complete comparison of their performance.

  3. The response does not fully address the robustness of query-independent token selection, particularly its ability to identify visually inconspicuous events that are highly relevant to the query.

Reviewer_jfvgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CoverVTG, an efficient framework for universal video temporal grounding. The main idea is to reduce redundant visual tokens before backbone encoding using coverage-constrained token selection. A lightweight recovery adapter is then introduced to compensate for information loss. The method achieves competitive performance with significantly lower computational cost.

Critical strengths
  1. The paper presents an interesting approach to reducing visual encoding costs for long videos.
  2. The coverage-constrained token selection is simple and intuitive, particularly for static scenes with substantial spatial-temporal redundancy.
  3. The experimental results demonstrate a promising accuracy-efficiency trade-off across multiple benchmarks.
Critical weaknesses
  1. Concerns about performance in dynamic scenes.

The proposed token selection seems reasonable for static backgrounds, where neighboring frames contain substantial redundant information. However, in dynamic scenes involving fast-moving objects or rapid camera motion, spatial and temporal consistency may not hold. The selected tokens could miss important object information across consecutive frames.

I would like to see a more detailed evaluation of the method under different motion conditions. For example, the authors could divide the test set into different difficulty levels based on object or camera motion and report the performance for each group. This would help clarify whether the proposed method remains effective in challenging dynamic scenes.

  1. Is the coverage constraint necessary?

The method requires selecting exactly one token from each row and column of the assignment matrix within a tube. While this constraint ensures spatial-temporal coverage, it may not be optimal for dynamic scenes, where important information could be concentrated in particular regions or frames.

I would be interested in an ablation that removes this constraint and selects tokens based only on their informativeness scores, while keeping the same token budget. Comparing these two strategies would help demonstrate whether the coverage constraint actually improves performance.

Questions for authors
  1. How does CoverVTG perform on videos with different levels of object motion and camera motion?

  2. What happens if the row-column coverage constraint is removed and tokens are selected solely based on informativeness scores under the same token budget?

1 comment
AuthorsOpenReview ↗

Response to Reviewer jfvg

We sincerely thank Reviewer jfvg for reviewing this paper.

Q1. Performance under different motion conditions.

We agree that rapid object motion and camera motion can challenge the local redundancy exploited by CoverVTG. To investigate this issue, we computed optical flow between neighboring frames and divided the Ego4D-NLQ evaluation videos into low- and high-motion groups according to their motion intensity. The fine-tuned CoverVTG-B results are:

Motion groupMethodR@1, IoU=0.3R@1, IoU=0.5
HighHieraMamba18.7913.01
HighCoverVTG-B24.0415.59
LowHieraMamba18.8413.09
LowCoverVTG-B25.4216.57

CoverVTG-B outperforms HieraMamba[1] in both groups, with improvements of 5.25/2.58 percentage points in the high-motion group and 6.58/3.48 percentage points in the low-motion group. Thus, its accuracy advantage over this reference persists under higher motion, although the margin becomes smaller.

We also acknowledge the observed sensitivity: CoverVTG-B scores 1.38/0.98 percentage points lower in the high-motion group, whereas HieraMamba's corresponding differences are 0.05/0.08 points. This behavior is consistent with the scope of our analysis in Section 2.3. Coverage ensures that retained observations are distributed across the spatial and temporal axes of each local tube, but it does not guarantee that the same moving object is observed at every time step. Rapid motion or abrupt changes can weaken the local smoothness assumption that motivates using neighboring observations as complementary information. The Recovery Adapter exchanges information among the retained observations; it cannot guarantee preservation of every transient event.

These results show that CoverVTG retains an accuracy advantage over HieraMamba in the evaluated high-motion subset, while motion sensitivity remains a limitation, as already discussed in Appendix A.2.

Q2. Is the coverage constraint necessary at the same token budget?

We agree that a matched-budget score-only ablation is needed. The lack of reliable high-level semantics before backbone encoding motivates a structural prior, but does not by itself demonstrate that coverage improves performance.

We therefore evaluated a variant that removes the coverage constraint and selects tokens using shallow informativeness scores. Both methods use PE-Core-B16 as the backbone, HieraMamba as the temporal grounding head, the same pretraining datasets and data configuration, and coarse-to-fine inference. We report zero-shot results on Ego4D-NLQ:

Selection strategyR@1, IoU=0.3R@1, IoU=0.5
Score-only selection without coverage constraints6.953.57
CCTA (CoverVTG-B)10.575.60

CCTA improves performance by 3.62 and 2.03 percentage points at IoU thresholds 0.3 and 0.5, respectively. This supports the utility of the coverage constraint in the evaluated configuration. Without coverage constraints, high shallow scores can concentrate selections in a few locations or frames, leaving other regions unobserved. This can reduce the diversity of complementary observations available to the Recovery Adapter. The adapter can still exchange information among the selected tokens, but it has fewer complementary observations to use when compensating for sparse-encoding degradation of the retained-token representations.

[1] An, Joungbin, and Kristen Grauman. "Hieramamba: Video temporal grounding via hierarchical anchor-mamba pooling." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

Reviewer_RpWaOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoverVTG is a lightweight approach to universal video temporal grounding. It uses Coverage-Constrained Token Assignment (CCTA) to prune patch tokens before visual encoding while maintaining local spatial and temporal coverage. Recovery Adapters exchange information among retained tokens within a frozen backbone. A HieraMamba head predicts temporal boundaries, while a clip-text contrastive loss supports coarse retrieval for long videos. Experiments on four benchmarks report competitive accuracy and lower latency than MLLM-based methods.

Critical strengths

S1. Pruning tokens before the backbone directly targets visual encoding cost, an important bottleneck for long videos.

S2. Results across four datasets are promising. Table 4 shows that CCTA reduces latency, while Recovery Adapters recover part of the lost accuracy.

Critical weaknesses

W1. The benefit of CCTA is not clearly isolated. An equal-budget resizing baseline is missing. Also, the average video lengths of TaCoS, Charades-STA, and ActivityNet-Captions fall below the 768-frame inference threshold, so CCTA may not be used on many samples.

W2. Key details of the VTC loss are missing, and the latency protocol for UniTime in Table 2 is unclear. Both limit reproducibility and the interpretation of the efficiency claims.

Questions for authors

Q1. Can CCTA be compared with aspect-ratio-preserving resolution reduction at matched token budgets and FPS?

Q2. How are positive and negative clip-text pairs defined for VTC? Are negatives sampled within a video, across the batch, or both? Please specify effective batch size and temperature.

Q3. At 2 FPS, 768 frames correspond to 384 seconds, longer than the average TaCoS, Charades-STA, and ActivityNet-Captions videos. What fraction of test samples actually use CCTA? Can results be broken down by video length?

Q4. Was UniTime's 17.02s latency on Charades-STA measured with the official implementation? Please clarify the timing scope and report visual encoding, LLM prefill, and decoding costs.

4 comments
AuthorsOpenReview ↗

Response to Reviewer RpWa (1)

We sincerely thank Reviewer RpWa for reviewing this paper.

Q1. Comparison with resolution reduction at matched token budgets and FPS.

We agree that resolution reduction is an important baseline. We conducted an additional zero-shot comparison on Ego4D-NLQ using PE-Core-B16 as the backbone and HieraMamba as the temporal grounding head. The resizing baseline uses the same pretraining datasets and data configuration as CoverVTG-B, and both methods use coarse-to-fine inference.

To match the encoder patch-token count, we adapt the resizing resolution to video duration while keeping the frame sampling rate at 2 FPS. Let DD denote video duration in seconds. The input spatial resolution is selected as

\begin{cases} 384, & D\le128, \cr 192, & 128<D\le512, \cr 128, & 512<D\le1152, \cr 96, & D>1152. \end{cases}$$ Thus, the comparison matches the number of visual patch tokens rather than merely using the same upper budget. For videos at most 128 seconds long, both methods retain the full $24\times24$ patch grid. For longer videos, the resizing baseline reduces spatial resolution in every frame, while CCTA retains an equal number of original-resolution patch observations distributed across space and time. | Method | R@1, IoU=0.3 | R@1, IoU=0.5 | |---|---:|---:| | Resolution reduction | 8.32 | 4.49 | | CoverVTG-B | **10.57** | **5.60** | CoverVTG-B improves the two metrics by **2.25 and 1.11 percentage points**, respectively. These results support the effectiveness of the combined coverage-constrained sparse observation and representation-recovery design relative to the resizing baseline. Whereas resizing reduces spatial detail in every frame, CCTA retains original-resolution patch observations distributed across neighboring frames, and the adapters exchange information among these complementary observations. **Q2. Positive and negative pairs, effective batch size, and temperature for VTC.** For each query, the positive visual representation is obtained by globally pooling the visual features of frames within its annotated ground-truth segment. Negative visual examples are drawn from both clips in the same video and clips from other videos in the batch. The fixed clip length is $T_{\mathrm{retrieval}}=8$ frames. Same-video negatives are required to have **zero temporal overlap** with the query's ground-truth segment. Negatives also include clips from other videos in the batch. We use a video-centric training strategy, and the **global batch size is 16 videos**. The temperature is inherited from PE-Core.
AuthorsOpenReview ↗

Response to Reviewer RpWa (2)

Q3. Actual inference threshold, fraction of videos using CCTA, and duration-stratified performance.

The reviewer is correct that the inference budget listed in Table 9, Nbudget=442,368N_{\mathrm{budget}}=442{,}368, implies a dense threshold of 768 frames. However, the benchmark evaluations use Nbudget=147,456N_{\mathrm{budget}}=147{,}456, the same budget as training. The larger setting was intended to describe an extended-capacity inference configuration on an A100 (40 GB), rather than the configuration used for the benchmark results. We apologize for the inaccurate distinction in the manuscript and have corrected Table 9 and the accompanying paragraph in Appendix A.3.

For the actual benchmark configuration,

=\frac{N_{\mathrm{budget}}}{H_pW_p} =\frac{147456}{24\times24} =256\ \mathrm{frames} =128\ \mathrm{seconds}\quad\text{at 2 FPS}.$$ Thus, CCTA is used during coarse retrieval for videos exceeding 256 sampled frames. Shorter videos are densely encoded, and fine grounding on the retrieved temporal window also uses dense encoding. The video-level activation frequencies are: | Benchmark | Videos at most 128 s | Videos longer than 128 s | Total videos | Videos using CCTA | |---|---:|---:|---:|---:| | Ego4D-NLQ | 0 | 415 | 415 | **100.00%** | | TaCoS | 5 | 20 | 25 | **80.00%** | | Charades-STA | 1,334 | 0 | 1,334 | **0.00%** | | ActivityNet-Captions | 2,714 | 2,171 | 4,885 | **44.44%** | CCTA is therefore active for all evaluated Ego4D-NLQ videos, most TaCoS videos, and a substantial subset of ActivityNet-Captions. It is not activated on the Charades-STA split. The duration-stratified **fine-tuned CoverVTG-B** results are: | Benchmark | Duration | Videos | R@1, lower IoU | R@1, higher IoU | |---|---|---:|---:|---:| | Ego4D-NLQ | $D\le128$ s | 0 | N/A | N/A | | Ego4D-NLQ | $D>128$ s | 415 | 24.59 | 15.98 | | TaCoS | $D\le128$ s | 5 | 72.80 | 64.17 | | TaCoS | $D>128$ s | 20 | 56.43 | 45.06 | | Charades-STA | $D\le128$ s | 1,334 | 69.17 | 52.40 | | Charades-STA | $D>128$ s | 0 | N/A | N/A | | ActivityNet-Captions | $D\le128$ s | 2,714 | 50.23 | 31.27 | | ActivityNet-Captions | $D>128$ s | 2,171 | 48.17 | 30.29 | The IoU thresholds are 0.3/0.5 for Ego4D-NLQ and TaCoS and 0.5/0.7 for Charades-STA and ActivityNet-Captions. On TaCoS, both CoverVTG-B and our HieraMamba[1] evaluation exhibit a substantial short-to-long performance gap: | Method | Short: R@[email protected] | Short: R@[email protected] | Long: R@[email protected] | Long: R@[email protected] | |---|---:|---:|---:|---:| | HieraMamba | 73.23 | 64.98 | 56.53 | 46.67 | | CoverVTG-B | 72.80 | 64.17 | 56.43 | 45.06 | This indicates that the duration-related performance gap is not unique to CoverVTG. A larger temporal search space and potentially more competing events offer plausible explanations, but the groups can also differ in content and query difficulty. **Q4. UniTime implementation, timing scope, and component costs.** We used UniTime’s official implementation to measure latency. The measured interval starts after video loading and ends when the model produces its final output; these numbers therefore represent model inference latency. The component-level measurements on an A100 are: | Component | Ego4D-NLQ | TaCoS | Charades-STA | ActivityNet-Captions | |---|---:|---:|---:|---:| | Visual encoding | 96.74 s | 54.52 s | 10.34 s | 17.48 s | | LLM prefill | 5.68 s | 3.51 s | 2.82 s | 3.16 s | | Decoding | 6.37 s | 4.17 s | 3.77 s | 3.93 s | | **Total** | **108.79 s** | **62.20 s** | **16.93 s** | **24.57 s** | [1] An, Joungbin, and Kristen Grauman. "Hieramamba: Video temporal grounding via hierarchical anchor-mamba pooling." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.
Reviewer_RpWaOpenReview ↗

Further Comment

Thank you for the additional experiments and clarifications. The matched-token resizing comparison, corrected inference configuration, and latency breakdown substantially address my main concerns and strengthen the evidence for the proposed accuracy–efficiency trade-off. I maintain my score.

AuthorsOpenReview ↗

Thank You for Your Constructive Feedback

Dear Reviewer RpWa,

Thank you for your careful review and constructive feedback. Your suggestions helped us strengthen the experimental comparisons and clarify the inference configuration and efficiency analysis. We appreciate your acknowledgment that the additional experiments and clarifications address your main concerns.

Best regards, Authors of Submission 16598

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank all reviewers for their constructive feedback and recognition of our work. Reviewer RpWa recognized the importance of reducing visual encoding costs and the effectiveness of CCTA and Recovery Adapters. Reviewer jfvg appreciated the intuitive coverage-constrained design and promising accuracy–efficiency trade-off. Reviewer mQoW highlighted the clear methodological framework and comprehensive evaluations across four benchmarks.

Building upon these recognized strengths, we conducted extensive additional experiments during the rebuttal to address concerns regarding the necessity of our design choices, comparisons with recent methods, and robustness under challenging conditions. We summarize the key findings below.

1. Controlled experiments substantiate the core technical contributions.

Our additional experiments systematically disentangle the benefits of coverage constraints and content-adaptive assignment. On Ego4D-NLQ (zero-shot), CCTA achieves 10.57/5.60 (R@1 at IoU 0.3/0.5), outperforming score-only selection without coverage (6.95/3.57), Random Permutation (8.97/5.01), and Phase Shift (9.43/5.27). The latter two satisfy identical coverage constraints and token budgets, demonstrating the additional benefit of maximum-weight assignment. Furthermore, CCTA outperforms resolution reduction (8.32/4.49) under matched visual-token counts and FPS. Together with the Recovery Adapter ablation, these results provide stronger evidence for our coverage-constrained observation and progressive representation-recovery design.

2. Additional comparisons strengthen the evidence for practical effectiveness.

We added comparisons with UniversalVTG[1], a closely related lightweight universal VTG method. Under the Unifier-off setting, CoverVTG-L achieves higher average grounding accuracy across all four benchmarks, with lower inference latency on the two long-video benchmarks. Following the reviewer's suggestion, we further evaluated both methods using the same GPT-5 Query Unifier. CoverVTG-L achieves 29.81/20.79 on Ego4D-NLQ, compared with UniversalVTG's 29.06/20.56. These results demonstrate CoverVTG's competitiveness under more comparable query-processing conditions.

3. New robustness analyses support the reliability of sparse observation and recovery.

CoverVTG-B maintains an accuracy advantage over HieraMamba[2] on high-motion videos (24.04/15.59 vs. 18.79/13.01). On a challenging 500-segment subset where query-relevant patches receive low shallow informativeness scores, performance (24.63/16.00) remains comparable to the full validation set (24.59/15.98). Furthermore, the Recovery Adapter improves retained-token feature similarity to dense-reference representations from 0.61 to 0.81 on the full Ego4D-NLQ validation set. These analyses support the effectiveness of coverage-preserving observation and complementary information exchange.

4. Clarifications of inference configurations and efficiency evaluation.

We clarified the inference-budget configurations and confirmed that CCTA is activated on 100% of Ego4D-NLQ, 80% of TaCoS, and 44.44% of ActivityNet-Captions videos. We also clarified the contrastive training settings and provided component-level UniTime[3] latency measurements using its official implementation, further strengthening the evidence supporting our efficiency claims.

Overall, the rebuttal substantially strengthens the empirical support for CoverVTG's central contribution: reducing expensive visual encoding through structured sparse observation while mitigating representation degradation through lightweight recovery. Through constructive discussions and positive feedback during the rebuttal, our paper achieved final reviewer scores of 3/3/3 (three Weak Accepts), with Reviewer mQoW raising their score and Reviewer RpWa acknowledging that their main concerns were substantially addressed. We believe these new results and clarifications reinforce the technical soundness, distinctiveness, and practical value of CoverVTG for efficient universal video temporal grounding.

[1] An, Joungbin, Agrim Jain, and Kristen Grauman. "UniversalVTG: A Universal and Lightweight Foundation Model for Video Temporal Grounding." arXiv preprint arXiv:2604.08522 (2026).

[2] An, Joungbin, and Kristen Grauman. "Hieramamba: Video temporal grounding via hierarchical anchor-mamba pooling." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[3] Li, Zeqian, et al. "Universal video temporal grounding with generative multi-modal large language models." Advances in Neural Information Processing Systems 38 (2026): 64426-64455.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.