- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents CoverVTG, a lightweight framework for Universal Video Temporal Grounding (VTG) that reduces the computational cost of visual encoding by performing token sparsification before the visual backbone encodes. The proposed Coverage-Constrained Token Assignment (CCTA) selects sparse visual tokens under spatial and temporal coverage constraints, while lightweight Recovery Adapters aggregate information from the preserved observations during visual encoding. Furthermore, based on spatiotemporal smoothness assumptions, the paper provides a theoretical analysis of local recoverability and adopts a coarse-to-fine inference strategy for temporal grounding. Experiments are conducted on four datasets — Ego4D-NLQ, TaCoS, Charades-STA, and ActivityNet-Captions — evaluating the model's zero-shot generalization, fine-tuned performance, inference efficiency, and the effectiveness of each module in the framework.
Critical strengths
-
The paper notes that the high computational cost brought by dense visual backbone encoding is an important bottleneck in long-video temporal grounding. The proposed framework is organized around coverage-constrained sparse observation, feature recovery, and temporal localization, resulting in a clear overall structure.
-
The paper conducts experiments on four benchmark datasets under both zero-shot and fine-tuned settings, and provides a fairly comprehensive evaluation of the proposed method through ablation studies, sampling strategy comparisons, and inference efficiency analysis.
Critical weaknesses
-
CCTA adopts a maximum-weight assignment strategy with spatial and temporal coverage constraints, while the Recovery Adapter employs a fairly conventional bottleneck attention design. The paper fails to adequately clarify the technical distinction between CCTA and existing pre-encoding token reduction approaches, such as MeToM (CVPR 2026), nor does it sufficiently validate the unique advantage of its coverage-constrained formulation.
-
The experiments do not include UniversalVTG (2026), which also targets lightweight universal VTG. Given that both methods share the same task setting and similar efficiency optimization goals, the absence of this comparison leaves CoverVTG's performance and efficiency advantages over recent related methods insufficiently verified.
-
The compared methods employ different backbone architectures and pretraining configurations, making it difficult to accurately disentangle CoverVTG's own contributions from the effects of model configuration differences on performance and efficiency gains. Evaluation under more strictly controlled comparisons is needed to further clarify the actual benefits brought by the method design.
-
CCTA relies only on shallow visual cues for token selection without considering query information; therefore, even when spatial and temporal coverage constraints are satisfied, it may miss key events that lack salient visual features or have short durations. The existing experiments fail to quantify the limitations of this strategy across different event characteristics and sparsity levels. Furthermore, Table 7 is based on only 10 videos and computes feature similarity on the retained token positions, without directly assessing the extent to which information in the discarded observations is preserved.
Questions for authors
-
What are the technical differences and advantages of CCTA over existing methods? What key technical differences exist between CCTA and existing pre-encoding token reduction methods, especially MeToM (CVPR 2026)? Compared with other coverage-preserving sampling strategies, what specific advantages can the maximum-weight assignment bring?
-
Why is the comparison with UniversalVTG missing? UniversalVTG (2026) is also a lightweight universal VTG method — why was it not included in the experimental evaluation? How does CoverVTG compare with this method in terms of grounding accuracy, model size, and inference efficiency?
-
How robust is the query-independent token selection strategy? Under different sparsity levels, is the query-independent token selection strategy sufficiently robust for events with unobtrusive visual features or short durations? Additionally, could the authors directly evaluate feature recovery at the discarded token positions, rather than only computing feature similarity on the retained token positions as in Table 7?
-
Can the actual contribution of the method be further verified through controlled experiments? Could the authors conduct controlled comparisons under matched backbone architectures and pretraining configurations, so as to more accurately disentangle the actual contribution of CoverVTG's method design to grounding performance and inference efficiency?
7 comments
Response to Reviewer mQoW (2)
Q3. Robustness of query-independent selection strategy, and retained-token recovery.
Robustness of query-independent selection strategy. CCTA uses query-independent shallow visual scores because reliable high-level query–patch alignment is not available in our pre-backbone representations. Coverage limits concentration in a few regions or frames, which can reduce the risk of leaving other local positions unobserved.
Short-event analysis. We evaluated the fine-tuned CoverVTG-B model on short-duration events in Ego4D-NLQ, using nested subsets of events lasting less than 1, 2, and 4 seconds, and report the results stratified by total video duration:
| Event duration | Video duration (s) | Videos | HieraMamba R@[email protected] | CoverVTG-B R@[email protected] |
|---|---|---|---|---|
| s | 123 | 8.66 | 10.19 | |
| s | 1 | 0.00 | 0.00 | |
| s | 12 | 2.38 | 3.12 | |
| s | 228 | 18.28 | 23.75 | |
| s | 1 | 0.00 | 0.00 | |
| s | 15 | 3.66 | 4.76 | |
| s | 266 | 20.34 | 27.11 | |
| s | 1 | 0.00 | 0.00 | |
| s | 15 | 7.59 | 9.23 |
Under the stated adaptive budget, the video-duration ranges correspond to nominal retention fractions of , , and , respectively, within the supported single-chunk regime. CoverVTG outperforms HieraMamba in the six rows with nonzero recall.
In the shorter-video group, recall increases as the cumulative event threshold is relaxed, while the longest-video group has lower recall. These observations are consistent with the difficulty of locating very short events in larger temporal search spaces. And at 2 FPS, sub-second events can also be affected by temporal sampling before patch selection.
What is recovered, and the expanded feature analysis. We clarify that “recovery” refers to compensating for sparse-encoding degradation of the retained-token representations, not explicitly reconstructing features at discarded token positions.
A retained token can still lose useful context when other observations are removed, causing its encoded representation to differ from dense encoding. CCTA retains complementary spatial and temporal observations, and the adapter exchanges information among them to mitigate this degradation.
We expand the original 10-video diagnostic to the full evaluated Ego4D-NLQ validation set:
| Method | Mean cosine similarity at retained positions |
|---|---|
| Sparse encoding without Recovery Adapter | 0.61 |
| Sparse encoding with Recovery Adapter | 0.81 |
For retained positions in video , the diagnostic compares sparse features with dense-reference features at the same locations. The increase from 0.61 to 0.81 supports improved dense-reference consistency at retained positions on the expanded set. This directly evaluates the practical function of the adapter; it is not presented as reconstruction accuracy at missing positions or proof that all discarded semantic information is preserved.
We appreciate the reviewer’s suggestion to evaluate feature recovery at discarded positions. Our adapter is designed to improve the representations of retained tokens by exchanging information among complementary observations, and does not explicitly produce features at discarded positions. Consequently, we cannot directly report reconstruction quality at those positions with the current architecture. Such an evaluation would require an additional reconstruction module. We will clarify this distinction and the scope of our retained-position evaluation in the revised manuscript.
Q4. Controlled comparisons under matched backbones and pretraining.
As reported in our response to Q2, we provide an initial comparison between CoverVTG-L and UniversalVTG, both of which use PE-Core-L as the backbone and HieraMamba as the grounding head. The UniversalVTG results used in this comparison are reported as Avg. R@1, which does not provide a breakdown at individual IoU thresholds. Moreover, matching the backbone and grounding head alone does not eliminate differences in pretraining. Once our full reproduction of UniversalVTG is complete, we will include more detailed controlled comparisons under matched settings in the revised manuscript.
Further Comment by Reviewer mQoW
I appreciate the authors' detailed response, but I still have several concerns:
-
The ablation studies support the effectiveness of CCTA as a whole. However, my concern is specifically about the necessity of maximum-weight assignment. The current experiments do not establish its advantages over alternative assignment strategies under the same coverage constraints.
-
The additional comparison with UniversalVTG under the Unifier-off setting is useful. However, since UniversalVTG originally employs an LLM-based Query Unifier, it would be helpful to evaluate CoverVTG with the same query preprocessing strategy and compare it against UniversalVTG with the Unifier enabled. This would provide a more complete comparison of their performance.
-
The response does not fully address the robustness of query-independent token selection, particularly its ability to identify visually inconspicuous events that are highly relevant to the query.
Response to Reviewer mQoW (1)
We sincerely thank Reviewer mQoW for reviewing this paper.
Q1. Technical differences from MeToM and the benefit of maximum-weight assignment.
Comparison with MeToM. MeToM is a relevant pre-encoding reduction approach: its Residual-Guided Patch Merging (RPM) uses codec metadata to merge low-information regions before visual encoding. CoverVTG instead combines coverage-constrained observation before semantic encoding and information exchange among the retained observations during encoding. Thus, our distinction is the coverage-constrained observation-and-recovery formulation, rather than pre-encoding reduction alone.
There is also a concrete difference in the structure of encoder inputs. MeToM explicitly states in Section 3.2.2 that frame-level token sequences are padded to the maximum sequence length within the batch, with padding masks applied in self-attention and feed-forward layers, which, to some extent, diminishes the efficiency gains derived from sparsification.. CCTA regularizes frame-level token counts within the same sparsification configuration.
MeToM's Figure 3 and Section 4.3 also distinguish visual-encoder savings from LLM-prefill savings:
Its Vision Tower acceleration is approximately 525/450=1.17×, a 14.3% latency reduction, while its overall TTFT acceleration is approximately 2.65×. The larger overall gain primarily comes from reducing LLM prefilling time. This clarifies the different computational emphasis: CoverVTG focuses on reducing visual-backbone observations for lightweight temporal grounding.
Maximum-weight assignment and complementary recovery.
For a local tube with K2 candidates, where K=L2, CCTA retains K observations by solving
\qquad \mathcal P_K=\{M\in\{0,1\}^{K\times K}:M\mathbf1=\mathbf1,\ M^\top\mathbf1=\mathbf1\}.$$ The constraints ensure coverage, while the objective selects the highest total shallow informativeness score among coverage-valid assignments. Table 5 compares CCTA with Phase Shift, a fixed coverage-preserving pattern: CCTA achieves **10.57/5.60**, versus **9.43/5.27**, improving R@1 at IoU 0.3/0.5 by **1.14/0.33 percentage points**. Our additional score-only ablation without the coverage constraint achieves **6.95/3.57**, compared with **10.57/5.60** for CoverVTG-B. These comparisons address content-adaptive assignment and the benefit of coverage, respectively. Table 4 further shows that, with CCTA and without coarse-to-fine inference, adding the adapter improves **5.27/2.38 to 6.42/4.07**, with latency increasing from **2.14 to 2.33 seconds**. This supports the practical benefit of information exchange under sparse observation while keeping the novelty claim focused on the overall formulation. **Q2. Comparison with UniversalVTG.** We agree that UniversalVTG[1] is a relevant lightweight baseline and should have been included. We initially omitted UniversalVTG because it uses an LLM-based Query Unifier to preprocess queries, whereas CoverVTG operates on the original queries. Although query unification improves grounding performance, this difference raised concerns about the comparability of the evaluation protocols. We acknowledge, however, that such protocol differences should be explicitly documented rather than used as a reason to exclude a relevant baseline. We provide the following comparison. UniversalVTG accuracy is taken from the Unifier-off row of its Table S1; CoverVTG-L accuracy is computed from our benchmark-specific fine-tuned results. Avg. R@1 averages IoU thresholds 0.3/0.5 for Ego4D-NLQ and TaCoS and 0.5/0.7 for Charades-STA and ActivityNet-Captions. | Method | Parameters | NLQ Avg. R@1 / latency | TaCoS Avg. R@1 / latency | Charades-STA Avg. R@1 / latency | ActivityNet-Captions Avg. R@1 / latency | |---|---:|---:|---:|---:|---:| | UniversalVTG, Unifier off | 735M | 21.05 / 6.32 s | 55.55 / 4.00 s | 63.03 / 1.59 s | 37.95 / 1.82 s | | CoverVTG-L | 754M | 22.99 / 4.74 s | 56.18 / 2.31 s | 63.75 / 1.84 s | 41.30 / 2.01 s | The table shows higher accuracy for CoverVTG-L than this Unifier-off configuration, with faster inference on the two long-video benchmarks but slightly slower inference on the two short-video benchmarks. Due to time constraints, we have not yet completed retraining UniversalVTG using original queries throughout training. Once retraining is complete, we will include the results in the revised manuscript. [1] An, Joungbin, Agrim Jain, and Kristen Grauman. "UniversalVTG: A Universal and Lightweight Foundation Model for Video Temporal Grounding." arXiv preprint arXiv:2604.08522 (2026).