acceptodds
Under review as a conference paper at ICLR 2027

TIDE: Temporal Information-Dense Evidence Compression for Video Temporal Grounding

Abstract

Video temporal grounding (VTG) usually requires Large Vision-Language Models (LVLMs) to localize query-relevant events and accurately predict their start and end timestamps in untrimmed videos, incurring substantial computational and memory costs. Existing token compression methods, mostly developed for general video understanding, reduce visual redundancy by pruning or merging tokens based on feature similarity or visual saliency. However, these appearance-based criteria are not well aligned with temporal grounding: precise localization relies on subtle state transitions and coherent entity evolutions across frames rather than visual appearance alone. Pruning or merging tokens based solely on visual appearance can fragment entity trajectories and attenuate temporal contrasts between adjacent states. To address this problem, we propose TIDE, a token compression framework that shifts the objective from removing appearance-based redundancy to preserving temporal evidence and maintaining structural continuity. TIDE identifies piecewise-stable states by tracking cross-frame token correspondences, allocates token budgets based on query-conditioned transition signals, and recovers peripheral context through soft state-compatible routing. At a visual token retention rate of 12.5% on ActivityNet-Captions, TIDE outperforms the full-token baseline in mIoU with Qwen3-VL-8B and achieves a 4.7 speedup for compression plus LLM prefilling with Qwen3-VL-4B. The code is available at https://anonymous.4open.science/r/TIDE-1125.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Temporal Grounding, Token Compression, Optimal Transport, Evidence Redistribution, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Qj39OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TIDE, a training-free visual token compression framework for video temporal grounding with large vision-language models. It aims to preserve temporal evidence needed for event localization, rather than relying only on visual similarity or saliency. TIDE tracks cross-frame token correspondences to guide query-conditioned budget allocation, diverse anchor selection, and context fusion that discourages mixing across temporal states. Experiments on ActivityNet-Captions and Charades-STA report higher mIoU than the compared compression methods. The paper further evaluates the design through component ablations and additional VideoQA experiments.

Critical strengths
  1. Extensive experimental evaluation.

The authors evaluate TIDE across multiple backbones and token retention ratios on Charades-STA and ActivityNet-Captions, with additional VideoQA experiments. They further demonstrate the ablation studies and many analyses.

  1. An interesting focus on state transitions.

The idea of preserving state transitions during token compression for VTG is interesting. The paper emphasizes that precise localization relies on subtle state transitions rather than visual appearance alone.

Critical weaknesses

Insufficient analysis and justification of TIDE’s modules.

  • Tables 3 and 4 show that the modules improve performance, but provide limited insight into how well each module serves its intended purpose. For example, TIDE still outperforms the compared compression methods without message passing in the reported setting. With several modules combined and limited analysis, it is difficult to identify which design choices are key to solving the stated problem and driving the gains.
  • The paper also needs to better explain why the specific rules in these modules are necessary to address the stated problem, beyond describing how they use state-transition information. One way to improve this would be to first define key requirements for VTG compression and then explain which requirements each module addresses. This would help readers understand why each module is needed.
Questions for authors
  • Were the baseline latencies in Table 6 remeasured or taken from SemVID? Several values appear identical to those reported in SemVID, although Section 5.4 states that the timings were measured in the authors’ environment. Could the authors explain this and clarify whether the reported speedup is based on measurements obtained under the same conditions?

  • How does TIDE compare with FlashVID in terms of accuracy and efficiency? FlashVID achieves competitive grounding performance in Tables 1–2, but its latency is missing from Table 6. Could the authors report its compression and LLM prefilling times using the same backbone, input settings, and token retention ratio as TIDE?

  • How does TSF balance feature contextualization with preserving temporal contrast? In lines 052–053, the authors note that averaging across states can weaken temporal contrast, yet TSF aggregates adjacent-frame features before estimating transitions and states. While Table 4 shows a performance benefit from message passing, could the authors clarify when this aggregation helps correspondence estimation?

5 comments
AuthorsOpenReview ↗

Summary: Module evidence, unified timing, and analysis of temporal contextualization

We sincerely thank the reviewer for the careful reading and constructive suggestions, and for recognizing our experimental coverage and focus on state transitions. The questions about component contributions, timing comparisons, and temporal smoothing helped us strengthen the experimental evidence and clarify the method. Our main responses are:

  1. The modules address distinct compression decisions, with their contributions measured separately (W1). VTG compression must allocate limited capacity, choose informative observations, and recover the context needed to interpret them. EDA, ESS, and BPF handle these decisions using TSF's shared temporal reference. The ANet removal studies identify context recovery and temporal feature construction as major contributors; adding message passing to the remaining pipeline improves mIoU by 1.42 points. We pair these ablations with allocation, repetition, and GT-boundary routing diagnostics to explain how each rule serves its intended purpose. See Response 1.

  2. The updated speedup comes from a same-condition retest (Q1). The original baseline timings were quoted from SemVID on L40, while TIDE was measured by us on A800. We have remeasured every method under one hardware, software, and input configuration. The replacement result is 4.44× for compression plus LLM prefill, with Total defined as the sum of the reported stage medians. We will update Table 6 and the corresponding statements throughout the paper. See Response 2.

  3. TIDE provides higher grounding accuracy with additional compression cost (Q2). The new retest includes FlashVID's compression and prefill times. On Qwen3-VL-4B/ANet at 12.5% nominal retention, TIDE gains 5.36 mIoU in the main accuracy evaluation and adds 13.08 ms in the timing test. Prefill times are nearly identical, so the additional cost is concentrated in compression. The accuracy advantage also holds across the remaining main-table settings. See Response 2.

  4. Affinity-guided contextualization retains useful temporal evidence beyond plain smoothing (Q3). TSF retains half of each token's own feature and aggregates similar content from adjacent frames before estimating changes and states. To verify that this contextualization retains localization cues, we compare it with fixed-position temporal averaging using the same self weight and temporal neighborhood. TIDE improves mIoU over that control by 0.38/1.53 points on ANet/Charades-STA. Holding contextualized features fixed also shows higher GT-boundary recall with matched predecessors than with fixed positions. Together, these results support preserving grounding evidence through selective contextualization and correspondence-based change estimation. See Response 3.

The added experiments, method explanations, and corresponding revisions will be marked in blue in the updated PDF. We hope this additional evidence addresses your concerns and would be glad to discuss any remaining questions.

AuthorsOpenReview ↗

Response 1 to W1: Connecting task requirements, module behavior, and localization gains

Context recovery, the shared temporal representation, and message passing provide substantial localization gains on ANet. New diagnostics connect allocation, selection, and fusion to their intended roles. Following your suggestion, we organize the design around four requirements:

RequirementModule and design rationale
Compare content across framesTSF: spatial-semantic affinities guide aggregation and predecessor matching, providing a shared reference for change, trajectories, and states.
Balance visual complexity and temporal evidence within a fixed budgetEDA: effective rank gives complex static frames an allocation signal; transition terms emphasize change. A within-unit two-sided maximum shares the signal across adjacent observations, with integer quotas preserving the budget exactly.
Select informative anchors while controlling repetitionESS: the predecessor-similarity discount addresses repetition across frames; MMR promotes feature diversity within each frame.
Recover discarded context while respecting state differencesBPF: local soft top-k routing recovers context. A query-and-change gate controls the state penalty, favoring compatible destinations while retaining fallback routes.

EDA decides how many tokens each frame receives, ESS decides which tokens remain, and BPF decides how unselected context contributes to them. TSF supplies their common temporal reference.

Localization contributions. Tables 3–4 use Qwen3-VL-4B on ANet at 12.5% retention:

VariantmIoUFull TIDE minus variant
Full TIDE42.13—
w/o Context Fusion40.19+1.94
w/o TSF40.61+1.52
w/o Message Passing40.71+1.42
w/o Temporal Discount40.96+1.17
w/o Trajectory40.97+1.16
Rank-only allocation41.12+1.01

Removing TSF disables the shared representation and its dependent signals. Removing context fusion retains allocation and anchor selection. Removing message passing uses raw features and recomputes trajectories, states, and downstream computations. Each row measures full-pipeline performance under the specified intervention.

As you observed, the remaining pipeline performs well without message passing. Its additional contribution is measured within that pipeline: 40.71 to 42.13 mIoU. On Qwen3-VL-4B/Charades-STA, disabling TSF, using rank-only allocation, removing temporal discount, or removing context fusion lowers mIoU by 0.60, 0.71, 0.75, or 0.30 points.

Behavior diagnostics. We use Qwen3-VL-4B at 12.5% retention, with a fixed 5,000-query ANet subset and all 3,720 Charades-STA queries:

  • EDA: relative to uniform allocation within the same unit, quotas before/after ANet GT starts are 1.12/1.09, and before/after ends 1.07/1.10. Charades-STA values are 1.01/1.03 and 1.05/0.99. Thus, ANet internal boundary neighborhoods receive 7%–12% more capacity on average.
  • ESS: with allocation and MMR fixed, temporal discount changes adjacent-frame position reuse from 0.51 to 0.50 on ANet and 0.51 to 0.49 on Charades-STA. Mean selected-token change increases from 0.11 to 0.12 and 0.07 to 0.08; trajectory reuse remains approximately 0.85/0.75.
  • BPF: with anchors fixed, the fraction of fusion weight crossing GT boundaries for positively gated sources within ±1 visual time step falls from 18.48% to 14.69% on ANet and 19.75% to 16.14% on Charades-STA. Admission rates change from 96.83% to 95.25% and 99.66% to 99.49%; zero-gate routing is unchanged.

Finally, w/o fusion / α=0 / full TIDE use the same selected anchors and yield 40.19/42.08/42.13 mIoU on ANet and 53.39/53.30/53.69 on Charades-STA. The α=0 comparison retains the other fusion rules while admission and routing weights respond to α. These controls separate context recovery from the state penalty: on ANet, recovery adds 1.89 points and the penalty another 0.05; on Charades-STA, the penalty adds 0.39 points. We will present performance and behavior results together to explain each design choice.

AuthorsOpenReview ↗

Response 2 to Q1 + Q2: Timing provenance, a unified retest, and comparison with FlashVID

Under the unified timing protocol, TIDE has a reported compression-plus-prefill total of 291.43 ms, giving 4.44× over Full-token from the stage statistics. Relative to FlashVID, it adds 13.08 ms (4.70%) while achieving 5.36 higher mIoU in the ANet/Qwen3-VL-4B main-table comparison.

Provenance and retest. The original Full-token, VisionZip, FastVID, and SemVID timing rows were quoted from SemVID Table 6 (L40); TIDE was measured by us on A800. The original 4.7× ratio combined these sources. We will correct the source description and replace the table with our retest of all six methods, including FlashVID, under the same hardware, software, and input settings.

The retest is conducted under the same configuration and experimental environment, using Qwen3-VL-4B, 12.5% nominal retention, three warm-up runs, and five repetitions. Compression is the method-specific net cost after subtracting shared Full-token visual/assembly overhead. Total sums the reported compression and prefill medians across videos; speedup divides Full-token Total by method Total. The scope covers compression and prefill, excluding video decoding, vision encoding, and autoregressive generation.

MethodCompression (ms)Prefill (ms)Total (ms; speedup)mIoU(%)
Full-token—1293.361293.36 (1.00×)42.93100.00
VisionZip556.73201.14757.87 (1.71×)32.3775.40
FastVID20.52183.90204.42 (6.33×)33.9479.06
FlashVID84.55193.80278.35 (4.65×)36.7785.65
SemVID65.16191.82256.98 (5.03×)38.4995.40
TIDE97.53193.90291.43 (4.44×)42.1398.14

Timing comes from the new 200-video test. Accuracy and relative Full-token scores (%) come from the ANet main table. Italic SemVID accuracy entries are quoted from its paper and use its own Full-token baseline; the other accuracy entries are our evaluations.

FlashVID and TIDE take 84.55/97.53 ms for compression and 193.80/193.90 ms for prefill. Their totals differ by 13.08 ms. This quantifies the additional cost associated with TIDE's localization gains. Methods use the same nominal retention; actual visual-token counts depend on each implementation. FlashVID retains more visual tokens than TIDE in this timing batch.

Accuracy across settings. TIDE has higher mIoU than FlashVID in all 12 main-table settings: two datasets, three backbones, and two retention ratios. At 12.5% retention:

DatasetBackboneFlashVID mIoUTIDE mIoUGain
ANetQwen3-VL-4B36.7742.13+5.36
ANetQwen3-VL-8B35.4742.75+7.28
ANetQwen3.5-4B40.2445.28+5.04
Charades-STAQwen3-VL-4B52.8253.69+0.87
Charades-STAQwen3-VL-8B54.9255.34+0.42
Charades-STAQwen3.5-4B52.3854.75+2.37

On the two Qwen3-VL backbones, TIDE at 12.5% retention also exceeds FlashVID at 25% on ANet (42.13/42.75 vs. 39.43/39.63). We will replace Table 6 with the unified retest, add FlashVID, and update the compression-plus-prefill speedup from 4.7× to 4.44× throughout the paper. Fixed-shape module profiling will retain its separate measurement conditions.

Reviewer_Z41pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is clearly written and well organized. The motivation, method, and experimental settings are easy to follow. The figures illustrate the proposed framework effectively, and the appendix provides sufficient implementation details and additional analyses. However, a citation rendering error appears on page 6, Section 5.1 (Benchmarks and Metrics): cite-caba2015activitynet is displayed directly after "ActivityNet-Captions" instead of being rendered as a properly formatted citation.

Critical strengths

S1. Strong empirical results. TIDE achieves the best mIoU among the compared compression methods across the main VTG settings. The improvements are particularly noticeable on ActivityNet-Captions at 12.5% token retention, where TIDE outperforms FlashVID by 5.36, 7.28, and 5.04 mIoU points across three Qwen models. The method also maintains competitive performance on Charades-STA and shows promising results on video question answering. These findings suggest that TIDE can preserve useful temporal information even under substantial token reduction.

S2. Reasonable and coherent method design. The use of a shared temporal state representation to guide token allocation, selection, and fusion is well motivated for VTG. In particular, BPF addresses the possibility that visually similar tokens from different temporal states may contain distinct information needed for event localization. The ablation studies support the contribution of the main components. For example, removing TSF reduces mIoU from 42.13 to 40.61, while removing context fusion reduces it to 40.19. These results provide reasonable evidence for the overall design.

Critical weaknesses

W1. Limited direct validation of the boundary-preservation mechanism. TSF identifies temporal states through cross-frame correspondences and feature changes, but these changes do not necessarily correspond to query-relevant event boundaries. Camera motion, background changes, or unrelated actions may also introduce significant temporal variations. Appendix Figure 8 provides a positive example in which TSF improves the ranking of an annotated boundary frame from 23/180 to 7/180. However, dataset-level quantitative evidence is still lacking, making it difficult to determine how consistently the proposed representation captures meaningful event transitions. Furthermore, the independent effect of the state-compatibility penalty in BPF is not directly isolated. Although Table 4 includes related fusion comparisons, they do not specifically measure how much of the performance improvement comes from this penalty rather than context recovery in general. W2. Unclear provenance and comparability of some experimental results. Several latency measurements in TIDE's Table 6 are identical to those reported in SemVID's Table 6, including the full-token prefilling latency and the compression and prefilling latencies of VisionZip, FastVID, and SemVID. However, TIDE describes these results as measurements obtained in its experimental environment. Although hardware details are provided, the provenance of these identical measurements remains unclear. Additionally, the Qwen3-VL results for SemVID are taken from the original paper rather than evaluated under the same pipeline. The authors explicitly acknowledge this distinction, but the differing full-token baselines make direct comparisons less straightforward. SemVID also shows a substantial performance drop on Qwen3.5-4B in the authors' reproduction. It would be helpful to clarify whether this reflects a limitation of SemVID's cross-backbone generalization or differences in its model-specific implementation.

Questions for authors

Q1. Validation of temporal state transitions. Appendix Figure 8 shows that TSF improves the ranking of an annotated event-boundary frame from 23/180 to 7/180. Could the authors provide dataset-level statistics comparing TSF with simple inter-frame differences in terms of alignment with ground-truth event boundaries, such as boundary-neighborhood recall or boundary ranking? Evaluating both the original transition signals and their query-conditioned counterparts would further help establish the proposed mechanism.

Q2. Independent contribution of state-compatible fusion. Although Table 4 includes a frame-boundary fusion variant, it does not directly isolate the state-compatibility penalty in BPF. What is the final VTG performance when the penalty coefficient in Equation (9) is set to zero while the selected tokens and other fusion operations remain unchanged? This would help distinguish state-aware boundary preservation from general context recovery.

Q3. Provenance and fairness of baseline comparisons. Several latency values in TIDE's Table 6 exactly match those reported by SemVID, while the full-token mIoU differs between the two studies (40.33 vs. 42.93 on Qwen3-VL-4B). Which latency measurements were independently obtained, and which, if any, were adopted from prior work? Were they measured under comparable input configurations and timing conditions? Additionally, could the authors clarify how evaluation differences affect the SemVID comparison and whether its substantial performance drop on Qwen3.5-4B reflects limited cross-backbone generalization or model-specific implementation differences?

8 comments
AuthorsOpenReview ↗

Summary: GT-boundary validation, isolated state-penalty controls, and fair comparisons

We thank the reviewer for the constructive suggestions and for recognizing the empirical results and coherent design. Our main responses are:

  1. TSF supplies feature states for compression; the LLM predicts query-specific intervals (W1/Q1). We directly evaluate this intermediate temporal reference against annotated event boundaries on both datasets. At ±2 s, TSF improves top-10% boundary recall over pooled frame differences by 1.64/1.04 points on ANet/Charades-STA. We also report query-weighted signals and fixed-feature correspondence controls, providing dataset-level evidence beyond the original visualization. See Response 1.

  2. The new fixed-anchor control isolates state compatibility from context recovery (W1/Q2). Table 3 attributes the fusion gain to recovering peripheral context after selection. We additionally set α=0 with the same anchors and other fusion rules to isolate the penalty itself. The penalty reduces the cross-GT-boundary weight fraction for positively gated sources by approximately 18%–21%. Its accuracy effect is setting-dependent: +0.05 mIoU on ANet and +0.39 on Charades-STA, so context recovery, not the penalty, accounts for most of the ANet fusion gain. The routing and accuracy controls jointly test its intended role. See Response 2.

  3. All six methods have now been remeasured under a unified timing protocol (W2/Q3). The original baseline timings were quoted from SemVID on L40, while TIDE was measured by us on A800. The new test uses the same hardware, software, and inputs for every method and yields 4.44× for compression plus prefill, computed from the reported stage medians. We will replace Table 6 and update the associated statements. See Response 3.

  4. Shared-baseline reruns confirm TIDE's accuracy advantage over SemVID (W2/Q3). The original manuscript explicitly identifies quoted SemVID accuracy entries and their normalization references. We have additionally run Full-token, SemVID, and TIDE in the same pipeline for each setting. TIDE leads in all five comparisons, by approximately 1.2–3.5 mIoU, directly addressing the evaluation-pipeline concern. See Response 3.

  5. The Qwen3.5-4B SemVID results are identified as our adaptation (W2/Q3). We compare that implementation with Full-token and TIDE under identical inference settings. The pronounced loss occurs for the SemVID adaptation on ANet, while TIDE retains approximately 93% of Full-token mIoU on both benchmarks. These results establish the implementation-level comparison; the cause of the adaptation's loss remains to be isolated. We will document the adaptation details and retain this attribution scope. See Response 3.

We will also correct the citation command in Section 5.1. We thank the reviewer for the careful reading and constructive suggestions, which have helped us improve the paper. The added results, source descriptions, and corresponding revisions will be marked in blue in the updated PDF. If any point remains unclear, we would be glad to discuss it further.

AuthorsOpenReview ↗

Response 1 to W1 + Q1: Dataset-level alignment with annotated event boundaries

We evaluate each query's own annotated start/end boundaries under the same candidate budget for every signal. TSF improves boundary recall over pooled frame differences on ANet and gives comparable or higher recall on Charades-STA. At a fixed candidate budget, the target boundaries compete with other changes in the video for the same ranked positions. This directly tests whether the transition signal prioritizes the query's annotated boundary neighborhoods.

Dataset-level evaluation. Using Qwen3-VL-4B, we evaluate ANet-val and Charades-STA, holding visual inputs, GT, and evaluation rules fixed. For each signal, we rank visual time steps and select the top 10% as candidates. Recall measures coverage of that query's GT boundary neighborhoods, using processor-provided timestamps and the stated tolerance.

Dataset / toleranceRandomPooled frame differenceTSF transitionQuery-weighted TSF
ANet, ±2 s35.8343.9345.5745.56
ANet, ±4 s58.1259.0060.9661.25
Charades-STA, ±1 s20.9124.9324.9624.85
Charades-STA, ±2 s37.8138.9039.9439.83

Values are recall (%). At ±2 s, TSF exceeds pooled frame differences by 1.64 points on ANet and 1.04 on Charades-STA. At the tighter ±1 s tolerance on Charades-STA, the scores are close (24.96 vs. 24.93). These are equal-budget comparisons against external event annotations.

Isolating correspondence. We hold contextualized features and the candidate budget fixed, comparing differences at fixed spatial positions with differences between matched predecessors:

Dataset / toleranceRecall: fixed position → matchedMedian normalized boundary rank: fixed position → matched
ANet, ±2 s44.32 → 45.570.138 → 0.132
Charades-STA, ±1 s24.27 → 24.960.300 → 0.303

Lower rank is better. Correspondence raises recall on both datasets with the same feature representation; the median ranking improvement occurs on ANet. This control identifies the effect of comparing matched content within the transition construction.

Query conditioning. Query-weighted and original TSF recalls differ by at most 0.29 points in the table. We also evaluate conditioning in the allocation and fusion stages where it is used: jointly removing it from EDA and BPF gives 41.36/53.47 mIoU on ANet/Charades-STA, versus 42.13/53.69 for full TIDE. This measures its joint contribution to the actual grounding task.

Connecting boundary evidence to compression. At 12.5% retention, for GT boundaries inside a temporal unit, adjacent ANet frames receive 7%–12% more tokens on average than uniform allocation within that unit. The original ANet ablation and added Charades-STA control show mIoU decreases of 1.52/0.60 points when TSF and its dependent signals are removed. This removal follows the paper's full fallback definition and measures the shared temporal reference's contribution to the pipeline.

Together, the query-specific GT diagnostic, fixed-feature correspondence control, allocation behavior, and localization ablations connect boundary alignment, retained capacity, and final grounding performance. We will include these results and their evaluation definitions in the revision.

AuthorsOpenReview ↗

Response 2 to W1 + Q2: Isolating the state penalty with fixed anchors

On Qwen3-VL-4B, the state penalty reduces cross-boundary fusion with the same selected anchors; its accuracy gain is small on ANet (+0.05 mIoU) and measurable on Charades-STA (+0.39). Table 3 measures the benefit of recovering peripheral context after selection. The added α=0 control specifically isolates the state penalty within that fusion stage.

We freeze the outputs of TSF, EDA, and ESS: frame budgets, anchor indices, pre-fusion features, and positions. Setting α=0 changes only the state-penalty coefficient. Similarity, temporal window, Gaussian prior, trajectory reward, admission rule, top-k routing, and weighted accumulation remain active. The w/o Context Fusion control returns the selected contextualized anchors directly. Within each dataset, all variants share the checkpoint, queries, preprocessing, and decoding, at 12.5% retention.

Datasetw/o fusion mIoUα=0 mIoUFull TIDE mIoU[email protected]: α=0 → TIDE
ANet40.1942.0842.1325.23 → 25.83
Charades-STA53.3953.3053.6935.43 → 36.88

On ANet, context recovery without the penalty adds 1.89 mIoU, and the penalty adds another 0.05. On Charades-STA, the penalty raises mIoU from 53.30 to 53.69 and [email protected] by 1.45 points; full fusion also exceeds the selected-anchor score of 53.39. Thus, the added control measures the state constraint's contribution separately from the context-recovery gain reported in Table 3: context recovery dominates on ANet, while the penalty's accuracy contribution is concentrated on Charades-STA and in the routing behavior below.

The fixed-anchor control holds the fusion rules constant while allowing source admission, destinations, and weights to respond to α. It measures the penalty's marginal effect in the implemented fusion process.

Routing near external GT boundaries. We use annotated boundaries to define two sides and measure the fraction of fusion weight routed from nearby non-anchor sources to anchors on the other side. Ratios are computed per query and then averaged. Sources are grouped by positive gate (e>0) or zero gate (e=0); the gate and anchors are identical between the two α settings.

Dataset / neighborhoodα=0 cross-boundary weight (%)Full TIDE (%)Relative reduction
ANet, ±1 step18.4814.6920.5%
ANet, ±2 steps12.479.8820.8%
Charades-STA, ±1 step19.7516.1418.3%
Charades-STA, ±2 steps12.7410.3918.4%

The table covers positively gated sources; steps are visual sampling steps. The ±1-step groups contain 4,833/3,719 valid queries. Admission rates change from 96.83% to 95.25% and 99.66% to 99.49%; mean output weights change from 0.587 to 0.577 and 0.579 to 0.578. Zero-gate cross-boundary ratios remain identical between α settings: 21.20%/21.47%.

The evidence is consistent with both functions of BPF: recovering discarded context and constraining its transport near event boundaries. The accuracy decomposition also shows that context recovery, not the state penalty, accounts for most of the ANet fusion gain; we will state this explicitly rather than attributing the whole gain to boundary awareness. Fixed-anchor accuracy controls isolate the penalty's contribution, and external-GT routing statistics verify its boundary-sensitive behavior. We will report these results alongside the existing fusion ablations.

Reviewer_ugQgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TIDE, a training-free visual token compression framework for video temporal grounding (VTG). Its central motivation is that appearance-based token pruning or merging may remove state transitions or mix information across event boundaries, impairing temporal localization. TIDE constructs a Temporal State Field through cross-frame token correspondences and uses this shared representation for query-conditioned frame budgeting, temporally discounted anchor selection, and state-aware context fusion.

Critical strengths

1. Relevant and well-motivated problem.

The paper identifies a meaningful distinction between preserving information for recognizing an event and preserving information for localizing its temporal boundaries. Addressing how discarded tokens are fused into retained anchors, in addition to which tokens are selected, is a useful perspective on compression for VTG.


2. Coherent integration of compression stages.

TIDE uses a shared temporal representation to coordinate allocation, selection, and fusion. This provides a coherent design, and the soft routing mechanism offers a practical way to recover context when few compatible anchors survive compression. The method requires no parameter updates.

Critical weaknesses

1. Insufficient justification of the query-relevance signal

TIDE estimates query relevance using the maximum cosine similarity between projected visual tokens and non-contextualized LLM input-token embeddings. However, compatibility with the same LLM input space does not by itself establish that cosine similarity between these representations is a reliable measure of semantic relevance. The paper does not directly validate this assumption for the evaluated backbones. Moreover, even if lexical alignment is adequate, the score is invariant to permutations of the same token IDs and does not incorporate surrounding linguistic context. It therefore cannot distinguish compositional differences expressed solely through token order. Since this signal guides both budget allocation and boundary-sensitive fusion, its quality and independent contribution require direct evaluation.


2. Incomplete assessment of broader video understanding

VideoQA is evaluated only on LLaVA-OneVision, whereas the VTG experiments use Qwen-family backbones. Consequently, the current results do not establish how VTG-oriented compression affects other video understanding capabilities of the same models that exhibit the reported grounding gains. VideoQA should also be evaluated on a backbone used for VTG, at matched retention ratios. Broader task coverage through benchmarks such as MVBench and MLVU, together with per-task results, would help determine whether prioritizing transition evidence preserves general video understanding or introduces capability-specific trade-offs.


3. Limited cross-family validation of the VTG mechanism

The VTG results cover multiple model sizes and architectures within the Qwen family, which provides useful evidence of consistency. However, the method depends on backbone-specific representation properties, including visual–text embedding similarity, cross-frame feature correspondence, and positional handling. Its effectiveness on an independent model family therefore remains uncertain. Furthermore, the component-removal ablations are confined to Qwen3-VL-4B on ActivityNet-Captions at 12.5% retention. The Charades-STA sensitivity experiments provide complementary evidence, but do not establish whether the same components explain the gains across backbones and datasets.

Questions for authors
  1. What evidence supports the semantic comparability of projected visual tokens and input text embeddings under cosine similarity? Can the authors validate the relevance scores and compare against a query-independent control and mismatched-query inputs? A controlled evaluation on relational or temporal query pairs would additionally clarify the consequences of using non-contextualized token representations.

  1. Can the authors evaluate TIDE on at least one independent VTG-capable video LLM, such as a suitable InternVL, InternVideo, or LLaVA variant, under matched token budgets? Repeating the key query-conditioning and state-penalty controls would help establish whether the proposed mechanism generalizes beyond Qwen-family representations. Additionally, can the authors report VideoQA results on a backbone used in the VTG experiments, at both 25% and 12.5% retention, and broaden the evaluation to benchmarks such as MVBench and MLVU? Per-task results would clarify whether the VTG-oriented compression preserves broader video understanding capabilities or introduces task-specific trade-offs.
14 comments
AuthorsOpenReview ↗

Summary: Query validation, VideoQA on the VTG backbone, and independent-family evaluation

We thank the reviewer for the constructive suggestions and for recognizing the motivation, shared temporal representation, and training-free design. Our main responses are:

  1. The query score provides an auxiliary lexical cue (W1/Q1). We use it to weight allocation and fusion decisions. To validate that role, we directly compare correct and mismatched queries on the same video, then repeat localization with query conditioning removed or replaced. Correct queries improve target-frame ranking, and query conditioning adds 0.77/0.22 mIoU on ANet/Charades-STA. We will describe the score according to this modest, empirically measured role. See Response 1.

  2. The same compressed video can support opposite temporal-order questions (W1/Q1). The scorer is word-order-invariant. The LLM interprets the complete query, including its word order; TIDE's compression retains the query-relevant visual tokens and their temporal information. We test this division of roles directly: opposite-order questions receive identical compressed visual tokens, and only the downstream question order changes. At 12.5% retention, TIDE achieves 68.75% accuracy versus 70.00% for Full-token, well above the text-only control. This demonstrates preservation of usable temporal evidence in the controlled order test. See Response 1.

  3. The VTG backbone retains substantial VideoQA ability under compression (W2/Q2). To directly assess broader abilities on the model used for grounding, we evaluate Full-token, TIDE at both retention ratios, and FlashVID at 12.5% on the same Qwen3-VL-4B checkpoint across LongVideoBench, Video-MME, MVBench, and MLVU. Across the two budgets, TIDE retains approximately 86%–96% of the Full-token benchmark scores. At matched nominal retention, TIDE scores higher than FlashVID on LongVideoBench and MLVU, with close scores on Video-MME and MVBench. The task breakdown also shows well-preserved localization and sequence performance on MVBench. See Response 2 for the complete per-task analysis.

  4. The pipeline advantage extends to an independent family, with setting-dependent component effects (W3/Q2). On InternVL2.5-8B, whose visual and language backbones differ from Qwen, TIDE exceeds FlashVID on both datasets at both retention ratios, by 0.80–1.77 mIoU. Repeating component removals and the query/state-penalty controls identifies context recovery as a consistent contributor and shows that individual temporal and query contributions vary by setting. The absolute Full-token results are reported alongside these comparisons to make the tested grounding capability clear. See Response 3.

The added experiments and corresponding revisions will be marked in blue in the updated PDF. If any point remains unclear, we would be glad to discuss it further.

AuthorsOpenReview ↗

Response 1 to W1 + Q1: Direct validation, query controls, and temporal-order reasoning

The correct-query control yields observed gains of 0.77/0.22 mIoU on ANet/Charades-STA. Direct diagnostics show measurable query-specific information, while the absolute frame-level AUROC supports describing the score as a low-cost lexical relevance cue. The cue requires no additional text encoder or parameter updates.

1. Direct score validation. On 200 ANet videos (100 short, ≤60 s; 100 long), we select two strictly non-overlapping events per video with similar description lengths. Holding the video and target event fixed, we score with the correct query, the other event's query, or the template alone. Sampling rules and seeds are fixed in advance. Each frame score averages its top 10% spatial-token scores; GT-inside/outside frames form positive/negative examples. Metrics are averaged over 399 valid targets; one of 400 targets has no positive sampled frame.

Scoring textAUROC (%)AP (%)
Correct query, full prompt52.6435.82
Same-video mismatched query, full prompt47.3732.43
Template only49.0633.18

Correct-minus-mismatched AUROC/AP differences are +5.27/+3.39 points. Correct-minus-template differences are +3.58/+2.64 points. Spatial-mean pooling and comparison restricted to the two event intervals give correct/mismatched AUROCs of 54.04/47.73 and 54.15/45.85. These diagnostics establish query sensitivity; the absolute AUROC indicates limited frame-level discrimination. We will therefore narrow the main-text description to a low-cost lexical relevance cue that provides an auxiliary signal for token allocation and context fusion.

2. Localization controls. With Qwen3-VL-4B at 12.5% retention, the downstream LLM always receives the correct ordered query. Query-independent EDA uses unweighted mean transition strength, and BPF uses a transition-only gate; the remaining rules are retained.

DatasetFlashVID mIoUQuery-independent TIDEFull TIDE
ANet36.7741.3642.13
Charades-STA52.8253.4753.69

Query conditioning adds 0.77/0.22 mIoU and 0.55/1.23 [email protected]. Query-independent TIDE still exceeds FlashVID by 4.59/0.65 mIoU: a substantial pipeline advantage persists after removing query conditioning.

Additional controls on all 3,720 Charades-STA queries yield:

Compressor settingmIoU
Query in EDA only53.44
Query in BPF only53.47
Mismatched query, seeds 17/29/4353.22/53.23/53.46
Cross-video mismatch, seed 1753.21
Template only53.26
Event-description tokens only53.25

The first mismatch control prioritizes another query from the same video, with cross-video fallback. Full TIDE scores highest among these controls. The single-path scores remain close to the query-independent score; the complete configuration has the highest reported mIoU.

3. Identical visual evidence, opposite word order. The maximum-cosine score is invariant to permutations of the same token IDs. We therefore test the retained evidence using 100 action pairs, 200 real videos, and 400 questions per method. Each action pair has two videos with opposite action orders; each video receives two opposite Yes/No order questions with the same token-ID multiset. Official action timestamps define GT. We verify identical compressed features, anchor indices, and budgets for the two questions.

MethodAccuracy (%)Both questions correct (%)All four per action pair correct (%)
Full-token70.0051.0027.00
TIDE, 25%71.0054.0031.00
TIDE, 12.5%68.7552.5031.00
Text-only50.002.500.00

Relative to Full-token, TIDE accuracy differences are +1.00 and −1.25 points at 25% and 12.5% retention, respectively. The shared visual evidence supports the LLM in answering both orderings while reading the complete ordered query.

AuthorsOpenReview ↗

Response 2 to W2 + Q2: VideoQA on the VTG backbone, with FlashVID and task-level comparisons

The same Qwen3-VL-4B checkpoint retains approximately 89.8%–96.2% of Full-token benchmark scores at 25% retention and 86.1%–92.7% at 12.5%. At 12.5%, TIDE exceeds FlashVID on LongVideoBench and MLVU, with scores within 0.08 points on Video-MME and MVBench.

We evaluate four benchmarks with 10,211 questions. Full-token, TIDE at two retention ratios, and FlashVID at 12.5% give 40,844 question evaluations. Within each benchmark, methods share video sampling, prompts, choices, decoding, and scoring.

BenchmarkQuestionsFull-tokenTIDE 25%TIDE 12.5%FlashVID 12.5%
LongVideoBench1,33758.5656.3254.3052.06
Video-MME2,70061.4157.3754.4854.56
MVBench4,00064.8062.0358.8358.90
MLVU, multiple-choice tasks2,17464.0257.5255.1353.13

Scores are percentages. The first three benchmarks use question-averaged accuracy; MLVU uses the official equal-weight average over seven tasks (M-avg). At 12.5%, TIDE's reported scores exceed FlashVID by 2.24/2.00 points on LongVideoBench/MLVU and are lower by 0.08/0.07 points on Video-MME/MVBench.

MVBench task-level analysis. We examine representative categories from MVBench's 20 tasks, focusing on localization, temporal structure, and object interactions on the same backbone used for VTG:

MVBench taskFull-tokenTIDE 25%TIDE 12.5%FlashVID 12.5%
Action Localization53.0059.5061.5056.50
Action Sequence73.5073.5072.0076.00
Scene Transition92.5093.0092.5091.50
Object Interaction79.5078.0079.0078.00
State Change64.5058.5054.0058.00
Moving Direction54.0045.5037.5036.50
Counterfactual Inference72.5067.0055.0057.00
  • Action localization improves under compression. TIDE raises accuracy over Full-token by 6.50/8.50 points at 25%/12.5% retention. At 12.5%, it also exceeds FlashVID by 5.00 points, showing strong localization performance.
  • Temporal order and scene transitions are well preserved. Action Sequence matches Full-token at 25% retention and remains within 1.50 points at 12.5%. Scene Transition matches or exceeds Full-token at both budgets. These results show that the retained representation supports action-order and scene-transition judgments under substantial compression.
  • Object interaction remains close to Full-token. At 12.5% retention, TIDE has a 0.50-point gap to Full-token and a 1.00-point lead over FlashVID. This extends the observed capability retention to interaction understanding on the same VTG backbone.

Together with Table 5, these results show that TIDE's grounding benefits coexist with strong general VideoQA performance under compression. Table 5 reports higher overall scores than Full-token on both VideoQA benchmarks with LLaVA-OneVision-7B at 25% retention, and the added Qwen3-VL-4B experiments show broad capability retention with accuracy competitive with FlashVID at the same nominal retention. We will include the complete analysis in the revised appendix.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and for the thoughtful feedback you have provided throughout the review process. We have revised the manuscript in response to your comments and uploaded the updated PDF. All revisions are marked in blue, including the headings and captions of newly added sections and tables, so the changes are easy to locate.

We recognize that reviewing is demanding, especially alongside other commitments. If your schedule allows, we would be grateful if you could look over our responses and the revised manuscript and share any further thoughts before the discussion period closes.

If any concern remains unresolved, or if we have misunderstood or not fully addressed any point, please let us know. We would welcome the opportunity to clarify further and to improve the manuscript.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.