acceptodds
Withdrawn from ICLR 2027

FATE: Frame-and-Token Evidence Preservation for Efficient MLLM-based Video Temporal Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) have shown strong capabilities for video temporal grounding (VTG), but processing video inputs incurs substantial computation and memory costs, especially as video duration increases. Existing efficiency methods typically reduce visual computation through frame selection before visual encoding or token compression after encoding, with most approaches focusing on individual reduction stages. However, these stages are closely connected in VTG: frame selection determines which temporal evidence is available for encoding, while token compression determines which fine-grained visual information remains available for localization. We introduce FATE, a training-free framework that coordinates visual reduction across these stages through shared temporal evidence. FATE determines where to look through coverage-preserving frame selection and how much to preserve through relevance-adaptive token allocation. After visual encoding, query-motion-guided token preservation further determines what to preserve within each temporal patch through informative seed selection and coverage-aware completion. Together, these components provide a consistent evidence-guided strategy for preserving grounding-relevant information across the visual processing pipeline without additional training of the grounding MLLM. Experiments on three VTG benchmarks under varying frame and visual-token budgets demonstrate consistent improvements on Ego4D-NLQ and ActivityNet Captions while maintaining competitive performance on Charades-STA.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Temporal Grounding, Visual Token Compression, Query-Aware Frame Selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kjRnOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Considering existing methods focus on either selecting informative frames before visual encoding or compressing visual tokens after encoding, this paper proposes a training-free framework FATE. FATE integrates these two stages to improve model's temporal grounding performance.

Critical strengths

FATE offers a solution to the limitation of handling frame selection and token compression as separate processes. It proposes a training-free framework FATE, making it practical to any MLLMs.

Critical weaknesses
  1. The abstract and introduction indicate shared temporal evidence as an important contribution, and Section 3.3 uses temporal evidence to guide both frame selection and token allocation. However, frame selection and token allocation based on visual-textual similarity are not novel. For example, "Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs" uses similarity to guide frame selection. "A TRAINING-FREE FRAMEWORK FOR LONG VIDEO UNDERSTANDING VIA VIDEO-QUERY-OPTIONS SIMILARITY" adjusts FPS and spatial resolution based on similarity. The experiments did not include comparisons with these methods, so it remains unclear whether "shared temporal evidence" is truly important.

  2. The experimental results do not support the necessity of the proposed method. In Table 3, the proposed frame-selection strategy improves mIoU by only 0.2 over simple Top-K selection. In Table 4, the full method exceeds Coverage Only by 0.1, while "Query + Coverage" and "Motion + Coverage" actually underperform Coverage Only, without sufficient analysis.

Questions for authors
  1. Why use frame-wise semantic similarity? In my view, for instance, "taking out a bottle" versus "putting in a bottle" are often difficult to distinguish based on a single frame alone. Have the authors considered a segment-wise semantic similarity?

  2. FATE integrates both "pre-encoding frame selection" and "post-encoding token compression." Have the authors evaluated the combination of these two existing categories of methods? For instance, could the "uniform" and "Top-k" approaches in Table 3 be replaced with other existing frame selection methods for comparison?

  3. The comparative method FlashVID is not specifically for TVG. Beyond SemVID, have the authors considered other compression methods designed for TVG tasks? Furthermore, have they evaluated FATE on general video understanding benchmarks, such as LongVideoBench?

  4. There is no analysis for Table 4.

3 comments

We sincerely thank the reviewer for the time and effort devoted to carefully evaluating our work. We particularly appreciate the thoughtful and critical feedback, which has encouraged us to reconsider how we present our contributions and to examine the effectiveness of our design choices more thoroughly. This review process has been a valuable learning experience for us, helping us better understand both the strengths and limitations of our work. We are genuinely grateful for these insights and the opportunity to improve our research, and we address the reviewer's concerns below.

Novelty and Effectiveness of Shared Temporal Evidence (W1)

We thank the reviewer for highlighting these relevant works. We agree that similarity-guided frame selection and adaptive visual resource allocation have been explored in Q-Frame and A Training-Free Framework for Long Video Understanding via Video-Query-Options Similarity (VQOS). However, FATE differs from these approaches in both its VTG-oriented temporal evidence construction and cross-stage visual reduction strategy, as summarized below.

AspectQ-FrameVQOSFATE
Target TaskGeneral video understandingLong-video understandingVideo temporal grounding
Relevance EvidenceQuery-frame similarityVideo-query-options similarityEvent-wise, relation-aware temporal evidence
Frame SelectionQuery-aware samplingAdaptive frame samplingCoverage-preserving selection
Visual Budget AllocationMulti-resolution adaptationDynamic resolution allocationPost-encoder token allocation
Fine-grained Token PreservationNot explicitly addressedNot explicitly addressedQuery-motion seeds + coverage completion

While Q-Frame and VQOS coordinate frame sampling and resolution adaptation before visual encoding, FATE coordinates pre-encoder frame selection with post-encoder token allocation and preservation. Moreover, FATE incorporates explicit temporal relationships between decomposed events, rather than relying solely on visual-textual similarity. These designs specifically address VTG, where preserving event progression and temporal context is important for accurate localization.

To further validate these contributions, we conducted additional controlled experiments on the full Ego4D-NLQ validation set (4,552 queries):

  • Relation-aware evidence: Temporal relation composition improves mIoU from 10.518% to 11.098% on 1,641 queries containing valid temporal relations (+0.580 pp, 95% CI: [+0.058, +1.085]), with all other components fixed.
  • Evidence-guided token allocation: With frame selection and coverage-only token selection fixed, FATE's adaptive allocation improves mIoU from 9.233% to 9.728% compared with SemVID's allocation (+0.495 pp, 95% CI: [+0.174, +0.794]).

These results provide additional evidence for the effectiveness of relation-aware temporal evidence and post-encoder adaptive token allocation.

Effectiveness of Frame Selection and Query-Motion-Guided Preservation (W2/Q4)

We appreciate the reviewer's concerns regarding the modest improvements in Tables 3 and 4. We conducted additional controlled experiments to further examine these two components.

1. Coverage-preserving frame selection. We further compared Top-K and FATE frame selection on all 4,552 Ego4D-NLQ validation queries under a 512-frame budget, keeping the subsequent token-processing pipeline identical.

Frame SelectionmIoU (%)
Top-K9.462
FATE9.789

Both settings use 512 frames and 12.5% token retention.

FATE improves mIoU by +0.327 pp, complementing the +0.2 pp improvement reported in Table 3 under 2,048 frames. These results show a consistent numerical advantage over Top-K across the two frame budgets, although the improvements remain modest.

2. Query-motion-guided token preservation. We further examined the contribution of query-motion guidance on the complete Ego4D-NLQ validation set using 2,048 frames and 12.5% token retention.

Token Preservation StrategymIoU (%)
Coverage Only10.090
Query + Coverage9.849
Motion + Coverage9.898
Query-Motion Seeds Only9.800
Query-Motion + Coverage (ρ = 0.25)10.200
Full FATE (ρ = 0.5)10.218

These results confirm that coverage-aware preservation is the primary contributor to fine-grained token selection. Introducing query relevance or temporal variation individually does not improve performance, potentially because either signal alone can prioritize less useful tokens at the expense of coverage. However, combining both signals with coverage achieves a modest improvement of +0.128 pp over Coverage Only.

We therefore view query-motion-guided selection as a complementary refinement rather than the principal source of FATE's performance gains. We will clarify these findings and provide a more detailed interpretation of Table 4 in the revised paper.


Frame-wise vs. Segment-wise Semantic Similarity (Q1)

We appreciate the reviewer's insightful example. We agree that frame-wise semantic similarity alone may not reliably distinguish temporally reversed actions, such as "taking out a bottle" and "putting in a bottle."

In FATE, frame-wise CLIP similarity serves as a lightweight relevance signal for guiding frame selection and token allocation, rather than directly determining event boundaries or recognizing fine-grained actions. The selected frames retain their temporal ordering, allowing the downstream MLLM to reason over the preserved visual evidence for final grounding. Moreover, frame-wise CLIP embeddings are query-independent and can be precomputed and reused across queries, avoiding repeated video-side encoding.

We agree that segment-wise similarity, particularly when computed using a temporally aware video encoder, could better capture action dynamics and directionality. However, this would introduce additional temporal modeling and potentially higher preprocessing costs. Exploring such evidence representations is a valuable extension, while our current framework focuses on coordinating visual reduction using lightweight, reusable temporal evidence.

Combination of Existing Frame Selection and Token Compression Methods (Q2)

We appreciate the reviewer's suggestion. Our evaluation prioritizes controlled comparisons with VTG-specific compression methods, particularly SemVID, while keeping the grounding backbone, frame budget, and token-retention ratio consistent.

Existing methods such as Q-Frame and VQOS jointly optimize frame sampling and input resolution. Combining their complete pipelines with post-encoder token compression would require adapting their resource-allocation strategies to a common VTG evaluation protocol.

To investigate the contributions of the two stages, we additionally conducted controlled experiments on all 4,552 Ego4D-NLQ validation queries, using 512 frames and 12.5% token retention.

Frame SelectionToken ProcessingmIoU (%)
Top-KFATE9.462
FATESemVID9.171
FATEFATE9.789

With frame selection fixed, FATE's token processing improves mIoU by +0.618 pp over SemVID's processing (95% CI: [+0.187, +1.034]). With token processing fixed, FATE's frame selection improves mIoU by +0.327 pp over Top-K.

These experiments isolate the contributions of the two stages, although they do not exhaustively evaluate combinations of existing frame selectors and token compressors. We will clarify this distinction in the revised paper.


VTG-Specific Baselines and General Video Understanding (Q3)

Baseline selection. We select SemVID as our primary baseline because it is a state-of-the-art training-free token compression method specifically designed for VTG, enabling controlled comparisons under the same grounding backbone and visual-token budget. Other VTG methods, such as GroundVTS and VTG-LLM, incorporate trainable modules or model-specific adaptations, making them less directly comparable under our training-free evaluation protocol. We additionally include FlashVID to examine how a general-purpose video compression method performs on VTG.

Evaluation scope. Our three VTG benchmarks cover long-, medium-, and short-form videos, allowing us to evaluate FATE across different temporal scales. We focus on VTG because accurate temporal localization requires preserving event progression and temporal context, rather than only identifying salient visual content. We agree that evaluation on LongVideoBench and other general video understanding benchmarks would help assess broader transferability. However, general VideoQA performance is not a claim established by our current experiments.

Reviewer_uJobOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FATE, a training-free framework for efficient video temporal grounding with MLLMs that coordinates pre-encoder frame selection and post-encoder token compression using a shared, relation-aware temporal evidence signal derived from query decomposition and video-text matching. The shared evidence guides "where to look" and "how much to preserve", while a query-motion-guided determines "what to preserve". Experiments on Ego4D-NLQ, ActivityNet, and Charades-STA with varying frame and token budgets show consistent gains over SemVID on long and medium videos and competitive performance on short videos, with favorable accuracy-efficiency trade-offs.

Critical strengths
  1. Introduces a well-motivated problem formulation and a novel cross-stage perspective that explicitly coordinates frame selection and token preservation through shared temporal evidence, rather than treating them as independent optimization stages.

  2. FATE provides a training-free approach that improves visual processing efficiency without requiring additional training or fine-tuning of the grounding MLLM, enhancing its practical applicability to existing models.

  3. FATE demonstrates consistent improvements on Ego4D-NLQ and ActivityNet Captions across multiple frame and token budgets.

Critical weaknesses
  1. Limited Analysis of the Efficiency and Generalizability of Relation-Aware Temporal Evidence.

While the proposed relation-aware temporal evidence is well-motivated, several aspects require further investigation. First, the framework introduces an additional VLM (CLIP ViT-L/14) for temporal evidence construction, raising questions about its computational overhead and whether the resulting efficiency gains remain significant when considering end-to-end inference costs. Furthermore, it would be valuable to analyze how the choice and capability of the VLM affect grounding performance. Second, the proposed relation modeling primarily focuses on explicit temporal relationships, such as before, after, and overlap. However, it remains unclear how the framework handles more complex event dependencies that cannot be characterized solely by temporal ordering, as suggested by the event structure in Fig. 2. Additionally, the temporal decay parameter tau imposes a fixed temporal proximity prior, which may not generalize well across events with different temporal scales or videos of varying durations. A sensitivity analysis of tau across different video lengths and temporal relationships would help clarify the robustness of the proposed formulation. Further analysis of these aspects would strengthen the practical efficiency and generalizability of the proposed approach.

  1. Limited Sensitivity and Analysis of Evidence-Guided Temporal Allocation.

While the proposed evidence-guided temporal allocation demonstrates improvements over uniform and Top-K baselines, several design choices require further investigation. First, the temporal block duration (15s) and anchor search radius (one anchor interval) are fixed without sensitivity analyses, although both directly influence the balance between temporal coverage and relevance-guided frame selection. Evaluating different configurations across varying video lengths would help establish the robustness of the proposed strategy. Second, the patch-level relevance in Eq. (4) is computed by averaging temporal evidence over selected frames, which may dilute sharp relevance peaks associated with short-duration events. Consequently, patches containing brief but important grounding evidence may receive fewer visual tokens than those exhibiting consistently moderate relevance. Although the paper compares relevance-adaptive and uniform token allocation, it lacks comparisons with alternative aggregation strategies and duration-stratified evaluations. Further experiments, particularly on short-moment localization, would strengthen the justification for these design choices.

  1. Limited Experimental Coverage and Generalizability.

Although FATE demonstrates promising results on VTG benchmarks, the experimental evaluation remains limited in several aspects. First, the comparisons primarily focus on SemVID and FlashVID, leaving out other relevant video efficiency methods that could provide a more comprehensive assessment of the proposed framework. Second, while improving VTG performance is important, it is equally necessary to ensure that the proposed visual reduction strategy does not compromise the general video understanding capabilities of MLLMs. However, no evaluations on other video understanding tasks, such as video question answering or video reasoning, are provided. Finally, although the authors include an additional experiment with Qwen3-VL-4B-Instruct in the Appendix, the main evaluation relies on TimeLens2-4B, and the backbone generalization analysis is restricted to a single dataset and token-retention setting. More extensive experiments across diverse MLLM architectures, model scales, and video understanding tasks would strengthen the generalizability and practical applicability of FATE.

Questions for authors
  1. Effectiveness and Validity of Query-Motion-Guided Token Selection.

In Eq. (5), motion relevance is estimated by comparing visual tokens at the same spatial position across adjacent temporal patches. However, this measures local feature variation rather than explicitly tracking object motion, potentially introducing sensitivity to camera movements or unrelated scene changes, particularly in egocentric videos such as Ego4D-NLQ. Could the authors further clarify how this formulation captures grounding-relevant motion and provide additional analysis to validate its effectiveness? Moreover, Table 4 shows that the full method achieves only a marginal improvement over coverage-only selection (10.2 vs 10.1 mIoU). It would be valuable to investigate the independent contribution of query-motion-guided seed selection and whether its additional complexity is justified by consistent improvements across different datasets and token budgets.

  1. Component-Wise Computational Cost Analysis.

Could the authors provide a more detailed breakdown of the computational overhead introduced by each component of FATE, including query decomposition, CLIP-based temporal evidence construction, frame selection, token-budget allocation, and query-motion-guided token preservation? While the current efficiency analysis reports reductions in visual encoding and grounding inference time, a component-wise analysis including preprocessing and auxiliary model inference costs would help clarify the overall computational efficiency and practical benefits of the proposed framework.

  1. Direct Validation of the Claimed Contributions.

Beyond the reported improvements in VTG performance, could the authors provide more direct evidence that the proposed components effectively address the specific problems identified in the paper? While the ablation studies demonstrate the contributions of individual components to overall grounding accuracy, they do not fully explain whether these improvements arise from better preservation of grounding-relevant information. For instance, quantitative or qualitative analyses of temporal evidence coverage, grounding-relevant token retention, and the effectiveness of cross-stage coordination would help clarify how each component addresses its intended problem. Such analyses would provide stronger support for the proposed methodology beyond aggregate performance gains.

7 comments

We sincerely thank the reviewer for the time and care devoted to evaluating our work. We particularly appreciate the thoughtful and rigorous questions, which have encouraged us to revisit several design choices and conduct more thorough experimental analyses. Beyond helping us improve the paper, this process has been a valuable learning experience and has deepened our understanding of both the strengths and limitations of our approach. We are genuinely grateful for this opportunity to learn and improve, and we address the reviewer's concerns below.

Computational Efficiency and Component-wise Overhead (W1/Q2)

We thank the reviewer for highlighting the importance of accounting for the additional computational costs introduced by FATE. To provide a more comprehensive efficiency analysis, we conducted a component-wise latency evaluation on 150 Ego4D-NLQ validation queries, using a maximum frame budget of 2,048. TimeLens2 retains 100% of visual tokens, while SemVID and FATE retain 12.5%.

Table R1. Component-wise Latency Breakdown (Mean Latency, seconds).

StageTimeLens2 (100%)SemVID (12.5%)FATE (12.5%)
Video Decoding17.5917.6617.61
Query Decomposition (API)——1.45
CLIP Video Encoding——34.91
CLIP Text Encoding & Similarity——0.01
Temporal Evidence Construction——0.03
Frame Selection0.190.590.91
Vision Encoder4.875.185.16
Token Allocation & Selection—4.010.56
MLLM Inference30.413.063.08
Cold-start Total53.0630.5063.73
Cached Total (w/ Decoding)53.0630.5028.80
Cached Total (w/o Decoding)35.4712.8411.19

Shared Temporal Evidence includes query decomposition, CLIP encoding, and temporal evidence construction; Coverage-Preserving Frame Selection corresponds to Frame Selection; Relevance-Adaptive Token Allocation and Query-Motion-Guided Token Preservation are included in Token Allocation & Selection. Cached execution reuses precomputed CLIP video features (loading overhead: approximately 0.004 s/query). Model loading is excluded.

Since CLIP video features are query-independent, they can be precomputed and reused across queries. Under this setting, FATE requires 28.80 s, compared with 30.50 s for SemVID and 53.06 s for TimeLens2, corresponding to latency reductions of 5.6% and 45.7%, respectively. Excluding video decoding, the reductions increase to 12.9% and 68.5%, respectively.

We will incorporate this complete breakdown and clarify the corresponding execution settings in the revised paper.

Effectiveness and Generalizability of Temporal Relation Modeling (W1)

We appreciate the reviewer's concerns regarding the effectiveness and generalizability of temporal relation modeling.

Effectiveness of temporal relations. To isolate the contribution of temporal relation modeling from query decomposition, we conducted an additional controlled ablation by setting the relation weight λr to 0 (decomposition only) or 0.5 (relation-aware), while keeping all other components unchanged. The experiments cover all 4,552 Ego4D-NLQ validation queries, using 2,048 maximum frames and 12.5% token retention.

Query Group# QueriesDecomposition OnlyRelation-AwareΔmIoU (pp)
Without valid relations2,9119.7229.7220.000
With valid relations1,64110.51811.098+0.580
All queries4,55210.00910.218+0.209

For queries containing valid temporal relations, relation-aware modeling improves mIoU by +0.580 percentage points, with a paired query-level bootstrap 95% confidence interval of [+0.058, +1.085]. The improvement over all queries is +0.209 percentage points (95% CI: [+0.026, +0.400]). These results provide direct evidence of the contribution of temporal relation composition, independently of query decomposition.

Scope of relation modeling. FATE focuses on explicit temporal dependencies, including before, after, overlap, first, and last, rather than general-purpose event-graph reasoning. Other semantic information, such as object attributes, spatial relationships, and locations, is retained within individual event descriptions for visual matching. More complex dependencies, such as causal or conditional relationships, are not explicitly modeled by the current temporal operators. We will clarify this design scope in the revised paper.

Temporal decay sensitivity. Our relation operators employ exponential temporal decay as a soft proximity prior, rather than imposing hard temporal windows. The default decay scales are τr = 8 s for ordering relations and τo = 2 s for overlapping relations.

To evaluate sensitivity to these choices, we independently varied each parameter while keeping all other settings unchanged.

τr (s)τo (s)mIoU (%)ΔmIoU (pp)Paired Bootstrap 95% CI (pp)
4210.186−0.032[−0.160, +0.097]
8210.218——
16210.162−0.056[−0.193, +0.080]
8110.181−0.036[−0.134, +0.061]
8410.222+0.005[−0.086, +0.094]

Across the evaluated parameter ranges, the overall mIoU changes by at most 0.056 percentage points, and all paired bootstrap confidence intervals include zero. These results indicate that FATE's aggregate grounding performance is relatively insensitive to moderate changes in the temporal decay scales, supporting the robustness of the default configuration within the tested ranges.

Sensitivity to Frame Selection Hyperparameters (W2)

We thank the reviewer for suggesting a sensitivity analysis of the temporal block duration and anchor search radius. We conducted additional grounding experiments on 600 Ego4D-NLQ validation queries from 221 videos, stratified by video duration.

We independently varied the temporal block duration (10/15/30 s) and anchor search radius (0/1/2 anchor intervals), while keeping all other components unchanged.

Table R2. Frame Selection Hyperparameter Sensitivity.

Block DurationAnchor RadiusmIoU (%)ΔmIoU (pp)Paired Bootstrap 95% CI (pp)
10 s111.133+0.353[−0.748, +1.459]
15 s110.781——
30 s110.069−0.712[−1.639, +0.223]
15 s09.817−0.964[−2.217, +0.279]
15 s210.744−0.037[−0.985, +0.931]

The default configuration (15 s, radius = 1) achieves competitive performance across the evaluated settings. Although the 10 s block yields a slightly higher mIoU, none of the tested alternatives shows a statistically established improvement over the default.

Analysis across video durations. We further examine performance across three video-duration groups (200 queries each), reporting mIoU differences relative to the default configuration.

Video DurationBlock 10 sBlock 30 sRadius 0Radius 2
Short (≤1111.933 s)+0.622+0.018−0.223−0.017
Medium (1111.933–1968.066 s)+0.911−0.378−0.317+1.099
Long (>1968.066 s)−0.475−1.774−2.351−1.192

Values represent ΔmIoU in percentage points.

The duration-stratified results show some numerical variation across video lengths, with larger decreases for the 30 s block and radius = 0 on longer videos. However, these trends remain statistically uncertain (e.g., the video-cluster bootstrap 95% CI for the 30 s block on long videos is [−4.089, +0.154] pp).

Overall, the results indicate that the default configuration is a reasonable choice across the evaluated settings, with no statistically established advantage from the tested alternatives. We will include this sensitivity analysis in the revised paper.

Reviewer_KoDzOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

FATE is a training-free framework for efficient MLLM-based video temporal grounding. It uses query-derived temporal evidence to select frames before visual encoding and allocate token budgets afterward. Query-motion seeds and coverage-aware completion determine which tokens to keep. The method is evaluated mainly with TimeLens2-4B on three VTG benchmarks.

Critical strengths

S1. The paper connects frame selection and token pruning through shared temporal evidence, rather than treating them as separate operations.

S2. FATE improves over SemVID on Ego4D-NLQ across the tested budgets. At 512 frames and 12.5% token retention, it reaches 10.5 mIoU versus 7.7 for SemVID (Appendix Table 7).

Critical weaknesses

W1. The end-to-end efficiency benefit is unclear. Table 2 reports only encoder and MLLM time and excludes query decomposition; the costs of CLIP scouting and token/frame selection are not broken out. These extra stages could offset the reported speedup.

W2. The need for the full token-preservation design is not well established. In Table 4, full FATE reaches 10.2 mIoU versus 10.1 with coverage-only selection. The paper also lacks a hybrid baseline combining FATE frame selection with SemVID token compression.

Questions for authors

Q1. Could the authors report matched end-to-end latency for uncompressed TimeLens2, SemVID, and FATE? Please break down query decomposition, CLIP evidence construction, frame selection, token allocation/preservation, visual encoding, and MLLM inference, distinguishing cached from uncached costs.

Q2. Could the authors evaluate SemVID token compression with FATE frame selection under identical frame and token budgets? This would help isolate the gains from frame selection and from the proposed token-preservation strategy.

1 comment

We sincerely appreciate the reviewer's careful and constructive feedback. Your comments have prompted us to revisit our experimental evaluation and conduct additional controlled studies, which have helped us gain a deeper understanding of both the strengths and limitations of our method. We are grateful for these valuable insights and address your concerns below.

W1/Q1: End-to-End Efficiency

We thank the reviewer for pointing out that our original efficiency evaluation did not fully account for the preprocessing overhead. To address this concern, we conducted a component-wise latency evaluation on the same 150 Ego4D-NLQ validation queries, using a maximum frame budget of 2,048. TimeLens2 retains 100% of visual tokens, while SemVID and FATE retain 12.5%. All methods use the same evaluation workload.

Table R1. Component-wise Latency Breakdown (Mean Latency, seconds).

StageTimeLens2 (100%)SemVID (12.5%)FATE (12.5%)
Video Decoding17.5917.6617.61
Query Decomposition (API)——1.45
CLIP Video Encoding——34.91
CLIP Text Encoding & Similarity——0.01
Temporal Evidence Construction——0.03
Frame Selection0.190.590.91
Vision Encoder4.875.185.16
Token Allocation & Selection—4.010.56
MLLM Inference30.413.063.08
Cold-start Total53.0630.5063.73
Cached Total (w/ Decoding)53.0630.5028.80
Cached Total (w/o Decoding)35.4712.8411.19

Mapping to FATE's methodological components: Shared Temporal Evidence comprises Query Decomposition, CLIP Video Encoding, CLIP Text Encoding & Similarity, and Temporal Evidence Construction; Coverage-Preserving Frame Selection corresponds to Frame Selection; Relevance-Adaptive Token Allocation and Query-Motion-Guided Token Preservation are included in Token Allocation & Selection. Cached execution reuses precomputed CLIP video features (feature loading: approximately 0.004 s/query). Model loading is excluded.

With precomputed CLIP video features, FATE achieves a mean latency of 28.80 s, compared with 30.50 s for SemVID and 53.06 s for TimeLens2, corresponding to reductions of 5.6% and 45.7%, respectively. Excluding video decoding, FATE requires 11.19 s, compared with 12.84 s and 35.47 s, corresponding to reductions of 12.9% and 68.5%, respectively.

We will revise the efficiency evaluation to report the complete breakdown and explicitly distinguish these execution settings.


W2/Q2: Hybrid Baseline with FATE Frame Selection and SemVID Token Compression

We thank the reviewer for suggesting this controlled comparison. We implemented a hybrid baseline that retains FATE's frame selection while replacing its entire token processing pipeline with SemVID's. All methods use identical frame-budget constraints and 12.5% token retention.

Max FramesFATE Frames + SemVID TokensFull FATEΔmIoU
5129.1719.789+0.618
20489.68510.218+0.533

All experiments were evaluated on the complete Ego4D-NLQ validation set (4,552 queries). These results show that, even with FATE frame selection held fixed, FATE's token processing pipeline consistently outperforms SemVID's across both frame budgets. Thus, the improvement cannot be attributed solely to frame selection.


W2: Contribution of Query-Motion-Guided Token Preservation

We thank the reviewer for this observation. We agree that the additional improvement from query-motion-guided seeding is relatively modest compared with the contributions of other components.

To provide a more precise comparison, we report the full-precision results on all 4,552 Ego4D-NLQ validation queries, using 2,048 frames and 12.5% token retention.

Token PreservationmIoU (%)
Coverage Only10.090
Query + Coverage9.849
Motion + Coverage9.898
Query-Motion + Coverage (ρ=0.25)10.200
Full FATE (ρ=0.5)10.218

Full FATE improves over Coverage Only by 0.128 mIoU percentage points. The results also show that combining query relevance and temporal variation with coverage achieves higher mIoU than using either signal individually.

These findings suggest that coverage-aware selection provides a strong foundation, while query-motion-guided seeding serves as a complementary refinement. Although its incremental gain is smaller than that of other components, it contributes positively to the overall performance. We will clarify the respective roles of these mechanisms in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.