- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Considering existing methods focus on either selecting informative frames before visual encoding or compressing visual tokens after encoding, this paper proposes a training-free framework FATE. FATE integrates these two stages to improve model's temporal grounding performance.
Critical strengths
FATE offers a solution to the limitation of handling frame selection and token compression as separate processes. It proposes a training-free framework FATE, making it practical to any MLLMs.
Critical weaknesses
-
The abstract and introduction indicate shared temporal evidence as an important contribution, and Section 3.3 uses temporal evidence to guide both frame selection and token allocation. However, frame selection and token allocation based on visual-textual similarity are not novel. For example, "Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs" uses similarity to guide frame selection. "A TRAINING-FREE FRAMEWORK FOR LONG VIDEO UNDERSTANDING VIA VIDEO-QUERY-OPTIONS SIMILARITY" adjusts FPS and spatial resolution based on similarity. The experiments did not include comparisons with these methods, so it remains unclear whether "shared temporal evidence" is truly important.
-
The experimental results do not support the necessity of the proposed method. In Table 3, the proposed frame-selection strategy improves mIoU by only 0.2 over simple Top-K selection. In Table 4, the full method exceeds Coverage Only by 0.1, while "Query + Coverage" and "Motion + Coverage" actually underperform Coverage Only, without sufficient analysis.
Questions for authors
-
Why use frame-wise semantic similarity? In my view, for instance, "taking out a bottle" versus "putting in a bottle" are often difficult to distinguish based on a single frame alone. Have the authors considered a segment-wise semantic similarity?
-
FATE integrates both "pre-encoding frame selection" and "post-encoding token compression." Have the authors evaluated the combination of these two existing categories of methods? For instance, could the "uniform" and "Top-k" approaches in Table 3 be replaced with other existing frame selection methods for comparison?
-
The comparative method FlashVID is not specifically for TVG. Beyond SemVID, have the authors considered other compression methods designed for TVG tasks? Furthermore, have they evaluated FATE on general video understanding benchmarks, such as LongVideoBench?
-
There is no analysis for Table 4.
3 comments
Effectiveness of Frame Selection and Query-Motion-Guided Preservation (W2/Q4)
We appreciate the reviewer's concerns regarding the modest improvements in Tables 3 and 4. We conducted additional controlled experiments to further examine these two components.
1. Coverage-preserving frame selection. We further compared Top-K and FATE frame selection on all 4,552 Ego4D-NLQ validation queries under a 512-frame budget, keeping the subsequent token-processing pipeline identical.
| Frame Selection | mIoU (%) |
|---|---|
| Top-K | 9.462 |
| FATE | 9.789 |
Both settings use 512 frames and 12.5% token retention.
FATE improves mIoU by +0.327 pp, complementing the +0.2 pp improvement reported in Table 3 under 2,048 frames. These results show a consistent numerical advantage over Top-K across the two frame budgets, although the improvements remain modest.
2. Query-motion-guided token preservation. We further examined the contribution of query-motion guidance on the complete Ego4D-NLQ validation set using 2,048 frames and 12.5% token retention.
| Token Preservation Strategy | mIoU (%) |
|---|---|
| Coverage Only | 10.090 |
| Query + Coverage | 9.849 |
| Motion + Coverage | 9.898 |
| Query-Motion Seeds Only | 9.800 |
| Query-Motion + Coverage (ρ = 0.25) | 10.200 |
| Full FATE (ρ = 0.5) | 10.218 |
These results confirm that coverage-aware preservation is the primary contributor to fine-grained token selection. Introducing query relevance or temporal variation individually does not improve performance, potentially because either signal alone can prioritize less useful tokens at the expense of coverage. However, combining both signals with coverage achieves a modest improvement of +0.128 pp over Coverage Only.
We therefore view query-motion-guided selection as a complementary refinement rather than the principal source of FATE's performance gains. We will clarify these findings and provide a more detailed interpretation of Table 4 in the revised paper.
Frame-wise vs. Segment-wise Semantic Similarity (Q1)
We appreciate the reviewer's insightful example. We agree that frame-wise semantic similarity alone may not reliably distinguish temporally reversed actions, such as "taking out a bottle" and "putting in a bottle."
In FATE, frame-wise CLIP similarity serves as a lightweight relevance signal for guiding frame selection and token allocation, rather than directly determining event boundaries or recognizing fine-grained actions. The selected frames retain their temporal ordering, allowing the downstream MLLM to reason over the preserved visual evidence for final grounding. Moreover, frame-wise CLIP embeddings are query-independent and can be precomputed and reused across queries, avoiding repeated video-side encoding.
We agree that segment-wise similarity, particularly when computed using a temporally aware video encoder, could better capture action dynamics and directionality. However, this would introduce additional temporal modeling and potentially higher preprocessing costs. Exploring such evidence representations is a valuable extension, while our current framework focuses on coordinating visual reduction using lightweight, reusable temporal evidence.
Combination of Existing Frame Selection and Token Compression Methods (Q2)
We appreciate the reviewer's suggestion. Our evaluation prioritizes controlled comparisons with VTG-specific compression methods, particularly SemVID, while keeping the grounding backbone, frame budget, and token-retention ratio consistent.
Existing methods such as Q-Frame and VQOS jointly optimize frame sampling and input resolution. Combining their complete pipelines with post-encoder token compression would require adapting their resource-allocation strategies to a common VTG evaluation protocol.
To investigate the contributions of the two stages, we additionally conducted controlled experiments on all 4,552 Ego4D-NLQ validation queries, using 512 frames and 12.5% token retention.
| Frame Selection | Token Processing | mIoU (%) |
|---|---|---|
| Top-K | FATE | 9.462 |
| FATE | SemVID | 9.171 |
| FATE | FATE | 9.789 |
With frame selection fixed, FATE's token processing improves mIoU by +0.618 pp over SemVID's processing (95% CI: [+0.187, +1.034]). With token processing fixed, FATE's frame selection improves mIoU by +0.327 pp over Top-K.
These experiments isolate the contributions of the two stages, although they do not exhaustively evaluate combinations of existing frame selectors and token compressors. We will clarify this distinction in the revised paper.
VTG-Specific Baselines and General Video Understanding (Q3)
Baseline selection. We select SemVID as our primary baseline because it is a state-of-the-art training-free token compression method specifically designed for VTG, enabling controlled comparisons under the same grounding backbone and visual-token budget. Other VTG methods, such as GroundVTS and VTG-LLM, incorporate trainable modules or model-specific adaptations, making them less directly comparable under our training-free evaluation protocol. We additionally include FlashVID to examine how a general-purpose video compression method performs on VTG.
Evaluation scope. Our three VTG benchmarks cover long-, medium-, and short-form videos, allowing us to evaluate FATE across different temporal scales. We focus on VTG because accurate temporal localization requires preserving event progression and temporal context, rather than only identifying salient visual content. We agree that evaluation on LongVideoBench and other general video understanding benchmarks would help assess broader transferability. However, general VideoQA performance is not a claim established by our current experiments.
We sincerely thank the reviewer for the time and effort devoted to carefully evaluating our work. We particularly appreciate the thoughtful and critical feedback, which has encouraged us to reconsider how we present our contributions and to examine the effectiveness of our design choices more thoroughly. This review process has been a valuable learning experience for us, helping us better understand both the strengths and limitations of our work. We are genuinely grateful for these insights and the opportunity to improve our research, and we address the reviewer's concerns below.
Novelty and Effectiveness of Shared Temporal Evidence (W1)
We thank the reviewer for highlighting these relevant works. We agree that similarity-guided frame selection and adaptive visual resource allocation have been explored in Q-Frame and A Training-Free Framework for Long Video Understanding via Video-Query-Options Similarity (VQOS). However, FATE differs from these approaches in both its VTG-oriented temporal evidence construction and cross-stage visual reduction strategy, as summarized below.
While Q-Frame and VQOS coordinate frame sampling and resolution adaptation before visual encoding, FATE coordinates pre-encoder frame selection with post-encoder token allocation and preservation. Moreover, FATE incorporates explicit temporal relationships between decomposed events, rather than relying solely on visual-textual similarity. These designs specifically address VTG, where preserving event progression and temporal context is important for accurate localization.
To further validate these contributions, we conducted additional controlled experiments on the full Ego4D-NLQ validation set (4,552 queries):
These results provide additional evidence for the effectiveness of relation-aware temporal evidence and post-encoder adaptive token allocation.