- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CueMiner, a training-free visual token compression framework that aims to preserve temporal reasoning accuracy by retaining both salient events and their preceding visual context. By combining bidirectional scoring, segment-wise information estimation, and distributional shift-based segmentation, CueMiner achieves competitive performance across multiple Video-LLMs under aggressive token compression.
Critical strengths
-
Strong Empirical Performance. CueMiner consistently outperforms existing visual token compression methods across multiple benchmarks and Video-LLMs, particularly under aggressive compression ratios. The ability to preserve substantial accuracy with only 5% of visual tokens demonstrates its practical potential.
-
Comprehensive Experimental Evaluation. The paper provides extensive evaluations across four Video-LLMs and multiple benchmarks, along with detailed ablation studies examining the contributions of individual components. These experiments provide empirical support for the effectiveness of the proposed framework.
-
Practicality and Generalizability. CueMiner operates without additional training or fine-tuning and can be integrated into existing Video-LLMs. Its query-independent compression strategy also offers potential benefits for efficiently processing multiple questions about the same video.
Critical weaknesses
[W1] Limited Evidence for Temporal Reasoning. Although the paper emphasizes preserving temporal reasoning capabilities, most evaluations rely on general video QA accuracy. The task-specific analyses are relatively limited, making it difficult to determine whether the improvements primarily reflect better temporal reasoning or general visual information preservation.
[W2] Simplified Modeling of Visual Token Distributions. CueMiner approximates visual token distributions using single diagonal Gaussians, which may not adequately capture complex or multimodal visual representations. Although the authors compare against GMMs, the experiments do not fully establish the robustness of these distributional assumptions across diverse video characteristics.
[W3] Limited Practical Efficiency Gains. Despite substantial reductions in retained visual tokens, the improvements in end-to-end inference latency are relatively modest in the main experiments. This raises concerns about whether the additional complexity of the proposed compression pipeline translates into meaningful practical benefits beyond accuracy preservation.
[W4] Limited Methodological Novelty. The proposed framework primarily combines established techniques, including surprisal-based scoring, distributional divergence, entropy-based allocation, and DPP selection. While their integration yields competitive results, the methodological contribution appears incremental, particularly given recent advances in spatio-temporal visual token compression.
Questions for authors
-
CueMiner performs forward and backward scoring only within individual temporal segments, while VTDS places segment boundaries at significant distributional changes. Since these changes may correspond to key events, could the proposed segmentation separate an event from the antecedent visual evidence needed to interpret it?
-
Although backward scoring improves downstream accuracy, it measures visual distinctiveness relative to future observations rather than relevance to subsequent events. Could the authors quantitatively evaluate whether backward scoring improves the retention of event-relevant antecedent evidence, for example through annotated temporal evidence intervals or controlled state-transition scenarios?
-
The current memory analysis focuses on incremental LLM-prefill memory, excluding other inference stages. Could the authors report end-to-end peak GPU memory, including segmentation, density estimation, and token selection? Furthermore, why does the simpler Top-k strategy outperform DPP in both accuracy and latency as the number of sampled frames increases?
-
ST-SimDiff already combines similarity-based redundancy reduction and temporal difference-based event preservation. Could the authors clarify the methodological distinction and, if possible, include a direct comparison under matched retention budgets?
I am willing to reassess my score during the discussion period if these concerns are adequately addressed.
5 comments
Question 1. We thank the reviewer for the interesting interpretation. Before addressing this concern, we first clarify the distinction between key event detection and segment boundary detection. First, we argue that key events occur when the overall visual-token distributions between consecutive frames exhibit substantial overlap, as indicated by a small symmetric KL divergence. Under such conditions, a key event represents a localized change in visual content rather than a substantial change in the overall scene. For example, in Figure 1, the appearance of a bus in Frame 3 is a key event because the overall visual content within the frame remains similar to that in Frame 2, but the small subset of visual tokens corresponding to the bus is unique under the visual-token distribution of preceding Frames 1 and 2. Thus, this case corresponds to the key event described in the reviewer's concern but would not produce a segment boundary.
In contrast, we argue that segment boundaries are characterized by substantial changes in the overall visual-token distributions between consecutive frames, as indicated by a large symmetric KL divergence between these distributions. For example, if Frame 3 in Figure 1 abruptly depicted a person standing on Mars, it would represent a scene transition rather than a key event. Conversely, this case would produce a segment boundary but would not correspond to the key event described in the reviewer's concern. Also, this is the case that the reviewer interprets as a key event, whereas it actually represents a scene transition.
Under this argument, the frame immediately before a segment boundary (the last frame of the preceding segment) is more likely to contain a key event within the preceding segment, whereas the frame immediately after a segment boundary (the first frame of the following segment) is more likely to contain antecedent visual evidence for a subsequent key event within the following segment, rather than antecedent visual evidence and a subsequent key event.
Question 2. As you suggest, we construct a benchmark of controlled state-transition scenarios to quantitatively evaluate whether backward scoring preserves event-relevant antecedent evidence (see benchmark sample in Figures 7, 8, 9, and 10). Through this quantitative evaluation, we confirm that backward scoring identifies and retains antecedent visual evidence in these scenarios. Specifically, through backward scoring, over 90% of the visual tokens corresponding to the state prior to the transition are retained on average, even at a visual-token retention ratio of 10% (see Table 17). The detailed experimental setup and results are described in the first paragraph of Appendix A.10.2.
Question 3. First, we report the end-to-end GPU memory overhead of CueMiner’s compression stages, including segmentation, density estimation, and token selection, in Figure 6 (Appendix A.6). The results show that compression adds only a small amount of GPU memory overhead.
Second, to explain why Top-k outperforms DPP-based selection in the large-frame regime, we first provide an insightful observation that the retained-token budget grows linearly with the number of sampled frames, whereas the number of visual features within the video grows logarithmically (see Figure 14). When the number of sampled video frames is sufficiently large, the token budget substantially exceeds the number of visual features since linear growth asymptotically dominates logarithmic growth (see region B in Figure 14). Thus, even though Top-k considers only the importance of individual visual tokens without accounting for inter-token redundancy, it can still select a visual-token subset with sufficiently diverse representation directions since the token budget substantially exceeds the number of visual features within a video when the number of sampled frames is large. This explains why Top-k outperforms DPP-based selection in the large-frame regime; see Appendix A.10.8 for the detailed analysis.
Question 4. We clarify two key methodological differences between ST-SimDiff and CueMiner. First, for similarity-based redundancy reduction, ST-SimDiff selects representative visual tokens based on similarity, without jointly optimizing token importance and redundancy. However, this overlooks informative tokens during redundancy reduction. In contrast, CueMiner uses DPP to simultaneously consider each token's importance and its similarity to other tokens, preserving informative tokens while reducing redundancy. Second, for temporal difference-based event preservation, ST-SimDiff compares visual tokens at the same spatial position across adjacent frames. However, such temporal-only methods do not account for the spatial dimension. Consequently, they are less effective at capturing visual content undergoing spatial displacement across frames (see Figure 5). In contrast, CueMiner considers visual tokens from all spatial positions across preceding frames, allowing it to recognize previously observed objects despite spatial displacement. Figure 5(c) shows that CueMiner effectively preserves visual tokens corresponding to objects undergoing spatial displacement.
As requested, we additionally compare CueMiner with ST-SimDiff under matched VLLMs and visual-token retention ratios of 20%, 10%, and 5%. The results show that CueMiner consistently and substantially outperforms ST-SimDiff across all four VLLMs and all three retention ratios. Please refer to the table below for the detailed comparison results.
| Method | LOV | LOV 1.5 | LOV 2.0 | Qwen3-VL | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Retention ratio | 20% | 10% | 5% | 20% | 10% | 5% | 20% | 10% | 5% | 20% | 10% | 5% |
| ST-SimDiff | 45.50 | 44.38 | 39.79 | 38.69 | 37.73 | 36.82 | 44.20 | 40.40 | 38.40 | 42.75 | 37.98 | 34.38 |
| CueMiner (Top-k) | 49.40 | 46.60 | 46.50 | 46.50 | 45.60 | 44.30 | 51.70 | 49.20 | 45.80 | 46.70 | 44.40 | 42.10 |
| CueMiner (DPP) | 50.00 | 47.30 | 47.40 | 47.60 | 47.40 | 45.60 | 53.50 | 51.50 | 48.70 | 47.60 | 45.90 | 43.20 |
We thank the reviewer for the insightful question and the opportunity for a meaningful discussion.
Weakness 1. We clarify that our paper already includes a task-specific evaluation directly related to temporal reasoning. Specifically, Table 12 reports accuracy on the State Change subtask of MVBench, which evaluates the ability of VLLMs to recognize changes in states over time. The results show that CueMiner achieves the highest accuracy among all compared compression methods across all four VLLMs at a 10% retention ratio and achieves the highest or tied-highest accuracy at a 5% retention ratio. These results provide task-specific evidence that CueMiner effectively preserves temporal reasoning capabilities even under aggressive visual-token compression.
Weakness 3. We acknowledge that the end-to-end inference speedup is modest because compression incurs additional overhead. However, the primary efficiency of CueMiner is memory reduction rather than maximizing inference speedup. Therefore, its practical efficiency benefits should be evaluated not only in terms of inference latency but also in terms of GPU memory consumption. As reported in Appendix A.5 (Table 7), the estimated incremental GPU memory usage during LLM prefill is reduced by up to approximately 8.3× relative to the vanilla model. These results demonstrate that CueMiner provides meaningful practical efficiency benefits through substantial GPU memory reduction while preserving accuracy, despite its modest end-to-end inference speedup.
Weakness 4. We clarify CueMiner's methodological novelty over existing approaches as follows. CueMiner introduces four key methodological advances that address distinct limitations of existing visual-token compression methods. First, spatio-temporal forward scoring (Section 3.2.1) considers visual observations across both temporal and spatial dimensions, whereas existing temporal-only methods compare tokens at fixed spatial positions, making them less effective at capturing visual content undergoing spatial displacement (Figure 5). Second, backward scoring (Section 3.2.2) explicitly identifies antecedent visual evidence preceding key events, which existing forward-only methods overlook despite its importance for temporal reasoning. Third, segment-wise estimation (Section 3.3) accounts for temporal redundancy across consecutive frames, whereas conventional frame-wise estimation overestimates the information content of visually repetitive tokens (Figure 3(a)). Fourth, visual-token distributional shift (Section 3.4) identifies segment boundaries based on changes in visual-token distributions, overcoming the limitations of cosine similarity between frame-level representations in separating visually distinct segments (Figure 3(b)). These contributions are not merely a combination of established techniques; each addresses a specific limitation of previous approaches. Moreover, whereas existing methods typically rely on heuristic metrics to estimate visual-token importance, CueMiner defines its forward, backward, and local scoring metrics based on the information-theoretic definition of information content (Equations 3, 6, and 9).