acceptodds
Under review as a conference paper at ICLR 2027

CueMiner: Efficient and Accurate Video Large Language Models via Context Mining

Abstract

How can we minimize memory consumption during Video Large Language Model (VLLM) inference while preserving temporal reasoning accuracy? VLLMs achieve strong temporal reasoning performance in video understanding by processing rich visual information across video frames, yet retaining the full visual representation incurs considerable memory overhead. To alleviate this overhead, existing approaches compress video representations by selectively retaining only visual information deemed important. Specifically, they excavate information associated with key events. However, these approaches do not consider preserving the visual evidence that temporally precedes such events, even though this evidence provides critical context for interpreting temporally subsequent key events. Consequently, aggressive compression degrades temporal reasoning accuracy in VLLMs. In this paper, we propose CueMiner, an efficient video compression method that preserves temporal reasoning accuracy even under aggressive compression. To preserve accuracy, CueMiner not only captures salient events, but also mines the temporally antecedent visual context that enables VLLMs to interpret how subsequent events develop over time. On MLVU with LLaVA-OneVision, CueMiner retains 99.4% of the accuracy at a retention ratio of 5%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Large Language Models, Visual Token Compression, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Y6sbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CueMiner, a training-free visual token compression framework that aims to preserve temporal reasoning accuracy by retaining both salient events and their preceding visual context. By combining bidirectional scoring, segment-wise information estimation, and distributional shift-based segmentation, CueMiner achieves competitive performance across multiple Video-LLMs under aggressive token compression.

Critical strengths
  1. Strong Empirical Performance. CueMiner consistently outperforms existing visual token compression methods across multiple benchmarks and Video-LLMs, particularly under aggressive compression ratios. The ability to preserve substantial accuracy with only 5% of visual tokens demonstrates its practical potential.

  2. Comprehensive Experimental Evaluation. The paper provides extensive evaluations across four Video-LLMs and multiple benchmarks, along with detailed ablation studies examining the contributions of individual components. These experiments provide empirical support for the effectiveness of the proposed framework.

  3. Practicality and Generalizability. CueMiner operates without additional training or fine-tuning and can be integrated into existing Video-LLMs. Its query-independent compression strategy also offers potential benefits for efficiently processing multiple questions about the same video.

Critical weaknesses

[W1] Limited Evidence for Temporal Reasoning. Although the paper emphasizes preserving temporal reasoning capabilities, most evaluations rely on general video QA accuracy. The task-specific analyses are relatively limited, making it difficult to determine whether the improvements primarily reflect better temporal reasoning or general visual information preservation.

[W2] Simplified Modeling of Visual Token Distributions. CueMiner approximates visual token distributions using single diagonal Gaussians, which may not adequately capture complex or multimodal visual representations. Although the authors compare against GMMs, the experiments do not fully establish the robustness of these distributional assumptions across diverse video characteristics.

[W3] Limited Practical Efficiency Gains. Despite substantial reductions in retained visual tokens, the improvements in end-to-end inference latency are relatively modest in the main experiments. This raises concerns about whether the additional complexity of the proposed compression pipeline translates into meaningful practical benefits beyond accuracy preservation.

[W4] Limited Methodological Novelty. The proposed framework primarily combines established techniques, including surprisal-based scoring, distributional divergence, entropy-based allocation, and DPP selection. While their integration yields competitive results, the methodological contribution appears incremental, particularly given recent advances in spatio-temporal visual token compression.

Questions for authors
  1. CueMiner performs forward and backward scoring only within individual temporal segments, while VTDS places segment boundaries at significant distributional changes. Since these changes may correspond to key events, could the proposed segmentation separate an event from the antecedent visual evidence needed to interpret it?

  2. Although backward scoring improves downstream accuracy, it measures visual distinctiveness relative to future observations rather than relevance to subsequent events. Could the authors quantitatively evaluate whether backward scoring improves the retention of event-relevant antecedent evidence, for example through annotated temporal evidence intervals or controlled state-transition scenarios?

  3. The current memory analysis focuses on incremental LLM-prefill memory, excluding other inference stages. Could the authors report end-to-end peak GPU memory, including segmentation, density estimation, and token selection? Furthermore, why does the simpler Top-k strategy outperform DPP in both accuracy and latency as the number of sampled frames increases?

  4. ST-SimDiff already combines similarity-based redundancy reduction and temporal difference-based event preservation. Could the authors clarify the methodological distinction and, if possible, include a direct comparison under matched retention budgets?

I am willing to reassess my score during the discussion period if these concerns are adequately addressed.

5 comments

We thank the reviewer for the insightful question and the opportunity for a meaningful discussion.

Weakness 1. We clarify that our paper already includes a task-specific evaluation directly related to temporal reasoning. Specifically, Table 12 reports accuracy on the State Change subtask of MVBench, which evaluates the ability of VLLMs to recognize changes in states over time. The results show that CueMiner achieves the highest accuracy among all compared compression methods across all four VLLMs at a 10% retention ratio and achieves the highest or tied-highest accuracy at a 5% retention ratio. These results provide task-specific evidence that CueMiner effectively preserves temporal reasoning capabilities even under aggressive visual-token compression.

Weakness 3. We acknowledge that the end-to-end inference speedup is modest because compression incurs additional overhead. However, the primary efficiency of CueMiner is memory reduction rather than maximizing inference speedup. Therefore, its practical efficiency benefits should be evaluated not only in terms of inference latency but also in terms of GPU memory consumption. As reported in Appendix A.5 (Table 7), the estimated incremental GPU memory usage during LLM prefill is reduced by up to approximately 8.3× relative to the vanilla model. These results demonstrate that CueMiner provides meaningful practical efficiency benefits through substantial GPU memory reduction while preserving accuracy, despite its modest end-to-end inference speedup.

Weakness 4. We clarify CueMiner's methodological novelty over existing approaches as follows. CueMiner introduces four key methodological advances that address distinct limitations of existing visual-token compression methods. First, spatio-temporal forward scoring (Section 3.2.1) considers visual observations across both temporal and spatial dimensions, whereas existing temporal-only methods compare tokens at fixed spatial positions, making them less effective at capturing visual content undergoing spatial displacement (Figure 5). Second, backward scoring (Section 3.2.2) explicitly identifies antecedent visual evidence preceding key events, which existing forward-only methods overlook despite its importance for temporal reasoning. Third, segment-wise estimation (Section 3.3) accounts for temporal redundancy across consecutive frames, whereas conventional frame-wise estimation overestimates the information content of visually repetitive tokens (Figure 3(a)). Fourth, visual-token distributional shift (Section 3.4) identifies segment boundaries based on changes in visual-token distributions, overcoming the limitations of cosine similarity between frame-level representations in separating visually distinct segments (Figure 3(b)). These contributions are not merely a combination of established techniques; each addresses a specific limitation of previous approaches. Moreover, whereas existing methods typically rely on heuristic metrics to estimate visual-token importance, CueMiner defines its forward, backward, and local scoring metrics based on the information-theoretic definition of information content (Equations 3, 6, and 9).

Question 1. We thank the reviewer for the interesting interpretation. Before addressing this concern, we first clarify the distinction between key event detection and segment boundary detection. First, we argue that key events occur when the overall visual-token distributions between consecutive frames exhibit substantial overlap, as indicated by a small symmetric KL divergence. Under such conditions, a key event represents a localized change in visual content rather than a substantial change in the overall scene. For example, in Figure 1, the appearance of a bus in Frame 3 is a key event because the overall visual content within the frame remains similar to that in Frame 2, but the small subset of visual tokens corresponding to the bus is unique under the visual-token distribution of preceding Frames 1 and 2. Thus, this case corresponds to the key event described in the reviewer's concern but would not produce a segment boundary.

In contrast, we argue that segment boundaries are characterized by substantial changes in the overall visual-token distributions between consecutive frames, as indicated by a large symmetric KL divergence between these distributions. For example, if Frame 3 in Figure 1 abruptly depicted a person standing on Mars, it would represent a scene transition rather than a key event. Conversely, this case would produce a segment boundary but would not correspond to the key event described in the reviewer's concern. Also, this is the case that the reviewer interprets as a key event, whereas it actually represents a scene transition.

Under this argument, the frame immediately before a segment boundary (the last frame of the preceding segment) is more likely to contain a key event within the preceding segment, whereas the frame immediately after a segment boundary (the first frame of the following segment) is more likely to contain antecedent visual evidence for a subsequent key event within the following segment, rather than antecedent visual evidence and a subsequent key event.

Question 2. As you suggest, we construct a benchmark of controlled state-transition scenarios to quantitatively evaluate whether backward scoring preserves event-relevant antecedent evidence (see benchmark sample in Figures 7, 8, 9, and 10). Through this quantitative evaluation, we confirm that backward scoring identifies and retains antecedent visual evidence in these scenarios. Specifically, through backward scoring, over 90% of the visual tokens corresponding to the state prior to the transition are retained on average, even at a visual-token retention ratio of 10% (see Table 17). The detailed experimental setup and results are described in the first paragraph of Appendix A.10.2.

Question 3. First, we report the end-to-end GPU memory overhead of CueMiner’s compression stages, including segmentation, density estimation, and token selection, in Figure 6 (Appendix A.6). The results show that compression adds only a small amount of GPU memory overhead.

Second, to explain why Top-k outperforms DPP-based selection in the large-frame regime, we first provide an insightful observation that the retained-token budget grows linearly with the number of sampled frames, whereas the number of visual features within the video grows logarithmically (see Figure 14). When the number of sampled video frames is sufficiently large, the token budget substantially exceeds the number of visual features since linear growth asymptotically dominates logarithmic growth (see region B in Figure 14). Thus, even though Top-k considers only the importance of individual visual tokens without accounting for inter-token redundancy, it can still select a visual-token subset with sufficiently diverse representation directions since the token budget substantially exceeds the number of visual features within a video when the number of sampled frames is large. This explains why Top-k outperforms DPP-based selection in the large-frame regime; see Appendix A.10.8 for the detailed analysis.

Question 4. We clarify two key methodological differences between ST-SimDiff and CueMiner. First, for similarity-based redundancy reduction, ST-SimDiff selects representative visual tokens based on similarity, without jointly optimizing token importance and redundancy. However, this overlooks informative tokens during redundancy reduction. In contrast, CueMiner uses DPP to simultaneously consider each token's importance and its similarity to other tokens, preserving informative tokens while reducing redundancy. Second, for temporal difference-based event preservation, ST-SimDiff compares visual tokens at the same spatial position across adjacent frames. However, such temporal-only methods do not account for the spatial dimension. Consequently, they are less effective at capturing visual content undergoing spatial displacement across frames (see Figure 5). In contrast, CueMiner considers visual tokens from all spatial positions across preceding frames, allowing it to recognize previously observed objects despite spatial displacement. Figure 5(c) shows that CueMiner effectively preserves visual tokens corresponding to objects undergoing spatial displacement.

As requested, we additionally compare CueMiner with ST-SimDiff under matched VLLMs and visual-token retention ratios of 20%, 10%, and 5%. The results show that CueMiner consistently and substantially outperforms ST-SimDiff across all four VLLMs and all three retention ratios. Please refer to the table below for the detailed comparison results.

MethodLOVLOV 1.5LOV 2.0Qwen3-VL
Retention ratio20%10%5%20%10%5%20%10%5%20%10%5%
ST-SimDiff45.5044.3839.7938.6937.7336.8244.2040.4038.4042.7537.9834.38
CueMiner (Top-k)49.4046.6046.5046.5045.6044.3051.7049.2045.8046.7044.4042.10
CueMiner (DPP)50.0047.3047.4047.6047.4045.6053.5051.5048.7047.6045.9043.20
Reviewer_yvdTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CueMiner is a training-free, question-independent visual token pruning method for video large language models. It combines bidirectional surprisal scoring, segment-wise information estimation, and distributional-shift-based segmentation, followed by budget allocation and DPP-based token selection. Experiments across multiple models and benchmarks show competitive accuracy, particularly at low token retention ratios.

Critical strengths
  1. Well-motivated and practically relevant. Preserving visual context preceding key events is a reasonable approach to retaining information needed for understanding state transitions. The training-free and question-independent design also offers potential for deployment and reuse across questions.

  2. Broad experimental coverage. The evaluation covers multiple models, benchmarks, retention ratios, and frame counts, with competitive overall accuracy.

  3. Substantial ablation studies. The appendix examines scoring, segmentation, budgeting, and selection strategies. Reporting cases where Top-k outperforms DPP also helps clarify the method’s practical trade-offs.

Critical weaknesses
  1. The relationship to CDPruner deserves a clearer discussion and comparison. CDPruner (NeurIPS 2025) already uses quality-weighted DPP for visual token pruning and includes video experiments. CueMiner primarily differs in its bidirectional and local surprisal scores and segment-wise processing. Although CDPruner is cited, the connection between their selection formulations is not sufficiently discussed, and a direct comparison is missing.

  2. The efficiency and applicability of DPP selection need clarification. Tables 22–23 show that, with more sampled frames, Top-k can achieve both higher accuracy and lower latency than DPP, while the DPP variant is slower than existing baselines in some settings. This does not undermine the proposed scoring approach, but suggests that efficiency depends on input size and selection strategy.

  3. Some implementation details remain underspecified. The appendix provides the DPP objective and budget constraints, but does not clearly describe the actual solver or the conversion from segment entropy to integer token budgets. Concise pseudocode covering budget rounding, capacity constraints, and numerical stability would improve reproducibility.

[1] Qizhe Zhang, Mengzhen Liu, Lichen Li, Ming Lu, Yuan Zhang, Junwen Pan, Qi She, and Shanghang Zhang. Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs. NeurIPS, 2025.

Questions for authors
  1. Which algorithm is used for DPP selection? Can a segment’s token budget exceed the feature dimension, and how is this case handled?

  2. Could the authors visualize the finally selected tokens to demonstrate the preservation of meaningful pre-event context? The current heatmaps and ablations support this interpretation, but do not fully distinguish it from sensitivity to general visual changes.

4 comments

We thank the reviewer for the insightful question and the opportunity for a meaningful discussion.

Weakness 1. We clarify two key methodological differences between CDPruner and CueMiner. First, for token quality estimation, CDPruner estimates token importance based on relevance to each user query, whereas CueMiner estimates token importance based on the visual content of the video itself. This leads to two key differences. (1) Question awareness: CDPruner is question-aware, requiring query-dependent token selection and introducing repeated compression overhead when serving multiple questions about the same video. In contrast, CueMiner is question-agnostic, allowing compressed visual tokens to be reused across multiple queries, making it more efficient for large-scale video question answering. (2) Video-specific importance estimation: CDPruner estimates token importance without explicitly modeling temporal relationships across video frames. In contrast, CueMiner incorporates the temporal characteristics of videos through forward scoring to capture key events, backward scoring to preserve antecedent visual evidence, and segment-wise local scoring to account for temporal redundancy. Second, for token budget allocation, CDPruner uses a uniform per-frame token budget in its video experiments, without considering differences in information content across frames. In contrast, CueMiner adaptively allocates token budgets across temporal segments based on their entropy, preserving more tokens in informative segments.

As requested, we additionally compare CueMiner with CDPruner under matched VLLMs and visual-token retention ratios of 20%, 10%, and 5%. The results show that CueMiner consistently and substantially outperforms CDPruner across all four VLLMs and all three retention ratios. Please refer to the table below for the detailed comparison results.

MethodLOVLOV 1.5LOV 2.0Qwen3-VL
Retention ratio20%10%5%20%10%5%20%10%5%20%10%5%
CDPruner45.3844.6839.4643.2639.4935.3342.1037.4036.2039.1836.3936.79
CueMiner (Top-k)49.4046.6046.5046.5045.6044.3051.7049.2045.8046.7044.4042.10
CueMiner (DPP)50.0047.3047.4047.6047.4045.6053.5051.5048.7047.6045.9043.20

Weakness 2. To clarify the efficiency and applicability of DPP-based selection, we distinguish between two cases based on the number of sampled frames: fewer frames and more frames. When the number of sampled frames is small, CueMiner with DPP-based selection outperforms other baselines, including Top-k selection, in accuracy (see Table 23 with 48-frame setting), while incurring only slightly higher inference latency than other methods (see Table 24 with 48-frame setting). However, as the number of sampled frames to be processed increases, the latency overhead introduced by DPP-based selection also increases, as the reviewer pointed out (see Table 24). To address this, we show that when processing a large number of sampled frames, CueMiner with Top-k selection outperforms all other baselines, including CueMiner with DPP-based selection, in both accuracy and inference latency (see Tables 23 and 24). Considering that the accuracy and efficiency of these selection strategies vary with the number of sampled frames, we provide a practical guideline: use DPP-based selection when processing fewer frames and Top-k selection when processing more frames. For an explanation of why DPP-based selection and Top-k selection outperform each other depending on the number of sampled frames, please refer to Appendix A.10.10.

Weakness 3. We have revised Appendix A.3 to provide concrete implementation details beyond the objectives for both token budgeting and DPP-based selection.

Question 1. First, we use a greedy selection algorithm. Starting from an empty set, we iteratively select the token with the largest marginal log-determinant gain until the allocated segment budget is reached. Details are provided in Appendix A.3.2. Second, in the experiments at visual-token retention ratios of 20%, 10%, and 5%, we did not observe any segment budget exceeding the feature dimension. However, as the reviewer points out, such cases can arise at higher retention ratios. Our current DPP selection design does not include a promising process for handling such cases, which we acknowledge as a limitation. To handle the case, we consider one possible solution to be imposing the constraint Bs≤dB_s \leq d at the budgeting stage and redistributing excess allocations among other segments, rather than handling such cases during DPP selection.

Question 2. As requested, we additionally provide multiple visualizations of the selected visual tokens beyond the forward and backward scoring heatmaps. In Appendix A.10.2, Figure 7 shows that backward scoring identifies ground-truth antecedent regions in controlled state-transition scenarios, while Figure 9 visualizes the final selected tokens and shows that CueMiner preserves both pre-event visual evidence and key-event tokens, unlike SURGE. Furthermore, Figure 14 in Appendix A.10.5 considers the more challenging case where a segment boundary separates an antecedent action from its subsequent key event, showing that CueMiner still preserves evidence of both the button press and the resulting ghost appearance across the two segments. Together, these visualizations demonstrate that CueMiner preserves meaningful antecedent visual evidence across controlled, real-world, and cross-segment scenarios.

Thanks for the rebuttal and the new experiments. The CDPruner comparison and the token visualizations in Appendix A.10 address most of my concerns, so I have raised my score to 3.

Thank you for taking the time to review our work and engage in such a thoughtful discussion. We genuinely enjoyed the exchange and found it very meaningful. We’re grateful for your careful feedback and for the time you devoted to helping us improve the paper.

Reviewer_VMF7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CueMiner, a training-free visual token compression framework for video large language models that preserves both salient events and their preceding visual context. The method combines bidirectional temporal scoring, segment-wise information estimation, and distributional-shift-based segmentation to identify informative visual tokens while reducing redundancy. Experiments across four video LLMs and four video understanding benchmarks demonstrate strong accuracy retention under aggressive compression, achieving competitive inference efficiency while retaining as few as 5% of visual tokens.

Critical strengths
  1. Effective Context-Aware Token Compression

CueMiner introduces bidirectional scoring to preserve both salient events and their preceding visual context, providing a simple and meaningful improvement over conventional token compression methods.

  1. Strong Empirical Results

The method achieves strong accuracy retention with as few as 5% of visual tokens across four video LLMs and four benchmarks, demonstrating its effectiveness and generalizability.

Critical weaknesses
  1. Limited Justification for DPP-Based Selection

Although DPP-based selection is a key component of CueMiner, the ablation results (Tables 22–23) show that simpler Top-k selection achieves better accuracy and lower latency with longer videos. This raises questions about the necessity of DPP in the proposed framework.

  1. Limited End-to-End Inference Speedup

Despite substantial token compression, the actual inference speedup remains relatively modest due to the additional compression overhead. A more detailed efficiency analysis would help establish the practical advantages of CueMiner.

Questions for authors

N/A

1 comment

We thank the reviewer for the insightful question and the opportunity for a meaningful discussion.

Weakness 1. Thus, DPP-based selection is still effective, since it outperforms other baselines, including Top-k selection, when the input contains relatively few sampled frames (see Table 23 with 48-frame setting). However, DPP incurs additional selection latency, which grows as the number of input frames increases (see Table 24). To address this trade-off, we evaluate CueMiner with Top-k selection for large video frames and show that Top-k achieves higher accuracy than DPP-based selection (see Table 23) while also attaining the lowest latency among the compared baselines (see Table 24). These results provide a practical guideline: use DPP-based selection when the input contains fewer frames, and Top-k selection when processing large video frames. For an explanation of why DPP-based selection outperforms Top-k selection with fewer sampled frames, whereas Top-k selection outperforms DPP-based selection with more sampled frames, please refer to Appendix A.10.8.

Weakness 2. We acknowledge that the end-to-end inference speedup is modest because compression incurs additional overhead. However, the primary efficiency objective of CueMiner is memory reduction rather than maximizing inference speedup. As reported in Table 7, the estimated incremental GPU memory usage during LLM prefill is reduced by up to approximately 8.3× relative to the vanilla model. Moreover, Figure 6 shows that compression incurs only a small GPU memory overhead of 0.188, 0.376, 0.572, and 0.752 GiB at 32, 64, 96, and 128 sampled frames, respectively, relative to the memory allocated at the start of compression.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.