acceptodds
Under review as a conference paper at ICLR 2027

RISE: Recurrent Evidence Consolidation with Kernelized Selection for Efficient Long-Video Understanding

Abstract

Although recent advancements in video large language models (Video LLMs) have enabled remarkable vision-language reasoning, long-video understanding suffer from the scalability issue due to substantial overhead of large-scale visual tokens. Existing approaches typically identify informative tokens based on local density or static saliency, retaining redundant semantically similar tokens while failing to sufficiently cover global video semantics. Towards this end, we propose a novel approach named Recurrent Evidence Consolidation with Kernelized Selection (RISE) for efficient long-video understanding. The core of our RISE is to compress redundant visual tokens from the views of both temporal consistency and information utility for efficient reasoning. In particular, our RISE first calculates the attention-based consistency scores to quantify recurrent patterns among spatio-temporally neighboring tokens in a sliding-window manner. These scores would be utilized to guide the anchor-based aggregation of evidence-consistent tokens from neighboring frames. In addition, we measure the conditional evidence gain according to a determinantal point process (DPP) kernel and iteratively select the most informative candidates from the remaining tokens. In this way, we maximize the information utility of the selected tokens within the constraints of computational budgets. Extensive experiments on four video benchmarks demonstrate that RISE consistently outperforms state-of-the-art baselines across various video LLM backbones. The code is available at https://anonymous.4open.science/r/RISE_4_Video_Compression

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
token compression, large language model, video question answering, video understanding

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_qrZWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RISE, a method for compressing video visual tokens that requires no additional training and is agnostic to the user's query. The core process consists of two stages: first, it aggregates recurring visual evidence through local matching and quality-weighted fusion across adjacent frames; subsequently, it employs a low-rank log-det objective to select complementary visual representations within a specified token budget. The authors evaluate the method on LLaVA-OneVision and Qwen3-VL using four video understanding benchmarks and various compression ratios, while also examining the impact of increasing the number of input frames under a fixed token budget.

Critical strengths
  1. The problem addressed is of practical value, and the method’s operational requirements are favorable. Redundancy in video-based visual tokens is a key factor affecting inference efficiency for long videos; RISE requires no additional training and employs a compression process independent of specific queries, making it valuable for deployment and reuse.
  2. Experiments cover a range of models and compression budgets, yielding compelling results. The paper evaluates the method across two model architectures and four benchmarks. Notably, with Qwen3-VL, RISE achieves average scores 2.7 and 3.3 points higher than the strongest baselines in the respective settings at retention rates of 20% and 10%, demonstrating its practical effectiveness.
  3. The authors provide extensive supplementary experiments. The appendix includes component ablation studies, and repeat experiments, while the main text examines inputs of 160 and 320 frames under fixed token budgets. These experiments provide a basis for assessing the method's usability and its capability to handle long-sequence inputs.
Critical weaknesses

1.There is a lack of sufficient comparison with highly relevant recent methods, and the specific contributions of the proposed approach have not been clearly established. The primary experiments compare the method only against FastV, VisionZip, FastVID, and AOT, failing to adequately cover recent video token compression techniques directly related to RISE.

  • UniComp (CVPR 2026) preserves essential video information through frame fusion, token budget allocation, and compression based on information uniqueness—approaches highly relevant to the "integrating redundant evidence and preserving complementary information" strategy proposed in this paper—yet the manuscript lacks corresponding discussion and experimental comparisons.
  • FlashVID (ICLR 2026) combines attention- and diversity-driven token selection with tree-based spatiotemporal merging;
  • HoliTom (NeurIPS 2025) enhances inference efficiency through global redundancy-aware temporal segmentation, spatiotemporal merging, and internal LLM compression;
  • STTM (ICCV 2025) investigates multi-granularity spatiotemporal token merging. Although the paper cites these works, it does not include them in the experimental evaluation.

In particular, while the paper characterizes FlashVID as representative of the "select-then-merge" paradigm, it provides no direct comparison; consequently, it fails to sufficiently demonstrate the specific advantages of the RISE approach, which emphasizes "integrating redundant evidence before selecting complementary information."

2.Regarding the efficiency comparison in Table 6 using different token retention rates, the conclusions require precise qualification. Specifically, the 796.8-second runtime for RISE at a 10% retention rate is compared against the 1,103.5-second and 1,181.2-second runtimes for FastVID and VisionZip at a 20% retention rate.

Based on my understanding of the field, the prefill time should be same for the same retention rate, making such a comparison unnecessary; what truly matters is the time spent on preprocessing prior to the prefill stage.

3.There are inconsistencies regarding the numerical values ​​and statistical metrics in the efficiency analysis of Figure 4.

In the main text, when discussing Figure 4(d), the relative accuracy of FastVID on Qwen3-VL is stated as 79.3%; however, the result corresponding to a 20% retention rate in Table 2 is 92.6%, and the value shown in the figure is also approximately 92.6%. This error significantly exaggerates the performance gap described in the text and should be corrected.

Questions for authors

See weakness. If the mentioned related works are adequately compared, demonstrating the superiority of this method, I would consider raising the score.

13 comments
AuthorsOpenReview ↗

Response to Weakness 1 (More baseline comparison and specific analysis)

W1: Insufficient comparisons with recent methods (e.g., UniComp, FlashVID, HoliTom, and STTM) make RISE's contributions unclear. Additional baseline comparisons and analysis are needed to clarify the advantages of its integrate-then-select strategy.

A1: Thank you for your comment.

  • More baseline comparison: We compare our method with several relevant recent methods as shown in the following table. These results demonstrate that RISE has comparable performance over other methods, showing the effectiveness of RISE.
  • Specific analysis of contribution: As shown in Fig. 2(b) in our paper, we have explicitly compared merge-then-select and select-then-merge under identical token budgets. Our findings indicate that selecting tokens before consolidation may prematurely discard recurrent evidence, whereas merging first accumulates temporally consistent information before evaluating its complementary contribution. This controlled comparison isolates the effect of processing order rather than differences between compression frameworks. And as the results in Table 1 show that our method performs better than FlashVID with LLaVA-OV, especially under 10% retention ratio, illustrating the reliability of RISE.

Table 1: Comparison with recent video token compression methods on LLaVA-OneVision under different retention ratios.

MethodRetention RatioVideoMMEEgoSchemaLVBVideoMMMUAvg.Rel. Score (%)
Vanilla100%58.5060.4056.6032.1051.90100.00
FlashVID10%57.4859.9355.9429.3350.6797.66
HoliTom10%57.0060.8656.2529.1150.8197.89
FrameFusion10%51.6755.2650.7132.0047.4191.35
OTPrune10%55.7858.3856.2528.2249.6695.68
RISE10%57.5061.2055.2031.8051.4099.10
FlashVID20%58.0060.1956.5530.4451.3098.86
HoliTom20%58.9360.6455.8729.3351.1998.64
FrameFusion20%56.7858.6253.9331.4450.1996.71
OTPrune20%57.5959.2355.2729.2250.3396.97
RISE20%58.4061.0056.3032.7052.10100.40
AuthorsOpenReview ↗

Response to Weakness 2 (Comprehensive efficiency analysis)

W2: Regarding the efficiency comparison in Table 6 using different token retention rates, the conclusions require precise qualification. Specifically, the 796.8-second runtime for RISE at a 10% retention rate is compared against the 1,103.5-second and 1,181.2-second runtimes for FastVID and VisionZip at a 20% retention rate. Based on my understanding of the field, the prefill time should be same for the same retention rate, making such a comparison unnecessary; what truly matters is the time spent on preprocessing prior to the prefill stage.

A2: Thank you for your comment and your insightful question. We will explain it and offer the results under the same retention ratio.

  • Response on the fairness of prefilling time comparison. As you have noted, FlashVID also compares methods under different retention ratios in its Table 6. We adopt a similar setting because it is difficult to simultaneously balance accuracy and prefilling time in a perfectly controlled manner. For a fair comparison, we therefore report prefilling time when our method achieves comparable or even higher accuracy.
  • Response on LLM forward time. The point about LLM forward time would be most direct if the number of retained tokens were exactly the same across methods. However, in practice, the actual number of retained tokens may differ slightly due to rounding and implementation details. For example, in the open-source implementation of FlashVID cited by the reviewer, some retention ratios lead to a slightly higher number of retained tokens, as also discussed in the project’s issue tracker. Consequently, the reported prefilling time naturally varies across different compression methods.
  • Results of our method: The prefilling time comprises both compression time and LLM forward time. We report our results of efficiency below under the same retention ratio. The results in Table 2 demonstrate that Compared with other methods, RISE exhibits obvious end-to-end acceleration and high inference speed. Specifically, compared with FlashVID, RISE reduces GPU memory by 6.5%, TTFT by 15.8%, and model latency by 9.5%, with comparable prefilling time (293.73 ms vs. 288.29 ms). Compared with HoliTom, RISE further reduces TTFT by 45.8% and model latency by 31.3%, while maintaining similar prefilling time. These results demonstrate that RISE achieves superior end-to-end inference efficiency and memory efficiency despite its additional compression overhead.

Table 2: Efficiency of RISE on Qwen3-VL under 10% retention ratio

MethodGPU Memory (GiB)Compression Time (ms)LLM Forward (ms)Prefilling Time (ms)TTFT (ms)Model Latency (ms)
HoliTom18.9461.33232.68294.011829.122705.71
FlashVID19.5853.15235.14288.291177.812054.63
RISE18.3161.90231.83293.73991.431859.95
AuthorsOpenReview ↗

Response to Weakness 3 (Typo in our paper)

W3: In the main text, when discussing Figure 4(d), the relative accuracy of FastVID on Qwen3-VL is stated as 79.3%; however, the result corresponding to a 20% retention rate in Table 2 is 92.6%, and the value shown in the figure is also approximately 92.6%. This error significantly exaggerates the performance gap described in the text and should be corrected.

A3: Thank you for your pointing out the problem. This is a typo when we correct it to a right figure. We will rectify it in the manuscript.

Reviewer_cHtWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RISE, a training-free and task-agnostic method for compressing visual tokens in videos. The approach begins by recursively merging redundant visual tokens based on local feature similarity between adjacent frames and their corresponding attention weights. Subsequently, it employs a global greedy selection strategy—guided by a low-rank log-det objective—to balance token importance against feature diversity within a fixed computational budget. The method is evaluated across two large-scale video models and four benchmarks, with assessments covering both post-compression accuracy and computational efficiency.

Critical strengths
  1. The research addresses a problem of practical significance. The proposed compression method—which requires no additional training and is agnostic to the specific task—offers clear deployment value, with compressed representations of the same video remaining reusable across different applications.
  2. The experiments encompass two models, multiple datasets, and various token retention ratios. Notably, when evaluated on Qwen3-VL at identical token retention ratios, RISE outperforms FastVID by an average of 3.1 to 3.3 points, demonstrating a superior trade-off between accuracy and compression rate.
  3. The paper includes component ablation studies, hyperparameter analysis, and repeated experiments, which facilitate a deeper understanding of the method's behavior. On LLaVA-OneVision, the model retains 10% of its visual tokens while maintaining an average score nearly equivalent to that of the uncompressed model—a result with significant practical utility.
Critical weaknesses
  1. Appendix Table 6 reports prefilling times but does not separately itemize the time spent on computing attention scores, cross-frame matching and merging, low-dimensional projection, or greedy selection; nor does it clarify whether these operations are fully accounted for in the reported timings. Consequently, it is currently impossible to determine whether the reported improvements in execution time already factor in the full cost of compression. The selection complexity is stated in the appendix as O(NK)O(NK). Given a fixed retention ratio, the selection budget grows with the input scale, meaning the compression phase itself could potentially become a bottleneck when processing long videos.
  2. In Table 7, the variant retaining cross-frame merging and using only Top-K selection scored an average of 44.4, compared with 44.7 for the full method. In this ablation of only two datasets, the additional benefit from complex selectors is smaller. In Figure 2(a), its improvement relative to Top-K is also uneven, with one data set declining, so the generalization in the text that Top-K is significantly lagging behind is stronger.
  3. Theorem 4.1 provides bounds on the "weighted feature retention" (as defined by the authors) under the assumption of intra-group consistency; however, the connection between this metric and the accuracy of downstream question-answering tasks remains unestablished. The retention of weighted feature statistics does not inherently guarantee the preservation of task-relevant evidence, such as transient critical events, temporal ordering, or the frequency of specific actions. While the derivation of marginal gains in Theorem 4.2 is valid, it primarily demonstrates that the greedy steps increase the defined log-det objective, without establishing that they necessarily increase information relevant to the task.
Questions for authors

see Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Weakness 1 (Comprehensive efficiency analysis)

W1: The reported efficiency analysis lacks a detailed breakdown of compression time and LLM forward time. Providing these measurements would help clarify the computational overhead and further demonstrate the practical efficiency of RISE.

A1: Thank you for your comment. The prefilling time comprises both compression time and LLM forward time. We further provide a comprehensive efficiency comparison with Vanilla, FastVID, HoliTom, FlashVID, and UniComp under the same experimental settings. As shown in Table 1, RISE reduces TTFT by 63.6% compared with Vanilla, demonstrating substantial inference acceleration. Compared with FastVID, FlashVID, and HoliTom, RISE achieves comparable TTFT, with differences within 2%. Importantly, RISE achieves higher accuracy under the same retention ratio, outperforming FastVID by 3.3 percentage points on Qwen3-VL, while incurring only 1.9% additional TTFT. Furthermore, compared with UniComp, RISE reduces compression overhead by 90.1%, TTFT by 36.0%, and GPU memory by 30.5%. These results demonstrate that RISE achieves a favorable trade-off between accuracy and efficiency, preserving more informative visual tokens while maintaining competitive inference speed.

Table 1: Efficiency comparison on Qwen3-VL under a 10% visual token retention ratio (except Vanilla at 100%).

MethodGPU Memory (GiB)Vision Encoding (ms)Compression (ms)LLM Forward (ms)Prefilling (ms)TTFT (ms)Model Latency (ms)
Vanilla19.95697.700.002027.782027.782725.483569.63
FastVID18.31697.7023.56251.54275.10972.801888.82
HoliTom18.94697.7061.33232.68294.01991.711868.30
FlashVID19.58697.7053.15235.14288.29985.991862.81
UniComp26.36697.70628.02224.05852.071549.772448.23
RISE18.31697.7061.90231.83293.73991.431859.95
AuthorsOpenReview ↗

Response to Weakness 2 (More ablation analysis)

W2: In Table 7, the variant retaining cross-frame merging and using only Top-K selection scored an average of 44.4, compared with 44.7 for the full method. In this ablation of only two datasets, the additional benefit from complex selectors is smaller. In Figure 2(a), its improvement relative to Top-K is also uneven, with one data set declining, so the generalization in the text that Top-K is significantly lagging behind is stronger.

A2: We further conduct relevant experiments on Qwen3-VL, as shown in Table 1. We conduct relevant experiments on retention ratios of 10% and 20%, and the experimental results show that our method is far superior to the method that only uses the topk attention score, further proving our insight

Table 1: Comparison between Top-K and DPP-based selection on Qwen3-VL under different visual token retention ratios.

MethodRetention RatioVideoMMMULVBVideoMMEEgoSchemaAvg.Rel. Score (%)
Vanilla100%58.0058.4063.7068.2062.10100.0
Top-K10%39.6756.2559.6764.2454.9688.5
DPP (RISE)10%51.0055.9060.5065.8058.3093.9
Top-K20%43.4457.9762.3365.8157.3992.4
DPP (RISE)20%53.9058.9062.4067.2060.6097.6
AuthorsOpenReview ↗

Response to Weakness 3 (Theoretical analysis)

W3: Theorem 4.1 provides bounds on the "weighted feature retention" (as defined by the authors) under the assumption of intra-group consistency; however, the connection between this metric and the accuracy of downstream question-answering tasks remains unestablished. The retention of weighted feature statistics does not inherently guarantee the preservation of task-relevant evidence, such as transient critical events, temporal ordering, or the frequency of specific actions. While the derivation of marginal gains in Theorem 4.2 is valid, it primarily demonstrates that the greedy steps increase the defined log-det objective, without establishing that they necessarily increase information relevant to the task.

A3:We thank the reviewer for this insightful comment. We provide two further clarifications:

  • Theoretical Guarantee. Beyond the evidence retention bound established in Theorem 4.1, we further show that the conditional log-det objective in Theorem 4.2 is normalized, monotone, and submodular. Specifically, its marginal gain satisfies the diminishing-returns property, ensuring that our greedy selection achieves the following approximation guarantee: F(Rgreedy)≥(1−1/e)F(R⋆),F(R_{\mathrm{greedy}})\geq (1-1/e)F(R^\star), where R⋆R^\star denotes the globally optimal subset under the same token budget.
  • Task Performance and Generalizability. RISE is designed for question-agnostic video compression, allowing the same compressed visual representations to be reused across different questions about a given video without additional compression overhead. To further validate its effectiveness on downstream tasks, we provide additional comparisons with recent baselines, including OTPrune, FlashVID, and HoliTom, on LLaVA-OneVision under matched retention ratios of 10% and 20%. These complementary theoretical and empirical evaluations support the effectiveness of our framework.

Table 2: Comparison with recent video token compression methods on LLaVA-OneVision under different retention ratios.

MethodRetention RatioVideoMMEEgoSchemaLVBVideoMMMUAvg.Rel. Score (%)
Vanilla100%58.5060.4056.6032.1051.90100.00
FlashVID10%57.4859.9355.9429.3350.6797.66
HoliTom10%57.0060.8656.2529.1150.8197.89
FrameFusion10%51.6755.2650.7132.0047.4191.35
OTPrune10%55.7858.3856.2528.2249.6695.68
RISE10%57.5061.2055.2031.8051.4099.10
FlashVID20%58.0060.1956.5530.4451.3098.86
HoliTom20%58.9360.6455.8729.3351.1998.64
FrameFusion20%56.7858.6253.9331.4450.1996.71
OTPrune20%57.5959.2355.2729.2250.3396.97
RISE20%58.4061.0056.3032.7052.10100.40
Reviewer_B2wnOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RISE, a training-free and question-agnostic visual token compression method for video LLMs. RISE first merges similar tokens across adjacent frames, weighted by an attention-based evidence score. It then greedily selects a fixed-budget subset that maximizes a DPP-style log-determinant objective over quality-weighted, low-rank token features. Three empirical observations in Section 3 motivate this merge-then-select design. The method is evaluated on LLaVA-OneVision and Qwen3-VL across four long-video benchmarks, against FastV, VisionZip, FastVID, and AOT.

Critical strengths
  1. Plug-and-play, question-agnostic design that holds up on a recent backbone. RISE needs no training and no access to the query, so one compressed representation can serve every question about the same video.

  2. A meaningful fixed-budget setting. One experiment keeps the token budget fixed while increasing the number of frames. This directly tests the practical goal of long-video compression, which is fitting more temporal context into the same budget.

Critical weaknesses
  1. The efficiency claim is not substantiated. RISE’s compression step is itself costly. It extracts attention scores, matches and merges tokens across all adjacent frames, projects tokens onto a low-rank subspace whose construction is unspecified, and runs a K-step sequential greedy selection. None of this cost appears in the LLM-side FLOPs (Fig. 4(d), Sec. 5.4). The comparison also covers only Vanilla and FastVID. Without a wall-clock comparison against recent methods, especially those with fast greedy subset selection such as FLoC [1], RISE’s efficiency advantage is unclear. Efficiency is the paper’s central motivation, so this is the main reason for my recommendation.

  2. Limited technical depth and positioning. Both stages recombine known components: similarity-thresholded weighted averaging (Stage 1) and standard greedy log-det DPP maximization (Stage 2). Closely related principled subset-selection methods are neither discussed nor compared: FLoC [1] (facility location) and OTPrune [2] (essentially the same log-det objective as Eq. (6)). The Section 3 observations are design ablations rather than new insights. The “semantic evidence” score that drives both merging and anchor selection is raw attention, which can favor semantically uninformative sink tokens [3]. Finally, the baselines are outdated: FlashVID, HoliTom, and DivPrune are cited but not compared.

[1] FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understanding, Cho et al., ICLR'26.

[2] OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transport, Chen et al., CVPR'26.

[3] Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs, Kim et al., ECCV'26

Questions for authors
  1. Please report end-to-end wall-clock latency and peak memory, with RISE’s compression overhead (Stage 1 and Stage 2) broken out from LLM prefill. Use matched retention ratios and compare against recent methods such as FLoC, DivPrune/OTPrune, FlashVID, and HoliTom. Please also specify how the low-rank subspace is obtained and what it costs per video.

  2. How does RISE compare with FLoC and OTPrune in accuracy under the same backbone and budget? Beyond adding the attention-based quality weight, what does Stage 2 contribute over these existing subset-selection formulations?

6 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1 (Comprehensive efficiency analysis)

W1 && Q1: RISE's efficiency advantage remains unclear due to the unreported compression overhead and limited comparisons with recent methods. End-to-end latency, peak GPU memory, and details of low-rank subspace construction are needed to substantiate its efficiency claims.

A1: Thank you for your comment. Below, we provide a point-by-point response to each of your concerns.

  • Efficiency Comparision with recent methods We further provide a comprehensive efficiency comparison with Vanilla, FastVID, HoliTom, FlashVID, and UniComp under the same experimental settings. As shown in Table 1, RISE reduces TTFT by 63.6% compared with Vanilla, demonstrating substantial inference acceleration. Compared with FastVID, FlashVID, and HoliTom, RISE achieves comparable TTFT, with differences within 2%. Importantly, RISE achieves higher accuracy under the same retention ratio, outperforming FastVID by 3.3 percentage points on Qwen3-VL, while incurring only 1.9% additional TTFT. Furthermore, compared with UniComp, RISE reduces compression overhead by 90.1%, TTFT by 36.0%, and GPU memory by 30.5%. These results demonstrate that RISE achieves a favorable trade-off between accuracy and efficiency, preserving more informative visual tokens while maintaining competitive inference speed.

Table 1: Efficiency comparison on Qwen3-VL under a 10% visual token retention ratio (except Vanilla at 100%).

MethodGPU Memory (GiB)Vision Encoding (ms)Compression (ms)LLM Forward (ms)Prefilling (ms)TTFT (ms)Model Latency (ms)
Vanilla19.95697.700.002027.782027.782725.483569.63
FastVID18.31697.7023.56251.54275.10972.801888.82
HoliTom18.94697.7061.33232.68294.01991.711868.30
FlashVID19.58697.7053.15235.14288.29985.991862.81
UniComp26.36697.70628.02224.05852.071549.772448.23
RISE18.31697.7061.90231.83293.73991.431859.95
  • Detailed Implementation: Our implementation follows Eq. (4) and Appendix C.2, Eq. (27). We stack the row-normalized visual features surviving Stage 1 into

XS∈RC×d,X_{\mathcal S}\in\mathbb{R}^{C\times d},

where C is the number of surviving candidates and d is the feature dimension. We construct the orthonormal basis using an uncentered randomized truncated SVD:

\qquad \Phi_r=V_r, \qquad r=32.$$ We then follow Eq. (27) and Eq. (4): $$z_i=\frac{\Phi_r^\top\bar{x}_i} {|\Phi_r^\top\bar{x}_i|_2}, \qquad b_i=\sqrt{q_i},z_i.$$ Thus, quality weighting is applied after projection and does not enter the construction of the subspace basis. The basis is constructed online once per video inference request after Stage 1 and reused throughout all K greedy selection steps. With fixed iteration settings, the costs of basis construction and feature projection are, respectively, $$T_{\mathrm{basis}}=O\left(Cdr+(C+d)r^2\right), \qquad T_{\mathrm{projection}}=O(Cdr).$$ On Qwen3-VL-8B with 32 frames and an RTX A6000, basis construction takes 4.54 ms and feature projection takes 0.36 ms on average. The complete low-rank representation construction, including normalization and quality scaling, takes 5.65 ms per request. All these costs are included in the reported compression time.
AuthorsOpenReview ↗

Response to Weakness 2 and Question 2 (More baseline comparison) (Part 1)

W2 && Q2: Limited technical depth and positioning. Both stages recombine known components: similarity-thresholded weighted averaging (Stage 1) and standard greedy log-det DPP maximization (Stage 2). Closely related principled subset-selection methods are neither discussed nor compared: FLoC [1] (facility location) and OTPrune [2] (essentially the same log-det objective as Eq. (6)). The Section 3 observations are design ablations rather than new insights. The “semantic evidence” score that drives both merging and anchor selection is raw attention, which can favor semantically uninformative sink tokens [3]. Finally, the baselines are outdated: FlashVID, HoliTom, and DivPrune are cited but not compared. How does RISE compare with FLoC and OTPrune in accuracy under the same backbone and budget? Beyond adding the attention-based quality weight, what does Stage 2 contribute over these existing subset-selection formulations? [1] FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understanding, Cho et al., ICLR'26. [2] OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transport, Chen et al., CVPR'26. [3] Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs, Kim et al., ECCV'26

A2: We thank the reviewer for highlighting the need to clarify our contributions.

  • Contributions of RISE: While weighted merging and log-det optimization are established techniques, RISE's contribution lies in a unified formulation that couples temporal evidence consolidation with global conditional selection, rather than either operation in isolation. We clarify three distinctions:
    • Different Motivations. FLoC approaches token compression from the perspective of representative coverage, while OTPrune focuses on distribution preservation through optimal transport. In contrast, our observations in Sec. 3 reveal that (i) token significance depends on its conditional evidence gain, (ii) recurrent evidence across neighboring video frames should be consolidated before selection, and (iii) segment-wise selection may fragment globally related semantic evidence. We mainly focus on the spatio-temporal scale. These findings motivate a different perspective on video compression.
    • Different Methodology. OTPrune primarily addresses visual token pruning through distribution alignment, whereas RISE explicitly models temporal recurrence in videos. Rather than directly selecting from original tokens, RISE first consolidates consistent cross-frame evidence through local matching and evidence-weighted aggregation, jointly updating token representations and their semantic evidence scores. These accumulated scores subsequently guide global conditional selection. The log-det objective serves as an established mathematical tool for measuring complementary information gain, rather than the source of our methodological contribution.
    • Unified Framework. Instead, RISE integrates recurrent evidence consolidation and global conditional selection into a unified, training-free framework. Local aggregation concentrates repeated evidence, while global selection preserves complementary semantics without explicit temporal segmentation. The controlled studies in Sec. 3 validate these design principles. We will further clarify these distinctions and extend comparisons with recent methods, including FLoC, OTPrune, FlashVID, HoliTom, and DivPrune.
AuthorsOpenReview ↗

Response to Weakness 2 and Question 2 (More baseline comparison) (Part 2)

A2: We offer the folllowing analysis.

  • Contributions of Stage 2: Beyond attention-based weighting, Stage 2 contributes in two important aspects. First, evidence-aware selection: Stage 2 operates on temporally consolidated anchors, whose representations and evidence scores have been jointly updated through cross-frame aggregation. Therefore, the selection process considers accumulated temporal evidence rather than isolated token importance. Second, global evidence allocation: Stage 2 evaluates the marginal contribution of each anchor conditioned on previously selected evidence across the entire video, enabling complementary information to be preserved without segment-wise budget constraints. The log-det objective serves as a principled tool for this purpose, while our contribution lies in its integration with temporal evidence consolidation. As validated in Fig. 2(a) and Appendix E.3, replacing conditional selection with score-based Top-K reduces performance even when Stage 1 is retained. Beyond that, we implement a FLoC-inspired similarity-scoring variant within the RISE pipeline and compared it with our attention-based scoring, as shown in Table 2. The variant replaces the importance signal while retaining the temporal matching, merge groups, merged features, and subsequent DPP selection procedure. The results show that our method in the Stage 2 is more likely to compress tokens in a better manner.

Table 2: Comparison with recent video token compression methods on Qwen3-VL under different retention ratios.

MethodRetention RatioVideoMMEEgoSchema TotalLVBVideoMMMU TotalAvg.Rel. Score (%)
Vanilla100%63.7068.2058.4058.0062.10100.00
FLoC†10%53.5261.0851.6129.2248.8678.68
RISE10%60.5065.8055.9051.0058.3093.88
FLoC†20%56.5664.5854.3833.8952.3584.30
RISE20%62.4067.2058.9053.9060.6097.58
  • Problems on uninformative sink tokens: First, as demonstrated in Section 3.3 Sink Tokens as an Obstacle to Fine-Grained Understanding in [1], temporal pruning can implicitly suppress the repeated retention of attention sink tokens. This finding is consistent with our recurrent evidence consolidation, which merges temporally similar tokens to reduce redundant evidence. These results suggest that our approach partially mitigates the concern raised by the reviewer. Second, the use of [CLS] attention for estimating token importance follows established practices in recent methods, including FlashVID, FastVID, and VisionZip. Furthermore, our additional ablation shows that replacing attention-based importance scores with similarity-based scores leads to performance degradation, providing empirical support for the effectiveness of attention-based evidence estimation. Nevertheless, we acknowledge that attention sinks remain an important issue deserving further investigation, and we will discuss potential sink-aware extensions in the Future Work section of the revised manuscript.

  • More experiments on different baselines: Since FLOC has not been open-sourced to date, we are unable to directly reproduce its results. Instead, we conduct a comparison with OTPrune as a representative alternative, and additionally include comparisons with other recent baselines. All experiments are performed on LLaVA-OV under retention ratios of 10% and 20%. As shown in the table, our method achieves the best overall performance, further validating the rationality and effectiveness of our approach. [1] Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs, Kim et al., ECCV'26

Table 3: Comparison with recent video token compression methods on LLaVA-OneVision under different retention ratios.

MethodRetention RatioVideoMMEEgoSchemaLVBVideoMMMUAvg.Rel. Score (%)
Vanilla100%58.5060.4056.6032.1051.90100.00
FlashVID10%57.4859.9355.9429.3350.6797.66
HoliTom10%57.0060.8656.2529.1150.8197.89
FrameFusion10%51.6755.2650.7132.0047.4191.35
OTPrune10%55.7858.3856.2528.2249.6695.68
RISE10%57.5061.2055.2031.8051.4099.10
FlashVID20%58.0060.1956.5530.4451.3098.86
HoliTom20%58.9360.6455.8729.3351.1998.64
FrameFusion20%56.7858.6253.9331.4450.1996.71
OTPrune20%57.5959.2355.2729.2250.3396.97
RISE20%58.4061.0056.3032.7052.10100.40
Reviewer_Q7SmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RISE, a training-free visual token compression method for long-video understanding. Selecting tokens by individual importance scores can retain redundant information while missing complementary evidence across the video. To address this, RISE first merges similar tokens across adjacent frames using attention-weighted aggregation, then selects a fixed-budget subset through an objective that considers both token importance and its marginal contribution to the selected set. Experiments on four benchmarks and two video LLMs show promising performance under aggressive compression. The overall design is intuitive and practically useful, but acceptance would require clearer evidence for the contribution of conditional selection and more consistent experimental reporting.

Critical strengths
  1. The proposed idea is well-motivated and intuitively clear. Consolidating repeated information before selection can prevent redundant content from occupying the limited token budget, while conditional selection encourages complementary information. The method requires no additional training and allows the same compressed representation to be reused across different questions, which makes it practically useful.

  2. The experimental results are promising across different architectures and compression settings. RISE preserves much of the original performance under aggressive compression and achieves better average results than the compared baselines. The fixed-budget experiment is also useful, as it shows that compression can help the model benefit from additional frames.

Critical weaknesses
  1. The additional contribution beyond temporal consolidation is not sufficiently established. The Introduction describes FrameFusion as an existing merge-then-select method, yet it is absent from the main comparisons. A comparison with this closely related approach would help clarify what RISE adds to the existing pipeline. Moreover, the component-wise ablation in Table 7 shows that replacing the proposed selector with top-k causes only a small performance drop, and this comparison is limited to a subset of the benchmarks on LLaVA-OneVision. A broader evaluation of consolidation followed by top-k is recommended to make the contribution of conditional selection clear.

  2. The rationale for treating temporal recurrence as stronger semantic evidence is not sufficiently validated. RISE accumulates importance when similar tokens recur across frames, which may favor persistent background content over brief but task-critical events. The aggregate results and successful qualitative examples provide limited insight into this potential bias. An analysis of failures involving short-lived events, together with an ablation that keeps temporal merging fixed while varying score accumulation, would help clarify when recurrence is a useful proxy for semantic importance.

Questions for authors
  1. Can the authors provide end-to-end latency and peak memory as the input video length increases, including the overhead of token compression? The reported prefilling time does not make clear whether temporal merging, low-rank projection, and greedy selection are included. I wonder whether the practical speedup persists when both the input and retained token counts grow.
8 comments
AuthorsOpenReview ↗

Response to Weakness 1 (More ablation studies and extensive experiments)

W1: RISE's contribution beyond temporal consolidation remains unclear. Additional comparisons with FrameFusion and broader evaluations of DPP versus top-k selection would help clarify the effectiveness and added value of the proposed approach.

A1: Thank you for your comment. We explain the relevant questions from the following two perspectives.

  • More ablation studies. We further compare our DPP-based selection with a top-k selection based on CLS attention on Qwen3-VL, under retention ratios of 10% and 20%. As shown in Table 1, directly selecting tokens via top-k CLS attention tends to retain redundant and highly similar tokens. In contrast, our DPP-based selection preserves a more diverse and informative token set, leading to better overall performance.

Table 1: Comparison between Top-K and DPP-based selection on Qwen3-VL under different visual token retention ratios.

MethodRetention RatioVideoMMMULVBVideoMMEEgoSchemaAvg.Rel. Score (%)
Vanilla100%58.0058.4063.7068.2062.10100.0
Top-K10%39.6756.2559.6764.2454.9688.5
DPP (RISE)10%51.0055.9060.5065.8058.3093.9
Top-K20%43.4457.9762.3365.8157.3992.4
DPP (RISE)20%53.9058.9062.4067.2060.6097.6
  • Extensive experiments. To further illustrate and validate our insight, we have added comparisons with existing merge-then-select methods, taking FrameFusion as a representative example. To further demonstrate the effectiveness of our approach, we additionally include FlashVID and HoliTom for comparison, where FlashVID also adopts a merge-then-select strategy. All experiments are conducted on LLaVA-OV under retention ratios of 10% and 20%. As shown in Table 2, our method consistently outperforms the baselines in terms of average performance, with significant improvements at the 10% retention ratio, thereby confirming the rationality and effectiveness of our approach.

Table 2: Comparison with recent video token compression methods on LLaVA-OneVision under different retention ratios.

MethodRetention RatioVideoMMEEgoSchemaLVBVideoMMMUAvg.Rel. Score (%)
Vanilla100%58.5060.4056.6032.1051.90100.00
FlashVID10%57.4859.9355.9429.3350.6797.66
HoliTom10%57.0060.8656.2529.1150.8197.89
FrameFusion10%51.6755.2650.7132.0047.4191.35
OTPrune10%55.7858.3856.2528.2249.6695.68
RISE10%57.5061.2055.2031.8051.4099.10
FlashVID20%58.0060.1956.5530.4451.3098.86
HoliTom20%58.9360.6455.8729.3351.1998.64
FrameFusion20%56.7858.6253.9331.4450.1996.71
OTPrune20%57.5959.2355.2729.2250.3396.97
RISE20%58.4061.0056.3032.7052.10100.40
AuthorsOpenReview ↗

Response to Weakness 2 (Comprehensive case analysis and ablation study)

W2: Is temporal importance accumulation a reliable measure of semantic evidence, or does it favor persistent content over important but less salient information? Can controlled ablations and further analysis justify this design?

A2: Thank you for your question. We provide further analysis on the importance accumulation in Table 3. The experiments are conducted on LLaVA-OV under 10% retention ratio on VideoMME. With temporal matching and merge groups fixed, RMS replaces score accumulation with qC=∑i∈Cqi2/∣C∣q_C=\sqrt{\sum_{i\in C}q_i^2/|C|}, Max uses qC=max⁡i∈Cqiq_C=\max_{i\in C}q_i, and Uniform replaces attention-weighted feature merging with an arithmetic mean while retaining the original score accumulation. Our results show that score accumulation achieves the best overall performance among the compared variants. We further provide two question-driven analyses to clarify this finding.

First, we believe that the suboptimal performance of top-k selection stems from its tendency to concentrate on the highest-scoring tokens in CLS attention, causing the model to repeatedly focus on a narrow or redundant subset of tokens and to rely heavily on model priors. In contrast, our merging-based approach fuses similar tokens from the previous frame with those in the current frame. As stated in Line 258, the tokens from the previous frame are removed after merging, which helps mitigate the issue of persistent focus on the same content across frames.

Second, building on the above, we note that top-k selection tends to prioritize events that the model deems most salient. However, our method is a query-agnostic, general-purpose token compression approach: the model does not have access to the question in advance. Relying solely on CLS attention may therefore cause the model to attend to a narrow set of tokens, and when the question concerns regions with lower attention scores, such as the background content mentioned in the question, the model may lack sufficient evidence to answer correctly.

Table 3: Ablation study of evidence accumulation on VideoMME.

MethodScore AccumulationFeature MergingAccuracy (%)
RMSqC=1∣C∣∑i∈Cqi2q_C=\sqrt{\frac{1}{\lvert C\rvert}\sum_{i\in C}q_i^2}Attention-weighted56.00
MaxqC=max⁡i∈Cqiq_C=\max_{i\in C}q_iAttention-weighted56.15
UniformqC=∑i∈Cqi2q_C=\sqrt{\sum_{i\in C}q_i^2}Uniform mean55.89
Ours (RISE)qC=∑i∈Cqi2q_C=\sqrt{\sum_{i\in C}q_i^2}Attention-weighted57.50
AuthorsOpenReview ↗

Response to Question 1 (Comprehensive efficiency analysis)

Q1: Can the authors provide end-to-end latency and peak memory as the input video length increases, including the overhead of token compression? The reported prefilling time does not make clear whether temporal merging, low-rank projection, and greedy selection are included. I wonder whether the practical speedup persists when both the input and retained token counts grow.

A3: The prefilling time comprises both compression time and LLM forward time. We report our results of efficiency below. The results in Table 4 demonstrate that Compared with other methods, RISE exhibits obvious end-to-end acceleration and high inference speed. Specifically, compared with FlashVID, RISE reduces GPU memory by 6.5%, TTFT by 15.8%, and model latency by 9.5%, with comparable prefilling time (293.73 ms vs. 288.29 ms). Compared with HoliTom, RISE further reduces TTFT by 45.8% and model latency by 31.3%, while maintaining similar prefilling time. These results demonstrate that RISE achieves superior end-to-end inference efficiency and memory efficiency despite its additional compression overhead.

Table 4: Efficiency of RISE on Qwen3-VL under 10% retention ratio

MethodGPU Memory (GiB)Compression Time (ms)LLM Forward (ms)Prefilling Time (ms)TTFT (ms)Model Latency (ms)
HoliTom18.9461.33232.68294.011829.122705.71
FlashVID19.5853.15235.14288.291177.812054.63
RISE18.3161.90231.83293.73991.431859.95

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal

Dear Area Chair and Reviewers, We sincerely thank the Area Chair and all reviewers for their careful evaluation, insightful feedback, and constructive discussions. We greatly appreciate the reviewers' recognition of our motivation, practical value, and experimental results. Throughout the rebuttal, we have provided detailed point-by-point responses, additional experiments, and methodological clarifications to address the raised concerns. We are encouraged that the reviewers have recognized several strengths of RISE:

  • RISE addresses an important and practical problem, offering a training-free and question-agnostic framework that enables compressed video representations to be reused across different questions (Reviewers B2wn, cHtW, qrZW).
  • RISE is well motivated and intuitively designed, effectively consolidating recurrent visual information and preserving complementary semantics under limited token budgets (Reviewer Q7Sm).
  • RISE achieves strong performance against recent baselines, consistently obtaining the highest average scores under matched retention ratios compared with UniComp, FlashVID, HoliTom, FrameFusion, and OTPrune across multiple video understanding benchmarks (Reviewers B2wn, cHtW, qrZW).
  • RISE demonstrates competitive computational efficiency, substantially reducing inference latency compared with uncompressed models while maintaining comparable end-to-end efficiency to recent compression methods with compression overhead fully accounted for (Reviewers B2wn, cHtW).

During the rebuttal, in response to the reviewers' constructive suggestions, we have substantially strengthened the evaluation and technical clarification of RISE. The summary is listed as follows:

  • Comprehensive baseline comparisons (Reviewers Q7Sm, B2wn, cHtW, qrZW): We add comparisons with recent methods, including UniComp, FlashVID, HoliTom, FrameFusion, and OTPrune, as well as a FLoC-inspired scoring variant. Extensive evaluations on LLaVA-OneVision and Qwen3-VL under retention ratios of 10% and 20% further demonstrate the effectiveness and generalizability of RISE.
  • Detailed efficiency analysis (Reviewers Q7Sm, B2wn, cHtW, qrZW): We provide a detailed breakdown of compression overhead, LLM forward time, TTFT, model latency, and GPU memory. RISE reduces TTFT by 63.6% compared with uncompressed inference while maintaining competitive efficiency against recent compression methods.
  • Extended ablations and temporal evaluation (Reviewers Q7Sm, B2wn, cHtW): We expand the DPP vs. Top-K comparison across four benchmarks and two retention ratios, conduct controlled ablations on temporal importance accumulation and feature aggregation, and introduce additional evaluation on TUNA-MCQ across two model backbones. These results further substantiate the contributions of temporal consolidation and global conditional selection.
  • Methodological and theoretical clarifications (Reviewers B2wn, cHtW, qrZW): We clarify the technical distinction between RISE and related approaches, particularly how local evidence accumulation and global redundancy suppression address coupled challenges. We also supplement the theoretical analysis with a greedy approximation guarantee for the submodular selection objective and corrected numerical inconsistencies in the manuscript.

We sincerely appreciate the reviewers' continued engagement and constructive suggestions, which have helped us improve the manuscript. We welcome any further discussion and hope that these clarifications will assist the Area Chair in the final assessment.

Best regards,

The authors of the RISE submission

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.