- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RESET, a training-free framework for visual token compression in VLMs. The paper argues that conventional token selection methods primarily estimate the importance of individual tokens without explicitly considering whether candidate tokens provide complementary information beyond the tokens already selected. To address this issue, RESET formulates visual token compression as a residual subspace selection problem and develops a unified framework for image and video understanding. For image compression, RESET iteratively selects tokens with the largest residual energy using a QRCP-based procedure, encouraging the retained tokens to capture complementary feature directions. For video compression, it constructs a low-rank historical reference subspace to remove information already explained by previous visual content before selecting tokens from the residual representation. The framework also incorporates grouped parallel execution to reduce the computational overhead of sequential token selection. Experiments across multiple VLM architectures and image and video understanding benchmarks show that RESET achieves competitive performance. Furthermore, since the method does not require access to attention information, it can be integrated with methods such as FlashAttention.
Critical strengths
1. A novel perspective on token importance and a unified framework for image and video compression. RESET takes a different perspective on visual token importance by evaluating a token based on its residual energy after accounting for the information already captured by previously selected tokens, rather than assessing its importance in isolation. This allows the selection process to favor tokens that contribute complementary feature directions, providing a meaningful perspective on reducing redundancy in visual token compression. The paper further extends this principle to video by introducing a low-rank historical reference subspace to account for information already represented in previous frames. While the selection procedure builds on established techniques such as QRCP, the contribution lies in applying residual based selection as a common principle for image and video compression. This provides a coherent framework for addressing both spatial and temporal redundancy.
2. Evaluation across image and video benchmarks. The paper evaluates RESET on multiple VLM architectures and image and video understanding benchmarks. It achieves the highest average performance retention among the compared methods at three token budgets on LLaVA-1.5-7B, and outperforms FluxMem on four of five video benchmarks. Ablation studies examine the effects of historical reference and group size, while latency measurements evaluate the efficiency of the method. Although the results are not uniformly better across tasks, the experiments provide empirical support for RESET's effectiveness under the evaluated settings.
Critical weaknesses
1. The methodological novelty of the token selection strategy remains insufficiently isolated. RESET builds on classical QRCP-based residual selection, while prior work such as ApET also adopts a reconstruction-based perspective. The paper does provide a matched comparison between ApET and RESET under the same image-compression pipeline and token budgets, showing consistent improvements in average performance retention. However, the experiments do not isolate the effects of the key design choices in RESET, particularly the use of residual-energy-based pivot selection and the iterative residual updates that account for previously selected directions.
2. The computational trade-offs and cross-method comparability are not fully characterized. Although RESET and FluxMem are both evaluated at 1 fps, the broader comparisons in Table 2 involve heterogeneous frame-sampling configurations, making it difficult to attribute performance differences solely to the compression methods. In addition, while the paper reports compression and end-to-end latency under different group sizes and context lengths, it does not provide a component-wise breakdown of the compression cost, particularly the overhead attributable to iterative QRCP selection versus historical subspace construction and updates. Such profiling would help clarify the practical efficiency benefits of RESET, especially under mild compression, where the paper acknowledges that residual-selection overhead becomes more significant.
Questions for authors
1. Contribution of the QRCP-based selection strategy. Table 1 already compares RESET with the original ApET selector under the same image-compression pipeline and token budgets. Could the authors provide additional ablations that isolate the contributions of iterative residual updates and QRCP-based pivot selection, for example by comparing against alternative residual-based selection criteria under the same pipeline? This would help clarify which design choices account for the observed performance gains and what methodological advantage QRCP provides beyond existing reconstruction-based selection approaches.
2. Fairness of video comparisons and the computational cost of selection. Table 2 reports different or unspecified frame-sampling settings for several video baselines, which limits the comparability of cross-method results. Could the authors provide matched-setting comparisons where feasible, or clarify which baseline results were obtained under the same input-frame configurations as RESET? In addition, could the authors specify the operations included in the reported compression latency and provide a breakdown of the costs of QRCP selection, historical subspace construction and updates? This would help quantify the overhead attributable to residual selection and clarify the practical efficiency benefits of RESET under different compression ratios.
3. Preservation of fine-grained spatial information after compression. Although the paper reports aggregate MME and GQA results and video spatial-understanding subtasks, the impact of aggressive token compression on localization and grounding remains unclear. Could the authors provide spatial-reasoning subtask results or evaluate RESET on a localization or grounding benchmark under different token budgets? Additionally, are the original positional indices of retained tokens preserved after pruning, or reassigned according to the compressed sequence, particularly for architectures using M-RoPE? Clarifying these points would help assess the preservation of fine-grained spatial information.
6 comments
Q3: Grounding Performance and Positional Preservation
Spatial Understanding results are already reported in Table 7. To further assess localization, we evaluate RESET and ApET on RefCOCO TestB using Qwen2.5-VL-7B, with matched inputs and generation settings. We report Accuracy@IoU ≥ 0.5.
| Token retention | RESET | ApET |
|---|---|---|
| 20% | 41.02 | 22.87 |
| 10% | 16.11 | 5.14 |
| 5% | 6.48 | 1.02 |
The uncompressed baseline scores 86.34. RESET consistently outperforms ApET, but both methods exhibit substantial degradation relative to the full input. These results reveal an important limitation of training-free token compression for fine-grained localization under aggressive budgets, which we will acknowledge in the revision.
Regarding positional information, RESET preserves the original spatial and temporal indices of retained tokens, including their M-RoPE coordinates, instead of reassigning positions based on the compressed sequence. Merged features inherit their retained anchor's coordinates. This maintains positional consistency for the remaining representations, although information from removed locations may still be lost.
Thank you for the detailed responses and additional experiments. I appreciate the efforts in addressing my previous concerns. Regarding the RefCOCO evaluation, could the authors additionally report grounding performance at 70% and 50% token retention? Furthermore, could the authors provide performance-versus-pruning-ratio and speedup-versus-pruning-ratio curves by combining these additional results with the existing results (20%, 10%, and 5% retention)? The speedup should account for the overhead introduced by RESET. These results would help better understand the trade-off between localization performance and compression efficiency, and clarify the practical applicability of the proposed method.
We thank the reviewer for highlighting the need to isolate the selection mechanism, improve the comparability of video results, and examine fine-grained spatial information. We provide additional selector ablations, matched video comparisons, component-level profiling, and grounding results below.
W1 & Q1: Contributions of Iterative Selection and Pivot Scoring
QRCP is a classical algorithm. RESET's contribution lies in using iterative residual selection with reference conditioning for image and causal video compression. Unlike a fixed reference used for scoring, RESET updates the remaining residuals after each selected token, accounting for directions already retained. In video, a compact historical reference additionally suppresses redundant information across frames within causal groups. To isolate these design choices, we compare three selectors within the same image-compression pipeline on MME using LLaVA-1.5-7B. Static Residual Top-K ranks tokens once without residual updates. Iterative Normalized Residual updates residuals but normalizes pivot scores by original feature energy. RESET uses iterative updates with absolute residual energy.
The large gap over Static Top-K highlights the value of iterative residual updates. Both absolute and normalized residual scoring identify complementary information, with comparable results overall. Neither feature magnitude nor normalized novelty is guaranteed to align with downstream relevance. We therefore adopt the simpler absolute residual criterion, which also performs best at the tightest budget.
W2 & Q2: Matched Video Comparisons and Compression Cost
Due to the limited rebuttal time, we focus our additional matched comparisons on the Real-Time tasks of OVO-Bench. We rerun CausalMem and StreamingTOM with the same Qwen2.5-VL-7B model, 1 fps video sampling, dynamic resizing, prompts, and generation settings as RESET. Their hyperparameters are adjusted to achieve similar overall visual-token retention. Actual retention rates are reported below.
RESET obtains the highest average accuracy in all three matched settings.
We further profile compression on a single NVIDIA A800 using three OVO-Bench clips sampled at 512, 1024, and 2048 frames. We report the costs of historical projection, iterative QRCP selection, historical subspace construction and updates, token merging, and other operations. The component-level results are summarized below.
All latency values are reported in milliseconds (ms). Other Ops. includes selection preparation, postprocessing, and remaining compression operations. The results show that all computation overhead increase with the token budget and video length. RESET achieves consistent E2E acceleration across all evaluated settings, with greater speedups on longer videos.