- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LRCP (Low-Rank Compressibility Guided Pruning), a training-free visual token compression method for large vision-language models. The method is motivated by two empirical observations: visual token representations exhibit substantial low-rank structure, and their dominant subspaces remain relatively stable after token removal. LRCP estimates a dominant subspace using PCA, ranks tokens by their squared projection residuals, and retains tokens that are poorly represented by that subspace. Discarded tokens are merged into their most similar retained neighbors. Compression is applied at the vision encoder output and an intermediate LLM layer. Experiments cover three image-based LVLMs and Video-LLaVA, with comparisons across multiple compression budgets. The reported results indicate a favorable accuracy–efficiency trade-off. However, the connection between the surrogate analysis and the implemented compression procedure, the isolation of scoring benefits from pruning schedules, and the task-relevance validation require further scrutiny.
Critical strengths
(1) LRCP addresses the substantial inference cost of visual tokens through a compact, training-free procedure. Its reliance on feature representations rather than attention matrices makes it compatible with implementations that avoid materializing attention weights.
(2) Rather than selecting tokens that best represent the dominant subspace, LRCP prioritizes deviations from it. This is an intuitive alternative when globally common features are redundant and localized details carry useful information.
(3) The paper examines effective rank and subspace stability across LLaVA-v1.5, LLaVA-NeXT, and Qwen2.5-VL. The supplementary analysis also considers stability under LRCP’s actual selection rule, addressing a relevant limitation of random-dropout analysis alone.
(4) Experiments include general visual question answering, text-related understanding, hallucination evaluation, and video question answering. LRCP achieves the highest reported average performance retention across the main image settings, including aggressive compression budgets.
(5) The paper compares alternative scoring rules, subspace construction strategies, merging configurations, and compression layers. It also states that measured latency includes preprocessing, PCA, selection, and merging overhead, and explains the distinction between average and final token budgets in the appendix.
Critical weaknesses
(1) Section 3.3 defines
which preserves the complete low-rank component for every original token while discarding only selected residual components. The implemented method instead outputs merged token vectors; it does not explicitly retain the full low-rank component, its per-token coefficients, or the original spatial arrangement. Stability of the estimated subspace does not establish preservation of these quantities. Two token sets can span similar subspaces while encoding different spatial configurations or semantic content. Consequently, the residual-ranking derivation is correct for the stipulated surrogate, but its connection to the actual compressed representation remains insufficiently justified. The analysis also does not incorporate the merging operation used in the final method.
(2) LRCP retains more tokens in early layers and fewer in later layers. For example, the reported average-64 configuration uses 96 tokens in the first half of the LLM and 32 in the second half. Baselines retain their native schedules and are matched by average token count. This is a useful comparison convention, but equal token-layer totals do not imply equivalent information availability, attention costs, or opportunities for multimodal integration. Although Appendix A.4 discusses computational differences, the accuracy contribution of the schedule remains unresolved. Applying competing scoring methods under LRCP’s exact schedule, and testing LRCP under a single-stage schedule, would better isolate the proposed criterion.
(3) Table 7 reports high overlap between selected patches and annotated informative regions. However, the manuscript does not sufficiently specify the overlap denominator, annotation procedure, region coverage, selection of examples, or treatment of tokens after intermediate-layer contextualization and merging. Counting any patch–region intersection can produce high overlap when informative regions occupy a substantial fraction of the image. Comparisons against random selection, spatially uniform selection, and competing methods under identical budgets are therefore essential. Without these controls, the reported percentages do not establish that high residuals preferentially identify task-relevant evidence rather than visually distinctive regions.
(4) The principal improvement over the strongest baseline on LLaVA-v1.5 at the average-64 setting is approximately 1.1 % in average performance retention. Some individual benchmarks favor competing methods. The paper provides no uncertainty estimates, and the latency margins over ApET in Table 8 are small enough to warrant repeated measurements. The manuscript discusses reconstruction-based methods such as ResPrune, but does not provide a corresponding experimental comparison. SVD-Prune is evaluated only in a limited supplementary setting. Stronger comparisons with the closest representation-based methods, matched schedules, and repeated timing measurements would make the contribution more convincing.
(5) Algorithm 1 mean-centers and normalizes , then merges and returns features derived from that transformed matrix. It does not clarify whether scoring uses transformed features while merging operates on original features, or whether the transformation is reversed before subsequent model processing. This distinction matters for a training-free intervention. The Qwen resolution configuration is also ambiguous: Section 4.1 describes minimum and maximum “pixels” of 256 and 2048, while Table 3 describes that range as visual token counts. Finally, the video evaluation uses eight sampled frames and three benchmarks, which supports transfer to sampled-video QA but provides limited evidence for the broader motivation concerning long videos and complex temporal understanding.
Questions for authors
(1) How does the surrogate representation correspond to the actual compressed token set? Can the authors quantify preservation of low-rank coefficients or low-rank reconstruction error after selection and merging, rather than measuring subspace similarity alone?
(2) How much of the improvement comes from the two-stage schedule? Please compare LRCP, random selection, ApET, and other competitive criteria under exactly the same pruning locations and per-stage token budgets.
(3) Which features are passed to the downstream model? Are centering and normalization used only for PCA and scoring? Are discarded tokens merged in the original feature space, and how are positional indices assigned to merged tokens?
(4) How is Table 7 computed? Please define the numerator and denominator, annotation source, region coverage, sample-selection procedure, and mapping from contextualized or merged tokens to patches. How do random and competing selections perform under the same metric?
(5) How robust is residual ranking to irrelevant high-frequency content? Can the authors provide controlled tests involving background texture, irrelevant text, noise, and small answer-critical objects, reporting both localization and downstream accuracy?
(6) Can the closest-method comparisons be expanded? In particular, how does LRCP compare with ResPrune and SVD-Prune across multiple budgets under matched schedules and merging operations?
(7) What is the variability of the reported efficiency gains? Please specify the timing dataset, batch size, generated-token counts, warm-up protocol, synchronization procedure, and repeated-run statistics. Are generation lengths controlled across methods?
(8) Please clarify the remaining evaluation details. What are the exact Qwen pixel-to-token settings, the video compression schedule, the video evaluation model and prompt, and the protocol used to select ranks and compression layers?
Interesting work; Wondering how it compares with PruneSID
Dear authors,
This is an interesting work! However, I believe the idea of low-rank PCA has already been proposed in PruneSID "Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity" from ICLR 2026. PruneSID does compression before entering the LLM, but I am still wondering how it compares with PruneSID.
Thanks for your work!