- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates adversarial vulnerabilities introduced by visual token compression in large vision-language models. It defines compression-specific failures (CSFs) as cases where full-token inference remains correct while compressed inference fails. Through controlled counterfactual analyses, the authors examine how token selection and representation drift contribute to these failures. Based on these findings, they propose CIRA, an encoder-only attack that combines Global Selection Hijacking (GSH) and Hidden-Evidence Preservation (HEP) to manipulate token priorities while preserving critical visual information. Experiments across multiple datasets, compressors, and models demonstrate its effectiveness under unknown compression configurations. The paper also introduces a cross-view selection-stabilization defense and evaluates its robustness against adaptive attacks.
Critical strengths
- The clean-conditioned, paired CSF evaluation and controlled retained-set interventions provide a clear framework for analyzing failures specifically associated with visual token compression.
- The combination of global priority reallocation and selective evidence preservation is well motivated, and the experiments demonstrate promising attack selectivity across multiple compressors, budgets, and LVLM architectures.
Critical weaknesses
- The novelty is somewhat limited, as compression-specific failures and token-ranking manipulation were previously investigated in [1, 2], while encoder-only attacks and probabilistic modeling of unknown compression budgets were explored in [3, 4]; the main novelty therefore lies in the particular combination of GSH and HEP.
- The mechanistic counterfactual analysis is restricted to LLaVA–VisionZip and relies on a post-hoc answer-aware oracle, leaving unclear whether the observed relationship between retained-set allocation, representation drift, and correctness generalizes to other compression mechanisms.
- The attack relies on encoder-side proxy rankings that may differ substantially from actual token-selection decisions, particularly for merging-based compressors and LLM-attention-based methods such as FastV, but this proxy mismatch is not systematically quantified.
[1] Sun et al., Does Acceleration Cause Hidden Instability in Vision Language Models? Uncovering Instance-Level Divergence Through a Large-Scale Empirical Study.
[2] Zhang et al., Less Is More — Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Models.
[3] Zhang et al., On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression.
[4] Mei et al., VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Models.
Questions for authors
- Can the retained-set counterfactual analysis and TCS defense be extended to compressors beyond VisionZip?
- Does HEP consistently outperform selective perturbation [1] and simpler feature-preservation baselines under matched optimization settings?
- Can the authors quantify the agreement between encoder-side proxy rankings and actual retained-token selections across different compressors, and relate this agreement to CSFR?
[1] Zhang et al., Less Is More — Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Models.
4 comments
[2/2] Response to Reviewer 5pKP
Question 3 & Weakness 3The attack relies on encoder-side proxy rankings that may differ substantially from actual token-selection decisions, particularly for merging-based compressors and LLM-attention-based methods such as FastV, but this proxy mismatch is not systematically quantified. Can the authors quantify the agreement between encoder-side proxy rankings and actual retained-token selections across different compressors, and relate this agreement to CSFR?
Thank you for this question! We traced native patch IDs and predictions on three benchmarks, 1,000 pairs each and four budgets.
-
Proxy–native agreement measurement.
- Proxy Top-D matches each compressor's native direct/representative capacity, including PruMerge's representative patches and FastV's actual decoder-side selections.
- The uniform reference fixes the native set and uniformly selects D patches from all 576 patches. We compute its exact expected Jaccard and average using the same observation weights as the measured adversarial retained-set Jaccard.
- For each selection rule, the displaced set contains patches retained on the clean input that lose retained/representative membership after attack. We also compute Jaccard between the proxy and native displaced sets.
-
CSFR analysis and results.
- Among observations clean-correct under full-token and native-compressed inference, we split within each compressor–dataset–budget setting by displaced-set Jaccard: lower is at or below the median; higher is above it. Tied values stay together.
- Each group contains at least 174 observations per setting. Group CSFR averages 12 dataset-budget settings equally; retained-set Jaccard averages all evaluated observations.
Compressor Retained Jaccard: proxy / uniform CSFR: lower / higher displaced agreement (%) VisionZip 0.457 / 0.084 17.44 / 21.23 VisPruner 0.329 / 0.103 14.65 / 16.01 PruMerge 0.543 / 0.102 30.77 / 37.49 FastV 0.328 / 0.103 15.76 / 16.80 - Per-compressor mean retained agreement is 3.17–5.41 times the uniform reference; all 48 setting means exceed their matched reference.
- The higher displaced-agreement groups have higher mean CSFR for all four compressors, by 3.79, 1.36, 6.72 and 1.04 percentage points, respectively.
Thank you for your response. It addressed my questions, and I will maintain my score.
Thank you for your time and constructive feedback, which have helped us improve the quality of our paper! We are pleased that our response addressed your questions.
[1/2] Response to Reviewer 5pKP
We thank the reviewer for the positive assessment and constructive questions.
Thank you for this question! We extended both to VisPruner, PruMerge and FastV. The answer-aware oracle selects post-hoc diagnostic exchanges at fixed failure states; CIRA's encoder-only optimization uses no reference answers.
Cross-compressor diagnostics.
TCS defense.
Thank you for this question! Building on prior ranking manipulation and encoder attacks, CIRA couples global priority inversion with targeted preservation of displaced tokens.
Contribution of HEP.
Matched preservation controls.