Prune Less, Pay More: Imperceptible Budget Inflation Attacks on Token Pruning in Vision-Language Models

The growing number of visual tokens processed by Vision-Language Models (VLMs) increases inference costs, motivating token pruning and compression techniques that reduce computational demand. Increasingly, these techniques adapt the number of retained tokens to the complexity of each input. The count is determined by statistical thresholds, learned importance predictors, or image-complexity estimators. In this work, we show that this flexibility introduces a hidden opportunity for adversarial cost amplification: an imperceptibly perturbed image can cause a VLM to retain substantially more visual tokens than its clean counterpart while preserving task performance. Consequently, users may incur greater resource consumption without noticing changes in either the image’s appearance or the quality of the model's response. Unlike existing attacks that manipulate token selection within a fixed budget to degrade accuracy, our attack targets the adaptive token budget itself. We propose the Budget Inflation Attack (BIA), which optimizes imperceptible image perturbations to induce content-adaptive compressors to retain more tokens, undermining their intended efficiency benefits. Across multiple adaptive pruning mechanisms and datasets, BIA increases retained visual token counts by significantly while leaving task accuracy essentially unchanged. This token inflation can increase inference computation, latency, and energy consumption. More importantly, it may translate into higher user charges when billing reflects the additional processing. Our findings reveal that adaptive visual token pruning creates an overlooked attack surface: adversaries can erode efficiency and consume resource budgets while preserving the apparent utility of the system. Our code is available at https://anonymous.4open.science/r/PLPM/README.md.