acceptodds
Under review as a conference paper at ICLR 2027

PixelPrune: Pixel-Level Adaptive Visual Token Reduction via Predictive Coding

Abstract

Document and GUI understanding are among the highest-value applications of Vision-Language Models (VLMs), yet they impose exceptionally heavy computational burden: fine-grained text and small UI elements demand high-resolution inputs that produce tens of thousands of visual tokens. We observe that this cost is largely wasteful: across document and GUI benchmarks, 29–78% of patches within an image are pixel-level duplicates of others. We propose PixelPrune, which exploits this pixel-level redundancy through predictive-coding-based compression, pruning redundant patches before the Vision Transformer (ViT) encoder. Because it operates in pixel space prior to any neural computation, PixelPrune accelerates the entire inference pipeline—ViT encoder, patch merger, and LLM decoder—and applies equally to training. PixelPrune is parameter-free and supports pixel-lossless () as well as controlled lossy () compression. Without any training, it stays on average within 0.9 points of full-token accuracy on documents and similarly close on GUI question answering. On GUI grounding, where predicting precise coordinates makes the model more sensitive to the mismatch between full-token training and pruned inference, distillation from the model's own full-token outputs restores full-token accuracy. Trained from scratch on GUI grounding with PixelPrune, a model matches full-token training and can be served with either pruned or full-token input. PixelPrune speeds up inference by 3.0–4.2 and training by 1.9.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, visual token pruning, efficient inference, predictive coding, pre-ViT compression, document understanding, GUI grounding, native-resolution ViT

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_shKwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PixelPrune, a method targeting high-resolution document and GUI scenarios that eliminates pixel-level redundancy via predictive coding prior to the vision encoder. Operating on pixel blocks aligned with the patch merger, the method predicts each block from its previously visited spatial neighbors, retaining only those that cannot be accurately predicted while preserving their original spatial coordinates. The exact matching mode requires no learned selector and guarantees exact pixel-level recoverability, whereas the near-exact matching mode further allows controlled reconstruction error. Experiments cover training-free document understanding, GUI question answering, GUI grounding adapted via knowledge distillation, and training with compressed inputs. The authors report a 3.0–4.2× TTFT speedup across two representative scenarios and an approximate 1.9× training speedup in the evaluated GUI training setup.

Critical strengths
  1. Clear problem formulation supported by concrete bottleneck analysis. The paper targets the computational cost of visual encoding under high-resolution inputs rather than solely reducing the number of visual tokens fed into the LLM. As shown in Figure 2, at a resolution of 4096 × 4096, the vision encoder accounts for 86%, 72%, and 75% of total prefill latency for Qwen3-VL-2B, 4B, and 8B, respectively. This evidence effectively links input redundancy, computational bottlenecks, and the motivation for pre-ViT compression.

  2. Simple, low-overhead method with strong accuracy under the default setting. The exact matching mode (τ = 0) requires no learned selector and does not rely on neural features or attention maps. Under a matched per-image token budget, PixelPrune achieves an average score of 68.3 across seven document understanding benchmarks on Qwen3-VL-2B, remaining close to Full (69.2) while outperforming all compared compression baselines. Matching the token budget on a per-image basis is a sound experimental design that controls for confounding factors more effectively than matching only dataset-level average retention ratios.

  3. Practical inference latency and memory measurements beyond token counts or theoretical FLOPs. Table 5 reports selector overhead, ViT latency, TTFT, and KV cache memory alongside FLOPs. These empirical measurements provide direct evidence that pre-ViT compression translates into tangible system-level gains, strongly supporting the practical value of the proposed approach.

Critical weaknesses
  1. Lack of quantitative comparison with IndexPrune, a closely related pre-ViT document compression method. Although IndexPrune is discussed in the Related Work section, it is not included in the experimental evaluation. The current experiments primarily compare against three post-ViT methods and two simple pre-ViT controls (Random and ConnComp), making it difficult to assess the practical advantages and trade-offs of the proposed predictive coding approach relative to this closely related baseline. A comparison on representative document benchmarks, or a clear explanation of why a fair comparison is infeasible, would help better establish the independent contribution of this work.

  2. The task-level interpretation of pixel-level recoverability goes beyond the theoretical guarantee. Proposition 1 establishes that a deterministic decoder applying the same prediction rule can reconstruct the original pixels from the compressed representation. During inference, however, the retained patches are fed directly into the VLM without reconstructing the omitted patches. Therefore, pixel-level recoverability does not necessarily imply equivalent feature representations or information aggregation after pruning. In Section 4.3.1, the authors attribute the performance degradation on GUI grounding to the mismatch between full-token training and pruned inference rather than information loss. However, this causal interpretation does not directly follow from Proposition 1. Although knowledge distillation restores performance, this result alone does not establish the mismatch as the sole cause of degradation. The authors should distinguish pixel-level recoverability from task-level performance preservation and moderate the corresponding claims accordingly.

Questions for authors
  1. Comparison with IndexPrune: Why was IndexPrune omitted from the experimental comparisons? Could the authors provide a comparison of accuracy, token retention ratio, and actual computational overhead on representative document benchmarks under the same backbone and input preprocessing settings? Such a comparison would help clarify the advantages and trade-offs of parameter-free predictive coding relative to learned text-saliency selection. If there are specific implementation or reproducibility challenges, please clarify.

  2. Recoverability and the interpretation of performance degradation: Beyond the performance recovery observed after knowledge distillation, is there additional evidence supporting the attribution of GUI grounding degradation to the training–inference mismatch discussed in Section 4.3.1? In particular, can the authors distinguish the effects of sparse spatial layouts from other changes in feature representations and information aggregation caused by token deletion?

Reviewer_3hX9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PixelPrune, a training-free method that removes redundant patches before the ViT in VLMs. Inspired by PNG/JPEG-LS predictive coding, it scans 32×32 blocks and predicts each from its left, upper and upper-left neighbors. A block is dropped if it exactly matches the prediction (τ=0), or is within a threshold (τ>0). Kept blocks keep their original positions. On document and GUI benchmarks with Qwen3-VL, it stays within about 1 point of full-token accuracy at 50–77% retention, beats post-ViT baselines at the same budget, and gives 3.0–4.2× faster prefill. GUI grounding drops 9–14 points without training but recovers after short distillation. Training with PixelPrune matches full-token training and is 1.9× faster.

Critical strengths
  1. Pruning before the ViT gives real end-to-end speedups, in both inference and training, which most post-ViT methods cannot offer. The method is very simple and easy to deploy.

  2. The experiments are broad: three model sizes, three architectures, documents, GUI QA and grounding, distillation, and training from scratch, plus Random and ConnComp controls at the same stage and budget.

Critical weaknesses
  1. Limited novelty. The core idea is to drop patches that are pixel-identical to their neighbors. This is close to ShowUI’s grouping of identical patches, RLT’s removal of repeated patches in video, and DocKylin’s blank-region cropping. The predictive-coding framing and the “recoverability guarantees” make it sound deeper than it is. Proposition 1 is nearly trivial, and the VLM never decodes the dropped patches anyway. In fact, the grounding results show that “recoverable in pixels” does not mean “usable by the model.”

  2. Missing obvious baseline and favorable setup. The simplest way to get fewer tokens is to lower the resolution (a smaller smart_resize pixel budget). This is not compared. The token budgets for baselines are set to whatever PixelPrune keeps (50–77%), which is the mild regime where most methods do fine. At τ=0 the budget cannot be chosen, so there is no evidence at higher compression. On grounding, only the pre-ViT methods get distillation. Finally, on natural images τ=0 keeps 80–100% of tokens, so the method mainly helps in domains with large blank areas.

Questions for authors
  1. How does PixelPrune compare with simply lowering the input resolution to the same number of tokens?

  2. Can you compare methods at fixed, more aggressive budgets (e.g., 25% and 10%), using τ>0 for PixelPrune?

  3. On grounding, how do post-ViT baselines perform with the same distillation recipe? How does PixelPrune compare with GUI-specific methods such as FocusUI or AQuaUI?

  4. Table 5 only compares latency with Full. What are the speedups of VisionZip and DivPrune at the same budget?

1 comment

Dear Reviewer 3hX9,

As the discussion period is nearing its end, we would appreciate your feedback on our rebuttal. We have provided additional results on resolution reduction, more aggressive compression, and latency comparisons under matched token budgets in response to your questions.

Could you please let us know whether these results address your concerns, and which concerns, if any, remain unresolved? We would be happy to clarify further.

Thank you for your time and consideration.

Reviewer_PbSCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PixelPrune uses prediction from neighboring pixel blocks to remove redundant blocks before the ViT while retaining their original coordinates, without learning additional selector parameters. It achieves near-full-input training-free performance on document understanding and GUI question answering, recovers GUI grounding accuracy through training adaptation, and reports 3.0–4.2× faster prefill and 1.9× faster training in the evaluated settings.

Critical strengths
  1. Operating directly on pixels reduces tokens before the ViT, saving computation across the vision encoder, patch mergers, and language model. Pruning complete merge blocks while retaining original coordinates provides a simple design aligned with the model architecture.
  2. The document evaluation covers seven tasks and extends across model scales and architectures. Per-image token-budget matching, prediction-strategy comparisons, and pruning-depth ablations provide substantial validation of the selection rule and its placement.
  3. The GUI experiments distinguish training-free from adapted performance and compare compressed and full-input training with matched initialization, data, and loss. The number of samples per step is controlled, and training-time and memory savings are measured.
Critical weaknesses
  1. ConnComp's per-image budget allocation is insufficiently specified. Section 4.1 describes ShowUI-inspired sampling from pixel-identical connected groups, but does not explain how component retention counts are allocated, rounded, and adjusted to match PixelPrune's exact per-image budget. These rules determine the retained positions and affect interpretation and reproduction of the accuracy advantage over ConnComp. Please provide mask-generation pseudocode specifying component quotas and the handling of the remaining budget, and state whether ShowUI's singleton-protection and component-visibility rules are retained.

  2. The GUI distillation recipe omits key configuration details needed to determine the training objective and data exposure. The main text and Appendix G report the mixed loss, 300 steps, eight GPUs, and learning rates, but omit the effective batch size and gradient accumulation, leaving the number of processed samples unclear. The CE answer-label source and the shared answer prefix and output-token range used for KL are also unspecified, limiting reproducibility of the accuracy recovery. Please provide the effective batch size, gradient accumulation, CE label source, and KL alignment.

Questions for authors

See Weaknesses

Comments on the paper

1 comment
Area_Chair_m9AaOpenReview ↗

With the authors' rebuttal now posted, I would ask each reviewer to respond in the discussion thread.

Now that the authors' rebuttal has been posted, I ask each reviewer to respond in the discussion thread—especially on whether your original concerns have been addressed or remain unresolved—so that our discussion can meaningfully inform the final recommendation before the discussion period closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.