acceptodds
Under review as a conference paper at ICLR 2027

Learning to Allocate Tokens for Vision Transformers

Abstract

Vision Transformers (ViTs) are a critical backbone of modern computer vision frameworks. Standard ViT architectures work by representing uniformly sized patches as single tokens in a sequence. However, this design is poorly aligned with the true nature of images, where information density is typically not spatially uniform. This results in uninformative image patches receiving the same token allocation as highly informative image patches. To address this limitation, we propose the APE (Adaptive Patch Encoder) architecture, an extension of the traditional ViT that estimates patch importances and dynamically allocates and merges tokens to maximize the information carried in the ViT input sequence. We show that APE is highly efficient while achieving state-of-the-art performance on benchmarks across classification, detection, segmentation, and VQA. Quantitative results across a diverse range of tasks validate that APE's learned saliency metric maximizes the compute-performance tradeoff without domain-specific degradation. Checkpoints and code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision architechtures, efficient transformers

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_eu9DOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Standard vision transformers split images into a grid of square patches (e.g. 16x16) uniformly. This creates a scenario where we spend unnecessary compute effort on non-semantically meaningful background features. To address this issue, this work proposes a methodology to classify tokens within a patch as background or foreground, merging/clustering background tokens, and providing the non-uniform tokens to later transformer layers for processing. This work assigns several tokens to a given patch position, filters them based on the entropy of their activations, performs merging for tokens belonging to background patches (where none of its tokens passed the filter), then finally forwards the merged information as well as original tokens that passed the filter to transformer layers for processing. This approach achieves a reduction in compute in terms of FLOPs and wall-clock time while maintaining baseline accuracy levels.

Critical strengths
  • Token reduction and efficiency gains: the pruning before transformer layers cuts down attention cost which creates a significant reduction of compute in terms of flops and wall clock time without major accuracy degradations
  • Token merging occurs in latent space and does not rely on more noisy pixel features, which prevents spending effort on patches with high frequency features but no semantically meaningful features
  • Test time scaling: t_select can be scaled at test time for an accuracy/compute tradeoff
Critical weaknesses
  • Difficult optimization of t_select and t_merge, both of these rely on straight through estimation to optimize since they induce hard thresholds during inference time. This produces a difficult to optimize surface that requires delicate learning schedules to work with.

  • Computing shannon entropy channel-wise is a proxy for activation channel spread, but this is not a true physical or statistical measure of "information" carried by the token

  • No enforced order guarantee: the nesting loss L_nest is just a soft penalty term of the total loss. The assumed token hierarchy in the paper is not mathematically guaranteed for every patch and the paper's approach is very optimistic on this ordering.

  • L_nest conflicts with the task specific loss when the most discriminative features would require a different activation spread

  • BFS background token merging introduces operations that may not perform optimally on a GPU or be easily parallelizable

Questions for authors
  • How many patches generated in your dataset truly follow monotonically decreasing entropy? What is observed accuracy in cases where this ordering is violated?
Reviewer_LY23OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Accelerating Vision Transformers (ViTs) is a classic research problem. This paper introduces a method called APE (Adaptive Patch Encoder), which uses an adaptive patch encoder to map each image patch into a variable number of tokens (ranging from 0 to 4). This approach compresses the number of tokens, thereby accelerating the Transformer. Specifically, APE employs a learnable global threshold to determine the token allocation scheme; patches assigned zero tokens are merged with neighboring regions, and RoPE (Rotary Positional Embedding) is subsequently used to address the resulting positional embedding challenges. Experimental results demonstrate the effectiveness of this method.

Critical strengths

The paper provides a systematic review of existing approaches and synthesizes their strengths to develop APE, which shows superior performance across various benchmarks—particularly in fine-grained recognition and tasks involving different domains.

Critical weaknesses

My primary concern regarding this method stems from the experimental results. As shown in Table 1, APE does not outperform APT in terms of both performance and computational cost. In fact, APT—which simply employs a finer patch partition—achieves results very close to those of APE (accuracy: 84.6 vs. 84.7; GFLOPs: 29.7 vs. 30.5). This suggests that APE does not offer a significant advantage in terms of overall energy-efficiency. Furthermore, the paper contains some internal contradictions; for instance, while it argues that the token-merging strategy used in ToMe leads to blurred boundaries, APE itself also performs token merging—and does so even more aggressively.

Questions for authors
  1. Why do the baselines in Table 1 and Table 7 differ, given that they appear to represent the same configuration?
  2. In Table 3(a), for the EVA-02-L model at 1536 resolution, APE shows a 2-point improvement in mAP over DART, yet the AP50 values ​​remain nearly identical. Is this data accurate? In other experimental settings, the gains in mAP do not appear to be this substantial.
  3. What is the training overhead? Is the model more difficult to train compared to a standard ViT?
  4. What is the final distribution of tokens learned across different models? This aspect is not addressed in the paper.
Reviewer_r3h9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposed APE, which assigns more tokens to important regions instead of a single token for each patch in vision transformers. It uses well-known techniques such as entropy-based selection and token merging. The authors ran various experiments across different models, datasets, and downstream tasks to evaluate their method.

Critical strengths

1- I liked the idea that not every patch needs a token, and a single patch can be expressed by more than one token.

2- Many experiments have been conducted across different downstream tasks.

3- The results show improved efficiency compared to standard ViTs.

Critical weaknesses

1- Using entropy is not well justified. I understand the idea that higher entropy might highlight some features, but it needs to be proven; also, lower entropy does not necessarily indicate a less salient part of the image. Is there any correlation between entropy and the image's salient regions?

2- I am not convinced the comparison between methods is fair since they have different FLOPs, runtime, and metrics. APE often uses more computation but yields relatively small improvements. How can you say this improvement justifies the additional compute cost?

3- In many experiments, I noticed marginal improvements compared to other methods. I assume you might get these improvements with different initial seeds! Multi-seed runs are needed when the results are so close!

4- The images in Figures 4 and 5 show that the neighbours are neighboring patches that are already visually very similar! (e.g., similar color or texture, such as a clear blue sky). It's unclear if pixel-wise merging can achieve a similar or better result.

5- It's not clear how the threshold is learned. What prevents the model from choosing a threshold of 0 or negative, or a very small value, to use all the candidate tokens? The model can easily ask for all tokens to get the lowest loss or highest accuracy!

6- The proposed APE method is a bit complex and a combination of different techniques, but the improvements in many experiments are not impressive!

7- Tables 2 and 12 have the same results for different model sizes! (ViT-B and ViT-L)!

Questions for authors

1- Why does higher entropy mean more useful information? Is there any proof or correlation between entropy and the salient and non-salient parts of the image?

2- How is the threshold learned? What prevents the model from choosing a very low threshold and selecting all the tokens?

3- Can the authors compare different methods using the same computational cost and runtime?

4- What happens if APE selects the same number of tokens as a standard ViT (only the tokens with the highest entropy)? Will it still improve metrics?

5- Can the authors provide multi-seed results where the results are very close?

6- Regarding object detection and segmentation, how does APE perform for small objects?

Reviewer_xXQ6OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes APE, an adaptive Vision Transformer architecture that dynamically allocates tokens using nested representations, entropy-based token selection, and token merging. Extensive experiments across image classification, object detection, semantic segmentation, and vision-language tasks demonstrate improved accuracy-efficiency trade-offs under different computational budgets.

Critical strengths
  1. Learning to allocate computational resources according to image content addresses a fundamental limitation of uniform tokenization, with potential applicability across diverse vision tasks.

  2. The method demonstrates competitive accuracy-efficiency trade-offs across multiple architectures, downstream tasks, and inference budgets, providing strong empirical support for its practical value.

Critical weaknesses
  1. The paper uses Shannon entropy of normalized feature activations as a proxy for task-relevant information, but this relationship is not rigorously established. The interpretation of entropy in the nested representation objective and token selection criterion also requires clarification.

  2. Although APE generally improves accuracy, it sometimes requires more computation than competing adaptive tokenization methods, particularly APT. This makes its overall computational advantage less conclusive under strictly matched budgets.

Questions for authors
  1. In Section 3.1, Eq. (2) encourages entropy to decrease across nested embeddings, while the explanation associates lower entropy with more informative representations. However, Eq. (3) selects tokens with higher entropy. Could the authors clarify the intended relationship between entropy and task-relevant information, and reconcile these formulations?

  2. Could the authors further clarify the advantages over APT and DART under matched end-to-end computational budgets, particularly when accounting for the accuracy-FLOPs trade-off and adaptive patch encoding overhead?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.