- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Standard vision transformers split images into a grid of square patches (e.g. 16x16) uniformly. This creates a scenario where we spend unnecessary compute effort on non-semantically meaningful background features. To address this issue, this work proposes a methodology to classify tokens within a patch as background or foreground, merging/clustering background tokens, and providing the non-uniform tokens to later transformer layers for processing. This work assigns several tokens to a given patch position, filters them based on the entropy of their activations, performs merging for tokens belonging to background patches (where none of its tokens passed the filter), then finally forwards the merged information as well as original tokens that passed the filter to transformer layers for processing. This approach achieves a reduction in compute in terms of FLOPs and wall-clock time while maintaining baseline accuracy levels.
Critical strengths
- Token reduction and efficiency gains: the pruning before transformer layers cuts down attention cost which creates a significant reduction of compute in terms of flops and wall clock time without major accuracy degradations
- Token merging occurs in latent space and does not rely on more noisy pixel features, which prevents spending effort on patches with high frequency features but no semantically meaningful features
- Test time scaling: t_select can be scaled at test time for an accuracy/compute tradeoff
Critical weaknesses
-
Difficult optimization of t_select and t_merge, both of these rely on straight through estimation to optimize since they induce hard thresholds during inference time. This produces a difficult to optimize surface that requires delicate learning schedules to work with.
-
Computing shannon entropy channel-wise is a proxy for activation channel spread, but this is not a true physical or statistical measure of "information" carried by the token
-
No enforced order guarantee: the nesting loss L_nest is just a soft penalty term of the total loss. The assumed token hierarchy in the paper is not mathematically guaranteed for every patch and the paper's approach is very optimistic on this ordering.
-
L_nest conflicts with the task specific loss when the most discriminative features would require a different activation spread
-
BFS background token merging introduces operations that may not perform optimally on a GPU or be easily parallelizable
Questions for authors
- How many patches generated in your dataset truly follow monotonically decreasing entropy? What is observed accuracy in cases where this ordering is violated?