acceptodds
Under review as a conference paper at ICLR 2027

Every Sparkle Gets a Token: Entropy-Guided Foveated Tokenization for Fine-Grained Visual Recognition

13 views

Abstract

Uniform image tokenization allocates the same number, locations, and scales of tokens to every image, even when discriminative evidence is concentrated within a small region. Existing token-reduction methods commonly identify redundancy only after a dense patch sequence has entered the backbone. We introduce SpiralFovea, a transparent, parameter-free selector that constructs a sparse visual sequence directly from image content. SpiralFovea uses local luminance entropy to identify spatially distributed anchors and extracts multi-scale rings of crops around them, allocating more tokens to locally informative regions while retaining coarser contextual coverage. These rings also define a deterministic centre-to-periphery ordering suitable for directional state-space models such as Mamba, while transformers receive explicit spatial descriptors for the resulting non-grid tokens. At 224 × 224 resolution, SpiralFovea uses at most 78 spatial tokens instead of the 196-token ViT-S/16 grid. Across four recognition tasks, it improves the reported mean accuracy of frozen ViT/DINO and ResNet–Mamba pipelines while increasing measured end-to-end throughput by 17.8–28.9%. On standard 200-way CUB at a matched sparse budget, SpiralFovea achieves 85.16% top-1 accuracy—within 0.45 percentage points of the dense model and 0.99 points above ToMe. At 518 × 518 resolution, it reduces a 1,369-token DINOv2 grid to approximately 120–200 tokens, revealing substantial computational savings but task-dependent accuracy trade-offs. Foreground-localization and inverted-entropy analyses further identify the method’s operating boundary: entropy is effective when discriminative structure is locally textured and spatially coherent, but becomes unreliable when high-entropy background clutter dominates.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
adaptive visual tokenization, foveated vision, fine-grained visual recognition, vision transformers, sparse computation, entropy-guided sampling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9wryOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SpiralFovea, a parameter-free, pre-backbone token selector for fine-grained image recognition. It computes local luminance entropy, divides the image into horizontal strips, and picks one entropy-maximum anchor per active strip (with a distance-based deduplication). Around each anchor it samples crops on four fixed concentric rings with increasing radius and crop size. Only these crops (at most 78 at 224x224, versus 196 grid patches) are passed to the backbone. For frozen ViT/DINO backbones, only a small positional MLP and a linear classifier are trained. For a ResNet-18 + Mamba model, the crops are fed in a fixed center-outward order per anchor, and the anchors are visited top to bottom.

The authors evaluate on WikiArt (genre, style), Flowers-102, PCam, and CUB-200. They report +1.7 to +2.1 mean accuracy over dense backbones, 85.16% on CUB (dense 85.61%, ToMe 84.17%), and +17.8% to +28.9% measured throughput on a T4 GPU. At 518x518 with DINOv2-L, tokens drop from 1,369 to about 120-200 with accuracy changes from +1.90 to -5.86 points. They also analyze anchor overlap with annotated objects and state that the benefit disappears when background entropy exceeds foreground entropy. The paper presents the method as a measured operating point rather than a general claim of superiority.

Critical strengths
  1. The authors narrow the scope of their claims and directly test their own assumptions. For example, they measure how often the anchors fall on the object, and analyze a cohort where the background entropy is higher than the foreground.

  2. The input-stage method is simple and interpretable: the selector needs no training and finishes before the backbone, so where it fails can be inspected.

Critical weaknesses
  1. The claim that the spiral order is useful is not supported by the experiments.
  • The authors argue that Mamba reads tokens sequentially and updates a single state, so reading the detailed crop at the anchor center first and the coarse surrounding context later should help. However, there is no ablation of the spiral order on the Mamba model. Table 7 is a ViT experiment, and it is not defined what "linear layout" changes.
  • It is not separated whether the Mamba gain comes from the different crop positions or from the spiral order. The central claim needs an ablation that separates these two to be supported.
  • The order between anchors is simply top to bottom, so the whole sequence is not "center to outside".
  • Since scan order may matter for Mamba, it needs an analysis of whether reading the key tokens first and the surrounding tokens later is actually the better choice. There is no analysis of the spiral in reverse order or in random order.
  1. The core design is not well explained, the selection is restricted, and the comparisons are insufficient.
  • There is no figure showing the ring configuration (radii, crop sizes, counts) and the token order. Figure 1 does not visually convey the core ring structure and ordering.
  • Anchor selection is not based on a global entropy ranking. Since there is exactly one anchor per strip, if a strip contains two high-entropy locations, only one is used. Anchors are also confirmed from the top strip downward rather than in entropy order, and when two anchors are close, the later one is discarded. So when the lower anchor is the more important one, information is lost.
  • There is no experimental comparison with other input-stage tokenizations (Elastic, SPoT, EPOC, DHT), and ToMe is the only properly trained baseline.
Questions for authors
  1. (Clarification) In Table 7, what exactly does "linear spatial layout" change? Is it the crop positions or the token order? And does the order matter for the ViT in this ablation?

  2. (Suggestion) Is there a comparison on the Mamba model where the same crops are fed in spiral, reversed, and random order? An analysis is needed to show whether reading the key tokens first is actually better.

  3. (Clarification/Suggestion) The entropy is computed on grayscale. Is the result the same as with RGB entropy on datasets where color is a distinguishing cue, such as Flowers and WikiArt? Beyond the 0.01-point difference on CUB, did you check whether the anchor positions change (e.g., with the anchor overlap analysis)?

Reviewer_4U4nOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Modern vision transformer typically takes as input patches obtained with a uniform grid. The paper argues that it is inefficient and proposes a backbone-free, entropy-guided tokeniser for images. The method first downsamples the image and divides it into multiple horizontal strips. Within each strip, it then computes local luminance entropy and use the location with maximum entropy as an anchor. It next selects around each anchor four rings at different sizes and angles which are then cropped and resized as tokens to the model. Before going through the backbone, the tokens are processed with a two-layer positional MLP.

The paper evaluates the proposed tokeniser in four small classification benchmarks. It shows that the method is more efficient than dense tokeniser while also performing better. It also compares favourably to other sparse tokenisers.

The paper provides analysis on the quality of selected anchors using annotated datasets, and ablation studies showing the important of different design choices.

Critical strengths
  • The paper is trying to tackle an interesting problem. It is conceptually appealing to build a backbone-free tokeniser for images that can compete with dense tokenisation while being more efficient.

  • The paper provides interesting analysis on the quality of the selected anchors.

  • The paper is candid about its limitations and provides a clear, separate section on this.

Critical weaknesses

Section 9 of the paper covers surprisingly well its weaknesses. Below are my own assessments:

  • Though the proposed method sounds, it is hand-crafted and weak in terms of novelty. The proposed anchor selection method is similar to that of many pre-deeplearning image descriptors. The method has many hyper-parameters whose values are provided without justification in the paper (the radii, the crop widths, the angular counts, the boundary margin, the deduce threshold, etc.). This make the method quite ad-hoc, which puts a major question mark on its applicability in wider scenarios.
  • The comparisons provided in the paper are not completely controlled. Compared to baselines, the proposed method has more parameters in the trainable positional MLP which the dense baseline does not have. The high-resolution study is confounded by the change of backbone.
  • It lacks a systematic comparison to other baselines on all considered benchmarks (only CUB results are provided in Table 2).
  • The paper provides results on only 5 small benchmarks which is far from the current standard.
  • The paper's writing is unclear and weird at times. For example, the a lot of space is used to provide a number in Eq. 6. Notations such as \rho and \theta are not introduced in Eq. 3.
Questions for authors

The paper is weak in the present form. Positive results from the following experiments would strengthen it:

  • Larger scale experiments on ImageNet or iNaturalist
  • A fair comparison to the dense tokeniser by also using the MLP in the latter
  • A controlled comparison to other sparse tokenisers.
  • Ablation on the hyper-parameters
  • In the same budget setting, add a comparison to dense 78 patches on a uniform grid.
Reviewer_PBqGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors introduces SpiralFovea, which turns out to be a token selection / pruning strategy that uses categorical Shannon entropy based on luminance to add importance scores for patches. The paper does not have a dedicated claims section, but features a single description of the contribution (L77, p.2):

The contribution is a measured operating point with an observable scope condition, rather than a universal superiority claim.

This is a rather evasive formulation, and makes it difficult to establish the significance of the work. The method is dubbed SpiralFovea, seemingly due to the fact that they use a spiral sequential ordering for sequential state space models, namely via a Mamba model. Later, it becomes clear that in certain experiments, the authors employ the spiral ordering strategy in standard attention based ViTs as well, seemingly without regard for the fact that any such ordering strategy will have no impact for permutation equivariant models.

The evaluation is on several smaller datasets along with a histopathology dataset. Each dataset are either small (CUB, OxfordFlowers), binary (PatchCamelyon), or uncommon (WikiArt) for general benchmarking in CV. The authors acknowledge that this is a limitation, however this does not clarify the intention or purpose of basing the experiments on these benchmarks, as opposed to more common datasets (e.g. ImageNet, iNaturalist, etc.).

The central teaser figure (Figure 1) is particularly difficult to interpret. Panel (a), labelled "WikiArt bird", appears to depict a human figure, and it is unclear what the label refers to. The source images are low-resolution and largely obscured by overlapping rectangles. While the caption identifies the cyan crosses as entropy anchors and the rectangles as crops, it does not explain what the different colours represent, which crops belong to which anchors, or how the proposed spiral ordering is constructed. It is also difficult to see how the figure demonstrates the eponymous spiral foveated sampling, or even if the selected anchors are meaningful selections / evidence for representing the image.

The pipeline in panel (c) provides little in terms of clarification, and consists primarily of a strict sequence of labelled processing stages. Given that the proposed sampling geometry is central to the contribution, I would expect the paper to illustrate it much more clearly. As presented, even the introductory figure makes it exceedingly difficult to understand what distinguishes the proposed method from other spatial sampling strategies.

In spite of careful reading, the contributions come across as unclear, with odd choices of benchmarks, and several questionable methodological choices, such as not really benchmarking their method with the related works the paper cite as relevant.

Critical strengths

The paper features a set of recent relevant works on tokenization, these seem appropriate for comparison.

The authors are open about limitations and acknowledges the issue on datasets and refrains from overclaiming on the significance on the work.

Critical weaknesses

1. Choice of Pure Shannon Entropy for Spatial Data

The papers central contribution is seemingly the choice of using Shannon entropy over quantized luminosity in each patch with 16 values, as a histogram. The quantization is global, seemingly to uniform selected bins, at half resolution. While this is only a mechanism to do the token selection, there are several significant problems with such an approach.

  • It disregards spatial structure completely, meaning high frequency and low frequency content is not distinguished in the selection mechanism at all. An image gradient between a light and dark region with maximal uniform distribution between the 16 bins has a maximal score (4 bits). A region with a meaningful texture pattern with high frequency content can achieve the exact same score, as long as the quantized distribution is equivalent. This metric is therefore completely unable to meaningfully distinguish patterns and textures, contrary to what is claimed in the paper (L319, p.6). A second order technique such as histogram of oriented gradients (HOG, Dalal and Triggs, 2005) is well known to convey significantly more meaningful information, and represents an established method that has been extensively discussed in previous works.
  • The global quantization policy is exceedingly sensitive to quantization noise. A region with pixel intensities between 0.45-0.47 will have a histogram entropy of 0 bits, whereas a neighboring patch with the same texture, but with intensities between 0.49 and 0.51 will have a corresponding entropy of approx. 1 bit. A tiny shift in luminosity can therefore significantly alter the score, and which anchors are selected by the model. This could possibly be somewhat remediated by quantization methods such as dithering, but it is unlikely to completely remove this arbitrary sensitivity.
  • To demonstrate the viability of this modelling decision, the authors carry out a random-anchor comparison. This is, however, insufficient. It establishes that their heuristic can outperform random placement, not that the chosen statistic is appropriate.

2. Issues with Spiral Ordering

As mentioned in the summary, the paper names their method after a spiral ordering in Mamba style sequential models. However, the paper confuses the effect of this ordering as something translatable to standard ViT models, which are known to be permutation equivariant.

Table 7 reports a 0.9 percentage-point reduction when replacing the spiral layout with a linear spatial layout. The experiment is performed using DINO-S/16, not Mamba. Moreover, L172–174, in the Mamba subsection, explicitly says “the linear-layout ablation in Section 7 provides the empirical evidence for the chosen order,” yet that ablation is on DINO-S/16. The manuscript interprets this experiment as evidence that the order provides useful structure for sequential fusion. As this reviewer reads the paper, there are only two reasonable interpretations:

  • If the authors altered only the ordering, the reported difference requires some level of further explanation, since the transformer should be equivariant to token ordering.
  • If the authors instead changed token locations, sampled content, or positional descriptors, the experiment does not actually isolate ordering, contrary to the claims in the paper. As such, none of these explanations establishes that the "spiral" sequence is beneficial for Mamba. Yet this is one of the paper's defining motivations, and the source of the name of the method.
Questions for authors
  • Exactly how do the authors intend to deal with sensitivity issues arising from quantization errors, as discussed in weaknesses (1) above?
  • Why was 16 quantization levels selected, and why are images downsampled to half resolution for the study?
  • How is histogram entropy of first order luminance features supposed to detect edges if spatial structure is completely ignored by histogram features?
  • The paper lacks a clear section of claims or contributions, except for the paragraph at the end of the introduction. The stated contributions are somewhat unclear for an ICLR submission. What are the authors stated claims as contributions to the field?
  • Why have the authors elected to avoid datasets such as ImageNet or iNaturalist, that could more convincingly demonstrate the capability of the method on established benchmarks? Is there a particular reason for the choice of datasets included in the study?
  • Why was ToMe excluded from the throughput comparisons? How meaningful is the inclusion of DynamicViT and MSViT when neither applies the intended training procedure?
  • How can a total of 78 spatial tokens produce an average attention reduction of 79%, when the stated quadratic calculation implies at least 83.9%, according to the papers own definition? Moreover, how is the upper bound of 78 tokens established from 116 candidate crops?
  • What is the meaning of the label in Figure 1a, "WikiArt bird"? What do the colours of the bounding boxes represent in the figure? How does it relate to the spiral ordering advertised in the paper?
  • What does Mamba actually contribute relative to simpler aggregation of the ResNet features? Note that this is wholly separate from the question of whether spiral ordering is useful within Mamba.
3 comments
AuthorsOpenReview ↗

Reason for choosing entropy and Quantization Sensitivity.

Clarification of Entropy-Based Spatial Selection.

We thank the reviewer for raising these concerns. We would first like to clarify an important aspect of our methodology. SpiralFovea does not compute a single histogram entropy for each predefined ViT patch and subsequently rank those patches. Instead, it constructs a spatially resolved local entropy map using overlapping 15 × 15 luminance windows on a downsampled 112 × 112 image. The resulting entropy map is used to identify spatially distributed local maxima, which serve as anchors for subsequent multi-scale crop extraction. Thus, entropy operates before patch projection and determines where the input representation should be sampled, rather than assigning importance scores to an existing sequence of predefined patches.

Role of Histogram Entropy and Spatial Structure.

Regarding the reviewer's concern about edge detection, we agree that first-order histogram entropy does not explicitly encode spatial arrangements or identify edge orientations. Two local windows with identical intensity histograms but different spatial arrangements will indeed receive identical entropy values. Therefore, we do not claim that histogram entropy directly detects edges or captures complete local image structure. Instead, it serves as a computationally inexpensive proxy for local intensity-distribution complexity, helping identify potentially informative regions for allocating spatial sampling density. The subsequent multi-scale ring sampler extracts image crops around the selected anchors. Unlike the histogram representation used for anchor selection, these crops retain their original spatial arrangements and are processed by the visual backbone, allowing the model to learn spatial structures and features from the sampled regions. We acknowledge that high entropy does not necessarily imply semantic importance and may also arise from textured backgrounds or noise. This is an inherent limitation of using local entropy as a sampling heuristic.

Quantization Sensitivity and Choice of Hyperparameters.

We also appreciate the reviewer's observation regarding the sensitivity of histogram entropy to quantization boundaries. We agree that small changes in pixel intensities near quantization thresholds can alter histogram frequencies and, consequently, local entropy values. Since anchor selection depends on local entropy maxima, such variations can potentially influence the selected anchor locations.

In our implementation, we use 16 uniformly spaced intensity bins over normalized luminance values. This choice provides a compact representation of local intensity distributions while keeping histogram computation inexpensive. We use half-resolution entropy computation (112 × 112 for a 224 × 224 input) to reduce preprocessing overhead. The resulting anchor coordinates are mapped back to the original image resolution before multi-scale crop extraction.

Regarding how we address quantization sensitivity, our current implementation uses a fixed, uniform quantization scheme to ensure consistent entropy computation across images. However, we acknowledge that this does not eliminate sensitivity to quantization boundaries. The current manuscript does not establish 16 bins or half-resolution computation as optimal, nor does it quantify the stability of anchor selection under alternative bin counts, quantization boundaries, or entropy-map resolutions. We will clarify that these are engineering hyperparameters rather than theoretically derived optimal values. A systematic sensitivity analysis, potentially including alternative quantization schemes, would provide a more complete characterization of the selector's robustness. We will also explicitly acknowledge this limitation in the revised manuscript.

Evidence Supporting Entropy-Based Anchor Selection.

To further investigate whether entropy-based anchor selection provides an advantage over arbitrary spatial sampling, we conducted an experiment using randomly selected anchors while retaining the same multi-scale ring-sampling strategy. We observed that random-anchor selection resulted in approximately 2.5–4 percentage points lower average accuracy across the evaluated datasets compared with entropy-based selection. This result suggests that the spatial locations identified through local entropy provide a more effective sampling strategy than random anchor placement under the same sampling configuration. While this does not establish entropy as a universally optimal measure of visual importance, it provides empirical evidence supporting its utility as a lightweight spatial proposal mechanism.

AuthorsOpenReview ↗

Clarification of the Main Novelty and Contributions.

Clarification of the Main Novelty and Contributions.

We thank the reviewer for pointing out that the contributions of the paper are not articulated clearly enough. We would like to clarify that the central contribution of SpiralFovea lies in its fundamentally different approach to visual token reduction.

1. Moving Token Selection from Latent Space to Input Image Space.

A major distinction between SpiralFovea and conventional token-pruning approaches is the stage at which spatial selection takes place. Much of the existing literature on visual token reduction operates in the latent space, where an image is first divided into patches, projected into token embeddings, and subsequently pruned, merged, or selected based on learned representations or attention characteristics. In contrast, SpiralFovea performs spatial selection directly on the input image, before patch projection and visual backbone processing. It determines where computational resources should be allocated before the visual representation is constructed, rather than reducing an already constructed token sequence. We believe this input-space perspective provides an important alternative direction for efficient visual recognition.

2. Entropy-Guided Adaptive Spatial Sampling.

To realize this approach, we introduce a lightweight, parameter-free spatial selection mechanism based on local luminance entropy. The resulting entropy map identifies spatially distributed sampling anchors, around which a multi-scale ring-based crop sampling strategy extracts local details and surrounding context. This allows the input image to be represented through spatially adaptive sampling rather than relying exclusively on a predefined uniform patch grid.

3. Architecture-Compatible Spatial Representation.

We investigate how the resulting spatial samples can be processed by different visual architectures, including Transformer-based backbones and Mamba-based sequential aggregation. This demonstrates how input-space adaptive sampling can be incorporated into different visual representation architectures without restricting the approach to a single backbone family.

4. Empirical Performance and Computational Efficiency.

We extensively evaluate SpiralFovea across four fine-grained visual recognition datasets and multiple visual backbones. Our results demonstrate that SpiralFovea achieves higher accuracy than the evaluated token-reduction baselines under comparable token budgets. Furthermore, compared with their corresponding unpruned backbones, SpiralFovea achieves higher throughput while maintaining competitive or improved recognition performance. For example, our experiments report throughput improvements ranging from 17.8% to 28.9% across the evaluated backbones. Additionally, our random-anchor experiments demonstrate that entropy-guided selection achieves approximately 2.5–3 percentage points higher average accuracy than random spatial selection under the same sampling strategy.

We acknowledge that the original manuscript did not sufficiently emphasize these contributions. We will revise the Introduction and contributions paragraph to make clear that the primary novelty is performing adaptive spatial selection directly in image space, before visual tokenization, with entropy-guided anchor selection and multi-scale sampling serving as the mechanisms that implement this idea.

AuthorsOpenReview ↗

ToMe missing throughput experiments and clarification of MSViT and DynamicViT

Additional End-to-End Throughput Comparison.

To provide a more comprehensive systems-level comparison, we additionally report the end-to-end throughput of existing token-reduction baselines under the same hardware and evaluation conditions. We include both ViT-S/16 and DINO-ViT-S/16 backbones to examine whether the observed efficiency trends generalize across different pretrained Transformer architectures.

MethodViT-S/16 (images/s)DINO-ViT-S/16 (images/s)
Dense / Uniform598612
ToMe690710
DynamicViT613620
MSViT605610
SpiralFovea (Ours)714721

All measurements are performed on a Tesla T4 GPU with a batch size of 32. The reported throughput includes the respective token-selection, preprocessing, and backbone inference overheads.

These additional measurements provide a more direct comparison of the end-to-end computational efficiency of SpiralFovea against existing token-reduction approaches. SpiralFovea achieves the highest measured throughput across both backbone configurations, demonstrating that its input-space selection mechanism introduces sufficiently low overhead to translate token reduction into practical inference speedups.

Clarification of DynamicViT and MSViT Comparisons.

We agree with the reviewer that DynamicViT and MSViT are not evaluated under their intended training procedures. Both methods employ learned selection mechanisms that typically require method-specific optimization, including distillation, budget regularization, or specialized training schedules.

In our experiments, we adopted a shared frozen-backbone protocol to maintain a common evaluation setting across methods. Consequently, these results should be interpreted as diagnostic comparisons under a shared protocol, rather than definitive comparisons against fully trained implementations of DynamicViT and MSViT.

We acknowledge that this protocol does not establish the intrinsic performance of these methods under their recommended training recipes. We will make this limitation more explicit in the revised manuscript and avoid interpreting these results as evidence of superiority over their fully trained versions. ToMe remains the more direct comparator in our current matched-budget evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.