- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper is about “content-adaptive visual tokenizers.” Instead of cutting the image into a fixed grid of patches like ViT, the method (ToMATOS) groups pixels into K regions that follow the image content and turns each region into one token. The number of tokens is fixed, but the shape and size of each region adapt to the image. The paper also aims to be faster and fully trainable, unlike earlier adaptive tokenizers (SuiT, SPiT, ∂HT), which are slow, rely on CPU superpixels or non-differentiable steps, or don’t let the user set the number of tokens.
A small CNN first computes a feature for every pixel. Each feature is split into two parts: a small clustering feature (6 channels, via PCA) and a token feature with a positional encoding. Flash-SNIC, a fast GPU version of the SNIC superpixel algorithm, places K seeds on a regular grid and grows regions from them. Each pixel joins the neighbouring region with the lowest cost, which mixes feature similarity and spatial distance (weighted by a compactness parameter m). To make this trainable, pixels near region borders are softly shared between the regions in their 3×3 neighbourhood using a softmax with temperature τ. Each region is then pooled into one token (mean and max pooling for classification, box pooling for distillation), and the tokens go into a standard ViT.
Critical strengths
Fast. The tokenizer runs fully on the GPU (Triton kernel), which makes it faster than other adaptive tokenizers: 12.7 ms vs 16–58 ms at Small size.
Controllable at inference. A single trained model can run with different numbers of regions and compactness values, which changes its speed–accuracy trade-off (Fig. 5). The paper does not explain how to choose these values in practice.
Transparent reporting. The authors report negative results (e.g. DINOv3 on INSID3, worse sufficiency than SuiT) and state how qualitative examples were selected. The implementation details are thorough.
Critical weaknesses
1: the core motivation is assumed, not shown, (or likely overstated.)
The paper treats object-aligned tokens as a goal and claims that patches mixing objects (“bleeding”) hurt the representation, but this is never tested. In a ViT, self-attention shares information across the whole image. That shared context is a main reason ViTs generalize well (e.g. to new objects, or to downstream uses such as robot policies), so a patch covering two objects is not obviously a problem. The evidence supports a narrower claim: sharper boundaries for attribution maps and masks at a fixed token budget. Why it matters: the paper’s significance rests on the broader claim. (See Q1, Q5.)
2: the reported gains cannot be attributed to the tokenizer.
Classification. ToMATOS uses a RegNetY-16GF distillation token, but the ViT baselines do not (App. B.1). DeiT with the same teacher reaches 81.2 (Small) and 83.4 (Base), which would make ToMATOS a tie (81.24) and a loss (82.66). Results come from single runs with gaps of a few tenths. Segmentation. The students use more tokens than the DINOv1/v3 teachers (1681 vs 1296). TokenCut is run without its bilateral solver. For DINOv3, pooling teacher features onto regions gives +0.04 IoU against +4.2 for the student (App. D.2), so the gain may come from the extra distillation training rather than from the regions. Why it matters: without matched patch-token baselines, the main experimental claims are not supported. (See Q3, Q4.)
Questions for authors
-
Can you give direct evidence that tokens mixing several objects hurt representation quality or generalization, beyond boundary-sensitive metrics? Otherwise, would you narrow the claim to attribution and mask sharpness at a fixed token budget?
-
How sensitive are the results to K, m, τ, the number of sweeps and the PCA size? Why are evaluation settings tuned per model (e.g. m = 0.05 vs 1.0 in Table C.1)? Given the content-adaptivity motivation, why is K fixed for all images, and how should a user choose it?
3.Can you report a patch ViT trained with exactly the same recipe (same RegNetY teacher, augmentation, epochs and initialization)? Were SuiT, SPiT and ∂HT trained with the teacher too? Can you report mean ± std over several seeds, and clarify the initialization of the Base model (App. B.1 vs Table B.1)?
- Can you compare against a patch-token student distilled with the same recipe and token count, and report TokenCut with its bilateral solver?
5.What is the expected benefit for the community over standard encoders? Can you compare at matched compute against a patch ViT with fewer tokens, and against simple token-merging or clustering baselines on a pretrained DINO or SigLIP2 model?
Rebuttal Summary
Dear Area Chair and Reviewers,
We thank the reviewers for their careful reading and constructive suggestions. Reflecting the rebuttal, we have updated the paper draft with all changes marked in green and summarize the discussions below.
How we addressed the main concerns
1. Separating region tokenization from distillation (zfGA W2, Q3-Q4; n3H9 W2). We now show that with the same distillation pipeline, ToMATOS improves IoU over the patch student in all six TokenCut comparisons across three datasets and two teacher groups (+0.9 to +6.2 points). In the INSID3 evaluation settings, ToMATOS also improves one-shot part correspondence by 2.11 points for DINOv2 and 2.33 for DINOv3. Both DINOv3 students remain below their teacher, while ToMATOS student still outperforms the patch student decoupling the impact of distillation and region tokens that was asked. Reviewer zfGA acknowledged that these controls clarified the contribution of region tokens and raised their score.
2. Practical benefit and runtime (zfGA W1, Q1, Q5; n3H9 W1). On DINOv3/ECSSD at 1,296 tokens, ToMATOS gains 2.75 IoU for 4.3 ms over the patch student. While bilateral refinement gives the patch student a small 1.59-point lead over ToMATOS it almost doubles runtime per image. Among the adaptive tokenizers evaluated, ToMATOS has the lowest measured latency: 3.6/5.0 ms for Small/Base versus 11.5/12.7 ms for SuiT as the next fastest option, albeit not differentiable.
3. What regions capture (zfGA W1, Q5). At tested 1,024 tokens, ToMATOS improves over both ToMe and k-means in IoU and mask boundary F-score.These regions define the tokens used by the classifier. At the same nominal masking fractions, removing highly attributed content lowers confidence by about 9.5 percentage points more for ToMATOS than for the patch ViT, meaning that our region tokens succinctly capture the main information necessary for the downstream task.
4. How the partition is learned (8yr2 W1-W2, Q1-Q2; n3H9 Q2; zfGA Q2). We clarified how soft assignments let task gradients move region boundaries while the interior pixels still receive gradient updates through the pooling of token features within a region. We also reported the asked region-size and aspect-ratio distributions, sensitivity to token count and compactness, and number of iterations checks, giving a better analytical understanding of how regions are shaped.
Overall, we believe the strongest advantage of ToMATOS is that learning the spatial support of each token improves faithfulness of attributions, which improves vision transformers across any tasks relying on attributions for interpretability or learning. Further, ToMATOS shows improvements in both within-image object grouping and cross-image part correspondence, while preserving classification accuracy and a predictable token budget at an order of magnitude less processing time than state-of-the-art approaches for region-aware tokenization.