acceptodds
Under review as a conference paper at ICLR 2027

ToMATOS: Tokens Meaningfully Aligned To Object Semantics

Abstract

Fixed-grid tokenization in vision transformers partitions images independent of their visual structure and disregard visual boundaries. This mismatch to actual image semantics limits their representation ability, with information of different object and background features bleeding into a single token. We introduce ToMATOS, an efficient, differentiable tokenization framework that represents images using content-adaptive region tokens with granular control. ToMATOS combines a lightweight feature extractor with a GPU-efficient feature clustering, resulting in a flexible and semantically meaningful tokenizer. We evaluate ToMATOS on supervised image classification and distilled vision foundation models of DINO variants over zero-shot segmentation tasks. In experiments ToMATOS demonstrates improved accuracy and attribution faithfulness. Notably, distilled region-token models outperform their foundation-model teachers, e.g. DINOv3, on sematic tasks highlighting the benefits of region-based tokens.

Primary area:
interpretability and explainable AI
Keywords:
Vision Transformers, Adaptive Tokenization, Region Tokens, Foundation model, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zfGAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper is about “content-adaptive visual tokenizers.” Instead of cutting the image into a fixed grid of patches like ViT, the method (ToMATOS) groups pixels into K regions that follow the image content and turns each region into one token. The number of tokens is fixed, but the shape and size of each region adapt to the image. The paper also aims to be faster and fully trainable, unlike earlier adaptive tokenizers (SuiT, SPiT, ∂HT), which are slow, rely on CPU superpixels or non-differentiable steps, or don’t let the user set the number of tokens.

A small CNN first computes a feature for every pixel. Each feature is split into two parts: a small clustering feature (6 channels, via PCA) and a token feature with a positional encoding. Flash-SNIC, a fast GPU version of the SNIC superpixel algorithm, places K seeds on a regular grid and grows regions from them. Each pixel joins the neighbouring region with the lowest cost, which mixes feature similarity and spatial distance (weighted by a compactness parameter m). To make this trainable, pixels near region borders are softly shared between the regions in their 3×3 neighbourhood using a softmax with temperature τ. Each region is then pooled into one token (mean and max pooling for classification, box pooling for distillation), and the tokens go into a standard ViT.

Critical strengths

Fast. The tokenizer runs fully on the GPU (Triton kernel), which makes it faster than other adaptive tokenizers: 12.7 ms vs 16–58 ms at Small size.

Controllable at inference. A single trained model can run with different numbers of regions and compactness values, which changes its speed–accuracy trade-off (Fig. 5). The paper does not explain how to choose these values in practice.

Transparent reporting. The authors report negative results (e.g. DINOv3 on INSID3, worse sufficiency than SuiT) and state how qualitative examples were selected. The implementation details are thorough.

Critical weaknesses

1: the core motivation is assumed, not shown, (or likely overstated.)

The paper treats object-aligned tokens as a goal and claims that patches mixing objects (“bleeding”) hurt the representation, but this is never tested. In a ViT, self-attention shares information across the whole image. That shared context is a main reason ViTs generalize well (e.g. to new objects, or to downstream uses such as robot policies), so a patch covering two objects is not obviously a problem. The evidence supports a narrower claim: sharper boundaries for attribution maps and masks at a fixed token budget. Why it matters: the paper’s significance rests on the broader claim. (See Q1, Q5.)

2: the reported gains cannot be attributed to the tokenizer.

Classification. ToMATOS uses a RegNetY-16GF distillation token, but the ViT baselines do not (App. B.1). DeiT with the same teacher reaches 81.2 (Small) and 83.4 (Base), which would make ToMATOS a tie (81.24) and a loss (82.66). Results come from single runs with gaps of a few tenths. Segmentation. The students use more tokens than the DINOv1/v3 teachers (1681 vs 1296). TokenCut is run without its bilateral solver. For DINOv3, pooling teacher features onto regions gives +0.04 IoU against +4.2 for the student (App. D.2), so the gain may come from the extra distillation training rather than from the regions. Why it matters: without matched patch-token baselines, the main experimental claims are not supported. (See Q3, Q4.)

Questions for authors
  1. Can you give direct evidence that tokens mixing several objects hurt representation quality or generalization, beyond boundary-sensitive metrics? Otherwise, would you narrow the claim to attribution and mask sharpness at a fixed token budget?

  2. How sensitive are the results to K, m, τ, the number of sweeps and the PCA size? Why are evaluation settings tuned per model (e.g. m = 0.05 vs 1.0 in Table C.1)? Given the content-adaptivity motivation, why is K fixed for all images, and how should a user choose it?

3.Can you report a patch ViT trained with exactly the same recipe (same RegNetY teacher, augmentation, epochs and initialization)? Were SuiT, SPiT and ∂HT trained with the teacher too? Can you report mean ± std over several seeds, and clarify the initialization of the Base model (App. B.1 vs Table B.1)?

  1. Can you compare against a patch-token student distilled with the same recipe and token count, and report TokenCut with its bilateral solver?

5.What is the expected benefit for the community over standard encoders? Can you compare at matched compute against a patch ViT with fewer tokens, and against simple token-merging or clustering baselines on a pretrained DINO or SigLIP2 model?

Reviewer_8yr2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Standard vision transformers split images into a grid of square patches (e.g. 16x16) uniformly. These slices arbitrarily cut across object boundaries and mixes distinct visual features into single tokens, which limits representation quality and interpretability. There exist alternatives to this rigid patch split, but these alternatives suffer from non-differentiability in grouping (making it more difficult to optimize models making use of them), are slowed down due to having a CPU-bound implementation, or are unable to select a fixed token budget. This paper aims provide a non-rigid patch split methodology while addressing these issues.

Critical strengths
  • Control over token count after patching: An explicit benefit of uniform patching is the guarantee of a certain number of tokens / compute budget being used during image processing. Baseline methods with non-uniform or adaptive patch splitting suffer from the inability to select a number of tokens to control compute/memory budget. TOMATOS' seed lattice spacing initialization guarantees the requested number of tokens K, giving users control independent of image input resolution.

  • GPU accelerated clustering: This work implements Flash-SNIC using a triton kernel, which iterates over the feature map with a fixed budget. This reduces the bottleneck from other CPU-bound clustering baselines.

  • Generated patches differentiability: Instead of rigidly assigning a nondifferentiable cluster id to every pixel, this work softens the assignment using a 3x3 grid incorporating neighboring pixel information. This creates a gradient of patching with respect to feature information, allowing the patching itself to be optimized during training.

Critical weaknesses
  • Assignment gradients only present at boundaries: All slots in a pixel's local 3x3 neighborhood could be assigned to the same region label when they're deep within the region's interior. In these scenarios the soft assignment collapses to a single region and there are no useful gradients. Assignment learning is strictly learned in terms of shifting existing borders and can't dynamically discover, split, or spawn new clusters from region interiors.

  • Heuristic assignment after fixed budget: After Flash-SNIC terminates (with its fixed set of iterations), pixels left unassigned are attached to the nearest seed in terms of 2D coordinates. Any learned feature similarities are then discarded and creates a discontinuity in assignments.

Questions for authors
  • How did authors select 3x3 for soft assignment? Why not a larger grid?
  • What is the average area of regions? What about aspect ratios?
Reviewer_n3H9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed a new framework called TOMATOS, a content-adaptive tokenization method that can replace the traditional fixed-grid-based ViTs. Their motivation is clear: the patch-based tokenization splits images on an arbitrary grid, causing object boundaries to fragment and unrelated object/background features to bleed into single tokens, which largely hurts the spatial semantic representation and model interpretability. Their proposed framework can now perform end-to-end tokenization during model training, as opposed to a separate step. The key contribution is the GPU-efficient clustering mechanism inspired by SNIC and the soft alignment to make the whole formulation fully differentiable. The key results are strong.

Critical strengths

The end2nd differentiable region tokenization with explicit control is a strong technical contribution, plus their interpretability and localization make the whole story very compelling. The paper also showed that DINO models can be distilled into region-token students, which leads to substantial gains on the zero-shot segmentation task, which is a very strong practical downstream application.

The other experimental results are diverse, i.e.. accuracy, latency, semantic boundary alignment, and attribution faithfulness evaluations are thorough.

Critical weaknesses

The overhead of dense feature extraction and Triton-kernel clustering makes it slower than simple guid-based ViTs, maybe more discussion on the runtime latency is helpful.

A deeper discussion on why DINOv3 (Table 3) student has a big performance drop (compared to the teacher) for the one-shot INSID3 task would strengthen the insights and analysis. I don't fully understand the analysis in lines 451-456.

Questions for authors

The DINOv3 drop is still unclear. Given the teacher's strong performance on dense feature representation, why does the proposed approach degrade cross-image correspondence? Can any extra distillation loss function be imposed to do a better job?

I like the overall idea, but it's less clear how critical the soft assignment is for the whole framework.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Reviewers,

We thank the reviewers for their careful reading and constructive suggestions. Reflecting the rebuttal, we have updated the paper draft with all changes marked in green and summarize the discussions below.

How we addressed the main concerns

1. Separating region tokenization from distillation (zfGA W2, Q3-Q4; n3H9 W2). We now show that with the same distillation pipeline, ToMATOS improves IoU over the patch student in all six TokenCut comparisons across three datasets and two teacher groups (+0.9 to +6.2 points). In the INSID3 evaluation settings, ToMATOS also improves one-shot part correspondence by 2.11 points for DINOv2 and 2.33 for DINOv3. Both DINOv3 students remain below their teacher, while ToMATOS student still outperforms the patch student decoupling the impact of distillation and region tokens that was asked. Reviewer zfGA acknowledged that these controls clarified the contribution of region tokens and raised their score.

2. Practical benefit and runtime (zfGA W1, Q1, Q5; n3H9 W1). On DINOv3/ECSSD at 1,296 tokens, ToMATOS gains 2.75 IoU for 4.3 ms over the patch student. While bilateral refinement gives the patch student a small 1.59-point lead over ToMATOS it almost doubles runtime per image. Among the adaptive tokenizers evaluated, ToMATOS has the lowest measured latency: 3.6/5.0 ms for Small/Base versus 11.5/12.7 ms for SuiT as the next fastest option, albeit not differentiable.

3. What regions capture (zfGA W1, Q5). At tested 1,024 tokens, ToMATOS improves over both ToMe and k-means in IoU and mask boundary F-score.These regions define the tokens used by the classifier. At the same nominal masking fractions, removing highly attributed content lowers confidence by about 9.5 percentage points more for ToMATOS than for the patch ViT, meaning that our region tokens succinctly capture the main information necessary for the downstream task.
4. How the partition is learned (8yr2 W1-W2, Q1-Q2; n3H9 Q2; zfGA Q2). We clarified how soft assignments let task gradients move region boundaries while the interior pixels still receive gradient updates through the pooling of token features within a region. We also reported the asked region-size and aspect-ratio distributions, sensitivity to token count and compactness, and number of iterations checks, giving a better analytical understanding of how regions are shaped.

Overall, we believe the strongest advantage of ToMATOS is that learning the spatial support of each token improves faithfulness of attributions, which improves vision transformers across any tasks relying on attributions for interpretability or learning. Further, ToMATOS shows improvements in both within-image object grouping and cross-image part correspondence, while preserving classification accuracy and a predictable token budget at an order of magnitude less processing time than state-of-the-art approaches for region-aware tokenization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.