ToMATOS: Tokens Meaningfully Aligned To Object Semantics

Fixed-grid tokenization in vision transformers partitions images independent of their visual structure and disregard visual boundaries. This mismatch to actual image semantics limits their representation ability, with information of different object and background features bleeding into a single token. We introduce ToMATOS, an efficient, differentiable tokenization framework that represents images using content-adaptive region tokens with granular control. ToMATOS combines a lightweight feature extractor with a GPU-efficient feature clustering, resulting in a flexible and semantically meaningful tokenizer. We evaluate ToMATOS on supervised image classification and distilled vision foundation models of DINO variants over zero-shot segmentation tasks. In experiments ToMATOS demonstrates improved accuracy and attribution faithfulness. Notably, distilled region-token models outperform their foundation-model teachers, e.g. DINOv3, on sematic tasks highlighting the benefits of region-based tokens.