- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes a method to learn semantic-aware discrete visual tokens. The main objective is to have discrete tokens that capture meaningful objects and concepts rather than focusing solely on image reconstruction. The main contribution is a sequential communication process, where the encoder progressively extracts information from image crops and accumulates it into discrete tokens. The decoding is then performed through flow matching.
Critical strengths
The idea of sequential communication is intuitive and makes sense to learn structured visual representation without object-level supervision.
Additionally, the fact that object centric tokens emerge from the sequential information accumulation highlights the good design of the method.
The paper is nicely written, the storyline is clear and easy to follow.
Critical weaknesses
A lot of computational overhead is introduced (the sequence modeling requires multiple forwards, do does the flow matching decoding, and at training the DINO forward is also required). It would be interesting to have an analysis of the efficiency of the method compared to other tokenizers.
Additionally, although the authors provide an ablation of semantic distillation, the fact that DINO features are distilled during training makes hard the understanding of whether the semantic structure emerged from sequential communication or just by the distillation process.
Questions for authors
Could the authors provide a comparison of training or at least inference costs against existing image tokenizers?
To what extent do the object-centric properties emerge without semantic distillation from pretrained models? Could the authors report the object-localization performance of the model trained without the distillation?
While the authors investigate how semantic performance evolves with the number of image crops, could they also report reconstruction fidelity as a function of the number of communication steps, to better characterize the trade-off between semantic information, reconstruction quality, and computational cost? It would be interesting to see if improving semantic performance always increases reconstruction or not.
4 comments
Rebuttal 2/3
Q2: Does the structure come from distillation rather than sequential communication? (Weakness 2, Question 2)
A2: We thank the reviewer for this insightful question, which goes to the heart of our claim. The object-aligned structure is produced by sequential communication, not by distillation: removing message distillation barely changes it, removing the local crops strongly degrades it although all distillation is kept, and a model trained without any distillation still learns object-aligned tokens.
It helps to distinguish the two uses of DINOv2 in COMiT. SREPA distills the global DINOv2 [CLS] feature into the message tokens. REPA [Yu et al., 2025] aligns intermediate image tokens of the decoding pathway with DINOv2 patch features; its loss is never applied to the message, and it is a standard convergence aid for diffusion transformers that prior tokenizers such as FlexTok use as well. The paper already contains the controlled comparison that swaps these two ingredients in otherwise identical COMiT-B models (same architecture, data and schedule, single global crop at test time; Appendices D.1 and D.5):
| Model | Message distillation (SREPA) | Sequential local crops | CSSD token mIoU [95% CI] | Large COCO objects localized by one token |
|---|---|---|---|---|
| COMiT-B (full) | ✓ | ✓ | 0.527 [0.509, 0.546] | 25.65% |
| COMiT-B, no SREPA | – | ✓ | 0.507 [0.490, 0.525] | 23.48% |
| COMiT-B, no local crops | ✓ | – | 0.338 [0.324, 0.352] | 17.39% |
| FlexTok d18-d18 (860M, REPA with DINOv2-L) | – | – | – | 21.74% |
The model without local crops keeps all DINOv2 supervision, yet its token mIoU drops by 0.189 and it falls below FlexTok on COCO. Conversely, the model without SREPA, whose supervision matches FlexTok's (REPA only, with a smaller teacher), loses only 0.020 in mIoU. On large objects in multi-object COCO images, it localizes as many objects with a single token as FlexTok-d18-d18 (23.48% vs. 21.74%, same mean best-token IoU of 0.386) with about a fifth of the parameters (174M vs. 860M), and its best matches are spread over twice as many token indices (117 vs. 59 of 256). SREPA therefore mainly determines what semantic content is accessible from the message (IN100: 82.91 vs. 72.26), which is its intended role, while sequential communication determines how this content is organized into object-aligned tokens.
Removing REPA as well is not feasible at ImageNet scale within the discussion period, and such a run would also be hard to interpret. REPA was introduced because it accelerates diffusion-transformer training by more than an order of magnitude, and FlexTok reports that removing it degrades the rFID of its d12-d12 model from 5.98 to 23.28 at the same budget, so a no-REPA run would conflate the absence of a teacher with under-training. To answer the question directly instead, we analyzed a COMiT-B model that we had trained on CLEVR with the flow-matching loss only, i.e., with the REPA and SREPA losses switched off, so that no DINOv2 signal reached the model (16 message tokens, vocabulary of 1k, local crops only). Its message tokens attend to individual objects (Figure 9): most of the 16 tokens lock onto a single object, while a few cover the background. Using the object positions provided with CLEVR, the attention peak of some token falls on 47.7% of the objects, compared with 25.4% for random peaks, which is even higher than for the same model trained with REPA and SREPA (43.1%; best of the 12 layers for each model).
Object-aligned tokens therefore emerge from communication-based training itself: at ImageNet scale, DINOv2 alignment adds semantic content and speeds up training, but it is not the source of the structure. We added the CLEVR analysis as a new paragraph "Objectness without semantic distillation" in Appendix C (Figure 9). In the new Appendix D.7, we further show that removing a single token aligned with a CLEVR object selectively changes that object in the reconstruction (Table 15, Figure 11).
Rebuttal 3/3/
Q3: How does reconstruction change with the number of crops? (Question 3)
A3: Thank you for this suggestion; the analysis turned out to be quite informative, and we added it to the paper as the new Appendix B.6 (Table 7). In short, semantic performance does not always improve together with reconstruction, and this is by design: without the global crop, every crop adds unobserved content and both improve together, whereas once the global crop has covered the scene, further crops improve relational semantics at unchanged reconstruction.
We encode ImageNet1k validation images with COMiT-B (200 images) and COMiT-L (100 images) using the four cropping policies of Section 3.5, and decode the message after every communication step with the setting of Table 4, using the same decoding noise for all steps so that differences come from the message only. PSNR follows Table 4; on this subset, the global crop alone gives a PSNR of 17.80 for COMiT-B, in line with 17.75 on the full validation set. The results for COMiT-B are summarized below (COMiT-L shows the same trends, Table 7):
| COMiT-B, number of local crops | 1 | 2 | 3 | 4 |
|---|---|---|---|---|
| PSNR, no global crop, raster-scan | 11.77 | 12.62 | 13.02 | 14.15 |
| PSNR, no global crop, adaptive | 12.22 | 13.82 | 14.79 | 15.11 |
| IN100 top-1, no global crop, raster-scan (Table 7) | 27.55 | 46.93 | 49.48 | 59.93 |
| PSNR, global crop + adaptive (17.80 with the global crop alone) | 17.90 | 17.79 | 17.74 | 17.70 |
Without the global crop, every step adds unobserved content, and reconstruction and semantics improve together: from one to four raster-scan crops, PSNR increases by 2.4 (2.6 for COMiT-L), while the IN100 accuracy of messages encoded with the same policy rises from 27.6% to 59.9%. The adaptive policy is ahead at every budget (by up to 1.8 in PSNR for COMiT-B and 2.0 for COMiT-L) and needs fewer steps: three adaptive crops (14.79) already exceed four raster-scan crops (14.15). With the global crop first, the message already describes the whole image after one step. Further crops keep PSNR within 0.1 for COMiT-B and within 0.25 for COMiT-L (the first adaptive crop even raises it, to 17.90 for COMiT-B), while relational probing improves: the three-crop adaptive policy of Table 3 (global crop followed by two adaptive crops) raises Visual Genome top-1 from 52.11 to 55.00 at an unchanged PSNR (17.79 vs. 17.80).
This is the intended behavior of a fixed-capacity message (Section 3.1): once the scene is covered, further observations refine semantic and relational content rather than pixel detail. The same decoupling appears across model scale, where moving from COMiT-L to COMiT-XL improves rFID from 3.67 to 3.50 without improving the semantic scores (Table 4), and across bottleneck sizes, where PSNR increases by 1.65 while the probing accuracy stays between 80% and 82% (Figure 5c). Since every local crop adds one encoder pass (A1), the single global crop remains the most economical default, and local crops are an optional test-time knob for tasks that benefit from them. The full analysis, including COMiT-L, is in the new Appendix B.6.
Once more, many thanks for the encouraging assessment and the helpful questions. We hope that the answers and the revised appendix address them; if so, we would be grateful if the reviewer would consider updating the rating. We would be happy to clarify anything further.
I thank the authors for their additional experiments and explanations. I remain positive about the paper and therefore keep my positive grade.
Rebuttal 1/3
We sincerely thank the reviewer for the positive and encouraging review. We are glad that the reviewer finds sequential communication an intuitive way to learn structured visual representations without object-level supervision, sees the emergence of object-centric tokens from sequential information accumulation as a sign of good design, and finds the paper nicely written with a clear storyline. We answer each question below; all new material in the revised PDF is shown in blue.
Q1: How costly is COMiT compared with other tokenizers? (Weakness 1, Question 1)
A1: Thank you for raising this point; cost is an important practical aspect, and the revision now includes a dedicated analysis. In short, COMiT encodes an image with a single forward pass, COMiT-B and COMiT-L decode at about the same cost per step as FlexTok-d12-d12 and FlexTok-d18-d18, and FlexTok-d18-d18 requires at least 2.4 times the total training compute of COMiT-L, which outperforms it on all semantic benchmarks.
We compare COMiT with FlexTok, the prior tokenizer with the same message length, vocabulary and type of flow-based decoder. To use identical assumptions for both methods, we count FLOPs analytically from the transformer passes that each method requires and the tokens in each pass, excluding the frozen VAEs of both.
Inference. A downstream model consumes the message, and with the default single global crop, used for all probing results, COMiT produces it with one pass of the shared transformer. This pass costs about 1.6 times a FlexTok encoder pass of the same depth and width, because it also processes the buffer tokens in which the new message is written. Even so, COMiT-B, with half the encoding FLOPs of FlexTok-d18, outperforms every FlexTok variant on IN100 and MSCOCO (Table 4). Sequential local crops are an optional test-time knob, each adding one encoder pass over a shorter sequence. Flow-matching decoding is shared by both methods (FlexTok uses a rectified-flow decoder), and it is needed only to reconstruct images, not for understanding. Per decoding step, COMiT-B costs the same as FlexTok-d12-d12, and COMiT-L costs 6% more than FlexTok-d18-d18 and 3.4 times less than FlexTok-d18-d28.
Training. Using DINOv2 at training time is standard for tokenizers with diffusion-transformer decoders: FlexTok also applies REPA, with a DINOv2-L teacher that is about 3.5 times as expensive as our DINOv2-B. In COMiT, a single DINOv2-B pass provides the targets of both REPA and SREPA and accounts for only 6% (COMiT-B) and 2% (COMiT-L) of the training FLOPs. Sequential encoding adds little to the training cost: since the number of crops is sampled with mode 1 and gradients are stopped through all but the last update, a training step contains on average 1.8 encoder passes, most of them over the short sequences of local crops, and at most two of them receive gradients. COMiT is trained in a single stage with one network and a frozen off-the-shelf VAE, without a discriminator, perceptual loss, proxy codes or decoder fine-tuning, whereas FlexTok first trains its own VAE and TiTok requires a separate decoder fine-tuning stage. Per training image, COMiT-L costs at most 1.23 times as much as FlexTok-d18-d18 and at most 0.55 times as much as FlexTok-d18-d28. Since COMiT is trained on about 256M images (200 ImageNet epochs) and FlexTok on about 781M, FlexTok-d18-d18 and FlexTok-d18-d28 require in total at least 2.4 and 5.5 times the training compute of COMiT-L, which outperforms both on all semantic benchmarks.
Overall, sequential communication does not make COMiT costly: it encodes in a single pass, its decoding cost per step is in the range of FlexTok's, and it reaches stronger semantics with less total training compute. We added this analysis as a new paragraph "Computational cost" in Appendix A.