acceptodds
Under review as a conference paper at ICLR 2027

COMiT: Learning Structured Visual Tokens through Sequential Communication

Abstract

Discrete image tokenizers provide a sequential interface for vision and multimodal models, but are typically optimized for reconstruction or compression and therefore tend to encode local appearance rather than object-level structure. We introduce COMiT, a communication-inspired framework for learning structured discrete visual representations. COMiT constructs a fixed-length latent message through sequential visual observations: at each step, a transformer processes a localized image crop and updates, refines, and reorganizes the existing token sequence. After several iterations, the resulting message conditions a flow-matching decoder that reconstructs the complete image. The encoder and decoder are implemented within a single transformer and trained end-to-end using flow-matching reconstruction and semantic representation-alignment objectives. COMiT substantially improves compositional generalization and relational reasoning over prior methods. Our experiments show that, while semantic alignment helps ground the representation, attentive sequential tokenization is critical for inducing more interpretable, objectcentric token structures.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Discrete visual representations, image tokenization, representation learning, vision transformers, generative modeling, flow matching, compositional generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NJgpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method to learn semantic-aware discrete visual tokens. The main objective is to have discrete tokens that capture meaningful objects and concepts rather than focusing solely on image reconstruction. The main contribution is a sequential communication process, where the encoder progressively extracts information from image crops and accumulates it into discrete tokens. The decoding is then performed through flow matching.

Critical strengths

The idea of sequential communication is intuitive and makes sense to learn structured visual representation without object-level supervision.

Additionally, the fact that object centric tokens emerge from the sequential information accumulation highlights the good design of the method.

The paper is nicely written, the storyline is clear and easy to follow.

Critical weaknesses

A lot of computational overhead is introduced (the sequence modeling requires multiple forwards, do does the flow matching decoding, and at training the DINO forward is also required). It would be interesting to have an analysis of the efficiency of the method compared to other tokenizers.

Additionally, although the authors provide an ablation of semantic distillation, the fact that DINO features are distilled during training makes hard the understanding of whether the semantic structure emerged from sequential communication or just by the distillation process.

Questions for authors

Could the authors provide a comparison of training or at least inference costs against existing image tokenizers?

To what extent do the object-centric properties emerge without semantic distillation from pretrained models? Could the authors report the object-localization performance of the model trained without the distillation?

While the authors investigate how semantic performance evolves with the number of image crops, could they also report reconstruction fidelity as a function of the number of communication steps, to better characterize the trade-off between semantic information, reconstruction quality, and computational cost? It would be interesting to see if improving semantic performance always increases reconstruction or not.

4 comments
AuthorsOpenReview ↗

Rebuttal 1/3

We sincerely thank the reviewer for the positive and encouraging review. We are glad that the reviewer finds sequential communication an intuitive way to learn structured visual representations without object-level supervision, sees the emergence of object-centric tokens from sequential information accumulation as a sign of good design, and finds the paper nicely written with a clear storyline. We answer each question below; all new material in the revised PDF is shown in blue.

Q1: How costly is COMiT compared with other tokenizers? (Weakness 1, Question 1)

A1: Thank you for raising this point; cost is an important practical aspect, and the revision now includes a dedicated analysis. In short, COMiT encodes an image with a single forward pass, COMiT-B and COMiT-L decode at about the same cost per step as FlexTok-d12-d12 and FlexTok-d18-d18, and FlexTok-d18-d18 requires at least 2.4 times the total training compute of COMiT-L, which outperforms it on all semantic benchmarks.

We compare COMiT with FlexTok, the prior tokenizer with the same message length, vocabulary and type of flow-based decoder. To use identical assumptions for both methods, we count FLOPs analytically from the transformer passes that each method requires and the tokens in each pass, excluding the frozen VAEs of both.

Inference. A downstream model consumes the message, and with the default single global crop, used for all probing results, COMiT produces it with one pass of the shared transformer. This pass costs about 1.6 times a FlexTok encoder pass of the same depth and width, because it also processes the buffer tokens in which the new message is written. Even so, COMiT-B, with half the encoding FLOPs of FlexTok-d18, outperforms every FlexTok variant on IN100 and MSCOCO (Table 4). Sequential local crops are an optional test-time knob, each adding one encoder pass over a shorter sequence. Flow-matching decoding is shared by both methods (FlexTok uses a rectified-flow decoder), and it is needed only to reconstruct images, not for understanding. Per decoding step, COMiT-B costs the same as FlexTok-d12-d12, and COMiT-L costs 6% more than FlexTok-d18-d18 and 3.4 times less than FlexTok-d18-d28.

Training. Using DINOv2 at training time is standard for tokenizers with diffusion-transformer decoders: FlexTok also applies REPA, with a DINOv2-L teacher that is about 3.5 times as expensive as our DINOv2-B. In COMiT, a single DINOv2-B pass provides the targets of both REPA and SREPA and accounts for only 6% (COMiT-B) and 2% (COMiT-L) of the training FLOPs. Sequential encoding adds little to the training cost: since the number of crops is sampled with mode 1 and gradients are stopped through all but the last update, a training step contains on average 1.8 encoder passes, most of them over the short sequences of local crops, and at most two of them receive gradients. COMiT is trained in a single stage with one network and a frozen off-the-shelf VAE, without a discriminator, perceptual loss, proxy codes or decoder fine-tuning, whereas FlexTok first trains its own VAE and TiTok requires a separate decoder fine-tuning stage. Per training image, COMiT-L costs at most 1.23 times as much as FlexTok-d18-d18 and at most 0.55 times as much as FlexTok-d18-d28. Since COMiT is trained on about 256M images (200 ImageNet epochs) and FlexTok on about 781M, FlexTok-d18-d18 and FlexTok-d18-d28 require in total at least 2.4 and 5.5 times the training compute of COMiT-L, which outperforms both on all semantic benchmarks.

Overall, sequential communication does not make COMiT costly: it encodes in a single pass, its decoding cost per step is in the range of FlexTok's, and it reaches stronger semantics with less total training compute. We added this analysis as a new paragraph "Computational cost" in Appendix A.

AuthorsOpenReview ↗

Rebuttal 2/3

Q2: Does the structure come from distillation rather than sequential communication? (Weakness 2, Question 2)

A2: We thank the reviewer for this insightful question, which goes to the heart of our claim. The object-aligned structure is produced by sequential communication, not by distillation: removing message distillation barely changes it, removing the local crops strongly degrades it although all distillation is kept, and a model trained without any distillation still learns object-aligned tokens.

It helps to distinguish the two uses of DINOv2 in COMiT. SREPA distills the global DINOv2 [CLS] feature into the message tokens. REPA [Yu et al., 2025] aligns intermediate image tokens of the decoding pathway with DINOv2 patch features; its loss is never applied to the message, and it is a standard convergence aid for diffusion transformers that prior tokenizers such as FlexTok use as well. The paper already contains the controlled comparison that swaps these two ingredients in otherwise identical COMiT-B models (same architecture, data and schedule, single global crop at test time; Appendices D.1 and D.5):

ModelMessage distillation (SREPA)Sequential local cropsCSSD token mIoU [95% CI]Large COCO objects localized by one token
COMiT-B (full)✓✓0.527 [0.509, 0.546]25.65%
COMiT-B, no SREPA–✓0.507 [0.490, 0.525]23.48%
COMiT-B, no local crops✓–0.338 [0.324, 0.352]17.39%
FlexTok d18-d18 (860M, REPA with DINOv2-L)–––21.74%

The model without local crops keeps all DINOv2 supervision, yet its token mIoU drops by 0.189 and it falls below FlexTok on COCO. Conversely, the model without SREPA, whose supervision matches FlexTok's (REPA only, with a smaller teacher), loses only 0.020 in mIoU. On large objects in multi-object COCO images, it localizes as many objects with a single token as FlexTok-d18-d18 (23.48% vs. 21.74%, same mean best-token IoU of 0.386) with about a fifth of the parameters (174M vs. 860M), and its best matches are spread over twice as many token indices (117 vs. 59 of 256). SREPA therefore mainly determines what semantic content is accessible from the message (IN100: 82.91 vs. 72.26), which is its intended role, while sequential communication determines how this content is organized into object-aligned tokens.

Removing REPA as well is not feasible at ImageNet scale within the discussion period, and such a run would also be hard to interpret. REPA was introduced because it accelerates diffusion-transformer training by more than an order of magnitude, and FlexTok reports that removing it degrades the rFID of its d12-d12 model from 5.98 to 23.28 at the same budget, so a no-REPA run would conflate the absence of a teacher with under-training. To answer the question directly instead, we analyzed a COMiT-B model that we had trained on CLEVR with the flow-matching loss only, i.e., with the REPA and SREPA losses switched off, so that no DINOv2 signal reached the model (16 message tokens, vocabulary of 1k, local crops only). Its message tokens attend to individual objects (Figure 9): most of the 16 tokens lock onto a single object, while a few cover the background. Using the object positions provided with CLEVR, the attention peak of some token falls on 47.7% of the objects, compared with 25.4% for random peaks, which is even higher than for the same model trained with REPA and SREPA (43.1%; best of the 12 layers for each model).

Object-aligned tokens therefore emerge from communication-based training itself: at ImageNet scale, DINOv2 alignment adds semantic content and speeds up training, but it is not the source of the structure. We added the CLEVR analysis as a new paragraph "Objectness without semantic distillation" in Appendix C (Figure 9). In the new Appendix D.7, we further show that removing a single token aligned with a CLEVR object selectively changes that object in the reconstruction (Table 15, Figure 11).

AuthorsOpenReview ↗

Rebuttal 3/3/

Q3: How does reconstruction change with the number of crops? (Question 3)

A3: Thank you for this suggestion; the analysis turned out to be quite informative, and we added it to the paper as the new Appendix B.6 (Table 7). In short, semantic performance does not always improve together with reconstruction, and this is by design: without the global crop, every crop adds unobserved content and both improve together, whereas once the global crop has covered the scene, further crops improve relational semantics at unchanged reconstruction.

We encode ImageNet1k validation images with COMiT-B (200 images) and COMiT-L (100 images) using the four cropping policies of Section 3.5, and decode the message after every communication step with the setting of Table 4, using the same decoding noise for all steps so that differences come from the message only. PSNR follows Table 4; on this subset, the global crop alone gives a PSNR of 17.80 for COMiT-B, in line with 17.75 on the full validation set. The results for COMiT-B are summarized below (COMiT-L shows the same trends, Table 7):

COMiT-B, number of local crops1234
PSNR, no global crop, raster-scan11.7712.6213.0214.15
PSNR, no global crop, adaptive12.2213.8214.7915.11
IN100 top-1, no global crop, raster-scan (Table 7)27.5546.9349.4859.93
PSNR, global crop + adaptive (17.80 with the global crop alone)17.9017.7917.7417.70

Without the global crop, every step adds unobserved content, and reconstruction and semantics improve together: from one to four raster-scan crops, PSNR increases by 2.4 (2.6 for COMiT-L), while the IN100 accuracy of messages encoded with the same policy rises from 27.6% to 59.9%. The adaptive policy is ahead at every budget (by up to 1.8 in PSNR for COMiT-B and 2.0 for COMiT-L) and needs fewer steps: three adaptive crops (14.79) already exceed four raster-scan crops (14.15). With the global crop first, the message already describes the whole image after one step. Further crops keep PSNR within 0.1 for COMiT-B and within 0.25 for COMiT-L (the first adaptive crop even raises it, to 17.90 for COMiT-B), while relational probing improves: the three-crop adaptive policy of Table 3 (global crop followed by two adaptive crops) raises Visual Genome top-1 from 52.11 to 55.00 at an unchanged PSNR (17.79 vs. 17.80).

This is the intended behavior of a fixed-capacity message (Section 3.1): once the scene is covered, further observations refine semantic and relational content rather than pixel detail. The same decoupling appears across model scale, where moving from COMiT-L to COMiT-XL improves rFID from 3.67 to 3.50 without improving the semantic scores (Table 4), and across bottleneck sizes, where PSNR increases by 1.65 while the probing accuracy stays between 80% and 82% (Figure 5c). Since every local crop adds one encoder pass (A1), the single global crop remains the most economical default, and local crops are an optional test-time knob for tasks that benefit from them. The full analysis, including COMiT-L, is in the new Appendix B.6.

Once more, many thanks for the encouraging assessment and the helpful questions. We hope that the answers and the revised appendix address them; if so, we would be grateful if the reviewer would consider updating the rating. We would be happy to clarify anything further.

I thank the authors for their additional experiments and explanations. I remain positive about the paper and therefore keep my positive grade.

Reviewer_pqRiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes COMiT, a 1D discrete image tokenizer. Instead of encoding the whole image in one pass, the model looks at a random sequence of crops and keeps rewriting a fixed-length FSQ message, and the same transformer then decodes the full image from this message with a flow-matching loss. A DINOv2 [CLS] alignment loss (SREPA) is added on the message tokens. Under frozen-token probing on IN100, a pair-disjoint MSCOCO split and Visual Genome relations, COMiT outperforms TiTok, FlexTok and SelfTok, while its reconstruction is weaker. The ablations suggest that SREPA mainly affects what information the message contains, and training with local crops mainly affects whether individual tokens align with objects.

Critical strengths
  1. The method is simple, using one network and a single flow-matching objective for both encoding and decoding, yet it achieves clearly better semantic probing than prior 1D tokenizers. The evaluation also goes beyond classification, covering compositional generalization and inter-object relations.

  2. The ablations are clear: removing SREPA mainly hurts classification probing, while removing local crops mainly hurts token–object alignment. This supports the claim that semantic alignment decides what the message encodes, and sequential observation decides where it is stored.

Critical weaknesses
  1. The main results use only a single global crop at test time, so the sequential “communication” mainly serves as a training-time bias. The no-local-crop ablation removes both recurrence and partial-view training, and the shared encoder/decoder is never compared with separate ones.

  2. The comparison with FlexTok is not under matched supervision, since SREPA distills DINOv2 [CLS] directly into the message. Without SREPA, COMiT-B falls below FlexTok on IN100, and it is unclear how much the results depend on the DINOv2 teacher.

  3. Reconstruction lags behind the larger FlexTok variants and SelfTok, which the paper treats as an intended trade-off. However, all understanding results come from frozen-token probes, and there is no downstream generation or MLLM experiment to support the motivation.

  4. Minor: Sec. 3.1 suggests the stop-gradient encourages greedy token use, but Appendix D.3 and Table 10 do not support this, as backpropagating through more updates gives similar token usage and no worse localization. Each tokenizer is also trained with a single run.

Questions for authors

See Weakness.

5 comments
AuthorsOpenReview ↗

Rebuttal 1/4

We thank the reviewer for the careful and thoughtful review. We appreciate that the reviewer finds the method simple yet clearly better at semantic probing than prior 1D tokenizers, values an evaluation that goes beyond classification, and finds that the ablations clearly separate what the message encodes from where it is stored. We answer each point below; all new material in the revised PDF is shown in blue.

Q1: Is sequential communication more than a training-time bias? (Weakness 1)

A1: We thank the reviewer for this thought-provoking question. Sequential communication is first of all a training principle, but its effect is not limited to training: the structure it induces persists when a single global crop is used at test time, and the trained model remains a working sequential communicator that integrates partial views and benefits from additional crops.

The analogy with masked image modeling is helpful here: masking shapes the representation of MAE, although the encoder sees the full image at test time. Likewise, all our localization results (CSSD and multi-object COCO, Appendices D.1 and D.5) and the compositional gain on unseen MSCOCO object pairs (+2.32 points over the model trained without local crops, with an affine readout on the pair-disjoint split) are measured with a single global crop. The structure lives in the tokens, not in the test-time procedure.

At the same time, COMiT keeps its sequential ability at test time. When only partial views are available, it integrates them step by step: from one to four local crops, reconstruction PSNR improves by 2.4 and the IN100 accuracy of messages encoded with the same policy rises from 27.6% to 59.9% (Table 7). The adaptive policy selects informative views, so that three adaptive crops already exceed four raster-scan crops in reconstruction. When the global crop is available, further crops add capabilities without retraining: relational probing on Visual Genome improves from 52.11 to 55.00 with only two adaptive crops, at unchanged reconstruction (Appendix B.6), and to 55.78 with nine raster-scan crops (Table 3).

Sequential communication is thus both a training principle that shapes the tokens and a mechanism that remains usable at test time. We keep the single global crop as the default because it gives the best overall trade-off at the lowest cost, one forward pass (Table 3, Appendix A), and the new Appendix B.6 (Table 7) shows how reconstruction and semantics evolve over the communication steps.

Q2: Is the effect due to recurrence or to partial-view training? (Weakness 1)

A2: This is a subtle and insightful question. In COMiT, recurrence and partial views are two sides of one mechanism, and it is their combination, the recurrent integration of partial views, that produces object-aligned tokens. A single partial view cannot describe the image, so partial views need recurrence to be integrated, while recurrence over the full view has nothing to integrate. Partial views also add no information: the local crops are windows of the same image at the same resolution, so the difference lies entirely in integrating the content over steps.

We can nevertheless compare independently trained checkpoints that differ in how the communication is set up (Appendices D.3 and D.4, plus our new evaluation of the fixed-number variant; CSSD token mIoU, single global crop at test time):

COMiT-B trainingCSSD token mIoU
Recurrent partial views, randomized number, 1 backpropagated update (default)0.527 [0.509, 0.546]
Recurrent partial views, randomized number, up to 4 backpropagated updates0.566 [0.545, 0.587]
Recurrent partial views, fixed number of 4 (new)0.552 [0.531, 0.571]
No partial views, no recurrence (no local crops)0.338 [0.324, 0.352]

All three recurrent variants, trained independently with different schedules and gradient depths, localize objects well, whereas the model without partial views localizes far less, at comparable training compute per image. The randomized protocol, in turn, drives semantic accessibility: with a fixed number of crops, localization stays high, but the message is pre-allocated to the expected crops (Table 11) and carries less accessible category information (Table 12).

A model trained with partial views but without recurrence (e.g., reconstructing the full image from a single random crop) would isolate the remaining factor, although such a model would lose the ability to benefit from aggregating additional crops at test time. It requires a new full ImageNet training, which is not feasible within the discussion period.

Taken together, the recurrent integration of partial views robustly drives localization, while randomizing the number of views makes the encoded content accessible; the underlying controls are described in Appendices D.1, D.3 and D.4.

AuthorsOpenReview ↗

Rebuttal 2/4

Q3: How does the shared encoder/decoder compare with separate networks? (Weakness 1)

A3: Sharing is a design choice motivated by the symmetry of the communication game; our claims about structure do not depend on it. A matched-budget comparison with separate networks requires additional ImageNet trainings, which is not feasible within the discussion period. All controlled ablations above share parameters, so the object-aligned structure is established within the shared design.

The shared design has two properties that we find valuable. First, the network reads and writes in one language: every recurrent update reads the previous message before rewriting it, so the encoder is already a listener, and with one network the same layers interpret the message during decoding and during the next encoding step. Second, there is no hand-designed capacity split between encoder and decoder (FlexTok, for instance, offers d18-d18 and d18-d28); the shared network allocates capacity between the two roles itself.

We therefore regard parameter sharing as a design that mirrors the symmetry of the communication game, not as the source of our results, which rest on controls within the shared design. Its motivation is given in the paragraph "Unified encoder and decoder" of Appendix A.

Q4: How much do the results depend on the DINOv2 teacher? (Weakness 2)

A4: Thank you for this important question. The object-aligned organization of the tokens does not depend on the teacher, whereas the amount of accessible category information does, by design.

Organization. Without SREPA, COMiT-B is trained under the same kind of supervision as FlexTok (REPA only) and with a smaller teacher (DINOv2-B instead of DINOv2-L). Its CSSD localization barely changes (0.507 vs. 0.527), and on multi-object COCO images it localizes as many large objects with a single token as FlexTok-d18-d18 (23.48% vs. 21.74%, same mean IoU) with a fifth of the parameters (Appendix D.5). Going further, we analyzed a COMiT model trained on CLEVR with flow matching only, without any DINOv2 supervision. Its tokens still attend to individual objects: the attention peak of some token falls on 47.7% of the objects, compared with 25.4% for random peaks and 43.1% for the same model trained with REPA and SREPA (Figure 9). Removing the token aligned with an object selectively changes that object in the reconstruction (Table 15, Figure 11).

Content. The IN100 comparison without SREPA (72.26 vs. 80.25 for FlexTok-d12-d12) is not matched: FlexTok-d12-d12 is trained on three times more images (about 781M vs. 256M) with a 3.5 times more expensive teacher. SREPA is part of COMiT; it reuses the teacher pass that REPA already requires (the teacher accounts for 2–6% of the training FLOPs, Appendix A), and it lets COMiT-B outperform every FlexTok variant on IN100 (82.91 vs. at most 81.54) with 174M parameters. Moreover, the mechanism adds semantics beyond the teacher: with identical REPA + SREPA, communication-based training adds +1.97 points on IN100 and +2.32 points on unseen MSCOCO compositions over a single-pass model (affine readout on the pair-disjoint split, five probe seeds).

As for any distilled tokenizer, including FlexTok, the amount of category information depends on distillation; the object-aligned structure, which is our main claim, does not. To make this explicit, the revision adds the CLEVR analysis without semantic distillation (new paragraph in Appendix C, Figure 9) and the token interventions (new Appendix D.7).

AuthorsOpenReview ↗

Rebuttal 3/4

Q5: Is the reconstruction trade-off justified without downstream experiments? (Weakness 3)

A5: We understand the reviewer's concern. The trade-off is moderate and suits the understanding-oriented use that COMiT targets: its reconstruction is competitive for a single-stage tokenizer and improves with scale, and frozen-token probing is the standard way to measure what a tokenizer delivers to downstream models.

Where reconstruction stands. COMiT-L's PSNR (17.81) is above that of every TiTok variant (15.60–17.52) and within 0.8 of every FlexTok variant (18.41–18.59). Its rFID (3.67; 3.50 for COMiT-XL) is better than that of FlexTok-d12-d12 (4.20) and ALIT (7.39), and behind tokenizers that rely on two-stage training with decoder fine-tuning (TiTok), larger models (FlexTok-d18) or larger data mixtures (SelfTok). COMiT reaches this in a single training stage, without adversarial or perceptual losses or decoder fine-tuning, and with three times fewer training images than FlexTok (Appendix A). Reconstruction improves steadily with model scale (rFID 11.06 → 3.67 → 3.50, Table 4) and bottleneck size (Figure 5c), so we see the gap as a matter of training budget rather than a limitation of the approach.

Why frozen probes. Frozen-token probing is the standard protocol for representation quality, as linear probing is for self-supervised learning: it measures the information delivered by the tokenizer without letting a large downstream model rewrite it. Importantly, our Visual Genome probe already is a lightweight cross-modal model: frozen T5 embeddings of the subject, predicate and object are appended to the frozen image tokens and processed jointly by a shallow Transformer (Appendix B), much like a multimodal model processes visual tokens together with text. In this setting, a small-scale proxy for how a multimodal model would use the tokens, COMiT-L achieves the best accuracy of all tokenizers (Table 4). Our other probes also go well beyond classification: compositional generalization to unseen object pairs (MSCOCO), dense segmentation (Appendix D.6), single-token object localization (Appendices D.1 and D.5), and attention-free readouts (Appendix D.2).

Downstream models. A conclusive generation or MLLM comparison requires training a large model on top of each tokenizer with large-scale paired data, which is beyond the compute and time of the discussion period.

COMiT thus trades a moderate amount of reconstruction fidelity for substantially more accessible and better organized semantics, which is the right trade-off for an understanding-oriented tokenizer. We will make this scope explicit; improved reconstruction and integration into multimodal models are discussed as future work in Appendix G, and the new Appendix B.6 shows that semantic gains do not require better reconstruction.

Q6: Does the stop-gradient really encourage greedy token use? (Weakness 4)

A6: We thank the reviewer for the careful reading, and we agree. The stop-gradient contributes only slightly; greedy token use is induced mainly by the randomized number of crops.

The full-scale controls show this directly: with a fixed number of crops, 36.7% of the message positions are under-used after the first local crop, whereas both randomized models leave at most 5.9% under-used (Table 11). The stop-gradient also contributes, but only slightly: with a single backpropagated update, slightly fewer positions are under-used after the global crop (0.0% vs. 0.4%) and after the first local crop (4.7% vs. 5.5%), i.e., where greedy use matters most, while backpropagating through up to four updates is ahead after two and three crops. Limiting backpropagation is therefore primarily a matter of efficiency, and allowing more updates is beneficial when memory permits: the variant with up to four backpropagated updates localizes objects even better (CSSD mIoU 0.566 vs. 0.527) and makes category information more accessible (Table 12).

The randomized number of crops is therefore the main driver of greedy token use, and the stop-gradient mainly enables efficient training; we have revised the corresponding sentence in Section 3.1 to say so (in blue in the revised PDF).

Reviewer_PJd4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

COMiT learns fixed-length discrete visual messages by sequentially incorporating local image crops. A shared transformer performs both message updates and flow-matching reconstruction, supported by DINOv2-based representation alignment. The paper evaluates semantic recognition, compositional generalization, relational prediction, and token localization, showing improved semantic accessibility and object alignment over several existing tokenizers.

Critical strengths
  1. The paper provides a coherent approach to learning structured visual tokens through recurrent updates of a fixed-capacity message. Its ablations offer useful evidence that semantic alignment and local-crop training have different effects: removing local crops substantially reduces token localization, whereas removing SREPA mainly affects recognition accuracy.
  2. Evaluation extends beyond reconstruction and includes meaningful supplementary controls: affine probes, crop-count randomization, backpropagation-depth ablations, and multi-object localization with confidence intervals. COMiT-L achieves the strongest semantic probing results in Table 4, and its localization advantage over FlexTok supports the practical relevance of the learned structure.
Critical weaknesses
  1. The specific benefit of sequential communication remains incompletely isolated. Removing local crops simultaneously removes recurrent refinement, additional visual observations, and computation. The improvements cannot be attributed specifically to sequential message rewriting. The benefit of sharing encoder and decoder parameters is also not directly tested.
  2. The evaluations only partially establish the claimed structure. Visual Genome negatives are sampled from other images, allowing object-category recognition to distinguish triplets without resolving their predicates. The results support semantic accessibility and spatial alignment more directly than general relational reasoning.
  3. Cross-tokenizer semantic gains are confounded by different alignment objectives. COMiT uses both REPA and SREPA, whereas FlexTok uses REPA alone. Consequently, the semantic advantage cannot be fully attributed to the proposed tokenization mechanism.
  4. The practical trade-off is incompletely quantified. COMiT improves semantic probing while sacrificing reconstruction quality, but the comparison does not report training cost or encoding throughput. Parameter counts alone do not characterize the cost of recurrent encoding, leaving the practical value of the representation–reconstruction trade-off difficult to assess.
Questions for authors
  1. Do the localization and compositional gains persist against compute-matched repeated-global encoding or non-recurrent aggregation of the same local crops?
  2. How does Visual Genome performance change with negatives that preserve the subject and object categories but reverse subject–object roles?
  3. Does ablating or replacing an object-aligned token selectively affect that object’s reconstruction or prediction?
  4. What is the effect of using separate encoder and decoder networks under a matched parameter budget?
5 comments
AuthorsOpenReview ↗

Rebuttal 1/4

We thank the reviewer for the positive and constructive review. We are glad that the reviewer finds COMiT a coherent approach to learning structured visual tokens, considers our ablations and supplementary controls meaningful, and notes that COMiT-L achieves the strongest semantic probing results and a localization advantage over FlexTok. The reviewer's questions prompted several new analyses, which we summarize below; all new material in the revised PDF is shown in blue.

Q1: Are the gains due to sequential rewriting rather than extra observations or compute? (Weakness 1, Question 1)

A1: We thank the reviewer for this question, which is central to our claims. The gains come from sequential rewriting: the local crops add no new observations, the model trained without them uses comparable compute, and every model trained with recurrent rewriting localizes objects, while the only model trained without it localizes far less.

Observations. The local crops are 96×96 windows of the same 256×256 image at the same resolution, so they contain no pixel that the global crop does not contain. The model without local crops encodes the entire image in a single pass; it therefore already is a non-recurrent aggregation of exactly the content covered by the local crops. Partial views therefore add no information. What they change is how information reaches the message: piece by piece, so that it has to be integrated and placed by rewriting.

Compute. Per training image, the model without local crops spends about the same compute as COMiT-B: it encodes the full image with gradients at almost every training step, whereas COMiT mostly encodes the short sequences of local crops, of which only the last receives gradients. Both models are trained on the same images for the same number of steps, with the same batch size and seed, and both are evaluated with the same single global crop, so the test-time computation is identical as well.

Recurrent rewriting. Across three independently trained checkpoints with different schedules and gradient depths, token–object localization is consistently high, whereas the model without local crops, trained with the same REPA + SREPA supervision, localizes far less (CSSD, single global crop, 95% bootstrap intervals):

COMiT-B trainingCSSD token mIoU
Recurrent local crops, randomized number, 1 backpropagated update (default)0.527 [0.509, 0.546]
Recurrent local crops, randomized number, up to 4 backpropagated updates0.566 [0.545, 0.587]
Recurrent local crops, fixed number of 4 (new)0.552 [0.531, 0.571]
No local crops (same REPA + SREPA supervision)0.338 [0.324, 0.352]

The benefit is not limited to localization. On the pair-disjoint MSCOCO split, where every test object pair is unseen during probe training, an affine readout of the message (five probe seeds) reaches 30.90 ± 0.38 for COMiT-B vs. 28.59 ± 0.72 without local crops (paired gain +2.32 ± 0.53, positive for every seed), again with identical supervision and the single global crop at test time.

The control that feeds the global crop K times at matched compute would require new full ImageNet trainings, which is not feasible within the discussion period. We expect it to behave like the model without local crops: when every step observes the same complete view, there is nothing to integrate, and the recurrence reduces to refining a single view. This is consistent with the new Appendix B.6, where steps that follow the global crop leave the reconstruction unchanged, whereas partial views are integrated step by step.

Once observations and compute are held fixed, what remains is the recurrent rewriting of partial views, and it is this mechanism that produces object-aligned tokens and the compositional gains. The underlying controls are described in Appendices D.1, D.3 and D.4, and the new Appendix B.6 shows how partial views are integrated over communication steps.

AuthorsOpenReview ↗

Rebuttal 2/4

Q2: What is the benefit of sharing the encoder and decoder? (Weakness 1, Question 4)

A2: Sharing is a design choice motivated by the symmetry of the communication game; our claims about structure do not depend on it. A matched-budget comparison with separate networks requires additional ImageNet trainings, which is not feasible within the discussion period. All controlled ablations above share parameters, so the object-aligned structure is established within the shared design.

The shared design has two properties that we find valuable. First, the network reads and writes in one language. In COMiT, the encoder is already a listener, since every recurrent update reads the previous message before rewriting it. With a single network, the layers that interpret message tokens during decoding are the same layers that interpret them during the next encoding step, so whatever the network writes, it can read by construction; separate networks would have to duplicate this message-reading function. Second, there is no hand-designed capacity split: a separate design must fix how parameters are divided between encoder and decoder (FlexTok, for instance, offers d18-d18 and d18-d28), whereas a shared network allocates capacity between the two roles itself.

Parameter sharing is thus best seen as a design that mirrors the symmetry of the communication game, while our structural claims rest on controls within the shared design. Its motivation is given in the paragraph "Unified encoder and decoder" of Appendix A.

Q3: Does Visual Genome test relational reasoning, e.g., with role-swapped negatives? (Weakness 2, Question 2)

A3: We thank the reviewer for this excellent suggestion. We agree that with negatives sampled from other images, recognizing the object categories suffices to reject most negatives, so the standard VG accuracy measures relational semantics jointly with object recognition. We therefore evaluated the trained probes with role-swapped negatives. COMiT remains the best tokenizer when the roles of subject and object must be resolved, although role assignment is challenging for every tokenizer.

For a ground-truth relation (s, p, o), the role-swapped negative (o, p, s) keeps the subject and object categories and the predicate but swaps the roles. We exclude relations whose reversal may also be correct: symmetric predicates (e.g., "next to", "near"), relations whose subject and object share the same category, and relations whose reverse is also annotated in the same image; 91% of the validation relations are retained. We report two metrics: Hard-16, the 16-way task of Table 4 in which one of the 15 cross-image negatives is replaced by the role-swapped triplet (chance 6.25%), and swap accuracy, the fraction of relations for which the probe scores the correct triplet above its reversal (chance 50%).

TokenizerHard-16 ↑Swap acc. ↑
TiTok-L20.155.2
ALIT25.657.2
SelfTok-51227.858.1
FlexTok d18-d1839.559.7
COMiT-B39.960.9
COMiT-L40.061.0
COMiT-XL40.161.4

All three COMiT models outperform every baseline in both metrics, and the ordering of the tokenizer families is preserved. COMiT-B, which trails FlexTok-d18-d18 on the standard VG task (52.11 vs. 54.46, Table 4), is even slightly ahead of it here. All tokenizers score above chance in swap accuracy, so role information is accessible from their tokens, and COMiT attains the highest values (up to 61.4%). Accuracy is lower than in the standard task for every tokenizer, since a role-swapped negative can only be rejected by resolving which object is the subject.

The role-swap evaluation therefore confirms that COMiT's relational advantage holds even when recognizing the objects is not sufficient and the probe has to resolve which object plays which role.

AuthorsOpenReview ↗

Rebuttal 3/4

Q4: Are the gains over FlexTok due to the tokenization or to SREPA? (Weakness 3)

A4: Table 4 compares complete systems, and we appreciate the opportunity to disentangle the two factors. The object-aligned structure and the compositional gains come from the tokenization mechanism, while SREPA makes category information accessible, as controls that hold the supervision fixed show.

Same supervision, with and without communication. The full and no-local-crops COMiT-B models use identical REPA + SREPA objectives. Communication-based training adds +1.97 IN100 points (82.91 vs. 80.94), +2.32 points on unseen MSCOCO compositions (A1), and object-aligned tokens (CSSD mIoU 0.527 vs. 0.338; 25.65% vs. 17.39% of large COCO objects localized by one token). The model without local crops, i.e., a single-pass tokenizer with REPA + SREPA, reaches FlexTok-level IN100 accuracy (80.94 vs. 80.25–81.54) but not the structure, so the remaining gains require the tokenization mechanism.

Same supervision as FlexTok. COMiT-B without SREPA uses REPA only, as FlexTok does, and with the smaller DINOv2-B instead of FlexTok's DINOv2-L. On multi-object COCO images, it localizes as many large objects with a single token as FlexTok-d18-d18 (23.48% vs. 21.74%, same mean IoU) with a fifth of the parameters, and spreads its best matches over twice as many token indices (Appendix D.5). Even without any teacher, a COMiT model trained on CLEVR with flow matching only still learns object-aligned tokens (new paragraph in Appendix C).

SREPA is part of the method, and it is cheap. Aligning the aggregate message with a global descriptor is natural for a 1D message that describes the whole scene. SREPA reuses the DINOv2 forward pass that REPA already requires, with a smaller teacher than FlexTok's, and it makes semantic content accessible with three times fewer training images than FlexTok (Appendix A). Training FlexTok with SREPA would be an interesting complementary experiment, but it requires retraining FlexTok (about 781M training images), which is not feasible within the discussion period.

Hence, the object-aligned structure that distinguishes COMiT stems from communication-based tokenization, whereas SREPA adds accessible category information at a small cost. The revision adds the CLEVR analysis without semantic distillation (Appendix C) and the cost comparison with FlexTok (Appendix A).

Q5: What are the training and inference costs of COMiT? (Weakness 4)

A5: Thank you for pointing out this omission; the revision now includes a dedicated cost analysis (new paragraph "Computational cost" in Appendix A), which compares COMiT under identical assumptions with FlexTok, the prior tokenizer with the same message length, vocabulary and type of flow-based decoder. In short, COMiT encodes an image with a single forward pass, COMiT-B and COMiT-L decode at about the same cost per step as FlexTok-d12-d12 and FlexTok-d18-d18, and FlexTok-d18-d18 requires at least 2.4 times the total training compute of COMiT-L, which outperforms it on all semantic benchmarks.

Recurrent encoding is optional at test time: all main results use the single global crop, and each additional local crop adds one encoder pass over a shorter sequence. During training, the number of crops is sampled with mode 1 and gradients are stopped through all but the last update, so a training step contains on average 1.8 encoder passes, at most two of which receive gradients; training is a single stage with a frozen off-the-shelf VAE and uses three times fewer training images than FlexTok. Compared with FlexTok-d18-d18, COMiT-L therefore gains +4.3 IN100, +6.7 MSCOCO and +2.0 VG points with fewer parameters and less training compute, at the cost of 0.8 in PSNR and a higher rFID (3.67 vs. 1.61), which is still lower than that of FlexTok-d12-d12 (4.20) and ALIT (7.39).

For understanding-oriented use, where a downstream model consumes the tokens, COMiT provides substantially more accessible semantics for a comparable or lower computational budget, which is the trade-off we target. The full analysis is in the new paragraph "Computational cost" in Appendix A.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers for their thoughtful and constructive reviews. We are encouraged that the reviewers find sequential communication an intuitive and coherent way to learn structured visual tokens without object-level supervision (NJgp, PJd4), the method simple yet clearly better at semantic probing than prior 1D tokenizers (pqRi), the evaluation broad, beyond reconstruction and classification (PJd4, pqRi), the ablations and controls clear and meaningful (PJd4, pqRi), and the paper nicely written (NJgp).

We have uploaded a revised version of the paper, in which all changes are marked in blue. Following the reviewers' suggestions, the revision mainly adds new analyses to the appendix:

  • a comparison of the computational cost of COMiT and FlexTok (Appendix A, paragraph "Computational cost");
  • reconstruction after every communication step and its relation to semantic probing (Appendix B.6, Table 7);
  • object-aligned tokens in a model trained without any semantic distillation (Appendix C, Figure 9);
  • token interventions showing that object-aligned tokens carry the information about their object, including object-specific interventions in multi-object scenes (Appendix D.7, Tables 14 and 15, Figures 10 and 11);

We address each reviewer’s questions in the individual responses. Since they largely overlap with the new material in the revised appendix, reviewers may also read the referenced appendix sections directly. This holds in particular for the qualitative results, which can only be shown in the revised PDF. To keep the responses quick to read, each answer states its main conclusion in bold, followed by the supporting details. We would be happy to discuss any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.