- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes GAMUT, a ViT-B/16 encoder trained jointly with SigLIP, DINO/iBOT and pixel-reconstruction objectives, so that a single 768-dimensional, uncompressed representation serves both understanding and diffusion-based generation. Training proceeds from a SigLIP2-feature MIM warm-up to joint pretraining and then decoder training. Latent channel regularization (LCR) prevents channel collapse, and latent interpolation decoding (LID) makes the decoder more robust to generated latents. GAMUT outperforms existing unified tokenizers on zero-shot classification and retrieval, ImageNet reconstruction and generation, and segmentation.
Critical strengths
The analysis of representation collapse is clear. Fig. 3 and Fig. 5 directly show the emergence of single-channel energy concentration and how LCR reverses it, and the observation in Fig. 4 that DINO collapses when trained from scratch is a useful reference. The rFID of 0.22 is strong, and the appendix provides thorough implementation details.
Critical weaknesses
- The description of VTP is inaccurate, and the core claim lacks direct validation. The paper states that VTP compresses high-dimensional features into a low-dimensional bottleneck and thereby sacrifices semantics, and attributes its gains over VTP to avoiding this compression. However, VTP’s image-text alignment uses the high-dimensional features before the bottleneck, and the low-dimensional bottleneck is used only for reconstruction. The real difference between the two is therefore whether the generative latent is 64- or 768-dimensional. If this is the core of the paper, the most direct experiment would compare 64, 256 and 768 dimensions with everything else fixed, reporting understanding, reconstruction and generation metrics together with compute cost. The paper does not do this.
- The “from scratch” claim is misleading, and the comparison with VTP is not matched. The introduction states that GAMUT “is trained entirely from scratch”, but Stage 1 distills features from SigLIP2-giant, and in Tab. 4 this step alone raises IN-1K accuracy by 5.22 points, while self-supervision and reconstruction together add only 1.15. SigLIP2-giant itself reaches 84.5 zero-shot on IN-1K, and the paper does not separate how much of this gain comes from extra training, from the teacher’s knowledge, and from the warm-up strategy itself. In addition, GAMUT uses 2.4B image-text pairs plus the knowledge of SigLIP2-giant, whereas VTP-B uses only 277M image-text pairs and no external teacher. Even so, gFID only improves from 3.88 to 3.81.
- The ablations are insufficient. The paper is relatively incremental and can be viewed as a combination of VTP and RAE with several optimizations. It introduces many modules, yet the ablation study occupies only half a page and a small Tab. 4. The understanding ablations report only IN-1K accuracy, and LCR and LID are evaluated only by gFID. Designs such as the non-affine LN and the noisy reconstruction branch are not ablated individually, and the interactions between components are not analyzed. As noted in Weakness 1, the authors also need a latent-dimension comparison to support their central claim.
- (Minor, but strongly recommended) Readability is poor, and the paper lacks a clear main thread. The main text contains only three figures, and the experiments rely almost entirely on numerical tables, while the most informative visual evidence is in the appendix: Fig. 6 shows how collapse destroys spatial structure, and Fig. 7 compares reconstructions. I suggest moving more figures into the main text to help readers. LID is also supported only by a single gFID number, with no comparison of decoding the same generated latents with and without LID. Meanwhile, SigLIP, DINO, iBOT, MIM, MAP, QK-Norm, non-affine LN, LCR and other modules appear one after another, and it is hard to tell which are existing components, which are new, and which are routine choices for training stability. After reading the method section, I can list every design, but I remain quite confused.
Questions for authors
See weakness.
-
Why does the Stage 1 MIM predict SigLIP2-giant features? Under the same training budget, how would understanding, reconstruction and generation change if it instead reconstructed raw pixels as in MAE, or used a teacher of the same size as the student, such as SigLIP2-B?
-
Could the authors add a control that skips MIM and instead trains Stage 2 for an extra 2B samples? This would separate the benefit of the warm-up from extra training versus from the teacher’s knowledge.
-
Are the RAE and VTP gFID numbers in Tab. 2 rerun with the same x-prediction, noise shift and input projection, or cited from the original papers?
9 comments
Response to Reviewer zEL4 - Part2
[W2, Q1, Q2] Clarification on the comparison with VTP
These questions share a common concern: whether the comparison with VTP is properly matched in terms of our current training configuration. We therefore first provide a collective clarification and summarize the key conclusions here. The corresponding detailed experimental results and discussions are shown in the following sections.
Data scale and training budget
Our current dataset size (2.4B samples) and training budget (14B seen samples) are primarily chosen to align with existing vision-language pretraining methods. Existing methods span a wide range of data scales: 400M for CLIP, 1B for DataComp, 2.5B for MetaCLIP, LAION-5B for OpenCLIP. And the recent state-of-the-art SigLIP family, employs the WebDataset-10B and trains for 40B seen samples. Except for the earliest works, mainstream specialized methods place considerable emphasis on data scaling, which is necessary for learning image-text alignment across a broad range of real-world concepts, and using dataset sizes and training budgets comparable to or greater than ours.
For visual understanding, data scaling undoubtedly contributes substantially to the our results reported in Table 1. However, our goal is not to claim superiority only over VTP, which indeed uses considerably less data. Rather, we scale our dataset to a level comparable to those of mainstream specialized methods and achieve comparable or stronger performance, thereby demonstrating the practical value of a unified representation.
For visual generation, by contrast, the experiments below show that the performance gains arise primarily from the representation properties induced by our training paradigm, whereas simply increasing the dataset size or training budget yields only limited improvements.
MIM warmup strategy
We would like to emphasize that "MIM with a trained teacher outputs as targets" is an established pretraining paradigm, rather than a new strategy proposed by us. In Section 3.2 of the original manuscript, we explicitly state that we are “following EVA” and cite the corresponding work. We provide further details and discussions in the subsequent section. The main effects of the MIM stage are summarized as follows.
For visual understanding, (1) the MIM warm-up stage is necessary under our current training budget since it substantially accelerates convergence. The model reported in our paper converges after 14B seen samples in Stage 2, whereas the variant without MIM continues to show a clear improvement trend even at 20B seen samples. (2) Regarding the comparison with VTP you mentioned, the no-MIM variant also outperforms VTP, demonstrating that our performance advantage over VTP does not depend on MIM.
For visual generation, we provide additional experiments showing that simply aligning the representation with a teacher model yields limited performance, whereas the masked modeling strategy itself improves generative capability.
Brief summary
As discussed in the preceding section, the key distinction between GAMUT and VTP is whether visual understanding and generation are supported through a single unified representation. This distinction is architectural rather than merely quantitative. Accordingly, we focuses on improving the absolute performance of this unified representation across multiple dimensions, including the techniques we proposed, along with the data and training scaling.
As for your concerns about the sources of our final performance gains, we provide the additional experimental results and discussions in the following sections.
Response to Reviewer zEL4 - Part3
[Q1] Why using MIM with a larger teacher
Our reported choice is informed by key findings from several influential prior works, as well as observations from our preliminary experiments. First, the paradigm of using a larger model’s outputs as MIM targets, along with its key supporting findings, was established primarily in the following works.
- Non-pixel-space targets can promote semantic modeling in MIM: As demonstrated by established works such as BEiT[1], iBOT[2], and MILAN[3].
- Using features from a pretrained CLIP model as MIM targets is an efficient pretraining strategy: This approach was first introduced in MILAN and later extended by EVA[4] and EVA-CLIP[5] into a two-stage training paradigm, MIM with a CLIP teacher followed by contrastive learning. EVA-CLIP firstly demonstrated its efficiency compared with contrastive learning alone. Furthermore, EVA-CLIP’s experiments also demonstrated that using teacher representations as MIM targets is more effective than directly aligning to the teacher representations.
- Larger CLIP teachers and longer MIM training can yield more robust performance: EVA-02[6] demonstrated this using a 1B-parameter teacher to train students including Base size or smaller. This approach has also been adopted in recent large multimodal models. For example, Seed 1.5-VL[4] uses a 4.4B-parameter teacher to train a student with 500M parameters.
Second, we conducted preliminary comparisons within this paradigm, with results summarized in the table below.
| Teacher Model | Mask Ratio | Linear Probing at Stage1 | Zero-shot at Stage2, 2B |
|---|---|---|---|
| SigLIP2-L | 75% | 76.92 | 70.02 |
| SigLIP2-g | 75% | 78.40 | 71.47 |
| SigLIP2-g | 50% | 77.34 | 70.59 |
Based on these results, we found that a larger teacher and a higher masking ratio yield better performance, which informed the final configuration used in our reported model.
[1] Bao, Hangbo, et al. "Beit: Bert pre-training of image transformers." arXiv preprint arXiv:2106.08254 (2021).
[2] Zhou, Jinghao, et al. "ibot: Image bert pre-training with online tokenizer." arXiv preprint arXiv:2111.07832 (2021).
[3] Hou, Zejiang, et al. "Milan: Masked image pretraining on language assisted representation." arXiv preprint arXiv:2208.06049 (2022).
[4] Fang, Yuxin, et al. "Eva: Exploring the limits of masked visual representation learning at scale." 2023 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, 2023.
[5] Sun, Quan, et al. "Eva-clip: Improved training techniques for clip at scale." arXiv preprint arXiv:2303.15389 (2023).
[6] Fang, Yuxin, et al. "Eva-02: A visual representation for neon genesis." Image and Vision Computing 149 (2024): 105171.
[7] Guo, Dong, et al. "Seed1. 5-vl technical report." arXiv preprint arXiv:2505.07062 (2025).
Response to Reviewer zEL4 - Part1
Thank you very much for your valuable feedback. Below, we provide additional discussion and experiments for your concerns point by point. We hope these additional experiments and discussions further clarify the contributions of our method and address your concerns. We look forward to your further feedback.
[W1] Clarification of our core claim and the description of VTP
We apologize for causing any misunderstanding. Our discussion of VTP is intended to highlight the difference in architectural design philosophy between GAMUT and VTP. The key distinctions are as follows:
To support this, we conduct the following experiments to compare VTP’s representations before and after compression with ours,
in which the metrics corresponding to:
Results show that VTP’s low-dimensional features perform substantially worse than its high-dimensional features on both linear probing and segmentation, directly supporting our statement that “bottleneck compression sacrifices semantics.” Conversely, VTP’s high-dimensional representations, which perform well on understanding tasks, yield poor generation results.
Thus, our central claim remains that we achieve a single unified representation. These results above show that introducing the bottleneck separates the representation spaces, leaving neither space capable of supporting all tasks. This motivates us removing the bottleneck, thus enabling all supervision objectives to shape a single representation space.
The dimensionality of 768 is a natural consequence of removing the bottleneck, since it's the width of the ViT backbone. Therefore, under the current framework, once the model size is fixed, the representation dimensionality is also determined, leaving no alternative dimensionality choices of the kind you mentioned.