acceptodds
Under review as a conference paper at ICLR 2027

GAMUT: Unified Visual Representation for Generation and Multimodal Alignment

Abstract

Visual representation learning has evolved through largely separate paradigms, including self-supervised learning, multimodal alignment, and visual tokenization. Despite substantial progress within each paradigm, this separation is increasingly at odds with the demands of modern vision and multimodal systems, which require diverse information to support open-ended tasks rather than representations specialized for a single capability. To address this issue, we introduce GAMUT, which learns a unified visual representation and brings these paradigms into a single joint pretraining framework. We develop three key techniques to ensure stable and efficient joint pretraining. First, we construct a three-stage training pipeline that progressively shapes the representation and avoid the prohibitive computational cost of prolonged multi-objective training. Second, we propose the latent channel regularization, which effectively mitigates the channel-wise energy distribution imbalance and prevents representation collapse Third, we introduce latent interpolation decoding to further improve the suitability of such representation for generative tasks. Together, these designs enable GAMUT to capture high-level semantics while preserving fine-grained visual details. As a result, GAMUT performs strongly across a diverse range of visual tasks, outperforming existing unified tokenizers and task-specific baselines on multiple benchmarks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
unified tokenizer, vision encoder pretraining

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zEL4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes GAMUT, a ViT-B/16 encoder trained jointly with SigLIP, DINO/iBOT and pixel-reconstruction objectives, so that a single 768-dimensional, uncompressed representation serves both understanding and diffusion-based generation. Training proceeds from a SigLIP2-feature MIM warm-up to joint pretraining and then decoder training. Latent channel regularization (LCR) prevents channel collapse, and latent interpolation decoding (LID) makes the decoder more robust to generated latents. GAMUT outperforms existing unified tokenizers on zero-shot classification and retrieval, ImageNet reconstruction and generation, and segmentation.

Critical strengths

The analysis of representation collapse is clear. Fig. 3 and Fig. 5 directly show the emergence of single-channel energy concentration and how LCR reverses it, and the observation in Fig. 4 that DINO collapses when trained from scratch is a useful reference. The rFID of 0.22 is strong, and the appendix provides thorough implementation details.

Critical weaknesses
  • The description of VTP is inaccurate, and the core claim lacks direct validation. The paper states that VTP compresses high-dimensional features into a low-dimensional bottleneck and thereby sacrifices semantics, and attributes its gains over VTP to avoiding this compression. However, VTP’s image-text alignment uses the high-dimensional features before the bottleneck, and the low-dimensional bottleneck is used only for reconstruction. The real difference between the two is therefore whether the generative latent is 64- or 768-dimensional. If this is the core of the paper, the most direct experiment would compare 64, 256 and 768 dimensions with everything else fixed, reporting understanding, reconstruction and generation metrics together with compute cost. The paper does not do this.
  • The “from scratch” claim is misleading, and the comparison with VTP is not matched. The introduction states that GAMUT “is trained entirely from scratch”, but Stage 1 distills features from SigLIP2-giant, and in Tab. 4 this step alone raises IN-1K accuracy by 5.22 points, while self-supervision and reconstruction together add only 1.15. SigLIP2-giant itself reaches 84.5 zero-shot on IN-1K, and the paper does not separate how much of this gain comes from extra training, from the teacher’s knowledge, and from the warm-up strategy itself. In addition, GAMUT uses 2.4B image-text pairs plus the knowledge of SigLIP2-giant, whereas VTP-B uses only 277M image-text pairs and no external teacher. Even so, gFID only improves from 3.88 to 3.81.
  • The ablations are insufficient. The paper is relatively incremental and can be viewed as a combination of VTP and RAE with several optimizations. It introduces many modules, yet the ablation study occupies only half a page and a small Tab. 4. The understanding ablations report only IN-1K accuracy, and LCR and LID are evaluated only by gFID. Designs such as the non-affine LN and the noisy reconstruction branch are not ablated individually, and the interactions between components are not analyzed. As noted in Weakness 1, the authors also need a latent-dimension comparison to support their central claim.
  • (Minor, but strongly recommended) Readability is poor, and the paper lacks a clear main thread. The main text contains only three figures, and the experiments rely almost entirely on numerical tables, while the most informative visual evidence is in the appendix: Fig. 6 shows how collapse destroys spatial structure, and Fig. 7 compares reconstructions. I suggest moving more figures into the main text to help readers. LID is also supported only by a single gFID number, with no comparison of decoding the same generated latents with and without LID. Meanwhile, SigLIP, DINO, iBOT, MIM, MAP, QK-Norm, non-affine LN, LCR and other modules appear one after another, and it is hard to tell which are existing components, which are new, and which are routine choices for training stability. After reading the method section, I can list every design, but I remain quite confused.
Questions for authors

See weakness.

  • Why does the Stage 1 MIM predict SigLIP2-giant features? Under the same training budget, how would understanding, reconstruction and generation change if it instead reconstructed raw pixels as in MAE, or used a teacher of the same size as the student, such as SigLIP2-B?

  • Could the authors add a control that skips MIM and instead trains Stage 2 for an extra 2B samples? This would separate the benefit of the warm-up from extra training versus from the teacher’s knowledge.

  • Are the RAE and VTP gFID numbers in Tab. 2 rerun with the same x-prediction, noise shift and input projection, or cited from the original papers?

9 comments
AuthorsOpenReview ↗

Response to Reviewer zEL4 - Part1

Thank you very much for your valuable feedback. Below, we provide additional discussion and experiments for your concerns point by point. We hope these additional experiments and discussions further clarify the contributions of our method and address your concerns. We look forward to your further feedback.

[W1] Clarification of our core claim and the description of VTP

We apologize for causing any misunderstanding. Our discussion of VTP is intended to highlight the difference in architectural design philosophy between GAMUT and VTP. The key distinctions are as follows:

  • VTP’s representations for understanding and generation remain separate. Just as you noted, it aligns images and text in the high-dimensional feature space before the bottleneck, while using the compressed feature space after the bottleneck for reconstruction and as the generative latent space. This design inherently separates the two representation spaces.
  • GAMUT has a single, fully unified representation space. We employ exactly the same feature serving understanding, reconstruction, and generation tasks.
  • The 768-dimensionality is a natural consequence of achieving a single unified representation, rather than a hyperparameter choice that we claim to be superior.

To support this, we conduct the following experiments to compare VTP’s representations before and after compression with ours,

Representation SpaceLinear Probing (Acc) ↑Segmentation (IoU) ↑gFID ↓
VTP - 768D (before the bottleneck)79.7679.6912.76
VTP - 64D (after the bottleneck)58.3452.043.88
GAMUT (Ours) - 768D80.4184.253.81

in which the metrics corresponding to:

  • Linear Probing (Acc): We apply average pooling to the features from each representation spaces, freeze the pooled representations, and train a separate linear classifier on ImageNet for each, to assess image-level semantic modeling.
  • Segmentation (IoU): Following the VOC setup in Table. 3 of our manuscript, we train a segmentation head on each frozen representation space to evaluate pixel-level semantic modeling.
  • gFID: We additionally train a generative model on VTP’s uncompressed representations. Since this space has the same dimensionality as ours, we use exactly the same training and evaluation configurations as in our experiments.

Results show that VTP’s low-dimensional features perform substantially worse than its high-dimensional features on both linear probing and segmentation, directly supporting our statement that “bottleneck compression sacrifices semantics.” Conversely, VTP’s high-dimensional representations, which perform well on understanding tasks, yield poor generation results.

Thus, our central claim remains that we achieve a single unified representation. These results above show that introducing the bottleneck separates the representation spaces, leaving neither space capable of supporting all tasks. This motivates us removing the bottleneck, thus enabling all supervision objectives to shape a single representation space.

The dimensionality of 768 is a natural consequence of removing the bottleneck, since it's the width of the ViT backbone. Therefore, under the current framework, once the model size is fixed, the representation dimensionality is also determined, leaving no alternative dimensionality choices of the kind you mentioned.

AuthorsOpenReview ↗

Response to Reviewer zEL4 - Part2

[W2, Q1, Q2] Clarification on the comparison with VTP

These questions share a common concern: whether the comparison with VTP is properly matched in terms of our current training configuration. We therefore first provide a collective clarification and summarize the key conclusions here. The corresponding detailed experimental results and discussions are shown in the following sections.

Data scale and training budget

Our current dataset size (2.4B samples) and training budget (14B seen samples) are primarily chosen to align with existing vision-language pretraining methods. Existing methods span a wide range of data scales: 400M for CLIP, 1B for DataComp, 2.5B for MetaCLIP, LAION-5B for OpenCLIP. And the recent state-of-the-art SigLIP family, employs the WebDataset-10B and trains for 40B seen samples. Except for the earliest works, mainstream specialized methods place considerable emphasis on data scaling, which is necessary for learning image-text alignment across a broad range of real-world concepts, and using dataset sizes and training budgets comparable to or greater than ours.

For visual understanding, data scaling undoubtedly contributes substantially to the our results reported in Table 1. However, our goal is not to claim superiority only over VTP, which indeed uses considerably less data. Rather, we scale our dataset to a level comparable to those of mainstream specialized methods and achieve comparable or stronger performance, thereby demonstrating the practical value of a unified representation.

For visual generation, by contrast, the experiments below show that the performance gains arise primarily from the representation properties induced by our training paradigm, whereas simply increasing the dataset size or training budget yields only limited improvements.

MIM warmup strategy

We would like to emphasize that "MIM with a trained teacher outputs as targets" is an established pretraining paradigm, rather than a new strategy proposed by us. In Section 3.2 of the original manuscript, we explicitly state that we are “following EVA” and cite the corresponding work. We provide further details and discussions in the subsequent section. The main effects of the MIM stage are summarized as follows.

For visual understanding, (1) the MIM warm-up stage is necessary under our current training budget since it substantially accelerates convergence. The model reported in our paper converges after 14B seen samples in Stage 2, whereas the variant without MIM continues to show a clear improvement trend even at 20B seen samples. (2) Regarding the comparison with VTP you mentioned, the no-MIM variant also outperforms VTP, demonstrating that our performance advantage over VTP does not depend on MIM.

For visual generation, we provide additional experiments showing that simply aligning the representation with a teacher model yields limited performance, whereas the masked modeling strategy itself improves generative capability.

Brief summary

As discussed in the preceding section, the key distinction between GAMUT and VTP is whether visual understanding and generation are supported through a single unified representation. This distinction is architectural rather than merely quantitative. Accordingly, we focuses on improving the absolute performance of this unified representation across multiple dimensions, including the techniques we proposed, along with the data and training scaling.

As for your concerns about the sources of our final performance gains, we provide the additional experimental results and discussions in the following sections.

AuthorsOpenReview ↗

Response to Reviewer zEL4 - Part3

[Q1] Why using MIM with a larger teacher

Our reported choice is informed by key findings from several influential prior works, as well as observations from our preliminary experiments. First, the paradigm of using a larger model’s outputs as MIM targets, along with its key supporting findings, was established primarily in the following works.

  • Non-pixel-space targets can promote semantic modeling in MIM: As demonstrated by established works such as BEiT[1], iBOT[2], and MILAN[3].
  • Using features from a pretrained CLIP model as MIM targets is an efficient pretraining strategy: This approach was first introduced in MILAN and later extended by EVA[4] and EVA-CLIP[5] into a two-stage training paradigm, MIM with a CLIP teacher followed by contrastive learning. EVA-CLIP firstly demonstrated its efficiency compared with contrastive learning alone. Furthermore, EVA-CLIP’s experiments also demonstrated that using teacher representations as MIM targets is more effective than directly aligning to the teacher representations.
  • Larger CLIP teachers and longer MIM training can yield more robust performance: EVA-02[6] demonstrated this using a 1B-parameter teacher to train students including Base size or smaller. This approach has also been adopted in recent large multimodal models. For example, Seed 1.5-VL[4] uses a 4.4B-parameter teacher to train a student with 500M parameters.

Second, we conducted preliminary comparisons within this paradigm, with results summarized in the table below.

Teacher ModelMask RatioLinear Probing at Stage1Zero-shot at Stage2, 2B
SigLIP2-L75%76.9270.02
SigLIP2-g75%78.4071.47
SigLIP2-g50%77.3470.59

Based on these results, we found that a larger teacher and a higher masking ratio yield better performance, which informed the final configuration used in our reported model.

[1] Bao, Hangbo, et al. "Beit: Bert pre-training of image transformers." arXiv preprint arXiv:2106.08254 (2021).

[2] Zhou, Jinghao, et al. "ibot: Image bert pre-training with online tokenizer." arXiv preprint arXiv:2111.07832 (2021).

[3] Hou, Zejiang, et al. "Milan: Masked image pretraining on language assisted representation." arXiv preprint arXiv:2208.06049 (2022).

[4] Fang, Yuxin, et al. "Eva: Exploring the limits of masked visual representation learning at scale." 2023 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, 2023.

[5] Sun, Quan, et al. "Eva-clip: Improved training techniques for clip at scale." arXiv preprint arXiv:2303.15389 (2023).

[6] Fang, Yuxin, et al. "Eva-02: A visual representation for neon genesis." Image and Vision Computing 149 (2024): 105171.

[7] Guo, Dong, et al. "Seed1. 5-vl technical report." arXiv preprint arXiv:2505.07062 (2025).

Reviewer_Nga4OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work introduces GAMUT, which uses high-dimensional patch representations as a latent space to unify multimodal alignment, visual perception, and generation. Its three-stage training pipeline begins with MIM warm-up targeting SigLIP2 features. Stage 2 jointly optimizes image–text alignment, DINO/iBOT self-supervision, and pixel reconstruction, introducing LCR to mitigate representation collapse caused by energy concentration in a single channel. Stage 3 freezes the encoder and trains a larger decoder, using noisy reconstruction and LID to improve robustness when decoding generated latents. Experiments show that GAMUT outperforms existing baselines across zero-shot classification and retrieval, ImageNet reconstruction and generation, and dense perception.

Critical strengths
  1. The work addresses a clear problem: preserving both semantics and visual details in a single high-dimensional representation without a compression bottleneck. It achieves balanced performance across alignment, reconstruction, generation, and dense perception.
  2. The observations and analysis of representation collapse are insightful. LCR and LID are clearly motivated and offer useful practical guidance for training unified representations.
  3. Generation experiments provide controlled comparisons using the same generative backbone and training duration. Implementation details and evaluation protocols are well documented, supporting reproducibility.
Critical weaknesses
  1. Why does the reconstruction objective exploit a single dominant channel as a shortcut? Could this be related to non-affine LayerNorm constraining each token’s total energy? A deeper mechanistic analysis would help establish whether LCR addresses the underlying cause or merely suppresses the symptoms.
  2. Stage 1 distills features from SigLIP2-G, while the DINO branch retains only local-to-global alignment and also relies on this warm-up. Is GAMUT’s semantic geometry shaped by joint training, or largely inherited from SigLIP2-giant? Measuring representation similarity to the teacher before and after Stage 2, for example using CKA, would clarify what the self-supervised and reconstruction objectives actually change.
  3. The method is simple and effective, but additional ablations on scaling behavior would strengthen its impact.
Questions for authors

Please see weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Nga4

Thank you for recognizing our motivation for unified representations and the value of our experimental findings. Below, we provide additional mechanistic analysis and experiments for the concerns you raised. We hope these additions adequately address your concerns and look forward to your further feedback.

Analysis of the dominant channel

To show what information the dominant channel provides for pixel reconstruction, we analyze the correlations between the values of the dominant channel and per-patch pixel statistics in the reconstructed images. The table below presents 2 statistics that exhibit moderate or strong correlations, reported as absolute Pearson correlation coefficients.

Pixel StatisticsDominant Channel (Before Norm)Dominant Channel (After Norm)
Luminance Mean0.810.82
Luminance Skewness0.430.45

Luminance Mean measures the average brightness of a patch, while Luminance Skewness is the standardized third central moment of its luminance distribution, indicating the direction and degree of asymmetry. These metrics indicate that:

  • The dominant channel indeed carries information useful for pixel reconstruction.
  • Its values exhibit strong correlations with these statistics both before and after normalization, thus there is no evidence linking this phenomenon to the non-affine LayerNorm. Furthermore, the affine parameters are fixed for each channel, whereas the correlations above indicate that the dominant channel takes varying values across patches. Thus, in principle, affine parameters cannot resolve this issue.
  • After introducing LCR, both correlations fall below 0.01. We believe this as evidence that LCR successfully disrupts this information shortcut, encouraging the information needed for reconstruction to be redistributed across the channels.

Mechanistically, when the feature space is unconstrained, a task such as pixel reconstruction, which requires recovering the input at the output, may encourage the model to develop such shortcuts that simplify the objective. In VAE-based methods, the KL regularization toward a standard Gaussian prior constrains the latent space and helps prevent similar behavior. From this perspective, as discussed in the manuscript, non-affine LayerNorm and LCR together constrain GAMUT’s representations to have zero mean and unit variance, while the former along channel dimension, and the latter along the spatial and sample dimensions. They thus serve a regularizing role analogous to KL regularization in VAEs.

Effects of Stages 1 and 2 on Shaping Semantic Geometry

We compare the representations learned by our model at the end of Stages 1 and 2 with two larger models, SigLIP2-g and DINOv2-g. We also includes SigLIP2-B to serve as a baseline for base-size encoder. Following your suggestion, we use per-image CKA, where we compute CKA between patch-level representations within each image and then average the scores over ImageNet validation set. The results are shown in the table below.

ModelSigLIP2-gDINOv2-g
SigLIP2-B0.39870.4739
GAMUT after Stage10.46150.5064
GAMUT after Stage20.29670.6246

The results show that:

  • After Stage 1, our model exhibits greater similarity to SigLIP2-g than SigLIP2-B does, owing to feature alignment during training. The MIM training paradigm also brings its representation geometry closer to that of DINOv2-g.
  • From Stage 1 to Stage 2, similarity to SigLIP2-g decreases, while similarity to DINOv2-g increases further. This demonstrates that joint pretraining in Stage 2 reshapes the representation space.

Furthermore, prior works such as REPA and RAE have demonstrated that DINOv2 outperform SigLIP2 on generation tasks. We therefore view the greater similarity to DINOv2 after Stage 2 as an advantage for unified representation. Moreover, the greater similarity to DINOv2 does not compromise GAMUT’s strong image–text alignment, as demonstrated by the zero-shot performance.

Further Scaling

Thank you very much for your suggestion. Further validating scaling behavior is a top priority for us. In the current work, we have already scaled unified representation pretraining to a data volume and training duration approaching those of specialized methods. We will continue to explore the performance at larger model scales.

Thank you very much for your time and valuable feedback. As the discussion period is coming to a close, we would like to kindly ask whether our responses have addressed your concerns and whether there are any remaining questions we could clarify or discuss in the time left. We look forward to hearing from you.

Reviewer_3mQzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GAMUT learns a shared high-dimensional visual representation through teacher-based masked pretraining, joint image-text alignment, self-supervision and reconstruction, followed by decoder refinement. Its main additions are latent channel regularization and latent interpolation decoding. Experiments demonstrate useful classification, retrieval, segmentation, reconstruction and generation capabilities, with substantial improvements over existing unified tokenizers on several tasks. Ablations support the practical value of the proposed components. However, reconstruction and generation are evaluated only on ImageNet at 256×256, while downstream multimodal understanding and the robustness of the training recipe remain insufficiently established.

Critical strengths
  1. The shared representation supports complementary capabilities. Strong retrieval and frozen-encoder segmentation results coexist with high-quality reconstruction, providing meaningful evidence that semantic abstraction and visual detail can be retained together.
  2. The stabilization techniques have demonstrated practical benefits. Channel regularization repairs a severe generation failure, and interpolation decoding improves its paired generation ablation. These findings offer useful guidance for training unified representations.
Critical weaknesses
  1. The evaluation does not fully establish the intended breadth of use. Generation is restricted to ImageNet 256×256 class-conditional synthesis, leaving higher-resolution and broader-distribution performance untested. Classification, retrieval and segmentation provide useful understanding evidence, but do not establish downstream MLLM/VQA performance. RAE’s higher-resolution experiments and UniTok’s broader generation and multimodal evaluations illustrate relevant validation directions. Although the authors acknowledge these limitations, they materially constrain the significance of the general-purpose representation claim.

  2. The robustness of the training recipe is insufficiently characterized. I appreciate that Stage 2 uses unit coefficients and that the ablations demonstrate compatibility in the reported configuration. However, component ablations do not establish robustness to loss weighting or stage timing. Appendix A.3 also describes activating LCR after detecting degeneration, without clearly specifying its activation schedule for the final model. This leaves uncertainty about how reliably the recipe can be reproduced and transferred to other training settings.

Questions for authors
  1. Do the benefits of LCR and LID persist at higher resolution or in a non-ImageNet generation setting? Is there evidence of downstream VQA performance under a matched language-model and adaptation protocol?
  2. When was LCR activated for the reported model, and did training require intervention based on observed instability? How sensitive are the results to reconstruction/LCR weights and warmup duration?
  3. Were the Table 2 generation baselines evaluated with matching prediction targets, noise-schedule treatment and comparable decoder training? How stable is the 0.07 gFID advantage over VTP?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 3mQz - Part1

Thank you for your positive assessment of the effectiveness of our overall framework and the practical value of the proposed techniques. Below, we provide additional discussion and experiments for your concerns. We hope these additions adequately address your concerns and look forward to your further feedback.

VQA performance

We have added a comparison of GAMUT, VTP, and SigLIP on VQA tasks in the following configuration:

  • Model architecture: We use Qwen3-1.7B as the LLM decoder and a two-layer MLP adapter to connect different encoders to the LLM.
  • Training: We follow a two-stage LLaVA-style training procedure. The first stage trains only the adapter on LLaVA-Pretrain dataset, while the second stage jointly trains the adapter and the LLM on LLaVA-NeXT-Data. All encoders remain frozen through both stages, and take 256 × 256 resolution input. We follow the LLaVA-NeXT framework and train with its anyres configuration, enabling the encoder to process larger input images despite its lower native resolution.
  • Evaluation: Using lmms-eval, we evaluate and report results on 6 VQA benchmarks: two general VQA benchmarks (VQA-v2 and GQA), two vision-centric benchmarks (MME-P and CV-Bench), and two OCR benchmarks covering natural scenes (TextVQA) and documents (DocVQA).

As shown in the table below, the ranking of the three models by average score is consistent with the zero-shot results in the manuscript, where GAMUT slightly outperforms SigLIP and substantially outperforms VTP. Across individual benchmarks, GAMUT and SigLIP perform comparably on VQA-v2, GQA, and MME-P. GAMUT shows larger gains on CV-Bench and TextVQA, which place greater emphasis on visual structure and details perception in natural scenes. On DocVQA, whose distribution differs substantially from our pretraining distribution, GAMUT underperforms SigLIP. Furthermore, both VTP and GAMUT, which use unified pretraining, outperform SigLIP on CV-Bench, suggesting that unified pretraining benefits visual perception.

ModelVQA-v2GQAMME-PCV-BenchTextVQADocVQAaverage
SigLIP68.855.268.454.341.333.353.5
VTP68.354.866.255.139.228.552.0
GAMUT (ours)68.955.268.155.843.931.753.9

High-resolution generation

We apologize that, due to limited time and computational resources, we are unable to provide text-to-image generation results during the rebuttal period. For your concerns about the effectiveness of the LCR and LID on more challenging generation setting , we have added a high-resolution generation experiments. Specifically, following the “upsample” configuration in RAE, we set the pixel decoder’s output patch size to twice the encoder’s patch size, and fine-tune the decoder to obtain generation results at 512 × 512 resolution. We compare the following configurations:

  • A latent space trained without LCR, with the decoder fine-tuned without LID;
  • A latent space trained with LCR, with the decoder fine-tuned without LID;
  • A latent space trained with LCR, with the decoder fine-tuned with LID.
LCRLIDgFID ↓
✗✗228.54
✓✗6.61
✓✓4.23

The findings are consistent with the ablation results at 256 × 256 resolution in the manuscript. Without LCR, single-channel energy dominance disrupts the latent distribution, preventing the generative model from learning to produce valid images. The benefits of LID for generation remain substantial at higher resolution.

AuthorsOpenReview ↗

Response to Reviewer 3mQz - Part2

LCR activation schedule

For the reported model, we observed the single-channel energy dominance at 61% of the second training stage completion and introduced LCR at that point. We subsequently found that introducing LCR and resuming training effectively resolved this issue, so we did not retrain from scratch.

However, this does not hinder transferability or reproducibility. For new training runs, we recommend enabling LCR from the start of the second training stage, as supported by the following experiments. We compare models trained with and without LCR enabled from the start of the second stage, each trained for 2B seen samples, and report the average zero-shot classification accuracy, reconstruction fidelity, and the energy share of the largest-energy channel.

ConfigClassification AVG (Top1-Acc) ↑Reconstruction (rFID) ↓Energy share of the largest channel (%) ↓
w/o LCR57.591.3224.7
w/ LCR57.741.350.17

The results show that energy concentration begins to emerge early in training. Introducing LCR from the start of the second training stage suppresses this trend while maintaining comparable classification and reconstruction performance. Therefore, when transferring or reproducing the training recipe, LCR should remain enabled throughout the second stage, without the need for intervention midway through training.

Loss weights for LCR and reconstruction

Following the setup in the previous discussion, we additionally ablate the loss weights for reconstruction and LCR under the same second-stage training budget of 2B seen samples. The results are shown in the table below.

  • LCR weight: The current training framework is insensitive to the LCR loss weight. As shown in the last column, even with a loss weight of 0.1, LCR still achieves its intended effect, maintaining a relatively uniform energy distribution across channels. We attribute this primarily to LCR operating directly on the output features, without any projection or normalization layers weakening its gradient. Thus, a small loss weight can still effectively shape the feature distribution.
  • Reconstruction weight: Reducing the reconstruction loss weight has little effect on classification performance, but slightly degrades reconstruction.
    • For reconstruction, we attribute the results to the pixel decoder being supervised solely by the reconstruction loss, and a smaller weight slows its updates early in training.
    • For classification, since our framework keeps higher-dimensional representations, which can more readily preserve information, reducing the conflict between semantic modeling and pixel reconstruction. Consequently, the semantic representation is insensitive to the reconstruction loss weight.
Reconstruction Loss WeightLCR Loss WeightClassification AVG (Top1-Acc) ↑Reconstruction (rFID) ↓Energy share of the largest channel (%) ↓
1.01.057.741.350.17
0.11.057.461.420.18
1.00.157.611.340.16

Warmup duration

The warmup duration was determined by monitoring performance metrics during Stage 1. In our eraly experiments, we observed a positive correlation between early Stage 2 performance and the linear probing performance of Stage 1 warmup. We therefore monitored Stage 1 warmup training using linear probing and found that performance plateaued at a high level at 2B seen-samples training, which informed our choice of the reported warmup duration. We have also added a curve showing this trend in Figure.4 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 3mQz - Part3

Generation configuration

The results of comparison methods in Table. 2 are cited from the respective papers, whose configurations are not fully identical. Some settings are identical across methods, while others warrant method-specific choices. Still others can be matched across methods, and yielding additional gains for our approach than the reported results. We discuss these cases in detail below and provide additional experimental results.

  • For the decoder training strategy, our noisy reconstruction branch follows RAE, while the LID branch is unique to our method.

  • For the noise schedule, our method follows exactly the same setup as RAE, using a timestep shift scaled according to the data dimensionality. VTP uses a similar shift strategy but with different hyperparameters. Since this operation depends on data dimensionality and VTP has lower-dimensional representations, retaining each method’s original configuration is reasonable.

  • For the prediction targets and training objectives, our method differs from both prior works. RAE uses v-prediction with an MSE loss in v space. VTP also uses v-prediction, combining an MSE loss in v space with an auxiliary cosine loss. Our GAMUT uses x-prediction and converts the predictions to v space to compute the MSE loss. We provide the following ablation study to examine the effects of the prediction parameterization and the loss space.

ConfiggFID ↓
x-prediction, v-space MSE (reported)3.81
v-prediction, v-space MSE (aligned with RAE)3.97
v-prediction, v-space MSE + cosine (aligned with VTP)3.64

The results show that :

  • Our prediction parameterization provides additional gains over the one used in RAE. This is consistent with the findings in JiT, which shows that x-prediction helps models better capture the underlying manifold structure of high-dimensional data.
  • Following VTP, adding an auxiliary cosine loss compensates for the performance drop associated with v-prediction and yields better results than the configuration used for our reported results.

Thus, as stated in the manuscript, we chose the reported DiT configuration for its simplicity and sound rationale. Further improvements remain possible through additions such as auxiliary losses. And we believe that this additional improvement under matched configurations further demonstrate that our advantage over VTP in generation performance is stable.

Thank you very much for your time and valuable feedback. As the discussion period is coming to a close, we would like to kindly ask whether our responses have addressed your concerns and whether there are any remaining questions we could clarify or discuss in the time left. We look forward to hearing from you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.