acceptodds
Under review as a conference paper at ICLR 2027

HUG: Learning and Transferring Visual Tokens for Unified Image Understanding and Generation

Abstract

We present HUG, a framework for unified image understanding and autoregres- sive image generation that combines visual tokenizer learning with vocabulary adaptation to a pretrained LLM. A shared vision encoder provides continuous features for understanding and discrete tokens for generation. HUG supplements post-quantization caption supervision and discrete-to-continuous alignment with direct caption and masked VAE-latent prediction before quantization. Controlled experiments show that direct pre-quantization supervision improves reconstruction fidelity and code-use diversity. Increasing the post-caption loss weight does not re- produce the pre-caption gains, and pre-quantization masked prediction outperforms its post-quantization counterpart in our setting. For vocabulary transfer, HUG builds on text-scale-aware normalization and uses Text-Aligned Normalization (TAN) with caption-supervised adaptation to initialize the LLM’s visual input and output weights. With the tokenizer fixed, the full TAN procedure outperforms the tested fixed and learned initialization alternatives without adding a downstream module. With a 0.6B Qwen3 language backbone, HUG achieves 0.85 on GenEval and 86.05 on DPG-Bench. Scaling to 4B yields 0.90 and 87.00, respectively, the highest overall scores on both benchmarks among the unified models evaluated, alongside competitive multimodal understanding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
image understanding, image generation, unified multimodal modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_i1tMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HUG, which achieves unified image understanding and generation by improving the visual tokenizer and visual vocabulary initialization. The method employs a shared SigLIP2 encoder, introduces caption and masked VAE-latent prediction before quantization, retains supervision for alignment between captions and continuous features after quantization, and then adapts the visual vocabulary via TAN to initialize the visual input and output weights of the LLM. The authors report competitive generation performance and verify each design component through ablation studies on reconstruction, initialization, and joint training.

Critical strengths
  1. It is valuable to study tokenizer training and vocabulary transfer separately. Comparing different initializations with a fixed tokenizer reveals the impact of vocabulary processing itself on training.
  2. The generation performance is strong. The 4B model achieves a GenEval score of 0.90 and a DPG-Bench score of 87.00. Table 3(c) also reports both understanding and generation results under joint training, demonstrating that the overall approach delivers practical effectiveness.
Critical weaknesses
  1. What would be the final understanding and generation performance if the pre-caption is removed? What would the performance be if VAE-latent prediction is retained but masking is not applied? Can these control experiments demonstrate whether the gains mainly come from additional supervision, or the design of pre-quantization supervision and masking?
  2. What performance can be achieved if the visual vocabulary is randomly initialized followed by the same caption training? Compared with this simple baseline, how much further improvement can TAN bring?
Questions for authors
  1. What are the results when directly predicting VAE latents without masking?
  2. randomly initialized followed by the same caption training?
1 comment

Dear Reviewer i1tM,

We sincerely thank you for your time and thoughtful feedback.

Performance without pre-caption. We trained a tokenizer with post-quantization caption supervision only and ran the same joint training as in Table 3(c). The setting is identical to the first row of Table 3(c) (L2 initialization,no D2C, no VAE-latent prediction, same downstream data and schedule), except that pre-caption is removed.

| Tokenizer | GenEval | DPG | TextVQA | MMB-EN |

TokenizerGenEvalDPGTextVQAMMB-EN
Post-caption only (new)0.4866.2037.6447.43
Pre- + post-caption0.5267.7637.1248.45

With pre-caption, GenEval increases by 0.04 and DPG by 1.56. The understanding results are mixed (TextVQA -0.52, MMB-EN +1.02), so we do not draw a conclusion for understanding. This is reasonable, since understanding uses the continuous branch and pre-caption is applied to the discrete branch. This comparison has two limitations. It is done on the simplest configuration (no D2C, no VAE-latent prediction) rather than on the full model, and the caption pathways are sampled differently in the two tokenizers (1/2 each for continuous and post without pre-caption, 1/3 each for the three pathways with it), which may also contribute to the difference.

VAE-latent prediction without masking. We trained a tokenizer with the mask ratio set to 0, so the auxiliary Transformer predicts the VAE latents from the complete pre-quantization latent. All other settings are the same as the last row of Table 3(c), including decoder training, reconstruction evaluation and downstream training with TAN initialization. In the paper the mask ratio is sampled from [0.1, 0.5].

VAE-latent predictionPPLPSNRSSIMrFIDLPIPSGenEvalDPGTextVQAMMB-EN
None25K15.730.423.80.530.5871.7238.8050.31
Post-quant., with mask29K16.500.563.50.49----
Pre-quant., w/o mask (new)36K18.030.623.50.430.6372.9440.3252.37
Pre-quant., with mask38K18.250.603.20.400.6273.1540.5152.09

The results without masking are close to those with masking. For the tokenizer, masking gives slightly better PSNR (+0.22), rFID (3.2 vs. 3.5) and LPIPS (0.40 vs. 0.43), and slightly lower SSIM. For the downstream model, the differences on all four benchmarks are below 0.3 and do not have a consistent direction. This shows that the improvement mainly comes from predicting VAE latents before quantization, and masking is not necessary. We will change the name of this objective from "masked VAE-latent prediction" to "VAE-latent prediction" and add both results to Tables 3(a) and 3(c).

Where the gains come from. From the table above, the additional visual supervision is useful by itself: applied after quantization, it improves all reconstruction metrics. Applying the same objective before quantization gives a larger improvement (PSNR 18.25 vs. 16.50, LPIPS 0.40 vs. 0.49), and these two rows use the same pathway sampling and loss weights. Masking has a small effect, as discussed above. As another control, doubling the weight of the post-quantization caption loss does not improve the results (PSNR 15.06 vs. 15.02 in Table 3(a)). We did not test all combinations: the two positions are compared with masking, and masking is only ablated for the pre-quantization position.

Random initialization followed by the same caption training. We ran this baseline and a variant where the random vocabulary is first rescaled to the text-embedding norm. In both cases only the token indices from the tokenizer are used. The caption training uses the same data, number of steps and optimization settings as TAN in Table 3(b), and in both cases the auxiliary LLM starts from the pretrained Qwen3-0.6B. After that, we transfer only the vocabulary to a newly loaded Qwen3-0.6B and run the same T2I training.

VocabularyNormCaption lossGenEval
Random, w/o caption0.93-0.15
Random + caption (new)0.670.860.12
Random + text norm + caption (new)1.030.830.14
TAN w/o adaptation0.93-0.34
TAN0.960.780.40

Norm is the average token norm of the transferred vocabulary, and caption loss is measured at the end of caption training. The caption loss of the two baselines (0.86 and 0.83) is close to that of TAN (0.78), which means the LLM can already use these vocabularies for captioning. However, they do not improve generation over random initialization without caption training (0.12 and 0.14 vs. 0.15). TAN is higher than this baseline by 0.26 to 0.28. These results suggest that the improvement of TAN mainly comes from the vocabulary learned by the tokenizer, and caption adaptation further improves it from 0.34 to 0.40.

Reviewer_gujjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

HUG combines visual-tokenizer learning with vocabulary transfer to a pretrained language model. A shared SigLIP2 encoder feeds continuous understanding features and discrete generation tokens. The tokenizer uses caption and masked VAE-latent prediction before quantization, in addition to post-quantization caption supervision and discrete-to-continuous alignment. Caption-adapted Text-Aligned Normalization (TAN) initializes tied visual input/output weights; its auxiliary LLM is discarded before downstream training. The complete systems report strong GenEval/DPG-Bench results, and controlled studies separate tokenizer quality from vocabulary initialization.

Critical strengths
  1. The paper examines two distinct interfaces rather than attributing every improvement to a single tokenizer. Fixed-tokenizer initialization experiments are particularly useful. Table 3(b) includes raw transfer, L2 normalization, fixed text-scale variants, adapted standard LayerNorm and an adapted projector control.

  2. Table 3(a) includes both post-caption reweighting and pre/post-MIM comparisons. With the other listed objectives retained, pre-MIM obtains PSNR/SSIM of 18.25/0.60 versus 16.50/0.56 for post-MIM. The reconstruction protocol uses all 50,000 ImageNet validation images and identically trained decoder architectures, rather than selected examples.

  3. The paper is well-written and easy to follow.

Critical weaknesses
  1. The pre-caption comparison changes both supervision location and pathway allocation. Appendix A.1, lines 833–839, samples continuous/post-caption pathways with probability 1/2 each without pre-caption, but continuous/pre/post pathways with probability 1/3 each when pre-caption is enabled. Thus Table 3(a) changes the continuous teacher's caption updates and the post-quantization caption exposure, while D2C and MIM remain active every step. Doubling the post-caption loss is helpful but does not match this scheduling change.

  2. The size and stability of TAN's incremental benefit need clearer evidence. The strongest adapted alternative, Proj.+L2, reaches 0.36 GenEval versus TAN's 0.40, while fixed L2 reaches 0.35 (Table 3(b)). These are meaningful differences, but no uncertainty or independent-run variability is provided. Table 3(c) supports TAN in joint training, although it compares against unadapted L2.

  3. The headline results lack sufficient inference and selection details for independent verification. Appendix A.5 names the benchmarks but does not specify generation token count/resolution, autoregressive sampling settings, decoder sampling/guidance settings, images per prompt, or checkpoint-selection criteria.

Questions for authors
  1. Can you separate the pre-caption effect from pathway scheduling using a matched allocation control in the existing tokenizer setting, and clarify whether any available downstream result removes only pre-caption?

  2. Can you provide per-prompt paired uncertainty for TAN versus Proj.+L2, plus any existing training/sampling-seed variation?

1 comment

Dear Reviewer gujj,

We sincerely thank you for your time and thoughtful feedback.

Pre-caption and pathway allocation. You are right. In Table 3(a), enabling pre-caption also changes how the caption pathways are sampled: 1/2 each for continuous and post without pre-caption, and 1/3 each for the three pathways with it. Doubling the post-caption weight only accounts for the post-quantization exposure, not for this change. We do not have an allocation-matched control yet (no pre-caption, continuous 1/3, post 1/3, and no caption loss in the remaining steps), so we cannot separate the two effects at this point. We will add this control to Table 3(a) and state the limitation in the paper. This issue does not affect the comparison between pre- and post-quantization VAE-latent prediction in Table 3(a), since both variants enable pre-caption and use the same pathway sampling and loss weights.

For your second question, we now have a downstream result without pre-caption. We trained a tokenizer with post-quantization caption supervision only and ran the same joint training as in Table 3(c). The setting is identical to the first row of Table 3(c) (L2 initialization, no D2C, no VAE-latent prediction, same downstream data and schedule), except that pre-caption is removed.

TokenizerGenEvalDPGTextVQAMMB-EN
Post-caption only (new)0.4866.2037.6447.43
Pre- + post-caption0.5267.7637.1248.45

With pre-caption, GenEval increases by 0.04 and DPG by 1.56. Understanding is mixed (TextVQA -0.52, MMB-EN +1.02), so we do not draw a conclusion for understanding. This comparison is done on the simplest configuration rather than on the full model, and it has the same difference in pathway sampling, which may also contribute to the gain.

Size and stability of the TAN gain. We also ran the two models on DPG-Bench.

InitializationGenEvalDPG
Proj.+L20.3661.5 ± 0.4
TAN0.4064.3 ± 0.6

TAN is better on both benchmarks. DPG is the mean and standard deviation of three sampling runs: 64.2 / 64.9 / 63.7 for TAN and 61.1 / 61.8 / 61.6 for Proj.+L2. The lowest TAN run is 1.9 above the highest Proj.+L2 run. GenEval was run once. We do not have results from different training seeds, and we did not compute per-prompt paired intervals.

You are also right that Table 3(c) compares TAN with L2 without adaptation; we do not have a joint-training result for Proj.+L2. An experiment requested by Reviewer i1tM is related to the role of caption adaptation. We trained a randomly initialized vocabulary with the same caption training as TAN (same data, steps and optimization settings as TAN in Table 3(b), both starting from the pretrained Qwen3-0.6B), with and without rescaling it to the text-embedding norm. The caption loss at the end is 0.86 and 0.83, close to 0.78 for TAN. However, GenEval is only 0.12 and 0.14, compared with 0.15 for random initialization without caption training and 0.40 for TAN. So the improvement of TAN is not simply due to the additional caption training.

Inference and checkpoint selection. We will add these details to Appendix A.5. The main results and all ablations use the same settings.

  • The LLM generates 729 visual tokens for each image, with temperature 1.0, top-k 1200 and top-p 0.95. We do not use classifier-free guidance.
  • The decoder is Lumina-Image 2.0 with the Euler sampler, 50 steps and guidance scale 1.0 (no guidance). It does not take the text prompt as input. For GenEval and DPG-Bench we decode at 512x512 for faster inference; the reconstruction results in Table 3(a) are at 1024x1024.
  • We generate four images for each prompt on both benchmarks. Within a run, all models use the same seeds.
  • For checkpoint selection, we built a validation set of 200 prompts retrieved from the training set, which has no overlap with GenEval or DPG-Bench. We generate images for these prompts with each saved checkpoint, score them with Gemini 3.0 Flash, and choose the checkpoint with the highest average score. We do this for the main results and for all ablation rows with generation results. The two benchmarks are not used for selection. Table 3(a) only reports reconstruction and compares checkpoints at the same training step. Since no guidance is used in either stage and the decoder does not see the text, the text-image alignment of the generated images comes only from the visual tokens predicted by the LLM.
Reviewer_fYH2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HUG, a framework for unified image understanding and autoregressive image generation, focusing on two aspects: learning effective discrete visual representations and transferring the learned visual vocabulary into a pretrained language model.

First, HUG introduces a dual-branch visual tokenizer built on a shared SigLIP2 encoder, where continuous features support image understanding and discrete features support image generation. Beyond conventional post-quantization caption supervision and discrete-to-continuous alignment, the authors introduce direct pre-quantization supervision through caption prediction and masked VAE-latent prediction. The experiments suggest that these objectives improve reconstruction quality and code-use diversity.

Second, the paper proposes Text-Aligned Normalization (TAN), which adapts the learned visual vocabulary to the embedding scale of a pretrained LLM through text-scale-aware normalization and caption-supervised adaptation.

The complete system combines Qwen3 language backbones with a pretrained Lumina-Image 2.0 decoder. HUG achieves competitive results on both multimodal generation and understanding benchmarks among the evaluated unified models.

Critical strengths

1. Promising empirical results.

The most compelling strength is the image-generation performance achieved by HUG. In Table 2, HUG-0.6B achieves 0.85 on GenEval and 86.05 on DPG-Bench, outperforming several larger unified models. Scaling to 4B further improves these results to 0.90 and 87.00, respectively. The model also maintains competitive performance on multimodal understanding benchmarks.

Although the comparisons involve different training recipes and image decoders, the results demonstrate that the proposed system is an effective approach to combining image understanding and generation.

2. Ablations provide useful evidence for the proposed design choices.

The paper includes several carefully designed comparisons. In Table 3(a), the authors investigate pre-quantization caption supervision, post-caption loss reweighting, and the placement of masked prediction. The results show that pre-quantization supervision provides improvements that cannot be reproduced simply by increasing the post-caption loss weight.

Similarly, Table 3(b) evaluates multiple vocabulary initialization strategies while keeping the tokenizer fixed. The results indicate that TAN with caption adaptation outperforms the tested alternatives. These experiments help distinguish the effects of representation learning from vocabulary initialization and make the paper more convincing than a purely system-oriented contribution.

Critical weaknesses

1. Limited methodological novelty and insufficient investigation of the underlying mechanisms.

The main innovations largely consist of combining and modifying existing techniques rather than introducing a fundamentally new modeling approach.

The shared-encoder continuous/discrete design closely follows previous unified visual tokenization frameworks, such as MANZANO[1]. The main tokenizer contribution consists of applying caption supervision and masked feature prediction before quantization, while retaining existing post-quantization objectives.

More importantly, directly supervising pre-quantization features is not a fresh idea in visual tokenization. For instance, DualToken[2] applies a self-distillation objective to the second-to-last layer features before quantization to preserve semantic representations. UniWeTok[3] also explicitly investigates supervision before and after quantization and introduces Pre-Post Distillation to improve representation learning. The authors should discuss these related works and clarify how their approach differs.

Besides, moving these objectives to the pre-quantization latent is a reasonable design choice, but the paper does not provide a sufficiently deep explanation of why this placement fundamentally improves discrete representation learning. This would also help distinguish the paper from prior works with similar technical contributions.

In particular, gradients from post-quantization objectives already propagate to the upstream encoder through STE. It remains unclear whether the improvements primarily arise from avoiding gradient approximation through quantization, introducing an additional optimization pathway, changing the distribution of supervision across features, or providing stronger visual reconstruction signals.

Similarly, TAN largely builds upon existing text-scale-aware normalization techniques. Its principal extension is caption-supervised vocabulary adaptation followed by transferring the resulting embeddings into the LLM. While useful, the conceptual distinction from existing normalization and vocabulary adaptation strategies is modest.

Overall, the contribution currently appears closer to a carefully engineered combination of existing techniques than a substantially new representation-learning or multimodal modeling paradigm. A more convincing analysis of the underlying mechanisms and a clearer distinction from closely related prior work would strengthen its significance.

2. Insufficient experimental evidence

The tokenizer evaluation in Table 3(a) relies primarily on PSNR, SSIM, and code-use perplexity. These measurements are useful but insufficient to comprehensively characterize the quality of visual tokens for autoregressive image generation.

In particular, code-use perplexity only measures the diversity of token utilization and does not directly establish better semantic organization, information preservation, or autoregressive predictability. Likewise, PSNR and SSIM are limited measures of perceptual reconstruction quality, especially when the reconstruction process involves a pretrained generative decoder. Additional metrics such as rFID, LPIPS, and token prediction performance would provide a more complete picture.

Moreover, Table 3 does not directly isolate the contribution of pre-quantization caption supervision under a matched unified-training setting. All its variants already include both pre- and post-quantization caption supervision, while the largest downstream improvements from the full tokenizer and TAN are demonstrated using substantially simplified training configurations.

Additionally, the headline results involve extensive multi-stage training, a pretrained diffusion-based image decoder, and large-scale internal and synthetic datasets. The comparisons in Tables 1 and 2 therefore do not isolate the contribution of the proposed techniques from differences in training data, decoder quality, and computational resources. Table 3(c) provides useful component-level evidence, but it does not include a complete factorial analysis of pre-caption, pre-MIM, D2C, and TAN, making it difficult to attribute the final performance gains precisely.

Consequently, while the empirical results are impressive, the evidence does not yet establish how much of the full-system improvement can be attributed specifically to the proposed contributions rather than the strong pretrained decoder, training data, and extensive downstream optimization.

[1] MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer, Arxiv 2025

[2] DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies, ICLR 2026

[3] UniWeTok: An Unified Binary Tokenizer with Codebook Size 2128\mathit{2^{128}} for Unified Multimodal Large Language Model, Arxiv 2026

Questions for authors

Q1. What is the primary mechanism underlying the benefits of pre-quantization supervision?

Since post-quantization losses already propagate gradients to the pre-quantization representation through STE, could the authors provide a more detailed explanation or empirical analysis of why direct pre-quantization supervision is superior?

For example, would the improvements persist when matching the magnitude and distribution of gradients received by the pre-quantization latent? Such an analysis would help establish whether the benefits arise from the quantization boundary itself or from additional supervision and optimization effects.

Q2. How do the proposed tokenizer objectives affect autoregressive token modeling and perceptual reconstruction quality?

Could the authors report additional reconstruction metrics, particularly rFID and LPIPS, for the variants in Table 3(a)? Furthermore, how do pre-caption and pre-MIM affect visual-token prediction loss, validation perplexity, or downstream generation performance when trained under matched conditions?

This would clarify whether increased code-use diversity and improved reconstruction translate into representations that are easier or harder for an autoregressive LLM to model.

Q3. How much of TAN's improvement is attributable to caption-supervised vocabulary adaptation rather than embedding-scale initialization?

The fixed normalization baselines in Table 3(b) achieve approximately 0.34–0.35 on GenEval, whereas TAN reaches 0.40. Could the authors investigate whether the improvement persists with longer downstream training or alternative learning rates?

Additionally, evaluating an adapted vocabulary with a comparable scale-matching parameterization but without TAN's specific normalization would help isolate the contribution of the proposed adaptation procedure. This would strengthen the argument that TAN provides benefits beyond better initialization and optimization conditioning.

Q4. Can the authors better isolate the contributions of the proposed components to the final unified model?

Table 3(c) demonstrates incremental improvements, but it does not separately evaluate the effects of pre-quantization supervision under the same joint-training configuration. A more complete component ablation, ideally including the combined and individual effects of pre-caption, pre-MIM, D2C, and TAN, would help establish their respective contributions.

It would also be valuable to clarify whether the improvements observed under simplified 0.6B training remain consistent when approaching the full training recipe.

4 comments

Dear Reviewer fYH2,

We sincerely thank you for your time and thoughtful feedback. Below, we address the main concerns and clarify several aspects of our work.

W1-a: The dual-branch design follows MANZANO. Yes, we follow the shared-encoder dual-branch interface of MANZANO, including sampling one pathway per step with a small LLM, and we do not claim it as our contribution. What we study are two steps that MANZANO does not discuss. The first one is how to transfer the learned visual vocabulary into the LLM. MANZANO simply extends the LLM embedding table with 64K image tokens. In our Table 3(b), with the tokenizer and decoder fixed, only changing the initialization of these weights changes GenEval from 0.15 (random) to 0.40 (TAN). The second one is where to supervise the discrete branch. MANZANO only supervises it after FSQ, which is the first row of our Table 3(a). We will rewrite the first contribution to make this clear.

W1-b: DualToken and UniWeTok. Thanks for pointing out these two works. We agree that supervising pre-quantization features is not new, and our original description was inaccurate. We will discuss both works in Sec. 2 and fix the statement in the introduction.

DualToken keeps the 26th-layer features of the SigLIP encoder close to their initial values with an L2 loss before quantization, to preserve semantics while the shallow layers learn pixel reconstruction. UniWeTok (Pre-Post Distillation) distills a SigLIP2 teacher into the latents both before and after quantization, and reports 0.10% / 41.51% / 51.32% zero-shot top-1 for pre-only / post-only / both.

Our pre+post caption supervision is similar to Pre-Post Distillation in structure. We replace feature distillation with caption prediction, and we do not claim this part as new. The main difference is in the detail objective. In both works, the loss before quantization is a semantic loss, and the visual details are learned through pixel reconstruction from the quantized features, i.e., always after quantization. Our tokenizer has no pixel reconstruction loss. The details come from VAE-latent prediction on the latent before the FSQ bottleneck. We also compare it with the same objective applied after quantization, with the same pathway sampling and loss weights: PSNR 18.25 vs. 16.50, LPIPS 0.40 vs. 0.49, rFID 3.2 vs. 3.5 (table in Part 2). Our observation is also different from UniWeTok. In UniWeTok, pre-only distillation cannot give semantics to the discrete latents. In our case, the detail objective applied only before quantization does transfer to the discrete tokens, and it is better than applying it after quantization.

W1-c: TAN and existing normalization. We agree that TAN is built on the text-scale-aware normalization of Li et al. and uses the same parameterization (LayerNorm with gain T/sqrt(D) and zero bias). The conceptual novelty here is limited, and the difference is in how it is used. In Li et al., the layer is placed after the visual projector, works on continuous features, and is kept in the model. TAN is applied to the 64K vectors of the discrete vocabulary only once after caption adaptation, and the output is used to initialize the visual input embeddings and output weights of the LLM. In addition, Li et al. only study the input side. Their analysis on a discrete vocabulary (Ovis-2.5) shows a norm gap but no clear update asymmetry. In our case the scale still matters a lot for generation (Raw 0.19 vs. L2 0.35). One reason is that these vectors are also the output classifier weights of visual tokens, so their scale directly affects the logits. We think the contribution of this part is mainly empirical: with the tokenizer fixed, the initialization of the visual vocabulary has a large effect on generation. More results are in Part 3 (Q3).

Q1: Why does pre-quantization supervision help? We have a structural explanation and some empirical evidence, but we admit that the mechanism is not fully verified.

In our tokenizer, the discrete branch outputs a 512-d latent z, and FSQ projects it to 6 dimensions with a linear layer before bounding and rounding. So the gradient of any post-quantization loss reaches z only through this 6-d projection (and the STE), and increasing the loss weight only rescales it. In contrast, a pre-quantization loss supervises all 512 dimensions of z directly.

For the four possible reasons you listed, our current results suggest the following. A stronger visual signal does help: VAE-latent prediction improves reconstruction even when applied after quantization, and most of the gain remains without the mask (see our response to Reviewer i1tM). Simply adding one more optimization pathway is not enough: the post-quantization variant has the same additional pathway but a much smaller gain, and doubling the post-caption weight gives no gain. The distribution of supervision is not a factor for the two VAE-latent variants, since they share the same pathway sampling, but it is a factor for the pre-caption comparison (see our response to Reviewer gujj). For avoiding the gradient approximation, we do not have a direct experiment: there are two steps between z and the quantized feature (the 512-to-6 projection and the rounding), and our comparison cannot tell which one matters.

On gradient matching, we only have the 2x weight experiment and have not done exact gradient-norm matching. The distribution of the gradient cannot be matched by a post-quantization loss because of the 6-d constraint above. We will make these limitations clear in the paper. To fully answer this question, two more experiments are needed: gradient-norm matching, and applying the objective after the projection but before rounding.

Q2 and W2: Evaluation metrics. We agree that code-use PPL only reflects utilization, and PSNR/SSIM are limited when a generative decoder is used. We add rFID and LPIPS for all rows in Table 3(a), and also the no-mask variant asked by Reviewer i1tM.

Post-cap weightD2CPre-capVAE-latent pred.PPLPSNRSSIMrFIDLPIPS
1----13.320.334.20.55
1Y--16K15.060.374.30.53
2Y--16K15.020.364.20.51
1YY-25K15.730.423.80.53
1YYPost, mask29K16.500.563.50.49
1YYPre, mask38K18.250.603.20.40
1YYPre, no mask (new)36K18.030.623.50.43

The full tokenizer is also the best on rFID and LPIPS, and pre- vs. post-quantization shows the same trend as PSNR. There are two cases where the new metrics do not fully agree with PSNR/SSIM. Pre-caption improves rFID (4.3 to 3.8) but not LPIPS (0.53). D2C improves PSNR and SSIM, but rFID and LPIPS are almost the same.

Q2, Q4 and W2: Downstream results and component isolation. We add two rows to Table 3(c): the tokenizer with post-caption only, and the no-mask variant. All rows use the same joint training (C.L. = pre- and post-caption).

Tokenizer objectivesInit.GenEvalDPGTextVQAMMB-EN
Post-cap only (new)L20.4866.2037.6447.43
C.L.L20.5267.7637.1248.45
C.L. + D2CL20.5469.0537.0148.30
C.L. + D2CTAN0.5871.7238.8050.31
C.L. + D2C + VAE pred. (mask)TAN0.6273.1540.5152.09
C.L. + D2C + VAE pred. (no mask, new)TAN0.6372.9440.3252.37

With the new first row, pre-caption is isolated under the unified training: GenEval +0.04, DPG +1.56, and mixed results on understanding. Now each of the four components has one step where only this component is changed. Both pre-quantization objectives increase the code-use PPL and improve downstream generation, so more diverse codes do not hurt generation in our setting. We did not measure the visual-token prediction loss separately. We acknowledge that this table is still sequential rather than a full factorial (which needs 16 tokenizer/decoder/LLM pipelines), that the pre-caption step also changes the pathway sampling, and that we did not repeat the ablations with the full training recipe (as noted in our Limitations).

Q3: Where does the gain of TAN come from? Please note that TAN is only an initialization, so we only claim that it helps under the training budgets we tested. To separate caption adaptation from scale matching, we add two baselines (also requested by Reviewer i1tM): a randomly initialized vocabulary trained with the same caption adaptation, with or without rescaling to the text norm.

VocabularyNormCaption lossGenEval
Random, w/o caption0.93-0.15
Random + caption (new)0.670.860.12
Random + text norm + caption (new)1.030.830.14
TAN w/o adaptation0.93-0.34
TAN0.960.780.40

(Norm is that of the transferred vocabulary. Caption loss is measured at the end of caption training.) The caption loss of the random vocabularies is close to TAN, so the LLM can already read them, but they bring no gain for generation. Scale matching alone gives 0.34-0.35 and caption training alone gives 0.12-0.14. Therefore the gain of TAN requires the tokenizer vocabulary, the text scale and caption adaptation together. For other learning rates, Table 3(c) uses joint training with lr 1.5e-4 (5e-5 in Table 3(b)), and TAN is still better than L2 (GenEval 0.58 vs. 0.54, DPG 71.72 vs. 69.05). Due to limited time, we have not run longer training. For the adapted vocabulary with scale matching but without TAN's normalization, the closest ones we have are Proj.+L2 (0.36) and the second baseline above. We have not run the exact setting (tokenizer vocabulary with Text-L2 and a learnable affine).

W2: Headline results. We agree that Tables 1 and 2 are system-level comparisons and cannot isolate the contribution of each component. We will state this clearly in the main text. We would like to clarify one point: the decoder does not take the text prompt as input, so the text-image alignment measured by GenEval and DPG-Bench cannot come from the decoder. In the table above, all rows use the same decoder configuration, downstream data and compute, and GenEval increases from 0.48 to 0.62 (DPG from 66.20 to 73.15). Table 3(b) only uses the public BLIP-3o-30M.

By construction, the post-quantization gradient at z lies in the 6-d row space of the 512-to-6 projection. In a small-scale probe, the pre-quantization caption gradient has only 1.4% of its energy in this subspace, so the two signals differ in distribution and not only in scale.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.