- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2(−1)
Read review
Paper summary
The paper describes how the different losses associate to vision tokenizers in multimodal model training correlate with downstream performance.
The authors connect different VQGAN tokenizers to pre-trained LLM backbones and train them using three "families" of losses. Then, they compute the "log-likelihood on the supervised tokens" of unconditional text and image generation, alongside Image-to-text (I2T) and text-to-image (T2I) conditional generation.
The four different losses are measured for various training lengths (as measured by quantity of data ) and model sizes. For each variation, downstream performance is computed for several benchmarks.
The main results show that, in their particular case, the I2T loss correlates well with downstream quality even when considering all different tokenizers, while the T2I correlates only when controlled against the tokenizer type.
The final section studies some small empirical phenomena: the additional semantic loss does not correlate with generation performance and strengthen image-token–word alignment, the text process in the backbone is modified when the model is fine-tuned to output image tokens and a larger vocabulary increases downstream performance.
Critical strengths
The experiments span a large number of configurations and cover a wide span of possibilities for tokenizer type and size. Losses are tracked and computed at different checkpoints, allowing to probe the dynamics of the considered quantities along the training.
Critical weaknesses
The authors present the work as a study on image tokenizers as "visual languages in unified multimodal model".
First of all, they only consider tokenizers that encode images into a single pre-trained backbone family and generate output image from image tokens. Thus, their picture of unified multimodal model is very narrow and disregards native multimodal models, or non VQGAN encoders, or separate generative output.
Moreover, it is not so much a study on the image tokenizers as a whole but on the specific quantity which is their loss. Very little is said about the internal interpretability, or their geometry of representation or else. I understand that studying the loss of the tokenizer is the main focus of the paper, and in itself it may be a valid one. However, the title suggests quite a different story than what it actually presented in the paper. The limited part of the paper dedicated to the study of differences between tokenizer type (Section 5) is not sufficiently interesting per se to warrant pubblication, and does not use in any meaningful way the loss as an analysis tool: Section 5.1, 5.3 and 5.4 would read mostly unchanged without the discussion on losses, and the finding of Section 5.2 could be done by benchmarking on downstream tasks without particular effort.
For these reasons, the paper is difficult to navigate, and - at least to me - it reads quite disjointed. The title and introduction state an objective which feels out of line with the rest of the paper. Thus, the authors fail to show what the point of their research exactly is.
The paper has not demonstrated how (and why) should the measurements of losses be actively used in training and or in understanding the outputs.
As an example, Section 5.1 claims that to compare tokenizers the rFID reconstruction metric is not sufficient, since it does not correlate with downstream tasks ("reconstruction alone is therefore insufficient for tokenizer selection."). However, neither does the "multimodal learnability", which for the GigaTok case does not change at all, and even for the UniTok case has only one value significantly moving out of three.
As another example, in Section 5.4 the paper shows that losses can be in general "non-monotonic" with three vocabulary sizes. The I2T loss may be coherent with the benchmark score, but its variation is less than 0.1% and with only three points this correlation would require more study to be truly validated.
In its current form the paper does not present a coherent narrative nor sufficient novelty or potential applications to warrant acceptance.
Questions for authors
The main clarification I ask the authors is to clearly state the premises and scope of the paper and the different sections.
-
"Multimodal learnability" is not specifically defined anywhere, even if it is considered a fundamental aspect of discussion. Crossreferencing the numbers presented in Table 3 and Figure 2 I assume it just to mean "log-likelihood on the supervised tokens". Is this the case?
-
In Section 5.2 the authors ask the question "Can the image tokenizer affect text modeling even when the text-only data, text tokenizer, and language backbone are all fixed?". They answer with "Thus, semantic supervision or a smaller image vocabulary can improve text modeling despite an unchanged text token space." This leaves me confounded. What do they mean by fixed language backbone here? If it is frozen, I do not understand how the loss can change. I imagine that fixed means that the starting point of training is the same for all of them? If this is the case, I don't find the results surprising. Having a shared embedding space in the backbone will change the parameters of the model; when the model is trained to output new tokens, necessarily the parameters to be changed will be those closer to the output, modifying also text generation.
-
In Table 3 the paper states that UniTok-sem "improves all three losses". This statement includes a 2.883 to 2.873 and a 1.670 to 1.655 reductions, which may not be significative above noise.
6 comments
Follow-up
W3 — The title suggests a different study than the paper presents.
The title refers to tokenizers’ role as the "visual language" of unified models. Task-specific loss is the analytical tool, analogous to analyses of text-only language modeling. The introduction makes this connection explicit: "we use task-specific validation losses during continual pretraining to characterize tokenizer effects."
W4 — Section 5 offers limited insights and does not clearly show what loss analysis adds beyond downstream benchmarking.
The purpose of the loss analysis is to examine how tokenizer choices affect different parts of joint image–text modeling and how those effects relate to downstream performance. Section 4 establishes how to interpret the measurements; Section 5 applies them to specific tokenizer design choices:
- Section 5.1: Task-specific losses let us examine whether designs leading to better reconstruction also make the resulting image token space easier for the unified model to learn. This connects to the fidelity–learnability finding in the generation-only setting.
- Section 5.2: Text loss measures how image tokenizer choices affect modeling of the same text data. The objective ablation shows that the text-loss gaps largely disappear when image-token prediction is removed, while the I2T gaps persist. This implicates image-token prediction in the text-side effect. The interference between image-token and text-token modeling is only observable under a shared objective, and we include this subsection to show why studying tokenizers in a unified setting yields findings beyond generation-only or understanding-only settings. We agree that text-only benchmarks applicable to pre-SFT models could be an alternative to text loss in this analysis, but text loss is defined at every checkpoint and across data and model scales, whereas downstream benchmarks are evaluated sparsely. It therefore provides a denser signal of the same text-side behavior.
- Section 5.3: Decomposing I2T loss reductions by word group shows that UniTok-sem’s gains concentrate on visually grounded object words. The loss analysis identifies where caption modeling improves.
- Section 5.4: Comparing task-specific losses reveals different vocabulary preferences. This motivates examining modeling difficulty together with reconstruction fidelity when interpreting vocabulary-size effects.
On the examples from Sections 5.1 and 5.4. Both of the reviewer’s examples treat the loss as a standalone predictor of downstream performance, which is not the role it plays here. Section 4.2.1 shows that residual shifts in the loss–performance relationship are associated with reconstruction fidelity (Figure 7). Neither is expected to predict benchmark performance on its own. The examples are instances of this divergence rather than counterexamples to it: for Section 5.1, a substantial reconstruction improvement (rFID 0.81→0.51) brings no consistent improvement in task losses or downstream performance; for Section 5.4, the normalized image-side losses favor IBQ-8192 while the downstream results favor IBQ-16384. In addition, we agree that the I2T differences in Section 5.4 are small. The revised appendix reports the loss ordering of the three IBQ variants across two 0.6B learning-rate and batch-size settings, showing that the pattern generalizes across settings.
W5 — The paper is difficult to navigate and reads disjointed, and it does not present a coherent narrative or sufficient novelty and potential applications.
The paper follows a single progression: Section 4 establishes how to interpret the task-specific losses in unified training — how they scale, and how they relate to generation and understanding benchmarks — and Section 5 uses those losses as a lens to study tokenizer design.
Our framework provides a per-task, cross-token-space account of what these losses mean and which of them are comparable across tokenizers, together with the tokenizer-design findings it enables (Findings 4–8). Neither is available from reconstruction metrics or single-axis generation-/understanding-only benchmarks, and the framework offers another perspective on studying tokenizers.
Questions
Q1 (definition of "multimodal learnability"). The reviewer's understanding is correct. We define multimodal learnability in the introduction as how well image and text tokens are jointly modeled under the shared autoregressive objective, and it is measured by the task-specific validation losses — the per-token negative log-likelihood on the supervised tokens, evaluated separately for the text, image, T2I, and I2T tasks. Following Liquid, we compute the loss only on the target-modality tokens, leaving the prompt and the conditioning modality unscored, so "learnability" refers to how well the same model predicts these targets under a fixed objective, data, and training setup. We have added this link explicitly in Section 5.1.
Follow-up
Q2 ("fixed language backbone"). "Fixed" means fixed architecture, fixed pretrained initialization, and fixed recipe; the model is not frozen and only the newly added image-token embeddings and output rows differ across tokenizers. We agree the wording was unclear and have refined it in the revised version.
The reviewer's reasoning — that training the model to accommodate new tokens changes parameters and thus modifies text generation — is partly what we observe, and on its own we would not find it surprising. What we found is more specific:
- The text-loss gap appears only under the joint objective. When we remove image-token prediction by reformatting T2I and unconditional samples into I2T order with the hyperparameters fixed, the text-side gaps between the UniTok and IBQ pairs largely disappear, while the I2T gaps persist. So the effect is caused by the image-token prediction objective but not the captioning (I2T) objective.
Q3 (is the UniTok-sem loss improvement significant above noise?). We agree that the quoted reductions (2.883→2.873, 1.670→1.655) are small. We have not yet assessed these gaps across seeds, but have checked whether the reductions persist across learning rates and batch sizes:
| Task | (1e-4, 512) | (3e-5, 512) | (3e-5, 2048) |
|---|---|---|---|
| I2T | −0.016 | −0.015 | −0.005 |
| T2I | −0.381 | −0.416 | −0.463 |
| Text | −0.015 | −0.010 | −0.004 |
UniTok-sem has lower I2T, T2I, and text losses in all three settings. The reductions are largest for T2I, while the smaller text and I2T improvements retain the same direction across settings. In the revised version, we have added this table to the appendix.
We thank the reviewer for the constructive feedback. We first clarify the paper’s scope, then address the weaknesses and questions.
Premises and scope of the paper and its sections
The paper is a controlled, loss-based study of how the image token space shapes joint image–text modeling in unified autoregressive (AR) multimodal models. It is not a survey of tokenizers or an interpretability study of tokenizer representations. Our premises are:
The sections are organized as follows:
Section 3 defines the testbed, and Section 2 introduces the tokenizers and design axes.
Section 4 establishes how to interpret these losses in unified training. Section 4.1 examines how each task’s loss scales and why the losses should be considered separately. Section 4.2 relates them to generation and understanding benchmarks, including I2T loss as a more consistent signal, vocabulary normalization for image-token losses, and how reconstruction fidelity is associated with residual shifts in the loss–performance relationship (Figure 7).
Section 5 uses the losses as a diagnostic lens to study the downstream effects of tokenizer design: fidelity vs. learnability (Section 5.1), effects on text modeling (Section 5.2), and the impact of semantic supervision (Section 5.3) and vocabulary size (Section 5.4) on multimodal learnability.
In the revised version, we have added boxed takeaways to each subsection in Sections 4–5 to clarify its role and findings.
Clarification on the paper summary
The reviewer's summary states that "the additional semantic loss does not correlate with generation performance and strengthen image-token–word alignment." We would like to clarify, since the second half does not match our claim and the first half is not a claim of the paper. We do not claim that semantic supervision is uncorrelated with generation performance. On the contrary, UniTok-sem improves generation and understanding relative to UniTok (Table 3), and our Section 5.3 studies how it does so: the I2T gain concentrates on visually grounded object words, and validation-set PMI between image tokens and words is higher for UniTok-sem on COCO object words, which suggest stronger object-level image-token–word association.
Weaknesses
We address the reviewer's concerns below, in the order they are raised.
W1 — The picture of a unified multimodal model is narrow: the study does not cover native multimodal models, non-VQGAN encoders, or separate generative output.
Our findings are established on Qwen3 and discrete, single-codebook tokenizers. As stated in the Limitations, extending the loss relationships or tokenizer rankings to other backbones and architectures requires matched retraining and evaluation.
We use this design for the following reasons:
W2 — The paper studies the loss rather than the tokenizers themselves; it says little about internal interpretability or representation geometry.
We agree that we do not study the tokenizer's internal representation geometry. Our focus is the tokenizer’s downstream effects, which we examine through task-specific losses. Section 5 explores how the image token space shapes joint modeling.