acceptodds
Under review as a conference paper at ICLR 2027

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

Abstract

Image tokenizers define the "visual language" of unified multimodal models, yet are commonly studied through isolated metrics or generation-/understanding-only evaluations. These evaluations do not fully capture how visual tokens behave when modeled jointly with text. We build a controlled pure-autoregressive testbed and track task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. We examine how these losses scale and relate to downstream performance, then use them to study multimodal learnability—how well image and text tokens are jointly modeled—and tokenizer design. We find that (1) losses should be analyzed by task, since they exhibit distinct scaling behavior and rank tokenizers differently; and (2) the loss–performance relationship depends on the predicted token space: for a fixed tokenizer, T2I and I2T losses correlate with generation quality, but across tokenizers, the T2I loss–performance relationship shifts with the image-token space, whereas I2T loss, computed over a shared text vocabulary, provides a more consistent signal. I2T loss also correlates with both generation and visual understanding performance after supervised finetuning. Using losses as a lens, we show that (3) better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and that (4) image tokenizer choice can affect text modeling under joint optimization. As case studies, we revisit three tokenizer design axes—the discriminator, semantic supervision, and vocabulary size—to examine their effects on joint modeling and downstream performance. Together, our testbed offers a complementary perspective on image tokenizers as visual languages, highlighting their interplay with text in joint multimodal training.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
image tokenizer, unified multimodal model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Xg8oOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2(−1)
Read review
Paper summary

The paper describes how the different losses associate to vision tokenizers in multimodal model training correlate with downstream performance.

The authors connect different VQGAN tokenizers to pre-trained LLM backbones and train them using three "families" of losses. Then, they compute the "log-likelihood on the supervised tokens" of unconditional text and image generation, alongside Image-to-text (I2T) and text-to-image (T2I) conditional generation.

The four different losses are measured for various training lengths (as measured by quantity of data ) and model sizes. For each variation, downstream performance is computed for several benchmarks.

The main results show that, in their particular case, the I2T loss correlates well with downstream quality even when considering all different tokenizers, while the T2I correlates only when controlled against the tokenizer type.

The final section studies some small empirical phenomena: the additional semantic loss does not correlate with generation performance and strengthen image-token–word alignment, the text process in the backbone is modified when the model is fine-tuned to output image tokens and a larger vocabulary increases downstream performance.

Critical strengths

The experiments span a large number of configurations and cover a wide span of possibilities for tokenizer type and size. Losses are tracked and computed at different checkpoints, allowing to probe the dynamics of the considered quantities along the training.

Critical weaknesses

The authors present the work as a study on image tokenizers as "visual languages in unified multimodal model".

First of all, they only consider tokenizers that encode images into a single pre-trained backbone family and generate output image from image tokens. Thus, their picture of unified multimodal model is very narrow and disregards native multimodal models, or non VQGAN encoders, or separate generative output.

Moreover, it is not so much a study on the image tokenizers as a whole but on the specific quantity which is their loss. Very little is said about the internal interpretability, or their geometry of representation or else. I understand that studying the loss of the tokenizer is the main focus of the paper, and in itself it may be a valid one. However, the title suggests quite a different story than what it actually presented in the paper. The limited part of the paper dedicated to the study of differences between tokenizer type (Section 5) is not sufficiently interesting per se to warrant pubblication, and does not use in any meaningful way the loss as an analysis tool: Section 5.1, 5.3 and 5.4 would read mostly unchanged without the discussion on losses, and the finding of Section 5.2 could be done by benchmarking on downstream tasks without particular effort.

For these reasons, the paper is difficult to navigate, and - at least to me - it reads quite disjointed. The title and introduction state an objective which feels out of line with the rest of the paper. Thus, the authors fail to show what the point of their research exactly is.

The paper has not demonstrated how (and why) should the measurements of losses be actively used in training and or in understanding the outputs.

As an example, Section 5.1 claims that to compare tokenizers the rFID reconstruction metric is not sufficient, since it does not correlate with downstream tasks ("reconstruction alone is therefore insufficient for tokenizer selection."). However, neither does the "multimodal learnability", which for the GigaTok case does not change at all, and even for the UniTok case has only one value significantly moving out of three.

As another example, in Section 5.4 the paper shows that losses can be in general "non-monotonic" with three vocabulary sizes. The I2T loss may be coherent with the benchmark score, but its variation is less than 0.1% and with only three points this correlation would require more study to be truly validated.

In its current form the paper does not present a coherent narrative nor sufficient novelty or potential applications to warrant acceptance.

Questions for authors

The main clarification I ask the authors is to clearly state the premises and scope of the paper and the different sections.

  • "Multimodal learnability" is not specifically defined anywhere, even if it is considered a fundamental aspect of discussion. Crossreferencing the numbers presented in Table 3 and Figure 2 I assume it just to mean "log-likelihood on the supervised tokens". Is this the case?

  • In Section 5.2 the authors ask the question "Can the image tokenizer affect text modeling even when the text-only data, text tokenizer, and language backbone are all fixed?". They answer with "Thus, semantic supervision or a smaller image vocabulary can improve text modeling despite an unchanged text token space." This leaves me confounded. What do they mean by fixed language backbone here? If it is frozen, I do not understand how the loss can change. I imagine that fixed means that the starting point of training is the same for all of them? If this is the case, I don't find the results surprising. Having a shared embedding space in the backbone will change the parameters of the model; when the model is trained to output new tokens, necessarily the parameters to be changed will be those closer to the output, modifying also text generation.

  • In Table 3 the paper states that UniTok-sem "improves all three losses". This statement includes a 2.883 to 2.873 and a 1.670 to 1.655 reductions, which may not be significative above noise.

6 comments

We thank the reviewer for the constructive feedback. We first clarify the paper’s scope, then address the weaknesses and questions.

Premises and scope of the paper and its sections

The paper is a controlled, loss-based study of how the image token space shapes joint image–text modeling in unified autoregressive (AR) multimodal models. It is not a survey of tokenizers or an interpretability study of tokenizer representations. Our premises are:

  • Model setup. A pure-AR unified model trained to predict both text and image tokens autoregressively. We extend a pretrained Qwen3 backbone with image-token embeddings and LM-head outputs and train it with a shared image–text objective (conditional next-token prediction).
  • Tokenizer setup. Discrete, single-codebook image tokenizers at a fixed resolution (256×256) and token length (16×16). The tokenizers are frozen and used to pre-tokenize images.
  • Metrics. Task-specific validation loss — the per-token negative log-likelihood on the supervised tokens — evaluated separately for text, image, T2I, and I2T prediction. This is how we measure multimodal learnability.

The sections are organized as follows:

  • Section 3 defines the testbed, and Section 2 introduces the tokenizers and design axes.

  • Section 4 establishes how to interpret these losses in unified training. Section 4.1 examines how each task’s loss scales and why the losses should be considered separately. Section 4.2 relates them to generation and understanding benchmarks, including I2T loss as a more consistent signal, vocabulary normalization for image-token losses, and how reconstruction fidelity is associated with residual shifts in the loss–performance relationship (Figure 7).

  • Section 5 uses the losses as a diagnostic lens to study the downstream effects of tokenizer design: fidelity vs. learnability (Section 5.1), effects on text modeling (Section 5.2), and the impact of semantic supervision (Section 5.3) and vocabulary size (Section 5.4) on multimodal learnability.

In the revised version, we have added boxed takeaways to each subsection in Sections 4–5 to clarify its role and findings.

Clarification on the paper summary

The reviewer's summary states that "the additional semantic loss does not correlate with generation performance and strengthen image-token–word alignment." We would like to clarify, since the second half does not match our claim and the first half is not a claim of the paper. We do not claim that semantic supervision is uncorrelated with generation performance. On the contrary, UniTok-sem improves generation and understanding relative to UniTok (Table 3), and our Section 5.3 studies how it does so: the I2T gain concentrates on visually grounded object words, and validation-set PMI between image tokens and words is higher for UniTok-sem on COCO object words, which suggest stronger object-level image-token–word association.

Weaknesses

We address the reviewer's concerns below, in the order they are raised.

W1 — The picture of a unified multimodal model is narrow: the study does not cover native multimodal models, non-VQGAN encoders, or separate generative output.

Our findings are established on Qwen3 and discrete, single-codebook tokenizers. As stated in the Limitations, extending the loss relationships or tokenizer rankings to other backbones and architectures requires matched retraining and evaluation.

We use this design for the following reasons:

  • Holding the backbone fixed is itself the control. Using Qwen3 keeps the backbone architecture and pretrained initialization consistent, avoiding confounding backbone changes with tokenizer differences.
  • The setup is chosen to give a shared modeling objective. We adopt a pure-AR model that predicts both text and image tokens, for which a single loss can be defined over both modalities. Other designs in this space (serial or hybrid AR + diffusion, decoupled modules, or separate tokenizers for the two tasks; see Related Work) change the objective or the architecture, so the image-token loss we compare across tokenizers is either not trivially defined (diffusion-based image generation) or not shared across the two tasks (separate tokenizers).

W2 — The paper studies the loss rather than the tokenizers themselves; it says little about internal interpretability or representation geometry.

We agree that we do not study the tokenizer's internal representation geometry. Our focus is the tokenizer’s downstream effects, which we examine through task-specific losses. Section 5 explores how the image token space shapes joint modeling.

W3 — The title suggests a different study than the paper presents.

The title refers to tokenizers’ role as the "visual language" of unified models. Task-specific loss is the analytical tool, analogous to analyses of text-only language modeling. The introduction makes this connection explicit: "we use task-specific validation losses during continual pretraining to characterize tokenizer effects."

W4 — Section 5 offers limited insights and does not clearly show what loss analysis adds beyond downstream benchmarking.

The purpose of the loss analysis is to examine how tokenizer choices affect different parts of joint image–text modeling and how those effects relate to downstream performance. Section 4 establishes how to interpret the measurements; Section 5 applies them to specific tokenizer design choices:

  • Section 5.1: Task-specific losses let us examine whether designs leading to better reconstruction also make the resulting image token space easier for the unified model to learn. This connects to the fidelity–learnability finding in the generation-only setting.
  • Section 5.2: Text loss measures how image tokenizer choices affect modeling of the same text data. The objective ablation shows that the text-loss gaps largely disappear when image-token prediction is removed, while the I2T gaps persist. This implicates image-token prediction in the text-side effect. The interference between image-token and text-token modeling is only observable under a shared objective, and we include this subsection to show why studying tokenizers in a unified setting yields findings beyond generation-only or understanding-only settings. We agree that text-only benchmarks applicable to pre-SFT models could be an alternative to text loss in this analysis, but text loss is defined at every checkpoint and across data and model scales, whereas downstream benchmarks are evaluated sparsely. It therefore provides a denser signal of the same text-side behavior.
  • Section 5.3: Decomposing I2T loss reductions by word group shows that UniTok-sem’s gains concentrate on visually grounded object words. The loss analysis identifies where caption modeling improves.
  • Section 5.4: Comparing task-specific losses reveals different vocabulary preferences. This motivates examining modeling difficulty together with reconstruction fidelity when interpreting vocabulary-size effects.

On the examples from Sections 5.1 and 5.4. Both of the reviewer’s examples treat the loss as a standalone predictor of downstream performance, which is not the role it plays here. Section 4.2.1 shows that residual shifts in the loss–performance relationship are associated with reconstruction fidelity (Figure 7). Neither is expected to predict benchmark performance on its own. The examples are instances of this divergence rather than counterexamples to it: for Section 5.1, a substantial reconstruction improvement (rFID 0.81→0.51) brings no consistent improvement in task losses or downstream performance; for Section 5.4, the normalized image-side losses favor IBQ-8192 while the downstream results favor IBQ-16384. In addition, we agree that the I2T differences in Section 5.4 are small. The revised appendix reports the loss ordering of the three IBQ variants across two 0.6B learning-rate and batch-size settings, showing that the pattern generalizes across settings.

W5 — The paper is difficult to navigate and reads disjointed, and it does not present a coherent narrative or sufficient novelty and potential applications.

The paper follows a single progression: Section 4 establishes how to interpret the task-specific losses in unified training — how they scale, and how they relate to generation and understanding benchmarks — and Section 5 uses those losses as a lens to study tokenizer design.

Our framework provides a per-task, cross-token-space account of what these losses mean and which of them are comparable across tokenizers, together with the tokenizer-design findings it enables (Findings 4–8). Neither is available from reconstruction metrics or single-axis generation-/understanding-only benchmarks, and the framework offers another perspective on studying tokenizers.

Questions

Q1 (definition of "multimodal learnability"). The reviewer's understanding is correct. We define multimodal learnability in the introduction as how well image and text tokens are jointly modeled under the shared autoregressive objective, and it is measured by the task-specific validation losses — the per-token negative log-likelihood on the supervised tokens, evaluated separately for the text, image, T2I, and I2T tasks. Following Liquid, we compute the loss only on the target-modality tokens, leaving the prompt and the conditioning modality unscored, so "learnability" refers to how well the same model predicts these targets under a fixed objective, data, and training setup. We have added this link explicitly in Section 5.1.

Q2 ("fixed language backbone"). "Fixed" means fixed architecture, fixed pretrained initialization, and fixed recipe; the model is not frozen and only the newly added image-token embeddings and output rows differ across tokenizers. We agree the wording was unclear and have refined it in the revised version.

The reviewer's reasoning — that training the model to accommodate new tokens changes parameters and thus modifies text generation — is partly what we observe, and on its own we would not find it surprising. What we found is more specific:

  • The text-loss gap appears only under the joint objective. When we remove image-token prediction by reformatting T2I and unconditional samples into I2T order with the hyperparameters fixed, the text-side gaps between the UniTok and IBQ pairs largely disappear, while the I2T gaps persist. So the effect is caused by the image-token prediction objective but not the captioning (I2T) objective.

Q3 (is the UniTok-sem loss improvement significant above noise?). We agree that the quoted reductions (2.883→2.873, 1.670→1.655) are small. We have not yet assessed these gaps across seeds, but have checked whether the reductions persist across learning rates and batch sizes:

Task(1e-4, 512)(3e-5, 512)(3e-5, 2048)
I2T−0.016−0.015−0.005
T2I−0.381−0.416−0.463
Text−0.015−0.010−0.004

UniTok-sem has lower I2T, T2I, and text losses in all three settings. The reductions are largest for T2I, while the smaller text and I2T improvements retain the same direction across settings. In the revised version, we have added this table to the appendix.

Reviewer_jVjKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that image tokenizers for unified multimodal models should be evaluated inside joint image–text autoregressive training, not by reconstruction metrics alone. It sets up a controlled testbed and uses separate validation losses for each task as the measuring stick. Its main findings are that I2T loss is the most reliable signal for comparing tokenizers, that better reconstruction doesn't guarantee better joint learning, and that the choice of image tokenizer can even affect text modeling.

These claims are tested on Qwen3 models from 0.6B to 4B with seven tokenizers. The evidence comes from scaling sweeps, correlations between loss and generation/VQA benchmarks, and ablations on discriminator, semantic supervision and vocabulary size.

Critical strengths

Strength 1: I2T loss is a tokenizer-agnostic diagnostic. I2T loss is computed over the shared text vocabulary, so it stays aligned with generation quality across tokenizers, unlike T2I loss. Measured before SFT, it also moderately predicts post-SFT VQA. It gives practitioners a cheap way to screen tokenizers without training each one all the way through SFT.

Strength 2: Controlled evidence that reconstruction != joint learnability, and that image tokenizers affect text modeling. In within-family ablations, better rFID (GigaTok-DINO) brings no downstream gain, while worse rFID with semantic supervision (UniTok-sem) improves losses, generation and VQA. Image tokenizer choice also shifts text loss, which ablation traces to the image-prediction objective. This directly challenges reconstruction-based tokenizer selection. The evidence is also the paper's least confounded.

Critical weaknesses

Weakness 1: Cross-family comparisons are confounded. The three tokenizer families differ in model size (IBQ about 0.1B, GigaTok about 0.6B, UniTok 0.8B), training data and image preprocessing. Even the continual-pretraining learning rate differs between families in Table 3. The headline claim that I2T loss is a consistent cross-tokenizer signal depends on these comparisons. The observed differences therefore cannot be cleanly attributed to the image token space.

Weakness 2: Loss–performance correlations pool model and data scales. The reported correlations (Figs. 4, 5 and 8) mix 0.6B–4B models and different data scales. The high r values may mostly show that larger scale lowers loss and raises scores, rather than measure tokenizer-level differences. This weakens the paper's main practical claim, that pretraining loss can be used to select tokenizers.

Questions for authors

None

3 comments

We thank the reviewer for recognizing the value of our findings. We understand the reviewer's central concerns to be the confounders in the cross-family tokenizer comparisons and the pooled loss-performance correlation analysis. We address these concerns below.

Weakness 1: Confounders in cross-family tokenizer comparisons

We clarify below which factors are under study, which are controlled, and which tokenizer pairs are compared.

  • "The three tokenizer families differ in model size (IBQ about 0.1B, GigaTok about 0.6B, UniTok 0.8B), training data and image preprocessing." Along with training objectives and architectural choices, these factors shape the visual token space produced by tokenizer training. Our cross-family comparison therefore evaluates complete tokenizers rather than isolating individual design factors. In Section 4, we evaluate I2T loss as an empirical diagnostic across these complete tokenizers: we do not attribute a cross-family difference to any one of these factors.

    In Section 5, we isolate individual design choices in the within-family ablations: discriminator (GigaTok vs GigaTok-DINO), semantic supervision (UniTok vs UniTok-sem), and vocabulary size (IBQ). Each comparison uses an identical training protocol on both sides, with only the intended axis changing. These conclusions are therefore not confounded by the cross-family differences identified by the reviewer.

  • "Even the continual-pretraining learning rate differs between families in Table 3." Table 3 in Section 5 compares tokenizers within each family instead of ranking all four tokenizers jointly. In the revised version, we have grouped the rows by family and explicitly stated that all comparisons are within-family, with identical learning rates, batch sizes, data mixtures, and step counts within each comparison.

  • "The headline claim that I2T loss is a consistent cross-tokenizer signal depends on these comparisons. The observed differences therefore cannot be cleanly attributed to the image token space." The claim concerns whether I2T loss tracks downstream performance across the evaluated tokenizers. It does not attribute performance differences to individual tokenizer design or training choices.

Weakness 2: Pooled loss-performance correlation analysis

We agree that pooling model and data scales could make the reported correlations reflect a scale trend rather than tokenizer-level differences. We clarify the purpose of Figure 4 and report controlled analyses for Figures 5 and 8.

In the figures, each point represents a configuration defined by tokenizer, model size, data scale, learning rate, and batch size. The three control levels are: (i) pooled, where all of these vary; (ii) fixed model size (0.6B), where model size is held fixed and tokenizer, data scale, learning rate, and batch size still vary; and (iii) fixed model and data scale (0.6B, 48M samples), where model size and data scale are fixed and tokenizer, learning rate, and batch size still vary. For the fixed-model, fixed-data analysis, we use one final checkpoint per run. Sample sizes are reported with each correlation.

Throughout, r is the Pearson correlation between the pre-SFT validation loss and the benchmark score, so the expected sign depends on the benchmark: negative for VQAv2, GQA, and GenAI-all (lower loss, higher score) and positive for MJHQ-30K gFID (lower loss, lower FID).

Figure 4 (within tokenizer, pre-SFT loss vs pre-SFT benchmarks)

Figure 4 examines within-tokenizer relationships: for a fixed tokenizer, it asks whether loss tracks generation quality across model sizes, data scales, and hyperparameters. It is not used to select or rank tokenizers. The controlled analyses for Figures 5 and 8 below address the reviewer's concern.

Figure 5 (across tokenizers, pre-SFT loss vs pre-SFT benchmarks)

I2T loss:

ControlI2T loss vs GenAI-allI2T loss vs MJHQ-30K
Pooled, no controlr=−0.86 (p<1e-5, n=185)r=+0.82 (p<1e-5, n=156)
Fixed model size (0.6B)r=−0.91 (p<1e-5, n=159)r=+0.82 (p<1e-5, n=156)
Fixed model and data scale (0.6B, 48M)r=−0.87 (p=3e-7, n=21)r=+0.68 (p=3e-4, n=23)

For T2I loss, the relationship is tokenizer-dependent: it does not align with text-to-image generation quality across tokenizers, as stated in the caption of Figure 5.

Figure 8 (across tokenizers, pre-SFT loss vs post-SFT benchmarks)

Generation (GenAI-all, MJHQ-30K):

ControlI2T loss vs GenAI-allI2T loss vs MJHQ-30K
Pooled, no controlr=−0.86 (p<1e-5, n=39)r=+0.88 (p<1e-5, n=39)
Fixed model size (0.6B)r=−0.84 (p<1e-5, n=37)r=+0.87 (p<1e-5, n=37)
Fixed model and data scale (0.6B, 48M)r=−0.89 (p=4e-6, n=16)r=+0.78 (p=4e-4, n=16)

Understanding (VQAv2, GQA):

ControlI2T loss vs VQAv2I2T loss vs GQA
Pooled, no controlr=−0.59 (p<1e-4, n=38)r=−0.62 (p<1e-4, n=39)
Fixed model size (0.6B)r=−0.50 (p=0.002, n=36)r=−0.55 (p<1e-3, n=37)
Fixed model and data scale (0.6B, 48M)r=−0.46 (p=0.074, n=16)r=−0.47 (p=0.068, n=16)

For the generation benchmarks, the correlations remain strong and statistically significant when model and data scale are fixed, suggesting that the relationship with I2T loss is not driven solely by scale. For VQAv2 and GQA, the correlations remain moderate (r = −0.46 and −0.47), although they do not reach statistical significance (p = 0.074 and 0.068). Their direction and magnitude are consistent with the moderate association reported in Section 4.2.2. For visual understanding, our claim is that pre-SFT I2T loss provides an informative diagnostic signal of post-SFT VQA performance across the evaluated configurations. Selecting tokenizers for understanding tasks still requires downstream evaluation, consistent with the limitation stated in Appendix F. We have added these controlled analyses to the revised version.

AuthorsOpenReview ↗

Request for the reviewer's feedback

Dear Reviewer jVjK,

Thank you again for the constructive review. As the deadline for the discussion period approaches, we would welcome your comments on our response. We are happy to provide any further clarification or results that would help. If the response addresses your concerns, we would be grateful if you would consider raising the rating.

Best,

Authors

Reviewer_QmpGOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper is an empirical study that treats image tokenizers as if they were visual languages, and asks whether a unified multimodal model's generative behavior is shaped more by the tokenizer it uses or by the training recipe it goes through. The authors train or adapt a unified model on several tokenizers drawn from different tokenizer families, and then study how tokenizer choice affects cross-entropy loss and downstream image-generation quality. Two observations drive the story. First, raw cross-entropy loss is not comparable across tokenizers because they use different codebooks and vocabulary sizes, so the paper introduces a normalization that lets loss and image quality be related fairly. Second, a tokenizer's characteristic performance profile tends to persist: it is not meaningfully reset by continued training or by mixing in more natural-image data, though the authors show that adding text-domain data does change the pattern. The overall picture is a careful measurement study that offers a cautionary lesson about comparing losses across tokenizers, an observation that tokenizer identity carries much of the downstream signal, and a practical guide to what does and does not shift that signal.

Critical strengths
  1. The paper turns a vague intuition into a concrete, checkable result: comparing cross-entropy loss across different tokenizers without normalizing for their different codebooks and vocabulary sizes is misleading, and once you normalize, the relationship between loss and downstream image quality becomes meaningful and consistent. This is consequential because it directly affects how practitioners read existing reports and choose tokenizers, and it is the kind of "obvious in hindsight" pitfall that a focused, well-measured study is exactly the right vehicle to expose. It is the finding I would most readily cite from this work.

  2. The study has a clear and useful conceptual angle: it reframes tokenizer selection as a matter of each tokenizer's intrinsic tendency, showing which factors leave that tendency intact and which genuinely change it. This matters because it gives a mental model for predicting behavior rather than just reporting isolated numbers, and it turns the paper from a benchmark table into something that offers real guidance for the community.

Critical weaknesses
  1. The findings beyond the main loss-comparability result are stated with more confidence than the evidence carries. The idea that a tokenizer's profile is intrinsic and unreset by continued training is drawn from very small gaps between settings, and the paper does not consistently report variance or error bars. Since the authors themselves note that several evaluations are sensitive to random seed and configuration, the small margins behind these conclusions may partly reflect noise rather than a real effect. Because these secondary findings carry a good share of the paper's claimed novelty, this is the concern most likely to shape my recommendation.

  2. The comparison across tokenizers is not fully controlled. Different tokenizer families are trained with different learning rates, and the training protocol is not identical across them. This is a plausible confound for a paper whose whole point is to fairly order tokenizers by their downstream effect: a difference attributed to the tokenizer itself could partly be an artifact of a different optimization setting. The headline conclusion would be much more trustworthy with the effect isolated from the training recipe.

Questions for authors
  1. Could you report the variance of your key metrics across seeds and configurations, especially for the findings that rest on small differences? I would like to know whether the intrinsic-tendency conclusions hold when the gaps are compared against the noise level you have already observed.
  2. How sensitive are the normalized-loss and downstream-quality conclusions to your specific normalization choice? If a reasonable alternative normalization were used, would the main comparability result survive unchanged?
2 comments

We thank the reviewer for recognizing the strengths of our paper and for the constructive questions. We understand the central concerns to be the significance and soundness of our secondary findings. We address these concerns below.

Weakness 1 and Question 1: Significance of the findings

We agree that the secondary findings rest on smaller gaps than the main loss-comparability result, and that those gaps need to be assessed against a noise reference. We therefore quantify the two axes of variation the reviewer names: random seed and optimization configuration.

  • Seed sensitivity under a fixed configuration (already reported in the appendix). Table 12 compares one GigaTok configuration (0.6B, lr=1e-4, bs=1024) under two data-shuffling seeds (42 and 37). VQAv2, GQA, GenAI-all, and MJHQ-30K differ by at most 1.9% in relative terms, whereas POPE (7.07%) and MME-P (4.98%) do not, which is why we exclude the latter two.

  • Consistency across optimization configurations (new analysis). We recompute the two within-family Table 3 comparisons at each of the three 0.6B settings for which both tokenizers in each comparison are available, using matched final checkpoints and pre-SFT losses:

ComparisonTask(1e-4, 512)(3e-5, 512)(3e-5, 2048)
UniTok → UniTok-semI2T−0.016−0.015−0.005
T2I−0.381−0.416−0.463
Text−0.015−0.010−0.004
GigaTok → GigaTok-DINOI2T−0.001+0.006−0.009
T2I+0.021+0.044+0.051
Text+0.011+0.005+0.001

At the final checkpoints, UniTok-sem improves all three losses across all three settings, whereas GigaTok-DINO shows no consistent task-loss improvement across settings — its T2I and Text losses rise at every setting, while its I2T gap changes sign across settings. These comparisons support the consistency of the observed trends across optimization settings. In the revised version, we have added the per-setting table to the appendix.

Weakness 2: The effect of optimization settings

We agree that a difference attributed to the tokenizer must not be an artifact of the optimization setting. We would like to clarify that the tokenizer comparisons we present control for optimization settings.

In Section 5, every effect we attribute to the image token space comes from a single-axis ablation within one tokenizer family, where the training protocol — learning rate, batch size, data mixture, preprocessing, and number of steps — is identical on both sides:

  • Discriminator: GigaTok vs GigaTok-DINO differ only in the discriminator.
  • Semantic supervision: UniTok vs UniTok-sem differ only in the added CLIP contrastive loss.
  • Vocabulary size: the IBQ variants differ only in codebook size.

Learning rates differ only between some families; these differences do not enter any comparison used to support a claim.

In Section 4, we present cross-family comparisons to show differences in scaling patterns and loss–performance relationships. All of these runs use the same optimization setting (lr=3e-5, bs=512). Hence, the optimization settings do not confound the results there.

To clarify the scope of the comparisons in Table 3 in Section 5, we have grouped the rows by family in the revised version and explicitly stated that all comparisons are within-family and that learning rate, batch size, data mixture, and step count are identical within each comparison.

Question 2: Sensitivity to normalization choice

We thank the reviewer for raising this concern. In Appendix D.6, we test an alternative normalization based on the empirical unigram entropy (H1H_1) of each tokenizer's image tokens on the validation set, rather than the maximum entropy (log⁡2B\log_2 B), using the same T2I-loss setting and set of points as in Figure 6. The results for GenAI-all are shown in Table 10. In the revised version, we have also added the numerical results for MJHQ-30K in Table 11 and moved the reference to Tables 10–11 into the main text. On both benchmarks, the comparability finding continues to hold under this empirical normalization.

BenchmarkNormalizationPearson rrR2R^2 from lossAdditional R2R^2 from tokenizer identity
GenAI-all (Table 10)None+0.0160.0000.972
Divide by log⁡2B\log_2 B−0.9570.9150.048
Divide by H1H_1−0.9530.9090.055
MJHQ-30K (Table 11)None+0.0390.0010.774
Divide by log⁡2B\log_2 B+0.8900.7920.046
Divide by H1H_1+0.8840.7810.057
AuthorsOpenReview ↗

Request for the reviewer's feedback

Dear Reviewer QmpG,

Thank you again for the careful reading and for the questions. As the deadline for the discussion period approaches, we would welcome your comments on our response. We are happy to provide any further clarification or results that would help. If the response addresses your concerns, we would be grateful if you would consider raising the rating.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.