acceptodds
Under review as a conference paper at ICLR 2027

Taming Visual Tokenizer Friendly for Large Language Model

Abstract

Extending pretrained language models to visual understanding and generation can compromise their existing language and reasoning capabilities. We investigate which properties of visual tokenization affect this trade-off and find that the organization of visual latents matters beyond their standalone visual quality. We develop LMTok, a 1D visual tokenizer whose latent organization is shaped by the next-token prediction objective of a pretrained language model, and compare it with a spatial 2D tokenizer under matched token lengths, codebook sizes, and downstream training budgets. Despite comparable reconstruction, generation, and visual understanding performance, training with LMTok helps the LLM better retain its language capabilities than training with spatial 2D tokens. With Qwen3-8B as the backbone, LMTok improves over spatial 2D tokenization by 1.72 points on MMLU-Pro, 3.40 points on MATH-500, and 5.40 points on MBPP, while maintaining comparable visual performance under unified training. This trend remains consistent when scaling the language-model backbone from 4B to 8B. Comparisons with existing 1D tokenizers and controlled ablations identify the autoregressive shaping as a key contributor to this improved compatibility. Overall, our results show that visual tokenizers should be evaluated not only by standalone visual quality, but also by how their sequence organization interacts with the autoregressive prior. The code and model will be released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Visual Tokenizer, Visual Generation, Unified Multimodal language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y9izOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the trade-off between acquiring visual capabilities and retaining language/reasoning abilities when extending pretrained Large Language Models (LLMs) to unified multimodal understanding and generation. The authors argue that the organization of visual latents significantly impacts this trade-off. They propose LMTok, a 1D visual tokenizer that decouples spatial feature extraction from latent sequence organization. LMTok uses a frozen LLM as an autoregressive (AR) prior during tokenizer training to shape the 1D token sequence via next-token prediction, while employing a cross-alignment mechanism to preserve 2D spatial semantics without fixing token-to-patch correspondence. Experiments show that under matched settings, LMTok achieves comparable standalone visual reconstruction and generation quality to spatial 2D tokenizers, but significantly mitigates language capability degradation during unified training on benchmarks like MMLU-Pro and MATH-500.

Critical strengths
  1. Novel and Insightful Perspective: Instead of solely chasing state-of-the-art visual reconstruction or generation metrics (which is the common trend in visual tokenizer research), the paper tackles a critical and often overlooked issue in unified multimodal models: the catastrophic forgetting of language capabilities. Formulating the tokenizer design as an "AR-shaping" problem to align with the LLM's native next-token prediction objective is a highly innovative and logical approach.

  2. Rigorous Controlled Ablations: The authors provide a very solid experimental methodology by strictly matching token lengths, codebook sizes, and downstream training budgets between the 1D LMTok and the 2D spatial baseline. The ablation studies thoroughly isolate the contributions of the AR prior, the 1D structure, and the cross-alignment, proving that the language retention gains come from the AR-guided sequence organization rather than just the 1D flexibility.

Critical weaknesses
  1. Misalignment of Evaluation Metrics and Baselines: The paper mainly compares LMTok against traditional spatial 2D tokenizers (like VQGAN derivatives) and older 1D tokenizers (TiTok). It fails to compare against state-of-the-art Unified Autoregressive Models (e.g., Emu3, Janus, Transfusion, Show-o) which use advanced techniques like next-scale prediction, decoupled encoders, or hybrid diffusion/autoregressive frameworks. Furthermore, it does not compare against standard VLMs (which use continuous vision encoders + projectors, avoiding discrete tokenization altogether). Without these comparisons, it is hard to judge if LMTok actually advances the frontier of unified modeling or just improves a suboptimal discrete baseline.

  2. Questionable Core Motivation and Practical Value: The core motivation is to prevent language degradation when an LLM is forced to predict discrete visual tokens autoregressively. However, the reason language degradation happens in unified discrete models is fundamentally due to modality interference and the massive distribution shift between text and pixel-level visual tokens. The current industry and academic consensus to solve this is either: (1) using frozen continuous vision encoders with projectors (standard VLMs), (2) decoupling the visual pathways for understanding and generation (like Janus), or (3) using hybrid models (text AR + visual Diffusion/Flow, like Transfusion). Forcing the LLM to natively predict discrete visual tokens—even if shaped by an AR prior—is an inherently inefficient and conflicting paradigm. The paper attempts to "patch" this paradigm rather than questioning its viability against continuous or hybrid alternatives.

Questions for authors
  1. How does LMTok perform when integrated into modern unified architectures (e.g., Janus or Transfusion) compared to their native tokenization or continuous representations?

  2. Given that modern VLMs and hybrid unified models successfully avoid language degradation by not forcing the LLM to autoregressively predict discrete visual tokens, what is the practical purpose of optimizing discrete tokenizers for language retention?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Y9iz

Thank you for the thoughtful review, and for the encouraging words on the perspective (“novel and insightful”) and the ablations (“very solid”). You raised two concerns. Here is the short version of our answers; details follow.

  • W1/Q1 (baselines, modern architectures): The language-retention advantage persists across all six benchmarks in a controlled Janus-style configuration.
  • W2/Q2 (why optimize discrete tokenizers): A purely discrete model reuses the full LLM toolchain, and its visual tokens are the component with the most room to improve. We agree that hybrid designs are strong; we see our work as complementary to them.

Q1 / W1. Integration into a decoupled architecture, and baselines

Following Q1, we integrate LMTok into a Janus-style setup: continuous SigLIP2 features for understanding, and discrete tokens (LMTok vs. the matched 2D tokenizer) as generation targets. Training is identical to Table 1 (Qwen3-4B, 100K steps).

Qwen3-4BMMLUMMLU-ReduxMMLU-ProMATH-500GSM8KMBPPGQAPOPEMME-PDPGGenEval
Text-only70.2067.1150.6747.080.5960.2–––––
Discrete input + 2D gen.65.7363.2243.7038.873.9153.849.9179.331253.884.267.6
Discrete input + LMTok gen.66.3764.8746.4643.875.9656.051.8782.511310.484.567.1
SigLIP2 input + 2D gen.64.1964.5943.4341.272.9354.662.2185.811584.683.167.1
SigLIP2 input + LMTok gen.66.6267.4445.0243.675.6657.862.3686.401556.782.667.3
  • LMTok is higher on all six language benchmarks (+2.5 on average), with matched understanding and generation.
  • Language degradation is still present with a decoupled encoder (MATH-500: 43.6 vs. 47.0 for text-only), so decoupling alone does not solve it. Language retention depends mainly on the visual sequence the backbone is trained to generate, and little on whether the input is discrete.

On system-level baselines: Emu3, Janus and Transfusion are trained on different and largely non-public data at different scales. An end-to-end comparison would confound the tokenizer with data, scale and architecture, and would not isolate the variable we study. Their generation interface, however, is the setting we control. Emu3 is a pure next-token predictor over discrete visual tokens, which is our main setting. Janus generates images by predicting discrete VQ indices in a trainable shared backbone, which is the experiment above.

Q2 / W2. Why optimize discrete tokenizers for language retention

Because a purely discrete model reuses the full LLM toolchain, and its visual tokens are the component with the most room left to improve.

  • What the discrete route offers. A shared categorical next-token interface gives exact token probabilities for both text and images, so token-based post-training carries over directly: DPO in Emu3, and GRPO over text and image tokens in T2I-R1 and Emu3.5. The same models reuse LLM serving stacks (vLLM in Janus and Emu3.5) and support interleaved, long-horizon generation (Emu3.5). These are motivations for the discrete interface, not results of this work.
  • Hybrid designs are strong alternatives, and we do not argue against them. They currently lead in image fidelity. However, they do not avoid language degradation automatically. LMFusion shows that, in the Transfusion setting, fine-tuning a pretrained LLM degrades language unless the text modules are frozen, and Show-o2 introduces a two-stage recipe specifically to retain language knowledge.
  • Decoupling and tokenizer design are complementary. The Janus-style experiment above shows both: degradation remains with a decoupled encoder, and LMTok reduces it.
  • What our study shows. “Modality interference” names the phenomenon, but it does not say which visual representation causes how much of it. We show that tokenizers with equal visual quality differ by 2–5 points on language benchmarks, and we identify the cause. Earlier work on discrete tokenizers (Emu3, UniTok) improved visual quality; compatibility with the language model is a new axis.

We have stated this scope explicitly in the revised introduction and related work.

Two small clarifications on our setup. The 2D baseline shares LMTok’s SigLIP2-initialized encoder, decoder and semantic supervision (rFID 0.98, gFID 1.84), so it is a matched semantic tokenizer. Both tokenizers are distilled from SigLIP2, and LMTok-MCQ approaches continuous SigLIP2 features in understanding (Table 3).

We hope the new experiment and the clarifications above address your concerns. If any further experiment or explanation would be helpful, we would be glad to provide it during the discussion period. Given the positive assessment of the perspective and of the ablations, we would be grateful if you would consider revisiting the evaluation of our work.

Reviewer_Y9izOpenReview ↗

Maintain my weak reject recommendation

Thank you for the detailed response and the additional Janus-style experiment and I have read. With the understanding encoder held fixed, LMTok improves all six language benchmarks while maintaining broadly comparable visual performance.

However, the additional experiments and rebuttal do not resolve my main concern regarding the significance and practical implications of the contribution. The experiments establish an improvement within a controlled discrete autoregressive setting, but do not establish when or why one should choose this setting over decoupled or hybrid alternatives. Improving language retention after requiring a shared LLM backbone to predict discrete visual tokens does not, by itself, demonstrate that this architectural choice offers a compelling tradeoff. Reusing token-based post-training and serving infrastructure is a reasonable motivation, but the associated benefits and costs are not evaluated here. I agree that direct comparisons against differently scaled full systems would not isolate the tokenizer’s contribution. My concern is therefore not the absence of every named system baseline, but the lack of evidence identifying a practical regime in which the proposed approach offers a meaningful advantage over alternative ways of combining language and image generation.

Table 4 also leaves questions about the methodological advance beyond existing AR-guided tokenization. The randomly initialized AR prior achieves broadly similar retention benefits, and the EOSTok reproduction is competitive on the reported language benchmarks. These results support the broader value of AR-guided visual representations, but make the distinct contribution of the proposed design less clear.

Taken together, these concerns leave me unconvinced that the contribution is sufficiently significant for acceptance, and I maintain my weak reject recommendation.

AuthorsOpenReview ↗

Follow-up to Reviewer Y9iz: scope of the claim and distinct contribution

Thank you for engaging further, and for confirming the Janus-style result, that system-level comparisons would not isolate the tokenizer, and that toolchain reuse is a reasonable motivation. Your comment also made us realize that our previous answer argued for the discrete route in general, which is more than this paper needs. Let us be precise.

  1. “When or why one should choose this setting over decoupled or hybrid alternatives”
  • We do not claim it should be chosen over hybrid designs. We have not compared the two, and the revision says so. Our claim is narrower: whenever a trainable LLM backbone generates images, language degrades, and the generation target determines how much.
  • When: a large and active family of models. Chameleon, Emu3, Emu3.5, Janus and Janus-Pro all generate images as discrete tokens in a trainable backbone, and Emu3.5 reports image generation and editing comparable to Gemini 2.5 Flash Image. Every such system faces the retention problem we measure.
  • Versus decoupled designs: it is not an either/or choice. Janus-style models still generate discrete tokens. In the Janus-style experiment you acknowledged, LMTok improves all six language benchmarks. The result applies inside option (2) of your original list.
  • Versus hybrid designs: the problem exists there too. In LMFusion’s own control, fine-tuning Llama-3 with the Transfusion recipe lowers HellaSwag by about 15% at first and leaves a persistent gap of about 7%; language is preserved only by freezing the text modules. Show-o2 needs a dedicated two-stage recipe for the same reason. There are two ways to protect language: isolate parameters, or make the generation target more compatible with the LLM. We provide the first controlled evidence for the second.
  • Why discrete at all: an open question we do not try to settle. Both routes are being pursued in parallel. Prior AR tokenizer papers (e.g., GigaTok, EOSTok) are judged on what they contribute within AR generation, without first proving that AR is preferable to diffusion. We ask for the same standard.
  1. Distinct contribution beyond existing AR-guided tokenization
  • The “broader value of AR-guided visual representations” for language retention was not known before this paper. LARP and EOSTok evaluate standalone generation only; neither involves a language model. The EOSTok retention numbers you cite were produced by us, as a control for our claim. A control that confirms a finding should not count against the paper that reports it.
  • Your original review listed the same evidence as a strength. It called the AR-shaping formulation “highly innovative and logical” and stated that the ablations show the gains “come from the AR-guided sequence organization”. AR-100M was already in Table 4 at that time. The only new row is the EOSTok control, which supports the same conclusion.
  • The result is more specific than “AR guidance helps”. Adding the same AR prior to 2D tokens lowers retention (48.8 vs. 52.6). Standalone generation does not predict the effect either: matched 2D and 1D tokenizers have the same gFID but a 3-point language gap.

We would be glad to hear whether the narrower claim in point 1 addresses your concern about scope. We also note that neither the review nor the follow-up identifies a soundness problem, while Soundness remains rated 1, and that Informativeness is rated 1 although the review calls the perspective “novel and insightful”. We would be grateful if you could point to the specific soundness issue, or reconsider these ratings.

AuthorsOpenReview ↗

A response to Reviewer Y9iz on attribution and on the standard of review

We thank Reviewer Y9iz for the discussion. The reviewer has not responded to our last follow-up. We nevertheless want to explain clearly why we disagree with the final assessment, because the disagreement concerns two questions of principle that matter beyond this paper: how credit for a finding should be assigned, and what standard a review should apply.

  1. We disagree with crediting this finding to EOSTok.

The final assessment argues that our contribution is less clear because “the EOSTok reproduction is competitive on the reported language benchmarks.” We see three reasons why this reading is mistaken.

  • The EOSTok numbers come from our own experiment. No public implementation of EOSTok was available, so we reimplemented it ourselves and trained it under our unified-training protocol. We added it as a control, to test whether our finding generalizes beyond our own tokenizer. A control that confirms a finding is evidence for that finding, not a reason to assign it to someone else.
  • The two papers make totally different claims. EOSTok argues that 1D tokens are better than raster-ordered 2D tokens for autoregressive image generation, and it evaluates only image generation on ImageNet. Our matched comparison points elsewhere: 1D and 2D tokenizers turn out to be comparable in reconstruction, generation and visual understanding. The advantage of AR-shaped 1D tokens lies in something EOSTok never measured, namely how well a pretrained LLM keeps its language ability under unified training.
  • No prior work reported this finding. To our knowledge, no prior work has shown that AR-guided 1D tokens help a pretrained LLM retain its language ability, or that the same prior on 2D tokens does not. For unified models built on LLMs, this is a central and practical question. For unified models built on pretrained LLMs, where losing language ability is one of the central costs, this is a practically important result.
  1. We disagree with the standard applied in this review.
  • On the discrete paradigm. The review calls direct prediction of discrete visual tokens “an inherently inefficient and conflicting paradigm,” yet in the same review it asks us to compare against Emu3 and Janus as “state-of-the-art Unified Autoregressive Models.” Both systems generate images by predicting discrete visual tokens with a trainable LLM backbone.
  • On hybrid models. The original review states that VLMs and hybrid models “successfully avoid language degradation.” The evidence says otherwise. LMFusion reports a persistent drop in language performance when Llama-3 is trained with the Transfusion recipe, and preserves language only by freezing its text modules. Our Janus-style experiment shows degradation with a decoupled encoder as well. The follow-up no longer repeats this premise, but the rating built on it has not changed.
  • On the ratings. Informativeness is rated 1, meaning “I learned little or nothing new,” while the same review calls our perspective “novel and insightful” and the problem “critical and often overlooked.” Soundness is rated 1, meaning “significant concerns that do not appear easy to address,” yet after three rounds no soundness problem has been identified, and our request to name one has not been answered.
  • On the scope of a paper. The remaining objection is that we do not show when discrete generation should be preferred over hybrid generation. We make no such claim, and a tokenizer study cannot settle it. Prior work on autoregressive tokenizers, including LARP, GigaTok and EOSTok itself, has been judged on what it contributes within that family. None of it was asked to first prove that the family should win.

Our position. We agree that hybrid designs perform better in some settings, and we find them promising. Which route will prove best, however, is a question the field is still answering, and it lies outside the scope of this paper. Several routes are being developed in parallel today, and that diversity is how a field makes progress* each line of work tests ideas the others cannot. Requiring a paper to reject other routes before it is allowed to advance its own, or ruling a direction out before the evidence is in, does not serve the community. We ask only that this work be judged on whether it answers its own question rigorously, and the reviewer’s original assessment of our ablations as “very solid” and “rigorous” already answers that.

Reviewer_fP9uOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether visual tokenization affects how well a pretrained LLM retains its language capabilities after unified multimodal training. It proposes LMTok, a 1D image tokenizer whose latent sequence is shaped by a training-time autoregressive prior, with cross-alignment used to preserve visual semantics. The authors compare it with a spatial 2D tokenizer under matched token lengths, codebook sizes, and downstream training settings. With Qwen3-4B and Qwen3-8B backbones, LMTok retains more language and reasoning performance while achieving broadly comparable visual reconstruction, understanding, and generation results. Ablations suggest that neither a 1D representation nor AR supervision alone accounts for the observed benefit.

Critical strengths
  1. A consequential evaluation question. The paper shows that visual tokenizers with similar standalone visual quality can have different effects on a pretrained LLM’s language performance after unified training. This makes language retention a useful criterion for tokenizer design, beyond reconstruction or generation quality.

  2. Consistent results with informative controls. LMTok improves language retention over the spatial 2D baseline across both Qwen3-4B and Qwen3-8B, while maintaining broadly comparable visual performance. The ablations further show that neither 1D latents without AR supervision nor adding the AR prior to spatial 2D tokens yields the same benefit.

Critical weaknesses
  1. Limited methodological novelty relative to prior AR-oriented tokenizers. Learned 1D visual tokens and an autoregressive training prior are closely related to earlier work, particularly LARP. The paper’s clearest new contribution is the language-retention evaluation, but it does not directly establish how much LMTok improves over the most relevant existing tokenizer designs under the same unified-training setup. This makes the method’s incremental contribution difficult to judge.

  2. The proposed mechanism is not fully isolated. The 1D and 2D tokenizers differ not only in sequence organization but also in their aggregation and semantic-alignment pathways. The ablations support an interaction between 1D latents and AR supervision, but do not directly show which property of the resulting token sequence causes better language retention. This limits the strength of the paper’s central explanatory claim.

Questions for authors
  1. Clarification on the comparison with prior work: What evidence supports an advantage for LMTok over existing image tokenizers designed for autoregressive modeling, particularly EOSTok, in the unified-training setting? A matched comparison, or a clear explanation of why one is not feasible, would materially affect my assessment of the contribution’s novelty.

  2. Clarification on the mechanism: What evidence shows that AR-guided sequence organization, rather than differences in visual information or semantic alignment, accounts for the language-retention gains? Can the authors provide a more controlled ablation that isolates the effect of AR-guided token organization from differences in visual information and semantic alignment?

  3. Could the authors report spatial-relation-specific results for LMTok and the matched 2D baseline—for example, the relevant GQA question types and MME position subtask—and, if available, a referring-expression grounding result? The aggregate visual-understanding scores may mask differences in spatial fidelity, which would affect my assessment of whether the language-retention gains come at a cost to visual capability.

6 comments
AuthorsOpenReview ↗

Response to Reviewer fP9u (1/3)

Thank you for the careful and constructive review. Your three questions pointed to the experiments the paper most needed, and we ran all of them. Short answers first; details follow.

  • Q1 (EOSTok, novelty): We reimplemented EOSTok and trained it in the same unified setup. It retains language as well as LMTok, which is what our explanation predicts; LMTok is clearly better in visual understanding.
  • Q2 (mechanism): We now report reconstruction, generation and understanding for every tokenizer in Table 4. Two tokenizers that differ only in the AR prior have the same visual quality but a 6.4-point gap in language.
  • Q3 (spatial ability): LMTok is better on relations and on par on VSR, but slightly lower on position questions. A separate grounding evaluation of the standalone models also indicates a potential localization limitation.

Q1 / W1. Advantage over existing AR-oriented tokenizers, in particular EOSTok

As of November 7, 2026, we could not locate official code or pretrained weights for EOSTok. We therefore reimplement EOSTok’s joint tokenizer–AR training design in our codebase, including IBQ, APR, and encoder/decoder semantic alignment, within our controlled training setup. Our reimplementation uses DINOv2 alignment on both the encoder (implicit alignment) and the decoder. We match the token count (256), codebook size (16,384 entries in a single codebook), and tokenizer-training budget to LMTok and the 2D baseline, rather than reproduce the original paper’s full training schedule. Its gFID is 1.76 under the same standalone evaluation protocol as for the other tokenizers (frozen tokenizer, separately trained LlamaGen 343M), vs. 1.81 for LMTok and 1.84 for the 2D baseline.

Table A. Unified training with Qwen3-4B (100K steps, same recipe as Table 1).

SettingMMLUMMLU-ReduxMMLU-ProMATH-500GSM8KMBPPGQAPOPEMME-PDPGGenEval
Text-only70.2067.1150.6747.080.5960.2–––––
Spatial 2D65.7363.2243.7038.873.9153.849.9179.331253.884.267.6
EOSTok (our reimpl.)66.1063.1446.7842.175.9355.847.5377.121123.684.167.7
LMTok66.3764.8746.4643.875.9656.051.8782.511310.484.567.1
  • Language retention is comparable. The EOSTok reimplementation improves over the 2D baseline on 5 of 6 benchmarks and is close to LMTok. We do not claim an advantage over EOSTok on this axis. The result supports our explanation: the benefit comes from AR supervision on a flexible 1D sequence, independent of the specific prior, quantizer or training pipeline.
  • LMTok's advantage is visual understanding (GQA 51.9 vs. 47.5, POPE 82.5 vs. 77.1, MME-P 1310 vs. 1124), at equal generation quality. EOSTok aligns to DINOv2, which is not language-aligned, and applies the alignment to encoder and decoder features, so semantics need not be recoverable from the quantized tokens. LMTok's cross-alignment supervises a feature map recovered from the quantized sequence with SigLIP2. A unified model needs both properties.
  • What is new relative to prior AR-oriented tokenizers. LARP and EOSTok show that AR supervision makes visual tokens easier to generate. We show three things that standalone generation cannot reveal. (1) Visual quality does not predict compatibility with an LLM: the 2D baseline and LMTok have similar gFID (1.84 vs. 1.81) but differ by 3.0 points in language. (2) With all other components matched, 1D and 2D tokenizers are comparable on visual tasks; the advantage of 1D appears only in joint modeling with language, and only with AR supervision. (3) The effect is general: it holds for a frozen prior, a prior trained from scratch, and the EOSTok pipeline.
AuthorsOpenReview ↗

Response to Reviewer fP9u (2/3)

Q2 / W2. Isolating AR-guided organization from visual information and semantic alignment

To examine whether the language-retention differences can be explained by visual information preservation or semantic quality, we evaluate reconstruction, generation, and standalone understanding for every tokenizer in Table 4, using the exact checkpoints used in unified training.

Table B. Standalone quality of every tokenizer in Table 4 (same checkpoints as used in unified training). "Lang. avg" is the mean of the four language columns. gFID uses the same LlamaGen 343M generator; understanding uses the LLaVA-1.5 recipe.

TokenizerAR supervisionMMLU-ProMATH-500GSM8KMBPPLang. avgrFID ↓gFID ↓GQA ↑MME-P ↑POPE ↑
TiToknone42.0438.267.7053.250.31.022.6642.98833.469.05
GigaToknone42.1737.271.1150.050.10.821.7948.70878.573.33
EOSTok (our reimpl.)joint AR generator46.7842.175.9355.855.21.021.7650.791078.974.62
Spatial 2Dnone43.7038.873.9153.852.60.981.8453.731229.378.65
Spatial 2Dfrozen Qwen3-0.6B42.2536.067.0249.848.81.001.8652.281135.977.69
Learned 1Dnone41.2436.269.8249.449.20.982.2052.621242.378.28
LMTok (1D)frozen Qwen3-0.6B46.4643.875.9656.055.60.971.8154.041246.681.17
Learned 1DAR-100M (joint)45.3942.173.5656.554.41.011.8953.381133.878.81
  1. Controlled ablation for the alignment and aggregation pathways. "Learned 1D without prior" and LMTok share the same query-based aggregation and the same cross-alignment. They differ only in the AR prior. Their reconstruction (rFID 0.98 vs. 0.97) and understanding (GQA 52.6 vs. 54.0, MME-P 1242 vs. 1247) are close, yet the language average differs by 6.4 points (MATH-500 by 7.6). The different alignment pathways of 1D and 2D tokenizers therefore cannot explain the gain.
  2. Visual information. Our five variants have rFID between 0.97 and 1.01, while their language average ranges from 48.8 to 55.6.
  3. Semantic quality does not track language retention. The three external tokenizers (TiTok, GigaTok, EOSTok reimpl.) all have weaker understanding than our SigLIP2-aligned variants, yet their language averages are 50.3, 50.1 and 55.2: the one trained with AR supervision is the one that retains language. Conversely, our 2D baseline and LMTok have similar understanding (GQA 53.7 vs. 54.0, MME-P 1229 vs. 1247), but the 2D baseline retains less language (52.6 vs. 55.6).
  4. Generation quality does not predict it either. Spatial 2D, Spatial 2D + prior and LMTok have gFID 1.84 / 1.86 / 1.81 and language averages 52.6 / 48.8 / 55.6.

The one factor that tracks language retention is the interaction described above: AR supervision applied to a flexible 1D sequence. 1D without AR supervision does not help (TiTok, GigaTok, Learned 1D without prior). AR supervision on a fixed raster layout does not help (Spatial 2D + prior). With both, all three tokenizers improve (LMTok, AR-100M, EOSTok reimpl.). Which statistic of the resulting sequence drives the effect is a question we leave for future analysis; App. B.4 shows that the learned order differs qualitatively from raster order.

AuthorsOpenReview ↗

Response to Reviewer fP9u (3/3)

Q3. Spatial-relation results

To examine whether LMTok’s flexible 1D representation comes at the expense of spatial understanding, we add complementary evaluations on the same final 4B and 8B unified checkpoints. We evaluate spatial relations and image-relative positions using GQA subsets, MME-Position, and VSR, distinguishing relational reasoning from position-specific performance.

Table C. Spatial evaluations on the unified checkpoints of Table 1.

BackboneTokenizerGQA-relationGQA relation-spatialGQA-positionMME-PositionVSR
Qwen3-4BSpatial 2D40.6442.0475.6230.0067.65
Qwen3-4BLMTok43.9144.7674.2236.6768.47
Qwen3-8BSpatial 2D40.3242.0477.4843.3369.11
Qwen3-8BLMTok41.0544.6076.4046.6768.43

What each column measures:

  • GQA-relation (5,308 questions): GQA's official relation category, one of the five semantic types reported by the GQA evaluation script. Questions require a relation between two objects.
  • GQA relation-spatial (2,424): the subset of the above whose reasoning program uses a spatial preposition (left/right of, in front of, behind, above, below, on top of, under). It excludes non-spatial relations such as wearing or holding.
  • GQA-position (644): GQA's position question types, which ask for an object's absolute location in the image. This set is disjoint from the two above.
  • MME-Position (30 images, 60 questions): the position sub-task of MME. Each image has two yes/no questions with opposite prepositions. We report acc+, the share of images with both questions answered correctly, which a constant answer cannot inflate.
  • VSR (2,195 statements): an independent benchmark (Liu et al., TACL 2023) of true/false spatial statements on COCO images; random split, test set.

The GQA slices are computed from the same per-question predictions as the aggregate GQA score, with the standard exact-match metric. Full definitions are given in [App. B.5] of the revised paper.

Findings. LMTok is higher on GQA relation questions at both scales, including spatial relations (+2.7 and +2.6), and on par on VSR (+0.8 and −0.7). MME-Position favors LMTok, but with 30 images the differences correspond to one or two images. On GQA-position the 2D baseline is slightly ahead at both scales (−1.4 and −1.1). This suggests a position-specific limitation in these evaluations, without establishing its cause.

Grounding. Our unified training data contains no referring-expression data, so we could not evaluate grounding on the unified models within the rebuttal period. The LLaVA-1.5 665K data used for our standalone understanding models (Table 3) does include a portion of referring-expression data, so we evaluated those models directly (Acc@IoU 0.5, Vicuna-7B, 256×256 input):

TokenizerRefCOCO valRefCOCOg val
Spatial 2D38.6532.52
LMTok35.5629.86

LMTok is about 3 points lower. Both scores are low in absolute terms at this resolution and with this amount of grounding data, so we treat the result as preliminary.

Reading these results together. Spatial-relation performance is maintained or improved, while the position and grounding results point to a possible limitation in fine-grained localization. The grounding numbers come from standalone LLaVA models and are reported only in this rebuttal; we have not measured this on the unified checkpoints, and we leave improving localization in AR-shaped 1D tokens to future work.

We hope these results address your concerns. We are happy to run further experiments or clarify anything, and we would be grateful if you would consider updating your assessment.

Reviewer_dBrWOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LMTok, an image tokenizer designed to help LLMs retain their language abilities during multimodal training. It converts each image into 256 learned tokens. During tokenizer training, a frozen language model predicts successive image tokens, and the prediction loss updates the tokenizer. Reconstruction and semantic losses preserve visual information. With Qwen3-4B and 8B, LMTok preserves more language and reasoning ability than a spatial 2D tokenizer under matched training settings, with similar visual performance. The main contribution is the study connecting tokenizer design to language retention; the design builds on earlier autoregressively trained tokenizers.

Critical strengths
  1. The paper shows that similar visual quality can hide different effects on an LLM's language ability. The main comparisons match token length, codebook size and downstream training. At 8B, LMTok improves MMLU-Pro by 1.72 points, MATH-500 by 3.4 and MBPP by 5.4 over the spatial baseline, with similar visual performance (Table 1). This is a useful finding that image reconstruction and generation scores alone would not establish.

  2. The ablations help narrow down the explanation. Table 4 shows that learned 1D tokens alone are insufficient, and adding the same autoregressive prior to spatial 2D tokens does not improve language retention. A small prior trained from scratch also gives substantial gains.

Critical weaknesses
  1. The tokenizer design builds closely on earlier work. LARP already combines learned query tokens with an autoregressive prior during tokenizer training, and the paper acknowledges this connection. EOSTok jointly trains a semantic 1D image tokenizer and an autoregressive generator. The main new finding concerns language retention, but EOSTok is not evaluated under the same multimodal training setup. This leaves unclear how much of the benefit is specific to LMTok's design.

  2. The explanation of the language gains needs stronger controls. Table 4 reports language scores without reconstruction and semantic-quality results for each tokenizer variant. Those results would help separate the effect of token organization from differences in the visual information preserved.

Questions for authors
  1. How does EOSTok compare with LMTok in language retention under the same multimodal training setup?

  2. How do the tokenizer variants in Table 4 compare in reconstruction and semantic quality?

2 comments
AuthorsOpenReview ↗

Response to Reviewer dBrW (1/2)

Thank you for the careful review and the positive assessment. You asked for two things, and both are now in the revised paper. Short answers first:

  • Q1 (EOSTok): We reimplemented EOSTok and trained it in the same unified setup. It retains language as well as LMTok, which is what our explanation predicts. LMTok is clearly better in visual understanding.
  • Q2 (Table 4 controls): We report reconstruction, generation and understanding for every tokenizer in Table 4. Visual quality is matched across our variants, while language retention is not.

Q1 / W1. Comparison with EOSTok under the same unified training

As of November 7, 2026, we could not locate official code or pretrained weights for EOSTok. We therefore reimplement EOSTok’s joint tokenizer–AR training design in our codebase, including IBQ, APR, and encoder/decoder semantic alignment, within our controlled training setup. Our reimplementation uses DINOv2 alignment on both the encoder (implicit alignment) and the decoder. We match the token count (256), codebook size (16,384 entries in a single codebook), and tokenizer-training budget to LMTok and the 2D baseline, rather than reproduce the original paper’s full training schedule. For a fair generation comparison, we use the same standalone evaluation protocol as for the other tokenizers: the tokenizer is frozen and a separate LlamaGen 343M generator is trained on its tokens (gFID 1.76, vs. 1.81 for LMTok and 1.84 for the 2D baseline).

Table A. Unified training with Qwen3-4B.

SettingMMLUMMLU-ReduxMMLU-ProMATH-500GSM8KMBPPGQAPOPEMME-PDPGGenEval
Text-only70.2067.1150.6747.080.5960.2–––––
Spatial 2D65.7363.2243.7038.873.9153.849.9179.331253.884.267.6
EOSTok (our reimpl.)66.1063.1446.7842.175.9355.847.5377.121123.684.167.7
LMTok66.3764.8746.4643.875.9656.051.8782.511310.484.567.1

Three observations:

  • Language retention is comparable. The EOSTok reimplementation is above the 2D baseline on 5 of 6 benchmarks and close to LMTok. We do not claim an advantage over EOSTok on this axis. The benefit comes from AR supervision on a flexible 1D sequence, independent of the specific prior, quantizer or pipeline.
  • LMTok’s advantage is visual understanding (GQA 51.9 vs. 47.5, POPE 82.5 vs. 77.1, MME-P 1310 vs. 1124), at equal generation quality. EOSTok aligns encoder and decoder features to DINOv2; LMTok’s cross-alignment supervises the quantized sequence with SigLIP2.
  • What is new relative to LARP and EOSTok. They show that AR supervision makes visual tokens easier to generate. We show three things that standalone generation cannot reveal. (1) Visual quality does not predict compatibility with an LLM: the 2D baseline and LMTok have similar gFID (1.84 vs. 1.81) but differ by 3.0 points in language. (2) With all other components matched, 1D and 2D tokenizers are comparable on visual tasks; the advantage of 1D appears only in joint modeling with language, and only with AR supervision. (3) The effect is general: it holds for a frozen prior, a prior trained from scratch, and the EOSTok pipeline, and it needs neither a special quantizer nor an auxiliary reconstruction loss.
AuthorsOpenReview ↗

Response to Reviewer dBrW (2/2)

Q2 / W2. Reconstruction and semantic quality of the Table 4 variants

Table B. Standalone quality of every tokenizer in Table 4 (same checkpoints as used in unified training). gFID uses the same LlamaGen 343M generator; understanding uses the LLaVA-1.5 recipe.

TokenizerAR supervisionMMLU-ProMATH-500GSM8KMBPPLang. avgrFID ↓gFID ↓GQA ↑MME-P ↑POPE ↑
TiToknone42.0438.267.753.250.31.022.6642.98833.469.05
GigaToknone42.1737.271.1150.050.10.821.7948.70878.573.33
EOSTok (our reimpl.)joint AR generator46.7842.175.9355.855.21.021.7650.791078.974.62
Spatial 2Dnone43.7038.873.9153.852.60.981.8453.731229.378.65
Spatial 2Dfrozen Qwen3-0.6B42.2536.067.0249.848.81.001.8652.281135.977.69
Learned 1Dnone41.2436.269.8249.449.20.982.2052.621242.378.28
LMTok (1D)frozen Qwen3-0.6B46.4643.875.9656.055.60.971.8154.041246.681.17
Learned 1DAR-100M (joint)45.3942.173.5656.554.41.011.8953.381133.878.81

The language differences are not explained by differences in preserved visual information or semantic quality:

  • Same visual information. Our five variants have rFID between 0.97 and 1.01, while their language average ranges from 48.8 to 55.6.
  • The cleanest control is Learned 1D without prior vs. LMTok. They share the architecture and the cross-alignment pathway and differ only in the AR prior. Reconstruction (rFID 0.98 vs. 0.97) and understanding (GQA 52.6 vs. 54.0, MME-P 1242 vs. 1247) are close, yet the language average differs by 6.4 points.
  • Semantic quality does not track language retention. The three external tokenizers (TiTok, GigaTok, EOSTok reimpl.) all have weaker understanding than our SigLIP2-aligned variants, yet their language averages are 50.3, 50.1 and 55.2: the one trained with AR supervision is the one that retains language. Conversely, our 2D baseline and LMTok have similar understanding (GQA 53.7 vs. 54.0), but the 2D baseline retains less language (52.6 vs. 55.6).

We hope these additions resolve both concerns. We are glad to clarify anything further, and we would be grateful if you would consider raising your score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their time and thoughtful feedback. We are encouraged that all three reviews recognize the two core elements of this work:

  • The finding. Visual tokenizers with similar visual quality can affect an LLM’s language ability very differently, which makes language retention a useful criterion for tokenizer design (dBrW, fP9u, Y9iz).
  • The controlled study. The matched comparisons and ablations narrow the explanation to AR supervision on flexible 1D tokens (dBrW, fP9u, Y9iz).

What this work contributes

  • A new evaluation axis. Visual quality does not predict how a tokenizer affects an LLM: matched 2D and 1D tokenizers have similar gFID but differ by 2–5 points on language benchmarks.
  • Beyond prior AR-oriented tokenizers. LARP and EOSTok show that AR supervision makes tokens easier to generate; we show it also makes 1D tokens easier for an LLM to learn with less language degradation.
  • Why discrete. A discrete model keeps the full LLM toolchain, and we see this study as complementary to hybrid and decoupled designs.

Changes in the revised paper:

  • Introduction. Clarified the scope of our claims (a shared, trainable autoregressive backbone) and sharpened the statement of contributions ([Sec. 1]; Y9iz).
  • Extended ablations. Added reconstruction, generation and understanding results for every tokenizer in Table 4 ([Table 4]; dBrW, fP9u).
  • EOSTok comparison. Added our reimplementation of EOSTok under the same unified training ([Table 1]; dBrW, fP9u).
  • Janus-style experiment. Added results with a decoupled continuous encoder for understanding ([Table 5]; Y9iz).
  • Spatial analysis. Added spatial-relation evaluations ([Table 6], [App. B.5]; fP9u).

AuthorsOpenReview ↗

Summary of the discussion for the Area Chair

We thank the AC for handling our submission and briefly summarize the discussion.

The findings are not in dispute. All three reviewers accept the main result. Reviewer dBrW recommends acceptance. Reviewer fP9u finds the evaluation question useful and states that the new experiments address several concerns. Reviewer Y9iz acknowledges the Janus-style result and that the evidence supports “the broader value of AR-guided visual representations”. No soundness problem has been raised in either round.

All three reviewers recognize the importance of the finding. Reviewer dBrW calls it “a useful finding that image reconstruction and generation scores alone would not establish”. Reviewer fP9u describes it as “a consequential evaluation question” that “makes language retention a useful criterion for tokenizer design”. Reviewer Y9iz writes that the paper “tackles a critical and often overlooked issue in unified multimodal models”. The results themselves are not in dispute: no soundness problem has been raised in either round, and Reviewers fP9u and Y9iz state that the new experiments address several concerns and support “the broader value of AR-guided visual representations”.

The other points concern scope. Reviewer fP9u asks which statistic of the token sequence causes the effect; we identify the causal factor through controlled ablations and state the finer mechanism as an open question. Reviewer Y9iz asks when discrete generation should be preferred over hybrid designs; we do not claim this, and the finding applies to any design in which a trainable backbone generates discrete tokens, including the decoupled setting we tested.

Revisions. The revised paper adds the EOSTok comparison, standalone quality of every tokenizer in Table 4, the Janus-style experiment, spatial evaluations, and an explicit scope statement.

We hope the AC will weigh whether a new, controlled finding about tokenizers and language retention is a sufficient contribution for ICLR. Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.