- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates the trade-off between acquiring visual capabilities and retaining language/reasoning abilities when extending pretrained Large Language Models (LLMs) to unified multimodal understanding and generation. The authors argue that the organization of visual latents significantly impacts this trade-off. They propose LMTok, a 1D visual tokenizer that decouples spatial feature extraction from latent sequence organization. LMTok uses a frozen LLM as an autoregressive (AR) prior during tokenizer training to shape the 1D token sequence via next-token prediction, while employing a cross-alignment mechanism to preserve 2D spatial semantics without fixing token-to-patch correspondence. Experiments show that under matched settings, LMTok achieves comparable standalone visual reconstruction and generation quality to spatial 2D tokenizers, but significantly mitigates language capability degradation during unified training on benchmarks like MMLU-Pro and MATH-500.
Critical strengths
-
Novel and Insightful Perspective: Instead of solely chasing state-of-the-art visual reconstruction or generation metrics (which is the common trend in visual tokenizer research), the paper tackles a critical and often overlooked issue in unified multimodal models: the catastrophic forgetting of language capabilities. Formulating the tokenizer design as an "AR-shaping" problem to align with the LLM's native next-token prediction objective is a highly innovative and logical approach.
-
Rigorous Controlled Ablations: The authors provide a very solid experimental methodology by strictly matching token lengths, codebook sizes, and downstream training budgets between the 1D LMTok and the 2D spatial baseline. The ablation studies thoroughly isolate the contributions of the AR prior, the 1D structure, and the cross-alignment, proving that the language retention gains come from the AR-guided sequence organization rather than just the 1D flexibility.
Critical weaknesses
-
Misalignment of Evaluation Metrics and Baselines: The paper mainly compares LMTok against traditional spatial 2D tokenizers (like VQGAN derivatives) and older 1D tokenizers (TiTok). It fails to compare against state-of-the-art Unified Autoregressive Models (e.g., Emu3, Janus, Transfusion, Show-o) which use advanced techniques like next-scale prediction, decoupled encoders, or hybrid diffusion/autoregressive frameworks. Furthermore, it does not compare against standard VLMs (which use continuous vision encoders + projectors, avoiding discrete tokenization altogether). Without these comparisons, it is hard to judge if LMTok actually advances the frontier of unified modeling or just improves a suboptimal discrete baseline.
-
Questionable Core Motivation and Practical Value: The core motivation is to prevent language degradation when an LLM is forced to predict discrete visual tokens autoregressively. However, the reason language degradation happens in unified discrete models is fundamentally due to modality interference and the massive distribution shift between text and pixel-level visual tokens. The current industry and academic consensus to solve this is either: (1) using frozen continuous vision encoders with projectors (standard VLMs), (2) decoupling the visual pathways for understanding and generation (like Janus), or (3) using hybrid models (text AR + visual Diffusion/Flow, like Transfusion). Forcing the LLM to natively predict discrete visual tokens—even if shaped by an AR prior—is an inherently inefficient and conflicting paradigm. The paper attempts to "patch" this paradigm rather than questioning its viability against continuous or hybrid alternatives.
Questions for authors
-
How does LMTok perform when integrated into modern unified architectures (e.g., Janus or Transfusion) compared to their native tokenization or continuous representations?
-
Given that modern VLMs and hybrid unified models successfully avoid language degradation by not forcing the LLM to autoregressively predict discrete visual tokens, what is the practical purpose of optimizing discrete tokenizers for language retention?
4 comments
Maintain my weak reject recommendation
Thank you for the detailed response and the additional Janus-style experiment and I have read. With the understanding encoder held fixed, LMTok improves all six language benchmarks while maintaining broadly comparable visual performance.
However, the additional experiments and rebuttal do not resolve my main concern regarding the significance and practical implications of the contribution. The experiments establish an improvement within a controlled discrete autoregressive setting, but do not establish when or why one should choose this setting over decoupled or hybrid alternatives. Improving language retention after requiring a shared LLM backbone to predict discrete visual tokens does not, by itself, demonstrate that this architectural choice offers a compelling tradeoff. Reusing token-based post-training and serving infrastructure is a reasonable motivation, but the associated benefits and costs are not evaluated here. I agree that direct comparisons against differently scaled full systems would not isolate the tokenizer’s contribution. My concern is therefore not the absence of every named system baseline, but the lack of evidence identifying a practical regime in which the proposed approach offers a meaningful advantage over alternative ways of combining language and image generation.
Table 4 also leaves questions about the methodological advance beyond existing AR-guided tokenization. The randomly initialized AR prior achieves broadly similar retention benefits, and the EOSTok reproduction is competitive on the reported language benchmarks. These results support the broader value of AR-guided visual representations, but make the distinct contribution of the proposed design less clear.
Taken together, these concerns leave me unconvinced that the contribution is sufficiently significant for acceptance, and I maintain my weak reject recommendation.
Follow-up to Reviewer Y9iz: scope of the claim and distinct contribution
Thank you for engaging further, and for confirming the Janus-style result, that system-level comparisons would not isolate the tokenizer, and that toolchain reuse is a reasonable motivation. Your comment also made us realize that our previous answer argued for the discrete route in general, which is more than this paper needs. Let us be precise.
- “When or why one should choose this setting over decoupled or hybrid alternatives”
- We do not claim it should be chosen over hybrid designs. We have not compared the two, and the revision says so. Our claim is narrower: whenever a trainable LLM backbone generates images, language degrades, and the generation target determines how much.
- When: a large and active family of models. Chameleon, Emu3, Emu3.5, Janus and Janus-Pro all generate images as discrete tokens in a trainable backbone, and Emu3.5 reports image generation and editing comparable to Gemini 2.5 Flash Image. Every such system faces the retention problem we measure.
- Versus decoupled designs: it is not an either/or choice. Janus-style models still generate discrete tokens. In the Janus-style experiment you acknowledged, LMTok improves all six language benchmarks. The result applies inside option (2) of your original list.
- Versus hybrid designs: the problem exists there too. In LMFusion’s own control, fine-tuning Llama-3 with the Transfusion recipe lowers HellaSwag by about 15% at first and leaves a persistent gap of about 7%; language is preserved only by freezing the text modules. Show-o2 needs a dedicated two-stage recipe for the same reason. There are two ways to protect language: isolate parameters, or make the generation target more compatible with the LLM. We provide the first controlled evidence for the second.
- Why discrete at all: an open question we do not try to settle. Both routes are being pursued in parallel. Prior AR tokenizer papers (e.g., GigaTok, EOSTok) are judged on what they contribute within AR generation, without first proving that AR is preferable to diffusion. We ask for the same standard.
- Distinct contribution beyond existing AR-guided tokenization
- The “broader value of AR-guided visual representations” for language retention was not known before this paper. LARP and EOSTok evaluate standalone generation only; neither involves a language model. The EOSTok retention numbers you cite were produced by us, as a control for our claim. A control that confirms a finding should not count against the paper that reports it.
- Your original review listed the same evidence as a strength. It called the AR-shaping formulation “highly innovative and logical” and stated that the ablations show the gains “come from the AR-guided sequence organization”. AR-100M was already in Table 4 at that time. The only new row is the EOSTok control, which supports the same conclusion.
- The result is more specific than “AR guidance helps”. Adding the same AR prior to 2D tokens lowers retention (48.8 vs. 52.6). Standalone generation does not predict the effect either: matched 2D and 1D tokenizers have the same gFID but a 3-point language gap.
We would be glad to hear whether the narrower claim in point 1 addresses your concern about scope. We also note that neither the review nor the follow-up identifies a soundness problem, while Soundness remains rated 1, and that Informativeness is rated 1 although the review calls the perspective “novel and insightful”. We would be grateful if you could point to the specific soundness issue, or reconsider these ratings.
A response to Reviewer Y9iz on attribution and on the standard of review
We thank Reviewer Y9iz for the discussion. The reviewer has not responded to our last follow-up. We nevertheless want to explain clearly why we disagree with the final assessment, because the disagreement concerns two questions of principle that matter beyond this paper: how credit for a finding should be assigned, and what standard a review should apply.
- We disagree with crediting this finding to EOSTok.
The final assessment argues that our contribution is less clear because “the EOSTok reproduction is competitive on the reported language benchmarks.” We see three reasons why this reading is mistaken.
- The EOSTok numbers come from our own experiment. No public implementation of EOSTok was available, so we reimplemented it ourselves and trained it under our unified-training protocol. We added it as a control, to test whether our finding generalizes beyond our own tokenizer. A control that confirms a finding is evidence for that finding, not a reason to assign it to someone else.
- The two papers make totally different claims. EOSTok argues that 1D tokens are better than raster-ordered 2D tokens for autoregressive image generation, and it evaluates only image generation on ImageNet. Our matched comparison points elsewhere: 1D and 2D tokenizers turn out to be comparable in reconstruction, generation and visual understanding. The advantage of AR-shaped 1D tokens lies in something EOSTok never measured, namely how well a pretrained LLM keeps its language ability under unified training.
- No prior work reported this finding. To our knowledge, no prior work has shown that AR-guided 1D tokens help a pretrained LLM retain its language ability, or that the same prior on 2D tokens does not. For unified models built on LLMs, this is a central and practical question. For unified models built on pretrained LLMs, where losing language ability is one of the central costs, this is a practically important result.
- We disagree with the standard applied in this review.
- On the discrete paradigm. The review calls direct prediction of discrete visual tokens “an inherently inefficient and conflicting paradigm,” yet in the same review it asks us to compare against Emu3 and Janus as “state-of-the-art Unified Autoregressive Models.” Both systems generate images by predicting discrete visual tokens with a trainable LLM backbone.
- On hybrid models. The original review states that VLMs and hybrid models “successfully avoid language degradation.” The evidence says otherwise. LMFusion reports a persistent drop in language performance when Llama-3 is trained with the Transfusion recipe, and preserves language only by freezing its text modules. Our Janus-style experiment shows degradation with a decoupled encoder as well. The follow-up no longer repeats this premise, but the rating built on it has not changed.
- On the ratings. Informativeness is rated 1, meaning “I learned little or nothing new,” while the same review calls our perspective “novel and insightful” and the problem “critical and often overlooked.” Soundness is rated 1, meaning “significant concerns that do not appear easy to address,” yet after three rounds no soundness problem has been identified, and our request to name one has not been answered.
- On the scope of a paper. The remaining objection is that we do not show when discrete generation should be preferred over hybrid generation. We make no such claim, and a tokenizer study cannot settle it. Prior work on autoregressive tokenizers, including LARP, GigaTok and EOSTok itself, has been judged on what it contributes within that family. None of it was asked to first prove that the family should win.
Our position. We agree that hybrid designs perform better in some settings, and we find them promising. Which route will prove best, however, is a question the field is still answering, and it lies outside the scope of this paper. Several routes are being developed in parallel today, and that diversity is how a field makes progress* each line of work tests ideas the others cannot. Requiring a paper to reject other routes before it is allowed to advance its own, or ruling a direction out before the evidence is in, does not serve the community. We ask only that this work be judged on whether it answers its own question rigorously, and the reviewer’s original assessment of our ablations as “very solid” and “rigorous” already answers that.
Response to Reviewer Y9iz
Thank you for the thoughtful review, and for the encouraging words on the perspective (“novel and insightful”) and the ablations (“very solid”). You raised two concerns. Here is the short version of our answers; details follow.
Q1 / W1. Integration into a decoupled architecture, and baselines
Following Q1, we integrate LMTok into a Janus-style setup: continuous SigLIP2 features for understanding, and discrete tokens (LMTok vs. the matched 2D tokenizer) as generation targets. Training is identical to Table 1 (Qwen3-4B, 100K steps).
On system-level baselines: Emu3, Janus and Transfusion are trained on different and largely non-public data at different scales. An end-to-end comparison would confound the tokenizer with data, scale and architecture, and would not isolate the variable we study. Their generation interface, however, is the setting we control. Emu3 is a pure next-token predictor over discrete visual tokens, which is our main setting. Janus generates images by predicting discrete VQ indices in a trainable shared backbone, which is the experiment above.
Q2 / W2. Why optimize discrete tokenizers for language retention
Because a purely discrete model reuses the full LLM toolchain, and its visual tokens are the component with the most room left to improve.
We have stated this scope explicitly in the revised introduction and related work.
Two small clarifications on our setup. The 2D baseline shares LMTok’s SigLIP2-initialized encoder, decoder and semantic supervision (rFID 0.98, gFID 1.84), so it is a matched semantic tokenizer. Both tokenizers are distilled from SigLIP2, and LMTok-MCQ approaches continuous SigLIP2 features in understanding (Table 3).
We hope the new experiment and the clarifications above address your concerns. If any further experiment or explanation would be helpful, we would be glad to provide it during the discussion period. Given the positive assessment of the perspective and of the ablations, we would be grateful if you would consider revisiting the evaluation of our work.