acceptodds
Under review as a conference paper at ICLR 2027

Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?

Abstract

Discrete visual tokens offer a native interface for multimodal modeling under a shared next-token prediction objective by representing vision in a discrete form analogous to language. However, quantization error is widely assumed to cause unavoidable semantic loss, fundamentally limiting the semantic fidelity of discrete visual tokens and thereby positioning continuous visual representations as an intrinsic upper bound. We revisit this assumption using multimodal understanding as a proxy for semantic fidelity and find that quantization error is unreliable, as discrete representations with substantially larger errors can achieve better understanding. Instead, the effective bit budget of discrete indices, reflecting the semantic information they convey, emerges as a key determinant of downstream performance. However, existing quantizers eventually saturate as additional codebooks contribute progressively less semantic information. As one approach to alleviating this saturation, we introduce Factorized Vector Quantization, which quantizes complementary, fixed-dimensional views of the same visual feature with symmetric parallel codebooks. It continues to improve at larger bit budgets, ultimately matching continuous visual representations in multimodal understanding. We train on industrial-scale multimodal data and achieve state-of-the-art performance among discrete-representation models, improving over UniAR and Emu3-Chat on a nine-metric average by 22.8% and 59.2%, respectively. To the best of our knowledge, our model is also the first discrete-representation model to perform competitively with strong continuous-representation models such as Qwen3-VL, including on fine-grained OCR tasks. Beyond these results, we unveil the key principles and a practical recipe that make such competitiveness possible. These results and practical insights jointly support the potential of discrete visual representations to underpin large-scale native multimodal foundation models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Discrete Visual Representations, Vector Quantization, Native Multimodal Modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kUXKOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether discrete visual tokens can serve as semantically lossless representations for multimodal foundation models. The authors first reveal that feature reconstruction MSE does not correlate with downstream multimodal understanding performance; instead, the effective bit budget per patch is the critical driver. To overcome the saturation bottleneck of existing multi-codebook quantizers (e.g., RVQ and PQ), the authors propose Factorized Vector Quantization (FVQ), which projects visual features into complementary, fixed-dimensional views via a tight-frame analysis operator and quantizes them with symmetric parallel codebooks. Scaled on industrial multimodal data, FVQ-8B matches strong continuous-representation baselines (e.g., Qwen3-VL-8B) and significantly outperforms previous discrete visual tokenizers (e.g., UniAR, Emu3) across general reasoning and dense OCR/document benchmarks.

Critical strengths
  1. Sustained Scaling without Quantizer Saturation:

Unlike RVQ (which suffers from decaying residual magnitudes) and PQ (which partitions channels into overly narrow slices), FVQ preserves the full feature receptive field via dense tight-frame projections. This theoretical guarantee ensures zero geometric distortion prior to quantization, enabling steady downstream gains as the bit budget scales up to 224 bits/patch.

  1. Bridging the Continuous–Discrete Performance Gap in Fine-Grained Tasks

This paper empirically proves that high-bit discrete representations can match or rival leading continuous models (e.g., reaching 867 on OCRBench and matching Qwen3-VL on DocVQA), fundamentally challenging the premise that discretization inherently sacrifices fine-grained semantics.

  1. Thorough Isolation of Variables

The core findings are supported across multiple vision backbones (Qwen3 ViT, MoonViT-v2, SigLIP2) and validated under strictly matched training regimes (e.g., controlled LLaVA-OneVision experiments), confirming that the benefits originate from the quantizer formulation rather than solely data scaling.

Critical weaknesses
  1. Inference Efficiency and LLM Context Overhead

Each visual patch is represented by LL discrete tokens. While the paper notes token embeddings are merged before entering the LLM, the total vocabulary size or embedding lookup parameters scale significantly, and sequence length handling is not thoroughly profiled against latency and memory footprint during autoregressive decoding.

  1. Absence of Unified Multimodal Generation Validation

A primary motivation for adopting discrete tokens over continuous representations is unifying multimodal understanding and autoregressive image generation (as highlighted in Emu3 and UniAR). The paper evaluates understanding tasks exclusively, leaving unaddressed whether the complementary dense representations in FVQ are equally amenable to autoregressive visual generation.

  1. Sensitivity to Projection Initialization and Codebook Utilization

The tight-frame matrix AA relies on a static QR decomposition of a random Gaussian matrix. The paper does not analyze codebook perplexity/utilization across different parallel codebooks or verify whether certain projection slices suffer from codebook collapse across different random seeds.

Questions for authors
  1. Inference Overhead & Latency:

How does the inference latency and GPU memory footprint of FVQ-8B compare to the continuous Qwen3-VL baseline during prefill and generation stages?

  1. Applicability to Visual Generation:

Given that discrete tokens are primarily motivated by unified autoregressive generation (e.g., Chameleon, Emu3), can FVQ tokens be seamlessly decoded back to pixel space autoregressively or via diffusion decoders without representation collapse?

  1. Codebook Code Utilization:

Did you observe balanced codebook utilization across the LL parallel codebooks? What is the codebook perplexity across different slices during training?

  1. Ablation on Frame Tightness:

How sensitive is downstream performance if the tight-frame constraint (A⊤A=ρIDA^\top A = \rho I_D) is relaxed to unconstrained random dense projections or learned linear layers?

1 comment

The authors have responded thoroughly to most of my questions with supporting measurements and ablation studies. I maintain my rating of 3 (weak acceptance).

Reviewer_PjM6OpenReview ↗
Rating
4
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether discrete visual representations can match continuous features for multimodal understanding. It argues that reconstruction error alone does not explain downstream performance and proposes FVQ, a quantizer using parallel codebooks over dense feature views. Shared-setting comparisons, quantizer ablations, and industrial-scale training show competitive understanding performance and support the usefulness of discrete visual interfaces.

Critical strengths
  1. What I appreciate most is the testable empirical conclusion: lower reconstruction MSE does not necessarily mean better understanding, and multiple codebooks can substantially narrow the gap between discrete and continuous representations. Table 2 is particularly informative: it compares VQ, SimVQ, RVQ, PQ, FVQ, and continuous representations using the same QwenViT features and the same VLM training and evaluation setup.

  2. FVQ's construction and pre-quantization recovery formula are clearly explained, and the main model's results on fine-grained OCR and chart tasks show practical value beyond the controlled study. The paper is clear and easy to follow. Some baseline details, including visual encoders, alignment, and training, still need clarification, but the overall presentation is strong. I consider this a good piece of work.

Critical weaknesses
  1. The effects of bit budget and FVQ's design are not fully separated. B=Llog⁡2CB=L\log_2 C is nominal capacity, not measured semantic information. In other words, the increased LL also changes parameters and computation, while correlated dense views need not add independent information. Discussions or further ablation studies are needed.

  2. The large-scale results establish a competitive system but do not isolate the quantizer's contribution. Should provide more details in Section 4.1 on data sources, mixture ratios, filtering procedures, and checks for overlap with evaluation benchmarks, as the current description gives only broad data types and token counts. The 22.8% and 59.2% gains over UniAR and Emu3-Chat are system-level comparisons with different training recipes. Table 2 does help at a smaller scale, but whether FVQ retains its advantage over continuous features and PQ under the same industrial recipe remains unclear.

Questions for authors
  1. What are DD, dd, codebook and interface parameter counts, LLM image-token counts, and training/inference costs for each setting?
  2. Does FVQ still improve over rotated PQ and dense projections without the tight-frame constraint at matched budgets, across several seeds?
  3. Is there a continuous or PQ baseline using the same industrial recipe? Please specify the data mixture and benchmark-overlap checks.
Reviewer_wghjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether discrete visual tokens can achieve semantic fidelity comparable to continuous visual representations.

The main contributions are:

  1. Shows that feature reconstruction error is not necessarily correlated with downstream multimodal understanding performance.
  2. Proposes Factorized Vector Quantization (FVQ), using fixed-dimensional dense projections and parallel codebooks to improve bit-budget scaling.
  3. Demonstrates competitive results against continuous representations in controlled experiments and strong performance among discrete-token-based VLMs after large-scale training.
Critical strengths
  1. Interesting problem and empirical observations. The relationship between discrete representation capacity and downstream multimodal understanding is worth investigating. The analysis of quantization error and bit-budget scaling provides useful insights.

  2. Promising results for discrete visual representations. FVQ shows consistent improvements over conventional quantization methods at larger bit budgets. The results on OCR and document understanding are particularly encouraging.

Critical weaknesses
  1. The controlled continuous baseline is relatively weak, limiting the main conclusion.

    • The central claim is largely supported by Table 2, where FVQ achieves 65.3 compared with 64.6 for the continuous baseline.
    • However, the absolute performance of this baseline is relatively low, particularly on MathVista (45.7) and OCRBench (676).
    • Matching a relatively weak continuous baseline does not establish that discrete representations can preserve the semantic capabilities of stronger, well-trained continuous VLMs.
    • A comparison under substantially stronger and sufficiently trained continuous baselines would be more convincing.
  2. The large-scale results do not sufficiently support the claim of semantic losslessness.

    • The proposed model still trails Qwen3-VL-8B on several important benchmarks, particularly MMMU-Pro (40.8 vs. 55.9), MMMU (64.3 vs. 69.6), and ChartQA (83.9 vs. 89.6).
    • These models are not trained under fully matched conditions, making it difficult to isolate the impact of representation discretization.
    • The results demonstrate that discrete representations can be competitive, but do not yet establish semantic equivalence to strong continuous representations.
Questions for authors
  1. Stronger baseline: Can the authors evaluate FVQ against a stronger continuous baseline under matched training data, model capacity, and training compute? This is the most important experiment for supporting the paper's central claim.

  2. Scaling at higher performance levels: Does the discrete-continuous performance gap remain small as the model and training scale increase? The current controlled experiments do not establish whether the conclusion holds in a stronger performance regime.

  3. Representation efficiency: FVQ uses up to 16 codebooks and 224 bits per visual patch. What are the additional costs in terms of token representation, training, and inference? How does the method compare with continuous representations under a matched computational budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.