- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether discrete visual tokens can serve as semantically lossless representations for multimodal foundation models. The authors first reveal that feature reconstruction MSE does not correlate with downstream multimodal understanding performance; instead, the effective bit budget per patch is the critical driver. To overcome the saturation bottleneck of existing multi-codebook quantizers (e.g., RVQ and PQ), the authors propose Factorized Vector Quantization (FVQ), which projects visual features into complementary, fixed-dimensional views via a tight-frame analysis operator and quantizes them with symmetric parallel codebooks. Scaled on industrial multimodal data, FVQ-8B matches strong continuous-representation baselines (e.g., Qwen3-VL-8B) and significantly outperforms previous discrete visual tokenizers (e.g., UniAR, Emu3) across general reasoning and dense OCR/document benchmarks.
Critical strengths
- Sustained Scaling without Quantizer Saturation:
Unlike RVQ (which suffers from decaying residual magnitudes) and PQ (which partitions channels into overly narrow slices), FVQ preserves the full feature receptive field via dense tight-frame projections. This theoretical guarantee ensures zero geometric distortion prior to quantization, enabling steady downstream gains as the bit budget scales up to 224 bits/patch.
- Bridging the Continuous–Discrete Performance Gap in Fine-Grained Tasks
This paper empirically proves that high-bit discrete representations can match or rival leading continuous models (e.g., reaching 867 on OCRBench and matching Qwen3-VL on DocVQA), fundamentally challenging the premise that discretization inherently sacrifices fine-grained semantics.
- Thorough Isolation of Variables
The core findings are supported across multiple vision backbones (Qwen3 ViT, MoonViT-v2, SigLIP2) and validated under strictly matched training regimes (e.g., controlled LLaVA-OneVision experiments), confirming that the benefits originate from the quantizer formulation rather than solely data scaling.
Critical weaknesses
- Inference Efficiency and LLM Context Overhead
Each visual patch is represented by discrete tokens. While the paper notes token embeddings are merged before entering the LLM, the total vocabulary size or embedding lookup parameters scale significantly, and sequence length handling is not thoroughly profiled against latency and memory footprint during autoregressive decoding.
- Absence of Unified Multimodal Generation Validation
A primary motivation for adopting discrete tokens over continuous representations is unifying multimodal understanding and autoregressive image generation (as highlighted in Emu3 and UniAR). The paper evaluates understanding tasks exclusively, leaving unaddressed whether the complementary dense representations in FVQ are equally amenable to autoregressive visual generation.
- Sensitivity to Projection Initialization and Codebook Utilization
The tight-frame matrix relies on a static QR decomposition of a random Gaussian matrix. The paper does not analyze codebook perplexity/utilization across different parallel codebooks or verify whether certain projection slices suffer from codebook collapse across different random seeds.
Questions for authors
- Inference Overhead & Latency:
How does the inference latency and GPU memory footprint of FVQ-8B compare to the continuous Qwen3-VL baseline during prefill and generation stages?
- Applicability to Visual Generation:
Given that discrete tokens are primarily motivated by unified autoregressive generation (e.g., Chameleon, Emu3), can FVQ tokens be seamlessly decoded back to pixel space autoregressively or via diffusion decoders without representation collapse?
- Codebook Code Utilization:
Did you observe balanced codebook utilization across the parallel codebooks? What is the codebook perplexity across different slices during training?
- Ablation on Frame Tightness:
How sensitive is downstream performance if the tight-frame constraint () is relaxed to unconstrained random dense projections or learned linear layers?
The authors have responded thoroughly to most of my questions with supporting measurements and ablation studies. I maintain my rating of 3 (weak acceptance).