- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies compute-optimal visual compression for multimodal pretraining. The authors argue that existing scaling laws mainly consider model size and training tokens while treating the visual tokenizer as fixed. To address this limitation, the paper extends a Chinchilla-style loss function by introducing two additional factors: token efficiency and tokenizer distortion. The proposed framework jointly considers model size, text tokens, visual tokens, and visual compression rate.
The central assumption is that tokenizer distortion acts as an irreducible loss floor that cannot be reduced by increasing decoder capacity. Based on this assumption, the authors derive several theoretical results, including that the optimal visual compression rate decreases as the training compute budget increases, and that image generation generally requires finer visual representations than image understanding. The experiments mainly consist of scaling-law analysis and closed-loop predictions on synthetic Gaussian-field decoders, along with evaluations of frozen CLIP, LLaVA, and Qwen3-VL models.
Critical strengths
-
A relevant research problem. Visual token compression directly affects both training cost and model performance in multimodal pretraining. Including the compression rate in compute-optimal resource allocation is a reasonable research direction.
-
Relatively complete mathematical analysis. The authors provide a systematic analysis of the proposed loss function, covering optimal resource allocation, the relationship between compression rate and compute budget, and the distinction between distortion floors and capacity absorption. The paper also includes controlled experiments to examine some of its theoretical assumptions.
Critical weaknesses
-
Limited novelty and insights. The main conclusions are fairly intuitive, particularly that larger compute budgets favor finer visual representations and that generation tasks require more visual details than understanding tasks. Most of the theoretical results follow directly from the assumed loss function and distortion-floor hypothesis. Despite the extensive mathematical analysis, I find the actual insights rather limited.
-
Insufficient training experiments and limited scale. The main experiments are conducted on synthetic data with very small decoders of up to 51M parameters. The experiments involving real VLMs mainly use frozen or pretrained models, with the largest being Qwen3-VL-8B. For a paper studying multimodal scaling laws, the lack of actual large-scale pretraining experiments is a major limitation. Evaluating frozen models up to 8B parameters is not sufficient to establish a reliable scaling law for multimodal pretraining.
-
Limited practical guidance for real pretraining. The paper aims to determine the optimal visual compression rate under a fixed compute budget, but it does not demonstrate how the proposed framework can guide actual VLM training decisions. In practice, the important question is how to balance model size, training tokens, and visual compression under realistic compute constraints. Without validating the predicted optimal configurations through real pretraining experiments, the practical value of the proposed scaling law remains unclear.
Questions for authors
-
Does the distortion-floor assumption hold in real VLMs? The main conclusions rely on assuming that tokenizer distortion is independent of model capacity. Can the authors provide evidence beyond synthetic experiments, particularly on whether larger models can compensate for information lost during compression?
-
Can the scaling law predict optimal compression in real pretraining? The closed-loop predictions are only validated on synthetic decoders. Can the authors demonstrate that the proposed framework correctly predicts the optimal compression rate under fixed compute budgets in real VLM pretraining?
-
Are the proposed efficiency and distortion metrics reliable? The paper uses feature cosine similarity and related proxies to measure token efficiency and distortion. How well do these metrics correlate with actual multimodal pretraining loss, especially across understanding and generation tasks?