acceptodds
Under review as a conference paper at ICLR 2027

Tokenizer Distortion as a Floor: Compute-Optimal Visual Compression for Multimodal Pretraining

Abstract

Multimodal scaling laws commonly regard the visual tokenizer as fixed infrastructure and allocate a FLOP budget solely between model parameters and tokens. This convention excludes the visual compression rate from resource allocation. We redefine multimodal pretraining as a joint optimization over parameters, text tokens, visual tokens, and the visual compression rate, with two empirically measured tokenizer curves entering a Chinchilla-style loss. The first curve captures the predictive information carried by each retained visual token. The second curve quantifies the distortion introduced by discarding complementary visual content. The core structural question asks whether larger parameter budgets can absorb this distortion. Across the suite of controlled decoders we train, the residual gap under matched effective visual data remains statistically invariant to model width. This observation identifies tokenizer distortion as a loss floor rather than a term reducible by increased model capacity. Because the floor stationarity condition lacks direct width elasticity, the optimal compression rate relaxes as the compute budget grows. When the efficiency curve is nearly flat, as with CLIP, the interior optimality condition vanishes and the optimum lies at the finest feasible code. A floor model fitted on small compute budgets recovers discrete configurations that remain optimal on held out larger budgets, and it outperforms direct transfer of compression rates from small budget regimes. Generative objectives raise the effective distortion weight, which can push the unconstrained optimal rate below unity so that the feasible optimum settles at the domain boundary. Pretraining optimizes over a static corpus while inference optimizes per query, so the corresponding Lagrangians are not scalar multiples of one another.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal scaling laws, Compute-optimal allocation, Vision-language pretraining, Visual tokenization

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_eznaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies compute-optimal visual compression for multimodal pretraining. The authors argue that existing scaling laws mainly consider model size and training tokens while treating the visual tokenizer as fixed. To address this limitation, the paper extends a Chinchilla-style loss function by introducing two additional factors: token efficiency and tokenizer distortion. The proposed framework jointly considers model size, text tokens, visual tokens, and visual compression rate.

The central assumption is that tokenizer distortion acts as an irreducible loss floor that cannot be reduced by increasing decoder capacity. Based on this assumption, the authors derive several theoretical results, including that the optimal visual compression rate decreases as the training compute budget increases, and that image generation generally requires finer visual representations than image understanding. The experiments mainly consist of scaling-law analysis and closed-loop predictions on synthetic Gaussian-field decoders, along with evaluations of frozen CLIP, LLaVA, and Qwen3-VL models.

Critical strengths
  1. A relevant research problem. Visual token compression directly affects both training cost and model performance in multimodal pretraining. Including the compression rate in compute-optimal resource allocation is a reasonable research direction.

  2. Relatively complete mathematical analysis. The authors provide a systematic analysis of the proposed loss function, covering optimal resource allocation, the relationship between compression rate and compute budget, and the distinction between distortion floors and capacity absorption. The paper also includes controlled experiments to examine some of its theoretical assumptions.

Critical weaknesses
  1. Limited novelty and insights. The main conclusions are fairly intuitive, particularly that larger compute budgets favor finer visual representations and that generation tasks require more visual details than understanding tasks. Most of the theoretical results follow directly from the assumed loss function and distortion-floor hypothesis. Despite the extensive mathematical analysis, I find the actual insights rather limited.

  2. Insufficient training experiments and limited scale. The main experiments are conducted on synthetic data with very small decoders of up to 51M parameters. The experiments involving real VLMs mainly use frozen or pretrained models, with the largest being Qwen3-VL-8B. For a paper studying multimodal scaling laws, the lack of actual large-scale pretraining experiments is a major limitation. Evaluating frozen models up to 8B parameters is not sufficient to establish a reliable scaling law for multimodal pretraining.

  3. Limited practical guidance for real pretraining. The paper aims to determine the optimal visual compression rate under a fixed compute budget, but it does not demonstrate how the proposed framework can guide actual VLM training decisions. In practice, the important question is how to balance model size, training tokens, and visual compression under realistic compute constraints. Without validating the predicted optimal configurations through real pretraining experiments, the practical value of the proposed scaling law remains unclear.

Questions for authors
  1. Does the distortion-floor assumption hold in real VLMs? The main conclusions rely on assuming that tokenizer distortion is independent of model capacity. Can the authors provide evidence beyond synthetic experiments, particularly on whether larger models can compensate for information lost during compression?

  2. Can the scaling law predict optimal compression in real pretraining? The closed-loop predictions are only validated on synthetic decoders. Can the authors demonstrate that the proposed framework correctly predicts the optimal compression rate under fixed compute budgets in real VLM pretraining?

  3. Are the proposed efficiency and distortion metrics reliable? The paper uses feature cosine similarity and related proxies to measure token efficiency and distortion. How well do these metrics correlate with actual multimodal pretraining loss, especially across understanding and generation tasks?

Reviewer_Tk4bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies compute-optimal scaling laws in multimodal pretraining by challenging the conventional assumption of treating the visual tokenizer as fixed infrastructure, reformulating resource allocation as a joint optimization over decoder parameters NN, text tokens DtD_t, visual tokens DvD_v, and the visual compression rate Rv≥1R_v \ge 1. Building on a two-source Chinchilla-style loss, the authors incorporate two empirical tokenizer curves, namely an efficiency curve η(Rv)\eta(R_v) defining the effective visual data volume D~v=Dvη(Rv)\tilde{D}_v = D_v \eta(R_v), and a distortion curve D(Rv)\mathcal{D}(R_v) penalizing discarded visual content. Through a matched residual decomposition protocol under matched D~v\tilde{D}_v, the work posits that tokenizer distortion acts as a loss floor invariant to model width NN rather than an error reducible through model capacity.

Critical strengths

This paper introduces the visual compression rate RvR_v as an endogenous decision variable within a constrained optimization framework. By doing so, it establishes a unified theoretical formulation that reconciles divergent practical designs, connecting fixed visual token bottleneck architectures with scale-dependent resolution-scaling strategies while rationalizing why dense visual codes during pretraining remain theoretically compatible with aggressive token pruning at inference time.

Critical weaknesses

W1: The authors might provide more details about the CLIP experiment, maybe an imperfect proxy for the visual tokenization process modeled by the scaling law. Appendix G varies image resolution using a frozen CLIP ViT-B/32 encoder, but does not specify how these changes affect the number of visual tokens processed or the associated FLOP cost. It also defines η(Rv)\eta(R_v) using the cosine similarity between a global image feature and the best-matching text query. This quantity is not directly a measure of predictive information per retained visual token in an autoregressive decoder. The authors should clarify how resolution is mapped to RvR_v, report the resulting token counts and compute costs, and explain why this global-feature proxy is suitable for the proposed allocation law. Since the CLIP analysis is supplementary to the controlled decoder experiments, this concern primarily limits the interpretation of the real-encoder results.

W2: The matched gaps in Figures 2(a) and 5(a–c) rise across the smaller decoder widths before approximately plateauing. The formal test concerns the corrected remainder rather than the plotted raw gap, but the paper does not make this distinction sufficiently clear when claiming width invariance. Besides, five of the six reported slope estimates in Table 4 lie above the upper endpoints of their stated 95% bootstrap intervals. The authors might need to clarify the interval construction and report the corrected remainders with uncertainty at each width. An analysis excluding the two smallest decoders would show whether the floor approximation holds only after a capacity threshold.

W3: All pretraining scaling-law fits and closed-loop selection experiments in the paper are conducted exclusively on synthetic streams of 16×1616 \times 16 Gaussian field images paired with 16 discrete attribute tokens under minimal compute budgets spanning only C∈[1013,3×1014]C \in [10^{13}, 3 \times 10^{14}] FLOPs. For the largest decoders on the IsoFLOP grid, the training duration is clamped to as few as 80 steps, placing these models in a severe transient optimization regime where the validation NLL reflects short-term learning rate dynamics rather than asymptotic capacity-data scaling.

Questions for authors

In Figure 5(a)–(c), the matched residual gaps ΔL\Delta L across all three synthetic fields exhibit a clear, monotonic upward trend as NN increases from 0.44M0.44\text{M} to 11.5M11.5\text{M} before visibly plateauing at larger widths. This suggests the presence of a negative interaction in the small-capacity regime where smaller decoders are too bottlenecked to exploit the fine visual codes and thus violate the additive separability in Assumption 1. Can the authors provide the per-width raw gaps and corrected remainders, explain the apparent slope–interval inconsistencies, and repeat the floor-versus-interaction analysis with the 0.44M and 0.93M decoders excluded? Does the resulting fit still select the same held-out configurations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.