Tokenizer Distortion as a Floor: Compute-Optimal Visual Compression for Multimodal Pretraining

Multimodal scaling laws commonly regard the visual tokenizer as fixed infrastructure and allocate a FLOP budget solely between model parameters and tokens. This convention excludes the visual compression rate from resource allocation. We redefine multimodal pretraining as a joint optimization over parameters, text tokens, visual tokens, and the visual compression rate, with two empirically measured tokenizer curves entering a Chinchilla-style loss. The first curve captures the predictive information carried by each retained visual token. The second curve quantifies the distortion introduced by discarding complementary visual content. The core structural question asks whether larger parameter budgets can absorb this distortion. Across the suite of controlled decoders we train, the residual gap under matched effective visual data remains statistically invariant to model width. This observation identifies tokenizer distortion as a loss floor rather than a term reducible by increased model capacity. Because the floor stationarity condition lacks direct width elasticity, the optimal compression rate relaxes as the compute budget grows. When the efficiency curve is nearly flat, as with CLIP, the interior optimality condition vanishes and the optimum lies at the finest feasible code. A floor model fitted on small compute budgets recovers discrete configurations that remain optimal on held out larger budgets, and it outperforms direct transfer of compression rates from small budget regimes. Generative objectives raise the effective distortion weight, which can push the unconstrained optimal rate below unity so that the feasible optimum settles at the domain boundary. Pretraining optimizes over a static corpus while inference optimizes per query, so the corresponding Lagrangians are not scalar multiples of one another.