acceptodds
Under review as a conference paper at ICLR 2027

Reconstruction Does Not Determine Generative Learnability: A Theory of Resource-Matched Visual Tokenization

Abstract

A visual tokenizer is typically optimized for reconstruction, while the downstream generator must learn the latent distribution it induces. We show that better reconstruction need not imply easier generation because reconstruction does not identify the generative geometry of the latent space: reconstruction-equivalent tokenizers can induce sharply different finite-resource learning problems. To quantify this gap, we develop Rate-Distortion-Generative Learnability (RDGL), supported by diffusion-scale representation complexity, latent-to-image transport bounds, finite-sample minimax limits, a full-spectrum deterministic equivalent from random matrix theory, and separate capacity and optimization laws. The resulting theory shows that the preferred representation is resource dependent and motivates Capacity-Matched Spectral Tokenization (CMST), which shapes latent geometry to the downstream data, model-capacity, and optimization budgets. Experiments support both the mechanism and the resulting design principle: reconstruction-preserving latent gauges raise fixed-budget gFID from 4.18 to 15.61, while CMST reduces gFID from 4.18 for the matched VA-VAE control to 2.96 under the same downstream protocol. The resulting principle is that a representation's complexity and geometry should be matched to the learner that must model it, rather than optimized for reconstruction alone. The anonymized code is available at https://anonymous.4open.science/r/ICLR2027-CMST/.

Primary area:
generative models
Keywords:
Visual Tokenization, Generative Learnability, Representation Learning, Latent Diffusion Models, Rate-Distortion, Random Matrix Theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ydo8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the reconstruction–generation dilemma in visual tokenization models, arguing that reconstruction quality does not necessarily determine generative learnability. The authors analyze the effects of latent geometry, training data, model capacity, and optimization budget. Based on a series of theoretical analyses (including Gaussian models, minimax analysis, and random matrix theory), they propose Capacity-Matched Spectral Tokenization (CMST), which adjusts the latent spectrum according to downstream computational resources. Experiments demonstrate improvements in generation quality under fixed training budgets.

Critical strengths

The reconstruction-preserving transformation experiment is interesting. It shows that identical reconstruction quality can lead to substantially different downstream generation performance.

CMST achieves meaningful empirical improvements over existing baselines.

Critical weaknesses

The main motivation and observations, particularly the reconstruction–generation trade-off, are already well established in prior work, such as [1], [2], and [3].

The paper presents an extensive number of theoretical results and proofs (the paper is 53 pages long!) to address what is fundamentally an empirical problem. However, most of these results are derived under Gaussian or linear assumptions, whereas the actual experiments involve nonlinear diffusion Transformers. Moreover, given the capabilities of modern generative AI systems (e.g., GPT-6 Astra, maybe with Ultra thinking level in codex or ChatGPT Pro), which can now help derive large numbers of lemmas and proofs relatively quickly (see, e.g., [4]), I believe it is increasingly important to ensure that theoretical complexity is clearly justified. In particular, theoretical results should provide concrete, precise, and ideally testable predictions that go beyond existing empirical observations, rather than merely offering post hoc explanations.

In this paper, I am not convinced that such an extensive theoretical framework is necessary, nor is it clear to me how closely the theoretical results are connected to the proposed method and experimental findings. I would therefore like the authors to clarify this point, as elaborated in the questions below.

[1] When Worse is Better: Navigating the Compression-Generation Trade-off in Visual Tokenization. NeurIPS 2025.

[2] Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models. CVPR 2025.

[3] GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation. ICCV 2025.

[4] https://github.com/openai/math

Questions for authors
  • The paper develops an extensive theoretical framework spanning over 50 pages. While the authors explicitly connect several theoretical results to the CMST objective, I am not convinced that the specific mathematical constructions are necessary to explain the observed phenomena or design the proposed method. Could the authors clarify what insights or design decisions fundamentally depend on these results, rather than on existing intuitions about representation complexity, spectral conditioning, and resource constraints?
  • While the paper quantitatively validates several theoretical results in Gaussian or local surrogate settings, it remains unclear whether the framework has quantitative predictive power for actual nonlinear diffusion models. After calibrating the surrogate parameters, can the proposed generative cost accurately predict or rank the optimal latent dimensions, spectral configurations, or downstream generation performance under different resource budgets? Such evidence would help distinguish a genuinely predictive theory from a theoretical motivation for an empirically effective regularizer.
Reviewer_5fuWOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how latent representations affect generation under limited data, model capacity, and training time. It derives spectral learning-error estimates under Gaussian and linear-denoiser assumptions and uses them to propose Capacity-Matched Spectral Tokenization (CMST), a regularizer that shapes latent spectra for a target generator budget. ImageNet experiments show improved generation and suggest that the preferred representation depends on downstream resources.

Critical strengths

s1) The reconstruction-preserving transformations provide a controlled demonstration that latent geometry affects generation quality independently of reconstruction. This is relevant to demonstrate the effect of latent geometry alone can alter the downstream sample quality.

Critical weaknesses

w1) The theory does not clarify the relationship between generation quality and the theoretical points. One can drive the representation to a degenerate state in which information varies along only one direction. Under the linear-denoiser and Gaussian approximations, this would imply low estimated generative difficulty, but it would not necessarily improve generation quality.

Why is this critical: The theory although correct does not produce any statement on the effect of representation on sample quality.

w2) Reducing the variance with soft gate does not remove its effect on decoder. In fact it can raise decoder's sensitivity to the information spread across this specific direction. So it would be possible to simply remove a given direction in which case the regularizer is not needed anymore and we can simply encode the information into a lower dimension.

Why is this critical: This is an observation on the method that can lead to reconstruction blow ups along the same directions the method is trying to surppres. If the direction is fully removed then the proposed regularizer is not needed.

Questions for authors

Q1) Can authors provide justification on why the regularizer is needed and how in its optimal form is this different from simply compressing the data into lower dimensions?

Reviewer_VTx1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper criticizes generative image tokenizers being optimized for reconstruction and shows that better reconstruction does not imply easier generation. It shows that reconstruction leaves the latent-space geometry underdetermined. This causes representations with identical reconstruction quality to differ substantially in generative tasks. Building on this, it develops a theoretical framework for generative learnability. They further propose Capacity-Matched Spectral Tokenization (CMST) to adapt the latent representation to the model, data and compute resources.

Critical strengths

The authors start with a pretrained visual tokenizer (VA-VAE) as their baseline and apply two types of transformations to its latent representations: (i) orthogonal rotations, which preserve the covariance spectrum, and (ii) increasingly anisotropic transformations, which stretch some latent directions while compressing others. Each transformation is compensated by its inverse in the decoder, ensuring identical image reconstructions.

The authors then train the same diffusion model from scratch on each transformed representation under identical conditions. While orthogonal transformations have negligible effect, increasing anisotropy substantially degrades generation quality. Additional experiments controlling for total latent variance and channel-wise standardization confirm that this degradation cannot be explained by scaling effects alone. These experiments provide compelling evidence that latent geometry influences generative learnability independently of reconstruction quality.

Critical weaknesses

Figure 12(a) is inconsistent with the experimental protocol stated in Appendix L.3. The experiment specifies covariance eigenvalues in [0.5,1.5][0.5,1.5] and the identity estimator Q^=N(0,I)\hat Q= \mathcal{N} (0,I)when n<dn<d. Under this protocol,

1dW22(N(0,Σ),N(0,I))=1d∑i(λi−1)2≤(1−0.5)2≈0.0858.\frac{1}{d} W_2^2 (\mathcal{N} (0, \Sigma), \mathcal{N} (0,I) ) = \frac{1}{d} \sum_i ( \sqrt{\lambda_i} - 1)^2 \leq (1-\sqrt{0.5})^2 \approx 0.0858. Nevertheless, the data-poor points in Figure 12(a), including n/d=1/4n/d=1/4 and 1/21/2, are visibly above 0.10.1.

Questions for authors

What explains the discrepancy in Figure 12(a)? Please provide the actual plotted values, normalization, and estimator configuration, and clarify whether correcting the discrepancy changes the conclusions of this experiment or any related numerical validation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.