- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks what autoencoder training actually buys you. It splits reconstruction error three ways (Eq. 4): error at the decoder floor with h optimized freely, error when only the 32-dim latent z can move, and the encoder's own error. The middle difference is the accessibility gap; the outer one is the encoder gap. Five parts of the validation. Section 2.1: a randomly initialized frozen decoder matches a trained one once h is about 5x the image dimension (about 2.5x the trained h), with a trained/random x z/h control on SD VAE (Fig. 1c). Section 2.2: over SVHN, CIFAR-10, MNIST and Speech, z-optimization reaches 1.1-1.7x while h-optimization reaches 5.0-27.1x. Section 2.3: the correction dh is high-dimensional — more than 100 PCs for 95% of its variance, r95 far above 32, with part of the variance inside col(W). Section 2.4: sweeping z/h from 1/64 to 1 closes the accessibility gap and grows the encoder gap. Section 2.5: under domain shift, per-sample h-optimization beats eight trained adaptation conditions. The stated conclusion is that training organizes high-dimensional capacity rather than creating it. The authors state in their limitations that the experiments are confined to convolutional autoencoders and a limited set of image and audio datasets, leaving generality across architectures and objectives untested.
Critical strengths
The paper's contribution is the analysis method itself. It freezes a randomly initialized decoder and optimizes the intermediate representation h per sample, measuring the reconstruction quality that is reachable without any weight learning. Prior work such as Deep Image Prior and Deep Decoder optimizes weights per image, which is why it cannot separate capacity from accessibility. GAN inversion and Intermediate Layer Optimization likewise use intermediate-layer expressivity as a solution to inverse problems, rather than treating the gap itself as an object of study. The paper itself acknowledges that the conceptual ancestor of the accessibility gap is the amortization gap of Cremer et al. (2018); even so, no prior work applies this decomposition in reconstruction space with weights held fixed. Reconstruction numbers in the autoencoder literature have until now been values in which decoder capacity and access through the bottleneck are confounded; this method lets the two be seen separately. The authors provide the control experiment that could have refuted their own claim. They cross a trained decoder with a randomly initialized one, and optimizing only z with optimizing h. This separates two explanations: that training creates reconstruction capacity in the decoder itself, and that the capacity is already present and training builds the path to reach it through a narrow latent. Optimizing h directly makes the randomly initialized decoder match or slightly beat the trained one, so the first is not supported; optimizing only z makes the trained decoder roughly 50x lower in error, so the second is supported. Conceptual papers often assert a conclusion of this kind and omit the control, but here a falsifiable measurement is actually presented. At least on one model.
Critical weaknesses
A substantial part of what this paper claims has already been measured at larger scale and in a different architecture, and that comparison is missing. The experiments consist entirely of a three-layer convolutional autoencoder with a 32-dimensional bottleneck, plus a frozen SD VAE evaluated on 32 images; there is no transformer backbone. The authors acknowledge this in their limitations. Yet Section 3.4 cites the modern tokenizer literature and notes that one of those works used a ViT backbone to measure the scaling effects of the encoder and the decoder separately, without the experiments performing any comparable measurement. In other words, part of what this paper claims is already known, and the remaining question is whether it holds at larger scale, which the paper does not answer. Moreover, the hard single-path bottleneck this framework requires is bypassed by design in modern autoencoders — in architectures that learn residuals or use a 3D causal structure, h is no longer everything downstream of z. The result that widening the bottleneck closes the accessibility gap is not a finding at the z/h = 1 endpoint but a structural consequence. The decoder's fc is a single affine map, so at z/h = 1 it becomes a square matrix, and if it is full rank it is surjective, which forces the telescoping identity to grow the encoder gap by the same amount. The paper's own numbers show this: the deterministic AEs only "approach zero," the VAEs retain a small residual gap, and the supplementary figure's axis runs negative. Since an attained minimum cannot have a negative accessibility gap, that endpoint displays optimizer error rather than training. The abstract nonetheless leads with this endpoint. The genuinely empirical content — the closure rate at intermediate ratios, the consistency across beta, the stability of the total gap — survives, but the headline does not. The central quantity is undefined, it is optimizer-dependent, and the core sections contain no external comparison. The paper never states what z ranges over when the reconstruction loss is minimized, so in the VAE panels the latent may be optimized off the training manifold. This is precisely the confound that the amortization literature it cites avoids by measuring against the variational family. In addition, the optimization budget differs across figures for the same h configuration, and there are no convergence curves or stated tuning criterion, so I cannot tell whether the reported gaps are a fact about training or a product of the optimization budget. Furthermore, there is no re-run of anything the paper compares against, so 5.0-27.1x is externally uncalibrated, and the one genuinely comparative experiment, on domain shift, receives only a single sentence of protocol in the appendix.
Questions for authors
Clarification requests
- Section 3.4 cites the modern tokenizer literature. Among those works, Hansen-Estruch et al. (2025) used a ViT backbone to measure the scaling effects of the encoder and the decoder separately, on data far exceeding ImageNet-1K, but the experiments contain no comparable measurement. Do you plan to apply this decomposition in their setting, or is there a reason you limited the scope to convolutional autoencoders?
- Could you report the singular values (or effective rank) of the trained decoder's fc matrix at the z/h = 1 configurations? If the route is full-rank affine, the gap closes at z/h = 1 by construction, so that endpoint appears to be a consequence of the parameterization rather than an empirical result. In addition, h-optimization is given 20,000 steps in Fig. 2 and 3,000 steps for the same h = 2,048 in Fig. 4, and no convergence curve for h-optimization is presented, so the two results cannot be placed side by side. Could you provide the convergence curve for that configuration?
- In Fig. 5b, the advantage of h-optimization varies across transfer pairs — it leads by a wide margin on CIFAR-10 → STL and Speech M → F, whereas on MNIST → USPS the learned adaptation conditions achieve lower MSE. The accessibility gap is a quantity defined independently of domain, so does this framework explain which pairs retain a large gap and which do not? Is the variation across pairs predicted by the size of the gap, or does it come from a separate cause?
- The accessibility gap in Section 2.3 occupies the classical approximation-gap slot of the Cremer et al. (2018) decomposition you cite, yet Section 3.2 calls it a "totally different restriction." Which is it? Relatedly, in the tokenizer regime that Section 3.4 invokes as motivation, Ramanujan et al. (2025) reach a conclusion about compressibility versus usability. You do not cite it, but are your results consistent with theirs, and does your framework explain or contradict them?
Suggestion for future work (not required for my assessment)
- The framework assumes a hard single-path bottleneck. Where residual or skip pathways let information bypass the latent, h is no longer everything downstream of z and the telescope's attribution breaks — this is the case for U-Nets, transformer blocks, diffusion backbones, and recent compressive autoencoders that learn residuals. Moreover, for the deployment-constrained settings of Section 3.4 (codecs, CSI feedback) the bottleneck is discrete and hierarchical (VQ-VAE, RVQ), where "optimize z freely" is not a well-defined diagnostic. Drawing that boundary, and extending the measurement to one discrete-bottleneck model, would say a lot about how far the result generalizes.
We appreciate your time and comments!
Re: A substantial part of what this paper claims has already been measured at larger scale and in a different architecture.We'd like to distinguish the substance. Hansen-Estruch et al. (2025) observe that scaling the encoder gives no reconstruction benefit while scaling the decoder does, and that larger bottlenecks improve reconstruction; they do not explain it, their goal being to introduce ViTok. Our decomposition is a candidate explanation for that pattern — the capacity sits in the decoder, and what limits its use is how much of it the bottleneck can address. The relation is explanatory rather than duplicative, and we will say so in the text instead of citing them for relevance alone.
Re: that endpoint displays optimizer error rather than training. The abstract nonetheless leads with this endpoint.We agree the endpoint is structural and we showed so with experiments in responses above. We want to point out that the abstract states a trend across ratios rather than a claim about the endpoint. And as we stated in Q2 below, even though the map at the endpoint (z/h=1) is surjective, it is badly conditioned.
Re: The paper never states what z ranges over […] so in the VAE panels the latent may be optimized off the training manifold.We should have stated the domain: z is unconstrained over Rz, warm-started at the encoder's μ, with no prior term in the per-image objective.
Your concern is that an unconstrained z* might land far outside the region the decoder was trained on. We checked, and it does not: at z = 2048 the optimized z* has a per-coordinate norm of 0.44 against a prior-typical 1.0, so it sits inside the prior's typical set. The encoder is the one at an extreme, driven by the KL to the prior's cheapest point at 0.0125. The VAE panels are not optimizing off-manifold; the useful code is priced out rather than far away.
For your questionsQ1. We cannot run a ViT-backbone tokenizer at that scale within the discussion period. We agree the scope here was chosen so every quantity in the telescope is exactly defined, which requires h to be everything downstream of a single z — true of our architecture and the frozen SD-VAE, not of most modern tokenizers (see Q5).
Q2. We scanned every square model we trained:
Your inference is correct: the route is full-rank affine, so the gap closes at z/h = 1 by construction. The conditioning rows are the finding discussed in responses above — surjective but badly conditioned, increasingly so with width. Stable rank should not be read as an effective dimension here as the spectrum decays too steeply for that, with the ten largest of 2,048 singular values carrying only about 15% of the energy.
Q3. The framework does not predict which pairs retain a large gap.
Q4. Our encoder gap is the analogue of the amortization gap, measured in reconstruction rather than the ELBO (5.6–12.1% on three of four datasets). The accessibility gap is not their approximation gap: theirs asks how well the variational family matches the true posterior, ours involves no posterior and asks which decoder states any bottleneck code can reach. The two occupy different slots.
Q5. Agreed, and we will draw the boundary explicitly. The telescope requires h to be everything downstream of z, which fails for U-Nets, transformer blocks, diffusion backbones and compressive autoencoders that learn residuals; and "optimize z freely" is not well defined for VQ-VAE or RVQ.