acceptodds
Under review as a conference paper at ICLR 2027

Capacity Is Not Accessibility in Autoencoders

Abstract

The high expressivity of trained decoders in autoencoder models is usually attributed to their learned weights. However, we find that even random convolutional decoders can achieve high-quality reconstructions when sufficiently high-dimensional intermediate representations are optimized directly. This raises the question: what does learning contribute if substantial capacity is already present in random decoders? We decompose autoencoder reconstruction errors into three components: a decoder-limited error, an encoder gap between the best bottleneck state and that predicted by the encoder, and an accessibility gap between the best intermediate representation and the best state reachable through the bottleneck. Across image and audio datasets, we show that increasing bottleneck dimensionality systematically reduces the accessibility gap while increasing the encoder gap, leaving the total gap comparatively stable. Training therefore does not merely provide capacity, it organizes high-dimensional capacity so that useful outputs can be reached through far fewer control variables. We further find that the accessibility gap spans many functionally relevant decoder directions and can extend under distribution shift. Together, these results suggest that poor reconstruction through a trained decoder can reflect restricted access to its capability rather than insufficient decoder capacity.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
autoencoders, representation learning, latent representations, bottleneck representation, decoder expressivity, amortized inference, transfer learning, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_F2x5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks what autoencoder training actually buys you. It splits reconstruction error three ways (Eq. 4): error at the decoder floor with h optimized freely, error when only the 32-dim latent z can move, and the encoder's own error. The middle difference is the accessibility gap; the outer one is the encoder gap. Five parts of the validation. Section 2.1: a randomly initialized frozen decoder matches a trained one once h is about 5x the image dimension (about 2.5x the trained h), with a trained/random x z/h control on SD VAE (Fig. 1c). Section 2.2: over SVHN, CIFAR-10, MNIST and Speech, z-optimization reaches 1.1-1.7x while h-optimization reaches 5.0-27.1x. Section 2.3: the correction dh is high-dimensional — more than 100 PCs for 95% of its variance, r95 far above 32, with part of the variance inside col(W). Section 2.4: sweeping z/h from 1/64 to 1 closes the accessibility gap and grows the encoder gap. Section 2.5: under domain shift, per-sample h-optimization beats eight trained adaptation conditions. The stated conclusion is that training organizes high-dimensional capacity rather than creating it. The authors state in their limitations that the experiments are confined to convolutional autoencoders and a limited set of image and audio datasets, leaving generality across architectures and objectives untested.

Critical strengths

The paper's contribution is the analysis method itself. It freezes a randomly initialized decoder and optimizes the intermediate representation h per sample, measuring the reconstruction quality that is reachable without any weight learning. Prior work such as Deep Image Prior and Deep Decoder optimizes weights per image, which is why it cannot separate capacity from accessibility. GAN inversion and Intermediate Layer Optimization likewise use intermediate-layer expressivity as a solution to inverse problems, rather than treating the gap itself as an object of study. The paper itself acknowledges that the conceptual ancestor of the accessibility gap is the amortization gap of Cremer et al. (2018); even so, no prior work applies this decomposition in reconstruction space with weights held fixed. Reconstruction numbers in the autoencoder literature have until now been values in which decoder capacity and access through the bottleneck are confounded; this method lets the two be seen separately. The authors provide the control experiment that could have refuted their own claim. They cross a trained decoder with a randomly initialized one, and optimizing only z with optimizing h. This separates two explanations: that training creates reconstruction capacity in the decoder itself, and that the capacity is already present and training builds the path to reach it through a narrow latent. Optimizing h directly makes the randomly initialized decoder match or slightly beat the trained one, so the first is not supported; optimizing only z makes the trained decoder roughly 50x lower in error, so the second is supported. Conceptual papers often assert a conclusion of this kind and omit the control, but here a falsifiable measurement is actually presented. At least on one model.

Critical weaknesses

A substantial part of what this paper claims has already been measured at larger scale and in a different architecture, and that comparison is missing. The experiments consist entirely of a three-layer convolutional autoencoder with a 32-dimensional bottleneck, plus a frozen SD VAE evaluated on 32 images; there is no transformer backbone. The authors acknowledge this in their limitations. Yet Section 3.4 cites the modern tokenizer literature and notes that one of those works used a ViT backbone to measure the scaling effects of the encoder and the decoder separately, without the experiments performing any comparable measurement. In other words, part of what this paper claims is already known, and the remaining question is whether it holds at larger scale, which the paper does not answer. Moreover, the hard single-path bottleneck this framework requires is bypassed by design in modern autoencoders — in architectures that learn residuals or use a 3D causal structure, h is no longer everything downstream of z. The result that widening the bottleneck closes the accessibility gap is not a finding at the z/h = 1 endpoint but a structural consequence. The decoder's fc is a single affine map, so at z/h = 1 it becomes a square matrix, and if it is full rank it is surjective, which forces the telescoping identity to grow the encoder gap by the same amount. The paper's own numbers show this: the deterministic AEs only "approach zero," the VAEs retain a small residual gap, and the supplementary figure's axis runs negative. Since an attained minimum cannot have a negative accessibility gap, that endpoint displays optimizer error rather than training. The abstract nonetheless leads with this endpoint. The genuinely empirical content — the closure rate at intermediate ratios, the consistency across beta, the stability of the total gap — survives, but the headline does not. The central quantity is undefined, it is optimizer-dependent, and the core sections contain no external comparison. The paper never states what z ranges over when the reconstruction loss is minimized, so in the VAE panels the latent may be optimized off the training manifold. This is precisely the confound that the amortization literature it cites avoids by measuring against the variational family. In addition, the optimization budget differs across figures for the same h configuration, and there are no convergence curves or stated tuning criterion, so I cannot tell whether the reported gaps are a fact about training or a product of the optimization budget. Furthermore, there is no re-run of anything the paper compares against, so 5.0-27.1x is externally uncalibrated, and the one genuinely comparative experiment, on domain shift, receives only a single sentence of protocol in the appendix.

Questions for authors

Clarification requests

  1. Section 3.4 cites the modern tokenizer literature. Among those works, Hansen-Estruch et al. (2025) used a ViT backbone to measure the scaling effects of the encoder and the decoder separately, on data far exceeding ImageNet-1K, but the experiments contain no comparable measurement. Do you plan to apply this decomposition in their setting, or is there a reason you limited the scope to convolutional autoencoders?
  2. Could you report the singular values (or effective rank) of the trained decoder's fc matrix at the z/h = 1 configurations? If the route is full-rank affine, the gap closes at z/h = 1 by construction, so that endpoint appears to be a consequence of the parameterization rather than an empirical result. In addition, h-optimization is given 20,000 steps in Fig. 2 and 3,000 steps for the same h = 2,048 in Fig. 4, and no convergence curve for h-optimization is presented, so the two results cannot be placed side by side. Could you provide the convergence curve for that configuration?
  3. In Fig. 5b, the advantage of h-optimization varies across transfer pairs — it leads by a wide margin on CIFAR-10 → STL and Speech M → F, whereas on MNIST → USPS the learned adaptation conditions achieve lower MSE. The accessibility gap is a quantity defined independently of domain, so does this framework explain which pairs retain a large gap and which do not? Is the variation across pairs predicted by the size of the gap, or does it come from a separate cause?
  4. The accessibility gap in Section 2.3 occupies the classical approximation-gap slot of the Cremer et al. (2018) decomposition you cite, yet Section 3.2 calls it a "totally different restriction." Which is it? Relatedly, in the tokenizer regime that Section 3.4 invokes as motivation, Ramanujan et al. (2025) reach a conclusion about compressibility versus usability. You do not cite it, but are your results consistent with theirs, and does your framework explain or contradict them?

Suggestion for future work (not required for my assessment)

  1. The framework assumes a hard single-path bottleneck. Where residual or skip pathways let information bypass the latent, h is no longer everything downstream of z and the telescope's attribution breaks — this is the case for U-Nets, transformer blocks, diffusion backbones, and recent compressive autoencoders that learn residuals. Moreover, for the deployment-constrained settings of Section 3.4 (codecs, CSI feedback) the bottleneck is discrete and hierarchical (VQ-VAE, RVQ), where "optimize z freely" is not a well-defined diagnostic. Drawing that boundary, and extending the measurement to one discrete-bottleneck model, would say a lot about how far the result generalizes.
1 comment

We appreciate your time and comments!

Re: A substantial part of what this paper claims has already been measured at larger scale and in a different architecture.

We'd like to distinguish the substance. Hansen-Estruch et al. (2025) observe that scaling the encoder gives no reconstruction benefit while scaling the decoder does, and that larger bottlenecks improve reconstruction; they do not explain it, their goal being to introduce ViTok. Our decomposition is a candidate explanation for that pattern — the capacity sits in the decoder, and what limits its use is how much of it the bottleneck can address. The relation is explanatory rather than duplicative, and we will say so in the text instead of citing them for relevance alone.

Re: that endpoint displays optimizer error rather than training. The abstract nonetheless leads with this endpoint.

We agree the endpoint is structural and we showed so with experiments in responses above. We want to point out that the abstract states a trend across ratios rather than a claim about the endpoint. And as we stated in Q2 below, even though the map at the endpoint (z/h=1) is surjective, it is badly conditioned.

Re: The paper never states what z ranges over […] so in the VAE panels the latent may be optimized off the training manifold.

We should have stated the domain: zz is unconstrained over Rzℝ^z, warm-started at the encoder's μ\mu, with no prior term in the per-image objective.

Your concern is that an unconstrained zz* might land far outside the region the decoder was trained on. We checked, and it does not: at z = 2048 the optimized zz* has a per-coordinate norm of 0.44 against a prior-typical 1.0, so it sits inside the prior's typical set. The encoder is the one at an extreme, driven by the KL to the prior's cheapest point at 0.0125. The VAE panels are not optimizing off-manifold; the useful code is priced out rather than far away.

For your questions

Q1. We cannot run a ViT-backbone tokenizer at that scale within the discussion period. We agree the scope here was chosen so every quantity in the telescope is exactly defined, which requires hh to be everything downstream of a single zz — true of our architecture and the frozen SD-VAE, not of most modern tokenizers (see Q5).

Q2. We scanned every square model we trained:

z = h = 51210242048
models303073
full rank in float6430/3030/3073/73
zero singular values000
matrix_rank on float32511–5121022–10242037–2044
condition number, AE5.9e46.0e41.8e5
condition number, VAE3.2e43.5e43.0e5
stable rank, AE7.711.113.9
stable rank, VAE8.811.911.8

Your inference is correct: the route is full-rank affine, so the gap closes at z/h = 1 by construction. The conditioning rows are the finding discussed in responses above — surjective but badly conditioned, increasingly so with width. Stable rank should not be read as an effective dimension here as the spectrum decays too steeply for that, with the ten largest of 2,048 singular values carrying only about 15% of the energy.

Q3. The framework does not predict which pairs retain a large gap.

Q4. Our encoder gap is the analogue of the amortization gap, measured in reconstruction rather than the ELBO (5.6–12.1% on three of four datasets). The accessibility gap is not their approximation gap: theirs asks how well the variational family matches the true posterior, ours involves no posterior and asks which decoder states any bottleneck code can reach. The two occupy different slots.

Q5. Agreed, and we will draw the boundary explicitly. The telescope requires hh to be everything downstream of zz, which fails for U-Nets, transformer blocks, diffusion backbones and compressive autoencoders that learn residuals; and "optimize zz freely" is not well defined for VQ-VAE or RVQ.

Reviewer_CwNZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper defines an (image or speech) autoencoder as a series of function mapping x→z→h→x^x\to z \to h \to \hat x, mediated by the encoder EE, part of the decoder FF, and the rest of the decoder GG. It first notes that, for high enough dimensional hh, one can use a random decoder GG and optimize hh directly, though this does not hold for zz or low-dimensional hh. It also notes that the differences between such high-dimensional optimized hh and the typical hh obtained from an AE are high-dimensional. Moreover, increasing the bottleneck width decreases the difference between optimizing hh and zz, but does increase the relative error of the encoder. Finally, the authors note that these differences persist across data. The main conclusions are not 100% clear to me.

Critical strengths

This paper tackles an interesting question about the role of capacity and bottleneck size in autoencoders.

Critical weaknesses

Results seem trivial / uninformative: This paper makes a big deal about the fact that, for a random decoder GG and high-dim intermediate decoder representation hh, optimizing hh directly allows you to do as well as (or even better than) the actual trained decoder. But this seems trivial: if the random GG is surjective (enough), you can reach any output by optimizing hh. Naturally, this is easier if hh is higher-dim! And indeed, when you use lower-dimensional hh (or similarly, you use zz instead, which is is a low-dim bottleneck), this fails.

The tricky thing for the decoder is precisely that the reps have gone through a bottleneck and are low-dim. The decoder can't possibly cover all of the output space, but if it learns enough about the data, it can learn which parts of the output space the data actually inhabit. The high-dim hh scenario really fails to capture what makes this problem difficult / interesting for the decoder. Did people actually think this was a decoder capacity problem? Asking since you claim that "Separating raw decoder capacity from learned accessibility is therefore necessary to understand what training actually contributes"

I have similar feelings about the "high-dimensional residual" finding. Yes, the optimized hh probably differs highly from the hh obtained via normal AE training. But this isn't surprising; the latter's range is constrained by the bottleneck, while the former is not. Why would one expect otherwise?

I again feel similar about the experiment where the bottleneck dimensionality is increased. If one buys that the bottleneck's initial low dimensionality explains the difficulty in optimizing zz rather than hh, then the accessibility gap must decrease as the ratio z/hz/h increases to 1. The fact that the gap between the encoder gap increases seems like a natural consequence. If optimizing zz did nothing over just using the encoder because zz was too low-dim anyway, then increasing the dimensionality of z makes it easier for optimizing it to do better than using the encoder. I feel like this deflationary account (that optimization is easier when you have a high-dimensional hh or zz, and harder otherwise) can explain many of your results.

Limited conclusions: The authors draw the following conclusions:

Narrow bottlenecks restrict which decoder states can be reached, producing a large accessibility gap, whereas wider bottlenecks reduce this restriction but increase encoder error. This trade-off suggests a structural tension in amortized representation learning between making a representation sufficiently expressive and making its useful states easy to identify. Whether this balance generalizes beyond the architectures studied here remains an open question

This largely gels with my deflationary account (though I don't think wider bottlenecks really "increase encoder error" so much as they make optimization of z∗z^* easier). But it's not clear to me what the big takeaway here is. What does it matter that wider bottlenecks do this? Does this inform autoencoder design somehow? In general, this finding seems kind of vague and high level, not super interesting from an interpretability perspective, but also not terribly useful.

Questions for authors
  • Do you agree with the deflationary account I outlined above?
  • Is there a practical takeaway from this result? Or a takeaway for some other specific audience?
1 comment

We appreciate your time and comments!

Re: if the random is surjective (enough), you can reach any output by optimizing. Naturally, this is easier if is higher-dim!

We agree with the surjectivity argument, and we acknowledge that the route is square and full rank on all models with z/h=1 (tested with h=512, 1024, 2048). However, upon scanning of all these model, we also found that the map is surjective but badly conditioned, with condition numbers of 1.8e5 (AE) and 3.0e5 (VAE) at z = h = 2048, up from 5.9e4 and 3.2e4 at 512. Driving hh along the weakest singular direction needs a latent component ~10⁵ times larger than along the strongest, against an encoder whose per-coordinate norm is 0.0125. So the non-zero value measured at that endpoint is not unstructured noise — and the growth of the condition number with width is also why a fixed optimize z budget saturates at z = 32 but not at z = 512 or 2048. (See F2x5 Q2 below)

In addition, if dimensionality were the determining factor, matched dimensionality would give matched performance, and it does not. At a matched 32 free scalars, a subset of hh closes 8.77% of the reachable gap against 6.59% for a subset of zz; the count is identical and only the location differs. And a randomly initialized decoder of the same depth and width, given 67× more optimization steps, is 17–1360× worse than the trained one and worse than predicting the mean image on MNIST; the dimensionality is identical and only the training differs. Dimensionality is necessary but not sufficient, which is what separating capacity from accessibility is for.

Re: Did people actually think this was a decoder capacity problem? Asking since you claim that "Separating raw decoder capacity from learned accessibility is therefore necessary to understand what training actually contributes"

We are not claiming the decoder lacks capacity; we claim the reverse — the capacity is present and the bottleneck cannot access it. The sentence you quote argues the two need separating precisely because they are usually conflated: a decoder that reconstructs poorly through its own encoder is normally read as a decoder that cannot reconstruct, and our measurement says that inference is wrong.

The reason it matters is a pattern the field already relies on: latent-diffusion pipelines reuse a trained VAE decoder as a general-purpose image generator, which treats its capacity as a reusable asset. Our decomposition measures how much of that asset a given bottleneck can reach.

Re: the latter's range is constrained by the bottleneck, while the former is not. Why would one expect otherwise?

The measurement that makes it surprising is on the other side of the decomposition. Exhaustive per-image optimization of the 32-dimensional latent, against the target, barely improves on the encoder that produced it — 5.6% on CIFAR-10, 12.1% on SVHN, 8.2% on Speech (means over 10 seeds at z = 32). The encoder is close to optimal given its bottleneck, so essentially all of the remaining shortfall is the route from zz to hh rather than the quality of inference. The natural prior is the opposite — that a better or more expensive encoder is where the headroom lies — and that is what the amortization-gap literature would predict.

Re: Do you agree with the deflationary account I outlined above?

Partly. We agree at the endpoint when z/h = 1 the map is surjective. But as we showed above, the map is also badly conditioned and dimensionality is not the only factor.

We disagree on your reading of what the bottleneck is. Your reading treats low dimensionality as a design that makes optimization hard. Our reading is that the compact code is coupled to the decoder it was trained with rather than being a general-purpose representation of the input, which is what an autoencoder bottleneck is usually taken to be. The decoder holds capacity the code cannot address, and that coupling is the cost of compactness.

Re: Is there a practical takeaway from this result? Or a takeaway for some other specific audience?

The audience is anyone reusing a frozen decoder — latent-diffusion pipelines, tokenizers, learned codecs. Two measured consequences: widening the bottleneck when transferring recovers real headroom, in an amount predictable from the accessibility gap at the source width; and starting from an intermediate layer recovers more than widening does, since scalars spent in hh buy 1.3× what scalars spent in zz buy at matched count. The design caution is that a bottleneck tuned for compactness is being tuned against reusability.

We also tested a fair version of the objection that our models are simply undertrained. At 3,000 / 9,000 / 30,000 training steps the accessibility gap widens with further training while the encoder gap shrinks (e.g., for AE at z=512, h=2048, the accessibility gap is 42.4%, 51.8% and 62.1% for the 3 steps listed above), so undertraining was holding our effect down rather than propping it up.

Reviewer_BbxsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a study of the reconstruction loss across different stages of an autoencoder. The authors decompose reconstruction errors into three components: "decoder-limited error, an encoder gap between the best bottleneck state and that predicted by the encoder, and an accessibility gap between the best intermediate representation and the best state reachable through the bottleneck". The authors conclude that poor reconstruction "can reflect restricted access to the decoder's capability" rather than "insufficient decoder capacity".

Critical strengths

The paper is investigating an interesting hypothesis. Also, the overall structure of the paper is easy to follow. The visualization and explanations are straightforward.

Critical weaknesses

I remain unconvinced about the conclusions in the paper. The approach freezes the weights of the models, and then independently optimize h or z for each image, but that does not result in a generalized mapping from images to representations. I would have expected two ablation studies:

Does the observed performance simply depend on the number of optimizable variables? For example, in this work, the authors either optimize z, or h. Instead, the authors could randomly select subsets of dimensions from the bottleneck representation (z) and an intermediate representation (h), while keeping the total number of optimizable variables constant. By varying the proportion of variables allocated to (z) versus (h), the authors could investigate whether reconstruction quality depends on where the optimization occurs. This ablation study can also be done on SD-VAE, where h can be chosen from different intermediate layers. It is possible that it doesn’t matter where the optimizable variables are. But, only the degrees of freedom (the number of optimizable variables) primarily influence the reconstruction error: the more the variables, the lower the error. An interesting trend can be observed through the result in Figure 2: the higher the number of optimizable variables is, the lower the reconstruction error is. If so, there is no insight in these experiments. It could be that if you increase the number of optimizable variables, then you get lower reconstruction error.

The approach in the paper involves optimizing representations for individual samples while keeping the network weights frozen. Although this is a valid approach for characterizing the reconstruction capabilities of a fixed decoder, it does not directly establish how training organizes representational capacity or how the resulting representations generalize. I would have expected comparisons between randomly initialized and trained decoders, using identical per-sample optimization procedures. This would have helped determine how much of the observed behavior is attributable to learned structure rather than network architecture and optimization dimensionality.

Questions for authors

I would love to hear the author's responses for the two issues pointed out in the weaknesses section.

In addition, the experimental evaluation appears to rely on small numbers of images. Figures 1(a) and 1(b) use only 16 images, while Figure 1(c) uses only 32 images, sampled from datasets such as CIFAR-10, MNIST, and Flowers102. The paper does not explain how these images were selected. Were they randomly sampled from the training or testing splits? Was sampling performed across different classes, and were the same images used across all experimental configurations?

More importantly, 16 or 32 images seem quite small to establish statistically reliable conclusions about the reconstruction error decomposition. The trends reported in the paper could be entirely sensitive to the specific images selected.

I recommend that the authors substantially increase the number of evaluated images and report the stability of their findings as a function of sample size. In particular, they should plot the estimated reconstruction error components against the number of evaluated images, using repeated random sampling and confidence intervals to demonstrate both convergence and statistical reliability.

Without such an analysis, it is difficult to determine whether the reported trends are robust properties of the models or artifacts of the limited number of samples used in the experiments.

1 comment

We appreciate your time and comments

Re: the authors could randomly select subsets of dimensions from the bottleneck representation (z) and an intermediate representation (h), while keeping the total number of optimizable variables constant. […] If so, there is no insight in these experiments.

We ran the ablation as suggested: 450 configurations, 5 seeds, 1,000 images, nothing retrained, random subsets of zz against random subsets of hh with the total count held fixed. The fraction of the reachable gap each budget closes:

free scalarssubset of zsubset of h
40.71% ± 0.040.92% ± 0.04
81.41% ± 0.101.96% ± 0.07
162.79% ± 0.144.26% ± 0.12
326.59% ± 0.238.77% ± 0.25

At every matched count, scalars spent in hh buy more than scalars spent in zz — about 1.3× more at 32.

Re: I would have expected comparisons between randomly initialized and trained decoders, using identical per-sample optimization procedures.

Thank you for the suggestion. We have run the comparison, and it changed the reading of results in Fig 1.

We used one architecture, CIFAR-10, z = 32, three decoder widths. One arm is trained; the random counterpart is the same model at its random initialization. Both arms then optimize the hidden state on the same 500 held-out images with the same step counts and learning rates.

harmencoderopt h ceiling
512trained0.0195910.007806
random0.0736440.063153
1,024trained0.0133710.003639
random0.0812810.062928
2,048trained0.0126660.002322
random0.0764450.058292

Mean-image baseline 0.065204. The random arm never gets meaningfully past it. Hence comparing two decoders on reconstruction error compares two different things once one of them has more free variables than the image has pixels. At that point it can be bent onto any target and a low error stops being a statement about the decoder. We tested this with the held-out-pixel protocol we use elsewhere: optimize the hidden state on a random half of the pixels, score the other half.

decoderfree numbersceilingunseen halfmean image
trained2,0480.00230.00490.0652
random, per channel at 2×6,1440.00320.19190.0661
random, per channel at 4×12,2880.000590.16750.0661

The trained decoder is thirteen times better than a constant image on pixels it never saw. Both random decoders are two and a half to three times worse than a constant, despite ceilings that look comparable or better.

The direction across mask fractions is the clearest part. Show a random decoder more of the image and it gets worse on the rest — 0.149, 0.192, 0.217 at 2× and 0.137, 0.167, 0.190 at 4× for a quarter, a half and three quarters visible. The trained decoder improves over the same range: 0.0128, 0.0049, 0.0039. Extra information helps a decoder that holds images and hurts one that only fits them.

So the claim that random decoder contains sufficient capacity is wrong and we will remove it in the updates. The accessibility gap on the other hand, is a separate measurement from the random-decoder comparison and is unaffected by this correction.

Re: plot the estimated reconstruction error components against the number of evaluated images, using repeated random sampling and confidence intervals.

We have done the test as you suggested using random AE models with h=512, 1024, 2048 on CIFAR-10. We individually optimized 256 images each for 20,000 steps and kept their MSEs. We then subsampled N = 8, 16, 32, 64, 128, 256 MSEs from the saved ones with 5 random seeds. We report the mean MSE ±\pm std for each h value, and also compared if larger h values would lead to lower MSE.

Nh = 512h = 1,024h = 2,0481,024 beats 5122,048 beats 1,024
80.0396 ± 0.00730.0246 ± 0.00430.0065 ± 0.001599.9%100%
160.0396 ± 0.00510.0246 ± 0.00300.0065 ± 0.0010100%100%
320.0396 ± 0.00340.0246 ± 0.00200.0065 ± 0.0007100%100%
640.0396 ± 0.00230.0246 ± 0.00130.0065 ± 0.0005100%100%
1280.0396 ± 0.00130.0246 ± 0.00080.0065 ± 0.0003100%100%
2560.03960.02460.0065100%100%
Reviewer_TU3uOpenReview ↗
Rating
1(−1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that poor reconstruction through a trained autoencoder can reflect restricted access to the decoder's capability, through the bottleneck, rather than a lack of decoder capacity. The authors first show that even random, frozen convolutional decoders reconstruct images accurately when a high-dimensional intermediate representation h is optimised directly for each image. Reconstruction capacity is therefore not inherently tied to training. They then decompose the reconstruction loss into three terms: the decoder-limited error (the floor reached by optimising h directly), the accessibility gap (how much the bottleneck limits what the decoder can reach) and the encoder gap (the loss caused by the encoder not producing the best bottleneck code). Experiments on images and speech, with convolutional AEs and VAEs, show that widening the bottleneck shrinks the accessibility gap while the encoder gap grows, leaving the total comparatively stable. A PCA study of the residual Δh between the encoder's intermediate state and the optimised one finds it high-dimensional and mostly outside the subspace reachable through the bottleneck. Under distribution shift, optimising h also reaches target-domain images that the bottleneck route reconstructs poorly. The authors conclude that learning does not create reconstruction capacity but organises it better, allowing to use a smaller number of dimension for the intermediate representation and for the bottleneck.

Critical strengths
  1. Breadth. The extensive width sweep, the range of models (AEs and VAEs), modalities (images and speech) and domain shifts, and the report of both normalised and unnormalised gaps all add to the generality of the results.

  2. Clarity. The paper is easy to read and to follow. The question is clear and the prose is good. The decomposition is intuitive and reusable as a diagnostic.

  3. PCA analysis. Sec. 2.3, the PCA analysis of the residual Δh = h* − h⁰, appears to be a new idea and adds a useful tool to that diagnostic.

Critical weaknesses
  1. Several results are either fixed by construction and then blurred by optimisation noise, or lack the controls needed to interpret them. Take the decomposition first. Eq. 2–4 define L_z* and L_h* as true minima. Since every state reachable through the bottleneck is also an admissible h, the true accessibility gap can never be negative, and the same holds for the encoder gap. What the experiments report, though, is the loss reached after a fixed number of Adam steps, with very different budgets: 300 z-steps against 20,000 h-steps in Fig. 2, and up to 20,000 z-steps against 3,000 h-steps in Fig. 4. Each reported gap is thus the true gap plus the difference of two optimisation errors whose sign is unknown. I suspect this is why Fig. 4a and S3a appear to show negative gaps at z/h = 1. At that end point, moreover, if F is an invertible linear layer, every h is reachable and the true gap is zero by construction, so its vanishing is not an empirical finding. Starting the h-optimisation from F(ẑ) and keeping the best iterate would guarantee a non-negative measured gap, and convergence curves would show how far each estimate sits from its minimum. The VAEs show the mirror image. There, the accessibility gap stays positive at z/h = 1 even though the z-optimisation gets at least as many steps as the h-optimisation, so optimisation error cannot be the explanation. My guess is posterior collapse: in linear VAEs, collapsed latent dimensions come with zero decoder columns (Lucas et al., 2019). If something similar happens here, W is close to rank-deficient, and part of h-space stays out of reach even when z and h have the same size (see question 5). That would be a finding in itself, but one about collapse rather than about decoder capacity. The second problem, missing controls, directly affects two analyses. In Fig. 4, the three normalised curves are linked by the identity 1 − T = (1 − a)(1 − e). Whenever the total T stays roughly flat, a falling accessibility gap a mechanically forces a rising encoder gap e; on top of that, each width is a newly trained model, so width and training effects are mixed. In Sec. 2.3, gradient steps on h always lie in the row space of the decoder Jacobian, so they naturally avoid directions that leave the output unchanged, and to first order Δh is the Jacobian's pseudo-inverse applied to the pixel residual. A high-dimensional, "functionally relevant" Δh is therefore what one would expect from any narrow code. The PCA dimension of the pixel residuals is the baseline it should be compared with.

  2. The conclusions go beyond what is measured. The decoder floor is obtained by optimising 2,048 free numbers per image, with the target in hand, for up to 20,000 steps and without any prior or constraint. This shows that some h reconstructs the image, not that the decoder has a usable capability. The optimised h leaves the region of h-space the decoder was trained on, and on MNIST it uses more numbers than the image has pixels. The intermediate-layer literature cited in Sec. 3.3 documents this cost quite explicitly. ILO notes that "intermediate layers can produce nonrealistic images and therefore inversion must be regularized" (Daras et al., 2021); Pividori et al. (2019) obtain representations that "although map to the target image, are not related to the distribution"; and Abdal et al. (2019) find that "only the embedding of faces is semantically meaningful". Yet the paper cites these works only for expressivity. Measuring even one cost, for instance on an inpainting or denoising task, or as the distance of h* from the codes seen during training, would make the capability claim concrete. Hierarchical VAEs (NVAE, VDVAE) would be the natural point of comparison: they give intermediate layers both an encoder and a learned prior, so access is amortised and paid for in KL. The central interpretation, that training "organizes high-dimensional capacity so that useful outputs can be reached through far fewer control variables" (abstract), is never tested directly. The paper's own Fig. 1 even suggests that training does more than organise: at the same width (2×), the trained decoder reaches clearly lower error than the random ones. A simple test would be to freeze the decoder body at its random initialisation, train only E and F, and compare the result with the fully trained autoencoder at the same bottleneck size.

Questions for authors

Questions

  1. How many free variables per image does each curve and bar of Fig. 1 use? The random decoders are optimised per RGB channel for CIFAR-10 and Flowers102, while the autoencoders map h = 2048 to the full (C, 32, 32) image (A.2.1). Could you also report the MSE values at 2×?
  2. Is h taken before or after the activation that follows decoder.fc? After a ReLU, the zero gap at z/h = 1 no longer holds by construction.
  3. How many samples N enter the PCA of Sec. 2.3, and are the principal components centred in Eq. 6?
  4. Is the warm start from the encoder used in Fig. 2, 4 and 5? A.2.2 states it only for Fig. 1c. On which images were learning rates and step counts selected?
  5. For the VAEs: how many latent units are active at each z/h, what is the singular-value spectrum of W at z/h = 1, is the reconstruction term summed or averaged over pixels, and does L_rec decode the posterior mean or a sample?
3 comments

We appreciate your time and comments! (response 1/3)

Re: "What the experiments report, though, is the loss reached after a fixed number of Adam steps, with very different budgets: 300 z-steps against 20,000 h-steps in Fig. 2, and up to 20,000 z-steps against 3,000 h-steps in Fig. 4."

The model we used in Fig. 2 (h=2048, z=32) is only one of the models we tested in Fig. 4 (as you noted). We used different optimization steps for different models in Fig. 4. In all cases, the loss gets very small but keeps decreasing with more training even beyond 20,000 steps. for example, when z = 1024 and h = 2048, MSE of opt_z is still decreasing from 0.0013636 at 16,000 steps to 0.0012690 at 64,000 steps; similarly, MSE of opt_h is 0.0013076 at 1,000 steps, 0.0012567 at 4,000 steps and 0.0011949 at 64,000 steps. So both opt_z and opt_h are undertrained, opt_h even more so, and the gap still exists.

Re: "At that end point, moreover, if F is an invertible linear layer, every h is reachable and the true gap is zero by construction, so its vanishing is not an empirical finding."

We did not claim the vanishing of accessibility gap at z/h = 1 to be a result by itself, it's the trend of change from z/h << 1 to z/h=1 that matters.

Re: "My guess is posterior collapse: in linear VAEs, collapsed latent dimensions come with zero decoder columns (Lucas et al., 2019). If something similar happens here, W is close to rank-deficient, and part of h-space stays out of reach even when z and h have the same size."

Lucas et al. predict a zero decoder column per collapsed dimension. Across all AE models the smallest column of WW is 0.77–0.84 of the mean, and WW is full rank in all VAE models. Hence the quiet dimensions are really unused, not unreachable. We found the real reason the accessibility gap stays positive at z/h = 1 in VAE models is the KL penalty. The stronger the KL penalty, the larger the mismatch. As we tested on SVHN, freezing the decoder bit-identically, deleting the KL (setting β\beta=0) and retraining the encoder alone removes 73.7% of the encoder gap at z/h = 1 (for h=2048).

Re: Whenever the total T stays roughly flat, a falling accessibility gap a mechanically forces a rising encoder gap e; on top of that, each width is a newly trained model, so width and training effects are mixed

First, the total loss (T) stays roughly flat is not by design, it is a result by itself. It means the ratio between L(h∗)L(h^*) and L(z0)L(z^0) stay relatively flat across different ratios of z/h, which is not so obvious before testing.

But we agree that each width is a model trained separately from others in Fig 4. to illustrate the effect more clearly, we reran the test with a single trained model where z=h=2048, and then randomly choosing different subsets from z to show the effect of varying z/h (only one random seed for the subsampling test). The result is below for the normalized accessibility gap:

z/h1/641/321/161/81/41/21
Fig. 4, trained at each width+85.7%+82.7%+76.2%+59.9%+38.3%+5.8%−7.5%
subset of one trained model+79.6%+78.7%+76.2%+62.6%+39.4%+9.2%−2.7%

The trend stays the same as before.

Re: A high-dimensional, "functionally relevant" Δh is therefore what one would expect from any narrow code. The PCA dimension of the pixel residuals is the baseline it should be compared with.

We agree this is the right baseline and we ran it with four datasets × 10 seeds, centred PCA on 4,000 images: Δh needs more components than the pixel residual while living in a smaller space (2,048 against 3,072) — r95r95 1,264 against 594 on CIFAR-10, and 1,240 against 283 on SVHN, i.e. 2.1 to 4.4× more. So Δh is not inherited from the pixel residual. However, there was a bug in our code here. While doing optimization on the 4,000 images, we threw all of them into one batch to save time, and used the mean loss of all 4,000 entries for gradient computing as conventional for batch processing. But we shouldn't have, since the 4,000 optimizations should be done independently, they are not a real batch. We fixed this bug, and found that the fraction of the residual the encoder can reach is 26–38% — 62–74% unreachable, at 17–24× chance — so "essentially orthogonal" overstates it and we will drop that phrasing. And we will report r95r95 rather than the participation ratio, which weights the head of a spectrum carrying a large common offset and is the wrong summary here.

Response (2/3)

Re: This shows that some h reconstructs the image, not that the decoder has a usable capability. […] Yet the paper cites these works only for expressivity.

This is a valuable input, and we have done two tests to clarify the case.

First, the inpainting test. We hide a random portion of the pixels, optimize h on the visible part only, and score the hidden part, which the optimizer never saw. We report the fraction of the encoder-to-ceiling gap that is closed on those unseen pixels — 0% means freeing h bought nothing the encoder did not have, and 100% means seeing part the image was as good as seeing all of it. CIFAR-10, z = 32, 500 held-out images.

h25% visible50% visible75% visible
51257.6%81.2%86.9%
1,02437.8%73.7%81.9%
2,04834.2%75.0%84.2%

At h = 2,048 with half the image hidden, the MSE of unseen pixels come out at 0.004896 against the encoder's 0.012645 and a full-information ceiling of 0.002318. So the ceiling is not purely an artifact of holding the target: three quarters of it survives when the scored pixels are withheld from the optimizer.

Second, the displacement. We calculated ‖h∗−h0‖/‖h0‖‖h^*−h_0‖/‖h_0‖, per image then averaged, measured on the same three-width at 3,000 steps: 1.55 at h = 512, 1.41 at 1,024, 1.31 at 2,048. The optimizer moves the hidden state by more than its own length, which is not a small correction to where the encoder put the image. It falls with width — a wider decoder reaches its ceiling from closer to h0h_0 — which is the opposite of what "more room to wander" would predict.

There are two caveats: 1. It depends on the budget: at h = 512, 3,000 steps gives 1.55 and 20,000 gives 5.87 while the ceiling itself improves by 3%, so a long run measures the optimizer's wandering and not the distance the decoder needs. And h∗h^* is not unique, so what we report is the displacement Adam happened to reach, which is an upper bound on the displacement the decoder requires. 2. These are the ceilings of the h sizes we ran for this reply. They are not the same as reported in Figs. 2 and 4.

There is an additional result regarding dataset: the test fails on MNIST: at 50% visible, freeing h makes the unseen pixels 35–84% worse than the encoder's reconstruction. The variable that separates the two datasets is not the width, and not the count of free variables against observed pixels — CIFAR-10 has a ratio of 2.67 free variables per pixel still closes 34% of the gap, while MNIST at the same ratio is negative. It is whether the optimizer can drive the visible residual to zero. On MNIST at h = 2,048 with a quarter shown, the visible residual reaches 3 × 10⁻⁶, and once the fit is exact nothing constrains the remaining directions. On CIFAR-10 the decoder cannot fit even the visible pixels exactly (residual 0.001–0.008), so it must stay on the part of its range that looks like a natural image.

Re: Hierarchical VAEs would be the natural point of comparison: they give intermediate layers both an encoder and a learned prior.

We think our decomposition does not apply to those models as it stands: the telescope requires hh to be everything downstream of a single zz, and a hierarchical VAE has latents at several resolutions with deterministic paths around them, so Lh−LzL_h − L_z is not defined. This is the same boundary we draw for U-Nets, transformer blocks and diffusion backbones in answer to Reviewer F2x5's Q5.

Re: The central interpretation, that training "organizes high-dimensional capacity so that useful outputs can be reached through far fewer control variables" (abstract), is never tested directly.

Thank you for the suggestion, we have made direct tests and now have two results. The first is the degrees-of-freedom ablation run at Reviewer Bbxs's suggestion, which holds the number of free scalars fixed and randomly subsample from them: at a matched 32 scalars a subset of hh closes 8.778.77% \pm 0.25 of the reachable gap against 6.596.59% \pm 0.23 for a subset of zz, about 1.3× more for the same budget (see full table in our reply to Reviewer Bbxs). The second is the conditioning result in response to Reviewer F2x5's Q2 — the z→hz \rightarrow h route at z = h is square and invertible, but with a condition number of 1.8e5 to 3.0e5 only a small set of its directions is usable in practice.

Response 3/3

Re: The paper's own Fig. 1 even suggests that training does more than organise [...] A simple test would be to freeze the decoder body at its random initialisation, train only E and F, and compare the result with the fully trained autoencoder at the same bottleneck size

Thank you for the suggestion, we have run the test as suggested. We used one architecture, CIFAR-10, z = 32, three decoder widths. The decoder body is frozen at its random initialization; the encoder and the z→hz \rightarrow h map are trained against it, with the same budget, data and optimizer as the fully trained model. Both arms then get the same per-image opt z and opt h on the same 500 held-out images.

harmencoderopt zopt haccessibility gap
512fully trained0.0195910.0151510.00780648.5%
frozen body, E and F trained0.0653200.0648160.0622873.9%
1,024fully trained0.0133710.0122200.00363970.2%
frozen body, E and F trained0.0654410.0650150.0620004.6%
2,048fully trained0.0126660.0116120.00232280.0%
frozen body, E and F trained0.0616060.0614080.0564098.1%

Mean-image baseline 0.065204.

Our statement that random decoder is already expressive is wrong, and training does create capacity rather than merely organize. We will correct this in the updated manuscript. However, the finding that encoder capacity exceeds what the bottleneck can reach — the accessibility gap, which is the novel part of our finding is unaffected.

Re: your questions

Q1. At ratio rr the random decoders use 3r×10243r × 1024 numbers against the trained decoder's 2,048 for the whole image, so the comparison is not matched. The honest version is stronger: MSE at 2× is 0.0159 random against 0.00233 trained, and a random decoder needs roughly 15,360 free numbers to match 2,048 trained ones. That figure reads off the minimum over searched cells; on a typical cell it is about 61,440, so what we quote is conservative.

Q2. Before the activation.

Q3. N = 4,000 against h = 2,048. We compute both centered and uncentered spectra and will report which each figure uses; they differ materially, because the residual carries a large common offset.

Q4. The warm start is used in every figure reporting a gap, not only Fig. 1c. The appendix documents it in one place only and we will fix that.

Q5. LrecL_{rec} is summed over pixels and decodes the posterior mean, never a sample. See response to Reviewer F2x5's Q2. On active units we decline to give a count, because the per-dimension KL is smooth near any threshold rather than bimodal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.