acceptodds
Withdrawn from ICLR 2027

SOLA: Stochastic One-Loss Autoencoders with Population-Derived Latent Geometry

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Variational autoencoders (VAEs) typically obtain stochastic latent representations by balancing reconstruction against a Kullback–Leibler (KL) regularization term, introducing a problem-dependent trade-off that can lead to posterior collapse. We ask whether a stochastic generative autoencoder can retain a principled KL interpretation without optimizing a KL loss at all. We introduce the *Stochastic One-Loss Autoencoder* (SOLA), which derives both latent stochasticity and sampling geometry directly from the encoded population. SOLA combines a rank-reduced orthogonal latent representation with random latent mixing, inducing Gaussian perturbations whose covariance is determined by the geometry of each encoded batch. We show that the resulting stochastic representation admits a geometry-matched Gaussian reference for which the batch-averaged KL divergence is exactly constant for fixed bottleneck dimension and noise scale, independently of the network parameters and learned singular spectrum. Its optimization gradient is therefore identically zero. Accordingly, SOLA is trained using reconstruction alone while retaining a stochastic latent representation. After training, the learned batch geometries are aggregated into a fixed global representation that supports deterministic inference and a population-derived Gaussian sampling rule, without a learned variance head, KL weight, or KL schedule. Across six synthetic, structured, and real-world datasets, SOLA achieves competitive reconstruction and generation using the same noise scale throughout. In contrast, both the preferred VAE KL weight and the qualitative response to changing it vary substantially across problems. Controlled ablations separate the effects of dimensionality reduction, orthogonal latent geometry, and stochastic mixing, while empirical diagnostics assess the latent statistics underlying the Gaussian sampling approximation. Comparisons with modified VAE objectives and alternative generative autoencoders, including KL annealing, Free Bits, -VAE, capacity-controlled -VAE, RAE, WAE, and VQ-VAE, further show that competitive generative performance can be obtained without explicitly optimizing a prior-matching objective or fitting a separate latent density for SOLA's native generator. The code is open-source and available in the supplementary material.

Primary area:
generative models
Keywords:
Variational Autoencoders, Generative Modeling, Stochastic Autoencoders, Latent Space Geometry, Low-Rank Representations, Singular Value Decomposition, Representation Learning, Posterior Collapse

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1h9ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

SOLA addresses the reconstruction–KL trade-off by combining an SVD bottleneck with population-scaled stochastic noise and reconstruction-only training. It derives a constant average KL divergence to a geometry-matched reference and uses aggregated latent statistics to construct a Gaussian generator. Experiments assess reconstruction and generation across six datasets using the same noise scale. Additional component ablations and CelebA controls with fitted Gaussian latent densities test the roles of the bottleneck, noise, and sampling procedure.

Critical strengths

The paper clearly presents a coherent training and sampling procedure, with a valid constant-KL derivation under positive noise scale and full-rank retained coordinates. This provides an interpretable reconstruction-only alternative to balancing reconstruction and KL penalties.

Using the same noise scale across six datasets supports the method's practical promise. Comparisons with tuned VAEs, other autoencoders, and component and fitted-density controls make the evidence for feasibility and reduced tuning more convincing.

Critical weaknesses

The constant-KL identity provides limited theoretical support for generation: both the noise covariance and reference distribution track the encoded batch's second moment. It does not establish that aggregated codes match the final sampling Gaussian; orthogonality and matching second moments are insufficient. The authors acknowledge the Gaussian approximation, so this limits the result's explanatory reach without invalidating its derivation.

The practical gain over simple alternatives appears modest. On CelebA, Table 10 reports FID 102.84 ± 6.44 for AE plus Gaussian fitting, versus native SOLA's 99.84 ± 4.12 in Table 9. This small difference leaves the benefit of the added design unclear. Combining existing components into one procedure is useful, but the advance appears incremental. The controlled autoencoder experiments establish feasibility, while advantages for stronger models and more complex data remain unknown.

Questions for authors

What practical advantage over AE plus Gaussian fitting most strongly supports SOLA's contribution?

How do the pooled Gaussian diagnostics support the approximation in the fixed basis used for generation?

Reviewer_rp3BOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SOLA, which adds Gaussian perturbations determined by encoded second moments to a per-batch low-rank SVD bottleneck and trains using reconstruction loss alone. The authors prove that the average KL divergence to a geometry-matched Gaussian reference for the same batch is constant, then aggregate batch geometries to construct a fixed Gaussian sampler. Experiments compare the method with VAEs and other approaches on six datasets.

Critical strengths
  1. The method is clearly defined, and the central algebraic derivation is correct under the stated conditions. The authors explicitly distinguish the constant-KL identity from the Gaussian approximation used for generation.

  2. The paper includes multiple training seeds, classical baselines, and component controls, and reports unfavourable results and computational overhead. The post-training Gaussian-fitting control is particularly useful for distinguishing representation quality from sampling-distribution mismatch.

Critical weaknesses
  1. A sufficiently important new research contribution has not been established. Low-rank bottlenecks, reconstruction-only training, and stochastic low-rank representations all have direct precedents. The main constant-KL result also holds for general full-rank coordinates with a reference distribution matched to those coordinates; it mainly describes an algebraic property of the chosen construction. It does not explain when the encoded population is suitable for the final Gaussian sampler or provide new insights into generation quality or failure boundaries. The paper does not engage substantively enough with current related approaches to establish an important unresolved gap or a compelling reason to adopt this method.

  2. Experimental scope and quality–cost evidence are insufficient to support the contribution. Missing evidence includes recent relevant strong methods, more challenging representative benchmarks such as ImageNet 256×256, modern backbones and larger latent scales, and distribution-coverage evaluation. On CelebA, a simple AE or compact G-only model with a fitted full-covariance Gaussian already approaches SOLA, while per-batch SVD has a clear training cost. The results do not establish quality, stability, or overall workflow benefits that justify this cost. This is a substantial gap in the scope and evidential basis of the experimental study, rather than an issue resolved by adding one or two baselines or correcting tables and prose.

Overall assessment: I recommend clear rejection. The paper provides feasibility evidence and a correct algebraic derivation, but does not establish sufficiently important new knowledge or a worthwhile practical advantage. My main concern is the adequacy of the research contribution, rather than whether the method runs or whether isolated reporting errors exist. A change in assessment would require substantive theoretical insight or convincing empirical conclusions.

Questions for authors
  1. Relative to direct prior work and current related single-stage or latent generative approaches, what important new knowledge does the paper establish, and which unresolved problem does it address? Which results substantiate that positioning?

  2. Beyond the identity under a geometry-matched reference, can the constant-KL result provide new testable conclusions about when actual Gaussian sampling works, generation quality, or failure boundaries?

  3. How can a convincing evaluation be established against recent relevant methods, on representative benchmarks with realistic difficulty, and across more modern backbones and latent scales? Please distinguish direct-sampling methods from pipelines requiring an additional generative model, and use consistent protocols and complete cost accounting.

  4. What meaningful advantage does SOLA provide over alternatives such as VRRAE or a simple AE with moment estimation? Please include training-time SVD, statistical aggregation, model selection, and sampling costs, and evaluate the trade-off between sample quality and distribution coverage.

Reviewer_BRqBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces an alternative to KL for training a AE-like model by introducing its own geometry, and essentially pointing out that imposing latent structure in the latent formation step of an AE can end up allowing losses which are pure reconstruction losses and completely cut out the KL-loss, which generally makes the latents Gaussian-like, by doing a batch-level norm + SVD + noise addition to argue for approximate Gaussian-like structure. The empirical results are quite weak and the novelty is low.

Critical strengths

Autoencoders have long had issues with their latent spaces that have in general led to a variety of jiggling around the KL term and the paper studies an important problem. Further, reconstruction losses in general have always been more high-signal than the KL loss, so the paper's thrust is correct.

Critical weaknesses
  • The empirical choice of datasets is extremely weak for a 2026 paper.

  • The resulting gain in terms of FID is not impressive. There are other major issues (see below in questions)

  • The so-called "recovery theorem" for a KL loss being constant is misleading, to the point of re-deriving elementary VAE math.

Questions for authors

https://arxiv.org/html/1903.12436v4 - RAE - quotes a much lower FID. Why is the FID reported in your paper so different ?

I assume this is due to the low epochs used. But then every method is being compared before convergence, which renders the comparison meaningless.

Even leaving this aside, the gap to the RAE is very slight and concerning given that the RAE paper is from 2019 - a lifetime ago in ML.

Please explain these discrepancies (also for CIFAR10). In general, all FID scores seem to be much worse than previously reported for all models, due to underfitting.

Please show an updated experimental section with more datasets suitable for 2026, or at least for CelebA and CIFAR-10/100 (if not ImageNet) carried out to completion, with definitive FID wins that are competitive on an absolute scale. Otherwise this paper in its current form carries almost zero empirical signal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.