acceptodds
Under review as a conference paper at ICLR 2027

Superposed Latent Autoencoder

Abstract

Autoencoders typically meet tight latent-memory budgets by making each latent representation smaller, sacrificing representational capacity. We ask a different question: can multiple wider latents be stored together instead? We introduce the Superposed Latent Autoencoder (SLAE), which preserves high-capacity latent representations while sharing storage through learned superposition. SLAE transforms latents into storage-friendly codes, binds them with randomized keys, superposes multiple codes into a single memory tensor, and learns to recover each latent before decoding. Under the same storage budget, SLAE replaces irreversible dimensional bottlenecks with structured interference that can be suppressed. Across CIFAR-10/100, SVHN, STL-10, Tiny ImageNet, and a wide range of memory budgets, SLAE substantially improves the reconstruction–memory tradeoff, reducing reconstruction error by up to 56% over conventional autoencoders at matched storage. Further analysis shows that SLAE's advantage comes from making wider representations usable under the same storage budget. These gains also extend beyond reconstruction: the information preserved by SLAE improves downstream classification by up to 16.79 percentage points under the same memory budget. Our results suggest a new principle for representation compression: instead of making every latent smaller, keep representations wide and let them share memory.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, Autoencoder, Latent Superposition, Memory-Efficient Learning, Neural Compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_787HOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper finds that letting several of them share one memory can retain more information than shrinking each latent independently. It proposes the Superposed Latent Autoencoder, which binds wide latents with randomized keys, superposes them into a single tensor at the same average storage, and recovers each latent before decoding. At matched storage, this reduces reconstruction error by up to 56% and improves downstream classification by up to 16.79 percentage points.

Critical strengths
  1. The paper is clearly written. The capacity–interference framing, the matched-storage setup, and the method are easy to follow, and the figures and notation line up with the claims.

  2. The experiments are well designed. Comparisons use a matched storage budget across datasets and budgets, and the capacity–interference breakdown plus the depth and joint-compression controls make the source of the gains interpretable.

Critical weaknesses
  1. The article lacks results based on large-scale datasets, such as ImageNet-1K/21K. The manifold distribution of large-scale datasets is highly complex, and this algorithm, which combines multiple sample latent variables, may fail on ImageNet-1K/21K.

  2. Additionally, in Appendix M, the comparative experiments between the SLAE and quantization is also insufficient. INT4 PTQ quantization often requires more advanced techniques (e.g., smooth/OBQ). Instead of the naive post-training per-channel uniform quantization with scales calibrated on the training set.

  3. Representation quality should be evaluated by using the latent itself as a frozen feature and training a fully connected classifier on top of it. On Table-9, this probe measures how much class-discriminative information survives lossy compression.

  4. For the figures in the paper, please use vector graphics instead of screenshots.

Questions for authors

Please see weakness

Reviewer_17T2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Superposed Latent Auto-Encoders (SLAE), a method to store latent representation of data under storage constraints. Suppose, you are given K images, the basic premise of the paper is that there a broadly two ways to store the representation with storage constrained to total C scalars: (1) Store each of the data in C/K dimension without any interference (this is what a conventional auto encoder implements), or (2) we can store each data in C dimensions (wider than the first latent space) but allow the model to superposed the features. During superposition, each of the features is bound with a distinct key, which can be used later to decode to the unique data point from the shared latent space. The method is implemented on the vision datasets (cifar, svhn, tiny imagenet), and shows significant improvement over the conventional auto-encoder (which is referred to as plain autoencoder in the manuscript).

Critical strengths
  1. The paper is very written, and describe the problem statement and motivation for the methodology very well.

  2. The idea of using superposition to store latent vectors is intuitive, and the anaylysis and ablation are provided on understanding where the gains come from.

  3. Strong improvement over the baselines in fig. 4 and table. 1

  4. The learned representation not only recovers the original input but also captures meaningful representations for the downstream tasks.

Critical weaknesses
  1. The datasets used a simple. One good application of this approach could be archiving in digital pathology and medical imaging [1]. I would encourge authors to further explore this direction to make the method practical more useful.

  2. Comparison with other methods (mentioned in the related works) is missing.

  3. Comparison with quantization based methods is another important baseline that the authors should compare with (paper has some initial results on cifar10 in the appendix).

  4. The storage analysis does not take into account storage used the randomized keys, used for binding.

I would be open to upgrading my score if the authors can provide evidence by comparing with the existing methods/baselines.

Kalra et al., Yottixel - An Image Search Engine for Large Archives of Histopathology Whole Slide Images

Questions for authors
  1. What is the overhead of storing the randomized keys?
Reviewer_U33WOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presented an approach to model compression in autoencoders that leverages randomized superposition to maintain a wider latent code per example under a fixed memory budget. The key idea is that by randomizing how latent codes are projected into a joint memory space, you could recover a better construction with the same amount of memory, compared to just making the latent dimension smaller. The key contributions of the work consist of how the authors conduct the randomization into the shared memory space, as well as the use of a recovery network to (if I understood correctly) do a sort of error correction/re-mapping of the latent codes to perform better reconstruction. The experimental protocol compares the proposed method with a memory equivalent autoencoder setting, and the supplemental material expands on this analysis with other methods of reducing autoencoder memory.

Critical strengths

Overall, the writing was clear and easy to follow, and I appreciated the simplicity and elegance of the idea of using superposition in this way. While I was not familiar with existing parameter superposition work, having read up about it in the process of the review, the approaches used by the authors seem to follow common approaches used in that domain. I find the demonstrated performance for the fixed memory size to be small, but notable and compelling nonetheless. Even if perhaps not ground-breaking, I thought the paper was still a solid piece of work, and I encourage the authors to continue working along these directions. I learned something in reading the paper and doing the review, and it also helped me recall older concepts I had seen in different ML contexts 1-2 decades ago, which I always appreciate in newer papers.

Critical weaknesses

I only have two large-scale suggestions for the manuscript (one philosophical and one technical).

The first point relates to the overall positioning/framing of the paper, and is not a technical criticism but more related to the purpose. I understood from the manuscript that the point here was really to deal with memory requirements for AE models, and I am sympathetic to and supportive of this, however it was hard for me to get excited about this since in the domains I work in, having a very compact (i.e., as small as possible) latent space is far more valuable than saving memory, and so, on a practical level for my own work and those of the companies we work with, they would never accept an expanded latent space purely on the basis of saving memory. I understand that the paper is explicit about its goal of saving memory and targets this, which I am fine with on a technical level. I just wanted to note that this was the biggest weakness of the paper in my eyes, since as exciting a technique as it is, I likely won't use or advocate for it with partners because it doesn't solve a problem we actually have and introduces additional latent dimensions we don't want. This said, I don't fault or reduce the score of the paper for this, since I can appreciate there are instances or problems where memory constraints might be the driving factor (e.g., low dimensional attention, among others). Perhaps a thought experiment for future work by the authors might be to explore ways of getting both compact compression as well as superposition (although I realize these are fundamentally at odds, due to how the superposition works).

The other technical detail that was missing from the paper was a more robust literature review regarding other methods for doing randomized compression and recovery. The paper mentions, e.g., the Johnson-Lindenstrauss result for linear reductions, but there was historically (and still today) work on using randomized collisions to recover models or data for machine learning applications, primarily from the "sketching" community (e.g., countmin sketch, bloom filters, locality sensitive hashing, matrix sketching, etc.) as well as from the quantization community (e.g., quantized auto-encoders) which is another mechanism to reduce memory (by just storing a larger matrix in lower precision). Under what situations or conditions would I prefer the paper's proposed approach versus some of these alternatives? This is currently unclear to me from the manuscript.

To be honest, I consider neither of these to be super consequential. In general, I am quite supportive of the manuscript. If the authors could address the usage/literature points above, or if other reviewers argue convincingly that there is major impact/significance to be found in addressing the memory/storage aspects that I don't resonate with yet, I would be willing to increase my score from weak accept to clear accept.

Minor typographical details that could be fixed in a final camera ready version, but that I don't think are critical: (1) Fig. 4 was a bit small and hard to read, (2) Line 328 missing space between "Level,E_r"

Questions for authors

The paper introduces a number of new hyper-parameters and other settings that need to be tuned (e.g., the "short warmup period" before superposition). Could the paper provide some guidance on how these are to be chosen in practice or how critical the correct choice of these values are in reality? Having to do all of this additional tuning on a per problem basis is a non-trivial extra cost if one wishes to implement this method.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript Incorporating Rebuttal Results and Clarifications

Dear Reviewers and Area Chair,

As the discussion phase draws to a close, we have uploaded a revised PDF incorporating additional results and clarifications discussed during the rebuttal. Highlighting new large-scale and medical-image evaluations, comparisons with alternative autoencoder architectures, and expanded strong quantization method experiments, in addition to SLAE warm-up sensitivity and key-overhead analyses in the appendices. The main-text additions are highlighted in blue.

Thank you for your constructive feedback, which has helped us strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.