acceptodds
Desk rejected at ICLR 2027

WithinVAE: Compact Visual Autoencoding with Self-Distilled Semantics

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual autoencoders must balance reconstruction fidelity, compactness, and semantic structure. Existing semantic tokenizers typically inherit semantics from pretrained recognition encoders, whose invariances can discard the appearance detail required for inversion. We show that this inheritance is unnecessary: semantic structure can be learned inside an autoencoder's own reconstructive representation. We introduce WithinVAE, a compact variational visual autoencoder with a sequential bottleneck, DINO/iBOT-style self-distillation from an exponential-moving-average copy of the autoencoder itself, and position-dependent token masking to achieve a 4-fold compression ratio, which is free of pretrained representation encoder or external semantic target. On ImageNet-1K at 256×256, WithinVAE improves all four reported reconstruction metrics over RPiAE, including PSNR by 1.36 dB, while using half its latent scalar capacity. Under frozen evaluation, the same latents improve linear probing by 6.5 points over RPiAE and 5.1 points over our reconstruction-only baseline, attributing the semantic gains to self-distillation rather than architecture or external supervision. An auxiliary latent-to-latent model reproduces the ordered-compression trend, and a pixel-to-latent model transfers the objective to a pretrained generative space. These results separate the origin of semantic supervision from its use: a compact visual autoencoder can organize semantic structure without borrowing a recognition representation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Variational autoencoder, image reconstruction, semantic representation learning, self-distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pYocOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

WithinVAE combines a compact variational autoencoder with global DINO-style and local iBOT-style self-distillation from an EMA teacher. Initialized from One-D-Piece, it uses position-dependent token masking to support variable-length prefix decoding.

Critical strengths
  • The paper investigates: whether an autoencoder’s own EMA teacher can improve semantic organization without explicit alignment to a separate pretrained recognition encoder.
Critical weaknesses
  • The manuscript repeatedly refers to Appendices A/B and Algorithms 1–3, but the provided 13-page document ends with references.
  • The contribution beyond One-D-Piece is insufficiently isolated.
  • The claim of learning semantics without external supervision needs qualification.
  • The generation experiments do not establish the benefit of self-distillation.
Questions for authors
  • Can the semantic gains be demonstrated without this external perceptual signal, or can the claim be narrowed accordingly?
  • How much prefix reliability is inherited from One-D-Piece? Can matched prefix-decoding comparisons against no masking, uniform masking, and Tail Token Drop isolate the proposed schedule?
  • Does self-distillation improve downstream generation relative to the reconstruction-only and masking-only variants under identical generator settings?
  • Can the referenced appendices and exact evaluation configurations be provided?
Reviewer_DV38OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new VAE method which attempts to combine a compact latent space together with strong semantics. The paper attempts to do this via a self-distillation method on its own latent representation, rather than the use of pre-trained encoders (which most existing methods use). The method is trained with a combination of reconstruction, perceptual, adversarial, and the semantic/distillation losses. The method is compared against existing autoencoding methods on reconstruction quality, as well as on downstream DiT performance, and furthermore the authors perform an ablation and compare against RPiAE on semantic/linear probe performance.

Critical strengths
  1. The proposed approach does achieve reasonably strong reconstruction performance under a compact latent (comparable or smaller than every other method compared against), and is able to handle semantic types of tasks quite well, even with just linear probes on the latent space.
Critical weaknesses
  1. While capable of outperforming RPiAE (the most similar method to the one proposed in this paper), the reconstruction results actually are quite a bit worse than other existing models shown in Table 1 (although the bolded locations is confusing and hides the fact that WithinVAE is not actually the best on any metric). Therefore, it seems like this method actually leads to a significant drop in reconstruction performance (27.7 to 22.66/22.33 PSNR, or 0.097 to 0.159/0.163 LPIPS for example), in order to get its claimed improvements in other categories (the semantic properties). However, I don't think the results shown give a strong enough reason why we should accept much worse visual reconstruction quality in exchange for better semantic performance (RPiAE already has 98.3 View, for example, which this approach increases to 99.1).

  2. Additionally, the difference in training times between the approaches seem significantly unbalanced. As far as I can tell, the current approach initializes itself using the One-D-Piece model, which has already been trained for 300 epochs, and then trains an additional 10 epochs. In comparison, as far as I can tell, the RPiAE model was trained for a total of 64 epochs.

Questions for authors
  1. Under more equal training times (for example, not initializing from the One-D-Piece model weights), how well does the proposed approach actually perform? Additionally, how can we reasonably say that this approach has "self-distilled semantics" if it initializes its weights from One-D-Piece (which itself is trained to align with the MaskGiT tokenizer)? There is no good disentanglement of where the semantics really all come from?
  2. It is my understanding that the "Global DINO" and "Local iBOT" found in Figure 2 refer to the type of loss, and not actually using those models directly. Is this true? If so, it might be good to make it more clear, as during my first reading of the paper, I thought those models were actually used.
Reviewer_sBejOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes WithinVAE, a compact image autoencoder which learns semantic structure inside its own reconstruction space, without borrowing a pretrained recoginition model. It starts from the One-D-Piece architecture and two ideas are added during training, position-dependent token masking and self-distillation with an EMA copy as the teacher. On ImageNet dataset, WithinVAE beats RPiAE on all reconstruction metrics. The authors conclude that semantic gains mainly come from self-distillation.

Critical strengths
  1. The mechanism is well designed and well explained. Position-dependent masking affects which tokens are reliable for reconstruction. Global self-distillation keeps the pooled representation consistent across views. It is not a random pile of tricks.
  2. The comparison is clean. WithinVAE and the reconstruction-only baseline share the same initialization, architecture and reconstruction loss supervision. The only difference is the self-distillation loss. So the gains can be fairly credited to self-distillation.
  3. There are useful practical results reported in this paper. The trained model decodes prefixes of different lengths without re-training, which is useful for generation models. The compression result also highlights its practical value with half the latent capacity and better reconstruction.
Critical weaknesses
  1. The initialization may carry external semantics. I read the One-D-Piece paper and found its first stage trains the encoder to match a pretrained tokenizer's logits. Therefore, the starting weights are very likely to contain semantic structure from an external model. This means the performance gains of self-distillation may result from the reservation of already learned structure, or by making the exisiting semantics easier to read linearly, instead of forming a new structure. A random initialization experiment may help.
  2. The claim "without external supervision" is too broad. In fact, this paper uses perceptual loss, which usually uses a pretrained VGGNet. This is also external supervision with recognition pretraining. And as mentioned above, the One-D-Piece initialization may carry a structure from a pretrained tokenizer. I also doubt that prefix decoding may partly come from One-D-Piece's Tail token drop training, not only from position masking.
Questions for authors

My main concern is the model initialization from One-D-Piece. Self-distillation has been proven effective in this paper, but the strong claim that semantics are organized inside the reconstruction space itself is not fully verified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.