acceptodds
Under review as a conference paper at ICLR 2027

Reconstruction is not Perception: A Simple Proof of Necessity but Insufficiency

Abstract

Masked autoencoders (MAE) provide a scalable approach to visual representation learning, yet strong classification performance often requires fine-tuning, richer probes, or longer pretraining. We ask whether this mismatch reflects downstream underdetermination: models with the same reconstruction loss can nevertheless have substantially different downstream utility. We study this question through reconstruction-loss level sets in the joint encoder–decoder parameter space. Controlled sweeps of ViT-B/16 MAEs on ImageNet-100 reveal substantial variation in linear-probe accuracy among checkpoints at matched reconstruction loss, with the variation generally increasing as reconstruction loss decreases. We derive a first-order local characterization of this phenomenon and adapt a classification-guided predictor–corrector traversal that improves or degrades downstream performance while keeping reconstruction loss nearly fixed. Across nine decoder–loss settings, these walks span – percentage points of accuracy, – the matched-loss pretraining spread, and extend beyond the pretraining accuracy range in every setting. The phenomenon also transfers to public ImageNet-1k checkpoints, where walks improve ViT-B/16 and ViT-L/16 by and percentage points at nearly unchanged reconstruction loss. Mechanistically, reconstruction and classification gradients remain nearly orthogonal throughout pretraining, while walk-induced spectral changes concentrate in a late-learned tail containing only – of target variance. These results show that reconstruction loss does not determine downstream representation quality and identify directions that reconstruction weakly constrains yet matter substantially for classification.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self supervised learning, masked autoencoder, reconstruction, classification, loss landscape

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iDfLOpenReview ↗
Rating
2
Confidence
2(−2)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the mismatch between reconstruction loss and downstream linear-probe performance in masked autoencoders. It studies downstream underdetermination: models with nearly identical reconstruction loss can exhibit different classification performance. The authors provide an theoretical analysis and use classification-guided level-set walks to improve or degrade linear-probe accuracy while keeping reconstruction loss nearly fixed. Controlled sweeps on ImageNet-100 and experiments on public ImageNet-1k checkpoints provide empirical evidence for the proof.

Critical strengths
  • The paper is well written and easy to follow.
  • Provide a math proof that reconstruction loss does not showing representation quality.
  • The level-set walk experiment demonstrate the a huge space that can be optimized.
Critical weaknesses
  • While this paper deepens our understanding of a recognized limitation in reconstruction-based learning methods, its practical guidance for improving self-supervised pre-training remains limited. Although classification-guided random walks offer a useful diagnostic tool, they rely on downstream labels and thus cannot directly provide solutions for purely self-supervised scenarios. Investigating whether gradients from other self-supervised objectives, such as contrastive learning or self-distillation, can also yield effective directions orthogonal to the reconstruction loss level sets would much more enhance the paper's practical insights. Could such directions improve downstream representations while maintaining reconstruction performance, without requiring classification labels for updates?

  • Spectral analysis does not fully disentangle the contributions of the encoder and the decoder. Specifically, the PCA basis is constructed from target image patches and applied to the decoder's reconstruction output, whereas classification performance is evaluated using encoder features. Since level-set random walks update both components simultaneously, changes in low-variance output directions do not in themselves reveal how class-relevant information within the encoder changes, nor do they clarify the extent to which the observed spectral effects reflect decoder adaptation. Employing a fixed-decoder setup and conducting separate analyses of encoder representations and decoder outputs would help elucidate this relationship and strengthen the proposed mechanistic explanation.

Questions for authors

See the weakness part.

Reviewer_37zsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper argues that MAE reconstruction loss underdetermines linear probe acc. It shows, large accuracy spread among ImageNet-100 ViT-B/16 checkpoints at matched reconstruction loss, a supervised predictor-corrector level-set walk that moves accuracy by 24–50 pp while holding reconstruction loss within 1.5e-3, and that reconstruction/classification gradients are nearly orthogonal and the walks only touch a tiny-variance spectral tail of the reconstructed patches.

Critical strengths
  • Matched loss spread across 78 pretraining runs is a clean, convincing demonstration that recon loss is not a usable proxy for prope quality. Also the gradient cosine tracking across training is a nice supporting fact
  • The walk machinery is carefullly engineered and transfers to public IN-1k checkpoints.
Critical weaknesses
  • The level set wlak uses labels, so finding directions that change probe acc at fixed recon loss is nearly guaranteed once the gradients are orthogonal. It is constrained supervised fine tuning and says nothing about what pretraining misses; the “+4.1 pp on ViT-B” is not a representation-learning gain.
  • No proof is delivered: Prop. 1 is Ky Fan on a linear AE, Thm. 1 is the chain rule. The title overclaims for what is an empirical study.
  • The spread comes from extreme weight decays (0 vs 0.6) at a single seed; seed variance is never reported, and reconstruction loss is only held fixed on one fixed mask set, so mask overfitting is unchecked.
  • (minor) close-form -> closed-form, and Fig 1 panel labels don't match the caption.
Questions for authors
  • What does the walk show beyond “supervised gradient ⊥\perp reconstruction gradient”? Is there any unsupervised signal that selects the good direction?
  • Does reconstruction loss stay fixed under fresh random masks (and on a held-out test set) along the walk?
  • What is the matched-loss accuracy spread across seeds at a single hyperparameter setting?
  • I also recommend to drop “proof” from the title or state precisely what is proved.
Reviewer_RRdwOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether reconstruction loss in masked autoencoders reflects downstream representation quality. The authors study classification performance at a fixed reconstruction loss. They provide a theoretical analysis of underdetermination and propose a classification-guided method to explore representations with different linear probe accuracies while preserving the reconstruction loss.

Critical strengths
  • The paper extends prior findings on reconstruction-based representation learning to MAEs.
  • The level-set walk provides an approach for exploring representations with different classification performance while approximately preserving the reconstruction loss.
Critical weaknesses
  • The spectral analysis focuses on reconstructed patches from the decoder, while classification relies on encoder representations. Changes in reconstructed patches do not necessarily reflect changes in encoder features relevant to classification.
  • Including decoder parameters may lower the gradient cosine similarity. Does the near orthogonality still hold for encoder parameters alone?
  • Theorem 1 assumes a locally optimal decoder, while the level set walk jointly updates the encoder and decoder. How does the theoretical analysis account for this difference?
Questions for authors
  • Can the proposed level-set walk be adapted to other objectives?
  • How sensitive are the observed accuracy differences to the reconstruction loss tolerance? Do they remain substantial under tighter constraints?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.