- Rating
- 2
- Confidence
- 2(−2)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the mismatch between reconstruction loss and downstream linear-probe performance in masked autoencoders. It studies downstream underdetermination: models with nearly identical reconstruction loss can exhibit different classification performance. The authors provide an theoretical analysis and use classification-guided level-set walks to improve or degrade linear-probe accuracy while keeping reconstruction loss nearly fixed. Controlled sweeps on ImageNet-100 and experiments on public ImageNet-1k checkpoints provide empirical evidence for the proof.
Critical strengths
- The paper is well written and easy to follow.
- Provide a math proof that reconstruction loss does not showing representation quality.
- The level-set walk experiment demonstrate the a huge space that can be optimized.
Critical weaknesses
-
While this paper deepens our understanding of a recognized limitation in reconstruction-based learning methods, its practical guidance for improving self-supervised pre-training remains limited. Although classification-guided random walks offer a useful diagnostic tool, they rely on downstream labels and thus cannot directly provide solutions for purely self-supervised scenarios. Investigating whether gradients from other self-supervised objectives, such as contrastive learning or self-distillation, can also yield effective directions orthogonal to the reconstruction loss level sets would much more enhance the paper's practical insights. Could such directions improve downstream representations while maintaining reconstruction performance, without requiring classification labels for updates?
-
Spectral analysis does not fully disentangle the contributions of the encoder and the decoder. Specifically, the PCA basis is constructed from target image patches and applied to the decoder's reconstruction output, whereas classification performance is evaluated using encoder features. Since level-set random walks update both components simultaneously, changes in low-variance output directions do not in themselves reveal how class-relevant information within the encoder changes, nor do they clarify the extent to which the observed spectral effects reflect decoder adaptation. Employing a fixed-decoder setup and conducting separate analyses of encoder representations and decoder outputs would help elucidate this relationship and strengthen the proposed mechanistic explanation.
Questions for authors
See the weakness part.