- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates the idea of approximating epiplexity, building on ideas introduced in prior work on learnable novelty. The idea is to approximate the complexity of a computationally bounded learner (observer) using a fixed randomly initialized model with a linear readout. The work shows that the spectral objective prefers spreading the singular values of the readout across many directions. The paper argues that this does not necessarily lead to representations that are useful for downstream tasks. To address this, they (1) construct eight masked views of an image together with relation and interaction tokens, which are intended to capture relationships between the different parts; and (2) they introduce an objective that measures how much of the representation covariance can be explained by the observer, using the log ratio of covariance determinants before and after prediction -- intuitively measuring explained covariance of the representations as opposed to spectral complexity of the weights as done originally.
In the experiments, they show that using the original spectral objective introduced in prior work on learnable novelty, but with an observer receiving masked views, leads to better downstream performance on ImageNet compared to the whole-image baseline. However, their own proposed compositional objective does not improve upon this. They conclude that spectral spreading, observer predictability, and downstream usefulness are distinct properties.
Critical strengths
- The direction of extracting structure from data is interesting, applying notions of epiplexity and forms of its approximation is underexplored.
- The scope of the work is good.
Critical weaknesses
The biggest weakness is that the study provides neither convincing new insights into approximating epiplexity nor practically useful insights into how it could lead to practical gains.
The paper's conclusion reads, "when the spectral approximation to epiplexity is used to train a representation, what is it actually rewarding?. In theory, the answer is spread." But I don't see how this is established beyond the fixed-trace setting. The authors then write, "At fixed readout energy, the score prefers to distribute spectral mass evenly across as many directions as it can." However, this constraint is not enforced during training. While the authors empirically observe spectral spreading, their theoretical result concerns the readout weights, whereas the experiments measure the covariance of the learned representations. It's not clear how the former explains the latter.
Additionally, the authors write that "switching the spectral observer from the whole image to masked views more than doubled linear-probe top-1 accuracy." But (1) you can switch many more things to get better accuracy (just adopt inductive biases studied in self-supervised works), and (2) increasing accuracy x2 (from 3% to 7%) doesn't establish where the gains came from. The comparison also changes the preprocessing and compute.
Lastly, the authors write, "recognizing the leaves is not the same as knowing how they connect" and propose "building objectives that reward the connections themselves." But I don't see how this follows from their results. The paper shows that using masked views (with more compute and other changes) leads to better downstream performance. That's all. It does not show that the proposed objective learns relationships between parts.
More concretely:
- Their proposed method does not improve downstream performance. The biggest gains come from using eight masked views with the original objective. Their proposed compositional objective performs worse on almost all tasks.
- The motivation for multiple masked views comes out of nowhere. It's a particular choice of the frozen observer. The masked views go through the observer, and the expectation is that its outputs can predict the representations of the trainable encoder. This introduces a specific inductive bias whose justification does not follow from epiplexity. Moreover, this setup uses more compute, and the comparison also changes preprocessing and score calibration.
- The motivation for the new objective is unclear. Why should maximizing the amount of representation covariance explained by the observer lead to compositionality? It's not clear how this captures relationships between image parts. The experiments also do not demonstrate that it does.
Questions for authors
See weaknesses.