- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper tackles the problem of joining the features gained by reconstruction and prediction, which offer qualitatively different representations. The solution is to supplement the training objective with an image-adaptive noise and a reliability gate. The former is to address the shortcomings of a uniform noise used in other research, while the latter is to control the degree of prediction loss, acting as a curriculum. This proposed change results in consistent improvement of image-based prediction task performances over several related baselines.
Critical strengths
- I think this paper identifies one of the fundamental questions that representation learning tries to answer: Is a representation good for reconstruction also good for prediction? It also goes on to clarify how existing approaches are not appropriate theoretically and explains why the proposed method is (somewhat) justified by the theoretical analysis as well. Furthermore, the experiments back up the claims by showing consistent increase in performance.
Critical weaknesses
- (Not really critical) There are some minor notational issues in both propositions that make it hard for the reader to appreciate their importance. See questions below.
Questions for authors
- Is there a reason why the authors focus on vision encoder instead of a general-purpose encoder?
- Could the authors explain the following notations and concepts used in the propositions?
- W(0): W is a map, but it's treated as a matrix.
- What does do?
- How can the significance of these propositions extend beyond the linear setting?
- The paper talks how an image-dependent noise is necessary to achieve the intended results, but will this be related to general augmentation techniques? Because many augmentation techniques can be thought of as being input-dependent perturbation, and it might bear some relevance to the main discussion.
Response to Reviewer 9GE5
Numbering follows the review.
W1 / Q2. Notation in both propositions. Fixed: we have revised Sec 3.2 so that every symbol is defined where it first appears.
What each proposition says, in words. Proposition 1: with a noise level that ignores the image, directions of larger variance are learned faster and more completely, whatever the level. Noise of this kind cannot change what reconstruction prefers. Proposition 2: when the level depends on the image, reconstruction no longer sees Σ but a covariance in which each image counts according to its signal rate, and that covariance can have a different order of directions. We have added these two sentences after the propositions.
Q1. Why a vision encoder and not a general-purpose encoder? Because our evidence is visual; the mechanism is not specific to vision. IAN needs only a token sequence: the saliency of Eq. (5) is computed from the affinity of token embeddings (L253–255). Our experiments cover ViT-B/L on ImageNet-1K and one vision-language model (CLIP, 83.35 → 85.28; App. B, L866–871), so we limit the claims to vision, as the abstract states (L030–031).
Q3. How can the significance of the propositions extend beyond the linear setting? As a prediction that we test in a ViT-B. The prediction: a noise level that ignores the image leaves the preference of reconstruction unchanged; only a level that depends on the image can change it.
The third row is the most direct answer: same noise levels, same distribution, only their link to the image removed. This control was requested by Reviewer m4GD; the full table is in our reply to that review. A derivation for nonlinear encoders remains open (Sec 5, L471–473).
Q4. Relation to general augmentation techniques, many of which are input-dependent perturbations. Yes, it is related, and the propositions add one distinction. The reviewer is right that the output of any augmentation depends on the input. What matters in Proposition 2 is whether its strength is chosen from the input.