- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper argues that two models with the same validation score may not be equally useful downstream, especially when a downstream model uses only one part of the representation rather than the whole backbone. A simple construction (Proposition 1) shows that swapping coordinates can keep validation and transfer accuracy unchanged while changing the usefulness of a fixed coordinate. To study this, the authors train a ViT from scratch (FACTOR-MAE) with a reconstruction branch and an understanding branch, and split its features into shared and unique codes. They then propose UCRS, a two-step rule for choosing regularization. First, the weight-decay strength is picked by a short kNN run. Then MI regularization is added only if backbone transfer improves and the unique code does not get worse.
Critical strengths
- Weight decay is compared against a calibrated same-strength reference, a control most regularization papers omit.
- Negative results are reported candidly, including the never-binding constraint, suite reversals and a false rejection.
Critical weaknesses
- Gains are tiny (about 0.12 points at 75% accuracy), and UCRS underperforms default weight decay.
- The constraint rarely acts: it never changes a decision in six settings and only three of nine.
- Reported data are hard to trust. For ViT-S/IN-100, Table 1/A1 select , while Table A5 selects , and the same transfer value 45.70 is attributed to both. For ViT-B/IN-100, the shared-kNN screening scores are 62–65 in Table A5 but 16.84–17.64 in App. C. In Table 4, all four directional contrasts have the same SD (0.120), and two different UCRS-vs-unique-only quantities share identical statistics (+0.022, 0.030, [.008, .037]).
- The paper is very hard to read.
Questions for authors
- Please explain the data problems listed in Weakness 3. Why does the selected for ViT-S/IN-100 differ between Table 1/A1 and Table A5? Why are the ViT-B/IN-100 screening scores so different between Table A5 and App. C? And why are the statistics in Table 4 so regular? Also, are the models in the nine-condition study the same ones reported in Table 1?
- Could you release the raw per-block results behind Tables 4 and A11?
- How much does the distilled ViT-Ti vary across random seeds? Is a 0.12-point difference outside that range?