- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper argues that label efficiency depends not simply on the number of labels, but on the extent to which the representation leaves structural choices open to supervision. If a model is locked into an incorrect structure before seeing labels, an “error floor” may arise that cannot be overcome no matter how many labels are added; however, if multiple candidate structures are preserved, even a frozen representation can overcome this. Experiments also showed that wider pretraining inputs and encoder adaptation improved performance; on ImageNet and robotics tasks, the Adapt method reached the same target risk with about 10 times fewer labels than the Frozen method.
Critical strengths
- A key strength of this paper is that it redefines label efficiency not as a simple sample complexity problem, but as a question of what structural choices the representation leaves open for supervision.
- The paper theoretically demonstrates that early commitment can create a risk floor and verifies, through practical experiments, that candidate retention and adaptation can mitigate this issue.
- The paper validates this central issue across different tasks—ECG, ImageNet, and robot manipulation—and enhances the persuasiveness of the experimental design by controlling for input coverage and encoder update permissions.
Critical weaknesses
- Since the main comparison baselines are skewed toward internal ablation studies focused on Scratch, Frozen, and Adapt, there is a lack of direct comparison with strong external methods. Because FixMatch is only applied under certain conditions on ImageNet, and the paper does not directly compare label efficiency with existing representation methods such as R3M and VIP on Robot, it is unclear how meaningful the reported 10× label reduction is relative to actual competing methods (e.g., SOTA methods).
- The scope of the theoretical analysis is limited. The key guarantees are based on two candidate structures and a known Gaussian design; extending these to situations where the candidate structures themselves must be learned—as in real-world deep learning—remains an open problem. Therefore, further verification is needed to determine to what extent the structural commitment results from the toy model generalize to complex representation learning in general.
- The label savings observed in the experiments depend on specific training recipes and a discrete label-budget grid. The authors themselves report that the risk varies depending on the backbone, objective, and update depth, and acknowledge that the savings ratio (such as 10×) could change if measurements near the target crossing were made more densely. Therefore, it is difficult to interpret the quantitative effects as a universal scaling law.
Questions for authors
Please refer to the weaknesses.