acceptodds
Under review as a conference paper at ICLR 2027

How Representations Shape Label Requirements

Abstract

Pretraining can reduce the need for labels, but what makes a representation label-efficient? Labels reveal both what to predict and which inputs should share a prediction. We isolate these roles in a regression family whose candidate groupings have equal response complexity. Choosing a grouping before labels arrive can make a target risk unattainable, regardless of label count. A frozen representation retaining both candidates allows a supervised readout to overcome this structural barrier, at a higher estimation cost. We give upper and lower bounds on optimal risk within the committed class that match within a factor of three, and computable label budgets for known Gaussian tasks. Experiments on electrocardiograms, ImageNet, and robot actions vary input coverage and encoder update permissions. Under fixed recipes and on the measured grids, adaptation reaches specified mean-risk targets with one-tenth as many labels on ImageNet and robot actions.

Primary area:
learning theory
Keywords:
Representation Learning, Label Complexity, Sample Complexity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pyi5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that label efficiency depends not simply on the number of labels, but on the extent to which the representation leaves structural choices open to supervision. If a model is locked into an incorrect structure before seeing labels, an “error floor” may arise that cannot be overcome no matter how many labels are added; however, if multiple candidate structures are preserved, even a frozen representation can overcome this. Experiments also showed that wider pretraining inputs and encoder adaptation improved performance; on ImageNet and robotics tasks, the Adapt method reached the same target risk with about 10 times fewer labels than the Frozen method.

Critical strengths
  1. A key strength of this paper is that it redefines label efficiency not as a simple sample complexity problem, but as a question of what structural choices the representation leaves open for supervision.
  2. The paper theoretically demonstrates that early commitment can create a risk floor and verifies, through practical experiments, that candidate retention and adaptation can mitigate this issue.
  3. The paper validates this central issue across different tasks—ECG, ImageNet, and robot manipulation—and enhances the persuasiveness of the experimental design by controlling for input coverage and encoder update permissions.
Critical weaknesses
  1. Since the main comparison baselines are skewed toward internal ablation studies focused on Scratch, Frozen, and Adapt, there is a lack of direct comparison with strong external methods. Because FixMatch is only applied under certain conditions on ImageNet, and the paper does not directly compare label efficiency with existing representation methods such as R3M and VIP on Robot, it is unclear how meaningful the reported 10× label reduction is relative to actual competing methods (e.g., SOTA methods).
  2. The scope of the theoretical analysis is limited. The key guarantees are based on two candidate structures and a known Gaussian design; extending these to situations where the candidate structures themselves must be learned—as in real-world deep learning—remains an open problem. Therefore, further verification is needed to determine to what extent the structural commitment results from the toy model generalize to complex representation learning in general.
  3. The label savings observed in the experiments depend on specific training recipes and a discrete label-budget grid. The authors themselves report that the risk varies depending on the backbone, objective, and update depth, and acknowledge that the savings ratio (such as 10×) could change if measurements near the target crossing were made more densely. Therefore, it is difficult to interpret the quantitative effects as a universal scaling law.
Questions for authors

Please refer to the weaknesses.

Reviewer_8m1LOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors study what makes a pretrained representation label-efficient. It separates two roles that labels play: estimating response values, and revealing which inputs should share a prediction. The key contributions are:

  • Committing to a grouping before seeing labels can make a target risk unreachable. In a row/column regression where the response is shared either along rows or along columns, a learner that commits to one grouping before seeing labels ("RF") has a risk floor that holds regardless of label count (0.375 in the running example). A frozen representation that keeps both groupings (2r−12r-1 dimensions) can get below this floor using labels alone.
  • The best achievable risk of such a committed learner is pinned down within a factor of three. Theorem 1 gives matching upper and lower bounds, and Eqs. 4-5 turn them into input-count and query requirements.
  • For a known representation and a Gaussian task, the required label fraction has a computable bound. It satisfies sε≤ρ∗≤(1+κ)sε/(1+κsε)s_\varepsilon \le \rho^* \le (1+\kappa)s_\varepsilon/(1+\kappa s_\varepsilon) (Eq. 8).
  • On real data, adapting the encoder reaches the same risk with fewer labels than freezing it. Scratch, Frozen and Adapt models are compared on ECG, ImageNet and ManiSkill3 robot actions. The results show that more pretraining inputs help (12/12 contrasts), that adaptation reaches the risk targets with about 10x fewer labels on ImageNet and robots (Table 1), and that an additive rule forecasts risk at a held-out budget.
Critical strengths

Quality

The experiment design tries to control for getting reasonable performance estimates and typical confounders

  • 52 paired seed groups and nested label masks;
  • pretraining presentations held fixed when the input pool varies;
  • closed-loop robot success reported next to offline risk (Fig. 5)

Originality

The authors argue for separating two distinct roles that labels play. Framing label efficiency as "which structural choices remain open to supervision" is an interesting angle. The toy model makes the difference between estimating responses and identifying structure concrete.

Critical weaknesses

Some key technical limitations and suggested improvements:

  1. The theory and the experiments feel disconnected. The finite-model result follows fairly directly from the setup. With no evidence for either grouping, an early commitment is wrong half the time and loses the excluded contrasts; this is the c/2 floor. Retaining both groupings is the standard bias-variance tradeoff of model selection. More importantly, per my understanding none of the neural experiments contains an RF-vs-retain contrast. As far as I can tell, the RF/Joint arms appear only in the simulations (App. E); please let me know if I'm missing something. The neural findings (more pretraining data helps, fine-tuning beats frozen, unfreezing more layers helps) agree with established transfer-learning results. Also, the theory explains why frozen features that retain candidates can work, but the headline empirical result is that adaptation wins. Couple of suggestions:

    • Add at least one neural experiment that mirrors the toy model, for example a pretraining setup where a frozen encoder either keeps or discards a candidate grouping.
    • Show the predicted floor, and how many labels it takes to get past it, in that setting.
  2. The "10x fewer labels" result depends on a coarse label grid, so the true ratio is uncertain. ImageNet uses {1, 3, 10, 30, 100}% labels and robots use {100, 300, 1k, 3k, 10k, 100k} actions. So 30% to 3% and 10k to 1k are two grid steps apart, and the true ratio could be anywhere from roughly 3x to 30x. The adjusted-interval ImageNet row (100% to 3%) shows how much the frontier moves. Suggestions:

    • Use a denser grid near the crossings, or interpolate the risk curves.
    • Consider moving the caveat from Sec. 5 into the abstract.
  3. The theory sections are hard to follow without the appendix. Secs. 3.2-3.3 introduce R_or, p_err, T*_RF,query, Gamma_m and the A-optimal design bound within about two pages, with little intuition for any of them. Suggestions:

    • Add a worked example in the main text.
    • Move secondary quantities to the appendix.
Questions for authors
  1. Does any neural experiment compare a representation that commits to one grouping before labels with one that keeps both? As far as I can tell, this comparison appears only in the finite simulations. If no such experiment exists, which neural result would have come out differently if the theory were false?

  2. How should a practitioner use Theorem 1 or Eq. (4) to choose between more pretraining data, more labels, or encoder adaptation? Beyond the point that a wrong early commitment loses c with probability p_err(B), I could not see how the bounds translate into a concrete decision.

  3. What is the Frozen-to-Adapt label ratio when the label grid is denser near the target crossings? The current grid only pins the ratio down to somewhere between roughly 3x and 30x on ImageNet and robot actions.

Reviewer_9xRxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how the representations learned by a shared encoder via self-supervised learning (without labels) affect the amount of labeled data required for a given performance in downstream tasks under different finetuning settings: training from scratch, freezing the encoder, or finetuning the encoder. The authors first consider a synthetic problem with two possible structures. They show that committing to one structure before observing labels performs poorly compared with keeping both structures in a shared representation. They then experiment on ECG, ImageNet and robotic datasets, comparing three approaches: 1) a model trained from scratch using labels; 2) a pretrained model for which the encoder is kept frozen in the presence of labels; and 3) a pretrained model for which the encoder is finetuned with the provided labels. They show that finetuning the pretrained encoder (specially given a low amount of labeled data) and increasing the diversity of unlabeled pretraining data generally improve performance.

Critical strengths

The paper contains extensive theoretical and empirical evaluations. The authors derive bounds on label requirements for Gaussian regression problems. They also consider a synthetic problem that illustrates how committing to a representation before observing labels can hinder downstream performance.

The empirical evaluation covers three different domains (ECG, ImageNet and robotic manipulation) and analyses both the effects of changing the amount of unlabeled pretraining data as well as the amount of labeled downstream data. They also consider different training strategies, including training from scratch, freezing the pretrained encoder, and finetuning it.

The figures are generally clear and informative.

Critical weaknesses

I have two main concerns with the paper.

The first relates to paper's presentation. The paper lacks clarity and is extremely hard to follow. The introduction, related works and method sections are particularly hard to understand. For example, from the introduction and related works sections, it is not clear to me exactly what the paper proposes to do and how this differs from existing literature. In general, most sentences are overly complex and abstract, making them extremely hard to understand even after re-reading them several times.

The second regards the contributions and novelty. As I already briefly mentioned, the main contributions of the paper are not clearly articulated and it is difficult to understand how the paper relates to existing literature. Moreover, the experimental results demonstrate that allowing a pretrained encoder to be finetuned on downstream tasks leads to better performance than keeping it frozen. From my understanding, this is neither novel nor surprising. In several areas of machine learning (e.g. computer vision, LLMs), it is well known that allowing (potentially only parts of) the pretrained encoder to be finetuned in the presence of labeled data can boost performance in downstream tasks. Therefore, it is not clear to me exactly how this paper improves over existing research.

Questions for authors
  1. What are the main contributions of this work and how do they relate to existing research on representation learning and/or finetuning (including insights from computer vision, LLMs, etc.)?

  2. What is the novelty of the work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.