acceptodds
Withdrawn from ICLR 2027

Frozen Foundation Models and Noisy Labels: Accuracy Headroom and Calibration Gaps

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Foundation-model encoders change the starting point for learning from noisy downstream labels: a visual representation is available before the corrupted supervision is observed. We ask how much accuracy headroom remains for noise-specific training when the encoder is immutable, and whether accurate readouts also yield reliable probabilities. Linear probes serve as matched diagnostics over the same frozen representation, not as substitutes for representation learning. Across the evaluated benchmarks, cross-entropy probing often approaches a matched clean-label readout and specialized noisy-label learning rules under moderate noise, while severe or structured corruption leaves meaningful accuracy headroom. Absolute accuracy also varies across encoders, so this finding is conditional on the tested representations. Yet similar top-1 accuracy can coexist with markedly different calibration errors. A population analysis separates changes in Bayes decisions from changes in posterior probabilities. We then evaluate Anchor-based Efficient Calibration (AEC), a post-hoc temperature-matching intervention that uses training logits rather than clean calibration labels. AEC often improves ECE and representative proper scores without changing predictions, but its fixed target can also worsen calibration. These results position decision quality and probability reliability as distinct objectives in noisy-label learning with frozen foundation models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Learning with Noisy Labels, Frozen Foundation Models, Probability Calibration, Linear Probing, Temperature Scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EyVDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies learning with noisy labels using frozen base model representations. By comparing standard cross-entropy loss and noise-sensitive linear outputs against clean label references, the paper show limited accuracy gains under moderate noise conditions, but significant improvements under more severe interference. The paper also examines probabilistic reliability separately and evaluates the AEC method. Experiments on both synthetic and real-world noisy label benchmarks show that high accuracy can coexist with poor calibration, while AEC generally improves calibration performance, though it may also lead to a certain degree of performance degradation.

Critical strengths

The problem setup is clear and well-defined, and the use of fixed encoder and readout family makes the comparison results easier to interpret. The matched clean-label reference is useful, and the paper appropriately avoids treating it as a mathematical upper bound. I also appreciate the cautious conclusions in the paper: it reports cases where robust training still leads to significant accuracy gains, while also pointing out scenarios in which AEC may degrade calibration performance.

Critical weaknesses

AEC selects the top 10% of examples with the highest confidence in each predicted class and adjusts their average confidence to 0.995. However, the main paper does not adequately justify why this target is appropriate across different predictors and data corruption settings. The authors acknowledge this limitation, and Figure 1 shows that calibration performance may worsen. A clearer analysis is needed regarding the rationale behind the target setting, the accuracy of anchor predictions, and observed failure cases, to clarify under what conditions AEC should be effective. The direct comparison of calibration methods focuses on a single dataset, which is insufficient to determine AEC's consistent advantage over other calibration strategies. It is particularly important to distinguish whether the gains come from high-confidence anchors, class-conditioned temperature scaling, or more general probability scaling. The NLL and Brier results only show two CE settings, while broader conclusions are primarily based on ECE. Since AEC may increase confidence in incorrect predictions, improvements in ECE alone are insufficient to fully assess probability quality. It is recommended that the main experimental discussion explicitly clarify whether changes in ECE align with NLL/Brier under different training rules and reported failure cases.

Questions for authors

How were the target confidence of 0.995 and the anchor fraction of 0.1 chosen? What evidence connects the target to anchor prediction accuracy, and how sensitive are the conclusions to these choices? Do AEC’s ECE improvements also correspond to improvements in NLL and Brier across GCE, SSR, and DivideMix, including the failure cases? How does its comparison with noisy-validation temperature scaling change in harder settings?

Reviewer_j3q4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies learning with noisy labels when the image encoder is a frozen foundation model and only a linear readout is trained. It asks how much accuracy is left for noise-aware training, and whether accurate readouts also give reliable probabilities. Cross-entropy (CE), GCE, SSR and DivideMix readouts over frozen DINOv2 ViT-B/14 features are compared against a matched clean-label probe trained with the same protocol. CLIP and an ImageNet ViT appear in one comparison. Benchmarks are CIFAR-10 with synthetic noise, CIFAR-10N, CIFAR-100N, Animal-10N and Clothing1M, and no clean labels are used for tuning.

A short population analysis splits clean Bayes regret into a decision shift and a learner gap (Eq. 4), and bounds calibration error by the clean–noisy posterior gap (Prop. 1). The paper then evaluates AEC. For each predicted class, AEC takes the 10% most confident training points and fits a temperature so that their mean confidence is 0.995, which leaves predictions unchanged.

The main findings are that CE is close to the clean probe under moderate noise (0.23 points at 50% symmetric noise), while severe or structured noise leaves several points of headroom. The same CE probe has ECE 51.59 against 8.34 for the clean probe. AEC lowers ECE in 38 of 40 rule-by-dataset cells.

Critical strengths
  1. The question is clear and practical, and the paper is careful about what it does not claim. For example, it does not treat the clean probe as an upper bound or AEC as recovering clean posteriors.
  2. The matched clean probe is the right kind of control. Only the labels change, so the effect of the training rule is separated from the representation.
  3. Accuracy and calibration are reported separately for four rules over ten noise settings, and the cases where AEC makes ECE worse are shown and discussed.
  4. AEC is cheap, needs no clean labels and never changes a prediction. The main numbers are internally consistent: every value quoted in the text and all 20 cells of Fig. 1 match the appendix tables.
Critical weaknesses
  1. The shared probe recipe appears under-trained, which would affect both headline findings. Even the clean probes are strongly underconfident. ECE minus error rate, a lower bound on underconfidence, is 6.1 points for DINOv2, 3.1 for ViT and 13.7 for CLIP on CIFAR-10, and 4.2 for DINOv2 on CIFAR-100. The noisy CLIP and ViT probes beat their clean counterparts (90.48 against 89.53 for CLIP). The clean accuracies are also below published linear probes for the same encoders. DINOv2 ViT-B/14 reaches 98.7 on CIFAR-10 and 91.3 on CIFAR-100 (Oquab et al., 2023, Table 8), against 97.74 and 87.45 here. CLIP ViT-B/16 reaches 96.2 on CIFAR-10 (Radford et al., 2021, Table 10), against 89.53 here. Weight decay, feature normalization and convergence are not reported. If the recipe is under-trained, headroom is measured against a weak reference, and part of each "calibration gap" comes from the recipe rather than the noise. AEC was not applied to the clean probe.

  2. The closest prior work is not cited. Zhu, Zhang, Gangrade and Scott (NeurIPS 2024), "Label Noise: Ignorance Is Bliss" (arXiv 2411.00079), already proposes the same recipe: frozen DINOv2 features and a linear head trained with plain ERM on noisy labels. It reports the same finding, that accuracy stays flat as noise grows and its level is set by feature quality, across several extractors. It also compares robust losses on DINOv2 features against CE. Corollary 1 here is essentially their Theorem 4, which requires every row of the transition matrix to be diagonally dominant and also proves the converse. Their relative-signal-strength bound is consistent with where this paper finds headroom. The finite-sample term scales with the inverse of the signal strength, which for one-hot clean posteriors is 0.44 at 50% and 0.11 at 80% symmetric noise. The part of this paper that goes beyond Zhu et al. is the calibration study, since their immunity result concerns only the argmax and they do not study calibration. The paper would be stronger built around that result (decisions survive the noise, probabilities do not), with Zhu et al.'s public readout protocol as the CE baseline. Other work further limits the novelty of the accuracy half. This includes robust linear and nearest-neighbour readouts over frozen foundation-model embeddings (Di Salvo et al.) and work on why self-supervised representations help under label noise (Xu et al.; C2D).

  3. The calibration gap is largely the expected result of fitting a noisy posterior with cross-entropy, and under moderate noise it is much smaller for other rules. Consider a predictor that exactly fits the noisy posterior, which is the setting of Prop. 1. Under one-hot clean labels and symmetric noise, its calibration error equals the noise rate. The CE probe's ECE is 51.59 and 75.92 at 50% and 80% noise. GCE, without any post-hoc step, has ECE below the clean probe on CIFAR-10N Aggregate, CIFAR-10N Worse and CIFAR-100N (5.54, 5.92 and 3.19 against 8.34, 8.34 and 16.77). On CIFAR-100N it is lower than CE with AEC (3.19 against 3.62). The gap is not limited to CE, though. GCE's ECE is 43.62 at 80% symmetric noise, 13.67 at 40% asymmetric noise and 12.24 on Clothing1M. The claims should be stated per training rule and noise level, and GCE's calibration under moderate noise deserves discussion.

  4. AEC's effect rests on a fixed target that is never justified. The paper notes that the target ξ = 0.995 need not equal anchor accuracy, but it offers no other way to set it, and the results follow the task's accuracy. After AEC, ECE is 2.3 to 5.2 on the DINOv2 CIFAR-10 settings, but 8.8 to 14.3 on Clothing1M, where accuracy is 62 to 70%. The appendix overview promises an AEC sensitivity analysis, but none is given. The paper does not say how the target (0.995) and the anchor fraction (0.1) were chosen, or whether test results were consulted. There is also no comparison of per-class against global temperatures, or of confidence-ranked against random anchors.

  5. The calibration baselines are thin. The only comparison is on one dataset, with no variability. Its only competitor that uses no clean labels is temperature scaling on noisy validation labels. That method targets the noisy posterior, so it cannot remove the noise-induced underconfidence (8.14 against 3.14 for AEC). Missing are noise-corrected temperature scaling with a transition matrix estimated from the same anchors, the cited noisy-label calibration methods (Penso et al., TransTS), and temperature scaling with a small clean budget. NLL and Brier are reported only for CE.

  6. The anchor purity in Table A.5 does not fit its definition. Eq. 13 counts anchors whose noisy label is correct. The selector ignores noisy labels and symmetric noise is independent of the image. Purity should therefore be near 0.5 at 50% noise and 0.2 at 80%, rising only to the extent that the linear head memorizes individual flipped labels. The paper reports 0.963 and 0.942. A toy simulation (768 dimensions, 50,000 points, logistic regression at several regularization strengths) gives 0.55 to 0.68 and 0.26 to 0.32.

  7. The coverage is narrow and some details are missing.

    • There is one default encoder, and seven of the eight settings with a clean reference use CIFAR-10 training images. The CIFAR-100 synthetic results listed in Appendix E.1 are absent.
    • DivideMix runs with its end-to-end settings on a linear head and falls 4.6 and 7.8 points below CE on CIFAR-100N and Clothing1M. Its reported runtime of 8 to 10 minutes is hard to reconcile with augmented views passing through a ViT-B/14 for 300 epochs at a standard input resolution. The resolution is not stated.
    • The GCE and SSR settings, the validation split, and what the three seeds control are not given.
    • Three captions promise bold entries that are not there, and Appendix B.1 is empty.
Questions for authors
  1. What weight decay and feature normalization does the probe use, and does it converge? How do the results change with a converged L2 logistic regression, and what do AEC and clean-validation temperature scaling do to the clean probe's ECE?
  2. How does this work relate to Zhu et al. (NeurIPS 2024), "Label Noise: Ignorance Is Bliss"? That paper already proposes frozen DINOv2 features with noise-ignorant ERM, compares robust losses on them, and proves essentially the result of Corollary 1 (their Theorem 4). What is new relative to it, and does AEC still help on top of their readout protocol with DINOv2 ViT-B/14?
  3. How were the target ξ = 0.995 and the anchor fraction κ = 0.1 chosen, and how sensitive are the results to them?
  4. How can anchor purity (Eq. 13) be 0.963 and 0.942 under 50% and 80% symmetric noise when the selector ignores the noisy labels?
  5. How does AEC compare with noise-corrected temperature scaling, the cited noisy-label calibration methods (Penso et al., TransTS), and temperature scaling with 20 to 500 clean labels, on several datasets with variability?
  6. Do you agree that, under moderate noise, the calibration gap is mainly a CE effect, given GCE's raw ECE? Can you report NLL and Brier for all four rules?
  7. Why do the noisy CLIP and ViT probes beat their clean counterparts in Table 3, and what are the standard deviations?
  8. At what resolution and with how many augmented views does DivideMix-LP run, and how does it finish in 8 to 10 minutes?
Reviewer_iJHzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper attempts to analyze the difference between accuracy and posterior probability of linear probabilistic classifiers operating on frozen representations provided by some encoders, when learning using noisy labels (LNL). The paper first establish that such a gap indeed exists, through straightforward theoretical analysis and empirical evaluation using linear probabilistic classifiers trained on frozen foundational model encodings and a few different losses. Then, the paper evaluates the effectiveness of Anchor-based Efficient Calibration (AEC) as a way to mitigate this gap.

The empirical evaluation considers a combination of a few frozen representations and three losses, and two sources of label noise (synthetic or human-annotations). This analysis shows that high accuracy does not guarantee good calibration, and that this gap can be backbone-dependent.

Critical strengths

None.

Critical weaknesses

The contributions of this paper are either already well-understood or trivial extensions of known results. In particular, it is understood that high accuracy does not guarantee good calibration of probabilistic models in general. Applying this to a setting where linear models are learned over outputs of a frozen encoder in the presence of noisy labels does not make this observation any more consequential. The paper does not provide any new insights such as characterizing when this gap might vanish in practice (beyond a trivial theoretical result under simplifying assumptions) or how one might mitigate its effects.

Questions for authors
  1. What are the main contributions of the paper? In the introduction, three contributions are mentioned, but the first two seem trivial and I am unclear what the main finding is based on the third one.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.