acceptodds
Under review as a conference paper at ICLR 2027

When Confidence Collapses: Robust Pseudo-Label Selection for Extremely Sparse Semi-Supervised Learning

Abstract

Semi-supervised learning (SSL) reduces annotation costs by exploiting abundant unlabeled data, yet its behavior under extremely sparse supervision remains poorly understood. We systematically investigate this regime and identify a common failure mode in confidence-driven pseudo-label selection, where an overfit classification head produces saturated confidence scores that become poorly discriminative. As a result, existing selection strategies can retain large numbers of unreliable pseudo-labels, causing training performance to peak prematurely and subsequently regress. Motivated by this observation, we propose a confidence-robust pseudo-label selection framework that combines a Confidence Calibration Prior with Weak-Strong Consistency. The Confidence Calibration Prior uses a fixed confidence floor and a per-class top- constraint to control pseudo-label selection without relying solely on the confidence distribution, while Weak-Strong Consistency improves the reliability of the selected supervision. We further use MoCo pretraining as a complementary representation initialization and evaluate the resulting framework under a unified protocol across three image-classification benchmarks and multiple label densities. Experiments show that our method consistently improves over representative confidence-driven baselines in the extremely sparse regime, while its advantage becomes dataset-dependent as the labeling budget increases. These results reveal an operating boundary of confidence-driven pseudo-label selection and motivate sparse-label SSL methods that are less dependent on the confidence distribution of the classification head.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
semi-supervised learning, sparse-label

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TftjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a pseudo-label selection framework for extremely sparse semi-supervised learning. It combines a fixed confidence floor and per-class top-K selection with weak-strong consistency, prototype supervision, and MMD regularization, and uses MoCo pretraining for representation initialization.

Critical strengths
  • The motivation to study unreliable pseudo-label selection under extremely sparse supervision is valuable.
  • The class-wise selection constraint provides a simple way to control pseudo-label quantity.
  • The training diagnostics provide useful observations of pseudo-label quality and training stability.
Critical weaknesses
  • The distinction between confidence collapse and confirmation bias is insufficiently explained, while the methodological novelty appears limited.
  • The discussion of prior sparse-label results is inaccurate. Existing methods already evaluate settings with four labels per class, and their original results do not support the claim of severe degradation.
  • Comparisons may be unfair because the proposed method uses additional MoCo pretraining, while several baselines are simplified reproductions.
  • The reason for reducing the unlabeled pool to 10,000 samples is unclear. The experiments do not distinguish label scarcity from insufficient total training data.
  • The main hyperparameters are insufficiently justified or analyzed.
  • Evaluation is limited to three datasets, and the method achieves the best performance in only four of nine settings. The legend in Figure 4 also needs better placement.
Questions for authors
  • Can the authors clarify the distinction between confidence collapse and confirmation bias, and better justify the methodological novelty?
  • Can the authors explain the discrepancy between the reproduced baseline results and those reported in the original papers, and evaluate the baselines with the same MoCo pretraining?
  • Can the authors explain the reduced unlabeled pool and verify the observations using larger pools?
  • Can the authors specify how the main hyperparameters are selected and provide sensitivity analyses?
  • Can the authors extend the evaluation to better establish the performance advantages?
Reviewer_1E95OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the failure of confidence-driven pseudo-label selection in semi-supervised learning (SSL) under extremely sparse supervision. The authors identify a phenomenon termed confidence collapse, in which classification heads produce highly confident predictions despite low pseudo-label accuracy. This behavior can cause existing SSL methods to accept unreliable pseudo-labels and experience substantial performance degradation during training. To address this issue, the paper proposes a two-stage framework combining MoCo v2 representation pretraining with a Confidence Calibration Prior (CCP) and Weak-Strong Consistency (WSC). CCP imposes a fixed confidence floor and a per-class top-K selection constraint, while the subsequent SSL stage incorporates pseudo-label consistency, prototype-based supervision, and MMD regularization. The method is evaluated against eight baselines on CIFAR-10, SVHN, and STL-10 with 4, 25, and 40 labeled examples per class. The reported results show that the proposed approach achieves the highest peak test accuracy across all three datasets at four labels per class. However, its advantages diminish or reverse as the labeling budget increases, particularly on SVHN. The paper further provides ablation studies and per-epoch diagnostics to investigate pseudo-label quality and training stability.

Critical strengths
  1. The paper addresses a practically relevant but challenging SSL regime and provides valuable diagnostics of confidence-driven selection failures. In particular, Figure 4 and Table 3 reveal that confidence-based methods may exhibit saturated prediction confidence while maintaining low pseudo-label accuracy, accompanied by considerable peak-to-final performance degradation. For example, on CIFAR-10 with four labels per class, FreeMatch exhibits an average peak-to-final accuracy drop of 8.32 percentage points, compared with 1.67 points for the proposed method. These observations are useful because they highlight the importance of analyzing pseudo-label reliability and training trajectories rather than relying exclusively on maximum confidence scores or peak accuracy.

  2. The paper evaluates nine methods across three datasets and three label densities. The proposed method achieves the highest reported peak accuracy at four labels per class on all three benchmarks, and the component ablations provide additional evidence that the full framework can be beneficial in this regime. Importantly, the authors also acknowledge substantial performance reversals at higher labeling densities, especially on SVHN with 25 labels per class. This analysis helps characterize the limitations of a fixed top-K selection budget and avoids presenting the proposed approach as universally superior.

Critical weaknesses
  1. A central concern is that the proposed framework incorporates several components beyond the Confidence Calibration Prior, including an additional 50 epochs of MoCo v2 pretraining, weak-strong augmentation, prototype-based supervision, and MMD regularization. In contrast, the main baseline comparisons do not provide the same MoCo initialization. Appendix H also reports substantially greater training time for the proposed method. Although Table 2 contains component-level ablations, the Weak-Strong Consistency-only configuration already includes MoCo pretraining, while the CCP-only configuration does not. Consequently, the experiments do not cleanly disentangle the contributions of representation pretraining, augmentation, auxiliary losses, and the proposed selection rule. Furthermore, Appendix C indicates that several recent baselines are simplified reproductions that omit components from their original formulations. For example, the CLFP reproduction excludes its supervised contrastive branch, while the CoVar reproduction omits its EMA teacher and pairwise loss. Single-point comparisons against FreeMatch are insufficient to establish that these simplified reproductions preserve the original methods' behavior in the extremely sparse-label regime. These issues are consequential because the main contribution is framed as a more robust pseudo-label selection mechanism. Without matched-pretraining comparisons and stronger component isolation, it is difficult to determine whether the observed improvements primarily arise from the proposed selection principle or from differences in representation quality, optimization, and implementation.

  2. The main evaluation relies on peak test accuracy, rather than final-epoch accuracy or checkpoint selection using a separate validation set. Selecting the maximum test accuracy across training epochs can introduce optimistic evaluation bias, particularly when methods exhibit substantial fluctuations or early performance peaks. More importantly, the treatment of divergent runs is inconsistent. Table 1 reports SVHN results with 40 labels per class for CGMatch and RegMixMatch using three seeds after excluding a divergent run, whereas Appendix E, Table 12 lists all four seeds and reports substantially different means: 64.23% for CGMatch and 68.92% for RegMixMatch, compared with 81.95% and 85.37% in Table 1. Meanwhile, a divergent FixMatch run on SVHN with 25 labels per class is retained in its four-seed average. The different handling of divergent seeds requires a consistent justification and correction. Additionally, some improvements in the extremely sparse regime are small relative to seed variability. For example, on SVHN with four labels per class, the proposed method improves over FreeMatch by only 0.73 percentage points, while the reported standard deviations are 3.35 and 1.72 points, respectively. The available results therefore do not establish a statistically decisive improvement in this setting.

Questions for authors
  1. Can the authors provide comparisons in which the strongest baselines receive the same MoCo-pretrained initialization as the proposed method? Additionally, what are the individual contributions of MoCo pretraining, prototype supervision, and MMD regularization when the remaining components are controlled?

  2. Why are the divergent seeds of CGMatch and RegMixMatch on SVHN with 40 labels per class excluded from Table 1, whereas the divergent FixMatch seed at 25 labels per class is retained? Can the authors provide a consistently computed four-seed summary for every method and clarify whether the test set was used to select the reported peak checkpoints?

  3. The per-class top-K constraint limits selection to at most 80 samples from each unlabeled batch of 128, which closely matches the reported keep ratio of approximately 62%. Can the authors compare CCP against simpler alternatives, such as retention-matched random sampling, global top-K selection, or class-balanced sampling applied to an existing confidence-driven method?

Reviewer_VbtDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an approach for semi-supervised learning under extremely scarce annotations. It identifies the main challenges of current approaches and addresses them with two elements: a confidence-calibrated prior that looks at the K most relevant samples in each batch (highest confidence per class), and a weak-strong consistency regularization term that keeps the labeled and unlabeled distributions close together. The experimental section benchmarks the approach on three datasets and against several state-of-the-art methods, and provides ablation studies for the two main elements of the approach. The authors also provide a discussion and an exploration of the training dynamics to further explain the method.

Critical strengths
  • The approach surpasses other competing methods in a scarcely labeled setup. This shows that the proposed techniques successfully stabilize training when supervision is as scarce as 4 samples per class.
  • The paper ablates the main elements of the approach and provides a useful analysis of the mechanisms by which it stabilizes training in the low-label regime.
Critical weaknesses
  • The novelty is somewhat limited. Current literature already addresses extremely sparse SSL, with some papers even reaching one labeled sample per class. Hence, the framing of the paper in the introduction should focus more on the insights provided and on the contribution of the proposed approach, and a bit less on the fact that the paper explores sparse SSL.
  • The Confidence-Calibrated Prior should be discussed further. The top-K selection mechanism is very similar to an adaptive per-class and per-batch threshold. The authors should further discuss the improvements that this approach provides over a selection mechanism based solely on prediction confidence.
  • It seems that the second contribution, Weak-Strong Consistency, consists of two terms: cross-entropy between predictions and prototypes, and a regularization term that keeps the labeled and unlabeled representations close (multi-kernel MMD regularizer). The text should further highlight the novelty of this part. The first term is a prototype-based loss (similar to "ProtoCon: Pseudo-label Refinement via Online Clustering and Prototypical Consistency for Efficient Semi-supervised Learning" - CVPR23, and "Improving Barely Supervised Learning by Discriminating Unlabeled Samples with Super-Class" - NeurIPS22), while the latter is very similar to the distribution alignment techniques used in SSL (similar to ReMixMatch or "Semi-Supervised Learning by Augmented Distribution Alignment" - ICCV19). The paper should delve into the details of these two elements so that the novelty of the contribution is clearly distinguished from previous work.
  • The method makes the model stable under sparse supervision but degrades performance in less scarce setups. The authors discuss this in the experimental section and point to Section 4.4 for further explanation. This explanation should be reviewed, as it does not address the lower performance of the approach in less scarce scenarios. The explanation of Table 3 and Figure 4 in the main text is not clear and seems to show some discrepancies with the reported experiments in the table/figure. While Subsection 4.5 attributes this lower performance to the top-K selection being too limiting, the paper should provide a more definitive justification for this behaviour.
  • The experimental scope is limited. The datasets used are relatively simple research-oriented datasets that do not guarantee the generalization of the method. The current experiments already show different behaviour when moving from CIFAR to STL-10 (as reported by the authors in Subsection 4.3), suggesting that more complex datasets introduce different training challenges. Along similar lines, the model architectures are limited to WideResNet and SmallCNN. Despite these being widely used datasets and models in SSL research, the authors do not address or discuss the challenges that the approach might face when generalizing to more complex setups.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.