acceptodds
Withdrawn from ICLR 2027

PEEK: Partial Label Learning Empowered by External Knowledge

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Partial Label Learning (PLL) is a challenging weakly supervised learning problem that aims to identify the ground-truth label from a candidate set containing exactly one true label and multiple false positives. Despite steady progress, existing PLL methods predominantly rely on closed-world supervision derived from the training data itself, leaving a clear gap in how to effectively incorporate external knowledge—especially the open-world semantic priors emerging from recent large-scale foundation models. In this paper, we first establish a theory which shows that external guidance should be assigned a higher weight in the early stage of training and down-weighted in the later, enabling a smooth transition from knowledge-assisted disambiguation to self-driven refinement. Building on this insight, we propose PEEK, a computationally efficient framework that leverages the open-world semantic knowledge of a frozen VLM (e.g., CLIP). PEEK dynamically fuses the model’s confidence with the VLM-derived confidence, either via simple global weighting or instance-adaptive weighting, thereby enabling an adaptive transition from external guidance to internal self-evolution. As a versatile plug-and-play module, PEEK significantly accelerates convergence and improves a wide range of state-of-the-art PLL baselines on standard benchmarks as well as challenging fine-grained datasets, without introducing additional training overhead.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
partial label learning, semi-supervised learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_MNM8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PEEK, a plug-and-play framework that incorporates external knowledge from a frozen CLIP model to improve label disambiguation in partial label learning. The key idea is to dynamically combine CLIP-derived confidence with the predictions of the trainable model, allowing external knowledge to guide early training while gradually shifting toward the model's own predictions. To achieve this, the authors develop two weighting strategies, namely global dynamic weighting and instance-specific weighting, motivated by a theoretical analysis. Experimental results show that PEEK accelerates convergence and improves the performance of existing PLL methods.

Critical strengths
  1. The paper introduces PEEK, a novel partial label learning approach that utilizes external knowledge from a pretrained CLIP model to improve label disambiguation.

  2. The paper is well-organized and clearly written, making it easy to follow.

  3. Extensive experiments demonstrate the effectiveness of the proposed method across various partial label learning settings.

Critical weaknesses
  1. Some part of the paper needs further improvement to meet ICLR submission. In particular, several formatting issues should be addressed, such as those in Table 11.

  2. The experimental results show inconsistent trends regarding the performance of ISW and GDW, with neither strategy consistently outperforming the other. Can the authors clarify under what conditions ISW is preferable to GDW, and vice versa? A more detailed analysis would help explain their respective advantages and applicability.

  3. Integrate both ISW and GDW can be better than use either one? It is better to give the experimental results.

  4. Since ISW already incorporates GDW through the global weighting factor, it would be valuable to investigate whether combining global decay with instance-specific weighting provides additional benefits over using either component alone. A dedicated ablation study would help clarify the contribution of each component.

  5. In Table 1, several results show that higher noise rates lead to better test accuracy, which is counterintuitive. Could the authors explain why increasing label ambiguity sometimes improves classification performance?

  6. Table 3 shows that the confidence component provides only marginal improvements (or even useless) at certain noise rates (e.g., q=0.3q=0.3 and q=0.7q=0.7). Could the authors explain why its contribution is limited in these settings and clarify under what conditions this component is particularly beneficial?

  7. How you choose your Static weight?

  8. For Table 9, the comparison can be unfair, why not use same backbone? The peak memory (15.73 GB) is not an unmanageable memory consumption.

Questions for authors

Please see the weakness.

Reviewer_muidOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses Partial Label Learning (PLL) and introduces PEEK, a framework that integrates the confidence scores of a frozen vision-language model (VLM) with predictions from a trainable model. By progressively reducing the weight assigned to the VLM, PEEK enables a gradual transition from external semantic knowledge to task-specific predictions. Experimental results demonstrate that PEEK consistently outperforms both the VLM’s zero-shot predictions and existing PLL baselines across standard and fine-grained benchmarks.

Critical strengths

1、The paper is well-organized and clearly written, making the proposed method easy to follow. 2、 The authors introduced two distinct integration strategies, simple global weighting and instance-adaptive weighting. 3、The proposed framework is flexible and broadly applicable, as it does not depend on a specific model architecture or loss function.

Critical weaknesses

1、The MSE-optimal coefficient depends on both predictors’ errors and their covariance, but GDW uses linear decay and ISW uses stability/entropy without proof that these estimate the optimum; the monotonic-error assumption is also not generally valid. The authors must separate proved results from heuristics, track model/VLM/fused errors during training, and compare against a diagnostic oracle weight. 2、Stable and confident predictions can still be wrong, and ISW may then suppress the VLM when external guidance is most needed. The authors should report fusion weights by PLL/VLM correctness, test whether ISW predicts relative reliability, construct VLM-confidently-wrong cases, and compare with calibrated confidence, disagreement, and direct reliability estimates. 3、Under identical settings, compare against VLM-only pseudo-labels, fixed-weight fusion, frozen-teacher distillation, confidence weighting without temporal stability, EMA consistency, and validation-based or training-progress decay. The key issue is whether PEEK’s specific weighting provides reproducible gains over simpler alternatives. 4、At high q, candidate sets may approach all classes, amplifying the VLM prior. Report candidate-set size distributions, candidate-restricted VLM accuracy, VLM-only pseudo-label accuracy, multiple fixed weights including the best validation-selected one, corrupted-VLM results, more seeds, confidence intervals, and paired tests; separate ambiguity robustness from external-knowledge robustness. 5、Renormalizing over candidates can make weak VLM support appear sharp. Report original and candidate-restricted confidence, compare methods that explicitly model candidate/non-candidate mass, and specify handling when the VLM’s preferred class is outside the candidate set. 6、“No training overhead” is unsupported. Separately report per-epoch overhead, one-time VLM prior cost, total training time under matched accuracy/budget, peak memory, and inference latency/memory; lower peak memory does not imply lower total cost. 7、The baselines selected for the standard PLL setting are not sufficiently up to date. The authors are encouraged to include more recent PLL methods for comparison, especially recent baselines that also introduce VLMs, to more comprehensively validate the effectiveness of PEEK.

Questions for authors

Refer to Weaknesses

Reviewer_v3VfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper solves the Partial Label Learning task that identifies the ground-truth label from a candidate set containing exactly one true label and multiple false positives. The authors first theoretically illustrate that external VLM guidance should assign higher weights in the early stages of training and lower weights in later stages. Based on this theory, the authors then proposed the PEEK method to fuse the model's confidence with the VLM-derived confidence through simple global weighting or instance-adaptive weighting.

Critical strengths

S1: Exquisitely designed modules.

S2: Sound Motivation.

S3: Clear Figures.

Critical weaknesses

W1: The presentation could be improved further.

This paper only has a method diagram and lacks a motivation diagram to highlight the difference between this method and other methods (e.g., existing PLL methods mainly rely on closed-world supervision from the training data itself, while peek uses open-world semantic priors emerging from large-scale base models).

W2: The method proposed in this article is based on pre-trained vlm, namely clip. However, this brings up a paradox: (1) If clip can correctly discover all ground-truth labels, why don't we just use clip; (2) If clip cannot accurately predict all positive labels (this is more likely to happen, because clip's classification effect is affected by the clip pre-training data set, if the clip pre-training set does not have data for some labels of PLL, clip should not be able to identify those labels), how does this method improve the accuracy of clip pseudo-labels?

Point 2 actually transforms the PLL problem into a traditional pseudo-label training problem. Although this method seems to use a lot of techniques to optimize pseudo labels, it seems to only compare other PLL methods, not the current sota method of pseudo-label training. and it lacks an overall design idea that is different from other pseudo-label training methods (which results in providing insufficient insight, which is just an accumulation of other existing research).

W3: Inexact definition.

This paper uses a lot of terms related to zero-shot, but does not give them an accurate definition, such as "zero-shot inference", "zero-shot confidence distribution", "zero-shot VLM"?

In addition, in the original CLIP paper [1], the definition of zero samples only means that the test data set and clip's pre-training data set are not the same data set, but for specific data, especially at the category level, clip's pre-training data set overlaps. In other words, clip's pre-training data set contains many categories of the test data set[2][3], and clip's classification performance has a very strong correlation with the frequency of categories appearing in the pre-training set. This seems inconsistent with the "zero-shot" that this work wants to express, because PLL is mainly concerned with classification problems, but clip's zero-shot capability is not at the class level?

[1] Learning transferable visual models from natural language supervision, ICML 2021.

[2] Demystifying clip data, ICLR2024.

[3] No" zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance, NeurIPS 2024.

Questions for authors

Please see questions in W2 and W3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.