- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the reliability of negative supervision in multi-label contrastive learning under partial annotation. The authors argue that treating unrecorded labels as verified negatives introduces fabricated negative pairs and that conventional negative-removal evaluations can lead to misleading conclusions. The paper makes three main contributions. First, it introduces an observation-aware diagnostic that distinguishes unexamined label coordinates (C1) from observed contradictions (C2) and measures negative-supervision contamination at the corpus, sampling-schedule, and training-loss levels. Second, it investigates how different negative-removal strategies affect learned representations, showing that the identities of removed pairs can matter more than their number. Third, it examines how evaluation vocabularies and incomplete ground-truth labels influence model rankings, demonstrating that conventional metrics can favor semantically meaningless representations. The analysis includes experiments on a renal histopathology dataset and supplementary evaluations involving Open Images, COCO, LVIS, and chest radiography datasets. The authors conclude with a six-item reporting protocol intended to improve the reliability of negative-mining research.
Critical strengths
- A meaningful investigation of negative supervision and evaluation reliability.
The paper addresses an important issue in multi-label contrastive learning: missing annotations are frequently treated as negative evidence, even when the corresponding labels were never examined.
A particularly valuable contribution is the distinction between contamination measured at the dataset level and contamination encountered by the actual training objective. The finding that the loss-side contamination rate can substantially exceed the corpus-level rate illustrates why conventional dataset statistics may be insufficient for understanding contrastive training.
The analysis of evaluation vocabulary is also informative. Demonstrating that a model trained with meaningless text can outperform real-text models under matched-vocabulary evaluation raises important questions about whether existing evaluation procedures measure semantic understanding or merely compatibility between learned representations.
- Extensive diagnostic controls and transparent experimental analysis.
The paper includes several useful experimental controls, including random removal at matched fractions, rule-level audits of removed pairs, separate analyses of observation-based removal strategies, and evaluation using shared held-out text.
The authors also acknowledge limitations in the statistical independence of certain runs and provide detailed per-seed results. The extension to datasets with different annotation mechanisms further strengthens the relevance of the problem.
These experiments demonstrate considerable effort toward identifying why apparently positive experimental conclusions may be misleading.
Critical weaknesses
- The definition and interpretation of the C1 and C2 categories contain a potentially important logical inconsistency.
The proposed diagnostic assigns a negative candidate to C1 whenever at least one required negative coordinate is unexamined. The C2 condition is evaluated only when C1 does not apply.
However, this ordering allows candidates that are already contradicted by an observed positive label to be assigned to C1.
For example, suppose a candidate requires labels a and b to be absent, where a is unexamined but b is observed positive. Under Definition 1, this candidate is assigned to C1 because a is unexamined. Nevertheless, the negative candidate is already known to be invalid because b is observed positive.
Therefore, C1 does not exclusively represent undecidable negatives, contrary to the interpretation repeatedly used in the paper.
This issue may also affect Remark 1, where the probability that a C1 candidate is a correct negative is expressed as the product of the absence probabilities of its unexamined required coordinates. Such an expression does not account for C1 candidates already contradicted by an observed positive coordinate, whose probability of being valid negatives is necessarily zero.
Since the distinction between undecidable and contradicted negatives underlies several central experiments, the authors should clarify whether the reported C1/C2 statistics and intervention results remain valid under a contradiction-first definition.
This is more than a terminological concern because the classification rule determines which pairs enter each removal intervention and how the resulting effects are interpreted.
- The generality of the conclusions is limited by differences in experimental objectives and incomplete cross-protocol evaluation.
Although the paper includes multiple public datasets, the most comprehensive analyses of contrastive negative removal and evaluation-vocabulary effects are conducted on a single renal histopathology pipeline.
The COCO and LVIS experiments primarily use sigmoid-based multi-label supervision rather than the same contrastive negative-pair objective. These results provide valuable evidence about missing-label treatment, but they do not directly establish that the same conclusions hold across different multi-label contrastive objectives, negative-sampling mechanisms, or representation-learning architectures.
Moreover, not all negative-removal strategies are evaluated under the shared held-out vocabulary. The observation-based and random-removal experiments conducted after the held-out evaluation budget was exhausted are assessed primarily using profile probes.
This is an important limitation because the paper itself demonstrates that changing the evaluation vocabulary can substantially alter the ranking of negative-removal strategies. Consequently, the beneficial effects of certain observation-based removal interventions have not been directly established under the shared held-out semantic evaluation protocol.
Finally, the held-out vocabulary results rely on four consecutive seeds that share much of their sampling streams, as acknowledged in Appendix H. Although the authors avoid presenting these as independent replicates, the limited independence reduces the statistical strength of this central evaluation result.
Overall, the experiments establish several important observations, but stronger evidence is required to support their general applicability to multi-label contrastive learning.
Questions for authors
Q1. C1/C2 classification consistency.
Under Definition 1, why does the C1 condition take precedence when a candidate contains both an unexamined required coordinate and another coordinate that is observed positive? Such a candidate is already known to be an invalid negative.
Could the authors report the fraction of C1 candidates that also contain observed positive contradictions and repeat the relevant analyses using a contradiction-first classification rule? In particular, how would this change the C1/C2 proportions, the observation-removal results, and the interpretation of Remark 1?
Q2. Validity of the prevalence-based estimate.
Remark 1 estimates the probability that a C1 candidate is a valid negative using the absence probabilities of unexamined coordinates. How does this expression handle candidates assigned to C1 that already contain an observed positive contradiction?
Could the authors clarify the assumptions required for this estimate and evaluate its accuracy after explicitly separating genuinely undecidable candidates from those already known to be incorrect?
Q3. Consistency across evaluation protocols.
The paper shows that matched-vocabulary and shared held-out-vocabulary evaluations can produce different conclusions. However, the observation-based removal strategies are not comprehensively evaluated under the shared held-out vocabulary.
Could the authors clarify whether the improvements from C2 partial-overlap removal persist under a shared held-out text evaluation? This would directly connect the negative-removal findings with the paper's central argument about evaluation validity.
Q4. Generalization across contrastive objectives.
The most extensive negative-removal experiments use one histopathology contrastive learning pipeline, while several public-dataset experiments rely on sigmoid-based multi-label objectives.
Could the authors provide additional evidence using a different multi-label contrastive learning formulation or explain which theoretical and empirical conclusions are expected to remain valid when the negative-pair construction and contrastive loss change?