acceptodds
Under review as a conference paper at ICLR 2027

Fabricated Negatives in Multi-Label Contrastive Learning: What the Loss Sees, What Removal Changes, and What the Metric Can Tell

Abstract

Multi-label contrastive training treats every label absent from a sample's record as a verified absence; under partial annotation this turns coordinates that nobody examined into negative supervision. We make the observation process explicit and report three findings, each of which reverses a conclusion the standard pipeline draws from the same runs. (1) Fabricated negatives are common and must be counted where the loss receives them. We define a fabrication rate from the label matrix and a recorded observation mask, split into an undecidable clause (coordinate not examined) and a contradicted clause (sample observed positive), and computed on CPU before training. On a renal-pathology corpus it is 0.26 over all pairs, 0.49 at the loss, and 0.52 as realized in training; on Open Images, 96% of train-tier candidate negatives cannot be certified, and the usual coverage approximation misses by 0.37. (2) Which pairs a rule removes, not how many, decides what removal does. Random removal of up to 66% of the candidates the loss receives leaves a profile probe unchanged; at the same volume, a knowledge-graph rule that our pipeline had called certification lowers it by 0.25, because 72% of what it removes are certifiably correct negatives, while removing only the contradicted pairs that partially overlap the target raises all three probes over eight seeds. Where hidden labels are known, the undecidable pairs are worth removing too: removal raises macro AUC on every seed of a pathology testbed, on COCO the gain from not treating unexamined labels as absent grows with the wrong-negative mass the diagnostic predicts before training (Spearman 1.0, declared in advance), and on the federated record of LVIS it raises macro AUC by 0.019 on every seed against complete labels. (3) The standard metric cannot see any of this. Scoring each model against its own training vocabulary ranks a control trained on meaningless text above every real-text arm and makes removal look beneficial and dose-dependent (Spearman +0.90); on shared held-out text no removal ranks first. Scoring recorded profiles in their closed world errs the other way: models that gain on complete labels lose on every seed, and a constant prediction outscores them all. We distill the findings into a six-item reporting protocol that makes a negative-mining claim auditable whichever arm ranks first.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multi-Label Contrastive Learning, False Negatives, Partial Annotation, Negative Mining, Class-Balanced Sampling, Evaluation Protocol, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JFUgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reliability of negative supervision in multi-label contrastive learning under partial annotation. The authors argue that treating unrecorded labels as verified negatives introduces fabricated negative pairs and that conventional negative-removal evaluations can lead to misleading conclusions. The paper makes three main contributions. First, it introduces an observation-aware diagnostic that distinguishes unexamined label coordinates (C1) from observed contradictions (C2) and measures negative-supervision contamination at the corpus, sampling-schedule, and training-loss levels. Second, it investigates how different negative-removal strategies affect learned representations, showing that the identities of removed pairs can matter more than their number. Third, it examines how evaluation vocabularies and incomplete ground-truth labels influence model rankings, demonstrating that conventional metrics can favor semantically meaningless representations. The analysis includes experiments on a renal histopathology dataset and supplementary evaluations involving Open Images, COCO, LVIS, and chest radiography datasets. The authors conclude with a six-item reporting protocol intended to improve the reliability of negative-mining research.

Critical strengths
  1. A meaningful investigation of negative supervision and evaluation reliability.

The paper addresses an important issue in multi-label contrastive learning: missing annotations are frequently treated as negative evidence, even when the corresponding labels were never examined.

A particularly valuable contribution is the distinction between contamination measured at the dataset level and contamination encountered by the actual training objective. The finding that the loss-side contamination rate can substantially exceed the corpus-level rate illustrates why conventional dataset statistics may be insufficient for understanding contrastive training.

The analysis of evaluation vocabulary is also informative. Demonstrating that a model trained with meaningless text can outperform real-text models under matched-vocabulary evaluation raises important questions about whether existing evaluation procedures measure semantic understanding or merely compatibility between learned representations.

  1. Extensive diagnostic controls and transparent experimental analysis.

The paper includes several useful experimental controls, including random removal at matched fractions, rule-level audits of removed pairs, separate analyses of observation-based removal strategies, and evaluation using shared held-out text.

The authors also acknowledge limitations in the statistical independence of certain runs and provide detailed per-seed results. The extension to datasets with different annotation mechanisms further strengthens the relevance of the problem.

These experiments demonstrate considerable effort toward identifying why apparently positive experimental conclusions may be misleading.

Critical weaknesses
  1. The definition and interpretation of the C1 and C2 categories contain a potentially important logical inconsistency.

The proposed diagnostic assigns a negative candidate to C1 whenever at least one required negative coordinate is unexamined. The C2 condition is evaluated only when C1 does not apply.

However, this ordering allows candidates that are already contradicted by an observed positive label to be assigned to C1.

For example, suppose a candidate requires labels a and b to be absent, where a is unexamined but b is observed positive. Under Definition 1, this candidate is assigned to C1 because a is unexamined. Nevertheless, the negative candidate is already known to be invalid because b is observed positive.

Therefore, C1 does not exclusively represent undecidable negatives, contrary to the interpretation repeatedly used in the paper.

This issue may also affect Remark 1, where the probability that a C1 candidate is a correct negative is expressed as the product of the absence probabilities of its unexamined required coordinates. Such an expression does not account for C1 candidates already contradicted by an observed positive coordinate, whose probability of being valid negatives is necessarily zero.

Since the distinction between undecidable and contradicted negatives underlies several central experiments, the authors should clarify whether the reported C1/C2 statistics and intervention results remain valid under a contradiction-first definition.

This is more than a terminological concern because the classification rule determines which pairs enter each removal intervention and how the resulting effects are interpreted.

  1. The generality of the conclusions is limited by differences in experimental objectives and incomplete cross-protocol evaluation.

Although the paper includes multiple public datasets, the most comprehensive analyses of contrastive negative removal and evaluation-vocabulary effects are conducted on a single renal histopathology pipeline.

The COCO and LVIS experiments primarily use sigmoid-based multi-label supervision rather than the same contrastive negative-pair objective. These results provide valuable evidence about missing-label treatment, but they do not directly establish that the same conclusions hold across different multi-label contrastive objectives, negative-sampling mechanisms, or representation-learning architectures.

Moreover, not all negative-removal strategies are evaluated under the shared held-out vocabulary. The observation-based and random-removal experiments conducted after the held-out evaluation budget was exhausted are assessed primarily using profile probes.

This is an important limitation because the paper itself demonstrates that changing the evaluation vocabulary can substantially alter the ranking of negative-removal strategies. Consequently, the beneficial effects of certain observation-based removal interventions have not been directly established under the shared held-out semantic evaluation protocol.

Finally, the held-out vocabulary results rely on four consecutive seeds that share much of their sampling streams, as acknowledged in Appendix H. Although the authors avoid presenting these as independent replicates, the limited independence reduces the statistical strength of this central evaluation result.

Overall, the experiments establish several important observations, but stronger evidence is required to support their general applicability to multi-label contrastive learning.

Questions for authors

Q1. C1/C2 classification consistency.

Under Definition 1, why does the C1 condition take precedence when a candidate contains both an unexamined required coordinate and another coordinate that is observed positive? Such a candidate is already known to be an invalid negative.

Could the authors report the fraction of C1 candidates that also contain observed positive contradictions and repeat the relevant analyses using a contradiction-first classification rule? In particular, how would this change the C1/C2 proportions, the observation-removal results, and the interpretation of Remark 1?

Q2. Validity of the prevalence-based estimate.

Remark 1 estimates the probability that a C1 candidate is a valid negative using the absence probabilities of unexamined coordinates. How does this expression handle candidates assigned to C1 that already contain an observed positive contradiction?

Could the authors clarify the assumptions required for this estimate and evaluate its accuracy after explicitly separating genuinely undecidable candidates from those already known to be incorrect?

Q3. Consistency across evaluation protocols.

The paper shows that matched-vocabulary and shared held-out-vocabulary evaluations can produce different conclusions. However, the observation-based removal strategies are not comprehensively evaluated under the shared held-out vocabulary.

Could the authors clarify whether the improvements from C2 partial-overlap removal persist under a shared held-out text evaluation? This would directly connect the negative-removal findings with the paper's central argument about evaluation validity.

Q4. Generalization across contrastive objectives.

The most extensive negative-removal experiments use one histopathology contrastive learning pipeline, while several public-dataset experiments rely on sigmoid-based multi-label objectives.

Could the authors provide additional evidence using a different multi-label contrastive learning formulation or explain which theoretical and empirical conclusions are expected to remain valid when the negative-pair construction and contrastive loss change?

Reviewer_U33HOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper discusses a scenario where, in multi-label contrastive learning, labels absent from a sample’s record are treated as negative, i.e., the image does not contain those labels. However, if the dataset is only partially annotated, this assumption is no longer necessarily true. In that case, an absent label may simply be unobserved rather than truly negative, and continuing to treat such labels as negatives can introduce problematic supervision.

This is the main idea I was able to extract clearly from the manuscript. Unfortunately, I am not very familiar with this specific research area, and I found the paper difficult to follow in terms of its central motivation, methodological contribution, and practical conclusion.

The comments below are based on my best understanding of the paper.

Critical strengths

The manuscript is motivated by an interesting scenario where sometimes researchers may neglect that a dataset is only partially labeled.

The paper is transparent about the failure of the authors’ previous work.

Critical weaknesses
  1. The central failure mode arises when unrecorded labels are treated as negatives. It is unclear to me how common this specific assumption is in current multi-label contrastive-learning systems. The paper itself cites papers addressing the distinction between missing and negative labels. Then the problem feels somewhat strawman unless the authors can demonstrate that this assumption is widespread in contemporary practice.

  2. Following 1, much of the core problem appears to be a data/annotation bookkeeping issue rather than a new contrastive-learning problem.

  3. The contribution is difficult to identify clearly, both because of the presentation and because of the relatively narrow scope of the problem.

Questions for authors
  1. How common it is for researchers not examining dataset that is partially annotated?
  2. If a dataset already maintains an observation mask and the loss ignores unexamined labels, what additional practical problem does the proposed framework solve?
  3. Can the authors clarify whether the main goal is to propose a method or to provide an auditing/reporting framework?
Reviewer_5kBuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates a common problem in multi-label contrastive learning: missing labels are often treated as negative labels, even though they may simply have not been examined. The paper shows that such false negatives are common and can become more frequent during training due to sampling. It compares three strategies for removing negative pairs: random removal, knowledge-graph-based removal, and observation-based removal. The results suggest that random removal has little effect, while removing certain contradicted pairs may improve representation quality in some evaluation settings. The paper also argues that evaluation using the same text descriptions seen during training can produce misleading results and recommends using held-out descriptions and meaningless-text controls.

Overall, the paper addresses a relevant problem and provides useful observations about negative sampling and evaluation. However, the presentation is difficult to follow, the experimental design is complicated, and the statistical evidence does not fully support some of the main conclusions.

Critical strengths

The paper addresses a clear and practically important problem in multi-label contrastive learning. An unrecorded label should not automatically be treated as a verified negative, as this can introduce incorrect supervision during training.

The distinction between unexamined labels and labels that directly contradict a proposed negative is useful. The paper also makes an important point that the frequency of problematic negatives should be measured during training, rather than only in the original dataset, because sampling can substantially change their frequency.

The comparison between training descriptions and held-out descriptions is valuable. It raises an important concern about whether the evaluation measures meaningful representations or simply agreement with text descriptions memorized during training.

The experiments on public datasets provide additional evidence that treating missing labels as negatives can hurt performance, particularly when complete label information is available.

Critical weaknesses

The paper lacks illustrative figures and clear visualizations of the experimental results, making it extremely difficult to follow. A simple illustration of different types of negative pairs and a clear comparison of the removal strategies would significantly improve the presentation.

The paper includes a pathology study, several public datasets, and multiple evaluation settings. However, these experiments are not organized around a clear and consistent scientific question. As a result, it is difficult to identify the main contribution and understand how each experiment supports the central conclusion.

The main claim that removing contradicted negative pairs improves representation learning is not fully supported by the statistical results. Although the average improvements across eight seeds are positive, the confidence intervals include zero for the main group and simple probes. Therefore, the evidence is insufficient to establish a consistent improvement.

Several comparisons are not completely fair or well controlled. The removal strategies differ not only in which negative pairs are removed, but also in how positive pairs are handled, how weights are assigned, and how the loss is calculated. These differences make it difficult to determine whether the improvements come from removing incorrect negatives or from other changes in training.

The random removal controls are approximately matched by the number of removed pairs, but they do not always use exactly the same training conditions. A more controlled comparison would be needed to isolate the effect of different removal strategies.

The statistical analysis is limited. Several experiments use only four random seeds, and the authors acknowledge that consecutive seeds share much of the sampling process. This raises concerns about the independence of the experiments and the reliability of the reported improvements.

The main pathology dataset cannot be publicly shared, which limits the reproducibility of the most important experiments. Although the public-dataset experiments are useful, it is not clear whether they provide sufficient independent validation of the main findings.

Questions for authors

Are the improvements from removing contradicted negatives statistically significant?

Can the authors provide fair comparisons that isolate the effect of negative-pair removal?

Which experiments provide the strongest evidence for the main conclusions?

Can the key findings be reproduced on public datasets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.