Fabricated Negatives in Multi-Label Contrastive Learning: What the Loss Sees, What Removal Changes, and What the Metric Can Tell
Abstract
Multi-label contrastive training treats every label absent from a sample's record as a verified absence; under partial annotation this turns coordinates that nobody examined into negative supervision. We make the observation process explicit and report three findings, each of which reverses a conclusion the standard pipeline draws from the same runs. (1) Fabricated negatives are common and must be counted where the loss receives them. We define a fabrication rate from the label matrix and a recorded observation mask, split into an undecidable clause (coordinate not examined) and a contradicted clause (sample observed positive), and computed on CPU before training. On a renal-pathology corpus it is 0.26 over all pairs, 0.49 at the loss, and 0.52 as realized in training; on Open Images, 96% of train-tier candidate negatives cannot be certified, and the usual coverage approximation misses by 0.37. (2) Which pairs a rule removes, not how many, decides what removal does. Random removal of up to 66% of the candidates the loss receives leaves a profile probe unchanged; at the same volume, a knowledge-graph rule that our pipeline had called certification lowers it by 0.25, because 72% of what it removes are certifiably correct negatives, while removing only the contradicted pairs that partially overlap the target raises all three probes over eight seeds. Where hidden labels are known, the undecidable pairs are worth removing too: removal raises macro AUC on every seed of a pathology testbed, on COCO the gain from not treating unexamined labels as absent grows with the wrong-negative mass the diagnostic predicts before training (Spearman 1.0, declared in advance), and on the federated record of LVIS it raises macro AUC by 0.019 on every seed against complete labels. (3) The standard metric cannot see any of this. Scoring each model against its own training vocabulary ranks a control trained on meaningless text above every real-text arm and makes removal look beneficial and dose-dependent (Spearman +0.90); on shared held-out text no removal ranks first. Scoring recorded profiles in their closed world errs the other way: models that gain on complete labels lose on every seed, and a constant prediction outscores them all. We distill the findings into a six-item reporting protocol that makes a negative-mining claim auditable whichever arm ranks first.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.