- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper analyzes bias induced by irreducible target noise in local contrastive losses for regression. They study a quadratic margin loss that labels nearest-neighbor pairs as “push” when the target gap exceeds the batch median, showing that the preferred pair distance is proportional to the conditional probability of a push label, which can occur even between inputs with equal conditional means. They further study when this biases predictions, finding that the Bayes predictor can fail to be a local minimum when the loss acts directly on scalar predictions, and deriving a local formula for the bias for nonconstant means. Experiments use synthetic data to validate the formula and show that bias can be partially avoided by moving the loss to an intermediate representation, though shared encoder updates trade geometry descent against protecting predictions on independent inputs. They also study the effects of loss placement on the Wine Quality dataset.
Critical strengths
- The overall idea is clear, showing that label noise can mislead a contrastive regression loss, causing it to separate inputs that should receive the same prediction. They demonstrate how this can distort predictions when the loss is applied directly to outputs, but can, in principle, be avoided when applied to an intermediate representation.
Critical weaknesses
-
The impact seems fairly limited. The analysis targets a specific formulation of the pair loss (nearest-neighbor pairs, a batch-median push/pull threshold, and a quadratic margin loss) rather than more commonly used contrastive regression methods like RNC or ConR, and the authors note their analysis doesn’t cover those objectives. Proposition 1 also only applies when the loss acts directly on scalar predictions, while most contrastive regression methods put the loss on representations. Even within this setting, the key condition for Proposition 1 (C_v > 0) isn’t guaranteed by the median rule, as the authors point out, and it’s never checked for the loss as actually used. As a result, it’s unclear how much the results carry over to more commonly used methods.
-
The experiments on synthetic data show inconsistent effects on prediction error: in the pure-noise control, prediction-level geometry increases excess risk on only one of the two input maps, mostly through a mean offset the pair loss can’t directly cause. The shared-update experiments measure prediction disturbance rather than regression risk, and the Wine Quality experiments cannot isolate label noise. The paper does not show when this issue meaningfully harms models in the real world, nor does it propose fixes if harms were to occur.
Questions for authors
-
Can the authors show a setting where this bias meaningfully hurts a trained model? For example, a real dataset with known or estimable label noise, or a synthetic task where prediction-level geometry consistently increases error across seeds and input maps.
-
Can the authors demonstrate whether this mechanism affects more commonly used contrastive regression losses such as RNC or ConR?
-
Proposition 1 requires C_v > 0, which the median rule doesn’t guarantee. Can the authors show this condition holds for the loss as used, either analytically or by checking it on the synthetic tasks?