acceptodds
Under review as a conference paper at ICLR 2027

LOCAL TARGET CONTRAST UNDER CONDITIONAL UNCERTAINTY

Abstract

Local contrastive losses for regression use target differences to choose which nearby representations to separate. These differences can contain irreducible noise. We analyze a quadratic margin loss by conditioning on the input batch, including the dependence introduced by a target-gap median. Its preferred pair distance is proportional to the conditional probability of a push label, which need not measure a difference in conditional means. When the loss acts directly on scalar predictions, we give a sufficient condition under which the Bayes predictor is not a local minimum of the combined population objective. An exact example has positive excess squared risk for every positive loss weight. For nonconstant means, a finite-design analysis identifies net pair forces; incompatible edge pref- erences can cancel. Intermediate geometry can instead use prediction-insensitive directions, but independent activation changes need not correspond to a shared encoder update. We test this distinction using separate fitting, protection, diag- nostic and evaluation inputs. The tested shared updates show a tradeoff between geometry descent and protection of predictions on independent inputs. On the re- lated red and white Wine Quality tasks, MLP and Transformer comparisons show small, mixed effects. The results separate a condition for objective bias from a general ranking of loss locations in training.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Contrastive Regression, Conditional Uncertainty, Pairwise Regularization, Representation Geometry, Excess Risk, Prediction Protection

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ye9EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper analyzes bias induced by irreducible target noise in local contrastive losses for regression. They study a quadratic margin loss that labels nearest-neighbor pairs as “push” when the target gap exceeds the batch median, showing that the preferred pair distance is proportional to the conditional probability of a push label, which can occur even between inputs with equal conditional means. They further study when this biases predictions, finding that the Bayes predictor can fail to be a local minimum when the loss acts directly on scalar predictions, and deriving a local formula for the bias for nonconstant means. Experiments use synthetic data to validate the formula and show that bias can be partially avoided by moving the loss to an intermediate representation, though shared encoder updates trade geometry descent against protecting predictions on independent inputs. They also study the effects of loss placement on the Wine Quality dataset.

Critical strengths
  1. The overall idea is clear, showing that label noise can mislead a contrastive regression loss, causing it to separate inputs that should receive the same prediction. They demonstrate how this can distort predictions when the loss is applied directly to outputs, but can, in principle, be avoided when applied to an intermediate representation.
Critical weaknesses
  1. The impact seems fairly limited. The analysis targets a specific formulation of the pair loss (nearest-neighbor pairs, a batch-median push/pull threshold, and a quadratic margin loss) rather than more commonly used contrastive regression methods like RNC or ConR, and the authors note their analysis doesn’t cover those objectives. Proposition 1 also only applies when the loss acts directly on scalar predictions, while most contrastive regression methods put the loss on representations. Even within this setting, the key condition for Proposition 1 (C_v > 0) isn’t guaranteed by the median rule, as the authors point out, and it’s never checked for the loss as actually used. As a result, it’s unclear how much the results carry over to more commonly used methods.

  2. The experiments on synthetic data show inconsistent effects on prediction error: in the pure-noise control, prediction-level geometry increases excess risk on only one of the two input maps, mostly through a mean offset the pair loss can’t directly cause. The shared-update experiments measure prediction disturbance rather than regression risk, and the Wine Quality experiments cannot isolate label noise. The paper does not show when this issue meaningfully harms models in the real world, nor does it propose fixes if harms were to occur.

Questions for authors
  1. Can the authors show a setting where this bias meaningfully hurts a trained model? For example, a real dataset with known or estimable label noise, or a synthetic task where prediction-level geometry consistently increases error across seeds and input maps.

  2. Can the authors demonstrate whether this mechanism affects more commonly used contrastive regression losses such as RNC or ConR?

  3. Proposition 1 requires C_v > 0, which the median rule doesn’t guarantee. Can the authors show this condition holds for the loss as used, either analytically or by checking it on the synthetic tasks?

Reviewer_ysTJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Contrastive objectives in regression problems often make up for the lack of positive/negative example pair labels by using distance either in true label, representation, or predicted label space to determine which pairs of examples should be pushed apart. In this work, the authors reveal that this can be at odds with the mean squared error objective typically used in regression problems. The authors demonstrate a number of theoretical results regarding the (in)compatibility of these objectives, including cases where the Bayes optimal predictor for the MSE loss is does not minimize the joint contrastive-MSE loss. Empirical results support the theoretical findings.

Critical strengths
  • Contrastive objectives and the typical MSE objective are both commonly used, and so demonstrating when these objectives interfere with one another is interesting. The theoretical results also help the reader build some intuition around both of these objectives. For instance, I thought the result showing that Bayes optimal predictors can be non-optimal for the joint loss, and that the geometric loss is invariant to a constant shift in the function output, helped me better understand the interplay between these two objectives.
Critical weaknesses
  • [W1] The paper is very difficult to understand. The introduction and abstract refer to terms like “loss weights”, “pair forces”, and “loss locations” without any elaboration. In section 3, the problem is introduced with no care as to indicating the potential domains of X and Y, or their intuitive relationship. What is presumed to be the risk of the predictor f is presented in equation (1) with no explanation. In section 5 describes using a transformer model when the task described has not been structured as a sequence modelling task (see Q1). Tasks are introduced abruptly with very little introduction (line 213 mentions the “branch tasks” as if the reader already knows that is a class of tasks they will be working over). At line 307 “update coordinates” are mentioned with no introduction [Q2]. “Protection” is frequently referred to, perhaps in reference to the work of Farajtabar et al. (2020), but this is never defined within the paper [Q2]. These terms, and several others, are not clear, and more care should be taken to describe these important experimental details to the reader.
  • [W2] The results within the paper are very limited in scope. It is not clear whether the simultaneous optimization of MSE and a contrastive loss is common (Q3). The synthetic data results involve a very specific generative model (equation 10), and only a single real dataset is considered where the results are not particularly insightful (unclear whether the geometric loss is helpful to add, or whether it should be added). Furthermore, some of the theoretical results also require heavy assumptions, such as those seen in Proposition 2.
Questions for authors
  • [Q1] When using the transformer on multivariate data such as the wine dataset, is each token a scalar that is embedded into higher dimensional space?
  • [Q2] What are “update coordinates”? What is “protection”? Are we keeping some predictions the same? Do “update coordinates” indicate that some of the weights are frozen?
  • [Q3] Are there some strong examples of researchers or practitioners using contrastive losses and mean squared error in tandem? Furthermore, is the specific contrastive form described in equation (3) common? It is not clear to me why we would always want to have half of the examples being defined as positive pairs and half described as negative pairs, with this constantly shifting.
Reviewer_5xm8OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper looks at a potential conflict between contrastive learning and regression, especially when the target has some inherent uncertainty. The authors focus on a quadratic-margin contrastive loss and show that applying it directly to predictions can move the solution away from the conditional mean, which is optimal for squared-error regression. They first analyze a simple setting and derive a sufficient condition for this to happen, then extend the analysis to nonconstant target means. The paper also includes synthetic experiments to see whether applying the contrastive loss to hidden representations can avoid this issue while still learning useful features.

Critical strengths
  • The question being studied is somewhat interesting and novel.
Critical weaknesses

The reviewer found the paper is difficult to follow because the introduction provides insufficient context and motivation. The introduction is too short to present meaningful information, yet the authors should have plenty of space to write it since the main manuscript is shorter than 8 pages.

Several parts in the paper are unclear. For example, solely on page 2:

  • Lines 092–093 state: “When a generator is known, we estimate it on test inputs using its conditional mean, rather than subtracting two noisy empirical MSE values.” The referent of “it” is unclear.
  • Around lines 097–098 and Equation (3), the distance DiD_i and margin rr are introduced without clearly specifying the space in which distance is measured or explaining the margin’s role.
  • More fundamentally, the nearest-neighbor pairing rule and batch-median push/pull criterion appear without sufficient motivation. Why is this particular objective worth studying, and how closely does it represent losses used in existing contrastive regression methods?

As such, it is very hard to read and understand the paper to clearly justify its merits.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.