acceptodds
Withdrawn from ICLR 2027

Does a Shared Temperature Imply a Shared Angular Scale in Probabilistic Contrastive Learning?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In probabilistic contrastive learning, a shared temperature is commonly interpreted as a shared similarity scale, but this interpretation does not hold for high-dimensional distributional class representations. We study the exact von Mises–Fisher (vMF) probabilistic score used by ProCo when representation dimension and class concentration grow jointly. We prove that the score retains a class-dependent leading angular gain , where is the mean resultant length. This gain enters Softmax competition, pairwise decision boundaries, and feature gradients. On real CIFAR-LT, ImageNet-LT, and iNaturalist representations, the theory accurately predicts boundary movements and local gradient changes under the full vMF score. Classwise temperature adjustment also changes the cosine-zero intercept and finite-dimensional response. We construct intercept-preserving and Pure Angular controls to separate the leading gain from these accompanying changes. Complete gain equalization yields a shared-scale cosine prototype rule at leading order; a finite-dimensional margin condition guarantees agreement of the two classifiers. Across 16 frozen representation settings, prediction agreement is 98.43–99.99%, with disagreements concentrated at small cosine margins. In controlled contrastive-only training with the training-frequency prior, Pure Angular editing improves both learned representations at all tested CIFAR-10/100 imbalance factors and retains positive changes on ImageNet-LT. Thus vMF concentration not only describes class distributions, but also forms a decision and learning scale in high-dimensional probabilistic contrastive learning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
contrastive learning, representation learning, von Mises–Fisher distribution, high-dimensional asymptotics, temperature scaling, long-tailed recognition, decision geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2taTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper theoretically and empirically investigates the effects of a shared temperature on classification scores and learning dynamics in ProCo, a probabilistic contrastive learning method based on the vMF distribution. Specifically, through a high-dimensional analysis of the vMF score, the authors demonstrate that a shared temperature does not necessarily imply a shared angular scale, as the effective angular gain gc=Ac/τg_c = A_c/\tau depends on the class-specific mean resultant length AcA_c. The paper further analyzes how this gain affects decision boundaries and feature gradients, and compares several approaches to equalizing the gain across classes. Experiments on multiple long-tailed image classification datasets are conducted to validate the theoretical predictions and examine their effects on representation learning.

Critical strengths

S1: Presentation. This paper is well organized and clearly written, with a logical progression from problem formulation to theoretical analysis and empirical validation.

S2: Theory. The theoretical analysis is carefully developed and sufficiently detailed. In particular, the uniform approximation guarantees for both the score and its derivative in high-dimensional settings are valuable.

S3: Focus. The distinction between a shared temperature and class-dependent effective angular gains provides an interesting perspective. The experimental design, which separates the different effects induced by temperature modifications, is also well motivated.

Critical weaknesses

W1: The novelty and significance of the theoretical contribution are somewhat limited. The central observation that the effective angular gain takes the form gc=Ac/τg_c = A_c/\tau can be derived relatively directly from the expected feature representation defined in the original ProCo formulation. Although the rigorous high-dimensional approximation guarantees are valuable, it remains unclear to what extent the main finding provides fundamentally new insights beyond those already implied by the existing formulation.

W2: The motivation and benefits of angular gain equalization are not sufficiently established. While the paper demonstrates that equalizing the gains affects decision boundaries and gradients, it does not provide a theoretical justification for why such equalization should improve generalization. Moreover, the empirical benefits depend on experimental conditions and the treatment of class priors. Consequently, it remains unclear under what conditions gain equalization is desirable or beneficial.

W3: The practical significance and generalizability of the proposed approach remain limited. The main representation-learning experiments focus on the probabilistic contrastive objective of ProCo, and the observed performance improvements are relatively modest. Furthermore, complete gain equalization brings the leading score geometry close to that of a conventional cosine-prototype classifier. Therefore, controlled comparisons with simpler cosine-based learning approaches are important for establishing the added value of the proposed intervention. The current results do not sufficiently clarify what practical advantages the proposed approach offers over such simpler alternatives.

Questions for authors

Q1: Novelty of the theoretical contribution: The effective angular gain gc=Ac/τg_c=A_c/\tau appears to follow relatively directly from the expected feature representation in the original ProCo formulation. Could the authors clarify what fundamentally new insights are provided by the high-dimensional analysis that cannot be obtained from the existing formulation or a straightforward first-order approximation?

Q2: Motivation for angular gain equalization: While gain equalization improves performance under certain experimental conditions, it also leads to substantial degradation depending on the treatment of class priors. Under what conditions do the authors expect gain equalization to be beneficial? Furthermore, is equalization intended as a practical approach to improving generalization, or primarily as an analytical intervention to understand the role of angular gains?

Reviewer_kaQ2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper analyzes the high-dimensional vMF score used by ProCo, identifies a class-dependent angular gain Ac/τA_c/\tau, and studies its effects on decisions and learning through controlled score interventions.

Critical strengths

The paper offers an interpretable connection between class-distribution geometry, temperature, and feature learning, supported by theoretical analysis and controlled experiments.

Critical weaknesses
  1. The incremental contribution and practical benefit over ProCo need stronger justification. In lines 043–057 and 084–092, the contribution is an analysis of the existing vMF score rather than a new distribution model. This can be valuable, but the distinction between a new theoretical insight and an improved learning method needs to be clearer. In Table 1 (lines 163–182), several improvements are small, particularly +0.17/+0.21 percentage points on ImageNet-LT. I notice three-seed have been considered, but I am wondering whether it is enough to show that the improvement is significantly distinguished from zero. Furthermore, Appendix M reports mixed joint-training outcomes (lines 2948–2961) and substantial uniform-prior degradation (lines 2970–2972). Please clarify the advance over ProCo, quantify uncertainty in the method differences, and explain the conditions under which the intervention is practically useful.

  2. The presentation does not make the mathematical scope and key mechanisms sufficiently accessible. In lines 064–068, “cosine-zero intercept” is introduced without an immediate intuitive explanation or equation reference, making the intervention difficult to follow. Section 2.1 (lines 098–107) introduces the Bessel-function notation and log-MGF score too abruptly. Theorem 1 does state its joint-growth and bounded-temperature assumptions (lines 119–124), but their practical meaning and connection to the evaluated settings should be explained more directly. In addition, the vMF model fixes the distributional shape through its direction and concentration, while actual class features can be asymmetric or multimodal, as acknowledged in lines 428–439. I think more discussion should be included and I am wondering how the difference between the real and vMF impact your main conclusion.

Questions for authors
  1. Novelty and benefit over ProCo (lines 043–057, 084–092 and Table 1) Is the novelty primarily the joint high-dimensional analysis of the existing score, or a generally useful modification of ProCo? Please distinguish these contributions and justify the practical improvement.
  2. Distributional scope (lines 055–057, 098–107(section 2.1), 428–439(section 4.2)) How should the Ac/τA_c/\tau interpretation be understood when class features have asymmetric, anisotropic, or multimodal structure that a single vMF cannot capture? Please clarify which conclusions concern the fitted vMF score and which concern actual class distributions, and provide a model-mismatch diagnostic if possible.
  3. Exposition and asymptotic regime I am not familiar with “cosine-zero intercept,” which makes the method description difficult to follow at its first occurrence in the part of Introduction. Could you give an intuitive explanation and refer immediately to qc(0;τ)q_c(0;\tau)? Please also define the modified Bessel function explicitly and explain how the stated expansion conditions relate to the dimensions, concentrations, and temperatures used in the experiments.
Reviewer_piyDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes the exact vMF probabilistic score of ProCo in the regime where the dimension and the concentration grow jointly. Using a uniform expansion that controls both the remainder and its derivative, the authors prove that even though all classes share the same temperature, the effective angular scale gc=Ac/τg_c = A_c/\tau differs across classes. They then show that this gain propagates into softmax competition, pairwise boundaries, and feature gradients, and that the resulting boundary shifts and gradient changes can be quantitatively predicted on CIFAR-LT, ImageNet-LT, and iNaturalist features. Finally, the paper proposes the Pure Angular intervention, which separates the temperature-dependent intercept and remainder. The authors report that full Pure Angular coincides with the cosine prototype classifier under a margin condition, and that it yields small performance improvements in contrastive-only training.

Critical strengths

Rigorous theoretical development connected to quantitative analysis. The paper precisely identifies the difference between the fixed-dimensional limit and the joint limit, and clearly shows that a shared temperature does not imply a shared angular scale. Through exact derivative expressions and a Riccati identity, it directly controls not only the error in the values but also the error in the derivatives, so its subsequent claims about boundary directions and gradients do not rely on formal differentiation. The same coefficients are independently rederived as directional cumulants, and the orders of the residuals are verified with high-precision Bessel computations. In addition, through diverse empirical validations, the paper quantitatively analyzes how closely the theoretically predicted values agree with the observed ones.

Clean decomposition of temperature effects and well-designed diagnostics. Using exact identities, the paper decomposes the differences among the three realizations (direct temperature, intercept-preserving, and Pure Angular) into a change in the intercept and a change in the finite-dimensional remainder. It then demonstrates these differences empirically with a diagnostic that varies only the score while holding the representations and statistics fixed. Furthermore, the paper presents a margin certificate as a sufficient condition under which full equalization coincides with the cosine prototype rule, and confirms that the predictions indeed coincide on all certified queries.

Critical weaknesses

W1. Limited novelty and significance of the core finding. The identification of the coefficient Ac/τA_c/\tau is trivial; the paper's substantive contribution lies in controlling the magnitude of the remainder term, and I am not sure whether this control alone constitutes a sufficiently meaningful contribution. Specifically, the fact that the linear coefficient is Ac/τA_c/\tau follows almost directly from the first cumulant of the log-moment-generating score. Since the paper itself presents the exact decomposition qc=tρAc+DΦ(⋅)q_c = t\rho A_c + D_{\Phi}(\cdot) in Proposition 2 (Eq. B.1), the linear term is already Ac/τA_c/\tau regardless of the dimension, and what the joint asymptotics actually add is the magnitude control showing that the remainder term is small. Moreover, the phenomenon in which the norm of the class mean vector AcμcA_c\mu_c determines the logit slope is structurally identical to the well-known observation that class weight norm imbalance in a linear classifier causes head bias in long-tailed settings, as well as to the τ\tau-normalization of Kang et al. However, this connection is not discussed, which makes it difficult to judge the novelty.

W2. Claims about representation learning are stronger than the evidence supports. The improvements in Table 1 are small, and in some cases, such as the ImageNet-LT Encoder, they fall within the standard deviation. On CIFAR-100/50, Direct temperature or Intercept-preserving outperforms Pure Angular, which is not consistent with the statement that "Pure Angular is consistently better." More importantly, the main results are confined to a single setting, namely contrastive-only training with a training-frequency prior. Under the uniform prior, the same intervention incurs large losses of −11 to −20 pp on CIFAR-10 IF50/100; under joint training, which is ProCo's original recipe, the results are mixed; and on ImageNet, β=1.5\beta=1.5 outperforms full equalization. This indicates that the generality of Pure Angular, the paper's main proposed method, is limited, and in light of this, the "continuous evidence chain" conclusion at the end of Sec. 3.4 appears somewhat overstated. Additionally, the absolute accuracy of contrastive-only training on ImageNet-LT (around 23% for the Encoder) is very low compared with standard ProCo, which raises doubts about the representativeness of this setting.

Questions for authors

Q1. What unique conclusions does the joint-regime analysis provide beyond Proposition 2? Since Proposition 2 gives the exact decomposition qc=tρAc+DΦq_c = t\rho A_c + D_\Phi at every dimension, the identification of the coefficient Ac/τA_c/\tau itself follows directly from this decomposition. I understand the contribution of Theorem 1 to be the magnitude control showing that the ρ\rho-dependence of the Bregman term is O(ν−1)O(\nu^{-1}) in both its value and its derivative. Is there a concrete case in which a conclusion fails without this control, for instance, one where a fixed-dimensional analysis would yield an incorrect prediction?

Q2. How does training with Pure Angular differ from training with a shared-scale cosine prototype score? In frozen evaluation, fully equalized Pure Angular agrees with the cosine prototype rule at a rate of 98.43–99.99%, with a gradient cosine about 1.0. Given this, I wonder whether the training improvement in Table 1 is actually distinguishable from simply the effect of training with a shared-scale cosine score on the class mean directions. Is there a baseline trained with a cosine prototype score under the same conditions? I would also appreciate a concrete explanation of how this work compares with class weight norm imbalance and with Kang et al.'s τ\tau-normalization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.