acceptodds
Under review as a conference paper at ICLR 2027

Anchor Divergence for Semantic Geometry in Contrastive Learning

Abstract

This paper concerns how semantic context determines geometry in learned vector representations. Similarity is typically measured using cosine similarity, which provides a single fixed geometry. Semantic similarity, however, is inherently context dependent: two images may be similar because they depict the same object, share a visual style, or are relevant to the same clinical finding. We show that contrastive representations naturally encompass a family of geometries that can be specialized to particular semantic structure. The key idea is to use an interplay between contrastive learning, exponential families, and information geometry to establish a correspondence between probability distributions over “anchors” and Bregman geometries on the representation space. We use this correspondence to define Anchor Divergences, a method for specifying context-specific semantic geometries on fixed representations. Under this correspondence, modeling the anchor distribution models the geometry itself. Experiments on retrieval show that anchor divergences provide an effective and efficient way to specify context-specific semantic similarity.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
contrastive learning, representation learning, information geometry, semantic similarity, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9GkCOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the "Anchor Divergence" method, which leverages the Bregman geometric structure induced by the distribution of "anchors" in contrastive learning to flexibly define similarity metrics within a fixed representation space, tailored to specific semantic contexts. By parameterizing the anchor distribution, the method not only provides an effective approximation of the underlying semantic geometry, but also utilizes an empirical Bayesian framework to incorporate broad semantic priors, thereby enhancing adaptability to specific tasks. Experiments in image retrieval and clinical diagnosis demonstrate that, compared to conventional cosine similarity, Anchor Divergence can more precisely focus on the specific semantic dimensions relevant to the user, while offering superior computational efficiency and sample adaptability.

Critical strengths
  1. The motivation of this paper is clear, and the proposed algorithm is highly novel. It transforms context-dependent similarity into an object that can be directly intervened upon and verified.
  2. Through experiments across various tasks in different domains, encompassing both parametric and non-parametric approaches, the authors have effectively demonstrated the validity of their method.
Critical weaknesses

(1) Existing baselines are insufficient for isolating the specific contribution of the divergence measure employed. Standard cosine similarity fails to leverage the target text, while ITI retains only the single best-matching anchor. In contrast, this paper utilizes the full distribution of anchor responses. Consequently, the observed performance gains may, in part, stem from the retention of more task-relevant information. I suggest comparing other simple distance metrics, applied to the same softmax probability vectors, to determine whether the specific geometric formulation used provides any additional benefit.

(2) While the paper offers a valuable unified interpretation, the empirical contribution of the proposed model requires a clearer presentation. The authors note that the "limited-anchor" softmax-KL formulation aligns with TIP-X, and that the strongest results, achieved under the "full-anchor" setting, stem primarily from this specific formulation. Although the parameterized model demonstrates advantages in certain limited-anchor scenarios, the current experiments need to more directly illustrate the conditions under which this new approach to distribution modeling provides distinct gains.

Questions for authors

See "Weakness".

1 comment

We thank the reviewer for recognizing the novelty of our framework for context-dependent similarity and the breadth of our experiments exploring its consequences.

  1. Contribution of the divergence measure

    KL provides an exact correspondence between anchor distributions and Bregman geometries, with the anchor distribution’s cumulant generating function as the generating potential. Modeling the anchor distribution therefore models a semantic geometry. This gives a systematic way to derive useful constructions like the nonlinear geometries induced by GMMs. This structure also enables efficient computation: after preprocessing each image against the convex potential, similarity can be computed exactly from compact cached statistics without storing full softmax vectors (Appendix F.4).

    To isolate the empirical effect of divergence choice, we compare distances on identical softmax responses over all 1,000 ImageNet class-name anchors:

    MethodAUPRC
    Cosine0.3509
    Euclidean0.6457
    Hellinger0.7293
    Total variation0.7213
    Symmetric KL0.7195

    Hellinger and total variation perform comparably to KL, showing that the gains from semantic anchoring extend across several distributional distances. We will include this comparison in the appendix.

  2. Empirical contribution of distributional modeling

    Useful structure with limited anchors. When acquiring anchors is costly or when only a limited quantity is available, parametric models can use the structure in the anchor space to enhance retrieval performance relative to softmax (Figures 4 and 6). Modeling therefore contributes beyond access to additional semantic information: distributional assumptions can improve geometry estimation from the same anchors.

    Controllable complexity and computation. Once fitted, parametric models avoid processing the full anchor bank for each new image. Table 4 quantifies this benefit: at 2172^{17} anchors, Gaussian takes 0.375 seconds versus 3.066 seconds for empirical softmax, including fitting, preprocessing, and retrieval. More expressive families, such as GMMs, can represent multiple semantic modes, with subsequent evaluation cost controlled by model complexity rather than the original bank size.

    Posterior adaptation of broad semantic knowledge. Section 4 summarizes broad anchors through prior parameters and updates them using scarce target anchors, thereby adapting the induced geometry. Figure 5 demonstrates the usefulness of this construction when target anchors are limited.

    With full representative target-anchor coverage, empirical softmax remains strongest in Figures 4 and 6. Our central contribution is a framework for defining a semantic geometry, offering the unified perspective recognized by the reviewer. We study this framework under different information and computational constraints and will make the benefits, tradeoffs, and supporting evidence summarized above more explicit in the experimental discussion.

Reviewer_NKMnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to define context-dependent similarity on fixed contrastive representations. Images are mapped to softmax distributions over semantic text anchors and compared through the KL divergence between these distributions. The authors show that this finite-anchor construction converges uniformly to a population Bregman divergence induced by the cumulant generating function of an anchor distribution. This perspective motivates Gaussian, Gaussian-mixture, and latent subspace models, together with posterior or posterior-like updates that combine broad semantic information with limited target anchors. Experiments on natural-image, sketch-based, and chest X-ray retrieval show that anchors can control the semantic attributes emphasized by retrieval, parametric models can improve performance at limited anchor budgets, and WordNet-based adaptation improves ImageNet retrieval when target anchors are scarce.

Critical strengths
  1. The paper addresses a meaningful problem in using pretrained representations for retrieval. A fixed image embedding can encode object identity, style, and other attributes, while the relevant notion of similarity depends on the task. Figure 3 directly examines this issue by varying the mixture of object and style anchors and measuring retrieval along both coordinates. Table 2 provides complementary evidence on CheXpert, where softmax anchor divergence improves average AUPRC over cosine similarity by 0.0128. These results support the usefulness of specifying retrieval semantics through text without changing the image encoder.

  2. The anchor-distribution perspective provides a coherent way to organize empirical, parametric, and adapted semantic geometries. Theorem 1 formalizes the population limit, while Figures 4 and 6 show how the relative benefits of parametric models and empirical softmax depend on anchor coverage. Figure 5 shows benefits from broad prior information at small target-anchor budgets, while target-only softmax performs best at full coverage. Table 4 also demonstrates a computational advantage for the Gaussian model at large anchor-set sizes. Together, these results help identify when different constructions are useful.

Critical weaknesses
  1. The adaptation experiments establish gains over target-only models, but the comparative benefit of the chosen update rules remains insufficiently explored. Section 5.4 explains why broad prior information is most useful when target anchors are scarce, and Figure 5 supports its usefulness in this regime. However, the experiment does not compare the proposed updates with other ways of combining the same broad and target information. For example, the latent spherical update in Appendix D.4 obtains a subspace from a weighted sum of the broad projector and target covariance. Its Bayesian interpretation motivates this regularization choice, but does not establish its advantage over other reasonable combinations. A focused comparison using the same text resources would strengthen the evidence for the adaptation component.

  2. Several details of the limited-anchor experiment require clarification because they could affect the interpretation of the gains. Appendix E.5 fixes the latent dimension at 128 for the adaptation experiments, but it is unclear whether this also applies to the target-only latent spherical model in Figure 5. With n target anchors, the centered covariance has rank at most n−1; if a larger subspace is retained, the selection of additional directions needs to be specified. This choice could influence the baseline at the smallest anchor budgets. Appendix D.4 also does not clearly specify the experimental value or selection rule for β in the spherical update. The construction and size of the WordNet anchor set, including its overlap with ImageNet class names or synsets, are also insufficiently documented. These details are needed to reproduce the comparison and interpret the target-anchor budget.

Questions for authors
  1. How do the proposed adaptation rules compare with other reasonable ways of combining the same broad and target information, such as a weighted empirical anchor mixture? Where an update is equivalent to a regularized estimator, please clarify that relationship and compare alternatives that are genuinely different. Reporting the broad model’s performance before target updating would also help quantify the incremental benefit of the target update.

  2. Does k = 128 apply to the target-only latent spherical curve in Figure 5? If so, how are directions beyond the empirical covariance rank handled? If such directions are retained, does the conclusion hold with a rank-aware choice or the 95% explained-variance rule used elsewhere? Please also specify β and its selection procedure for the spherical update, and describe the WordNet anchor construction, size, and overlap with ImageNet class names or synsets.

3 comments

We thank the reviewer for their thoughtful assessment and constructive feedback. We are glad they found the paper clear, easy to follow, and informative. We address the minor or fixable soundness concerns through additional comparisons, ablations, and clarifications below.

  1. Contribution of target updating and comparison with empirical mixing

    Our central goal is to understand how semantic context determines geometry in contrastive representations. Through the contrastive softmax, we establish a correspondence between text-anchor distributions and semantic geometries on fixed image embeddings. Modeling the anchor distribution therefore models the geometry itself, providing a systematic way to derive useful constructions like the nonlinear geometries induced by GMMs.

    Empirical mixing is a natural instance of this broad framework, as illustrated in Figure 3. We compare posterior updating with qw=(1−w)qbroad+wqtargetq_w=(1-w)q_{\mathrm{broad}}+wq_{\mathrm{target}}, selecting ww using 2,000 separate training images. Results average 3 runs using overlap-removed WordNet and 10,000 ImageNet validation queries against a 50,000-image gallery.

    AUPRC across all classes

    Method / target anchors0163264128256
    Latent spherical0.46710.47380.48020.48590.52450.5493
    GMM0.44490.46800.49130.51350.52330.5470
    Empirical mixture0.50420.50810.51080.51450.52740.5602

    Zero target anchors denotes the broad-only baseline.

    Posterior updating improves both parametric models over their broad-only baselines, while empirical mixing achieves the highest aggregate AUPRC.

    AUPRC on classes absent from the target-anchor set

    Method / target anchors064128256
    Latent spherical0.46710.47910.51190.5301
    GMM0.44490.50670.50900.5217
    Empirical mixture0.50420.50530.50430.5163

    Only query classes are restricted; the gallery retains all 1,000 classes.

    Posterior models outperform calibrated empirical mixing on these remaining classes. Parametric modeling can capture semantic structure in the text space that benefits classes beyond those explicitly named by the target anchors, and posterior adaptation can retain and refine this structure by using the broad-anchor prior to regularize target-specific estimation.

    These complementary strengths illustrate the framework's purpose: studying and adapting semantic geometry under different information and computational constraints. We do not claim that any one construction is universally superior.

  2. Rank selection and update hyperparameters

    Rank. Figure 5 fixes k=128k=128 for both target-only and adapted models. At small budgets, eigensolver-selected null-space directions complete the target-only subspace, avoiding restriction to very few dimensions.

    Method / Target anchors216641282561,000
    Target-only, rank 1280.25430.29240.40460.51030.53500.5627
    Target-only, 95% variance0.00150.14480.42600.50430.53610.5334
    Adapted, rank 1280.45670.46300.47740.51370.53500.5625
    Evaluated on full dataset

    Adaptation outperforms both target-only variants at small anchor budgets.

    Choice of β\beta. The spherical posterior-like update uses the weighting derived for the Gaussian model in Appendix D.3. For nn target anchors, β=n2(1ϵ−11+ϵ)\beta=\frac{n}{2}\left(\frac{1}{\epsilon}-\frac{1}{1+\epsilon}\right). We fix ϵ=10−8\epsilon = 10^{-8} (Appendix E.5) mainly to avoid covariance degeneracy, without tuning it for adaptation. The resulting large β\beta prioritizes directions supported by target-anchor variation, while the broad WordNet projector guides the selection of remaining directions.

    We will add these implementation details and the rank ablation to the appendix.

  3. WordNet construction and ImageNet overlap

    ImageNet-1K uses a subset of WordNet synsets. Our NLTK-derived bank contains 148,730 distinct lemma strings; 1,892 match ImageNet class-synset lemmas or dataset class-name aliases. We remove these strings, refit parametric models, and reevaluate broad-only retrieval:

    MethodFull WordNetAfter overlap removal
    Cosine0.35090.3509
    Softmax KL0.55890.4910
    Gaussian0.37930.3771
    Latent spherical0.45620.4533
    GMM (K = 256)0.45650.4307

    Full ImageNet validation AUPRC

    All remain above cosine. We will clarify WordNet’s relationship to ImageNet and their overlap in the main text, with construction and filtering details in the appendix.

I appreciate the authors' efforts to address my concerns. The rebuttal substantially addresses my second weakness and partially addresses the first. I continue to find the broader semantic-geometry framework interesting and appreciate the additional evidence. However, the comparative utility of the proposed posterior adaptation remains insufficiently established. I therefore maintain my original assessment. Given the differing opinions among reviewers, I defer to the AC's judgment on the final recommendation.

We’re glad that the reviewer appreciates the paper’s significance and continues to find the framework interesting. We continue to think that the correspondence between anchor distributions and semantic geometries provides a foundation for using statistical techniques to understand and make use of semantic structure in learned representations. Our experiments show that updating broad anchor distributions with target information can improve the induced image geometry; the remaining concern is the comparative utility of the particular posterior updates. We appreciate the reviewer’s engagement throughout the discussion.

Reviewer_we2WOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies context-dependent semantic similarity in contrastive representations, based on the single fixed notion of cosine similarity. The authors introduce anchor divergences to specify semantic geometry through distributions over text anchors. The proposed framework connects KL divergence between anchors’ softmax distributions to Bregman divergence. An empirical-Bayes-style approach is also designed to adapt broad anchor distributions using task-specific anchors.

Critical strengths
  • The problem is well motivated, and the method’s validity is supported by the semantic coordinate experiment.
  • The clinical retrieval experiments show meaningful quantitative improvements.
Critical weaknesses
  • The adaptation experiments do not fully isolate the benefit of the proposed updating strategy. Fig. 5 compares the posterior models using broad WordNet anchors and limited target anchors, against models receiving only the target anchors. However, it would be difficult to isolate the improvement coming from the adaptation procedure and the available broad semantic information.
  • The overlap between WordNet anchor construction and ImageNet class names need clearer clarity.
Questions for authors
  • How much does adaptation improve over the corresponding broad-only model?
1 comment

We thank the reviewer for the thoughtful assessment and for recognizing the paper’s motivation, theoretical contribution, and empirical appeal.

  1. Separating broad semantic information from target adaptation

    To isolate the contribution of target updating, we compare each posterior model with its corresponding broad-only prior. We use WordNet after removing direct ImageNet class-name and synonym matches, so the comparison also tests whether updating remains useful after overlap removal.

    All methods are evaluated on 10,000 ImageNet validation queries against a 50,000-image training gallery. The results below are the average of 3 runs.

    Method / target anchors0163264128256
    Latent spherical0.46710.47380.48020.48590.52450.5493
    GMM0.44490.46800.49130.51350.52330.5470

    AUPRC; zero target anchors denotes the broad-only model.

    At 256 target anchors, updating improves AUPRC by 0.0822 for latent spherical and 0.1021 for GMM. Thus, target-specific updating contributes substantially beyond the broad geometry, and these gains persist after removing direct ImageNet-name overlap from the broad anchor bank.

    We will add the corresponding broad-only baselines to the adaptation plot and include the WordNet–ImageNet overlap ablation in the appendix.

  2. WordNet–ImageNet overlap

    ImageNet-1K classes correspond to a subset of WordNet synsets. Our WordNet bank contains 148,730 lemma strings, including 1,892 matching ImageNet class-synset lemmas or class-name aliases. We remove these strings and reevaluate the broad-only geometries:

    MethodFull WordNetAfter overlap removal
    Cosine0.35090.3509
    Softmax KL0.55890.4910
    Gaussian0.37930.3771
    Latent spherical0.45620.4533
    GMM (K = 256)0.45650.4307

    ImageNet validation AUPRC; broad-only models; evaluated on the full dataset

    Removing direct overlap reduces softmax-KL and GMM performance, while Gaussian and latent spherical change only slightly. All filtered models remain above cosine. Thus, direct target-name inclusion contributes to performance, but does not fully explain the benefit of broad anchors.

    Given the expansive vocabulary of a broad anchor set, some degree of overlap with the target set is naturally expected. However, including target names among many other anchors differs from concentrating the anchor distribution on target semantics: the induced geometry depends on how probability mass is distributed, not simply on which terms are present. Adaptation can therefore refine the geometry even when the broad bank already contains the target names. We will clarify this in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.