- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces the "Anchor Divergence" method, which leverages the Bregman geometric structure induced by the distribution of "anchors" in contrastive learning to flexibly define similarity metrics within a fixed representation space, tailored to specific semantic contexts. By parameterizing the anchor distribution, the method not only provides an effective approximation of the underlying semantic geometry, but also utilizes an empirical Bayesian framework to incorporate broad semantic priors, thereby enhancing adaptability to specific tasks. Experiments in image retrieval and clinical diagnosis demonstrate that, compared to conventional cosine similarity, Anchor Divergence can more precisely focus on the specific semantic dimensions relevant to the user, while offering superior computational efficiency and sample adaptability.
Critical strengths
- The motivation of this paper is clear, and the proposed algorithm is highly novel. It transforms context-dependent similarity into an object that can be directly intervened upon and verified.
- Through experiments across various tasks in different domains, encompassing both parametric and non-parametric approaches, the authors have effectively demonstrated the validity of their method.
Critical weaknesses
(1) Existing baselines are insufficient for isolating the specific contribution of the divergence measure employed. Standard cosine similarity fails to leverage the target text, while ITI retains only the single best-matching anchor. In contrast, this paper utilizes the full distribution of anchor responses. Consequently, the observed performance gains may, in part, stem from the retention of more task-relevant information. I suggest comparing other simple distance metrics, applied to the same softmax probability vectors, to determine whether the specific geometric formulation used provides any additional benefit.
(2) While the paper offers a valuable unified interpretation, the empirical contribution of the proposed model requires a clearer presentation. The authors note that the "limited-anchor" softmax-KL formulation aligns with TIP-X, and that the strongest results, achieved under the "full-anchor" setting, stem primarily from this specific formulation. Although the parameterized model demonstrates advantages in certain limited-anchor scenarios, the current experiments need to more directly illustrate the conditions under which this new approach to distribution modeling provides distinct gains.
Questions for authors
See "Weakness".
We thank the reviewer for recognizing the novelty of our framework for context-dependent similarity and the breadth of our experiments exploring its consequences.
Contribution of the divergence measure
KL provides an exact correspondence between anchor distributions and Bregman geometries, with the anchor distribution’s cumulant generating function as the generating potential. Modeling the anchor distribution therefore models a semantic geometry. This gives a systematic way to derive useful constructions like the nonlinear geometries induced by GMMs. This structure also enables efficient computation: after preprocessing each image against the convex potential, similarity can be computed exactly from compact cached statistics without storing full softmax vectors (Appendix F.4).
To isolate the empirical effect of divergence choice, we compare distances on identical softmax responses over all 1,000 ImageNet class-name anchors:
Hellinger and total variation perform comparably to KL, showing that the gains from semantic anchoring extend across several distributional distances. We will include this comparison in the appendix.
Empirical contribution of distributional modeling
Useful structure with limited anchors. When acquiring anchors is costly or when only a limited quantity is available, parametric models can use the structure in the anchor space to enhance retrieval performance relative to softmax (Figures 4 and 6). Modeling therefore contributes beyond access to additional semantic information: distributional assumptions can improve geometry estimation from the same anchors.
Controllable complexity and computation. Once fitted, parametric models avoid processing the full anchor bank for each new image. Table 4 quantifies this benefit: at 217 anchors, Gaussian takes 0.375 seconds versus 3.066 seconds for empirical softmax, including fitting, preprocessing, and retrieval. More expressive families, such as GMMs, can represent multiple semantic modes, with subsequent evaluation cost controlled by model complexity rather than the original bank size.
Posterior adaptation of broad semantic knowledge. Section 4 summarizes broad anchors through prior parameters and updates them using scarce target anchors, thereby adapting the induced geometry. Figure 5 demonstrates the usefulness of this construction when target anchors are limited.
With full representative target-anchor coverage, empirical softmax remains strongest in Figures 4 and 6. Our central contribution is a framework for defining a semantic geometry, offering the unified perspective recognized by the reviewer. We study this framework under different information and computational constraints and will make the benefits, tradeoffs, and supporting evidence summarized above more explicit in the experimental discussion.