acceptodds
Withdrawn from ICLR 2027

Semantics as Distributions: Structural Measurement for Vision-Language Representation Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Semantics in vision-language representation spaces is distributional: a semantic concept is expressed through a distribution of cross-modal correspondence patterns rather than a single pairwise response. How such distributional structure should enter alignment measurement, however, remains largely unexplored. Empirically, correspondence vectors collected from image-text pairs sharing the same semantic concept form reproducible and concept-dependent distributions, whose organization we term semantic structural fingerprints. Motivated by this observation, we introduce a decision-theoretic formulation that recasts cross-modal similarity measurement as discrimination between latent matched and mismatched correspondence distributions. By locally approximating the Bayes-optimal log-likelihood ratio to second order, we derive a principled measurement of correspondence strength and structural organization. Based on this formulation, we develop Distributional Structural Measurement (DSM), a computational realization that learns match- and mismatch-associated structures through signed low-rank components at complexity. Experiments across architectures and retrieval benchmarks consistently demonstrate the effectiveness of DSM, while statistical analyses validate the reproducibility and distinctiveness of semantic structural fingerprints. Together, these results support a distributional view of cross-modal semantic measurement. Code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Semantic Similarity Learning, Cross-modal Alignment, Distributional Structural Measurement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Jzb4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new way of scoring similarity in vision-language alignment. Most methods reduce a visual-textual pair to a single cosine score at the final scoring step. The authors argue that this scalar discards how correspondence evidence is organized across feature dimensions. For each caption word, the authors build a word-conditioned visual feature through attention over image patches or regions and multiply it coordinate-wise with the word feature. The result is a correspondence vector z whose sum becomes the cosine similarity. According to the authors, z vectors from occurrences of the same concept form reproducible, concept-specific distributions, which they call ‘semantic structural fingerprints’. Building on this observation, they frame similarity as Bayes-optimal discrimination between matched and mismatched correspondence distributions. A second-order Taylor expansion of the log-likelihood ratio gives a linear term plus a quadratic term. The proposed method, so-called DSM, implements the quadratic term as the difference of two low-rank positive semidefinite forms. The resulting score is normalized, averaged over caption words and trained with a triplet loss. Experiments on Flickr30K and MS-COCO with five backbones include in-domain retrieval, cross-dataset transfer and zero-shot classification on nine benchmarks.

Critical strengths

The main strength of this paper is that it changes only the final scoring step and still obtains consistent gains. Within each backbone, the representation pipeline is held fixed and only the measurement function differs, so the improvement can be attributed to the scoring rule itself. Because cosine similarity sums the coordinate-wise correspondence into a single number, information about which dimensions agree is usually lost. The paper shows that a small learned head can recover part of this information. The added cost is modest. Compared with an unconstrained quadratic head (Full-W), DSM uses 15–3115 – 31 times fewer parameters while matching or exceeding its performance (Table 7). So, overall the proposed method is simple and inexpensive, and it gives clear gains in several settings, especially for the non-CLIP backbones.

Critical weaknesses

Although the authors present DSM as a new scoring method, closely related work is not discussed. This includes low-rank bilinear pooling for multimodal, factorization machines and low-rank bilinear similarity learning. These methods also model second-order interactions through low-rank factorization, and some of them operate on the same element-wise product of visual and textual features. Without a comparison with this line of work, it is hard to judge what DSM adds.

The paper splits the second-order interaction into a branch for match-associated directions and a branch for mismatch-associated directions, and places this design within a Bayes-optimal discrimination framework. Whether the sign split matters has not been tested experimentally. A comparison with a single PSD branch or an unsigned low-rank form of the same rank would be needed. The theory, for its part, motivates the general form of the measurement but does little beyond that. The trained model shares one matrix across all concepts, is optimized with a ranking loss and is wrapped in a normalization that is not part of the derivation.

The improvements are consistent, but they are largest for the non-CLIP backbones. For CLIP, the gain is only about 1∼21\sim 2 Rsum points, and all results appear to come from single runs, so it is unclear whether these differences are statistically reliable. Appendix A12 also shows that the benefit depends on the word-level pipeline: with globally pooled features, DSM performs no better than cosine similarity. The gain may come partly from how the measurement interacts with word-level aggregation, not from the measurement alone.

The fingerprint analysis is interesting, but several points are unclear. First, what ‘semantics as distributions’ means is not well defined. Theorem 1 states that when two correspondence distributions have equal means but different second moments, any linear score, including cosine similarity, gives the same expected response, whereas a quadratic response can tell them apart. This sounds reasonabe. But the fingerprint analysis compares only the mean profile of zz for each concept, which is first-order information. The second-order contrast which is the basis of the method, seems not estimated. Second, the paper does not say which checkpoint the fingerprints were extracted from. If they were taken from a model trained with DSM, the argument becomes partly circular. The intended logic is that concepts differ in structure and DSM is therefore needed. A fingerprint obtained after DSM training could instead reflect structure that DSM itself induced.

Questions for authors

It is recommended that the authors review previous research and check whether it overlaps with the current work, for example, Fukui et al. (2016), 'Multimodal compact bilinear pooling for visual question answering and visual grounding', Kim et al. (2017), 'Hadamard product for low-rank bilinear pooling', Rendle (2010), 'Factorization machines', and Liu et al. (2015), 'Low-rank similarity metric learning in high dimensions'.

To show that the sign split matters, is it possible to compare DSM with (a) a single PSD branch (A = UUᵀ) and (b) an unsigned low-rank symmetric A, both at the same rank and parameter count? If the signed form performs better, this would directly support the central design choice. In addition, since removing the first-order term leaves performance almost unchanged (Table 3), is the linear term needed in practice?

Computing means and standard deviations over at least three seeds for Table 1, especially for the CLIP backbones, where the margins are 1∼21\sim 2 Rsum points would be helpful?

Appendix A12 shows that DSM gives no gain with globally pooled features. To separate the effect of the measurement from that of word-level aggregation, could you add a word-level cosine baseline with learned word weights or attention pooling over words? If DSM still outperforms this baseline, the gain can be attributed more confidently to the measurement itself.

From which checkpoint were the fingerprints extracted? Repeating the analysis on encoders never trained with DSM, and comparing the six concepts against random token groups or mean-centered profiles, would show that the fingerprints reflect concept-specific structure, not structure induced by DSM or shared across all tokens.

Reviewer_5h7uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to improve cross-modal similarity measurement beyond the conventional aggregation of coordinate-wise image-text correspondences into a single scalar. The authors introduce a distributional perspective on semantic correspondence, arguing that matched and mismatched image-text pairs exhibit informative structural differences that may not be captured by cosine similarity. Motivated by a local second-order approximation to the Bayes-optimal log-likelihood ratio, they propose Distributional Structural Measurement (DSM), which combines first-order correspondence evidence with positive and negative low-rank quadratic responses to capture cross-dimensional interactions efficiently. The method is trained using a triplet ranking objective without explicitly estimating the underlying correspondence distributions. Experiments on Flickr30K, MS-COCO, cross-dataset transfer, and zero-shot classification report improvements over conventional similarity measurements. Ablation studies, semantic fingerprint analyses, and near-tie evaluations provide additional evidence for the utility of second-order structural information.

Critical strengths

Comprehensive experimental evaluation. The paper provides a relatively thorough empirical evaluation across multiple vision-language backbones, retrieval datasets, cross-dataset transfer settings, and zero-shot classification benchmarks. Importantly, the authors compare DSM against several alternative similarity measurements and conduct targeted ablations of the first-order and second-order components, learned structural directions, and score normalization. The near-tie analysis further provides relevant evidence that structural information can improve discrimination when conventional cosine similarity is ambiguous. These experiments offer meaningful support for the practical effectiveness of the proposed method.

Critical weaknesses
  1. Insufficient connection between the distributional theory and the implemented method. The paper motivates DSM through a second-order Taylor approximation of the Bayes-optimal log-likelihood ratio (Eq. 5). However, the corresponding quadratic coefficients depend on the derivatives of the underlying matched and mismatched probability densities, which are neither estimated nor explicitly connected to the learned DSM parameters. Furthermore, the second-moment contrast in Eq. (7) and the rank-constrained optimal solution in Proposition 3 arise from a different objective than the Bayes likelihood-ratio formulation. Although Proposition 3 provides a valid justification for the signed low-rank parameterization, it does not establish that the globally shared DSM parameters learned through triplet ranking loss approximate either theoretical solution. The input-dependent normalization in Eq. (10) introduces an additional discrepancy. Consequently, the proposed distributional framework appears to motivate the architecture rather than rigorously characterize the actual measurement being learned, weakening the paper's central theoretical contribution.

  2. Unclear methodological novelty and insufficient evidence linking learned structures to semantic fingerprints. The implemented DSM is essentially a learnable, normalized low-rank quadratic scoring function over coordinate-wise image-text correspondences. While the experiments demonstrate the usefulness of second-order interactions, it remains unclear what is fundamentally new compared with existing nonlinear or structured similarity learning approaches. Moreover, the proposed semantic structural fingerprints are defined through concept-conditioned distributional moments, but these statistics are not explicitly used in training, and the learned structural directions are not shown to correspond to them. Since DSM does not explicitly utilize semantic concept labels or estimate concept-conditioned distributions, it is unclear whether the learned structures genuinely capture concept-specific semantic fingerprints or merely reflect general statistical differences between matched and mismatched image-text pairs.

Questions for authors
  1. Coordinate-basis dependence of semantic structural fingerprints. The correspondence vector is defined as (z=x\odot y), making its coordinate-wise structure dependent on the representation basis. A shared orthogonal transformation of the visual and textual embeddings preserves their cosine similarity but generally changes the coordinate-wise correspondence patterns and their associated structural fingerprints. To what extent are the proposed semantic structural fingerprints invariant or robust to such transformations? Does their semantic interpretation rely on the specific coordinate basis learned by each pretrained encoder?

  2. Relationship to prior structured and probabilistic similarity measurements. The paper discusses GSMN and ESL as prior approaches to structured similarity measurement. Could the authors more clearly distinguish DSM from existing learnable nonlinear similarity functions, particularly in terms of its scoring-function expressiveness and the specific contribution of its distributional formulation? Additionally, how does DSM relate to Variational Adapter for Cross-modal Similarity Representation (VACSR, ICML 2026), which also uses element-wise cross-modal similarity representations and explores probabilistic similarity modeling? Clarifying these relationships would help assess the novelty of DSM's proposed measurement framework.

Reviewer_nddPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that cosine similarity ignores how the agreement is distributed across dimensions. The authors observe that correspondence vectors of the same concept show reproducible patterns ("semantic structural fingerprints"), and derive a scoring function from a second-order approximation of the log-likelihood ratio between matched and mismatched correspondences. The proposed method, DSM, adds signed low-rank quadratic terms to a weighted cosine score with O(rd) cost. Experiments on Flickr30K and MS-COCO with five backbones show consistent gains over cosine, bilinear, and block-wise measurements, as well as in transfer and zero-shot classification.

Critical strengths
  • The idea that the measurement function itself, and not only the encoder, shapes alignment is interesting and somewhat overlooked.

  • The method is simple, cheap (O(rd)), and can be plugged into existing pipelines.

  • The controlled comparison, where only the measurement function changes, is a good design, and the gains are consistent across backbones.

  • The paper is easy to follow and generally well written.

Critical weaknesses

My main concern regarding this work is the gap between the distributional framing and the actual method. In practice, DSM is a learned low-rank quadratic scoring head on z = v ⊙ t, trained with a triplet loss. The derivation is per concept and local around a reference point z0z_0, while DSM uses a single shared matrix A, adds a normalization that is not part of the derivation, and never estimates any distribution. Below I outline additional concerns:

  • The related work is missing important lines of research. Probabilistic embeddings such as PCME [1] and PCME++ [2] explicitly model image-text semantics as distributions, which is very close to the motivation of this paper. Low-rank bilinear pooling with Hadamard products [3] is also closely related to the proposed quadratic form on z.

  • The fingerprint analysis does not really support the second-order claim. The profiles in Appendix A.7 are dimension-wise aggregates (first-order), not the contrast ΔGcΔG_c that the theory relies on. Since the word feature tct_c is almost fixed for a given concept, concept-dependent profiles of zz are somewhat expected. Also, only six concepts are studied. A spectral analysis of ΔGcΔG_c, similar to the eigen-analysis of kernel-matrix differences in SPEC [4], would directly support the claimed structure.

  • With global features, DSM performs the same as cosine (535.8 vs 536.8 Rsum, Table 13), and global cosine is even better than fine-grained cosine with the same backbone (536.8 vs 534.1). This suggests that the gains may come from fixing the word-level averaging pipeline rather than from distributional structure.

  • The near-tie analysis is biased by construction. Pairs are selected where cosine is close to chance, so almost any independently trained model would do better on them. An independently trained cosine model (different seed) should be included as a control.

  • In zero-shot classification, all fine-tuned CLIP variants, including DSM, are below the original CLIP (e.g., 58-60% vs around 67% on ImageNet for ViT-B/16). The pretrained CLIP numbers should be reported for context.

Minor weaknesses:

  • No standard deviations or multiple seeds are reported, while the gains on CLIP are small (about 1-5 Rsum).
  • The "DSM w/o second-order" row in Table 3 is identical to the weighted cosine row in Table 1, although the normalization should make them different. Please clarify.
  • Figure 8 relies on t-SNE fitted independently for each model, so the comparison is only qualitative. Embedding comparison methods such as SPEC [4] could quantify which semantic clusters are captured differently by the DSM- and cosine-trained correspondence spaces.

[1] Chun et al. "Probabilistic Embeddings for Cross-Modal Retrieval", CVPR 2021

[2] Chun. "Improved Probabilistic Image-Text Representations", ICLR 2024

[3] Kim et al. "Hadamard Product for Low-rank Bilinear Pooling", ICLR 2017

[4] Jalali et al. "Towards an Explainable Comparison and Alignment of Feature Embeddings", ICML 2025

Questions for authors
  • What happens if only the positive or only the negative branch is used? And how does DSM compare with an unconstrained rank-r symmetric matrix?

  • If the structural argument is general, why does DSM not help with global features?

  • How does an independently trained cosine model perform on the near-tie pairs?

  • What is the zero-shot accuracy of the original pretrained CLIP with the same prompts?

  • Does ΔGcΔG_c (not the mean profile) show concept-dependent structure, and which checkpoint is used for the fingerprint analysis?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.