acceptodds
Withdrawn from ICLR 2027

Alignment with Different Pairs: Cross-Modal Similarity and Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We distinguish three questions about cross-modal alignment: what structure each modality has, whether the two modalities organize designated partners alike, and whether the deployed scorer ranks those partners highly. Sensitivity to random shuffling does not establish that relational agreement orders pairings by original-pair fidelity, and correlation with recall does not establish its value for checkpoint selection. We introduce hard negative re-pairing: an external sentence encoder guides a one-to-one reassignment to semantically similar captions from other images, while representations and, in retrieval experiments, all image–text scores remain fixed. On COCO, hard negative re-pairing with no original partners retained yields higher centered kernel alignment (CKA) than the mean under random re-pairing retaining most original partners. Across 22 global image–text checkpoints, neighborhood agreement retains a larger fraction of its original-minus-random contrast than assigned-pair recall. On Flickr30k, selecting checkpoints by validation agreement rather than validation recall costs 5.75 percentage points of held-out bidirectional Recall@1, despite a positive agreement–recall correlation. Relational agreement alone does not establish original-pair fidelity, and checkpoint-selection criteria should be judged by the held-out retrieval performance of their choices.

Primary area:
interpretability and explainable AI
Keywords:
cross-modal alignment, representation similarity, centered kernel alignment, vision-language models, cross-modal retrieval, checkpoint selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_SMjmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper substantiates several narrowly scoped conclusions quite thoroughly, but does not establish equally convincingly what important misconceptions these conclusions correct, what new mechanisms they reveal, or what practically meaningful research decisions they would change. I view it as a carefully executed diagnostic study with some methodological value, but limited substantive insight and demonstrated broader significance. My concern is primarily the importance of the contribution rather than the internal consistency of the experiments. Stronger evidence about meaningful semantic errors, objective-versus-optimization limitations, or the causes of the dataset-dependent behavior would be more consequential than further confirmations that the diagnostics are not interchangeable.

Critical strengths

The paper distinguishes marginal representation structure, cross-modal relational agreement, and retrieval performance. It introduces hard negative re-pairing to examine the relationship between structural agreement and original-pair fidelity, and compares checkpoint selection by neighborhood agreement against selection by validation recall. The framing is clear, and the interventions are reasonably controlled: one-to-one re-pairing preserves the sample collections, representations and retrieval scores remain fixed, and the external selector does not consult the evaluated models’ outcomes. These properties make the construction more informative than random shuffling alone.

Critical weaknesses
  1. The re-pairing experiment offers limited insight into semantic alignment. Equation 2 explicitly favors semantically similar replacements by minimizing the total cosine dissimilarity between original and replacement text partners in an external encoder’s embedding space.Consequently, lower original-pair fidelity need not imply weaker semantic alignment, and retained CKA may reflect a reasonable response to preserved semantics rather than a failure to detect genuine semantic errors. The authors acknowledge that original-pair fidelity does not assess replacement validity and that close substitutes are expected to preserve some relational agreement. However, the experiments do not distinguish valid alternative descriptions from semantically similar captions containing incorrect attributes or relations; the object-overlap analysis does not resolve this, and no human validity assessment was conducted. A targeted assessment separating these cases would clarify whether the diagnostic overlooks meaningful semantic errors. Without that distinction, the result remains primarily a caution about exact pair identity.
  2. Non-monotonicity does not establish a substantive failure of CKA-based pair recovery. The ordering reversal demonstrates that CKA does not globally order pairings by original-pair fidelity. However, an optimization objective need not rank every suboptimal candidate monotonically by recovery accuracy to be useful. The original pairing still achieves the highest CKA in the main comparison, approximately 0.719 versus 0.659 for complete semantic re-pairing.The FAQ experiment moves closer to evaluating practical usefulness, but selecting the highest-CKA candidate changes recovery accuracy by only −0.25, −0.73, and +0.41 percentage points relative to randomly selecting an already optimized run.These results do not establish whether imperfect recovery mainly reflects the objective, the solver, or limited variation within the candidate pool. Comparing the ground-truth pairing’s objective with the selected candidates under identical preprocessing and anchors would help clarify this issue. Figure 2 cannot substitute for that comparison because Appendix C.6 uses additional feature standardization.
  3. The checkpoint-selection result supports a narrower recommendation than the discussion suggests. The 5.75-percentage-point held-out Recall@1 gap provides concrete evidence that agreement-based selection can be inferior within the evaluated panel.The paired bootstrap and model-pool sensitivity analyses are useful, but they do not establish the reliability of the entire selection procedure across new validation samples, model pools, and distributions. Moreover, the experiment concerns trained retrievers for which the target validation metric is already available. Demonstrating that an indirect criterion performs worse in this setting offers limited guidance about situations where a proxy is needed. In particular, Section 6’s claim that relational agreement is worth using “only before a retriever exists” is broader than the evidence supports.The practical contribution would be stronger with a clearer account of the conditions under which agreement-based selection is informative or misleading.
  4. A potentially more informative finding remains insufficiently explained. I find the dataset-dependent sign reversal in Appendix E.5 more promising than the repeated demonstrations of non-equivalence: the agreement–recall correlation is negative on COCO but positive on Flickr30k for the same checkpoints, and this difference persists after matching sample sizes at k=50k=50.The follow-up analysis locates most of the reversal in cross-family ordering changes, but does not identify the underlying data or model factors responsible.Explaining this discrepancy could provide more substantive insight into when structural agreement is useful and why it sometimes diverges from retrieval performance.
Questions for authors
  1. How can you distinguish retained agreement due to valid alternative descriptions from insensitivity to genuine semantic errors?
  2. What explains the opposite agreement–recall correlations on COCO and Flickr30k at k=50k=50, even after matching sample sizes?
Reviewer_ZbhHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores the limitations of existing cross-modal alignment diagnostic methods in evaluating original-pair fidelity and retrieval performance. The authors introduce a controlled evaluation method called ‘hard negaive re-pairing’, which involves replacing original image-text pairs with semantically similar alternatives while keeping the underlying representations and retrieval scores constant. Experiments on COCO show that higher CKA scores or neighborhood agreement do not necessarily imply the original pairs are accurate, nor do they guarantee superior retrieval performance. Further experiments on Flickr30k demonstrate that selecting models based on neighborhood agreement can result in worse held-out retrieval performance compared to selecting models based on the Recall@1 metric.

Critical strengths

1.Propose more discriminative structured control for evaluating cross-modal alignment diagnostics. Previous studies, relied on random shuffling to assess cross-modal agreement, may conflate original-pair fidelity with shared semantic structure. By employing a hard negative re-pairing strategy, the authors examine the effect of semantically similar replacements while controlling for original-pair fidelity. Notably, even when the original pairs are entirely replaced, the CKA score remains higher than that of a random-shuffling baseline where 80% of the original pairs are preserved This provides a counterexample to the assumption that higher CKA necessarily indicates higher original-pair fidelity. 2.Connect the conceptual distinction between relational agreement and retrieval performance to practical checkpoint-selection outcomes. Across 22 global image–text models, the retention of neighborhood agreement, Recall@10, and Recall@1 exhibits a consistent ordering, demonstrating that the observed separation is not specific to a single model. The Flickr30k model-selection experiment further shows that even when metrics are positively correlated, selecting a model based on one metric does not necessarily guarantee strong performance on the target task.

Critical weaknesses

1.The paper constructs hard negatives by excluding original partners and identical normalized captions, but this does not guarantee that a replacement caption is not also a valid description of the reassigned image. Although the authors examine COCO object-category overlap to characterize the semantic similarity of replacements, they do not directly evaluate the factual accuracy of these captions as descriptions. Consequently, assigned-pair Recall calculated based on these re-assigned pairs may not fully reflect true semantic retrieval performance. This limitation constrains the generalizability of the experimental results to the evaluation of actual semantic matching quality. 2.Experiments on the Flickr30k dataset indicate that selecting models based on neighborhood agreement can lead to a significant loss in held-out Recall. However, the primary held-out evaluation is limited to one dataset and a fixed pool of 16 pretrained models. Although the paper provides retrospective COCO analyses and model-pool sensitivity analyses, the results are not consistent across all model families, and the correlation between agreement and recall differs substantially across datasets. The authors have acknowledged and discussed these phenomena; thus, they do not invalidate the core conclusions. Nevertheless, the current evidence remains insufficient to determine whether these effects persist to a similar degree across other datasets or different pools of models.

Questions for authors

1.Could the authors conduct a more rigorous evaluation of image–caption semantic validity to estimate how many replacement captions remain valid descriptions of their reassigned images under complete hard negative re-pairing? If semantically valid and invalid replacements were evaluated separately, would the observed separation between CKA and Recall still hold? 2. In the Flickr30k model-selection experiment, the two selection criteria choose SigLIP models with different model sizes and input resolutions (B/16 256 vs. L/16 384). It remains unclear whether neighborhood agreement would still select a worse retriever when candidate models have comparable capacities and input resolutions. Could the authors further investigate this setting to clarify the reliability and scope of neighborhood agreement as a model-selection criterion?

Reviewer_xrcmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether cross-modal representation similarity reliably reflects original image-text pair fidelity and retrieval performance. It distinguishes marginal representation structure, relational agreement, and retrieval quality, and introduces hard negative re-pairing to test whether relational diagnostics can distinguish original pairs from semantically similar alternatives. Experiments show that CKA can remain high even when no original pairs are retained, and neighborhood agreement preserves substantially more of its original-to-random contrast than retrieval recall under hard negative re-pairing. In a checkpoint-selection experiment on Flickr30k, selecting a model by neighborhood agreement leads to a 5.75 percentage-point reduction in held-out bidirectional Recall@1 compared with selection by validation recall. The paper highlights important limitations of using relational similarity as a proxy for retrieval quality.

Critical strengths
  1. The paper makes a useful conceptual distinction between marginal representation geometry, cross-modal relational agreement, and retrieval performance. Its controlled interventions help clarify that these quantities can vary independently, providing a more careful interpretation of commonly used alignment diagnostics.

  2. The hard negative re-pairing experiments provide an informative counterexample to the assumption that high relational agreement necessarily indicates faithful original-pair matching. The evaluation covers multiple image-text encoders and includes a held-out checkpoint-selection experiment with paired bootstrap confidence intervals, connecting the diagnostic analysis to a practical retrieval outcome.

Critical weaknesses
  1. The empirical scope of the checkpoint-selection conclusion remains limited. Although the re-pairing analysis covers 22 global image-text checkpoints, the selection experiment considers 16 CLIP, SigLIP, and SigLIP2 checkpoints on a single Flickr30k split. The reported 5.75-point gap is meaningful, but it does not establish how reliably neighborhood agreement performs across datasets, additional model families, or checkpoints from the same training run. The paper also leaves the extension of the selection result to CKA unresolved.

  2. The interpretation of original-pair fidelity requires further qualification. Hard negative re-pairing deliberately assigns semantically similar captions to different images, demonstrating that relational diagnostics can remain high when designated correspondences change. However, exact original-pair retention is not equivalent to semantic correctness: an alternative caption may still describe an image accurately. Although the annotation-based control strengthens the robustness of the counterexample, the evaluation does not fully establish how often the replacement pairs are genuinely invalid under semantic relevance judgments. Consequently, the experiments support a limitation of these diagnostics, but the broader implications for cross-modal alignment evaluation and practical model selection remain insufficiently established.

Questions for authors
  1. Can the authors validate the checkpoint-selection result on additional retrieval datasets and model families, including checkpoints sampled from a single training trajectory? It would also be useful to evaluate CKA-based checkpoint selection directly.

  2. How often do the hard-negative replacement captions remain semantically valid descriptions of their assigned images? Can the authors supplement exact-pair fidelity with human-annotated or multi-positive relevance judgments to distinguish incorrect pairings from alternative but valid descriptions?

Reviewer_BoeyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies what cross-modal similarity diagnostics, such as CKA and neighborhood agreement, tell us about image–text pairings and retrieval quality. Its main argument is that these diagnostics are often read as evidence about the original image and caption pairs, or about retrieval performance, when they measure something different.

Critical strengths

None.

Critical weaknesses

The writing in the paper makes it extremely difficult to understand the paper after multiple readings.

L014: “Sensitivity to random shuffling does not establish that relational agreement orders pairings by original-pair fidelity, and correlation with recall does not establish its value for checkpoint selection.” —> Sensitivity to random shuffling, how is fidelity defined here, which checkpoint selection?

L021: What does “original-minus-random contrast” and “assigned pair recall” mean ?

L020: “Global image–text checkpoints” -> what is global image-text checkpoints,

L030: what odes it mean by “organize corresponding examples similarly” ?

L042: The paper gets into technical terms like local intrinsic dimensionality (LID), neighbor radii, and effective rank of the covariance spectrum in introduction without giving the reader proper background and what these terms mean.

L049: The decrease in CKA under random shuffling motivates recovering image–caption partners by maximizing CKA (Maniparambil et al., 2024). That control does not establish an ordering by original-pair fidelity—the fraction of original pairs retained—because it leaves semantically similar replacements untested. —> very difficult to understand what the authors are saying in this.

L129: What does it mean to say the “pick can be verified”?

The paper fails to give a high level overview of the goal of the paper. Without that, the details add little to the reader’s understanding. The authors need to significantly improve the writing in the paper.

Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.