acceptodds
Under review as a conference paper at ICLR 2027

The Retrieval–Grounding Tradeoff in Cross-Modal Connectors, and How to Navigate It

Abstract

We ask whether a topology-preserving, invertible flow connector traces a better alignment-vs-structure frontier than projection heads for cross-modal (image<->text) translation, under the Aristotelian Representation Hypothesis (Gröger et al., 2026) that neural representations share local-neighborhood but not global-metric structure. Across ten connectors and two pre-registered testbeds — within-CLIP (pre-aligned) and cross-model (independent encoders) — the flow does not win: transport flows land off the target manifold; a guided conditional-generation flow buys single-shot retrieval but not manifold fidelity; and a one-line InfoNCE term on the same MLP beats every flow (42.9 vs. 27.9) at a fraction of the cost. Our central result is causal: a connector's grounding (how decodably image-specific its output is) is driven by proximity to the target manifold, which we establish with a projection/shuffle/intervention triad — so optimizing a connector for retrieval pushes its outputs off the manifold, a marginal manifold cost of discriminative pressure that is a hard retrieval–grounding exclusion when the spaces are pre-aligned and softens to a frontier otherwise. The mechanism is prescriptive: a train-time nearest-real-target pull on a contrastive objective navigates the tradeoff, filling the previously-empty top-right and Pareto-dominating every learned connector in both regimes (+11 R@1 over regression at equal grounding), and — the key control — beating post-hoc DeCap-style and subspace projections that target the same proximity. We scope all claims and release full pre-registration, code, and per-seed results.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
cross-modal representation learning, flow matching, modality gap, retrieval–grounding tradeoff, multimodal connectors

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_2dCZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the relationship between retrieval performance and grounding quality in cross-modal connectors that translate between frozen image and text embedding spaces. The authors initially hypothesize that invertible, topology-preserving flow-based connectors may provide a better alignment–structure tradeoff than standard projection heads, and evaluate multiple regression, orthogonal, and flow-based mappings in both a within-CLIP setting and a cross-model setting with independently trained encoders. Contrary to the initial hypothesis, deterministic flows do not outperform simple regression mappings, while a conditional generative flow improves retrieval but still produces outputs that are relatively far from the target embedding manifold. A simple InfoNCE term added to an MLP substantially outperforms the flow-based connectors in retrieval. The paper then argues that retrieval and grounding are governed by a tradeoff between discriminability and proximity to the target manifold. Grounding is measured using a frozen DeCap-style text decoder and is found to correlate strongly with nearest-target embedding distance. Based on shuffle and subspace interventions, the authors argue that proximity to real target embeddings, rather than simple subspace membership, is the relevant mechanism. They subsequently propose F6, which augments a contrastive mapping objective with a train-time nearest-real-target pull. By varying the strength of this term, F6 traces a retrieval–grounding frontier and outperforms standard learned connectors and post-hoc projection baselines at comparable grounding levels.

Critical strengths

The main strengths of the paper are that (i) it conducts a relatively unusual hypothesis-driven study that reports several negative results rather than forcing the original flow-based hypothesis to succeed, and (ii) it provides a useful empirical perspective on the distinction between retrieval-optimal and decoder-compatible cross-modal representations.

First, the paper is commendably willing to falsify its own initial hypothesis. The experiments show that deterministic and stochastic flow connectors do not produce a clearly superior retrieval–structure frontier, and that a simple contrastive signal added to an MLP is substantially more effective for retrieval than the more complicated flow machinery. In the cross-model setting, adding InfoNCE raises MLP R@1 from 22.4 to 42.9, whereas the conditional generative flow reaches 27.9. The authors also expose the pre-registered kill conditions and explicitly report which ones failed. This makes the empirical narrative more credible than a purely post-hoc comparison built around only successful results.

Second, the paper provides an interesting analysis of retrieval versus grounding. Across the tested connectors, grounding strongly tracks proximity to real target embeddings, and the authors attempt to distinguish this effect from generic distributional or subspace membership through shuffle and projection interventions. The resulting F6 objective is also a useful practical consequence of the analysis: combining a contrastive retrieval objective with a nearest-real-target pull produces a controllable retrieval–grounding frontier and improves the tradeoff relative to post-hoc DeCap-style and subspace projections.

Critical weaknesses

The first major concern is the paper’s presentation and submission compliance. The manuscript is extremely compressed and difficult to follow even after careful reading. It introduces a large number of internal identifiers and experimental labels—e.g., F1/F2/F3/F5/F6, KC1/KC2/KC3/KC4/KC-F5/KC-NAV, and P-NAV2—while frequently moving between pre-registered hypotheses, failed kill conditions, revised interpretations, and supplementary results. This makes it unnecessarily difficult to reconstruct the main scientific narrative and distinguish the original hypotheses from the claims that ultimately survive the experiments. In addition, the submitted manuscript appears to omit the required AI Use Statement: the main text proceeds from the limitations and reproducibility discussion directly to the references without such a section. :chatgpt-content-reference{index="0"} I view the latter primarily as a submission-policy/compliance issue rather than a scientific flaw, but together with the unusually dense presentation it substantially reduces the clarity and professionalism of the submission.

The second major concern is that the paper’s central “grounding” and causal claims appear broader than the evidence currently supports. Grounding is operationalized using a DeCap-style decoder trained on text embeddings and measured through CIDEr, so it is natural that connector outputs closer to the target text-embedding manifold are easier for this decoder to interpret. The shuffle and subspace interventions are useful controls, and the authors further repeat the analysis with a noise-augmented decoder, but both probes remain within essentially the same decoder-based measurement paradigm. :chatgpt-content-reference{index="1"} :chatgpt-content-reference{index="2"} Moreover, the strongest retrieval–grounding exclusion is specific to the pre-aligned within-CLIP regime, while the cross-model setting exhibits a considerably softer frontier; the authors themselves later narrow the more general conclusion to a manifold-fidelity regularity. :chatgpt-content-reference{index="3"} Although the DINOv2→bge replication is helpful, the experiments still focus on text target spaces, COCO/Flickr data, and closely related grounding probes. :chatgpt-content-reference{index="4"} I therefore find the empirical observation interesting, but the stronger framing of a general causal account of cross-modal grounding is not yet fully established.

Questions for authors

How general is the proposed grounding mechanism beyond the DeCap-style decoder? Since grounding is operationalized as the CIDEr improvement of a decoder trained specifically on text embeddings, proximity to real text embeddings may naturally benefit this decoder. Could the authors validate the same connector ordering and proximity relationship using a substantially different decoder architecture, a separately trained generative model, or a non-decoder-based grounding metric? Such an experiment would help distinguish a general representational property from compatibility with the chosen decoder family.

Could the authors moderate or more precisely scope the causal claim? The shuffle experiment establishes that image–output correspondence matters, and the subspace intervention shows that removing out-of-subspace energy alone does not improve grounding. However, these controls do not appear to independently manipulate nearest-real-target proximity while holding other semantic and geometric properties fixed. Could the authors provide an intervention that directly moves outputs toward or away from matched real targets while approximately preserving retrieval-relevant directions, and measure the resulting change in grounding?

How sensitive is F6 to the construction and size of the real-target memory used for the nearest-target pull? The proposed objective relies on nearest real target embeddings during training. Could the authors report how performance changes when the memory is substantially subsampled, restricted to semantically unrelated examples, or constructed from a shifted dataset? This would clarify both the scalability of F6 and whether its gains depend on dense access to the training target manifold.

What is the computational cost of F6 relative to the simple contrastive MLP? One of the paper’s central findings is that a one-line InfoNCE objective is cheaper and more effective than flow-based connectors. However, F6 introduces nearest-real-target search during training. Could the authors report training FLOPs, wall-clock time, and memory or indexing cost for the contrastive MLP, F5, and F6?

How stable are the retrieval–grounding correlations given that the headline correlation is computed over only approximately ten connector families? The paper reports Spearman correlations such as −0.89 and −0.98, but these are based on a small number of heterogeneous connector classes rather than a large independent sample. Could the authors quantify uncertainty over the connector-level correlation or repeat the analysis across substantially more hyperparameterized variants within each connector family?

Can the authors clarify which conclusion is intended to generalize across regimes? Within-CLIP shows a strong exclusion between high retrieval and high grounding, whereas the cross-model setting contains mappings with both relatively strong retrieval and grounding. The manuscript eventually narrows the universal statement to “manifold-fidelity regularity,” but this qualification appears relatively late. Reframing the paper around the narrower claim earlier would make the contribution easier to interpret.

Could the authors simplify the exposition and terminology substantially? The manuscript introduces many identifiers and pre-registration labels—F1/F2/F3/F5/F6, KC1/KC2/KC3/KC4/KC-F5/KC-NAV, P-NAV2, several manifold statistics, and two regimes—in a very compressed presentation. Even after careful reading, tracking which hypotheses are original, falsified, revised, or retained requires substantial effort. A conventional separation into hypotheses, experimental results, revised mechanism, and final method would substantially improve accessibility.

Reviewer_Ek9NOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1(−1)
Read review
Paper summary

This paper studies the mechanism of connectors that map image embeddings to a text embedding space. It also tests whether invertible flow-based connectors outperform simple unconstrained projection heads, and finds that they do not. Additionally, the paper claims that the performance of grounding is driven by proximity to real text embedding, and therefore, proposes F6 that uses a regularization term to pull the outputs of connectors to the nearest real target embeddings to maintain the performance.

Critical strengths
  1. F6 is intuitive and improves the retrieval performance without sacrificing the grounding performance too much in the cross-model setting.
  2. The failures of flow-based connectors are reported fairly. The key results are tested on multiple backbones and datasets with multiple seeds.
Critical weaknesses
  1. The central results are largely known. Grounding is measured with a decoder that is trained on real text embeddings, so it is expected to improve the connector's grounding performance by pulling the outputs to those embeddings. Such a phenomenon has already been studied in DeCap, which aims to bridge exactly the gap between outputs and real text embeddings.

  2. In Figure 3, the performance curves of F6 and DeCap-proj barely overlap in retrieval. It is hard to assess the grounding performance of the two methods at the same retrieval level, especially for the within-CLIP setting.

  3. The paper's writing is hard to follow. It mixes three storylines together: the failure of invertible flow-based connectors, the retrieval-grounding tradeoff, and the introduction of F6. The title and abstract highlight different ones. Much of the text reads like an experiment log, with labels like RQ2, KC1, KC-SEL, P-NAV2 rarely explained in the main content. The introduction also gives little background information without stating the problems the current connectors have or explaining the hypothesis it is based on.

Questions for authors
  1. Could you extend F6's sweep toward lower retrieval level and DeCap-proj's sweep toward higher, so the two curves can be compared at matched R@1 on both testbeds?
  2. The retrieval-grounding tradeoff rests on a premise that a single output should serve grounding and retrieval simultaneously. Is it possible to use the output for retrieval and its DeCap-projected version for grounding? What prevents this if not?
Reviewer_4MEeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I found the paper difficult to follow. Motivated by the Aristotelian Representation Hypothesis, the paper investigates whether invertible, topology-preserving flow connectors achieve a better alignment–structure tradeoff than unconstrained projection heads in cross-modal translation. The authors compare several deterministic and generative flow variants with identity, Procrustes, ridge regression, MLP, and contrastive baselines in both a pre-aligned within-CLIP setting and an unaligned cross-model setting. The results largely reject the initial hypothesis: the flow-based connectors do not outperform regression-based methods, and invertibility does not provide a reliable certificate of grounding.

Critical strengths
  1. The experiments part seems comprehensive with a lot of different baselines; aligned within-CLIP; mapping within COCO, and transferring within Flickr30k.
  2. The authors use shuffle, subspace projection, and nearest-target intervention to control the experiments, it might be interesting in some way.
Critical weaknesses
  1. This paper is hard to follow. So if this paper want to propose a new flow-based “connector”, it could be logically, however, the experiment itself reject this hypothesis. If this paper just giving a new loss term with train-time nearest-real-target, it is not novel, a lot of previous works, ranging from cross languages translation, embedding convertor, and some other cross-* works.
  2. A lot of AI-slops, no AI usage claim: “What we do not claim” in L077; “image→5captions” in L187; a lot of slash in the paragraph; the mechanism is prescriptive in the L356; the notation of F1-F6 in the subscetion titles. No formal Discussion section. Most importantly, no AI usage section which required by this year’s venue.
Questions for authors
  1. I encourage the authors to conduct a careful editorial revision, particularly to clarify the definitions, organize the contributions, and strengthen the transitions between the flow hypothesis, the grounding analysis, and the proposed F6 objective (also, delete the expression like F1-F6).
  2. If the authors want to present a new flow-based work, it would be novel, but it needs more work to prove it is stronger than baselines; if the authors want to just give a near-to-real-target, the author should explain what is the increment things comparing to those old related works.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.