acceptodds
Under review as a conference paper at ICLR 2027

Ground, Relate, and Match: Optimal-Transport-Guided Graph Grounding for Compositional Image–Text Matching

Abstract

Dual-encoder vision-language models such as CLIP achieve strong zero-shot image-text matching, but global similarity can overlook fine-grained compositional structure, including entity binding, attributes, and directed semantic relations. Existing training-free methods introduce local evidence, yet often score components independently or reason directly over fractional alignments that do not represent coherent entity groundings. We introduce LEM (Local Evidence Matters), a training-free framework for compositional image-text matching through structured graph alignment. LEM parses paired captions into canonical directed semantic graphs and constructs a caption-neutral region graph from localized image proposals. A dustbin-augmented entropic partial optimal transport problem identifies promising entity-region correspondences while allowing entities and regions to remain unmatched. LEM converts the resulting transport plan into a compact set of injective grounding hypotheses and reweights them using the original entity-identity evidence. It then evaluates typed entity properties and directed relations over each coherent grounding and marginalizes their semantic evidence over the same support for both captions. Candidate-conditioned comparisons distinguish competing attributes, predicates, and argument directions, while independent absolute scores remain comparable across images.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
image–text matching, visual grounding, optimal transport, graph alignment, structured inference, training-free learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qk5JOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose LEM (Local Evidence Matters), a training-free method for compositional image-text matching. The key idea is to incorporate structured local evidence into VLMs such as CLIP. Specifically, LEM aligns caption entities and relations with image regions using structured optimal transport. In experiments, LEM improves some results on Winoground and SugarCrepe.

Critical strengths

The LEM method is training-free and it can be easily integrated with existing VLMs without modification of the backbone. This makes LEM applicable to frozen pretrained models such as CLIP and NegCLIP.

Critical weaknesses
  1. The performance improvements are limited and inconsistent across tasks. While LEM shows performance gains on several subtasks of each benchmark, the improvements are not consistent. Performance drops are significant on some metrics such as relation replacement.

  2. The experimental evaluation is small scale. The experiments mainly focus on Winoground and SugarCrepe. Evaluation on additional compositional benchmarks, as well as standard image-text retrieval benchmarks such as COCO or Flickr30K, would help demonstrate generalization and verify that LEM does not hurt general retrieval performance.

  3. The method appears to require some benchmark-specific tuning. Several important weights are selected separately for each benchmark and backbone, raising a question of how well the method would work on a new dataset without additional calibration.

  4. The additional complexity is not sufficiently discussed. LEM introduces Grounding DINO and iterative OT optimization on top of CLIP. Reporting runtime and computational overhead would help clarify the practical tradeoff between performance and complexity.

Questions for authors

I do not have specific questions, but please address the weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer qk5J

Thank you for the review. We take the four weaknesses in order.

W1. Limited and inconsistent improvements

We agree. Under 5-fold cross-validated weight selection (see W3), 5 of the 12 comparisons with the global baselines in Table 1 are significant gains and none is a significant loss.

One result is consistent across backbones: with the submission weights (selected on Winoground with CLIP B/32, w_glob = 0.55) transferred unchanged to OpenCLIP ViT-L/14, SigLIP, SigLIP2 and EVA02-L/14, Winoground Image improves on all five backbones (+3.3 to +8.0 points, significant on four). The relation-replacement drop is just as consistent under the same weights (−2.8 to −5.1 points). Both come from the directed relation term. We will narrow the claims to paired-image discrimination and list the drop as a limitation.

W2. Evaluation scale and retrieval

We added the full seven-subset SugarCrepe, ARO-VG, MMVP-VLM, the four backbones above and four more baselines (CE-CLIP, BindCLIP, BLIP ITM, VQAScore). LEM does not improve ARO-VG or MMVP-VLM, and on SugarCrepe only the node-only variant beats the global baseline on average (76.94 vs. 75.41). These runs use the submission weights, not cross-validated selection; we will rerun them under that protocol. We have not run COCO or Flickr30K retrieval and make no claim about it. The revision will test LEM as a re-ranker over the global top-k.

W3. Benchmark-specific tuning

You are right. The submission chose its weights per benchmark from sweeps on the evaluation sets (Appendix C.7). All Table 1 results are now selected by 5-fold cross-validation within each benchmark: each fold selects one of 630 weight configurations on the other four folds, and only held-out predictions are scored. The multi-backbone and additional-benchmark runs still use the submission weights, selected on Winoground and SugarCrepe with CLIP B/32, and we label them as such.

The weights remain task dependent: cross-validation picks w_rel = 0 on Swap-Att and w_glob ≥ 0.90 on Repl-Rel, and the NegCLIP folds on Winoground disagree. A new task would need a small calibration set, and we will say so. The exception is Winoground Image, where the CLIP B/32 weights transfer unchanged to four other backbones (W1).

W4. Runtime

On one Tesla T4, over 100 Winoground image-caption pairs, the mean time per pair is 33 ms for global CLIP, 562 ms for ComCLIP and 710 ms for LEM (medians 34, 569 and 614 ms). Grounding DINO takes 429 ms, crop encoding 173 ms, OT 20 ms, and the remaining steps 88 ms. LEM is about 21× slower than global CLIP and 26% slower than ComCLIP on average. Detection dominates; OT is about 3% of the cost. We will report this cost next to the accuracy results.

Reviewer_FybeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Global image-text similarity can miss compositional differences, such as which attribute belongs to which object or which entity acts on which. This paper improves image-text matching by incorporating structured local evidence.

The authors propose LEM, which parses the caption into a relation graph and proposes image regions with Grounding DINO. It then aligns the two with unbalanced optimal transport, using an attribute-aware node cost and a quadratic relation cost that compares a relation prompt with its reversed version. The resulting local score is fused with the CLIP score.

The authors evaluate LEM on Winoground and three SugarCrepe tasks using CLIP and NegCLIP. Ablations examine entity-attribute weighting, relation weighting, and fusion-weight transfer to a larger backbone. A localization error analysis examines Grounding DINO proposals against COCO annotations.

Critical strengths
  1. Joint entity and relation alignment is well motivated. Correctly grounding individual objects does not establish their relationships. The proposed formulation allows relational evidence to influence entity assignments.

  2. The approach requires no model training. As a training-free approach, it can work for downstream benchmarks out of the box.

Critical weaknesses
  1. It's hard to tell what the paper itself contributes. Compositional matching is not a new task and has been explored in a similar way by ComCLIP. Within the pipeline, Grounding DINO does the localization, Stanza does the parsing, and every cost is just a rescaled dual-encoder similarity. The OT alignment sits on top of all this as a fairly thin layer. I can't tell what the paper contributes beyond the pre-packaged capabilities from the pretrained models it uses.

  2. The results do not demonstrate a convincing improvement in compositional matching. Only one of twelve comparisons against global baselines in Table 1 shows a significant gain. Relation replacement, which directly tests a capability the method aims to improve, drops significantly on both backbones. Increasing the relation weight also generally hurts this task. I find it difficult to reconcile these results with the claimed benefit of explicit relation modeling.

  3. The evaluation is too limited to establish broader effectiveness. The main results cover only Winoground and three SugarCrepe tasks, with ComCLIP as the only local compositional baseline. The larger-backbone experiment is useful, but broader comparisons with stronger compositional methods and evaluation on additional benchmarks are needed. Given the inconsistent gains in the tested settings, the evidence does not establish that LEM is a reliable, broadly applicable improvement over existing matching methods.

Questions for authors
  1. How were the evaluation-specific weights selected? Were the weights (e.g., fusion and relation weights) chosen on separate held-out data or on the evaluation sets themselves?

  2. How much does each component contribute? Could the authors ablate the key components, e.g., balanced vs. unbalanced OT and removing the quadratic relation term? An oracle experiment with ground-truth boxes would also help separate the method's contribution from detector errors.

  3. What is the inference overhead? How does the per-pair cost compare with CLIP and ComCLIP, and which stage (detection, crop encoding, or OT) dominates it?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Fybe

Thank you for the review. We answer your three questions first, since they changed the paper most.

Q1. How were the weights selected?

On the evaluation sets. Appendix C.7 states that α, λ_attr and w_glob were chosen per benchmark and backbone from the sweeps in §4.3, and those sweeps were run on Winoground and SugarCrepe. Only the shared parameters in Table 3 were set on separate examples. This is not held-out selection, and you are right to question it.

We redid the main results with 5-fold cross-validation inside each benchmark, keeping each Winoground group in one fold. Each fold picks one of 630 configurations (5 attribute weights × 6 relation weights × 21 fusion weights) on the other four folds, and only held-out predictions are scored. Under exact McNemar tests, the significant gains for CLIP are Winoground Group 7.50 → 12.00 (p = 0.022), Image 9.25 → 21.00 (p < 0.001) and Swap-Obj 63.01 → 69.51 (p = 0.004). For NegCLIP, only Image (11.25 → 18.25, p < 0.001) and Repl-Rel (76.46 → 77.52, p = 0.028) are significant. This table replaces Table 1.

Image rises from 14.00 in the submission to 21.00. Of this, 1.75 points come from a code refactor made after the submitted runs (the submission weights give 15.75 with the current code), and 5.25 points come from the wider search: all five CLIP folds choose λ_attr = 1.0, w_rel = 0.6 and w_glob = 0.25, which gives the local score more weight than the submission's w_glob = 0.55. ComCLIP numbers also change slightly because we now use its official code.

Q2. How much does each component contribute?

  • Unbalanced vs. balanced OT: sweeping the marginal penalty from 0.01 to balanced OT, the default behaves like balanced OT, and stronger relaxation significantly hurts Swap-Att on both backbones. Unbalancedness contributes nothing, so we remove it from the title and the contributions.
  • Relation term: removing it brings Winoground Image back to 9.75 (global baseline 9.25) and raises the SugarCrepe average. The Winoground gain comes from this term, and so do most SugarCrepe losses.
  • Oracle with ground-truth COCO boxes on SugarCrepe: Swap-Obj 65.85, Swap-Att 62.76 and Repl-Rel 67.00, against 61.38, 68.32 and 67.21 with Grounding DINO boxes (Table 1) and 63.01, 66.52 and 71.34 for the global baseline. Better boxes help Swap-Obj (+4.47) but leave Repl-Rel unchanged, so the Repl-Rel loss comes from the relation evidence, not from detection. Ground-truth boxes lower Swap-Att, and we will analyse this in the revision. Winoground has no box annotations.

Q3. What is the inference overhead?

On one Tesla T4, over 100 Winoground image-caption pairs, the mean time per pair is 33 ms for global CLIP, 562 ms for ComCLIP and 710 ms for LEM. Inside LEM: Grounding DINO 429 ms, crop encoding 173 ms, OT 20 ms, other steps 88 ms. Detection dominates; OT is about 3% of the cost.

W1. What does the paper contribute?

After these ablations, less than we claimed. The contribution that remains is a directed relation score computed over consistent entity groundings, which improves image-side discrimination without training. With the submission weights (selected on Winoground with CLIP B/32, w_glob = 0.55) transferred unchanged to OpenCLIP ViT-L/14, SigLIP, SigLIP2 and EVA02-L/14, Winoground Image improves on all five backbones (+3.3 to +8.0 points, significant on four) and exceeds ComCLIP on all five. We will not claim more than that.

W2. Only one of twelve comparisons is a significant gain

Under cross-validation, 5 of the 12 comparisons with the global baselines are significant gains and none is a significant loss. Against ComCLIP, CLIP+LEM is significantly lower on Repl-Rel (71.19 vs. 73.19, p = 0.018). With a fixed relation weight, the term that gives the Winoground Image gain also hurts relation replacement: in Repl-Rel the predicate changes but the roles do not, so a direction score only adds noise. A symmetric relation cost removes the drop but also the Winoground Image gain. We will present this as a trade-off.

W3. Limited evaluation

We added the full seven-subset SugarCrepe, ARO-VG, MMVP-VLM, the four backbones above, and CE-CLIP, BindCLIP, BLIP ITM and VQAScore. The results are mixed. LEM beats ComCLIP, the training-free local baseline, on Winoground Image, but with the relation term it is below the global baseline on six of seven SugarCrepe subsets and gives no gain on ARO-VG or MMVP-VLM. These runs use the submission weights, not cross-validated selection; we will rerun them under that protocol. We will limit the claims to paired-image discrimination.

If these responses address your concerns, we would be grateful if you would consider updating your score.

I appreciate the authors providing additional ablation study and analysis, which clarified some of my confusions. However, the new results suggest an even narrower contribution than originally reported and major changes to the original claims, with mixed effect on the key components. I choose to maintain my original scores.

Reviewer_XdLCOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies compositional image-text matching: determining whether an image supports the entity identities, attribute bindings, and subject-object relations expressed in a caption. The proposed training-free method, LEM, parses captions into entity-relation graphs, uses Grounding DINO to extract image regions, and computes node and relation matching costs with frozen CLIP or NegCLIP encoders. Structured unbalanced optimal transport then produces soft correspondences between regions and textual entities, and the resulting local matching score is fused with global image-text similarity.

The main methodological addition is to place node correspondence and pairwise relational evidence within a shared optimization objective, so that entity assignments are jointly constrained by relation costs while poorly supported correspondences can receive less transport mass. The evaluation focuses on Winoground and three SugarCrepe subtasks, with improvements in selected settings but mixed results overall.

Critical strengths
  1. Clear motivation and a reasonable training-free formulation. Global image-text similarity can miss attribute binding and relation direction, while independent local matching does not necessarily ensure that multiple correspondences are jointly consistent. Combining node and relational evidence addresses this limitation directly. Keeping the detector, parser, and encoders frozen makes the approach applicable as an inference-time enhancement, and unbalanced transport provides a reasonable mechanism for handling missed detections, duplicate boxes, and unreliable regions. These aspects give the method a concrete rationale and potential practical value.

  2. Demonstrated gains in some settings, accompanied by statistical analysis. For example, on the CLIP backbone, Winoground Image Score improves from 9.25% to 14.00%, a statistically significant increase. Reporting confidence intervals and significance tests helps distinguish reliable improvements from uncertain gains. This supports the usefulness of the approach for at least some compositional matching settings.

Critical weaknesses
  1. The methodological contribution and its independent benefit are not yet sufficiently established. Compositional matching, local-region scoring, and training-free inference already have direct precedents, particularly ComCLIP, which is closely related in both task and overall approach. The contribution therefore rests primarily on the proposed costs and joint unbalanced alignment. More convincing comparisons are needed to establish a substantive advantage over simpler local aggregation and alternative structured alignment approaches.

The main table directly compares against only one prior local compositional method, ComCLIP, and the evaluation is concentrated on Winoground and three SugarCrepe subtasks. Controlled comparisons using the same detector, crops, prompts, and encoder are needed to distinguish the benefit of the joint optimization from that of adding local evidence. The additional computation also matters: training-free inference still requires detection, encoding multiple regions and region pairs, and iterative optimization. Given the modest and inconsistent improvements, clearer evidence of the efficiency-performance trade-off is needed to establish practical value.

  1. The gains are inconsistent, and the relation results are at odds with a central motivation of the method. On SugarCrepe relation replacement, CLIP accuracy decreases from 71.34% to 67.21%, and NegCLIP accuracy decreases from 76.46% to 74.25%; both decreases are statistically significant. Winoground Text Score also decreases. The evidence therefore supports benefits in selected compositional matching settings rather than a general improvement in relational understanding.

The source of the relational evidence also deserves closer examination. The pair-crop score still relies on a frozen encoder comparing forward and reversed relation descriptions, while the directional gate mainly checks whether individual region identities agree with the assigned subject and object. In my view, this mechanism reorganizes existing evidence but does not necessarily provide stronger action or relation recognition. In particular, when the subject and object belong to the same entity category, comparing their bare entity names cannot distinguish their roles. This limitation is consequential given the observed degradation on relation replacement.

Questions for authors

Overall assessment: The paper has a reasonable motivation and a concrete methodological design, with evidence of effectiveness on some metrics. However, the necessity and independent contribution of the added alignment mechanism are not sufficiently isolated, the direct comparisons are limited, and the relation modeling does not yield consistent gains on relation replacement.

Can you isolate the contribution of joint structured alignment from the additional local evidence by comparing it against independent matching and simple aggregation using the same detector, regions, prompts, and encoder? Which results demonstrate that the joint optimization itself is necessary?

What explains the statistically significant degradation on SugarCrepe relation replacement for both backbones? Can the analysis distinguish limitations of the pair-crop relation score, the endpoint-identity gate, and the alignment procedure? In particular, how does the method handle subject and object roles when their bare entity names are identical?

What is the end-to-end inference cost relative to the global baseline and ComCLIP under comparable settings, including detection, region-pair encoding, and optimization? This comparison would help assess whether the observed gains justify the additional computation.

1 comment
AuthorsOpenReview ↗

Response to Reviewer XdLC

Thank you for the careful review. We answer your three questions directly.

Q1. Is the joint optimization itself necessary?

Not in the form we claimed. With the same detections, crops, prompts and CLIP ViT-B/32, the gain comes from the directed relation term evaluated over consistent groundings, not from the joint OT itself.

  • Node level, without the relation term: mean pooling, max pooling and Hungarian assignment give Winoground Group 7.25, 8.50 and 7.75, against 8.00 for OT. None of these differences is significant. OT beats only mean pooling, on Swap-Att (68.32 vs. 62.31, p < 0.001).
  • Relation level: removing the relation term brings Winoground Image back to 9.75, close to the global baseline (9.25). Applying the same relation scores after node matching, instead of inside the optimization, lowers Group by 5 and Image by 4 of the 400 examples, and is equal or slightly better on SugarCrepe.

These ablations use the submission configuration, so the node-only numbers match §4.3 and Table 1 (Group 8.00, Swap-Att 68.32). We will drop the claim that the joint optimization is necessary.

Q2. What causes the drop on relation replacement?

The directed relation score. Removing the endpoint gate, removing pair crops, or applying relation scores after node matching each changes Repl-Rel by less than one point. Making the relation cost symmetric removes the drop (Repl-Rel stays at 70.1–70.8 for every relation weight, while the directed version falls from 70.70 to below 65), but it also removes the Winoground Image gain. With ground-truth COCO boxes, Repl-Rel is still 67.00 against 71.34, so detection is not the cause.

We agree with your reading: LEM reorganizes the encoder's evidence and does not add relation recognition. In Repl-Rel the predicate changes but the roles do not, so a direction score only adds noise. With fixed weights the drop is consistent, −2.8 to −5.1 points on all five backbones we tested (CLIP ViT-B/32, OpenCLIP ViT-L/14, SigLIP, SigLIP2, EVA02-L/14). Under cross-validated weights it disappears for CLIP (71.19 vs. 71.34, p = 0.89), but only because the selected fusion weight is 0.90–0.95, and LEM is then still below ComCLIP (71.19 vs. 73.19, p = 0.018). We will remove the claim of improved relational understanding and present the relation term as a trade-off.

Identical subject and object names: the method does not separate their roles. The endpoint gate scores both assignments equally; only the directed pair-crop prompt differs. The case is rare (33 of 400 Winoground examples, 8 of 1,406 Repl-Rel examples), LEM is not worse than the baseline on either subset (Winoground Image 3/33 vs. 0/33; Repl-Rel 8/8 vs. 7/8), and it does not explain the Repl-Rel drop. We will list it as a limitation.

Winoground Text decreases under cross-validation (CLIP 30.25 → 27.25, NegCLIP 30.50 → 27.25), though not significantly (p = 0.25 and 0.16). We will report it.

Q3. What is the inference cost?

On one Tesla T4, over 100 Winoground image-caption pairs, the mean time per pair is 33 ms for global CLIP, 562 ms for ComCLIP and 710 ms for LEM (medians 34, 569 and 614 ms). Inside LEM: Grounding DINO 429 ms, crop encoding 173 ms (133 ms for pair crops), OT 20 ms, other steps 88 ms. LEM is 26% slower than ComCLIP on average (8% at the median) and about 21× slower than global CLIP (18× at the median). Detection dominates; OT is about 3% of the cost.

Comparisons beyond ComCLIP

We added CE-CLIP, BindCLIP, BLIP ITM and VQAScore, the four additional backbones above, the full seven-subset SugarCrepe, ARO-VG and MMVP-VLM. LEM beats ComCLIP, the training-free local baseline, on Winoground Image on all five backbones. It stays well behind cross-attention models, is below the global baseline on six of seven SugarCrepe subsets with the relation term on, and does not help on ARO-VG or MMVP-VLM. These runs use the submission weights (selected on Winoground and SugarCrepe with CLIP B/32), not cross-validated selection; we will rerun them under that protocol.

Main table

The numbers you cite (Winoground Image 9.25 → 14.00) will be replaced by a cross-validated table, because the submission chose its weights from sweeps on the evaluation sets (Appendix C.7). In that table CLIP+LEM reaches 21.00 on Image (p < 0.001) and 12.00 on Group (p = 0.022). We redid these results with 5-fold cross-validation within each benchmark: each fold selects one of 630 weight configurations on the other four folds, and only held-out predictions are scored.

If these responses address your concerns, we would be grateful if you would consider updating your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.