- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose LEM (Local Evidence Matters), a training-free method for compositional image-text matching. The key idea is to incorporate structured local evidence into VLMs such as CLIP. Specifically, LEM aligns caption entities and relations with image regions using structured optimal transport. In experiments, LEM improves some results on Winoground and SugarCrepe.
Critical strengths
The LEM method is training-free and it can be easily integrated with existing VLMs without modification of the backbone. This makes LEM applicable to frozen pretrained models such as CLIP and NegCLIP.
Critical weaknesses
-
The performance improvements are limited and inconsistent across tasks. While LEM shows performance gains on several subtasks of each benchmark, the improvements are not consistent. Performance drops are significant on some metrics such as relation replacement.
-
The experimental evaluation is small scale. The experiments mainly focus on Winoground and SugarCrepe. Evaluation on additional compositional benchmarks, as well as standard image-text retrieval benchmarks such as COCO or Flickr30K, would help demonstrate generalization and verify that LEM does not hurt general retrieval performance.
-
The method appears to require some benchmark-specific tuning. Several important weights are selected separately for each benchmark and backbone, raising a question of how well the method would work on a new dataset without additional calibration.
-
The additional complexity is not sufficiently discussed. LEM introduces Grounding DINO and iterative OT optimization on top of CLIP. Reporting runtime and computational overhead would help clarify the practical tradeoff between performance and complexity.
Questions for authors
I do not have specific questions, but please address the weaknesses
Response to Reviewer qk5J
Thank you for the review. We take the four weaknesses in order.
W1. Limited and inconsistent improvements
We agree. Under 5-fold cross-validated weight selection (see W3), 5 of the 12 comparisons with the global baselines in Table 1 are significant gains and none is a significant loss.
One result is consistent across backbones: with the submission weights (selected on Winoground with CLIP B/32, w_glob = 0.55) transferred unchanged to OpenCLIP ViT-L/14, SigLIP, SigLIP2 and EVA02-L/14, Winoground Image improves on all five backbones (+3.3 to +8.0 points, significant on four). The relation-replacement drop is just as consistent under the same weights (−2.8 to −5.1 points). Both come from the directed relation term. We will narrow the claims to paired-image discrimination and list the drop as a limitation.
W2. Evaluation scale and retrieval
We added the full seven-subset SugarCrepe, ARO-VG, MMVP-VLM, the four backbones above and four more baselines (CE-CLIP, BindCLIP, BLIP ITM, VQAScore). LEM does not improve ARO-VG or MMVP-VLM, and on SugarCrepe only the node-only variant beats the global baseline on average (76.94 vs. 75.41). These runs use the submission weights, not cross-validated selection; we will rerun them under that protocol. We have not run COCO or Flickr30K retrieval and make no claim about it. The revision will test LEM as a re-ranker over the global top-k.
W3. Benchmark-specific tuning
You are right. The submission chose its weights per benchmark from sweeps on the evaluation sets (Appendix C.7). All Table 1 results are now selected by 5-fold cross-validation within each benchmark: each fold selects one of 630 weight configurations on the other four folds, and only held-out predictions are scored. The multi-backbone and additional-benchmark runs still use the submission weights, selected on Winoground and SugarCrepe with CLIP B/32, and we label them as such.
The weights remain task dependent: cross-validation picks w_rel = 0 on Swap-Att and w_glob ≥ 0.90 on Repl-Rel, and the NegCLIP folds on Winoground disagree. A new task would need a small calibration set, and we will say so. The exception is Winoground Image, where the CLIP B/32 weights transfer unchanged to four other backbones (W1).
W4. Runtime
On one Tesla T4, over 100 Winoground image-caption pairs, the mean time per pair is 33 ms for global CLIP, 562 ms for ComCLIP and 710 ms for LEM (medians 34, 569 and 614 ms). Grounding DINO takes 429 ms, crop encoding 173 ms, OT 20 ms, and the remaining steps 88 ms. LEM is about 21× slower than global CLIP and 26% slower than ComCLIP on average. Detection dominates; OT is about 3% of the cost. We will report this cost next to the accuracy results.