- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose an incremental model architecture that designs a lightweight trainable branch attached to a frozen dual encoder. The effectiveness of the method is validated on datasets such as Flickr30K Entities.
Critical strengths
- The overall narrative is clear, and the paper is well-written.
- The authors validate the retrieval performance, plug-and-play capability, and grounding effectiveness across multiple different benchmarks.
Critical weaknesses
- The proposed method largely combines several techniques that have been explored in prior work, including sparse fine-grained alignment [1], retrieval assisted by grounding [2], counterfactual discrimination [3], and compositional local matching [4]. The authors have not sufficiently demonstrated the unique value of this specific combination. [1] Bica, I., et al. Improving Fine-Grained Understanding in Image-Text Pre-Training. ICML, 2024. [2] Plummer, B. A., et al. Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. ICCV, 2015. [3] Yuksekgonul, M., et al. When and Why Vision-Language Models Behave like Bags-of-Words, and What to Do About It? 2022. [4] Jiang, K., et al. ComCLIP: Training-Free Compositional Image and Text Matching. NAACL, 2024.
- The complete training pipeline of PhraseBind relies on additional phrase-region spatial annotations, which may limit its scalability in scenarios lacking fine-grained grounding annotations. Although the authors demonstrate some cross-dataset transferability on RefCOCOg, both RefCOCOg and the training dataset Flickr30K Entities are based on natural images and share a similar phrase-to-region grounding task format. Thus, these results are insufficient to fully prove the method's generalization capability under different data distributions or application scenarios. It is recommended that the authors further evaluate the method on datasets with different linguistic description characteristics, such as RefCOCO+, and explore its applicability in cross-domain scenarios like medical imaging or remote sensing. At the very least, the limitations imposed by the need for additional spatial annotations on practical deployment should be discussed.
- The paper introduces a confidence head to measure the reliability of visual grounding for phrases, but it employs binary classification supervision using phrase-level binding labels. These two concepts are not entirely equivalent: a phrase may be accurately grounded to a corresponding visual region, yet its described attributes or relations may be inconsistent with the image content. Therefore, it remains unclear whether the learned confidence represents the reliability of visual grounding or merely the semantic matching degree between the phrase and the image.
- Although the paper evaluates the method on multiple datasets, the existing experiments mainly compare against frozen CLIP/SigLIP backbones, the authors' reproduced hard-negative learning methods, and their self-constructed token-local and phrase-local variants. There is a lack of direct comparison with more representative methods for fine-grained vision-language alignment and compositional matching. For instance, SPARC and ComCLIP, which investigate sparse local vision-language alignment and compositional image-text matching respectively, are highly relevant to the core design of this paper.
- The Point accuracy used for grounding evaluation in the experiments appears somewhat singular. It is advisable to consider more region-level grounding metrics, such as IoU-based accuracy following a clear heatmap-to-box conversion.
Questions for authors
Could the authors provide more details on how the phrase-level confidence labels are constructed?
3 comments
Annotations, cross-domain transfer, confidence labels and region metrics (part 2 of 2)
2. Annotations and transfer. Composition gains do not need spatial annotations: a same-budget variant without any spans, masks, boxes or confidence targets still improves SugarCrepe to 84.62±0.14 and ARO-VG to 56.19±0.53. Region labels serve grounding, raising pointing from 38.62±3.65 to 60.14±0.13. Flickr30K Entities is the only annotated source, so the referring results below, including location-free RefCOCO+, are zero-shot.
| Pointing gain [95% CI] | SigLIP-B/16 | CLIP ViT-B/16 |
|---|---|---|
| RefCOCOg Test-U | +15.76 [14.48, 17.06] | +28.29 [27.01, 29.59] |
| RefCOCO+ val | +14.67 [13.32, 16.02] | +24.09 [22.70, 25.52] |
| RefCOCO+ Test-A | +19.51 [17.59, 21.46] | +25.66 [23.82, 27.54] |
| RefCOCO+ Test-B | +9.05 [7.06, 11.08] | +20.36 [18.19, 22.54] |
Every split improves significantly on both backbones. After removing per seed the target images that also appear in the COCO caption-training sample, with the frozen model scored on the same subset, SigLIP gains remain 15.3 to 16.0 on RefCOCOg Test-U, 19.0 to 20.2 on Test-A and 8.3 to 9.4 on Test-B. On CLIP, PhraseBind also improves SugarCrepe by +0.64 [0.38, 0.91] and ARO-VG by +0.89 [0.75, 1.03] while keeping 99.14% of frozen-correct retrieval decisions.
Remote sensing. Following the suggestion, we evaluated the same checkpoints zero-shot on DIOR-RSVG, a remote-sensing visual grounding benchmark, with no remote-sensing training. We use the complete author test split, 7,500 expressions over 6,102 images, and the same pointing and top-8 box rules.
| DIOR-RSVG, % | Frozen | PhraseBind | Difference [95% CI] |
|---|---|---|---|
| SigLIP pointing | 11.37 | 16.02±3.40 | +4.64 [3.73, 5.55] |
| SigLIP IoU≥.5 | 2.93 | 3.85±0.33 | +0.92 [0.52, 1.33] |
| CLIP pointing | 5.15 | 25.38±0.66 | +20.23 [19.31, 21.16] |
| CLIP IoU≥.5 | 1.84 | 7.76±0.38 | +5.92 [5.34, 6.50] |
Both metrics improve with CIs above zero on both backbones. On CLIP, pointing rises almost fivefold and IoU≥.5 more than fourfold. Phrase-region binding learned from natural-image annotations therefore transfers to overhead imagery and its referring expressions without any in-domain labels.
3. Confidence labels. A phrase gets target 1 if its Entities annotation has a box and 0 otherwise. The confidence therefore estimates grounding reliability, whether the phrase has a localizable referent, which is the distinction the reviewer draws from attribute and relation agreement, and the data confirm it. Removing confidence lowers pointing by 0.52 [0.15, 0.90] and IoU≥.5 by 0.50 [0.06, 0.94], while its semantic-option AUROC is .4900 to .4939 on SugarCrepe and .5003 to .5054 on ARO-VG. Semantic binding is handled by the counterfactual-trained support term.
5. Region-level metric. The box, fixed before testing, encloses the top-8 evidence patches and is scored at IoU≥.5 in original coordinates. On CLIP, IoU≥.5 improves with CIs above zero on Entities, 16.99 to 19.24, +2.25 [0.66, 3.85], and on all five referring splits: RefCOCOg val +5.02 [3.63, 6.40], Test-U +4.86 [3.88, 5.86], RefCOCO+ val +2.91 [1.97, 3.88], Test-A +4.33 [2.94, 5.70] and Test-B +2.30 [0.92, 3.65]. On SigLIP it improves significantly on four of five referring splits, such as +3.68 [2.39, 4.96] on RefCOCO+ Test-A, and Entities IoU≥.5 is 17.29±0.34 against 18.75. IoU≥.5 also improves on DIOR-RSVG on both backbones, as shown above. The router selects the eight most discriminative patches by Eqs. 2 and 3, so the box measures where the evidence lies. Pointing is the metric aligned with this design and gains +28.84 on SigLIP and +41.53 on CLIP.
Dear Reviewer,
We sincerely thank you for your thoughtful review and constructive feedback. In our rebuttal, we have made every effort to address your concerns by clarifying the target construction and providing additional experimental evidence on component ordering, computational efficiency, and generation performance.
As the discussion period draws to a close, we would be very grateful if you could kindly review our responses and share your thoughts. Should any concerns remain, we would be more than happy to provide further clarification.
Thank you again for your consideration and support!
Best regards,
Authors
Value of the combination and direct comparison with ComCLIP and SPARC (part 1 of 2)
Thank you for the detailed suggestions. All runs use seeds 42/43/44, mean ± SD in %, with β=.002 and α=.95 chosen on Flickr30K validation and frozen before testing; CIs are paired 95% image-cluster bootstrap intervals with 10,000 resamples.
1 and 4. Value of the combination, and representative comparisons. PhraseBind keeps the frozen encoder's global decisions intact and adds phrase-level spatial evidence and binding, a combination no prior method offers.
Hard-negative objectives buy composition by changing the global model, so retrieval and Winoground fall and no grounding is produced. PhraseBind changes retrieval by +0.04 [−0.17, 0.24], keeps 99.46% of frozen-correct decisions and leaves Winoground unchanged, while raising pointing by +28.84 [27.41, 30.32], ARO-VG by +1.57 [1.42, 1.72] and SugarCrepe by +0.39 [0.19, 0.60]. Adding the PhraseBind branch to each trained adapter raises ARO-VG by a further +0.70 [0.62, 0.78] on NegCLIP, +1.17 [1.07, 1.26] on TripletCLIP and +0.65 [0.56, 0.74] on StructureCLIP, plus +1.07 [0.90, 1.26] SugarCrepe on TripletCLIP, with retrieval within 0.11 points of each parent and Entities pointing of 60.26 to 60.73.
ComCLIP. We ran the official ComCLIP code, keeping its object masking, attribute-object concatenation, region overlay, relation regions, GRiT dense regions and score normalization. The paid-API parser is replaced by Qwen2.5-7B-Instruct with the two official prompts and greedy decoding, on the same frozen backbones; we call this ComCLIP with an open-source parser. All 7,511 SugarCrepe and 52,685 ARO-VG pairs are scored, and no sample is dropped.
PhraseBind is ahead in all four cells with every CI above zero. On these dual encoders, replacing the image embedding with parse-driven region crops falls below the frozen model in every cell, while learned phrase-to-patch evidence raises it. The two are complementary: adding PhraseBind's local score to ComCLIP with fixed 0.95/0.05 weights and no tuning lifts ComCLIP by +0.35 [0.11, 0.59] and +1.34 [1.20, 1.48] on SigLIP and by +0.67 [0.41, 0.94] and +0.81 [0.67, 0.94] on CLIP.
ComCLIP's image embedding changes with every query, so on Flickr30K we evaluate it as reranking of each query's frozen top-10 candidates. It lowers SigLIP mean R@1 to 82.95, and PhraseBind is ahead by +1.38 [0.68, 2.08]. On CLIP it gives no gain over the frozen model, −0.10 [−0.92, 0.71]. ComCLIP keeps 96.21% and 95.82% of frozen-correct decisions, against 99.46% and 99.14% for PhraseBind. It also needs a 7B LLM parse per caption and an LLM matching call per caption-image pair, 0.64 s of measured compute per caption option before caching. PhraseBind uses a rule-based extractor and one forward pass, 3.88 to 3.91 ms per image at batch 16 on the same RTX 4090.
SPARC. SPARC is a full pre-training method, so we test its sparse token-patch alignment as a matched frozen-backbone control with the same supervision, parameters, 1,600 updates and coefficients. It reaches similar accuracy, pointing 61.85±0.16 and ARO-VG 52.16±0.75, at twice PhraseBind's cost: median forward 127.28 to 128.99 ms against 62.01 to 62.48 ms at FP32 and batch 16, peak memory 1,687.03 against 990.69 MiB, and at most eight inspectable phrase units per caption for PhraseBind.