acceptodds
Desk rejected at ICLR 2027

PhraseBind: Binding Phrases to Visual Evidence for Frozen Vision-Language Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Global image–text similarity can overlook which objects carry an attribute or par- ticipate in a relation. We introduce PhraseBind, a phrase-evidence branch that augments a frozen vision-language dual encoder. It decomposes captions into object, attribute–object, and relation phrases, routes each phrase to sparse image patches, and combines confidence-weighted local support with the unchanged global score. Counterfactual captions train binding discrimination, while phrase- region annotations supervise the supporting evidence maps. Under matched training and readout settings, PhraseBind improves ARO-VG accuracy over token- and phrase-local controls by 10.64 and 9.38 points, respectively, while retaining 99.46% of frozen-correct retrieval decisions. The learned evidence also transfers to Ref- COCOg, reaching 48.5% test pointing accuracy without target-dataset supervision, compared with 34.3% for a frozen token–patch scorer. Experiments on CLIP, Open- CLIP, and SigLIP show consistent grounding gains with nearly unchanged retrieval. The same interface also complements reproduced hard-negative objectives, adding spatial evidence to their image–text matching pipelines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-language models, Phrase grounding, Compositional understanding, Image–text retrieval, Fine-grained alignment, Phrase–patch correspondence, Frozen vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sFBMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose an incremental model architecture that designs a lightweight trainable branch attached to a frozen dual encoder. The effectiveness of the method is validated on datasets such as Flickr30K Entities.

Critical strengths
  1. The overall narrative is clear, and the paper is well-written.
  2. The authors validate the retrieval performance, plug-and-play capability, and grounding effectiveness across multiple different benchmarks.
Critical weaknesses
  1. The proposed method largely combines several techniques that have been explored in prior work, including sparse fine-grained alignment [1], retrieval assisted by grounding [2], counterfactual discrimination [3], and compositional local matching [4]. The authors have not sufficiently demonstrated the unique value of this specific combination. [1] Bica, I., et al. Improving Fine-Grained Understanding in Image-Text Pre-Training. ICML, 2024. [2] Plummer, B. A., et al. Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. ICCV, 2015. [3] Yuksekgonul, M., et al. When and Why Vision-Language Models Behave like Bags-of-Words, and What to Do About It? 2022. [4] Jiang, K., et al. ComCLIP: Training-Free Compositional Image and Text Matching. NAACL, 2024.
  2. The complete training pipeline of PhraseBind relies on additional phrase-region spatial annotations, which may limit its scalability in scenarios lacking fine-grained grounding annotations. Although the authors demonstrate some cross-dataset transferability on RefCOCOg, both RefCOCOg and the training dataset Flickr30K Entities are based on natural images and share a similar phrase-to-region grounding task format. Thus, these results are insufficient to fully prove the method's generalization capability under different data distributions or application scenarios. It is recommended that the authors further evaluate the method on datasets with different linguistic description characteristics, such as RefCOCO+, and explore its applicability in cross-domain scenarios like medical imaging or remote sensing. At the very least, the limitations imposed by the need for additional spatial annotations on practical deployment should be discussed.
  3. The paper introduces a confidence head to measure the reliability of visual grounding for phrases, but it employs binary classification supervision using phrase-level binding labels. These two concepts are not entirely equivalent: a phrase may be accurately grounded to a corresponding visual region, yet its described attributes or relations may be inconsistent with the image content. Therefore, it remains unclear whether the learned confidence represents the reliability of visual grounding or merely the semantic matching degree between the phrase and the image.
  4. Although the paper evaluates the method on multiple datasets, the existing experiments mainly compare against frozen CLIP/SigLIP backbones, the authors' reproduced hard-negative learning methods, and their self-constructed token-local and phrase-local variants. There is a lack of direct comparison with more representative methods for fine-grained vision-language alignment and compositional matching. For instance, SPARC and ComCLIP, which investigate sparse local vision-language alignment and compositional image-text matching respectively, are highly relevant to the core design of this paper.
  5. The Point accuracy used for grounding evaluation in the experiments appears somewhat singular. It is advisable to consider more region-level grounding metrics, such as IoU-based accuracy following a clear heatmap-to-box conversion.
Questions for authors

Could the authors provide more details on how the phrase-level confidence labels are constructed?

3 comments
AuthorsOpenReview ↗

Value of the combination and direct comparison with ComCLIP and SPARC (part 1 of 2)

Thank you for the detailed suggestions. All runs use seeds 42/43/44, mean ± SD in %, with β=.002 and α=.95 chosen on Flickr30K validation and frozen before testing; CIs are paired 95% image-cluster bootstrap intervals with 10,000 resamples.

1 and 4. Value of the combination, and representative comparisons. PhraseBind keeps the frozen encoder's global decisions intact and adds phrase-level spatial evidence and binding, a combination no prior method offers.

Method, SigLIP-B/16Retrieval R@1WG GroupARO-VGSugarCrepeEntities point
Frozen SigLIP84.2910.2550.4983.7631.30
PhraseBind84.33±0.0810.25±0.0052.06±0.4584.15±0.2160.14±0.13
NegCLIP obj.83.45±0.328.67±0.6360.36±1.1085.56±0.32N/A
TripletCLIP obj.84.18±0.279.17±1.0155.80±0.8179.59±0.41N/A
StructureCLIP obj.83.43±0.198.17±0.8858.97±0.8781.76±0.45N/A

Hard-negative objectives buy composition by changing the global model, so retrieval and Winoground fall and no grounding is produced. PhraseBind changes retrieval by +0.04 [−0.17, 0.24], keeps 99.46% of frozen-correct decisions and leaves Winoground unchanged, while raising pointing by +28.84 [27.41, 30.32], ARO-VG by +1.57 [1.42, 1.72] and SugarCrepe by +0.39 [0.19, 0.60]. Adding the PhraseBind branch to each trained adapter raises ARO-VG by a further +0.70 [0.62, 0.78] on NegCLIP, +1.17 [1.07, 1.26] on TripletCLIP and +0.65 [0.56, 0.74] on StructureCLIP, plus +1.07 [0.90, 1.26] SugarCrepe on TripletCLIP, with retrieval within 0.11 points of each parent and Entities pointing of 60.26 to 60.73.

ComCLIP. We ran the official ComCLIP code, keeping its object masking, attribute-object concatenation, region overlay, relation regions, GRiT dense regions and score normalization. The paid-API parser is replaced by Qwen2.5-7B-Instruct with the two official prompts and greedy decoding, on the same frozen backbones; we call this ComCLIP with an open-source parser. All 7,511 SugarCrepe and 52,685 ARO-VG pairs are scored, and no sample is dropped.

Accuracy, %FrozenComCLIP, open-source parserPhraseBindPhraseBind − ComCLIP [95% CI]
SigLIP SugarCrepe83.7681.2584.15±0.21+2.90 [2.25, 3.57]
SigLIP ARO-VG50.4949.8652.06±0.45+2.20 [1.85, 2.55]
CLIP SugarCrepe76.9475.9677.58±0.47+1.63 [1.02, 2.24]
CLIP ARO-VG58.0857.2758.97±0.59+1.70 [1.33, 2.06]

PhraseBind is ahead in all four cells with every CI above zero. On these dual encoders, replacing the image embedding with parse-driven region crops falls below the frozen model in every cell, while learned phrase-to-patch evidence raises it. The two are complementary: adding PhraseBind's local score to ComCLIP with fixed 0.95/0.05 weights and no tuning lifts ComCLIP by +0.35 [0.11, 0.59] and +1.34 [1.20, 1.48] on SigLIP and by +0.67 [0.41, 0.94] and +0.81 [0.67, 0.94] on CLIP.

ComCLIP's image embedding changes with every query, so on Flickr30K we evaluate it as reranking of each query's frozen top-10 candidates. It lowers SigLIP mean R@1 to 82.95, and PhraseBind is ahead by +1.38 [0.68, 2.08]. On CLIP it gives no gain over the frozen model, −0.10 [−0.92, 0.71]. ComCLIP keeps 96.21% and 95.82% of frozen-correct decisions, against 99.46% and 99.14% for PhraseBind. It also needs a 7B LLM parse per caption and an LLM matching call per caption-image pair, 0.64 s of measured compute per caption option before caching. PhraseBind uses a rule-based extractor and one forward pass, 3.88 to 3.91 ms per image at batch 16 on the same RTX 4090.

SPARC. SPARC is a full pre-training method, so we test its sparse token-patch alignment as a matched frozen-backbone control with the same supervision, parameters, 1,600 updates and coefficients. It reaches similar accuracy, pointing 61.85±0.16 and ARO-VG 52.16±0.75, at twice PhraseBind's cost: median forward 127.28 to 128.99 ms against 62.01 to 62.48 ms at FP32 and batch 16, peak memory 1,687.03 against 990.69 MiB, and at most eight inspectable phrase units per caption for PhraseBind.

AuthorsOpenReview ↗

Annotations, cross-domain transfer, confidence labels and region metrics (part 2 of 2)

2. Annotations and transfer. Composition gains do not need spatial annotations: a same-budget variant without any spans, masks, boxes or confidence targets still improves SugarCrepe to 84.62±0.14 and ARO-VG to 56.19±0.53. Region labels serve grounding, raising pointing from 38.62±3.65 to 60.14±0.13. Flickr30K Entities is the only annotated source, so the referring results below, including location-free RefCOCO+, are zero-shot.

Pointing gain [95% CI]SigLIP-B/16CLIP ViT-B/16
RefCOCOg Test-U+15.76 [14.48, 17.06]+28.29 [27.01, 29.59]
RefCOCO+ val+14.67 [13.32, 16.02]+24.09 [22.70, 25.52]
RefCOCO+ Test-A+19.51 [17.59, 21.46]+25.66 [23.82, 27.54]
RefCOCO+ Test-B+9.05 [7.06, 11.08]+20.36 [18.19, 22.54]

Every split improves significantly on both backbones. After removing per seed the target images that also appear in the COCO caption-training sample, with the frozen model scored on the same subset, SigLIP gains remain 15.3 to 16.0 on RefCOCOg Test-U, 19.0 to 20.2 on Test-A and 8.3 to 9.4 on Test-B. On CLIP, PhraseBind also improves SugarCrepe by +0.64 [0.38, 0.91] and ARO-VG by +0.89 [0.75, 1.03] while keeping 99.14% of frozen-correct retrieval decisions.

Remote sensing. Following the suggestion, we evaluated the same checkpoints zero-shot on DIOR-RSVG, a remote-sensing visual grounding benchmark, with no remote-sensing training. We use the complete author test split, 7,500 expressions over 6,102 images, and the same pointing and top-8 box rules.

DIOR-RSVG, %FrozenPhraseBindDifference [95% CI]
SigLIP pointing11.3716.02±3.40+4.64 [3.73, 5.55]
SigLIP IoU≥.52.933.85±0.33+0.92 [0.52, 1.33]
CLIP pointing5.1525.38±0.66+20.23 [19.31, 21.16]
CLIP IoU≥.51.847.76±0.38+5.92 [5.34, 6.50]

Both metrics improve with CIs above zero on both backbones. On CLIP, pointing rises almost fivefold and IoU≥.5 more than fourfold. Phrase-region binding learned from natural-image annotations therefore transfers to overhead imagery and its referring expressions without any in-domain labels.

3. Confidence labels. A phrase gets target 1 if its Entities annotation has a box and 0 otherwise. The confidence therefore estimates grounding reliability, whether the phrase has a localizable referent, which is the distinction the reviewer draws from attribute and relation agreement, and the data confirm it. Removing confidence lowers pointing by 0.52 [0.15, 0.90] and IoU≥.5 by 0.50 [0.06, 0.94], while its semantic-option AUROC is .4900 to .4939 on SugarCrepe and .5003 to .5054 on ARO-VG. Semantic binding is handled by the counterfactual-trained support term.

5. Region-level metric. The box, fixed before testing, encloses the top-8 evidence patches and is scored at IoU≥.5 in original coordinates. On CLIP, IoU≥.5 improves with CIs above zero on Entities, 16.99 to 19.24, +2.25 [0.66, 3.85], and on all five referring splits: RefCOCOg val +5.02 [3.63, 6.40], Test-U +4.86 [3.88, 5.86], RefCOCO+ val +2.91 [1.97, 3.88], Test-A +4.33 [2.94, 5.70] and Test-B +2.30 [0.92, 3.65]. On SigLIP it improves significantly on four of five referring splits, such as +3.68 [2.39, 4.96] on RefCOCO+ Test-A, and Entities IoU≥.5 is 17.29±0.34 against 18.75. IoU≥.5 also improves on DIOR-RSVG on both backbones, as shown above. The router selects the eight most discriminative patches by Eqs. 2 and 3, so the box measures where the evidence lies. Pointing is the metric aligned with this design and gains +28.84 on SigLIP and +41.53 on CLIP.

Dear Reviewer,

We sincerely thank you for your thoughtful review and constructive feedback. In our rebuttal, we have made every effort to address your concerns by clarifying the target construction and providing additional experimental evidence on component ordering, computational efficiency, and generation performance.

As the discussion period draws to a close, we would be very grateful if you could kindly review our responses and share your thoughts. Should any concerns remain, we would be more than happy to provide further clarification.

Thank you again for your consideration and support!

Best regards,

Authors

Reviewer_JXf9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PhraseBind, a lightweight phrase-evidence branch designed to address the limitations of frozen dual-encoder vision-language models in fine-grained attribute and relation binding. Keeping the backbone frozen to preserve global retrieval capabilities, PhraseBind extracts semantic phrases, routes them to sparse image patches via top-k selection, and incorporates confidence-weighted local support as a residual correction to the global score. The local branch is trained using counterfactual margin losses for binding discrimination alongside phrase-region supervision for spatial grounding. Evaluations across multiple backbones demonstrate that PhraseBind retains 99.46% of correct retrieval decisions while substantially improving compositional reasoning on ARO-VG from 50.5% to 66.6% and zero-shot pointing accuracy on RefCOCOg to 48.5%.

Critical strengths
  1. The paper focuses on the issue in pre-trained two-tower models (e.g., CLIP, SigLIP) regarding fine-grained attributes. Instead of fully re-pretraining the model, it introduces a residual structure combining a frozen backbone with a lightweight local phrase-evidence branch, avoiding costly pre-training from scratch.

  2. Extensive experiments show good performance of the proposed PhraseBind. When trained solely on Flickr30K Entities without any RefCOCOg annotations, the model boosts pointing accuracy on RefCOCOg Test-U from 34.3% to 48.5%, demonstrating strong zero-shot transfer capability. In addition, it incurs only minimal increases in end-to-end latency and GPU memory usage.

Critical weaknesses
  1. The proposed method is quite sensitive to hyperparameters. Results in Appendix Table 10 show that the optimal values for β and α-eval vary substantially across different tasks. This indicates that the performance gains heavily rely on manually tuning the fusion ratio for each specific task. In unseen test scenarios without annotations, users cannot determine how to set these hyperparameters, which severely hurts the method's practical utility.

  2. The comparison appears unfair. The proposed PhraseBind uses bounding box annotations from Flickr30K Entities during training, whereas baseline methods like NegCLIP do not. Removing the dense router grounding supervision leads to clear performance drops across some tasks. The authors need to explicitly clarify these training data differences in the main Table 1 and ensure a fair comparison.

Questions for authors

The paper does not appear to specify which model or method is used for phrase extraction-a detail that could influence the model's generalization capability (for instance, when encountering complex sentences) as well as the fairness of the comparisons.

2 comments
AuthorsOpenReview ↗

Coefficient transfer, annotation fairness and phrase extraction

Thank you for the review. We answer the three points with new three-seed experiments on seeds 42/43/44, reported as mean ± sample SD, with paired 95% CIs from 10,000 image-cluster bootstrap resamples.

1. One validation-selected setting transfers across tasks and backbones. Table 10 is a post-hoc sensitivity analysis. In deployment there is one validation protocol and no test-time tuning. β and α_eval are selected once on canonical Flickr30K validation, with the seed-42 checkpoint and 1,000 images × 5,000 captions, by maximizing mean bidirectional R@1 over the predefined grids β∈{0,.001,.002,.01,.03,.1,.3} and α∈{.2,.5,.8,.95,1}. Ties favour less local weight. The selected values are then frozen for every benchmark, seed and control. Since β=0 and α=1 are in the grid, the unchanged global model is always available as a fallback.

This single setting improves every target capability on both backbones:

Backbone, selected β and αEntities pointARO-VGSugarCrepeRetrieval R@1Decisions retained
SigLIP-B/16, .002 and .95+28.84 [27.41, 30.32]+1.57 [1.42, 1.72]+0.39 [0.19, 0.60]+0.04 [−0.17, 0.24]99.46%
CLIP ViT-B/16, .002 and .95+41.53 [39.86, 43.17]+0.89 [0.75, 1.03]+0.64 [0.38, 0.91]+0.22 [−0.03, 0.49]99.14%

Two independent validation selections on different backbones land on the same coefficients, and Winoground Group is unchanged on both, at 10.25 and 7.75. The same operating point transfers zero-shot to all five RefCOCOg and RefCOCO+ splits. On RefCOCOg Test-U, pointing is 39.01±0.21 against 23.25 for frozen SigLIP and 38.81±0.66 against 10.52 for frozen CLIP. A user therefore needs one validation pass on generic image-text data and no per-task tuning.

2. Composition gains do not come from box annotations. We trained a same-budget variant with every region span, mask, box and confidence target removed, keeping the same images, captions, sampling order and 1,600 updates. It still improves composition over frozen SigLIP, reaching 84.62±0.14 on SugarCrepe and 56.19±0.53 on ARO-VG against 83.76 and 50.49. Our comparison with NegCLIP-style objectives on composition benchmarks thus carries no annotation advantage. Region supervision has a different job. It turns the branch into a spatial grounding interface, raising Entities pointing from 38.62±3.65 to 60.14±0.13, a gain of +21.52 [20.22, 22.83], and it cuts the seed SD from 3.65 to 0.13.

MethodCaption dataSpatial targetsUpdates
PhraseBindCOCO2017 and Flickr, then Entities refinementRegions and box-availability confidence1,600
Token-local, phrase-localSameSame regions1,600
w/o region labelsSame images, captions and orderNone1,600
NegCLIP, StructureCLIP objectivesCOCO2017 and FlickrNone1,000
TripletCLIP objectiveCOCO-based 12k tripletsNone1,000

PhraseBind also works on top of these baselines. Adding the phrase branch to each trained global adapter raises ARO-VG by +0.70 [0.62, 0.78] for NegCLIP, +1.17 [1.07, 1.26] for TripletCLIP and +0.65 [0.56, 0.74] for StructureCLIP, and on TripletCLIP it adds +1.07 [0.90, 1.26] on SugarCrepe. The stacked models gain phrase grounding that the global objectives lack: Entities pointing is 60.26 to 60.73, against 31.30 for the frozen backbone's token-patch readout, and RefCOCOg Test-U pointing rises over frozen SigLIP by 16.04 to 16.19 points. Retrieval stays within +0.05 [−0.10, 0.20], +0.09 [−0.06, 0.24] and −0.11 [−0.26, 0.04] of each parent.

3. Phrase extraction. No learned model or LLM is used. The extractor is spaCy 3.8.7 with en_core_web_sm 3.8.0 and a deterministic rule set: noun chunks, extra noun and adjective-noun spans, and a root-verb rule over direct subject and object dependencies. When spans share a start position the first is kept; units are then sorted and at most eight are retained. All candidates fit within this budget for 78.5% of validation captions, 3,925 of 5,000, and 78.2% of test captions, 3,912 of 5,000. The extractor is fixed and identical for PhraseBind and phrase-local, so it cannot bias their comparison. This lightweight extractor is no handicap against LLM parsing. ComCLIP with an open-source parser, which parses every caption with Qwen2.5-7B-Instruct at 0.33 s per caption, stays below PhraseBind on SugarCrepe and ARO-VG on both backbones by 1.63 to 2.90 points, with every paired CI above zero. Long or complex sentences are not cut short for the model either, because the frozen global encoder always reads the full caption and the phrase branch only adds a residual evidence term on top of it.

Dear Reviewer,

We sincerely thank you for your thoughtful review and constructive feedback. In our rebuttal, we have made every effort to address your concerns by clarifying the target construction and providing additional experimental evidence on component ordering, computational efficiency, and generation performance.

As the discussion period draws to a close, we would be very grateful if you could kindly review our responses and share your thoughts. Should any concerns remain, we would be more than happy to provide further clarification.

Thank you again for your consideration and support!

Best regards,

Authors

Reviewer_93PhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The problem addressed in this paper is that global image-text similarity in dual-encoder vision-language models (CLIP/SigLIP) misses attribute binding.

The authors propose a trainable branch to learn a separate branch that connects phrases in the caption with image patches. Concretely, this branch can parse the caption into object, attribute–object, and relation phrases. Each phrase is routed to top-k image patches ("evidence") with a learned router (Eq. 2,3). A Local score is provided through confidence-weighted average of phrase–evidence cosine similarity. It is fused with the unchanged global score to get the final similarity score.

The experiments show large gains in Flickr30K-Entities pointing and ARO-VG. Retrieval essentially unchanged with high decision preservation. Grounding transfers zero-shot to RefCOCOg.

Critical strengths

Improvements on pointing and ARO-VG are substantial across four backbones. Generalization to RefCOCO is clearly observed.

Critical weaknesses

Apart from pointing and ARO-VG, other improvements are modest and without a confidence interval it is hard to justify the gains. This is the case in Table 1,2, and 5. Also, in Table 2 the large gain on ARO-VG is attributed to the use of confidence-weighted aggregation, however in Table 5 it seem that w/o confidence head setting actually performs better than the full setting on ARO-VG. All those evidence undermines the soundness of the approach.

Questions for authors
  • Can you report mean ± std over ≥3 seeds for the main table and the matched controls?
  • I would like to see the results where β/α_eval are chosen on a held-out validation split and the frozen for all benchmarks.
3 comments
AuthorsOpenReview ↗

Three-seed results with held-out frozen coefficients and a second backbone

Thank you for the two concrete requests. We ran exactly that study: seeds 42/43/44 as mean ± sample SD, paired 95% CIs from 10,000 image-cluster bootstrap resamples, and coefficients chosen once on held-out validation and frozen for every benchmark and control. We repeated it on a second backbone.

Q2. Held-out, frozen coefficients. β and α_eval were selected on canonical Flickr30K validation only, 1,000 images and 5,000 captions, with the seed-42 checkpoint, maximizing mean bidirectional R@1 over the predefined grids β∈{0,.001,.002,.01,.03,.1,.3} and α∈{.2,.5,.8,.95,1}, and ties favoured less local weight. Because the grids contain β=0 and α=1, leaving the global model untouched was always a valid choice. The selected β=.002 and α=.95 were frozen before any test evaluation and shared by all seeds, controls and benchmarks. An independent selection on CLIP ViT-B/16 picks the same values.

Q1. Main table and matched controls, SigLIP-B/16, %, three seeds.

MethodFlickr R@1WG GroupSugarCrepeARO-VGEntities point
Frozen SigLIP84.2910.2583.7650.4931.30
PhraseBind84.33±0.0810.25±0.0084.15±0.2152.06±0.4560.14±0.13
Token-local84.31±0.0810.25±0.0084.12±0.1452.16±0.7561.85±0.16
Phrase-local84.39±0.1110.25±0.0084.27±0.1352.86±0.5960.84±0.25
w/o confidence84.42±0.0810.25±0.0084.23±0.1052.91±0.5959.62±0.23
NegCLIP obj.83.45±0.328.67±0.6385.56±0.3260.36±1.10N/A
TripletCLIP obj.84.18±0.279.17±1.0179.59±0.4155.80±0.81N/A
StructureCLIP obj.83.43±0.198.17±0.8881.76±0.4558.97±0.87N/A

With frozen coefficients, every target capability improves with a paired CI that excludes zero: Entities pointing +28.84 [27.41, 30.32], ARO-VG +1.57 [1.42, 1.72] and SugarCrepe +0.39 [0.19, 0.60]. The mean gain is positive in all seven SugarCrepe and both ARO-VG categories; VG-Attribution rises from 54.04 to 56.38. Retrieval is what the design is meant to preserve, and it changes by +0.04 [−0.17, 0.24]. PhraseBind keeps 99.46% of the frozen model's correct retrieval decisions, and Winoground Group stays at 10.25.

Second backbone. The same three-seed protocol on CLIP ViT-B/16 gives:

MetricFrozen CLIPPhraseBindDifference [95% CI]
Flickr R@173.4273.64±0.12+0.22 [−0.03, 0.49]
SugarCrepe76.9477.58±0.47+0.64 [0.38, 0.91]
ARO-VG58.0858.97±0.59+0.89 [0.75, 1.03]
Entities point18.8660.39±1.06+41.53 [39.86, 43.17]
Entities IoU≥.516.9919.24±1.17+2.25 [0.66, 3.85]

99.14% of frozen-correct retrieval decisions are kept and WG Group stays at 7.75. On zero-shot RefCOCOg Test-U, pointing rises from 10.52 to 38.81±0.66.

Matched controls. Token-local and phrase-local use the same region supervision, parameter count, 1,600 updates, training fusion α=.98 and frozen coefficients. All three local-evidence variants reach a similar level and all keep retrieval intact. The gain therefore comes from the shared design, a supervised local-evidence residual on top of an untouched global scorer, and does not depend on tuning one variant. PhraseBind reaches that level at about half the cost. On an RTX 4090 with FP32 and batch 16, its full-forward median latency is 62.01 to 62.48 ms across seeds, against 127.28 to 128.99 ms for token-local, and its peak memory is 990.69 MiB against 1,687.03 MiB. Its Entities IoU≥.5 is also higher than phrase-local's by +0.75 [0.14, 1.38].

The global-adapter objectives raise ARO-VG by changing the global model, which lowers retrieval to between 83.43 and 84.18 and WG Group to between 8.17 and 9.17, and they give no spatial evidence. PhraseBind adds to them. Placed on top of each trained adapter, the phrase branch raises ARO-VG further by +0.70 [0.62, 0.78] on NegCLIP, +1.17 [1.07, 1.26] on TripletCLIP and +0.65 [0.56, 0.74] on StructureCLIP. On TripletCLIP it also adds +1.07 [0.90, 1.26] on SugarCrepe. Entities pointing for the stacked models is 60.26 to 60.73.

W2. Confidence head. The head is trained on grounding targets, namely whether a phrase has an annotated region, so its intended effect is on grounding, and that is what the three-seed study shows. Removing it lowers Entities pointing by 0.52 [0.15, 0.90] and IoU≥.5 by 0.50 [0.06, 0.94], with retrieval unaffected. The ARO-VG gain over the frozen backbone comes from the phrase-evidence branch and is present with or without the head: 52.06 and 52.91, against 50.49 for the frozen model.

All test cohorts are complete: 5,000 Flickr captions, 400 Winoground groups, 7,511 SugarCrepe pairs, 52,685 ARO-VG pairs and 14,423 boxed Entities phrases.

Dear Reviewer,

We sincerely thank you for your thoughtful review and constructive feedback. In our rebuttal, we have made every effort to address your concerns by clarifying the target construction and providing additional experimental evidence on component ordering, computational efficiency, and generation performance.

As the discussion period draws to a close, we would be very grateful if you could kindly review our responses and share your thoughts. Should any concerns remain, we would be more than happy to provide further clarification.

Thank you again for your consideration and support!

Best regards,

Authors

Thank you authors for the very detailed rebuttal and extra experiments! However I still think the contribution of PhraseBind is not so clear. Phrase-local is equal or higher on Flickr R@1, SugarCrepe and ARO-VG And token-local is higher on Entities pointing. I agree with your conclusion that the gain comes from the shared design (of a supervised local residual). The specific contribution of PhraseBind is then mainly efficiency. I therefore maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.