- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses a gap between sentence-level image–text alignment and object-level localization. The authors propose ProtoLIP, a small trainable prototype layer added to a frozen vision–language model. An offline, text-derived taxonomy groups prototypes into semantic families, and lexical matching selects the families relevant to a query. The selected prototypes pool image features, which are scored against the full query. Additional area and overlap losses encourage more compact and separated spatial assignments.
The resulting matching score is a sum of prototype contributions, which can also be grouped by semantic family and distributed spatially. Experiments on COCO, VOC20, ADE20K, and Flickr30K Entities show improved localization over the frozen backbone. The authors also examine transfer to other backbones, matching and retrieval performance, taxonomy coverage, and component ablations.
Critical strengths
- The localization improvements are substantial for a small addition to a frozen model. On the COCO object-query benchmark, Pointing improves from 0.426 to 0.594 and Energy from 0.332 to 0.515. Improvements also appear on VOC20, ADE20K, and Flickr30K Entities, with additional gains on public SigLIP and FLAIR backbones. These results make the approach interesting beyond a single model or dataset. The absence of spatial supervision and the modest 2.23M trainable-parameter budget strengthen its practical appeal.
- The paper includes useful controls and reports important trade-offs. The parameter-matched ablations help separate the effects of prototype learning, family routing, and spatial regularization. The matched/unmatched query analysis and full COCO-80 evaluation address some concerns about vocabulary coverage and query selection. The authors also report retrieval declines and unsuccessful localization transfer to DreamLIP rather than presenting the method as uniformly beneficial. This makes the empirical contribution more convincing.
Critical weaknesses
- An exact sum is not, by itself, a faithful spatial explanation. Equation 4 guarantees that the spatial contributions sum to the matching score because each prototype assignment is normalized. However, the prototype score is computed after pooling features and applying a nonlinear MLP with normalization. Redistributing that score according to the pooling weights does not establish that each location has the claimed predictive influence. The decomposition is mathematically valid as a score allocation, but the stronger interpretation of these maps as the spatial evidence responsible for the prediction needs additional support. This matters because the connection between explanation and prediction is a central motivation of the paper.
- The experiments demonstrate localization more directly than object-level disentanglement. Pointing and Energy measure whether evidence falls within a target region; they do not directly test whether different objects within a sentence receive correctly separated contributions. This distinction is especially important when two objects share a semantic family, such as a man and a woman, because the explicit separation loss operates only between families. The COCO protocol also treats all instances of a category as valid targets, so it cannot establish instance-specific separation. The qualitative examples are helpful, but a controlled evaluation of same-family objects, attribute binding, or relation-dependent queries would better support the central claim. Table 5 further shows that most gains occur for taxonomy-matched queries, which limits how broadly the current results establish open-vocabulary disentanglement.
Questions for authors
- What interpretation of Equation 4 is intended: an exact allocation of the score, or a faithful measure of spatial influence? Can you provide an intervention-based check showing whether removing highly attributed regions changes the matching score more than removing appropriate control regions?
- How well does ProtoLIP separate two objects belonging to the same semantic family? Results for queries such as “the man” versus “the woman,” or two instances distinguished by an attribute or relation, would help determine whether the method resolves the object-level ambiguity motivating the paper.
- How are negative image–text pairs constructed for AUC and balanced accuracy, and how is the classification threshold selected? This is important for interpreting why these metrics improve while caption Recall@1 declines. In particular, do the negatives include captions with the same object nouns but incorrect attributes or relations?
- How are ties and empty detection maps handled in the MM Grounding DINO comparison? Rasterizing a box produces a flat region with many maximum-valued cells, so the Pointing result may depend on the tie-breaking rule. Does the conclusion remain similar when using the highest-scoring box center or another detector-appropriate localization measure?
2 comments
Response to Question 3:
We thank the reviewer for this question. For AUC and BAcc, we evaluate all image–text combinations, with positive pairs and negative pairs (1,200 positives and 1,438,800 negatives for ). For BAcc, the threshold is selected to maximize BAcc on the evaluation set, as no separate validation split is available. AUC and Recall@1 are threshold-independent.
Our negatives are naturally occurring mismatched pairs, which may include captions sharing object nouns but differing in attributes or relations. However, we do not explicitly construct hard negatives for the main evaluation.
Attribute ablation. To investigate the Recall@1 decline, we removed attribute words (e.g., colors and quantities) from captions.
| Model | Original I→T / T→I R@1 | Without Attributes I→T / T→I R@1 |
|---|---|---|
| ItemizedCLIP | 0.170 / 0.191 | 0.106 / 0.117 |
| ProtoLIP | 0.139 / 0.177 | 0.089 / 0.105 |
Both models show reduced Recall@1 after attribute removal, suggesting that attribute information contributes to retrieval. The I→T gap narrows from 0.031 to 0.017, while the T→I gap remains nearly unchanged (0.014 to 0.012). ProtoLIP exhibits a slightly smaller decline, potentially indicating lower sensitivity to attribute information.
We further evaluated 22 captions selected from Flickr30K Entities. We modified actions or relations to "standing beside" while preserving the images and object nouns, then compared the scores of the original and modified captions.
| Model | Original Caption Preferred |
|---|---|
| ItemizedCLIP | 16/22 (72.7%) |
| ProtoLIP | 17/22 (77.3%) |
Both models generally prefer the original captions, with no clear evidence of degraded relation-level discrimination in ProtoLIP.
Response to Question 4:
We thank the reviewer for highlighting this evaluation issue. For MM-Grounding-DINO, we rasterize predicted bounding boxes onto a grid, assigning each box's confidence score to its covered cells and summing scores in overlapping regions. Pointing selects the maximum-valued cell, with ties resolved by the first cell in flattened grid order. Queries with no detections receive zero localization scores.
To address the concern about tie-breaking, we conducted an additional evaluation using the center of the highest-confidence predicted box instead of the maximum location in the rasterized map.
| Dataset | Change in Pointing (Box Center − Rasterized) |
|---|---|
| COCO | +0.007 |
| Flickr30K Phrase | +0.009 |
| ADE20K-150 Object | +0.017 |
The center-based evaluation yields slightly higher Pointing scores across all three datasets. These small differences indicate that the rasterization tie-breaking rule has a limited impact on the reported results, and the overall comparison remains similar under this detector-appropriate localization measure.
We sincerely thank the reviewer for the careful evaluation of our paper. We appreciate the reviewer’s careful consideration of our work and the questions raised regarding its underlying mechanisms and evaluation methodology. We hope that our responses and additional experiments have addressed the reviewer's concerns. We also welcome any further questions and discussion.
Regarding Q2, the challenges involving fine-grained attributes and relations reflect broader limitations of CLIP-based models, as well as limitations of our current approach. We believe these are important directions for future research and deserve further investigation.
Regarding Q3, our additional experiments suggest that attribute information contributes to retrieval performance in both models, while our preliminary relation-level hard-negative evaluation shows no clear degradation in ProtoLIP's relation discrimination. These findings help clarify the observed retrieval trade-offs, although further investigation is needed to fully understand the Recall@1 decline.
Regarding Q4, although our additional experiments show only minor changes in the overall results, we recognize that the tie-breaking strategy may still meaningfully affect localization for certain individual images. This is an important evaluation detail that we had previously overlooked, and we appreciate the reviewer for bringing it to our attention.
Response to Question 1:
We thank the reviewer for this question Eq(4) is intended as an exact spatial allocation of the matching score. This allocation is not post hoc, as the same spatial assignment weights are used to construct the prototype representations for computing the matching score. Since these weights sum to one for each prototype, the signed spatial contributions exactly recover the matching score when summed over all locations.
However, we agree that exact score decomposition does not necessarily mean that removing a region will produce an equivalent score change. To examine this, we conducted an additional intervention experiment. For each query, we replaced the top 10% of patches ranked by signed spatial attribution with the mean patch feature of the same image and recomputed the matching score. We compared this with replacing the bottom 10% and a random 10% of patches.
ADE20K-150 full validation set (16,628 queries)
Replacing the top-attributed regions produces a larger mean score drop than the random control (0.00299 vs. 0.00006), with a paired Wilcoxon test yielding p<10−190.
Flickr30K Entities
Values indicate the mean matching-score drop after replacing 10% of patches.
We observe the same trend on Flickr30K Entities, with significant differences between the top-attribution and random interventions for both phrase queries (p=1.65×10−118) and full-caption queries (p=3.24×10−17).
These results suggest that regions receiving higher attribution have a greater effect on the matching score when perturbed. We hope this clarifies the intended interpretation of Eq.(4) and addresses the reviewer's concern about spatial attribution.
Response to Question 2:
We thank the reviewer for highlighting this limitation. ProtoLIP can potentially distinguish objects within the same semantic family through query-dependent prototype contributions. Even when queries share the same family routing, their full text embeddings can produce different similarity scores sk(I,T), leading to different spatial evidence maps.
To investigate this, we evaluated 45 images containing both a man and a woman, comparing ProtoLIP with the frozen ItemizedCLIP backbone.
ProtoLIP improves Pointing for both queries. However, the heatmaps remain highly similar (cosine similarity: 0.997 for ProtoLIP vs. 0.967 for ItemizedCLIP), indicating limited fine-grained separation.
We also qualitatively observed differentiation between more visually distinct concepts within the same family, such as "car" and "boat".
We acknowledge that intra-family separation remains a limitation, as our regularization operates at the family level rather than explicitly enforcing instance-level separation. Further investigation is needed to determine how much this limitation arises from the frozen backbone versus prototype aggregation and regularization.