acceptodds
Under review as a conference paper at ICLR 2027

ProtoLIP: From Sentence-Level to Object-Level Evidence Disentanglement

Abstract

Query-conditioned vision–language models enable fine-grained interpretation by revealing which visual content supports a given textual query and how this ev- idence changes across queries. However, semantically, sentence-level evidence does not necessarily decompose into object-specific contributions, while spatially, object-level evidence can remain entangled with co-occurring objects and sur- rounding scene context. Across multiple VLM architectures and independent benchmarks, we observe persistent object-level evidence entanglement. More- over, exposed evidence maps do not necessarily correspond to the evidence that directly constitutes the model’s prediction. To disentangle visual evidence at both semantic and spatial levels, we introduce ProtoLIP, a lightweight prototype- mediated evidence layer that organizes reusable visual prototypes into text-derived semantic families and uses coarse-to-fine evidence routing, where semantic fami- lies constrain prototype eligibility and the complete query determines fine-grained prototype contributions. Our studies show that ProtoLIP improves evidence lo- calization and separation across query granularities, achieving average relative gains of 29% in Pointing and 43% in Energy across four object- and phrase- level OOD benchmarks. Its localization gains also transfer to independently pre- trained VLMs, with larger improvements observed in several transfer settings. On the primary backbone, ProtoLIP also improves image–text matching discrimina- tion while remaining competitive with a spatially supervised grounding model in object-level localization. Crucially, ProtoLIP constructs its image–text matching score directly from localized prototype evidence, enabling exact decomposition across prototypes, semantic families, and spatial evidence without spatial annota- tions or backbone retraining.

Primary area:
interpretability and explainable AI
Keywords:
fine-grained vision-language, multimodal learning, interpretability, prototype learning, evidence disentanglement

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ijXeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a gap between sentence-level image–text alignment and object-level localization. The authors propose ProtoLIP, a small trainable prototype layer added to a frozen vision–language model. An offline, text-derived taxonomy groups prototypes into semantic families, and lexical matching selects the families relevant to a query. The selected prototypes pool image features, which are scored against the full query. Additional area and overlap losses encourage more compact and separated spatial assignments.

The resulting matching score is a sum of prototype contributions, which can also be grouped by semantic family and distributed spatially. Experiments on COCO, VOC20, ADE20K, and Flickr30K Entities show improved localization over the frozen backbone. The authors also examine transfer to other backbones, matching and retrieval performance, taxonomy coverage, and component ablations.

Critical strengths
  1. The localization improvements are substantial for a small addition to a frozen model. On the COCO object-query benchmark, Pointing improves from 0.426 to 0.594 and Energy from 0.332 to 0.515. Improvements also appear on VOC20, ADE20K, and Flickr30K Entities, with additional gains on public SigLIP and FLAIR backbones. These results make the approach interesting beyond a single model or dataset. The absence of spatial supervision and the modest 2.23M trainable-parameter budget strengthen its practical appeal.
  2. The paper includes useful controls and reports important trade-offs. The parameter-matched ablations help separate the effects of prototype learning, family routing, and spatial regularization. The matched/unmatched query analysis and full COCO-80 evaluation address some concerns about vocabulary coverage and query selection. The authors also report retrieval declines and unsuccessful localization transfer to DreamLIP rather than presenting the method as uniformly beneficial. This makes the empirical contribution more convincing.
Critical weaknesses
  1. An exact sum is not, by itself, a faithful spatial explanation. Equation 4 guarantees that the spatial contributions sum to the matching score because each prototype assignment is normalized. However, the prototype score is computed after pooling features and applying a nonlinear MLP with normalization. Redistributing that score according to the pooling weights does not establish that each location has the claimed predictive influence. The decomposition is mathematically valid as a score allocation, but the stronger interpretation of these maps as the spatial evidence responsible for the prediction needs additional support. This matters because the connection between explanation and prediction is a central motivation of the paper.
  2. The experiments demonstrate localization more directly than object-level disentanglement. Pointing and Energy measure whether evidence falls within a target region; they do not directly test whether different objects within a sentence receive correctly separated contributions. This distinction is especially important when two objects share a semantic family, such as a man and a woman, because the explicit separation loss operates only between families. The COCO protocol also treats all instances of a category as valid targets, so it cannot establish instance-specific separation. The qualitative examples are helpful, but a controlled evaluation of same-family objects, attribute binding, or relation-dependent queries would better support the central claim. Table 5 further shows that most gains occur for taxonomy-matched queries, which limits how broadly the current results establish open-vocabulary disentanglement.
Questions for authors
  1. What interpretation of Equation 4 is intended: an exact allocation of the score, or a faithful measure of spatial influence? Can you provide an intervention-based check showing whether removing highly attributed regions changes the matching score more than removing appropriate control regions?
  2. How well does ProtoLIP separate two objects belonging to the same semantic family? Results for queries such as “the man” versus “the woman,” or two instances distinguished by an attribute or relation, would help determine whether the method resolves the object-level ambiguity motivating the paper.
  3. How are negative image–text pairs constructed for AUC and balanced accuracy, and how is the classification threshold selected? This is important for interpreting why these metrics improve while caption Recall@1 declines. In particular, do the negatives include captions with the same object nouns but incorrect attributes or relations?
  4. How are ties and empty detection maps handled in the MM Grounding DINO comparison? Rasterizing a box produces a flat region with many maximum-valued cells, so the Pointing result may depend on the tie-breaking rule. Does the conclusion remain similar when using the highest-scoring box center or another detector-appropriate localization measure?
2 comments

Response to Question 1:

We thank the reviewer for this question Eq(4) is intended as an exact spatial allocation of the matching score. This allocation is not post hoc, as the same spatial assignment weights are used to construct the prototype representations for computing the matching score. Since these weights sum to one for each prototype, the signed spatial contributions exactly recover the matching score when summed over all locations.

However, we agree that exact score decomposition does not necessarily mean that removing a region will produce an equivalent score change. To examine this, we conducted an additional intervention experiment. For each query, we replaced the top 10% of patches ranked by signed spatial attribution with the mean patch feature of the same image and recomputed the matching score. We compared this with replacing the bottom 10% and a random 10% of patches.

ADE20K-150 full validation set (16,628 queries)

Region replacedMean dropMedian dropDrop rate
Top 10%0.002990.0009662.2%
Bottom 10%-0.000560.0000053.5%
Random 10%0.000060.0000050.2%

Replacing the top-attributed regions produces a larger mean score drop than the random control (0.00299 vs. 0.00006), with a paired Wilcoxon test yielding p<10−190p<10^{-190}.

Flickr30K Entities

EvaluationQueriesTop 10%Bottom 10%Random 10%
Phrase3,9160.005130.000070.00007
Full caption9980.00410-0.000220.00019

Values indicate the mean matching-score drop after replacing 10% of patches.

We observe the same trend on Flickr30K Entities, with significant differences between the top-attribution and random interventions for both phrase queries (p=1.65×10−118p=1.65\times10^{-118}) and full-caption queries (p=3.24×10−17p=3.24\times10^{-17}).

These results suggest that regions receiving higher attribution have a greater effect on the matching score when perturbed. We hope this clarifies the intended interpretation of Eq.(4) and addresses the reviewer's concern about spatial attribution.

Response to Question 2:

We thank the reviewer for highlighting this limitation. ProtoLIP can potentially distinguish objects within the same semantic family through query-dependent prototype contributions. Even when queries share the same family routing, their full text embeddings can produce different similarity scores sk(I,T)s_k(I,T), leading to different spatial evidence maps.

To investigate this, we evaluated 45 images containing both a man and a woman, comparing ProtoLIP with the frozen ItemizedCLIP backbone.

ModelQueryPointing ↑Target EnergyOther Energy
ItemizedCLIPMan0.2670.2910.293
ProtoLIPMan0.4440.4130.458
ItemizedCLIPWoman0.3110.3150.311
ProtoLIPWoman0.5110.4560.415

ProtoLIP improves Pointing for both queries. However, the heatmaps remain highly similar (cosine similarity: 0.997 for ProtoLIP vs. 0.967 for ItemizedCLIP), indicating limited fine-grained separation.

We also qualitatively observed differentiation between more visually distinct concepts within the same family, such as "car" and "boat".

We acknowledge that intra-family separation remains a limitation, as our regularization operates at the family level rather than explicitly enforcing instance-level separation. Further investigation is needed to determine how much this limitation arises from the frozen backbone versus prototype aggregation and regularization.

Response to Question 3:

We thank the reviewer for this question. For AUC and BAcc, we evaluate all N2N^2 image–text combinations, with NN positive pairs and N(N−1)N(N-1) negative pairs (1,200 positives and 1,438,800 negatives for N=1200N=1200). For BAcc, the threshold is selected to maximize BAcc on the evaluation set, as no separate validation split is available. AUC and Recall@1 are threshold-independent.

Our negatives are naturally occurring mismatched pairs, which may include captions sharing object nouns but differing in attributes or relations. However, we do not explicitly construct hard negatives for the main evaluation.

Attribute ablation. To investigate the Recall@1 decline, we removed attribute words (e.g., colors and quantities) from captions.

ModelOriginal I→T / T→I R@1Without Attributes I→T / T→I R@1
ItemizedCLIP0.170 / 0.1910.106 / 0.117
ProtoLIP0.139 / 0.1770.089 / 0.105

Both models show reduced Recall@1 after attribute removal, suggesting that attribute information contributes to retrieval. The I→T gap narrows from 0.031 to 0.017, while the T→I gap remains nearly unchanged (0.014 to 0.012). ProtoLIP exhibits a slightly smaller decline, potentially indicating lower sensitivity to attribute information.

We further evaluated 22 captions selected from Flickr30K Entities. We modified actions or relations to "standing beside" while preserving the images and object nouns, then compared the scores of the original and modified captions.

ModelOriginal Caption Preferred
ItemizedCLIP16/22 (72.7%)
ProtoLIP17/22 (77.3%)

Both models generally prefer the original captions, with no clear evidence of degraded relation-level discrimination in ProtoLIP.

Response to Question 4:

We thank the reviewer for highlighting this evaluation issue. For MM-Grounding-DINO, we rasterize predicted bounding boxes onto a 64×6464 \times 64 grid, assigning each box's confidence score to its covered cells and summing scores in overlapping regions. Pointing selects the maximum-valued cell, with ties resolved by the first cell in flattened grid order. Queries with no detections receive zero localization scores.

To address the concern about tie-breaking, we conducted an additional evaluation using the center of the highest-confidence predicted box instead of the maximum location in the rasterized map.

DatasetChange in Pointing (Box Center − Rasterized)
COCO+0.007
Flickr30K Phrase+0.009
ADE20K-150 Object+0.017

The center-based evaluation yields slightly higher Pointing scores across all three datasets. These small differences indicate that the rasterization tie-breaking rule has a limited impact on the reported results, and the overall comparison remains similar under this detector-appropriate localization measure.

We sincerely thank the reviewer for the careful evaluation of our paper. We appreciate the reviewer’s careful consideration of our work and the questions raised regarding its underlying mechanisms and evaluation methodology. We hope that our responses and additional experiments have addressed the reviewer's concerns. We also welcome any further questions and discussion.

Regarding Q2, the challenges involving fine-grained attributes and relations reflect broader limitations of CLIP-based models, as well as limitations of our current approach. We believe these are important directions for future research and deserve further investigation.

Regarding Q3, our additional experiments suggest that attribute information contributes to retrieval performance in both models, while our preliminary relation-level hard-negative evaluation shows no clear degradation in ProtoLIP's relation discrimination. These findings help clarify the observed retrieval trade-offs, although further investigation is needed to fully understand the Recall@1 decline.

Regarding Q4, although our additional experiments show only minor changes in the overall results, we recognize that the tie-breaking strategy may still meaningfully affect localization for certain individual images. This is an important evaluation detail that we had previously overlooked, and we appreciate the reviewer for bringing it to our attention.

Reviewer_CquxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose ProtoLIP, a lightweight framework designed to resolve severe object-level semantic and spatial entanglement in existing fine-grained vision-language models like ItemizedCLIP and FLAIR. Without requiring any spatial annotations or backbone retraining, ProtoLIP introduces only 2.23M trainable parameters via an offline LLM-constructed coarse semantic family routing and a fine-grained prototype pooling mechanism. It directly formulates image-text matching scores as an additive combination of prototype and spatial region contributions. Extensive zero-shot localization experiments across multiple granularities and backbones verify its effectiveness.

Critical strengths
  1. The paper accurately identifies a critical bottleneck in fine-grained vision-language models like ItemizedCLIP and FLAIR: despite achieving sentence-level alignment, they suffer from severe entanglement in object-level semantic decomposition and spatial evidence. ProtoLIP effectively decouples object-level evidence without requiring any spatial annotations or backbone retraining.

  2. Introducing only 2.23M trainable parameters (1.48% of the frozen backbone), ProtoLIP combines offline LLM-built coarse semantic family routing with fine-grained prototype pooling. Modeling image-text matching scores directly as an additive combination of prototype and spatial region contributions avoids the disconnect between post-hoc attribution maps and actual prediction scores.

  3. The evaluation thoroughly covers multi-granularity out-of-distribution (OOD) benchmarks ranging from single words and phrases to full sentences. It systematically validates transferability across different frozen pre-trained backbones (e.g., SigLIP, FLAIR) and retrained same-data controls, achieving zero-shot localization performance comparable to supervised MM Grounding DINO.

Critical weaknesses

Major Weaknesses

  • Although Eq. (4) claims exact spatial decomposition of S(I,T) via sum_n beta_{k,n}(I) = 1, the pooled feature tilde_g_k(I) in Eq. (2) passes through a nonlinear residual MLP phi_theta with GELU and L2 normalization before computing similarity s_k(I,T), and the map H truncates negative values. Thus, q_k(I,T) * beta_{k,n}(I) is merely a post-hoc algebraic redistribution, not a true causal contribution.

  • In Eqs. (3) and (5), since prototype weights r_k and spatial assignments beta_{k,n}(I) are independent of query T, the normalization constant sum_{j in E_T} r_j cancels out in Eq. (5). Consequently, M_c(n; I,T) degenerates into a static query-agnostic map M_c(n; I), rendering L_area and L_overlap unable to prevent intra-family spatial entanglement.

  • Deterministic lexical routing causes severe long-tail mismatches (up to 50% on ADE20K) with negligible Pointing gains. Furthermore, direct ProtoLIP consistently degrades strict Top-1 retrieval (I@1/T@1) on COCO, Flickr30K, and in-domain sets. The 50/50 score fusion in Table 1 also breaks exact decomposability, while notable matching performance drops occur on Multi-positive SigLIP-0.3M and DreamLIP-0.3M.

Minor Weaknesses

  • In Table 3, under the "Matching" column, the header specifies "BAcc | AUC". However, except for row 3 (Family Prototype: 0.863 | 0.934), the AUC and BAcc values in all other five rows (e.g., Frozen ItemizedCLIP: 0.889 | 0.831, and Full ProtoLIP: 0.932 | 0.862) appear to be swapped. This misalignment can be verified by cross-checking the COCO results reported in Figure 5b.

  • Equation (1) in Section 3.2 directly uses image patch features x_n, the total patch count N, and the similarity function sim(., .) without defining them in the main text. Moreover, Section 3.3 mentions the positive-evidence normalized map H in prose without providing a mathematical formulation, and M_c(n; I,T) in Eq. (5) is inconsistently simplified to M_c(n) in Eqs. (6)–(7).

  • Line 440 contains a duplicated-word typo ("MM MM Grounding DINO"), and Line 23 has an isolated period at the beginning of the line. Furthermore, the Pointing and Energy metrics in Appendix Table 8 are formatted as four-digit integers without decimal points (e.g., 5868 and 5143), which is inconsistent with the [0,1] decimal format used elsewhere (e.g., 0.594 in Table 3 or .594 in Table 11).

Questions for authors
  1. Given the nonlinear MLP and L2 normalization in Eq. (2), as well as the cancellation of query-dependent terms in Eq. (5), how do the authors theoretically justify that q_k(I,T) * beta_{k,n}(I) reflects true patch contributions, and how is intra-family multi-object spatial decoupling guaranteed?

  2. The main COCO experiments use a custom-filtered 557-image subset, whereas Pointing drops from 0.594 to 0.389 on full COCO-80 (Table 4). Can the authors provide external baselines (NACLIP and FLAIR) on full COCO-80 and clarify why soft routing via text embeddings was not used instead of brittle lexical matching?

  3. Direct ProtoLIP systematically degrades Top-1 retrieval across Figure 5b and Tables 1, 6, and 10, while the 50/50 backbone score fusion in Table 1 violates exact additive decomposability. Can the authors resolve this retrieval degradation without relying on non-decomposable backbone scores?

2 comments

Response to Critical Weakness 1 and Question 1:

We thank the reviewer for raising this question. We would like to clarify how our spatial evidence is computed and what the decomposition represents.

First, our matching score is computed at the prototype level. Here, qk(I,T)q_k(I,T) is the contribution of prototype kk to the final matching score. We distribute this contribution using the spatial assignments βk,n(I)\beta_{k,n}(I). Since ∑nβk,n(I)=1\sum_n \beta_{k,n}(I)=1, we have ∑nqk(I,T)βk,n(I)=qk(I,T)\sum_n q_k(I,T)\beta_{k,n}(I)=q_k(I,T). This equality holds even with the nonlinear MLP and L2 normalization, since both are applied before computing qk(I,T)q_k(I,T). Second, βk,n(I)\beta_{k,n}(I) is not an arbitrary post hoc assignment. The same weights are used to construct the prototype representation, g~k(I)=∑nβk,n(I)xn\widetilde{g}_k(I)=\sum_n\beta_{k,n}(I)x_n, which is used in the matching computation. We also conducted patch-level intervention experiments to check whether regions with higher attribution have a greater effect on the prediction. We replaced the top-attributed 10% of regions and compared the score changes with bottom-attributed and randomly selected regions.

Dataset / EvaluationQueriesTop 10%Bottom 10%Random 10%
ADE20K-15016,6280.00299-0.000560.00006
Flickr30K Entities (phrase)3,9160.005130.000070.00007
Flickr30K Entities (full caption)9980.00410-0.000220.00019

Mean matching-score drop after replacing 10% of image regions.

On ADE20K-150, replacing the top-attributed regions produces a larger mean score drop than replacing random regions (0.00299 vs. 0.00006), with a paired Wilcoxon test yielding p<10−190p<10^{-190}. Similar trends are observed on Flickr30K Entities for both phrase queries and full captions. These results suggest that regions receiving higher attribution have a greater influence on the model's predictions.

Third, the cancellation in Eq. (5) does not make the final spatial evidence query-independent. Eq. (5) defines a family-level spatial map, whereas the final evidence map is C(n;I,T)=∑kqk(I,T)βk,n(I)C(n;I,T)=\sum_k q_k(I,T)\beta_{k,n}(I). Even when two queries activate the same prototype family, their full text embeddings can produce different similarity scores sk(I,T)s_k(I,T), leading to different prototype contributions and spatial evidence.

Different prototypes within the same family can learn different spatial assignments, allowing the model to distinguish their evidence through query-dependent reweighting.

Response to Question 2:

We thank the reviewer for this suggestion. We additionally evaluated NACLIP and FLAIR using the same evaluation protocol as Table 4. The results are shown below.

MethodPointingEnergy
ItemizedCLIP0.3130.250
NACLIP0.4350.329
FLAIR (merged-30M)0.2480.270
ProtoLIP (Ours)0.3890.358

On full COCO-80, ProtoLIP achieves the highest Energy score (0.358), while NACLIP obtains a higher Pointing score (0.435 vs. 0.389). ProtoLIP also outperforms FLAIR and ItemizedCLIP on both metrics. These results show that ProtoLIP remains competitive on the full benchmark, although the methods differ in their training data and architectures.

Regarding text embedding based routing, we explored this direction during development before. We found that selecting prototype families based on global sentence embeddings can be unreliable for queries containing multiple objects. Since a global embedding represents the overall sentence, it may emphasize dominant concepts while giving less weight to other explicitly mentioned objects. This can cause relevant prototype families to be overlooked.

We therefore adopted lexical matching to identify eligible prototype families. We acknowledge that this approach also has limitations. Extending the mapping with additional expressions, potentially assisted by LLMs, could improve its handling of synonyms and alternative descriptions. Combining lexical matching with more flexible semantic routing is another direction worth exploring.

Response to Question 3:

We thank the reviewer for raising this important question. We believe this reflects a trade-off of our work. First, we would like to clarify what we observe in our experiments. As shown in table 10, under the in domain evaluation with the same training data, direct protolip improves from 0.845 to 0.958, while image recall@1 slightly decreases from 0.368 to 0.347. However, recall@5 and recall@10 improve from 0.406 to 0.485 and from 0.434 to 0.553, respectively. These results suggest that protolip improves overall matching discrimination, although there is still a small drop in top-1 retrieval.

We think this may be related to the different types of visual evidence used for global image text matching and object-level localization. Clip-based models learn global image-text alignment through contrastive learning. Their matching scores may depend not only on the target objects, but also on backgrounds, scene context, and object co-occurrence. These cues can be useful for distinguishing similar image-text pairs and may help top-1 retrieval. However, information that helps global matching is not always the same as the visual evidence humans would expect for locating a specific object. For example, background or co-occurrence cues may improve retrieval without directly supporting the presence of the target object.

This is one of the reasons we introduce prototypes in our work. Our goal is to encourage the model to capture features shared by similar objects across different scenes, rather than relying too heavily on scene specific correlations. This may help the model produce spatial evidence that focuses more on the target objects, but it may also change the ranking behavior of the pretrained backbone.

For example, when several candidate images contain similar objects but differ in their backgrounds or surrounding objects, the original backbone may use these contextual differences to identify the correct match. Protolip places more emphasis on object-level evidence, and may be less sensitive to some of these differences. As a result, the correct match may still appear among the top few candidates but not always rank first. We believe this is one possible explanation for why auc and recall@5/10 improve while recall@1 slightly decreases.

One possible direction to address this is hard-negative ranking. We could introduce a ranking objective that focuses on highly similar negative pairs, helping the model distinguish subtle differences between images and captions that share similar objects. At the same time, we would retain the existing spatial localization constraints to avoid encouraging the model to rely too heavily on background cues. since the matching score would still be computed from prototype contributions, the additive decomposition would remain unchanged. This could help improve top-1 retrieval without giving up object-level interpretability, although it would need to be tested in future work. Finally, we would like to clarify that the main goal of protolip is object-level evidence localization and disentanglement. Thus, we aim to decompose matching scores into prototype contributions that can be linked to spatial evidence.

We sincerely thank the reviewer for carefully reading our paper. The reviewer raised several useful points and identified minor issues in our manuscript that we had overlooked. We have revised the paper based on this feedback and hope that our revisions and responses have addressed the reviewer's concerns. We also welcome any further questions or comments.

Reviewer_n7EYOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProtoLIP, a lightweight layer on top of frozen VLM features, to disentangle evidence for a queried object from its co-ocurring objects and context. The layer has a set of learnable prototype grouped into 39 semantic families, which an LLM builds offline from training captions. The image-text matching score is decomposed into prototype contributions, which are pooled from patch-level evidence . Trained without spatial annnotaions, ProtoLIP reports significant gains in evidence localization on four OOD benchmarks and achieves competitive localization performance with GroundingDINO.

Critical strengths
  1. The proposed module is simple, lightweight, and interpretable by design, requiring no spatial annotations or backbone training.
  2. The method improves evidence localization performance consistently across several backbones and benchmarks.
Critical weaknesses
  1. The method relies on a fixed semantic taxonomy matched by exact strings, which may limit generalization performance. Exact string matching may be problematic in cases of synonyms, ambiguous words, negation, or out-of-dictionary objects.
  2. The hyper-parameters including tau and lambda are neither justified nor analyzed. The paper does not state the data or method of which these values are chosen, and it reports no analyses on them.
Questions for authors
  1. How are the hyper-parameters such as tau and lambda chosen?
  2. GroundingDINO is a powerful grounding model trained on COCO, but Fig. 6 shows it has a relatively low score on COCO. Can the authors explain this gap?
1 comment
AuthorsOpenReview ↗

Follow up on Rebuttal

Dear Reviewer,

We would like to kindly follow up on our rebuttal and revised manuscript. We have addressed your critical concerns regarding semantic taxonomy generalization and hyperparameter selection, including additional analyses and sensitivity experiments. We also reran the evaluation to verify the reported results.

We would greatly appreciate your feedback on whether these additions address your concerns, or if any substantial issues remain. Thank you again.

Best

Comments on the paper

2 comments
AuthorsOpenReview ↗

Unified Response to Reviewers and Summary of Revisions

Dear Area Chair and Reviewers,

Thank you for your careful reviews and valuable suggestions. We have revised the manuscript and provided additional experiments and analyses in our responses. For ease of reference, we summarize our responses and revisions below.

  1. How are the hyperparameters set, and do the results require precise tuning? (n7EY)

We have added the default hyperparameters and training configuration, together with a sensitivity analysis varying one parameter at a time. Pointing and Energy remain relatively stable across the tested ranges, indicating that precise tuning is unnecessary within these ranges. Our response also explains that the regularization parameters were selected empirically during development and kept fixed across the evaluated backbones.

PDF locations: Appendix B; Appendix H, Table 14.

  1. Are the MM Grounding DINO results on COCO reproducible? How are bounding boxes, tied maxima, and empty detections handled? (n7EY, ijXe)

Our repeated evaluation confirms the COCO Pointing/Energy scores of 0.476/0.446, supporting their reproducibility under our protocol. The revised manuscript details detection thresholds, 64×64 rasterization, overlapping-box score accumulation, and normalization. Our response also explains tie-breaking and empty detections and provides a highest-confidence box-center evaluation. Pointing increases by 0.007, 0.009, and 0.017 on COCO, Flickr30K Phrase, and ADE20K, respectively, indicating that the overall comparison is robust to this alternative localization rule.

PDF locations: Section 4.4; Appendix B.

  1. How are negatives and thresholds constructed for AUC/BAcc? Are attribute and relation information examined? (ijXe)

Our response clarifies that evaluation uses all image–text combinations, with corresponding pairs as positives and the remaining pairs as negatives. The BAcc threshold maximizes BAcc on the evaluation set; AUC and Recall@1 are threshold-independent. Removing attributes reduces Recall@1 for both models, showing that both use attribute information. In the 22-caption relation/action comparison, ProtoLIP prefers the original caption in 17/22 cases versus 16/22 for ItemizedCLIP, providing no clear evidence of degraded relation discrimination in this small test.

  1. How does exact decomposition hold, and is the spatial evidence connected to the prediction? (Cqux, ijXe)

We clarify that nonlinear mapping and normalization precede prototype contribution computation, preserving the equality between summed signed spatial contributions and the matching score. Spatial allocation uses the same weights as prototype pooling. Added interventions on ADE20K and Flickr30K Entities produce larger average score drops for highly attributed regions than for random or low-attribution regions. This supports the predictive relevance of the attributed regions under the tested intervention.

PDF locations: Appendix B.2; Appendix I, Equations (15)–(18), Table 15.

  1. Does cancellation in the family-map normalization make the final evidence query-independent? Can objects within the same family be distinguished? (Cqux, ijXe)

We distinguish the family regularization map from the final evidence map, whose prototype contributions depend on the complete query embedding. Queries activating the same family can therefore produce different evidence. In our 45-image man/woman experiment, ProtoLIP improves Pointing from 0.267 to 0.444 for “man” and from 0.311 to 0.511 for “woman.” These results demonstrate improved localization, although the highly similar heatmaps indicate that localization gains do not necessarily imply fine-grained separation.

PDF locations: Section 3.4; Appendix I, Equation (19) and accompanying discussion.

AuthorsOpenReview ↗

Unified Response to Reviewers and Summary of Revisions (Continued)

  1. How does the fixed taxonomy handle synonyms, unmatched queries, and open-vocabulary inputs? Why use lexical rather than soft routing? (n7EY, Cqux, ijXe)

We explain alias mapping, full-bank fallback for unmatched queries, and scoring with the complete text embedding. Coverage and matched/unmatched analyses show that routing provides the largest localization gains on matched queries, while fallback largely preserves unmatched-query performance. The in-domain forced-fallback comparison retains similar matching discrimination while improving strict retrieval, clarifying routing’s effect. Our response also explains that lexical matching retains multiple explicitly mentioned objects whose families may be overlooked by global sentence-embedding routing.

PDF locations: Section 3.1; Appendix A; Appendix C, Tables 5–6.

  1. Does ProtoLIP remain effective on full COCO-80, and can external baselines be included? (Cqux)

Across all 80 COCO categories, ProtoLIP achieves Pointing/Energy of 0.389/0.358 versus 0.313/0.250 for frozen ItemizedCLIP, confirming that gains extend beyond the caption-linked subset. Our response adds NACLIP and FLAIR results: ProtoLIP outperforms FLAIR on both metrics and achieves the highest Energy, while NACLIP achieves the highest Pointing.

PDF locations: Section 4.1; Appendix B.1, Table 4.

  1. Why do matching metrics improve while Top-1 retrieval declines? Does fusion preserve decomposability? (Cqux, ijXe)

Under the same-data in-domain evaluation, Direct ProtoLIP improves AUC from 0.845 to 0.958 and several Recall@5/10 metrics, showing stronger matching discrimination and broader-cutoff retrieval despite lower Recall@1. The 50/50 fusion experiment recovers Flickr30K Top-1 retrieval above the frozen backbone. We distinguish these uses: Direct ProtoLIP retains prototype-level decomposition, while fusion improves retrieval without preserving exact decomposition of the fused score. Hard-negative ranking is discussed as a future direction within the prototype-scoring structure.

PDF locations: Section 4.2, Table 1; Appendix C.1, Table 6; Appendix F, Table 10.

Thank you again for your time and constructive feedback. We have added experiments and analyses to the appendix to further support our findings, and we hope these additions address the reviewers’ concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.