acceptodds
Withdrawn from ICLR 2027

Negation-Aware Reranking Framework for Zero-Shot Referring Image Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Zero-shot referring image segmentation reduces the reliance on task-specific expression–mask annotations by using pretrained vision–language models and class-agnostic mask proposal networks. However, existing methods show performance degradation on expressions containing negation, as CLIP-based holistic matching may select a candidate with high target compatibility despite violating the negation constraint imposed by the referring expression. To address this challenge, we introduce NARF, a training-free negation-aware framework that reranks candidate masks using a base score that preserves the base method's original assessment, a target-compatibility score that measures compatibility with the intended entity, and a negation-violation score that penalizes detected violations. We further construct a Negation-Focused Evaluation Set from existing RIS datasets for systematic evaluation. Across five zero-shot RIS methods, NARF improves aggregate oIoU and mIoU and provides further gains when combined with NegationCLIP, demonstrating the complementarity of negation-aware representation enhancement and candidate-level constraint verification.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Zero-Shot Referring Image Segmentation, Vision-Language Models, Negation Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rvpKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a computationally efficient strategy for zero-shot referring image segmentation with negated referring expressions. The strategy incorporates three scores: a base score and a target-compatibility score to assess the alignment between candidate objects and the target described in the expression, and a negation-violation score to penalize candidates that violate the negation constraints. Experimental results demonstrate competitive performance with low computational requirements.

Critical strengths
  1. The paper is clearly written and easy to follow.

  2. The proposed method achieves competitive performance with low computational cost.

Critical weaknesses
  1. The paper argues that the base score can introduce substantial bias. However, the proposed method initially uses this score to select the top-(K) candidate masks for subsequent processing. This raises a concern that valid target masks may already be excluded at this stage. How can the target-compatibility and negation-violation scores mitigate this initial bias, particularly when the correct mask is absent from the selected candidates? Further analysis of candidate recall and sensitivity to (K) would help address this concern.

  2. The operation in Eq. (9) is unclear. Please clarify its purpose, define the relevant terms, and provide an example illustrating how it works.

  3. The paper does not present failure cases. Including representative failure examples and discussing their causes would help readers better understand the method’s limitations.

Questions for authors

see above mentioned

Reviewer_eGDqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies negation understanding in zero-shot referring image segmentation and proposes NARF, a training-free reranking framework that combines target compatibility with explicit negation constraint verification. The authors also construct a negation-focused evaluation set from existing RIS benchmarks. Experiments across five zero-shot RIS methods show improvements in overall segmentation performance, with additional ablation studies supporting the proposed design.

Critical strengths
  1. Plug-and-play design. The proposed framework requires no additional training and can be integrated into existing zero-shot RIS methods without modifying their architectures.
  2. Consistent improvements. Experiments across five zero-shot RIS baselines demonstrate improvements in overall oIoU and mIoU, suggesting the framework's applicability across different methods.
Critical weaknesses
  1. Limited methodological novelty. The proposed approach offers limited methodological innovation in addressing negation understanding in zero-shot RIS, as it mainly relies on existing techniques rather than introducing a substantially novel solution.
  2. Lack of fine-grained evaluation across negation types. The paper lacks a detailed analysis of performance across different negation types, making it difficult to assess the effectiveness of DCV on accessory absence (e.g., "without glasses") and RCV on spatial versus action relations (e.g., "not near" vs. "not holding").
Questions for authors
  1. Inference efficiency. NARF introduces additional LLM and CLIP computations. I suggest reporting the end-to-end inference latency and computational overhead relative to the baselines.
  2. Constraint parsing details. Section 3 does not clearly explain how the target and constraint parsers are implemented. Could the authors clarify whether they are rule-based or LLM-based?
  3. Comparison with simpler reranking. I suggest comparing NARF with simpler reranking strategies to better isolate the benefits of explicit negation verification.
Reviewer_jHtVOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a training-free zero-shot referring image segmentation method that re-ranks candidate masks based on a target-compatibility score and a negation-violation score. The idea is motivated by the observation that current vision-language representations may not handle expressions containing negation well. To address this, the paper introduces two negation types, Canditate-local negation and Target-reference negation, to verify negation constraints during mask selection. When applied to existing zero-shot RIS models, the proposed method consistently improves performances on negation-focused evaluation sets derived from existing benchmarks.

Critical strengths
  • The paper targets the underexplored problem of negation in zero-shot RIS where the models largely rely on pretrained vision-language models.
  • The proposed method improves performance across five zero-shot RIS models on negation-focused evaluation sets, and also shows complementary effects when combined with NegationCLIP.
Critical weaknesses
  • Limited technical novelty: The proposed verification method mainly combines existing CLIP similarity scores and spatial heuristics to rerank the candidate mask scores, without demonstrating that these scores reliably indicate negation constraint violations. The claim that negation constraints must be explicitly verified during mask selection, which is highlighted in the introduction section, is not well justified in the paper or by the proposed method. In particular, DCV seems just like a simple negative similarity penalty.
  • Limited evaluations: The proposed method is evaluated only on a negation-focused evaluation set (0.943.61% of the original datasets). The paper does not report results on the full RIS benchmarks. Moreover, while handling negation is a valid and promising research problem, the paper does not sufficiently justify its significance in real-world RIS applications, including how frequently such expressions occur and how important it is to handle them.
  • Insufficient justification: Although the paper evaluates the complementary effects of NegationCLIP in Table 2 and ablations in Table 4, these experiments do not sufficiently show the necessity of the proposed type-dependent verification framework. In particular, the full framework does not consistently outperform the single-verification variants (DCV-only or RCV-only). Moreover, the paper lacks sufficient analysis of whether categorizing negation into the two types is the most appropriate strategy. Additional comparisons with alternative or finer-grained categorizations would help justify the proposed design.
  • Extra costs: It requires an extra Qwen3-8B model to route each referring expression into two types, Candidate-local and Target-reference negations. It requires additional cost for this routing, but the paper does not analyze the additional inference time or computational overhead introduced by this process.
Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.