- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RefCon, a framework for MLLM-based referring image segmentation that explicitly models both referent and non-referent visual information. The method introduces Referent Tokens and Contrast Tokens to represent target and non-target image patches, respectively. A Referent–Contrast Decoder (RCD) progressively refines and contrasts these representations, while a Region-Aware Contrastive Transport (RCT) loss combines contrastive learning and optimal transport to enhance discrimination and spatial coverage. Experiments on multiple benchmarks, including RefCOCO, RefCOCO+, RefCOCOg, gRefCOCO, and FP-RefCOCO variants, demonstrate consistent performance improvements over existing methods. Ablation studies further validate the effectiveness of the proposed components.
Critical strengths
-
Clear motivation and coherent framework design. The paper addresses the limitation of referent-centric representations in existing MLLM-based segmentation methods by explicitly incorporating non-referent information. The proposed Referent–Contrast Decoder and Region-Aware Contrastive Transport loss form a coherent framework for progressively distinguishing target regions from non-target regions. This provides a meaningful perspective for improving fine-grained referring segmentation.
-
Strong experimental performance and comprehensive validation. The proposed method achieves consistent improvements across multiple benchmarks, including referring expression segmentation, generalized referring segmentation, and false-premise segmentation. Furthermore, extensive ablation studies examine the contributions of individual components, token configurations, and decoder designs, providing empirical support for the effectiveness of the proposed framework.
Critical weaknesses
-
Insufficient evidence for the discriminative role of Contrast Tokens. Although the paper emphasizes learning what to exclude, Contrast Tokens are supervised using randomly sampled non-referent image patches, which may include irrelevant background regions rather than semantically confusing objects. The paper lacks quantitative analysis demonstrating whether these tokens capture meaningful exclusion cues beyond general background information. This weakens the central motivation and the claimed contribution of explicit non-referent modeling.
-
Limited justification for the effectiveness of optimal transport in reducing token redundancy. The proposed RCT loss uses optimal transport to encourage spatial coverage and reduce redundancy among tokens. However, the paper mainly supports this claim through qualitative visualizations and improvements in segmentation performance, without directly measuring token diversity, spatial overlap, or coverage. Consequently, it remains unclear whether the optimal transport component achieves the intended representation-level effects or primarily serves as an additional regularization mechanism.
Questions for authors
-
Regarding the discriminative role of Contrast Tokens: Can the authors provide quantitative evidence demonstrating that the generated Contrast Tokens capture meaningful exclusion information rather than general background features? In particular, how do the results differ when using randomly selected background patches versus semantically confusing non-referent regions?
-
Regarding the effectiveness of optimal transport: Can the authors provide quantitative analysis of token diversity, spatial overlap, and region coverage before and after introducing the optimal transport loss? Such evidence would help clarify whether the observed performance improvements result from reduced token redundancy and improved spatial coverage, as claimed.
3 comments
Response 2/2
W2/Q2: Effectiveness of optimal transport in reducing token redundancy and improving spatial coverage.
We appreciate the reviewer's request for quantitative evidence of OT's effect on token diversity and spatial coverage. The OT objective in RCT operates on both Referent and Contrast Tokens, encouraging each token set to collectively represent different portions of its corresponding ground-truth region rather than concentrating on the same limited patches. This is particularly important because a limited set of tokens must represent a much larger image region. OT does not impose fixed token-to-subregion assignments; instead, it encourages the learned tokens to distribute their correspondence across their respective regions.
To directly evaluate whether OT reduces token redundancy and promotes distributed spatial coverage, we have added three quantitative representation-level measurements for contrast tokens in Appendix Table 10, also shown below. We measure token utilization to determine how many tokens participate in representing the non-referent region, spread evenness to quantify how evenly background patches are distributed across Contrast tokens, and role confusion to evaluate how effectively Contrast Token representations distinguish non-referent patches from referent patches. The table reports these measurements for Contrast Tokens, using cosine similarities between their MLLM-generated hidden states and image-patch embeddings; similar trends are observed for Referent Tokens.
| RCT configuration | Tokens utilized (of 5) ↑ | Spread evenness ↑ | Role confusion ↓ |
|---|---|---|---|
| Without RCT | 2.0 | 0.32 | 30% |
| + OT | 4.9 | 0.76 | 20% |
| + OT + contrastive (RCT) | 4.8 | 0.72 | 1% |
For token utilization and spread evenness, each ground-truth non-referent patch is assigned to its highest-cosine-similarity Contrast Token. Utilization is the average number of tokens receiving at least one patch assignment per image, while spread evenness is the normalized entropy of the per-token assignment counts. For role confusion, each image patch is scored by its mean cosine similarity across Contrast Tokens, and we report 1 − AUC, treating non-referent patches as positives and referent patches as negatives. Lower role confusion indicates stronger discrimination between the two regions.
The results directly demonstrate that OT reduces token concentration and promotes more balanced representation of background patches. Without RCT, only 2.0 tokens receive background-patch assignments on average, with low spread evenness (0.32), indicating that a few tokens dominate. Adding OT increases utilization to 4.9 and spread evenness to 0.76, showing that nearly all tokens participate and background-patch assignments become substantially more balanced. Full RCT maintains this distribution (4.8; 0.72), while contrastive supervision further reduces role confusion from 20% to 1%. Thus, OT promotes distributed spatial representations, while contrastive supervision strengthens referent–non-referent discrimination.
We have also added per-token visualizations in Appendix Fig. 7, showing predicted Contrast Token locations, individual token–patch similarity maps, and background partitions. Without RCT, a few tokens dominate the background responses; OT distributes responses across different background portions, while contrastive supervision further reduces unwanted responses within the referent region. Table 10 quantifies how many Contrast Tokens receive non-referent patch assignments and how evenly those assignments are distributed. Figure 7 shows where those assignments fall in the image.
The representation-level improvements are also reflected in segmentation performance (Table 3(b)): adding OT improves cIoU from 81.5/84.6 to 82.2/85.6, while full RCT further improves it to 83.1/86.6.
Together, the newly added Appendix Table 10 and Fig. 7 provide quantitative and visual evidence that OT reduces concentration among tokens and encourages them to represent different portions of their corresponding regions. These representation-level findings, combined with the controlled segmentation ablations, support OT's intended contribution beyond an unspecified regularization effect.
We hope these clarifications and additional results address your concerns. We would appreciate your reconsideration of the overall assessment in light of the revised evidence. We’re happy to provide further details if helpful.
Dear Reviewer,
Thank you for your review. We have addressed your concerns in our rebuttal and revised manuscript. We would appreciate it if you could reconsider your assessment and revise your overall rating in light of our response.
Response 1/2
Thank you for taking the time to review our paper and for your constructive feedback. We respond to each of your concerns below.
W1/Q1: Discriminative role of Contrast Tokens and hard-negative selection.
We appreciate the reviewer's concern regarding the discriminative role of Contrast Tokens. RefCon does not aim to discover specific semantically confusing objects or hard negatives; instead, its objective is to collectively represent the broader non-referent (background) region and explicitly exploit this evidence to distinguish the intended referent. Since a limited set of Contrast Tokens must represent a much larger set of non-referent patches, random sampling exposes the MLLM to diverse valid background locations, while Region-Aware Contrastive Transport (RCT) encourages these tokens to cover different portions of the background rather than concentrate on a limited subset of patches. Specifically, its optimal-transport (OT) objective encourages distributed coverage of the non-referent region without assigning predetermined subregions to individual tokens, while its contrastive objective strengthens referent–non-referent discrimination. Thus, the objective is broad, discriminative background representation rather than selecting particular semantic distractors.
To directly evaluate whether selecting harder negatives improves performance, we have added a comparison of random selection against two referent-guided alternatives in Appendix Table 12, also shown below:
Feature-space selection chooses non-referent patches closest to Referent Tokens in feature space, whereas spatial-location selection chooses patches closest to Referent Token locations. These are practical ways to select potentially confusing negatives without additional annotations. Selecting specific same-category or different-category background objects would require semantic or instance-level annotations beyond the referent mask, which are not generally available under standard referring segmentation supervision. Neither hard-negative strategy improves segmentation; random selection achieves the highest cIoU on both splits. This supports representing the broader background rather than restricting Contrast Tokens to visually similar or nearby distractors.
We also quantify whether explicitly exploiting Contrast Tokens provides useful exclusion evidence for identifying the referent. Before decoding, Table 3(c) shows that pseudo masks obtained by contrasting RefCon-generated Referent and Contrast Token activation maps achieve 49.8/50.8 cIoU, compared with 39.8/42.5 for PaDT, which generates only referent-oriented visual tokens. After decoding, the newly added Appendix Table 14 evaluates spatial maps derived from refined visual tokens and corresponding refined dense image features. By aggregating Referent and Contrast Token responses separately and subtracting the Contrast responses, RefCon achieves 80.1/76.3 cIoU, compared with 47.5/50.6 for PaDT. For final segmentation, Table 3(b) shows that explicit Referent–Contrast modeling improves cIoU over referent-only decoding from 80.7/83.5 to 83.1/86.6. These results demonstrate the benefit of explicitly exploiting non-referent evidence in token-derived spatial localization and final segmentation.
The quantitative results support the discriminative value of explicitly modeling non-referent evidence, while the newly added hard-negative comparison shows that the evaluated referent-guided selection strategies do not outperform random selection combined with RCT. We further analyze how OT distributes background coverage across Contrast Tokens in W2, with quantitative evidence in Appendix Table 10 and visualizations in Appendix Fig. 7.