acceptodds
Withdrawn from ICLR 2027

RefCon: Learning What to Include & Exclude for MLLM Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) have enabled language-guided image segmentation, including referring expression segmentation. However, MLLM-based segmentation methods remain referent-centric; when exclusion information is used, it is typically encoded as sparse point prompts rather than explicit non-referent image-patch representations maintained throughout decoding. Consequently, explicit non-referent image-patch evidence generally remains absent from the MLLM-generated representation used for dense prediction, leaving the referent represented primarily through what should be included. We introduce RefCon, a framework that represents each referent through complementary MLLM-generated referent and non-referent image-patch representations and maintains both throughout decoding for repeated contrast. Given an image and instruction, the MLLM autoregressively generates two image-grounded token sets: Referent Tokens, corresponding to referred-object patches, and Contrast Tokens, corresponding to non-referent patches. To exploit this complementary evidence, we introduce a Referent–Contrast Decoder (RCD) that repeatedly refines both branches with image features and explicitly contrasts their token and dense-feature representations throughout refinement. To structure these token sets for role separation and non-redundant coverage, we further propose a Region-Aware Contrastive Transport (RCT) loss: its contrastive component aligns Referent Tokens with foreground evidence and Contrast Tokens with non-referent evidence while separating their roles, whereas its optimal-transport component reduces within-set redundancy and encourages collective coverage without assigning predefined subregions to individual tokens. Together, RefCon introduces explicit referent–non-referent representation into MLLM segmentation, with complementary inclusion and exclusion evidence generated by the MLLM and repeatedly contrasted throughout dense decoding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLMs, referring expression segmentation, referring expression comprehension, finegrained understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sqtsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RefCon, a framework for MLLM-based referring image segmentation that explicitly models both referent and non-referent visual information. The method introduces Referent Tokens and Contrast Tokens to represent target and non-target image patches, respectively. A Referent–Contrast Decoder (RCD) progressively refines and contrasts these representations, while a Region-Aware Contrastive Transport (RCT) loss combines contrastive learning and optimal transport to enhance discrimination and spatial coverage. Experiments on multiple benchmarks, including RefCOCO, RefCOCO+, RefCOCOg, gRefCOCO, and FP-RefCOCO variants, demonstrate consistent performance improvements over existing methods. Ablation studies further validate the effectiveness of the proposed components.

Critical strengths
  1. Clear motivation and coherent framework design. The paper addresses the limitation of referent-centric representations in existing MLLM-based segmentation methods by explicitly incorporating non-referent information. The proposed Referent–Contrast Decoder and Region-Aware Contrastive Transport loss form a coherent framework for progressively distinguishing target regions from non-target regions. This provides a meaningful perspective for improving fine-grained referring segmentation.

  2. Strong experimental performance and comprehensive validation. The proposed method achieves consistent improvements across multiple benchmarks, including referring expression segmentation, generalized referring segmentation, and false-premise segmentation. Furthermore, extensive ablation studies examine the contributions of individual components, token configurations, and decoder designs, providing empirical support for the effectiveness of the proposed framework.

Critical weaknesses
  1. Insufficient evidence for the discriminative role of Contrast Tokens. Although the paper emphasizes learning what to exclude, Contrast Tokens are supervised using randomly sampled non-referent image patches, which may include irrelevant background regions rather than semantically confusing objects. The paper lacks quantitative analysis demonstrating whether these tokens capture meaningful exclusion cues beyond general background information. This weakens the central motivation and the claimed contribution of explicit non-referent modeling.

  2. Limited justification for the effectiveness of optimal transport in reducing token redundancy. The proposed RCT loss uses optimal transport to encourage spatial coverage and reduce redundancy among tokens. However, the paper mainly supports this claim through qualitative visualizations and improvements in segmentation performance, without directly measuring token diversity, spatial overlap, or coverage. Consequently, it remains unclear whether the optimal transport component achieves the intended representation-level effects or primarily serves as an additional regularization mechanism.

Questions for authors
  1. Regarding the discriminative role of Contrast Tokens: Can the authors provide quantitative evidence demonstrating that the generated Contrast Tokens capture meaningful exclusion information rather than general background features? In particular, how do the results differ when using randomly selected background patches versus semantically confusing non-referent regions?

  2. Regarding the effectiveness of optimal transport: Can the authors provide quantitative analysis of token diversity, spatial overlap, and region coverage before and after introducing the optimal transport loss? Such evidence would help clarify whether the observed performance improvements result from reduced token redundancy and improved spatial coverage, as claimed.

3 comments
AuthorsOpenReview ↗

Response 1/2

Thank you for taking the time to review our paper and for your constructive feedback. We respond to each of your concerns below.

W1/Q1: Discriminative role of Contrast Tokens and hard-negative selection.

We appreciate the reviewer's concern regarding the discriminative role of Contrast Tokens. RefCon does not aim to discover specific semantically confusing objects or hard negatives; instead, its objective is to collectively represent the broader non-referent (background) region and explicitly exploit this evidence to distinguish the intended referent. Since a limited set of Contrast Tokens must represent a much larger set of non-referent patches, random sampling exposes the MLLM to diverse valid background locations, while Region-Aware Contrastive Transport (RCT) encourages these tokens to cover different portions of the background rather than concentrate on a limited subset of patches. Specifically, its optimal-transport (OT) objective encourages distributed coverage of the non-referent region without assigning predetermined subregions to individual tokens, while its contrastive objective strengthens referent–non-referent discrimination. Thus, the objective is broad, discriminative background representation rather than selecting particular semantic distractors.

To directly evaluate whether selecting harder negatives improves performance, we have added a comparison of random selection against two referent-guided alternatives in Appendix Table 12, also shown below:

Contrast Token selectionRC+ testB cIoU ↑RCg val cIoU ↑
Default (random)83.1086.60
Feature-space (visually similar negatives)82.9186.25
Spatial-location (nearby negatives)82.6085.84

Feature-space selection chooses non-referent patches closest to Referent Tokens in feature space, whereas spatial-location selection chooses patches closest to Referent Token locations. These are practical ways to select potentially confusing negatives without additional annotations. Selecting specific same-category or different-category background objects would require semantic or instance-level annotations beyond the referent mask, which are not generally available under standard referring segmentation supervision. Neither hard-negative strategy improves segmentation; random selection achieves the highest cIoU on both splits. This supports representing the broader background rather than restricting Contrast Tokens to visually similar or nearby distractors.

We also quantify whether explicitly exploiting Contrast Tokens provides useful exclusion evidence for identifying the referent. Before decoding, Table 3(c) shows that pseudo masks obtained by contrasting RefCon-generated Referent and Contrast Token activation maps achieve 49.8/50.8 cIoU, compared with 39.8/42.5 for PaDT, which generates only referent-oriented visual tokens. After decoding, the newly added Appendix Table 14 evaluates spatial maps derived from refined visual tokens and corresponding refined dense image features. By aggregating Referent and Contrast Token responses separately and subtracting the Contrast responses, RefCon achieves 80.1/76.3 cIoU, compared with 47.5/50.6 for PaDT. For final segmentation, Table 3(b) shows that explicit Referent–Contrast modeling improves cIoU over referent-only decoding from 80.7/83.5 to 83.1/86.6. These results demonstrate the benefit of explicitly exploiting non-referent evidence in token-derived spatial localization and final segmentation.

The quantitative results support the discriminative value of explicitly modeling non-referent evidence, while the newly added hard-negative comparison shows that the evaluated referent-guided selection strategies do not outperform random selection combined with RCT. We further analyze how OT distributes background coverage across Contrast Tokens in W2, with quantitative evidence in Appendix Table 10 and visualizations in Appendix Fig. 7.

AuthorsOpenReview ↗

Response 2/2

W2/Q2: Effectiveness of optimal transport in reducing token redundancy and improving spatial coverage.

We appreciate the reviewer's request for quantitative evidence of OT's effect on token diversity and spatial coverage. The OT objective in RCT operates on both Referent and Contrast Tokens, encouraging each token set to collectively represent different portions of its corresponding ground-truth region rather than concentrating on the same limited patches. This is particularly important because a limited set of tokens must represent a much larger image region. OT does not impose fixed token-to-subregion assignments; instead, it encourages the learned tokens to distribute their correspondence across their respective regions.

To directly evaluate whether OT reduces token redundancy and promotes distributed spatial coverage, we have added three quantitative representation-level measurements for contrast tokens in Appendix Table 10, also shown below. We measure token utilization to determine how many tokens participate in representing the non-referent region, spread evenness to quantify how evenly background patches are distributed across Contrast tokens, and role confusion to evaluate how effectively Contrast Token representations distinguish non-referent patches from referent patches. The table reports these measurements for Contrast Tokens, using cosine similarities between their MLLM-generated hidden states and image-patch embeddings; similar trends are observed for Referent Tokens.

RCT configurationTokens utilized (of 5) ↑Spread evenness ↑Role confusion ↓
Without RCT2.00.3230%
+ OT4.90.7620%
+ OT + contrastive (RCT)4.80.721%

For token utilization and spread evenness, each ground-truth non-referent patch is assigned to its highest-cosine-similarity Contrast Token. Utilization is the average number of tokens receiving at least one patch assignment per image, while spread evenness is the normalized entropy of the per-token assignment counts. For role confusion, each image patch is scored by its mean cosine similarity across Contrast Tokens, and we report 1 − AUC, treating non-referent patches as positives and referent patches as negatives. Lower role confusion indicates stronger discrimination between the two regions.

The results directly demonstrate that OT reduces token concentration and promotes more balanced representation of background patches. Without RCT, only 2.0 tokens receive background-patch assignments on average, with low spread evenness (0.32), indicating that a few tokens dominate. Adding OT increases utilization to 4.9 and spread evenness to 0.76, showing that nearly all tokens participate and background-patch assignments become substantially more balanced. Full RCT maintains this distribution (4.8; 0.72), while contrastive supervision further reduces role confusion from 20% to 1%. Thus, OT promotes distributed spatial representations, while contrastive supervision strengthens referent–non-referent discrimination.

We have also added per-token visualizations in Appendix Fig. 7, showing predicted Contrast Token locations, individual token–patch similarity maps, and background partitions. Without RCT, a few tokens dominate the background responses; OT distributes responses across different background portions, while contrastive supervision further reduces unwanted responses within the referent region. Table 10 quantifies how many Contrast Tokens receive non-referent patch assignments and how evenly those assignments are distributed. Figure 7 shows where those assignments fall in the image.

The representation-level improvements are also reflected in segmentation performance (Table 3(b)): adding OT improves cIoU from 81.5/84.6 to 82.2/85.6, while full RCT further improves it to 83.1/86.6.

Together, the newly added Appendix Table 10 and Fig. 7 provide quantitative and visual evidence that OT reduces concentration among tokens and encourages them to represent different portions of their corresponding regions. These representation-level findings, combined with the controlled segmentation ablations, support OT's intended contribution beyond an unspecified regularization effect.

We hope these clarifications and additional results address your concerns. We would appreciate your reconsideration of the overall assessment in light of the revised evidence. We’re happy to provide further details if helpful.

Dear Reviewer,

Thank you for your review. We have addressed your concerns in our rebuttal and revised manuscript. We would appreciate it if you could reconsider your assessment and revise your overall rating in light of our response.

Reviewer_dt2DOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RefCon for MLLM-based referring segmentation. It generates referent and non-referent visual tokens, refines them through a two-stream decoder, and introduces contrastive and optimal-transport supervision to improve their representations. Experiments across multiple referring segmentation benchmarks demonstrate consistent improvements over existing methods.

Critical strengths

Explicitly indicating what should be excluded during segmentation is well motivated. Maintaining non-referent visual evidence throughout decoding is a reasonable way to improve target discrimination.

Critical weaknesses

1-The contrastive loss aligns each Contrast Token with the entire non-referent region, which may contain highly heterogeneous objects and background features. Since these tokens share nearly identical positive sets, it is unclear how they learn complementary spatial evidence rather than similar global representations. Although OT is introduced to encourage coverage, its interaction with the contrastive objective remains insufficiently explained. Per-token similarity maps and quantitative representation analysis would provide stronger evidence.

2-Although the MLLM predicts visual tokens associated with specific image patches, the decoder operates on their hidden states without explicitly preserving the original spatial correspondence. Moreover, OT matches these hidden states to the entire corresponding region. Given that maintaining explicit patch-grounded representations throughout decoding is a central contribution, the lack of direct validation of this correspondence is concerning.

Questions for authors

1-How accurately do the MLLM-generated Referent and Contrast Tokens correspond to their intended regions? Such token-level grounding accuracy is important for validating the proposed representation mechanism.

2-What is the advantage of explicitly subtracting cross-stream features? Table 6 shows that directly concatenating the two feature streams achieves nearly identical performance.

8 comments
AuthorsOpenReview ↗

Response 1/4

Thank you for taking the time to review our paper and for your constructive feedback. We respond to each of your concerns below.

W1: Heterogeneous non-referent regions, token diversity, and the interaction of OT and contrastive supervision.

We appreciate the reviewer's observation that Contrast Tokens share the same heterogeneous non-referent patches as positives under the contrastive loss, potentially leading to redundant representations. The contrastive objective encourages Contrast Tokens to align with non-referent evidence and distinguish it from the intended referent. However, RefCon aims for Contrast Tokens to collectively represent the broader non-referent (background) region, encompassing visually similar distractors, neighboring objects, and diverse background features. Achieving this requires different Contrast Tokens to capture evidence from different spatial portions of the background, rather than concentrating their responses on the same background patches, which contrastive supervision alone does not explicitly enforce because all Contrast Tokens share the same set of non-referent patches as positives.

To address potential redundancy among Contrast Tokens, we introduce optimal transport (OT) alongside the contrastive loss as part of Region-Aware Contrastive Transport (RCT). OT encourages different Contrast Tokens to capture evidence from different background patches, allowing them to collectively represent the broader non-referent region rather than concentrating on the same spatial locations. Importantly, OT does not prescribe fixed subregions or predefined token-to-patch assignments. The same principle applies to Referent Tokens over foreground patches. Under the contrastive loss (Eqs. 8–10), Contrast Tokens are attracted toward non-referent patches and other Contrast Tokens, while being repelled from referent patches and Referent Tokens. Conversely, Referent Tokens are attracted toward referent patches and other Referent Tokens, while being repelled from non-referent patches and Contrast Tokens. Thus, OT encourages distributed spatial coverage within both token sets, while contrastive supervision strengthens referent–non-referent discrimination.

To directly verify OT's effectiveness in reducing token concentration and its interaction with contrastive supervision, we have added quantitative representation-level analysis in Appendix Table 10, also shown below. Token utilization measures how many Contrast Tokens participate in representing background patches, spread evenness measures how uniformly these patches are distributed across tokens, and role confusion measures how well Contrast Tokens distinguish non-referent from referent patches. We evaluate these metrics using cosine similarities between MLLM-generated Contrast Token hidden states and image-patch embeddings; similar trends are observed for Referent Tokens.

RCT configurationContrast Tokens utilized (of 5) ↑Spread evenness ↑Role confusion ↓
Without RCT2.00.3230%
+ OT4.90.7620%
+ OT + contrastive (RCT)4.80.721%

For utilization and spread evenness, each ground-truth background patch is assigned to its most cosine-similar Contrast Token. Utilization counts tokens receiving at least one assignment, while evenness is the normalized entropy of per-token assignment counts. Role confusion is 1 − AUC of mean Contrast Token–patch cosine similarity, treating background patches as positives and referent patches as negatives.

The results provide direct quantitative evidence that OT reduces the concentration of background-patch assignments on a few Contrast Tokens and promotes distributed spatial representations. Without RCT, only 2.0 Contrast Tokens are utilized on average, with spread evenness of 0.32, indicating that background-patch assignments concentrate on a few dominant tokens. Introducing OT increases utilization to 4.9 and evenness to 0.76, showing that nearly all Contrast Tokens participate with substantially more balanced spatial assignments. Adding contrastive supervision maintains this distribution (4.8; 0.72) while reducing role confusion from 20% to 1%, demonstrating that Contrast Token representations more reliably distinguish background patches from referent patches. Thus, the two objectives complement each other: OT distributes spatial evidence among tokens, while contrastive supervision strengthens the discrimination between the intended referent and non-referent regions.

AuthorsOpenReview ↗

Response 2/4

We have also added per-token visualizations in Appendix Fig. 7, showing predicted Contrast Token locations, individual Contrast Token–patch similarity maps, and background partitions based on maximum cosine similarity. These qualitatively confirm the quantitative findings: OT distributes token responses across different portions of the background, while adding contrastive supervision further reduces unwanted similarity responses within the referent region.

These representation-level improvements also translate into better final segmentation performance in Table 3(b): adding OT to RCD increases cIoU from 81.5/84.6 to 82.2/85.6, while full RCT further improves it to 83.1/86.6 on RefCOCO+ testB / RefCOCOg val, respectively.

Together, Appendix Table 10 and Fig. 7 provide quantitative and visual evidence that OT promotes complementary spatial coverage among Contrast Tokens, while contrastive supervision strengthens referent–non-referent discrimination while largely preserving distributed spatial coverage. Table 3(b) further confirms that these complementary effects improve final segmentation performance.

W2: Spatial correspondence of tokens during decoding.

We appreciate the reviewer's observation regarding the spatial correspondence of generated Referent and Contrast Tokens. We clarify that MLLM-generated Referent and Contrast Tokens are intended to encode broader spatial evidence from the referent and non-referent regions, respectively, beyond their initially predicted patches. Although each token initially corresponds to a specific image patch, its hidden-state representation is intended to capture semantically related evidence across a broader portion of its corresponding region, rather than remain restricted to that patch. The optimal-transport (OT) objective further encourages different Referent and Contrast Tokens to capture evidence from different portions of their respective regions, allowing each token set to collectively represent the broader foreground or background without enforcing fixed token-to-patch assignments or predefined subregions.

The Referent–Contrast Decoder (RCD) is designed to progressively refine these representations using evidence from the opposite stream, so that non-referent evidence informs refinement of the referent representations and vice versa. As described in Sec. 3.1 (Eqs. 2–6), MLLM-generated Referent and Contrast Token hidden states initialize separate decoder streams. Each RCR layer first performs Independent Two-Way Refinement, jointly refining tokens and dense image features within each stream using their corresponding referent or non-referent evidence. Cross-Stream Contrast then updates both streams' token and dense-feature representations relative to the opposite stream. Repeating these stages progressively strengthens referent–non-referent discrimination throughout decoding.

To evaluate how well the generated Referent and Contrast Token representations localize the intended referent before decoding, and how RCD refinement improves their localization, we have added Appendix Table 14, shown below. We construct localization maps directly from token representations before and after decoding, evaluate their cIoU against ground-truth referent masks, and compare RefCon with PaDT, which generates only referent-oriented tokens. These maps are evaluated independently of the final segmentation predicted by the Referent-stream mask task token. Appendix Table 14 combines the decoder-free results already reported in Table 3(c) with the new post-decoder evaluations.

ModelBefore decoder RC+ testB cIoU ↑Before decoder RCg val cIoU ↑After decoder RC+ testB cIoU ↑After decoder RCg val cIoU ↑
PaDT39.842.547.550.6
RefCon49.850.880.176.3

Before decoding, RefCon constructs pseudo masks from MLLM-generated token activation maps by averaging Referent and Contrast Token responses separately, subtracting the Contrast response from the Referent response, and normalizing and thresholding the positive result. PaDT averages its referent-only token activation maps. RefCon achieves 49.8/50.8 cIoU, outperforming PaDT by 10.0/8.3 points. After decoding, RefCon derives spatial maps from refined Referent and Contrast Token representations and their corresponding refined dense image features, aggregates responses within each stream, and subtracts the Contrast responses from the Referent responses. PaDT aggregates its refined referent-oriented token responses. RefCon achieves 80.1/76.3 cIoU, compared with 47.5/50.6 for PaDT. RefCon gains 30.3/25.5 cIoU points through decoder refinement, versus only 7.7/8.1 for PaDT. These results demonstrate that RefCon's generated token representations already provide stronger referent-localization evidence before decoding, and that their refined representations identify the intended referent substantially more accurately afterward.

AuthorsOpenReview ↗

Response 3/4

Together, Table 3(c) and the newly added Appendix Table 14 provide quantitative evidence that Referent and Contrast Token representations encode useful region-level evidence before decoding and become substantially more effective at localizing the intended referent through RCD refinement, by capturing broader foreground and background evidence, with OT encouraging different tokens within each set to represent distinct portions of their corresponding regions rather than redundant evidence.

Q1: Grounding accuracy of MLLM-generated Referent and Contrast Tokens.

We appreciate the reviewer's request for quantitative validation of the generated visual tokens' spatial grounding. We have added a token-level grounding analysis in Appendix Table 11, also shown below, measuring the percentage of MLLM-predicted patch IDs that fall within their intended ground-truth regions: the referent (foreground) region for Referent Tokens and the non-referent (background) region for Contrast Tokens.

ModelReferent Token accuracy ↑Contrast Token accuracy ↑
PaDT36.7%—
RefCon82.5%83.3%

RefCon achieves 82.5% Referent Token grounding accuracy, outperforming PaDT by 45.8 percentage points, while its Contrast Tokens achieve 83.3% grounding accuracy. These results directly demonstrate that the MLLM learns to generate both token types at spatial locations within their intended image regions, validating the initial patch grounding of the complementary Referent and Contrast representations.

Q2: Explicit subtraction versus concatenation in Cross-Stream Contrast.

We appreciate the reviewer's observation regarding the similar performance of subtraction and concatenation. As described in Sec. 3.1 (Eqs. 4–6), each RCR layer uses a Cross-Stream Contrast operation, denoted by Cϕ(A,B)C_{\phi}(A,B), to update both streams' token and dense image-feature representations. Here, AA denotes the representation being updated, BB denotes the corresponding evidence from the opposite stream, and ϕ\phi is a learned linear projection. Our subtraction formulation, Cϕ(A,B)=ϕ([A;A−B])C_{\phi}(A,B)=\phi([A;A-B]), preserves the current representation while explicitly incorporating its difference from the opposite stream. The concatenation variant instead uses Cϕ(A,B)=ϕ([A;B])C_{\phi}(A,B)=\phi([A;B]), providing both representations directly to the projection. For Referent and Contrast Tokens, BB is the mean of the opposite stream's generated tokens (Eq. 5). Task tokens are paired by role across streams (Eq. 4), while dense image features are contrasted at corresponding spatial locations (Eq. 6).

Their similar performance is mathematically expected because both formulations can represent equivalent linear transformations. For concatenation, the learned projection produces W1A+W2BW_1A+W_2B, whereas the subtraction formulation produces W1A+W2(A−B)=(W1+W2)A−W2BW_1A+W_2(A-B)=(W_1+W_2)A-W_2B. Since the projection weights are learned, either formulation can express the same cross-stream transformation. Explicit subtraction directly exposes the directional feature difference, while concatenation allows the linear projection to learn it implicitly.

To quantify how effectively different Cross-Stream Contrast designs separate the final Referent and Contrast Token representations, we have added their cosine similarity on RefCOCO+ testB in the newly added Appendix Table 13, alongside the segmentation results from Table 6. We compare RCD, which applies Cross-Stream Contrast at every RCR layer, with a late-contrast variant that applies the operation only once after independent refinement. This allows us to examine the importance of repeated cross-stream interaction alongside the choice between subtraction and concatenation.

RCD variantRC+ testB cIoU ↑RCg val cIoU ↑RC+ cosine similarity ↓
Late Cross-Stream Contrast78.078.9+0.407
RCD: Concatenation [A;B][A;B]83.086.4−0.557
RCD: Subtraction [A;A−B][A;A-B]83.186.6−0.558

Both RCD variants achieve nearly identical separation between the final Referent and Contrast Token representations (−0.557 versus −0.558), alongside nearly identical segmentation performance (83.0/86.4 versus 83.1/86.6). In contrast, applying Cross-Stream Contrast only once in the late-contrast variant yields substantially higher cosine similarity (+0.407) and lower cIoU (78.0/78.9). Unlike RCD, where each RCR layer refines both streams and then contrasts their updated representations, the late-contrast variant lacks repeated cross-stream interaction. The results demonstrate that applying Cross-Stream Contrast at every RCR layer produces substantially greater separation between the final Referent and Contrast representations than applying it only once.

Reviewer_QbTUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces RefCon, an MLLM-based method for referring image segmentation that has the MLLM generate both Referent Tokens (patches in the target mask) and Contrast Tokens (patches off the target mask), rather than referent-only representations as in prior work. The two token sets drive two parallel decoder streams that are repeatedly contrasted with each other during decoding, with only the referent stream producing the final mask. An auxiliary contrastive and optimal-transport loss keeps the two token sets distinct and spread over their regions. The evaluation is done on RefCOCO, RefCOCO+, and RefCOCOg for mIoU and no-target accuracy, showing better performance than existing methods. Ablation studies show that the proposed components are helpful.

Critical strengths
  1. The proposed method outperforms existing methods in reference segmentation tasks
  2. Ablation studies show the proposed component is helpful.
Critical weaknesses
  1. The writing and figures can be improved. The method figure is very crowded, and hard to parse. It doesn't really help me to understand how the method works. The writing of the method is also very hard to parse.
  2. The analysis about why the method works is somehow missing. The proposed method is reasonable. But why the final mask supervision, which already contains the negative pixels suppression, is not enough here, and how the negative patches work. Except for the teaser and method figures, there is not any predicted negative patches visualization. Providing them might be helpful to understand why the method works better.
  3. One question to the proposed method is that, how the model knows which patches to predict as the non-deterministic contrast patches? Any patches can be valid in the background, and the supervision can be very noisy. Is this an issue or not?
Questions for authors

The clarification for #2 and #3 critical weaknesses will be helpful.

4 comments
AuthorsOpenReview ↗

Response 1/3

Thank you for taking the time to review our paper and for your constructive feedback. We respond to each of your concerns below.

W1: Clarity of method presentation and figures.

We appreciate the reviewer's feedback regarding the clarity of the methodology and architecture figure. We have revised the main architecture figure (Fig. 2), increasing the spacing between components to make the information flow and individual operations easier to follow. We have also added a detailed architectural illustration in Appendix Fig. 10, which separately explains (A) initialization of the Referent and Contrast streams, (B) Independent Two-Way Refinement and Cross-Stream Contrast within each RCR layer, and (C) Region-Aware Contrastive Transport (RCT) supervision. In particular, the expanded illustration clarifies how task tokens, generated visual tokens, and dense image features interact across the two streams.

We have also refined the Introduction, methodology description (Sec. 3), and Fig. 2 caption to clarify how the MLLM-generated tokens initialize the decoder, how Independent Two-Way Refinement jointly updates tokens and dense image features within each stream, and how Cross-Stream Contrast subsequently updates both streams relative to each other. These revisions provide a clearer explanation of RefCon's architecture and operations while preserving the original methodological formulation.

W2: Why final-mask supervision alone is insufficient and how negative patches improve segmentation.

We appreciate the reviewer's observation that final-mask supervision already penalizes incorrectly included background pixels. However, suppressing background pixels through the final-mask loss differs from explicitly representing non-referent image evidence alongside the referent and repeatedly exploiting it throughout decoding. The final-mask loss penalizes incorrect predictions but does not explicitly establish separate non-referent token representations that can inform progressive referent refinement during decoding.

RefCon addresses this by enabling the MLLM to generate both Referent Tokens associated with foreground patches and Contrast Tokens associated with the broader non-referent region (Sec. 3.1). Rather than selecting specific negative objects, Contrast Tokens collectively represent broader background evidence. Unlike approaches relying on referent-only representations or decoder prompts [1–3], RefCon maintains separate Referent and Contrast streams throughout decoding, repeatedly using non-referent evidence to distinguish and refine the intended referent. Each RCR layer first performs Independent Two-Way Refinement, jointly updating tokens and dense image features within each stream, followed by Cross-Stream Contrast, which updates both streams relative to each other (Eqs. 2–6). Region-Aware Contrastive Transport (RCT) further strengthens these representations: contrastive supervision aligns Referent Tokens with foreground evidence and Contrast Tokens with non-referent evidence while separating the two representations, whereas optimal transport encourages distributed spatial coverage within each token set without predefined token-to-subregion assignments.

The benefit of explicitly modeling and progressively exploiting background evidence is demonstrated by the component ablations in Table 3(b) shown in the below table:

ConfigurationRC+ testB cIoU ↑RCg val cIoU ↑
Referent-only (5 tokens)80.783.5
RCD without RCT81.584.6
RCD + OT82.285.6
Full RefCon (RCD + RCT)83.186.6

The referent-only baseline already suppresses background pixels through final-mask supervision, achieving 80.7/83.5 cIoU. By introducing the Referent–Contrast Decoder (RCD), which progressively refines the Referent stream using explicit non-referent evidence from the Contrast stream, performance improves to 81.5/84.6 even without RCT. Adding OT further improves performance to 82.2/85.6, while full RCT, combining OT and contrastive supervision, achieves 83.1/86.6. These results demonstrate that explicitly maintaining and exploiting non-referent evidence during decoding provides benefits beyond final-mask supervision, with additional improvements from RCT's representation-level constraints.

AuthorsOpenReview ↗

Response 2/3

We further quantify the contribution of Contrast Tokens to referent localization by comparing RefCon's spatial maps derived from both Referent and Contrast Token representations with those of PaDT, which generates only referent-oriented visual tokens, before and after decoding, using cIoU against ground-truth referent masks in the below table (Table 3(c) and newly added Appendix Table 14):

EvaluationModelRC+ testB cIoU ↑RCg val cIoU ↑
Before decodingPaDT39.842.5
RefCon49.850.8
After decodingPaDT47.550.6
RefCon80.176.3

Before decoding, RefCon constructs pseudo masks by subtracting the averaged Contrast Token activation maps from the averaged Referent Token activation maps, achieving 49.8/50.8 cIoU, compared with 39.8/42.5 for PaDT, which generates only referent-oriented visual tokens. This demonstrates the benefit of incorporating explicit non-referent evidence into spatial localization at the MLLM level. After decoding, spatial maps are constructed from refined visual tokens and their corresponding refined dense image features. RefCon aggregates Referent and Contrast Token responses separately and subtracts the Contrast responses, achieving 80.1/76.3 cIoU, compared with 47.5/50.6 for PaDT. Thus, RefCon's token-derived spatial-map cIoU improves by 30.3/25.5 points through decoding, compared with 7.7/8.1 for PaDT, supporting the effectiveness of progressively refining and contrasting both streams. These evaluations examine the visual-token representations separately from the final segmentation predicted by the Referent-stream mask task token.

To directly address the request for predicted negative-patch visualizations, we have added Appendix Fig. 7, showing MLLM-predicted Contrast Token locations, individual Contrast Token–patch similarity maps, and background partitions. The figure illustrates how different Contrast Tokens represent different portions of the non-referent region, rather than being restricted to predefined negative objects. It also visualizes the effect of RCT: OT distributes token responses across the background, while contrastive supervision reduces unwanted responses within the referent region.

Together, the controlled ablations, token-derived spatial-map evaluations, and predicted Contrast Token visualizations explain the benefit beyond final-mask supervision: Contrast Tokens provide explicit non-referent representations, RCT encourages distributed and discriminative spatial evidence, and RCD repeatedly exploits these representations to refine the intended referent throughout decoding.

AuthorsOpenReview ↗

Response 3/3

W3: Non-deterministic Contrast Token predictions and supervision reliability.

We appreciate the reviewer's concern regarding the non-deterministic supervision of Contrast Tokens. RefCon does not require Contrast Tokens to predict a unique or predetermined set of non-referent patches; instead, a limited number of Contrast Tokens collectively represent a much larger non-referent (background) region, with each token encouraged to capture spatial evidence from a broader subregion rather than remaining restricted to its initially predicted patch.. During training, Contrast Token targets are randomly sampled from the ground-truth non-referent region (Sec. 3.1), exposing the MLLM to diverse valid background locations rather than restricting supervision to a selected subset of negatives. This sampling is complemented by Region-Aware Contrastive Transport (RCT): its optimal-transport (OT) objective encourages different Contrast Tokens to represent different portions of the non-referent region without predefined token-to-subregion assignments, while contrastive supervision aligns them with non-referent patches and distinguishes them from referent patches. Thus, random sampling provides diverse background supervision, while RCT encourages distributed and discriminative token representations.

To directly evaluate whether more selective Contrast Token target selection improves performance, we have added a comparison of random selection against two referent-guided alternatives in Appendix Table 12 shown below:

Contrast Token selectionRC+ testB cIoU ↑RCg val cIoU ↑
Default (random)83.1086.60
Feature-space (visually similar negatives)82.9186.25
Spatial-location (nearby negatives)82.6085.84

Feature-space selection chooses non-referent patches closest to Referent Tokens in feature space, whereas spatial-location selection chooses patches closest to Referent Token locations in the image. Neither hard-negative selection strategy improves segmentation of referent; random selection achieves the highest cIoU on both splits. These results support learning from diverse valid background patches rather than restricting Contrast Tokens to particular subsets, such as negatives that are visually similar to or spatially close to the intended referent.

We further validate whether the MLLM learns to predict patches within the intended non-referent region. The newly added Appendix Table 11 reports 83.3% Contrast Token grounding accuracy, measured as the percentage of MLLM-predicted Contrast Token patch IDs falling within the ground-truth non-referent region. As discussed in our response to W2, the token-derived localization results in Appendix Table 14 show that RefCon's complementary Referent and Contrast representations localize the intended referent more accurately than PaDT's referent-only representations, both before and after decoding. Together with the grounding results in Appendix Table 11, these findings demonstrate that stochastic background-patch supervision produces meaningful non-referent token predictions and representations that contribute to identifying the intended referent.

We have also added Appendix Fig. 7, showing MLLM-predicted Contrast Token locations and individual token–patch similarity maps, illustrating their spatial correspondence with different portions of the non-referent region.

Together, these results support random sampling as an effective strategy for learning Contrast Token representations from the broader non-referent region. The grounding accuracy, token-derived localization results, and hard-negative comparisons provide quantitative validation, while RCT encourages distributed and discriminative representations without requiring predetermined negative-patch targets.

References: [1] UGround (Qian et al., ICML 2026); [2] SAM4MLLM (Chen et al., ECCV 2024); [3] PaDT (Su et al., ICLR 2026).

We hope these clarifications and additional results address your concerns. We would appreciate your reconsideration of the overall assessment in light of the revised evidence. We’re happy to provide further details if helpful.

Dear Reviewer,

Thank you for your review. We have addressed your concerns in our rebuttal and revised manuscript. We would appreciate it if you could reconsider your assessment and revise your overall rating in light of our response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.