acceptodds
Withdrawn from ICLR 2027

Evidence Is Not Extent: Separating Referential Evidence, Object Extent, and Boundary Trust for Referring Image Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Parameter-efficient referring image segmentation asks one dense score to name the referent, cover its extent, and vouch for its boundary. However, evidence that identifies the referent is often sparse, such as a color patch or an anchor object, while the final mask must cover the full object extent. Therefore, sharpening the score may truncate the object, whereas a shape prior that completes it can spill onto same-category distractors. We introduce E3-RIS, separating these decisions by authority: each source may change only the pixels it is entitled to change. Prompt-Anchored Referential Evidence (PARE) pools prompt-initialized queries into one posterior that names the referent. Evidence-Constrained Bidirectional Projection (ECBP) lets frozen SAM add or remove pixels only inside a boundary band where the posterior agrees. Boundary Trust-Region Arbitration (BTRA) admits a direction-fixed edit when a learned score clears its threshold, and otherwise keeps the projected mask. On RefCOCO val, E3-RIS lifts mIoU over TALENT from 77.8 to 78.5 and [email protected] from 40.1 to 44.2. It also exceeds TALENT in mIoU across all nine official RefCOCO, RefCOCO+, and RefCOCOg splits.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
referring image segmentation, vision-language, parameter-efficient tuning, Segment Anything Model, boundary refinement, computer vision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XwaUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes E3-RIS, a referring image segmentation framework that separates target identification, object-extent recovery, and the decision of whether a boundary correction should be accepted. PARE estimates referential evidence, ECBP uses proposals from a frozen SAM model to make constrained local mask edits, and BTRA learns to judge the reliability of these edits. Experiments on referring segmentation benchmarks show improvements in mask quality, particularly at high IoU thresholds, together with a series of component and controlled comparisons.

Critical strengths

The paper addresses a meaningful problem in referring segmentation. Evidence that identifies the correct object does not necessarily describe its complete spatial extent, and separating these two roles provides a reasonable motivation for the proposed framework.

The method is carefully evaluated. In addition to standard ablations, the authors compare constrained editing with unrestricted replacement, include matched-capacity and matched-coverage controls, and study failures caused by unreliable mask changes. These experiments provide useful support for the boundary-editing design rather than attributing all improvements to additional model capacity.

The improvement in high-overlap segmentation is encouraging. On RefCOCO val, the B-scale model improves [email protected] from 40.1% to about 44.2%, while mIoU increases from 77.8% to about 78.5%. BTRA also outperforms the matched-capacity generic refiner in [email protected] (44.25% versus 42.94%; Table 4), suggesting that learning when to accept an edit can be useful.

Critical weaknesses
  1. The practical accuracy-efficiency trade-off is not sufficiently established.

    The reported gains, especially at [email protected], are valuable, but the complete system also introduces considerable inference overhead. Under the reported B-scale setting, latency increases from 49.56 ms for TALENT to 154.60 ms for E3-RIS, and peak memory increases from 0.80 GiB to 3.36 GiB (Table 13). This does not mean that the additional computation is unjustified, since accurate boundaries can be more important than mean IoU in some applications. However, the current evaluation does not clearly show which operating points offer the best balance between segmentation quality and computational cost. In addition, oIoU decreases on six of the nine reported splits for the B-scale comparison against TALENT (Table 1), indicating that the gains depend in part on the evaluation metric. A comparison of accuracy, high-overlap success rate, latency, and memory across reduced-cost configurations would make the practical contribution clearer.

  2. The additional benefit of the full BTRA ensemble needs more direct validation.

    The existing ablations do support the proposed editing and arbitration mechanisms, so the concern is not a lack of component studies. However, relative to ECBP, adding BTRA increases mIoU by only 0.22 points on RefCOCO val (78.23% to 78.45%) while increasing [email protected] by 1.62 points (42.63% to 44.25%). At the same time, latency rises from 101.70 ms to 154.60 ms (Table 13). These results suggest that BTRA has a particular advantage for highly accurate masks, but they do not show whether all three arbiters are needed to obtain that benefit. It would be useful to compare the current ensemble with a single arbiter or a lighter arbitration design under the same evaluation protocol.

Overall, the paper presents a well-motivated framework with convincing evidence for safer boundary editing and better high-overlap masks. My main reservation is whether the complete inference pipeline is necessary to achieve these benefits at a practical computational cost. A stronger efficiency-quality comparison, particularly with a reduced-cost BTRA configuration, could change my assessment.

Questions for authors
  1. Could the authors report an accuracy-latency-memory comparison across lighter E3-RIS variants, including the effect on [email protected]?
  2. How much of BTRA's high-overlap improvement remains when using one arbiter instead of the three-member ensemble, and what is the corresponding latency?
  3. Could the authors explain the oIoU decreases in the B-scale TALENT comparison and clarify which types of examples benefit most from the boundary-editing process?
Reviewer_PY5zOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes E3-RIS, a parameter-efficient framework for referring image segmentation that separates referential evidence, object extent, and boundary refinement into three stages. Specifically, PARE uses multiple prompt-initialized queries to extract referential evidence and generate an initial segmentation mask. ECBP then incorporates a frozen SAM model to refine object extent, allowing pixel additions or removals only within evidence-constrained boundary regions. Finally, BTRA uses a learned confidence score to selectively accept boundary modifications while preserving the original predictions when confidence is insufficient.

Critical strengths

Clear motivation and well-designed framework

Critical weaknesses

Limited methodological novelty. The framework mainly combines existing techniques, with limited algorithmic innovation.

Questions for authors

1.While the paper distinguishes E3-RIS from existing SAM-based and boundary refinement methods, the individual components are closely related to established techniques. Could the authors further clarify what is fundamentally new in the proposed authority-constrained editing strategy beyond their combination?

2.Since ECBP and BTRA only perform local modifications around the initial mask, how does the framework handle cases where PARE selects the wrong target instance? Could the authors clarify how strongly the final performance depends on the correctness of the initial mask?

3.Table 4 shows that ECBP with posterior veto achieves lower mIoU than local editing without the veto (78.23 vs. 78.33). Could the authors explain this result and compare both settings with the same BTRA refinement module? This would help clarify whether the proposed authority constraints are responsible for the final performance improvements.

Others(Not so important): Some part is too small in Figure.2. Figure.1 should show the main different from E3-RIS to other RIS method.

Reviewer_u7RpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes E3-RIS, a parameter-efficient referring image segmentation framework that separates three roles that are typically handled by a single dense prediction (referential evidence, object extent, and boundary refinement). PARE uses multiple prompt-initialized queries to improve referent identification, ECBP constrains SAM-based mask expansion and removal using the predicted posterior, and BTRA selectively accepts local boundary edits while keeping the current mask by default. The proposed framework achieve strong performance over RefCOCO variants.

Critical strengths
  • Well-motivated problem formulation: The distinction between referential evidence, object extent, and boundary refinement is intuitive and clearly motivated. The paper identifies a meaningful limitation of using a single dense score to handle these different objectives.

  • Strong empirical performance: E3-RIS achieves strong results across RefCOCO, RefCOCO+, and RefCOCOg, with particularly notable improvements at high IoU thresholds.

  • Extensive analysis and ablations: The paper provides thorough stage-wise analyses, ablations, and controlled comparisons, which help clarify where the performance gains come from and support the design choices of ECBP and BTRA

Critical weaknesses
  • Potentially SAM-specific motivation: The paper argues that referential evidence and object extent should be treated separately. While this is intuitive, it is not fully clear whether this is a general limitation of RIS or a problem that becomes particularly important when SAM is used for mask completion. Standard RIS models are already trained with full instance-level mask supervision, so it would be helpful to show that the evidence-vs.-extent conflict consistently appears across different RIS frameworks, including those without SAM. This would better establish the generality of the proposed problem formulation.

  • Limited empirical contribution of PARE: PARE is an important component of the paper's overall motivation, but its standalone gain is relatively modest. On RefCOCO val, it improves mIoU from 77.46 to 77.80 (+0.34). Given this small margin, the improvement may potentially be affected by training variance. The full upstream experiments are reported with a single training seed, while multi-seed results are only provided for the reduced-budget setting. Reporting multiple full-training runs with mean and standard deviation would provide stronger evidence that the gain is robust. In addition, PARE reduces distractor leakage but also decreases target completeness, suggesting that it may produce a more conservative posterior that is later complemented by SAM. More ablations on the number of queries, prompt initialization, and localization supervision would help clarify the specific contribution of PARE.

  • Several heuristic and hand-designed components: The proposed pipeline includes several manually designed choices, such as boundary bands, addition/removal rings, posterior thresholds, and separate edit thresholds. Although the paper provides sensitivity and transfer analyses, the overall design still feels somewhat heuristic. It would be helpful to provide stronger justification for why these particular constraints are necessary, or to compare against simpler or more learnable alternatives. This would make the proposed authority-separation principle feel less engineering-driven.

  • Unclear applicability to GRES: The experiments focus on standard RIS benchmarks such as RefCOCO, RefCOCO+, and RefCOCOg. It is therefore unclear how naturally the proposed framework extends to generalized referring expression segmentation (GRES), where the single-referent or single-mask assumption may not always hold. Since the paper presents evidence, extent, and boundary separation as a general principle, an evaluation or discussion in the GRES setting would strengthen this claim.

  • Missing comparisons with recent strong and efficient baselines: The comparison with recent strong RIS models such as One-Ref [A] and Latent-VG [B] appears to be missing. This is particularly important because some recent single-encoder approaches achieve strong segmentation performance with relatively efficient inference. Since E3-RIS introduces an additional SAM-based refinement pipeline, it would be useful to compare not only accuracy but also inference latency, memory usage, and overall computational cost against these methods. Such comparisons would help clarify whether the performance gains from incorporating SAM justify the additional inference overhead.

[A] Xiao et al., OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling, NeuRIPS 2024.

[B] Yu et al., Latent Expression Generation for Referring Image Segmentation and Grounding, ICCV 2025.

Questions for authors

Could the authors please comment on whether the proposed evidence–extent–boundary decomposition can be extended to the GRES setting, where multiple or even no referents may be present? It would be helpful to understand whether the current design assumptions still hold in this more general setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.