acceptodds
Under review as a conference paper at ICLR 2027

Referential Recalibration for Fine-Grained Referring Expression Segmentation

Abstract

Fine-grained referring expression segmentation extends classic RES from segmenting an entire referred object to localizing a specific part within a host-object. Most existing methods formulate this task as a direct mapping from a holistic expression to a target mask, which works well for object-level grounding. However, when the referred target is a fine-grained part of an object, the model is prone to assigning the referential focus to the host-object rather than the target-part. Moreover, even when the target-part is partially distinguished, the resulting score map may remain incomplete or drift into irrelevant regions. To address these issues, we introduce Referential Attribution Reformulation, which turns the host-object from a source of interference into useful guidance for target-part localization, and Bidirectional Collaborative Rectification, which further refines the initial target-part score map by recovering missing regions and suppressing drift-prone activations using internal target-part support and structural affinity. Our framework is training-free and improves VLM-based segmentation without additional fine-grained annotations or parameter updates. Extensive experiments demonstrate consistent improvements across different vision-language segmentation paradigms, supporting the effectiveness and broad compatibility of the proposed inference-time framework.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Referential Recalibration, Multimodal Large Language Models, Fine-grained Visual Grounding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_67cUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a training-free framework for fine-grained referring expression segmentation. Referential Attribution Reformulation (RAR) decomposes an expression into host-object and target-part descriptions, using host localization to constrain subsequent part localization. Bidirectional Collaborative Rectification (BCR) then refines the part score map through coordinated propagation and suppression, guided by internal text-to-image attention and visual feature affinity.

The framework is evaluated on RefCOCOm using LISA, M²SA, and Qwen3-VL combined with SAM. Additional experiments assess compatibility with object-level RES on RefCOCO, RefCOCO+, and RefCOCOg. The paper also provides component ablations, a comparison with SAMRefiner, parameter sensitivity analyses, qualitative results, and inference cost measurements.

Critical strengths

1. The paper addresses a meaningful problem and reports improvements across multiple base models. Fine-grained part localization remains substantially more difficult than object-level localization. In Table 3, val-Part mIoU improves from 21.3 to 24.4 for LISA-7B, from 24.5 to 27.4 for M²SA, and from 33.4 to 39.8 for Qwen3-VL+SAM. These results suggest practical value in improving existing models without parameter updates and constitute the strongest evidence supporting acceptance.

2. The design is motivated by the task and has preliminary ablation support. Host localization preserves instance-level context, while the part description focuses the subsequent prediction on the intended region. BCR coordinates expansion and suppression rather than applying either indiscriminately. The full framework outperforms either component alone in Table 4, and collaborative rectification improves mIoU from 23.0 to 24.4 in Table 5. These findings provide initial evidence for complementary contributions from the proposed components.

Critical weaknesses

1. The experimental evidence conflicts. Table 4 reports 23.1 for “LISA+BCR,” whereas Table 9 reports 22.1 for the apparently corresponding “Original Expression+BCR” setting. Both are evaluated on RefCOCOm val-Part, but no setting difference is explained. This discrepancy affects the estimated standalone contribution of BCR.

2. Insufficient specification of the algorithm and its integration across segmentation paradigms. Section 5.3 describes Qwen3-VL predicting a box for SAM, but does not explain the source of the score maps used by BCR, where rectification occurs, or how attention and visual representations are spatially aligned.

3. Incomplete ablation design. The experiments lack a “host-constrained image+original expression” control for RAR, as well as separate propagation-only and suppression-only ablations for BCR. Moreover, all core ablations are conducted on LISA-7B, although the largest gains occur with Qwen3-VL+SAM. Therefore, the current evidence does not adequately establish which design choices explain the improvements across paradigms.

Questions for authors

1. Do the 23.1 result in Table 4 and the 22.1 result in Table 9 correspond to the same experimental setting? If not, please specify the differences.

2. Could you provide minimal controls that isolate the source of the gains? In particular, “host-constrained image+original expression” on LISA, and RAR-only, BCR-only, and full-framework results on Qwen3-VL+SAM would help distinguish the effects of spatial restriction from those of referential reformulation and collaborative rectification.

1 comment

Thank you for the detailed response, which has addressed my methodological concerns. My remaining concern is that the fine-grained experiments mainly rely on relatively older base models and baselines, so it is unclear whether the framework still helps with more recent and stronger models. Are there more recent methods on RefCOCOm that could be included for comparison? If so, could the authors evaluate the framework on top of such a stronger recent model?

Reviewer_C4xMOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free inference framework for fine-grained referring expression segmentation (RES). Referential Attribution Reformulation (RAR) uses an instruction-following language model to separate an expression into host-object and target-part descriptions, which enables a base RES model to segment the input image twice for object and part score map. Bidirectional Collaborative Rectification (BCR) refines the resulting score map using text-to-image prefill attention and visual patch affinities

Critical strengths

Confusing a referred part with its host is a meaningful failure mode. Using the host to establish the visual scope before asking for the part is intuitive and potentially useful, particularly when a frozen model already has stronger object-level than part-level localization ability. The author also discusses in detail the influence of latency on operations in practice.

Critical weaknesses
  1. The host-guided object-to-part localization strategy is closely related to HOPS[1], which also uses affinity-guided propagation to recover weak part responses. The authors should clarify the incremental contribution of RAR and BCR beyond these established designs. Suppressing irrelevant responses is also connected to TALENT[2]. The authors are encouraged to discuss this relevant work.
  2. Although the authors report the additional inference cost, I am not yet convinced whether this latency statistic only represents the doubled cost caused by two backbone computations in RAR, or includes the LLM's computation latency (and memory usage). Moreover, it is unclear whether LLama-3-8B-Instruct performs expression decomposition online or precomputes and caches its outputs. Please clarify whether the reported inference time and GPU memory usage include this LLaMa.
  3. RAR could potentially use a smaller language model. Separating host-object and target-part descriptions appears to be a relatively simple language task, making an additional 8B model potentially excessive. I encourage the authors to explore smaller language models and report their decomposition quality and inference overhead. Furthermore, the evaluated base pipelines already incorporate multimodal language models, which could potentially perform this decomposition. The paper does not discuss this possibility.

[1] Li, X., Lin, D., Gao, S., Liu, Y., He, J., Li, J., Liu, R., Guo, Q., Yang, K. and Feng, W., 2026. HOPS: Hierarchical Open-vocabulary Part Segmentation with Attention-Aware Filtering and Affinity-Guided Enhancement. In CVPR (pp. 27719-27729).

[2] Jin, S., Yu, S., Zhang, B., Yao, C., Liu, M. and Xiao, J., 2026. TALENT: Target-aware Efficient Tuning for Referring Image Segmentation. in CVPR (pp. 7472-7482).

Questions for authors
  1. The “LISA + BCR” setting in Table 4 is insufficiently specified. Lines 416–418 suggest that the initial target-part score map is obtained directly from the original expression. However, BCR also requires the host-object score map SoS_o and target-part token indices to construct AtA_t. How are these obtained when RAR is removed? If expression decomposition and host-object localization are retained, this setting still uses part of RAR.
  2. The handling of negative cosine similarities in BCR is unclear. In Eq. (3), A(i,j)A(i,j) can be negative, so the aggregated maps RpR_p and RdR_d in Eqs. (4)–(5) may also be negative despite the nonnegative evidence terms. Consequently, the “positive” update could decrease target-part scores, while subtracting the “negative” update could increase them, contrary to the stated propagation and suppression roles. Have the authors considered this case? Please clarify this.
Reviewer_aPpJOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a method called referential recalibration for fine-grained referring expression segmentation. In particular, referential attribution reformulation is proposed to turn the host-object into useful guidance for target-part localization. In addition, bidirectional collaboration rectification is designed to refine the initial target-part score map and suppressing drift-prone activations by using internal structural affinity. The whole method is training-free and improves VLM-based segmentation without fine-tuning. Experiments on fine-grained and classical RES benchmarks proved the effectiveness of the proposed method.

Critical strengths

(1) The motivation is well presented of the referential recalibration for fine-grained referring expression segmentation.

(2) The illustrations are mostly clear and intuitive of the referential attribution reformulation and the bidirectional collaboration rectification.

(3) The proposed training-free method exploits host-object guidance and internal structural affinity which sounds interesting for the community.

Critical weaknesses

(1) The whole framework exploits host-object guidance and internal structural affinity to improve performance in a training-free manner. However, the guidance and affinity rely on the statistics of the base model, which are prone to failure if the adjustment is inappropriate. There is no discussion on the robustness of the whole framework.

(2) On Page 4, Line 203-204, regions outside the host-object score map are blurred to reduce interference. Though sounds reasonable, this strategy highly relies on the assumption of accurate host-object score map. If the score map is inaccurate, this strategy may fail.

(3) On Page 6, Line 304-305, the final score map is refined by adding the weighted positive correction and subtracting the weighted negative correction. However, there is no visualization and discussion on how the two terms behave. In addition, such a formula may exist partial cancellation if the positive or the negative correction is inaccurate.

(4) On Page 6, Line 309-310, the refined score map is converted to final mask following the original mask generation procedure of base model. Since the whole method is training-free, the adjusted score map is not aligned with the base model. There is no discussion on the suitability between the adjusted score map and the base model.

Questions for authors

No.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.