acceptodds
Under review as a conference paper at ICLR 2027

LatentRefine-SegZero: Latent-Space Refinement for Referring Segmentation via Reflective Reasoning

11 views

Abstract

Significant progress has been made in referring segmentation by combining multimodal large language models (MLLMs) with the Segment Anything Model (SAM). However, balancing in-domain accuracy and out-of-domain generalization remains challenging. SFT-based methods achieve strong in-domain segmentation performance, but updating pretrained MLLM parameters with pixel-level losses may limit generalization to out-of-domain referring expressions. RL-based methods that avoid such updates typically control SAM through discrete box and point prompts, limiting fine-grained mask refinement. To mitigate this trade-off, we propose LatentRefine-SegZero, a latent-space refinement framework for referring segmentation based on reflective reinforcement learning. Our method comprises three components: (1) , which uses reflective reasoning to assess proposal reliability and select a refinement pathway; (2) , which converts proposal representations into supplementary latent prompts for mask refinement based on reliable proposals; and (3) , which converts reflective representations into new latent prompts for target relocalization. GRPO optimizes proposal generation and reliability assessment; pixel-level supervision trains lightweight adaptation modules without updating pretrained MLLM backbone weights. Experiments demonstrate that LatentRefine-SegZero achieves a favorable balance between in-domain segmentation accuracy and out-of-domain generalization. It leads the compared methods without pixel-loss updates to pretrained MLLM parameters by 1.4–3.9 cIoU points across all eight RefCOCO-series splits, while outperforming the compared methods with such updates on RefAdv and PR-Bench. It also improves over the Qwen2.5-VL+SAM2 baseline in both domains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
referring image segmentation, multimodal large language models, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_H2xsOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LatentRefine-SegZero, a reliability-aware framework for referring segmentation. It uses reflective reasoning to assess an initial MLLM-generated proposal and routes it to either semantic alignment or semantic relocalization, with lightweight connectors mapping MLLM features into SAM-compatible prompts.

Critical strengths
  1. The paper proposes a clear reliability-aware refinement strategy for referring segmentation. By separating reliable proposals from unreliable ones and routing them to semantic alignment or semantic relocalization, the method provides a meaningful way to handle different failure modes rather than applying the same refinement strategy to all cases.

  2. The experimental evaluation is relatively comprehensive and consistently supports the proposed design. The method is tested on both in-domain and out-of-domain benchmarks, and the routing, connector structure, point anchors, and pre-alignment strategy are all examined through ablations, providing reasonable empirical support for the framework.

Critical weaknesses

Critical Weaknesses

  1. The core reliability supervision and optimization objective are internally inconsistent with the method description. The paper defines proposal reliability semantically, i.e., a proposal can still be considered reliable when it refers to the intended target despite imperfect box alignment. However, the training labels are constructed purely from geometric overlap: proposals with IoU >0.7>0.7 are labeled reliable and those with IoU <0.3<0.3 unreliable. This conflates semantic correctness with box overlap in the routing variable that determines the refinement pathway. In addition, Eq. (12) defines both proposal-generation and reflection rewards as unweighted sums, whereas the appendix reports actual weights of 0.1/0.3/0.3/0.30.1/0.3/0.3/0.3 and 0.1/1.0/1.00.1/1.0/1.0, respectively. These discrepancies make both the semantic definition of reliability and the mathematical training objective inconsistent with the supervision actually used.

  2. The claimed OOD generalization and relocalization mechanism is not directly established by the current evaluation. The paper attributes improved OOD generalization to avoiding pixel-loss updates to pretrained MLLM parameters, but the compared methods differ simultaneously in architecture, backbone, segmentation model, training data, and optimization strategy, so this causal explanation is not isolated. Moreover, RefAdv and PR-Bench evaluate BBox IoU and Box [email protected] derived from predicted masks rather than pixel-level segmentation quality. The semantic relocalization pathway also remains strongly dependent on the initial proposal through its point anchors: removing these anchors reduces RefCOCOg val performance from 68.0 to 60.2 cIoU. Together, these results support improved localization robustness, but do not yet establish that the proposed mechanism preserves mask-level OOD segmentation or performs relocalization independently of the initial spatial proposal.

Questions for authors

Questions for Authors

  1. Why does the submitted implementation not appear to reproduce the pathway-specific training protocol described in the paper? The paper states that semantic alignment is specialized on approximately 100K examples with initial box IoU >0.7>0.7, while semantic relocalization is specialized on approximately 50K examples with IoU <0.3<0.3. However, the provided alignment pipeline loads the generic stage1_cache_train.json without an explicit IoU-based filtering step, and DirectRejectSftDataset used for relocalization retains both accept and reject records, while the reject-only condition is commented out. Please clarify which exact data-construction and training pipeline was used to produce the reported results, and whether the released implementation corresponds to that pipeline.

  2. How should the paper's semantic definition of proposal reliability be reconciled with its IoU-based supervision? The reflection prompt defines a proposal as reliable when it identifies the intended target and no better semantic match exists, even if the box is not perfectly aligned. In contrast, the training labels are assigned purely from box IoU, with IoU >0.7>0.7 treated as reliable and IoU <0.3<0.3 as unreliable. For example, how is a semantically correct proposal with IoU <0.3<0.3 labeled, and how is a geometrically overlapping but semantically incorrect proposal treated? More generally, what quantity is the reflective reasoning module intended to predict: semantic correctness, geometric box quality, or a combination of the two?

  3. How are proposals in the intermediate IoU range handled, and how does this relate to inference-time routing? Proposals with IoU between 0.30.3 and 0.70.7 are excluded from the reliable/unreliable supervision used for pathway specialization, whereas at inference time the model must make a binary routing decision for arbitrary proposals. What behavior is expected for this intermediate region, and is the learned reliability decision calibrated to such cases?

  4. Why is the GRPO objective inconsistent between the main text and the appendix? Eq. (12) defines the proposal-generation and reflective-reasoning rewards as unweighted sums, whereas Appendix A.4 assigns weights of 0.1/0.3/0.3/0.30.1/0.3/0.3/0.3 to the proposal-generation rewards and 0.1/1.0/1.00.1/1.0/1.0 to the reflective-reasoning rewards. These are mathematically different optimization objectives. Which objective was actually used to train the models reported in the experiments, and why does the main formulation omit these weights?

  5. Which Qwen2.5-VL checkpoint is used to generate the data for pathway specialization? Appendix A.3 states that initial proposals are generated using Qwen2.5-VL, while Stage 1 subsequently updates the Qwen2.5-VL policy using GRPO. It is therefore unclear whether the alignment and relocalization datasets are constructed from proposals and reflection outputs produced by the original pretrained checkpoint or by the GRPO-optimized checkpoint. Please clarify the exact checkpoint used at each stage of data generation and specialization.

  6. How should the claimed connection between frozen pretrained MLLM weights and improved OOD generalization be interpreted from the current evidence? A central motivation of the paper is that avoiding mask-level pixel-loss updates to the pretrained MLLM preserves generalization. However, the compared methods differ simultaneously in architecture, backbone, segmentation module, training data, and optimization strategy, and the appendix explicitly acknowledges that this grouping does not establish a causal relationship. Does the paper intend to claim a causal effect of freezing pretrained MLLM weights, or only an empirical correlation between the proposed training design and the observed OOD performance?

  7. What is the intended scope of the OOD segmentation claim, given that the reported OOD metrics are box-based? RefAdv and PR-Bench are evaluated using BBox IoU and Box [email protected] derived from the predicted masks rather than pixel-level segmentation metrics such as cIoU or mIoU. These results therefore directly measure whether the correct target is localized, but do not establish whether its segmentation mask generalizes accurately. Should the OOD results be interpreted primarily as target-localization generalization rather than mask-level referring-segmentation generalization?

  8. How should the semantic relocalization pathway be interpreted when the initial point prompts themselves lie on an incorrect object or region? The relocalization pathway removes the original proposal embedding from the final SAM input but still retains the two initial point embeddings as spatial anchors. The ablation shows that removing these point anchors reduces performance from 68.0 to 60.2 cIoU on RefCOCOg val, indicating substantial dependence on the initial spatial proposal. If those initial points are themselves incorrect, what information allows the pathway to perform genuine target relocalization rather than local correction around the original proposal?

6 comments

We sincerely thank the reviewer for recognizing the value of our reliability-aware optimization framework and the effectiveness of our experimental evaluation. We would like to clarify that this work is not merely an incremental improvement, but rather proposes a systematic framework involving multiple aspects, including model architecture, data construction, and the training pipeline. Some design details were not sufficiently clarified, which may have led to certain misunderstandings on the reviewers' part. In the following, we will clarify the relevant design choices and address the reviewers' questions point by point.

W1:The core reliability supervision and optimization objective are internally inconsistent with the method description.

1. The core reliability supervision.(Q2)

IoU is not an accurate measure of semantic correctness; rather, it serves as a geometric proxy for constructing reliability supervision when semantic correctness annotations are unavailable. When constructing the reflective reasoning training data, the MLLM only outputs the initial bounding box and points, without providing explicit annotations of proposal semantic correctness. Therefore, based on visual inspection, we empirically select samples with IoU > 0.7 and IoU < 0.3 as reliable and unreliable samples, respectively, while excluding ambiguous samples in the intermediate range.

Additional clarification: During the reflective reasoning stage, the image provided as input to the MLLM contains the initial proposal box visualized on the original image. We do not use images with visualized masks as input because the color of large-area masks may interfere with the MLLM's judgment.

2. Clarification of the Optimization Objective in Eq. (12). (Q4)

The reward functions actually used in our experiments are fully consistent with the weighted configurations described in Appendix A.4. Eq. (12) is intended to summarize the components of the two reward functions: the Proposal Generation Reward consists of format, IoU, box distance, and point distance rewards, while the Reflective Reasoning Reward consists of format, consistency, and reliability decision rewards. To keep the main text concise, the original manuscript omitted the weighting coefficients of individual reward terms and presented their specific weighted configurations collectively in Appendix A.4. This does not imply that all reward terms have a weight of 1. We apologize that our effort to keep the main text concise led to this misunderstanding.

Revision: We have corrected Equation (12) and the erroneous original description of data processing in Appendices A.3 and A.4. All revisions are highlighted in blue.

W2:The claimed OOD generalization and relocalization mechanism is not directly established by the current evaluation.

1. Are the OOD Generalization Improvements Truly Attributable to Avoiding Pixel-loss Updates?(Q6)

We believe that the results in Table 1 have demonstrated that avoiding pixel-loss updates to pretrained MLLM parameters is responsible for the improvement in OOD generalization. We elaborate on this observation below.

① Specifically, methods that update pretrained MLLM parameters with pixel-level losses consistently exhibit significant performance degradation on RefAdv and PR-Bench, despite differences in model architectures, parameter scales, segmentation models, training data, and in-domain performance.

② In contrast, methods that avoid such updates consistently maintain better OOD performance, comparable to or even exceeding the baseline, despite similar configuration differences. This consistent pattern suggests that avoiding pixel-loss updates to pretrained MLLM parameters is crucial for preserving OOD generalization.

These observations motivated us to propose LatentRefine-SegZero. Our reflective routing, dual-branch optimization, and multi-stage training strategies are all designed to avoid pixel-loss updates to pretrained MLLM parameters.

W2(Continuing from the previous response to W2.)

To address the reviewer's concern, we further conduct controlled experiments. We perform three controlled ablations using the same Qwen2.5-VL (7B) + SAM2 architecture and Alignment Connector, all without joint connector pre-alignment. Specifically, Setting 2 freezes the pretrained MLLM and updates only the connector; Setting 3 updates both the MLLM and connector using pixel-level supervision; and Setting 4 first optimizes the MLLM through mixed GRPO (using the weights obtained from Stage 1 training of LatentRefine-SegZero), and then freezes the MLLM during connector training.

MethodRefCOCO testARefCOCO+ testARefCOCOg testRefAdv BBox IoUPR-Bench BBox IoU
1. Baseline (Qwen2.5-VL-7B + SAM2)76.171.464.80.380.50
2. Loss Only for Connector (Alignment Connector, No Joint Pre-Alignment)76.572.065.10.350.44
3. Pixel Loss for MLLM and Connector (Alignment Connector, No Joint Pre-Alignment)81.878.274.70.230.29
4. LatentRefine-SegZero (Alignment Connector, No Joint Pre-Alignment)80.876.773.60.480.56

Comparing Settings 2 and 3, we observe that avoiding pixel-loss updates to the pretrained MLLM improves OOD performance on RefAdv and PR-Bench from 0.23/0.29 to 0.35/0.44, indicating that avoiding pixel-loss updates to pretrained MLLM parameters is responsible for maintaining OOD generalization. Comparing Settings 2 and 4, we observe that GRPO initialization further improves OOD performance.

We first enhance the MLLM's reasoning capabilities through GRPO, and then restrict pixel-level supervision to the connector to avoid disrupting the pretrained MLLM and its acquired capabilities, thereby further improving OOD generalization.

2. Are BBox Metrics Sufficient to Demonstrate Pixel-level OOD Segmentation Generalization?(Q7)

Our use of bounding-box metrics for OOD evaluation is primarily constrained by the annotation availability of the datasets, rather than an intentional avoidance of pixel-level segmentation metrics. Specifically, as shown in Fig. 2, many OOD test sets adopted by existing studies (e.g., ReasonSeg) still exhibit language distributions relatively close to the RefCOCO training set, making it difficult to adequately evaluate model generalization under substantial distribution shifts. But, RefAdv and PR-Bench exhibit more significant distribution shifts, but neither dataset provides pixel-level mask annotations. Therefore, we can only compute BBox IoU and Box [email protected] by comparing the bounding boxes decoded from predicted masks with the ground-truth bounding boxes.

Since the bounding boxes are decoded from the predicted masks, our conclusion regarding OOD segmentation generalization remains well-founded. To address the reviewer's concern regarding the insufficient evaluation of pixel-level segmentation generalization, we further use SAM3 to construct mask annotations for RefAdv and PR-Bench and supplement our evaluation with pixel-level segmentation metrics such as cIoU, thereby further validating the model's segmentation generalization capability under genuine OOD scenarios.

MethodRefAdv cIoUPR-Bench cIoU
LISA0.280.35
LENS0.250.29
SegAgent0.300.33
VisionReasoner0.430.47
Ours0.440.49

3. Does Semantic Relocalization Truly Overcome the Spatial Constraints of the Initial Proposal?(Q8)

The Semantic Relocalization branch is not a local refinement. As shown in Fig. 3, Semantic Alignment feeds the generated Embedding together with the initial box and point prompts into SAM, whereas Semantic Relocalization feeds only the Embedding generated by its Connector, without using the initial box and points as explicit prompts. Nevertheless, we retain the initial point anchors as auxiliary spatial priors to facilitate alignment between MLLM Hidden States and the SAM Prompt Space, since removing them would substantially increase the difficulty of cross-modal prompt alignment.

To further address this concern, we evaluate the initial proposals and both branch outputs on 100 RefCOCO test samples whose initial proposals deviate from the GT regions. We report cIoU and recovery rate, with successful recovery measured by bounding-box [email protected]. The results are shown below:

MethodcIoURecovery Rate ([email protected])
Initial Proposal12.10/100
Semantic Alignment14.30/100
Semantic Relocalization47.944/100

Semantic Alignment yields only a marginal cIoU improvement (12.1 to 14.3) and recovers none of the samples. In contrast, Semantic Relocalization achieves a cIoU of 47.9 and a recovery rate of 44%, demonstrating its ability to overcome the spatial constraints of inaccurate initial proposals despite retaining initial point anchors as auxiliary priors for cross-modal alignment.

Q1 & Q5: Why Does the Submitted Code Appear Unable to Reproduce the Branch-specific Training Pipeline Described in the Paper?

We sincerely thank the reviewer for pointing out these issues. We acknowledge that these are indeed significant oversights in our work.

These issues mainly arose because, during manuscript preparation, the complete experimental code was stored in an internal network environment and could not be exported as a whole. We could only migrate the code by manually copying individual files. During this process, we inadvertently omitted the code related to the construction and processing of the branch-specific datasets. Meanwhile, the description of the dataset construction in the appendix was also inaccurate, resulting in both the initially released code and the manuscript failing to fully reflect the actual experimental pipeline.

In the actual experiments, we used the model obtained after mixed GRPO training to perform inference again on the candidate samples and constructed two branch-specific datasets based on the reflection decisions generated by the model. Samples predicted as accept were saved to aligned_accept_train.json for training the Semantic Alignment branch, while samples predicted as reject were saved to direct_reject_train.json for training the Semantic Relocalization branch.

Regarding the code implementation, the correct data-loading configuration is as follows: the Semantic Alignment branch loads aligned_accept_train.json, while the Semantic Relocalization branch loads direct_reject_train.json. DirectRejectSftDataset supports both accept and reject records because the two branches share the same dataset class. However, since the input data have already been partitioned upstream, the two branches do not mix these two types of samples during training.

To further address the reviewer's concerns about the training pipeline, we have added the complete training algorithm to the revised appendix.

Training of LatentRefine-SegZero algorithm

Require: Training annotations D\mathcal D, base MLLM Gθ0\mathcal G_{\theta_0}, and pretrained SAM.

Ensure: Trained policy and two specialized refinement pathways.

Data preparation

  1. Build proposal reasoning data Dprop\mathcal D_{\mathrm{prop}} from the training splits of RefCOCO, RefCOCO+, and RefCOCOg.

  2. Generate and cache initial proposals using Gθ0\mathcal G_{\theta_0}; associate them with GT masks to form Dcache\mathcal D_{\mathrm{cache}}.

  3. Build reflection data Drefl\mathcal D_{\mathrm{refl}} using IoU labels (>0.7>0.7: reliable; <0.3<0.3: unreliable).

Stage 1: Mixed GRPO training

  1. Train Gθ0\mathcal G_{\theta_0} jointly on Dprop\mathcal D_{\mathrm{prop}} and Drefl\mathcal D_{\mathrm{refl}} with mixed GRPO to obtain Gθ⋆\mathcal G_{\theta^\star}.

Stage 2: Latent prompt connector training

  1. Freeze the MLLM backbone weights θ⋆\theta^\star and SAM.

Part I: Joint connector pre-alignment

  1. Jointly train both connectors and shared queries QsharedQ_{\mathrm{shared}} on all Dcache\mathcal D_{\mathrm{cache}} using GT mask supervision.

Part II: Pathway specialization

  1. For each x=(I,q,g(0))∈Dcachex=(I,q,g^{(0)}) \in \mathcal{D}_{\mathrm{cache}}, compute
\mathcal{G}_{\theta^\star} \left( \operatorname{Render}(I,b^{(0)}), u_{\mathrm{refl}}(q) \right).$$ 8. Partition by predicted reliability: $$\mathcal D_{\mathrm{align}} = \\{(x,r_x) \mid x\in\mathcal D_{\mathrm{cache}}, \hat{\rho}_x=\mathrm{reliable}\\}$$ $$\mathcal D_{\mathrm{reloc}} = \\{(x,r_x) \mid x\in\mathcal D_{\mathrm{cache}}, \hat{\rho}_x=\mathrm{unreliable}\\}$$ *Approximately 230K alignment and 70K relocalization examples.* 9. Initialize $Q_{\mathrm{prop}}$ and $Q_{\mathrm{refl}}$ from the trained shared queries. 10. Freeze the relocalization components; train the alignment connector and $Q_{\mathrm{prop}}$. 11. Freeze the alignment components; train the relocalization connector, $Q_{\mathrm{refl}}$, and newly initialized LoRA on $\mathcal D_{\mathrm{reloc}}$ with reflective context and GT masks. **Return:** Trained policy and specialized refinement pathways.
Reviewer_jG2AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates MLLM+SAM-based referring segmentation and proposes LatentRefine-SegZero, aiming to balance the strong in-domain segmentation performance of SFT-based methods with the out-of-domain generalization capabilities of RL-based methods. Specifically, the framework first employs GRPO to optimize the MLLM for initial proposal generation and reliability assessment. It then uses pixel-level supervision to train QFormer-based connectors that map MLLM hidden representations into SAM's prompt embedding space for segmentation refinement. The proposed method achieves the best performance among the compared methods that do not use pixel-level losses to update pretrained MLLM parameters on RefCOCO/+/g, while outperforming the compared methods that do use such updates on RefAdv and PR-Bench.

Critical strengths
  1. The overall training pipeline is reasonably designed, combining reinforcement learning for proposal generation and reliability assessment with pixel-level supervision for latent prompt refinement.

  2. The idea of distinguishing between different types of segmentation failures and applying corresponding correction strategies is intuitive and well motivated.

Critical weaknesses

The paper does not sufficiently clarify the key differences and advantages of the proposed method over closely related work. Moreover, the experimental results are not yet sufficiently convincing to fully support the claimed contributions, particularly regarding reflective reasoning, the two-stage connector design, and the effectiveness of the overall training framework.

Questions for authors
  1. Fairness of comparisons and incomplete OOD segmentation evaluation. The comparison in Table 1 is not entirely fair, as the competing methods differ in their MLLM backbones, model sizes, SAM versions, and training data sizes. These differences make it difficult to attribute the reported performance gains to the proposed method rather than to variations in model configurations or training settings. The evaluation on the OOD benchmarks (RefAdv and PR-Bench) only reports bounding-box localization metrics, without evaluating pixel-level segmentation performance. This limits the evidence supporting the effectiveness of the proposed latent refinement mechanism on OOD data.

  2. Insufficient differentiation from closely related work. The paper lacks a detailed discussion and comparison with LENS, which similarly adopts latent connectors to map MLLM representations into segmentation prompts. The authors should clarify the fundamental technical differences between these approaches and demonstrate the advantages introduced by the proposed framework.

  3. Limited evidence for the effectiveness of reflective reasoning. The contribution of the proposed reflective reasoning mechanism is not sufficiently validated. In Figure 4, the full method achieves only marginal improvements over the alignment-only variant (approximately 0.5–0.8 cIoU points). An oracle-routing baseline, where samples are assigned to the appropriate refinement pathway using ground-truth reliability information, would help assess the effectiveness of the learned reliability assessment. Table 4 only compares the full method against the relocalization-only variant, without reporting alignment-only results, making the OOD evaluation of the routing mechanism incomplete.

  4. Marginal benefits of the two-stage connector. As shown in Table 2, the proposed two-stage connector achieves only marginal improvements over the one-stage connector. These results do not provide sufficiently strong evidence to justify the additional architectural complexity of the two-stage design.

  5. Insufficient ablation of the key training components. The paper lacks a systematic ablation study isolating the contributions of its main training stages. In particular, it does not report the performance of (i) the GRPO-based training stage alone or (ii) the pixel-level supervised refinement stage alone, compared with the complete framework.

3 comments

We sincerely thank the reviewer for the constructive feedback and for recognizing the motivation of combining reinforcement learning with latent prompt refinement and using different correction strategies for different segmentation failures. We address the five questions below, drawing on the existing paper and supplementary experiments.

Q1 & Q5. Fairness of comparisons and incomplete OOD segmentation evaluation.

1. Fairness of comparisons

We believe that the results in Table 1 have demonstrated that avoiding pixel-loss updates to pretrained MLLM parameters is responsible for the improvement in OOD generalization. We elaborate on this observation below.

① Specifically, methods that update pretrained MLLM parameters with pixel-level losses consistently exhibit significant performance degradation on RefAdv and PR-Bench, despite differences in model architectures, parameter scales, segmentation models, training data, and in-domain performance.

② In contrast, methods that avoid such updates consistently maintain better OOD performance, comparable to or even exceeding the baseline, despite similar configuration differences. This consistent pattern suggests that avoiding pixel-loss updates to pretrained MLLM parameters is crucial for preserving OOD generalization.

These observations motivated us to propose LatentRefine-SegZero. Our reflective routing, dual-branch optimization, and multi-stage training strategies are all designed to avoid pixel-loss updates to pretrained MLLM parameters.

To address the reviewer's concern, we further conduct controlled experiments. We perform three controlled ablations using the same Qwen2.5-VL (7B) + SAM2 architecture and Alignment Connector, all without joint connector pre-alignment. Specifically, Setting 2 freezes the pretrained MLLM and updates only the connector; Setting 3 updates both the MLLM and connector using pixel-level supervision; and Setting 4 first optimizes the MLLM through mixed GRPO (using the weights obtained from Stage 1 training of LatentRefine-SegZero), and then freezes the MLLM during connector training.

MethodRefCOCO testARefCOCO+ testARefCOCOg testRefAdv BBox IoUPR-Bench BBox IoU
1Baseline (Qwen2.5-VL-7B + SAM2)76.171.464.80.380.50
2Loss Only for Connector (Alignment Connector, No Joint Pre-Alignment) 76.572.065.10.350.44
3Pixel Loss for MLLM and Connector (Alignment Connector, No Joint Pre-Alignment)81.878.274.70.230.29
4LatentRefine-SegZero (Alignment Connector, No Joint Pre-Alignment) 80.876.773.60.480.56

Comparing Settings 2 and 3, we observe that avoiding pixel-loss updates to the pretrained MLLM improves OOD performance on RefAdv and PR-Bench from 0.23/0.29 to 0.35/0.44, indicating that avoiding pixel-loss updates to pretrained MLLM parameters is responsible for maintaining OOD generalization. Comparing Settings 2 and 4, we observe that GRPO initialization further improves OOD performance.

2. incomplete OOD segmentation evaluation

Our use of bounding-box metrics for OOD evaluation is primarily constrained by the annotation availability of the datasets, rather than an intentional avoidance of pixel-level segmentation metrics. Specifically, as shown in Fig. 2, many OOD test sets adopted by existing studies (e.g., ReasonSeg) still exhibit language distributions relatively close to the RefCOCO training set, making it difficult to adequately evaluate model generalization under substantial distribution shifts. In contrast, RefAdv and PR-Bench exhibit more significant distribution shifts, but neither dataset provides pixel-level mask annotations. Therefore, we can only compute BBox IoU and Box [email protected] by comparing the bounding boxes decoded from predicted masks with the ground-truth bounding boxes.

Since the bounding boxes are decoded from the predicted masks, our conclusion regarding OOD segmentation generalization remains well-founded. To address the reviewer's concern regarding the insufficient evaluation of pixel-level segmentation generalization, we further use SAM3 to construct mask annotations for RefAdv and PR-Bench and supplement our evaluation with pixel-level segmentation metrics such as cIoU, thereby further validating the model's segmentation generalization capability under genuine OOD scenarios.

MethodRefAdv cIoUPR-Bench cIoU
LISA0.280.35
LENS0.250.29
SegAgent0.300.33
VisionReasoner0.430.47
Ours0.440.49

Q2. Differences from LENS

1. We have already presented comparisons with LENS in multiple places in the original manuscript, including textual analysis and quantitative experimental results.

The details are as follows:

①、Related Work – Supervised Learning from Representations to Masks (Page 3, Lines 150–151)

The original text states: “EVF-SAM explores early fusion of visual and language features, SAM4MLLM learns point prompts generated by an MLLM, and LENS maps MLLM representations into segmentation prompts.”

②、Table 1 – Comparison with State-of-the-Art Methods (Page 8, Line 393)

We include LENS in the Pixel-loss updates to pretrained MLLM category for quantitative comparison and report both in-domain and out-of-domain results. LENS achieves better performance on in-domain benchmarks, whereas our method performs better on the two OOD benchmarks, RefAdv and PR-Bench. The substantial degradation in LENS's OOD performance further supports our observation that directly updating the pretrained MLLM backbone with narrow pixel-level supervision may compromise its general reasoning capabilities and lead to overfitting to the training distribution. In contrast, LatentRefine-SegZero achieves better OOD generalization while maintaining strong in-domain performance.

③、Appendix B.2 – Grouping of Comparison Methods (Page 21, Lines 1117–1120)

The original text states: “LENS is included in the same group despite also using reinforcement learning: its joint training objective combines GRPO with a supervised segmentation loss, and its joint-training stage unfreezes the MLLM. Thus, the presence of reinforcement learning alone does not determine its placement.”

2. Based on the above discussion, we further summarize the core technical differences between LatentRefine-SegZero and LENS in the following two aspects:

①、The two methods differ in their training optimization mechanisms.

LENS adopts joint training with GRPO and supervised segmentation loss, and unfreezes the MLLM during the joint-training stage, allowing pixel-level losses to update the pretrained MLLM parameters. In contrast, although our method also employs GRPO and pixel-level supervision, the two optimization processes are decoupled: GRPO updates the MLLM through policy gradients to optimize proposal generation, error reflection, and reliability assessment, while pixel-level losses only update lightweight connectors and other related components, thereby avoiding direct updates to the pretrained MLLM backbone parameters through pixel-level supervision.

②、The two methods differ in their core technical designs.

LENS mainly focuses on mapping internal MLLM representations into segmentation prompts usable by SAM, whereas our method further introduces reflection-based proposal reliability assessment and dynamically selects between two optimization pathways based on the assessment results: for reliably localized proposals, Semantic Alignment supplements semantic information; for unreliably localized proposals, Semantic Relocalization constructs new latent prompts to relocate the target.

Q3. Limited Evidence for the Effectiveness of Reflective Reasoning

To further verify the reflective reasoning and dynamic routing mechanisms, we additionally report the Oracle Routing baseline and the results of the alignment pathway on OOD datasets. Oracle Routing uses the GT Mask to calculate Mask IoU(RefAdv and PR-Bench use BBox IOU): samples with IoU below 0.3 are routed to the Semantic Relocalization branch, while samples with IoU above 0.3 are routed to the Semantic Alignment branch. The results are shown below:

BenchmarkMetricAlignment OnlyRelocalization OnlyLatentRefine-SegZeroOracle Routing
RefCOCOg valcIoU74.075.376.181.2
RefAdvBBox IoU0.480.330.470.50
PR-BenchBBox IoU0.550.380.560.59

1. The Gap Between Oracle Routing and LatentRefine-SegZero

There is still a considerable gap between LatentRefine-SegZero and Oracle Routing on in-domain data. One reason is that, to avoid mask visualizations interfering with the original images, bounding box visualizations are used as the actual inputs to the multimodal large language model (MLLM). Compared with masks, bounding boxes are less effective in assessing semantic reliability. Secondly, the MLLM itself suffers from hallucinations, making it difficult for the model to accurately identify its own errors through reflective reasoning. Nevertheless, the results of Oracle Routing also demonstrate the great potential of this approach.

2. OOD Generalization Limitations and the Role of Dynamic Routing.

On OOD datasets, neither LatentRefine-SegZero nor Oracle Routing achieves significant improvements. This is mainly because the Semantic Alignment and Semantic Relocalization branches are trained only on in-domain data and do not possess the generalization capabilities of the MLLM itself. However, thanks to the dynamic routing mechanism, LatentRefine-SegZero does not exhibit the significant performance degradation observed with Relocalization Only.

Q4. Marginal Benefits of the Two-Stage Connector

We agree that accuracy improvement over the one-stage connector is limited. On RefCOCOg val in Table 2, alignment remains at 72.6 cIoU, while relocalization improves from 67.5 to 68.0. The motivation for the two-stage design also concerns parameter efficiency.

The one-stage connector performs its transformations in the MLLM hidden space, with dimension H = 3584. The two-stage connector first aggregates semantic information in a lower-dimensional latent space, D = 1024, and then constructs SAM prompts with dimension P = 256. For the connector configurations analyzed in our rebuttal, the parameter counts are:

Nsingle=29H2+HP+35H+P=373,549,824N_{\mathrm{single}} = 29H^2 + HP + 35H + P = 373{,}549{,}824

Ntwo=26D2+2HD+44D+26P2+2DP+32P=36,884,480N_{\mathrm{two}} = 26D^2 + 2HD + 44D + 26P^2 + 2DP + 32P = 36{,}884{,}480

ConnectorParameters per connectorAlignment cIoURelocalization cIoU
One-stage373.55M72.667.5
Two-stage36.88M72.668.0

This is a reduction of approximately 90.13% in connector parameters while maintaining alignment accuracy and slightly improving relocalization accuracy. We did not previously report the number of connector parameters separately, mainly because this structure was designed with reference to the dimensional mapping design of the two-layer MLP in the classical Transformer, and we initially regarded it as an empirical architectural choice. We apologize that the original manuscript did not provide a sufficient explanation, which may have caused the reviewer's concerns. We will supplement the appendix with a relevant parameter count analysis.

Reviewer_JvoCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Recent state-of-the-art RES methods combine MLLMs with SAM. However, SFT-based methods tend to generalize less well to out-of-domain data, while RL-based methods struggle with fine-grained mask refinement. This work improves the RL-based approach by training the MLLM to assess its own bounding-box and point proposals. Based on this assessment, it activates either the alignment or relocalization module, which converts proposal information and MLLM representations into latent prompts for SAM. Compared with explicit prompts, these latent prompts support finer mask refinement while keeping SAM frozen. The results show substantially better out-of-domain accuracy than the compared SFT-based methods while maintaining competitive in-domain performance.

Critical strengths

The proposed reliability-aware SAM prompt refinement strategy is well designed. Refining the initial explicit proposals through latent prompts is also a clever choice, allowing the model to convey additional information to improve mask quality while keeping SAM itself frozen. The routing strategy could also inspire future work on more specialized refinement pathways for different types of errors.

Critical weaknesses

The main contribution is the reliability-aware routing strategy for proposal refinement. However, the analysis provides limited insight into why it works better. Fig. 4 suggests that the relocalization pathway alone already achieves close-to-best in-domain results, while the additional improvement from the alignment pathway is relatively marginal. Table 4 provides relevant evidence that routing all inputs through relocalization leads to much worse OOD results, but it is unclear whether this is because the pathway was specialized on unreliable proposals rather than trained to handle both reliable and unreliable ones. An experiment training a single pathway on all samples would provide more insight into the necessity of the dual-path design. Qualitative analysis of the two pathways’ failure modes would also be helpful.

Questions for authors
  1. Reflective reasoning evaluates the bounding-box proposal without inspecting the proposed points. Do you observe failures when different objects have strongly overlapping bounding boxes-for example, two football players who substantially overlap in the image-or when the box covers the correct target but the points fall on another object? Qualitative examples would help clarify whether the reliability judgment adequately captures the information needed by the refinement pathways. f such cases are rare in the benchmarks, a controlled stress test could use synthetic examples constructed by compositing two objects using their ground-truth masks to create varying degrees of overlap.

  2. Why does the proposal-generation GRPO reward omit the IoU of the mask produced by SAM from the predicted box and points? This would directly measure how useful the geometric proposals are for segmentation, beyond their agreement with ground-truth coordinates. Have you evaluated this reward, or is its omission motivated by computational cost or training considerations?

2 comments

We sincerely thank the reviewer for the positive evaluation and constructive suggestions. We appreciate the recognition of our reliability-aware SAM prompt refinement strategy, the use of latent prompts to improve mask quality while keeping SAM frozen, and the potential of specialized refinement pathways. We address the concerns about the dual-path design, proposal reliability, and GRPO reward below.

W1: Necessity of the Dual-Path Design and Analysis of Its Operating Mechanism

1. Necessity of the Dual-Path Design

We thank the reviewer for the suggestion. To address the reviewer's concern about whether the OOD performance degradation of the Relocalization Pathway is caused by its specialized training on unreliable samples, we added two single-path training experiments using all samples to further verify the necessity of the dual-path design. Specifically, both experiments use the MLLM after mixed GRPO training as the initialization and separately specialize Semantic Alignment and Semantic Relocalization using all training samples, without reliability-based partitioning. Both are evaluated on the same in-domain and out-of-domain benchmarks.

MethodRefCOCO testARefCOCO+ testARefCOCOg testRefAdv BBox IoUPR-Bench BBox IoU
Baseline (Qwen2.5-VL-7B + SAM2)76.171.464.80.380.50
Semantic Alignment (All samples used for specialized training, initialized from the mixed-GRPO-trained model)80.876.773.60.480.56
Semantic Relocalization (All samples used for specialized training, initialized from the mixed-GRPO-trained model)82.479.375.60.250.27
LatentRefine-SegZero82.879.876.90.470.56

The experimental results show that even when all training samples are used to specialize the Relocalization Pathway, its in-domain performance is relatively strong, but its OOD performance still degrades significantly, indicating that this issue is not caused solely by the reliability-based partitioning of training samples. In contrast, the Alignment Pathway maintains good OOD generalization, but its in-domain performance is relatively limited. Through reliability-aware dual-path routing, LatentRefine-SegZero achieves the best performance on all three in-domain benchmarks while maintaining OOD generalization comparable to Alignment Only, further validating the complementarity and necessity of the dual-path design.

2. Analysis of Its Operating Mechanism

To further analyze the operating mechanisms, we selected 100 samples from the RefCOCO test set whose initial proposals deviate from the GT regions, and evaluated the initial proposals and the outputs of both branches. We report cIoU and recovery rate, where successful recovery is measured by bounding-box [email protected]. The results are shown below:

MethodcIoURecovery Rate ([email protected])
Initial Proposal12.10/100
Semantic Alignment14.30/100
Semantic Relocalization47.944/100

Semantic Alignment yields only a marginal cIoU improvement (from 12.1 to 14.3) and recovers none of the samples. In contrast, Semantic Relocalization achieves a cIoU of 47.9 and a recovery rate of 44%.

By retaining the original point and box information as inputs to SAM, Semantic Alignment achieves more stable OOD performance. However, this also limits its ability to learn when the original points and boxes have deviated from the target, meaning that the alignment branch can essentially only make local corrections. Retaining the original point and box information for refinement is therefore a double-edged sword.

Q1. Reliability Assessment When Boxes Overlap or Points Are Incorrect

We thank the reviewer for pointing out this important edge case. The current reflection module assesses the reliability of target localization based on an image with the proposal box visualized and the referring expression, without explicitly evaluating the candidate points. Therefore, we agree that box-level reliability does not guarantee the reliability of the complete geometric prompts. In particular, when objects overlap heavily, or when the candidate box is correct but the point lies on a distracting object, the model may accept the proposal box, while the subsequent alignment branch may still be affected by incorrect point prompts. Due to time constraints, we will include an analysis of this edge case in the final version of the paper.

Q2. Why the Proposal-Generation GRPO Reward Does Not Include SAM Mask IoU

We thank the reviewer for the suggestion. We agree that Mask IoU can more directly measure the effectiveness of proposals for the segmentation task. The current use of coordinate-level rewards is mainly motivated by GRPO training efficiency, rather than the belief that Mask IoU is unsuitable as a reward. Specifically, each input requires sampling 8 rollouts. Introducing an online Mask IoU reward would require additional SAM mask prediction and IoU computation for each proposal. Even with shared image features, this would still significantly increase GPU computational overhead, memory usage, and training latency. Therefore, we use box IoU, coordinate error, and point location rewards as proxy metrics to reduce training costs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.