- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents LatentRefine-SegZero, a reliability-aware framework for referring segmentation. It uses reflective reasoning to assess an initial MLLM-generated proposal and routes it to either semantic alignment or semantic relocalization, with lightweight connectors mapping MLLM features into SAM-compatible prompts.
Critical strengths
-
The paper proposes a clear reliability-aware refinement strategy for referring segmentation. By separating reliable proposals from unreliable ones and routing them to semantic alignment or semantic relocalization, the method provides a meaningful way to handle different failure modes rather than applying the same refinement strategy to all cases.
-
The experimental evaluation is relatively comprehensive and consistently supports the proposed design. The method is tested on both in-domain and out-of-domain benchmarks, and the routing, connector structure, point anchors, and pre-alignment strategy are all examined through ablations, providing reasonable empirical support for the framework.
Critical weaknesses
Critical Weaknesses
-
The core reliability supervision and optimization objective are internally inconsistent with the method description. The paper defines proposal reliability semantically, i.e., a proposal can still be considered reliable when it refers to the intended target despite imperfect box alignment. However, the training labels are constructed purely from geometric overlap: proposals with IoU are labeled reliable and those with IoU unreliable. This conflates semantic correctness with box overlap in the routing variable that determines the refinement pathway. In addition, Eq. (12) defines both proposal-generation and reflection rewards as unweighted sums, whereas the appendix reports actual weights of and , respectively. These discrepancies make both the semantic definition of reliability and the mathematical training objective inconsistent with the supervision actually used.
-
The claimed OOD generalization and relocalization mechanism is not directly established by the current evaluation. The paper attributes improved OOD generalization to avoiding pixel-loss updates to pretrained MLLM parameters, but the compared methods differ simultaneously in architecture, backbone, segmentation model, training data, and optimization strategy, so this causal explanation is not isolated. Moreover, RefAdv and PR-Bench evaluate BBox IoU and Box [email protected] derived from predicted masks rather than pixel-level segmentation quality. The semantic relocalization pathway also remains strongly dependent on the initial proposal through its point anchors: removing these anchors reduces RefCOCOg val performance from 68.0 to 60.2 cIoU. Together, these results support improved localization robustness, but do not yet establish that the proposed mechanism preserves mask-level OOD segmentation or performs relocalization independently of the initial spatial proposal.
Questions for authors
Questions for Authors
-
Why does the submitted implementation not appear to reproduce the pathway-specific training protocol described in the paper? The paper states that semantic alignment is specialized on approximately 100K examples with initial box IoU , while semantic relocalization is specialized on approximately 50K examples with IoU . However, the provided alignment pipeline loads the generic
stage1_cache_train.jsonwithout an explicit IoU-based filtering step, andDirectRejectSftDatasetused for relocalization retains bothacceptandrejectrecords, while the reject-only condition is commented out. Please clarify which exact data-construction and training pipeline was used to produce the reported results, and whether the released implementation corresponds to that pipeline. -
How should the paper's semantic definition of proposal reliability be reconciled with its IoU-based supervision? The reflection prompt defines a proposal as reliable when it identifies the intended target and no better semantic match exists, even if the box is not perfectly aligned. In contrast, the training labels are assigned purely from box IoU, with IoU treated as reliable and IoU as unreliable. For example, how is a semantically correct proposal with IoU labeled, and how is a geometrically overlapping but semantically incorrect proposal treated? More generally, what quantity is the reflective reasoning module intended to predict: semantic correctness, geometric box quality, or a combination of the two?
-
How are proposals in the intermediate IoU range handled, and how does this relate to inference-time routing? Proposals with IoU between and are excluded from the reliable/unreliable supervision used for pathway specialization, whereas at inference time the model must make a binary routing decision for arbitrary proposals. What behavior is expected for this intermediate region, and is the learned reliability decision calibrated to such cases?
-
Why is the GRPO objective inconsistent between the main text and the appendix? Eq. (12) defines the proposal-generation and reflective-reasoning rewards as unweighted sums, whereas Appendix A.4 assigns weights of to the proposal-generation rewards and to the reflective-reasoning rewards. These are mathematically different optimization objectives. Which objective was actually used to train the models reported in the experiments, and why does the main formulation omit these weights?
-
Which Qwen2.5-VL checkpoint is used to generate the data for pathway specialization? Appendix A.3 states that initial proposals are generated using Qwen2.5-VL, while Stage 1 subsequently updates the Qwen2.5-VL policy using GRPO. It is therefore unclear whether the alignment and relocalization datasets are constructed from proposals and reflection outputs produced by the original pretrained checkpoint or by the GRPO-optimized checkpoint. Please clarify the exact checkpoint used at each stage of data generation and specialization.
-
How should the claimed connection between frozen pretrained MLLM weights and improved OOD generalization be interpreted from the current evidence? A central motivation of the paper is that avoiding mask-level pixel-loss updates to the pretrained MLLM preserves generalization. However, the compared methods differ simultaneously in architecture, backbone, segmentation module, training data, and optimization strategy, and the appendix explicitly acknowledges that this grouping does not establish a causal relationship. Does the paper intend to claim a causal effect of freezing pretrained MLLM weights, or only an empirical correlation between the proposed training design and the observed OOD performance?
-
What is the intended scope of the OOD segmentation claim, given that the reported OOD metrics are box-based? RefAdv and PR-Bench are evaluated using BBox IoU and Box [email protected] derived from the predicted masks rather than pixel-level segmentation metrics such as cIoU or mIoU. These results therefore directly measure whether the correct target is localized, but do not establish whether its segmentation mask generalizes accurately. Should the OOD results be interpreted primarily as target-localization generalization rather than mask-level referring-segmentation generalization?
-
How should the semantic relocalization pathway be interpreted when the initial point prompts themselves lie on an incorrect object or region? The relocalization pathway removes the original proposal embedding from the final SAM input but still retains the two initial point embeddings as spatial anchors. The ablation shows that removing these point anchors reduces performance from 68.0 to 60.2 cIoU on RefCOCOg val, indicating substantial dependence on the initial spatial proposal. If those initial points are themselves incorrect, what information allows the pathway to perform genuine target relocalization rather than local correction around the original proposal?
6 comments
W2(Continuing from the previous response to W2.)
To address the reviewer's concern, we further conduct controlled experiments. We perform three controlled ablations using the same Qwen2.5-VL (7B) + SAM2 architecture and Alignment Connector, all without joint connector pre-alignment. Specifically, Setting 2 freezes the pretrained MLLM and updates only the connector; Setting 3 updates both the MLLM and connector using pixel-level supervision; and Setting 4 first optimizes the MLLM through mixed GRPO (using the weights obtained from Stage 1 training of LatentRefine-SegZero), and then freezes the MLLM during connector training.
| Method | RefCOCO testA | RefCOCO+ testA | RefCOCOg test | RefAdv BBox IoU | PR-Bench BBox IoU |
|---|---|---|---|---|---|
| 1. Baseline (Qwen2.5-VL-7B + SAM2) | 76.1 | 71.4 | 64.8 | 0.38 | 0.50 |
| 2. Loss Only for Connector (Alignment Connector, No Joint Pre-Alignment) | 76.5 | 72.0 | 65.1 | 0.35 | 0.44 |
| 3. Pixel Loss for MLLM and Connector (Alignment Connector, No Joint Pre-Alignment) | 81.8 | 78.2 | 74.7 | 0.23 | 0.29 |
| 4. LatentRefine-SegZero (Alignment Connector, No Joint Pre-Alignment) | 80.8 | 76.7 | 73.6 | 0.48 | 0.56 |
Comparing Settings 2 and 3, we observe that avoiding pixel-loss updates to the pretrained MLLM improves OOD performance on RefAdv and PR-Bench from 0.23/0.29 to 0.35/0.44, indicating that avoiding pixel-loss updates to pretrained MLLM parameters is responsible for maintaining OOD generalization. Comparing Settings 2 and 4, we observe that GRPO initialization further improves OOD performance.
We first enhance the MLLM's reasoning capabilities through GRPO, and then restrict pixel-level supervision to the connector to avoid disrupting the pretrained MLLM and its acquired capabilities, thereby further improving OOD generalization.
2. Are BBox Metrics Sufficient to Demonstrate Pixel-level OOD Segmentation Generalization?(Q7)
Our use of bounding-box metrics for OOD evaluation is primarily constrained by the annotation availability of the datasets, rather than an intentional avoidance of pixel-level segmentation metrics. Specifically, as shown in Fig. 2, many OOD test sets adopted by existing studies (e.g., ReasonSeg) still exhibit language distributions relatively close to the RefCOCO training set, making it difficult to adequately evaluate model generalization under substantial distribution shifts. But, RefAdv and PR-Bench exhibit more significant distribution shifts, but neither dataset provides pixel-level mask annotations. Therefore, we can only compute BBox IoU and Box [email protected] by comparing the bounding boxes decoded from predicted masks with the ground-truth bounding boxes.
Since the bounding boxes are decoded from the predicted masks, our conclusion regarding OOD segmentation generalization remains well-founded. To address the reviewer's concern regarding the insufficient evaluation of pixel-level segmentation generalization, we further use SAM3 to construct mask annotations for RefAdv and PR-Bench and supplement our evaluation with pixel-level segmentation metrics such as cIoU, thereby further validating the model's segmentation generalization capability under genuine OOD scenarios.
| Method | RefAdv cIoU | PR-Bench cIoU |
|---|---|---|
| LISA | 0.28 | 0.35 |
| LENS | 0.25 | 0.29 |
| SegAgent | 0.30 | 0.33 |
| VisionReasoner | 0.43 | 0.47 |
| Ours | 0.44 | 0.49 |
3. Does Semantic Relocalization Truly Overcome the Spatial Constraints of the Initial Proposal?(Q8)
The Semantic Relocalization branch is not a local refinement. As shown in Fig. 3, Semantic Alignment feeds the generated Embedding together with the initial box and point prompts into SAM, whereas Semantic Relocalization feeds only the Embedding generated by its Connector, without using the initial box and points as explicit prompts. Nevertheless, we retain the initial point anchors as auxiliary spatial priors to facilitate alignment between MLLM Hidden States and the SAM Prompt Space, since removing them would substantially increase the difficulty of cross-modal prompt alignment.
To further address this concern, we evaluate the initial proposals and both branch outputs on 100 RefCOCO test samples whose initial proposals deviate from the GT regions. We report cIoU and recovery rate, with successful recovery measured by bounding-box [email protected]. The results are shown below:
| Method | cIoU | Recovery Rate ([email protected]) |
|---|---|---|
| Initial Proposal | 12.1 | 0/100 |
| Semantic Alignment | 14.3 | 0/100 |
| Semantic Relocalization | 47.9 | 44/100 |
Semantic Alignment yields only a marginal cIoU improvement (12.1 to 14.3) and recovers none of the samples. In contrast, Semantic Relocalization achieves a cIoU of 47.9 and a recovery rate of 44%, demonstrating its ability to overcome the spatial constraints of inaccurate initial proposals despite retaining initial point anchors as auxiliary priors for cross-modal alignment.
Q1 & Q5: Why Does the Submitted Code Appear Unable to Reproduce the Branch-specific Training Pipeline Described in the Paper?
We sincerely thank the reviewer for pointing out these issues. We acknowledge that these are indeed significant oversights in our work.
These issues mainly arose because, during manuscript preparation, the complete experimental code was stored in an internal network environment and could not be exported as a whole. We could only migrate the code by manually copying individual files. During this process, we inadvertently omitted the code related to the construction and processing of the branch-specific datasets. Meanwhile, the description of the dataset construction in the appendix was also inaccurate, resulting in both the initially released code and the manuscript failing to fully reflect the actual experimental pipeline.
In the actual experiments, we used the model obtained after mixed GRPO training to perform inference again on the candidate samples and constructed two branch-specific datasets based on the reflection decisions generated by the model. Samples predicted as accept were saved to aligned_accept_train.json for training the Semantic Alignment branch, while samples predicted as reject were saved to direct_reject_train.json for training the Semantic Relocalization branch.
Regarding the code implementation, the correct data-loading configuration is as follows: the Semantic Alignment branch loads aligned_accept_train.json, while the Semantic Relocalization branch loads direct_reject_train.json. DirectRejectSftDataset supports both accept and reject records because the two branches share the same dataset class. However, since the input data have already been partitioned upstream, the two branches do not mix these two types of samples during training.
To further address the reviewer's concerns about the training pipeline, we have added the complete training algorithm to the revised appendix.
Training of LatentRefine-SegZero algorithm
Require: Training annotations , base MLLM , and pretrained SAM.
Ensure: Trained policy and two specialized refinement pathways.
Data preparation
-
Build proposal reasoning data from the training splits of RefCOCO, RefCOCO+, and RefCOCOg.
-
Generate and cache initial proposals using ; associate them with GT masks to form .
-
Build reflection data using IoU labels (: reliable; : unreliable).
Stage 1: Mixed GRPO training
- Train jointly on and with mixed GRPO to obtain .
Stage 2: Latent prompt connector training
- Freeze the MLLM backbone weights and SAM.
Part I: Joint connector pre-alignment
- Jointly train both connectors and shared queries on all using GT mask supervision.
Part II: Pathway specialization
- For each , compute
We sincerely thank the reviewer for recognizing the value of our reliability-aware optimization framework and the effectiveness of our experimental evaluation. We would like to clarify that this work is not merely an incremental improvement, but rather proposes a systematic framework involving multiple aspects, including model architecture, data construction, and the training pipeline. Some design details were not sufficiently clarified, which may have led to certain misunderstandings on the reviewers' part. In the following, we will clarify the relevant design choices and address the reviewers' questions point by point.
W1:The core reliability supervision and optimization objective are internally inconsistent with the method description.
1. The core reliability supervision.(Q2)
IoU is not an accurate measure of semantic correctness; rather, it serves as a geometric proxy for constructing reliability supervision when semantic correctness annotations are unavailable. When constructing the reflective reasoning training data, the MLLM only outputs the initial bounding box and points, without providing explicit annotations of proposal semantic correctness. Therefore, based on visual inspection, we empirically select samples with IoU > 0.7 and IoU < 0.3 as reliable and unreliable samples, respectively, while excluding ambiguous samples in the intermediate range.
Additional clarification: During the reflective reasoning stage, the image provided as input to the MLLM contains the initial proposal box visualized on the original image. We do not use images with visualized masks as input because the color of large-area masks may interfere with the MLLM's judgment.
2. Clarification of the Optimization Objective in Eq. (12). (Q4)
The reward functions actually used in our experiments are fully consistent with the weighted configurations described in Appendix A.4. Eq. (12) is intended to summarize the components of the two reward functions: the Proposal Generation Reward consists of format, IoU, box distance, and point distance rewards, while the Reflective Reasoning Reward consists of format, consistency, and reliability decision rewards. To keep the main text concise, the original manuscript omitted the weighting coefficients of individual reward terms and presented their specific weighted configurations collectively in Appendix A.4. This does not imply that all reward terms have a weight of 1. We apologize that our effort to keep the main text concise led to this misunderstanding.
Revision: We have corrected Equation (12) and the erroneous original description of data processing in Appendices A.3 and A.4. All revisions are highlighted in blue.
W2:The claimed OOD generalization and relocalization mechanism is not directly established by the current evaluation.
1. Are the OOD Generalization Improvements Truly Attributable to Avoiding Pixel-loss Updates?(Q6)
We believe that the results in Table 1 have demonstrated that avoiding pixel-loss updates to pretrained MLLM parameters is responsible for the improvement in OOD generalization. We elaborate on this observation below.
① Specifically, methods that update pretrained MLLM parameters with pixel-level losses consistently exhibit significant performance degradation on RefAdv and PR-Bench, despite differences in model architectures, parameter scales, segmentation models, training data, and in-domain performance.
② In contrast, methods that avoid such updates consistently maintain better OOD performance, comparable to or even exceeding the baseline, despite similar configuration differences. This consistent pattern suggests that avoiding pixel-loss updates to pretrained MLLM parameters is crucial for preserving OOD generalization.
These observations motivated us to propose LatentRefine-SegZero. Our reflective routing, dual-branch optimization, and multi-stage training strategies are all designed to avoid pixel-loss updates to pretrained MLLM parameters.