acceptodds
Under review as a conference paper at ICLR 2027

ContourVLA: A Closed-Loop Perception-Action Contour Policy for Generalized Referring Expression Segmentation

Abstract

Generalized referring expression segmentation (GRES) requires dynamically balancing high-level semantics for identifying a variable number of language-specified referents with fine-grained visual evidence for precise boundary delineation. This requirement challenges existing cascaded vision-language architectures, which typically rely on static feature interfaces and single-pass mask prediction, limiting adaptive perception and geometric correction. We introduce ContourVLA, a vision-language-action policy that recasts GRES as a closed-loop visuomotor process, in which editable contours serve as explicit policy states that condition multimodal perception and are updated by geometric action chunks. Evolution-Aware Semantic Scheduling (EASS) couples contour-guided bidirectional boundary sampling with state-conditioned routing of multilevel multimodal features, adapting perception to each contour state. Following supervised initialization, Dustbin-Augmented Entropic Credit Transport GRPO (DECT-GRPO) jointly optimizes discrete grounding and continuous contour actions with instance-level credits. Its rollout rewards and credits are derived from soft prediction-target correspondences that account for false positives and missed targets. ContourVLA improves gIoU over the strongest evaluated baselines by 8.7, 2.8, and 2.7 points on gRefCOCO val, testA, and testB, respectively, and achieves the highest mIoU across all eight RefCOCO, RefCOCO+, and RefCOCOg splits.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Perception-Action Contour Policy, Generalized Referring Expression Segmentation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4Qi7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ContourVLA reformulates generalized referring expression segmentation (GRES) as a closed-loop perception–action process, where editable contours act as explicit geometric states and are iteratively refined through predicted actions. The framework introduces Evolution-Aware Semantic Scheduling (EASS) to adaptively sample boundary evidence and route multilevel multimodal features according to the current contour state, together with a Geometric Action Decoder for contour updates. It further proposes DECT-GRPO, which uses dustbin-augmented entropic transport to provide instance-level credit assignment for jointly optimizing grounding and contour actions. Experiments on gRefCOCO, RefCOCO, RefCOCO+, and RefCOCOg show strong improvements over prior methods.

Critical strengths
  1. Novel closed-loop formulation for GRES. The paper reformulates GRES from conventional single-pass mask prediction into a closed-loop perception–action process, where editable contours serve as explicit geometric states to guide iterative multimodal perception and boundary refinement. This provides a conceptually meaningful alternative to static vision-language segmentation pipelines.

  2. Well-designed adaptive multimodal routing and instance-level policy optimization. The proposed EASS dynamically routes multilevel multimodal representations according to the evolving contour state, allowing the model to select semantic information that is more appropriate for different refinement stages. In addition, DECT-GRPO introduces dustbin-augmented entropic transport to establish soft prediction–target correspondences and provide instance-level credit assignment for both grounding and contour actions. These two designs make the perception and optimization processes more adaptive to instance-specific states and prediction quality.

Critical weaknesses
  1. Training and inference efficiency. ContourVLA relies on a relatively heavy training pipeline, including large-scale grounding pre-training, multi-dataset supervised fine-tuning, DECT-GRPO post-training, and iterative contour refinement during inference. However, the paper does not provide a comprehensive efficiency comparison with representative GRES methods in terms of end-to-end inference latency, GPU memory consumption, or total training cost. Therefore, it remains unclear whether the reported performance gains justify the additional computational and training overhead.

  2. Attribution of the performance gains. The main results combine several factors, including large-scale grounding pre-training, joint training on multiple RES/GRES datasets, the proposed contour-based architecture, and RL post-training. Since the methods in the main comparison are evaluated under their respective training configurations, it is difficult to fully disentangle how much of the improvement comes from the proposed closed-loop formulation itself versus the stronger training recipe and additional supervision. Although the controlled HiMTok experiment partially addresses this concern, it still evaluates the complete ContourVLA pipeline rather than isolating these factors under a strictly matched training and computation budget.

Questions for authors
  1. Clarification on computational efficiency. Could the authors provide a more comprehensive efficiency comparison with representative GRES methods, including end-to-end inference latency, peak GPU memory consumption, and total training cost? The current framework involves large-scale grounding pre-training, multi-stage optimization, RL post-training, and iterative contour refinement, so understanding the computational overhead relative to the reported performance gains would materially affect the assessment of its practical value.

  2. Clarification on the contribution of DECT-GRPO in single-target RES. A key motivation of DECT-GRPO is instance-level credit assignment among multiple predicted referents. However, conventional RefCOCO, RefCOCO+, and RefCOCOg primarily involve a single referred instance. Could the authors clarify how much the instance-level credit mechanism contributes in this single-target setting?

4 comments
AuthorsOpenReview ↗

Response (Part I)

We thank the reviewer for recognizing the novelty of ContourVLA's closed-loop contour formulation and appreciating the designs of adaptive multimodal routing and instance-level policy optimization. We are also grateful for your comments on the attribution of the performance gains, computational efficiency, and contribution of DECT-GRPO in single-target RES.

1. Attribution of the Performance Gains [W2]

More rigorous ablation experiments validate the contributions of the proposed architecture, the grounding pretraining, joint SFT training on multiple RES/GRES datasets, and RL post-training. (a) Architecture: As shown in Rows 1 and 3 in Table R9, replacing the native HiMTok semantic segmentation architecture with the closed-loop contour architecture improves most metrics, even if large-scale grounding pretraining (i.e., directly using the HiMTok pretrained checkpoint) and RL post-training are disabled. Rows 2 and 4 further show that the gIoU advantages of contour evolution further increase across all three splits after applying GRPO. (b) Pretraining: Row 4 in Table R9 shows that ContourVLA still achieves good performance even when using the HiMTok pretrained checkpoint without large-scale grounding pretraining, while the comparison with the full ContourVLA (last row of Table 1 of the manuscript) demonstrates that the latter achieves higher performance. (c) RL post-training: The comparisons between Rows 3 and 4 of Table R9 and Rows 1 and 5 in Table 4 of the main paper further validate the contribution of RL post-training. (d) Joint SFT training: This stage supervises GAD and EASS to learn contour evolution. Directly applying GRPO without this stage would make GAD and EASS unable to learn contour evolution from reward feedback alone, resulting in poor segmentation performance.

Table R9: Contributions of contour-evolution architecture and RL optimization without Stage-I grounding pretraining.

Variantval gIoUval cIoUtestA gIoUtestA cIoUtestB gIoUtestB cIoU
native HiMTok architecture + SFT (Table 8, B)73.6±0.270.9±0.374.7±0.275.4±0.472.5±0.372.3±0.3
native HiMTok architecture + SFT + GRPO (newly added)74.0±0.471.2±0.374.9±0.275.8±0.372.7±0.272.4±0.4
ContourVLA architecture + SFT (newly added)76.5±0.271.1±0.275.2±0.376.5±0.272.7±0.372.0±0.1
ContourVLA architecture + SFT + GRPO (Table 8, C)80.4±0.172.0±0.478.8±0.477.0±0.273.9±0.372.2±0.2

2. Computational Efficiency [W1, Q1]

Efficiency comparison experiments with representative GRES methods demonstrate that ContourVLA achieves high segmentation accuracy with lower inference overhead, outperforming the three representative GRES baselines in both end-to-end inference latency and peak GPU memory consumption. The detailed results are presented in Table R10.

           Table R10: Segmentation Performance and End-to-End Inference Overhead on gRefCOCO.

Methodval gIoUval cIoUtestA gIoUtestA cIoUtestB gIoUtestB cIoUEnd-to-end Latency (ms/expression) ↓Peak GPU Memory (GB) ↓
RAS-13B74.670.577.577.069.467.8522.631.5
HiMTok-8B72.170.473.474.971.771.91149.418.3
Text4Seg-13B74.869.875.174.368.067.14905.331.1
ContourVLA-3B83.572.380.378.174.471.8445.18.4

(a) Inference Efficiency. Under the same GPU, numerical precision, and test sample conditions, ContourVLA achieves 14.8% lower end-to-end latency than the fastest baseline (RAS-13B) and 54.1% lower peak GPU memory consumption than the baseline with the lowest memory usage (HiMTok-8B). Meanwhile, it achieves the highest gIoU across all three gRefCOCO splits.

(b) Training Cost and Performance Gains. ContourVLA-3B completes three stage training on 16 NVIDIA H20 GPUs, consuming a total of 1,580 GPU hours. Grounding pretraining accounts for 76% of the total cost (1200 GPU hours). SFT consumes 160 GPU hours. DECT-GRPO post-training consumes 220 GPU hours, improving gIoU and cIoU on gRefCOCO val by respectively 5.9 and 4.9 percentage points over SFT. Note that the contour evolution architecture can directly start downstream SFT from an existing HiMTok-8B checkpoint without requiring the 1,200 GPU hour grounding pretraining. This configuration achieves higher gIoU on gRefCOCO val than the native HiMTok segmentation architecture (see the "Attribution of the Performance Gains" section). For reference, retraining HiMTok-8B on the same type of H20 GPUs consumes 2,290 GPU hours, requiring 44.9% more than our ContourVLA-3B.

Please refer to the following response for the continuation of our detailed discussion.

AuthorsOpenReview ↗

Response (Part II)

3. Contribution of DECT-GRPO to Single-Target RES [Q2]

DECT-GRPO contributes to single-target RES, although each expression in this task corresponds to only one ground-truth target. In this task, the VLIM may predict multiple instances, and DECT-GRPO can still assign different instance-level credits based on the matching quality between each prediction and the target. To quantify the benefits of this mechanism, we follow the ablation setting in Table 4 of the revised manuscript and compare full DECT-GRPO with classical Shared-advantage GRPO across five test splits of three RES datasets. The results are presented in Table R11.

                Table R11: Impact of Instance-Level Credit Modulation on Single-Target RES Performance.

MethodRefCOCO testA mIoURefCOCO testA cIoURefCOCO testB mIoURefCOCO testB cIoURefCOCO+ testA mIoURefCOCO+ testA cIoURefCOCO+ testB mIoURefCOCO+ testB cIoURefCOCOg test mIoURefCOCOg test cIoU
Without instance-level credit modulation (Shared-advantage GRPO)82.381.880.078.779.579.272.668.375.376.3
With instance-level credit modulation (Full DECT-GRPO)83.782.981.179.280.780.774.269.376.777.3

Compared with the ablated version, DECT-GRPO improves both mIoU and cIoU across all five test splits. These results demonstrate that distinguishing the quality of different predictions for the same expression can also improve single-target RES performance.

AuthorsOpenReview ↗

Gentle reminder before the discussion deadline

Dear Reviewer 4Qi7,

Thank you again for your time and thoughtful efforts in reviewing our paper. In response to your valuable comments, we have analyzed the attribution of the performance gains, demonstrated the computational efficiency of ContourVLA, and evaluated the contribution of DECT-GRPO in single-target RES.

As the discussion period is coming to a close, we would like to kindly ask whether you have any remaining questions or concerns. If so, please feel free to let us know.

Many thanks,

Authors

AuthorsOpenReview ↗

Gentle reminder before the discussion deadline

Dear Reviewer 4Qi7,

Thank you again for your time and thoughtful feedback. As the discussion deadline approaches, we would appreciate your feedback on our rebuttal and welcome any remaining questions. Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_tZ9SOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ContourVLA, which constructs a policy to recast GRES as a closed-loop visuomotor process. It contains EASS which replaces static features with adaptive boundary sampling based observation and GAD which uses adaptive observation to predict action chunk to update the contours initialization by VLIM. They also propose DECT-GRPO to jointly optimize grounding tokens and contour actions to replace standard GRPO. Experiments show that it achieves good performance on many GRES and RES benchmarks.

Critical strengths
  1. The proposed re-observation and EASS modules show good performance. Table 2 shows that full EASS improves both gIoU and cIoU over frozen observations. Table 6 separates dynamic contour descriptors from static instance attributes and further shows the benefits of adaptive perception.

  2. Table 3 shows the success of DECT-GRPO over standard RL baselines. DECT-GRPO improves over shared-advantage GRPO by 3.0 gIoU and 5.1 instance-F1 points. Tables 7 and 8 add objective ablations and transfer to another pretrained backbone to verify this.

Critical weaknesses
  1. In Table 1, ContourVLA gets the highest mIoU across all eight RES splits, but the authors ignore DeRIS-L even though they evaluated DeRIS-B, which is released in the same paper, and it reports higher mIoU than ContourVLA on all eight splits. In addition, while Table 8 controls for backbone initialization and downstream data, the native HiMTok baseline only gets continued SFT, whereas ContourVLA also receives RL, making the compute budgets and input representations unmatched.

  2. The model is jointly trained on four COCO-based datasets (plus 139M Stage-I instances). Appendix A.1 notes that only official training splits are used, but it does not check whether training images from one dataset appear in the val/test splits of another dataset.

Questions for authors
  1. Can the authors report results for the released DeRIS-L under the evaluation protocol? Also, in Table 8, how does the native HiMTok baseline (Variant B) perform if post-trained with GRPO?

  2. gRefCOCO, RefCOCO, RefCOCO+, and RefCOCOg all use COCO images. When you mix their training sets together, do any val/test images from one dataset appear in the training set of another dataset?

  3. Can the authors report performance broken down by expression type and initial grounding quality? And Eq. 14 preserves the advantage of rollout, so how do false-positive instances receive positive advantages and impact training?

4 comments
AuthorsOpenReview ↗

Response (Part I)

We thank the reviewer for recognizing the meaningful controlled support of the state-conditioned re-observation and EASS modules, and the success of DECT-GRPO. We also thank the reviewer for the valuable suggestions on comparison baselines, data protocols, and instance credit mechanism validation.

1. Comparison Baselines [W1, Q1]

The comparison with DeRIS-L validates the advantages of ContourVLA, while further controlled experiments verify the contribution of the contour evolution architecture.

(a) Comparison with DeRIS-L. The official DeRIS-L RES task is jointly trained on RefCOCO, RefCOCO+, and RefCOCOg, while its GRES task is trained separately on gRefCOCO. Since the released DeRIS-L checkpoints are trained separately for RES and GRES, whereas ContourVLA uses a single checkpoint for both tasks, we jointly retrain DeRIS-L on the same four-dataset mixture and evaluate a single checkpoint across all splits using the same evaluator to ensure a fair comparison. The results are shown in Table R5.

              Table R5: Comparison with DeRIS-L demonstrating the competitiveness of ContourVLA.

ModelgRefCOCO val gIoUgRefCOCO val cIoUgRefCOCO testA gIoUgRefCOCO testA cIoUgRefCOCO testB gIoUgRefCOCO testB cIoURefCOCO val mIoURefCOCO val cIoURefCOCO testA mIoURefCOCO testA cIoURefCOCO testB mIoURefCOCO testB cIoU
DeRIS-L76.170.274.673.967.565.782.679.383.480.280.578.3
ContourVLA-3B83.572.380.378.174.471.882.481.383.782.981.179.2
ModelRefCOCO+ val mIoURefCOCO+ val cIoURefCOCO+ testA mIoURefCOCO+ testA cIoURefCOCO+ testB mIoURefCOCO+ testB cIoURefCOCOg val mIoURefCOCOg val cIoURefCOCOg test mIoURefCOCOg test cIoU
DeRIS-L76.674.579.678.872.969.776.674.077.177.5
ContourVLA-3B77.576.480.780.774.269.376.676.976.777.3

On GRES, ContourVLA-3B outperforms the jointly retrained DeRIS-L in both gIoU and cIoU across all three gRefCOCO splits, with gIoU gains of 5.7–7.4 percentage points. On conventional RES, ContourVLA-3B also leads on most metrics, with only slight decreases on a few metrics. These results demonstrate that ContourVLA achieves more pronounced performance advantages on GRES while maintaining competitive performance on RES.

(b) Fair RL Baselines. We add the results of native HiMTok mask baseline (Variant B) post-trained with GRPO (Row 2 of Table R6). We also conduct experiments of ContourVLA without LocateAnything Stage-I pretraining, with or without additional GRPO post-training (Rows 3 and 4 of Table R6). All variants start from the same HiMTok-8B checkpoint and use the same SFT schedule (30,000 steps), while the variants with RL optimization receive an additional 8,000 GRPO steps.

Table R6: Contributions of contour-evolution architecture and RL optimization without Stage-I grounding pretraining.

Variantval gIoUval cIoUtestA gIoUtestA cIoUtestB gIoUtestB cIoU
native HiMTok architecture + SFT (Table 8, B)73.6±0.270.9±0.374.7±0.275.4±0.472.5±0.372.3±0.3
native HiMTok architecture + SFT + GRPO (newly added)74.0±0.471.2±0.374.9±0.275.8±0.372.7±0.272.4±0.4
ContourVLA architecture + SFT (newly added)76.5±0.271.1±0.275.2±0.376.5±0.272.7±0.372.0±0.1
ContourVLA architecture + SFT + GRPO (Table 8, C)80.4±0.172.0±0.478.8±0.477.0±0.273.9±0.372.2±0.2

Under the same backbone and SFT budget, replacing HiMTok's native segmentation architecture with contour evolution achieves higher gIoU values across all three splits (Rows 1 and 3). After applying GRPO to both architectures, the gIoU advantages of contour evolution further increase across all three splits (Rows 2 and 4). The consistent results under both training settings validate the effectiveness of the contour-evolution architecture on GRES.

2. Cross-dataset Image Overlap [W2, Q2]

The images used for training do not overlap with any validation or test split of the four datasets. Following LocCa (NeurIPS 2024, Appendix C), we collect the COCO image IDs from all validation and test splits across the four datasets and remove every training sample with a matching image ID from the combined training set.

Please refer to the following response for the continuation of our detailed discussion.

AuthorsOpenReview ↗

Response (Part II)

3. Instance Credit Mechanism Validation [Q3]

The results grouped by single/multiple/no-target expressions and initial grounding quality show that ContourVLA achieves good segmentation quality for single- and multiple-target expressions, as well as high rejection accuracy for no-target expressions. Furthermore, the results against the hard-assignment instance-credit comparator show the superiority of the soft instance credit assignment (Eqs. 11~13 in the manuscript). Training statistics further show that the proportion of false positive instances receiving positive advantages is relatively low and gradually decreases as training progresses. This demonstrates that ContourVLA achieves fine-grained instance-level optimization while preserving the overall policy optimization direction.

(a) Evaluation by Single/Multiple/No-target Expressions. Table R7 reports the results grouped by expression type on the gRefCOCO val and testA/testB splits (there are no single-target expressions in val). We use mIoU+ and N-acc for with-target and no-target expressions. The results show that both metrics are high, demonstrating the advantages of ContourVLA in contour segmentation and no-target rejection.

    Table R7: ContourVLA's ability in handling different expression types on gRefCOCO.

Expression typeMetricval Countval Result (%)testA CounttestA Result (%)testB CounttestB Result (%)
Single-targetmIoU+0—5,91780.25,64673.2
Multiple-targetmIoU+5,324848,83580.55,74475.7
No-targetN-acc8,90583.24,44880.14,67374.1

(b) Impact of Initial Grounding Quality. Table R8 shows the final segmentation performance of ContourVLA grouped by initial grounding quality on gRefCOCO val. We define the initial grounding quality qq as the sum of the maximum IoUs obtained through one-to-one matching between initial predicted boxes and ground-truth boxes, divided by the larger of the two box counts. As the initial grounding quality increases, the final mIoU+ gradually improves. However, the difference between the medium- and high-quality groups is only 0.7 percentage points, indicating that contour evolution can compensate for the localization errors of initial detection boxes.

Table R8: Impact of initial grounding quality on final segmentation performance .

Initial grounding quality (qq)Number of expressionsmIoU+ (%)
Low (q<0.50q < 0.50)48574.8
Medium (0.50≤q<0.750.50 \leq q < 0.75)56584.3
High (q≥0.75q \geq 0.75)4,27485

(c) Instance-credit Comparator. Table 4 of the manuscript compares the experimental configuration using hard assignment instance credit (Row 3) with ContourVLA's soft instance credit assignment (Row 5) on gRefCOCO val. The results show that the Instance [email protected] and gIoU improve by respectively 2.0 and 1.3 percentage points, demonstrating that soft-matching instance credit assignment helps improve instance prediction quality and overall segmentation performance.

(d) Frequency and Impact of False Positives Receiving Positive Advantages. We sample rollouts from three checkpoints corresponding to the early, middle, and late stages of training, using the same one-to-one matching criterion as Instance [email protected]. Among all false-positive instances at each checkpoint, the proportions receiving positive instance advantages are respectively 17.6%, 8.5%, and 4.3%. Furthermore, within the same rollout with Ag≠0A_{g} \neq 0, when the instance credit of a false positive is lower than that of a true positive (cFP<cTPc_{\mathrm{FP}} < c_{\mathrm{TP}}, which typically occurs because DECT-GRPO uses dustbin-augmented soft matching, tending to assign higher instance credit to true positives and lower credit to false positives), Eq. 14 results in a lower advantage for the false positive than for the true positive. Therefore, when Ag>0A_{g} > 0, the false positive receives a smaller positive advantage; when Ag<0A_{g} < 0, the false positive receives a larger negative advantage and incurs a stronger penalty, which is beneficial for distinguishing true positives from false positives. This demonstrates that although DECT-GRPO preserves the sign of the rollout-level advantage, instance-level credit assignment differentiates the optimization strength according to instance quality. Furthermore, the proportion of false-positive instances receiving positive advantages decreases as training progresses, indicating that such erroneous positive updates become less frequent.

Thanks for the author's response. Most of my concerns are solved.

AuthorsOpenReview ↗

Thank you for acknowledging that your concerns have been addressed

We sincerely thank you for your positive recommendation and are glad that most of your concerns have been addressed. We have clarified the contributions of different components of ContourVLA by adding controlled comparisons. The instance-level credit assignment mechanism has been validated to distinguish prediction quality while preserving the rollout-level optimization direction.

Thank you again for your positive comments and valuable suggestions.

Reviewer_KHyDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a closed-loop perception-action framework, ContourVLA, for generalized referring expression segmentation. It consists of Evolution-Aware Semantic Scheduling for contour-conditioned feature sampling and semantic routing, a Geometric Action Decoder for iterative contour refinement, and DECT-GRPO for joint optimization of grounding and contour actions through instance-level credit modulation. The method achieves competitive results on GRES and RES benchmarks, with a few ablation studies.

Critical strengths
  1. It is interesting to formulate GRES as a perception-action process with editable contours.
  2. The proposed ContourVLA achieves competitive or state-of-the-art performance across multiple GRES and RES benchmarks.
  3. The main experiments demonstrate the effectiveness of EASS, re-observation, and DECT-GRPO.
Critical weaknesses
  1. Given the differences in large-scale pretraining data and optimization budgets, it is difficult to isolate the contributions of the proposed architecture from those of additional training and optimization.
  2. The substantial gIoU improvements may be partly driven by better no-target rejection rather than contour accuracy, but the current evaluations do not sufficiently disentangle these contributions.
  3. The main ablations are limited to support central claims. Some important results are only presented in the appendix. Several loss terms and hyperparameters also lack sufficient ablation or sensitivity analysis.
Questions for authors
  1. ContourVLA uses 139M LocateAnything training instances for Stage-I grounding pretraining, which may contribute substantially to its advantage over existing methods. Appendix A.10.3 shows that ContourVLA remains effective without Stage-I pretraining. However, the comparison still involves additional RL optimization and unmatched training budgets, making it difficult to isolate the contribution of the proposed architecture. Could the authors provide more controlled comparisons to disentangle the effects of architecture, pretraining, and optimization?
  2. On gRefCOCO val, ContourVLA exceeds its strongest baselines by 8.7 gIoU but only 1.8 cIoU. Could the authors report no-target accuracy for the main baselines and separately evaluate target-present and no-target expressions? Why are the improvements substantially larger (+8.7 gIoU) on val than on testA/testB (+2.8/+2.7)? Unlike most baselines, ContourVLA achieves higher gIoU on val than on testA. Could the authors explain this discrepancy and clarify whether the improvements observed in the validation ablations also generalize to the test splits?
  3. Key results for state-conditioned routing and joint grounding and action optimization is only presented Appendix. Could the authors move these results into the main paper? How were the seven SFT loss weights and other highly specific hyperparameters selected? Could the authors provide loss-component ablations (particularly SDF, smoothness, and spacing regularization) and sensitivity analyses for key hyperparameters?
3 comments
AuthorsOpenReview ↗

Response (Part I)

We thank the reviewer for finding ContourVLA's perception–action process interesting, recognizing its state-of-the-art performance across multiple GRES and RES benchmarks, and acknowledging the effectiveness of EASS, re-observation, and DECT-GRPO. We also thank the reviewer for the valuable suggestions on controlled comparisons, performance gain attribution, and key hyperparameter analysis. We have revised the manuscript accordingly.

1. Controlled Comparisons [W1, Q1]

More rigorous controlled experiments isolate the contributions of the proposed architecture. The contributions of architecture, pretraining, and optimization are further disentangled.

(a) Architecture and Pretraining. To examine the contribution of the proposed architecture, we compare HiMTok's native segmentation architecture with ContourVLA's contour evolution architecture using the same HiMTok-8B backbone. All variants use identical downstream training data and sampling distributions, with 30,000 SFT steps (global batch size of 128). The two RL variants undergo an additional 8,000 steps, each using 32 image–expression pairs with 8 rollouts per pair. To control for pretraining differences, all variants are initialized from the same HiMTok-8B checkpoint, skipping Stage-I pretraining on 139M LocateAnything instances. The results are shown in Table R1.

Table R1: Contributions of contour-evolution architecture and RL optimization without Stage-I grounding pretraining.

Variantval gIoUval cIoUtestA gIoUtestA cIoUtestB gIoUtestB cIoU
native HiMTok architecture + SFT (Table 8, B)73.6±0.270.9±0.374.7±0.275.4±0.472.5±0.372.3±0.3
native HiMTok architecture + SFT + GRPO (newly added)74.0±0.471.2±0.374.9±0.275.8±0.372.7±0.272.4±0.4
ContourVLA architecture + SFT (newly added)76.5±0.271.1±0.275.2±0.376.5±0.272.7±0.372.0±0.1
ContourVLA architecture + SFT + GRPO (Table 8, C)80.4±0.172.0±0.478.8±0.477.0±0.273.9±0.372.2±0.2

These results have been incorporated into Table 8 (as variants D and E) of the manuscript. Under the same backbone and SFT budget, comparisons between Rows 1/3 and Rows 2/4 show that replacing HiMTok's native segmentation architecture with contour evolution improves most metrics, both with and without GRPO post-training. Comparing Rows 3 and 4 further demonstrates the contribution of RL optimization. Row 4 shows that even when the HiMTok pretrained checkpoint is used (Stage-I grounding pretraining is removed), good performance is still achieved; the comparison of this row with the full ContourVLA (last row of Table 1 of the manuscript) shows that the full ContourVLA configuration achieves higher gIoU across all three gRefCOCO splits.

(b) Optimization. Rows 1 and 5 of Table 4 of the manuscript further demonstrate the performance gains from RL optimization. We have clarified that Row 1 (SFT only) means the ablation without GRPO. The comparison of Rows 3 and 4 of Table R1 further demonstrates that the gains from RL optimization also hold under a different experimental setting.

2. Performance Gain Attribution [W2, Q2]

More detailed experimental analyses show that the substantial gIoU improvements arise from both no-target rejection and target-present segmentation quality. These findings also explain the differences in performance gains between val and testA/testB and demonstrate the generalization ability of ContourVLA.

(a) No-target Rejection and Contour Accuracy. In Table R2, we use no-target accuracy (N-acc) to measure the ability to correctly return empty predictions for no-target expressions, and mean mask IoU on target-present expressions (mIoU+) to evaluate segmentation quality when targets are present. ContourVLA outperforms all baselines on both metrics across the three splits, indicating improvements in both aspects, with more pronounced gains in no-target rejection.

   Table R2: ContourVLA's high no-target rejection (N-acc) and target-present segmentation quality (mIoU+) across the three splits.

Methodsval gIoUval cIoUval N-accval mIoU+testA gIoUtestA cIoUtestA N-acctestA mIoU+testB gIoUtestB cIoUtestB N-acctestB mIoU+
RAS-13B74.670.569.982.477.577.068.880.169.467.861.172.8
HiMTok-8B72.170.471.972.573.474.974.873.071.771.969.872.5
DeRIS-B73.968.074.972.272.571.676.671.365.263.164.665.4
ContourVLA83.572.383.284.080.378.180.180.474.471.874.174.5

Please refer to the following response for the continuation of our detailed discussion.

AuthorsOpenReview ↗

Response (Part II)

(b) Cross-split Differences in gIoU. The differences in performance gains between val and testA/testB are mainly related to the proportion of no-target expressions and ContourVLA's advantage in no-target rejection. During RL post-training, DECT-GRPO assigns higher rewards to correct empty predictions than to false positives for the same no-target expression, encouraging empty outputs when no matching target exists. This reward difference is reflected in the evaluation metrics: under gIoU, each correct rejection receives a full score of one for that expression; whereas under cIoU, correct empty predictions contribute neither intersection nor union pixels, while incorrect nonempty predictions increase the union. This distinction is particularly relevant to val, where no-target expressions account for 62.58%, substantially higher than testA (23.17%) and testB (29.09%). Consequently, the advantage in no-target rejection contributes more strongly to the gIoU gains on val. For ContourVLA itself, both N-acc and mIoU+ are higher on val than on testA, which also explains its higher val gIoU.

(c) Generalization to the Test Splits. We extend the key ablation experiments on the two central mechanisms of ContourVLA, namely dynamic re-observation in EASS and instance-level advantage modulation in DECT-GRPO, to testA and testB. As shown by comparing Rows 1/2 and 3/4 of Table R3, both mechanisms improve performance on these splits, confirming that the improvements observed in the validation ablations generalize to the test splits.

  Table R3: Key ablations validating the generalization to test splits.

VarianttestA gIoU ↑testA cIoU ↑testB gIoU ↑testB cIoU ↑
Frozen observations72.470.366.865.2
Full EASS77.475.170.969.5
Shared-advantage GRPO79.276.673.170.7
Full DECT-GRPO80.378.174.471.8

3. Key Hyperparameter Analysis [W3, Q3]

Sensitivity analyses of the seven SFT loss weights validate the robustness of ContourVLA to variations in these hyperparameters. In addition, we conduct parameter-wise sensitivity analyses of the maximum action magnitude δmax⁡\delta_{\max} in GAD, the transport entropy coefficient ε\varepsilon in DECT-GRPO, and the null-match score b∅b_{\emptyset}. Using the default value x0x_{0} of each parameter as a reference, we evaluate {0.5,1.0,1.5,2.0}x0\{0.5, 1.0, 1.5, 2.0\}x_{0}, varying only one parameter at a time while keeping all other training configurations unchanged. As shown in Table R4, within 0.5–2.0 times the default values, the maximum gIoU decrease is 2.2 percentage points for the seven SFT loss weights and 3.4 percentage points for the three additional hyperparameters. These results indicate that the method exhibits good robustness to key hyperparameter variations within the evaluated ranges. In practice, all seven SFT loss weights and the three additional hyperparameters were selected through grid search on the validation set.

Table R4: Robustness to hyperparameter variations (measured by gIoU, %).

StageParameterDefault x0x_{0}0.5x00.5x_{0}1.0x01.0x_{0} (default)1.5x01.5x_{0}2.0x02.0x_{0}
SFTλground\lambda_{\mathrm{ground}}1.076.677.677.175.8
SFTλtraj\lambda_{\mathrm{traj}}2.076.777.676.375.4
SFTλsmooth\lambda_{\mathrm{smooth}}0.0477.477.677.076.7
SFTλspacing\lambda_{\mathrm{spacing}}0.1077.477.676.976.1
SFTλregion\lambda_{\mathrm{region}}0.7077.077.677.076.5
SFTλSDF\lambda_{\mathrm{SDF}}0.17876.277.676.575.8
SFTλrouter\lambda_{\mathrm{router}}1.076.877.676.776.0
SFTδmax⁡\delta_{\max}0.0577.277.677.476.5
DECT-GRPOε\varepsilon0.0581.683.582.180.1
DECT-GRPOb∅b_{\emptyset}0.2581.883.583.282.5

Furthermore, we individually set λSDF\lambda_{\mathrm{SDF}}, λsmooth\lambda_{\mathrm{smooth}}, and λspacing\lambda_{\mathrm{spacing}} to zero to examine their effects. Removing these three terms reduces gIoU/cIoU to 75.9/66.2, 77.2/66.9, and 77.0/66.7, respectively. These results indicate that signed-distance supervision (SDF) plays an important role in contour geometry learning, while smoothness and spacing regularization provide further performance gains.

Finally, we have moved the key results on state-conditioned routing and joint grounding–action optimization (originally Tables 6 and 7 in the appendix) to Section 4.3 of the main paper.

AuthorsOpenReview ↗

Gentle reminder before the discussion deadline

Dear Reviewer KHyD,

Thank you again for your time and thoughtful efforts in reviewing our paper. In response to your valuable comments, we have conducted controlled comparisons to clarify the contributions of the designed contour evolution architecture and RL optimization. Furthermore, we have validated that the substantial gIoU improvements arise from both no-target rejection and target-present segmentation quality. Lastly, we have verified ContourVLA's robustness to hyperparameter variations.

As the discussion period is coming to a close, we would like to kindly ask whether you have any remaining questions or concerns. If so, please feel free to let us know.

Many thanks,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summarized Response to All Reviewers

We thank the Reviewers, Area Chairs, and Program Chairs for their time and thoughtful feedback on ContourVLA. We have posted detailed responses to the comments from Reviewer KHyD, Reviewer tZ9S, and Reviewer 4Qi7 in their respective discussion threads.

We appreciate the reviewers’ recognition of several aspects of our work.

  • Novel closed-loop perception-action architecture. Reviewer KHyD found it “interesting to formulate GRES as a perception-action process with editable contours,” and Reviewer 4Qi7 described this formulation as “a novel closed-loop formulation for GRES” and “a conceptually meaningful alternative to static vision-language segmentation pipelines.”
  • Adaptive multimodal routing and instance-level policy optimization. Reviewer 4Qi7 highlighted the “well-designed adaptive multimodal routing and instance-level policy optimization.” Reviewer tZ9S found that the state-conditioned re-observation and EASS modules have “meaningful controlled support” and described the post-training experiments as “substantive evidence for DECT-GRPO.”
  • Competitive performance across GRES and RES benchmarks. Reviewer KHyD noted that ContourVLA achieves “competitive or state-of-the-art performance across multiple GRES and RES benchmarks.”

In response to the reviewers’ concerns, the experiments have been substantially enriched by adding nine sets of additional experiments and analyses. A revised manuscript using blue\color{blue}{\mathit{blue}} color to mark revisions is uploaded.

Our major revisions include the following.

  • Detailed ablation studies have been conducted to clarify the contributions of the contour evolution architecture and GRPO and analyze the impact of Stage-I grounding pretraining.
  • Performance gain attribution has been studied to verify the advantages of ContourVLA in no-target rejection and single-/multiple-target segmentation.
  • Core mechanism validation has been conducted to verify the roles of dynamic re-observation, instance-level advantage modulation, and the instance credit mechanism.
  • Robustness and generalization analysis has been carried out to verify robustness to variations in key hyperparameters and under different initial grounding quality levels, and validate the generalization of key mechanisms.

The following table maps the revisions described in the rebuttal to their corresponding changes in the revised manuscript and the reviewer comments they address.

PageLocationRevisionReviewer
7–8§4.2, Table 1Added a comparison with DeRIS-L jointly retrained on all four datasets.tZ9S W1/Q1
8–10§4.3, Tables 3, 5Moved the state-conditioned routing and joint grounding–action tables to §4.3.KHyD W3/Q3
30–31A.10.3, Table 8Added two matched-budget controls for native HiMTok and contour evolution.All three¹
31A.10.4, Table 9Compared no-target rejection and target-present segmentation across all splits.KHyD W2/Q2
31–32A.10.4, Tables 10–11Reported performance by expression type and initial grounding quality.tZ9S Q3
32-33A.10.5, Table 12Tested re-observation and instance credit on gRefCOCO testA and testB.KHyD W2/Q2
33A.10.5, Table 13Evaluated instance-level credit modulation on five single-target RES test splits.4Qi7 Q2
7, 32§3.4, A.10.4Clarified Eq. 14 and tracked how often false positives had positive advantages.tZ9S Q3
33-34A.10.6, Table 14 + textTested hyperparameter sensitivity and ablated three SFT loss terms individually.KHyD W3/Q3
34A.10.7, Table 15 + textReported end-to-end latency, peak GPU memory, and training cost by stage.4Qi7 W1/Q1

¹ The matched SFT and GRPO controls in Table 8 address KHyD W1/Q1, tZ9S W1/Q1, and 4Qi7 W2.

We thank the reviewers again for the questions and suggestions that helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.