acceptodds
Under review as a conference paper at ICLR 2027

Prompt Optimization Enables Improving and Self-Distilling Image Segmentation Models

Abstract

Text-Promptable segmentation models (PSMs) like SAM 3 and Falcon Perception can generate masks from natural language. However, we know from other model types like LLMs that prompt optimization can greatly improve outputs, and that these improved outputs can be used to self-train the model for recursive self-improvement. To test this for segmentation models, we study whether PSMs can be improved without ground-truth labels by optimizing the prompts using multimodal LLMs. First, we test and find that training-free iterative prompt refinement, where an MLLM inspects the image and current mask to propose improved prompts, raises segmentation quality by approximately 20 IoU points over category-name prompts. Then, we train MLLMs via GRPO to compress multi-step refinement into a single prompt rewrite, which achieves similar quality at a lower inference cost. Finally, we close the loop by distilling optimized masks back into the segmentation model, so that it produces improved masks directly from the original prompt. Through this work, we enable label-free self-improvement for PSMs for the first time and provide a foundation for autonomous adaptation to new domains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
promptable image segmentation, multimodal large language models, reinforcement learning, self-distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_P5PUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript improves MLLMs by optimizing text prompts to produce substantially better masks and image segmentation results. Training-free is an advantage. Generally, the manuscript is easy to read.

Critical strengths

It is claimed that a promising accuracy improvement can be attained.

Critical weaknesses

First, baseline methods such as SAM3Agent and EVOLSAM3 based on MLLMdriven prompt rewriting already exist, which undermines the novelty of the proposed approach. Second, as a submission to ICLR, the paper lacks sufficient contributions. The socalled selfdistillation scheme described in the paper is not selfdistillation in a rigorous sense. Third, aside from the nonoriginal framework, the paper is deficient in experimental comparisons across different methods. Furthermore, implementation details (including critical ones) of the method are omitted, violating the selfcontainment principle of academic papers and rendering the experiments hard to reproduce. In addition, the existing typesets e.g.“the PSM S understands”must be corrected.

Questions for authors
  1. Please provide necessary details of the proposed method. For inatance, how to carry out iteratively optimize the text prompt for segmentation models and what are the step-by-step procedures are very important. Withou these details, readers cannot experimentally follow the paper at all.
  2. Sufficient experimental comparison and analyes must be offered.
  3. Analyses on the shortcomings of the proposed method and possible modifications are also expected.
2 comments

We thank the reviewer for noting that the manuscript is "easy to read". The weaknesses are not numbered in the review, so we number them in the order in which they appear and address them together with the three questions. New results are marked in blue in the revised paper.

[W1]: Novelty with respect to SAM 3 Agent and EVOL-SAM3. We agree that both methods already use an MLLM to rewrite prompts for SAM 3, and we build on this idea. The difference lies in what happens after the rewriting. SAM 3 Agent [1] and EVOL-SAM3 [2] search for a good prompt at inference time with several MLLM calls per image, and they do not train a model. In contrast, we use the resulting masks for training. First, we train the MLLM with RL, so that a single rewrite is sufficient. Second, we fine-tune the segmenter on the improved masks, so that it returns better masks for the category name alone and no MLLM is needed at inference. We compare with both methods in [W3/Q2].

[W2]: Contributions and the term "self-distillation". We use the term because teacher and student are the same network with the same initial weights. The teacher is the segmenter with the rewritten prompt, the student is the segmenter with the category name, and no second segmentation model provides the training masks. We acknowledge that an MLLM chooses the prompt of the teacher, so the term can be understood more narrowly. If the reviewer prefers, we will call this stage "self-training with prompt-refined pseudo-labels" in the revision.

[W3/Q2]: Comparison with other methods. We agree that comparisons across methods strengthen the paper. To address that point, we run SAM 3 Agent and EVOL-SAM3 from their released code and add an OPRO-style optimization of the instruction [3]. All methods use the same MLLM, the same segmenters and the same 600 test tasks (100 per dataset). We repeat the comparison with a model that is seven times larger, Qwen 3.8 27B, which we also train with our RL setup. Both external methods use greedy decoding and the category name as query, and EVOL-SAM3 runs in the text-only mode of its released code, since Falcon does not accept box prompts. We report mean IoU in percent.

Table 1: Comparison with existing methods at the same model size.

MLLMMethodMLLM calls per imageFalconSAM 3.1
noneCategory name as prompt060.247.1
Qwen 3.5 4BSAM 3 Agent253.050.6
Qwen 3.5 4BEVOL-SAM35 to 671.058.8
Qwen 3.5 4BOPRO172.458.4
Qwen 3.5 4BOne rewrite, MLLM without RL171.451.1
Qwen 3.5 4BOne rewrite, RL-trained MLLM (ours)174.868.0
Qwen 3.8 27BSAM 3 Agent2 to 374.672.9
Qwen 3.8 27BEVOL-SAM3569.464.7
Qwen 3.8 27BOne rewrite, RL-trained MLLM (ours)174.769.0

With a 4B model, our method outperforms every existing method on both segmenters with a single MLLM call per image. It is 21.8 and 17.4 points above SAM 3 Agent, 3.8 and 9.2 points above EVOL-SAM3, and 2.4 and 9.6 points above OPRO, and all six differences are significant. The comparison with the MLLM without RL shows that the gain comes from RL training, which adds 3.4 points on Falcon and 17.0 points on SAM 3.1. SAM 3 Agent scores low with a 4B model because it does not complete a valid run for about a quarter of the tasks. With a 27B model, our method outperforms EVOL-SAM3 on both segmenters and is level with SAM 3 Agent on Falcon, while SAM 3 Agent is 4.0 points above our method on SAM 3.1. Table 1 is Tab. 3 of the revised paper.

Neither existing method changes the segmenter, whereas our second step does. After fine-tuning on the improved masks, Falcon improves from 60.1 to 68.2 IoU when it only receives the category name, and to 72.0 with a longer training budget, without any MLLM at inference.

References:

[1] Carion et al.: SAM 3: Segment Anything with Concepts. In ICLR, 2026

[2] Ye et al.: Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Prompting. arXiv, 2025

[3] Yang et al.: Large Language Models as Optimizers. In ICLR, 2024

[W4/Q1]: Missing details and step-by-step procedure. The instructions, tool definitions and decoding settings are given in App. D of the revision, and we agree that the procedure could have been explained more clearly. The revision contains one algorithm box for each of the three stages (Algs. 1 to 3 in App. F). Briefly, the stages work as follows.

Iterative refinement (Sec. 4). We first prompt the segmenter with the category name. The MLLM then receives the image and the image with the resulting mask drawn on top, and answers with a new prompt, which we pass to the segmenter. This is repeated up to ten times, and the MLLM always sees all previous prompts and masks. Finally, the MLLM or a separate verifier call selects one of the masks.

RL training (Sec. 5). The MLLM receives the image, the category name and the mask of the category name, and outputs one new prompt. The reward is the IoU between the mask of this prompt and the ground-truth mask. We train with GRPO using 8 sampled prompts per task, 32 tasks per update, 100 updates and LoRA with rank 64.

Self-distillation (Sec. 6). For each training image, we generate a mask with the rewritten prompt. We then fine-tune the segmenter to produce this mask when it is given the category name. We train 2.36M parameters for 200 or 1,000 steps with batch size 4 and learning rate 1e-5.

We will release the code and the outputs for all tasks upon acceptance.

[W5]: Typesetting. We have fixed "the PSM S understands" in the revision.

[Q3]: Shortcomings and possible modifications. The revision contains a limitations section (App. A) with the following points. First, iterative refinement does not improve monotonically, and the last prompt is often worse than the first rewrite. A selection step is therefore required, and we now provide one that works without ground truth. Second, fine-tuning the segmenter brings little gain on Fashionpedia, also with ground-truth masks (+0.7 points). Third, the category name returns an empty mask for 22% of the training images, and removing these images accounts for a substantial part of the gain from fine-tuning. Fourth, SAM 3 Agent with a 27B MLLM remains better than our method on SAM 3.1. Finally, RL training needs ground-truth masks for the reward, so only the fine-tuning of the segmenter works without masks of the target dataset.

Reviewer_c83TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether text-promptable segmentation models can be improved through multimodal prompt optimization and self-distillation. First, a frozen multimodal language model iteratively rewrites prompts by inspecting images and segmentation overlays. Second, GRPO trains a prompt optimizer to produce an effective single rewrite, using ground-truth mask IoU as the reward. Third, masks generated from refined prompts supervise segmentation students that retain the original category prompts at inference. Validation covers six datasets, two segmentation backends, multiple language models, and experiments on generalization and prompting styles. The results demonstrate useful prompt-rewriting and self-distillation gains, although the label-free framing and some reported RL comparisons require clarification.

Critical strengths
  1. The self-distillation experiment demonstrates a practically useful improvement: Falcon increases from 57.33 to 66.20 IoU while retaining category-only prompts at inference. This supports the central idea that improved prompt-conditioned predictions can become persistent model improvements.

  2. The evaluation covers six datasets, two segmentation backends, and multiple prompt optimizers, with additional generalization and failure analyses. Paired photograph-level confidence intervals and explicit reporting of regressions help characterize where the method succeeds and fails.

Critical weaknesses
  1. The broad label-free framing is not supported for the complete RL-based pipeline: Section 5 uses annotated masks to compute rewards, and the main distillation experiments assume known target categories. These supervision requirements materially limit the claim of autonomous adaptation without ground-truth labels.
  2. The approximately 20-point improvement primarily reflects per-trajectory maxima identified retrospectively using ground-truth IoU, rather than reliably selected outputs. For SAM 3.1, fixed-length refinement increases the maximum from 41.90 to 68.99 IoU, while the final mask reaches only 41.96, making this distinction consequential to the claimed improvement.
  3. The reported RL gains differ substantially between Table 2 and Appendix Table 4 without sufficient explanation: for Qwen 4B, Falcon gains are 1.15 versus 12.39 points, and SAM gains are 3.01 versus 24.99 points. Until the baseline and evaluation differences are reconciled, the incremental benefit of RL remains difficult to assess.
Questions for authors
  1. What accounts for the different RL gains in Tables 2 and 4? Please specify whether the baseline checkpoints, decoding settings, parsing rules, or evaluated tasks differ.

  2. Does “label-free” refer only to student training, or also to prompt-optimizer training and target-category identification? Which experiments establish adaptation to a target domain without using that domain’s annotated masks?

  3. The RL-mask student exceeds the base-rewrite student by 1.07 IoU points, but their training sets differ because failed rewrites are excluded. Does this advantage persist when both students use the same training tasks?

2 comments

We would like to thank the reviewer for recognizing that the self-distillation experiment shows "a practically useful improvement" and for valuing our "explicit reporting of regressions". Below, we address the three weaknesses together with the questions that belong to them. We start with the discrepancy between Tables 2 and 4, since it affects how the other results should be read. New results are marked in blue in the revised paper.

In all new experiments, a rewrite is one new prompt that the MLLM proposes after it sees the image and the mask obtained with the category name. We evaluate on the same 600 test tasks (100 per dataset) and report mean IoU in percent.

[W3/Q1]: Different RL gains in Tables 2 and 4 of the paper. The two tables differ because they use different evaluation protocols, which we clarify below. Of the four factors that the reviewer lists, the baseline checkpoints and the evaluated tasks are the same in both tables, while the decoding settings and the parsing rules differ. In Table 2, the Base column is measured with native tool calls, reasoning, a limit of 1,024 tokens and up to eight attempts to repair an invalid output. The RL column is measured with our earlier JSON output format, a limit of 512 tokens and no repair. Table 4 uses the JSON format for both columns. With this format, the pretrained Qwen 3.5 4B returns an invalid output for about 10% of the tasks, which count as zero. This lowers its score in Table 4 and makes the RL gain appear larger than in Table 2.

Neither table isolates the effect of RL. We therefore evaluate each model before and after RL with a single protocol, which uses the same tasks, output format, token limit and decoding for both.

Table A: The same model before and after RL under a single protocol.

ModelRL setupFalcon, before and after RLSAM 3.1, before and after RL
Qwen 3.5 0.8Bas submitted49.3 and 62.9 (+13.5)33.3 and 41.2 (+7.9)
Qwen 3.5 4Bas submitted71.3 and 72.1 (+0.9)50.4 and 52.9 (+2.5)
Qwen 3.5 4Bupdated71.4 and 74.8 (+3.4)51.1 and 68.0 (+17.0)
Qwen 3.5 9Bupdated68.9 and 71.6 (+2.7)58.0 and 68.8 (+10.8)
Qwen 3.8 27Bupdated73.2 and 74.7 (+1.6)53.7 and 69.0 (+15.3)

The first two rows answer the question about the incremental benefit of RL in the submission. Under matched conditions, RL improved the 0.8B model substantially, while the gain for the 4B model was small. For the rows marked "updated", we adjusted the RL hyperparameters and train with a constant learning rate and 8 sampled rewrites per task, with the same number of rollouts per update. We selected these settings on 600 training tasks, and the test tasks were not used for the selection. RL then improves Qwen 3.5 4B by 3.4 points on Falcon (95% CI 1.8 to 5.0) and by 17.0 points on SAM 3.1 (95% CI 13.9 to 20.1), while the gain of the 27B model on Falcon is not significant. For reference, the category name alone reaches 60.2 on Falcon and 47.1 on SAM 3.1. In the revision, Tab. 2 reports the results with the adjusted hyperparameters under this protocol, and App. C describes the protocol and the training details.

[W2]: The 20-point improvement is an oracle. We agree that the per-trajectory maximum is selected with the ground truth and should be read as an upper bound. To address that point, we add a selection step without ground truth. A second MLLM call, the verifier, receives the image and the 11 masks of a trajectory (the mask of the category name and the ten refinements) and picks one. We apply it to all 1,800 trajectories of Qwen 3.8 27B, Gemma 4 31B and Muse Glimmer 30B, with the same MLLM or Qwen 3.8 27B as verifier.

Table B: Selecting one mask per trajectory. Ranges are over the three MLLMs and the two verifiers.

SegmenterCategory nameLast refinementVerifierMaximum (uses ground truth)
Falcon55.668.6 to 69.973.5 to 74.679.5 to 80.7
SAM 3.142.146.5 to 54.371.1 to 73.675.4 to 77.7

Without ground truth, refinement with a verifier improves over the category name by 17.9 to 19.0 points on Falcon and by 29.0 to 31.5 points on SAM 3.1. On SAM 3.1, where the last refinement alone only reaches 46.5 to 54.3, the verifier recovers 85% to 89% of the oracle gain. The revision reports the verifier numbers in the abstract and in Sec. 4 (Tab. 5), and marks the maximum as an oracle.

[W1/Q2]: Scope of "label-free". We agree that the scope of the term should be stated more precisely. In the submission, the term refers to the fine-tuning of the segmenter, while the MLLM was trained with rewards computed from masks of the same datasets, and all experiments assume known category names. To answer which experiments establish adaptation without target-domain masks, we run a new one. We fine-tune Falcon on masks from MLLMs that never saw a mask of the target dataset: Qwen 3.5 4B and Qwen 3.8 27B without any RL training, and Qwen 3.5 4B after RL training on the other five datasets only. All runs use the same training images, 2.36M trainable parameters and 200 steps, and we report the mean of 5 seeds.

Without fine-tuning, Falcon reaches 60.1 on the 600 test tasks. Fine-tuning on the masks of the three MLLMs gives 67.7, 68.1 and 67.8, and fine-tuning on ground-truth masks gives 69.8. Without any mask of the target dataset, we thus recover about 80% of the gain of supervised fine-tuning. Known category names remain an assumption. The only experiment with predicted categories is the reconstruction in App. N of the revision, and we now state this in Sec. 6.

[Q3]: RL advantage on matched training tasks. We repeated the experiment with identical training tasks and 5 seeds, and find that the advantage of the RL-trained policy persists. In this setting, fine-tuning on the masks of the RL-trained Qwen 3.5 0.8B is 2.0 points better than fine-tuning on the masks of the same model without RL (95% CI 1.2 to 2.9), compared with 1.07 points in the submission. With the updated RL training, the difference grows to 4.9 points (95% CI 3.4 to 6.4). For Qwen 3.5 4B, it is 0.3 points and not significant after 200 steps. Since all runs saturate at about 68 after 200 steps, we repeat the experiment with 1,000 steps and 3 seeds. The difference for the 4B model is then 0.9 points (71.9 compared with 70.9, 95% CI 0.4 to 1.6), and ground-truth masks give 73.9.

In addition, we add a simple control. The category name returns an empty mask for 22% of the training tasks, and removing these tasks before training already gives 67.3 after 200 steps and 70.8 after 1,000 steps. The RL-trained models are 1.1 to 1.2 points above this filter at 1,000 steps. This control is reported in Sec. 6 of the revision (Tab. 4).

We hope these clarifications sufficiently address your concerns. If this resolves your concerns, we would be grateful if you could consider adjusting your score.

Reviewer_YUqEOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how multimodal large language models (MLLMs) can optimize text prompts for text-promptable segmentation models (PSMs) and transfer the resulting segmentation gains back into the segmentation models through self-distillation. The authors first explore the potential of prompt refinement through training-free iterative prompt optimization. They then train a single-step prompt-rewriting policy using GRPO with segmentation IoU as the reward. Finally, masks generated from the rewritten prompts serve as pseudo-labels for fine-tuning the segmentation model, improving its performance when given the original category-name prompts. Extensive experiments on six datasets spanning multiple domains evaluate the effectiveness of each stage.

Critical strengths
  1. This problem is of practical significance, and the various stages of the proposed method are logically interconnected. Single-step rewriting reduces the need for multi-turn reasoning, while self-training eliminates reliance on an external rewriter during the deployment phase. Together, these elements provide a clear path toward optimizing both performance and cost.

  2. The experiments are extensive and incorporate valuable internal controls. The work involves two influential text-promptable segmentation models (PSMs), multiple MLLMs in different architectures and sizes, and six datasets, covering leave-one-dataset-out generalization, prompt-style guidance, and a comparison of different pseudo-label sources. These experiments help clarify the scope of the method's applicability.

Critical weaknesses
  1. The individual stages lack sufficient comparisons with existing methods

(a) Section 5 trains an MLLM with GRPO to improve text-prompt rewriting, but the experiments primarily compare the model before and after training, without comparisons against other learned prompt optimization methods. For example, Promptist [1] also uses reinforcement learning to optimize prompts. Although it targets text-to-image generation, its training approach provides a relevant reference.

(b) The core of Section 6 is supervised fine-tuning of the segmentation model using pseudo-masks generated with optimized prompts. This is closely related to existing pseudo-label self-training and unsupervised domain adaptation methods. However, the current experiments primarily compare different pseudo-label sources and do not sufficiently establish an advantage over existing approaches. Representative baselines with matched supervision and training budgets are needed.

[1] Hao Y, Chi Z, Dong L, et al. Optimizing prompts for text-to-image generation[J]. Advances in Neural Information Processing Systems, 2023, 36: 66923-66939.


  1. Insufficiently Controlled Self-Distillation Experiments

a. If Section 6 aims to demonstrate self-training without target-domain mask annotations, the prompt-rewriting MLLM should also be trained without using target-domain ground-truth masks to compute RL rewards. Otherwise, the overall pipeline still relies on target-domain mask supervision, even though the student directly uses pseudo-labels, and therefore does not establish target-domain label-free adaptation. If such annotations are available, a comparison against direct SFT with ground-truth masks is needed, using the same training images, trainable modules, and training budget, to assess the relative benefits of pseudo-label training. The differing experimental configurations in Appendices H and I do not provide this controlled comparison.

b. Table 13 leaves the incremental contribution of RL to the final self-distillation gains insufficiently isolated. Students trained on pseudo-labels generated with pre-RL rewrites already achieve a mean IoU of 65.13, compared with 66.20 for RL-based rewrites and 57.33 for the original model. Thus, under the reported settings, most of the improvement is already achieved through prompt rewriting and self-distillation without RL, while RL provides an additional 1.07 IoU points. Although this difference is statistically supported by test-set resampling, the two rewrite conditions use different effective training sets, and the reported confidence intervals do not capture training variability. Comparisons on matched training samples and repeated training runs would help isolate the magnitude and robustness of RL’s additional contribution.

Questions for authors

Please refer to weaknesses.

2 comments

We thank the reviewer for pointing out the "practical significance" of the problem and for noting that our experiments "incorporate valuable internal controls". Below, we address the missing comparisons and the control of the self-distillation experiments, and we report new experiments for all four points. New results are marked in blue in the revised paper.

In all new experiments, a task is one image with one category name. A rewrite is one new prompt that the MLLM proposes after it sees the image and the mask obtained with the category name. We evaluate on the same 600 test tasks (100 per dataset) and report mean IoU in percent.

[W1(a)]: Comparison with other learned prompt optimizers. We agree that a comparison with other learned prompt optimizers strengthens Sec. 5. The closest learned optimizer is Promptist [1], which trains a language model to rewrite prompts for text-to-image models, first with supervised fine-tuning on manually engineered prompts and then with RL on the quality of the generated image. Our RL stage follows the same idea for segmentation models, with two differences. Our policy sees the image and the current mask, and it starts directly from a pretrained MLLM, since no engineered prompts exist for segmentation models. To isolate the learning rule, we train an alternative learned optimizer, reward-filtered imitation, which samples 8 rewrites per task and trains on the best one, as in ReST [2]. It uses the same model (Qwen 3.5 4B), training data, number of rollouts and evaluation as GRPO, so that only the learning rule differs. In addition, we compare with the closest existing methods, which optimize the prompt without training: SAM 3 Agent [3], EVOL-SAM3 [4] and an OPRO-style optimization of the instruction [5]. We run SAM 3 Agent and EVOL-SAM3 from their released code, with the same 4B model and with a model that is seven times larger (Qwen 3.8 27B), which we also train with GRPO.

Table 1: Prompt optimization methods at the same model size.

MLLMMethodMLLM calls per imageFalconSAM 3.1
noneCategory name as prompt060.247.1
Qwen 3.5 4BRewrite without training171.451.1
Qwen 3.5 4BGRPO (ours)174.868.0
Qwen 3.5 4BReward-filtered imitation172.453.0
Qwen 3.5 4BOPRO172.458.4
Qwen 3.5 4BEVOL-SAM35 to 671.058.8
Qwen 3.5 4BSAM 3 Agent253.050.6
Qwen 3.8 27BGRPO (ours)174.769.0
Qwen 3.8 27BEVOL-SAM3569.464.7
Qwen 3.8 27BSAM 3 Agent2 to 374.672.9

With a 4B model, GRPO outperforms every other method on both segmenters, and it needs a single MLLM call per image. It is 2.4 points above reward-filtered imitation on Falcon and 15.1 points above it on SAM 3.1. Among the existing methods, the smallest margins are 2.4 points on Falcon (OPRO, 95% CI 0.6 to 4.2) and 9.2 points on SAM 3.1 (EVOL-SAM3, 95% CI 6.2 to 12.3). SAM 3 Agent scores low with a 4B model because it does not complete a valid run for about a quarter of the tasks. With a 27B model, GRPO outperforms EVOL-SAM3 on both segmenters and is level with SAM 3 Agent on Falcon. On SAM 3.1, SAM 3 Agent with the 27B model is 4.0 points above GRPO (95% CI 1.6 to 6.3).

For the GRPO results in Table 1, we adjusted the RL hyperparameters and train with a constant learning rate and 8 sampled rewrites per task. We selected these settings on 600 training tasks, and the test tasks were not used for the selection. Table 1 corresponds to Tabs. 2 and 3 of the revised paper, and the training details are given in App. C.

References:

[1] Hao et al.: Optimizing Prompts for Text-to-Image Generation. In NeurIPS, 2023

[2] Gulcehre et al.: Reinforced Self-Training (ReST) for Language Modeling. arXiv, 2023

[3] Carion et al.: SAM 3: Segment Anything with Concepts. In ICLR, 2026

[4] Ye et al.: Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Prompting. arXiv, 2025

[5] Yang et al.: Large Language Models as Optimizers. In ICLR, 2024

[W1(b)]: Comparison with pseudo-label self-training. We agree that pseudo-label baselines with matched supervision and budget strengthen Sec. 6. We therefore add two of them to Table 2 below. The first is plain self-training, where Falcon is trained on its own masks for the category name. The second removes empty masks before training, which is a common confidence filter. We did not run domain-adaptation methods that need a different training loop, and we state this in the limitations (App. A).

[W2(a)]: Target-domain masks and ground-truth fine-tuning. Indeed, the MLLM in the submission was trained with rewards computed from masks of the target datasets. We address this in both ways the reviewer suggests. First, we use MLLMs that never saw a mask of the target dataset: Qwen 3.5 4B and Qwen 3.8 27B without any RL training, and Qwen 3.5 4B after RL training on the other five datasets only. Second, we add fine-tuning on ground-truth masks. All rows of Table 2 use the same training images, the same 2.36M trainable parameters and 200 training steps. They also use the same tasks, namely the 2,456 of 3,000 tasks for which every MLLM returned a valid rewrite. Each row is the mean of 5 seeds, and the standard deviation over seeds is at most 0.5 points. At test time, Falcon only receives the category name.

Table 2: Falcon after fine-tuning on different masks. Without fine-tuning, Falcon reaches 60.1.

Training masksMasks of the target dataset usedIoU
Ground truthyes69.8
Rewrite by Qwen 3.8 27B, no RLno68.1
Rewrite by Qwen 3.5 4B, RL on the other five datasetsno67.8
Rewrite by Qwen 3.5 4B, no RLno67.7
Category name, empty masks removedno67.3
Category name (plain self-training)no57.9

Without any mask of the target dataset, the rewrite masks recover about 80% of the gain of ground-truth masks (7.6 to 8.0 of 9.7 points). The revision no longer uses the term "label-free" and states that RL rewards use masks.

Table 2 also provides an additional insight. The category name returns an empty mask for 542 of the 2,456 training tasks. Plain self-training on these masks hurts, but removing the empty masks already gives 67.3. After 200 steps, the rewrite masks are only 0.4 to 0.8 points above this filter. This baseline is included in Tab. 4 of the revision.

[W2(b)]: Contribution of RL to self-distillation. The reviewer asks whether the 1.07 points from RL in Table 13 hold on matched training samples and repeated runs. With identical training tasks and 5 seeds, the difference for the Qwen 3.5 0.8B model of the submission is 2.0 points (95% CI 1.2 to 2.9), so it was not caused by the different training sets. With the adjusted RL hyperparameters from [W1(a)], the masks of this model improve from 54.6 to 73.5 IoU, and the difference between the two students grows to 4.9 points (95% CI 3.4 to 6.4).

For Qwen 3.5 4B, RL improves the masks from 74.6 to 78.1 IoU, but after 200 steps the student only improves by 0.3 points, which is not significant. Since all students saturate at about 68 after 200 steps, we repeat the experiment with 1,000 steps and 3 seeds. Falcon then reaches 73.9 with ground-truth masks, 72.0 with the RL-trained 0.8B model, 71.9 with the RL-trained 4B model, 71.1 with the 4B model that was RL-trained on the other five datasets, 70.9 with the 4B model without RL, and 70.8 with the empty-mask filter. With this budget, RL adds 0.9 points for the 4B model (95% CI 0.4 to 1.6) and 7.0 points for the 0.8B model, and both RL-trained models are about 1.1 points above the filter. Thus, RL improves self-distillation for both model sizes, with a large effect for the small model and an effect of about one point for the 4B model. We have revised Sec. 6 accordingly, and Tab. 4 of the revision reports both training budgets.

We hope these clarifications sufficiently address your concerns. If it's possible to revisit the score or if there are any additional questions you would like to explore, we would greatly appreciate it.

Comments on the paper

1 comment

Dear Reviewers, Dear AC,

We sincerely thank all three reviewers for their efforts at ICLR and for the thoughtful and constructive feedback.

We are encouraged that the reviews highlight several strengths of our work: (1) the practical significance of improving promptable segmentation models without manual annotation (YUqE), (2) the practically useful improvement through self-distillation and our explicit reporting of regressions (c83T), and (3) the clarity of the manuscript (P5PU).

We have now updated the paper with a revised version that addresses the questions mentioned in the reviews. The most important changes are (new results highlighted in blue in the latest revision):

  • We added comparisons with SAM 3 Agent, EVOL-SAM3, and OPRO at the same model size, as well as with reward-filtered imitation as an alternative learning rule (Sec. 5.1, Tab. 3).
  • We added a verifier that selects masks without ground truth, and we now mark the per-trajectory maximum as an oracle (Sec. 4, Tab. 5).
  • We repeated the self-distillation experiments with matched training data, ground-truth and pseudo-label baselines, and MLLMs that never received masks of the target dataset (Sec. 6, Tab. 4).
  • We added step-by-step algorithm boxes for all three stages (App. F) and a limitations section (App. A).

We have addressed the comments of each reviewer individually in our corresponding responses. We would be happy to address any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.