- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The manuscript improves MLLMs by optimizing text prompts to produce substantially better masks and image segmentation results. Training-free is an advantage. Generally, the manuscript is easy to read.
Critical strengths
It is claimed that a promising accuracy improvement can be attained.
Critical weaknesses
First, baseline methods such as SAM3Agent and EVOLSAM3 based on MLLMdriven prompt rewriting already exist, which undermines the novelty of the proposed approach. Second, as a submission to ICLR, the paper lacks sufficient contributions. The socalled selfdistillation scheme described in the paper is not selfdistillation in a rigorous sense. Third, aside from the nonoriginal framework, the paper is deficient in experimental comparisons across different methods. Furthermore, implementation details (including critical ones) of the method are omitted, violating the selfcontainment principle of academic papers and rendering the experiments hard to reproduce. In addition, the existing typesets e.g.“the PSM S understands”must be corrected.
Questions for authors
- Please provide necessary details of the proposed method. For inatance, how to carry out iteratively optimize the text prompt for segmentation models and what are the step-by-step procedures are very important. Withou these details, readers cannot experimentally follow the paper at all.
- Sufficient experimental comparison and analyes must be offered.
- Analyses on the shortcomings of the proposed method and possible modifications are also expected.
2 comments
[W4/Q1]: Missing details and step-by-step procedure. The instructions, tool definitions and decoding settings are given in App. D of the revision, and we agree that the procedure could have been explained more clearly. The revision contains one algorithm box for each of the three stages (Algs. 1 to 3 in App. F). Briefly, the stages work as follows.
Iterative refinement (Sec. 4). We first prompt the segmenter with the category name. The MLLM then receives the image and the image with the resulting mask drawn on top, and answers with a new prompt, which we pass to the segmenter. This is repeated up to ten times, and the MLLM always sees all previous prompts and masks. Finally, the MLLM or a separate verifier call selects one of the masks.
RL training (Sec. 5). The MLLM receives the image, the category name and the mask of the category name, and outputs one new prompt. The reward is the IoU between the mask of this prompt and the ground-truth mask. We train with GRPO using 8 sampled prompts per task, 32 tasks per update, 100 updates and LoRA with rank 64.
Self-distillation (Sec. 6). For each training image, we generate a mask with the rewritten prompt. We then fine-tune the segmenter to produce this mask when it is given the category name. We train 2.36M parameters for 200 or 1,000 steps with batch size 4 and learning rate 1e-5.
We will release the code and the outputs for all tasks upon acceptance.
[W5]: Typesetting. We have fixed "the PSM S understands" in the revision.
[Q3]: Shortcomings and possible modifications. The revision contains a limitations section (App. A) with the following points. First, iterative refinement does not improve monotonically, and the last prompt is often worse than the first rewrite. A selection step is therefore required, and we now provide one that works without ground truth. Second, fine-tuning the segmenter brings little gain on Fashionpedia, also with ground-truth masks (+0.7 points). Third, the category name returns an empty mask for 22% of the training images, and removing these images accounts for a substantial part of the gain from fine-tuning. Fourth, SAM 3 Agent with a 27B MLLM remains better than our method on SAM 3.1. Finally, RL training needs ground-truth masks for the reward, so only the fine-tuning of the segmenter works without masks of the target dataset.
We thank the reviewer for noting that the manuscript is "easy to read". The weaknesses are not numbered in the review, so we number them in the order in which they appear and address them together with the three questions. New results are marked in blue in the revised paper.
[W1]: Novelty with respect to SAM 3 Agent and EVOL-SAM3. We agree that both methods already use an MLLM to rewrite prompts for SAM 3, and we build on this idea. The difference lies in what happens after the rewriting. SAM 3 Agent [1] and EVOL-SAM3 [2] search for a good prompt at inference time with several MLLM calls per image, and they do not train a model. In contrast, we use the resulting masks for training. First, we train the MLLM with RL, so that a single rewrite is sufficient. Second, we fine-tune the segmenter on the improved masks, so that it returns better masks for the category name alone and no MLLM is needed at inference. We compare with both methods in [W3/Q2].
[W2]: Contributions and the term "self-distillation". We use the term because teacher and student are the same network with the same initial weights. The teacher is the segmenter with the rewritten prompt, the student is the segmenter with the category name, and no second segmentation model provides the training masks. We acknowledge that an MLLM chooses the prompt of the teacher, so the term can be understood more narrowly. If the reviewer prefers, we will call this stage "self-training with prompt-refined pseudo-labels" in the revision.
[W3/Q2]: Comparison with other methods. We agree that comparisons across methods strengthen the paper. To address that point, we run SAM 3 Agent and EVOL-SAM3 from their released code and add an OPRO-style optimization of the instruction [3]. All methods use the same MLLM, the same segmenters and the same 600 test tasks (100 per dataset). We repeat the comparison with a model that is seven times larger, Qwen 3.8 27B, which we also train with our RL setup. Both external methods use greedy decoding and the category name as query, and EVOL-SAM3 runs in the text-only mode of its released code, since Falcon does not accept box prompts. We report mean IoU in percent.
Table 1: Comparison with existing methods at the same model size.
With a 4B model, our method outperforms every existing method on both segmenters with a single MLLM call per image. It is 21.8 and 17.4 points above SAM 3 Agent, 3.8 and 9.2 points above EVOL-SAM3, and 2.4 and 9.6 points above OPRO, and all six differences are significant. The comparison with the MLLM without RL shows that the gain comes from RL training, which adds 3.4 points on Falcon and 17.0 points on SAM 3.1. SAM 3 Agent scores low with a 4B model because it does not complete a valid run for about a quarter of the tasks. With a 27B model, our method outperforms EVOL-SAM3 on both segmenters and is level with SAM 3 Agent on Falcon, while SAM 3 Agent is 4.0 points above our method on SAM 3.1. Table 1 is Tab. 3 of the revised paper.
Neither existing method changes the segmenter, whereas our second step does. After fine-tuning on the improved masks, Falcon improves from 60.1 to 68.2 IoU when it only receives the category name, and to 72.0 with a longer training budget, without any MLLM at inference.
References:
[1] Carion et al.: SAM 3: Segment Anything with Concepts. In ICLR, 2026
[2] Ye et al.: Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Prompting. arXiv, 2025
[3] Yang et al.: Large Language Models as Optimizers. In ICLR, 2024