- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses the sensitivity of promptable segmentation models to prompt variation. Even when different clicks, bounding boxes, or text prompts refer to the same object, models such as SAM can produce substantially different masks.
To address this problem, the authors propose Best-Teaches-Rest (BTR), a post-training objective that encourages consistent predictions across prompts referring to the same target. BTR selects the prediction with the highest ground-truth IoU as the teacher and trains the remaining predictions to match it. The method also includes a warmup stage to improve teacher reliability and a KL regularization term to limit changes from the pretrained model.
The authors evaluate BTR on SAM and SAM 3 using click, box, and text prompt variations. The results show substantial improvements in segmentation accuracy and prompt robustness over the frozen pretrained models. BTR also outperforms matched augmented supervised fine-tuning (Aug-SFT) baselines and shows strong zero-shot transfer for spatial prompts on LVIS and RAOS. Additional experiments on RClicks demonstrate improvements under real-user click distributions.
Critical strengths
1. The paper addresses an important practical problem and provides convincing improvements in prompt robustness.
I find the problem well motivated. A segmentation model should produce similar masks when different valid prompts refer to the same object. The paper evaluates this problem carefully by considering not only average IoU but also worst-prompt IoU and variation across prompts.
The experimental results are strong, particularly for spatial prompts. On EntitySeg, BTR improves average click IoU from 47.84 to 81.97 and worst-prompt IoU from 37.66 to 76.04. The improvements also transfer to unseen datasets such as LVIS and RAOS. In addition, the RClicks results show that the method is effective with real-user clicks, which strengthens the practical relevance of the work.
2. BTR shows meaningful improvements over well-controlled Aug-SFT baselines.
The authors compare BTR with augmented supervised fine-tuning under matched training data, prompt groups, optimization steps, and model configurations. This is an important comparison because simply exposing the model to diverse prompts already improves robustness considerably.
BTR further improves average IoU from 80.11 to 81.97 over Aug-SFT-Equal-λ, while also improving worst-prompt IoU from 73.12 to 76.04 and reducing prompt variation from 3.46 to 2.79. These consistent improvements across different metrics support the effectiveness of the proposed objective beyond standard prompt augmentation. I consider this a meaningful contribution, especially given the strong performance of the fine-tuned baseline.
Critical weaknesses
1. The improvements are less consistent for text prompts, which limits the generality of the proposed approach.
Although BTR performs well with click and box prompts, the results for text prompts are less convincing. On LVIS, the click+text model reduces noun-phrase IoU from 57.98 to 55.83, while prompt variation slightly increases. This contrasts with the strong improvements observed for spatial prompts.
The authors discuss the ambiguity of noun phrases and the differences between spatial and text prompts. However, this suggests that aligning predictions across prompts may not always be beneficial, particularly when text prompts are ambiguous or when their distribution differs from the training data.
Since the paper presents BTR as a general objective that can be applied across different prompt types, I think the limited generalization to text prompts is an important weakness.
2. The effectiveness of BTR appears to depend substantially on the pretrained model.
The results on RAOS show a notable difference between SAM and SAM 3. While BTR improves SAM 3 substantially, applying BTR to the original SAM decreases average click IoU from 27.67 to 15.87. The worst-prompt IoU also decreases considerably.
The authors acknowledge this limitation and suggest that BTR works best when the pretrained model already has sufficient knowledge of the target domain. However, this raises a question about when the proposed objective can be expected to improve robustness. Since BTR uses the model's own best prediction as the teacher, it may reinforce incorrect predictions when the pretrained model performs poorly on the target domain.
These results do not undermine the strong improvements on SAM 3, but they limit how broadly the method can be applied without considering the capabilities of the base model.
Questions for authors
1. Could the authors clarify why BTR does not generalize as well to text prompts?
On EntitySeg, BTR improves noun-phrase segmentation, but on LVIS, the click+text model performs worse than the frozen SAM 3 baseline. Is this mainly caused by differences in the text prompt distributions, or does the consistency objective itself make it difficult to handle ambiguous noun phrases? Understanding this distinction would help clarify whether the limitation comes from the training data or from the proposed objective.
2. What determines whether BTR improves or degrades performance on an unseen domain?
On RAOS, BTR substantially improves SAM 3 but degrades the performance of the original SAM. The authors suggest that this depends on the pretrained model's knowledge of the target domain. Could the authors provide more insight into how teacher quality affects this behavior? In particular, it would be useful to know whether there are measurable conditions under which BTR is likely to improve robustness rather than reinforce incorrect predictions.
3 comments
Thank you for the additional experiments.
W1/Q1. Text-prompt generalization, So, as I understand it,
-
the fact that both Aug-SFT and BTR degrade relative to frozen SAM 3 suggests that the performance drop is not specific to the BTR consistency objective.
-
In addition, BTR performs better than Aug-SFT in the matched text-only setting, which addresses much of my original concern. I consider this concern substantially mitigated.
W2. Dependence on the base model / Q2. What determines transfer to unseen domains. Thank you for the clarification. I appreciate that the authors acknowledge the limitation and have clarified it in the revised paper.
I maintain my original rating.
Response to Reviewer 6qTF’s follow-up comment
Thank you for the follow-up and for carefully reading our response.
Your understanding of W1/Q1 is correct on both points, and we are glad that the additional experiments addressed much of the concern. We also appreciate your comments on W2/Q2; the corresponding limitation and clarifications are reflected in the revised paper.
Thank you again for your constructive feedback, which has helped us improve the paper.
Response to Reviewer 6qTF: text-prompt transfer and dependence on the base model
We thank the reviewer for the constructive feedback. Changes in the revised PDF are marked in blue, and the section and table numbers below refer to the revision.
W1/Q1. Text-prompt generalization. Clicks and boxes specify the target's location directly, so their signal keeps the same form across domains. Noun phrases instead depend on a dataset's category vocabulary and phrasing distribution. LVIS has a long-tail vocabulary with concepts and expressions that EntitySeg covers poorly, so text alignment transfers less readily than spatial alignment.
To separate the objective from the training data, we ran additional LVIS evaluations. Due to time constraints, we could train the matched Aug-SFT baseline only in the text-only setting, so we compare it against BTR (text) and include the BTR (click+text) result for reference:
In the matched text-only comparison, BTR exceeds Aug-SFT by 2.51 points in IoU_avg, but both fall below frozen SAM 3. The drop is therefore not specific to the consistency objective and points to source-domain specialization from post-training on EntitySeg. BTR (text) and BTR (click+text) reach nearly the same IoU_avg (55.80 vs. 55.83), so we expect the same explanation to hold for the click+text model the reviewer refers to. These results are now in Table 1, with the discussion in Sec. 6 and App. A.3.
W2. Dependence on the base model. When the base model covers an instance so poorly that its predictions are all similarly inaccurate, teacher selection can become unstable, and the training signal over iterations may align to a suboptimal mask. BTR limits this in three ways: the teacher is selected by ground-truth IoU and supervised by the GT anchor, stop-gradient blocks consistency gradients from reaching the teacher (App. C.2), and consistency is enabled only after warmup (Sec. 4.4). In-domain, the stable-but-wrong fraction falls from 34.14% (frozen) to 14.36% (App. G.1), so accuracy and consistency improve together. How far this carries over to an unseen domain depends on the base model, as analyzed in Q2. We have added these conditions to the corresponding paragraph in Sec. 6.
Q2. What determines transfer to unseen domains.
(i) Teacher quality. We varied the teacher selection rule on SAM 3. On RAOS, the best-IoU teacher reaches a click IoU_avg of 52.38, compared with 49.23 for a center-click teacher, 48.63 for a random teacher, and 49.44 for matched Aug-SFT, which corresponds to the GT-teacher (full results in Table 4(c) and in our response to Reviewer Ts2L, W1/Q1). Since all training used EntitySeg only, accuracy-based teacher selection in the source domain also benefits the unseen domain.
(ii) Transfer conditions. Baseline accuracy alone does not predict transfer: frozen SAM (ViT-H) and SAM 3 have similar RAOS click IoU_avg (27.67 vs. 29.19), yet BTR yields 15.87 and 52.38, respectively. We believe transfer depends on whether the base model responds to prompt variation similarly in the source and target domains, which in turn depends on its architecture and pretrained representation. We have not yet established a quantitative predictor for this. Sec. 6 now includes this discussion and outlines such a predictor as future work.