acceptodds
Under review as a conference paper at ICLR 2027

PinPoint: Training-Free Referring Image Segmentation via Prompt Disambiguation

20 views

Abstract

Modern referring image segmentation pipelines couple a vision-language model (VLM) for grounding with a promptable segmenter such as the Segment Anything Model (SAM) for mask generation. Prior training-free instances of this recipe often trail fine-tuned and reinforcement-learning (RL)-tuned specialists, and it has been unclear whether the gap comes from the VLM's grounding, SAM's capacity, or the prompt. We show that prompt ambiguity is an important bottleneck in this architecture; a VLM-proposed bounding box (bbox) leaves SAM to guess which pixels inside the bbox belong to the object the expression denotes. Interior points are the natural disambiguator, but where they are placed matters; naively sampled points can land on boundaries, distractors, and background clutter, and even hurt performance compared to the bbox alone. Supervised and RL-tuned methods improve spatial prompts through task-specific training; we show that informative point prompts can instead be constructed without such training. At a matched budget of up to five interior points and on the same frozen stack, our selector improves cumulative Intersection-over-Union (cIoU) by 12–18 points across RefCOCO(+/g) splits. We introduce **PinPoint**, a deterministic, training-free point selector that fuses four visual cues into a consensus map, selects compact, spatially diverse points away from boundaries, and uses the frozen VLM to label each point and filter low-confidence labels. Without any task-specific training, PinPoint is competitive with supervised and RL-tuned specialists while issuing only two VLM calls per query.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual grounding, referring image segmentation, reasoning segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_T8wkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PinPoint, a training-free referring image segmentation framework using frozen vision-language models (VLMs) and SAM. Given a VLM-predicted bounding box, PinPoint combines color saliency, edge density, local entropy, and a Gaussian spatial prior to select informative interior points. A frozen VLM then labels these points as positive or negative, followed by confidence filtering and SAM-based segmentation. Experiments on RefCOCO, RefCOCO+, RefCOCOg, and ReasonSeg demonstrate competitive performance without task-specific training. Controlled comparisons and ablations investigate the contributions of point selection, semantic verification, and filtering.

Critical strengths
  1. Convincing improvements under controlled experimental settings. The paper keeps the VLM, SAM, predicted bounding boxes, and maximum point budget fixed when comparing prompt-selection strategies. PinPoint consistently improves over bounding-box-only prompting and uniform point sampling. The additional comparisons with non-uniform point selectors in Appendix H further strengthen the empirical evidence.

  2. A practical training-free framework. PinPoint improves segmentation without task-specific model updates. The ablation studies demonstrate that point selection, semantic verification, and confidence filtering jointly contribute to performance, highlighting the importance of prompt construction in frozen VLM–SAM pipelines.

Critical weaknesses
  1. Limited methodological novelty. The overall select-then-label framework is closely related to SAM4MLLM's PQPP, which also samples points within predicted bounding boxes and uses a VLM to determine their labels. Furthermore, automatic point selection using visual cues has been explored in SAMAug. PinPoint primarily combines existing visual cues, spatial suppression, VLM verification, and confidence filtering. The distinct methodological contribution beyond these established techniques is therefore somewhat limited.

  2. Insufficient evidence for the necessity of the full four-cue design. Appendix H shows that a simpler saliency-plus-Gaussian-prior selector with Soft-NMS already achieves 73.5/77.3/70.9 cIoU on RefCOCO val/testA/testB, compared with 75.9/81.7/71.1 for the full method. The additional gains are modest on some splits, particularly testB (+0.2). Moreover, Table 6 shows that adding the selected points as positive prompts alone does not improve upon bounding-box-only segmentation; substantial gains emerge after VLM-based point labeling and confidence filtering. These results raise questions about the independent contribution of the proposed four-cue selection mechanism.

  3. Incomplete comparisons with recent training-free RIS methods. Although the paper includes several relevant baselines, it does not discuss or compare recent training-free approaches such as RefChess (ICML 2026) and Hybrid Global-Local (CVPR 2025). While these approaches differ in their inference mechanisms, discussing their technical differences and providing comparisons under compatible protocols would help establish PinPoint's competitiveness within recent training-free RIS research.

  4. Unclear hyperparameter selection protocol. Appendix B repeatedly refers to both validation and testB performance when justifying the default hyperparameter settings. It is therefore unclear whether testB results influenced parameter selection. If so, the independence of the reported test evaluation would be compromised. The authors should clarify whether all hyperparameters were fixed using validation data before examining test results.

  5. Limited evidence of generalizability and practical efficiency. The main experiments rely on a single Qwen2.5-VL-7B and SAM 2-L configuration. It remains unclear whether the proposed point-selection strategy and confidence filtering generalize to other foundation models, especially since verbalized confidence scores can be model-dependent. Additionally, although point selection itself is lightweight, the complete pipeline requires two VLM calls and approximately 4.95 seconds per query on an A6000 GPU (Appendix F). More direct accuracy–latency comparisons would help clarify its practical advantages over alternative training-free methods.

Questions for authors
  1. Methodological novelty: What are the key technical differences between PinPoint and SAM4MLLM's PQPP or SAMAug, beyond the choice and combination of point-selection heuristics?

  2. Four-cue design: Given the strong results of the simpler saliency-plus-Gaussian-prior baseline in Appendix H, what additional evidence supports the necessity of the edge and entropy cues? Under what conditions do they provide consistent benefits?

  3. Recent training-free baselines: Could the authors discuss and, where feasible, compare with RefChess (ICML 2026) and Hybrid Global-Local (CVPR 2025), considering differences in backbone models and inference budgets?

  4. Hyperparameter selection: Were all default hyperparameters determined exclusively using validation data before examining testB results? Please clarify the experimental protocol.

  5. Generalization and efficiency: Does PinPoint achieve consistent improvements with other VLM or SAM backbones? How does its end-to-end accuracy–latency trade-off compare with alternative training-free pipelines?

Reviewer_n9BJOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PinPoint, a training-free RIS method that improves the interface between a frozen VLM and SAM by adding informative interior point prompts to VLM-predicted bounding boxes. PinPoint selects spatially reliable points using multiple visual cues, verifies them as positive or negative with the frozen VLM, and filters low-confidence points before SAM segmentation. The method achieves strong performance on RefCOCO/+/g and ReasonSeg without task-specific training, showing that prompt construction and semantic point verification are important for training-free VLM+SAM segmentation.

Critical strengths
  1. Strong performance without task-specific training: PinPoint achieves competitive performance on RefCOCO, RefCOCO+, RefCOCOg, and ReasonSeg without any task-specific fine-tuning or reinforcement learning. This is particularly appealing because it improves a fully frozen VLM+SAM pipeline while keeping the method training-free.

  2. Extensive empirical analysis: The paper provides a broad set of controlled experiments, including stage-wise ablations, hyperparameter sensitivity, point-selection comparisons, confidence-filter analysis, runtime breakdown, and GRES evaluation. These analyses help clarify which components contribute to the final performance and make the empirical study relatively thorough.

Critical weaknesses
  1. Motivation is clear but somewhat pipeline-specific: The motivation is intuitive and clearly presented. A VLM-predicted bounding box can be ambiguous for SAM because it may contain the target together with distractors, boundaries, and background. The question of whether informative point prompts can be constructed without task-specific training is also well motivated. However, the identified problem appears relatively specific to the training-free VLM-to-SAM pipeline rather than being a general limitation of referring image segmentation. Moreover, the ablation suggests that much of the gain comes from VLM-based point verification rather than point selection itself, which somewhat weakens the paper’s central motivation around informative point placement.

  2. Unclear efficiency of repeated VLM inference: PinPoint requires two VLM forward passes per query. One for localization and another for point verification. While the second VLM call improves performance, it also dominates the additional inference cost, with the two VLM stages accounting for roughly 95% of the total runtime. It would be helpful to compare against a one-call baseline, such as directly generating both the bounding box and informative points in a single VLM pass, to better understand whether the performance gain justifies the additional inference cost.

  3. Limited broader insight for the RIS community: The main finding of the paper appears to be that carefully selected and semantically verified point prompts can serve as a better interface between a frozen VLM and SAM. While this is useful for improving training-free VLM+SAM pipelines, it is less clear how broadly this insight advances referring image segmentation beyond this particular modular setting. The method does not directly improve referential grounding or segmentation representations, but mainly improves how two existing foundation models are connected. Therefore, the broader takeaway for the RIS community may be somewhat limited unless the authors can demonstrate that the proposed insight generalizes to other RIS architectures or leads to a more general principle for referring segmentation.

Questions for authors

Could the authors clarify how PinPoint should be interpreted in the GRES setting? The current gRefCOCO evaluation compares only against a bounding-box-only baseline, and no-target accuracy remains unchanged because PinPoint does not modify the predicted box set. Would it be possible to compare against dedicated GRES methods or further discuss whether PinPoint provides benefits beyond mask refinement for already localized instances?

Reviewer_bUY5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper tackles the problem of referring image segmentation, where the goal is to obtain a binary segmentation mask from an input image and text prompt describing the desired segmentation target. Their novel PinPoint framework uses a VLM (Qwen2.5-VL-7B) and SAM ( SAM 2-L) to derive the segmentation mask without fine-tuning either of these pre-trained models.

First the VLM generates candidate bounding boxes, then PinPoint finds candidate point prompts per bounding box. Each point prompt gets labeled foreground or background + uncertainty estimate by the same VLM. After filtering the points based on uncertainty, SAM is prompted to generate one binary mask for each bounding box's set of point prompts, fusing these masks with "OR" yields the final binary mask.

The model outperforms other training-free methods and is on par with approaches that utilize fine-tuning / RL-tuning on all four benchmarks (RefCOCO, RefCOCO+, RefCOCOg, ReasonSeg val).

Critical strengths
  1. Training-Free: The proposed algorithm does not require any model parameter fine-tuning or reinforcement learning runs. This is the main strength of the proposed algorithm, as this allows incorporating future models (new VLMs, or Point Prompt Segmentation Models such as SAM) without compute heavy retraining.
  2. Strong Results: Both PinPoint variants achieve SOTA results across all four benchmarks in the training-free category and are also on par with methods which include training / fine-tuning.
Critical weaknesses

Relying on classical saliency maps: Classical algorithms for estimating saliency maps are very sensitive to hyperparameters and PinPoint introduces a lot of tunable hyperparameters (hps listed in table 3, description of the components in lines 196 - 202). The method therefore might require tuning hyperparameters when applied to other data.

Questions for authors
  1. Feature maps instead of classical saliency maps? Similar to the point mentioned in the critical weaknesses. There is a general question on why not using feature maps of existing models instead of using classical saliency maps. For example by using SAM itself as to predict a saliency / binary mask from a bounding box and use it to sample point prompts, placing points more based on semantics instead of visual texture. Has something similar been tried and or why are classical saliency maps preferred here?
  2. Performance of two VLM calls: As stated in line 413, calling VLMs attributes to 95% of the wall-clock-time. Even if batching (line 265, line 1115) reduces the call count, the compute would still scale with number of points ×\times number of candidate boxes?

Minor Notes:

  • Lines 367 - 379, talking about both Table 1 and Table 2 at the same time can be a bit confusing on what to look at. Might be good to go through one table after another, not both at the same time.
  • Since this method uses four distinct cues / saliency maps, it would be good to have a few visual examples of these maps in the appendix. For example, the motivation behind CmsmC_{msm} is to use "edge density and entropy as smoothness guards" (line 215) which would be good to demonstrate with a visual example.
  • Parts of the introduction read more like a related work section, resulting in the actual related work section being quite small.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.