acceptodds
Withdrawn from ICLR 2027

LLMSeg: Training-Free Text-Prompted Image Segmentation with B-Splines

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Promptable segmentation models such as SAM 3 achieve remarkable open-vocabulary performance but require dedicated architectures, large-scale annotation engines, and task-specific training. We ask a different question: can a general-purpose multimodal large language model (MLLM) segment images directly, without an added segmentation head, mask decoder, or training? We present LLMSeg, a training-free pipeline that turns a vision-capable LLM into a text-prompted image segmenter. LLMSeg decomposes segmentation into three LLM-driven stages. First, detect localizes all instances matching a free-form text query and describes their coarse morphology. Next, segment traces each instance using closed periodic B-splines confined to its detected box. These splines define an outer boundary and optional hole loops. Finally, refine iteratively proposes focus boxes around residual errors and retraces confined splines. The process ends when the mask stabilizes or the pass budget is exhausted. The output is a binary mask with inspectable intermediate geometry. We evaluate LLMSeg across ten referring and reasoning segmentation splits (ReasonSeg, RefCOCO, RefCOCO+, and RefCOCOg). We also evaluate open-vocabulary instance segmentation (LVIS, COCO, and SA-Co), semantic segmentation, and zero-shot detection (ODinW13 and RF-100VL). With an off-the-shelf Gemini-3.7-Flash backend, LLMSeg is competitive with and often surpasses the specialist SAM 3 model on identical items. It reaches 69.6 gIoU on ReasonSeg val without additional segmentation-specific training. Backend model updates can improve accuracy without retraining the pipeline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
lllms, image segmentation, sam3, text-prompted segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ixgGOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors investigate whether a general-purpose multimodal large language model (MLLM) can perform text-prompted image segmentation without relying on a dedicated segmentation head or a pixel-level decoder trained for segmentation. To this end, they propose a three-stage pipeline consisting of a series of MLLM calls. In the first stage, the MLLM is prompted with an input image and a text query to localize all matching object instances, producing bounding boxes along with brief descriptions of their morphology and topology. In the second stage, the model traces the boundaries of each identified instance by predicting control points for closed, periodic B-splines of degree 1 or 3. These splines are then densely sampled and rasterized into binary segmentation masks using the even–odd polygon filling rule (Foley et al., 1996). In the third stage, the predicted masks and their contours are overlaid on the original image and fed back into the MLLM for iterative refinement. At each iteration, the model identifies regions with potential segmentation errors, proposes localized focus boxes, and retraces the spline contours within those regions. This process continues until the masks stabilize or a predefined iteration or computational budget is exhausted. Using an off-the-shelf Gemini-3.7-Flash model as the backend, the authors report segmentation performance competitive with SAM 3 across referring, reasoning, and open-vocabulary instance segmentation benchmarks, demonstrating the potential of general-purpose MLLMs to perform segmentation without task-specific segmentation architectures.

Critical strengths

The paper is well written and clearly presented, making it easy to read and follow. The authors demonstrate that frontier MLLMs, such as Gemini-3.7-Flash, can localize objects from free-form text queries and be systematically prompted to generate reasonably accurate geometric contours and iteratively refine them. The evaluation is comprehensive, covering a broad range of benchmarks, including referring, reasoning, and open-vocabulary segmentation. The proposed method also achieves competitive performance with SAM 3 on several benchmarks.

Critical weaknesses

However, the paper has several key weaknesses:

The training-free claim is overstated, and the technical novelty is limited. The method is presented as a training-free segmentation pipeline that relies solely on a general-purpose MLLM as its vision component. In my view, this framing is too strong. The authors conduct their primary and most extensive evaluations using Gemini-3.7-Flash, a model from a family that Google explicitly describes as having object detection and segmentation capabilities in its official documentation (https://ai.google.dev/gemini-api/docs/image-understanding). While the authors do not perform additional model training, the pipeline relies heavily on localization and segmentation capabilities already present in the underlying MLLM. Consequently, the results are less surprising than the paper’s framing suggests, and the distinction between a training-free pipeline and segmentation capabilities likely acquired during the backend’s pretraining or instruction tuning deserves greater emphasis.

Furthermore, the idea of using language models to perform segmentation by generating polygons or other text-encoded mask representations has been explored in prior work, including VisionLLM and Text4Seg (both cited by the authors), as well as PolyFormer and LLaFS (not cited). As a result, the technical novelty of the proposed approach appears limited, with the primary contribution lying in the prompting strategy and iterative refinement procedure. The paper would benefit from a clearer discussion of how these contributions differ from prior approaches, and of how much of the observed performance can be attributed to the proposed pipeline rather than to the existing capabilities of the underlying MLLM.

Segmentation performance is highly dependent on the underlying MLLM. Related to the previous point, the effectiveness of the proposed approach appears to depend heavily on the underlying MLLM. All benchmarks except ReasonSeg are evaluated with a single proprietary frontier model, Gemini-3.7-Flash. Although the authors evaluate 21 other MLLM backends, these experiments are limited to the 200-item ReasonSeg validation set. The results show substantial variation, with final gIoU ranging from 14.2 to 72.9 (the latter achieved by GPT-6 Astra), and the refinement stage lowers performance on 11 of the 21 backends. This suggests that the method’s effectiveness largely depends on the pretrained localization and geometric reasoning capabilities of the backbone rather than on the pipeline itself. Furthermore, with the same open-weight Gemma-4-31B backend, the SAM 3 Agent baseline substantially outperforms LLMSeg on ReasonSeg val (73.1 vs. 56.2 gIoU; Tables 7 and 11), raising concerns about whether the approach remains competitive without a frontier proprietary backend.

High inference cost The inference cost is also considerably high. As shown in Table 16, LLMSeg takes approximately 153 seconds per image with Gemini-3.7-Flash, compared to just 11 seconds for SAM 3 Agent using the same LLM backbone. Its estimated serving cost is also nearly twice as high ($15.6 vs. $8.3 per 1,000 images). Inference is even slower with open-weight backends, taking approximately 1,900–2,150 seconds per image with Gemma 4. These substantial computational costs raise serious concerns about the practical applicability of the proposed approach.

Additional Experiments Needed to Better Isolate the Proposed Contributions. In my opinion, the paper would greatly benefit from additional experiments that could better isolate the contributions of the proposed approach:

(a) Evaluation on smaller open-source models: The authors do not evaluate smaller open-source models (4–13B parameters) that can run on modest GPUs, or models such as LLaVA-1.5 and Llama-3.2 Vision, which have not been explicitly trained for object localization and bounding box prediction. Such experiments would help address my concerns about whether the pipeline's performance largely stems from the specialized training of the underlying MLLMs.

(b) Comparison with SAM using Stage 1 bounding boxes: A natural baseline would be to feed the bounding boxes predicted in Stage 1 of LLMSeg into SAM or SAM 3 as box prompts. This would help assess the effectiveness of the spline tracing in Stage 2 compared to SAM-based segmentation.

Omission of Relevant Prior Work (Minor) Although not a major weakness, the paper overlooks several relevant prior works on LLM-based polygon segmentation, such as PolyFormer (Liu et al., CVPR 2023) and LLaFS (Zhu et al., CVPR 2024). It also omits training-free open-vocabulary segmentation methods that leverage VLM attention maps, including PnP-OVSS (Luo et al., CVPR 2024), NACLIP (Hajimiri et al., WACV 2025), and Segmentation From Attention (Hossain et al., TMLR 2026).

Questions for authors

I would request the authors to look into the weakness section. I would appreciate if the authors could answer the following questions:

  1. Does the performance of the approach depends on whether the models have been explicitly trained to predict bounding boxes and geometry? I would greatly appreciate if the authors could evaluate their approach on MLLMs that are not trained to ground phrases and report results on at least one referring and one open-vocabulary benchmark.

  2. How does the proposed approach scale to smaller models?

Reviewer_7pjbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces LLMSeg, a method that use VLMs zero-shot for open-vocabulary segmentation and reference segmentation. The method directly let VLMs predict bounding box coordinates and control points, use code to extract B-SPLINES and get the mask, overlap the mask with input images, and feed back to VLMs for refinement. It is evaluated on ReasonSeg, RefCOCO, RefCOCO+, and RefCOCOg for referral segmentation, ADE20K-150 and Cityscapes for semantic segmentation, and ODinW13 and RF-100VL for detection. Compared to SAM3 with the same VLM backbone, it performs better on ReasonSeg, on-par on RefCOCO, and worse on RefCOCO and RefCOCOg. It generally outperforms SAM3 for open-vocabulary semantic segmentation and detection. Analysis about different backbones and proposed components are provided.

Critical strengths
  1. The paper is well presented. The figures help to clarify the paper quickly, and the writing is very easy to follow.
  2. The experiment is comprehensive. It evaluated many different tasks, across various VLM backbones, and provides ablation study and analysis of different backbones.
  3. The final general performance is on-par to the best method (SAM3 baseline) on referral segmentation and the best on open-vocabulary semantic segmentation and detection.
Critical weaknesses
  1. The contribution is mainly experimental. It is good to see that VLMs are able to do dense segmentation without external learning based methods for decent performance. However, this is also not very surprising, given that the VLMs have seen many tasks of detection and its great support for coding.
  2. Method wise, there is clear limitations to really apply this method for segmentation. It will not be as sharp as learning based methods, and its speed will be slow.
Questions for authors

I do not have more questions for the authors. It is a solid paper, but I am not sure whether it is significant enough for acceptance.

Reviewer_dAjSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a training-free method for text-prompted image segmentations, including grounding, open-vocabulary instance segmentation and zero-shot detection through using MLLM with the help of B-SPLINES. They conduct many experiments to show the validation of the proposed method and achieve comparable performance.

Critical strengths

The proposed method relies on a single MLLM rather than combining multiple existing foundation models, resulting in a more unified framework. It aims to fully exploit the rich knowledge and capabilities encoded in the pretrained MLLM to address multiple segmentation tasks. Moreover, the authors provide extensive experimental evaluations and detailed explanations to validate the effectiveness of the proposed approach.

Critical weaknesses

The main paper does not sufficiently justify the necessity and suitability of the proposed method. In particular, the motivation for introducing B-splines remains unclear. Why are B-splines chosen over other alternative representations, and what specific advantages do they offer for the targeted segmentation tasks? More importantly, the authors do not adequately explain why B-splines are particularly well suited to MLLM-based segmentation or how their mathematical properties align with the representations and capabilities of pretrained MLLMs. The authors should provide stronger theoretical insights and empirical evidence to justify this design choice and demonstrate that the integration of B-splines with MLLMs is well motivated rather than an empirical combination of existing techniques. In my opinion, a well-presented methodology should allow readers to understand the complete pipeline and the relationships between its key components directly from the main paper, without having to piece together information scattered across different sections or the appendix. I strongly encourage the authors to provide a more systematic and self-contained explanation of the proposed framework, clearly describing the inputs, outputs, and connections between consecutive stages.

Questions for authors
  1. The motivation for relying exclusively on a single pretrained MLLM to achieve training-free segmentation is not sufficiently justified. Although avoiding the combination of multiple foundation models could simplify the overall framework, the authors do not clearly explain why a single MLLM is sufficient or particularly suitable for segmentation tasks. While pretrained MLLMs have demonstrated strong semantic understanding capabilities, it remains unclear whether their internal representations contain sufficient spatial information to support accurate pixel-level segmentation. Moreover, the authors should clarify the specific limitations of existing training-free segmentation methods that combine multiple foundation models and explain how their proposed framework addresses these limitations. Without such justification, the decision to rely exclusively on a single MLLM appears to be more of an architectural preference than a well-motivated methodological choice
  2. The motivation for adopting B-splines as the geometric representation is not sufficiently justified. Although B-splines provide a compact and smooth contour representation, it remains unclear why they are particularly suitable for MLLM-based segmentation compared with simpler alternatives such as polygons. In particular, the authors do not adequately explain how the properties of B-splines align with the inherent capabilities of pretrained MLLMs or why an MLLM can reliably generate the corresponding control points without task-specific training. Moreover, the experimental results in Table 4 show that polygon-based representations achieve comparable or even slightly better segmentation performance than B-splines. Although the authors demonstrate that B-splines require fewer output coordinates, this primarily supports their representational compactness rather than their necessity or superiority for segmentation. Furthermore, the local control property of B-splines does not appear to be directly exploited in the refinement stage, which instead relies on adding or removing independently generated regions. Therefore, the authors should provide a stronger justification for choosing B-splines, clarify their specific advantages in the context of MLLM-based segmentation, and conduct more controlled comparisons with alternative contour representations to establish the necessity and effectiveness of this design choice.
  3. The presentation of the proposed method needs to be improved to enhance clarity and readability. Although the individual components are described in Section 3, their connections are not sufficiently explained in a clear and coherent manner. For example, the authors introduce instance detection, B-spline control polygon generation, and mask rasterization separately, but the complete process of associating the generated polygons with the detected instances and converting them into instance masks is not explicitly presented in a unified description. In particular, it would be helpful to clarify how the correspondence between instance identities, bounding boxes, and generated control polygons is established and maintained, especially when multiple instances overlap.
  4. The proposed method relies heavily on iterative refinement to improve segmentation quality, which raises concerns about its inference efficiency and computational complexity. According to the methodology, each refinement iteration requires up to two additional MLLM calls, potentially resulting in 18 calls per image under the default setting of eight refinement passes. Although the authors provide analyses of the number of refinement passes and MLLM calls, the actual inference latency and computational cost remain insufficiently investigated. In particular, repeatedly invoking a large MLLM may introduce substantial inference overhead, even when only a single foundation model is employed. Moreover, while the proposed batching strategy reduces the number of calls required for multiple instances, the computational cost of each call may still increase with the number and complexity of the detected instances.I suggest that the authors provide a more comprehensive efficiency analysis, including average inference time per image, token consumption, and computational or API costs, together with comparisons against SAM 3 and SAM 3 Agent under consistent evaluation settings. It would also be valuable to investigate the trade-off between segmentation accuracy and inference cost across different numbers of refinement iterations. Such analyses are necessary to clarify the practical advantages and scalability of the proposed training-free framework.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.