- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors investigate whether a general-purpose multimodal large language model (MLLM) can perform text-prompted image segmentation without relying on a dedicated segmentation head or a pixel-level decoder trained for segmentation. To this end, they propose a three-stage pipeline consisting of a series of MLLM calls. In the first stage, the MLLM is prompted with an input image and a text query to localize all matching object instances, producing bounding boxes along with brief descriptions of their morphology and topology. In the second stage, the model traces the boundaries of each identified instance by predicting control points for closed, periodic B-splines of degree 1 or 3. These splines are then densely sampled and rasterized into binary segmentation masks using the even–odd polygon filling rule (Foley et al., 1996). In the third stage, the predicted masks and their contours are overlaid on the original image and fed back into the MLLM for iterative refinement. At each iteration, the model identifies regions with potential segmentation errors, proposes localized focus boxes, and retraces the spline contours within those regions. This process continues until the masks stabilize or a predefined iteration or computational budget is exhausted. Using an off-the-shelf Gemini-3.7-Flash model as the backend, the authors report segmentation performance competitive with SAM 3 across referring, reasoning, and open-vocabulary instance segmentation benchmarks, demonstrating the potential of general-purpose MLLMs to perform segmentation without task-specific segmentation architectures.
Critical strengths
The paper is well written and clearly presented, making it easy to read and follow. The authors demonstrate that frontier MLLMs, such as Gemini-3.7-Flash, can localize objects from free-form text queries and be systematically prompted to generate reasonably accurate geometric contours and iteratively refine them. The evaluation is comprehensive, covering a broad range of benchmarks, including referring, reasoning, and open-vocabulary segmentation. The proposed method also achieves competitive performance with SAM 3 on several benchmarks.
Critical weaknesses
However, the paper has several key weaknesses:
The training-free claim is overstated, and the technical novelty is limited. The method is presented as a training-free segmentation pipeline that relies solely on a general-purpose MLLM as its vision component. In my view, this framing is too strong. The authors conduct their primary and most extensive evaluations using Gemini-3.7-Flash, a model from a family that Google explicitly describes as having object detection and segmentation capabilities in its official documentation (https://ai.google.dev/gemini-api/docs/image-understanding). While the authors do not perform additional model training, the pipeline relies heavily on localization and segmentation capabilities already present in the underlying MLLM. Consequently, the results are less surprising than the paper’s framing suggests, and the distinction between a training-free pipeline and segmentation capabilities likely acquired during the backend’s pretraining or instruction tuning deserves greater emphasis.
Furthermore, the idea of using language models to perform segmentation by generating polygons or other text-encoded mask representations has been explored in prior work, including VisionLLM and Text4Seg (both cited by the authors), as well as PolyFormer and LLaFS (not cited). As a result, the technical novelty of the proposed approach appears limited, with the primary contribution lying in the prompting strategy and iterative refinement procedure. The paper would benefit from a clearer discussion of how these contributions differ from prior approaches, and of how much of the observed performance can be attributed to the proposed pipeline rather than to the existing capabilities of the underlying MLLM.
Segmentation performance is highly dependent on the underlying MLLM. Related to the previous point, the effectiveness of the proposed approach appears to depend heavily on the underlying MLLM. All benchmarks except ReasonSeg are evaluated with a single proprietary frontier model, Gemini-3.7-Flash. Although the authors evaluate 21 other MLLM backends, these experiments are limited to the 200-item ReasonSeg validation set. The results show substantial variation, with final gIoU ranging from 14.2 to 72.9 (the latter achieved by GPT-6 Astra), and the refinement stage lowers performance on 11 of the 21 backends. This suggests that the method’s effectiveness largely depends on the pretrained localization and geometric reasoning capabilities of the backbone rather than on the pipeline itself. Furthermore, with the same open-weight Gemma-4-31B backend, the SAM 3 Agent baseline substantially outperforms LLMSeg on ReasonSeg val (73.1 vs. 56.2 gIoU; Tables 7 and 11), raising concerns about whether the approach remains competitive without a frontier proprietary backend.
High inference cost The inference cost is also considerably high. As shown in Table 16, LLMSeg takes approximately 153 seconds per image with Gemini-3.7-Flash, compared to just 11 seconds for SAM 3 Agent using the same LLM backbone. Its estimated serving cost is also nearly twice as high ($15.6 vs. $8.3 per 1,000 images). Inference is even slower with open-weight backends, taking approximately 1,900–2,150 seconds per image with Gemma 4. These substantial computational costs raise serious concerns about the practical applicability of the proposed approach.
Additional Experiments Needed to Better Isolate the Proposed Contributions. In my opinion, the paper would greatly benefit from additional experiments that could better isolate the contributions of the proposed approach:
(a) Evaluation on smaller open-source models: The authors do not evaluate smaller open-source models (4–13B parameters) that can run on modest GPUs, or models such as LLaVA-1.5 and Llama-3.2 Vision, which have not been explicitly trained for object localization and bounding box prediction. Such experiments would help address my concerns about whether the pipeline's performance largely stems from the specialized training of the underlying MLLMs.
(b) Comparison with SAM using Stage 1 bounding boxes: A natural baseline would be to feed the bounding boxes predicted in Stage 1 of LLMSeg into SAM or SAM 3 as box prompts. This would help assess the effectiveness of the spline tracing in Stage 2 compared to SAM-based segmentation.
Omission of Relevant Prior Work (Minor) Although not a major weakness, the paper overlooks several relevant prior works on LLM-based polygon segmentation, such as PolyFormer (Liu et al., CVPR 2023) and LLaFS (Zhu et al., CVPR 2024). It also omits training-free open-vocabulary segmentation methods that leverage VLM attention maps, including PnP-OVSS (Luo et al., CVPR 2024), NACLIP (Hajimiri et al., WACV 2025), and Segmentation From Attention (Hossain et al., TMLR 2026).
Questions for authors
I would request the authors to look into the weakness section. I would appreciate if the authors could answer the following questions:
-
Does the performance of the approach depends on whether the models have been explicitly trained to predict bounding boxes and geometry? I would greatly appreciate if the authors could evaluate their approach on MLLMs that are not trained to ground phrases and report results on at least one referring and one open-vocabulary benchmark.
-
How does the proposed approach scale to smaller models?