- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies open ad-hoc segmentation, where a target region is defined through combinations of parts, relations, and exclusions rather than a familiar category name. It proposes VASA, a training-free framework combining a vision-language model with SAM3 and a visual execution harness. The framework maintains a persistent working mask that the agent can inspect and modify through addition, removal, and replacement operations. Planning, execution constraints, visual scrutiny, and error recovery support this iterative construction process. The authors also introduce PARS, a benchmark derived from PartImageNet annotations with detailed textual definitions of target regions. Experiments across six vision-language models, together with evaluations on RefCOCOm and ReasonSeg, report substantial improvements on fine-grained segmentation and competitive performance on conventional reasoning segmentation.
Critical strengths
- The persistent working mask provides a practical mechanism for constructing targets through multiple segmentation operations while preserving intermediate progress. This directly addresses a limitation of repeatedly searching for a single prompt that retrieves the entire target.
- Improvements across six vision-language models and on RefCOCOm support the usefulness of the framework beyond one backbone or the proposed benchmark. The RefCOCOm results are particularly consequential because they use the dataset’s original expressions.
Critical weaknesses
- For Qwen3-VL 32B Thinking, Table 1 reports SAM3 Agent’s overall scores as 46.4/48.1/48.0, which Table 4 identifies as short-query results, whereas VASA’s corresponding scores match its long-query results. The paper should clarify whether the main comparison uses identical queries or method-specific query settings, as this affects the interpretation and attribution of the reported gains.
- PARS focuses primarily on body-region definitions derived from existing PartImageNet annotations, limiting the evidence for generalization to broader open-ended compositions and relations. Additionally, the annotated examples used to generate its descriptions overlap the evaluation set, leaving the effect of this overlap on the reported performance unresolved.
Questions for authors
- Which query settings were used for each method in Table 1, and why do the Qwen3-VL 32B Thinking results for SAM3 Agent match the short-query row of Table 4? Does VASA retain its advantage when both methods receive identical queries?
- How many evaluation images were used as annotated demonstrations for description generation? Can you report the existing results separately for demonstration images and the remaining evaluation images to assess whether performance differs materially between these groups?
- Can you provide an ablation that removes persistent mask state from the complete framework while retaining the other harness components where feasible and using a comparable computational budget?
4 comments
Rebuttal (2/2)
[Q2] How many evaluation images were used as annotated demonstrations for description generation? Can you report the existing results separately for demonstration images and the remaining evaluation images to assess whether performance differs materially between these groups?
We first note that the annotated demonstrations are used to generate class-level descriptions, not as inputs to the segmentation agent during evaluation. Nevertheless, we recognize the concern that these descriptions may fit the demonstration images better than the remaining images. For each open ad-hoc concept in PARS, we use 5 unique images out of 100–150 images as annotated demonstrations. As requested, we report the Table 1 results separately for these two groups below, with Qwen3-VL 32B Thinking as the VLM. The results show little performance difference between demonstration images and the remaining images, with VASA’s gains persisting in both groups.
| Evaluation group | SAM3 Agent gIoU | VASA gIoU | Gain |
|---|---|---|---|
| Demonstration images | 47.0 | 57.7 | +10.7 |
| Remaining images | 46.3 | 56.9 | +10.6 |
We also validate PARS’s text queries by randomly sampling 30 examples covering 30 different classes and asking a third-party annotator to draw a segmentation mask from each image and its long-form description. Comparing these masks with the ground truth assesses how well the descriptions convey the intended open ad-hoc concepts. The masks achieved 75.6 gIoU, 76.6 cIoU, and 12.9 xIoU, supporting that PARS’s text queries communicate the intended target regions.
VASA’s gains also generalize beyond PARS and its query-generation procedure. VASA outperforms SAM3 Agent across all three splits of RefCOCOm, which uses human-written referring expressions, and remains competitive on ReasonSeg. These results provide complementary evidence across datasets and query sources.
R3. Ablation of persistent mask state
[Q3] Can you provide an ablation that removes persistent mask state from the complete framework while retaining the other harness components where feasible and using a comparable computational budget?
We use cumulative ablation because the components are designed around a shared working mask and persistent states. Removing that state would require redesigning the planning, scrutiny, and recovery instructions that depend on it. Retaining those instructions without their underlying state would not preserve their functionality. Table 3 therefore evaluates how these components progressively enable effective visual construction.
VASA’s success comes from coordinating the working-mask mechanism with the other harness components. Expanding an agent’s action space without adequate guidance introduces more ways to fail. Adding a working mask alone to SAM3 Agent reduces gIoU from 46.4 to 40.0, whereas the complete harness achieves 56.9. The working-mask mechanism alone is not enough. With the complete harness, VASA substantially outperforms the baselines, combining the flexibility of working-mask operations with the reliability of guided execution. This cumulative analysis supports their coordinated operation, although it does not isolate the contribution of persistent state through a leave-one-out comparison.
Both methods already receive the same maximum interaction budget of 20 rounds. SAM3 Agent often stops early after unsuccessful prompt refinement, leaving much of its budget unused. In contrast, VASA uses the available budget effectively through working-mask updates, visual scrutiny, and error recovery, enabling continued progress toward an accurate segmentation.
Dear Reviewer Dpnj,
As the discussion period ending soon, we would like to thank you again for your time and energy you have devoted to our paper so far. We will continue monitoring this thread and respond promptly to any additional questions or concerns. We look forward to hearing from you.
Best regards,
The Authors
Thank you for clarifying the query settings in Table 1 and showing that the demonstration overlap in PARS does not materially affect the results. However, the ablation still does not isolate the persistent working mask from the other harness components, which leaves the source of the improvement unclear since adding the working mask alone lowers gIoU from 46.4 to 40.0 in Table 3. Because PARS also remains limited to the PartImageNet "body" category and Table 1 still compares methods under different query forms, I will keep my original rating.
Rebuttal (1/2)
Dear Reviewer Dpnj,
We appreciate your recognition of 1) the practical value of VASA’s harness design built around a persistent working mask, and 2) VASA's effectiveness across six VLMs and multiple benchmarks. We provide a detailed response to your concerns below. We will incorporate these clarifications into the revised manuscript.
R1. Clarification on the query form
We report the query form used for each method in the table below. We use short queries for methods designed to accept short noun phrases, such as OVSeg, Semantic-SAM, and SAM3. The only exception is SAM3 Agent, where we also use short queries to compare against its stronger performance, as long queries considerably reduce its accuracy.
Table 1 compares the two methods under their respective best-performing query settings, while Table 4 compares them under identical query conditions:
VASA outperforms SAM3 Agent under both matched query conditions. The larger long-query gap largely reflects SAM3 Agent’s performance drop. Table 1 therefore uses SAM3 Agent’s stronger short-query result, yielding a 10.5-point gain. Although this comparison uses different query forms, Table 4 confirms that VASA’s advantage persists with identical inputs.
VASA handles customized target queries that SAM3 Agent struggles with. Such queries reflect how users specify targets in real-world open ad-hoc segmentation. Our long-query experiments capture this setting. SAM3 Agent scores 22.1 gIoU points lower with detailed descriptions than with short category names, while VASA scores 5.9 points higher.
R2. PARS scope and description-generation overlap
Current evaluations do include cross-object relations and various compositional targets. Cross-object relationships are already represented in our RefCOCOm, such as "torso of the guy on the right" (Figure 8), and VASA outperforms SAM3 Agent on this dataset. PARS includes different body parts for different object classes, which are unconventional and unnamable, such as "segment all cylindrical engine parts, including the propellers if present, located under the wings, excluding the wings…" (Figure 4), and "Annotate the entire animal from the neck down, excluding all limbs and the head, which is the part with the face, eyes, nose, mouth, and the hair on the head" (Figure 9). We therefore construct PARS in an efficient way that ensures broad coverage without needing labor-intensive manual mask annotations.
Our work introduces a practical, understudied task and an effective solution, with PARS providing an initial benchmark comparable in size to ReasonSeg. Together with evaluations on RefCOCOm and ReasonSeg, PARS demonstrates VASA’s effectiveness and establishes the foundation for further research. Further expanding its scale and coverage is valuable but requires substantial annotation and validation effort. We leave this important direction to future work.