- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Tarot-SAM3, a training-free referring expression segmentation framework using frozen MLLM, SAM3, and DINOv3 components. Its ERI module interprets referring expressions and prepares text and bounding-box prompts, while MSR combines mask selection with point-prompt refinement. The framework aims to improve grounding for both explicit and implicit expressions without training the underlying models. Experiments show substantial improvements over recent training-free methods: with the same Qwen2.5-VL 7B backbone, RefCOCO testA cIoU increases from 73.7% to 78.5%, and ReasonSeg test cIoU increases from 65.0% to 71.2% relative to EVOL-SAM3.
Critical strengths
The paper studies a relevant limitation of training-free segmentation systems. Both interpreting a complex referring expression and selecting the correct segmentation mask can be difficult, especially when the target is specified indirectly. Combining structured interpretation with multiple prompt types is a reasonable way to address these difficulties.
The experimental results are encouraging. The improvements on RefCOCO and ReasonSeg are substantial under the reported same-backbone comparison, and the paper considers different MLLM sizes rather than relying on one model configuration.
The authors also provide useful ablations of ERI, MSR, reasoning-prompt options, and mask-selection strategies. These results suggest that several parts of the framework contribute to performance, although they do not completely resolve the contribution of each step under a fixed inference budget.
Critical weaknesses
-
The contribution of the multi-stage reasoning process is not fully separated from its inference cost.
Tarot-SAM3 combines structured MLLM reasoning, multiple prompt types, repeated mask selection, DINOv3 features, and optional iterative refinement. However, the paper does not report a complete end-to-end latency or average MLLM/SAM3 call-count comparison with SAM3 Agent and EVOL-SAM3. This is important for understanding how much additional inference is required to obtain the reported gains. In Table 3, adding OPM after IPS improves cIoU by 1.1 points on RefCOCO testA and 0.3 points on ReasonSeg test. These improvements may still be useful, but their computational cost is not shown. Although the paper includes several reasoning and selection ablations, a comparison with simpler structured prompting or selection rules using the same candidates and inference budget would better establish the necessity of the full procedure. Since the competing methods also use iterative inference, the current evidence does not establish that Tarot-SAM3 is slower than them.
-
The mask-selection and refinement procedure is not described consistently enough for reproduction.
The main text specifies a three-round MLLM selection procedure over shuffled text, bounding-box, and point-mask candidates, accepting a result only when all three selections agree (Eq. 10). However, Appendix Algorithm 5 begins with a single
MLLMbest(MText, MBBox)call, without showing the point candidate or the repeated-consensus rule. There is also a difference between the exclusive mask-discrepancy regions defined in Eq. (11) and the intersection-based construction ofABestin Algorithm 5, line 14. These differences may be simplifications of the pseudocode, rather than problems with the actual implementation. Nevertheless, the paper should clarify the exact sequence of candidate selection, discrepancy calculation, and positive/negative point updates so that the reported procedure can be reproduced reliably.
Overall, the method achieves meaningful accuracy improvements and includes substantial experimental analysis. My concerns are mainly about whether the full multi-stage procedure provides a clear advantage at a comparable computational budget and whether the implementation can be reproduced from the current description. A consistent algorithm specification and matched-budget analysis would make the contribution more convincing.
Questions for authors
- Could the authors provide end-to-end latency and average MLLM/SAM3 call counts for Tarot-SAM3 and the iterative baselines under comparable conditions, including the additional cost of OPM?
- How do RPO and IPS compare with simpler prompting or candidate-selection rules when using the same candidates and inference budget? How often does IPS obtain unanimous agreement rather than falling back to the text mask?
- Could the authors reconcile Eq. (10) and Eq. (11) with Appendix Algorithm 5, especially the two-candidate selection in line 1 and the construction of
ABestin line 14? Were prompt designs and hyperparameters fixed without consulting final test labels? - How does ERI handle ambiguous target granularity? In Figure 1, a skate and its blade may both be plausible interpretations of the expression, while Figure 17(B) shows a different difficulty: the requested unusually colored region is expanded into larger leaf areas. The authors already acknowledge the latter as a failure case. Could they clarify how the method distinguishes alternative valid target interpretations from object-versus-attribute-region grounding errors, and provide a more systematic evaluation?
7 comments
Follow-up Response to Reviewer J2Mp
W1.3 (also Q2) — Comparisons under Comparable Inference Budgets — Effectiveness Beyond Additional Model Calls
Q: A comparison with simpler structured prompting or selection rules using the same candidates and inference budget would better establish the necessity of the full procedure.
A: To distinguish the benefits of our methodological design from those of simply increasing inference computation, we conducted additional experiments comparing stronger baseline inference budgets, simpler prompt construction strategies, and alternative mask selection rules. The results are reported in Tables 6, 17, and 18, with further analysis in Appendix C.4.9–C.4.10.
(1) Additional inference calls alone do not close the performance gap. Increasing the MLLM or SAM3 budgets of the baselines does not consistently improve their performance. For example, on ReasonSeg, EVOL-SAM3 with 20.43 SAM3 calls, exceeding our 19.10, achieves 71.1/63.1, compared with our 75.8/71.2. These results further suggest that simply increasing MLLM calls or candidate masks can introduce additional selection ambiguity and inconsistent judgments in existing methods, potentially degrading their performance rather than improving it. In contrast, our structured reasoning and consensus-based selection make more effective use of the adequate inference budget.
(2) Simpler prompt construction strategies (Table 17). We compare our ERI with direct multi-prompt generation, simple structured prompting, and their combinations with RPO.
Table 17: Comparison of different prompt construction strategies. Results are reported as gIoU/cIoU (%). Paired MLLM call counts correspond to RefCOCO / ReasonSeg.
| Prompting Strategy | MLLM Calls | RefCOCO | ReasonSeg |
|---|---|---|---|
| Direct Multi-Prompt | 2/2 | 63.3 / 59.9 | 47.7 / 47.2 |
| Simple Structured Prompting | 4/4 | 64.4 / 57.8 | 56.1 / 51.6 |
| RPO + Direct Multi-Prompt | 12.14 / 7.74 | 72.1 / 65.6 | 65.4 / 59.7 |
| RPO + Structured Prompting | 14.32 / 9.65 | 64.8 / 57.9 | 61.5 / 56.0 |
| Our ERI | 13.97 / 9.02 | 77.8 / 74.2 | 72.1 / 65.7 |
Effectiveness of our prompting strategy. Compared with Direct Multi-Prompt, ERI improves gIoU/cIoU by +14.5/+14.3 points on RefCOCO and +24.4/+18.5 points on ReasonSeg. To ensure a more rigorous comparison, we additionally incorporate RPO into the baseline prompting strategies, providing them with the key target-specific and relational cues identified by our framework. Notably, ERI still outperforms RPO + Structured Prompting by +13.0/+16.3 and +10.6/+9.7 points, respectively, while requiring fewer MLLM calls (13.97 vs. 14.32 and 9.02 vs. 9.65). These results demonstrate that our advantage lies not merely in providing informative cues or increasing inference calls, but in how our prompting strategy organizes and utilizes them.
(3) Simpler mask selection strategies (Table 18). We further compare IPS against alternative selection rules using the same candidate mask types, including deterministic selection, consensus voting, single MLLM selection, iterative selection, and pairwise comparisons.
Table 18: Comparison of mask selection rules on RefCOCO TestA. Results are reported as gIoU/cIoU (%).
| Selection Rule | MLLM Calls | RefCOCO (TestA) |
|---|---|---|
| Always Text | 0 | 66.4 / 61.9 |
| Always BBox | 0 | 74.9 / 70.8 |
| Always Point | 0 | 63.4 / 61.1 |
| Consensus Voting | 0 | 77.8 / 74.5 |
| Single Selection | 1 | 70.9 / 66.5 |
| Single Selection (Structured) | 1 | 71.3 / 66.0 |
| Single Selection (Iteration) | 4.61 | 72.3 / 64.9 |
| Double-beats | 2.85 | 73.8 / 71.5 |
| IPS (Ours) | 3 | 79.6 / 77.4 |
Effectiveness of our mask selection strategy. To isolate the contribution of IPS, we compare different selection strategies using the same Text, BBox, and Point mask candidates. These include MLLM-free strategies (direct mask selection and Consensus Voting), single-round MLLM selection with or without structured instructions, iterative selection that continues until two consecutive judgments agree, and Double-beats, which performs repeated pairwise comparisons with tie-breaking rounds. Unlike these alternatives, IPS evaluates the candidates under three independently shuffled orders, accepts only unanimous selections, and conservatively falls back to the Text mask when judgments disagree. As shown in Table 18, IPS achieves the best performance (79.6/77.4 gIoU/cIoU) with only 3 MLLM calls, outperforming Double-beats by +5.8/+5.9 points under a comparable inference budget (3.00 vs. 2.85 calls). It also surpasses iterative selection by +7.3/+12.5 points, despite requiring fewer calls (3.00 vs. 4.61). These results demonstrate that repeated MLLM evaluations alone do not guarantee better mask selection; instead, IPS improves selection reliability through order-robust consensus and conservative fallback.
Follow-up Response to Reviewer J2Mp
W2 — Consistency of Mask Selection and Refinement — Clarification of Algorithm 5
Q: Algorithm 5 does not explicitly include the three-round consensus selection in Eq. (10), and its discrepancy-region construction appears inconsistent with Eq. (11).
A: We thank the reviewer for carefully identifying these inconsistencies and apologize for the oversimplified pseudocode in Algorithm 5. We clarify that the actual implementation follows Eqs. (10) and (11), while Algorithm 5 omits certain intermediate operations. We have revised Alg. 5 in the Appendix.
Specifically, we have clarified the following three steps in the revised Algorithm 5:
-
Mask selection. The MLLM evaluates Text, BBox, and Point candidates across three shuffled rounds. A candidate is accepted only when all three selections agree; otherwise, the Text mask is retained.
-
Discrepancy analysis. We compute the exclusive regions and , following Eq. (11).
-
Point-prompt updates. The revised pseudocode explicitly specifies how positive and negative point prompts are generated and updated to guide SAM3 in correcting residual segmentation errors.
Furthermore, upon acceptance, we will publicly release our implementation to facilitate reproducibility and encourage the community to explore and extend our mask refinement mechanism in related segmentation tasks.
Q2 — IPS Consensus and Fallback Frequency — Quantitative Analysis of Three-Round Agreement
Q: How often does IPS obtain unanimous agreement rather than falling back to the Text mask?
A: We thank the reviewer for this question. We conducted additional experiments to quantify the three-round consensus frequency and Text-mask fallback rate, with detailed results reported in Table 19 (Appendix C.4.11).
Table 19: Analysis of IPS consensus behavior and Point-mask contributions. Results are reported on RefCOCO TestA and ReasonSeg Test. The ΔgIoU/cIoU values indicate improvements relative to the corresponding Text-mask fallback baseline.
| Selection Outcome | RefCOCO TestA: Queries (%) | RefCOCO TestA: ΔgIoU/cIoU | ReasonSeg Test: Queries (%) | ReasonSeg Test: ΔgIoU/cIoU |
|---|---|---|---|---|
| Unanimous Text | 26.0 | 0.0 / 0.0 | 22.1 | 0.0 / 0.0 |
| Unanimous BBox | 14.8 | +35.5 / +25.0 | 4.3 | +7.4 / +11.9 |
| Unanimous Point | 14.1 | +3.8 / +3.0 | 0.9 | +5.2 / +14.6 |
| IPS (Fallback to Text) | 45.1 | +4.3 / +10.9 | 72.7 | +6.2 / +20.2 |
Consensus frequency. On RefCOCO TestA, the three shuffled evaluations achieve unanimous agreement in 54.9% of queries, comprising 26.0% Text, 14.8% BBox, and 14.1% Point selections, while the remaining 45.1% trigger the Text-mask fallback. On the more challenging ReasonSeg Test, unanimous agreement occurs in 27.3% of queries (22.1% Text, 4.3% BBox, and 0.9% Point), with 72.7% triggering fallback. These findings clarify how IPS balances reliable consensus-based selection with conservative fallback, particularly when handling complex and implicitly specified referring expressions.
Q3 — Validation-Based Configuration — Test-Set Independence and Reproducibility
Q: Were prompt designs and hyperparameters fixed without consulting final test labels?
A: We thank the reviewer for raising this important concern. We confirm that all prompt designs and hyperparameters were determined exclusively using validation splits, without consulting final test labels, and remained fixed throughout test evaluation. We have explicitly clarified this evaluation protocol in the revised main paper. Furthermore, to assess the stability of our validation-fixed configurations, we conducted three independent runs across four benchmarks, reporting mean ± standard deviation in Table 20 (Appendix C.4.12).
Table 20: Test-set performance with configurations fixed on validation splits. Results are reported as mean ± standard deviation of gIoU/cIoU (%) over repeated runs.
| Method | RefCOCO TestA | RefCOCO+ TestA | RefCOCOg Test(U) | ReasonSeg Test |
|---|---|---|---|---|
| SAM3 Agent | 68.55 ± 0.78 / 63.40 ± 1.27 | 61.50 ± 2.25 / 55.57 ± 1.25 | 61.50 ± 0.66 / 57.63 ± 1.11 | 64.17 ± 1.02 / 57.30 ± 3.39 |
| EVOL-SAM3 | 75.73 ± 0.79 / 72.70 ± 1.05 | 71.27 ± 0.93 / 66.19 ± 1.40 | 67.70 ± 0.46 / 64.73 ± 0.75 | 70.63 ± 0.38 / 64.57 ± 0.45 |
| Tarot-SAM3 | 80.00 ± 0.76 / 77.60 ± 0.93 | 75.20 ± 1.04 / 71.80 ± 1.19 | 72.97 ± 0.78 / 70.37 ± 0.97 | 75.77 ± 0.95 / 71.10 ± 0.66 |
The results show that Tarot-SAM3 consistently outperforms both SAM3 Agent and EVOL-SAM3 across all four benchmarks, with relatively small standard deviations, supporting the robustness and reproducibility of our reported performance.
Response to Reviewer J2Mp
W1.1 (also Q1) — End-to-End Inference Cost — Comprehensive Efficiency Comparison
Q: The paper does not report a complete end-to-end latency or average MLLM/SAM3 call-count comparison with SAM3 Agent and EVOL-SAM3.
A: We thank the reviewer for highlighting the importance of computational efficiency. To address this concern, we have added Table 6 and related analysis in the revised main paper.
Table 6: Performance and inference cost comparison across different computational budgets. Higher gIoU/cIoU indicates better segmentation performance, while lower inference costs are preferable. ΔOPM denotes the difference between Tarot-SAM3 with and without OPM.
Accuracy–efficiency trade-off. On RefCOCO, Tarot-SAM3 outperforms SAM3 Agent by +11.3/+14.2 gIoU/cIoU, while achieving 26.5% lower latency and consuming approximately 7.5× fewer tokens. On the more challenging ReasonSeg benchmark, our method surpasses EVOL-SAM3 by +4.9/+6.2 points, with comparable MLLM calls (9.94 vs. 9.47) and fewer tokens (36,153 vs. 64,906), albeit at higher latency (33.40s vs. 15.95s). We believe these substantial accuracy gains justify a meaningful and acceptable accuracy–efficiency trade-off, particularly for challenging reasoning-based segmentation tasks where accuracy is prioritized. This perspective is further supported by the positive assessments of our empirical performance from Reviewer TZ7v ("strong results") and Reviewer rNVs ("promising results").
W1.2 (also Q1) — OPM Contribution and Computational Overhead — Targeted Refinement at Modest Additional Cost
Q: Adding OPM after IPS improves cIoU by only 1.1 points on RefCOCO and 0.3 points on ReasonSeg, but its computational cost is not reported.
A: We agree that the contribution of OPM should be evaluated together with its computational overhead. We therefore explicitly report the incremental performance gains and inference costs introduced by OPM in Table 6. On RefCOCO, OPM improves gIoU/cIoU by +0.8/+1.1 points, requiring an additional 0.70s, 1.73 SAM3 calls, and 2,545 tokens. On ReasonSeg, it improves gIoU/cIoU by +0.5/+0.3 points, requiring an additional 1.17s, 0.22 MLLM calls, 0.66 SAM3 calls, and 1,074 tokens. Importantly, OPM introduces no additional peak GPU memory overhead on either benchmark. Although the average improvements are modest and introduce additional computational overhead, OPM addresses residual over- and under-segmentation errors that cannot be fully resolved by initial mask selection. Its adaptive point-prompt updates provide an additional opportunity to correct inaccurate mask boundaries of existing methods. Importantly, the reusability and transferability of our design were also acknowledged by Reviewer TZ7v ("components are reusable") and Reviewer rNVs ("the mask proposal, selection, and refinement algorithms are portable and potentially beneficial to related methods"). These observations further highlight OPM's potential value beyond its incremental performance gains within Tarot-SAM3.