acceptodds
Under review as a conference paper at ICLR 2027

Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentation

11 views

Abstract

Referring Expression Segmentation (RES) bridges vision and language but traditionally relies on extensive annotated datasets and struggles with implicit queries. While the Segment Anything Model 3 (SAM3) excels in promptable segmentation, its application to RES exposes **two critical challenges**: 1) SAM3 lacks the intrinsic capacity to decode lengthy, implicit expressions, and 2) it is highly vulnerable to both erroneous grounding priors and biased evaluation preferences from Multimodal Large Language Models (MLLMs), leaving its mask predictions unrefined. To address this, we introduce **Tarot-SAM3**, a novel training-free framework driven by the **key idea** of dual-stage refinement over MLLM-derived grounding cues and SAM3 mask predictions. Our **technical contribution** is two-fold. First, an Expression Reasoning Interpreter (**ERI**) is designed to guide the MLLM through structured parsing and evaluation-aware rephrasing, translating arbitrary queries into error-tolerant, heterogeneous prompts for SAM3 mask generation and performing consensus-driven mask selection within each prompt type. Second, the Mask Self-Refining (**MSR**) phase establishes a DINOv3-guided feedback loop, which identifies the most reliable mask and adaptively reconfigures point prompts to automatically resolve over- and under-segmentation. Extensive experiments demonstrate that Tarot-SAM3 achieves state-of-the-art results among MLLM-SAM3 architectures across MLLM scales, while under the Qwen2.5-VL 7B setting, it delivers **21.3%** and **28.2%** average relative improvements over the SAM3 Agent on explicit and implicit benchmarks, respectively. Open-world evaluations confirm our framework's superior generalization and robustness.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Segment Anything Model 3, Training-free, Referring Expression Segmentation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_J2MpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Tarot-SAM3, a training-free referring expression segmentation framework using frozen MLLM, SAM3, and DINOv3 components. Its ERI module interprets referring expressions and prepares text and bounding-box prompts, while MSR combines mask selection with point-prompt refinement. The framework aims to improve grounding for both explicit and implicit expressions without training the underlying models. Experiments show substantial improvements over recent training-free methods: with the same Qwen2.5-VL 7B backbone, RefCOCO testA cIoU increases from 73.7% to 78.5%, and ReasonSeg test cIoU increases from 65.0% to 71.2% relative to EVOL-SAM3.

Critical strengths

The paper studies a relevant limitation of training-free segmentation systems. Both interpreting a complex referring expression and selecting the correct segmentation mask can be difficult, especially when the target is specified indirectly. Combining structured interpretation with multiple prompt types is a reasonable way to address these difficulties.

The experimental results are encouraging. The improvements on RefCOCO and ReasonSeg are substantial under the reported same-backbone comparison, and the paper considers different MLLM sizes rather than relying on one model configuration.

The authors also provide useful ablations of ERI, MSR, reasoning-prompt options, and mask-selection strategies. These results suggest that several parts of the framework contribute to performance, although they do not completely resolve the contribution of each step under a fixed inference budget.

Critical weaknesses
  1. The contribution of the multi-stage reasoning process is not fully separated from its inference cost.

    Tarot-SAM3 combines structured MLLM reasoning, multiple prompt types, repeated mask selection, DINOv3 features, and optional iterative refinement. However, the paper does not report a complete end-to-end latency or average MLLM/SAM3 call-count comparison with SAM3 Agent and EVOL-SAM3. This is important for understanding how much additional inference is required to obtain the reported gains. In Table 3, adding OPM after IPS improves cIoU by 1.1 points on RefCOCO testA and 0.3 points on ReasonSeg test. These improvements may still be useful, but their computational cost is not shown. Although the paper includes several reasoning and selection ablations, a comparison with simpler structured prompting or selection rules using the same candidates and inference budget would better establish the necessity of the full procedure. Since the competing methods also use iterative inference, the current evidence does not establish that Tarot-SAM3 is slower than them.

  2. The mask-selection and refinement procedure is not described consistently enough for reproduction.

    The main text specifies a three-round MLLM selection procedure over shuffled text, bounding-box, and point-mask candidates, accepting a result only when all three selections agree (Eq. 10). However, Appendix Algorithm 5 begins with a single MLLMbest(MText, MBBox) call, without showing the point candidate or the repeated-consensus rule. There is also a difference between the exclusive mask-discrepancy regions defined in Eq. (11) and the intersection-based construction of ABest in Algorithm 5, line 14. These differences may be simplifications of the pseudocode, rather than problems with the actual implementation. Nevertheless, the paper should clarify the exact sequence of candidate selection, discrepancy calculation, and positive/negative point updates so that the reported procedure can be reproduced reliably.

Overall, the method achieves meaningful accuracy improvements and includes substantial experimental analysis. My concerns are mainly about whether the full multi-stage procedure provides a clear advantage at a comparable computational budget and whether the implementation can be reproduced from the current description. A consistent algorithm specification and matched-budget analysis would make the contribution more convincing.

Questions for authors
  1. Could the authors provide end-to-end latency and average MLLM/SAM3 call counts for Tarot-SAM3 and the iterative baselines under comparable conditions, including the additional cost of OPM?
  2. How do RPO and IPS compare with simpler prompting or candidate-selection rules when using the same candidates and inference budget? How often does IPS obtain unanimous agreement rather than falling back to the text mask?
  3. Could the authors reconcile Eq. (10) and Eq. (11) with Appendix Algorithm 5, especially the two-candidate selection in line 1 and the construction of ABest in line 14? Were prompt designs and hyperparameters fixed without consulting final test labels?
  4. How does ERI handle ambiguous target granularity? In Figure 1, a skate and its blade may both be plausible interpretations of the expression, while Figure 17(B) shows a different difficulty: the requested unusually colored region is expanded into larger leaf areas. The authors already acknowledge the latter as a failure case. Could they clarify how the method distinguishes alternative valid target interpretations from object-versus-attribute-region grounding errors, and provide a more systematic evaluation?
7 comments
AuthorsOpenReview ↗

Response to Reviewer J2Mp

W1.1 (also Q1) — End-to-End Inference Cost — Comprehensive Efficiency Comparison

Q: The paper does not report a complete end-to-end latency or average MLLM/SAM3 call-count comparison with SAM3 Agent and EVOL-SAM3.

A: We thank the reviewer for highlighting the importance of computational efficiency. To address this concern, we have added Table 6 and related analysis in the revised main paper.

Table 6: Performance and inference cost comparison across different computational budgets. Higher gIoU/cIoU indicates better segmentation performance, while lower inference costs are preferable. ΔOPM denotes the difference between Tarot-SAM3 with and without OPM.

DatasetMethodgIoU/cIoU ↑Latency (s) ↓MLLM Calls ↓DINOv3 Calls ↓SAM3 Calls ↓Tokens ↓Memory (GiB) ↓
SAM3 Agent (7B)69.1 / 64.316.826.5101.8196,3505.67
SAM3 Agent (7B, Raise MLLM)66.1 / 63.226.4814.9401.67231,6495.67
SAM3 Agent (7B, Raise SAM3)64.7 / 62.917.037.03012.74107,1205.67
RefCOCOEVOL-SAM3 (7B)76.3 / 73.75.638.7206.919,3584.21
EVOL-SAM3 (7B, Raise MLLM)74.8 / 71.88.2716.29011.9813,2274.21
EVOL-SAM3 (7B, Raise SAM3)75.2 / 72.07.6414.38015.5511,4744.21
Tarot-SAM3 (7B, w/o OPM)79.6 / 77.411.6613.171.0011.5910,2767.56
ΔOPM+0.8 / +1.1+0.70+2.67+0.00+1.73+2,5450.00
Tarot-SAM3 (7B)80.4 / 78.512.3615.841.0013.3212,8217.56
SAM3 Agent (7B)63.0 / 53.526.516.9101.4698,8235.65
SAM3 Agent (7B, Raise MLLM)64.2 / 57.534.849.3601.58134,6295.65
SAM3 Agent (7B, Raise SAM3)65.0 / 59.125.037.55017.71106,3325.65
ReasonSegEVOL-SAM3 (7B)70.9 / 65.015.959.4705.9264,9064.21
EVOL-SAM3 (7B, Raise MLLM)70.4 / 64.416.8210.9407.0771,2764.21
EVOL-SAM3 (7B, Raise SAM3)71.1 / 63.116.4513.90020.43114,8384.21
Tarot-SAM3 (7B, w/o OPM)75.3 / 70.932.239.720.6518.4435,0797.65
ΔOPM+0.5 / +0.3+1.17+0.22+0.00+0.66+1,074+0.00
Tarot-SAM3 (7B)75.8 / 71.233.409.940.6519.1036,1537.65

Accuracy–efficiency trade-off. On RefCOCO, Tarot-SAM3 outperforms SAM3 Agent by +11.3/+14.2 gIoU/cIoU, while achieving 26.5% lower latency and consuming approximately 7.5× fewer tokens. On the more challenging ReasonSeg benchmark, our method surpasses EVOL-SAM3 by +4.9/+6.2 points, with comparable MLLM calls (9.94 vs. 9.47) and fewer tokens (36,153 vs. 64,906), albeit at higher latency (33.40s vs. 15.95s). We believe these substantial accuracy gains justify a meaningful and acceptable accuracy–efficiency trade-off, particularly for challenging reasoning-based segmentation tasks where accuracy is prioritized. This perspective is further supported by the positive assessments of our empirical performance from Reviewer TZ7v ("strong results") and Reviewer rNVs ("promising results").


W1.2 (also Q1) — OPM Contribution and Computational Overhead — Targeted Refinement at Modest Additional Cost

Q: Adding OPM after IPS improves cIoU by only 1.1 points on RefCOCO and 0.3 points on ReasonSeg, but its computational cost is not reported.

A: We agree that the contribution of OPM should be evaluated together with its computational overhead. We therefore explicitly report the incremental performance gains and inference costs introduced by OPM in Table 6. On RefCOCO, OPM improves gIoU/cIoU by +0.8/+1.1 points, requiring an additional 0.70s, 1.73 SAM3 calls, and 2,545 tokens. On ReasonSeg, it improves gIoU/cIoU by +0.5/+0.3 points, requiring an additional 1.17s, 0.22 MLLM calls, 0.66 SAM3 calls, and 1,074 tokens. Importantly, OPM introduces no additional peak GPU memory overhead on either benchmark. Although the average improvements are modest and introduce additional computational overhead, OPM addresses residual over- and under-segmentation errors that cannot be fully resolved by initial mask selection. Its adaptive point-prompt updates provide an additional opportunity to correct inaccurate mask boundaries of existing methods. Importantly, the reusability and transferability of our design were also acknowledged by Reviewer TZ7v ("components are reusable") and Reviewer rNVs ("the mask proposal, selection, and refinement algorithms are portable and potentially beneficial to related methods"). These observations further highlight OPM's potential value beyond its incremental performance gains within Tarot-SAM3.

AuthorsOpenReview ↗

Follow-up Response to Reviewer J2Mp

W1.3 (also Q2) — Comparisons under Comparable Inference Budgets — Effectiveness Beyond Additional Model Calls

Q: A comparison with simpler structured prompting or selection rules using the same candidates and inference budget would better establish the necessity of the full procedure.

A: To distinguish the benefits of our methodological design from those of simply increasing inference computation, we conducted additional experiments comparing stronger baseline inference budgets, simpler prompt construction strategies, and alternative mask selection rules. The results are reported in Tables 6, 17, and 18, with further analysis in Appendix C.4.9–C.4.10.

(1) Additional inference calls alone do not close the performance gap. Increasing the MLLM or SAM3 budgets of the baselines does not consistently improve their performance. For example, on ReasonSeg, EVOL-SAM3 with 20.43 SAM3 calls, exceeding our 19.10, achieves 71.1/63.1, compared with our 75.8/71.2. These results further suggest that simply increasing MLLM calls or candidate masks can introduce additional selection ambiguity and inconsistent judgments in existing methods, potentially degrading their performance rather than improving it. In contrast, our structured reasoning and consensus-based selection make more effective use of the adequate inference budget.

(2) Simpler prompt construction strategies (Table 17). We compare our ERI with direct multi-prompt generation, simple structured prompting, and their combinations with RPO.

Table 17: Comparison of different prompt construction strategies. Results are reported as gIoU/cIoU (%). Paired MLLM call counts correspond to RefCOCO / ReasonSeg.

Prompting StrategyMLLM CallsRefCOCOReasonSeg
Direct Multi-Prompt2/263.3 / 59.947.7 / 47.2
Simple Structured Prompting4/464.4 / 57.856.1 / 51.6
RPO + Direct Multi-Prompt12.14 / 7.7472.1 / 65.665.4 / 59.7
RPO + Structured Prompting14.32 / 9.6564.8 / 57.961.5 / 56.0
Our ERI13.97 / 9.0277.8 / 74.272.1 / 65.7

Effectiveness of our prompting strategy. Compared with Direct Multi-Prompt, ERI improves gIoU/cIoU by +14.5/+14.3 points on RefCOCO and +24.4/+18.5 points on ReasonSeg. To ensure a more rigorous comparison, we additionally incorporate RPO into the baseline prompting strategies, providing them with the key target-specific and relational cues identified by our framework. Notably, ERI still outperforms RPO + Structured Prompting by +13.0/+16.3 and +10.6/+9.7 points, respectively, while requiring fewer MLLM calls (13.97 vs. 14.32 and 9.02 vs. 9.65). These results demonstrate that our advantage lies not merely in providing informative cues or increasing inference calls, but in how our prompting strategy organizes and utilizes them.

(3) Simpler mask selection strategies (Table 18). We further compare IPS against alternative selection rules using the same candidate mask types, including deterministic selection, consensus voting, single MLLM selection, iterative selection, and pairwise comparisons.

Table 18: Comparison of mask selection rules on RefCOCO TestA. Results are reported as gIoU/cIoU (%).

Selection RuleMLLM CallsRefCOCO (TestA)
Always Text066.4 / 61.9
Always BBox074.9 / 70.8
Always Point063.4 / 61.1
Consensus Voting077.8 / 74.5
Single Selection170.9 / 66.5
Single Selection (Structured)171.3 / 66.0
Single Selection (Iteration)4.6172.3 / 64.9
Double-beats2.8573.8 / 71.5
IPS (Ours)379.6 / 77.4

Effectiveness of our mask selection strategy. To isolate the contribution of IPS, we compare different selection strategies using the same Text, BBox, and Point mask candidates. These include MLLM-free strategies (direct mask selection and Consensus Voting), single-round MLLM selection with or without structured instructions, iterative selection that continues until two consecutive judgments agree, and Double-beats, which performs repeated pairwise comparisons with tie-breaking rounds. Unlike these alternatives, IPS evaluates the candidates under three independently shuffled orders, accepts only unanimous selections, and conservatively falls back to the Text mask when judgments disagree. As shown in Table 18, IPS achieves the best performance (79.6/77.4 gIoU/cIoU) with only 3 MLLM calls, outperforming Double-beats by +5.8/+5.9 points under a comparable inference budget (3.00 vs. 2.85 calls). It also surpasses iterative selection by +7.3/+12.5 points, despite requiring fewer calls (3.00 vs. 4.61). These results demonstrate that repeated MLLM evaluations alone do not guarantee better mask selection; instead, IPS improves selection reliability through order-robust consensus and conservative fallback.

AuthorsOpenReview ↗

Follow-up Response to Reviewer J2Mp

W2 — Consistency of Mask Selection and Refinement — Clarification of Algorithm 5

Q: Algorithm 5 does not explicitly include the three-round consensus selection in Eq. (10), and its discrepancy-region construction appears inconsistent with Eq. (11).

A: We thank the reviewer for carefully identifying these inconsistencies and apologize for the oversimplified pseudocode in Algorithm 5. We clarify that the actual implementation follows Eqs. (10) and (11), while Algorithm 5 omits certain intermediate operations. We have revised Alg. 5 in the Appendix.

Specifically, we have clarified the following three steps in the revised Algorithm 5:

  1. Mask selection. The MLLM evaluates Text, BBox, and Point candidates across three shuffled rounds. A candidate is accepted only when all three selections agree; otherwise, the Text mask is retained.

  2. Discrepancy analysis. We compute the exclusive regions RDBest=MBest∖MPointR_D^{\mathrm{Best}} = M_{\mathrm{Best}} \setminus M_{\mathrm{Point}} and RDPoint=MPoint∖MBestR_D^{\mathrm{Point}} = M_{\mathrm{Point}} \setminus M_{\mathrm{Best}}, following Eq. (11).

  3. Point-prompt updates. The revised pseudocode explicitly specifies how positive and negative point prompts are generated and updated to guide SAM3 in correcting residual segmentation errors.

Furthermore, upon acceptance, we will publicly release our implementation to facilitate reproducibility and encourage the community to explore and extend our mask refinement mechanism in related segmentation tasks.


Q2 — IPS Consensus and Fallback Frequency — Quantitative Analysis of Three-Round Agreement

Q: How often does IPS obtain unanimous agreement rather than falling back to the Text mask?

A: We thank the reviewer for this question. We conducted additional experiments to quantify the three-round consensus frequency and Text-mask fallback rate, with detailed results reported in Table 19 (Appendix C.4.11).

Table 19: Analysis of IPS consensus behavior and Point-mask contributions. Results are reported on RefCOCO TestA and ReasonSeg Test. The ΔgIoU/cIoU values indicate improvements relative to the corresponding Text-mask fallback baseline.

Selection OutcomeRefCOCO TestA: Queries (%)RefCOCO TestA: ΔgIoU/cIoUReasonSeg Test: Queries (%)ReasonSeg Test: ΔgIoU/cIoU
Unanimous Text26.00.0 / 0.022.10.0 / 0.0
Unanimous BBox14.8+35.5 / +25.04.3+7.4 / +11.9
Unanimous Point14.1+3.8 / +3.00.9+5.2 / +14.6
IPS (Fallback to Text)45.1+4.3 / +10.972.7+6.2 / +20.2

Consensus frequency. On RefCOCO TestA, the three shuffled evaluations achieve unanimous agreement in 54.9% of queries, comprising 26.0% Text, 14.8% BBox, and 14.1% Point selections, while the remaining 45.1% trigger the Text-mask fallback. On the more challenging ReasonSeg Test, unanimous agreement occurs in 27.3% of queries (22.1% Text, 4.3% BBox, and 0.9% Point), with 72.7% triggering fallback. These findings clarify how IPS balances reliable consensus-based selection with conservative fallback, particularly when handling complex and implicitly specified referring expressions.


Q3 — Validation-Based Configuration — Test-Set Independence and Reproducibility

Q: Were prompt designs and hyperparameters fixed without consulting final test labels?

A: We thank the reviewer for raising this important concern. We confirm that all prompt designs and hyperparameters were determined exclusively using validation splits, without consulting final test labels, and remained fixed throughout test evaluation. We have explicitly clarified this evaluation protocol in the revised main paper. Furthermore, to assess the stability of our validation-fixed configurations, we conducted three independent runs across four benchmarks, reporting mean ± standard deviation in Table 20 (Appendix C.4.12).

Table 20: Test-set performance with configurations fixed on validation splits. Results are reported as mean ± standard deviation of gIoU/cIoU (%) over repeated runs.

MethodRefCOCO TestARefCOCO+ TestARefCOCOg Test(U)ReasonSeg Test
SAM3 Agent68.55 ± 0.78 / 63.40 ± 1.2761.50 ± 2.25 / 55.57 ± 1.2561.50 ± 0.66 / 57.63 ± 1.1164.17 ± 1.02 / 57.30 ± 3.39
EVOL-SAM375.73 ± 0.79 / 72.70 ± 1.0571.27 ± 0.93 / 66.19 ± 1.4067.70 ± 0.46 / 64.73 ± 0.7570.63 ± 0.38 / 64.57 ± 0.45
Tarot-SAM380.00 ± 0.76 / 77.60 ± 0.9375.20 ± 1.04 / 71.80 ± 1.1972.97 ± 0.78 / 70.37 ± 0.9775.77 ± 0.95 / 71.10 ± 0.66

The results show that Tarot-SAM3 consistently outperforms both SAM3 Agent and EVOL-SAM3 across all four benchmarks, with relatively small standard deviations, supporting the robustness and reproducibility of our reported performance.

Reviewer_rNVsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work tackles the RES task by building a pipeline using frozen MLLM, DINOv3, and SAM3 models. They first design a module to rephrase the input text into multiple SAM-friendly text prompts and bounding boxes, which are used to generate mask candidates. The candidates are then filtered and selected through voting, DINOv3 similarity, and MLLM judgments. A final refinement strategy uses DINOv3 to identify positive and negative point prompts for mask refinement. Evaluations on RES benchmarks show state-of-the-art results among training-free methods. On ReasonSeg, which requires reasoning over complex input text, the proposed method achieves the best results on most metrics.

Critical strengths

The proposed method achieves promising results without fine-tuning any layers. Its components, including the mask proposal, selection, and refinement algorithms, are portable and potentially beneficial to related methods. The benchmark results also look good. Notably, its 7B variant surpasses other methods using 72B backbones on some benchmarks, demonstrating the empirical effectiveness of this work.

Critical weaknesses

My main concern is that several design choices appear heuristic and ad hoc. For instance, are all the intermediate steps in Eqs. (2–6), from the target-centric description T_D to the shorter and longer descriptions T_S,T_L, necessary? An MLLM could potentially generate the rephrased descriptions and candidate prompts directly using detailed instructions and in-context examples. Although the ablations support the usefulness of prompt augmentation and rephrasing, they do not isolate the benefit of the proposed intermediate decomposition over a simpler direct-prompting strategy.

The proposed method also relies heavily on a pretrained MLLM to interpret the input text. However, the experiments only evaluate Qwen2.5-VL variants, leaving the generalizability of the prompting strategy to other MLLM families unverified.

Questions for authors
  1. The technical section is difficult to follow and needs proofreading. Several important implementation details are also unclear, raising reproducibility concerns.
  • P5, L221 / Eq. (5): The text states that the MLLM conditions on Nt,PRa,BRaN_t, P_R^a, B_R^a, whereas Eq. (5) uses Nt,Nr,BRaN_t, N_r, B_R^a. Is PRaP_R^a the text prompt corresponding to one element of NrN_r? Please clarify this notation and explain how multiple reference objects are handled.
  • Eq. (7): Does a text-mask candidate need to exceed the overlap threshold with any one of BTIni,BTS,BTLB_T^{\mathrm{Ini}}, B_T^S, B_T^L, or with all three? Please specify the quantification over kk. The left-hand side should also denote the filtered mask set rather than an individual mask indexed by ii.
  • L238 / Algorithm 1, L3: Please clarify if the individual candidates MTexti,MBBXjM_{\mathrm{Text}}^i, M_{\mathrm{BBX}}^j are directly the multiple masks return from SAM3.
  • How are DIVERSETOP3, Reliability, Group, MAJORITY, and GUARDEDUPDATE implemented? The appendix describes their intended roles but does not fully specify their operations.
  • What is the implementation details of SpS_p in Eq. (8)?
  1. What is the contribution of the point-derived mask? Table 12 compares branch-level selection strategies and the final IPS result, but does not isolate the benefit of adding the point candidate. A comparison between IPS using only Text and BBox masks and IPS using all three mask types would help establish its effectiveness.

  2. The IPS module needs further analysis. How frequently do the three shuffled evaluations reach unanimous agreement, and how frequently is the Text-mask fallback triggered? Please also report the selection frequency of each mask type, distinguishing unanimous Text-mask selections from fallback selections.

  3. Beyond the examples in Figs. 3, 4, and 7, it would be helpful to provide a few complete examples showing all the outcomes of the proposed pipeline.

4 comments
AuthorsOpenReview ↗

Response to Reviewer rNVs

We sincerely thank the reviewer for the thoughtful and constructive feedback, and apologize for our delayed response. We have carefully considered each concern and conducted additional experiments and analyses to provide more comprehensive evidence and clarification. Below, we address the reviewer's comments point by point.


W1 — Necessity of Intermediate Decomposition — Structured Reasoning Outperforms Direct Prompt Generation

Q: Several design choices appear heuristic and ad hoc. Are all the intermediate steps in Eqs. (2–6), from the target-centric description to the shorter and longer descriptions, necessary? An MLLM could potentially generate the rephrased descriptions and candidate prompts directly using detailed instructions and in-context examples. The ablations do not isolate the benefit of the proposed intermediate decomposition over simpler direct prompting.

A: Yes, each intermediate step serves a distinct and complementary purpose in our reasoning-guided grounding pipeline. To directly investigate whether our intermediate reasoning decomposition provides benefits beyond simpler prompting strategies, we conducted additional comparisons with direct multi-prompt generation and structured prompting alternatives, both with and without RPO. The results are reported in Table 17 (Appendix C.4.9), together with a detailed discussion.

Table 17: Comparison of prompt construction strategies. Results are reported as gIoU/cIoU (%), with MLLM calls shown separately for RefCOCO and ReasonSeg.

Prompting StrategyMLLM Calls (RefCOCO / ReasonSeg)RefCOCOReasonSeg
Direct Multi-Prompt2 / 263.3 / 59.947.7 / 47.2
Simple Structured Prompting4 / 464.4 / 57.856.1 / 51.6
RPO + Direct Multi-Prompt12.14 / 7.7472.1 / 65.665.4 / 59.7
RPO + Structured Prompting14.32 / 9.6564.8 / 57.961.5 / 56.0
Our ERI13.97 / 9.0277.8 / 74.272.1 / 65.7

Key observations:

  1. Intermediate decomposition substantially improves grounding accuracy. Compared with Direct Multi-Prompt, ERI improves gIoU/cIoU by +14.5/+14.3 points on RefCOCO and +24.4/+18.5 points on ReasonSeg. Simple Structured Prompting also performs considerably worse, indicating that directly generating multiple prompts or merely introducing structured instructions is insufficient to achieve the same grounding quality. By progressively constraining target interpretation and preserving semantic consistency, ERI also helps mitigate MLLM hallucinations and stochastic inconsistencies, reducing the propagation of erroneous grounding cues into subsequent segmentation.

  2. The benefits cannot be explained simply by additional MLLM calls. Compared with RPO + Structured Prompting, ERI achieves improvements of +13.0/+16.3 points on RefCOCO and +10.6/+9.7 points on ReasonSeg, while requiring fewer MLLM calls (13.97 vs. 14.32 and 9.02 vs. 9.65, respectively). Furthermore, ERI outperforms RPO + Direct Multi-Prompt by +8.6 and +6.0 cIoU points, with only 1.83 and 1.28 additional MLLM calls. These results suggest that how reasoning is organized matters more than simply increasing prompting complexity or inference iterations.

  3. Task-oriented decomposition provides complementary benefits. Rather than asking the MLLM to generate all descriptions and spatial prompts in a single step, ERI progressively refines the target interpretation before constructing heterogeneous prompts. This allows the intermediate descriptions to serve distinct grounding purposes, reducing reliance on a single potentially inaccurate MLLM interpretation, which is also noted by Reviewer J2Mp's positive recognition that "our combining structured interpretation with multiple prompt types is a reasonable way to address these difficulties," particularly when referring expressions specify targets indirectly.

AuthorsOpenReview ↗

Follow-up Response to Reviewer rNVs

W2 — Generalization Across MLLM Families — Consistent Effectiveness Beyond Qwen2.5-VL

Q: The proposed method also relies heavily on a pretrained MLLM to interpret the input text. However, the experiments only evaluate Qwen2.5-VL variants, leaving the generalizability of the prompting strategy to other MLLM families unverified.

A: We thank the reviewer for highlighting this important concern. To evaluate the generalizability of Tarot-SAM3 beyond Qwen2.5-VL family, we conducted additional experiments with two distinct MLLM backbones, Qwen3-VL-32B and LLaVA1.5-7B, covering different model architectures and capacities. For fair comparisons, we evaluated SAM3 Agent, EVOL-SAM3, and Tarot-SAM3 using the same MLLM backbone within each group. The results are reported in Table 21 (Appendix C.4.13).

Table 21: Generalization across different MLLM backbones. Results are reported as gIoU/cIoU (%) on RefCOCO TestA and ReasonSeg Test.

MLLM BackboneMethodRefCOCO TestAReasonSeg Test
Qwen3-VL-32BSAM3 Agent72.9 / 70.268.4 / 67.6
Qwen3-VL-32BEVOL-SAM342.3 / 39.869.7 / 62.5
Qwen3-VL-32BTarot-SAM3 (Ours)73.8 / 69.376.2 / 71.6
LLaVA1.5-7BSAM3 Agent8.6 / 7.921.3 / 19.6
LLaVA1.5-7BEVOL-SAM346.6 / 39.550.1 / 43.0
LLaVA1.5-7BTarot-SAM3 (Ours)49.2 / 48.257.8 / 49.6

Key observations:

  1. Strong generalization to Qwen3-VL-32B. Tarot-SAM3 achieves the highest gIoU on RefCOCO testA (73.8) and the best gIoU/cIoU on ReasonSeg (76.2/71.6). In particular, it outperforms SAM3 Agent by +7.8/+4.0 points and EVOL-SAM3 by +6.5/+9.1 points on ReasonSeg. Although SAM3 Agent achieves a slightly higher RefCOCO cIoU (70.2 vs. 69.3), our framework demonstrates strong performance across both benchmarks.

  2. Effectiveness with a weaker MLLM backbone. Even with LLaVA1.5-7B, Tarot-SAM3 consistently outperforms both baselines across all evaluated metrics. Compared with EVOL-SAM3, it improves gIoU/cIoU by +2.6/+8.7 points on RefCOCO and +7.7/+6.6 points on ReasonSeg. These results are particularly encouraging because they demonstrate that our framework can improve segmentation performance even when the underlying MLLM has relatively limited reasoning capabilities.

AuthorsOpenReview ↗

Follow-up Response to Reviewer rNVs

Q1 — Technical Clarity and Reproducibility — Clarifications of Notation and Implementation

Q: Several equations and implementation details are unclear, particularly Eqs. (5), (7), (8), Algorithm 1, and the mask selection operations.

A: We thank the reviewer for carefully identifying these ambiguities and apologize for the insufficient clarity. We address each concern below and will revise the corresponding equations and algorithm descriptions to improve reproducibility.

1. Reference-object notation (Eq. 5). PRaP_R^a denotes the SAM3 text prompt corresponding to a reference object nRa∈Nrn_R^a \in N_r. When multiple reference objects are involved, each has its own prompt, grounding box BRaB_R^a, and relational criterion PCaP_C^a. We will correct Eq. (5) to consistently use (Nt,PRa,BRa)(N_t,P_R^a,B_R^a).

2. Overlap filtering (Eq. 7). A Text mask is retained if its bounding box overlaps with at least one of BTIniB_T^{\mathrm{Ini}}, BTSB_T^S, or BTLB_T^L above the threshold, rather than all three. We will explicitly introduce the existential quantifier over kk and correct the left-hand side to denote the filtered mask set.

3. Mask candidates (Algorithm 1). MTextiM_{\mathrm{Text}}^i and MBBXjM_{\mathrm{BBX}}^j originate from SAM3 predictions under different prompts. Before selection, empty masks are removed, consistency filtering is applied, and union masks may be constructed for multi-object targets. Thus, these symbols represent processed candidate masks rather than all raw SAM3 outputs.

4. Mask selection operations. These operations are high-level abstractions of multiple underlying functions and decision rules. Their key implementations are summarized below:

OperationImplementation Details
DIVERSETOP3Compute each candidate's maximum mask IoU with every prompt route. Rank by the number of agreeing routes, mean cross-route agreement, and SAM3 confidence. Greedily retain up to three candidates that satisfy the diversity constraint against every selected mask. Text candidates prioritize per-prompt union masks, followed by individual masks when needed.
ReliabilityUse unique masks from augmented Text prompts as voters. Weight each by SAM3 confidence, maximum BBox IoU, cross-prompt consistency, and anchor agreement, normalized by the number of unique masks per prompt. Score BBox candidates by weighted mean IoU, retaining only those meeting the anchor's support level and additional overlap constraints for high-confidence anchors. Select the highest-scoring eligible candidate after incorporating confidence; otherwise retain the anchor.
GroupProcess Text Top-3 and a nonduplicate Full Description mask sequentially. Assign each candidate to the first group where its IoU with every member meets the grouping threshold; otherwise create a new group. Group-level Text support and SAM3 confidence are determined by their maximum member-level values.
MAJORITYEach BBox mask supports its uniquely preferred Text group based on maximum group IoU, weighted by confidence, stability, consensus, and the margin over the runner-up. Text support, BBox support, and SAM3 confidence each vote for their unique top group; ties abstain. Select the first mask of the group receiving at least two votes; otherwise return the first mask of the first group.
GUARDEDUPDATEStarting from the majority-selected mask, each applicable semantic, spatial, or containment check either proposes an alternative candidate or abstains. We accept the update only when all non-abstaining proposals identify a single distinct alternative; if no alternative is proposed or different alternatives conflict, we retain the current mask. Short/Long BBox evidence provides additional proposals by verifying candidate agreement and containment. This conflict-aware update is deterministic and introduces no additional hyperparameters.

We will expand these operational definitions in the revised appendix and release our implementation code upon publication, providing the complete decision rules and facilitating reproducibility.

5. DINOv3 similarity map (Eq. 8). For each sampled anchor pp, we extract its final-layer DINOv3 feature and compute cosine similarity with every spatial feature: Sp(x)=⟨F(p),F(x)⟩∥F(p)∥2∥F(x)∥2.S_p(x)=\frac{\langle F(p),F(x)\rangle}{\|F(p)\|_2\|F(x)\|_2}. The resulting anchor-specific similarity maps are aggregated using the pixel-wise harmonic mean to guide SAM3 point-prompt generation.

AuthorsOpenReview ↗

Follow-up Response to Reviewer rNVs

Q2 — Contribution of Point-Derived Masks — Improved Selection Through Complementary Evidence

Q: What is the contribution of the point-derived mask? Table 12 does not isolate the benefit of adding the point candidate. A comparison between IPS using only Text and BBox masks and IPS using all three mask types would help establish its effectiveness.

A: We thank the reviewer for this valuable suggestion. To explicitly evaluate the contribution of Point masks, we conducted an additional ablation comparing IPS with Text and BBox candidates only against IPS with all three prompt types. The results are provided in Table 19 (Appendix C.4.11).

Table 19: Analysis of IPS consensus behavior and Point-mask contributions. Results are reported on RefCOCO TestA and ReasonSeg Test. The ΔgIoU/cIoU values indicate improvements relative to the corresponding Text-mask fallback baseline.

Selection OutcomeRefCOCO TestA: Queries (%)RefCOCO TestA: ΔgIoU/cIoUReasonSeg Test: Queries (%)ReasonSeg Test: ΔgIoU/cIoU
Unanimous Text26.00.0 / 0.022.10.0 / 0.0
Unanimous BBox14.8+35.5 / +25.04.3+7.4 / +11.9
Unanimous Point14.1+3.8 / +3.00.9+5.2 / +14.6
IPS w/o Point (Fallback to Text)42.3+3.4 / +8.868.3+4.1 / +10.9
IPS w/ Point (Fallback to Text)45.1+4.3 / +10.972.7+6.2 / +20.2

Key observations:

  1. Effectiveness of unanimous Point selection. When the three shuffled MLLM evaluations unanimously select the Point candidate, it outperforms the corresponding Text-mask fallback by +3.8/+3.0 gIoU/cIoU on RefCOCO and +5.2/+14.6 on ReasonSeg. These results demonstrate that Point masks provide complementary segmentation evidence beyond Text and BBox candidates, particularly for challenging reasoning-based expressions.

  2. More reliable consensus verification. Although adding Point masks slightly increases the Text fallback rate (42.3% → 45.1% on RefCOCO; 68.3% → 72.7% on ReasonSeg), selection performance improves the gains on fallback items from 3.4/8.8 to 4.3/10.9 and from 4.1/10.9 to 6.2/20.2 gIoU/cIoU, respectively. This suggests that Point masks provide complementary visual evidence that improves the reliability of mask selection. By introducing an additional candidate, IPS becomes more selective in accepting unanimous MLLM judgments, while retaining the conservative Text fallback when selections disagree.


Q3 — IPS Consensus and Fallback Analysis — Explicit Selection Frequency Across Mask Types

Q: The IPS module needs further analysis. How frequently do the three shuffled evaluations reach unanimous agreement, and how frequently is the Text-mask fallback triggered? Please also report the selection frequency of each mask type, distinguishing unanimous Text-mask selections from fallback selections.

A: We also conducted additional experiments to quantify three-way agreement, mask-type selection frequencies, and Text-mask fallback behavior. The detailed analysis is provided in previous Table 19 (Appendix C.4.11).

Key observations:

  1. Consensus frequency. On RefCOCO TestA, the three shuffled evaluations achieve unanimous agreement in 54.9% of queries, comprising 26.0% Text, 14.8% BBox, and 14.1% Point selections, while the remaining 45.1% trigger the Text-mask fallback. On the more challenging ReasonSeg Test, unanimous agreement occurs in 27.3% of queries (22.1% Text, 4.3% BBox, and 0.9% Point), with 72.7% triggering fallback. These findings clarify how IPS balances reliable consensus-based selection with conservative fallback, particularly when handling complex and implicitly specified referring expressions.

Q4 — Additional Qualitative Examples — Comprehensive Visualization of Segmentation Results

Q: Beyond the examples in Figs. 3, 4, and 7, it would be helpful to provide a few complete examples showing all the outcomes of the proposed pipeline.

A: We have provided additional qualitative visualizations in Figs. 11–16 of the Appendix, covering segmentation results across explicit and implicit RES benchmarks, open-world scenarios, and 360° panoramic images. In particular, Figs. 15 and 16 further illustrate the contributions of MSR and DINOv3, respectively, through component-level comparisons. These examples provide a more comprehensive view of Tarot-SAM3's segmentation performance and the effectiveness of its key components.

Reviewer_TZ7vOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Tarot-SAM3 is a training-free pipeline for explicit and implicit referring expression segmentation. It couples a frozen MLLM with SAM3 and DINOv3. The Expression Reasoning Interpreter (ERI) guides the MLLM through six structured reasoning dimensions to rewrite the query. It then builds augmented text and box prompts for SAM3, picks one mask per prompt type by IoU-based consensus voting, and derives point prompts from DINOv3 similarity. Mask Self-Refining (MSR) accepts the MLLM's final mask choice only when three shuffled-order queries agree. It then compares the chosen mask with the point-prompt mask and adds or shifts corrective points to fix over- and under-segmentation.

Critical strengths
  1. The training-free results at 7B are strong. The 7B variant beats the 72B training-free baselines on RefCOCO testA and ReasonSeg test. It also beats several fine-tuned methods on ReasonSeg (Tabs. 1–2).
  2. The position-bias analysis (Fig. 2, Tab. 6) quantifies a concrete failure mode of pipelines that rank masks by pairwise MLLM comparison.
  3. Some components are reusable. Adding TA/BA/MSR to EVOL-SAM3 improves RefCOCO testA by +2.9/+3.1 gIoU/cIoU (Tab. 5). The ablations are extensive, and code is provided.
Critical weaknesses
  1. Several headline claims contradict the paper's own tables. The claim of state-of-the-art results "across MLLM scales" fails at 3B (four of nine RefCOCO-family splits, ReasonSeg val cIoU) and at 72B (RefCOCO+ testB, ReasonSeg test-short cIoU). SAM3 Agent with Gemini 2.5 Pro beats the 72B variant on ReasonSeg test cIoU and RefCOCO testB (Tab. 15) but is left out of the main tables. Measured against EVOL-SAM3's published ReasonSeg numbers instead of the authors' re-evaluations, the 7B test margin shrinks from +4.9/+6.2 to +3.3/+3.8.
  2. Computational cost is never reported. Each query chains many MLLM calls (Eqs. 1–6, Alg. 2, Eq. 10, and up to three MSR rounds) and also runs SAM3 and DINOv3 repeatedly. Without comparisons at matched compute, the claim that the 7B variant beats 72B baselines is hard to interpret.
  3. Design choices are tuned on test splits: hyperparameters on RefCOCO testA (Tab. 4) and all ablations on test splits. Every result is a single run despite the random shuffling in Eq. (10), and several component gains are only 0.3–1.1 points.
  4. Novelty is limited. The method mainly combines known ingredients: structured prompting, prompt augmentation, IoU-voting ensembles and feature-similarity point prompts for SAM.
Questions for authors
  1. Tab. 6 shows that two orderings give a consistent winner in only 8.3% of pairs. What fraction of queries pass the three-way agreement in Eq. (10)? How accurate are the accepted cases compared with the fallback cases? If most queries fall back to MTextM_{\mathrm{Text}}, do the IPS gains come from the fallback rule rather than from consensus?
  2. Per query, how many MLLM, SAM3 and DINOv3 calls do Tarot-SAM3, SAM3 Agent and EVOL-SAM3 make? How many tokens do they generate, and what are their latency and peak memory? How do the methods compare at matched compute?
  3. Why were EVOL-SAM3's ReasonSeg results re-evaluated rather than taken from its paper, and what explains the gap?
  4. What are the results when all design choices are fixed on validation splits, reported as mean ± std over several runs?
8 comments
AuthorsOpenReview ↗

Response to Reviewer TZ7v

We sincerely thank the reviewer for the thoughtful and constructive feedback, and apologize for our delayed response. We have carefully considered each concern and conducted additional experiments and analyses to provide more comprehensive evidence and clarification. Below, we address the reviewer's comments point by point.


W1.1 — SOTA Claims Across MLLM Scales — We have Revised the Overgeneralized Statement

Q: Several headline claims contradict the paper's own tables. The claim of state-of-the-art results "across MLLM scales" fails at 3B (four of nine RefCOCO-family splits, ReasonSeg val cIoU) and at 72B (RefCOCO+ testB, ReasonSeg test-short cIoU).

A: We apologize for the overly broad statement and thank the reviewer for identifying this issue. We acknowledge that Tarot-SAM3 does not achieve the best performance on every benchmark split across all MLLM scales. We have revised the corresponding claims in the abstract and main text to "across most MLLM scales", rather than implying consistent SOTA performance in every setting.


W1.2 — Gemini 2.5 Pro Comparison — Preserving Fair Same-Backbone Comparisons

Q: SAM3 Agent with Gemini 2.5 Pro beats the 72B variant on ReasonSeg test cIoU and RefCOCO testB (Tab. 15) but is left out of the main tables.

A: We appreciate this observation. We initially presented this comparison in Table 16 because Gemini 2.5 Pro belongs to a different MLLM family, whereas our main tables emphasize controlled comparisons under matched Qwen2.5-VL backbones. Directly mixing different MLLM families in the main tables could compromise comparison fairness for both our method and existing baselines, as performance differences may arise from backbone capabilities rather than the frameworks themselves. Nevertheless, we agree that this strong baseline deserves greater visibility and have highlighted its results more explicitly in the revised manuscript. Notably, our 7B model already outperforms SAM3 Agent with Gemini 2.5 Pro on RefCOCO testA cIoU (78.5 vs. 77.8) and ReasonSeg test gIoU (75.8 vs. 74.0), as reported in Table 16. While Gemini 2.5 Pro performs better on certain other metrics, these results demonstrate that our framework remains competitive with stronger MLLM-based alternatives.


W1.3 (also Q3) — Re-evaluation of EVOL-SAM3 — Ensuring Consistent Single-Query Evaluation

Q: Measured against EVOL-SAM3's published ReasonSeg numbers instead of the authors' re-evaluations, the 7B test margin shrinks from +4.9/+6.2 to +3.3/+3.8.

A: We thank the reviewer for raising this concern. We identified an issue in EVOL-SAM3's publicly released ReasonSeg evaluation code, where multiple referring expressions associated with the same image are processed together as a single query, potentially introducing additional semantic information beyond the individual referring expression.

To ensure a fair and consistent single-query evaluation protocol, we corrected only the text input handling, without modifying EVOL-SAM3's model architecture or inference strategy. This difference in query handling explains the discrepancy between the published and our re-evaluated results. Importantly, even when EVOL-SAM3 benefits from additional semantic information by processing multiple referring expressions jointly, Tarot-SAM3 still outperforms its originally reported results by +3.3/+3.8 gIoU/cIoU on ReasonSeg test, further demonstrating the effectiveness of our framework.

AuthorsOpenReview ↗

Follow-up Response to Reviewer TZ7v

W2 (also Q2) — Computational Cost and Inference Efficiency — Comprehensive Evaluation under Matched Backbone Sizes

Q: Computational cost is never reported. Each query chains many MLLM calls (Eqs. 1–6, Alg. 2, Eq. 10, and up to three MSR rounds) and also runs SAM3 and DINOv3 repeatedly. Without comparisons at matched compute, the claim that the 7B variant beats 72B baselines is hard to interpret.

A: We thank the reviewer for highlighting this important concern. We agree that inference costs are essential for interpreting performance improvements, particularly when comparing methods with different reasoning pipelines and backbone sizes. Due to the limited time available for additional experiments, we have prioritized a comprehensive inference-cost evaluation of all three methods under the same 7B MLLM backbone size. The corresponding 72B inference-cost experiments are still in progress, and we intend to provide these results in a subsequent rebuttal update once completed.

Importantly, we have added Table 6 to the main manuscript, reporting end-to-end latency, average MLLM/DINOv3/SAM3 calls, token consumption, and GPU memory. We additionally evaluate SAM3 Agent and EVOL-SAM3 with increased MLLM and SAM3 inference budgets to investigate whether additional computation can reproduce our performance gains.

Table 6: Performance and inference cost comparison across different computational budgets. Results are reported as gIoU/cIoU (%).

DatasetMethodgIoU/cIoU ↑Latency (s) ↓MLLM Calls ↓DINOv3 Calls ↓SAM3 Calls ↓Tokens ↓Memory (GiB) ↓
SAM3 Agent (7B)69.1 / 64.316.826.5101.8196,3505.67
SAM3 Agent (7B, Raise MLLM)66.1 / 63.226.4814.9401.67231,6495.67
SAM3 Agent (7B, Raise SAM3)64.7 / 62.917.037.03012.74107,1205.67
RefCOCOEVOL-SAM3 (7B)76.3 / 73.75.638.7206.919,3584.21
EVOL-SAM3 (7B, Raise MLLM)74.8 / 71.88.2716.29011.9813,2274.21
EVOL-SAM3 (7B, Raise SAM3)75.2 / 72.07.6414.38015.5511,4744.21
Tarot-SAM3 (7B)80.4 / 78.512.3615.841.0013.3212,8217.56
SAM3 Agent (7B)63.0 / 53.526.516.9101.4698,8235.65
SAM3 Agent (7B, Raise MLLM)64.2 / 57.534.849.3601.58134,6295.65
SAM3 Agent (7B, Raise SAM3)65.0 / 59.125.037.55017.71106,3325.65
ReasonSegEVOL-SAM3 (7B)70.9 / 65.015.959.4705.9264,9064.21
EVOL-SAM3 (7B, Raise MLLM)70.4 / 64.416.8210.9407.0771,2764.21
EVOL-SAM3 (7B, Raise SAM3)71.1 / 63.116.4513.90020.43114,8384.21
Tarot-SAM3 (7B)75.8 / 71.233.409.940.6519.1036,1537.65
Key observations:
  1. Accuracy–efficiency trade-off under matched backbone sizes. On RefCOCO, Tarot-SAM3 outperforms SAM3 Agent by +11.3/+14.2 gIoU/cIoU, while achieving lower latency (12.36s vs. 16.82s) and substantially fewer tokens (12,821 vs. 96,350). It also surpasses EVOL-SAM3 by +4.1/+4.8 points, although EVOL-SAM3 remains faster. On ReasonSeg, Tarot-SAM3 improves upon EVOL-SAM3 by +4.9/+6.2 points, at the cost of increased latency (33.40s vs. 15.95s). While this introduces additional computational overhead, the substantial accuracy improvements on challenging reasoning-based segmentation tasks demonstrate a meaningful accuracy–efficiency trade-off.

  2. Additional inference calls alone do not close the performance gap. Increasing the MLLM or SAM3 budgets of the baselines does not consistently improve their performance. For example, on ReasonSeg, EVOL-SAM3 with 20.43 SAM3 calls, exceeding our 19.10, achieves 71.1/63.1, compared with our 75.8/71.2. Similarly, on ReasonSeg, SAM3 Agent with increased MLLM calls still achieves only 64.2/57.5 gIoU/cIoU with 9.36 MLLM calls, whereas Tarot-SAM3 achieves 75.8/71.2 with a comparable 9.94 MLLM calls. These results further suggest that simply increasing MLLM calls or candidate masks can introduce additional selection ambiguity and inconsistent judgments in existing methods, potentially degrading their performance rather than improving it. In contrast, our structured reasoning and consensus-based selection make more effective use of the adequate inference budget.

AuthorsOpenReview ↗

Follow-up Response to Reviewer TZ7v

W3 (also Q4) — Validation-Based Hyperparameter Selection and Stability — Consistent Performance Across Repeated Runs

Q: Design choices are tuned on test splits: hyperparameters on RefCOCO testA (Tab. 4) and all ablations on test splits. Every result is a single run despite the random shuffling in Eq. (10), and several component gains are only 0.3–1.1 points. What are the results when all design choices are fixed on validation splits, reported as mean ± std over several runs?

A: We sincerely apologize for the confusion caused by our presentation. All hyperparameters were selected using validation splits and kept fixed during test evaluation. Table 4 reports sensitivity results on RefCOCO testA, which may have misleadingly suggested that test data were used for hyperparameter tuning. We have clarified this distinction in the revised manuscript.

To further address concerns about stochastic MLLM judgments and experimental stability, we conducted three independent runs using validation-fixed configurations for Tarot-SAM3 and both baselines under the same 7B backbone. The results are provided in Table 20 (Appendix C.4.12), reporting the mean ± standard deviation of gIoU/cIoU (%) across four benchmarks.

Table 20: Test-set performance with configurations fixed on validation splits. Results are reported as mean ± standard deviation of gIoU/cIoU (%) over repeated runs.

MethodRefCOCO TestARefCOCO+ TestARefCOCOg Test(U)ReasonSeg Test
SAM3 Agent68.55 ± 0.78 / 63.40 ± 1.2761.50 ± 2.25 / 55.57 ± 1.2561.50 ± 0.66 / 57.63 ± 1.1164.17 ± 1.02 / 57.30 ± 3.39
EVOL-SAM375.73 ± 0.79 / 72.70 ± 1.0571.27 ± 0.93 / 66.19 ± 1.4067.70 ± 0.46 / 64.73 ± 0.7570.63 ± 0.38 / 64.57 ± 0.45
Tarot-SAM380.00 ± 0.76 / 77.60 ± 0.9375.20 ± 1.04 / 71.80 ± 1.1972.97 ± 0.78 / 70.37 ± 0.9775.77 ± 0.95 / 71.10 ± 0.66

Key observations:

  1. Consistent performance gains. Tarot-SAM3 achieves the highest mean gIoU/cIoU across all four benchmarks, outperforming EVOL-SAM3 by 3.93–5.27 / 4.90–6.53 points, respectively, with all configurations fixed on validation splits.

  2. Strong performance stability. Despite stochastic MLLM judgments and mask-order shuffling, Tarot-SAM3 maintains standard deviations below 1.2 points across all eight metrics. Our method exhibits lower variability than SAM3 Agent on 7 of 8 metrics, particularly on ReasonSeg cIoU (0.66 vs. 3.39) and RefCOCO+ gIoU (1.04 vs. 2.25). Compared with EVOL-SAM3, we also achieve lower standard deviations on several metrics, including RefCOCO gIoU/cIoU (0.76/0.93 vs. 0.79/1.05) and RefCOCO+ cIoU (1.19 vs. 1.40). Although our standard deviations are slightly higher on some other metrics, these variations remain limited relative to the performance gains, indicating that our improved accuracy does not come at the expense of substantially reduced stability.

  3. Consistent incremental contributions. Our ablation study (Table 3) progressively introduces each component under otherwise identical experimental settings, showing consistent incremental improvements on both RefCOCO and ReasonSeg. Although some individual gains are modest (0.3–1.1 points), they follow a systematic accumulation of improvements rather than isolated performance fluctuations. Furthermore, the three-run evaluations confirm that the overall gains remain stable despite stochastic MLLM judgments, supporting the effectiveness of our progressive design.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.