acceptodds
Under review as a conference paper at ICLR 2027

Light SAM3 Agent: Efficient Agentic Segmentation via Latent NPs Extraction and Action Space Slimming

14 views

Abstract

SAM3 Agent has demonstrated strong performance in reasoning segmentation tasks by combining multimodal reasoning with SAM3's precise mask generation. However, its substantial token consumption and long inference time limit its use in real-world applications. In this paper, we analyze the interaction behavior of SAM3 Agent and identify two key sources of redundancy in its original design: i) agent loop redundancy: SAM3 Agent relies on effective noun phrases (NPs) to obtain valid masks from SAM3, yet in challenging cases, identifying such an NP can consume many or even all available interaction rounds, leading to excessive MLLM inference cost. ii) action space redundancy: SAM3 Agent defines four executable actions, but we find that some contribute very little and may even have negative effects due to the additional context they introduce. To address these issues, we propose Light SAM3 Agent, a training-free acceleration framework that improves token efficiency by optimizing the agent loop and action space. Specifically, for agent loop redundancy, we propose Latent NPs Extraction, which explores alternative NPs in the latent space via probabilistic sampling, thereby compressing multiple rounds of NP exploration into a single round while avoiding costly explicit token generation. Moreover, for action space redundancy, we propose Action Space Slimming, which evaluates the invocation frequency and empirical utility of each action, and removes low-value actions from the candidate action set. Light SAM3 Agent achieves an average end-to-end speedup of 2.11× on the ReasonSeg test set while reducing token consumption by 38.8%. Meanwhile, with a cleaner reasoning context and more effective NP exploration, Light SAM3 Agent improves gIoU and cIoU over SAM3 Agent by 6.2% and 3.4%.

Primary area:
optimization
Keywords:
Agentic Segmentation, Multimodal Large Language Models, Inference Efficiency, AI Agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tVYnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Light SAM3 Agent, a training-free framework that reduces interaction costs in reasoning-based segmentation. The method comprises two components: (1) LNE, which reuses the same MLLM inference prefix, selects different branches based on the probability distribution of the first token of noun phrases, greedily completes candidates, and continues trying when SAM3 returns an empty mask; and (2) ASPS, which removes two actions—checking masks one by one and explicitly reporting no target—while simplifying prompts. On the ReasonSeg benchmark, gIoU improves from 63.6 to 69.8, decoded tokens decrease from 529 to 324, and the per-sample latency speedup achieves a mean of 2.11× and a median of 1.73×.

Critical strengths

Strength 1 Identifies a specific and practically meaningful system bottleneck.

Correct semantic reasoning does not guarantee that the generated phrase can be effectively localized by SAM3. To address this failure mode where "the target is correctly understood but the phrasing is unsuitable for the segmenter," reusing existing inference prefixes and attempting alternative phrasings on demand is a reasonable, training-free optimization. It avoids rerunning full inference for every wording adjustment and offers direct practical value.

Strength 2 The combined method yields substantial empirical gains.

The main experiments, using the same backbone and segmentor, simultaneously improve both quality and efficiency. Furthermore, the paper provides component ablations, budget analyses, cross-model experiments, and sample-level latency distributions. It further distinguishes budget-exhaustion recovery from other prediction changes, helping to clarify the sources of the observed benefits.

Critical weaknesses

Weaknesses 1 The unique contribution of LNE has not been sufficiently disentangled from cache reuse, on-demand decoding, and implementation details.

In practice, LNE performs "first-token ranking branching + explicit autoregressive completion"; terms like "latent extraction" and "probabilistic sampling" in the abstract may overstate its distinction from standard decoding search. Existing Explicit-3/5NPs baselines neither use the same candidate budget nor generate all candidates upfront, making it impossible to isolate the value of the first-token branching strategy. Although the 40-round single-NP baseline is informative, it does not control for identical caching and early-stopping conditions.

Weaknesses 2 ASPS interprets low usage frequency as low utility but does not adequately verify whether the removed capabilities are truly redundant.

The paper notes that "average IoU when invoking a certain tool" suffers from hard-case selection bias: checking masks one by one may occur precisely on the most difficult samples, so a low conditional IoU does not imply the action is ineffective. Meanwhile, ASPS also rewrites tool descriptions and CoT templates, making it difficult to attribute gains solely to action removal. Moreover, replacing explicit no-target judgment with budget exhaustion is not equivalent: when no target exists, the system may keep trying until the budget runs out; in the presence of similar distractors, it may produce non-empty yet incorrect candidates.

Questions for authors
  1. After fixing the template, caching, deduplication, candidate budget, and stopping rules, how much additional quality and end-to-end latency improvement does first-token ranking branching provide compared to standard on-demand sampling?

  2. Can controlled experiments validate both the benefits and capability costs of tool removal? I suggest three settings: "original four tools + compact prompts," "two-tool ASPS," and "retaining tools on demand for hard cases." Evaluation could leverage gRefCOCO, which includes no-target and multi-target expressions, to cover dimensions missing from the current evaluation. However, given time constraints, I would accept key ablations only.

5 comments
AuthorsOpenReview ↗

Response to Reviewer tVYn (1/3)

W1: [Disentangling LNE's Contribution and Terminology Clarification]

Thank you for your helpful comments. We agree with your precise summary of our implementation. In the revised paper, we have replaced "probabilistic sampling" with "probability-guided branching" to avoid misunderstanding. We retain “latent” to refer to alternative NPs that are not selected in the original greedy response but can be decoded from the same reasoning prefix.

Under the experimental setup of Table 3, we additionally evaluated Explicit-7/9NPs and observed performance degradation. Inspection of the generation logs showed increasingly repetitive phrasings and reduced NP diversity. Given this trend, we did not further expand the upfront candidate list to 15.

Instead, we implemented On-demand NP Rewriting (ODR) to match LNE's candidate budget (K=15K=15) while generating alternatives only when needed. When SAM3 returns no mask, a fresh MLLM session uses the image, question, and previously failed NPs to generate a new NP for the same target. This process repeats until SAM3 returns a non-empty mask or the candidate budget is exhausted. The results on ReasonSeg are as follows:

MethodVal Tokens ↓Val Rounds ↓Val gIoU ↑Test Tokens ↓Test Rounds ↓Test gIoU ↑
Explicit-3NPs3922.2169.04152.4166.2
Explicit-5NPs3992.2168.34032.2467.1
Explicit-7NPs4212.2666.34232.2766.3
Explicit-9NPs4172.1966.64202.2266.2
ODR3512.4071.23792.4068.4
Light SAM3 Agent (Ours)3152.1970.23242.2569.8

Note: ODR rewriting contributes to Tokens but not Rounds. This experiment was conducted on an NVIDIA RTX A6000 GPU.

Under the same K=15K=15 budget and on-demand execution, ODR requires fresh MLLM sessions for NP rewriting, whereas LNE decodes alternatives from the existing reasoning prefix. Light SAM3 Agent uses fewer tokens and outer agent rounds on both splits. Although ODR achieves higher Val gIoU, Light SAM3 Agent improves Test gIoU by 1.4% with 14.5% fewer tokens.

Q1: [LNE vs. Standard On-Demand Sampling Under Identical Settings]

Thank you for suggesting this controlled comparison. Under strictly identical templates, KV cache reuse, deduplication rules, candidate budget (K=15K=15), and early-stopping rules, we evaluated LNE against standard decoding strategies (Top-pp, Top-kk, and Beam Search) on ReasonSeg. The paper uses greedy decoding for LNE to ensure reproducibility. For a fairer and more comprehensive comparison, we additionally implemented LNE variants using Top-pp and Top-kk. The results are summarized below:

MethodVal gIoU (%) ↑Val cIoU (%) ↑Val Time (s) ↓Val Rounds ↓Val Tokens ↓Test gIoU (%) ↑Test cIoU (%) ↑Test Time (s) ↓Test Rounds ↓Test Tokens ↓
Top-p70.258.411.52.3334468.956.713.32.27337
Top-k70.158.312.82.3434569.056.714.22.23328
Beam Search69.656.511.92.3462868.055.713.32.24511
LNE (Top-p)70.058.811.82.1831269.558.913.52.18314
LNE (Top-k)70.458.710.52.1731069.558.512.62.19314
LNE (Greedy)70.558.510.62.1931469.257.512.62.19315

Note: Due to limited compute availability, this experiment was conducted on an NVIDIA RTX PRO 6000D GPU.

Under identical settings, LNE (Greedy) improves gIoU over standard Top-p/Top-k sampling by 0.3%–0.4% on Val and 0.2–0.3 % on Test, while reducing end-to-end latency by 7.8–17.2% and 5.3–11.3%, respectively. Compared with Beam Search, it improves gIoU by 0.9/1.2 % and reduces latency by 10.9%/5.3% on Val/Test. Test cIoU also improves by 0.8 % over standard sampling and 1.8 % over Beam Search.

The token savings reflect different decoding overheads: standard sampling can repeatedly generate duplicate candidates that are subsequently discarded, while Beam Search decodes branches that are not ultimately used. LNE explores ranked first-token alternatives and greedily completes one candidate at a time, limiting both sources of overhead.

AuthorsOpenReview ↗

Response to Reviewer tVYn (2/3)

W2&Q2: [Controlled Validation and Capability Analysis of ASPS]

We sincerely appreciate your suggestions on controls and gRefCOCO evaluation, and your consideration of the time constraints. Our analyses and experiments follow.

1. Analysis of the Removed Actions (W2):

ASPS involves a system-level trade-off between the potential benefits of individual actions and the interaction overhead and decision complexity they introduce. We analyze the two removed actions below.

(a) On examine each mask:

Across all 6 ReasonSeg cases where SAM3 Agent invoked examine each mask, ASPS (direct mask observation in select masks and return) improved 3/6 , left 2/6 unchanged , and degraded 1/6 . Two of the improved cases originally had 0 IoU due to execution errors in the additional examination session, highlighting the execution risks of added workflow complexity. Together with prior findings that additional close-up views do not necessarily improve visual reasoning [R1], these results suggest limited practical benefit from the additional examination step in these cases.

(b) On report no mask vs. Deterministic No-Mask Rule:

We understand the concerns about false positives under distractors and wasted computation when no target exists. However, allowing the MLLM to invoke report no mask can prematurely terminate the search even when the target is present. On ReasonSeg, removing this action recovers a substantial proportion of these false negatives:

SplitFalse-Negative "No Mask" CallsRecovered by ASPSRecovered by ASPS + LNE (Ours)
Val75/7 (71.4%)6/7 (85.7%)
Test167/16 (43.8%)10/16 (62.5%)

These premature give-ups mostly stem from phrasing failures. Removing report no mask allows the agent to continue searching, while LNE further improves recovery through alternative NP branches. However, this continued search can incur unnecessary computation when no target exists. This overhead is limited when no-target cases are rare but becomes important when they are frequent. We therefore evaluate no-target handling and the effect of the interaction budget on gRefCOCO in Response 3.

References

[R1] Woojun Jung, Jaehoon Go, Mingyu Jeon, Sunjae Yoon, and Junyeong Kim. Visual Funnel: Resolving Contextual Blindness in Multimodal Large Language Models. CVPR Findings, 2026. https://arxiv.org/abs/2512.10362

AuthorsOpenReview ↗

Response to Reviewer tVYn (3/3)

2. Controlled Experiments on gRefCOCO (Q2):

Due to time constraints, we randomly sampled 3,000 cases from the gRefCOCO val split (62.6% no-target, 37.4% multi-target, no single-target; consistent with the full val split). Following the reviewer's suggestions, we compare:

  • Compact Four-Tool: retains all four tools with a simplified CoT template and a system prompt matched to ASPS in token length (within 5%). This isolates tool removal from prompt simplification as far as possible.
  • Adaptive Tools v1/v2: start from the Compact Four-Tool prompt with the two tools removed, and add each tool back only when a rule-based trigger fires:
    • examine each mask: following its original definition ("to distinguish overlapping mask(s), to examine whether very small mask(s) are correct, or both"), it is enabled when any returned mask covers <3% of the image, or SAM3's overlap deduplication removes candidate masks.
    • report no mask: since SAM3 Agent only allows segment phrase when the context contains a single image, we enable it v1: from the second round onward; v2: after two consecutive empty SAM3 returns.
Max RoundsMethodTokens ↓Rounds ↓Time (s) ↓gIoU ↑cIoU ↑N-acc ↑
10SAM3 Agent8294.5625.443.241.741.0
10Compact Four-Tool5403.6111.836.938.524.4
10Adaptive Tools v14473.019.648.137.743.5
10Adaptive Tools v25283.5612.144.738.038.7
10ASPS7805.5417.755.140.355.8
5ASPS5043.5911.458.941.362.5

On the evaluated gRefCOCO subset, ASPS achieves higher gIoU and N-acc than the original and compact four-tool agents, but requires more interaction rounds than the adaptive variants. Reducing the budget lowers latency and increases N-acc, highlighting the importance of budget selection for no-target handling.

We also ran these settings on ReasonSeg (Val + Test combined):

MethodTokens ↓Rounds ↓Time (s) ↓gIoU ↑
SAM3 Agent5452.5221.664.5
Compact Four-Tool3892.5014.266.7
Adaptive Tools v13882.4713.966.2
Adaptive Tools v23762.4914.666.2
ASPS3762.6014.667.2
Light SAM3 Agent3152.1912.269.5

Note: Due to limited compute availability, both experiments above were conducted on a single NVIDIA RTX PRO 6000D GPU.

These controls show that ASPS achieves higher gIoU and no-target accuracy than the four-tool and adaptive alternatives on a subset containing only no-target and multi-target expressions, while Light SAM3 Agent further improves both quality and efficiency on ReasonSeg.

Reviewer_CCZaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Light SAM3 Agent, a training-free framework for reducing the inference cost of agentic segmentation. Latent NPs Extraction (LNE) reuses a reasoning prefix and explores alternative noun phrases through first-token branching and greedy completion, testing candidates on demand with SAM3. Action Space Slimming (ASPS) retains two core tools and simplifies the associated instructions. Experiments on ReasonSeg and RefCOCOg report improvements in segmentation quality, decoded-token consumption, and inference latency, with additional component ablations, budget analyses, and evaluations across multiple MLLM backbones.

Critical strengths
  1. Meaningful improvements to a practical inference bottleneck. Without additional training, the method improves ReasonSeg test gIoU from 63.6 to 69.8 while reducing decoded tokens from 529 to 324. Improvements also extend to the three evaluated RefCOCOg splits. The measured latency distributions and evaluations across multiple backbones provide useful evidence that the practical gains are not confined to a single configuration.

  2. Useful ablations and failure analysis. The paper goes beyond aggregate performance by evaluating individual components, alternative NP-generation strategies, and interaction budgets. The supplementary analysis shows that LNE primarily improves accuracy by recovering searches that otherwise exhaust the budget, whereas simply allowing more reformulation rounds does not consistently provide comparable gains. This offers a useful insight into the limitations of the original agent.

Critical weaknesses
  1. The methodological advance and its position relative to closely related methods are not sufficiently established. LNE’s central operation --- branching at the first NP token and greedily completing each alternative --- has a close precedent in CoT-decoding [R1]. Its potentially valuable contribution is integrating shared-prefix reuse with failure-triggered generation and SAM3-based stopping, but the paper does not develop this distinction clearly enough. Describing the procedure as "latent" extraction or "probabilistic sampling" also overstates its difference from explicit, deterministic branch decoding. At the task level, the cited EVOL-SAM3 [R2] already explores alternative prompts through SAM3 feedback, yet it is absent from the experimental comparisons. Explicit-3NPs/5NPs do not represent its verification and evolutionary search. EcoAct, ReWOO, and LLMCompiler are useful general references, but do not resolve this task-specific comparison gap.

  2. The existing controls do not sufficiently separate the proposed design choices from other changes in execution. For LNE, the explicit 3/5-NP comparisons simultaneously change candidate generation, candidate budget, and eager versus on-demand execution. Random LNE tests ordering within the same candidate-generation mechanism; it does not compare alternative generators under identical prefix reuse, budgets, and stopping conditions. Its test gIoU of 69.5 versus 69.8 for ranked LNE, alongside higher token consumption of 339 versus 324, suggests a useful ordering benefit but does not establish the independent advantage of the branching mechanism. For ASPS, tool removal is accompanied by changes to the instructions and CoT template, which the single-tool ablations do not disentangle. Figure 2’s invocation-conditioned IoU also cannot establish low tool utility independently of sample difficulty. Finally, the supplementary implementation includes conversation pruning and SAM3 image-feature caching, but it is unclear which baselines share these mechanisms.

[R1] Xuezhi Wang and Denny Zhou. Chain-of-Thought Reasoning Without Prompting. NeurIPS, 2024. https://arxiv.org/abs/2402.10200

[R2] Kai Ye, Xiaotong You, Jianghang Lin, Jiayi Ji, Pingyang Dai, and Liujuan Cao. Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Prompting. arXiv:2512.24702, 2025. https://arxiv.org/abs/2512.24702

Questions for authors
  • Referring to Weakness 1: What is the specific methodological advance over existing branching and prompt-search approaches, and how does Light SAM3 Agent compare with EVOL-SAM3 in terms of quality and execution cost? Please explain the relevant trade-offs and why it was omitted from the comparisons.

  • Referring to Weakness 2: Which prompt, history-management, caching, and budget settings are shared across the compared methods, and which existing results isolate candidate generation from scheduling and tool removal from prompt simplification? Please also clarify the adaptations to the external efficiency baselines, the measurement boundaries, and the absolute runtimes underlying Figure 4.

7 comments
AuthorsOpenReview ↗

Response to Reviewer CCZa (1/5)

W1 & Q1: [Methodological Advance over CoT-Decoding and Comparison with EVOL-SAM3]

Thank you for your constructive comments. Below are our responses to your concerns.

1. Distinction from CoT-Decoding [R1] and Methodological Contribution:

In essence, CoT-Decoding [R1] is a text-only, open-loop decoding strategy designed to elicit intrinsic reasoning paths without prompting, whereas LNE is an agentic, closed-loop strategy designed to bridge the phrasing–grounding gap between an MLLM and an external visual tool. Specifically, LNE differs from CoT-Decoding in three key aspects:

  1. Branching Position:

    • CoT-Decoding branches at the very first token of the entire output, decoding each resulting response to completion.
    • LNE branches specifically at the first NP token (t⋆t^\star) in the fixed CoT field (Appendix A.1), reusing the shared reasoning prefix to decode only short candidate NPs and avoiding the costly generation of multiple full reasoning chains.
  2. Triggering Mechanism:

    • CoT-Decoding is eager: it must fully decode all kk reasoning paths upfront before scoring and aggregating answers.
    • LNE is on-demand and driven by external tool feedback: a new NP branch is decoded only when the previous candidate fails to be grounded by SAM3.
  3. Stopping and Verification Criterion:

    • CoT-Decoding relies on internal model confidence (token probability margins or confidence-weighted voting across all kk completed paths), which provides no external correctness guarantee.
    • LNE couples branching with external grounded verification: SAM3 provides deterministic execution feedback for each candidate NP and terminates the search immediately once a valid mask is returned.

In addition, we hope LNE offers a broader insight for agent efficiency optimization: in agent workflows that invoke parameter-sensitive tools with deterministic feedback (e.g., search agents where a query yields zero hits), localized parameter-level branching from a frozen reasoning state (1) eliminates redundant multi-turn MLLM re-reasoning, and (2) prevents failed attempts from accumulating in the context and degrading subsequent decisions.

2. Terminology Clarification ("Latent" & "Probabilistic Sampling"):

In the revised paper, we have replaced "probabilistic sampling" with "probability-guided branching" to avoid overstating its difference from deterministic branch decoding. We retain "latent" to describe the candidate NPs, indicating that alternative NPs originate from the next-token vocabulary distribution without being explicitly generated in the MLLM's CoT reasoning text. We have also revised the second paragraph of Section 3.2 to further clarify LNE’s methodological contribution.

AuthorsOpenReview ↗

Response to Reviewer CCZa (2/5)

3. Comparison with EVOL-SAM3 [R2]:

While EVOL-SAM3 also explores alternative prompts via SAM3 feedback, its feedback loop requires invoking the MLLM to visually judge and evolve candidate masks across multiple turns. In contrast, LNE only checks whether SAM3 returns a non-empty mask, requiring no extra MLLM calls.

To clarify and further demonstrate LNE's advantages in both performance and cost, we conducted two comparisons on ReasonSeg following the reviewer's request:

  • NP-Search Mechanism Comparison (ASPS + Evol-search vs. Light SAM3 Agent): ASPS + Evol-search replaces LNE's local NP recovery with EVOL-SAM3's text-based evolutionary search (population size 4; up to 2 evolutionary updates). It retains confidence/semantic gating, visual arena selection, and semantic mutation, while excluding the geometric box branch and final text–box arbitration.
SplitMethodTokens ↓Rounds ↓Extra MLLM Calls ↓Time (s/sample) ↓gIoU (%) ↑cIoU (%) ↑
ValASPS + Evol-search3512.392.2314.870.558.7
ValLight SAM3 Agent3142.19—10.670.558.5
TestASPS + Evol-search3552.401.9519.068.456.1
TestLight SAM3 Agent3152.19—12.669.257.5
  • End-to-End Quality and Cost Comparison (Evol-SAM3 vs. Light SAM3 Agent):
SplitMethodTokens ↓MLLM Calls ↓Time (s/sample) ↓gIoU (%) ↑cIoU (%) ↑
ValSAM3 Agent5422.5020.065.558.1
ValEvol-SAM36110.0326.470.267.7
ValEvol-SAM3 w/o Intuition Elite Generation377.1417.369.360.8
ValLight SAM3 Agent3142.1910.670.558.5
TestSAM3 Agent5462.5322.064.357.0
TestEvol-SAM3609.9927.972.466.4
TestEvol-SAM3 w/o Intuition Elite Generation367.1218.169.762.8
TestLight SAM3 Agent3152.1912.669.257.5

Note: Due to limited compute availability, both experiments above were conducted on an NVIDIA RTX PRO 6000D GPU.

Under the same ASPS pipeline, LNE matches Evol-search on Val gIoU and outperforms it on Test (+0.8 gIoU, +1.4 cIoU), while requiring no extra MLLM calls and running 1.40× / 1.51× faster. Against the full EVOL-SAM3 pipeline, EVOL-SAM3 achieves higher accuracy, but at the cost of ~10 MLLM calls per sample and 2.5× / 2.2× higher latency than Light SAM3 Agent (even slower than the original SAM3 Agent). EVOL-SAM3 generates short outputs across multiple calls, resulting in fewer decoded tokens but higher end-to-end latency due to repeated multimodal prefilling, segmentation, and verification.

For a fairer comparison, we also disabled Intuition Elite Generation in EVOL-SAM3. This variant shows no consistent gIoU advantage over Light SAM3 Agent, although it retains higher cIoU. Its end-to-end latency remains approximately 1.6×/1.4× that of Light SAM3 Agent on Val/Test.

Why EVOL-SAM3 was not included in the main comparisons. We regard EVOL-SAM3 as complementary rather than a direct competitor, for three reasons:

  1. Paradigm mismatch: EVOL-SAM3 is a test-time scaling framework that reformulates reasoning segmentation as an evolutionary search, relying on a hand-crafted multi-stage pipeline (meta-planning, semantic gating, pairwise visual arenas, and semantic mutation). In contrast, our work optimizes the existing ReAct workflow without introducing additional complex reasoning steps.
  2. Different optimization objectives: EVOL-SAM3 aims to raise the upper bound of zero-shot segmentation, essentially trading additional computation for higher accuracy. Our core objective is to improve the efficiency and reduce the interaction burden of the native SAM3 Agent; the accuracy gains are a by-product of this efficiency-oriented design, arising from removing redundant context, reducing tool misuse, and recovering failed searches.
  3. Asymmetry in prompt modalities (pure concept text prompts vs. hybrid geometric box prompts): Both SAM3 Agent and Light SAM3 Agent strictly interact with SAM3 through pure concept text prompts (NPs). In contrast, EVOL-SAM3's final Heterogeneous Arena additionally prompts the MLLM to predict bounding boxes and feeds them as geometric box prompts into SAM3 (with fallback priority to the box-derived mask), introducing an extra spatial-localization modality beyond text-prompted agentic segmentation.
AuthorsOpenReview ↗

Response to Reviewer CCZa (3/5)

W2 & Q2: [Disentangling Design Choices and Experimental Controls]

Thank you for these suggestions. We provide additional controlled experiments and clarify the implementation details below.

1. Isolating Candidate Generation, Candidate Budget, and Eager vs. On-Demand Execution in LNE:

We clarify that Explicit-3/5NPs were not intentionally designed with mismatched candidate budgets. Under the experimental setup of Table 3, we further evaluated Explicit-7/9NPs and observed clear performance saturation and degradation: inspecting the generation logs revealed that prompting the MLLM to output more candidates upfront in a single turn leads to repetitive phrasings and lower NP diversity.

To further isolate candidate generation under a matched budget (K=15K=15) and on-demand execution, we implemented On-demand NP Rewriting (ODR). When SAM3 returns no mask, a fresh MLLM session uses the image, question, and previously failed NPs to generate a new NP for the same target. This process repeats until SAM3 returns a non-empty mask or the candidate budget is exhausted. The results on ReasonSeg are as follows:

MethodVal Tokens ↓Val Rounds ↓Val gIoU ↑Test Tokens ↓Test Rounds ↓Test gIoU ↑
Explicit-3NPs3922.2169.04152.4166.2
Explicit-5NPs3992.2168.34032.2467.1
Explicit-7NPs4212.2666.34232.2766.3
Explicit-9NPs4172.1966.64202.2266.2
ODR (On-demand, K=15K=15)3512.4071.23792.4068.4
Light SAM3 Agent (Ours)3152.1970.23242.2569.8

Note: This experiment was conducted on an NVIDIA RTX A6000 GPU. ODR rewriting contributes to Tokens but not Rounds.

Under the same K=15K=15 budget and on-demand execution, Light SAM3 Agent improves Test gIoU over ODR by 1.4% with 14.5% fewer tokens.

2. Comparing Alternative Candidate Generators under Identical Prefix Reuse, Budgets, and Stopping Conditions:

The reviewer notes that Random LNE only tests ordering within the same generation mechanism. We therefore compare LNE against standard decoding strategies (Top-pp, Top-kk, and Beam Search) under strictly identical templates, KV cache reuse, deduplication rules, candidate budget (K=15K=15), and early-stopping rules on ReasonSeg. The paper uses greedy decoding for LNE for reproducibility; for completeness, we also implement LNE variants with Top-pp and Top-kk:

MethodVal gIoU (%) ↑Val cIoU (%) ↑Val Time (s) ↓Val Rounds ↓Val Tokens ↓Test gIoU (%) ↑Test cIoU (%) ↑Test Time (s) ↓Test Rounds ↓Test Tokens ↓
Top-p70.258.411.52.3334468.956.713.32.27337
Top-k70.158.312.82.3434569.056.714.22.23328
Beam Search69.656.511.92.3462868.055.713.32.24511
LNE (Top-p)70.058.811.82.1831269.558.913.52.18314
LNE (Top-k)70.458.710.52.1731069.558.512.62.19314
LNE (Greedy)70.558.510.62.1931469.257.512.62.19315

Note: Due to limited compute availability, this experiment was conducted on an NVIDIA RTX PRO 6000D GPU.

Reviewer_T25xOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is targeting to improve the efficiency of agentic segmentation based on SAM3 Agent workflow. The authors firstly study the interaction process of SAM3 Agent, and identify two redundancies which are additional MLLM loop and less effective actions. Based on these observations, the authors propose a lightweight SAM3 Agent pipeline with two designs targeting to solve the two overheads. Latent NPs Extraction (also known as LNE in this paper) reuses the same reasoning prefix and generated distribution to generate alternative phrase candidates when the initial selection fails, avoiding repeated full MLLM reasoning. Action Space Slimming (also known as ASPS) removes two actions from the original SAM3 Agent action space and simplifies the corresponding action instructions. Main experiments include ReasonSeg and RefCOCOg with different MLLM backbones. The authors report improvements in efficiency and segmentation quality, but these improvements are not consistent across all settings.

Critical strengths
  • The observations and the overall design make sense.The paper identifies two practical sources of overhead in SAM3 Agent, and the two designs built around these observations are reasonable and well connected to the problems. In particular, I think LNE directly addresses a core bottleneck in SAM3 Agent: repeating expensive MLLM reasoning. Reusing the same reasoning prefix to explore alternative candidates is a reasonable way to avoid this repeated computation while still allowing the agent to try more choice.
  • The experiments are fairly comprehensive and clearly show the trade-offs between different designs. Although the method is not consistently better in every setting, the paper reports both improvements and regressions, making it clear where each design helps and where it does not.
Critical weaknesses
  • The motivation and experimental evidence for ASPS are not convincing enough. Figure 2 reports metrics when each action is used, but this is not a causal experiment showing that these two actions (examine mask and no mask) should be removed. I do not think a lower IoU necessarily means that an action is useless, as there could be other explanations such as the action being used more often on difficult tasks. Also, line 291 mentions that the action instructions are simplified at the same time. So the final results actually contains two modification: (1) removal of two actions, (2) change of prompt. Given the current experiment, it is hard for me to distinguish which change actually contributes to the improvement.
  • Although LNE makes sense in this case, its novelty is limited and its connection to existing work is not clearly discussed. As far as I know, CoT-Decoding was also doing very similar thing. It's core idea is also reusing the prefix and decoding with altenative topK candidates. LNE seems like applying this strategy directly on the SAM3 Agent scenario. The authors should discuss this connection and clarify what LNE contributes beyond adapting an existing decoding strategy to SAM3 Agent.
Questions for authors
  1. I think the efficiency analysis in Section 3.2 is a bit too optimistic and does not discuss the worst case. In the worst case, we should reasonably assume that it requires the same number of full MLLM decoding rounds as the original SAM3 Agent, but with many additional SAM3 calls. The existing analysis does not discuss this case. Could the authors also report whether this case happens and how often in the actual evaluation?
  2. Tables 1 and 2 show that Light SAM3 Agent performs better than SAM3 Agent in terms of IoU. I think this needs a more insightful explanation. Intuitively, ASPS removes some potentially helpful actions to improve system efficiency, so I would expect that Light SAM3 Agent might perform worse than SAM3 Agent. However, the actual results are better. Can the authors should provide experiments to identify where these gains come from?
4 comments
AuthorsOpenReview ↗

Response to Reviewer T25x (1/2)

W1 & Q2: [Causality of ASPS and Sources of Performance Gains]

1. Understanding ASPS’s IoU Gains (W1 & Q2):

Thank you for raising this concern. We agree with the reviewer that Figure 2 shows correlation rather than strict causality, and that selection bias exists on hard cases.

ASPS aims to reduce decision complexity and interaction overhead while retaining the two core actions. To intuitively demonstrate the advantage of this design, we conducted a granular statistical analysis of the two pruned actions on ReasonSeg (corresponding to Table 1), comparing SAM3 Agent, ASPS, and Light SAM3 Agent:

  • Analysis 1: On examine each mask

    Across all 6 ReasonSeg cases where SAM3 Agent invoked this tool, replacing it with ASPS (direct mask observation in select masks and return) improved 3/6, left 2/6 unchanged, and degraded 1/6. Thus, examine each mask shows no clear benefit even on the hard cases where it is invoked; instead, 2 of the 3 improved cases originally got 0 IoU due to execution errors in the additional examination session.

  • Analysis 2: On report no mask

    In SAM3 Agent, report no mask can lead to false negatives by prematurely abandoning the search when targets exist. The table reports recovery by ASPS and ASPS + LNE on these cases from Table 1, excluding budget-exhaustion failures.

    SplitFalse-Negative "No Mask" CallsRecovered by ASPSRecovered by ASPS + LNE (Ours)
    Val75/7 (71.4%)6/7 (85.7%)
    Test167/16 (43.8%)10/16 (62.5%)

    Most premature give-ups stem from phrasing failure rather than target absence, which ASPS and LNE's alternative NPs effectively recover.

Removing the two tools also yields a bonus: it enables a streamlined system prompt and a shorter context (forming ASPS). Appendix D.1 (Table 6) confirms that ASPS's gIoU gains (+4.4% Val, +3.7% Test) stem from improved predictions on normal, non-exhausted cases (B: +5.3% / +4.2%) rather than fewer budget-exhaustion failures (A: -1.0% / -0.5%).

2. Disentangling Action Removal from Prompt Simplification (W1): To distinguish action removal from prompt simplification, we evaluated:

  • Compact Four-Tool: retains all four tools with a simplified CoT template and a system prompt matched to ASPS in token length (within 5%).
  • Adaptive Tools v1/v2: start from Compact Four-Tool without the two tools, re-enabling them on demand via rule-based triggers (examine each mask for <3% area or overlap-deduplicated masks; report no mask from Round 2 in v1 or after two consecutive empty returns in v2).

We evaluated these settings on ReasonSeg (Val + Test combined) and on 3,000 cases randomly sampled from gRefCOCO Val (62.6% no-target, 37.4% multi-target, matching the full split ratio):

Results on ReasonSeg:

MethodTokens ↓Rounds ↓Time (s) ↓gIoU ↑
SAM3 Agent5452.5221.664.5
Compact Four-Tool3892.5014.266.7
Adaptive Tools v13882.4713.966.2
Adaptive Tools v23762.4914.666.2
ASPS3762.6014.667.2
Light SAM3 Agent (ASPS + LNE)3152.1912.269.5

Results on gRefCOCO Val (3,000 sampled cases):

Max RoundsMethodTokens ↓Rounds ↓Time (s) ↓gIoU ↑cIoU ↑N-acc ↑
10SAM3 Agent8294.5625.443.241.741.0
10Compact Four-Tool5403.6111.836.938.524.4
10Adaptive Tools v14473.019.648.137.743.5
10Adaptive Tools v25283.5612.144.738.038.7
10ASPS7805.5417.755.140.355.8
5ASPS5043.5911.458.941.362.5

Note: Due to limited compute availability, both experiments above were conducted on a single NVIDIA RTX PRO 6000D GPU.

On ReasonSeg, shortening the four-tool prompt alone captures most of ASPS's gain, with action removal adding only a small increment. On gRefCOCO, the same compact prompt collapses, whereas removing the two actions restores and surpasses the baseline. Prompt shortening drives the efficiency and most of the ReasonSeg gain, while action removal keeps the compact prompt viable on no-target-heavy data.

AuthorsOpenReview ↗

Response to Reviewer T25x (2/2)

3. Additional IoU Gains from More Effective NP Exploration (Q2): We conducted a paired comparison between ASPS (w/o LNE) and ASPS+LNE based on Table 3 on ReasonSeg (200 Val and 779 Test samples) under identical prompts and a 10-round budget, tracking raw IoU changes without thresholds:

SplitImprovedUnchangedDegraded
Val1916813
Test6865853

ASPS+LNE improves gIoU by 1.2%/2.5% on Val/Test. Appendix B (Table 4) attributes most of these gains to recovering budget-exhaustion failures.

W2: [Novelty of LNE and Distinction from CoT-Decoding]

Thank you for the valuable opportunity to clarify LNE's novelty and difference from CoT-Decoding. In essence, CoT-Decoding is a text-only, open-loop decoding strategy designed to elicit intrinsic reasoning paths without prompting, whereas LNE is an agentic, closed-loop strategy designed to bridge the phrasing–grounding gap between an MLLM and an external visual tool. Specifically, LNE differs from CoT-Decoding in three key aspects:

  1. Branching Position:

    • CoT-Decoding branches at the very first token of the entire output, decoding each resulting response to completion.
    • LNE branches specifically at the first NP token (t⋆t^\star) in the fixed CoT field (Appendix A.1), reusing the shared reasoning prefix to decode only short candidate NPs and avoiding the costly generation of multiple full reasoning chains.
  2. Triggering Mechanism:

    • CoT-Decoding is eager: it must fully decode all kk reasoning paths upfront before scoring and aggregating answers.
    • LNE is on-demand and driven by external tool feedback: a new NP branch is decoded only when the previous candidate fails to be grounded by SAM3.
  3. Stopping and Verification Criterion:

    • CoT-Decoding relies on internal model confidence (token probability margins or confidence-weighted voting across all kk completed paths), which provides no external correctness guarantee.
    • LNE couples branching with external grounded verification: SAM3 provides deterministic execution feedback for each candidate NP and terminates the search immediately once a valid mask is returned.

In addition, we hope LNE offers a broader insight for agent efficiency optimization: in agent workflows that invoke parameter-sensitive tools with deterministic feedback (e.g., search agents where a query yields zero hits), localized parameter-level branching from a frozen reasoning state (1) eliminates redundant multi-turn MLLM re-reasoning, and (2) prevents failed attempts from accumulating in the context and degrading subsequent decisions.

Q1: [Empirical Occurrence of Worst-Case Scenarios and SAM3 Costs]

Thank you for highlighting this important worst-case scenario. We agree that the theoretical worst case is exactly as the reviewer described. In practice, necessary system prompt updates for ASPS and LNE introduce slight behavioral variance, occasionally causing rounds to exceed the baseline. We thus report both the requested same-round case and the broader setting (same or more rounds), plus average SAM3 calls on the ReasonSeg benchmark:

SplitMethodAvg. SAM3 CallsSame rounds, more SAM3 callsSame or more rounds, more SAM3 calls
ValSAM3 Agent1.30——
ValLNE1.633.00%5.50%
ValLight SAM3 Agent1.855.00%9.50%
TestSAM3 Agent1.32——
TestLNE1.513.59%4.62%
TestLight SAM3 Agent1.735.65%8.99%

Overall, the worst-case occurrence is low, with a minor increase in average SAM3 calls. Although SAM3 cost is relatively small, for completeness we optimized it by caching image embeddings within each case to avoid repeated image encoding, with the measured latency reported below:

SettingTotal time (s)
One image, one query0.236 ± 0.012
One image, ten queries1.052 ± 0.070

As shown above, each subsequent SAM3 query takes only ~0.09s (vs. ~0.24s for the first query), which further minimizes the overhead of additional SAM3 invocations in the aforementioned worst-case scenario.

Thanks for the response. I think most of the questions are solved. However, the two weakness are not fully addressed.

For the first weakness, i think the current two analysis are relying on a relative small samples, like analysis 1 is using 6 failure samples and analysis 2 is using 23 samples. I'm worried about whether this is statistical significant or not. Meanwhile, Compact Four-Tool gives two different directions given the results. On ReasonSeg it improves the result, but on gRefCOCO it degrades the results. I cannot judge which component works in this case.

For the second weakness, given the explanation, looks like the novelty is only on how to apply CoT-Decoding on this SAM3 agent scenario. It is good that the authors connect those two fields, but it means no algorithm side contribution to be honest.

Given the current responses, i actually want to downgrade my rating. I think a middle point between 2 and 3 will be fair for this paper. I'll keep the rating as 3 given the quality and reasonable application, but let the AC to make the final decision. To AC: I would recommand 2 if we care more about algorithm side contribution.

Thank you for your further feedback. We would like to clarify the scope of the two analyses: the six cases cover all invocations of examine each mask by the original SAM3 Agent on ReasonSeg Val/Test, rather than six failure cases. We analyzed all these cases: compared with the original SAM3 Agent, ASPS improved three, left two unchanged, and degraded one. The 23 cases are false-negative report no mask decisions when the target is present, rather than all invocations of this tool. Regarding your concern about statistical significance, as shown in Figure 2, both tools are invoked relatively infrequently by SAM3 Agent, resulting in a limited number of cases for these analyses. These analyses explain specific failure modes and how ASPS improves these cases.

The opposite trends you noted concern prompt simplification alone, i.e., SAM3 Agent → Compact Four-Tool. In contrast, further removing the two actions under the compact prompt, i.e., Compact Four-Tool → ASPS, improves gIoU on both datasets: from 66.7% to 67.2% on ReasonSeg, and from 36.9% to 55.1% on gRefCOCO. These results indicate that the effect of prompt simplification varies across datasets, while action removal provides additional gains on both. Thus, prompt simplification alone does not consistently reproduce the benefits of ASPS across datasets.

Thank you for recognizing the quality and practical value of our work and for maintaining your current rating. We respect your considerations regarding algorithmic novelty and hope these clarifications help you and the AC assess our contribution in light of the method design and its demonstrated gains in inference efficiency and segmentation quality.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank all reviewers for their constructive suggestions and valuable insights. Below is a summary of our response.

We are encouraged by the reviewers’ recognition of the following strengths.

  1. Motivation and Method Design. The paper identifies two practical sources of overhead in SAM3 Agent and addresses them through more efficient noun-phrase search and a simpler two-tool workflow (T25x). When the agent identifies the right object but SAM3 cannot ground its phrasing, LNE can try another phrase from the same reasoning prefix, avoiding a full reasoning round for each wording change (T25x, tVYn). ASPS retains only phrase segmentation and mask selection.
  2. Efficiency Gains and Experimental Validation. With the same backbone and segmenter, Light SAM3 Agent raises ReasonSeg test gIoU from 63.6 to 69.8, reduces decoded tokens by 38.8%, and achieves a 2.11× mean per-sample speedup (CCZa, tVYn). The component ablations clarify where each design helps (T25x), and the failure analysis shows that LNE improves gIoU primarily by recovering searches that would otherwise exhaust the interaction budget (CCZa).

We have carefully considered all reviewers' suggestions, performed additional experiments, and provided detailed clarifications to address the raised concerns. A summary follows.

  1. Clarifying LNE’s novelty and its distinction from CoT-Decoding. CoT-Decoding uses text-only, open-loop decoding to elicit alternative reasoning paths. LNE uses a closed loop between the language model and SAM3 to find a phrase that grounds the intended object. (T25x W2, CCZa W1/Q1)

    • Branching position. CoT-Decoding branches at the first token of the entire response. LNE branches at the first token of the noun phrase, reuses the preceding reasoning, and decodes only a short alternative phrase.
    • Triggering mechanism. CoT-Decoding generates its candidate paths upfront. LNE tries another phrase only when SAM3 returns no mask for the previous one.
    • Verification and stopping. CoT-Decoding scores completed paths using the model’s confidence in their answers. LNE uses SAM3’s output as the stopping signal. The first nonempty mask set ends the phrase search and goes to the agent for selection.
    • Broader insight for agent efficiency. When a tool gives a clear failure signal, an agent can retry its argument from the same reasoning state. This saves repeated reasoning and keeps failed attempts out of the context for later decisions. A search agent reformulating a query after an empty result is one potential application.
  2. Requests to compare LNE with alternative search methods.

    • Decoding and phrase rewriting. We added controlled comparisons to distinguish the benefit of reusing the reasoning prefix from the benefit of ranking candidate phrases. Reuse makes a failed phrase cheaper to rewrite, and ranked branching further reduces decoding cost under matched search conditions. (CCZa W2/Q2, tVYn W1/Q1, Appendix A)
    • EVOL-SAM3. We compared the phrase-search methods within the same agent and then compared the complete systems. LNE recovers failed phrases without extra language-model calls. Full EVOL-SAM3 achieves higher test accuracy, while Light SAM3 Agent uses fewer language-model calls and returns a prediction sooner. (CCZa W1/Q1, Appendix C.2)
  3. Concerns about the respective contributions of prompt simplification and tool removal. We separated these two changes in Action Space Slimming (ASPS). Prompt simplification accounts for most of the gain on ReasonSeg. Under a compact prompt, removing the two actions further improves segmentation on ReasonSeg and has a larger effect on gRefCOCO. The no-target and multi-target evaluation also clarifies how the interaction budget affects empty-mask predictions. (T25x W1, CCZa W2/Q2, tVYn W2/Q2, Appendix D.2)

  4. Requests to explain the gains in segmentation quality and end-to-end efficiency.

    • Segmentation quality. LNE recovers searches that would otherwise exhaust the interaction budget. ASPS improves predictions on cases that finish within budget and helps recover targets that the original agent prematurely declared absent. (T25x Q2, tVYn W2, Appendices B and D.1)
    • End-to-end efficiency. We clarified how latency is measured from the start of agent execution to the final mask, including all SAM3 retries. We also reported how often extra tool calls occur and how image-feature caching reduces their cost. (T25x Q1, CCZa Q2)

We have incorporated the relevant experimental results into the revised manuscript and marked the changes in red.

Although we did not have enough time to discuss fully with the reviewers, we believe our responses substantially address the raised concerns, and the additional experiments have strengthened our work considerably. We sincerely thank the reviewers for their insightful feedback and the ACs for their dedication and guidance throughout the review process.

AuthorsOpenReview ↗

Follow-up on the Reviewers’ Comments

We thank the reviewers for continuing the discussion after reading our responses. We address two questions: which change accounts for the ASPS gains, and what LNE contributes to agent inference beyond existing first-token branching methods.

1. How much of the ASPS gain comes from prompt simplification and how much from tool removal? (Reviewers T25x and tVYn)

ASPS removes the mask-examination and no-mask actions and simplifies the tool instructions and reasoning template accordingly. To distinguish the contributions of these changes to gIoU, we added the Compact Four-Tool control during the rebuttal. It retains all four original tools, uses a simplified reasoning template, and has a system prompt comparable in length to that of two-tool ASPS. Comparing this control with the original agent shows the effect of prompt simplification alone. Comparing it with ASPS shows what tool removal adds (Appendix D.2, Tables 11–12).

  • On ReasonSeg, prompt simplification accounts for most of the gain. With all four tools retained, Compact Four-Tool already captures most of the gIoU improvement that ASPS achieves over the original agent. Removing the two actions yields a small further increase.
  • On the gRefCOCO validation subset, tool removal puts ASPS ahead of the original agent. Simplifying the prompt alone lowers gIoU below the original agent’s. Removing the two actions recovers that loss and raises gIoU above the original baseline.

Prompt simplification relies on manual design. Our experiments show that, starting from the same base prompt, removing the two tools improves gIoU on both datasets.

2. What is LNE’s methodological contribution beyond CoT-Decoding?

In the follow-up discussion, tVYn noted that the controlled comparisons largely addressed their concerns about LNE and increased their confidence in its practical benefits. T25x and CCZa continued to express reservations about LNE’s methodological contribution beyond CoT-Decoding. We would like to clarify four points.

  1. LNE was designed to improve inference efficiency. We first identified redundancy in the SAM3 Agent loop and addressed it on the MLLM side by reusing the reasoning prefix to retry noun phrases. Developing a new general-purpose sampling algorithm was outside the scope of this work.

  2. First-token branching achieved the best combination of accuracy and efficiency among the methods compared. With cache reuse, candidate budgets, and stopping conditions matched, the default greedy LNE achieved the highest validation gIoU on ReasonSeg. On the test set, LNE with top-k completion tied for the highest gIoU, the fewest decoded tokens, and the lowest end-to-end latency (Appendix A.3, Table 4).

  3. SAM3 feedback governs on-demand branching and early stopping. LNE tries another phrase only when SAM3 returns no mask and stops the phrase search as soon as SAM3 returns a nonempty mask set.

  4. A more effective and efficient interaction paradigm for the MLLM and SAM3. The MLLM uses SAM3’s feedback to revise the phrase it sends for segmentation while preserving its interpretation of the target.

We sincerely thank all three reviewers for taking part in the follow-up discussion and for the feedback that helped us further strengthen this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.