acceptodds
Withdrawn from ICLR 2027

SAGE-Grasp: An End-to-End Target-Driven 6-DoF Grasping Framework via Cross-Modal Semantic Addressing and Gumbel-Routed Experts

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language-conditioned 6-DoF grasping in clutter requires identifying the referred object while generating physically feasible grasps. Existing task-agnostic detectors disregard semantic intent, whereas segment-then-grasp pipelines are susceptible to upstream segmentation errors and often discard useful contextual information. Thus, we propose SAGE-Grasp, which establishes a unified semantic-geometric representation that couples target understanding with full-scene physical reasoning. Cross-Modal Semantic Addressing (CMSA) and a Gumbel-routed mixture-of-experts architecture provide complementary mechanisms for semantic grounding and expert-routed grasp prediction, enabling the model to focus on the instructed target while retaining the context required for feasible grasping. SAGE-Grasp is jointly trained with text, visual, and zero queries, enabling target-aware semantic grounding while preserving task-agnostic geometric affordances. Experiments on a target-driven extension of GraspNet-1Billion demonstrate stronger target adherence than cascaded baselines and competitive scene-level grasping performance. These results indicate that target-conditioned grasping can be integrated into full-scene prediction while retaining task-agnostic scene-level capability.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Language-Conditioned Grasping, 6-DoF Grasp Detection, Target-Conditioned Grasp Prediction, Semantic-Geometric Representation, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CHzGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies 6-DoF grasping of targets specified by text or visual queries in cluttered scenes. SAGE-Grasp combines frozen SAM-CLP semantic priors with full scene 3D geometric features, and predicts target related grasps through a hybrid expert grasping head consisting of query aware candidate sampling, CMSA, and Gumbel-routed MoE. The model combines text, visual, and zero query training to simultaneously support target conditional grasping and task independent grasping. The validation scope is a fixed set of 88 object description vocabulary and visual queries within the scene, which does not yet include real robot experiments.

Critical strengths
  1. In Table 1, text queries on Seen HR@1 From the strongest listed baseline of 12.75% to 37.04%, SGSR increased from 19.21% to 43.51%. There have also been improvements on Similar and Novel. CF@1 Simultaneously improving indicates that the results involve the geometric feasibility of grasping candidates, rather than just target localization. 
  2. Unified support for target conditions and task independent prediction has systematic value. The same model supports text, in scene vision, and zero query. Under zero query, 60.50, 51.99, and 23.16 APs are obtained on three partitions. The author also explicitly discloses the boundaries of fixed vocabulary, cache apriori, and lack of robot execution verification, which helps to accurately understand the conclusions that experiments can support.
Critical weaknesses

W1. Compared to cascading baselines, this article simultaneously changes the scene context, candidate sampling, semantic condition training, and prediction head, making it difficult to determine where the main benefits come from. In Table 3, CMSA brings benefits on Seen HR@1 The gain is only 0.49 percentage points, while the SGSR gain brought by Novel is 0.52 percentage points. In addition, the control without MoE lacks capacity matching explanation, making it difficult to distinguish the roles of expert routing and additional parameter quantities. This limits the experimental support for the two main architectural contributions.

W2. Lack of real machine grasp experiments, and the verification of physical feasibility is still limited to offline proxy indicators. The existing results are based on GraspNet data and analysis of collision and force closure assessments, and the performance of predicted grasping on real robots has not been validated yet. Due to the emphasis on the physical feasibility in cluttered scenes and the role of preserving context in reducing collisions, whether real perception errors, calibration errors, and execution deviations will weaken these advantages will directly affect their practical value. The authors disclosed the limitations of the lack of robot execution validation in Section 5, but currently there is still a lack of evidence linking offline metrics improvement to actual target grasping success rates.

Questions for authors

Q1. Does the cascading baseline receive a target clipped point cloud or a complete scene? How to achieve target screening and confidence ranking? Can you provide a target screening comparison that uses the same geometric backbone, the same SAM-CLIP prior, and preserves all scene inputs to clarify the benefits of the unified prediction framework itself?

Q2. Does "w/o CMSA" simultaneously remove gating supervision and contrast loss? What alternative head is used for 'w/o MoE' and does the number of parameters match?

Q3. Under the same backbone and training budget, how much scene level AP can a model trained only with zero queries achieve?

Q4. The model methods compared in Table 2 are outdated, and some new methods have not been compared yet (such as EconomicGrasp, etc.)

2 comments

I thank the authors for their response. However, my concerns regarding the real-world robot experiments and the performance evaluation in Table 2 have not been satisfactorily addressed. Therefore, I will maintain my original score.

AuthorsOpenReview ↗

Additional Real-World Validation and Clarification of Scene-Level Performance.

Thank you for your continued feedback. We have further strengthened the manuscript by adding real-world robotic experiments and clarifying the evaluation results in Table 2.

Real-world validation. We have conducted additional physical grasping experiments using a Flexiv robotic manipulator equipped with an Intel RealSense D405 RGB-D camera and a parallel-jaw gripper. Four objects with different geometric properties were evaluated, with 10 trials per object. SAGE-Grasp achieved success rates of 100% on the banana, 80% on the strawberry, 100% on the medicine box, and 100% on the zebra-shaped toy, yielding an overall success rate of 95.0% (38/40). The experimental setup, quantitative results, and qualitative grasping sequences are provided in Appendix G. These additional experiments provide direct physical evidence of the feasibility of SAGE-Grasp in real-world robotic execution.

Scene-level evaluation (Table 2). We acknowledge that SAGE-Grasp currently underperforms the two additional baselines in scene-level grasp detection. However, the primary objective of SAGE-Grasp is not to surpass specialized task-agnostic detectors in scene-level AP, but to integrate target-conditioned grasp prediction with full-scene geometric reasoning within a unified framework, while retaining scene-level grasping capability. This distinction reflects the central motivation of SAGE-Grasp: enabling semantic target-conditioned grasping without abandoning general scene-level grasp prediction.

We appreciate the reviewer's feedback, which has helped us strengthen the physical evaluation and clarify the objectives and limitations of the proposed framework.

Reviewer_DvkROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SAGE-Grasp for target-conditioned 6-DoF grasping in cluttered scenes. It combines frozen SAM–CLIP semantic priors with 3D geometric features through query-aware candidate sampling, a Cross-Modal Semantic Addressing module (CMSA), and a Gumbel-routed mixture-of-experts prediction head. Joint training with textual, visual, and null queries supports both target-conditioned and scene-level grasping. Experiments on GraspNet-1Billion show substantial improvements over the reported target-conditioned baselines, while ablations demonstrate additional benefits from CMSA and the MoE head.

Critical strengths

1.The method addresses a practically relevant problem with a coherent design. Target-conditioned grasping requires identifying the intended object while accounting for surrounding geometry. SAGEGrasp incorporates target semantics into a full-scene prediction pipeline and supports both targetconditioned and target-agnostic operation within one model.

2.The reported results provide encouraging evidence for the overall framework. In Table 1, textualquery Seen HR@1 improves from 12.75% for EconomicGrasp to 37.04%, and SGSR improves from 19.21% to 43.51%. Table 3(a) also reports lower performance when either CMSA or the MoE head is removed across all three splits, although repeated training runs would be needed to establish the robustness of the smaller differences. The evaluation retains queries with no target seeds or no collision-free candidates and counts them as failures, which helps avoid overstating performance

Critical weaknesses

1.The comparisons do not fully separate the benefits of the proposed architecture from those of retaining full-scene geometry. A full-scene grasp detector followed by semantic target filtering or reranking would provide an important control. The adaptation of the cascaded baselines to cropped inputs is also insufficiently specified. In addition, Table 2 omits GSNet and EconomicGrasp, despite their relevance and the inclusion of EconomicGrasp in Table 1, leaving the claimed competitive scene-level performance incompletely supported.

2.The ablations support incremental benefits from CMSA and MoE, but do not establish their advantages over simpler alternatives. Comparing CMSA with simpler fusion while keeping other semantic pathways fixed would clarify its specific contribution. A capacity-matched prediction head would similarly help distinguish MoE routing benefits from additional capacity. Appendix B reports only one training run per configuration, leaving the robustness of smaller gains—such as the 0.49- point Seen HR@1 improvement from CMSA—uncertain.

Questions for authors

1.Were the cascaded baselines evaluated with original weights, retrained, or fine-tuned for targetcropped inputs? Could you add a full-scene grasp-generation baseline followed by target filtering or reranking, using comparable semantic information and proposal budgets? Including GSNet and EconomicGrasp in Table 2 would also clarify the scene-level comparison.

2.What exactly replaces CMSA and MoE in Table 3(a), and how are their associated losses handled? Could you compare against simpler semantic fusion and a capacity-matched standard prediction head, while reporting computational costs? Repeated training runs for the key comparisons would help assess the stability of the observed gains.

3.How is the target visual-query region selected among SAM proposals at test time: by user input, automatic retrieval, or matching to annotations? Please clarify how selection failures are handled and how the training region bank is separated from test data. These details are necessary to interpret the visual-query results under the correct input assumptions.

Reviewer_BWVNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses target-driven grasping in cluttered scenes and proposes a 6-DoF grasp prediction method conditioned on visual or text queries. Using semantic priors extracted by SAM and CLIP , the model combines target information with scene geometry to predict feasible grasps on the queried object. Rather than isolating the target before grasp prediction, the method predicts target-conditioned grasps within the full scene, retaining the surrounding geometric context.

Critical strengths

The authors equip several baselines with the same SAM–CLIP retriever, reducing the confounding effects of different upstream semantic models and making the comparison fairer.

Critical weaknesses
  1. Retaining full-scene geometry and using soft gating help mitigate the information loss caused by hard segmentation. However, it remains unclear how these choices improve robustness to segmentation errors. The method still relies on SAM–CLIP semantic priors, and preserving geometric context does not necessarily compensate for incorrect target identification when those priors are unreliable.

  2. The overview does not help me form a clear high-level understanding of the method. Instead, it introduces scattered implementation details before establishing the overall picture, making the approach harder to follow.

  3. In Sections 3.2–3.3, I can follow how the network is designed, but the reasons behind these design choices remain unclear. The network feels more like a carefully crafted artifact than a scalpel designed for the problem at hand. The authors should explain more clearly which difficulty each design addresses and why simpler alternatives would be insufficient.

  4. Section 3.5 explains the basic role of each loss term, but does not convincingly justify why these terms need to be combined in this particular way or how their combination addresses the specific challenges of the problem. Further explanation and experimental evidence are needed.

Questions for authors

The method still relies on SAM–CLIP semantic priors. Full-scene geometry and soft gating can reduce information loss from hard segmentation. However, when these priors are incorrect, what helps the model identify the correct target? Why is it more robust than modular methods that directly use segmentation results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.