acceptodds
Under review as a conference paper at ICLR 2027

SECOND-Grasp: Semantic Contact-guided Dexterous Grasping

Abstract

Achieving reliable robotic manipulation, such as dexterous grasping, requires a synergy between physically stable interactions and semantic task guidance, yet these objectives are often treated as separate, disjoint goals. In this paper, we investigate how to integrate physically stable dexterous grasping with language-guided grasp generation to enable robotic grasping that is both physically reliable and semantically aligned with task intent. To this end, we propose SECOND-Grasp (SEmantic CONtact-guided Dexterous Grasping), a unified framework that enables robotic hands to dynamically adjust grasping strategies based on semantic reasoning while ensuring physical feasibility. We begin by obtaining coarse contact proposals through vision-language reasoning to infer where contacts should occur based on object properties, followed by segmentation to localize these regions across views.To further ensure consistency across multiple viewpoints, we introduce Semantic-Geometric Consistency Refinement (SGCR), which refines initial contact predictions by enforcing semantic consistency across views and removing geometrically invalid regions, yielding reliable 3D contact maps. Lastly, we derive a feasible hand pose for each contact map via inverse kinematics, generating a supervision signal for policy learning. Our approach, trained on DexGraspNet, consistently outperforms baselines in lifting success rate on both seen and unseen categories, achieving 98.2% and 97.7%, respectively, while also improving intent-aware grasping by 12.8% and 26.2%. We further show that SECOND-Grasp realizes semantic intent through distinct, physically executable grasping strategies across diverse objects and robotic hand embodiments.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dexterous Grasping, Vision-Language Models, Reinforcement Learning, 3D Vision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_D9Y5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose SECOND-Grasp to combine semantical manipulation information from a VLM and the physical validity of a dexterous grasping policy to do semantical guided dexterous grasping. First, they use a VLM to propose multi-view contact proposals from object images. Second, they proposed a novel Semantic-Geometric Consistency Refinement method to post-process VLM generated multi-view contact proposals. Then, they use IK to get a feasible hand pose from the refined contact proposal, and train a goal conditioned RL policy to execute that proposal. Experiments in simulation verified that SECOND-Grasp improves grasping success rate and intention awareness.

Critical strengths

Strengths

  1. Writing is clear and easy to follow.
  2. Experimental verifications in simulation is sound. Performance improvement is consistent.
Critical weaknesses

Weaknesses

  1. No real-robot experiments.
  2. The strengths of the VLM could potentially affect the performance of the overall pipeline, but no experimental comparisons were carried out between different VLMs.

Minor typo (not a critical technical weakness): Intro cites DexGraspNet 2.0 (line 97), but actual experiments cite and are carried out on DexGraspNet 1.0. So the citation of DexGraspNet 2.0 in line 97 seems to be a typo, and could be moved to related works section instead.

Questions for authors

Questions

  1. How are hand-object collision and hand-hand self-penetration handled in section 4.3 (hand pose prior via inverse kinematics)?
  2. Is the grasp intention completely inferred from the object's appearance using the VLM, or do humans need to provide one or two sentences to tell the VLM which parts to grasp? More specifically, in the comparisons in Table 1, do all baselines receive the same amount of information, or does the author's method require additional human-language guidance? If there are multiple places to grasp (like a mug with two handles), is it possible to use human prompt to precisely steer the system to choose one place?
  3. How does the order of the K proposed graspable regions affect the multi-view refinement? For example, what if one view proposes two graspable regions, where No. 1 is the handle, No. 2 is the lid, while its neighbouring view proposes the opposite (No. 1 is the lid and No. 2 is the handle)? Does the cross-view accumulation know how to match this order-discrepancy?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer D9Y5 (1/2)

We sincerely thank the reviewer for their positive assessment and recommendation toward acceptance. We are glad that the reviewer found our paper clear and well-presented, with consistent performance improvements supported by sound experimental evaluations. We provide further clarifications to address their remaining questions below.


[W1] Real-robot experiments.

Although deployment in real robots has not yet been conducted, we have included in our initial submission, some supporting experiments through supplementary material to demonstrate the robustness of our framework under realistic conditions. (please see index.html for demo videos)

● Real-scene Contact Maps

We have evaluated SGCR on the WildRGB-D data, aiming to investigate the robustness of SGCR under sim-to-real domain shift. While the data contains noisy depth estimation and imperfect geometric observations compared to simulators, SGCR precisely refines and captures the contact map aligning with a given grasp intent. We have also provided visualization of contact maps at each stage of SGCR on this real-world date in Figure 6.

● Simulated Real-world Noise

We have further evaluated robustness under simulated real-world uncertainties by introducing three types of perturbations: (1) action execution noise, (2) control latency, and (3) contact dynamics variations. As reported in Table 13, although these perturbations inevitably degrade overall grasping performance, intent-aware grasping performance remains relatively robust, with only modest reductions in ISR.

Overall, these results suggest that SECOND-Grasp can preserve semantic grasping intent under realistic execution uncertainties, providing encouraging evidence for its potential applicability to real-world robotic systems.


[W2] Comparing different VLM backbones.

We conduct additional experiments with different VLM backbones for initial 2D contact region proposals. To systematically assess the sensitivity of our SECOND-Grasp to backbone selection, we compare our default Qwen3-VL-8B against two alternatives: (1) Qwen3-VL-4B, to examine the effect of model size within the same family, and (2) Qwen3.5-VL-9B, to examine the effect of model version at a comparable parameter scale. The results are summarized in the table below.

VLMBBox IoU ↑Unseen GSR ↑Unseen ISR ↑
Qwen3-VL-4B57.0294.9289.16
Qwen3.5-VL-9B38.8795.3572.09
Qwen3-VL-8B (Ours)51.6997.6588.87

(1) vs. Qwen3-VL-4B

Interestingly, Qwen3-VL-4B achieves higher BBox IoU (57.02 vs. 51.69) and slightly higher ISR (89.16% vs. 88.87%), despite its smaller parameter size. Although its GSR is somewhat lower (94.92% vs. 97.65%), these results suggest that larger VLM capacity is not necessarily required for accurate semantic contact localization or intent-aware grasping.

(2) vs. Qwen3.5-VL-9B

Despite its lower BBox IoU (38.87), Qwen3.5-VL-9B achieves comparable downstream GSR (95.35%). However, when evaluated under a metric that requires precise semantic and geometrical grounding (ISR), its performance is relatively low (81.40%), following the trend in BBox prediction. The result suggests that although our SGCR refines the contact maps, the initial visual grounding performance matters for accurate intent-aware grasping.

Overall, despite noticeable variations in BBox IoU across VLM backbones, SECOND-Grasp maintains consistently high GSR and ISR, suggesting robustness to backbone selection and variations in initial semantic localization quality.


+ Minor Regarding Citation. We thank the reviewer for catching this typo. The intended reference was DexGraspNet 1.0, which was used throughout our experiments, and we have corrected the citation in the revised manuscript.

AuthorsOpenReview ↗

Author Response to Reviewer D9Y5 (2/2)

[Q1] How are collisions and self-penetration handled during IK?

The IK stage (Sec. 4.3) does not explicitly enforce hand–object or self-collision constraints and thus does not guarantee collision-free poses. However, our PCA-based contact partitioning encourages structured fingertip placements that reduce hand–object penetration compared to vanilla IK (see our response "1. Grasp generation is not limited to semantic understanding" in W1-(a) to Reviewer m4iX). Importantly, the IK-derived poses serve as soft and flexible guidance rather than directly executable grasps, allowing the RL policy to learn physically stable grasping behaviors through simulation-based interactions.

To further validate this design, we compare our IK-derived priors against those from existing grasp generators under identical RL training settings (see our response "Pose Prior Effectiveness in Policy Learning (Q2)" in W3 to Reviewer m4iX). The results demonstrate competitive grasp success rates and higher intent success rates, supporting the effectiveness of our lightweight IK approach as guidance for learning physically stable, intent-aligned grasps.

[Q2] How are grasp intents inferred, and is guidance consistent across baselines?

We would like to clarify that during training, SECOND-Grasp uses VLM proposals to identify and localize plausible graspable object parts from multi-view images, without requiring human-language instructions. At test time, we evaluate two distinct settings:

1. Standard Grasp-and-Lift (Table 1): The policy learns grasping behavior through semantic contact and pseudo-pose supervision during training. At test time, the learned policy directly predicts grasping actions from object observations, without requiring additional VLM inference, contact maps, pseudo poses, IK, or human-language guidance (Appendix C.1). Thus, our method does not receive additional human-language guidance compared with the baselines.

2. Intent-Aware Grasping (Table 2, Fig. 4): We additionally support explicitly specified grasp intents, where users can indicate the desired object part through language, and the VLM localizes the corresponding region to guide grasp execution. When multiple similar parts exist (e.g., two handles), SGCR uses VLM confidence scores and cross-view consistency to identify a high-confidence seed region, followed by geometric refinement to obtain a localized contact target.

In summary, human-language guidance is optional and is used only for explicitly intent-conditioned grasping, not for the standard grasp-and-lift comparisons in Table 1.

[Q3] How does the order of the K affect the multi-view refinement?

We would like to clarify that SGCR processes each semantic part independently, rather than jointly refining multiple unordered region proposals. Specifically, the VLM identifies multiple graspable parts and localizes each part across views under a consistent semantic identity (k) (e.g., handle or lid). SGCR then performs cross-view confidence accumulation and geometric refinement separately for each part, using only the corresponding masks across views. Therefore, differences in the ordering of part proposals do not affect the refinement, as correspondence is established through semantic part identity rather than proposal index. We will clarify this procedure in Sec. 4.1 of the revised manuscript.

The authors have addressed my concerns. My concern on experimental setting of Table 1 has been settled. The question regarding the graspable region has also been answered. I have no further questions or concerns and would maintain the current evaluation.

We are glad that our clarifications on the evaluation setting and graspable region have resolved your concerns. We sincerely appreciate your thoughtful feedback and continued recommendation for acceptance.

Reviewer_9TNeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SECOND-Grasp, a framework for semantically guided dexterous grasping. It uses a pretrained VLM and a segmentation model to propose object-part contact regions from multiple views, refines the resulting contact maps through Semantic-Geometric Consistency Refinement (SGCR), and derives pseudo hand poses through inverse kinematics. The contact maps and pose priors guide reinforcement learning, with a vision-based policy obtained through teacher-student distillation. Experiments evaluate grasp success, intent alignment, cross-dataset generalization, and several sources of sensing and execution variation.

Critical strengths
  1. The framework provides a coherent way to connect semantic object regions with executable dexterous grasping. In particular, converting contact maps into IK-derived pose priors for policy learning allows the policy to improve on the intermediate pose rather than requiring that pose to be directly executable. The IK study and learning curves support this design choice. This integration is a substantive contribution because it connects semantic guidance to the acquisition of physically successful grasping behavior.
  2. The reported empirical results provide substantial evidence of effective grasping and generalization. Improvements span seen and unseen categories, state-based and vision-based policies, and several additional object datasets. The five-seed comparison with DemoGrasp and the sensing and execution perturbation studies further strengthen this evidence. This breadth is consequential to my assessment because it suggests that the overall framework's benefits extend beyond a single object split or nominal simulation condition.
Critical weaknesses
  1. SGCR's claimed improvement in contact-map quality is not yet sufficiently explained or validated. For a convex solid, the condition in Eqs. (7)-(8) accepts every candidate point, so it does not by itself ensure the claimed compact local region. Appendix B.2 mentions additional seed-component and neighborhood rules, and Table 7 introduces a threshold, but their relationship to the equations remains unclear. The evaluation also leaves the contribution of each refinement stage uncertain: Table 3 reports grasp success and diversity for SGCR as a whole, while Table 7's zero-threshold baseline is inconsistently described. Clarifying the implementation and evaluating semantic localization across refinement stages would address both issues. Since SGCR is a central methodological contribution, these gaps limit confidence that its proposed operations explain the claimed improvement.

  2. The evidence for reliable semantic alignment remains incomplete across both contact generation and evaluation. Appendix B.3 discusses error propagation and tests depth noise and camera coverage, but does not quantify incorrect part predictions or localization errors; a region predicted incorrectly across all views may still receive strong cross-view support. Although Section 5.3 states that baselines use the same evaluation setting, the construction and validation of the reference regions for mSAD/ISR, and their relationship to the generated maps, are also unclear. Thus, high intent-alignment scores could demonstrate adherence to a supplied target without independently validating its semantic correctness. A targeted grounding audit and a clearer evaluation-reference protocol would strengthen this central claim of the paper.

Questions for authors
  1. How do the top-10% confidence seeds relate to the seed component and nearby candidates in Appendix B.2? Please define the neighborhood restriction, the quantity thresholded at 0.6, and its relationship to Eqs. (7)-(8). What geometry supports the volume-membership test, including in the partial-coverage and noisy-depth experiments?
  2. Does the zero-threshold setting in Table 7 retain semantic refinement? Please clarify the labels and, if feasible, report no refinement, semantic refinement only, and full SGCR using fixed evaluation regions, including an intent-alignment metric.
  3. Could the authors quantify semantic grounding on a small audited sample, or provide a controlled localization-error study with representative failure cases? It would be helpful to distinguish corrected errors from those reaching the final grasp, including the limitation when an incorrect part is predicted consistently across views.
  4. How are the target regions for mSAD/ISR constructed and validated? Are the same reference regions and requested intents used for all methods, and how do these references relate to the generated contact maps?
7 comments
AuthorsOpenReview ↗

Author Response to Reviewer 9TNe (1/2)

We thank the reviewer for thoughtful and constructive feedback. We appreciate the positive assessment of our IK-guided policy learning design, which bridges semantic guidance and physically executable grasping, as well as the extensive experimental results demonstrating robustness and generalization. We address the reviewer's concerns and provide further clarifications below.


[W1, Q1, Q2] Clarification on SGCR

We acknowledge that the original formulation was insufficiently precise and have revised the manuscript to clarify the implementation and associated experiments.

● Local-convexity and Eq (8)

Regarding Eq. (8), which explains our local-convexity based geometric refinement, we have corrected the equation as follows:

Qlocal(k)={q∈Q∣rk(q)≥ζ},\mathcal{Q}_{\text{local}}^{(k)} = \lbrace q \in \mathcal{Q} \mid r_k(q) \geq \zeta \rbrace,

where rk(q)=N(k)(q)/M(k)(q)r_k(q)=N^{(k)}(q)/M^{(k)}(q). Here, N(k)(q)N^{(k)}(q) refers to the number of seed points qsq_s satisfying (q,qs)∈C(O)(q,q_s) \in \mathcal{C}(\mathcal{O}). M(k)(q)M^{(k)}(q) denotes the total number of seed points in Qseed(k)\mathcal{Q}^{(k)}_{\mathrm{seed}}. Basically, for each candidate point q∈Q(k)q \in \mathcal{Q}^{(k)}, we compute the proportion of locally convex pairs among all pairs formed with the seed points qs∈Qseed(k)q_s \in \mathcal{Q}^{(k)}_{\mathrm{seed}}. A candidate is retained only if this proportion meets the threshold ζ\zeta, which controls the strictness of geometric refinement. For example, ζ=0.0\zeta=0.0 retains all candidate points in Q(k)\mathcal{Q}^{(k)} as Qlocal(k)\mathcal{Q}^{(k)}_{\mathrm{local}}, which is equivalent to disabling geometric refinement. ζ=1.0\zeta=1.0 retains only points that form locally convex pairs with every seed point. Accordingly, the final contact map is obtained by combining the seed region with the retained candidates: Qfinal(k)=Qseed(k)∪Qlocal(k)\mathcal{Q}^{(k)}_{\mathrm{final}} = \mathcal{Q}^{(k)}_{\mathrm{seed}} \cup \mathcal{Q}^{(k)}_{\mathrm{local}}. We will shortly upload the manuscript with revised equations.

● Threshold (ζ\zeta) and labeling inconsistency in Table 7.

We acknowledge that labeling �0��0� as "w/o SGCR" may have caused confusion. We originally used this label because semantic refinement only updates confidence scores without changing the contact regions, while �1��1� retains all candidate points. Consequently, both settings produce identical contact maps and IK-derived pseudo poses, resulting in the same policy guidance and training objectives. We apologize for this ambiguity and will label the setting simply as �2��2� in Table 7, clarifying this distinction in the revised manuscript. We will shortly upload the manuscript with an updated Table 7.

Metricζ\zeta = 0.0ζ\zeta = 0.2ζ\zeta = 0.4ζ\zeta = 0.6ζ\zeta = 0.8ζ\zeta = 1.0
Refinement Ratio (%)0.020.424.728.533.944.7
Seen GSR ↑94.98 ± 3.2795.82 ± 3.0196.87 ± 1.4697.30 ± 1.1097.00 ± 1.3695.90 ± 0.86
Unseen GSR ↑95.56 ± 2.4795.94 ± 2.3996.06 ± 2.0396.95 ± 1.3395.25 ± 1.2295.47 ± 0.88

● Semantic localization across refinement stages.

We refer the reviewer to Figures 4, 6, and 10, which visualize the progressive refinement of semantic contact regions across three stages: initial contact map, semantic refinement, and geometric refinement. As illustrated, SGCR transforms coarse semantic predictions into more localized and geometrically coherent contact regions, providing actionable guidance for dexterous policy learning. These visualizations further highlight SGCR's role in improving the spatial precision and geometric consistency of semantic contact guidance.

● Geometry for convexity validation.

For synthetic objects, we use the provided object meshes to evaluate the volume-membership criterion in Eq. (7), including the noisy-depth and partial-coverage experiments. For real-world WildRGB-D, where ground-truth object meshes are unavailable, we reconstruct object meshes from multi-view RGB-D observations using TSDF fusion and apply the same local-convexity criterion to the reconstructed geometry.

AuthorsOpenReview ↗

Author Response to Reviewer 9TNe (2/2)

[W2, Q3, Q4] Semantic Alignment Reliability and Evaluation Protocol of mSAD/ISR

We thank the reviewer for highlighting the distinction between semantic grounding correctness and downstream intent alignment. We clarify these concerns below.

● Semantic Alignment Reliability

Identifying and localizing common object parts (e.g., handles, lids, and bodies) is an established capability of modern pretrained VLMs. SGCR instead focuses on refining these predictions into geometrically coherent contact regions suitable for dexterous policy learning (Figs. 4, 6, and 10). To further assess grounding reliability, we additionally evaluate VLM predictions against human-annotated object-part regions using bbox IoU across different VLM backbones (see our response for W2 to Reviewer D9Y5). This enables us to independently assess initial semantic localization accuracy and examine its relationship to downstream grasping performance.

● Reference Regions and Interpretation of mSAD/ISR

We clarify that the reference regions for mSAD and ISR are derived from our SGCR-generated contact maps in Sec 5.3 and Table 2. For baseline methods requiring a point-based grasp condition, we provide the center point of the corresponding contact map, ensuring that all methods are guided toward the same intended object region. mSAD measures the distance between executed fingertip contacts and the reference region, while ISR additionally requires successful grasp-and-lift execution within the specified contact-distance threshold. Thus, these metrics evaluate adherence to a common semantic contact target, rather than independently validating the semantic correctness of VLM predictions. We will clarify this reference construction and evaluation protocol in Appendix A.3.

Thank you for the detailed rebuttal

There is still one point that I would like to clarify: How do the geometric filtering rules work? Are there any additional spatial neighborhood restrictions? If so, how do they apply to different object and part scales? Also, how do "seed component" and the top 10% of the most confident candidate points relate?

Reviewer_m4iXOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SECOND-Grasp for intent-aware dexterous grasping. A VLM (Qwen3-VL) and SAM2 propose graspable part regions from multi-view images of an object. These regions are lifted into 3D and refined through semantic and geometric refinement stages to produce a contact map. An IK-based optimization then generates a pseudo grasp pose using the refined contact region as a constraint. Finally, a goal-conditioned RL policy is trained with these pseudo grasp poses as references, serving as both a reward-shaping target and an observation condition. Policies trained with the reference grasps from the proposed method achieve higher success rates and better intent alignment than recent baselines.

Critical strengths
  • The problem of grasps that are both stable and part-specific is relevant.
  • The perception pipeline is training-free and needs no grasp annotations, and Figure 8 shows the pseudo pose speeds up early RL training.
Critical weaknesses

1. Overstated claims

(a) Unified semantic–physical reasoning. The paper’s central claim is that SECOND-Grasp "bridges semantic intent and physical feasibility through contact-level reasoning," which the introduction describes as "jointly grounding" the two. However, I think this claim is overstated. I read "jointly" as meaning the two inform each other: physical feasibility should shape which regions are chosen to contact, or the policy should reason about the semantic contacts while acting. That is not what happens in the paper. As I understand it, the proposed method has two separate stages, and each ignores the others' objective.

  • 1st stage (grasp generation): it handles semantics but not physics. The IK objective (Eq. 9) only pulls fingertips toward the region. It does not check collision, penetration or force closure, so nothing verifies that the chosen region can actually support a stable grasp. The authors themselves say the pseudo pose is not an executable grasp (App. C.2), and Table 8 shows only 3.78% Cov.@2mm
  • 2nd stage (policy learning): it handles physics but not semantics. The policy tracks a reference pose, and the contact map enters only as a reward bonus for touching the target region. The policy itself does no semantic or contact reasoning

This "propose a grasp, then train a goal-conditioned policy" structure is essentially UniDexGrasp [1] with a different proposal source, and swapping the source does not make the framework unified. The headline results in Table 1 also use no contact map or pseudo pose at test time, so they involve no semantic reasoning at all. I’d suggest a more accurate framing, such as "VLM-derived part-level reference poses as reference poses for training and conditioning an RL grasping policy."

(b) Cross-embodiment generalization. The paper claims to generalize across hands, but for the Allegro Hand it only shows pose-generation statistics (Table 10) and one qualitative rollout (Fig. 9). There are no policy results and no comparison with cross-embodiment methods [2]. Since the main contribution is the policy, I don’t think the claim holds without a full policy evaluation. Otherwise, it should be limited to the contact-map and IK stage.

2. Limited technical novelty

Given the two-stage view above, I don't see a substantial new idea in either stage

  • Lifting multi-view VLM predictions to 3D parts or affordances is well studied [3, 4, 5]
  • Contact- and part-conditioned dexterous grasp generation is an active area [6, 7, 8, 9, 10]
  • Goal- and affordance-conditioned grasp-and-lift RL is established [1, 11, 12, 13, 14]

The paper needs more detailed discussions on how the proposed method differs from these works are required to establish its novelty.

3. Evaluation does not match the two-stage design

Because the stages are independent, I think each should be tested separately:

  • The generated grasps should be evaluated on physical quality (Q1, penetration, simulation success) and intent alignment against grasp-synthesis methods [7, 8, 10]. Table 8 compares with some generators, but only on runtime and contact coverage.
  • The policy should be trained with reference poses from different sources (DexGraspNet ground truth, existing generators, and the proposed IK poses) under the same RL setup.

The ablation in Table 3 shows that adding the pseudo pose helps (+3–4 GSR), but not whether the IK poses are better references than poses from other sources. SGCR itself adds only +0.4–0.8 GSR. It is also unclear what the first row of Table 3 uses: the text describes the pseudo pose as an alternative to dataset-provided poses, while Sec. 5.1 states that no ground-truth poses are used.


References

[1] Y. Xu et al. UniDexGrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. CVPR 2023.
[2] H. Yuan et al. Cross-embodiment dexterous grasping with reinforcement learning. ICLR, 2025.
[3] M. Liu et al. PartSLIP: Low-shot part segmentation for 3D point clouds via pretrained image-language models. CVPR 2023.
[4] A. Abdelreheem et al. SATR: Zero-shot semantic segmentation of 3D shapes. ICCV 2023.
[5] Y. Kuang et al. RAM: Retrieval-based affordance transfer for generalizable zero-shot robotic manipulation. CoRL 2024.
[6] S. Liu et al. ContactGen: Generative contact modeling for grasp generation. ICCV 2023.
[7] P. Li et al. GenDexGrasp: Generalizable dexterous grasping. ICRA 2023.
[8] J. He et al. DexVLG: Dexterous vision-language-grasp model at scale. ICCV 2025.
[9] Y.-L. Wei et al. Grasp as you say: Language-guided dexterous grasp generation. NeurIPS 2024.
[10] Y.-L. Wei et al. AffordDexGrasp: Open-set language-guided dexterous grasp with generalizable-instructive affordance. ICCV 2025.
[11] W. Wan et al. UniDexGrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist-specialist learning. ICCV 2023.
[12] H. Yuan et al. DemoGrasp: Universal dexterous grasping from a single demonstration. arXiv:2509.22149, 2025.
[13] C. Mao et al. Universal dexterous functional grasping via demonstration-editing reinforcement learning. arXiv:2512.13380, 2025.
[14] P. Mandikal and K. Grauman. Learning dexterous grasping with object-centric visual affordances. ICRA 2021.

Questions for authors
  • What is the physical quality of the IK pseudo poses compared with grasp-synthesis baselines given the same target regions?
  • How does the policy perform when trained with references from DexGraspNet ground truth or existing part-conditioned generators while using the same rewards?
  • How are intents chosen at evaluation, and how is c handled in the unconditioned setting?
8 comments
AuthorsOpenReview ↗

Author Response to Reviewer m4iX (1/3)

We thank the reviewer for thoughtful feedback, and for recognizing the significance of our problem setting, annotation-free perception pipeline, and the benefits of IK-derived pose priors for policy learning. We address the concerns and provide clarifications below.


[W1, Q3] Overstated Claims

(a) Unified semantic-physical reasoning

● Clarification on terminology usage
Our use of the term "unified" and “jointly” was intended to emphasize the integration of semantic guidance and physical feasibility within a single grasp-planning framework, rather than to imply bidirectional interaction during action or joint optimization between the two stages. We acknowledge that this wording may overstate the architectural coupling between the components. We will change the terminology to more accurately reflect the proposed framework, and upload the revised manuscript shortly.

Nevertheless, we would like to clarify that the proposed framework is not merely a combination of two independent components. Both incorporate considerations of semantic intent and physical feasibility, as detailed below.

1. Grasp generation is not limited to semantic understanding. We leverage VLM-encoded physical knowledge to identify plausible graspable regions, which SGCR further refines using 3D geometric constraints. Additionally, PCA-based contact partitioning in IK (Sec. 4.3) promotes physically plausible fingertip placements while reducing hand-object penetration. As shown below, this design improves physics-related metrics (Q1, penetration, and Cov.@5mm) over variants without PCA-based partitioning, demonstrating that grasp generation incorporates geometric considerations beyond semantic guidance alone. | IK Configuration | Q1 ↑ | Penetration (mm) ↓ | Cov.@5mm (%) ↑ | |:---|---:|---:|---:| | Vanilla IK (w/o partitioning) | 0.113 | 1.553 | 14.538 | | Random contact partitioning | 0.103 | 1.031 | 5.544 | | PCA-based partitioning (Ours) | 0.184 | 1.120 | 15.110 |

2. Policy learning explicitly incorporates semantic intent. The contact maps generated by SGCR provide semantic guidance during policy optimization. Specifically, we optimize the contact-conditioned reward (r_contact, Eq. 13) alongside the physical task reward (r_task, Eq. 14). This encourages the policy not only to achieve physically successful grasps but also to establish contacts at semantically intended object regions. Additional ablation reported in the table below shows r_contact generally improves the Grasping Success Rate (GSR), but its benefit on the Intent Success Rate (ISR) is more noticeable, demonstrating that semantic contact guidance contributes to policy learning beyond pose guidance alone. | Method | Seen GSR ↑ | Seen ISR ↑ | Unseen GSR ↑ | Unseen ISR ↑ | |:---|---:|---:|---:|---:| | w/o rcontact | 97.16 | 72.09 | 95.35 | 76.74 | | Ours | 98.24 | 86.52 | 97.65 | 88.87 |

● Comparison with UniDexGrasp and contribution framing
Please refer to our response to “2. Limited Technical Novelty” below, where we clarify our distinct methodological contributions and their positioning relative to prior work.

● Clarification on test-time semantic guidance (Q3)
We clarify that SECOND-Grasp is evaluated in two different test time modes. Table 1 evaluates standard grasping without explicit test-time semantic conditioning, corresponding to c=∅c=\emptyset in Sec. 3. On the other hand, Table 2 explicitly evaluates intent-aware grasping, where pseudo-pose derived from our SGCR is used as conditions for the policy to follow different semantic contact intentions without retraining. We will clarify this distinction in the revised manuscript.

(b) Cross-embodiment generalization

We additionally conduct a full policy evaluation, reporting GSR on the Allegro Hand. As shown in the table below, ours achieves consistent performance trends across the Allegro and Shadow Hands (Table 1), demonstrating that the proposed framework can be effectively applied to different hand embodiments, beyond contact-map and pose generation alone.

MetricGOTT [15]RobustDexGrasp [16]DemoGraspOurs
GSR ↑87.1087.0094.1895.41

Furthermore, we clarify the distinction between SECOND-Grasp and CrossDex [2]. As the reviewer acknowledged, our perception pipeline is training-free and requires no grasp annotations. Unlike CrossDex, which learns a cross-embodiment policy using multi-hand pose annotations, SECOND-Grasp leverages a shared semantic contact representation and lightweight IK to generate embodiment-specific grasp references without grasp annotations or additional training for perception and grasp generation.

[15] Liu et al., GOTT: Object-centric Dexterous Manipulation with a Reusable Cross-Embodiment Primitive, CoRLW 2026.
[16] Zhang et al., RobustDexGrasp: Robust Dexterous Grasping of General Objects, CoRL 2025.

AuthorsOpenReview ↗

Author Response to Reviewer m4iX (2/3)

[W2] Limited Technical novelty

While semantic grounding, contact-based grasp generation, and goal-conditioned grasping RL have been studied extensively, SECOND-Grasp addresses a distinct research question: How can semantic contact knowledge be leveraged to supervise physically stable and intent-aligned dexterous grasping without annotated grasp poses? To this end, our key contribution is a framework that automatically derives semantic contact supervision and translates it into physically executable grasping behaviors through policy learning. We clarify our distinctions relative to the three categories below:

● Semantic Affordance Grounding [3–5]

While prior works primarily focus on localizing semantic affordances in 3D, SECOND-Grasp further transforms these predictions into contact-level guidance tailored to dexterous policy learning. Specifically, we first leverage VLM reasoning to identify plausible graspable object parts (Fig. 5), followed by our SGCR module, which refines these predictions into geometrically coherent contact regions. The resulting contact maps provide actionable supervision for intent-aware dexterous grasping, as demonstrated by the improvements in the table reported at the "SGCR's Contribution to Intent-Aware Grasping" section at our response for W3 below.

● Contact-conditioned Grasp Generation [6–10]

While prior methods rely on data-driven grasp generation or learned contact representations, SECOND-Grasp directly derives embodiment-specific pseudo-pose priors from semantic contact maps through lightweight IK, without training a grasp generator or requiring annotated grasp poses. This approach makes SECOND-Grasp substantially more efficient (0.187s vs. 11.329s inference time, Table 8). Importantly, these poses serve as intermediate guidance rather than final executable grasps, allowing the policy to further refine grasping behavior through physical interaction. As also recognized by Reviewer 9TNe as a core strength of our work, this design effectively bridges semantic contact guidance and physically executable grasping.

● Contact-conditioned Policy Learning [1, 11–14]

Unlike UniDexGrasp [1], which uses grasp proposals as goal conditions for policy learning, SECOND-Grasp leverages semantic contact maps not only to derive pseudo-pose priors but also to directly guide policy optimization through contact-conditioned rewards. Thus, our contribution goes beyond replacing the grasp proposal source, as semantic contact intent is explicitly incorporated into the policy learning objective alongside physical task rewards. Our ablation above removing the contact reward further confirms its effectiveness, showing improvements in both GSR and, more notably, ISR beyond pose guidance alone.

AuthorsOpenReview ↗

Author Response to Reviewer m4iX (3/3)

[W3, Q1, Q2, Q3] Insufficient evaluation

We thank the reviewer for suggesting additional experiments to better highlight the contributions of our framework. In response, we conduct three additional experiments that could provide more direct assessments of the contribution of our SECOND-Grasp.

● Grasp Quality and Intent Alignment (Q1)

First, extending the study in Table 8, we evaluate the physical grasp quality (Q1, penetration, and sim. success), and semantic intent alignment (mSAD) of different grasping methods. Here, we also include two additional grasp-generation baselines: ContactGen[6] and Grasp as You Say [9]. As shown in the table, our IK-derived poses achieve the highest Q1 (0.184), lowest penetration (1.120), and best intent alignment (mSAD: 1.829 cm) among all compared methods, while requiring substantially less computation (0.187 s). Although our simulation success rate (38%) is slightly below the best baseline (42%), these results demonstrate that our lightweight IK approach provides physically plausible and semantically aligned pose priors without requiring fully executable grasp configurations, supporting their intended role as efficient guidance for downstream policy learning.

MethodTime (s) ↓Q1 ↑Penetration ↓Sim. Success ↑mSAD (cm) ↓
GenDexGrasp [7]11.3290.0708.26620.00%2.220
UGG9.2220.1141.27938.00%2.994
DGA0.6220.1126.11642.00%3.271
ContactGen [6]12.9160.0957.44520.00%3.072
Grasp as You Say [9]5.6660.0697.17416.00%3.811
Ours (IK)0.1870.1841.12038.00%1.829

● Pose Prior Effectiveness in Policy Learning (Q2)

To isolate the effect of reference-pose sources, we train policies using DexGraspNet ground-truth poses and poses from the grasp-generation baselines above, under identical RL settings. As shown below, our IK-derived priors achieve the highest unseen-category ISR (88.87%) while maintaining competitive GSR (97.65%), nearly matching the best baseline (UGG: 97.67%). Notably, our method improves ISR by 14.45 percentage points over the strongest competing pose source, demonstrating the effectiveness of our lightweight IK priors in guiding physically successful, intent-aligned grasping.

MethodUnseen GSR ↑Unseen ISR ↑
DexGraspNet GT90.7055.80
GenDexGrasp [7]88.3774.42
UGG97.6767.44
DGA90.7053.49
ContactGen [6]88.3873.54
Grasp as You Say [9]93.0269.77
Ours (IK)97.6588.87

● SGCR's Contribution to Intent-Aware Grasping

The primary contribution of SGCR lies in improving semantic contact alignment rather than grasp success alone. Although GSR remains high without SGCR (97.85%/96.88%), successful grasping does not necessarily imply alignment with the intended semantic regions. Our additional ablation above reveals substantially larger improvements in ISR than in GSR across both seen and unseen categories, demonstrating that SGCR plays a crucial role in preserving semantic intent during grasp execution, beyond its modest gains in overall grasp success.

MethodSeen GSR ↑Seen ISR ↑Unseen GSR ↑Unseen ISR ↑
w/o SGCR97.8571.6396.8876.74
Ours98.2486.5297.6588.87

+ Clarification on Table 3 Ablation. We acknowledge that the description in Sec. 5.1 may have caused confusion regarding the baseline in Table 3. The statement that ground-truth poses are not used applies specifically to our full model, which instead employs IK-derived pseudo poses. Therefore, the first row of Table 3 with no pseudo pose uses DexGraspNet ground-truth poses as policy inputs, without our proposed pose-prior reward. For clarity, we summarize the revised component configurations of Table 3 below. We will incorporate this clarification into the revised manuscript and share them shortly.

GT PosePseudo PoseContact MapSGCR
✔✘✘✘
✘✔✘✘
✘✔✔✘
✘✔✔✔

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We thank all reviewers once again for their thoughtful and constructive feedback. In addition to addressing the comments in our rebuttal responses, we have revised the manuscript and uploaded an updated version to incorporate the reviewers' suggestions and clarify our contributions (in blue text). The major revisions are summarized below:

1. (m4iX W1) Clarification of Contribution Claims: Removed the terms "unified" and "joint" from the abstract and introduction to avoid overstating the architectural integration of our framework. (L16–19, L49–54, L106–108)

2. (m4iX W2) Related Work and Novelty: Revised the Related Work section to better position SECOND-Grasp and explicitly distinguish our technical contributions across the three research directions highlighted by the reviewer. (L117–186)

3. (m4iX Q3, D9Y5 Q2) Training and Evaluation Settings: Added a paragraph in Sec. 6.1 clarifying our training procedure and the two evaluation settings: standard grasp-and-lift and intent-aware grasping. (L355–365)

4. (m4iX W3) Table 3 Ablation Settings: Clarified the baseline configuration in Sec. 6.5 and revised Table 3 by adding a "GT Pose" column to explicitly distinguish ground-truth pose supervision from our proposed components. (L460–476, Table 3)

5. (9TNe W1) SGCR Geometric Refinement: Corrected the local-convexity formulation to accurately describe the threshold-based geometric refinement criterion used in SGCR. (L293–309)

6. (9TNe W1) Local-Convexity Threshold Analysis: Revised Appendix B.3 and Table 7 to clarify the interpretation of the threshold (ζ\zeta), particularly the (ζ=0.0\zeta=0.0) setting, and its relationship to geometric refinement. (L916–947)

7. (D9Y5 Q3) Cross-View Semantic Correspondence: Clarified the VLM-based contact proposal procedure, specifying that (kk) denotes a consistent semantic part identity across views rather than the ordering of proposals. (L221–227)

8. (D9Y5 Minor Typo) Citation Correction: Corrected the DexGraspNet citation in the introduction to refer to DexGraspNet 1.0, consistent with our experimental setup. (L96)

We appreciate the reviewers' valuable suggestions, which have helped us improve the clarity and presentation of our work. We sincerely hope that these revisions, together with our additional experiments and detailed rebuttal responses, adequately address the remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.