- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose SECOND-Grasp to combine semantical manipulation information from a VLM and the physical validity of a dexterous grasping policy to do semantical guided dexterous grasping. First, they use a VLM to propose multi-view contact proposals from object images. Second, they proposed a novel Semantic-Geometric Consistency Refinement method to post-process VLM generated multi-view contact proposals. Then, they use IK to get a feasible hand pose from the refined contact proposal, and train a goal conditioned RL policy to execute that proposal. Experiments in simulation verified that SECOND-Grasp improves grasping success rate and intention awareness.
Critical strengths
Strengths
- Writing is clear and easy to follow.
- Experimental verifications in simulation is sound. Performance improvement is consistent.
Critical weaknesses
Weaknesses
- No real-robot experiments.
- The strengths of the VLM could potentially affect the performance of the overall pipeline, but no experimental comparisons were carried out between different VLMs.
Minor typo (not a critical technical weakness): Intro cites DexGraspNet 2.0 (line 97), but actual experiments cite and are carried out on DexGraspNet 1.0. So the citation of DexGraspNet 2.0 in line 97 seems to be a typo, and could be moved to related works section instead.
Questions for authors
Questions
- How are hand-object collision and hand-hand self-penetration handled in section 4.3 (hand pose prior via inverse kinematics)?
- Is the grasp intention completely inferred from the object's appearance using the VLM, or do humans need to provide one or two sentences to tell the VLM which parts to grasp? More specifically, in the comparisons in Table 1, do all baselines receive the same amount of information, or does the author's method require additional human-language guidance? If there are multiple places to grasp (like a mug with two handles), is it possible to use human prompt to precisely steer the system to choose one place?
- How does the order of the K proposed graspable regions affect the multi-view refinement? For example, what if one view proposes two graspable regions, where No. 1 is the handle, No. 2 is the lid, while its neighbouring view proposes the opposite (No. 1 is the lid and No. 2 is the handle)? Does the cross-view accumulation know how to match this order-discrepancy?
4 comments
Author Response to Reviewer D9Y5 (2/2)
[Q1] How are collisions and self-penetration handled during IK?
The IK stage (Sec. 4.3) does not explicitly enforce hand–object or self-collision constraints and thus does not guarantee collision-free poses. However, our PCA-based contact partitioning encourages structured fingertip placements that reduce hand–object penetration compared to vanilla IK (see our response "1. Grasp generation is not limited to semantic understanding" in W1-(a) to Reviewer m4iX). Importantly, the IK-derived poses serve as soft and flexible guidance rather than directly executable grasps, allowing the RL policy to learn physically stable grasping behaviors through simulation-based interactions.
To further validate this design, we compare our IK-derived priors against those from existing grasp generators under identical RL training settings (see our response "Pose Prior Effectiveness in Policy Learning (Q2)" in W3 to Reviewer m4iX). The results demonstrate competitive grasp success rates and higher intent success rates, supporting the effectiveness of our lightweight IK approach as guidance for learning physically stable, intent-aligned grasps.
[Q2] How are grasp intents inferred, and is guidance consistent across baselines?
We would like to clarify that during training, SECOND-Grasp uses VLM proposals to identify and localize plausible graspable object parts from multi-view images, without requiring human-language instructions. At test time, we evaluate two distinct settings:
1. Standard Grasp-and-Lift (Table 1): The policy learns grasping behavior through semantic contact and pseudo-pose supervision during training. At test time, the learned policy directly predicts grasping actions from object observations, without requiring additional VLM inference, contact maps, pseudo poses, IK, or human-language guidance (Appendix C.1). Thus, our method does not receive additional human-language guidance compared with the baselines.
2. Intent-Aware Grasping (Table 2, Fig. 4): We additionally support explicitly specified grasp intents, where users can indicate the desired object part through language, and the VLM localizes the corresponding region to guide grasp execution. When multiple similar parts exist (e.g., two handles), SGCR uses VLM confidence scores and cross-view consistency to identify a high-confidence seed region, followed by geometric refinement to obtain a localized contact target.
In summary, human-language guidance is optional and is used only for explicitly intent-conditioned grasping, not for the standard grasp-and-lift comparisons in Table 1.
[Q3] How does the order of the K affect the multi-view refinement?
We would like to clarify that SGCR processes each semantic part independently, rather than jointly refining multiple unordered region proposals. Specifically, the VLM identifies multiple graspable parts and localizes each part across views under a consistent semantic identity (k) (e.g., handle or lid). SGCR then performs cross-view confidence accumulation and geometric refinement separately for each part, using only the corresponding masks across views. Therefore, differences in the ordering of part proposals do not affect the refinement, as correspondence is established through semantic part identity rather than proposal index. We will clarify this procedure in Sec. 4.1 of the revised manuscript.
The authors have addressed my concerns. My concern on experimental setting of Table 1 has been settled. The question regarding the graspable region has also been answered. I have no further questions or concerns and would maintain the current evaluation.
We are glad that our clarifications on the evaluation setting and graspable region have resolved your concerns. We sincerely appreciate your thoughtful feedback and continued recommendation for acceptance.
Author Response to Reviewer D9Y5 (1/2)
We sincerely thank the reviewer for their positive assessment and recommendation toward acceptance. We are glad that the reviewer found our paper clear and well-presented, with consistent performance improvements supported by sound experimental evaluations. We provide further clarifications to address their remaining questions below.
[W1] Real-robot experiments.
Although deployment in real robots has not yet been conducted, we have included in our initial submission, some supporting experiments through supplementary material to demonstrate the robustness of our framework under realistic conditions. (please see index.html for demo videos)
● Real-scene Contact Maps
We have evaluated SGCR on the WildRGB-D data, aiming to investigate the robustness of SGCR under sim-to-real domain shift. While the data contains noisy depth estimation and imperfect geometric observations compared to simulators, SGCR precisely refines and captures the contact map aligning with a given grasp intent. We have also provided visualization of contact maps at each stage of SGCR on this real-world date in Figure 6.
● Simulated Real-world Noise
We have further evaluated robustness under simulated real-world uncertainties by introducing three types of perturbations: (1) action execution noise, (2) control latency, and (3) contact dynamics variations. As reported in Table 13, although these perturbations inevitably degrade overall grasping performance, intent-aware grasping performance remains relatively robust, with only modest reductions in ISR.
Overall, these results suggest that SECOND-Grasp can preserve semantic grasping intent under realistic execution uncertainties, providing encouraging evidence for its potential applicability to real-world robotic systems.
[W2] Comparing different VLM backbones.
We conduct additional experiments with different VLM backbones for initial 2D contact region proposals. To systematically assess the sensitivity of our SECOND-Grasp to backbone selection, we compare our default Qwen3-VL-8B against two alternatives: (1) Qwen3-VL-4B, to examine the effect of model size within the same family, and (2) Qwen3.5-VL-9B, to examine the effect of model version at a comparable parameter scale. The results are summarized in the table below.
(1) vs. Qwen3-VL-4B
Interestingly, Qwen3-VL-4B achieves higher BBox IoU (57.02 vs. 51.69) and slightly higher ISR (89.16% vs. 88.87%), despite its smaller parameter size. Although its GSR is somewhat lower (94.92% vs. 97.65%), these results suggest that larger VLM capacity is not necessarily required for accurate semantic contact localization or intent-aware grasping.
(2) vs. Qwen3.5-VL-9B
Despite its lower BBox IoU (38.87), Qwen3.5-VL-9B achieves comparable downstream GSR (95.35%). However, when evaluated under a metric that requires precise semantic and geometrical grounding (ISR), its performance is relatively low (81.40%), following the trend in BBox prediction. The result suggests that although our SGCR refines the contact maps, the initial visual grounding performance matters for accurate intent-aware grasping.
Overall, despite noticeable variations in BBox IoU across VLM backbones, SECOND-Grasp maintains consistently high GSR and ISR, suggesting robustness to backbone selection and variations in initial semantic localization quality.
+ Minor Regarding Citation. We thank the reviewer for catching this typo. The intended reference was DexGraspNet 1.0, which was used throughout our experiments, and we have corrected the citation in the revised manuscript.