acceptodds
Under review as a conference paper at ICLR 2027

PointDex: Dexterous Grasp Generation via Point-based Hand-Object-Contact Modeling

17 views

Abstract

Dexterous grasp generation requires precise reasoning over the spatial and functional relationships among hands, objects, and their contacts. Yet existing approaches often struggle to model these interdependent entities effectively, and provide only limited means for steering the generated grasps. We introduce PointDex, an effective, steerable framework for dexterous grasp generation that captures hand–object interactions by representing grasp pose, object contact, and hand contact as a unified set of 3D points in a structured space. This explicit, disentangled yet jointly shared representation allows a single transformer to simultaneously denoise all point types, and naturally supports flexible steering by conditioning on any subset of points while denoising the rest. Given a 3D object and a language instruction, PointDex jointly generates posed hand keypoints and paired object–hand contact points, and subsequently recovers wrist pose and joint angles via inverse kinematics. On the DexGYS and Dexonomy benchmarks, PointDex improves language-conditioned grasp success by 16.6 points over the strongest prior method and reduces steering contact distance by up to 80%, while unifying diverse steering modes within a single model.

Primary area:
applications to robotics, autonomy, planning
Keywords:
dexterous grasping, grasp generation, generative models, 3D representation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_i4G9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PointDex targets language-conditioned dexterous grasp generation for the Shadow Hand. It represents the grasp pose as 29 posed hand keypoints and the contacts as 16 slots, each holding a pair of points: one on the object surface and one on the canonical open hand. All of these are 3D point tokens, and a DiT trained with rectified flow denoises them jointly under Diffusion Forcing-style per-token noise levels. The method also uses DETR-style slot matching, existence prediction, surface-sampled dummy contacts, a pair attention bias, and a Langevin corrector. Kabsch alignment and QP-based IK then recover the wrist pose and joint angles. Any subset of contact tokens can be held clean as a condition, so a single model supports object-only, hand-only, and paired contact steering. On DexGYS, PointDex reaches a success rate of 83.76% versus 67.14% for DextER, with higher Q1 and lower penetration. Under steering it greatly reduces contact distances, e.g., d_pair drops from 31.10 to 6.14 mm. It shows similar gains on Dexonomy, including the unseen-grasp-type splits, where its success rate is about 33% versus about 11% for DextER.

Critical strengths
  1. PointDex represents the grasp pose, object contacts, and hand contacts all as 3D point tokens, which avoids rotation parameterizations and implicit forward kinematics in the pose representation. Combined with per-token noise levels, a single model supports object-only, hand-only, and paired contact steering with any number of steering contacts, without extra modules or retraining. This matters to my assessment because prior methods each support only one steering interface, whereas here steering reduces to conditioning on a subset of tokens.
  2. Strong and consistent empirical results. PointDex clearly outperforms prior methods in unsteered generation, with higher success, better grasp quality, and less penetration. Under steering, where all methods receive the same contacts, it follows the specified contacts more closely in every mode while also achieving higher success. These advantages hold on both benchmarks and extend to unseen grasp types, which suggests good generalization. The ground-truth floor also gives the contact distances a clear reference. These results are the main basis for my positive assessment.
Critical weaknesses
  1. Unclear attribution of gains. The pose-only variant already reaches 79.81% SR, 12.7 points above DextER, and contact tokens add only about 4 points. Replacing keypoints with joint angles (−19.1), removing the Langevin corrector (−7.3), and removing the pair bias (−5.6) all matter more, yet the latter two are absent from the contribution narrative. The paper attributes the keypoint advantage to sharing a frame with the contacts, but no joint-angle variant without contact tokens tests this, and the joint-angle variant itself is not described. This matters because the paper's central claim is that the shared point representation drives grasp quality.
  2. The role of language and the practical scope are not established. Although the model is conditioned on language, no ablation removes or swaps the instruction, so it is unclear how much the results depend on it. All experiments use the Shadow Hand in simulation, none use single-view or noisy point clouds, and IK residuals, IK failure rates, and inference time are not reported. This matters because these factors determine how far the benchmark results carry over to practical use.
Questions for authors
  1. Many ablation gaps are within 1–5 points, and a few are not fully monotonic; for example, removing the pair bias gives 78.14%, slightly below the pose-only 79.81%. It would help to see the mean and standard deviation over several seeds for Tables 1, 3, 6, and 9.
  2. Per-token noise levels are the key design behind multi-mode steering. Could the authors compare against a simple alternative: the same backbone trained with a shared noise level and steered at inference by inpainting or guidance? This would show more directly why the design is needed.
  3. The Langevin corrector scales the injected noise by (1−t)^p, which, relative to standard Langevin dynamics, effectively lowers the sampling temperature late in sampling and may concentrate samples in high-density regions. Could the authors also report P-FID and diversity for the ablations, especially the one without Langevin, to show whether part of the SR gain comes at the cost of diversity? In addition, PointDex's P-FID is less favorable than DextER's in Table 1 and on the seen and unseen-object splits of Table 11, but better on the two unseen-grasp splits. Do the authors have an explanation for this?
  4. Do the authors have any real-robot results, or preliminary results with single-view or noisy point clouds? Positive results here would further strengthen my assessment of the paper.
  5. If the authors can address points 1–4 during the discussion period, I would be happy to reconsider my score.
5 comments
AuthorsOpenReview ↗

Response to reviewer i4G9 [1/3]

(W1) Unclear attribution of gains

G1 in the general response reports the joint-angle variant without contact tokens that the reviewer asks for, and App. A.1.5 now describes both joint-angle variants. They replace the keypoint pose tokens with the wrist pose and the 22 joint angles. In the variant without contacts, the contact slots become learned tokens, so the token count stays the same. All four variants in this comparison disable the local geometry embedding, since it queries object features at token positions in the object frame.

The keypoint pose gives the largest SR gain: +34.1 points without contacts (45.89% → 79.94%) and +19.6 with them (63.30% → 82.89%). The contacts add a smaller but clear gain, from 79.91 ± 0.25% to 83.21 ± 0.27% over three seeds (Table 3), and 17.4 points with a joint-angle pose. Sharing one frame between the pose and the contacts pays off most in steering. Given the same contacts, the keypoint model follows them 1.9 to 2.8 times more closely than the joint-angle model (d_pair 7.38 vs. 20.57 mm), at a much higher SR (80.95% vs. 63.93%).

The Langevin corrector and the pair bias are supporting components rather than contributions. The corrector is a generic sampler applied only at inference. Without it, PointDex still reaches 77.02 ± 0.41% SR, above every re-evaluated baseline (T(R,O) Grasp 73.19 ± 0.10%, DextER 65.17%). The pair bias is needed because the object point and the hand point of each contact are separate tokens, and it binds the two points of each contact. Without it, d_pair rises from 7.43 mm to 12.53 mm. The revised manuscript states the role of both components in the ablation discussion (Sec. 4.4).

(W2-1) Role of language and practical scope

On the DexGYS test set, we replaced the instruction at test time with one from a different object, or with an empty string. Action match is the fraction of grasps whose nearest same-object ground-truth grasp belongs to the instructed action; chance level is 28–31%. Con. is the contact-map distance to the instructed action’s ground-truth grasps, as in Table 1.

InstructionSR (%)Action match (%)Con.
Original83.2168.90.23
Different object’s70.630.80.81
Empty67.332.10.96

Our IK always returns a solution, so we count an IK failure when the IK-recovered hand does not match the generated keypoints. For each sample, we compute the mean distance between the 29 generated keypoints and the same keypoints obtained by forward kinematics on the IK-recovered pose. A sample counts as a failure when this mean exceeds 10 mm (DexGYS test set, 9,971 grasps).

Settingsmeanmedianmean > 10 mm
w/o contact steering1.06 mm0.88 mm0.14%
pair steering1.20 mm1.02 mm0.14%

Feeding the ground-truth keypoints to the same IK gives a residual of 0.07 mm.

(W2-2) Inference Time

We timed the methods below on DexGYS with one protocol (one NVIDIA B200 GPU, 256 test inputs, one grasp per input, batch size 16; mean batch time / 16). PointDex takes 32.9 ms per grasp, comparable to the diffusion model DexGYSNet (34.1 ms) and below DextER (83.4 ms); without the corrector it takes 10.7 ms. The IK, run on the CPU, adds 29.0 ms.

Method (DexGYS)Batch 16 (ms/grasp)IK/postprocess (ms/grasp)NFE
DexGYSNet34.1–100 + 1 (QGC)
DextER (bf16)83.4–1 per token
T(R,O) Grasp24.72.020
PointDex w/o corrector10.729.026
PointDex (Ours)32.929.0114

Times are wall-clock milliseconds per grasp, with torch.cuda.synchronize around each call, from the object point cloud and the instruction to the 28-D Shadow Hand pose. They include text and object encoding, all sampling or decoding steps, and refinement or test-time optimization (QGC for DexGYSNet); data loading is excluded. All methods run in fp32 except DextER, whose released inference uses bf16. The IK column is timed separately and adds to the time column: the keypoint IK of PointDex (8 iterations, on the CPU) and the Pyroki retargeting of T(R,O) Grasp on the CPU; the other methods output joint angles directly. NFE counts denoiser or decoder passes; DextER decodes autoregressively, one pass per generated token.

AuthorsOpenReview ↗

Response to reviewer i4G9 [2/3]

(W2-3, Q4) Other hands and real-robot results

To test another hand, we trained PointDex on the Inspire hand, which has 12 joints driven by only 6 actuators (the other 6 are mechanically coupled). The data uses the Dexonomy objects with newly synthesized Inspire-hand grasps, each paired with a templated instruction that names its grasp type: 182,061 grasps for training and 12,929 held-out grasps of objects seen in training for evaluation, with SR measured in MuJoCo as in DextER's Dexonomy evaluation. Only the keypoint layout, the canonical hand point cloud and the IK change; the architecture and sampler are unchanged. PointDex reaches 28.87% SR without steering and 36.32% with pair steering, against 18.01% and 20.04% for DextER trained on the same data. We also ran real-robot experiments with the Inspire RH56F1 hand, where PointDex succeeds in 24 of 40 trials (15 of 20 on seen objects and 9 of 20 on unseen ones); please see G2 for the setup and per-object results.

(W2-4, Q4) Partial point clouds

We evaluate the DexGYS-trained PointDex zero-shot on simulated partial observations. For each test input, two virtual cameras view the object: one 0.3 m behind the ground-truth wrist, facing the object, and one 0.3 m above and to the side of the midpoint between the wrist and the object. Only the points visible from either camera are kept, found by hidden-point removal (Katz et al., 2007); this leaves 42% of the cloud (median), which is resampled back to 10,000 points. Only the model input changes: the model is not retrained, and all metrics are computed against the full object, as in Table 1.

Object cloudSR (%) ↑P-FID ↓CD ↓Con. ↓Q1 ↑Pen. (cm) ↓
Full83.210.3511.0420.2331.5670.135
Partial82.100.3451.1490.2541.3040.220

PointDex keeps its SR and P-FID on partial point clouds (83.21% → 82.10%, 0.351 → 0.345), while other metrics degrade moderately. Because one camera sits behind the GT wrist, as in DextER's protocol, the visible part usually includes the side the hand approaches from.

References

  1. S. Katz, A. Tal, R. Basri. Direct Visibility of Point Sets. ACM Transactions on Graphics (Proc. SIGGRAPH), 26(3), 2007.

(Q1) Seeds and variance for the ablations

We have updated Tables 1, 3, 6, and 9 (Tables 1, 3, 6, and 13 in the revised manuscript) to report results as mean ± standard deviation over three random seeds; please refer to the revised manuscript for details. Some baselines in Table 1 don’t have publicly available official training code, checkpoints, or both, so we could not run them across multiple seeds during the rebuttal period. We will re-implement and train these baselines and report their mean ± standard deviation in the camera-ready version. Regarding the number of generated samples, we use one sample per input.

(Q2) Comparison with a shared noise level and inpainting

We trained the same backbone with one shared noise level for all points, using the same data and schedule, and steered it at inference with RePaint-style inpainting: at each step, the given points are re-noised to the current noise level and written back. Results on DexGYS with all contacts given:

Noise scheme (DexGYS)No steering SR (%)Object-only SR (%) / d_obj (mm)Hand-only SR (%) / d_hand (mm)Pair SR (%) / d_pair (mm)
Per-point noise (ours)83.2183.22 / 6.2683.23 / 5.8581.13 / 7.43
Shared noise + inpainting82.3782.56 / 17.6983.63 / 7.1082.70 / 40.67

Inpainting keeps SR but barely follows the contacts: d_obj is 17.69 mm instead of 6.26 mm, and d_pair 40.67 mm instead of 7.43 mm. With per-point noise levels, the given points stay clean at every step, so the network conditions on them directly instead of on noised copies.

AuthorsOpenReview ↗

Response to reviewer i4G9 [3/3]

(Q3) The corrector may concentrate samples at the cost of diversity

Concentration in high-density regions. Yes. Turning off only the corrector in the same model lowers P-FID from 0.351 to 0.144, below DextER’s 0.228. In the P-FID feature space, the corrector raises precision from 0.900 to 0.926 and lowers recall from 0.853 to 0.777 (Kynkäänniemi et al., 2019), so samples move toward high-density regions of the data distribution.

SR gain at the cost of diversity. Partly, yes. The corrector raises SR from 77.02% to 83.21% and reduces per-input diversity:

P-FID ↓SR (%) ↑Overall δt / δr / δq ↑
PointDex0.35183.218.81 / 77.9 / 12.22
PointDex w/o corrector0.14477.028.83 / 78.5 / 13.84

We use the corrector by default because SR is the more important metric for our goal. Since it is applied only at inference, it can be turned off, and even without it PointDex outperforms DextER (65.17%) and T(R,O) Grasp (73.19%) in SR (77.02%).

Split-wise P-FID (Table 11; Table 16 in the revised manuscript). The pattern combines two effects.

  • Seen and unseen-object splits. PointDex trails DextER because of the corrector. Without it, PointDex reaches 0.22 / 0.21, close to or below DextER’s 0.21 / 0.29.
  • Unseen-grasp splits. Even without the corrector, PointDex is far below DextER (1.00 / 0.93 vs. 1.68 / 1.71). The corrector raises PointDex’s P-FID by about 0.3 (to 1.31 / 1.21), but this gap is larger, so PointDex stays ahead.

To see where the samples lie, we compare each model’s samples on the unseen-grasp-type split with the training grasps of the same object, labeling each sample with its most similar seen grasp type. PointDex without the corrector stays close to these training grasps (0.26; 0.24 on the seen split), whereas DextER moves away from them (1.02; 0.22 on the seen split). Since the training grasps are closer to the unseen ground truth (1.20) than DextER’s samples are (1.68), staying near them gives PointDex the lower P-FID, without generating new grasp types.

Unseen-grasp-type splitP-FID to training grasps ↓P-FID to ground truth ↓
Training grasps (reference)-1.20
PointDex (w/o corrector)0.261.00
DextER1.021.68
Reviewer_jJhjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

PointDex generates dexterous grasps from a 3D object and a text instruction. It writes the hand pose as 29 keypoints and the contacts as 16 slots, each holding a pair of points (one on the object, one on a canonical open hand); a grasp uses up to 16 slots and the unused ones are filled with dummy pairs. One transformer denoises all 61 points together with flow matching. Each contact point has its own noise level, so the user can fix any subset of contacts and the model generates the rest; this gives object-only, hand-only and pair steering from one model. Wrist pose and joint angles are recovered from the keypoints by inverse kinematics (IK).

Critical strengths
  1. In all six settings of Tables 4 and 7 the contact distance is within 2 mm of the ground-truth floor. One model handles any number of contacts and all three steering modes.

  2. Grasp quality is clearly better. DexGYS success goes from 67.14% to 83.76% with the best Q1 and the lowest penetration (Table 1). It also has the best SR on all four Dexonomy splits (Table 11).

  3. The ground-truth floor rows and the corrector ablation with the same number of forward passes (Table 6) make the numbers easy to interpret. Code and checkpoints are promised.

Critical weaknesses
  1. The gain comes from the keypoints and the sampler, not from the contacts, as without any contact tokens the model still gets 79.81% (Table 3), and without the corrector it drops to 76.42% (Table 6). Yet Sec. 4.2 claims the contacts “drive grasp quality” and that the gain “requires both parts”.

  2. The mechanism of training with a separate noise level per input and then keeping the user’s inputs clean while generating the rest was introduced by UniDiffuser (Bao et al., 2023) and is used in many later works such as Diffusion Forcing. In grasping, UGG (Lu et al., 2024), which the paper cites, already trains with separate noise levels for object, hand and contact anchors and conditions by setting one of them to clean; PointDex only does this per contact point, yet presents it as a contribution and cites neither UniDiffuser nor UGG for it.

  3. The baselines may be weak because of how the authors ran them, not because of the methods. T(R,O) Grasp gets 32.46% without user contacts (Table 11) but only 7.92% with them (Table 7), and DextER’s object distances on DexGYS (17.23/31.10 mm) are 3–5 times larger than on Dexonomy (5.59/6.66), which looks like a bug in the setup.

  4. The main result (83.76% vs. 67.14% SR on DexGYS, Table 1) is not an apples-to-apples comparison, as the 83.76% was measured by the authors, but the 67.14% is copied from the DextER paper (Table 1 caption), so a different simulator setup, number of samples or seeds could explain part of the gap.

  5. In every steering experiment (Tables 4, 5 and 7) the contacts given to the model are, as far as we can tell, taken from the ground-truth grasp of the test sample (Sec. 4.3 points to App. A.1.3, which computes the contacts from each sample’s posed hand, and the GT floor in App. A.1.4 uses the same grasp), so the model is told part of the answer; this is why SR with contacts given (88.87% hand-only on DexGYS, 44.61–47.57% on Dexonomy) is higher than SR with only the object and text (83.76% and 39.37%). A real user has no ground-truth grasp to copy from, and no experiment uses user-given, noisy or conflicting contacts, so the steered SR numbers do not tell us what a user would get.

  6. The per-point noise scheme is never compared against the simple alternative, where a model is trained with one shared noise level for all points and steered by re-noising the given contacts to the current noise level at each step (inpainting), which also avoids the “clean and noisy points together” problem the paper uses to motivate its scheme. Nor is it compared with one shared noise level per contact pair (τio=τih)(\tau^o_i=\tau^h_i), or with q=0, i.e. no training points forced to be fully clean, so the main claim behind contribution 2 is not tested.

  7. No seeds or variance are reported, but several conclusions rest on gaps of 1–3 points (Tables 3, 9 and 10). For example, β=4 is chosen in Table 9 for its highest pair SR, but the SR spread over β=2–8 is only 0.7 points and β=8 has the lowest dpair , so this choice may be noise.

  8. P-FID is worse than DextER’s (0.33 vs. 0.20 in Table 1, 0.35 vs. 0.21 in Table 11) and this is never discussed. MaskDexGrasp (Zuo et al., 2026) uses the same Shadow Hand, takes text, and lets the user choose which fingers to use, so it is a natural baseline for the hand-only setting of Tables 4 and 7, but it is only cited.

Questions for authors
  1. The model without any contact tokens already reaches 79.81% and the Langevin corrector in the sampler alone is worth 7.3 SR points. Can you show an experiment where the contacts, and not the keypoints or the sampler, are responsible for the gain over DextER?

  2. Why is UniDiffuser not cited, and why is the UGG design, which already trains with separate noise levels and conditions by fixing some inputs to clean, not mentioned? What does PointDex add beyond applying this per contact point?

  3. Did you evaluate the DextER checkpoint on DexGYS without any contacts given? If yes, report that SR next to the copied 67.14% in Table 1; if not, run it and report. Also add your retrained T(R,O) Grasp scores to Table 1, as you did for Dexonomy in Table 11.

  4. Why does T(R,O) Grasp drop from 32.46% to 7.92% when it is given ground-truth contacts, and why are DextER’s object-side distances 3–5 times larger on DexGYS than on Dexonomy? Both suggest a problem in the baseline setup rather than in the methods.

  5. Do the steering contacts in Tables 4, 5 and 7 come from the ground-truth grasp of each test sample? How does PointDex behave with contacts that do not come from the ground-truth grasp, for example contacts from another grasp of the same object, contacts with added noise, or a pair that cannot be reached? Report SR and contact distance for such cases.

  6. Compare against a model trained with one shared noise level and steered by re-noising the given contacts at each step (inpainting), against tied pair noise (τio=τih)(\tau^o_i=\tau^h_i), and against q=0.

  7. Report SR over at least three seeds for Tables 1, 3, 6, 9 and 10, and state the number of samples generated per input.

  8. Why is P-FID worse than DextER’s on both benchmarks, and can MaskDexGrasp be added as a baseline for the hand-only setting?

  9. Why is the Langevin corrector needed? It is a generic sampling trick that is not tied to keypoints, contacts or per-point noise levels, yet it is worth 7.3 SR points (Table 6). Would the diffusion baselines in Table 1 gain a similar amount if sampled the same way? Can it be added at least to DextER’s sampler for a matched row in Table 1? If so, add such a row; if the corrector cannot be applied to them, explain why and state that the fair comparison is PointDex without it (76.42%) against DextER (67.14%). In either case, the “16.6 points over the strongest prior method” claim in the abstract, contributions and conclusion should be lowered accordingly.

  10. The generated keypoints are free 3D points, and IK fits the Shadow Hand to them as closely as it can. How far are the keypoints of the fitted Shadow Hand from the generated ones on average, and how often is the gap large because the generated points do not form a valid Shadow Hand pose? Also, why does giving the model one average point per link (88.87%, Table 4) work better than giving it the exact ground-truth hand points (83.23%, Table 5), although the exact points carry more information?

5 comments
AuthorsOpenReview ↗

Response to reviewer jjhj [1/4]

(W1, Q1, Q9) The gain comes from keypoints and the sampler, not from contacts

We agree that the keypoint pose gives the larger SR gain, but the contacts also contribute, and they are what makes steering possible. With three seeds, removing all contact tokens lowers SR from 83.21 ± 0.27% to 79.91 ± 0.25% (Table 3), a 3.3-point drop well beyond the seed variation, and removing only the object or only the hand contacts already costs 1.4 to 1.7 points. With a joint-angle pose, the contacts add 17.4 points (45.89% → 63.30%, G1). The SR gain from the contacts is therefore smaller than that from the keypoints, but it is consistent and clearly not zero.

The larger role of the contacts is steering. A model without contact tokens cannot take contacts as a condition, whereas PointDex follows the given contacts closely in every mode. As the reviewer notes, its contact distances are within 2 mm of the ground-truth floor in all six settings of Tables 4 and 7. The shared frame with the keypoint pose matters here as well: with the same contacts, the keypoint model lands 2.8 times closer to them than the joint-angle model (d_pair 7.38 vs. 20.57 mm, G1). We revised Sec. 4.2 to state this attribution: the keypoint pose drives the largest SR gain, the contacts add a consistent gain on top of it, and the shared frame matters most for steering.

The sampler does not account for the margin over the baselines either. Without the Langevin corrector, PointDex still reaches 77.02% SR, above DextER (65.17%) and every other baseline (G3). Nor does the corrector give every model the same gain. Added to the diffusion baselines, it raises DexGYSNet from 57.92% to 63.74% but leaves T(R,O) Grasp unchanged (73.19% vs. 73.08%), so PointDex without the corrector still exceeds every corrected baseline. DextER decodes discrete tokens autoregressively and has no noise level or score to correct, so its fair comparison is with PointDex without the corrector (77.02% vs. 65.17%). We keep the corrector because it raises the SR of PointDex from 77.02% to 83.21%. As suggested, we replaced the +16.6-point claim over DextER with the margin over the strongest baseline, T(R,O) Grasp: +10.0 points (+3.8 without the corrector).

(W2, Q2) Per-token noise levels already exist (UniDiffuser, Diffusion Forcing, UGG)

We agree. The General Response credits the mechanism to Diffusion Forcing and discusses UniDiffuser and UGG, and the revised manuscript cites all three (Sec. 2). UniDiffuser and UGG assign one noise level per modality. Compared with UGG, PointDex (1) uses per-token rather than per-modality noise levels, (2) represents the object point and the hand point of each contact separately, and (3) accepts a variable number of contact points as the condition. Coarser noise schemes do not achieve this (W6). One noise level per contact pair breaks object-only steering (79.54% SR and 8.54 mm, vs. 83.22% and 6.26 mm), and one shared noise level with inpainting barely steers (d_pair 40.67 mm vs. 7.43 mm).

AuthorsOpenReview ↗

Response to reviewer jjhj [2/4]

(W3, Q4) Baselines may be weak because of how they were run

T(R,O) Grasp. The released code does not support contact-conditioned generation, so we re-implemented it from the paper (Eq. 21, Sec. IV-C). During diffusion inference, sampling is guided by the gradient of a contact loss with two terms: a geodesic term that rotates the palm so its normal opposes the object normal at the contact center, and a distance term between the contact points and the link centers, weighted by a Gaussian heatmap around the contact center.

This formulation assumes a single contiguous contact region. The contacts in our steering benchmark are scattered across several faces of the object, so a single contact center is not well defined. In the submitted version, we therefore dropped the geodesic term. It also applied the distance term to the whole hand as one rigid body rather than per link as in Eq. 21, which we fix below.

During the rebuttal, we have revised the re-implementation to follow the original more closely:

  • Geodesic term restored. When a palm contact is given, we use the object normal at that contact to define the reference palm orientation. This tells T(R,O) Grasp which contact the palm should touch, which is extra ground-truth information that favors the baseline.
  • Per-link distance term restored. As in Eq. 21, the distance term is applied between the contacts and each link center rather than to the hand as a rigid body. Since there is no single contact center, contacts are weighted uniformly.

With these changes, T(R,O) Grasp keeps SR higher (19.69% / 72.08% on Dexonomy / DexGYS) but barely moves toward the contacts (d_obj 35.59 / 22.99 mm, vs. 43.78 / 23.32 mm without contacts), while the submitted version gets closer (15.69 / 18.19 mm) at the cost of SR (7.92% / 48.12%).

MethodSteering inputDexonomy SR (%)Dexonomy d_obj (mm)DexGYS SR (%)DexGYS d_obj (mm)
T(R,O) GraspNone32.4643.7873.1923.32
T(R,O) Grasp, submittedGT contacts7.9215.6948.1218.19
T(R,O) Grasp, re-implementationGT contacts19.6935.5972.0822.99
PointDexGT contacts47.575.1083.226.26
GT floorGT contacts-3.22-4.70

Since T(R,O) Grasp is designed for a single contact region, we also evaluate both methods in that setting: the region is the 64 object-grid points around one randomly chosen GT contact. With a single region, the contact loss applies exactly as in the paper, including the Gaussian heatmap and the geodesic term. PointDex receives either the region center or four points from the region (its center and three farthest-point samples). Here, d_obj is the distance to the region center / to the closest point of the region.

MethodSteering inputDexonomy SR (%)Dexonomy d_obj (mm)DexGYS SR (%)DexGYS d_obj (mm)
T(R,O) Grasp, re-implementationOne local region12.4732.30 / 21.2746.7621.39 / 12.27
PointDex, 1 pointOne local region37.596.35 / 4.0083.227.63 / 4.43
PointDex, 4 pointsOne local region35.295.31 / 2.2782.956.76 / 3.16

Even with the input it is designed for, T(R,O) Grasp stays 21.27 / 12.27 mm from the region on Dexonomy / DexGYS, and its SR drops to 12.47% / 46.76%. PointDex with four points reaches 2.27 / 3.16 mm at 35.29% / 82.95% SR. Across both settings, T(R,O) Grasp either follows the contacts or keeps its SR, but not both: its guidance steers the generated link poses through test-time gradients, trading one for the other, whereas PointDex conditions generation on the contacts as point tokens. The implementation details are added to App. A.1.6 (Table 11).

DextER distances. The gap comes from how the released checkpoints were trained, not from our setup. Unlike the released Dexonomy checkpoint, the released DexGYS checkpoint was trained with contact_position_dropout = 0.5: in half of the training samples, contact positions are dropped and only the contact link names remain. We suspect this teaches the model to treat positions as optional and follows them loosely. We retrained DextER on DexGYS with dropout 0 using the released code. It gets closer to the contacts (d_obj / d*_pair 9.77 / 13.25 mm, vs. 17.23 / 31.10 mm) with similar SR (62.41% vs. 61.15%), but remains well above PointDex (5.62 / 6.14 mm). The retrained numbers are reported in Table 12 of the revised manuscript.

Model (DexGYS, pair steering*)Training contact dropoutd_obj (mm)d*_pair (mm)SR (%)
DextER, released checkpoint0.517.2331.1061.15
DextER, retrained09.7713.2562.41
PointDex-5.626.1482.89
GT floor-4.705.02-
AuthorsOpenReview ↗

Response to reviewer jjhj [3/4]

(W4, Q3) The main SR comparison is not apples-to-apples

In the submission, Table 1 took the baseline numbers from the DextER paper because DextER is the only baseline that released training/test code and checkpoints, so the other baselines could not be re-run. To keep the setup consistent with those numbers, we evaluated PointDex with DextER’s official evaluation code, which we confirmed reproduces DextER’s reported SR on its released checkpoint with its default greedy decoding (67.71% vs. 67.14%).

To address the concern directly, we re-evaluated DextER and PointDex with this same pipeline and added T(R,O) Grasp, as requested by the reviewer. All three methods are evaluated over three random seeds, with DextER sampled at temperature 2.0 (65.17% SR, Table 1). T(R,O) Grasp has no language input, so we condition it on language by projecting PointDex’s frozen CLIP text embedding and concatenating it to T(R,O) Grasp’s timestep embedding.

Under this shared protocol, PointDex is best on every metric except P-FID (see W3), and its SR exceeds that of the strongest baseline, T(R,O) Grasp, by 10.0 points (83.21% vs. 73.19%). For the camera-ready version, we will re-evaluate the remaining baselines the same way.

Method (DexGYS)P-FID ↓CD ↓Con. ↓SR (%) ↑Q1 ↑Pen. (cm) ↓
DextER (temperature 2.0)0.228 ± 0.0121.683 ± 0.0550.369 ± 0.00365.17 ± 0.610.780 ± 0.0020.405 ± 0.001
T(R,O) Grasp1.826 ± 0.0331.214 ± 0.0070.262 ± 0.00173.19 ± 0.101.220 ± 0.0040.179 ± 0.001
PointDex (Ours)0.351 ± 0.0071.042 ± 0.0180.233 ± 0.00183.21 ± 0.271.567 ± 0.0060.135 ± 0.001

(W5, Q5) Steering contacts come from the GT grasp

To evaluate steering with contacts that are not the ground-truth ones, we add noise to the contacts while keeping them consistent with the language instruction. To avoid contacts that contradict the instruction, we take the contact region of the ground-truth grasp, that is, the object points within 18.3 mm of the posed hand and the hand points within 18.3 mm of the object. We then sample k points uniformly at random within this region, on the object side or the hand side. The sampled contacts follow the intent of the instruction, but they differ from the ground-truth contacts. Results on the full DexGYS test set with k = 1:

Steering contacts (k = 1)SR (%)d_obj (mm)d_hand (mm)
None83.21––
Object, GT contact83.1019.9 → 7.5–
Object, sampled in GT region83.4422.1 → 11.2–
Hand, GT contact83.10–8.2 → 6.4
Hand, sampled in GT region82.77–13.4 → 11.8

d_obj and d_hand are computed as in Table 5, and a → b compares grasps generated without and with steering on the same points. The GT-contact rows give one ground-truth contact, and the sampled rows give one random point of the ground-truth contact region. With the sampled contacts, SR stays at the unsteered level, and the grasps still move toward the given points (d_obj 22.1 → 11.2 mm, d_hand 13.4 → 11.8 mm), by a margin similar to that with the ground-truth contacts. Steering therefore remains effective even when the contacts are not taken from the exact ground-truth contact locations.

(W6, Q6) No comparison with simpler noise schemes

We trained the three alternatives the reviewer suggested: (a) one shared noise level steered by inpainting, (b) one noise level per contact pair, and (c) per-point noise with q = 0 on DexGYS under the same setting. Shared noise with RePaint-style inpainting keeps SR but barely steers (d_obj 17.69 mm, d_pair 40.67 mm). In inpainting, the given contacts are re-noised to the current noise level at every step, so the network never receives their exact positions. With per-point noise, the network receives the given contacts clean at every step. One noise level per pair slightly improves pair steering (d_pair 6.22 mm) but breaks object-only steering: SR drops to 79.54%, 3.7 points lower, and d_obj rises to 8.54 mm. With q = 0, SR drops by 0.9–1.9 points and the distances grow by 0.6–1.3 mm in every mode. Per-point noise with q = 0.08 is the only scheme that keeps both SR and steering accuracy in every mode.

Noise scheme (DexGYS)No steering SR (%)Object-only SR (%) / d_obj (mm)Hand-only SR (%) / d_hand (mm)Pair SR (%) / d_pair (mm)
Per-point noise, q = 0.08 (ours)83.2183.22 / 6.2683.23 / 5.8581.13 / 7.43
(a) Shared noise + inpainting82.3782.56 / 17.6983.63 / 7.1082.70 / 40.67
(b) Shared noise per pair82.5879.54 / 8.5482.56 / 7.3280.47 / 6.22
(c) Per-point noise, q = 082.2681.44 / 7.3481.34 / 6.4280.24 / 8.71
GT floor-- / 4.70- / 3.96- / 4.98
Reviewer_mNCeOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

PointDex represents dexterous grasps entirely as 3D points: hand keypoints, object contact points, and hand contact points, jointly denoised by one diffusion Transformer. Independent per-group noise levels let a single model support object, hand, and paired-contact steering at inference. It is evaluated on DexGYS and Dexonomy for the Shadow Hand, with language-conditioned success rate, steering fidelity, and grasp-quality metrics, plus ablations. The headline result is a large gain in simulated grasp success rate.

Critical strengths
  1. Keypoints, object contacts, and hand contacts living in the same 3D space removes the rotation discontinuities and nonlinear parameterizations of joint angles and makes proximity, penetration, and the "what touches where" relation direct geometric computations. The independent noise levels are the key mechanism that turns one trained model into object-level, hand-level, and correspondence-level steering, which is exactly the capability prior steering interfaces expose only through separate, single-mode models.
  2. On DexGYS, success rate rises from 67.14 (DexER) to 83.76 (+16.62), with better Chamfer distance (1.03 vs. 1.46), contact metric (0.23 vs. 0.34), grasp quality Q1 (1.56 vs. 0.89), and penetration (0.13 vs. 0.37), while diversity is essentially tied. Steering fidelity is strong: pair distance drops from 31.10 mm (DexER) to 6.14 mm, within roughly 1.1 mm of the nearest-point ground-truth floor of 5.02 mm (Table 4), and the same pattern holds on Dexonomy (Table 7). The representation ablations (Tables 2–3) show that both keypoints and contacts are necessary, which is the right evidence for the central claim.
Critical weaknesses
  1. Every result is for the Shadow Hand on DexGYS/Dexonomy in Isaac Gym and MuJoCo; there is no real-robot grasping, no second hand, and no human-grasp data. The paper acknowledges this, but simulation success rates are an optimistic measure of real reliability, and the claimed hand-agnosticism of the representation is not demonstrated on any other hand.
  2. Table 1 states that baseline numbers are taken from DexER. For the comparison that carries the +16.6 claim, the authors therefore rely on another paper's reported figures rather than a unified evaluation pipeline, leaving open differences in simulator setup, test protocol, and checkpoint selection. The steering baselines (Tables 4, 7) are evaluated from checkpoints by the authors, but the most important generation table is not.
  3. PointDex's P-FID is 0.33 against DexER's 0.20 (Table 1), the one metric where the strongest baseline is better, and the paper does not discuss this apparent success-versus-coverage tradeoff. More broadly, contact-aware grasp generation and contact steering interfaces are well established (RegionGrasp, MaxkDexGrasp, ClickDiff, DexER, ContactGen), and the individual components — rectified flow, DETR slot matching, the Langevin corrector, pair attention — are existing; the contribution is the unified point representation and the noise-level conditioning, not a new generative model.
Questions for authors

Why does P-FID worsen (0.33 vs. 0.20) while success and quality improve substantially? Is PointDex concentrating probability on a narrower set of successful modes? Please report mode coverage/recall and discuss the quality–diversity tradeoff. Can you re-run the generation baselines in Table 1 under your exact evaluation pipeline (same simulator, contact/penetration thresholds, and test manifest), rather than quoting DexER's numbers, and confirm the success-rate gap is unchanged? How large is the residual between the generated keypoints and the IK-recovered articulated pose, and do the steering distances change if contacts are measured on the IK-recovered hand rather than on the generated keypoints? The representation is described as hand-agnostic. Can you demonstrate transfer to a second hand (or to human grasp data), even at small scale, and do you have any real-robot validation planned?

3 comments
AuthorsOpenReview ↗

Response to reviewer mNCe [1/2]

(W1, Q4) Shadow Hand only, simulation only

We thank the reviewer for this suggestion. We address it with two new experiments, described in G2. First, we trained PointDex on the Inspire hand, whose kinematics differ from those of the Shadow Hand. Adapting PointDex required only a new keypoint layout, canonical hand point cloud, and IK. PointDex reaches 1.6 times DextER's SR without steering and 1.8 times with pair steering (Table R1), so the gains of PointDex are not specific to the Shadow Hand. Second, we deployed our model on a real robot, an OpenArm arm with an Inspire RH56F1 hand. There it succeeded in 24 of 40 trials, including 9 of 20 on unseen objects (Table R2). We leave the extension to human grasp data for future work.

(W2, Q2) Table 1 baselines are copied from DextER

In the submission, Table 1 took the baseline numbers from DextER, the only baseline that released training/test code and checkpoints. We evaluate PointDex with DextER's official evaluation code. We’ve confirmed this code on the released DextER checkpoint reproduces its reported SR with its default greedy decoding (67.71% vs. 67.14%). We re-evaluated DextER and PointDex with the same pipeline and also added T(R,O) Grasp as requested by the reviewer jJhj. All three methods are evaluated over three random seeds, with DextER sampled at temperature 2.0 (65.17% SR, Table 1). Since T(R,O) Grasp has no language input, we project PointDex's frozen CLIP text embedding and concatenate it to its timestep embedding. PointDex is best on every metric except P-FID (see W3), and its SR exceeds that of the strongest baseline, T(R,O) Grasp, by 10.0 points (83.21% vs. 73.19%). The remaining baselines will be re-evaluated the same way for the camera-ready version.

Method (DexGYS)P-FID ↓CD ↓Con. ↓SR (%) ↑Q1 ↑Pen. (cm) ↓
DextER (temperature 2.0)0.228 ± 0.0121.683 ± 0.0550.369 ± 0.00365.17 ± 0.610.780 ± 0.0020.405 ± 0.001
T(R,O) Grasp1.826 ± 0.0331.214 ± 0.0070.262 ± 0.00173.19 ± 0.101.220 ± 0.0040.179 ± 0.001
PointDex (Ours)0.351 ± 0.0071.042 ± 0.0180.233 ± 0.00183.21 ± 0.271.567 ± 0.0060.135 ± 0.001

(W3-1, Q1) P-FID is worse than DextER's

Please see G3. In brief, PointDex trades some diversity for quality, but even after this trade its diversity stays on par with the baselines. With the corrector, recall falls from 0.853 to 0.777, while coverage remains at DextER's level (0.958 vs. 0.952) and well above that of T(R,O) Grasp (0.785). This concentration raises SR from 77.02% to 83.21%. All four metrics are computed from k-nearest neighbors in the P-FID feature space, with k = 3 for precision and recall and k = 5 for density and coverage. This analysis and the discussion of the trade-off are added to the corrector ablation.

(W3-2) The components already exist

Please see our G1 in general response. As the reviewer notes, the contribution is the shared point representation rather than a new generative model. The General Response clarifies this contribution and explains why each established component is needed.

While the individual components are established, PointDex is the first to provide a unified steering interface for dexterous grasp generation. Because the pose and both sides of every contact are 3D points with independent noise levels, one model accepts object contacts, hand contacts, or full correspondences, in any number and together with language. Each prior method supports one fixed form of control:

  • RegionGrasp takes a single object region and always places the thumb tip on it, so the user cannot choose the hand part.
  • ClickDiff requires both sides of every contact, with the hand side given only per finger.
  • ContactGen treats contact only as an internal variable sampled from the object and offers no user control.
  • MaskDexGrasp re-samples selected fingers of an existing grasp and takes no contact location.
  • DextER is the closest: it uses the Shadow Hand and language, and accepts contacts as a token prefix. However, each contact is one entry that couples a hand link with an object position. It therefore cannot take object contacts alone, which is why it has no object-only results in Tables 4 and 7. It also specifies the hand side only as a whole link, so it cannot target a specific point on the hand.

Contact steering in prior work is therefore established only in single, fixed forms; an interface that unifies them within one model is not.

AuthorsOpenReview ↗

Response to reviewer mNCe [2/2]

(Q3) IK residual, and steering distances measured on the IK-recovered hand

IK residual. For each sample, we run forward kinematics on the IK-recovered Shadow Hand pose and compute the distance to the 29 generated keypoints. The distance is averaged over keypoints (DexGYS test set, 9,971 grasps, one sample per input).

Settingsmeanmedianmean > 10 mm
w/o contact steering1.06 mm0.88 mm0.14%
pair steering1.20 mm1.02 mm0.14%

For reference, the same IK applied to the ground-truth keypoints gives a residual of 0.07mm.

Steering distances. All steering distances in the paper are already measured on the IK-recovered hand, so they include any IK residual. We pose the canonical hand point cloud with the IK-recovered joint angles and measure contact distances on it (Appendix A.1.4). Measuring them on the generated keypoints is not possible: the 29 keypoints are sparse and do not define the hand surface, where contacts lie.

Dear Reviewer mNCe,

Thank you again for your review. We know this is a busy period, so before the discussion closes on November 18 (AoE), here is a brief summary of how our response and the revised manuscript address each of your concerns:

  • Second hand and real robot (W1, Q4). We trained PointDex on a second hand, the Inspire hand, changing only the keypoint layout, the canonical hand point cloud and the IK. It reaches 28.87% SR vs. 18.01% for DextER trained on the same data, and 36.32% vs. 20.04% with pair steering. On a real robot (OpenArm with an Inspire RH56F1 hand), PointDex succeeds in 24 of 40 trials vs. 13 of 40 for DextER, including 9 of 20 vs. 3 of 20 on unseen objects (Sec. 4.6, App. A.2.5–A.2.6).

  • Same evaluation pipeline as DextER (W2, Q2). In the submission, PointDex was evaluated with DextER's official evaluation code, which reproduces DextER's reported SR on its released checkpoint (67.71% vs. 67.14%). In the revision, we re-evaluated DextER and PointDex with this same pipeline over three seeds and added T(R,O) Grasp (Table 1). PointDex reaches 83.21 ± 0.27% SR vs. 73.19 ± 0.10% for T(R,O) Grasp and 65.17 ± 0.61% for DextER (sampled at temperature 2.0), and the headline result is now based on this evaluation (+10.0 points over the strongest baseline).

  • P-FID and diversity (W3, Q1). Without the Langevin corrector, PointDex has a lower P-FID than DextER (0.144 vs. 0.228), and its SR (77.02%) is still above every baseline. The corrector concentrates samples in high-density regions and raises SR to 83.21%, while coverage stays at DextER's level (0.958 vs. 0.952), so no modes are dropped. We added this analysis to the corrector ablation.

  • Contribution (W3). We revised Sec. 1 to clarify the contributions: the shared 3D-point representation of the pose and the contacts, and the unified multi-mode steering it enables (object-only, hand-only and pair contacts in one model). We also discussed how PointDex differs from prior contact-conditioned grasp generation methods in our response. A new ablation shows that the point representation drives the gain: replacing joint angles with keypoints raises SR by 34.1 points without contacts and by 19.6 points with them.

  • IK residual (Q3). The fitted Shadow Hand lies on average 1.06 mm from the generated keypoints (1.20 mm with pair steering), and only 0.14% of samples exceed 10 mm. All steering distances in the paper are measured on the IK-recovered hand, so they include this residual.

If any point is still unclear, we would be glad to address it. If your concerns are resolved, we would be grateful if you would consider updating your score.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response to all reviewers [1/4]

We thank all reviewers for their constructive feedback. We appreciate that they highlighted (1) the point representation without rotation parameterizations, (2) a single model that supports all three steering modes, and (3) strong and consistent results on both benchmarks, with steering close to the ground-truth floor.

Following these suggestions, we have updated the manuscript, with all changes highlighted in blue. Below, we summarize the key additions; detailed answers to each reviewer appear in the individual threads.

(1) Additional experiments

  • Real robot experiment: Sec. 4.6, Sec. A.2.6
  • New hand (Inspire hand) experiment: Sec. 4.6, Sec. A.2.5
  • New baseline method (T(R,O) Grasp): Tab. 1

(2) Additional analyses

  • Mean and standard deviations for the experiments (Tab. 1, 3, 6, 13, 14)

Some concerns were raised by more than one reviewer. We address them below, each tagged with the comments it answers, and the individual threads refer to these items.

AuthorsOpenReview ↗

General response to all reviewers [2/4]

G1. Contribution and novelty (mNCe W3-2; jJhj W1, W2, Q1, Q2; i4G9 W1)

Contribution. The core contribution of PointDex is to introduce a structured 3D point representation for grasp pose (as hand keypoint coordinates) and hand–object contact (as corresponding point coordinate pairs between the hand and the object) in a shared 3D coordinate frame. This allows the model to effectively learn the hand-object-contact relations and easily quantify proximity, penetration, and contact as distances between 3D points, rather than functions of rotations and forward kinematics [1, 2, 3, 4]. Our experiments show that it improves both the quality of unsteered grasps and the accuracy of steered grasps. Furthermore, we propose a unified transformer decoder that jointly denoises the hand keypoints and hand-object contact pairs such that each of them has its own noise level. This allows any subset of contacts to be freely fixed, serving as a grasping condition: one model that steers with object-only, hand-only, or pair contacts without retraining (Tables 4, 7). The table below ablates on how the pose and contacts are represented, and all four models are trained without the local geometry embedding (DexGYS):

Pose representation / use of contactsNo steering SR (%)Pair SR (%)d_obj (mm)d_hand (mm)d_pair (mm)
Joint angles w/o contacts45.89----
Joint angles w/ contacts63.3063.9310.049.3120.57
Keypoints w/o contacts79.94----
Keypoints w/ contacts82.8980.955.394.547.38
GT floor--4.703.964.98

These results show

  • The point-based representation of grasp pose gives the largest gain: Replacing joint angles with keypoints raises success rate by 34.1 percentage points without contact point pairs (45.89% -> 79.94%) and by 19.6 points with them (63.30% -> 82.89%).
  • Contact point pairs help with both pose representations: They add 17.4 points with joint angles and 3.0 with keypoints. Joint angles describe the hand in its own kinematic frame, so without contacts nothing ties the hand to the object surface, and the contact pairs supply this link. Keypoints already place the hand in the object frame, and the keypoint model already reaches 79.94% without contacts, which leaves less room for improvement, so the contacts add less on top of them.
  • The shared 3D point representation matters most for steering. Given the same contacts, the keypoint model ends up 1.9 ~ 2.8 times closer to them (d_pair 7.38 vs 20.57mm), at a much higher SR (80.95% vs. 63.93%).

We revised Sec. 4.2, which had credited the gain mainly to the contacts.

Difference from prior work. No prior method allows one model to follow contacts on the object side, the hand side, or both, at the level of individual points. PointDex enables it by representing the pose and the contacts as 3D points in one shared frame. The two closest methods show where the difference lies:

  • DextER [5] represents each contact as a hand-link token followed by its 3D position. Every position is tied to a whole link, so that DextER cannot take object-side contacts alone or target a specific point on a link.
  • UGG [6] also generates the object, the hand, and the contacts jointly, but with a different representation. Its hand is a pose with joint angles rather than points, and its contact anchors are points on the object near the hand, with no corresponding point on the hand. It also gives all anchors one shared noise level, so it can condition only on a whole modality. PointDex represents the pose as hand keypoints and each contact as a hand–object point pair in a shared 3D coordinate frame, and gives every point its own noise level so that any subset of points on either side can serve as the condition.

We do not claim the individual components as new. Contacts in grasp generation [5, 6, 7, 8, 9, 10], per-token noise levels [6, 11, 12], set matching [13], and rectified flow on point sets [14] all come from prior work. What is new is how they are combined in a coherently structured point representation for effective steerable grasping:

  • Keypoints put the hand pose in the object frame as 3D points.
  • Each contact is a hand–object point pair in the same frame.
  • Each of these points has its own noise level, so any subset of contacts can be fixed as the condition.

Together, these enable one model to support object-only, hand-only, and pair steering, and in each mode it follows the given contacts more closely than the strongest baseline for that mode (Tables 4, 7).

Revisions. We updated the contributions in Sec. 1 to (i) a shared 3D-point representation of the grasp pose and the contacts and (ii) unified multi-mode steering enabled by it. We also updated Sec. 4.2 with the ablation above and the related work to credit Diffusion Forcing and contrast PointDex with UniDiffuser, UGG, and DextER.

AuthorsOpenReview ↗

General response to all reviewers [3/4]

G2. A different robot hand and real-robot experiments (mNCe W1; i4G9 W2, Q4)

To test whether PointDex generalizes beyond the Shadow Hand and beyond simulation, we trained it on the Inspire hand and evaluated it both in simulation and on a real robot.

Inspire hand in simulation. We synthesize the Inspire-hand data with the grasp generator of Dexonomy [15] on the Dexonomy objects. For each grasp type, the generator fits the hand template of that type to the object surface and refines the grasp in MuJoCo. Only grasps that pass the simulation filtering of Dexonomy are kept. Each grasp carries a contact annotation and a templated instruction that names its grasp type (App. A.2.5). We train on 182,061 grasps and evaluate on 12,929 held-out grasps of objects seen in training, measuring success rate in MuJoCo following DextER's Dexonomy evaluation. The Inspire hand has 12 joints but only 6 actuators. The other 6 are mechanically coupled. Our IK fits the wrist to the palm keypoints, picks each finger's joint angles from a precomputed angle-to-keypoint lookup table, and sets the coupled joints from the hand’s coupling equations. Adapting PointDex to this hand requires only a new keypoint layout, canonical hand point cloud and IK; the architecture and sampler are unchanged. We train both DextER and PointDex from scratch on this data, with all other training settings identical to those of the original Dexonomy experiments. PointDex reaches 1.6 times DextER’s SR without steering (28.87% vs. 18.01%) and 1.8 times with pair steering (36.32% vs. 20.04%), and gains more from steering (+7.5 vs. +2.0 points). This confirms that PointDex’s gains are not specific to the Shadow Hand.

Table R1. Dexonomy-Inspire Evaluation

MethodNo steeringPair
DextER18.0120.04
T(R,O) Grasp19.56–
PointDex28.8736.32

Real robot. To further validate PointDex beyond simulation, we ran real-robot experiments with the Inspire hand.

  • Hardware: The setup consists of OpenArm 7-DoF arm with an Inspire RH56F1 hand and a single ZED Mini camera on the robot head.
  • Eval objects: For evaluation, we use objects from AutoDex [16]. We chose it because it releases the object meshes and its household objects are commercially available, so we can synthesize training grasps on the same meshes and test on the same physical objects.
  • Model training: We pretrain PointDex on 74,852 RH56F1 grasps that we synthesize on the Dexonomy objects with the Dexonomy grasp generator and fine-tune it on RH56F1 grasps that we synthesize on the AutoDex objects with the Dexonomy grasp generator. The model then generates grasps in the object frame without steering.
  • Pipeline: Each trial runs three steps. (1) FoundationPose [17] estimates the object pose from the camera image; the camera is not an input to PointDex. (2) PointDex generates grasps in the object frame without steering, and the grasps are transformed to the robot frame using the estimated pose. (3) One grasp that admits an arm IK solution and a collision-free approach is executed.

We test four AutoDex objects, two seen during training and two unseen, with ten trials per object. A trial succeeds if the object is lifted by 10 cm and held for 2 s. PointDex succeeds in 24 of 40 trials: 15 of 20 on seen objects and 9 of 20 on unseen objects (Table R2). We added this experiment to the revised manuscript (Sec. 4.6 and App. A.2.6), and we leave the extension to human grasp data for future work.

Table R2. Real-robot evaluation (Inspire RH56F1)

SplitObjectDextERPointDex
seenattached_container5/106 / 10
seenplant_mister5/109 / 10
unseencontainer_pink0/105 / 10
unseenknife_sharpener3/104 / 10
All13/4024/40
AuthorsOpenReview ↗

General response to all reviewers [4/4]

G3. Effect of the Langevin corrector (mNCe W3-1; jJhj W1, W8, Q1, Q8, Q9; i4G9 W1, Q3)

The Langevin corrector controls a trade-off between diversity and grasp quality, but it is not the source of PointDex's advantage over the baselines.

P-FID and the quality–diversity trade-off. On PointDex, the corrector pushes samples toward high-density regions of the learned distribution. P-FID penalizes both low fidelity and low diversity, so we also measure precision and recall [18] and density and coverage [19], which score fidelity (precision, density) and diversity (recall, coverage) separately. The corrector raises precision and density and lowers recall, while coverage stays at DextER's level (0.958 vs. 0.952). Samples therefore concentrate without dropping modes. Without the corrector, PointDex's P-FID (0.144) is below DextER's (0.228):

Method (DexGYS)CorrectorSR (%) ↑P-FID ↓Precision ↑Recall ↑DensityCoverage ↑
DexGYSNet✗57.924.900.6470.6460.6040.709
DexGYSNet✓63.743.370.7280.6000.7600.756
DextER–65.170.2280.8310.8780.9160.952
T(R,O) Grasp✗73.191.830.9100.5761.1240.785
T(R,O) Grasp✓73.080.640.8950.7681.0700.942
PointDex✗77.020.1440.9000.8531.0710.970
PointDex✓83.210.3510.9260.7771.1660.958

PointDex outperforms the baselines regardless of the corrector. Without the corrector, PointDex reaches 77.02% SR, above every baseline, including DexGYSNet with the corrector. The corrector does not help every model in the same way. Added to T(R,O) Grasp with the same settings, it leaves SR unchanged (73.19% → 73.08%). T(R,O) Grasp covers the data distribution narrowly (recall 0.576), so the corrector spreads its samples instead of concentrating them. Only a model that has already learned a wide-coverage distribution, such as PointDex (coverage 0.970), can turn this concentration into higher quality and SR. Note that DextER decodes autoregressively and has no score function, so the corrector cannot be applied to it.

References

[1] Y. Xu, W. Wan, J. Zhang, H. Liu, Z. Shan, H. Shen, R. Wang, H. Geng, Y. Weng, J. Chen, T. Liu, L. Yi, H. Wang. UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policy. CVPR, 2023.

[2] Z. Weng, H. Lu, D. Kragic, J. Lundell. DexDiffuser: Generating Dexterous Grasps with Diffusion Models. IEEE RA-L, 2024.

[3] Y. Zhong, Q. Jiang, J. Yu, Y. Ma. DexGrasp Anything: Towards Universal Robotic Dexterous Grasping with Physics Awareness. CVPR, 2025.

[4] Y. Zhou, C. Barnes, J. Lu, J. Yang, H. Li. On the Continuity of Rotation Representations in Neural Networks. CVPR, 2019.

[5] J. Lee, E. Park, M. Cho. DextER: Language-Driven Dexterous Grasp Generation with Embodied Reasoning. arXiv:2601.16046, 2026.

[6] J. Lu, H. Kang, H. Li, B. Liu, Y. Yang, Q. Huang, G. Hua. UGG: Unified Generative Grasping. ECCV, 2024.

[7] S. Liu, Y. Zhou, J. Yang, S. Gupta, S. Wang. ContactGen: Generative Contact Modeling for Grasp Generation. ICCV, 2023.

[8] Y. Wang, C. Guo, L. Cheng, H. Jiang. RegionGrasp: A Novel Task for Contact Region Controllable Hand Grasp Generation. ECCV, 2024.

[9] P. Li, Z. Wang, M. Liu, H. Liu, C. Chen. ClickDiff: Click to Induce Semantic Contact Map for Controllable Grasp Generation with Diffusion Models. ACM MM, 2024.

[10] B. Zuo, L. Zhou, H. Xu, J. Yan, Z. Yu, Z. Liu, Y. Wang. MaskDexGrasp: Generative Masked Modeling for Part-Aware Dexterous Grasp Synthesis. CVPR, 2026.

[11] B. Chen, D. Martí Monsó, Y. Du, M. Simchowitz, R. Tedrake, V. Sitzmann. Diffusion Forcing: Next-Token Prediction Meets Full-Sequence Diffusion. NeurIPS, 2024.

[12] F. Bao, S. Nie, K. Xue, C. Li, S. Pu, Y. Wang, G. Yue, Y. Cao, H. Su, J. Zhu. One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale. ICML, 2023.

[13] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko. End-to-End Object Detection with Transformers. ECCV, 2020.

[14] T. Sun, L. Zhu, S. Huang, S. Song, I. Armeni. Rectified Point Flow: Generic Point Cloud Pose Estimation. NeurIPS, 2025.

[15] J. Chen, Y. Ke, L. Peng, H. Wang. Dexonomy: Synthesizing All Dexterous Grasp Types in a Grasp Taxonomy. RSS, 2025.

[16] M. Choi, G. Kim, J. Kim, T. Kim, T. Ha, J. Lim, H. Joo. AutoDex: An Automated Real-World System for Dexterous Grasping Data Collection. arXiv:2606.23689, 2026.

[17] B. Wen, W. Yang, J. Kautz, S. Birchfield. FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects. CVPR, 2024.

[18] T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, T. Aila. Improved Precision and Recall Metric for Assessing Generative Models. NeurIPS, 2019.

[19] M. F. Naeem, S. J. Oh, Y. Uh, Y. Choi, J. Yoo. Reliable Fidelity and Diversity Metrics for Generative Models. ICML, 2020.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.