- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PointDex targets language-conditioned dexterous grasp generation for the Shadow Hand. It represents the grasp pose as 29 posed hand keypoints and the contacts as 16 slots, each holding a pair of points: one on the object surface and one on the canonical open hand. All of these are 3D point tokens, and a DiT trained with rectified flow denoises them jointly under Diffusion Forcing-style per-token noise levels. The method also uses DETR-style slot matching, existence prediction, surface-sampled dummy contacts, a pair attention bias, and a Langevin corrector. Kabsch alignment and QP-based IK then recover the wrist pose and joint angles. Any subset of contact tokens can be held clean as a condition, so a single model supports object-only, hand-only, and paired contact steering. On DexGYS, PointDex reaches a success rate of 83.76% versus 67.14% for DextER, with higher Q1 and lower penetration. Under steering it greatly reduces contact distances, e.g., d_pair drops from 31.10 to 6.14 mm. It shows similar gains on Dexonomy, including the unseen-grasp-type splits, where its success rate is about 33% versus about 11% for DextER.
Critical strengths
- PointDex represents the grasp pose, object contacts, and hand contacts all as 3D point tokens, which avoids rotation parameterizations and implicit forward kinematics in the pose representation. Combined with per-token noise levels, a single model supports object-only, hand-only, and paired contact steering with any number of steering contacts, without extra modules or retraining. This matters to my assessment because prior methods each support only one steering interface, whereas here steering reduces to conditioning on a subset of tokens.
- Strong and consistent empirical results. PointDex clearly outperforms prior methods in unsteered generation, with higher success, better grasp quality, and less penetration. Under steering, where all methods receive the same contacts, it follows the specified contacts more closely in every mode while also achieving higher success. These advantages hold on both benchmarks and extend to unseen grasp types, which suggests good generalization. The ground-truth floor also gives the contact distances a clear reference. These results are the main basis for my positive assessment.
Critical weaknesses
- Unclear attribution of gains. The pose-only variant already reaches 79.81% SR, 12.7 points above DextER, and contact tokens add only about 4 points. Replacing keypoints with joint angles (−19.1), removing the Langevin corrector (−7.3), and removing the pair bias (−5.6) all matter more, yet the latter two are absent from the contribution narrative. The paper attributes the keypoint advantage to sharing a frame with the contacts, but no joint-angle variant without contact tokens tests this, and the joint-angle variant itself is not described. This matters because the paper's central claim is that the shared point representation drives grasp quality.
- The role of language and the practical scope are not established. Although the model is conditioned on language, no ablation removes or swaps the instruction, so it is unclear how much the results depend on it. All experiments use the Shadow Hand in simulation, none use single-view or noisy point clouds, and IK residuals, IK failure rates, and inference time are not reported. This matters because these factors determine how far the benchmark results carry over to practical use.
Questions for authors
- Many ablation gaps are within 1–5 points, and a few are not fully monotonic; for example, removing the pair bias gives 78.14%, slightly below the pose-only 79.81%. It would help to see the mean and standard deviation over several seeds for Tables 1, 3, 6, and 9.
- Per-token noise levels are the key design behind multi-mode steering. Could the authors compare against a simple alternative: the same backbone trained with a shared noise level and steered at inference by inpainting or guidance? This would show more directly why the design is needed.
- The Langevin corrector scales the injected noise by (1−t)^p, which, relative to standard Langevin dynamics, effectively lowers the sampling temperature late in sampling and may concentrate samples in high-density regions. Could the authors also report P-FID and diversity for the ablations, especially the one without Langevin, to show whether part of the SR gain comes at the cost of diversity? In addition, PointDex's P-FID is less favorable than DextER's in Table 1 and on the seen and unseen-object splits of Table 11, but better on the two unseen-grasp splits. Do the authors have an explanation for this?
- Do the authors have any real-robot results, or preliminary results with single-view or noisy point clouds? Positive results here would further strengthen my assessment of the paper.
- If the authors can address points 1–4 during the discussion period, I would be happy to reconsider my score.
5 comments
Response to reviewer i4G9 [2/3]
(W2-3, Q4) Other hands and real-robot results
To test another hand, we trained PointDex on the Inspire hand, which has 12 joints driven by only 6 actuators (the other 6 are mechanically coupled). The data uses the Dexonomy objects with newly synthesized Inspire-hand grasps, each paired with a templated instruction that names its grasp type: 182,061 grasps for training and 12,929 held-out grasps of objects seen in training for evaluation, with SR measured in MuJoCo as in DextER's Dexonomy evaluation. Only the keypoint layout, the canonical hand point cloud and the IK change; the architecture and sampler are unchanged. PointDex reaches 28.87% SR without steering and 36.32% with pair steering, against 18.01% and 20.04% for DextER trained on the same data. We also ran real-robot experiments with the Inspire RH56F1 hand, where PointDex succeeds in 24 of 40 trials (15 of 20 on seen objects and 9 of 20 on unseen ones); please see G2 for the setup and per-object results.
(W2-4, Q4) Partial point clouds
We evaluate the DexGYS-trained PointDex zero-shot on simulated partial observations. For each test input, two virtual cameras view the object: one 0.3 m behind the ground-truth wrist, facing the object, and one 0.3 m above and to the side of the midpoint between the wrist and the object. Only the points visible from either camera are kept, found by hidden-point removal (Katz et al., 2007); this leaves 42% of the cloud (median), which is resampled back to 10,000 points. Only the model input changes: the model is not retrained, and all metrics are computed against the full object, as in Table 1.
| Object cloud | SR (%) ↑ | P-FID ↓ | CD ↓ | Con. ↓ | Q1 ↑ | Pen. (cm) ↓ |
|---|---|---|---|---|---|---|
| Full | 83.21 | 0.351 | 1.042 | 0.233 | 1.567 | 0.135 |
| Partial | 82.10 | 0.345 | 1.149 | 0.254 | 1.304 | 0.220 |
PointDex keeps its SR and P-FID on partial point clouds (83.21% → 82.10%, 0.351 → 0.345), while other metrics degrade moderately. Because one camera sits behind the GT wrist, as in DextER's protocol, the visible part usually includes the side the hand approaches from.
References
- S. Katz, A. Tal, R. Basri. Direct Visibility of Point Sets. ACM Transactions on Graphics (Proc. SIGGRAPH), 26(3), 2007.
(Q1) Seeds and variance for the ablations
We have updated Tables 1, 3, 6, and 9 (Tables 1, 3, 6, and 13 in the revised manuscript) to report results as mean ± standard deviation over three random seeds; please refer to the revised manuscript for details. Some baselines in Table 1 don’t have publicly available official training code, checkpoints, or both, so we could not run them across multiple seeds during the rebuttal period. We will re-implement and train these baselines and report their mean ± standard deviation in the camera-ready version. Regarding the number of generated samples, we use one sample per input.
(Q2) Comparison with a shared noise level and inpainting
We trained the same backbone with one shared noise level for all points, using the same data and schedule, and steered it at inference with RePaint-style inpainting: at each step, the given points are re-noised to the current noise level and written back. Results on DexGYS with all contacts given:
| Noise scheme (DexGYS) | No steering SR (%) | Object-only SR (%) / d_obj (mm) | Hand-only SR (%) / d_hand (mm) | Pair SR (%) / d_pair (mm) |
|---|---|---|---|---|
| Per-point noise (ours) | 83.21 | 83.22 / 6.26 | 83.23 / 5.85 | 81.13 / 7.43 |
| Shared noise + inpainting | 82.37 | 82.56 / 17.69 | 83.63 / 7.10 | 82.70 / 40.67 |
Inpainting keeps SR but barely follows the contacts: d_obj is 17.69 mm instead of 6.26 mm, and d_pair 40.67 mm instead of 7.43 mm. With per-point noise levels, the given points stay clean at every step, so the network conditions on them directly instead of on noised copies.
Response to reviewer i4G9 [3/3]
(Q3) The corrector may concentrate samples at the cost of diversity
Concentration in high-density regions. Yes. Turning off only the corrector in the same model lowers P-FID from 0.351 to 0.144, below DextER’s 0.228. In the P-FID feature space, the corrector raises precision from 0.900 to 0.926 and lowers recall from 0.853 to 0.777 (Kynkäänniemi et al., 2019), so samples move toward high-density regions of the data distribution.
SR gain at the cost of diversity. Partly, yes. The corrector raises SR from 77.02% to 83.21% and reduces per-input diversity:
| P-FID ↓ | SR (%) ↑ | Overall δt / δr / δq ↑ | |
|---|---|---|---|
| PointDex | 0.351 | 83.21 | 8.81 / 77.9 / 12.22 |
| PointDex w/o corrector | 0.144 | 77.02 | 8.83 / 78.5 / 13.84 |
We use the corrector by default because SR is the more important metric for our goal. Since it is applied only at inference, it can be turned off, and even without it PointDex outperforms DextER (65.17%) and T(R,O) Grasp (73.19%) in SR (77.02%).
Split-wise P-FID (Table 11; Table 16 in the revised manuscript). The pattern combines two effects.
- Seen and unseen-object splits. PointDex trails DextER because of the corrector. Without it, PointDex reaches 0.22 / 0.21, close to or below DextER’s 0.21 / 0.29.
- Unseen-grasp splits. Even without the corrector, PointDex is far below DextER (1.00 / 0.93 vs. 1.68 / 1.71). The corrector raises PointDex’s P-FID by about 0.3 (to 1.31 / 1.21), but this gap is larger, so PointDex stays ahead.
To see where the samples lie, we compare each model’s samples on the unseen-grasp-type split with the training grasps of the same object, labeling each sample with its most similar seen grasp type. PointDex without the corrector stays close to these training grasps (0.26; 0.24 on the seen split), whereas DextER moves away from them (1.02; 0.22 on the seen split). Since the training grasps are closer to the unseen ground truth (1.20) than DextER’s samples are (1.68), staying near them gives PointDex the lower P-FID, without generating new grasp types.
| Unseen-grasp-type split | P-FID to training grasps ↓ | P-FID to ground truth ↓ |
|---|---|---|
| Training grasps (reference) | - | 1.20 |
| PointDex (w/o corrector) | 0.26 | 1.00 |
| DextER | 1.02 | 1.68 |
Response to reviewer i4G9 [1/3]
(W1) Unclear attribution of gains
G1 in the general response reports the joint-angle variant without contact tokens that the reviewer asks for, and App. A.1.5 now describes both joint-angle variants. They replace the keypoint pose tokens with the wrist pose and the 22 joint angles. In the variant without contacts, the contact slots become learned tokens, so the token count stays the same. All four variants in this comparison disable the local geometry embedding, since it queries object features at token positions in the object frame.
The keypoint pose gives the largest SR gain: +34.1 points without contacts (45.89% → 79.94%) and +19.6 with them (63.30% → 82.89%). The contacts add a smaller but clear gain, from 79.91 ± 0.25% to 83.21 ± 0.27% over three seeds (Table 3), and 17.4 points with a joint-angle pose. Sharing one frame between the pose and the contacts pays off most in steering. Given the same contacts, the keypoint model follows them 1.9 to 2.8 times more closely than the joint-angle model (d_pair 7.38 vs. 20.57 mm), at a much higher SR (80.95% vs. 63.93%).
The Langevin corrector and the pair bias are supporting components rather than contributions. The corrector is a generic sampler applied only at inference. Without it, PointDex still reaches 77.02 ± 0.41% SR, above every re-evaluated baseline (T(R,O) Grasp 73.19 ± 0.10%, DextER 65.17%). The pair bias is needed because the object point and the hand point of each contact are separate tokens, and it binds the two points of each contact. Without it, d_pair rises from 7.43 mm to 12.53 mm. The revised manuscript states the role of both components in the ablation discussion (Sec. 4.4).
(W2-1) Role of language and practical scope
On the DexGYS test set, we replaced the instruction at test time with one from a different object, or with an empty string. Action match is the fraction of grasps whose nearest same-object ground-truth grasp belongs to the instructed action; chance level is 28–31%. Con. is the contact-map distance to the instructed action’s ground-truth grasps, as in Table 1.
Our IK always returns a solution, so we count an IK failure when the IK-recovered hand does not match the generated keypoints. For each sample, we compute the mean distance between the 29 generated keypoints and the same keypoints obtained by forward kinematics on the IK-recovered pose. A sample counts as a failure when this mean exceeds 10 mm (DexGYS test set, 9,971 grasps).
Feeding the ground-truth keypoints to the same IK gives a residual of 0.07 mm.
(W2-2) Inference Time
We timed the methods below on DexGYS with one protocol (one NVIDIA B200 GPU, 256 test inputs, one grasp per input, batch size 16; mean batch time / 16). PointDex takes 32.9 ms per grasp, comparable to the diffusion model DexGYSNet (34.1 ms) and below DextER (83.4 ms); without the corrector it takes 10.7 ms. The IK, run on the CPU, adds 29.0 ms.
Times are wall-clock milliseconds per grasp, with torch.cuda.synchronize around each call, from the object point cloud and the instruction to the 28-D Shadow Hand pose. They include text and object encoding, all sampling or decoding steps, and refinement or test-time optimization (QGC for DexGYSNet); data loading is excluded. All methods run in fp32 except DextER, whose released inference uses bf16. The IK column is timed separately and adds to the time column: the keypoint IK of PointDex (8 iterations, on the CPU) and the Pyroki retargeting of T(R,O) Grasp on the CPU; the other methods output joint angles directly. NFE counts denoiser or decoder passes; DextER decodes autoregressively, one pass per generated token.