- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PiHand proposes a three-stage framework for physics-based dexterous hand–object interaction generation conditioned on target object trajectories. It trains physics-based motion-tracking experts from mocap demonstrations, distills them into a goal-conditioned VAE interaction prior, and learns downstream manipulation policies through bounded latent residual exploration. Experiments on GRAB and DexYCB show improvements over action-space exploration and a hand-only prior, while additional catching and handover tasks demonstrate reuse of the learned prior.
Critical strengths
The proposed interaction prior achieves consistent improvements over the implemented baselines. On GRAB, it improves tabletop manipulation success from 60.0% with the hand-only prior to 74.0%, with low variance across three random seeds. The method also demonstrates reuse of the learned prior for DexYCB manipulation, catching, and bimanual handover.
Critical weaknesses
The paper’s methodological novelty is limited relative to prior art in latent-control and interaction-aware policy learning. The expert-distillation, conditional VAE, and latent-space RL framework closely mirrors PULSE, while established works such as OmniGrasp, MaskedManipulator, and InterPrior have already explored generative control leveraging motion or interaction priors. Consequently, the manuscript fails to clearly delineate what fundamentally new capability or technical insight its formulation introduces.
Furthermore, the experimental evaluation lacks rigor: the proposed method is primarily benchmarked against action-space exploration and a hand-only prior. Because the hand-only baseline strips object information from both expert training and the distilled prior/decoder, this comparison conflates the intrinsic benefits of the proposed representation with those of richer interaction supervision. To substantiate the claimed methodological advance, the authors must evaluate against stronger interaction-aware baselines and provide fine-grained, representation-level ablations.
Questions for authors
-
What are the key methodological differences between PiHand and PULSE/OmniGrasp and InterPrior? Beyond adapting latent-space control to dexterous hand–object interactions, what is the main technical contribution?
-
Can the authors compare PiHand with a stronger interaction-aware generative controller under the same training data, observations, and simulation settings?
-
How much of the performance improvement comes from interaction-aware supervision itself, rather than the specific design of the proposed interaction prior? Can the authors provide controlled ablations of object geometry and contact information?
-
How does bounded latent residual exploration compare with directly fine-tuning the distilled policy using the same pretrained model and downstream training budget?
-
The paper claims approximately tenfold higher training efficiency based on downstream training epochs. Does this advantage remain when the costs of expert training and prior distillation are included?
2 comments
Q3: Performance improvement decomposition We ablate the sources of our performance gain (tabletop manipulation, held-out SR in %):
| Experts | Prior conditioned on | Variant | SR (%) |
|---|---|---|---|
| Hand motion only | Hand state | Hand-Only Prior | 60.0 |
| Hand-object interaction | Hand state | 68.2 | |
| Hand-object interaction | Object trajectory | Ours | 74.0 |
| Hand-object interaction | Object trajectory | Prior without contact inputs | 75.6 |
| Hand-object interaction | Object trajectory | Prior without geometry input | 70.0 |
- Interaction supervision accounts for 8 of the 14 points, object-trajectory conditioning for the other 6. A prior distilled from our interaction experts but conditioned only on the hand state, as in the hand-only paradigm, reaches 68.2%, compared with 60.0% for the Hand-Only Prior and 74.0% for ours. It does so without the curriculum and pre-grasp reward that the Hand-Only Prior uses.
- The contact input is redundant. It is a binary contact map with one flag per hand link, which the decoder receives for the current state and the encoder for the future reference frames. The joint-to-surface vectors already carry this information, since a joint in contact has a vector of near-zero length. Removing the contact map from the prior therefore does not lower success (75.6%, within evaluation noise).
- The geometry input matters. It consists of joint-to-surface vectors from each of the joints per hand to the nearest point on the object surface, which the decoder receives for the current state. It is the prior's only geometry input, and removing it costs 4 points (70.0%).
In both input ablations we remove the input from the prior only, retrain the prior, and train the task policy on top of it with its full observation unchanged. The new rows are single runs.
Q4: Bounded latent residual vs. fine-tuning the distilled policy The bounded residual is about 22 points better. Fine-tuning the distilled prior and decoder with PPO, from the same pretrained prior and training regimen, reaches 51.7% against 74.0% for our residual. The numbers are in the table “Adaptation of our distilled prior” of our response to Reviewer ypyx (part 1). Another advantage of our latent residual approach is that the frozen prior is shared across tasks, so each new task adds only a small residual policy instead of a fine-tuned copy of the full controller.
Q5: Efficiency including pretraining Including pretraining, PiHand needs more compute than the baselines for our three tasks mainly due to the difficulty of tracking different mocap manipulations. Pretraining is a one-time cost, however, and each task on top of it costs about the same as for the baselines (GPU-hours, with the held-out SR in % from the paper in parentheses):
| Method | Pretraining (experts + distillation) | Tabletop | Catching | Handover |
|---|---|---|---|---|
| Action Space Exploration | none | 35.8 (59.1) | 20.0 (31.0) | 19.9 (11.9) |
| Hand-Only Prior | 59.5 | 41.8 (60.0) | 19.9 (1.0) | 19.9 (5.0) |
| Ours | 284.3 | 38.1 (74.0) | 20.0 (87.0) | 19.9 (51.9) |
We thank the reviewer for the detailed review and for acknowledging the consistent improvements and the reuse of the prior across tasks.
Q1: Differences to PULSE/Omnigrasp and InterPrior, and main contribution PiHand makes three contributions that build on each other. First, physically accurate interaction experts for the full GRAB dataset, with one single-handed and one bimanual expert per object (84.8% tracking success, Table 1). These highly accurate experts will be released for future research. Second, an interaction prior distilled from these experts that needs only the object trajectory. Third, task learning through a bounded residual on the frozen prior. PULSE and Omnigrasp do not use interaction experts: they learn a prior over human motion only and leave the interaction with the object to exploration. Our Hand-Only Prior baseline is our reimplementation of Omnigrasp for dexterous hands (App. D.2), and the supervision from our experts alone accounts for about 8 of the 14 points between it and our method (Q3). MaskedManipulator and InterPrior distill interaction experts into goal-conditioned whole-body controllers. MaskedManipulator has no RL stage for new tasks, and InterPrior fine-tunes the whole controller with RL and is suboptimal on the dexterous manipulation task (details in our response to Reviewer ypyx, Q1). PiHand keeps prior and decoder frozen, so one prior serves tabletop manipulation, catching and handover through a small task-specific residual. Q2 to Q4 test these contributions.
Q2: Stronger interaction-aware baseline We reimplemented InterPrior and MaskedManipulator in our setting and distilled both from our own interaction experts, with the same data, observations, simulation and object-trajectory conditioning. Task RL adds at most about 14 points to their distilled starting level, while our residual lifts our frozen prior from 30.8% to 74.0% (tabletop manipulation, held-out SR in %):
Neither method has public training code (InterPrior releases inference code and a pretrained model), so both follow the papers: InterPrior as a Transformer CVAE prior over state history and goal tokens, post-trained with RL and concurrent distillation, and MaskedManipulator as a Transformer diffusion policy trained with DAgger. MaskedManipulator has no task-RL stage, so we add one in the style of DSRL (Wagenmaker et al., 2025), where an RL policy steers its initial noise. We reimplement InterPrior with exploration noise adapted to our action space and an adaptive learning rate to ensure effectiveness of RL fine-tuning. DSRL improves MaskedManipulator only from 35.8% to 37.4%, and its tracking on the training sequences stayed flat throughout RL.