acceptodds
Withdrawn from ICLR 2027

PiHand: Latent Hand–Object Interaction Priors for Physics-Based Policy Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Synthesizing physically grounded dexterous hand–object interactions is important for VR/AR, gaming, and robotics, yet remains challenging because subtle contact dynamics, diverse object geometries, and target trajectories must be jointly respected while maintaining natural grasps. Existing methods either track mocap trajectories with limited transfer to novel objects, or rely on hand-only generative priors that fail to capture coupled hand–object motion, sacrificing contact fidelity and sample efficiency. To tackle these challenges, we introduce PiHand, a Physically grounded Interaction-aware Hand motion synthesis framework. PiHand first trains expert policies that track mocap demonstrations in simulation, distills them into a unified hand–object interaction prior using a goal-conditioned VAE, and finally solve diverse dexterous manipulation tasks through bounded residual exploration in its latent space. With only a single reward and no complex curriculum training, this yields general tabletop manipulation policies that generalize to unseen objects and trajectories on GRAB and DexYCB. The same prior transfers to object catching and bimanual handover, two tasks that are rarely seen in its training data. Across all tasks, PiHand consistently outperforms baselines in all metrics. We will release the processed data and model checkpoints.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
physics-based control, dexterous manipulation, imitation learning, interaction priors, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vx4kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PiHand proposes a three-stage framework for physics-based dexterous hand–object interaction generation conditioned on target object trajectories. It trains physics-based motion-tracking experts from mocap demonstrations, distills them into a goal-conditioned VAE interaction prior, and learns downstream manipulation policies through bounded latent residual exploration. Experiments on GRAB and DexYCB show improvements over action-space exploration and a hand-only prior, while additional catching and handover tasks demonstrate reuse of the learned prior.

Critical strengths

The proposed interaction prior achieves consistent improvements over the implemented baselines. On GRAB, it improves tabletop manipulation success from 60.0% with the hand-only prior to 74.0%, with low variance across three random seeds. The method also demonstrates reuse of the learned prior for DexYCB manipulation, catching, and bimanual handover.

Critical weaknesses

The paper’s methodological novelty is limited relative to prior art in latent-control and interaction-aware policy learning. The expert-distillation, conditional VAE, and latent-space RL framework closely mirrors PULSE, while established works such as OmniGrasp, MaskedManipulator, and InterPrior have already explored generative control leveraging motion or interaction priors. Consequently, the manuscript fails to clearly delineate what fundamentally new capability or technical insight its formulation introduces.

Furthermore, the experimental evaluation lacks rigor: the proposed method is primarily benchmarked against action-space exploration and a hand-only prior. Because the hand-only baseline strips object information from both expert training and the distilled prior/decoder, this comparison conflates the intrinsic benefits of the proposed representation with those of richer interaction supervision. To substantiate the claimed methodological advance, the authors must evaluate against stronger interaction-aware baselines and provide fine-grained, representation-level ablations.

Questions for authors
  1. What are the key methodological differences between PiHand and PULSE/OmniGrasp and InterPrior? Beyond adapting latent-space control to dexterous hand–object interactions, what is the main technical contribution?

  2. Can the authors compare PiHand with a stronger interaction-aware generative controller under the same training data, observations, and simulation settings?

  3. How much of the performance improvement comes from interaction-aware supervision itself, rather than the specific design of the proposed interaction prior? Can the authors provide controlled ablations of object geometry and contact information?

  4. How does bounded latent residual exploration compare with directly fine-tuning the distilled policy using the same pretrained model and downstream training budget?

  5. The paper claims approximately tenfold higher training efficiency based on downstream training epochs. Does this advantage remain when the costs of expert training and prior distillation are included?

2 comments

We thank the reviewer for the detailed review and for acknowledging the consistent improvements and the reuse of the prior across tasks.

Q1: Differences to PULSE/Omnigrasp and InterPrior, and main contribution PiHand makes three contributions that build on each other. First, physically accurate interaction experts for the full GRAB dataset, with one single-handed and one bimanual expert per object (84.8% tracking success, Table 1). These highly accurate experts will be released for future research. Second, an interaction prior distilled from these experts that needs only the object trajectory. Third, task learning through a bounded residual on the frozen prior. PULSE and Omnigrasp do not use interaction experts: they learn a prior over human motion only and leave the interaction with the object to exploration. Our Hand-Only Prior baseline is our reimplementation of Omnigrasp for dexterous hands (App. D.2), and the supervision from our experts alone accounts for about 8 of the 14 points between it and our method (Q3). MaskedManipulator and InterPrior distill interaction experts into goal-conditioned whole-body controllers. MaskedManipulator has no RL stage for new tasks, and InterPrior fine-tunes the whole controller with RL and is suboptimal on the dexterous manipulation task (details in our response to Reviewer ypyx, Q1). PiHand keeps prior and decoder frozen, so one prior serves tabletop manipulation, catching and handover through a small task-specific residual. Q2 to Q4 test these contributions.

Q2: Stronger interaction-aware baseline We reimplemented InterPrior and MaskedManipulator in our setting and distilled both from our own interaction experts, with the same data, observations, simulation and object-trajectory conditioning. Task RL adds at most about 14 points to their distilled starting level, while our residual lifts our frozen prior from 30.8% to 74.0% (tabletop manipulation, held-out SR in %):

Controller (same experts, data, observations, simulation)Without task RLWith task RL
InterPrior, adapted31.545.1
MaskedManipulator + DSRL35.837.4
Ours30.874.0

Neither method has public training code (InterPrior releases inference code and a pretrained model), so both follow the papers: InterPrior as a Transformer CVAE prior over state history and goal tokens, post-trained with RL and concurrent distillation, and MaskedManipulator as a Transformer diffusion policy trained with DAgger. MaskedManipulator has no task-RL stage, so we add one in the style of DSRL (Wagenmaker et al., 2025), where an RL policy steers its initial noise. We reimplement InterPrior with exploration noise adapted to our action space and an adaptive learning rate to ensure effectiveness of RL fine-tuning. DSRL improves MaskedManipulator only from 35.8% to 37.4%, and its tracking on the training sequences stayed flat throughout RL.

Q3: Performance improvement decomposition We ablate the sources of our performance gain (tabletop manipulation, held-out SR in %):

ExpertsPrior conditioned onVariantSR (%)
Hand motion onlyHand stateHand-Only Prior60.0
Hand-object interactionHand state68.2
Hand-object interactionObject trajectoryOurs74.0
Hand-object interactionObject trajectoryPrior without contact inputs75.6
Hand-object interactionObject trajectoryPrior without geometry input70.0
  1. Interaction supervision accounts for 8 of the 14 points, object-trajectory conditioning for the other 6. A prior distilled from our interaction experts but conditioned only on the hand state, as in the hand-only paradigm, reaches 68.2%, compared with 60.0% for the Hand-Only Prior and 74.0% for ours. It does so without the curriculum and pre-grasp reward that the Hand-Only Prior uses.
  2. The contact input is redundant. It is a binary contact map with one flag per hand link, which the decoder receives for the current state and the encoder for the future reference frames. The joint-to-surface vectors already carry this information, since a joint in contact has a vector of near-zero length. Removing the contact map from the prior therefore does not lower success (75.6%, within evaluation noise).
  3. The geometry input matters. It consists of joint-to-surface vectors from each of the joints per hand to the nearest point on the object surface, which the decoder receives for the current state. It is the prior's only geometry input, and removing it costs 4 points (70.0%).

In both input ablations we remove the input from the prior only, retrain the prior, and train the task policy on top of it with its full observation unchanged. The new rows are single runs.

Q4: Bounded latent residual vs. fine-tuning the distilled policy The bounded residual is about 22 points better. Fine-tuning the distilled prior and decoder with PPO, from the same pretrained prior and training regimen, reaches 51.7% against 74.0% for our residual. The numbers are in the table “Adaptation of our distilled prior” of our response to Reviewer ypyx (part 1). Another advantage of our latent residual approach is that the frozen prior is shared across tasks, so each new task adds only a small residual policy instead of a fine-tuned copy of the full controller.

Q5: Efficiency including pretraining Including pretraining, PiHand needs more compute than the baselines for our three tasks mainly due to the difficulty of tracking different mocap manipulations. Pretraining is a one-time cost, however, and each task on top of it costs about the same as for the baselines (GPU-hours, with the held-out SR in % from the paper in parentheses):

MethodPretraining (experts + distillation)TabletopCatchingHandover
Action Space Explorationnone35.8 (59.1)20.0 (31.0)19.9 (11.9)
Hand-Only Prior59.541.8 (60.0)19.9 (1.0)19.9 (5.0)
Ours284.338.1 (74.0)20.0 (87.0)19.9 (51.9)
Reviewer_ypyxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses synthesis of physically plausible dexterous hand–object interaction conditioned only on a target object trajectory. PiHand is a three-stage framework: (i) per-object PPO tracking experts that convert GRAB mocap into physically valid interactions, using reference-anchored integrated residual actions and a contact-aware reward; (ii) distillation of the expert pool into a goal-conditioned VAE — a learnable prior over a 64-dim latent conditioned on the future object trajectory only, a privileged encoder, and a deterministic decoder producing PD-target updates, trained with DAgger-style BC plus weighted KL; and (iii) PPO exploration that outputs a bounded residual on the frozen prior mean, requiring only a single object-tracking reward and no curriculum.

Critical strengths

Clear motivation and well-defined positioning. The paper clearly identifies two gaps: tracking-only methods fail to generalize to novel objects, while hand-only priors fail to model hand–object contact coupling. The core insight of embedding interactions directly into the latent space is compelling.

An interaction-aware prior beats hand-only priors and action-space RL is isolated causally. Both baselines are re-implemented under identical simulator, embodiment, and physics, and are even given extra supervision (pre-grasp reward, 3-level curriculum) that PiHand does not use; performance differences can therefore be attributed to the prior itself. The evidence is also consistent across axes that matter: unseen objects and trajectories (GRAB), cross-dataset transfer, and cross-task transfer, plus robustness to 4× mass (−11.7% vs −52.8%/−67.2% for baselines) and action noise. This is consequential because it converts a widely believed hypothesis (PULSE-style latent priors should include the object) into well-controlled evidence, and the latent-vs-action-residual and residual-bound ablations give actionable design guidance.

Critical weaknesses

The conceptual delta over the PULSE/Omnigrasp recipe and concurrent InterPrior is thin and not delineated. The overall recipe, distill tracking experts into a conditional-VAE latent space, then residual RL on the frozen decoder, is acknowledged to follow PULSE, and the cited concurrent InterPrior distills full-reference HOI experts into a goal-conditioned variational controller with RL fine-tuning, which is very close in spirit. The paper's contribution is therefore largely "the PULSE recipe instantiated for dexterous HOI, with careful engineering," and neither a sharp conceptual distinction nor any empirical differentiation from InterPrior is provided. This is consequential because it bounds the novelty assessment even though execution quality is high.

The generality claims exceed the evidence in several load-bearing places. (i) Everything is simulation-only, with rigid objects, a single MANO-based floating bimanual hand (no arm/body), and no sim-to-real signal; (ii) the "single reward, no curriculum" framing applies only to tabletop — catching and handover add fingertip-proximity and wrist-economy terms whose necessity is not ablated; (iii) handover evaluation is restricted to 160/1442 pre-filtered physically stable start frames, so the 51.9% SR is conditioned on a favorable selection and the absolute number remains modest; (iv) the hand-only-prior baseline's near-total failure on catching (1.0% SR) versus its competent tabletop performance (60.0%) is unexplained, raising the question of whether the largest transfer gains partly reflect baseline fragility rather than prior strength. Each is individually fixable, but together they determine how far the "general, reusable interaction prior" claim can be taken.

Questions for authors
  • Can you precisely state the technical delta over InterPrior (conditioning, distillation objective, exploration scheme, embodiment) and, if feasible, provide a comparison or a reason it is not comparable? Absent that, which design choices in PiHand do you believe are responsible for the gains over hand-only priors?

  • Why does the hand-only prior collapse to 1.0% SR on catching while remaining competitive on tabletop? Is this a tuning artifact (e.g., its decoder never seeing object states), and does it persist with a stronger hand-only baseline?

  • What is handover SR over all 1,442 held-out start frames (without the stability pre-filter), and how much of the failure mass is grasp establishment versus the transfer itself?

  • How sensitive are the catching/handover results to the task-specific reward terms (fingertip proximity, wrist economy)? Does the interaction prior still outperform baselines under the bare object-tracking reward on these tasks?

2 comments

We thank the reviewer for the thorough review and for recognizing the controlled comparison.

Q1: Delta over InterPrior InterPrior’s authors note that it is “not designed for fine-grained finger dexterity or in-hand manipulation”. It targets whole-body loco-manipulation and is trained on the OMOMO subset of InterAct, mostly large objects without ground-truth hand motion, so their finger behaviour comes from a reference-free reward that closes the hand around the object. PiHand targets exactly this gap: its prior is distilled from experts that track real finger motion and contacts, and it is conditioned only on the object trajectory. The second difference is how the distilled prior is adapted to different tasks: InterPrior fine-tunes the whole controller with RL, whereas PiHand keeps prior and decoder frozen and learns new tasks through a bounded latent residual. The distillation stage remains similar as both distill experts with DAgger into a conditional VAE.

The table below shows that the adaptation scheme alone has a huge impact on the performance. Starting from the same prior our second stage distilled, with the same reward, observations and training budget we compare different adaptation schemes used in RL-post training on the tabletop manipulation task. From the table below, we can see that InterPrior style post-training is much worse than our proposed post training scheme.

Adaptation of our distilled priorSR (%)
Frozen prior, no exploration policy30.8
Frozen prior + residual in action space49.6
Fine-tune prior and decoder (InterPrior-style post-training)51.7
Fine-tune prior and decoder + KL to the original prior32.3
Frozen prior + bounded latent residual (ours)74.0

We also reimplemented the VAE distillation and RL fine-tuning of InterPrior in our dexterous hand setting, with the same experts, data, observations, simulation and object-trajectory conditioning. While the VAE prior distilled using InterPrior is similar to our prior, its stage 3 post training only improves the SR by about 14 points (from 31.5% to 45.1%). In contrast, our stage-3 post training improves SR from 30.8% to 74%, demonstrating the advantage of our method for complex dexterous manipulation. We refer to table under Q2 of our response to Reviewer vx4k for the detailed comparison.

Which design choices drive the gains over hand-only priors? The interaction supervision accounts for about 8 of the 14 points, and conditioning on the object trajectory for the remaining 6. A prior distilled from our interaction experts but conditioned only on the hand state, as in the hand-only paradigm, reaches 68.2%, between the Hand-Only Prior (60.0%) and ours (74.0%) (see our response to vx4k, Q3). Because the prior is distilled from experts that physically interact with the objects, its latent space already contains grasps and manipulation, and exploration only has to select and refine them.

Q2: Why does the Hand-Only Prior collapse on catching? The Hand-Only Prior’s success rate increases from 1.0% to 56.0% when wrist-economy penalties are removed (App. E.4), showing that the observed collapse strongly depends on these penalties. In the submitted table, the Action Space Exploration row was trained without these penalties. We have now performed controlled post-training ablations, using the same reward configuration for all methods within each column of the table below. PiHand remains clearly ahead in all three settings.

MethodWith wrist economyWithout wrist economyObject-tracking reward only
Action Space Exploration31.043.048.0
Hand-Only Prior1.056.047.0
Ours87.093.084.0

Q3: Handover SR decomposition on all 1,442 start frames We filter start frames because a mocap frame copied into simulation is not necessarily force-closed, so the object can slip out of the hand before any policy acts. The filter involves no policy and selects the same states for all methods (App. E.5). We evaluated every one of the 1,442 held-out start frames once per method. Our method stays ahead on stable and unstable frames alike (SR in %):

MethodStable frames (160)Unstable frames (1,282)All frames (1,442)
Action Space Exploration11.98.89.2
Hand-Only Prior5.01.72.1
Ours51.918.822.5

For every method, most failures occur before the transfer begins, because the grasp copied from the mocap frame cannot be established in simulation. The table below follows the episodes through the handover: each column gives the share of episodes that pass a stage, out of those that reached it (all 1,442 frames, %, episode counts in parentheses):

MethodGrasp establishedCarried until the right hand touchesExchange completedHeld after the exchangeSuccess
Action Space Exploration25.7 (371 / 1,442)55.3 (205 / 371)98.5 (202 / 205)65.3 (132 / 202)9.2
Hand-Only Prior11.7 (168 / 1,442)81.0 (136 / 168)64.7 (88 / 136)34.1 (30 / 88)2.1
Ours37.4 (540 / 1,442)97.8 (528 / 540)93.0 (491 / 528)66.0 (324 / 491)22.5

Our advantage lies in establishing the grasp and in carrying the object. In the exchange and afterwards, our policy is on par with or slightly behind Action Space Exploration, because we reach these stages far more often.

Q4: Bare object-tracking reward for catching and handover On catching, our method keeps its lead under the object-tracking reward alone (ours 84.0% vs. hand-only 47.0% and action space exploration 48.0%). On handover, no method solves the task without the fingertip-proximity term (about 5% or below for all methods), because nothing draws the receiving hand to the object, and because the priors were not trained with the wrist zones (App. E.5). The single-reward framing applies to tabletop manipulation only, and we will state this clearly in the abstract and introduction.

Reviewer_NbdqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this work, the authors propose a date-driven interaction prior for dexterous manipulation. Starting with mocap data and then fine-tuning with RL, they can learn good policies. The method was evaluated on standard benchmarks, as well as on new, more complicated tasks.

Critical strengths

Dexterous manipulation is a very relevant problem, and the proposed method transfers to more complicated tasks such as catching and handover.

Critical weaknesses

Insufficient evaluations and baseline comparisons. See Questions.

The authors considered only one hand morphology. The additional ones would be relevant for showing how generalizable the approach is.

Questions for authors
  1. In line 480, it is stated that it ‘maintain 84.8% SR on the full GRAB’ in comparison to ManipTrans? Does it mean that ManipTrans is evaluated on the full GRAB dataset? If yes, why is that omitted from Table 1?
  2. Have you compared the proposed approach to relevant baselines, such as the Omnigrasp (Luo et al., 2024a) and PULSE (Luo et al., 2024b)?
1 comment

We thank the reviewer for the review.

Q1: Is ManipTrans evaluated on the full GRAB dataset? ManipTrans is evaluated only on GRAB50, as in Table 1, since it trains one policy per sequence which is time-consuming and does not support bimanual manipulation of a single object. The 84.8% in line 480 is our own result on the full dataset, showing that our per-object experts scale beyond GRAB50. We will rephrase that sentence for clarification.

Q2: Comparison with Omnigrasp and PULSE PULSE and OmniGrasp follow similar prior distillation and RL post training design yet PULSE focuses on locomotion while OmniGrasp can do loco-manipulation for humanoid. Our Hand-Only Prior baseline is our reimplementation of Omnigrasp, with its PULSE-style prior, adapted to dexterous hands, since the original methods are built for a full-body humanoid. It distills a prior from hand-only tracking experts, conditioned only on proprioception, and trains an object-aware RL policy as a residual on the prior mean, with Omnigrasp’s object observations, pre-grasp reward and trajectory generator (Sec. 4.2, App. D.2). Our method outperforms it in all three tasks.

Insufficient evaluations and baselines We added InterPrior- and MaskedManipulator-style controllers trained on our data, direct fine-tuning of the distilled policy, and ablations of the prior’s supervision and inputs. Results show that our method outperforms both baselines and the interaction supervision, the object-trajectory conditioning and the geometry input contribute to the performance gains, for details please see our responses to Reviewers ypyx and vx4k.

One hand morphology We follow prior works on physics-based motion synthesis which usually consider a single hand morphology for evaluation. Extending PiHand to other models is possible but we consider it as future work (App. A).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.