- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ConAct, a policy for multi-fingered robot hands that augments a shared human–robot keypoint action representation with contact intensity. A video-based hand tracker provides per-vertex contact probabilities, which are pooled into contact labels for the 21 MANO keypoints of each hand. A diffusion transformer predicts future hand keypoints and contact intensities from language instructions, object point clouds, and current hand keypoints. At deployment, the predicted contact intensities displace the keypoint targets along the finger-closing directions, and contact-weighted inverse kinematics converts them into robot commands.
The policy is pretrained on approximately one million human-video episodes and fine-tuned using 200 task-specific human demonstrations together with different amounts of teleoperated data. Evaluation is conducted on one unimanual and two bimanual tasks using an RB-Y1 robot equipped with two 20-DoF Wuji hands. The paper compares ConAct with VITRA, EgoVLA, and GR00T N1.7, and additionally reports a distractor evaluation and ablations of contact pretraining, graded contact labels, and contact-driven displacement.
Critical strengths
-
The contact-augmented action representation is clear and practically relevant. ConAct extends the shared human–robot keypoint representation with a contact intensity at each MANO keypoint. This addresses an important limitation of point-based transfer: trajectories specify where the hand moves but not how firmly the robot should press. The representation also remains compatible with robot-hand inverse kinematics.
-
The real-robot evaluation directly targets the low-teleoperation setting. The method is evaluated on one unimanual and two bimanual tasks using an RB-Y1 robot with two 20-DoF hands. The same human and teleoperated demonstration pools are used across methods under 10%, 50%, and 100% teleoperation regimes. The reported gains are particularly clear when only a small amount of teleoperation data is available.
-
The ablations provide useful evidence for the proposed design. The results show performance drops when large-scale contact pretraining, graded contact intensities, or contact-driven displacement are removed. These results support the practical value of the full contact-based pipeline, although they do not completely rule out simpler alternative control strategies.
Critical weaknesses
-
The paper does not sufficiently validate contact intensity as a proxy for physical pressing force. The contact labels are obtained by pooling per-vertex contact probabilities from an existing hand tracker. The tracker itself uses tactile-glove, pressure-pad, and geometric supervision during training, but the paper does not establish a quantitative relationship between the pooled intensity and the actual force applied by the robot. Therefore, the experiments demonstrate that the signal improves task success, but provide limited evidence that the policy learns the physical strength of human contact.
-
The benefit of learned graded contact is not fully separated from simpler closure strategies. At deployment, a single displacement gain of () cm is shared across all keypoints, objects, and tasks, and no direct force or torque measurements or sensitivity analysis are reported. In addition, the no-force variant still uses contact-weighted inverse kinematics, while the binary-label variant uses the same displacement scale. A tuned fixed-offset baseline or a tuned binary-contact policy using the same point representation and IK pipeline would provide a more direct comparison.
-
The experimental coverage is limited relative to the paper’s broader claims. The contact-specific ablations are conducted only on Pick & Place with 10% teleoperation data, even though the paper emphasizes bimanual manipulation. The evaluation uses one robot, a fixed camera configuration, and three tasks, with only 16 trials for each bimanual task and no reported variability estimates. The distractor experiment mainly evaluates the named-object segmentation and point-based observation design, rather than the contact representation itself.
Questions for authors
-
How should the pooled contact probability be interpreted: as contact area, contact likelihood, or pressing strength? Is there any calibration or correlation analysis against measured robot force or torque?
-
How was () cm selected, and how sensitive are the results to this value across different objects and tasks?
-
Can the authors compare against a tuned fixed closure offset and a tuned binary-contact policy using the same point representation, IK solver, and deployment pipeline?
-
Do the contact-pretraining, binary-label, and force-injection ablations show similar trends on the two bimanual tasks?
-
Can the authors report separate full-success and partial-success rates for the bimanual tasks, together with variability across repeated evaluations?
5 comments
Response to Reviewer uUwZ [2/3]
Q2. Choice of δ: how was δ = 5 cm selected, and how sensitive are the results to it across objects and tasks?
We chose δ from a coarse sweep over {2.5, 5, 10} cm on the robot, with one value shared across all keypoints, objects and tasks; Table R3 reports this sweep together with δ = 0.
Table R3. Success rate (%) of Ours (10% teleop) across δ; 16 trials per setting (on Pick & Place, 4 objects × 4 positions with one trial each, as in Table 3), except δ = 5 cm, which is Table 1 (64 trials on Pick & Place).
δ Pick & Place Flip Plastic Bag Replace Tissue Roll 0 25.0 0.0 9.4 2.5 cm 50.0 6.3 46.9 5 cm 56.3 15.6 53.1 10 cm 56.3 15.6 40.6
What matters is a non-zero displacement, not its exact scale: removing it (δ = 0) lowers success on every task, whereas success changes much less across δ = 2.5–10 cm, a 4× range, except on Flip Plastic Bag, where all values are low (at most 15.6%).
δ thus acts as a scale hyperparameter rather than a per-object force calibration: one value works across four Pick & Place objects, a deformable bag and a thin core. How firmly each keypoint presses relative to the others comes from the predicted intensity, and replacing it with a binary or constant value lowers success (Table R4).
δ was chosen on these same trials; at δ = 2.5 or 10 cm, ConAct would still exceed every 10%-teleop baseline in Table 1 on Pick & Place and Replace Tissue Roll, and at δ = 10 cm also on Flip Plastic Bag. We do not claim robustness beyond these objects and this hand. The revised PDF states how δ was chosen in Sec. 3.3 and adds the sweep to App. J (Table 7).
W2 & Q3. Simpler closure strategies: the binary-label variant reuses δ = 5 cm and the no-force variant keeps contact-weighted IK. How does ConAct compare with a tuned binary-contact policy on the same point representation, IK solver and pipeline?
We compare (1) simpler closure strategies at a matched contact intensity and (2) the two uses of contact at deployment, all on the same point representation, IK solver and deployment pipeline (Pick & Place, 10% teleop; 16 trials per variant, while Ours is Table 1 with 64 trials).
(1) Simpler closure strategies. Table 3's binary variant (labels thresholded at 0.2) applies the full δ whenever contact is predicted, whereas ConAct's intensities rarely exceed 0.5, so it closes the hand much harder. We therefore replaced the intensity in Eq. 3 with s = 0.34, the mean human-label intensity where the binary label marks contact; ConAct predicts a similar 0.29 on average where its intensity exceeds 0.2, so s, if anything, presses slightly harder. The binary policy applies s only where it predicts contact; the fixed offset applies it to every keypoint at all times (1.7 cm). Both keep contact-weighted IK, so only the closure rule differs.
Table R4. Closure strategies (success rate, %).
Closure Intensity in Eq. 3 Success Ours predicted, graded 56.3 Binary contact (matched s) s where contact is predicted 43.8 Fixed offset (matched s) s everywhere, always 18.8
ConAct outperforms both simpler strategies (43.8 and 18.8%), suggesting that the benefit comes from graded contact rather than from closing the hand harder. The fixed offset performs worst, as it closes the fingers before they reach the object.
(2) Displacement and contact-weighted IK. Table 3's w/o displacement variant (δ = 0) keeps contact-weighted IK. To separate the two, we also replace contact-weighted IK with uniform weights, with and without the displacement.
Table R5. Uses of contact at deployment (success rate, %).
Contact-weighted IK Uniform IK Displacement (δ = 5 cm) 56.3 50.0 No displacement (δ = 0) 25.0 12.5
The displacement accounts for most of the gain (removing it costs 31.3–37.5 points), and contact-weighted IK adds a smaller benefit (6.3–12.5 points); without either, success falls to 12.5%.
We have added these variants to the revised appendix (App. J, Table 8) and corrected one detail in Sec. 4.3 and App. D: the binary-label variant binarizes the labels during fine-tuning only, not during pretraining as the submitted version said.
Response to Reviewer uUwZ [3/3]
W3, Q4 & Q5. Bimanual coverage and variability: do the contact ablations show similar trends on the bimanual tasks, and what are the full and partial success rates with variability across repeated evaluations?
We repeated the three contact ablations of Table 3 on Flip Plastic Bag and Replace Tissue Roll with the protocol of Table 1 (16 trials per task, same initial layouts), and evaluated every method of Table 1 four times on both tasks to measure run-to-run variability.
Table R6. Contact ablations (success rate, %, 10% teleop), one run of 16 trials each (64 for Ours on Pick & Place); Ours is Table 1's run, the first of the four repeats averaged in Tables R7 and R8. Bimanual: partial success counts 0.5, as in the paper.
Pick & Place Flip Plastic Bag Replace Tissue Roll Ours 56.3 15.6 53.1 w/o contact labels 43.8 12.5 43.8 Binary labels 37.5 12.5 34.4 w/o displacement (δ = 0) 25.0 0.0 9.4
The bimanual tasks show a similar ordering to Pick & Place (w/o contact labels and binary labels tie on Flip Plastic Bag), with the largest drop from removing the displacement.
For Q5, Tables R7 and R8 report full success (both subtasks completed) and success with partial credit (Table 1's metric; completing only the first subtask counts 0.5) for every bimanual setting of Table 1.
Table R7. Full success (%), mean ± standard deviation over four repeats of 16 trials. All repeats use the same 16 initial layouts, so the standard deviation measures the variability of real-robot execution, not variation across different sets of initial layouts.
Flip 10% Flip 50% Flip 100% Tissue 10% Tissue 50% Tissue 100% VITRA 0.0 ± 0.0 17.2 ± 3.1 48.4 ± 3.1 0.0 ± 0.0 32.8 ± 6.0 50.0 ± 5.1 EgoVLA 9.4 ± 3.6 7.8 ± 3.1 35.9 ± 3.1 0.0 ± 0.0 0.0 ± 0.0 34.4 ± 6.3 GR00T N1.7 0.0 ± 0.0 12.5 ± 0.0 29.7 ± 3.1 0.0 ± 0.0 6.3 ± 5.1 20.3 ± 3.1 Ours 10.9 ± 3.1 26.6 ± 3.1 56.3 ± 5.1 17.2 ± 3.1 45.3 ± 6.0 59.4 ± 6.3
Table R8. Success with partial credit (%), mean ± standard deviation over four repeats of 16 trials; the first repeat is the run reported in Table 1.
Flip 10% Flip 50% Flip 100% Tissue 10% Tissue 50% Tissue 100% VITRA 7.0 ± 1.6 20.3 ± 3.1 48.4 ± 3.1 6.3 ± 2.6 51.6 ± 6.0 57.0 ± 6.9 EgoVLA 10.9 ± 3.1 17.2 ± 1.8 42.2 ± 3.1 12.5 ± 2.6 15.6 ± 2.6 48.4 ± 3.1 GR00T N1.7 8.6 ± 1.6 17.2 ± 1.8 30.5 ± 3.9 12.5 ± 0.0 25.0 ± 5.1 35.2 ± 1.6 Ours 14.8 ± 1.6 30.5 ± 4.7 59.4 ± 5.1 50.8 ± 3.0 61.7 ± 1.6 74.2 ± 1.6
ConAct achieves the highest mean success on both metrics in every bimanual setting.
We have added the bimanual ablations and the full and partial success over four repeats to the revised appendix (App. J, Tables 9 and 10). Sec. 5 already states the evaluation scope (one robot, a fixed egocentric camera, three tasks); in the camera-ready version, we will also clarify that the distractor experiment tests the observation design, not the contact representation.
We thank the reviewer again for the concrete suggestions, which led to the gain sweep, the binary-contact baseline and the bimanual ablations.
Response to Reviewer uUwZ [1/3]
We thank the reviewer for the detailed review and for the concrete experiments suggested. We address the weaknesses and questions below.
By construction, the intensity combines contact likelihood and contact area: it is a fixed weighted average (MANO skinning weights) of the tracker's per-vertex contact probabilities around each keypoint. Empirically, it also correlates with measured hand force, consistent with how Sec. 3.1 defines it: a graded measure of how firmly each keypoint presses, not a calibrated force (Sec. 5). The paper related it to pressing strength only through task success; we now test it directly. Our main evidence is (1) the intensity labeled on human hands against measured hand force, since ConAct learns how firmly to press from human videos; as a supplementary consistency check, we also compare (2) the intensity predicted on the robot with the robot's joint effort.
(1) Human labels. Intensity rises with pressing force, even among contact frames alone, so it carries more than whether contact occurs. We ran the tracker that labels our demonstrations (Anonymous, 2026) on the held-out test splits of OpenTouch (Song et al., 2025; egocentric video with a tactile glove) and PVDB (Grady et al., 2022; pressure pad), and correlated the pooled intensity with the measured hand force.
Table R1. Spearman ρ between pooled contact intensity and measured human hand force (per frame: mean intensity over the 21 keypoints vs. total hand force; OpenTouch 23k frames from 190 sequences, PVDB 90k frames from 1,668 sequences), with 95% intervals from a bootstrap over sequences.
Contact area alone correlates with force about as strongly (ρ = 0.75–0.78), so the intensity reflects contact extent that co-varies with pressing strength.
(2) Supplementary check on the robot. Predicted intensity also rises with the robot hand's joint effort on all three tasks. The hand has no force or tactile sensing, so we use its only contact-related signal, the joint effort reported by the Wuji Hand 2 driver, which rises when a joint is obstructed [1]. We correlate the intensity that Ours (10% teleop) predicts on held-out teleoperation episodes with the recorded effort.
Table R2. Spearman ρ between predicted intensity and Wuji Hand 2 joint effort on held-out teleoperation episodes (per frame: mean intensity over the 21 keypoints vs. summed effort over the 20 finger joints; 90 / 45 / 45 episodes), with 95% intervals from a bootstrap over episodes.
We have added this analysis to the revised appendix (App. J, Table 6) and aligned the remaining "force" wording with this definition: the abstract, Sec. 3.3 and the conclusion now describe ConAct as displacing the robot hand's keypoint targets rather than producing force, and App. J notes that the intensity is closely tied to contact extent.
[1] Wuji Technology. Wuji Hand ROS 2 driver documentation. https://github.com/wuji-technology/wujihandros2.