acceptodds
Under review as a conference paper at ICLR 2027

ConAct: Contact as Action for Multi-Fingered Robot Hand Policies from Human Videos

Abstract

Policies for multi-fingered robot hands are typically trained on teleoperated demonstrations, which are slow and costly to collect, especially for bimanual tasks. Human videos offer a scalable alternative, yet they show where the hands move but not how firmly they press. We argue that this missing contact is a key reason why policies pretrained on human videos still rely on many teleoperated demonstrations. To address this limitation, we introduce ConAct (Contact as Action), a bimanual policy for multi-fingered hands that learns from human videos with few teleoperated demonstrations. Specifically, our key idea is to extend the point-based action representation shared by human and robot hands with contact, i.e., a contact intensity at each 3D keypoint, which we label automatically on about 1M egocentric video episodes without any tactile sensor or manual annotation. This design not only lets the policy learn from human videos how firmly to press, but also lets us turn the predicted contact into force along the normals of the robot hand at deployment, without any force sensing. On a real bimanual robot with two 20-DoF hands, ConAct outperforms three Vision-Language-Action models pretrained on human videos on all three tasks at every amount of teleoperated demonstrations. For example, with only 20 teleoperated demonstrations in total, it more than doubles the average success rate of the strongest baseline (17.2% 41.7%) and already matches a baseline trained with ten times more. Our ablations further show that both the contact intensity and the force it drives contribute substantially to this gain. Moreover, on a bimanual task, ConAct more than quadruples the success rate of the best baseline with only five teleoperated demonstrations (12.5% 53.1%). Videos are available at https://anonymous39306.github.io/conact.

Primary area:
applications to robotics, autonomy, planning
Keywords:
human-to-robot, 3D, robot learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uUwZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ConAct, a policy for multi-fingered robot hands that augments a shared human–robot keypoint action representation with contact intensity. A video-based hand tracker provides per-vertex contact probabilities, which are pooled into contact labels for the 21 MANO keypoints of each hand. A diffusion transformer predicts future hand keypoints and contact intensities from language instructions, object point clouds, and current hand keypoints. At deployment, the predicted contact intensities displace the keypoint targets along the finger-closing directions, and contact-weighted inverse kinematics converts them into robot commands.

The policy is pretrained on approximately one million human-video episodes and fine-tuned using 200 task-specific human demonstrations together with different amounts of teleoperated data. Evaluation is conducted on one unimanual and two bimanual tasks using an RB-Y1 robot equipped with two 20-DoF Wuji hands. The paper compares ConAct with VITRA, EgoVLA, and GR00T N1.7, and additionally reports a distractor evaluation and ablations of contact pretraining, graded contact labels, and contact-driven displacement.

Critical strengths
  1. The contact-augmented action representation is clear and practically relevant. ConAct extends the shared human–robot keypoint representation with a contact intensity at each MANO keypoint. This addresses an important limitation of point-based transfer: trajectories specify where the hand moves but not how firmly the robot should press. The representation also remains compatible with robot-hand inverse kinematics.

  2. The real-robot evaluation directly targets the low-teleoperation setting. The method is evaluated on one unimanual and two bimanual tasks using an RB-Y1 robot with two 20-DoF hands. The same human and teleoperated demonstration pools are used across methods under 10%, 50%, and 100% teleoperation regimes. The reported gains are particularly clear when only a small amount of teleoperation data is available.

  3. The ablations provide useful evidence for the proposed design. The results show performance drops when large-scale contact pretraining, graded contact intensities, or contact-driven displacement are removed. These results support the practical value of the full contact-based pipeline, although they do not completely rule out simpler alternative control strategies.

Critical weaknesses
  1. The paper does not sufficiently validate contact intensity as a proxy for physical pressing force. The contact labels are obtained by pooling per-vertex contact probabilities from an existing hand tracker. The tracker itself uses tactile-glove, pressure-pad, and geometric supervision during training, but the paper does not establish a quantitative relationship between the pooled intensity and the actual force applied by the robot. Therefore, the experiments demonstrate that the signal improves task success, but provide limited evidence that the policy learns the physical strength of human contact.

  2. The benefit of learned graded contact is not fully separated from simpler closure strategies. At deployment, a single displacement gain of (δ=5\delta=5) cm is shared across all keypoints, objects, and tasks, and no direct force or torque measurements or sensitivity analysis are reported. In addition, the no-force variant still uses contact-weighted inverse kinematics, while the binary-label variant uses the same displacement scale. A tuned fixed-offset baseline or a tuned binary-contact policy using the same point representation and IK pipeline would provide a more direct comparison.

  3. The experimental coverage is limited relative to the paper’s broader claims. The contact-specific ablations are conducted only on Pick & Place with 10% teleoperation data, even though the paper emphasizes bimanual manipulation. The evaluation uses one robot, a fixed camera configuration, and three tasks, with only 16 trials for each bimanual task and no reported variability estimates. The distractor experiment mainly evaluates the named-object segmentation and point-based observation design, rather than the contact representation itself.

Questions for authors
  1. How should the pooled contact probability be interpreted: as contact area, contact likelihood, or pressing strength? Is there any calibration or correlation analysis against measured robot force or torque?

  2. How was (δ=5\delta=5) cm selected, and how sensitive are the results to this value across different objects and tasks?

  3. Can the authors compare against a tuned fixed closure offset and a tuned binary-contact policy using the same point representation, IK solver, and deployment pipeline?

  4. Do the contact-pretraining, binary-label, and force-injection ablations show similar trends on the two bimanual tasks?

  5. Can the authors report separate full-success and partial-success rates for the bimanual tasks, together with variability across repeated evaluations?

5 comments
AuthorsOpenReview ↗

Response to Reviewer uUwZ [1/3]

We thank the reviewer for the detailed review and for the concrete experiments suggested. We address the weaknesses and questions below.


W1 & Q1. Interpreting contact intensity: is it contact area, contact likelihood or pressing strength, and how does it relate to physical force?

By construction, the intensity combines contact likelihood and contact area: it is a fixed weighted average (MANO skinning weights) of the tracker's per-vertex contact probabilities around each keypoint. Empirically, it also correlates with measured hand force, consistent with how Sec. 3.1 defines it: a graded measure of how firmly each keypoint presses, not a calibrated force (Sec. 5). The paper related it to pressing strength only through task success; we now test it directly. Our main evidence is (1) the intensity labeled on human hands against measured hand force, since ConAct learns how firmly to press from human videos; as a supplementary consistency check, we also compare (2) the intensity predicted on the robot with the robot's joint effort.

(1) Human labels. Intensity rises with pressing force, even among contact frames alone, so it carries more than whether contact occurs. We ran the tracker that labels our demonstrations (Anonymous, 2026) on the held-out test splits of OpenTouch (Song et al., 2025; egocentric video with a tactile glove) and PVDB (Grady et al., 2022; pressure pad), and correlated the pooled intensity with the measured hand force.

Table R1. Spearman ρ between pooled contact intensity and measured human hand force (per frame: mean intensity over the 21 keypoints vs. total hand force; OpenTouch 23k frames from 190 sequences, PVDB 90k frames from 1,668 sequences), with 95% intervals from a bootstrap over sequences.

OpenTouchPVDB
All frames0.77 [0.72, 0.80]0.66 [0.64, 0.67]
Contact frames only0.51 [0.40, 0.59]0.45 [0.40, 0.49]

Contact area alone correlates with force about as strongly (ρ = 0.75–0.78), so the intensity reflects contact extent that co-varies with pressing strength.

(2) Supplementary check on the robot. Predicted intensity also rises with the robot hand's joint effort on all three tasks. The hand has no force or tactile sensing, so we use its only contact-related signal, the joint effort reported by the Wuji Hand 2 driver, which rises when a joint is obstructed [1]. We correlate the intensity that Ours (10% teleop) predicts on held-out teleoperation episodes with the recorded effort.

Table R2. Spearman ρ between predicted intensity and Wuji Hand 2 joint effort on held-out teleoperation episodes (per frame: mean intensity over the 21 keypoints vs. summed effort over the 20 finger joints; 90 / 45 / 45 episodes), with 95% intervals from a bootstrap over episodes.

Pick & PlaceFlip Plastic BagReplace Tissue Roll
Ours0.73 [0.71, 0.75]0.51 [0.43, 0.57]0.68 [0.66, 0.70]

We have added this analysis to the revised appendix (App. J, Table 6) and aligned the remaining "force" wording with this definition: the abstract, Sec. 3.3 and the conclusion now describe ConAct as displacing the robot hand's keypoint targets rather than producing force, and App. J notes that the intensity is closely tied to contact extent.

[1] Wuji Technology. Wuji Hand ROS 2 driver documentation. https://github.com/wuji-technology/wujihandros2.

AuthorsOpenReview ↗

Response to Reviewer uUwZ [2/3]

Q2. Choice of δ: how was δ = 5 cm selected, and how sensitive are the results to it across objects and tasks?

We chose δ from a coarse sweep over {2.5, 5, 10} cm on the robot, with one value shared across all keypoints, objects and tasks; Table R3 reports this sweep together with δ = 0.

Table R3. Success rate (%) of Ours (10% teleop) across δ; 16 trials per setting (on Pick & Place, 4 objects × 4 positions with one trial each, as in Table 3), except δ = 5 cm, which is Table 1 (64 trials on Pick & Place).

δPick & PlaceFlip Plastic BagReplace Tissue Roll
025.00.09.4
2.5 cm50.06.346.9
5 cm56.315.653.1
10 cm56.315.640.6

What matters is a non-zero displacement, not its exact scale: removing it (δ = 0) lowers success on every task, whereas success changes much less across δ = 2.5–10 cm, a 4× range, except on Flip Plastic Bag, where all values are low (at most 15.6%).

δ thus acts as a scale hyperparameter rather than a per-object force calibration: one value works across four Pick & Place objects, a deformable bag and a thin core. How firmly each keypoint presses relative to the others comes from the predicted intensity, and replacing it with a binary or constant value lowers success (Table R4).

δ was chosen on these same trials; at δ = 2.5 or 10 cm, ConAct would still exceed every 10%-teleop baseline in Table 1 on Pick & Place and Replace Tissue Roll, and at δ = 10 cm also on Flip Plastic Bag. We do not claim robustness beyond these objects and this hand. The revised PDF states how δ was chosen in Sec. 3.3 and adds the sweep to App. J (Table 7).


W2 & Q3. Simpler closure strategies: the binary-label variant reuses δ = 5 cm and the no-force variant keeps contact-weighted IK. How does ConAct compare with a tuned binary-contact policy on the same point representation, IK solver and pipeline?

We compare (1) simpler closure strategies at a matched contact intensity and (2) the two uses of contact at deployment, all on the same point representation, IK solver and deployment pipeline (Pick & Place, 10% teleop; 16 trials per variant, while Ours is Table 1 with 64 trials).

(1) Simpler closure strategies. Table 3's binary variant (labels thresholded at 0.2) applies the full δ whenever contact is predicted, whereas ConAct's intensities rarely exceed 0.5, so it closes the hand much harder. We therefore replaced the intensity in Eq. 3 with s = 0.34, the mean human-label intensity where the binary label marks contact; ConAct predicts a similar 0.29 on average where its intensity exceeds 0.2, so s, if anything, presses slightly harder. The binary policy applies s only where it predicts contact; the fixed offset applies it to every keypoint at all times (1.7 cm). Both keep contact-weighted IK, so only the closure rule differs.

Table R4. Closure strategies (success rate, %).

ClosureIntensity in Eq. 3Success
Ourspredicted, graded56.3
Binary contact (matched s)s where contact is predicted43.8
Fixed offset (matched s)s everywhere, always18.8

ConAct outperforms both simpler strategies (43.8 and 18.8%), suggesting that the benefit comes from graded contact rather than from closing the hand harder. The fixed offset performs worst, as it closes the fingers before they reach the object.

(2) Displacement and contact-weighted IK. Table 3's w/o displacement variant (δ = 0) keeps contact-weighted IK. To separate the two, we also replace contact-weighted IK with uniform weights, with and without the displacement.

Table R5. Uses of contact at deployment (success rate, %).

Contact-weighted IKUniform IK
Displacement (δ = 5 cm)56.350.0
No displacement (δ = 0)25.012.5

The displacement accounts for most of the gain (removing it costs 31.3–37.5 points), and contact-weighted IK adds a smaller benefit (6.3–12.5 points); without either, success falls to 12.5%.

We have added these variants to the revised appendix (App. J, Table 8) and corrected one detail in Sec. 4.3 and App. D: the binary-label variant binarizes the labels during fine-tuning only, not during pretraining as the submitted version said.

AuthorsOpenReview ↗

Response to Reviewer uUwZ [3/3]

W3, Q4 & Q5. Bimanual coverage and variability: do the contact ablations show similar trends on the bimanual tasks, and what are the full and partial success rates with variability across repeated evaluations?

We repeated the three contact ablations of Table 3 on Flip Plastic Bag and Replace Tissue Roll with the protocol of Table 1 (16 trials per task, same initial layouts), and evaluated every method of Table 1 four times on both tasks to measure run-to-run variability.

Table R6. Contact ablations (success rate, %, 10% teleop), one run of 16 trials each (64 for Ours on Pick & Place); Ours is Table 1's run, the first of the four repeats averaged in Tables R7 and R8. Bimanual: partial success counts 0.5, as in the paper.

Pick & PlaceFlip Plastic BagReplace Tissue Roll
Ours56.315.653.1
w/o contact labels43.812.543.8
Binary labels37.512.534.4
w/o displacement (δ = 0)25.00.09.4

The bimanual tasks show a similar ordering to Pick & Place (w/o contact labels and binary labels tie on Flip Plastic Bag), with the largest drop from removing the displacement.

For Q5, Tables R7 and R8 report full success (both subtasks completed) and success with partial credit (Table 1's metric; completing only the first subtask counts 0.5) for every bimanual setting of Table 1.

Table R7. Full success (%), mean ± standard deviation over four repeats of 16 trials. All repeats use the same 16 initial layouts, so the standard deviation measures the variability of real-robot execution, not variation across different sets of initial layouts.

Flip 10%Flip 50%Flip 100%Tissue 10%Tissue 50%Tissue 100%
VITRA0.0 ± 0.017.2 ± 3.148.4 ± 3.10.0 ± 0.032.8 ± 6.050.0 ± 5.1
EgoVLA9.4 ± 3.67.8 ± 3.135.9 ± 3.10.0 ± 0.00.0 ± 0.034.4 ± 6.3
GR00T N1.70.0 ± 0.012.5 ± 0.029.7 ± 3.10.0 ± 0.06.3 ± 5.120.3 ± 3.1
Ours10.9 ± 3.126.6 ± 3.156.3 ± 5.117.2 ± 3.145.3 ± 6.059.4 ± 6.3

Table R8. Success with partial credit (%), mean ± standard deviation over four repeats of 16 trials; the first repeat is the run reported in Table 1.

Flip 10%Flip 50%Flip 100%Tissue 10%Tissue 50%Tissue 100%
VITRA7.0 ± 1.620.3 ± 3.148.4 ± 3.16.3 ± 2.651.6 ± 6.057.0 ± 6.9
EgoVLA10.9 ± 3.117.2 ± 1.842.2 ± 3.112.5 ± 2.615.6 ± 2.648.4 ± 3.1
GR00T N1.78.6 ± 1.617.2 ± 1.830.5 ± 3.912.5 ± 0.025.0 ± 5.135.2 ± 1.6
Ours14.8 ± 1.630.5 ± 4.759.4 ± 5.150.8 ± 3.061.7 ± 1.674.2 ± 1.6

ConAct achieves the highest mean success on both metrics in every bimanual setting.

We have added the bimanual ablations and the full and partial success over four repeats to the revised appendix (App. J, Tables 9 and 10). Sec. 5 already states the evaluation scope (one robot, a fixed egocentric camera, three tasks); in the camera-ready version, we will also clarify that the distractor experiment tests the observation design, not the contact representation.

We thank the reviewer again for the concrete suggestions, which led to the gain sweep, the binary-contact baseline and the bimanual ablations.

Reviewer_wqEDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+2)
Read review
Paper summary

ConAct learns policies for multi-fingered robot hands from human videos with limited teleoperated demonstrations. It extends a shared human–robot keypoint representation with per-keypoint contact intensities, automatically derived from a pretrained hand tracker. A diffusion policy predicts future hand keypoints and contact intensities. During deployment, the predicted intensities adjust the keypoint targets along finger-closing directions before inverse kinematics, encouraging firmer contact through position control. Experiments on three real-world manipulation tasks show improved performance over the evaluated human-video-pretrained baselines, especially with limited teleoperation data.

Critical strengths
  1. ConAct connects automatically estimated human contact with robot control through a simple keypoint-based interface. This provides a concrete way to use contact predictions during manipulation without requiring tactile sensing on the deployed robot.

  2. The gains in the low-data regime support its practical value for reducing the cost of collecting dexterous robot demonstrations.

Critical weaknesses

1.In the abstract, the authors claim that "This design not only lets the policy learn from human videos how firmly to press, but also lets us turn the predicted contact into force along the normals of the robot hand at deployment, without any force sensing." However, I think this claim is insufficiently supported. The contact intensity is a weighted aggregation of contact probabilities, rather than a pressure or force estimate. At deployment, it scales a positional offset; the resulting force depends on the controller, hand mechanics, and object properties. The experiments demonstrate improved task performance, but do not validate a relationship between the predicted intensity and the force required or applied.

2.The contact-to-control mapping is heuristic and insufficiently validated. ConAct converts predicted contact intensity into a positional offset using a fixed gain of 5 cm. The paper does not explain how it was selected or report sensitivity to this choice. Since the resulting force depends on object properties and robot mechanics, it remains unclear how robust this mapping is beyond the tested settings. A gain-sensitivity analysis would help establish whether the reported benefits hold over a reasonable parameter range.

Questions for authors

How well does ConAct perform when fine-tuned only on the human demonstrations, without any teleoperated robot data? What failure modes are resolved by adding the small number of robot demonstrations?

4 comments
AuthorsOpenReview ↗

Response to Reviewer wqED [1/2]

We thank the reviewer for the clear assessment and for noting the practical value of ConAct in the low-data regime. We address the weaknesses and the question below.


W1. Contact and force: the abstract claims that predicted contact is turned into force, but the intensity aggregates contact probabilities and the resulting force depends on the controller, hand and object. Is this claim supported?

As defined in Sec. 3.1, the intensity is a graded measure of how firmly each keypoint presses; it is not a calibrated force (Sec. 5). It correlates with how firmly the human hand presses and, as a supplementary check, with the robot hand's joint effort. The paper supported this only through task success; we now test it directly. Our main evidence is (1) the intensity labeled on human hands against measured hand force, since ConAct learns how firmly to press from human videos; as a supplementary consistency check, we also compare (2) the intensity predicted on the robot with the robot's joint effort.

(1) Human labels. Intensity rises with pressing force, even among contact frames alone, so it carries more than whether contact occurs. We ran the tracker that labels our demonstrations (Anonymous, 2026) on the held-out test splits of OpenTouch (Song et al., 2025; egocentric video with a tactile glove) and PVDB (Grady et al., 2022; pressure pad), and correlated the pooled intensity with the measured hand force.

Table R1. Spearman ρ between pooled contact intensity and measured human hand force (per frame: mean intensity over the 21 keypoints vs. total hand force; OpenTouch 23k frames from 190 sequences, PVDB 90k frames from 1,668 sequences), with 95% intervals from a bootstrap over sequences.

OpenTouchPVDB
All frames0.77 [0.72, 0.80]0.66 [0.64, 0.67]
Contact frames only0.51 [0.40, 0.59]0.45 [0.40, 0.49]

(2) Supplementary check on the robot. Predicted intensity also rises with the robot hand's joint effort on all three tasks. The hand has no force or tactile sensing, so we use its only contact-related signal, the joint effort reported by the Wuji Hand 2 driver, which rises when a joint is obstructed [1]. We correlate the intensity that Ours (10% teleop) predicts on held-out teleoperation episodes with the recorded effort.

Table R2. Spearman ρ between predicted intensity and Wuji Hand 2 joint effort on held-out teleoperation episodes (per frame: mean intensity over the 21 keypoints vs. summed effort over the 20 finger joints; 90 / 45 / 45 episodes), with 95% intervals from a bootstrap over episodes.

Pick & PlaceFlip Plastic BagReplace Tissue Roll
Ours0.73 [0.71, 0.75]0.51 [0.43, 0.57]0.68 [0.66, 0.70]

We have added this analysis to the revised appendix (App. J, Table 6) and aligned the remaining "force" wording with this definition. In particular, the abstract and contribution 3 now state that ConAct displaces the robot hand's keypoint targets rather than converting contact into force, and Sec. 3.3, Sec. 4, Table 3 and the conclusion now describe it as a (contact-driven) displacement instead of "force injection".

[1] Wuji Technology. Wuji Hand ROS 2 driver documentation. https://github.com/wuji-technology/wujihandros2.

AuthorsOpenReview ↗

Response to Reviewer wqED [2/2]

W2. Contact-to-control gain: how was δ = 5 cm selected, and do the benefits hold over a reasonable range of the gain?

We chose δ from a coarse sweep over {2.5, 5, 10} cm on the robot, with one value shared across all keypoints, objects and tasks; Table R3 reports this sweep together with δ = 0.

Table R3. Success rate (%) of Ours (10% teleop) across δ; 16 trials per setting (on Pick & Place, 4 objects × 4 positions with one trial each, as in Table 3), except δ = 5 cm, which is Table 1 (64 trials on Pick & Place).

δPick & PlaceFlip Plastic BagReplace Tissue Roll
025.00.09.4
2.5 cm50.06.346.9
5 cm56.315.653.1
10 cm56.315.640.6

What matters is a non-zero displacement, not its exact scale: removing it (δ = 0) lowers success on every task, whereas success changes much less across δ = 2.5–10 cm, a 4× range, except on Flip Plastic Bag, where all values are low (at most 15.6%).

δ thus acts as a scale hyperparameter rather than a per-object force calibration: one value works across four Pick & Place objects, a deformable bag and a thin core. How firmly each keypoint presses relative to the others comes from the predicted intensity, and replacing it with a binary or constant value lowers success (e.g., 43.8% for a binary closure at the mean contact intensity of the human labels, close to what ConAct predicts during contact, and 18.8% for a constant offset, which closes the fingers before they reach the object, vs. 56.3%; Reviewer uUwZ, W2 & Q3).

δ was chosen on these same trials; at δ = 2.5 or 10 cm, ConAct would still exceed every 10%-teleop baseline in Table 1 on Pick & Place and Replace Tissue Roll, and at δ = 10 cm also on Flip Plastic Bag. We do not claim robustness beyond these objects and this hand. The revised PDF states how δ was chosen in Sec. 3.3 and adds the sweep to App. J (Table 7).


Q1. Human-only fine-tuning: how does ConAct perform without any teleoperated robot data, and which failure modes do the few robot demonstrations resolve?

(1) Trained on the human demonstrations as is, every method fails, and (2) retargeting the human hand labels to the robot hand makes ConAct the only human-only policy that succeeds: the pixel-based baselines face a perception gap on top of the embodiment gap, whereas ConAct's gap lies mainly in embodiment.

(1) Human only. We fine-tuned ConAct, GR00T N1.7 and VITRA on the 200 human demonstrations alone (0% teleop) and deployed them as in Table 1.

Table R4. Success rate (%) without teleoperated robot data; 16 trials per task (the 10% teleop row is Table 1, with 64 on Pick & Place). Baselines use the human demonstrations converted to their action spaces (App. I); EgoVLA has no human-only variant, as its hand retargeting is trained on teleoperated demonstrations.

Pick & PlaceFlip Plastic BagReplace Tissue Roll
GR00T N1.7, human only0.00.00.0
VITRA, human only0.00.00.0
ConAct, human only0.00.00.0
ConAct, human only, retargeted hand labels25.09.437.5
ConAct, 10% teleop56.315.653.1

The robot hand looks unlike any hand in the baselines' training images, so they produce no meaningful actions. ConAct sees both hands as the same 21 keypoints and approaches the correct object, but predicts a human-sized hand (hand span 0.87–0.95 of the robot's). Even after deployment rescales the bone lengths (App. E), the predicted wrist placement and finger directions fit a smaller hand, so the fingers do not close around the object.

(2) What robot data resolves. We retargeted the human hand labels to Wuji Hand 2 by per-frame kinematic fitting of its 21 keypoints, keeping the contact labels and still using no robot demonstrations. This policy succeeds on all three tasks, reaching 25.0 / 9.4 / 37.5%, i.e., 44–71% of the 10% teleop success rates. Much of what the few robot demonstrations provide is the robot's hand geometry, which a kinematic model can partly supply. We have added these results and the failure analysis to the revised appendix (App. J, Table 11) and will summarize them in Sec. 4 in the camera-ready version.

We thank the reviewer again for the constructive comments, which led to a direct test of the contact-force relation, a gain-sensitivity analysis and a human-only study.

Thank you for the response. The experiment demonstrating the correlation between human-annotated intensity and measured hand forces is compelling. The validation of parameter choices and the additional results on human-only fine-tuning further strengthen the paper. I am therefore happy to raise my score.

AuthorsOpenReview ↗

Thank you for your updated assessment

Dear Reviewer wqED,

Thank you for carefully reading our response and the revised paper, and for raising your score. We are glad that you find the correlation between the human-annotated intensity and the measured hand forces compelling, and that the validation of the parameter choices and the human-only fine-tuning results further strengthen the paper. All of these are included in the revised paper (App. J) and will remain in the final version.

Thank you again for your time and constructive review, which helped us improve the paper.

Best regards,
The Authors

Reviewer_bu8WOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ConAct, which emphasis to extract contact information from the human video, and use that to make the policy apply enough force. The main idea is to augment 3D hand keypoint actions with contact intensity, allowing the policy to predict both hand motion and contact. The authors use a pretrained hand tracker to generate contact labels from human videos and convert predicted contact intensities into finger-closing offsets during robot execution.

Critical strengths
  1. The main idea of using contact information is very interesting. It is a common problem from human data only that you don't know what is the actual force applied. The paper is a solid effort in that direction

  2. Having a separate contact head for the policy architecture is an interesting design to capture the contact information.

  3. The authors show the results in real world and performs relatively well, and the ablation shows the method is helpful.

Critical weaknesses
  1. My main concern is the contact estimation part, which is one of the most important pieces to the proposed method. However, this part relies on an anonymous project website, making it difficult to fully understand and evaluate how contact is estimated from videos alone. Although the appendix provides some details, I think the paper would benefit from a more complete explanation and validation of this critical component.

  2. I find the policy architecture interesting, particularly the use of a separate contact prediction head. Another interesting design choice is that, instead of incorporating contact into the keypoint targets during training, the authors modify the predicted keypoints at inference time using Equation (3). I would encourage the authors to provide more intuition behind these choices. What are the advantages of this design compared with directly incorporating contact into the action targets during training?

  3. The predicted contact intensity is derived from contact probabilities rather than measured forces. At deployment, it is converted into a position offset to generate force. However, it is unclear how well this contact intensity reflects how firmly the human hand presses, especially across objects with different physical properties and contact conditions. I wonder whether the authors could provide further analysis or validation of this relationship.

  4. The real-world experiments primarily focus on grasping and pick-and-place-like tasks, where applying sufficient grip force is particularly important. It would be interesting to evaluate whether the proposed contact representation also benefits more general dexterous manipulation tasks, such as in-hand manipulation or tasks requiring more precise contact regulation.

  5. From the videos, the robot motions appear relatively slow and occasionally jittery. It would be helpful if the authors could discuss what causes these behaviors (whether it is because the contact prediction is noisy?)

Questions for authors
  1. Clearly explain the contact estimation method in the main paper and provide some quantitative analysis of its accuracy. Since this is a key component of the proposed approach, it is important to understand how reliable the estimated contact labels are.

  2. Provide more intuition behind the policy architecture and design choices, particularly the use of a separate contact prediction head and the decision to modify the predicted keypoints at inference time rather than incorporating contact directly into the keypoint predictions during training.

  3. Briefly explain why a fixed value of delta = 5 cm is used in Equation (3). How sensitive is the performance to this parameter, and is this value related to the scale or calibration of the estimated contact intensities?

  4. Consider evaluating the method on manipulation tasks beyond grasping and pick and place. Alternatively, provide some discussion of how the proposed approach might generalize to more challenging dexterous manipulation tasks that require precise contact regulation.

  5. The robot motions in the videos appear somewhat jittery. Could the authors analyze whether this is related to the contact predictions and the resulting keypoint offsets? It would also be helpful to discuss potential ways to improve the smoothness of execution.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bu8W [1/3]

We thank the reviewer for the careful reading and for finding the use of contact and the separate contact head interesting. We address the weaknesses and questions below.


W1 & Q1. Contact estimation: can it be explained in the paper itself, and how reliable are its labels?

We will move the description of the tracker from the appendix into Sec. 3; its contact predictions are accurate on held-out data, and the pooled intensity correlates with the ground truth (Table R1). The labels come from a video hand tracker with a per-vertex contact head on the MANO mesh, trained with tactile-glove, pressure-pad and geometric contact supervision (Anonymous, 2026). We pool its 778 per-vertex contact probabilities to our 21 keypoints by a fixed weighted average over MANO skinning weights (Sec. 3.1, App. A).

We evaluated the tracker on the held-out test splits of OpenTouch (Song et al., 2025; egocentric video with a tactile glove; 23k hand-frames) and PVDB (Grady et al., 2022; pressure pad; 90k hand-frames), whose training splits supervise its contact output (App. A), both at the mesh vertices and at the 21 pooled keypoints that ConAct trains on.

Table R1. Contact estimation accuracy on the held-out test splits.

OpenTouchPVDB
Vertex contact, AUROC0.980.98
Pooled contact intensity, Spearman ρ0.730.65

The tracker ranks contact vertices almost perfectly, and the pooled intensity correlates well with the ground truth pooled in the same way. Its paper also reports a vertex contact AUROC of 0.96 on HOT3D and 0.94 on ARCTIC, two egocentric datasets with contact derived from hand and object meshes (Anonymous, 2026). We have added this evaluation to the revised appendix (App. J, Table 5).


W2 & Q2. Design intuition: why a separate contact head, and why apply contact to the keypoints at deployment rather than folding it into the training targets?

Both choices keep the keypoint targets, the action space shared by human and robot, identical for both, and leave the robot-specific force scale to a single deployment gain, since contact intensity is a graded proxy rather than a calibrated force (W3, Q3).

Separate contact head. A pressing human hand stays on the object surface, so how firmly it presses is not visible in its keypoints. Folding contact into the targets would shift the human keypoints away from the observed trajectory. Our teleoperated demonstrations carry no contact labels, so the same grasp would then have different targets for the two embodiments. A separate head keeps the targets as observed, so that the human trajectory directly serves as a target for the robot (Sec. 1), while the head learns from human videos how firmly to press. Its input is detached so that the contact loss does not change the predicted keypoints (Sec. 3.2).

Applying contact at deployment. How far the intensity should move the robot's fingers depends on the robot hand and controller, not on the human data. Applying it at deployment reduces this to one gain δ, set for the robot without retraining, and success is far less sensitive to its scale than to removing it (Q3). We will add this rationale to Sec. 3.

AuthorsOpenReview ↗

Response to Reviewer bu8W [2/3]

W3. Contact intensity and force: how well does the predicted intensity reflect how firmly the human hand presses, across objects and contact conditions?

As defined in Sec. 3.1, the intensity is a graded measure of how firmly each keypoint presses; it is not a calibrated force (Sec. 5). It correlates with how firmly the human hand presses and, as a supplementary check, with the robot hand's joint effort. The paper supported this only through task success; we now test it directly. Our main evidence is (1) the intensity labeled on human hands against measured hand force, since ConAct learns how firmly to press from human videos; as a supplementary consistency check, we also compare (2) the intensity predicted on the robot with the robot's joint effort.

(1) Human labels. Intensity rises with pressing force, even among contact frames alone, so it carries more than whether contact occurs. On the same test splits, which also record hand force, we correlated the pooled intensity with the measured hand force.

Table R2. Spearman ρ between pooled contact intensity and measured human hand force (per frame: mean intensity over the 21 keypoints vs. total hand force; OpenTouch 23k frames from 190 sequences, PVDB 90k frames from 1,668 sequences), with 95% intervals from a bootstrap over sequences.

OpenTouchPVDB
All frames0.77 [0.72, 0.80]0.66 [0.64, 0.67]
Contact frames only0.51 [0.40, 0.59]0.45 [0.40, 0.49]

(2) Supplementary check on the robot. Predicted intensity also rises with the robot hand's joint effort on all three tasks. The hand has no force or tactile sensing, so we use its only contact-related signal, the joint effort reported by the Wuji Hand 2 driver, which rises when a joint is obstructed [1]. We correlate the intensity that Ours (10% teleop) predicts on held-out teleoperation episodes with the recorded effort.

Table R3. Spearman ρ between predicted intensity and Wuji Hand 2 joint effort on held-out teleoperation episodes (per frame: mean intensity over the 21 keypoints vs. summed effort over the 20 finger joints; 90 / 45 / 45 episodes), with 95% intervals from a bootstrap over episodes.

Pick & PlaceFlip Plastic BagReplace Tissue Roll
Ours0.73 [0.71, 0.75]0.51 [0.43, 0.57]0.68 [0.66, 0.70]

We have added this analysis to the revised appendix (App. J, Table 6) and aligned the remaining "force" wording with this definition: the abstract, Sec. 3.3 and the conclusion now describe ConAct as displacing the robot hand's keypoint targets rather than producing force.


Q3. Choice of δ: why δ = 5 cm, how sensitive is the performance to it, and is it tied to the scale of the estimated intensities?

We chose δ from a coarse sweep over {2.5, 5, 10} cm on the robot, with one value shared across all keypoints, objects and tasks; Table R4 reports this sweep together with δ = 0.

Table R4. Success rate (%) of Ours (10% teleop) across δ; 16 trials per setting (on Pick & Place, 4 objects × 4 positions with one trial each, as in Table 3), except δ = 5 cm, which is Table 1 (64 trials on Pick & Place).

δPick & PlaceFlip Plastic BagReplace Tissue Roll
025.00.09.4
2.5 cm50.06.346.9
5 cm56.315.653.1
10 cm56.315.640.6

What matters is a non-zero displacement, not its exact scale: removing it (δ = 0) lowers success on every task, whereas success changes much less across δ = 2.5–10 cm, a 4× range, except on Flip Plastic Bag, where all values are low (at most 15.6%).

δ thus acts as a scale hyperparameter rather than a per-object force calibration: one value works across four Pick & Place objects, a deformable bag and a thin core. How firmly each keypoint presses relative to the others comes from the predicted intensity, and replacing it with a binary or constant value lowers success (e.g., 43.8% for a binary closure at the mean contact intensity of the human labels, close to what ConAct predicts during contact, and 18.8% for a constant offset, which closes the fingers before they reach the object, vs. 56.3%; Reviewer uUwZ, W2 & Q3). The scale is tied to that of the intensities: the 99th percentile of the predicted intensity is 0.26–0.41 on our tasks, so at δ = 5 cm the displacement stays below about 2 cm in 99% of cases.

δ was chosen on these same trials; at δ = 2.5 or 10 cm, ConAct would still exceed every 10%-teleop baseline in Table 1 on Pick & Place and Replace Tissue Roll, and at δ = 10 cm also on Flip Plastic Bag. We do not claim robustness beyond these objects and this hand. The revised PDF states how δ was chosen in Sec. 3.3 and adds the sweep to App. J (Table 7).

[1] Wuji Technology. Wuji Hand ROS 2 driver documentation. https://github.com/wuji-technology/wujihandros2.

AuthorsOpenReview ↗

Response to Reviewer bu8W [3/3]

W4 & Q4. Beyond grasping: does the contact representation help in-hand manipulation or tasks that need precise contact regulation?

The representation is not specific to grasping: hand keypoint trajectories with a contact intensity at each keypoint describe general hand–object interaction, and our tasks already go beyond holding an object firmly (flipping a deformable bag and pushing it to a target; pulling a 2 cm core off its holder and placing a new roll). We kept the tasks in this range so that demonstrations stay simple to collect, and have not yet evaluated in-hand manipulation.

For tasks such as in-hand reorientation, we expect the limit to be the precision of the contact labels rather than the framework, as contact estimated from ordinary videos may miss small, rapidly changing fingertip contacts. Labeling contact with a tactile glove, which measures contact and pressure directly, would address this without changing the policy, since ConAct already learns per-keypoint contact from labels. On the deployment side, Eq. 3 displaces the targets only along the finger-closing direction, which suits contacts made by closing the fingers; regulating tangential contact would add a second direction to Eq. 3, a natural next step once such glove labels are available. We will discuss this scope and extension in the limitations.


W5 & Q5. Slow and jittery motion: is it caused by noisy contact predictions, and how could execution be smoother?

The jitter comes from how we execute the predicted trajectory, not from the policy or its contact predictions. The joint targets are interpolated linearly between 20 Hz steps, and after each chunk the robot holds its last command for 100 ms so that the arms settle before re-planning (App. E), which makes the motion stop and go. The predicted trajectories themselves are smooth, as the interactive 3D viewer on our project page shows. Blending consecutive chunks instead of holding would remove these pauses; we kept this simple scheme so that the evaluation reflects the policy itself, and will note this in the limitations.

We thank the reviewer again for the constructive comments, which led to a direct evaluation of our contact labels and a clearer account of the design.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all Reviewers

We thank all reviewers for their valuable comments and constructive feedback. We summarize here our responses to the two concerns shared by all reviewers; details are in our response to each reviewer.

(1) Contact intensity and force. We tested the intensity directly: the intensity labeled on human hands correlates with measured hand force on two public datasets, and, as a supplementary check, the intensity predicted on the robot correlates with the robot hand's joint effort. As defined in Sec. 3.1, the intensity is a graded measure of how firmly each keypoint presses, not a calibrated force (Sec. 5); the revised PDF aligns the remaining "force" wording with this definition.

(2) Sensitivity to δ. We swept δ ∈ {0, 2.5, 5, 10} cm on all three tasks. What matters is a non-zero displacement, not its exact scale: removing it (δ = 0) lowers success on every task, whereas success changes much less across δ = 2.5–10 cm, a 4× range, except on Flip Plastic Bag, where all values are low (at most 15.6%).

Revised PDF. We have uploaded a revised PDF, with all changes in blue. It (i) replaces the remaining "force" wording with "contact-driven displacement" (abstract, Sec. 1, 3.3, 4.3, Table 3, conclusion), (ii) states how δ was chosen (Sec. 3.3) and corrects the binary-label description (Sec. 4.3, App. D), and (iii) adds App. J with all new analyses of this rebuttal (Tables 5–11). The main text stays within 9 pages; the remaining relocations to the main text (tracker description and design rationale in Sec. 3, a summary of the human-only results in Sec. 4, the grasping scope in the limitations) will be made in the camera-ready version with the additional page.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.