acceptodds
Under review as a conference paper at ICLR 2027

Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation

Abstract

A primary bottleneck in contact-rich manipulation is the difficulty of collecting real-world data. Sim-to-real reinforcement learning offers a scalable alternative, but the simulation-reality gap prevents information-dense modalities like touch from being effectively used. Existing sim-to-real methods often mitigate this gap by simplifying tactile data into coarse low-dimensional features – sacrificing the richness required for complex manipulation. In this work, we introduce Center-of-Pressure (CoP), an effective tactile representation grounded in physical principles that preserves the dominant normal-force and contact-location information while maintaining robustness for sim-to-real transfer. To support this representation, we propose a sensor calibration scheme based on differentiable dynamics, enabling the estimation of taxel orientations without requiring ground-truth force measurements. We evaluate CoP on two blind, challenging contact-rich manipulation tasks: peg-in-hole insertion and ball balancing. Across both tasks, policies conditioned on CoP transfer to a multi-fingered hand without any task-specific real-world training, and outperform both coarse binary-contact and raw-taxel baselines. Analysis of learned policy states further suggests that CoP-conditioned policies encode task-relevant physical properties, such as object mass, as an emergent byproduct of control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dexterous Manipulation, Tactile Representation, Sim-to-Real

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zBLbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Center of Pressure (CoP) as an intermediate tactile representation for sim to real RL. CoP consists of a normal force vector and a contact location per fingertip, chosen to sit between coarse binary contact and raw taxel arrays. A parametric stress distribution model gives a differentiable bidirectional mapping between taxel forces and CoP, with the CoP force recovered in closed form by regularized least squares. Unknown taxel orientations are calibrated by matching the wrench inferred from tactile readings to measured joint torques under static equilibrium, using differentiable kinematics and requiring no ground truth force sensor. Policies are trained with PPO in IsaacLab and transferred zero shot to Allegro hand with XELA uSkin, on two blind tasks including peg in hole insertion and ball balancing.

Critical strengths
  • The representation is a well chosen abstraction and the chosen tasks and evaluation are strong enough to demonstrate its effectiveness.
  • The calibration method is a practical contribution, estimating taxel orientations by matching tactile inferred wrench against joint torques under static equilibrium removes the dependence on a high precision force sensor.
Critical weaknesses
  • The abstraction loses information from raw sensory data, since only the surface normal component is used and shear force is not used, while shear carries slip and tangential alignment information required for insertion.
  • The calibration reliability and accuracy is not measured and reported.
Questions for authors
  • What is the computational cost of the taxel to CoP mapping at inference?
  • The masked sensor result shows CoP degrading more than simplified representations. Does a representation at intermediate fidelity exist that retains CoP's performance while matching binary's robustness, for example by including the active taxel count or a confidence term alongside CoP?
1 comment

We thank the reviewer for the constructive comments. Both weaknesses are fair, and the second prompted new experiments, reported below.

W1. Only the normal component is used, and shear carries slip and tangential information. We agree this is a real limitation, and we state it in Sec. 3.4 and Sec. 6(B). Two clarifications. First, the restriction comes from the simulator, not the representation. The stress-distribution model includes shear (Sec. 3.2), and the calibration data excite shear responses (Sec. 3.3). We drop the tangential component because IsaacLab's shear estimates were unreliable for our fingertip geometries. A channel that disagrees between simulation and hardware would undermine the alignment the representation exists to provide. Second, the results show how far a normal-only signal goes: 78% blind insertion across six shapes, and 63% under OOD initialization. The policy is not blind to tangential interaction, since normal forces at several differently oriented fingertips, with contact locations and joint positions over time, partially constrain the net wrench on the peg. We expect shear to help further, particularly for slip, once simulated shear is reliable. We have made the normal-only scope explicit wherever the paper says "3D force vector" (Sec. 1, Sec. 3.1).

W2. Calibration reliability and accuracy are not measured. We agree, and have run two additional evaluations on the real-hardware calibration recording consisting of 2,400 frames. Full per-fold, per-joint and per-taxel results are in Appendix A. Accuracy. We ran 5-fold blocked cross-validation. Folds are contiguous time blocks with a 10-frame purge gap, so temporally adjacent frames never fall in both training and test sets. The joint-torque RMSE on held-out contacts is 0.266 ± 0.022 N·m, close to the 0.260 ± 0.006 N·m on training contacts, so the calibration does not overfit. Reliability. We ran 50 independent calibrations, from 5 data subsets (the cross-validation training sets) and 10 random initializations each, and measured the geodesic angle between learned orientations for each taxel. Across initializations on the same data, the median per-taxel spread is 0.9°, and 27 of 30 taxels stay below 10°. Across data subsets, the median spread is 3.2° and the maximum is 28.2°. Two taxels converge to different orientations from different initializations, with spreads of 33° and 52°. Held-out RMSE nevertheless varies by only 0.001 N·m (std) across initializations, so these two orientations are weakly identified by the torque objective and do not affect torque prediction.

Q1. Computational cost of the taxel-to-CoP mapping at inference. The mapping is closed-form. Each fingertip needs one weighted average for the contact position and one regularized 3×3 solve for the force. Cost is linear in the number of active taxels, with no iterative optimization and no learned encoder. It takes 0.15 ms on average and at most 0.4 ms per step for all four fingertip sensor arrays, against a control period of 50 ms. We now report this in Sec. 3.2.

Q2. cop degrades more under masking. Could an intermediate representation keep cop's performance with bin's robustness, for example with an active-taxel count or a confidence term? Two observations first. Under 40% masking, cop loses the most relative to its own unmasked result (78% to 62%), but it remains the highest in absolute terms: 62%, against 57% for vec, 52% for bin and 30% for taxel. The gaps to vec and bin are within trial noise, so we claim only that cop is no worse. Also, taxel masking is not among the training randomizations (Table 6), so this is a zero-shot test under an unseen corruption. The reviewer's suggestion fits our framework well. The mapping already yields two confidence signals per fingertip: the active-taxel count and the least-squares residual of the CoP fit. Because the mapping is bidirectional, we could also train with taxel dropout in simulation: simulated CoP to taxel forces, random masking, then back to CoP. The policy would then see CoP errors with the same structure as on hardware. We have added this to Sec. 4.1 as a promising future direction.

Reviewer_e1udOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a compact contact representation for sim-to-real dexterous manipulation, combining a resultant force and contact location. A differentiable stress-distribution model maps between taxel readings and this CoP descriptor. Taxel orientations are calibrated using joint-torque consistency, and recurrent policies are trained in simulation for blind insertion and ball balancing before hardware evaluation. The practical contribution is the alignment and calibration pipeline together with real closed-loop manipulation, rather than invention of the classical pressure-center concept or a complete arbitrary-contact wrench representation.

Critical strengths

The paper addresses a relevant tradeoff between rich sensor-specific observations and compact transferable physical summaries. Real hardware experiments on two feedback-dependent tasks provide stronger application evidence than simulation alone. The calibration method avoids requiring an external ground-truth force sensor and is described with an explicit data budget. Representation ablations compare magnitude, vector, position, binary, proprioceptive, and raw-taxel inputs. These positive details should be retained when interpreting the limitations.

Critical weaknesses

The real insertion improvement also needs better uncertainty reporting. CoP succeeds in 47 of 60 trials across six shapes, versus 40 of 60 for the vector-only baseline. This is a promising empirical difference, but ten trials per shape do not establish robustness across training seeds. Simulation curves already include multiple seeds; the missing information concerns which trained policies were transferred and how real trials were distributed across them. Trial-level data, paired resets, experiment order, and a confidence interval for the performance difference would help. Overlap between separate intervals would not itself establish non-significance, and I do not make that inference here. Completion time is conditioned on success and therefore compares different selected subsets.

Questions for authors
  1. Can you provide per-trial Table 2 durations, exact aggregation code or definitions, and corrected overall means and variability?

  2. How many independently trained policies contributed to each real-world table, and can the CoP versus vector-only difference be repeated across seeds with paired initial conditions?

1 comment

We thank the reviewer for the statistical scrutiny. Rechecking the tables in response exposed an aggregation error in Table 2, corrected below. We also agree with the reviewer's framing of the contribution: the alignment and calibration pipeline with real closed-loop evaluation, not the classical pressure-center concept (Sec. 2, Sec. 3.1).

1. Table 2: per-trial durations, aggregation, corrected overall values (Q1). The Overall entries for vec and cop were inconsistent with their per-ball columns. Overall is the mean and standard deviation over all 40 trials (4 balls × 10 trials). The corrected values are 4.06 ± 2.2 s for vec and 4.35 ± 1.7 s for cop; the other rows are unaffected. We have also corrected the bolding: vec is best on the tennis ball and moon ball. The conclusion in Sec. 4.2 stands. cop and vec are comparable on this task, and both last 1.7 to 1.8 times as long as the next-best robot policy (mag, 2.40 s). Per-trial durations for every trial are now in the supplementary material.

2. Uncertainty for the insertion results. We agree that Table 1 needs intervals. From the reported counts, with 95% Wilson intervals for rates and Newcombe intervals for differences: Comparison Counts Difference 95% interval cop vs vec, in distribution 47/60 vs 40/60 +11.7 pts −4.3 to +26.9 cop vs bin, in distribution 47/60 vs 32/60 +25.0 pts +8.0 to +40.1 cop vs taxel, in distribution 47/60 vs 29/60 +30.0 pts +12.8 to +44.9 cop vs vec, OOD init. 38/60 vs 25/60 +21.7 pts +3.8 to +37.7 cop vs bin, OOD init. 38/60 vs 12/60 +43.3 pts +26.0 to +57.1 The advantage over binary contact and raw taxels, which is the claim in the abstract, is supported at this sample size. The in-distribution advantage over vec is suggestive but not established, and is clearer under OOD initialization. We have revised Sec. 4.1 to say exactly this and added intervals to Table 1. These intervals reflect trial-to-trial variation of the deployed policies. They do not capture variation across training seeds, which we address next.

3. Number of trained policies, seeds, paired resets, experiment order (Q2). One independently trained policy per representation contributed to each real-world table: seed 0 at the final training iteration, with no selection based on hardware performance. Trials were run in blocks per method. Initial conditions were re-randomized at every reset rather than paired across methods. For insertion, the peg yaw is re-randomized and its position varies with manual reset noise (Sec. 4.1). For ball balancing, the ball is placed at a random point on a circle with a radius of about half the plate's side length. We have documented this in a new "Real-world evaluation protocol" paragraph. We agree that this does not establish robustness across training seeds, and that a multi-seed comparison with paired resets is the right way to do so. We were unfortunately not able to run it. Each additional seed of cop and vec alone requires 120 further insertion trials, every one with a manual reset, plus regular cool-down pauses for the hand. We have instead limited our claims to what the evidence supports. The revised text states that real-world results come from one trained policy per representation, reports the intervals above, and describes the in-distribution advantage of cop over vec as suggestive. Single-seed deployment is now listed as a limitation in Sec. 6. Training returns for cop and vec vary little across five seeds in simulation (Fig. 13), which suggests the deployed policies are not atypical, although we recognize this is indirect evidence for hardware.

4. Completion time is conditioned on success. The reviewer is right that each method's time is averaged over a different subset of trials. We considered averaging over all trials instead. Failed trials would then enter at the maximum allowed time, so the mean would depend on an arbitrary choice of time limit and would largely restate the success rate. We have therefore kept time over successful trials, as a secondary metric to be read together with success rate and not as a ranking on its own. This matches our reading in Sec. 4.1: the simplified representations are fast mainly because they succeed only from favorable initial yaws. The Table 1 caption now states this.

Reviewer_687nOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a solution for sim-to-real gap in dexterous manipulation. The central premise is that binary contact indicators are insufficient for fine-grained manipulation tasks and a representation carrying more physics information like contact force and position would be helpful. A policy trained in IsaacLab using PPO and combined with domain randomization and system identification was evaluated on A16-DoF allegro hand with contact sensors on two tasks - peg-in-hole insertion for different geometries and ball balancing on plate.

Critical strengths

Strengths

  1. Physics grounded representation - Using a point force wrench seems like a practical approach. The authors also explicitly acknowledge the trade-off between single point wrench and multi-contact.
  2. Tasks are well designed and results support it well - Success rate for peg-in-hole is 78 for CoP, 67% for force vector only and 53 for binary shows evidence of active object interaction (Table 1). Resilience to Out of distribution OOD initializations are also insightful - 63% vs 42% for force vector only and 20% for binary
  3. A comprehensive system engineering and ablation add a lot insightful observations - especially important for a method that works on real robot. different observation modalities, OOD initializatons, validating GRU vs history stacking across multiple seeds and calibration, system identification and delay modeling are extremely useful information.
Critical weaknesses
  1. Some inconsistencies needs to be addressed - In section 4.2, first sentence in "Real-world performance" - it says only cop, vec and taxel policies are successful. Why is this the case? Especially when "mag" seems to achieve similar average return during training fig 13.b
  2. Fig 13 indicates that 5 random seeds were used for training - which one was deployed to obtain the results in Table 1-2? In general, more details on checkpoint selection, whether or not policies trained on different seeds are independently tested on hardware, how the initial states are chosen can be valuable addition.
Questions for authors
  1. In table 2, vec only 4.52s almost matches the CoP performance 4.60s. How to interpret this result? Would a baseline which has force magnitude + position help isolate?
  2. Might have missed this, but during training what is the contact inputs to the policy?
  3. It might be useful to add the equivalent simulation results of table 1 and 2.
1 comment

We thank the reviewer for the careful reading and the constructive suggestions.

W1. Sec. 4.2 says only cop, vec and taxel succeed, yet mag reaches a similar return in Fig. 13b. The reviewer is right; that sentence was inaccurate. In simulation (Fig. 13b), mag, vec and cop reach similar returns, taxel learns but plateaus lower, and base, bin and pos fail, as Appendix E.5 states. The gap between mag and vec/cop appears only on hardware: 2.40 s against 4.06 s and 4.35 s (Table 2). A scalar force magnitude is thus enough to learn the task in simulation, but it does not transfer. Our hypothesis is that, without force direction, the policy cannot tell where on the fingertip the plate presses, and so cannot compensate when real contact locations deviate from simulation. We have rewritten the sentence to separate the simulation and hardware findings.

W2. Which seed was deployed, how checkpoints were selected, and how initial states were chosen. One policy per representation was deployed: seed 0 at the final training iteration, with no selection based on hardware performance. For insertion, the peg yaw is re-randomized at every reset and its position varies with manual reset noise (Sec. 4.1). For ball balancing, the initial ball position is re-randomized at every reset along a circle with a radius of about half the plate's side length. We have added a "Real-world evaluation protocol" paragraph with these details. We agree that deploying one seed leaves seed-to-seed variation on hardware unmeasured, and we now state this as a limitation in Sec. 6. A multi-seed hardware evaluation was unfortunately beyond what we could carry out. Every trial needs a manual reset and the hand needs regular cool-down pauses, so each additional seed adds substantially to an already long protocol. As indirect support, training returns for cop and vec vary little across five seeds in simulation (Fig. 13), which suggests the deployed policies are not atypical.

Q1. In Table 2, vec (4.52 s) almost matches cop (4.60 s). How should this be read, and would a magnitude + position baseline help? First, a correction. While rechecking Table 2 we found that the Overall entries for vec and cop were aggregated incorrectly. The corrected values are 4.06 s and 4.35 s. The reading is unchanged: the two are comparable on this task, and each is ahead on two of the four balls. Our interpretation is that the position channel is largely redundant here. We use only the surface-normal force component (Sec. 3.4), so the force direction is the surface normal at the contact point, and on a curved fingertip that direction largely determines the contact location. In ball balancing the plate also rests at a nearly fixed place on each fingertip, so the task signal is how load shifts between fingers. In insertion the peg moves within the hand and contact locations vary widely, which is where explicit position appears to help: 47/60 vs 40/60 in distribution, and 38/60 vs 25/60 under OOD initialization. We thank the reviewer for the baseline suggestion, and considered it carefully. For the same reason as above, magnitude + position carries the same information as cop under normal-only forces: the direction follows from the position on the fingertip. It would differ from cop only in how direction is encoded, so we expect it to perform close to cop rather than isolate a component. The existing ablations already separate the components: mag (2.40 s), pos (1.55 s) and vec (4.06 s), against cop (4.35 s).

Q2. What are the contact inputs to the policy during training? The policy receives the same representation in training and deployment. In simulation, IsaacLab's ContactSensor gives the contact force and position on each taxel-covered fingertip region. We express both in the sensor frame and keep the surface-normal force component (Sec. 3.4). The ablations are derived from this: mag is the force norm, vec the force only, pos the position only, and bin a thresholded contact flag per sensor array. taxel inputs are generated by passing the simulated CoP through the inverse taxel-CoP mapping (Appendix B, Fig. 9). Observation noise and a 50–100 ms contact delay are applied during training (Table 6). On hardware, the same quantities are computed from raw taxel readings with the forward mapping. We now state this explicitly in Sec. 4.

Q3. Add the simulation equivalents of Tables 1 and 2. We appreciate the suggestion. Simulation performance of every representation, across five seeds, is reported in Fig. 13 (Appendix E.5), and we now point to it from Sec. 4.1 as well as Sec. 4.2. On insertion, all representations reach very similar returns in simulation (Fig. 13a), so the differences in Table 1 arise mainly in transfer rather than in training. For this reason we kept the tables focused on hardware, where the representations separate. Fig. 13b likewise shows the mag case discussed under W1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.