acceptodds
Under review as a conference paper at ICLR 2027

Joint-Force Matching: Contact Mechanics as Supervision for RL Critics

Abstract

Contact forces improve robot learning, and grasping demonstrations record them at every contact site and time step. In reinforcement learning (RL), the critic estimates the return of each action and guides the policy. Existing methods, however, predict each contact force or take it as an input, and they leave out how the forces load the hand's joints. Under elastic contact, these loads are the gradient of a single energy. Which structure in contact forces improves the critic's value estimates has remained unclear. We propose joint-force matching (JFM), which maps measured contact forces through the hand's Jacobians to joint loads and trains the negative joint-angle gradient of a scalar energy on the critic's representation to match them. JFM keeps force prediction and adds this gradient target. In simulated multi-finger grasping, JFM's critics have a lower return error than those of a force-prediction baseline at every pretraining seed, and JFM's offline-to-online policy succeeds on 33 of 158 held-out trajectories in closed loop against the baseline's 18. A free-vector head reaches a four to five times lower loss on the same joint loads, yet its critics have a higher mean return error than JFM's. We prove that JFM's energy targets the conservative component of frictional joint loads; the non-conservative remainder sets a floor on JFM's loss, and a free vector can also fit it. JFM keeps its advantage over the baseline when the energy gradient fed to the critic is zeroed at evaluation. JFM's gain in value estimates therefore comes from requiring the joint loads to be the gradient of one energy computed from the critic's representation.

Primary area:
reinforcement learning
Keywords:
physics-informed representation learning, dexterous grasping, offline reinforcement learning, auxiliary objectives, value estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2juzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Joint-Force Matching (JFM), an auxiliary objective that uses contact mechanics to improve RL critic representations for dexterous grasping. JFM maps measured contact forces to joint loads through the hand’s kinematic Jacobians and trains a scalar energy function whose negative joint-angle gradient matches these loads. By sharing a backbone with the critic, this objective encourages mechanically informed features while retaining standard force-prediction and RL losses. Experiments on simulated LEAP-hand grasping show improved agreement with demonstration returns compared with force prediction alone, while a direct joint-load predictor achieves lower auxiliary loss but worse mean critic return error. Policy improvements are less conclusive: matched offline policies perform similarly, whereas a separate offline-to-online setup shows greater grasp coverage for JFM.

Critical strengths
  • The idea of connecting contact forces with joint loads and use the signal to shape energy head is very interesting.
  • The free-vector ablation is helpful. A vector head trained on same joint-load targets fits more accurately, yet produces worse mean critic return error than JFM. This supports the benefit of the energy-gradient constraint.
  • Across all three training seeds, JFM’s value estimates are closer to the recorded demonstration returns than those of the force-prediction baseline.
  • JFM retains its critic advantage when the predicted joint-load input is removed at evaluation, suggesting that the benefit is retained in the learned model rather than depending solely on that extra input.
Critical weaknesses
  • Limited policy improvement under matched training. JFM covers 149 of 158 initial conditions versus 148 for the force-prediction baseline, providing little evidence that better critic estimates translate into better grasping performance.
  • A comparison with a separate energy network, or a variant that prevents the energy loss from updating the shared backbone, would help establish whether shaping the critic’s representation is responsible for the gains.
  • Missing metric. An initial condition counts as solved if any of 32 attempts succeeds, potentially masking low per-attempt success. The choice of 32 attempts should be justified, and average success rates over attempts with confidence intervals should be reported.
  • Insufficient proof of generalizability. Experiments use one simulated hand, and training and validation contain similar demonstrations. Performance on unseen objects, other embodiments, and real hardware remains untested.
Questions for authors
  • Can you compare against a separate energy network or a variant that blocks the JFM loss from updating the shared backbone to isolate the benefit of representation shaping?
  • Why was coverage defined using 32 attempts per initial condition? Can you report aggregated success rates for each initial condition with confidence intervals too?
  • Can you provide evaluation on held-out objects, other hand embodiments, and on real world hardware?
1 comment
AuthorsOpenReview ↗

Reply to Reviewer 2juz: Higher Grasp Success, Gains from the Shared Representation, and Generalization

We thank Reviewer 2juz for recognizing JFM's value, including its energy-gradient constraint and its value estimates, which are significantly closer to the demonstrated returns than those of FP (force prediction).

  1. Limited policy improvement under matched training:

little evidence that better value estimates translate into better grasping.

JFM's policy does grasp more often than FP's under matched training: over three training seeds, its success rate (SR) per attempt is 55.0% against FP's 52.5% at H=50 (p=0.005; H: policy steps before the object's support is removed). Table 3 notes that JFM's and FP's policies cover 149 and 148 of the 158 initial conditions at H=100, near the ceiling of the coverage test. We additionally evaluated the SR per attempt, which separates them: all 32 attempts on each initial condition count, for all three training seeds (%, mean ± std [95% CI over initial conditions]):

HJFMFPFV (free-vector)
10062.2±2.6 [57.1, 67.4]62.1±1.1 [57.0, 67.1]59.9±6.0 [54.7, 65.1]
5055.0±3.2 [49.1, 60.7]52.5±5.0 [46.9, 57.9]53.7±3.7 [48.1, 59.1]

JFM is the highest at both horizons. Paired on the same initial conditions and seeds, it is above FP at H=50 (p=0.005) and above FV at H=100 (p=0.012). The method with the better value estimates is also the one that grasps more often.

  1. Unclear whether shaping the critic's representation is responsible for the gain:

lack of a comparison separating the JFM loss on the shared backbone from the energy network.

Shaping the critic's representation is responsible for the gain: in the comparison that separates the two, JFM's gain over FP disappears once the JFM loss is blocked from the shared backbone. Sec. 5.4 (Table 4) zeroed the predicted joint-load input of JFM's critic at evaluation: its return error (distance between value estimates and demonstrated returns; lower is better) stayed below FP's at all three seeds (6.30/7.55/7.37 against 6.58/7.76/7.48), so the gain lies in the trained critic. We additionally ran the comparison: we blocked the JFM loss from the shared backbone, which produces the critic's representation, so the loss still trains the energy network, which fits the joint loads (matching loss 1.4–1.6×10⁻³; JFM 1.1–1.3×10⁻³), but no longer shapes that representation; all else is unchanged. Return error on Table 1's 8,000 validation samples:

SeedJFMFPJFM loss blocked from the backbone
426.316.587.96
437.517.7688.41
447.337.487.63

Without that shaping, the return error is the highest at every seed, and its excess over JFM's has a 95% CI excluding zero: the gain comes from that shaping. At seed 43 the blocked critic diverges on 201 samples; on the other 7,799 its error is 8.32 (JFM 7.60, FP 7.84).

  1. Missing metric:

coverage may mask a low SR per attempt;

no average SR, overall or for each initial condition, with confidence intervals.

The SR per attempt behind the coverage is 52–62% for every method at both horizons (table under item 1, with 95% CIs). For each of the 158 initial conditions (96 attempts each, Wilson 95% interval within ±10 points), the median SR is 74.5/69.8/69.8% (JFM/FP/FV) at H=100 and 61.5/54.2/60.4% at H=50, highest for JFM at both horizons.

  1. Why 32 attempts per initial condition?

32 attempts make coverage a reliable test: an initial condition from which a policy succeeds in 9% or more of its attempts is counted as covered with probability ≥ 95% (1 − 0.91^32). It is also the number of parallel environments in our simulation (App. B).

  1. Insufficient proof of generalizability:

one simulated hand;

no unseen objects, other hands or real hardware.

These points ask how far JFM extends beyond the tested hand and objects, a question for future work; they should not be grounds for dismissing JFM's value. The paper asks what in contact forces improves the critic's value estimates (Sec. 1) and answers by changing the supervision on the same hand, 71 objects and data (Sec. 5.2–5.4). For this question, simulation is the clearer design: it gives exact forces, Jacobians and returns, so differences between methods trace to the supervision; IQL (ICLR 2022) and SPR (ICLR 2021) are likewise evaluated in simulation. The result holds across 71 objects; unseen ones test transfer, a separate question from which supervision improves the critic. JFM is not tied to this hand: it maps contact forces to the joints through the hand's Jacobians, a construction that applies to any articulated hand (Sec. 3.1, 6). Each of these extensions builds on the supervision this paper identifies.

  1. Training and validation contain similar demonstrations.

This similarity does not explain JFM's advantage: each validation demonstration is held out from training, one per combination of object, pose and variant (Sec. 5.1), and all methods share this split, so similarity affects them equally.

Reviewer_EpLnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Joint-Force Matching (JFM), which uses contact forces and hand Jacobians to train a neural energy function. The learned energy gradient is used to improve RL critics for robotic grasping.

Critical strengths

Using neural physics to improve RL is good to explore.

Critical weaknesses
  • Baselines. The paper compares against FP and FV. It would be helpful to include comparisons with existing methods, especially prior work on neural dynamics and physics-based RL.
  • Evaluation. The paper mainly focuses on return error and return rank. I would prefer to see standard grasping success rates, as the current coverage metric counts success if any of 32 attempts succeeds. The main offline comparison also shows a small improvement (149/158 vs. 148/158).
  • Real-world validation. The experiments are limited to simulation. Real-world grasping experiments would help demonstrate the practical value of the method.
  • Presentation. I found the paper difficult to follow, especially the motivation and main method. A simpler explanation with more intuition would make the contribution clearer.
Questions for authors
  • Can the authors compare against existing published methods?
  • Can the authors report standard per-attempt success rates and provide real-world evaluation?
  • Why does the offline-to-online comparison use different pretraining budgets for JFM and FP? A comparison with matched budgets would be more convincing.
1 comment

We thank Reviewer EpLn for recognizing that using neural physics to improve RL is a direction worth exploring. We believe there are misunderstandings of the paper, which we hope this reply resolves: the paper already reports the matched-budget comparison (Evaluation), explains its motivation and method (Presentation), and uses the objective of published methods as its baseline (Baselines). Our additional evaluation over three seeds further confirms JFM's effectiveness: its policy grasps more often than FP's.

Evaluation:

standard per-attempt success rates are missing, and JFM's gain over FP (force prediction) in coverage is small;

the offline-to-online comparison uses different pretraining budgets;

the paper mainly reports return error and return rank.

Under matched budgets, JFM's per-attempt success rate is above FP's over three training seeds: 55.0% against 52.5% at H=50 (p=0.005; H: policy steps before support removal). Sec. 5.1 trained JFM, FP and FV (free-vector) at three pretraining seeds with the same network, data, objectives and budgets. Table 3 notes that their policies cover 149, 148 and 144 of 158 initial conditions at H=100, near the test's ceiling. We additionally evaluated all three seeds' policies and counted every attempt, 32 per initial condition (%, mean ± std):

HJFMFPFV
10062.2±2.662.1±1.159.9±6.0
5055.0±3.252.5±5.053.7±3.7

JFM is highest at both horizons. Sec. 5.5 states the differing budgets of its offline-to-online pair; the matched comparison is the main study above. Return error and return rank are the main measures because the paper asks what in contact forces improves the critic's value estimates (Sec. 1). Return error is the distance between the critic's values and the demonstrated returns, and return rank their rank correlation (Sec. 5.1). Table 1 (Sec. 5.2) showed that JFM's return error is lower and its return rank higher than FP's at every seed (p=0.027 and p=0.008).

Presentation:

the motivation and main method are difficult to follow; a simpler explanation with more intuition is needed.

Sec. 1 and Fig. 1 explained the motivation and the method, and the review's own summary states the method correctly. The motivation: the critic estimates the return of each action, and contact forces add a dense signal to the one scalar per step that the task reward gives it. Existing methods treat each contact force as a separate measurement and leave out how the forces load the hand's joints.

The method: through the hand's kinematics, each contact force loads the joints between its contact site and the palm. For elastic contact, one energy produces all these joint loads, as the energy stored in a stretched spring produces its force. JFM computes the joint loads from the measured forces through the hand's Jacobians, computes a scalar energy from the critic's representation, and trains the energy's negative gradient with respect to the joint angles to match these loads (Fig. 1c).

JFM thus supervises how the representation changes with the joint angles, and the critic reads the same representation. Whether a grasp holds depends on how the contact loads respond to joint motion, so this supervision can help the critic separate grasps that hold from grasps that slip.

Baselines:

comparisons with existing published methods, especially neural dynamics and physics-based RL, are missing.

The comparison with published methods is in the paper: FP is the force-prediction objective of published methods, and all compared methods share one published learner, IQL (ICLR 2022). Sec. 2 reviewed predictive and world models that learn to forecast forces or touch; FP is this objective on the same network, data and budgets as JFM, so their difference traces to JFM's supervision. JFM also predicts the contact forces more accurately than FP at every seed (Table 1). Neural-dynamics and physics-based RL methods learn equations of motion or differentiate a simulator. Comparing them with JFM changes the learner and the model class and asks which physics prior is better, a separate study; JFM is an auxiliary loss that can be added to the critic of such learners.

Real-world validation:

real-world grasping experiments are missing.

We agree that real-world grasping experiments would demonstrate JFM's practical value. This paper studies theory and mechanism; for its question, simulation is the clearer design: it gives exact forces, Jacobians and returns, so method differences trace to the supervision. IQL and SPR (ICLR 2021) are likewise evaluated in simulation. On a physical hand, tactile noise, calibration and actuation errors mix with the supervision and confound this analysis, so real-world validation deserves a dedicated study. The paper prepares for it: no policy receives the measured forces at evaluation (App. B), so demonstrations recorded with tactile gloves could train critics for hands without force sensing (Sec. 6).

Reviewer_hcANOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Joint-Force Matching (JFM), a physics-informed auxiliary supervision method for reinforcement learning critics in contact-rich manipulation. The key idea is to map measured contact forces into joint loads through the robot's Jacobians and train the gradient of a scalar energy function to match these loads. By sharing the backbone between the energy model and the critic, JFM aims to learn more physically meaningful representations and improve value estimation. Experiments on simulated dexterous grasping demonstrate improvements in critic accuracy and closed-loop grasping performance compared with force-prediction baselines.

Critical strengths

S1. Physics-informed representation learning. The paper introduces a meaningful connection between contact mechanics and critic representation learning. Rather than directly predicting contact forces, JFM imposes an energy-gradient constraint on joint loads, providing a physically motivated inductive bias.

S2. Solid theoretical formulation. The paper provides a theoretical analysis of the decomposition of joint loads into conservative and non-conservative components, clarifying what can be represented by the proposed scalar energy function.

S3. Promising experimental results. JFM consistently improves critic return estimation across the three main-study pretraining seeds and achieves better coverage in the offline-to-online closed-loop grasping experiment.

Critical weaknesses

W1. Lack of real-world experiments. All experiments are conducted in simulation, without validation on a physical robot. Since the proposed method relies heavily on contact mechanics and accurate contact information, its robustness to real-world sensor noise, contact uncertainty, and modeling errors remains unclear. This limits the evidence for its practical applicability.

W2. Insufficiently controlled closed-loop evaluation. In the main offline evaluation, JFM shows only a marginal improvement over the force-prediction baseline (149/158 vs. 148/158 trajectories covered). Although the offline-to-online evaluation demonstrates a larger gain (33/158 vs. 18/158), the methods use different pretraining budgets and some different model configurations. Moreover, the reported closed-loop results lack multiple independent training seeds, making it difficult to confidently attribute the improvement to JFM.

W3. Limited validation of the conservative-force assumption. JFM explicitly models the conservative component of joint loads, while real-world manipulation frequently involves friction, sliding, and contact transitions that introduce non-conservative dynamics. The paper does not sufficiently investigate how these effects influence the learned representation or downstream performance.

Questions for authors

Q1. Real-world validation: Could the authors provide grasping experiments on a physical LEAP Hand to demonstrate the practical effectiveness of JFM? If real-world experiments are infeasible, evaluating robustness to contact-force noise, contact-location errors, and variations in friction would strengthen the paper.

Q2. Fair closed-loop comparison: Could the authors evaluate JFM, FP, and FV under identical training budgets and model configurations, with multiple random seeds? Reporting per-attempt success rates in addition to trajectory coverage would also provide a clearer assessment of policy reliability.

Q3. Non-conservative contact dynamics: How does JFM perform under varying friction coefficients, sliding contacts, and stick-slip transitions? It would be valuable to analyze whether the magnitude of the non-conservative residual correlates with critic accuracy and grasping performance.

Q4. Representation learning mechanism: Could the authors provide additional ablations, such as stopping the JFM gradients from updating the shared backbone or comparing against a curl-regularized free-vector predictor, to more directly demonstrate that the improvement comes from the proposed energy-gradient constraint?

1 comment
AuthorsOpenReview ↗

Reply to Reviewer hcAN: Higher Grasp Success under Matched Training, the Source of the Gain, and Frictional Contact

We thank Reviewer hcAN for recognizing JFM's physically motivated energy-gradient constraint, solid theoretical formulation and consistent improvement of the critic's return estimates across three pretraining seeds.

W2, Q2. Insufficiently controlled closed-loop evaluation; fair closed-loop comparison:

marginal improvement over FP (force prediction) in the main evaluation;

different budgets and configurations in the offline-to-online evaluation;

no multiple training seeds or per-attempt success rates.

Under identical budgets and configurations, JFM's policy grasps more often than FP's over three training seeds: its success rate per attempt is 55.0% against FP's 52.5% at H=50 (p=0.005; H: policy steps before support removal). Sec. 5.1 trained JFM, FP and FV (free-vector) at three pretraining seeds with the same network, data, objectives and budgets. Table 3 notes that their policies cover 149, 148 and 144 of 158 initial conditions at H=100, near the test's ceiling. We additionally evaluated all three seeds' policies and counted every attempt, 32 per initial condition (%, mean ± std):

HJFMFPFV
10062.2±2.662.1±1.159.9±6.0
5055.0±3.252.5±5.053.7±3.7

JFM is highest at both horizons. Paired by initial condition and seed, it is above FP at H=50 (p=0.005) and above FV at H=100 (p=0.012). Sec. 5.5 states the differing budgets of its offline-to-online pair; the matched comparison is the main study above.

Q4. Representation learning mechanism:

does the gain come from the energy-gradient constraint? (stop-gradient ablation; curl-regularized FV)

The gain comes from the energy-gradient constraint, which acts through the shared backbone. Sec. 5.3 (Table 1) removed only the constraint: FV regresses the same joint loads with a vector head of matched capacity and the same routing to the critic. FV fits the loads four to five times more closely, yet its critics' mean return error (distance of value estimates from demonstrated returns) is above JFM's by 0.55 [0.22, 1.02] on 8,000 validation samples. We additionally ran the requested ablation, stopping the JFM gradients at the shared backbone; the energy network still fits the joint loads (matching loss 1.4–1.6×10⁻³; JFM 1.1–1.3×10⁻³). Return error on the same samples:

SeedJFMFPJFM gradients stopped
426.316.587.96
437.517.7688.41
447.337.487.63

Without these gradients, the return error is highest at every seed, and its excess over JFM's has a 95% CI excluding zero: the constraint improves the critic by shaping the shared representation. At seed 43 it diverges on 201 samples; on the rest its error is 8.32 (JFM 7.60). A curl penalty is a soft form of JFM's constraint: a curl-free field over joint angles is the gradient of a scalar, which JFM's energy is by construction; Table 1 compares the two extremes, FV (no constraint) and JFM.

W3, Q3. Limited validation of the conservative-force assumption:

how friction, sliding, contact transitions and the non-conservative residual influence the critic and grasping.

Proposition 1 (Sec. 4) analyzed frictional contact: the joint loads split into a conservative component, which JFM's energy targets, and a non-conservative remainder, which adds a constant to JFM's loss and leaves its gradient unchanged. Sec. 5.3 measured the remainder's effect: FV also fits the remainder, yet its critic is less accurate (Q4) and its policy grasps less often than JFM's at H=100 (W2). Fitting the non-conservative part thus helps neither. The demonstrations are frictional grasps in which contacts form and break. Proposition 1 holds at any friction coefficient, in sticking and in sliding; sweeping these conditions, or the remainder's size per state, is follow-up work built on this result.

W1, Q1. Lack of real-world experiments:

no experiments on a physical LEAP Hand;

unclear robustness to sensor noise and contact-location errors.

We agree that validation on a physical LEAP Hand under sensor noise and contact errors matters for JFM's practical use. This paper studies theory and mechanism: what in contact forces improves the critic's value estimates. For this question, simulation is the clearer design: it gives exact forces, Jacobians and returns, so method differences trace to the supervision. IQL (ICLR 2022) and SPR (ICLR 2021) are likewise evaluated in simulation. On hardware, tactile noise, calibration and actuation errors mix with the supervision and confound this analysis, so real-world validation deserves a dedicated study. Measured forces are JFM's training targets: no policy receives them at evaluation (App. B), so sensor noise never enters its inputs. Zero-mean noise in the force labels adds a constant to JFM's loss, like the remainder in Proposition 1, and leaves its gradient unchanged. Contacts are read at 17 sites fixed on the hand, so their locations come from its kinematics.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Authors' Rebuttal for the Chairs

We thank everyone involved in this discussion, including the AC, SAC and PC, and the three reviewers for their questions. All three reviews recognize the idea: JFM draws "a meaningful connection between contact mechanics and critic representation learning" (hcAN), connecting contact forces with joint loads is "very interesting" (2juz), and using neural physics to improve RL is "good to explore" (EpLn). Below we summarize how each concern is addressed through the paper itself and supplementary tests.

Most concerns ask for what the submission already contains. The choice of return error and return rank, the inputs the policy receives at evaluation, the matched network, data, objectives and budgets of the compared methods (Sec. 5.1, App. B), the analysis of frictional contact (Sec. 4, Proposition 1) and the budgets of the offline-to-online pair (Sec. 5.5) are all in the paper. Our replies pointed to each location and explained it again in plain terms.

The concern that better value estimates may not yield more grasps is settled by the per-attempt success rates all three reviewers asked for. Over three training seeds under matched budgets and configurations, JFM's policy succeeds on 55.0% of its attempts against force prediction's 52.5% at H=50 (p=0.005), and it is the highest of the three methods at both horizons. The method with the better value estimates also grasps more often.

The question of where the gain comes from is answered by the paper's controls and the requested ablation. The free-vector baseline (FV), identical to JFM except that it drops the energy-gradient constraint, fits the joint loads four to five times more closely, yet its critic's return error is above JFM's by 0.55 [0.22, 1.02] (Sec. 5.3). Zeroing the predicted load at evaluation leaves JFM's advantage in place (Sec. 5.4), and blocking the JFM loss from the shared backbone raises the return error above JFM's at every seed. The gain comes from shaping the representation the critic reads, as the paper states.

The remaining requests are beyond this paper's scope. A physical hand, other embodiments, unseen objects, neural-dynamics learners and friction sweeps extend JFM to new settings and do not bear on whether its claims hold. For the paper's question, simulation is the clearer design: exact forces, Jacobians and returns let differences between methods trace to the supervision. IQL (ICLR 2022) and SPR (ICLR 2021) are likewise evaluated in simulation. No concern identified an error in any result or a claim without evidence.

How the discussion ended. Although none of the three reviewers responded to our replies or asked a follow-up question, we have tried our best to answer the questions the reviewers raised. The review by EpLn summarizes our method correctly and names no specific concern about the paper's reliability; we hope the paper and our reply clear up the misunderstandings there. If any reviewer still sees a substantive problem that warrants rejection, we would welcome having it pointed out.

In sum, every concern in the reviews is answered in the submission, settled by the evidence above, or beyond the paper's scope, and none was left unanswered in the discussion. We hope the AC, SAC and PC will weigh the reviews in this light, together with the paper's theory and its experiments over three pretraining seeds, 71 objects and 158 held-out initial conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.